LLM Observability: Instrument, Score, Contain

LLM observability records how a model or agent produced an answer (prompts, retrieval, tool calls, tokens, cost, latency, quality scores) so you can explain a bad output. IBM Think treats it as real-time behavioral data. Datadog notes failures often surface as wrong answers, not errors.

Gartner said on 30 March 2026 that explainable AI will drive these investments to 50% of GenAI deployments by 2028, up from 15% then. Ranking pages still stop at glossaries and tool lists.

Instrument traces, score quality online, alert on SLOs, then contain without waiting on a deploy.

Key Takeaways

• A healthy HTTP 200 can still be a wrong answer, a truncated completion, or a tool loop. Traces plus quality scores are how you see it.
• OpenTelemetry GenAI conventions are still Development as of 17 July 2026. Pin a generation and inspect what your SDK actually emits.
• Treat online evals as quality SLIs. Do not run an LLM-as-judge on every request.
• Drift has two layers: live eval-score drop, and a golden set that no longer matches production.
• Containment is a prompt-version rollback, a feature-flag kill switch, or a cheaper fallback. It is not a 30-minute deploy revert.

What Is LLM Observability?

Classic monitoring asks whether the service is up and within budget: latency, error rate, token usage, cost, throughput.

Observability asks whether you can explain this bad answer from a trace. Reconstruct the prompt, retrieved chunks, tool calls, and the eval score that moved. braintrust.dev draws that split: monitoring tracks whether the system is within limits; observability reconstructs how one output was produced.

CPU and memory can look fine while outputs go weird. Record individual model calls, tool executions, and the data between them so you can explain one output and score thousands of runs you will never read.

Map HTTP-era signals onto LLM-era ones before you buy another dashboard.

An AI gateway can stamp identity and route keys. It does not replace the span tree. Gateway logs start at the proxy hop, so they miss the tool argument that went wrong three steps earlier.

Why It Matters in 2026

Gartner also projected the GenAI models market above $25 billion in 2026 and $75 billion by 2029. Pankaj Prasad's line in that release is the operating shift: traditional observability optimized for speed and cost; priority is moving toward factual accuracy, logical correctness, and sycophancy.

Truncations still look like HTTP 200s if you never record finish_reason.

Hamel Husain's October 2024 consulting split is still the 2026 complaint. Treat it as an estimate, not a survey.

On r/mlops, the recurring failure is features that demo, ship, then degrade for weeks with nobody watching quality. u/tehlucaa (June 2026) described the model as "doing something slightly wrong some percentage of the time."

How It Works: The Production Loop

braintrust.dev operationalizes the sequence most teams skip: instrument tracing across retrieval, prompts, model calls, and tools, then add evals, then monitor quality, cost, and latency so a score drop can page you.

Instrument the Span Tree

Export spans off the request path. Default metadata (model, tokens, duration) is enough for cost and latency SLOs.

Turn on content (gen_ai.input.messages, output.messages, system_instructions) only when you need to debug a hallucination. Prompts carry PII. On r/LangChain, u/BeatTheMarket30 (March 2026) called sending customer chat to hosted LangSmith unimaginable without encryption.

Start retention high enough that you can explain a complaint you cannot predict. Marc Klingen landed on traces first because evals that look good still leave users having a bad time when the test set never matched live intent.

Score Quality Online

Attach scores to live traces. Langfuse supports boolean scores, so a hallucination-check failure rate is just the share of true. OpenTelemetry v1.38 added gen_ai.evaluation.result as the vendor-neutral attach point (name, score, label, explanation).

Do not judge every request. Towards AI sequences deterministic checks first, a risk band second, an LLM-as-judge on the uncertain slice, and humans last. Comet samples 1-5% of production traces through a judge.

Dat Ngo (Arize, AI Engineer) put the cost constraint in one line: you pay for every eval you run.

"Just because you can eval something doesn't mean you always should." (Dat Ngo, AI Engineer, 11:52)

Jason Lopatecki on the same Mastra panel: adding one line of prompt instruction can break about 15 other behaviors. Offline evals are unit tests for that class of regression.

Arize's eval guide (updated 31 August 2026) uses the same evaluator before release and after, on sampled production. Separate retrieval scores from generation scores so a RAG miss is not blamed on the model.

Set Cost and Latency SLOs

Page on p95/p99, not p50. LLM latency is fat-tailed. Record TTFT, time-per-output-token, per-step duration, and total loop time.

Cost is not the number in the provider console. Compute it on every span against a versioned price list, including cache reads and writes as separate lines. If that table is stale, the dashboard lies.

Aparna Dhinakaran (July 2025) reported the session-level pattern: long sessions led to higher costs, slower completions, and more drift.

Contain, Then Diagnose

Tianpan notes a standard deploy rollback can take about 30 minutes while a misbehaving model ships bad outputs in seconds. FutureAGI calls prompt rollback the common remedy: a one-click revert that ships in seconds.

LaunchDarkly versions prompts as AI Configs so you can compare token usage, cost, latency, and quality per variation, then roll back.

Failing traces become eval-set entries. Towards AI sequences the same close: promote confirmed production failures back into offline tests.

What to Trace in an LLM Request

Google's head results name traces. They do not teach the tree.

The May 2026 OTel post opens with the on-call question: your agent took 45 seconds; was it the model, a slow tool, or a retry loop? Default export is metadata only. Content is opt-in.

The span tree you want looks like this:

root / invoke_agent
 ├─ chat {model}            # messages, tokens, finish reason, duration, TTFT
 ├─ execute_tool {name}     # args, result, error, duration
 │   └─ http / auth / retry # not in GenAI conventions; add or you mis-blame the prompt
 ├─ retrieval / memory
 └─ chat {model}            # second pass after tools

v1.37.0 span names SHOULD be {gen_ai.operation.name} {gen_ai.request.model}. v1.37 inference operations include chat, generate_content, and text_completion, plus a separate execute_tool span. Later conventions add invoke_agent (Hodge: v1.41 splits it into client vs internal).

Status on essentially all gen_ai.* fields: Development. Shared server.address is Stable.

John Hodge dated the snapshot on 17 July 2026: no GenAI-specific span, event, metric, or attribute is Stable. Conventions moved to semantic-conventions-genai.

Core semconv v1.42.0 (12 June 2026) deprecated gen_ai.*. v1.43.0 (3 July 2026) ships none.

"OpenTelemetry-compatible" is not a schema contract. Frameworks emit mixed generations. Pin, inspect, normalize.

Appropri8 (January 2026): a single agent run is often 5-10 LLM calls plus 15-20 tool calls. Logs show events. They do not show which LLM call caused which tool timeout.

Nango (21 July 2026): GenAI conventions capture the model call and execute_tool name, duration, and error. They do not capture auth, token refresh, retries, rate limits, or incoming webhooks. Add those spans or you will "fix the prompt" for a 429.

Braintrust tracing (20 July 2026): an LLM app can return a successful response even if retrieval, tool calls, or model steps failed.

MCP tool servers sit on that execute_tool branch. If you only log the final model message, you will miss a mangled argument to a correct tool.

Production Metrics That Should Page You

Merge vendor lists. Do not copy one "7 key metrics" post into your SLO table. Skip training-era scores such as perplexity.

Quality SLOs, alert routing, runbooks, on-call ownership, and per-tenant attribution are why you instrument. Ranking pages skip them.

Cost Attribution in Three Layers

AWS Cloud Financial Management plus TrueFoundry and Traceloop converge on the same three layers.

Billing is the provider console or CUR. Exact, and too coarse: one number per API key. TrueFoundry's worked example is a single Anthropic row on a shared key.

Treat the shared-key dollar figure as illustrative, not a benchmark.

Telemetry is tokens on every span (gen_ai.usage.*). Multiply against a versioned price list. Cache reads, cache writes, audio, and image are separate line items.

Attribution tags user, feature, tenant, and cost_center before provider selection, at the gateway. High-cardinality stays on the trace. Low-cardinality projects to metrics.

AWS's finance questions are the right ones: which team drives tokens, whether the right model is on the task, how well prompt caching works, how much spend dies on failed requests, and cost per session. AWS claims five levers can cut spend 30-50% depending on workload. Treat that as AWS guidance, not an industry average.

On r/LLMDevs, the useful metric is saved cost with no task-quality regression. Tag each call with a reason code (duplicate, over-modelled, context bloat, retry waste, loop waste, frontier-required) and sample by cost bucket.

SKU numbers age fast. As of 5 September 2026, Langfuse Cloud Core is $29/mo and LangSmith Plus is $39/seat/mo. Self-host Langfuse under MIT is $0.

The Langfuse and LangSmith figures are operational context, not a buying guide.

Two Kinds of Drift

Vendor copy collapses "drift" into one word. On-call feels two different failures.

Production Eval-Score Drift

Production eval-score drift is the rolling-mean rubric drop on live traffic. Tianpan notes a quality-score drop is often the first regression that never appears in infra metrics. Token-cost-per-request often fires first: a verbose prompt, a retry loop, or retrieved-context bloat.

One worked incident: p99 latency hit 12 seconds because a prompt change unlocked verbose behavior. The service was up. The model was misbehaving.

W&B (3 June 2026) published the counterexample that should live on your runbook wall. Retrieval drift lowered cost and latency while quality died.

The degraded scenario ingested 28% more input tokens, yet total cost fell (9.318¢ vs 9.492¢) and latency fell (8.2s vs 9.8s) because the model wrote a shorter, worse answer. Early warning was the retrieval span: an outdated doc scored 0.631.

Aggregate metrics lie. The component trace tells you where.

Eval-Dataset Drift

Eval-dataset drift is why CI stays green while production is wrong. FutureAGI (May 2026) splits three dataset moves.

A Jaccard index on top-k overlap is a cheap corpus check. Cadence: monthly by default, weekly under fast change, and immediately after a vendor model bump or corpus re-index.

Treat golden-v3-2026-05 as a versioned artifact. W&B's reminder for API systems: providers might update models without announcement. Pin versions; floating aliases are a liability.

Alerts and Incident Response

Ranking pages skip what to page on and how to contain a silent quality incident.

What to Alert On

• Eval-score or groundedness drop (quality SLI, including boolean failure rate)
• Cost spike or tokens-per-request doubling
• Latency SLO, TTFT, or per-step duration
• Tool-error rate and retry loops
• finish_reason anomalies (length truncations still look like HTTP 200s)
• Safety, prompt-injection, and PII guardrail hits
• Step-duration anomaly: an agent stuck eight minutes on a node that is usually 20 seconds

Step-duration is the signal practitioners keep asking vendors for. u/ultrathink-art in r/LangChain (June 2026): traces are great retrospectively, and nothing proactively surfaced the hung step.

Do not page an LLM-as-judge to decide whether to page. The Mastra panel rule: if you have a deterministic runtime expectation (checkout happened, JSON extracted), throw an exception and hook PagerDuty. Quality-of-poem failures should not wake anyone.

Copy a SEV shape from your own baselines, not from a generic runbook's sample multipliers.

Four Incident Classes

Failure is distributional. A slice of inputs goes wrong. The same prompt can be correct at 9am and hallucinated at 9:15, the reproduction gap Tianpan flags for on-call.

Root cause is a research question: yesterday's prompt edit, a silent provider model bump, or this morning's re-index. Classify before you "fix."

Agent Traces vs Single-Call Traces

Last-call logs are not a session. Nested agents need a different trace unit than a single chat completion.

Datadog on LinkedIn (September 2026): application logs may capture an agent's final API call without showing which prompt, retrieved content, or tool result led to the action.

Traceloop (February 2026): provider-native tracing only sees the model side. It does not see your tools, your business logic, or the path between the app and the model.

Nested invoke_agent / execute_tool is the unit. Langfuse said a run that was twelve spans last year is six hundred today.

Trajectory evals ask whether the path was right, not only whether the last message looks fine. Arize (August 2026): the final answer is only one part of an agent eval. Failure modes look like success.

One LinkedIn example: a single question took an agent 85 MCP turns through retrieval-only tools.

Langfuse also flags memory as the layer observability often cannot see. If you do not instrument memory ops, you will debug the model for a stale fact the agent fetched from its own store.

On r/LangChain, u/DripSkylarkII (August 2026) put the eval difference in one line: the final output can look fine while something three steps back was wrong.

AI harness tools multiply this fan-out. Session or thread is the unit of analysis, not a single span.

How to Choose a Stack

Do not rank a top 10. Pick a job.

Helicone joined Mintlify on 3 March 2026 and is in maintenance mode. Data stays accessible. Do not plan new proxy features there.

2026 changed who owns the category. ClickHouse acquired Langfuse, announced 16 January 2026, the same day ClickHouse posted a $400 million Series D.

Reuters put the round at a $15 billion valuation. MIT core and self-host had no planned licensing changes.

Cisco closed Galileo on 22 May 2026 (galileo.ai still carries the product). Traceloop joined ServiceNow in March 2026; OpenLLMetry remains Apache-2.0.

Datadog Agent Observability is the APM-adjacent SKU. As of 5 September 2026, that page lists a Free plan up to 40k LLM spans per month and Pro at $160/mo for the first 100k LLM spans.

Additional on-demand usage is billed after that. Datadog says it bills only LLM-provider calls; tool, workflow, agent, embedding, and retrieval spans are free on that page.

On Reddit, Langfuse is the default named OSS, and ClickHouse self-host gets ops-heavy. Phoenix is praised as "just the app and the db."

LangSmith is a non-starter where customer chat cannot leave the boundary. Datadog wins when the team already lives there.

Watch the Langfuse OTEL SDK. u/alxdan (March 2026) found it attaching to the global TracerProvider and uploading unrelated gen_ai.* spans from other tools in-process.

Marc Klingen uses Datadog for ops and still does not treat it as the LLM product. APM traces are ephemeral. The workflow is pour-in and troubleshoot timing, not replay this prompt with this tool payload.

Common Mistakes to Avoid

Reading the Last API Log as the Session

Provider logs show the model hop. Agents are a tree. You will miss the tool argument, the retrieval miss, and the nested invoke_agent.

Inspect the first diverging step, not the last message.

Alerting Only on HTTP Errors

Wrong answers return 200. Truncations return 200 with finish_reason=length. Page on eval-score SLIs, cost-per-request, and step duration.

Error rate is necessary and not sufficient.

Treating OpenTelemetry as a Finished Contract

Listicles call OTel "the standard." Hodge's 17 July 2026 snapshot still says Development, with gen_ai.* deprecated out of core semconv. Pin the generation your SDK emits. Filter infra spans or the LLM UI is noise.

Running a Judge on Every Request

Always-on LLM-as-judge is a cost bug. Deterministic checks first. Sample a slice of production traces for the expensive judge.

Yes/no questions beat 1-10 scores. Average repeated judge runs if you need stability. On r/mlops, LLM-as-judge without slice-level human anchors is "the judge grading the judge."

Rolling Back a Deploy for a Prompt Incident

A prompt-version flag ships in seconds. A deploy revert does not. Pin model versions and kill-switch the route.

Then ask whether yesterday's template, this morning's index, or a silent provider bump caused the slice to move.

When Cheaper Metrics Lie

Keep the W&B retrieval-drift case on the runbook wall. Cheaper, faster aggregates can still mean quality is dead.

Containment for that class is a corpus pin or a prompt-version rollback, not a Kubernetes revert.