LLM Observability

LLM observability is the practice of collecting, analyzing, and visualizing traces, metrics, and logs from language model applications to understand system behavior in production. It extends traditional observability principles to AI-specific signals. It tells you what your AI system is actually doing -- not just whether it returned a 200 status code, but whether the response was accurate, grounded, and cost-effective.

Why traditional observability is not enough

Traditional application monitoring tracks request latency, error rates, throughput, and resource utilization. These metrics tell you whether your system is up and responsive. They do not tell you whether your LLM application is producing good outputs.

An LLM endpoint can return a 200 with a perfectly formatted response that is entirely hallucinated. Traditional monitoring sees a healthy request. LLM observability sees a quality failure. The gap between "the system is running" and "the system is working correctly" is what LLM observability fills.

Core signals

Traces. A trace captures the full lifecycle of a request through your AI pipeline -- from the user query through retrieval, prompt assembly, model inference, and post-processing. Each step is a span with timing, inputs, outputs, and metadata. OpenTelemetry provides the industry-standard framework for structured tracing, and its semantic conventions for LLMs define how to instrument model calls. Traces let you see exactly what context the model received and what it produced. For a detailed breakdown, see LLM tracing.

Token metrics. Input tokens, output tokens, and total tokens per request. These directly determine cost and correlate with latency. Tracking token usage per pipeline, per model, and per user segment reveals optimization opportunities.

Quality scores. Per-request or sampled quality evaluations -- faithfulness, relevance, format compliance. When tracked over time, quality scores expose drift that latency and error rate metrics cannot detect.

Cost attribution. LLM API spend broken down by pipeline, model, customer segment, and time period. Production LLM costs can grow fast, and cost observability prevents surprise bills and enables informed model-routing decisions.

Error classification. Beyond HTTP errors, LLM-specific failures: rate limits, context window overflows, safety filter triggers, malformed outputs, and model refusals. Each failure type has a different root cause and remediation path.

Observability in practice

A typical LLM observability setup captures:

1. Every model call with its full prompt (or a hash of it), response, latency, token counts, and model version
2. Retrieval step details -- which documents were retrieved, their relevance scores, and how they were assembled into context
3. Quality evaluations on a sample of production traffic (100% is too expensive for most teams; 5-10% gives statistical coverage)
4. Alerts on latency spikes, error rate increases, quality score drops, and cost anomalies

This data feeds debugging workflows. When a user reports a bad answer, you trace the request ID back through the pipeline and see exactly where things went wrong -- bad retrieval, prompt assembly error, model hallucination, or post-processing bug.

Connection to the LLMOps stack

LLM observability is one pillar of the broader LLMOps practice. It works alongside evaluation (which defines what "quality" means), CI/CD (which gates deploys on quality), and agent observability (which extends tracing to multi-agent systems).

The LLM observability guide covers how to instrument your application, choose metrics, set up alerting, and select tooling. For teams evaluating platforms, the tools pricing comparison includes observability capabilities across major vendors. See also our Langfuse alternative and Arize alternative comparisons for observability-focused platforms.