Observability for LLM services

You cannot operate what you cannot see. LLM services need the usual three pillars (metrics, logs, traces) plus a few LLM-specific measurements, because the failure modes are different: a service can be fast, healthy and wrong; or slow only for long prompts; or saturated in memory while the CPU is idle.

#The metrics that matter

User-facing (alert on these, tie them to SLOs):

• TTFT histogram (p50, p95, p99)
• Inter-token latency or tokens/s per request
• Request rate, error rate, timeout rate
• Requests finished by reason: stop, length (hit max tokens), error, cancelled

Capacity (explain why latency moved):

• Requests running and waiting per replica
• KV-cache utilisation (%)
• Prompt tokens/s and generation tokens/s
• Prefix-cache hit rate
• Batch size and preemption or swap counts
• GPU memory, utilisation, temperature, power and error counters

Business:

• Tokens in and out per customer or feature, and cost per million tokens
• Cost per resolved task, where you can define "resolved"

#Where to get them

• vLLM exposes Prometheus metrics at /metrics (queue length, running requests, cache usage, TTFT and per-token latency histograms). Exact metric names change between versions, so open /metrics on your version and note them.
• llama.cpp server exposes Prometheus metrics at /metrics when started with --metrics.
• GPU metrics come from the NVIDIA DCGM exporter (or nvidia-smi for quick checks).
• Gateway metrics give end-to-end latency as users experience it, including queueing before the engine (gateways).
• Application metrics for retrieval (no-result rate, similarity scores) and tool calls (error rate, latency).

#Histograms, percentiles and why averages lie

Suppose 100 requests: 95 take 0.5 s and 5 take 20 s. The mean is 0.95 × 0.5 + 0.05 × 20 = 1.5 s. It looks acceptable, yet 5 percent of users waited 20 seconds. The p95 is about 0.5 s (borderline) and the p99 is 20 s: the percentiles show the problem and the mean does not.

Record latency as a histogram with buckets suited to your SLO (for TTFT, for example 0.1, 0.25, 0.5, 1, 2, 5, 10, 30 s). Histograms from many replicas can be summed and then queried for percentiles; averages of per-replica percentiles are wrong. A typical PromQL query (adapt the metric name to your engine):

histogram_quantile(0.95, sum by (le) (rate(ttft_seconds_bucket[5m])))

#A starter Prometheus config

scrape_configs:
  - job_name: vllm
    scrape_interval: 15s
    static_configs:
      - targets: ["vllm-1:8000", "vllm-2:8000"]
  - job_name: gpu
    static_configs:
      - targets: ["dcgm-exporter:9400"]
  - job_name: gateway
    static_configs:
      - targets: ["gateway:9100"]

#Alerts worth having

Alert on symptoms (users hurting) and use capacity metrics for diagnosis. Alerting on every cause creates noise that teams learn to ignore.

#Burn-rate alerts

A better way to alert on an SLO than a fixed threshold is the burn rate: how fast you are consuming the error budget.

budget      = 1 − SLO                       (99.5% SLO → 0.5% budget)
burn rate   = observed error rate / budget

If the error rate is 7.2 percent and the budget is 0.5 percent, the burn rate is 14.4. At 14.4 you would consume 14.4 × (1 hour / 720 hours) = 2% of the 30-day budget in one hour: a good "page now" threshold. A slower leak of 6 times burn consumes 5 percent in 6 hours: a "page soon" level. A burn of 1 consumes exactly the whole budget over 30 days.

A common pattern pairs a long and a short window to avoid noise and to clear quickly:

You can express "error" as failed requests, requests slower than the TTFT threshold, or both, so a latency SLO uses the same machinery.

#Structured request records

Log a request record per call, as JSON, with a request ID that follows the request through every hop:

{
  "ts": "2026-11-11T09:14:22.381Z",
  "request_id": "7b1f...",
  "route": "/v1/chat/completions",
  "model": "support-8b", "release": "support-bot-2026-11-03",
  "key_id": "k_42", "tenant": "acme",
  "prompt_tokens": 1810, "completion_tokens": 212,
  "queue_ms": 35, "ttft_ms": 410, "decode_ms": 2380, "total_ms": 2825,
  "finish_reason": "stop", "status": 200,
  "cache_hit_tokens": 1500, "retries": 0
}

This one table answers most operational questions: who is using what, which release is slow, whether caching is working, which customers drive cost. Log the release ID with every request so a regression lines up with a deploy (rollouts).

Be deliberate about prompt and response text: it can contain personal or confidential data. Either do not log it, sample it, redact it, or protect it with strict retention and access rules (PII, data leakage). Metadata-only logs are the safe default.

#Tracing

Propagate a request ID (and a trace context header) from client to gateway to engine. A trace of queue wait → prefill → decode for slow requests tells you immediately whether the fix is capacity (queue), prompt size (prefill) or output length (decode).

For RAG and agents, trace each stage (retrieval, rerank, each tool call, each model call) as spans in one trace.

#Cardinality and cost of observability

Every unique combination of label values makes a new time series. Labels like user_id, request_id or full prompt text can create millions of series and bring down your metrics store. Rules:

• Bounded labels only on metrics: model, route, status class, release, tenant tier.
• High-cardinality detail goes in logs and traces, not metrics.
• Set retention by value: high-resolution metrics for days, rollups for months, sampled traces for days.
• Sample traces (for example all errors, all slow requests, 1 percent of the rest).

#Dashboards

Design for the question being asked:

1. Overview (for everyone): request rate, error rate, TTFT p95, tokens/s, replicas up. Green or red against the SLO.
2. Capacity (for on-call): running and waiting requests, KV-cache use, GPU memory and utilisation, per replica.
3. Release (for deploys): metrics split by release ID, canary versus stable.
4. Cost (for owners): tokens and spend per tenant or feature, cost per million tokens, utilisation.
5. Quality (for the team): sampled evaluation scores, feedback rate, refusal rate, output-length drift.

#Quality is observable too

Latency and errors do not tell you the model got worse. A new quantization, prompt or retrieval index can return 200 OK in 300 ms and be wrong more often. Sample real traffic into your evaluation pipeline and track scores over time, with intervals (online evaluation). Track proxy signals too: refusal rate, average output length, share of length finishes, empty or malformed outputs, feedback ratios.

#Observability checklist

• TTFT, token rate, errors and timeouts as histograms or rates, per release
• Running and waiting requests, KV-cache use, GPU health
• Alerts tied to the SLO with burn rates, plus capacity warnings
• Structured request records with request ID and release ID
• Prompt and response logging policy decided and enforced
• Traces covering queue, prefill, decode and dependencies
• Bounded metric labels; retention set
• Dashboards for overview, capacity, release, cost and quality
• A runbook link in every alert (reliability patterns)

#Common mistakes

1. Averages instead of percentiles.
2. Alerting on causes (GPU at 90 percent) instead of symptoms (TTFT above SLO).
3. Logging full prompts by default.
4. High-cardinality labels.
5. No release ID on requests, so regressions cannot be tied to deploys.
6. Only server-side metrics, missing queueing and network time users experience.
7. No quality signal, so a bad model update is invisible.

#Exercise

Your SLO is 99.9 percent of requests with TTFT under 1.5 s over 30 days. In the last hour 1.2 percent of requests exceeded 1.5 s.

1. What is the budget and the burn rate?
2. How long until the monthly budget is gone at this rate?
3. What would you do?

Answers:

1. Budget is 0.1 percent. Burn rate 1.2 / 0.1 = 12.
2. At burn rate 12 the 30-day budget lasts 30 days / 12 = 2.5 days.
3. Page: this is nearly the "page now" level. Check the capacity dashboard for waiting requests and KV usage, compare by release ID for a recent deploy, and mitigate (scale out, shed load, roll back). Then write the incident up and adjust capacity or limits.