Structured logging for LLM services

Logs are the oldest observability tool and the one most often done badly. A typical first attempt is a pile of print lines: unstructured, uncorrelated, and either far too chatty or silent exactly when you need them. For an LLM service, where one request touches a gateway, retrieval, a GPU server and maybe several tools, good logging is the difference between a ten-minute diagnosis and a lost afternoon.

This lesson designs a logging approach from the questions you need to answer.

#Start from the questions

Before choosing a library, write the questions you will ask during an incident or a cost review:

If a field answers no question you would actually ask, do not log it. If you cannot answer a question you care about, add the field.

#Logs, metrics and traces: the division of labour

Use metrics to notice a problem, traces to locate it and logs to explain it. All three share a correlation ID so you can jump between them (observability overview).

#One event per request

The most useful pattern for a request/response service is the wide event (also called a canonical log line): at the end of each request, emit a single JSON record containing everything you know about it.

{
  "ts": "2026-11-11T09:14:22.381Z",
  "level": "info",
  "event": "request_completed",
  "service": "gateway",
  "request_id": "7b1f4c2e9a",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "route": "/v1/chat/completions",
  "tenant": "acme",
  "key_id": "k_42",
  "model": "support-8b",
  "release": "support-bot-2026-11-03",
  "stream": true,
  "prompt_tokens": 1810,
  "completion_tokens": 212,
  "cache_hit_tokens": 1500,
  "queue_ms": 35,
  "ttft_ms": 410,
  "decode_ms": 2380,
  "total_ms": 2825,
  "finish_reason": "stop",
  "status": 200,
  "retries": 0,
  "replica": "llm-3",
  "client_ip_hash": "a91c..."
}

Why one event beats many lines:

• One place to look. Instead of ten lines you must stitch together, you get the answer in one record.
• Cheap to aggregate. Counting tokens per tenant is a field sum, not text parsing.
• Predictable volume. One event per request means volume scales with traffic, not with how many log statements developers added.

Add more events only for things that are not a request summary: startup and shutdown, model load and unload, config changes, circuit breaker transitions, scaling events, rejected requests.

#A schema you can keep stable

Treat log fields like an API. Rules that save pain:

1. Consistent names across services. request_id everywhere, not reqId in one service and request-id in another.
2. Fixed types. ttft_ms is always an integer; never sometimes a string like "410ms".
3. Units in the name (_ms, _bytes, _tokens).
4. Few, bounded enumerations for level, event, finish_reason, status.
5. A version field (schema: 3) so downstream queries can adapt.
6. Document it in the repository, and review changes like code.
7. Never log secrets. No API keys, tokens, passwords or full authorisation headers. Log a key ID, not the key.

#Correlation: follow the request

Every request needs an ID created at the edge and passed to every downstream call:

• Accept an incoming X-Request-ID header if it is well-formed, otherwise generate one.
• Pass it (and the W3C traceparent header if you use tracing) to the model server, retrieval service and tools.
• Return it in the response headers, so a user's bug report can include it.
• Include it in every log line the request produces, including errors from deep inside libraries.

In Python, contextvars carry the ID through async code without threading it through every function:

import contextvars, json, logging, time, uuid
from fastapi import FastAPI, Request

request_id_var = contextvars.ContextVar("request_id", default="-")

class JsonFormatter(logging.Formatter):
    def format(self, record):
        base = {"ts": time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(record.created)) + f".{int(record.msecs):03d}Z",
                "level": record.levelname.lower(), "msg": record.getMessage(),
                "request_id": request_id_var.get()}
        base.update(getattr(record, "fields", {}))          # structured extras
        if record.exc_info:
            base["error_type"] = record.exc_info[0].__name__
        return json.dumps(base, separators=(",", ":"))

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
log = logging.getLogger("gateway"); log.addHandler(handler); log.setLevel(logging.INFO)

app = FastAPI()

@app.middleware("http")
async def request_log(request: Request, call_next):
    rid = request.headers.get("x-request-id") or uuid.uuid4().hex[:16]
    token = request_id_var.set(rid)
    t0 = time.perf_counter()
    status, extra = 500, {}
    try:
        response = await call_next(request)
        status = response.status_code
        response.headers["x-request-id"] = rid
        return response
    finally:
        log.info("request_completed", extra={"fields": {
            "event": "request_completed", "route": request.url.path, "status": status,
            "total_ms": int((time.perf_counter() - t0) * 1000),
            "tenant": getattr(request.state, "tenant", None), **getattr(request.state, "usage", {}), **extra}})
        request_id_var.reset(token)

The handler that fills request.state.usage (token counts, cache hits, timings) adds those fields to the same event. For streaming responses, emit the event when the stream ends (or is cancelled), since that is when TTFT, decode time and completion tokens are known.

#What to log for each hop

#The prompt question

The most consequential logging decision for an LLM service is whether to record prompt and completion text.

Why teams want to: debugging bad answers, building evaluation sets, understanding user needs.

Why it is risky: prompts contain personal data, secrets pasted by users, confidential documents and sometimes credentials. A log store is typically readable by more people, kept longer and backed up more widely than your primary database (data leakage, PII, compliance).

A policy that works for most teams:

1. Default: metadata only. Lengths, token counts, hashes, language, detected categories, flags such as contains_pii_detected.
2. Sampled content with redaction (for example 1 percent), stored in a separate, access-controlled, short-retention index.
3. Full content on demand for a single request ID, with an explicit reason and an audit record, for support cases.
4. Always off for sensitive tenants and categories.
5. Hash for dedupe. A hash of the prompt lets you spot repeated prompts and cache opportunities without storing the text (use a keyed hash so it cannot be reversed by guessing).

Whatever you choose, write it down, tell users, and make it enforceable in code (a single log_content() function guarded by policy), not by convention.

#Levels: what each means

A per-request debug switch is worth building: an internal header or a tenant flag that raises the level for that request only, with the debug output tagged and short-lived.

#Volume, sampling and cost

Do the arithmetic early:

events/day = requests/s × 86,400
bytes/day  = events/day × bytes per event

At 1,000 requests per second a verbose schema is a real bill. Levers:

• Trim fields. A 1 KB event is generous; most can be 300 to 500 bytes.
• Sample successes, keep all failures. Keep 100 percent of errors, slow requests and rejected requests; keep, say, 10 percent of fast successes, and record the sample rate in each event so counts can be re-weighted. (Metrics still count everything.)
• Tier retention. Keep detail for 7 days, aggregated rollups for months.
• Compress. JSON compresses 5 to 10 times; most log stores do it for you.
• Do not log inside hot loops. Per-token log lines are an anti-pattern: 300 tokens × 1,000 req/s is 300,000 lines per second. If you need token-level timing, record it as a histogram or a few summary fields.

#Logging errors usefully

An error line should answer what failed, for whom, and what was tried:

{"level":"error","event":"upstream_failed","request_id":"7b1f4c2e9a","upstream":"llm-3",
 "error_type":"ConnectTimeout","timeout_ms":2000,"attempt":2,"will_retry":true,"fallback":"llm-1"}

• Log once per failure at the point where it is handled, not at every layer that re-raises it (this produces ten lines for one error).
• Include error type and a stable error code, not only the message, so you can count by cause.
• Include what you decided to do (retry, fall back, give up), since that is what an on-call engineer needs.
• Keep stack traces for unexpected exceptions; do not attach them to routine, expected errors such as client disconnects.
• Remove any content that came from the prompt: exception messages from validation libraries sometimes echo input.

#Logging for streaming and long requests

Streaming makes timing richer and logging trickier:

• Record ttft_ms when the first token is sent, and total_ms at the end.
• Record finish_reason: stop, length, client_disconnect, timeout, error.
• A client that disconnects mid-stream still produced work. Log it, along with completion_tokens so far, so you can quantify wasted GPU time (reliability patterns).
• For agent runs lasting minutes, log a start event, one event per step and an end event with a run ID, rather than a single event at the end that may never arrive if the process dies.

#Testing your logging

Logging code is code. Test it:

1. Schema tests: parse a sample of events and check required fields and types.
2. A "no secrets" test: run requests containing fake keys and card numbers through the service and assert they never appear in log output (a canary approach, data leakage).
3. Correlation test: send a request with a known ID and confirm every service logs it.
4. Volume test: during load tests, measure log bytes per request and extrapolate (load testing).
5. Failure test: kill the log destination and confirm the service does not block or crash. Logging must never take down serving; use a bounded buffer and drop on overflow, counting drops in a metric.

#Where logs go

On containers, write logs to stdout as JSON and let the platform collect them (Docker deployment). Do not write log files inside containers, and do not make the application responsible for shipping them. The next lesson covers collectors, storage, redaction, retention and cost.

#Common mistakes

1. Free-text logs that no query can aggregate.
2. No request ID, so an incident means matching timestamps by eye.
3. Logging prompts by default.
4. Per-token or per-chunk log lines.
5. Different field names in each service.
6. Logging the same error at every layer.
7. Unbounded log volume with no sampling or retention plan.
8. Logging that blocks the request path.
9. Secrets in logs, including in URLs and exception messages.

#Exercise

Your gateway handles 200 requests/s. Each event is 800 bytes. Failures are 2 percent of traffic and you want to keep all of them plus 10 percent of successes.

1. How much log data per day without sampling?
2. How much with sampling?
3. What must each event record so your dashboards still count correctly?

Answers:

1. 200 × 86,400 × 800 B = 13.8 GB per day.
2. Kept fraction 0.02 + 0.98 × 0.10 = 0.118, so about 1.6 GB per day, an 88 percent reduction.
3. A sample_rate field (1.0 for kept failures, 0.1 for sampled successes), so counts from logs can be re-weighted. Dashboards for totals and rates should come from metrics, which count every request regardless of log sampling.

#Going deeper

• Add the wide event to your gateway and compare how long an incident query takes before and after.
• Continue with log pipelines, cost and retention.
• Pair logs with metrics and tracing.