CI for models: tests, canaries, shadow traffic and rollback

Ordinary software has tests that return pass or fail. A model change can pass every unit test and still make the product worse. A delivery pipeline for LLM services therefore has to produce evidence about behaviour, not only about whether the code runs.

#The pipeline

PR ─▶ static checks ─▶ unit/contract tests ─▶ build bundle ─▶ offline evals ─▶ staging load test
                                                                                   │
production ◀─ progressive rollout (shadow → canary → full) ◀─ gates ◀──────────────┘

Each stage should answer one question and fail fast.

#The release bundle

Version the things that determine behaviour, together:

release: 2026-11-18.3
engine_image: registry.example.com/serving/vllm@sha256:9c1e…
model: { name: llama-8b-instruct, revision: 4f2a9c1, checksum: sha256:… }
serving_config: { max_model_len: 16384, gpu_memory_utilization: 0.90, quant: fp8 }
prompts: { system: v14, rag_template: v7 }
evals: { golden: { accuracy: 0.842, ci95: [0.821, 0.862] }, safety: pass, format_valid: 0.991 }
perf: { ttft_p95_ms: 640, tokens_per_s_per_gpu: 2450 }

Store it in Git or a release registry, deploy by bundle id, roll back by bundle id. A rollback that restores the image but not the prompt is not a rollback.

#Offline evaluation as a gate

• Run the same golden set against the current production bundle and the candidate, on the same day, with the same judge.
• Compare with uncertainty: a 0.8-point difference on 200 examples is noise. Require the confidence interval of the difference to exclude a regression threshold (statistics).
• Slice results (by language, task type, length). A flat average can hide a 10-point drop for one group.
• Keep a never-train-on-it holdout and a small set of known past incidents as permanent regression tests.
• Budget the runtime: CI evals that take two hours get skipped. Use a fast smoke set per PR and the full set nightly or on release.

#Shadow traffic

Mirror a sample of production requests to the candidate and discard its responses (or store them for analysis).

• Compare: latency, token counts, finish reasons, refusal rates, format validity, and a judge score of candidate versus production answers on the same inputs.
• Watch for side effects: tool calls in shadow mode must be disabled or sent to a sandbox.
• Mind capacity: shadowing 10% of traffic needs about 10% extra GPU capacity.
• Mind privacy: shadow outputs contain user data; apply the same retention rules.

#Canary with gates

Send a small share (1 to 5%) of live traffic to the candidate, with automatic analysis:

Hold at each step (1%, 5%, 25%, 50%, 100%) long enough to gather enough samples: low-volume metrics need hours, not minutes. Use statistical tests or sequential analysis rather than eyeballing. Tools such as Argo Rollouts or Flagger automate steps and rollback; the metric queries are yours to define (GitOps and progressive delivery).

Choose canary users carefully: random assignment by user id (sticky), not per-request, so a conversation does not flip between versions mid-way.

#Rollback

Plan for it before you need it:

• Keep the previous bundle ready: its weights cached on nodes, its image pulled, ideally a few replicas still running during the rollout.
• One switch: a config or routing change, not a rebuild.
• Define triggers (automatic for errors and latency; human decision for quality) and an owner.
• Roll back state carefully: caches keyed by model version, vector indices tied to an embedding model, stored conversation formats. Changing the embedding model means re-indexing; it cannot be rolled back with a flag.
• Practise: roll back in staging monthly and time it.

#Database and data changes

Prompt and index changes are releases too. Version prompts as code, review them in PRs, and run evals on them. For retrieval indices, build the new index in parallel, evaluate, then switch an alias; keep the old one until the canary passes.

#What the release should record

After each release, keep: the bundle, eval results, canary analysis, who approved, and the rollback result if any. This is your audit trail and the data for "what changed?" during an incident (incident response).

#Common mistakes

1. Gates on health only, missing quality regressions.
2. Averages only, hiding a regression in a segment.
3. Per-request canary assignment, flipping a conversation between versions.
4. Shadow tools with side effects.
5. No warm previous version, making rollback 15 minutes long.
6. Embedding model changes treated as a flag flip.
7. Evals so slow they are skipped.

#Exercise

Your candidate scores 84.2% versus production's 83.6% on 150 golden examples, with identical latency. A colleague wants to skip the canary.

1. Is the improvement real?
2. What does the canary add?
3. What would you gate on?

Answers:

1. Almost certainly not distinguishable: 0.6 points on 150 examples is well inside sampling noise (roughly ±3 points per estimate).
2. Real-traffic behaviour, rare inputs and quality signals the golden set does not cover, with limited blast radius.
3. Error rate, TTFT p95, cost per request, format validity and a sampled quality score, with thresholds and minimum sample sizes defined in advance.

#Going deeper

• Write your gate thresholds and the evidence required for each.
• Time a rollback in staging.
• Next: incident response and postmortems.