From laptop to production: what actually changes

A model that answers well in your terminal is a prototype. A service is something other people depend on while you are asleep. The model is the same; everything around it changes. This course is about that "everything around it", and this first lesson sets the map.

#The four things production adds

Everything else (security, evaluation, compliance) rides on top of these, and each has its own course in the catalogue (security, evaluation).

#A worked scenario

You built a support assistant: an 8B model on a single GPU box, behind a small API, used happily by you and two colleagues. Marketing wants to put it in the customer app next month, with an expected 3,000 conversations a day.

What changes?

• Someone will send a 90,000-token prompt, or ask for 50,000 output tokens, and the server will stall or crash.
• Peak hour will bring 40 people at once. Your single-user speed was 90 tokens/s; the server's memory was fine for 2 users.
• The GPU box will reboot for a driver update at 2 p.m. on a Tuesday.
• A new model version will pass all your quick checks and subtly give worse refund advice.
• The bill will arrive, and the finance team will ask what a conversation costs.
• A customer will paste a card number into the chat, and you will be asked where it went.

None of these show up in a prototype. All of them are predictable, which means they can be designed for.

#Start with service level objectives

You cannot design capacity or alerts until you say what "good" means. For LLM services the user-visible numbers are:

• TTFT (time to first token): how long until anything appears. Driven by queueing and prefill (how generation works).
• Inter-token latency (or tokens per second per user): how fast the answer streams. Driven by decode speed and batch size.
• Error rate: failed, timed-out or truncated requests.
• Availability: the fraction of time the service accepts requests.
• Quality: the fraction of answers that meet your bar, measured by evaluation (evaluation course).

A usable SLO names a percentile, a threshold and a window, for example: "p95 TTFT below 1 s and error rate below 0.5 percent, measured over 30 days." Use percentiles, not averages: a mean of 1 second can coexist with a p99 of 12 seconds, and the people on the p99 are the ones who complain.

#Error budgets: turning an SLO into numbers

An SLO of 99.5 percent leaves a 0.5 percent error budget. Over 30 days (43,200 minutes) that is:

Each extra nine costs far more than the last: it usually means redundant hardware, automated failover and a larger on-call burden. Pick the lowest target your users genuinely need. Many internal tools are fine at 99 percent; a customer-facing checkout assistant may need 99.9.

The error budget is a management tool: when you have budget left, ship changes faster; when it is spent, slow down and fix reliability. A single GPU server cannot honestly promise 99.9 percent, because one reboot per month already uses the budget.

#Failure modes unique to LLM serving

1. Out-of-memory under load. Memory use grows with concurrent sequences, because each holds a KV cache. With a model using 128 KiB per token and 8K contexts, each sequence costs 1 GiB: 2 users need 2 GiB, 20 users need 20 GiB. A server that was fine at 2 users crashes at 20 (VRAM math).
2. Head-of-line blocking. One 30,000-token prompt can stall short requests behind it unless the scheduler chunks prefill.
3. Cold starts. Loading a 40 GB model takes tens of seconds to minutes. Autoscaling that ignores this makes latency worse, not better (autoscaling).
4. Silent quality regressions. A new quantization or model version still returns HTTP 200. Only evaluations catch it (regression testing in CI).
5. Unbounded requests. A client that asks for 100,000 output tokens can monopolise a GPU. Always enforce limits (reliability patterns).
6. Slow-burn cost. A prompt that grows by 200 tokens per release quietly doubles your bill over a year.
7. Dependency failures. The vector store, the tool API or the auth service goes down, and the LLM service goes with it unless you design for it.
8. Abuse. Open endpoints are scanned and used for free compute and data extraction (stack security).

#The maturity ladder

Most teams should aim for stage 3 before putting real customers on it, and stage 2 before anyone outside the team. Do not build stage 4 for a stage 1 workload.

#A minimal production checklist

• At least two replicas behind a health-checked load balancer
• Hard limits: max input tokens, max output tokens, request timeout, per-client rate limit
• Authentication and TLS; the model server not directly reachable
• Metrics scraped every 15 seconds and alerts tied to your SLOs
• A documented, tested way to roll back to the previous model version
• A cost dashboard: GPU hours, tokens served, cost per million tokens
• Evaluations that run on every change to the model, prompt or retrieval
• Logging policy: what is recorded, for how long, who can read it
• A runbook for the top three alerts and an on-call owner

#A reference architecture

users ──▶ edge / CDN ──▶ gateway (auth, limits, routing, retries)
                              │
                 ┌────────────┼────────────┐
                 ▼            ▼            ▼
            replica 1    replica 2    replica N     (model servers on GPUs)
                 │            │            │
                 └─── metrics, logs, traces ───▶ observability stack
                              │
                      model registry / storage (pinned, versioned weights)

The rest of this course takes each box in turn: capacity and cost next, then GPUs and deployment targets, gateways, autoscaling, observability, rollouts and reliability, ending with a lab that ships a monitored deployment.

#Build, buy or mix?

Self-hosting trades per-token fees for engineering time, on-call and idle capacity. The decision is mostly about volume, privacy and how much operational work you can take on, so count the people as well as the GPUs (capacity planning).

#People and process

Production is a team property, not just a technical one:

• Ownership: a named owner for the service, the model release and the on-call rota.
• Change management: how a prompt change, a model swap or a config edit reaches users (rollouts).
• Incident response: who is paged, how decisions are made, how reviews happen.
• Documentation: a runbook that someone else can follow at 3 a.m.

#Common mistakes

1. Going to production on a single server, then calling a reboot an incident.
2. Load testing with one user, or with a single prompt that hits the cache.
3. No limits, trusting clients to behave.
4. Averages instead of percentiles in dashboards.
5. No rollback plan for model changes.
6. Treating the model like a database (stateless, deterministic) when it is a statistical component.
7. Ignoring cost until the first invoice.

#Exercise

Your support bot must be available 99.9 percent of the time over 30 days, with p95 TTFT under 1.5 s. You currently run one server and reboot it monthly for patches (10 minutes), plus an average of two crashes a month with 8 minutes of recovery each.

1. How much downtime does the SLO allow?
2. How much do you currently use?
3. What would you change?

Answers:

1. 0.1 percent of 43,200 minutes is about 43 minutes.
2. 10 + 2 × 8 = 26 minutes in a normal month, 60 percent of the budget, with no room for a bad day, and a single driver issue could exceed it entirely.
3. Add a second replica behind a health-checked gateway and do rolling restarts so that patches cause no downtime, add alerts and automatic restart for crashes, and run a game day to confirm that killing one replica leaves users unaffected (monitored deployment lab).