Capacity planning and cost per token

Capacity planning answers one question: how many GPUs do I need for this traffic at this quality of service? Cost planning answers the follow-up: what does each token really cost me? Both start with the same measurements, and both are easy to get wrong in the same way: by planning from averages and single-user benchmarks.

#Step 1: describe the traffic

Measure or estimate, per request type (chat, summarise, extract, agent step):

• requests per second at peak (not average; plan for the busy hour and add a burst factor),
• average and p95 input tokens,
• average and p95 output tokens,
• the time-of-day pattern (a 9-to-5 service has 8 busy hours, not 24).

Do not guess. Instrument the prototype, or look at the logs of whatever the service replaces. If you have nothing, estimate with ranges and plan for the high end.

#Step 2: concurrency with Little's law

The number of requests in flight at once equals arrival rate times time in system:

concurrency = requests_per_second × seconds_per_request

If peak is 5 requests/s and a typical answer takes 8 s end to end, you need to support about 40 concurrent sequences. This number, not requests per second, determines your KV-cache memory and batch size (concurrency).

Seconds per request is TTFT + output_tokens / per-user decode speed. It is not fixed: it rises with load, which raises concurrency further. That feedback loop is why systems fall off a cliff near capacity.

#Step 3: check memory

Concurrency times context times KV bytes must fit alongside the weights:

KV memory = concurrent_sequences × tokens_per_sequence × kv_bytes_per_token

Worked: 40 concurrent sequences, 3,000 tokens each (prompt plus answer), 128 KiB per token (an 8B model):

40 × 3,000 × 128 KiB = 14.6 GiB

Add 4.5 GiB of weights and about 1.5 GiB overhead: about 20.6 GiB. A 24 GiB GPU fits it with 14 percent headroom. If your tokens per sequence doubled, it would not. Quantizing the KV cache halves this (KV cache). If the memory does not fit on one GPU, concurrency per GPU is capped by memory, not compute, and you need more GPUs even if tokens/s would be enough.

#Step 4: measure throughput on your hardware

Use your own load test (load testing) with a realistic prompt mix and record aggregate output tokens per second at your target concurrency, plus the TTFT and per-user speed at that load. Do not trust a single-user benchmark: batching raises aggregate throughput while lowering per-user speed.

Example measurements for one GPU (illustrative):

If your SLO says p95 TTFT under 1.5 s and at least 30 tokens/s per user, the usable capacity is around 12 concurrent sequences at about 600 tokens/s, not the 700 tokens/s at 32 users.

#Step 5: compute GPUs needed

Two checks, take the larger:

by throughput:  tokens_needed_per_s = peak_requests_per_s × avg_output_tokens
                gpus_t = ceil(tokens_needed_per_s / (usable_tokens_per_s_per_gpu × 0.7))

by concurrency: gpus_c = ceil(peak_concurrency / usable_concurrency_per_gpu)

gpus = max(gpus_t, gpus_c) + 1

The 0.7 keeps you at 70 percent of the usable ceiling so spikes do not break your SLO. The + 1 is a spare for failures and rolling updates (N+1).

#End-to-end example

Traffic: 5 requests/s at peak, 300 average output tokens, 8 s per request. Measured usable capacity per GPU at your SLO: 600 tokens/s and 12 concurrent sequences.

tokens needed        = 5 × 300 = 1,500 tok/s
gpus_t = ceil(1,500 / (600 × 0.7)) = ceil(1,500 / 420) = 4
peak concurrency     = 5 × 8 = 40
gpus_c = ceil(40 / (12 × 0.7)) = ceil(40 / 8.4) = 5
gpus   = max(4, 5) + 1 = 6

The concurrency check dominates, which is typical when the SLO limits per-user speed. Six GPUs for 5 requests per second looks like a lot; the levers below can change that dramatically (a smaller or quantised model, a quantised KV cache, shorter prompts, caching, a faster GPU).

#Cost per million tokens

cost_per_M = gpu_cost_per_hour / (tokens_per_s × 3600 × utilisation) × 1,000,000

Utilisation is the lever people forget. A GPU that is busy 25 percent of the day costs four times more per token than the same GPU busy all day. The calculator above lets you see this.

Worked example (illustrative numbers). A GPU costs 1.50 per hour and delivers 600 usable tokens/s.

Notice the factor of four between 20 and 80 percent. Many teams discover that their expensive GPUs average 15 to 25 percent utilisation, because they sized for the peak hour and the rest of the day is quiet.

#Count input tokens too

Output tokens are the slow ones, but prompts also cost: prefill uses compute, and long prompts occupy memory. A request with 3,000 input and 300 output tokens is mostly prefill by token count, though prefill is far faster per token than decode. For planning, a simple honest approach is to cost per request from measurements:

cost_per_request = gpu_cost_per_hour / requests_served_per_hour_at_target_utilisation

Measure requests per hour in your load test with the real prompt mix, then divide. It includes everything.

#Ways to lower cost, in order of effort

1. Raise utilisation: batch more requests, use continuous batching, schedule offline jobs into quiet hours, and scale replicas to the daily curve (autoscaling).
2. Quantize: FP8 or 4-bit weights fit more concurrency per GPU, and a quantised KV cache fits more sequences. Verify quality with evals first (measuring quality).
3. Cache: prefix caching removes repeated prefill (prompt caching).
4. Shorten prompts and outputs: trim system prompts, retrieve fewer chunks, set sensible max_tokens.
5. Right-size the model: route easy requests to a smaller model (gateways).
6. Speculative decoding: faster per-user decode at low concurrency (speculative decoding).
7. Cheaper hardware or pricing: reserved capacity for the baseline, spot for tolerant batch work (GPU choices).

#Compare against an API fairly

Hosted APIs price input and output tokens separately and usually quote per million. To compare, convert your numbers to the same split and include what you leave out of the GPU rent:

Self-hosting usually wins when you have steady high volume, privacy needs or latency control, and loses at low or spiky volume. A rough break-even: take the monthly API bill, subtract what you would still need from the API (for example a frontier model for hard cases), and compare with the monthly self-hosting cost at your realistic utilisation, including people time.

#A reusable calculator

import math

def plan(peak_rps, avg_out_tokens, secs_per_req, usable_tps, usable_conc,
         gpu_cost_hr, utilisation=0.5, headroom=0.7, spare=1):
    tokens_needed = peak_rps * avg_out_tokens
    concurrency = peak_rps * secs_per_req
    gpus_t = math.ceil(tokens_needed / (usable_tps * headroom))
    gpus_c = math.ceil(concurrency / (usable_conc * headroom))
    gpus = max(gpus_t, gpus_c) + spare
    cost_per_m = gpu_cost_hr / (usable_tps * 3600 * utilisation) * 1e6
    monthly = gpus * gpu_cost_hr * 24 * 30
    return dict(tokens_needed=tokens_needed, concurrency=concurrency,
                gpus_by_tokens=gpus_t, gpus_by_concurrency=gpus_c, gpus=gpus,
                cost_per_million_tokens=round(cost_per_m, 2), monthly_cost=round(monthly))

print(plan(5, 300, 8, 600, 12, 1.50))

Change one input at a time and see what moves the answer: usually utilisation, per-GPU concurrency and tokens per request.

#Sensitivity: what to verify first

Always recompute with your own measurements. Prices and hardware change quickly, so treat any example number, including the ones here, as a template.

#Common mistakes

1. Planning for average traffic, then missing the SLO at peak.
2. Using a single-user benchmark as the capacity number.
3. Forgetting KV-cache memory, which caps concurrency before compute does.
4. Ignoring utilisation when quoting cost per token.
5. Comparing GPU rent to API price without people and idle costs.
6. No spare: a single failure or rolling update causes an outage.

#Exercise

A service has a peak of 3 requests/s, 250 average output tokens, 10 s per request. One GPU measures 450 usable tokens/s and 10 concurrent sequences at your SLO, costs 2.00 per hour, and you expect 35 percent average utilisation.

1. How many GPUs do you need with 70 percent headroom and one spare?
2. What is the cost per million output tokens at 35 percent utilisation?

Answers:

1. Tokens needed 3 × 250 = 750; gpus_t = ceil(750 / (450 × 0.7)) = ceil(2.38) = 3. Concurrency 3 × 10 = 30; gpus_c = ceil(30 / (10 × 0.7)) = ceil(4.29) = 5. So max(3, 5) + 1 = 6 GPUs. Concurrency dominates.
2. 2.00 / (450 × 3,600 × 0.35) × 1,000,000 = 2.00 / 567,000 × 1e6 ≈ 3.53 per million output tokens. Raising utilisation to 70 percent would halve it, which is why scheduling and autoscaling matter.