Units and vocabulary: FLOPs, bytes, bandwidth and the balance point
The numbers every inference calculation uses: parameters to bytes, FLOPs per token, memory bandwidth, compute throughput, arithmetic intensity and the machine balance point, with a short runnable calculator.
Inference performance is arithmetic with a few units. If you can do these conversions, every later lesson is easier.
#The vocabulary
| Term | Meaning | Typical figure |
|---|---|---|
| Parameter | One learned number in the model | 8 billion for an 8B model |
| Bytes per parameter | The storage precision | 2 (16-bit), 1 (8-bit), 0.5 (4-bit) |
| FLOP | One floating-point operation (a multiply or an add) | |
| FLOP/s (or TFLOP/s) | Compute throughput | about 1,000 TFLOP/s on a datacenter GPU at low precision |
| Memory bandwidth | Bytes per second the memory can deliver | about 3,350 GB/s on an H100, about 1,000 GB/s on a high-end consumer GPU, 50 to 100 GB/s on a typical PC's RAM |
| Arithmetic intensity | FLOPs performed per byte read | |
| Balance point | Compute divided by bandwidth | about 295 FLOP per byte for 989 TFLOP/s over 3,350 GB/s |
| Prefill | Process the whole prompt at once | compute-bound |
| Decode | Generate one token at a time | memory-bound |
Figures change with each hardware generation; the method does not.
#The two ceilings
For any workload there are two speed limits, and the slower of the two wins:
memory limit = bandwidth / bytes read per token
compute limit = FLOP/s / FLOPs per tokenFor a model with P parameters stored at b bytes each, one decode token at batch size 1 reads about P × b bytes and costs about 2P FLOPs.
#A calculator you can run
def ceiling(params_b, bytes_per_param, bandwidth_gbs):
gb = params_b * bytes_per_param # GB of weights read per token
return bandwidth_gbs / gb # tokens per second, one stream
print("8B at 16-bit on 3350 GB/s:", round(ceiling(8, 2, 3350)), "tok/s")
print("8B at 4-bit on 3350 GB/s:", round(ceiling(8, 0.5, 3350)), "tok/s")
print("8B at 4-bit on 100 GB/s:", round(ceiling(8, 0.5, 100)), "tok/s")
def compute_ceiling(params_b, tflops):
return tflops * 1e12 / (2 * params_b * 1e9)
print("compute ceiling 8B at 989 TFLOP/s:", round(compute_ceiling(8, 989)), "tok/s (batch 1, ideal)")
print("balance point:", round(989e12 / 3350e9), "FLOP per byte")Output:
8B at 16-bit on 3350 GB/s: 209 tok/s
8B at 4-bit on 3350 GB/s: 838 tok/s
8B at 4-bit on 100 GB/s: 25 tok/s
compute ceiling 8B at 989 TFLOP/s: 61812 tok/s (batch 1, ideal)
balance point: 295 FLOP per byteRead it: the memory ceiling (209 tokens per second) is about 300 times lower than the compute ceiling (about 62,000). At batch size 1, decode is therefore memory-bound by a huge margin. Quantising to 4-bit moves the memory ceiling up four times. A machine with slower memory (the 100 GB/s line is typical of PC RAM) is slower in proportion to its bandwidth.
#Why batching helps
Reading the weights once can serve several tokens at the same time. With a batch of B sequences, the bytes read per step are almost unchanged but the FLOPs are 2P × B, so arithmetic intensity grows with B. Throughput therefore rises with batch size until the work reaches the balance point and becomes compute-bound (roofline lesson).
#Prefill versus decode in one table
| Prefill | Decode | |
|---|---|---|
| Tokens per pass | many (the prompt) | one per sequence |
| Arithmetic intensity | high | low |
| Limited by | compute | memory bandwidth |
| What helps | faster compute, lower precision tensor cores, flash attention | faster memory, fewer bytes (quantisation), batching |
| You feel it as | time to first token | tokens per second |
#Common unit mistakes
- GB versus GiB. 1 GiB is 1.074 GB. Weights and bandwidth are usually quoted in different units; convert before dividing.
- Bits versus bytes. 4-bit is half a byte.
- Peak versus achieved. Real kernels reach 50 to 90 percent of peak bandwidth and often much less of peak compute. Treat ceilings as upper bounds.
- Forgetting the KV cache. Long contexts add cache reads to every token (context cost).
#Exercise
- A 70B model at 4-bit on a GPU with 1,000 GB/s. What is the memory ceiling?
- The same model on two such GPUs with the weights split between them (ideal case, memory limits only). What is the new ceiling, and what real-world effects will lower it?
Answers: 1. Weights about 70 × 0.5 = 35 GB, so 1,000 / 35 ≈ 29 tokens per second. 2. Each GPU reads half the weights at the same time, so the ideal ceiling is about 57 tokens per second; communication between GPUs, uneven splits and kernel overheads reduce it (model parallelism).
Quick quiz
+10 XP per correct answer, +25 XP for a perfect score
1. About how many FLOPs does one token cost for a model with P parameters?
2. A GPU has 989 TFLOP/s and 3,350 GB/s. What is its balance point?
3. Decode at batch size 1 has an arithmetic intensity of about 1 FLOP per byte at 16-bit. Which limit applies?
4. What happens to arithmetic intensity as batch size grows?
5. Why is a 4-bit model faster at decode than the same model at 16-bit on the same GPU?
Found this useful?
Send it to someone who is learning the same thing.