Prerequisite check, and how to measure from day one

#Self-check

You are ready for this course if you can answer these without help (answers at the end):

How many bytes does a 7B-parameter model take at 8-bit?
Why is decode slower per token than prefill, per token processed?
What does a KV cache store, and why does it grow with context?
What is the difference between latency and throughput?
Roughly how fast is a typical PC's RAM compared with a datacenter GPU's memory?

If several are shaky, read Foundations, VRAM math and memory bandwidth first. They are short and free.

#Measure two things now

Before any technique, know where your machine stands. If you have a local runtime, take these two readings.

#1. A decode speed

With llama.cpp:

bash

llama-bench -m MODEL.gguf -ngl 99 -p 512 -n 128

The tg128 row is decode tokens per second; pp512 is prefill. With Ollama, run a prompt with --verbose and read the eval rate. Write the number down together with the model, the quantisation, the context size and the hardware.

#2. A memory reading

Note how much memory the model uses once loaded (GPU memory from nvidia-smi, or the activity monitor on a Mac). Compare it with weights plus KV cache from the calculator (VRAM calculator).

#Compare with the ceiling

Use the calculator from the previous lesson. Divide your bandwidth by the weight size, and compare:

Your measurement as a share of the ceiling	Meaning
Above 80%	The system is close to its memory limit; only fewer bytes (quantisation) or batching will help
40 to 80%	Normal range; overheads exist and may be tunable
Below 40%	Something is wrong or limiting: partial offload, throttling, a generic build, or launch overhead (the tuning playbook)

If you do not have a machine to measure, use the numbers published in the lessons and treat the exercises as paper calculations; you lose the hands-on feel, not the concepts.

#Habits that make later benchmarks trustworthy

Write down the setup (hardware, versions, model, settings) with every number.
Warm up before measuring; the first runs include loading.
Repeat and look at the spread.
Change one thing between measurements.
Measure what you serve: real prompt lengths, real concurrency.
Report percentiles for latency, not only averages (benchmarking).

#Exercise

Run the two measurements and the calculator for your own machine. Write one sentence on where you sit relative to the ceiling, and one guess at why.

Answers to the self-check: 1. About 7 GB. 2. Decode processes one token per weight read while prefill spreads each read over many tokens, so decode wastes the compute. 3. Keys and values of earlier tokens, one set per token, so it grows linearly. 4. Latency is time per request; throughput is work per unit time across requests. 5. A PC's RAM is about 50 to 100 GB/s, a datacenter GPU's memory is 2,000 to 8,000 GB/s, so roughly 30 to 100 times faster.

Next: Prefill, decode and the roofline.