Lab: run a real model with your engine and benchmark it against llama.cpp
Capstone for the engine course: make your engine load real weights, validate its logits against a reference implementation, measure speed honestly, add an optimisation you can explain, and write the report comparing tokens/s with llama.cpp and with the bandwidth ceiling.
Goal: turn the toy engine into one that runs a small real open model, verify it is correct, measure it honestly, and explain the gap to llama.cpp and to the hardware ceiling.
You need: the code from this course (Python reference and, ideally, a C/C++ port of the hot loops), a small open model in GGUF or safetensors format (a 0.5B to 1B parameter model is a practical choice), llama.cpp built from source (instructions), and a few hours.
Honest limits. This lab is a project brief, not a tested script: the pure-Python engine is far too slow for billion-parameter models (it processes a handful of tokens per minute), so the realistic route is a Python reference for correctness plus a compiled port for speed. The reference numbers, such as your hardware's bandwidth, must be measured on your machine.
Pro content
Keep reading with Inference Academy Pro
Unlock this full lab, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.