Your first run in ten minutes
Install a runtime, download a small model, chat with it, call it from code, and check how fast it is, with the commands for each step and what to do when something goes wrong.
This lesson gets you from nothing to a model answering on your machine. The commands use Ollama, the simplest runtime; later lessons cover alternatives (runtime landscape). Interfaces and model names change, so if a command differs, check the project's current documentation.
#1. Install
Download the installer from the Ollama site, or use your package manager. Then check:
ollama --version#2. Pick a small model
Choose something that fits in memory with room to spare. As a guide, a 3 to 4 billion parameter model at 4-bit needs about 2 to 3 GB. Not sure what fits? Use the VRAM calculator or the Can I run it? pages.
ollama pull llama3.2:3bThe name is an example; browse the library for current models.
#3. Chat
ollama run llama3.2:3bType a question, press enter, and read the answer. Exit with /bye. The model is now running on your machine, and nothing left it.
#4. See the speed
ollama run llama3.2:3b --verbose "Explain what a token is in two sentences."At the end it prints statistics. The eval rate is generation speed in tokens per second; the prompt eval rate is how fast it read your prompt. Write both down: they are your baseline for the Hardware and Performance tracks (measuring speed).
#5. Call it from code
Ollama exposes an OpenAI-compatible API on port 11434:
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama3.2:3b", "messages": [{"role": "user", "content": "Say hello."}]}'Any tool or library that speaks the OpenAI API can use http://localhost:11434/v1 as its base URL (the API).
#If something goes wrong
| Symptom | Likely cause | Fix |
|---|---|---|
| Very slow generation | The model does not fit in GPU memory and runs on the CPU | Pick a smaller model or a smaller quant |
connection refused |
The server is not running | Start the app or ollama serve |
| Out of memory | The model plus context is too big | Smaller model or shorter context |
| Downloads stall | Network or disk space | Retry; check free space |
| Odd or repetitive output | A very small model, or sampling settings | Try another model; see sampling |
#What you now have
A working local model, a speed baseline and an API endpoint. The rest of the course explains why the speed is what it is, how to fit larger models, how to pick the right runtime, and how to build on top.
#Exercise
Run the model with a short prompt and then a prompt of several paragraphs. Compare the prompt eval rate and the eval rate. Which changed more, and why?
Answer: the prompt eval rate (prefill) is much higher than the eval rate (decode), because reading a prompt can use the hardware's compute in parallel while generation reads all weights for each token. The two phases are explained in how generation works.
Next: What is local LLM inference?.
Quick quiz
+10 XP per correct answer, +25 XP for a perfect score
1. Why start with a small model?
2. What does the tokens-per-second figure measure?
3. What does an OpenAI-compatible local API let you do?
Found this useful?
Send it to someone who is learning the same thing.