What is local LLM inference?

Inference means using a trained model to generate output. Local inference does that on hardware you control (a laptop, a desktop GPU, a home server, a phone) instead of sending your prompt to a hosted API. The model is a file on your disk, the computation happens on your chip, and nothing has to leave your machine.

#The four pieces

1. Weights: the trained parameters, usually a multi-gigabyte file (for example .gguf or .safetensors). A "7B" model has about seven billion numbers; at 4-bit precision that is roughly 4 to 5 GB.
2. Tokenizer and chat template: the tokenizer turns text into token IDs (tokens); the chat template wraps your messages in the exact format the model was trained on. A wrong template is the most common cause of "the model seems dumb".
3. Runtime: software that loads the weights and runs the math: llama.cpp, Ollama, vLLM, MLX and others (runtimes).
4. Hardware: memory holds the weights and the context; memory bandwidth and compute decide how fast tokens appear (hardware).

prompt text -> tokenizer -> [ runtime + weights on your hardware ] -> token IDs -> text

Everything in this course is some variation of choosing, fitting, speeding up or operating those four pieces.

#What actually happens when you press Enter

1. Your text is split into tokens and wrapped in the chat template.
2. The runtime does prefill: it processes the whole prompt in parallel and builds a memory of it (the KV cache).
3. It then does decode: it predicts one token, appends it, and repeats, one token at a time, until the model emits a stop token or hits your limit.
4. Each token is chosen from a probability distribution by sampling (sampling).
5. Tokens are converted back to text and streamed to you.

Steps 2 and 3 are where your hardware matters, and how generation works explains why they behave so differently.

#Why run locally

• Privacy: prompts and documents never leave your machine. For legal, medical, financial or proprietary material this can be the deciding factor (but see privacy for what local does and does not guarantee).
• Cost at volume: no per-token fee once you own the hardware. Steady high usage can be dramatically cheaper.
• Offline use and predictable latency: works on a plane or in an air-gapped network, with no rate limits or outages you do not control.
• Control: pin an exact model version, choose sampling, add grammars and constrained output, fine-tune, inspect everything. Hosted models can change underneath you.
• Learning: nothing teaches how these systems work like running one.

#Why you might not

• Quality ceiling. The very best models are far larger than consumer hardware can hold. Open models at 7 to 70B are strong and improving, but for the hardest reasoning tasks hosted frontier models are still ahead.
• Hardware, power and maintenance are yours. GPUs cost money, use electricity, heat a room and need updates.
• Speed varies a lot. A 7B model on a good GPU feels instant; a 70B model on a laptop CPU does not.
• For occasional use, a hosted API is usually cheaper and simpler.
• You own the safety work: moderation, prompt-injection defence, logging policy (security).

Rule of thumb: local wins when privacy, offline use, steady high volume or deep customisation matter more than peak model quality.

#A back-of-envelope cost break-even

Prices change constantly, so treat this as a method, not a quote.

Suppose a hosted model costs about 1 per million output tokens and your application generates 20 million tokens a month, so 20 per month. A used 24 GB GPU and a PC might cost 1,200 up front plus a few dollars of electricity per month. At 20 per month the break-even is years, so hosted wins. At 2 billion tokens a month (2,000 per month hosted), the hardware pays for itself in a few weeks. And if your volume is high but your quality bar needs a frontier model, the comparison no longer applies.

The honest summary: volume and quality requirements decide. Estimate your monthly token count first. The cost-per-token lesson shows how to do this properly.

#What size model fits what hardware?

A rough guide for 4-bit quantised models (more in VRAM math):

You do not need a GPU to start. A recent laptop with 16 GB of RAM can run a 3 to 8B model at a usable speed, especially on Apple Silicon.

#Common myths

• "Local models are toys." Small models are limited, but models in the 7 to 32B range handle most summarising, extraction, rewriting, Q&A and coding-assistance tasks well.
• "You need an expensive GPU." Helpful, not required: quantisation (quantization) shrinks models enough for ordinary machines.
• "Local means private by default." The model is private; your logs, plugins, browser extensions and chat apps might not be. Check what each tool stores.
• "Bigger is always better." A well-chosen 8B with a good prompt often beats a badly configured 70B, and it responds ten times faster.

#Your first run in five minutes

If you have never run a model, do this before reading further:

# install Ollama from ollama.com, then:
ollama run llama3.2:3b
>>> Explain what a token is in two sentences.

Notice three things: how long the first token takes (prefill), how fast the words stream (decode), and how much memory the model takes (watch Activity Monitor or nvidia-smi). Then try a larger model and compare. You have just run an inference stack, and the rest of the course explains what you saw.

#How this course is organised

1. Foundations (this track): tokens, generation, sampling.
2. Hardware: sizing with the VRAM calculator, bandwidth, Apple Silicon.
3. Quantization: GGUF, K-quants, and what each costs in quality.
4. Runtimes: Ollama, llama.cpp, vLLM, MLX.
5. Performance, serving, applications, edge, evaluation: the practical layers on top.

Every track ends with checks you can run on your own machine. If you want the production, security and evaluation angles, start from the course catalogue.

#Exercise

You have a laptop with 16 GB of RAM and no discrete GPU. You want a private assistant for summarising meeting notes.

1. Which size class should you try first?
2. What would you measure to decide whether it is good enough?

Answers:

1. A 7 to 8B model at 4-bit (about 5 GB), which leaves room for the OS and a few thousand tokens of context.
2. Time to first token on a typical notes-sized prompt, tokens per second while generating, and the quality of summaries on five real meeting notes compared with what you would write yourself. If quality is lacking, try a 14B model at a lower quant if memory allows, or accept a hosted API for non-sensitive notes.