llama.cpp: build, run and serve

Older tutorials use ./main. That binary was renamed to llama-cli (and ./server to llama-server) in 2024. If a command in a blog post does not exist, the name is probably outdated.

llama.cpp is the engine behind much of the local-LLM ecosystem. Running it directly gives you the most control and the most information about what is happening. You can install a prebuilt binary in a minute, or build it to match your hardware.

#Install the easy way

brew install llama.cpp       # macOS / Linux with Homebrew
# or download prebuilt binaries from the project's GitHub releases
# (pick the build for your OS and GPU backend: CUDA, Vulkan, Metal, CPU)

Prebuilt binaries are fine for most people. Build from source when you need a specific backend, the very latest model support or custom options.

#Build from source

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build                       # macOS: Metal is on by default
# NVIDIA:  cmake -B build -DGGML_CUDA=ON
# Vulkan:  cmake -B build -DGGML_VULKAN=ON
# AMD:     cmake -B build -DGGML_HIP=ON
cmake --build build --config Release -j

Binaries land in build/bin/. Requirements: a C++ compiler, CMake, and for GPU backends the matching toolkit (CUDA Toolkit, Vulkan SDK or ROCm). Flag names occasionally change, so check the build instructions in the repository's README for your version.

A quick way to confirm the backend: start a model with -ngl 99 and read the log. It says how many layers were offloaded and to which device. If it reports 0 layers, you built a CPU-only binary or the flag is missing.

#The toolbox

#Run

# Chat, pulling a model from Hugging Face on first use
llama-cli -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M -ngl 99 -c 8192

# Serve on http://localhost:8080 (web UI included)
llama-server -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M \
  -ngl 99 -c 16384 --port 8080

Downloaded files are cached (the location depends on OS and version). You can also point at a file: -m /path/model.gguf.

After the server starts, open http://localhost:8080 for the chat UI, or call it:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "local",
  "messages": [{"role": "user", "content": "Name three uses of a KV cache."}],
  "max_tokens": 200
}'
curl http://localhost:8080/health          # readiness check

The server speaks the OpenAI API, so any compatible client works (serving).

#The flags you will use daily

Run llama-server --help for the full list for your version; it is long, and options evolve. Generate commands interactively with the command builder.

#Context, slots and memory

-c is the total context across all slots, and -np slots share it:

-c 32768 -np 4   ->   each slot gets 8,192 tokens

KV cache memory grows with -c (VRAM math); all of it is reserved when the server starts, so a too-large -c shows up immediately as an out-of-memory error rather than later. Slots matter for throughput under load (concurrency).

#Offloading and partial fits

-ngl 99 puts every layer on the GPU. If the model does not fit, lower -ngl (for example -ngl 24) to keep the rest on the CPU. It works, but each CPU layer runs at system-memory speed, so prefer a smaller quant or a shorter context first (GPU offload).

#Threads

-t matters for CPU inference and for any layers left on CPU. Match your physical core count; hyper-threads rarely help, because decode is bound by memory bandwidth, not by instruction throughput.

#Safe defaults for a first server

llama-server -m model.gguf -ngl 99 -c 8192 -np 2 \
  --host 127.0.0.1 --port 8080 --jinja

Keep --host 127.0.0.1 until you have put authentication and TLS in front (production endpoint lab); the server has only a basic API-key option.

#Troubleshooting

#Common mistakes

1. Following old tutorials that use ./main, ./server or removed flags.
2. Forgetting -ngl, then wondering why the GPU is unused.
3. Setting -c huge "just in case", then running out of memory.
4. Using all logical threads instead of physical cores.
5. Binding to 0.0.0.0 without authentication.
6. Comparing speed without noting quant, context and flags.

#Exercise

You have a 12 GiB GPU and want to serve a 14B model (Q4_K_M, about 8.3 GiB) to two users, each with 8K context (KV about 192 KiB per token).

1. What -c and -np do you use?
2. Does it fit?

Answers:

1. -np 2 and -c 16384 (2 × 8,192).
2. KV is 16,384 × 192 KiB = 3 GiB. Total with 8.3 GiB of weights and about 1 GiB of overhead is about 12.3 GiB, which does not fit in 12 GiB. Options: quantise the KV cache (-ctk q8_0 -ctv q8_0 with flash attention, saving about 1.5 GiB), reduce to 6K context per slot, or use a smaller quant.