The runtime landscape: which engine when?

There is no single best way to run a model locally. Different runtimes are built for different jobs: tinkering on a laptop, squeezing a model onto odd hardware, serving a hundred users, or shipping one executable. Choosing well saves weeks, and choosing badly (for example vLLM for a single-user laptop, or Ollama for a 100-user API) is a classic and avoidable mistake.

#The layers underneath

Most tools you will meet are layers on a few engines:

• ggml / llama.cpp is the C/C++ inference engine behind Ollama, LM Studio, llamafile, Jan and many apps. It is portable, quantisation-first and runs on almost anything.
• PyTorch-based servers (vLLM, SGLang, TGI) run models in 16-bit or compressed GPU formats with batching built for throughput.
• MLX is Apple's own array framework for Apple Silicon.
• Vendor stacks (TensorRT-LLM and others) squeeze maximum speed from specific GPUs, at the cost of build complexity.

Knowing the layer tells you what to expect: a front-end built on llama.cpp inherits its model support and its quirks, and its updates.

#Each runtime in a paragraph

llama.cpp. The Swiss army knife. Builds for every major backend, supports partial offload (some layers on GPU, rest on CPU), the widest set of quant types, grammar and JSON-schema constrained output, speculative decoding, embeddings and a built-in web UI. You assemble flags yourself, which is both its power and its learning curve (llama.cpp lesson).

Ollama. A daemon plus CLI plus model registry around llama.cpp. ollama run <model> downloads and chats. Modelfiles customise prompts and parameters. It provides a native API and an OpenAI-compatible one. Defaults are conservative (notably a small context window), and concurrency is basic (Ollama lesson).

LM Studio and Jan. Desktop apps with model browsers, chat UIs and a local server. Ideal if you prefer clicking to typing. LM Studio can also run MLX models on Macs.

vLLM. A Python server for GPUs focused on throughput: continuous batching and PagedAttention let one GPU serve many streams efficiently. It prefers Hugging Face weights and GPU-oriented quantisations (AWQ, GPTQ, FP8). Heavier to install, GPU-first, excellent for teams and products (vLLM lesson).

MLX and mlx-lm. Apple's framework designed around unified memory. Often fast on newer chips, with built-in quantisation and LoRA fine-tuning. Mac only (MLX lesson).

llamafile. Packs a model and llama.cpp into a single executable that runs on several operating systems. Great for distribution to non-technical users.

SGLang and TGI. Other production-grade GPU servers. SGLang is known for its structured-generation and prefix-caching features; Hugging Face's TGI integrates tightly with the Hub. Evaluate them against vLLM for your workload.

#Decision tree

1. Just trying models on my laptop → Ollama or LM Studio.
2. Want full control, odd hardware, partial CPU/GPU offload, grammars → llama.cpp.
3. On a Mac and want best speed or want to fine-tune → MLX (or llama.cpp with Metal; benchmark both).
4. Serving a team or product on NVIDIA GPUs → vLLM (or SGLang/TGI after testing).
5. Shipping a single executable to users → llamafile or embed llama.cpp.
6. Running in a browser or on a phone → see edge options.
7. Need maximum throughput on a fixed NVIDIA fleet → consider vendor stacks, after you have measured that the simpler engines are not enough.

#Questions to ask

#The same model, different engines

Most engines expose an OpenAI-compatible API, so your application code does not need to change when you swap runtimes (serving):

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="x")   # Ollama
# base_url="http://localhost:8080/v1"   # llama-server
# base_url="http://localhost:8000/v1"   # vLLM

Things that do differ between engines: model names, which sampling parameters are accepted (min-p, repeat penalty and similar are often runtime-specific), tool-calling parsers, structured-output syntax and error messages. Keep these behind a small adapter so you can switch without touching business logic.

#Comparing runtimes fairly

Benchmarks of runtimes are only meaningful if you hold things constant:

• The same model and equivalent precision. GGUF Q4_K_M and AWQ 4-bit are not identical in size, speed or quality.
• The same context length and concurrency.
• The same prompts, including prompt length (prefill-heavy versus decode-heavy workloads favour different engines).
• Warm-up first, then measure several runs.
• Report both prompt processing and generation speed, and for servers, throughput under load with latency percentiles (load testing).

A single tokens-per-second number from a blog post, measured on someone else's hardware, rarely transfers. Measure yours (benchmark lab).

#Migrating between runtimes

#Common mistakes

• Using vLLM for a single-user laptop (heavy, GPU-only) or Ollama for a 100-user API (not designed for it).
• Comparing runtimes with different quantizations or context sizes.
• Ignoring defaults: Ollama's default context is small, so set it explicitly.
• Choosing by benchmark headlines rather than by your workload.
• Exposing a runtime with no authentication to a network (stack security).
• Running with out-of-date versions: runtimes evolve quickly, and new model architectures need new releases.

#Exercise

For each, pick a runtime and justify it in one sentence:

1. A designer wants to try ten models on a MacBook Air with no terminal use.
2. A startup serves a support bot to 200 concurrent users on two A100 GPUs.
3. A hobbyist has an old laptop with 16 GB of RAM and a small integrated GPU.
4. A researcher on a 128 GB Mac Studio wants to fine-tune a 7B model with LoRA.

Answers: (1) LM Studio, since a GUI with a model browser needs no setup. (2) vLLM behind a gateway, for continuous batching and high concurrency. (3) llama.cpp or Ollama with a small quantised model on CPU, where portability and low overhead matter. (4) MLX (mlx-lm), which has built-in LoRA and uses the unified memory efficiently.