Ollama quickstart and Modelfiles

Ollama wraps llama.cpp in a background service with a model registry and a simple command line and API. It is the fastest route from "nothing installed" to "chatting with a local model", and a good base for small applications.

#Install and run

# macOS / Linux / Windows installers at ollama.com
ollama run llama3.1:8b          # downloads (~4.9 GB) and opens a chat
ollama pull qwen2.5:14b         # download only
ollama list                     # downloaded models
ollama ps                       # loaded models, memory use, CPU/GPU split
ollama show llama3.1:8b         # parameters, template, licence
ollama rm llama3.1:8b           # delete

ollama ps is your first debugging tool: it shows whether the model landed fully on GPU (100% GPU) or spilled to CPU, and how much memory it is using.

Model names look like name:tag, where the tag is typically a size and quantisation (llama3.1:8b, qwen2.5:14b-instruct-q4_K_M). A bare name uses a default tag. Browse the library at ollama.com to see sizes and quants before pulling.

Inside ollama run, helpful slash commands include /set parameter num_ctx 8192, /show info, /save <name> and /bye. Type /? to see them all.

#The API

The server listens on localhost:11434.

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "Explain KV cache in one sentence."}],
  "stream": false,
  "options": {"num_ctx": 8192, "temperature": 0.3}
}'

Main endpoints:

Responses stream by default as one JSON object per line; pass "stream": false for a single response. An OpenAI-compatible endpoint is at http://localhost:11434/v1, so existing OpenAI clients work by changing the base URL:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")   # key is ignored
r = client.chat.completions.create(model="llama3.1:8b",
        messages=[{"role": "user", "content": "Hello!"}])
print(r.choices[0].message.content)

Some options (such as num_ctx) are only available through the native API or a Modelfile, not the OpenAI-style one. The /api/chat endpoint also supports structured output with a format JSON schema (structured output) and tool calling for models that support it (function calling).

#Modelfiles

A Modelfile defines a named variant of a model: its base, parameters, system prompt and (optionally) template.

FROM llama3.1:8b
PARAMETER num_ctx 16384
PARAMETER temperature 0.3
SYSTEM """You are a concise senior engineer. Answer in bullet points."""

ollama create reviewer -f Modelfile
ollama run reviewer
ollama show reviewer --modelfile     # inspect the full Modelfile of any model

Useful instructions: FROM (base model or a GGUF path), PARAMETER (num_ctx, temperature, top_p, top_k, repeat_penalty, stop, seed), SYSTEM, TEMPLATE (the prompt format; change it only if you know the model's template) and ADAPTER (a LoRA). Because a Modelfile sits on top of an existing model, creating one costs almost no disk space.

#Import your own GGUF

# Modelfile
FROM ./my-model-Q4_K_M.gguf
PARAMETER num_ctx 8192

ollama create my-model -f Modelfile

If the GGUF does not carry the right chat template, you must provide a TEMPLATE that matches the model's format, or output quality will suffer. The model card normally gives it.

#Memory, context and the "gotcha"

Gotcha: the default context length is modest (a few thousand tokens, and it has changed between versions). If long prompts seem "forgotten", raise num_ctx (why).

Raising the context raises KV-cache memory (VRAM math); Ollama allocates it when the model loads, so a large num_ctx can push the model partly onto the CPU. After changing it, run ollama ps and look for 100% GPU.

Other behaviour to know:

• Models stay loaded for a few minutes after the last request (keep_alive), so the next request is fast; then they unload to free memory.
• The first request after loading is slower (weights load, caches warm).
• Ollama picks GPU layers automatically; if the model does not fit, it splits layers between GPU and CPU.

#Useful environment variables

How you set them depends on how Ollama runs:

# Linux (systemd service)
sudo systemctl edit ollama.service
#   [Service]
#   Environment="OLLAMA_KEEP_ALIVE=30m"
#   Environment="OLLAMA_FLASH_ATTENTION=1"
sudo systemctl daemon-reload && sudo systemctl restart ollama

# macOS (menu bar app)
launchctl setenv OLLAMA_KEEP_ALIVE "30m"     # then restart the app

# Running the server by hand
OLLAMA_KEEP_ALIVE=30m ollama serve

On Windows set the variables in the system environment settings and restart Ollama. Names and defaults change between releases, so check the current documentation for yours.

#Security

Ollama has no built-in authentication, and binding it to all interfaces exposes your GPU and models to the whole network. If other machines must reach it:

1. Keep it on 127.0.0.1 and put a reverse proxy in front with TLS and API keys (production endpoint lab).
2. Or restrict with a firewall to specific hosts.
3. Do not forward the port to the internet.
4. Remember that anyone who can reach /api/pull can make your server download arbitrary models.

See stack security.

#Troubleshooting

#When Ollama is the wrong tool

• Many concurrent users: use vLLM, or llama-server behind a gateway (runtime landscape).
• Fine-grained control (offload split, custom builds, speculative decoding flags): use llama.cpp directly (llama.cpp).
• Production monitoring and rollout discipline: add a gateway and metrics around it, or choose a server built for it.

#Exercise

1. A model answers well in ollama run but poorly through your app's OpenAI-compatible calls with long prompts. What is the likely cause, and how do you fix it?
2. ollama ps shows 45%/55% CPU/GPU for a 14B model at a 32K context on a 12 GB GPU. What are three fixes?

Answers:

1. The OpenAI-style endpoint cannot set num_ctx per request, so the context stayed at the default and long prompts were truncated. Create a Modelfile variant with a larger num_ctx and call that model name (or use the native API with options.num_ctx).
2. The model plus a 32K cache does not fit in VRAM. Reduce the context (for example to 8K), use a smaller quant (Q4_K_M instead of a higher one) or enable flash attention with OLLAMA_KV_CACHE_TYPE=q8_0 to halve the cache, or choose a smaller model.