The OpenAI-compatible API

llama.cpp, Ollama, vLLM, LM Studio and MLX all expose (nearly) the same HTTP API as OpenAI. That means one client code path works everywhere, and migrating between local and hosted is a configuration change. It is the single most useful compatibility fact in local inference, and also a place where subtle differences bite.

#Endpoints you'll use

Runtimes also have their own non-OpenAI endpoints (health checks, metrics, tokenization, model management), which are useful but not portable.

#Anatomy of a chat request

{
  "model": "local",
  "messages": [
    {"role": "system", "content": "You are concise."},
    {"role": "user", "content": "Give me 3 uses of the KV cache."}
  ],
  "temperature": 0.3,
  "top_p": 0.95,
  "max_tokens": 300,
  "stream": false,
  "stop": ["\n\nUser:"],
  "seed": 42
}

• messages: the conversation, each with a role (system, user, assistant, and tool for tool results) and content. The server applies the model's chat template (tokens and context).
• temperature, top_p: sampling (sampling).
• max_tokens: a cap on the reply length. Always set it.
• stop: strings that end generation.
• seed: for reproducibility where supported.
• stream: return tokens as they are generated.

The response (non-streaming):

{
  "id": "chatcmpl-123",
  "model": "local",
  "choices": [{"index": 0, "finish_reason": "stop",
               "message": {"role": "assistant", "content": "1. ..."}}],
  "usage": {"prompt_tokens": 28, "completion_tokens": 96, "total_tokens": 124}
}

finish_reason is useful: stop means the model finished, length means it hit max_tokens (the answer may be cut off), and tool_calls means the model wants a tool run. usage tells you the token counts you pay for in time and memory.

#curl

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role":"user","content":"Give me 3 uses of KV cache."}],
    "temperature": 0.3
  }'

#Python (official SDK)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed", timeout=120)

stream = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Explain prefill vs decode briefly."}],
    stream=True,
    max_tokens=300,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Even when the server needs no key, the SDK requires api_key to be a non-empty string.

#What a stream looks like on the wire

With stream: true, the server sends server-sent events: lines starting with data:, each a JSON chunk, ending with data: [DONE].

data: {"choices":[{"delta":{"role":"assistant"},"index":0}]}
data: {"choices":[{"delta":{"content":"Pref"},"index":0}]}
data: {"choices":[{"delta":{"content":"ill"},"index":0}]}
...
data: {"choices":[{"delta":{},"finish_reason":"stop","index":0}]}
data: [DONE]

Each delta.content is a fragment, usually one token (not always a whole word). Concatenate them to build the message. Many runtimes can also include a final usage chunk if you pass stream_options: {"include_usage": true}. Streaming matters for user experience: a 10-second generation feels instant when text starts appearing after half a second.

#JavaScript

import OpenAI from "openai";
const client = new OpenAI({ baseURL: "http://localhost:11434/v1", apiKey: "ollama" });
const r = await client.chat.completions.create({
  model: "llama3.1:8b",
  messages: [{ role: "user", content: "Hello" }],
});
console.log(r.choices[0].message.content);

Do not call a local server directly from browser code on a public page: it would need CORS and would expose the endpoint. Put your own backend in between.

#Embeddings

r = client.embeddings.create(model="nomic-embed-text", input=["first text", "second text"])
vectors = [d.embedding for d in r.data]

Use a dedicated embedding model, not a chat model (local RAG). Batch several inputs per call for speed.

#Listing models and checking the connection

curl http://localhost:8080/v1/models

If this returns JSON, the server is reachable and OpenAI-compatible. It is also the first thing to try when a client reports "connection error".

#Parameters outside the OpenAI schema

Local runtimes support extra sampling controls (min_p, top_k, repeat_penalty, mirostat, grammars). Pass them through the SDK with extra_body:

client.chat.completions.create(
    model="local", messages=msgs,
    extra_body={"min_p": 0.05, "top_k": 40, "repeat_penalty": 1.1},
)

Unknown fields are usually ignored silently, so a typo or an unsupported option does nothing and you will not be told. Verify by observing behaviour.

#Compatibility notes

• The model field: llama.cpp (single model) often ignores it; Ollama and vLLM require the real name.
• Advanced features (tools, JSON schema, logprobs, n > 1, vision inputs) vary by runtime and version: test them (structured output, function calling).
• Tool calling needs the right template or parser on the server (--jinja in llama.cpp, a parser flag in vLLM).
• Error shapes differ: some servers return OpenAI-style error objects, others plain text.
• Rate-limit headers and usage details are usually absent.
• Local servers have no auth by default. Bind to 127.0.0.1 or add an API key before exposing them (production lab).

#Errors, timeouts and retries

from openai import OpenAI, APIConnectionError, APITimeoutError, APIStatusError
import time

client = OpenAI(base_url="http://localhost:8080/v1", api_key="x", timeout=120, max_retries=0)

def chat(msgs, tries=3):
    for i in range(tries):
        try:
            return client.chat.completions.create(model="local", messages=msgs, max_tokens=300)
        except (APIConnectionError, APITimeoutError):
            time.sleep(2 ** i)                         # backoff
        except APIStatusError as e:
            if e.status_code in (429, 503):             # busy: queue full or overloaded
                time.sleep(2 ** i); continue
            raise
    raise RuntimeError("server unavailable")

Guidelines: always set a timeout and a max_tokens; retry connection errors and 429/503 with backoff and a small cap; do not retry after a stream has started producing output; log finish_reason and token counts. See reliability patterns for the server side of this conversation.

#Keep a thin adapter

Differences between runtimes are small but real. Isolate them in one place:

import os
from openai import OpenAI

PROVIDERS = {
    "ollama":   dict(base_url="http://localhost:11434/v1", model="llama3.1:8b"),
    "llamacpp": dict(base_url="http://localhost:8080/v1",  model="local"),
    "vllm":     dict(base_url="http://localhost:8000/v1",  model="Qwen/Qwen2.5-7B-Instruct"),
}
cfg = PROVIDERS[os.environ.get("LLM_PROVIDER", "ollama")]
client = OpenAI(base_url=cfg["base_url"], api_key=os.environ.get("LLM_API_KEY", "x"))

def complete(messages, **kw):
    return client.chat.completions.create(model=cfg["model"], messages=messages, **kw)

Your application calls complete(...), and switching runtimes is an environment variable.

#Common mistakes

1. No max_tokens, so a runaway generation holds a slot for minutes.
2. No timeout on the client.
3. Assuming every OpenAI feature works on every local server.
4. Calling the server from browser code or exposing it without auth.
5. Ignoring finish_reason: "length", which means the answer was cut off.
6. Hard-coding the model name instead of configuring it.

#Exercise

Your app streams a reply and sometimes the text ends mid-sentence.

1. What field tells you why?
2. What are two fixes?

Answers:

1. finish_reason on the final chunk. A value of length means generation stopped at max_tokens (or at the context limit).
2. Raise max_tokens (and make sure the context window has room for prompt plus reply), and/or instruct the model to be shorter. Optionally detect length and offer a "continue" action.