Local RAG from scratch

Models do not know your private documents. Retrieval-Augmented Generation (RAG) fetches relevant passages at question time and puts them in the prompt, so the model answers from your text rather than from memory. Done locally, none of your documents leave your machine. It is the most useful pattern in applied LLM work, and the one most often done badly.

docs -> chunks -> embeddings -> vector index
question -> embedding -> top-k similar chunks -> prompt -> local LLM -> answer

#1. Choose models

• Embedding model: small and fast, such as nomic-embed-text or bge-small, served by Ollama or llama.cpp (--embeddings). Embedding models are not chat models: they turn text into a vector so that similar meanings are close together.
• Chat model: any instruct model with enough context for k chunks plus the question and answer (context budget).
• Optionally a reranker (a small cross-encoder) to re-order the top candidates.

Embedding conventions matter. Some models are trained with task prefixes: nomic-embed-text expects search_document: before document text and search_query: before queries. Others (such as the BGE family) suggest an instruction string for queries. Read the model card, and use the same conventions every time. Always use the same embedding model for indexing and querying, and re-index if you change it.

#2. Chunking

Split into 200 to 500 token passages with 10 to 20 percent overlap, preferably along headings or paragraphs. Chunks that are too big dilute meaning (the embedding averages many topics, and you waste prompt space); too small lose context (a chunk that says "it expires after 30 days" is useless without knowing what "it" is).

Practical improvements:

• Split on structure first (headings, paragraphs, list items), then by size.
• Prepend context to each chunk: the document title and section heading, so the chunk is self-describing.
• Keep metadata with each chunk: source file, page or section, date and access group.
• Treat tables and code carefully: keep a table together or convert rows into sentences.
• Strip boilerplate (headers, footers, navigation) that pollutes every chunk.

#3. The code

A compact version first:

import numpy as np
from openai import OpenAI

emb = OpenAI(base_url="http://localhost:11434/v1", api_key="x")
llm = OpenAI(base_url="http://localhost:11434/v1", api_key="x")

def embed(texts):
    r = emb.embeddings.create(model="nomic-embed-text", input=texts)
    v = np.array([d.embedding for d in r.data], dtype="float32")
    return v / np.linalg.norm(v, axis=1, keepdims=True)

def chunk(text, size=1200, overlap=200):
    return [text[i:i + size] for i in range(0, len(text), size - overlap)]

docs = open("handbook.txt").read()
chunks = chunk(docs)
index = embed(chunks)                          # (n, d) normalized

def ask(q, k=4):
    scores = index @ embed([q])[0]             # cosine similarity
    top = [chunks[i] for i in np.argsort(-scores)[:k]]
    context = "\n\n---\n\n".join(top)
    msg = f"Answer ONLY from the context. If missing, say you don't know.\n\nContext:\n{context}\n\nQuestion: {q}"
    r = llm.chat.completions.create(model="llama3.1:8b", temperature=0.1,
                                    messages=[{"role": "user", "content": msg}])
    return r.choices[0].message.content

print(ask("What is the vacation policy?"))

Because the vectors are normalised, the dot product equals cosine similarity. For more than about 100,000 chunks, move from NumPy to a vector store (SQLite-vec, Chroma, Qdrant, pgvector).

#A more complete version

Persistence, metadata, prefixes, citations and an abstention-friendly prompt:

import json, os, re, numpy as np
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="x")
EMBED, CHAT = "nomic-embed-text", "llama3.1:8b"

def embed(texts, kind):                        # kind: "document" or "query"
    texts = [f"search_{kind}: {t}" for t in texts]
    out = []
    for i in range(0, len(texts), 32):         # batch the calls
        r = client.embeddings.create(model=EMBED, input=texts[i:i+32])
        out += [d.embedding for d in r.data]
    v = np.array(out, dtype="float32")
    return v / np.linalg.norm(v, axis=1, keepdims=True)

def split(path, size=1200, overlap=200):
    text = open(path).read()
    paras, chunks, cur = re.split(r"\n\s*\n", text), [], ""
    for p in paras:
        if len(cur) + len(p) > size and cur:
            chunks.append(cur); cur = cur[-overlap:] + "\n\n" + p
        else:
            cur = (cur + "\n\n" + p).strip()
    if cur: chunks.append(cur)
    return [{"id": f"{os.path.basename(path)}#{i}", "source": path, "text": c} for i, c in enumerate(chunks)]

def build(paths, out="index"):
    items = [c for p in paths for c in split(p)]
    vecs = embed([c["text"] for c in items], "document")
    np.save(f"{out}.npy", vecs); json.dump(items, open(f"{out}.json", "w"))

def load(out="index"):
    return np.load(f"{out}.npy"), json.load(open(f"{out}.json"))

def retrieve(q, vecs, items, k=4):
    scores = vecs @ embed([q], "query")[0]
    order = np.argsort(-scores)[:k]
    return [(items[i], float(scores[i])) for i in order]

PROMPT = """Answer the question using ONLY the sources below. Cite sources like [id].
If the sources do not contain the answer, reply exactly: I don't know based on the provided documents.

{sources}

Question: {q}"""

def answer(q, vecs, items, k=4):
    hits = retrieve(q, vecs, items, k)
    sources = "\n\n".join(f"[{c['id']}]\n{c['text']}" for c, _ in hits)
    r = client.chat.completions.create(model=CHAT, temperature=0.1, max_tokens=400,
        messages=[{"role": "user", "content": PROMPT.format(sources=sources, q=q)}])
    return r.choices[0].message.content, hits

if __name__ == "__main__":
    build(["handbook.txt", "faq.txt"])
    vecs, items = load()
    text, hits = answer("What is the vacation policy?", vecs, items)
    print(text); print([h[0]["id"] for h in hits])

Why this is better: the index is built once and reloaded (no re-embedding on every run), chunks have IDs for citations, the prompt gives an explicit abstention phrase, and the document and query prefixes follow the embedding model's convention.

#How big is the index?

vector memory = chunks × dimensions × bytes per number
100,000 × 768 × 4 bytes = 307 MB     (float32)

NumPy handles hundreds of thousands of chunks comfortably on a laptop. A vector store adds persistence, filtering by metadata, approximate nearest-neighbour search (so queries stay fast at millions of vectors) and updates without rebuilding. Quantising vectors to 8-bit or binary shrinks them further with small recall loss.

#Common failure modes

#Improve it

1. Hybrid search (BM25 keywords plus vectors) usually gives a big win for names, codes and numbers. Combine the two rankings with reciprocal rank fusion:

def rrf(rankings, k=60):
    score = {}
    for ranking in rankings:                    # each is a list of ids, best first
        for rank, i in enumerate(ranking):
            score[i] = score.get(i, 0) + 1 / (k + rank + 1)
    return sorted(score, key=score.get, reverse=True)

2. Re-ranker: retrieve 20 candidates cheaply, then re-order them with a cross-encoder and keep the top 4 to 5. This raises precision at small k.
3. Query rewriting: turn a follow-up ("and for contractors?") into a standalone question using the chat history.
4. Citations: return chunk IDs and show the sources, so users can verify and you can debug.
5. Metadata filters: restrict by product, date or document type.
6. Permissions: enforce access control in the retrieval step, never in the prompt (data leakage).
7. An evaluation set: write 30 to 100 questions with the expected answer and the chunk that contains it, and measure Recall@k and answer quality so changes are measurable (testing RAG, measuring quality).
8. Abstention tests: include questions that are not answerable from the documents.

#Security notes

Retrieved text goes into the prompt, so documents are an injection channel: a chunk that says "ignore previous instructions" can steer the model (prompt injection). Treat the corpus as untrusted unless you control who writes to it, label sources as data in the prompt, and keep the answering model away from powerful tools.

#Common mistakes

1. Tuning the prompt before checking retrieval.
2. Chunks of arbitrary character length that split sentences and tables.
3. Different embedding models or prefix conventions at index and query time.
4. Stuffing too many chunks into the prompt: more is not better (context).
5. No way to say "I don't know", so the model guesses.
6. Never measuring: "it seems fine" on five questions.

#Exercise

You index a 300-page manual: about 150,000 words.

1. Roughly how many tokens is that, and how many chunks of 400 tokens with 15 percent overlap?
2. How much memory do the vectors need with a 768-dimension float32 embedding?
3. A user asks for a part number (A-4471-B) and gets irrelevant passages. What do you change?

Answers:

1. 150,000 words × 1.33 ≈ 200,000 tokens. Each chunk advances by about 340 new tokens (400 minus 15 percent overlap), so about 590 chunks.
2. 590 × 768 × 4 bytes ≈ 1.8 MB: trivial. (Even 100 manuals would be under 200 MB.)
3. Add keyword (BM25) search alongside vectors and fuse the rankings, since part numbers are rare tokens that embeddings represent poorly. Also consider keeping the part number in metadata for exact lookup.