Private RAG on your own hardware with Ollama & LLaMA 3

Some documents are never allowed near a third-party API — medical records, legal discovery, internal financials. For those, "just call OpenAI" is a non-starter. The good news is that a fully local retrieval-augmented-generation stack is genuinely shippable now. The honest news is that it costs you some quality, and you should know exactly where.

The whole pipeline runs on one machine

RAG has three moving parts, and all three have local options. Embeddings turn text into vectors — a small sentence-transformer runs fine on CPU. A vector store holds those vectors; for a single box, an embedded store like a local pgvector or even an in-memory index is plenty. And generation is a local model served by Ollama. Nothing crosses the network.

// rag/answer.ts — retrieve locally, generate locally, never leave the box.
import { embed, search } from "./store";
import { ollama } from "./ollama";

export async function answer(question: string) {
  const queryVec = await embed(question);          // local embeddings
  const chunks = await search(queryVec, { topK: 5 });

  const context = chunks.map((c) => c.text).join("\n---\n");
  return ollama.generate({
    model: "llama3:8b",
    prompt: `Answer using only the context.\n\n${context}\n\nQ: ${question}`,
  });
}

That's the entire hot path: embed the question, pull the top-k relevant chunks, stuff them into the prompt, and let a local LLaMA 3 generate the answer. The chunks are the grounding — the model isn't recalling facts, it's reading the few passages you handed it.

Where local genuinely wins

• Data never leaves. This is the whole point. No DPA negotiation, no "is this PII in a log somewhere" anxiety, no per-token bill that scales with usage.
• Cost is fixed. Once the hardware is paid for, a thousand queries cost the same as ten. For high-volume internal tools, that flips the economics entirely.
• Latency is predictable. No rate limits, no noisy-neighbor slowdowns. A warm local model answers in a steady beat.

Where the cloud still wins — be honest about it

An 8B local model is not a frontier model, and pretending otherwise burns trust. On genuinely hard reasoning — multi-step synthesis, subtle instructions, long-context coherence — a hosted frontier model is still meaningfully better. Retrieval quality matters more than model size here, but it doesn't erase the gap.

Make retrieval the thing you tune

The single biggest quality lever in a local stack isn't the model — it's what you put in front of it. Chunk documents thoughtfully (semantic boundaries beat fixed token windows), tune your top-k, and add a re-ranking pass if recall is noisy. A mediocre model with excellent retrieval beats a great model fed irrelevant context, every time.

Local RAG isn't about matching the cloud everywhere. It's about being good enough on the 90% of questions where keeping data in-house is non-negotiable.

Start with the local pipeline, measure it against real questions from real users, and only reach for the cloud on the queries that demonstrably need it — assuming the data policy even allows it. For a surprising number of internal tools, it never has to.