You can run a full RAG pipeline on your own machine with no API key and nothing leaving
the laptop. Ollama serves both the embedding model
(nomic-embed-text) and a local chat model (llama3.1:8b) on
localhost; sqlite-vec stores the vectors in a single SQLite file.
The flow is: chunk your documents, embed each chunk through Ollama locally, store the vectors in a
vec0 table, then at query time embed the question, retrieve the nearest chunks, and
hand them to the local model for a cited answer. No OpenAI, no Pinecone, no LangChain.
Key Takeaways
- Nothing leaves the machine. — Ollama runs the embedding model and the chat model on localhost. Your documents, your questions, and the generated answers all stay on disk and in local memory. There is no API key because there is no remote API.
- The vector store is a file, not a service. — sqlite-vec registers a vec0 virtual table inside an ordinary SQLite database. The "vector database" is a file you can copy, back up, and delete. There is nothing else running and nothing to host.
- No framework did the orchestration. — No LangChain, no agent framework. Chunk, embed, store, retrieve, prompt: each step is a few lines of plain code you can read end to end. The whole thing fits in your head when something breaks.
- The trade is speed for privacy. — Local embeddings are slightly less precise than the best hosted models, and an 8B model on CPU is slow. For a corpus you cannot send to a third party, that is a trade worth making — and on a GPU it stops being slow.
Why run RAG locally at all?
Most RAG tutorials assume two hosted calls — a hosted embedding model and a hosted chat model — and so they assume an API key and an account and a per-token bill. That is a fine default when your corpus is public. It is the wrong default the moment the documents are the sensitive thing: legal files, medical notes, internal company knowledge, anything you cannot upload to a third party without a conversation with someone in compliance. For that case the relevant question is not which model scores half a point higher on a benchmark. It is whether the data is allowed to leave your control at all, and the cleanest answer is to make sure it never does.
I built the hosted version of this first. Ask
Tom answers questions from my published writing using OpenAI embeddings and a small OpenAI model,
with the vectors in a single SQLite file. This post is the same pipeline with the two model calls
pointed at localhost instead of the cloud. The shape does not change. What changes is
that you can pull the network cable and it keeps working — which, for a private corpus, is the entire
point.
What is the full local stack?
Three pieces do the work, and none of them is a hosted service. Ollama runs the models; sqlite-vec is the vector store; the orchestration is plain code. The only thing that touches a network is the one-time model download.
| Stage | What I use (local) | Why this and not the hosted alternative |
|---|---|---|
| Embeddings | Ollama + nomic-embed-text (768-dim) | Runs on localhost with no key; purpose-built for retrieval and accurate enough for prose. The hosted model is marginally better but leaves the machine. |
| Vector store | sqlite-vec (a vec0 virtual table) | The vectors live in one SQLite file. No vector database to host, bill, or back up — and nothing to authenticate to. |
| Generation | Ollama + llama3.1:8b | The retrieved chunks do the heavy lifting, so a mid-size local model is enough. No remote chat API, no per-token cost. |
| Orchestration | Plain Python, no framework | Chunk, embed, store, retrieve, prompt are one function each. A framework would hide steps that are already small. |
How do I set up Ollama for embeddings and generation?
Install Ollama, then pull one embedding model and one chat model. After this step you can work entirely offline; the pull is the only moment anything is downloaded.
# 1. Install Ollama (macOS/Linux), then start the local server
# macOS: download the app; Linux: curl -fsSL https://ollama.com/install.sh | sh
# The server listens on http://127.0.0.1:11434
# 2. Pull the two models you need (one-time network use)
ollama pull nomic-embed-text # embeddings, 768-dim, retrieval-tuned
ollama pull llama3.1:8b # local chat model for generation
# 3. Install the Python pieces
pip install sqlite-vec requests
That is the whole dependency list. sqlite-vec gives you the vector table;
requests talks to the local Ollama server. There is no SDK that phones home and no key
to set. From here, every step runs against 127.0.0.1.
What is the build sequence, start to finish?
Six steps. The first four build the index once; the last two are the live request, and both of those also run locally.
- Collect the corpus. Read your documents — Markdown, text, exported notes — keeping a source path or URL with each one so answers can cite back.
- Chunk. Split each document into overlapping word windows so a passage near a boundary survives intact in at least one chunk.
- Embed. Send each chunk to Ollama's local
/api/embeddingsendpoint and get back a 768-dimension vector. - Store. Insert each vector, plus its text and source, into a sqlite-vec
vec0table in one SQLite file. - Retrieve. At query time, embed the question locally and run a similarity search to pull the closest chunks.
- Generate. Hand the top chunks and the question to the local chat model and return the answer with citations.
How do I generate embeddings locally with Ollama?
The part people skip when chunking is overlap. If you cut documents into hard, non-overlapping blocks, a sentence that answers the exact question gets split across two chunks and retrieved as neither. Overlapping windows fix that for almost nothing. Each chunk then goes to Ollama's local embeddings endpoint — note the URL is a loopback address, so the request never leaves the machine:
import requests
OLLAMA = "http://127.0.0.1:11434"
def chunk(text, size=220, overlap=40):
words = text.split()
step = size - overlap
for start in range(0, len(words), step):
window = words[start:start + size]
if window:
yield " ".join(window)
def embed(text):
# local call — nothing leaves the machine, no API key
r = requests.post(f"{OLLAMA}/api/embeddings", json={
"model": "nomic-embed-text",
"prompt": text,
})
r.raise_for_status()
return r.json()["embedding"] # 768-dim vector size and overlap are the only two knobs, and a couple hundred words with a
small overlap is a sane default for prose. The embedding dimension here is 768 because that is what
nomic-embed-text returns; that number has to match the table you declare next, and it
has to stay fixed between ingest and query or the search returns nonsense.
How does sqlite-vec store and search the vectors?
This is the piece that replaces a hosted vector database. You load the extension, declare a
vec0 virtual table with the embedding dimension, and from then on it is just SQL.
Storage is an INSERT; search is a SELECT ordered by distance:
import sqlite3, sqlite_vec
from sqlite_vec import serialize_float32
db = sqlite3.connect("local-rag.db")
db.enable_load_extension(True)
sqlite_vec.load(db)
db.enable_load_extension(False)
# the entire "vector database": one virtual table in one file
db.execute("""
CREATE VIRTUAL TABLE chunks USING vec0(
embedding float[768],
source TEXT,
body TEXT
)
""")
# ingest: one row per chunk
for source, text in documents:
for piece in chunk(text):
vec = embed(piece)
db.execute(
"INSERT INTO chunks(embedding, source, body) VALUES (?, ?, ?)",
[serialize_float32(vec), source, piece],
)
db.commit()
# query: nearest chunks to the embedded question
q_vec = embed(question)
rows = db.execute("""
SELECT source, body, distance
FROM chunks
WHERE embedding MATCH ?
ORDER BY distance
LIMIT 6
""", [serialize_float32(q_vec)]).fetchall()
That LIMIT 6 is the top-k handed to the model. The whole store is the
local-rag.db file: you can copy it, back it up with cp, or delete it to
start over. There is no migration and no service, because there is no vector database — just a file
that happens to know how to do nearest-neighbour search.
How do I turn retrieved chunks into a cited answer locally?
The retrieved chunks go into the prompt, the question follows, and the local model answers constrained to what it was given. Because retrieval already narrowed the context to the relevant passages, the generation model does not have to be large — it has to follow instructions and stay inside the context. The call goes to the same local Ollama server:
def answer(question, rows):
context = "\n\n".join(f"[{src}] {body}" for src, body, _ in rows)
prompt = (
"Answer using only the context below. Cite the [source] you used.\n\n"
f"Context:\n{context}\n\nQuestion: {question}"
)
r = requests.post(f"{OLLAMA}/api/generate", json={
"model": "llama3.1:8b",
"prompt": prompt,
"stream": False,
})
r.raise_for_status()
return r.json()["response"]
Every line of that runs on 127.0.0.1. The question is embedded locally, the search runs
against a local file, and the answer is generated by a local model. At no point is there a request to
a third-party API, which is the whole reason to build it this way. The system prompt — here folded
into the instruction — is where you would set tone, persona, or refusal rules, exactly as you would
in a hosted build.
What does going local actually cost?
Two things, and it is worth being honest about both. The first is embedding quality:
nomic-embed-text is good, but the best hosted embedding models are a notch more precise,
so on a hard retrieval query the local store may occasionally miss a passage the hosted one would
catch. The second is speed: an 8B model generating a paragraph on CPU is measured in seconds, and
embedding a large corpus is a coffee break rather than an instant. A GPU buys most of that back; on a
capable machine the per-query latency feels like any local LLM chat.
That trade is the through-line. Every choice here — the local embedding model instead of the hosted one, the file instead of the service, the framework left out — buys the same thing: the documents never leave your control. For where this kind of retrieval layer earns its keep in the first place, I made the strategy argument in Build a Second Brain That Answers Back on ctaio.dev. If you would rather have the framework conveniences than read plain code, the LlamaIndex version of this same local stack trades a little legibility for a lot of scaffolding. And if your corpus is public anyway, the hosted Ask Tom build is the easier ride. Pick by the data, not the fashion.
FAQ
Can you really run RAG with no API key at all?
Yes. The only reason most RAG tutorials need an API key is that they call a hosted embedding model and a hosted chat model. Replace both with models running under Ollama on localhost and the keys disappear, because there is no remote service to authenticate against. Ollama exposes an HTTP server on 127.0.0.1:11434 that speaks to models you have pulled to disk; your code talks to that local port for both embeddings and generation. The vector store was never a hosted service in this design either — sqlite-vec keeps the vectors in a file. The single moment anything touches the network is the one-time ollama pull to download a model; after that you can disconnect entirely and the pipeline keeps working.
Which Ollama models should I use for embeddings and generation?
For embeddings, nomic-embed-text is the sensible default: it is small, fast, purpose-built for retrieval, and produces 768-dimension vectors that are more than good enough for prose. For generation, a mid-size instruct model such as llama3.1:8b is a reasonable starting point — large enough to write a coherent grounded answer, small enough to run on a laptop with a GPU or, more slowly, on CPU. The retrieved chunks do most of the work, so you do not need a frontier-scale model here; you need one that follows instructions and stays inside the context you give it. If answers feel thin, step up to a larger local model and trade speed for quality. Pin whichever models you choose and treat the version as part of your setup, because embedding dimensions must match between ingest and query.
How does this compare to the OpenAI version of the same pipeline?
Architecturally it is identical — chunk, embed, store in sqlite-vec, retrieve, generate — which is the point: I built the hosted version first and this is the same shape with the two model calls pointed at localhost instead of OpenAI. The differences are practical, not structural. Hosted embeddings (text-embedding-3-small at 1536 dimensions) are marginally more precise and far faster to batch; gpt-4.1-mini generates more fluent answers than an 8B local model. In exchange, the local build sends nothing to a third party and costs nothing per call. Choose by the corpus: for public writing the hosted version is the easier ride, but for documents you cannot legally or sensibly upload, local is the only honest option.
Is local RAG fast enough to actually use?
On a machine with a capable GPU, yes — embedding a few thousand chunks takes minutes once, and per-query latency is dominated by the chat model generating tokens, which feels like any local LLM chat. On CPU only, it works but you will feel it: an 8B model generating a paragraph on CPU is measured in seconds, not milliseconds, and embedding a large corpus is a coffee break rather than an instant. The fixed costs (ingesting and embedding the corpus) happen once; the per-query cost is one small embedding plus one generation. If latency matters and you have no GPU, shrink the generation model or reduce how many chunks you retrieve. Speed is the honest price of keeping everything local, and a GPU buys most of it back.
When should I keep RAG local instead of using a cloud API?
Keep it local when the documents themselves are the sensitive asset: legal files, medical records, unreleased financials, internal company knowledge, anything under a confidentiality or data-residency obligation. In those cases the question is not which model is marginally better but whether the data is allowed to leave your control at all, and a fully local pipeline answers that cleanly because nothing is transmitted. Keep it cloud when the corpus is public or low-stakes and you want the best retrieval quality and the least operational fuss. The decision is about the data, not the technology — the stack is nearly the same either way, which is exactly why you can prototype on the cloud version and move to local without a rewrite.
Ready to Transform Your AI Strategy?
Get personalized guidance from someone who's led AI initiatives at Adidas, Sweetgreen, and 50+ Fortune 500 projects.
