How to Build a Local RAG Pipeline with Ollama and LlamaIndex

Thomas Prommer Technology Executive & AI Strategy Consultant Connect on LinkedIn

Published: June 16, 2026

Updated: June 16, 2026

You build a local RAG pipeline by running two models in Ollama — an embedding model (nomic-embed-text) and a generation model (llama3.1) — and letting LlamaIndex do the orchestration. LlamaIndex reads a folder of documents, chunks them, embeds each chunk through Ollama, and stores the vectors in a local index you persist to disk. At query time it embeds the question, retrieves the nearest chunks, and asks the local Llama model to answer from them. There is no API key and no data leaves the machine, because both the embeddings and the generation run on Ollama locally.

Why run RAG on your own machine at all?

There is a version of RAG that starts with an API key, a hosted embedding endpoint, and a managed vector database, and ships your documents to three vendors before it answers a single question. For private documents, that is the wrong default. I built the opposite: a retrieval pipeline where the embeddings, the vectors, and the answers all stay on one machine. Ollama runs the models, LlamaIndex runs the orchestration, and the only thing crossing a network is nothing.

The honest framing is a trade, not a free lunch. A local 8B-class model is slower than a frontier API and its answers have a lower ceiling. What you buy in return is that your data never leaves your control and every query costs you nothing but electricity. For a body of sensitive notes, contracts, or research, that is the trade I take every time. The rest of this post is the actual build, in the order the data flows through it.

What is the full stack?

Four pieces, and none of them is a hosted service. Ollama serves the models, LlamaIndex orchestrates, a local store holds the vectors, and your own documents are the corpus.

How do I install Ollama and pull the models?

Two commands of setup. Install Ollama, then pull the two models you need — one to embed, one to answer. After this, both run on localhost and nothing in the rest of this build touches a network:

# install Ollama (macOS / Linux), then pull the two models
ollama pull nomic-embed-text   # the embedding model
ollama pull llama3.1           # the generation model (8B)

# install the Python pieces
pip install llama-index \
            llama-index-llms-ollama \
            llama-index-embeddings-ollama

Ollama serves an HTTP API on http://localhost:11434 in the background. The two llama-index-*-ollama packages are the adapters that let LlamaIndex talk to it for embeddings and generation respectively.

What is the build sequence, start to finish?

Six steps: the first four build and persist the index, the last two are the live query. You run the build once, then query as often as you like against the persisted store.

1. Point Ollama at both models. Tell LlamaIndex to use the local embedding model and the local generation model — no API key anywhere.
2. Load the documents. Read a folder of files into LlamaIndex with its document reader.
3. Chunk and embed. LlamaIndex splits each document into nodes and embeds each one through Ollama with nomic-embed-text.
4. Persist the index. Write the vector index to a folder on disk so you never re-embed unless the documents change.
5. Retrieve. At query time, embed the question and pull the nearest nodes from the persisted index.
6. Generate. Hand the retrieved nodes and the question to the local llama3.1 model and return a grounded answer.

How do I wire Ollama into LlamaIndex?

The whole "no API key" property comes from these few lines. You set LlamaIndex's global embedding model and language model to the Ollama-backed ones, and from then on every step in the pipeline runs locally without you thinking about it:

from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.core import Settings

# both models served by local Ollama — no API key, no egress
Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")
Settings.llm = Ollama(model="llama3.1", request_timeout=120.0)

Setting these on Settings makes them the defaults for the whole pipeline. The request_timeout is generous on purpose: a local model on modest hardware can take a while to generate, and you would rather wait than have the call time out mid-answer.

How does LlamaIndex build and persist the index?

This is where the documents become a searchable index. You read a folder, build the vector index (LlamaIndex chunks and embeds in the process), and persist it to disk so the expensive embedding pass happens exactly once:

from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    StorageContext,
    load_index_from_storage,
)

PERSIST_DIR = "./storage"

# build once: read docs -> chunk -> embed via Ollama -> index
documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)
index.storage_context.persist(persist_dir=PERSIST_DIR)

# every run after: reload from disk instead of re-embedding
storage = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
index = load_index_from_storage(storage)

The first block is the slow one — embedding every chunk on your own hardware — and you pay it once. The second block is what every subsequent run does: load the persisted index in about a second. Build once, reload forever, rebuild only when the documents change.

How does a query turn into a grounded answer?

With the index loaded, querying is one call. LlamaIndex embeds the question with the same local model, retrieves the nearest nodes, assembles them into a prompt, and hands that to the local llama3.1 model to answer:

# retrieve the top chunks and answer from them, all locally
query_engine = index.as_query_engine(similarity_top_k=4)
response = query_engine.query(
    "What did the Q3 architecture review conclude about the data layer?"
)
print(response)            # the grounded answer
print(response.source_nodes)  # the chunks it used, so you can cite them

That similarity_top_k=4 is how many chunks get handed to the model — enough context to answer, few enough to keep the local prompt fast. The source_nodes are the retrieved chunks, which is how you cite the answer back to the documents it came from. Every part of that call — the embedding, the retrieval, the generation — ran on your machine. Nothing left.

What is the honest limitation?

Speed and the answer ceiling. A local 8B model is slower to generate than a frontier API and its answers do not reach as high — that is the cost of running everything on your own hardware, and it is a deliberate trade, not an oversight. You buy privacy and zero marginal cost, and you pay for it in latency and in the quality gap between an 8B local model and a frontier one. For private documents, that trade is the right one most of the time: a slightly slower, slightly less polished answer matters far less than the documents never leaving your control.

That trade is the through-line of the whole build. Every choice here — Ollama instead of a hosted endpoint, a local index instead of a managed vector database, an 8B model instead of a frontier one — is the same discipline: keep the data on the box and spend the quality budget only where the documents are not sensitive enough to need it. For why this kind of retrieval layer earns its keep in the first place, I made the strategy argument in Build a Second Brain That Answers Back on ctaio.dev. If you would rather host the vectors than run them fully local, the Ask Tom build on one SQLite file is the same idea with sqlite-vec doing the storage. And for the full case on what running retrieval entirely on your own hardware buys you, I laid it out in Run RAG Locally With No API Key.

FAQ