Building a Zero-Cost Local RAG Pipeline using Python and Ollama
Retrieval-Augmented Generation (RAG) lets you query your custom private documents using large language models (LLMs) without sending sensitive data to third-party APIs. In this tutorial, we will build an entirely local
Retrieval-Augmented Generation (RAG) lets you query your custom private documents using large language models (LLMs) without sending sensitive data to third-party APIs.
In this tutorial, we will build an entirely local, zero-cost RAG pipeline using Python, Ollama, and ChromaDB.
Prerequisites
- Python 3.9+ installed
-
Ollama installed locally (
ollama pull mistral)
Step 1: Install Required Dependencies
Install ChromaDB (vector database) and the Ollama Python client:
pip install chromadb ollama
Step 2: Initialize Vector Database & Embeddings
We will store our knowledge base documents as vector embeddings in ChromaDB:
import chromadb
import ollama
# Initialize local persistent vector store
client = chromadb.PersistentClient(path="./rag_db")
collection = client.get_or_create_collection(name="local_knowledge")
# Sample document knowledge base
documents = [
"Our enterprise platform policy requires JWT rotation every 24 hours.",
"System maintenance occurs every Sunday from 2 AM to 4 AM UTC.",
"Database read replicas are deployed across US-East and EU-Central regions."
]
# Generate embeddings locally via Ollama and store in ChromaDB
for i, doc in enumerate(documents):
response = ollama.embeddings(model="mistral", prompt=doc)
collection.add(
ids=[f"doc_{i}"],
embeddings=[response["embedding"]],
documents=[doc]
)
print("Knowledge base successfully embedded and indexed locally!")
Step 3: Querying the Local RAG Pipeline
Now, retrieve context relevant to a user prompt and generate an accurate response:
def query_rag(user_query: str):
# 1. Embed the incoming question
query_embedding = ollama.embeddings(model="mistral", prompt=user_query)["embedding"]
# 2. Query top 2 matching document chunks
results = collection.query(
query_embeddings=[query_embedding],
n_results=2
)
retrieved_context = " ".join(results["documents"][0])
# 3. Pass context and question to the local LLM
prompt = f"""
Answer the question based ONLY on the context below:
Context: {retrieved_context}
Question: {user_query}
"""
response = ollama.generate(model="mistral", prompt=prompt)
return response["response"]
# Example execution
answer = query_rag("When does system maintenance happen?")
print("AI Response:\n", answer)
Conclusion
By pairing Ollama with ChromaDB, you get a production-ready, local RAG pipeline with zero API costs and 100% data privacy.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.