Multi-tenant RAG: stop one customer's documents leaking into another's answers

Retrieval-augmented generation (RAG) is the standard way to let people "chat with their documents": split the documents into chunks, embed them, store them in a vector database, and at question time retrieve the closest chunks and hand them to an LLM as context.

Nearly every tutorial stops there. That's fine for one person's notes. It's a problem the moment two customers share the same deployment, because the vector search has no idea whose documents it is searching. Ask "what's our pricing?" and the most similar chunk might belong to someone else.

I built Enterprise RAG Engine, an open-source FastAPI + LangChain service, to show what that boundary looks like in code. Here's how it works, a demo, and the bug I found in my own isolation logic while writing this post.

The idea: tag on write, filter on read

Isolation has to happen in the retrieval step, not after it. If you retrieve across everyone and filter afterwards, you've already handed other tenants' text to your process, and one forgotten filter later it's in an answer.

So every chunk is stamped with its tenant when it's stored:

def add_documents(documents, tenant_id=None):
    store = get_vector_store()
    for doc in documents:
        doc.metadata = {**doc.metadata, "tenant_id": tenant_id or "default"}
    return len(store.add_documents(documents))

And every search is restricted to one tenant by the vector store itself, so other tenants' chunks never come back:

def similarity_search(query, k, tenant_id=None):
    store = get_vector_store()
    filter_ = {"tenant_id": tenant_id or "default"}
    return store.similarity_search(query, k=k, filter=filter_)

Chroma and Pinecone both support metadata filters, so this works the same on either backend.

Grounded answers, with receipts

Isolation keeps the wrong documents out. The prompt keeps the model honest about the right ones:

You are an enterprise knowledge assistant. Answer the user's question using
ONLY the provided context. If the context does not contain the answer, say you
don't have enough information — never invent facts. Be concise and cite which
source(s) you drew from by their number, e.g. [1], [2].

Each response returns the source snippets next to the answer, so a user (or an auditor) can check the claim instead of trusting it.

The demo

Two companies upload one document each: Acme a leave policy, Globex a pricing guide. Then both ask the same question: "How many days of paid annual leave do I get?" These are the real responses from the engine running locally on Llama 3.2:

Acme

According to [1], full-time employees receive 24 days of paid annual leave per calendar year.

sources: acme-leave-policy.md

Globex

I don't have enough information to answer your question. The provided context only discusses pricing plans for Globex Ltd's services, but does not mention anything about paid annual leave or employee benefits. [1]

sources: globex-pricing.md

Globex never sees Acme's policy. It searches only its own pricing guide, finds nothing relevant, and says so rather than guessing.

▶ Watch the 2-minute demo

The bug I found in my own code

While writing this post, I wrote a test for the obvious edge case: what happens when a request doesn't send a tenant at all?

The search function originally looked like this:

filter_ = {"tenant_id": tenant_id} if tenant_id else None

No tenant meant no filter, which meant searching every tenant's documents. Ingest already treated a missing tenant as "default"; search didn't. The test failed and returned Acme's, Globex's and the default documents together.

The fix is the one-liner you saw above, tenant_id or "default", plus a test so it can't come back:

def test_missing_tenant_only_sees_the_default_tenant(store):
    docs = vector_store.similarity_search("leave days", k=5, tenant_id=None)
    assert {d.metadata["source"] for d in docs} == {"default.md"}

The lesson generalises: a missing scope should fall back to the narrowest scope, never the widest. Optional parameters are where that breaks.

The rest of the guard rails

• API-key auth on every request (X-API-Key), with keys read only from environment variables.
• Rate limiting per key (an in-memory sliding window; swap in Redis when you run more than one instance).
• Keys bound to tenants: API_KEY_TENANTS maps each API key to the tenants it may use. Asking for any other tenant returns 403 before a search runs, and a key bound to one tenant doesn't need to send tenant_id at all.
• Upload validation: only .txt and .md, 5 MB maximum, valid UTF-8.
• No secrets in git, enforced by secret scanning in CI.

What it doesn't do yet

Honesty matters more than a feature list, so:

• No per-user auth. Keys can be bound to tenants, but there's no JWT layer for individual end users yet; that's on the roadmap.
• The rate limiter is single-instance.
• Ingestion is text and Markdown only for now.

Run it yourself, free

No paid LLM API key needed. It runs a local model through Ollama:

git clone https://github.com/CyberAli-eng/enterprise-rag-engine.git
cd enterprise-rag-engine
cp .env.example .env
# in .env: API_KEYS=<your secret>, LLM_PROVIDER=ollama,
#          LLM_MODEL=llama3.2:3b, OLLAMA_BASE_URL=http://ollama:11434/v1
docker compose --profile local up --build

Prefer a hosted model? Set LLM_PROVIDER to openai, groq or together instead. Chroma or Pinecone is one more variable.

The code is MIT-licensed: github.com/CyberAli-eng/enterprise-rag-engine. Issues and PRs are welcome, especially for the Redis rate limiter and PDF ingestion.