The real cost of "just add RAG"

A data scientist in meeting with architects, engineers and product managers says "we'll just add RAG," and everyone settles and agrees. The model will read our docs and answer questions. It sounds like a feature you could ship by the next sprint.

While this is achievable for next meetings demo, the final product takes a lot longer. My view: RAG is cheap to demo and expensive to own, and nearly all of the cost sits in the parts the demo skips.

The demo and the product

A RAG demo takes a day because the hard parts are optional. You split some documents into chunks, embed them, put them in a vector store, then paste the top few matches into a prompt. You might go ahead and ask it five questions you already know the answers to, it gets four right, and everyone is impressed and started applauding.
In reality, a reliable version takes a more than a quarter because it has to work on questions you didn't pick, over documents you didn't clean, for users who aren't allowed to see everything.

Where the cost hides

The first cost is keeping content fresh. Loading documents once is a script, while keeping them up to date is a system. Pages get edited, PDFs get replaced, whole wiki spaces get archived, and each change has to reach the index or the assistant keeps quoting last year's policy. Someone has to own that pipeline and notice when it stops without an error.

Permissions come next, and they're the cost most likely to end up in an incident review. Your source systems know who can read which document. A chunk in a vector store knows nothing unless you copy the access rules onto it and filter on them for every query. Get that wrong and the assistant will happily summarise the salary spreadsheet for whoever asks. The demo indexed everything under one account, so the question never came up.

Then there are stale and contradicting documents. Plenty of companies have two or three versions of the travel policy, and retrieval has no idea which one is current. It ranks by similarity, so the old draft that happens to use the user's wording wins. The model then blends two versions into one confident answer that matches neither. Messy documentation goes in, and the assistant repeats the mess back in fluent sentences.

Evaluation is the cost teams skip most often, and the one I'd fund first. How do you know the answers are right? "It looked good in the demo" doesn't count. You need a set of real questions with known correct answers, and you need to check two things separately: whether retrieval brought back the right passages, and whether the model used them faithfully. Without that, every change to chunk size, prompt or model is a guess, and you find out it was a bad guess from a user.

Last is the running cost, which never ends. Switch embedding models and you re-embed the whole corpus, because vectors from different models can't be compared. Upgrade the chat model and the answers shift in ways your eval set has to catch. Every query pays for an embedding, a search and a prompt stuffed with retrieved text, so the cost per question grows with every chunk you add to the context.

When it's worth it

RAG earns its cost in a specific situation. The corpus is too big to fit in a prompt and changes too often to curate by hand. People ask in their own words, which don't match the terms in the documents, so keyword search misses. And they want an answer with a source they can click, rather than a list of ten links to read.

Internal support over a large, well maintained knowledge base fits that description. So does search across contracts or research papers, where the reader needs the exact wording and wants to check the source.

Cheaper things to try first

Most of the time I'd try these before building a pipeline.

Start with better search. A lot of "we need RAG" turns out to mean "our search is bad." Good titles, synonyms and a decent keyword index fix more than people expect, and the result is a link the user can check for themselves.

If the assistant would give wrong answers because the documents are wrong, fix the documents. Deleting the old travel policy is cheaper than teaching a retriever to ignore it.

Look at what people actually ask. If a few dozen questions cover most of the traffic, have someone who knows the answers write them down as a curated FAQ. It's boring, and it stays correct as long as someone owns it.

And check whether you need retrieval at all. If everything relevant fits in the model's context window, put it all in the prompt. Prompt caching makes the repeated part cheaper, and you've removed a whole class of bugs where the system retrieved the wrong chunk.

My rule of thumb

I say yes to RAG when two things exist before anyone writes code: twenty real questions with the correct answers written down, and a named person who owns the documents those answers come from.

The questions become the eval set. The owner keeps the content fresh, decides which policy is current, and knows who's allowed to read what. If nobody can produce either one, start there. It costs less than the pipeline, and the pipeline would need it anyway.