Dev.to AI 🤖 Ai 👁 0 📖 10 min read

How RAG Works: Embeddings, Vector Search, and Why Your AI App Needs Retrieval

A language model knows a lot, but it does not know your data. It has never seen your company's internal documents, your course notes, last week's policy update, or the contents of your database. It also has a training cu

A language model knows a lot, but it does not know your data. It has never seen your company's internal documents, your course notes, last week's policy update, or the contents of your database. It also has a training cutoff, so it can be out of date, and when it does not know something it sometimes produces a confident, fluent, wrong answer. Fine-tuning a model on your data is possible but slow, expensive, and awkward to keep current.

The dominant practical solution is Retrieval-Augmented Generation (RAG): instead of asking the model to remember your information, you look up the relevant pieces at question time and hand them to the model along with the question. RAG powers a large share of the AI features people use today, from document Q&A to customer support assistants to internal knowledge search. This article explains how it works end to end, the building blocks involved, where it breaks, and how to build and evaluate a basic version yourself.

The Core Idea in One Paragraph

When a user asks a question, a RAG system (1) searches a knowledge base for the passages most relevant to the question, (2) inserts those passages into the prompt as context, and (3) asks the model to answer using that context, ideally with citations. The model is not retrieving anything itself. Your code does the retrieval, and the model does the reading and writing. That division is the whole trick.

The Two Phases

A RAG system has an indexing phase (done ahead of time) and a query phase (done for every question).

Indexing:

Collect documents (PDFs, web pages, wikis, tickets, database records).
Clean and split them into chunks.
Convert each chunk into an embedding (a vector of numbers).
Store the chunks and embeddings in a searchable index.

Querying:

Convert the user's question into an embedding using the same model.
Find the stored chunks whose embeddings are closest to it.
Build a prompt containing those chunks plus the question.
Have the model generate an answer grounded in that context.

Let's go through the pieces.

Embeddings: Turning Meaning Into Numbers

An embedding is a list of numbers (commonly hundreds to a few thousand) that represents the meaning of a piece of text. An embedding model is trained so that texts with similar meanings end up close together in this high-dimensional space, even if they share no words. "How do I reset my password?" and "I forgot my login credentials" would land near each other, while "Quarterly revenue report" would land far away.

This is why embedding-based search is called semantic search. Traditional keyword search matches the literal words you typed. Semantic search matches what you meant. That makes it much better at handling paraphrases and vague questions, and sometimes worse at exact identifiers like error codes or product SKUs, which is a limitation we will return to.

"Close" needs a measurement. The most common one is cosine similarity, which compares the angle between two vectors:

go
func cosine(a, b []float64) float64 {
var dot, na, nb float64
for i := range a {
dot += a[i] * b[i]
na += a[i] * a[i]
nb += b[i] * b[i]
}
return dot / (math.Sqrt(na) * math.Sqrt(nb))
}

A result near 1 means the vectors point in nearly the same direction (very similar meaning), near 0 means unrelated. Many embedding models produce normalized vectors, in which case cosine similarity equals a simple dot product, which is faster to compute.

Chunking: The Underrated Step

You cannot embed an entire 200-page manual as a single vector and expect useful retrieval, because one vector would blur every topic together. So documents are split into chunks. How you split is one of the biggest factors in quality, and it gets less attention than it deserves.

Key trade-offs:

Chunks too small lose context. A sentence like "It must be renewed annually" is meaningless without knowing what "it" refers to.
Chunks too large dilute the signal, waste prompt space, and make retrieval less precise.
A common starting point is chunks of a few hundred tokens with some overlap between neighbors so ideas that straddle a boundary are not lost.
Structure-aware splitting often beats fixed-size splitting. Splitting on headings, paragraphs, or function boundaries in code keeps coherent units together.
Attach metadata to each chunk: source document, title, section, page number, date. This enables citations and filtering later.

A simple chunker:

go
func chunk(text string, size, overlap int) []string {
words := strings.Fields(text)
var chunks []string
step := size - overlap
for start := 0; start < len(words); start += step {
end := start + size
if end > len(words) {
end = len(words)
}
chunks = append(chunks, strings.Join(words[start:end], " "))
if end == len(words) {
break
}
}
return chunks
}

Real systems use smarter splitters, but the structure is the same.

Vector Search and Vector Databases

Once every chunk has an embedding, answering a query means finding the nearest stored vectors to the query vector. The brute-force approach is to compute similarity against every chunk, which is perfectly fine for thousands of chunks. For millions, it becomes too slow, so systems use approximate nearest neighbor (ANN) algorithms that trade a small amount of accuracy for large speedups. A popular one is HNSW, which builds a layered graph of vectors so a search can hop quickly toward the closest region instead of scanning everything.

Where do you store them? Options include:

Dedicated vector databases such as Qdrant, Weaviate, Milvus, and Pinecone, built for large-scale similarity search.
Extensions to databases you may already use, such as pgvector for PostgreSQL, which lets you store embeddings in a column and query by distance with SQL. For many applications this is the simplest path, because your vectors live next to your relational data and you avoid running another system.
Embedded or in-memory libraries such as FAISS or lightweight local stores, which are good for prototypes and smaller datasets.

An example of what a pgvector query looks like:

sql
SELECT id, content, source
FROM chunks
ORDER BY embedding <=> $1 -- cosine distance to the query embedding
LIMIT 5;

A practical tip: you do not need a specialized vector database on day one. Start with what you already run, and move to dedicated infrastructure only if scale or performance demands it.

Assembling the Prompt

After retrieval, you build a prompt that gives the model the context and clear instructions. A typical structure:

text
You are a helpful assistant. Answer the question using ONLY the context below.
If the answer is not in the context, say you don't know.
Cite the source id in brackets after each claim, like [doc-3].

Context:
[doc-3]
[doc-7]
[doc-12]

Question: How long do customers have to request a refund?

Several details matter here:

Grounding instruction. Telling the model to use only the provided context, and to admit when the answer is missing, reduces fabrication.
Citations. Labeling chunks with identifiers and asking the model to cite them lets users verify answers, and lets you check them programmatically.
Ordering and size. Models can pay less attention to material buried in the middle of a long context. Keep the retrieved set focused rather than stuffing in as much as possible.
Making Retrieval Better

Naive RAG, embed everything and take the top five matches, works surprisingly well as a demo and often disappoints in production. Teams typically add several refinements.

Hybrid search. Combine semantic search with traditional keyword search (such as BM25) and merge the results. This recovers the exact-match strength of keywords for things like error codes, names, and identifiers while keeping the flexibility of embeddings.

Reranking. Retrieve a generous candidate set (say, 30 chunks) cheaply, then use a more accurate but slower reranker model to score each candidate against the question and keep the best few. Reranking is one of the highest-return upgrades available.

Query rewriting. User questions are often short, ambiguous, or dependent on earlier conversation ("what about the second option?"). Rewriting the question into a standalone search query, or generating several variants, improves recall.

Metadata filtering. Restrict search to the right product, date range, language, or access level before similarity ranking. This both improves relevance and, importantly, enforces permissions.

Better chunk context. Prepending a short description of the document or section to each chunk before embedding helps a chunk remain interpretable on its own.

Agentic retrieval. Instead of one retrieval step, an agent decides what to search for, reads results, and searches again if needed, which helps with multi-part questions. This ties into the broader move toward agents in AI systems, at the cost of more latency and complexity.

Where RAG Fails

Knowing the failure modes is what separates a demo from a dependable system.

Retrieval misses the right chunk. If the relevant passage is not retrieved, the model cannot use it, and it may guess. Most RAG failures are retrieval failures, not generation failures. Debug by inspecting what was retrieved before blaming the model.

Retrieval finds plausible but wrong chunks. Semantically similar is not the same as correct. A question about "2024 refund policy" may retrieve the 2022 policy.

The model ignores or misreads context. Even with the right passages present, a model can overlook them, blend them with its prior knowledge, or draw an unsupported conclusion.

Questions that need reasoning across many documents. "Summarize trends across all our incident reports" is not a lookup problem. Retrieving a handful of chunks will not cover it. Such tasks need different approaches, such as pre-computed summaries or structured queries.

Stale or duplicate data. Out-of-date documents in the index produce out-of-date answers. You need a pipeline to update, deduplicate, and delete content.

Prompt injection through documents. If retrieved text contains instructions ("ignore previous instructions and..."), a model may follow them. Documents from untrusted sources should be treated as untrusted input, and systems that take actions based on retrieved content need guardrails.

Access control leaks. If your index mixes documents with different permissions, retrieval can surface content the user should not see. Enforce permissions at retrieval time through metadata filters, not by hoping the model will refrain from revealing things.

Evaluating a RAG System

You cannot improve what you do not measure, and "it seems to work" is not measurement. Evaluate the two stages separately.

Retrieval quality: build a test set of real questions, each labeled with the passage(s) that should be retrieved. Measure how often the correct chunk appears in the top results (recall at k) and how highly it is ranked.

Answer quality: check whether answers are correct, whether they are supported by the retrieved context (faithfulness), whether they actually address the question, and whether citations point to real supporting text. Evaluations can use human review, automated checks, and carefully validated model-based graders.

Start small: even 30 to 50 representative questions with known answers will reveal problems quickly and give you a baseline to compare every change against. When you change chunk size, swap an embedding model, or add reranking, rerun the set and see whether things actually got better.

RAG, Fine-Tuning, and Long Context

Students often ask how RAG relates to other approaches.

RAG is best for knowledge that changes, is large, needs citations, or has access restrictions. Updating knowledge means updating the index, not retraining.
Fine-tuning is best for changing a model's behavior, style, or format, or teaching a narrow skill, not for reliably injecting facts.
Long context windows, now large enough to hold entire books in some models, let you paste whole documents into a prompt. That is often the simplest solution for small knowledge sets, but cost and latency grow with every token, accuracy can degrade on very long inputs, and it does not scale to large or constantly changing corpora.

These are complementary, not competing. Many systems combine retrieval to find relevant material with a long-context model to reason over it.

Build One This Weekend

A minimal RAG project is a great way to understand all of this:

Pick a small corpus: your own notes, a documentation site, or a few PDFs.
Write a script to extract text and split it into overlapping chunks, attaching source metadata.
Embed each chunk using an embedding model (hosted API or a local model) and store the vectors in pgvector, SQLite with a vector extension, or even a plain in-memory list.
Write a query function: embed the question, find the top-k chunks by cosine similarity, and print them. Stop here first and check whether the retrieved chunks are actually good. This single step teaches the most.
Add generation: build the grounded prompt with citations and call a language model.
Write down 30 test questions with expected sources and measure retrieval recall.
Improve one thing at a time, such as chunk size, hybrid search, or reranking, and re-measure.
Key Takeaways
RAG gives a language model access to your data by retrieving relevant passages at question time and including them in the prompt.
Embeddings map text to vectors so that similar meaning means nearby vectors, enabling semantic search measured with cosine similarity or dot product.
Chunking is a major quality lever, and the right strategy depends on your documents.
You do not need a dedicated vector database to start. pgvector or an in-memory index is often enough, and ANN indexes like HNSW matter at larger scale.
Real-world quality comes from hybrid search, reranking, query rewriting, metadata filtering, and good prompts with citations.
Most failures are retrieval failures, so inspect retrieved chunks first, and enforce permissions in the retrieval layer.
Evaluate retrieval and answers separately using a small, real test set before and after every change.

RAG is not glamorous, and it is not a single product you install. It is a pipeline of ordinary engineering steps (parsing, chunking, indexing, searching, prompting, and testing) wrapped around a language model. That is exactly why it rewards developers who care about data quality and evaluation, and why it is one of the most useful practical skills in AI engineering today.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.