Four layers of checks for a RAG vector index, with the code

When a RAG answer is wrong, the model usually gets the blame and the index that fed it gets a pass. I have learnt to look at the index first. This is a companion to a longer article on my site, which holds the full set of checks, their sources and the comparison of 25 evaluation tools. Here is the method and the two results I keep coming back to.

What I built and ran

On 23 October 2026 I built an index from the 31 Markdown pages of the MCP specification (revision 2026-07-28, fetched at a pinned commit). The pages became 536 chunks under a cap of 400 tokens, split by heading first and then by paragraph, with the heading path prepended to each chunk. The embeddings are OpenAI's text-embedding-3-small, called through OpenRouter; the store is Chroma, embedded, with its default HNSW settings (16 neighbours, a build effort of 100, a search effort of 100). The checks run in four layers, cheapest first: the documents and their chunks, the embeddings and the index, retrieval against a golden set, and the answer with a judge. The golden set is 55 questions I wrote from the spec pages, labelled by page rather than by chunk id, so the labels survive a change of chunker: 30 factoid, 6 multi-hop, 8 paraphrases in plain words, 6 keyword queries and 5 out of scope. The code and the questions are on GitHub as rag-index-validation, under an MIT licence.

A page-level hit can hide a chunk-level miss

With the 50 answerable questions, the right page was in the top 5 for 49 of them (hit rate@5 0.980, recall@5 0.957, MRR@10 0.856). The one miss was a paraphrase in plain words. The result I keep thinking about, though, is one the metrics called a hit. The question "Is it OK to ask the user for their password in a pop-up form?" is answered by one sentence of the elicitation page. That page was in the top 5, at rank 2, through a different chunk about identifying the user; the chunk with the sentence sat at rank 6, with a similarity of 0.381. The generator was given the top 5 chunks and answered "I don't know". A label on the page said retrieval worked; a check of whether the model could answer from what it was given said otherwise. So I label pages, measure recall at the k the prompt carries, and read the misses by hand.

The check that gets skipped: approximate against exact

Every production vector store uses an approximate index, most often HNSW, which trades a fraction of the true neighbours for speed. The check is the same everywhere: run each query through the index and through a brute-force search over the same vectors, and divide the overlap of the two top-k sets by k.

# check_index.py, abridged
def exact_top_k(matrix, query, k):
    scores = matrix @ query  # unit vectors, so the dot product is the cosine
    return list(np.argsort(-scores)[:k])

truth = [set(ids[j] for j in exact_top_k(matrix, q, 10)) for q in queries]
for ef in (10, 32, 100):
    col.modify(configuration={"hnsw": {"ef_search": ef}})
    res = col.query(query_embeddings=queries.tolist(), n_results=10, include=[])
    recall = np.mean([len(set(res["ids"][i]) & truth[i]) / 10 for i in range(len(queries))])

On my index the result was recall@10 of 0.998 for the golden questions at ef_search 10, 32 and 100 alike, 1.000 for a sample of stored vectors, at about half a millisecond per query. The caveat matters more than the number: 536 vectors are far too few for an HNSW graph to miss anything, so this demonstrates the method, not a result. The point of running it early is that it costs minutes, and the number is there to watch when the index reaches the sizes, the filters and the quantization where it stops being 0.998. Filters, quantization and a model swap are the three things that move it, and each deserves its own repeat of the measurement.

What I did not do

One embedding model, no hosted store, no quantization, no filtered search. The judge saw six answers. None of the synthetic question generators was run. Your numbers will differ with the model, the chunker, the corpus and the day. The full article has the checks for each layer with their sources, the chunk-size evidence, the judge results and the tools table: https://pournasserian.com/writing/validating-a-vector-database-for-rag