Knowledge Graph Evaluation: Scoring 12 Real Decisions

Three weeks ago I started scoring my knowledge graph against the answer I would have reached without it. It earns its place: 6 of 12 decisions came out sharper, 2 errors never reached real work, and 73 minutes of research time saved.

The Metric That Cannot Fail

Retrieval told me none of that. 21 of 21 queries, 100% recall, 8s median. Recall and latency describe the index, not the work. No input makes them come back bad.

Freeze Your Answer First

None of that score exists unless the old answer is written down first: the options, the criteria, the recommendation, a confidence number. Then query. Write the baseline afterwards and you rebuild a past self who conveniently agreed with whatever came back.

If your retrieval has never contradicted you, nobody has checked.

What would it take for your own retrieval layer to return an answer that changes your mind, and would you be able to prove it did?

I write field notes from real builds: AI integration, automation, and the parts that break in production. New posts every two weeks. Use the RAG requirements template to audit the source, access and action boundaries in your own context layer.