Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 8 min read

Fix Hallucinations at the Data Layer, Not the Prompt

A language model predicts the most likely continuation of the text in front of it. When the right enterprise facts are missing, that continuation can be fluent and wrong. Feed it structured sources, relationships, and pr

Fix Hallucinations at the Data Layer, Not the Prompt

A language model predicts the most likely continuation of the text in front of it. When the right enterprise facts are missing, that continuation can be fluent and wrong. Feed it structured sources, relationships, and provenance, and bound what it is allowed to do on this task, so the guess has less room to fill.

Dan Klein, co-founder and CTO of Scaled Cognition and a professor at UC Berkeley, joined in April 2026. Sudhir Hasbe, President and Chief Product Officer at Neo4j, joined in April 2026.

How do you ground retrieval when similar vectors point at the wrong domain?

Filter similarity results with structured relationships and metadata before the model answers. Hasbe separates consumer models, trained on public web text, from enterprise systems, which must run on internal knowledge those models never saw. Fine-tuning on that knowledge is expensive, and he does not expect every company to hand its IP to a general model. The model still predicts a next word. Your data architecture is what makes the decision checkable.

Plain vector retrieval is where that gap shows up. Hasbe retells a Spotify case: similarity search on a pop song could surface country music, because the vector space only measures mathematical closeness. Metadata about the song was what restricted results to the domain the listener was in. He sees the same pattern in enterprise RAG. Teams that started on pure vector search were landing around 60 to 65 percent accuracy. Combining vector search with a graph filter on metadata and relationships, what he calls GraphRAG, is where he has seen customers move above 90 percent. Adding targeted tools on a reasoning engine, a predefined query for a question people ask often, is where he says accuracy can move above 95 to 97 percent. The graph is also easier to inspect than a vector space: you can look at the entities and the relationships and explain why a result was kept.

"You all have to figure out what's your data architecture to provide that enterprise knowledge to the agentic system so that you can make right decisions and you can audit the decisions"

Sudhir Hasbe, President and Chief Product Officer at Neo4j, on Chain of Thought ep 55

Hallucinations Are a Data Arch… - Chain of Thought | AI Agents, Infrastructure & Engineering - Apple Podcasts

Podcast Episode Β· Chain of Thought | AI Agents, Infrastructure & Engineering Β· April 16 Β· 52m

favicon podcasts.apple.com

Record the rationale with the decision. In Hasbe's sales example, the CRM stores the approved discount. The negotiation that justified it lives in email and Slack: the customer's budget, the sales leadership's approval after the manager discussion on Slack, the bet that the account will grow. A context graph is that missing layer. Future agent actions can read why the exception was granted, which the system of record never kept.

He describes three ways teams place that graph. A semantic map leaves data in place and records where it lives, what terms mean, and which reference sets apply. A second pattern pulls in the slice that is too scattered to reason over, such as a customer split across sales, marketing, and service clouds. A third puts company knowledge in one graph and runs agents on it. He says most teams start with the first or the second.

A minimal sketch to adapt:

def answer_with_graph(query, graph, vectors, allowed_tools):
    hits = vectors.similarity_search(query)
    context = graph.filter_by_metadata(hits)
    action = call_model(query, context, tools=allowed_tools)
    trace_span(action, provenance=context.sources)
    return action

Why does a second model checker fail to catch hallucinations?

The checker fails on the same hard cases as the model it reviews. Klein's name for the base system is a plausibility engine: it assembles likely text, and likely is easy to confuse with true.

"Ultimately, they are assembling plausible outputs on the fly. And it's so easy for those things to be plausible, but not true. The bottom line is we have built not truth engines, not reliability engines, we've built plausibility engines."

Dan Klein, co-founder and CTO at Scaled Cognition, on Chain of Thought ep 54

He walks a conversation of about 20 turns. A model that is right 80 percent of the time, checked by another at 80 percent, looks like a 4 percent miss if you assume the errors are independent. He then drops that assumption. Hard instances fail together, so 80 percent checking 80 percent lands around 82 percent right, barely above the model alone. Each of those 20 turns is another chance to miss. Sometimes the checker catches an error. Sometimes it adds one, and the design reaches for a third model, then 10 or 15, with the token bill and the latency that implies. He would rather the model avoid the mistake in its own operation.

Prompt text will not close that gap. Klein says a prompt has no precise semantics and no guarantee the model will follow it. When it ignores you, the recourse is rewording, capital letters, and exclamation points. Around the third exclamation point, he says, you can tell this is not a path to a controllable technology. On the output side you get tokens, and tokens do not tell you what the system is allowed to do. An action can be allowed or refused. Information can be allowed to move or blocked. He argues the model has to give actions and information that status, including a distinction between what it knows and what it does not, and what it can and cannot do. A token generator has no such distinction. If the string happens to match the facts, people call it truth. If it does not, they call it a hallucination. The model did not separate the two.

A retrieved passage can still be restated into something else. Klein asks how to distinguish a changed answer from a paraphrase that preserves its meaning. He also describes an iceberg: a wrong balance is the kind people notice; a plausible refund policy that is not your refund policy looks right to a developer reading the output. Looks right and is right are not the same.

"First of all, it's not very effective. So what can happen is, OK, you have multiple models. They can all fail. Sometimes one catches the error. Sometimes it introduces an error. And then you need a third model."

Dan Klein, co-founder and CTO at Scaled Cognition, on Chain of Thought ep 54

How many tools should you give one reasoning engine?

Give it a small set that matches the use case, chosen before the call. Hasbe says customers who attach hundreds of tools to one reasoning engine watch accuracy fall as the list grows. The same slide happens in a chat when you keep appending context after the model has started answering. An architect who had worked at Intuit and later at Salesforce told him the relevant set for that use case had to be chosen on purpose, bounded rather than open-ended. Customer lookup for a fraud step and customer lookup for a marketing step can share a name and still be the wrong tool. Extra unused tools also spend tokens, so the bound is a cost control and an accuracy control.

Hasbe treats the knowledge graph as the place that lookup happens. For a fraud flow, pick the 10 or 15 tools that belong to it, confirm the intent, and send that list. Uber, in an account he relays from a Neo4j customer summit, uses the graph as a business validation layer for a harder version of the same idea: thousands of jurisdiction combinations for Uber Eats, including payment and minimum-wage rules that change when a delivery crosses from Seattle toward Bellevue. The graph holds those rules so an agent deciding a payment can be checked against the combination that applies.

How do you tell a demo score from a system you can ship?

Ask whether the scenario is right every time, on the order of 100 runs in a row. Klein says counting how many benchmark scenarios a model got right does not match an enterprise bar. A system at 80 or 90 percent cannot ship; he illustrates with 1 in 10 customers told a flight was booked when it was not. A system that gets 70 percent of scenarios right every single time, however they are approached, is the one he calls hugely valuable. Set the metric to the tolerance you have for a wrong action. A wrong FAQ answer and a wrong account balance do not share a bar, because the second one leads someone into a financial decision.

FAQ

Is hallucination a model bug or a data gap?
Hasbe frames the enterprise case as a data architecture problem: the models were trained on public information, and internal knowledge has to be supplied or the system will answer from what it already has. Klein adds that even a retrieved answer can be replaced with different wording, so retrieval alone does not define the error. Treat grounding and the model's notion of information as two checks.

Can prompting alone make an agent reliable?
Klein says a prompt is a hint. Natural language is ambiguous, and nothing forces the system to follow the wording. Rewording, capital letters, and exclamation points are the recourse he describes, and he does not treat them as a guarantee. Put allow-or-refuse on actions, and record where information came from.

How many tools should an agent have?
Hasbe reports that hundreds of tools on one reasoning engine make results worse as the count grows. Select a bounded set for the use case, and use the graph to decide which set applies. Fewer irrelevant tools also means fewer tokens.

Takeaway

  • Supply enterprise knowledge through a data architecture the agent can audit: sources, relationships, and provenance.
  • Filter vector hits with metadata so a nearby embedding cannot cross into the wrong domain.
  • Store the decision and the rationale that email and Slack held, so a later action can read why an exception was granted.
  • Bound tools to the use case before the model runs, and trace the action against that context.
  • Judge a ship candidate by repeated success on a scenario, on the order of 100 runs, and match the bar to the cost of a wrong action.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.