The Semantic Debt Trap: Why AI Cannot Query Your Raw Data Lakehouse

Agents shouldn't write raw SQL against your lakehouse for any number that ends up in a board report. Your lakehouse holds your data. It doesn't hold what your business means by it.

The benchmarks point the same way, with caveats. In a 2023 data.world benchmark (43 business questions on an insurance schema, zero-shot), GPT-4 writing SQL got 16.7% right. Through a knowledge graph of the same data it got 54.2%. Same model, same questions. On the questions that needed more tables, SQL accuracy was 0%. Models have improved since: in dbt Labs' 2026 rerun, current models writing raw SQL scored 64.5% on 11 questions. That's a vendor test, and still not a number I'd put in a board report.

SQL skill isn't the gap. Meaning is. Three failures keep showing up:

• Polymorphic columns. amount means gross, net or a credit memo depending on another column. The model writes SUM(amount). Valid SQL, wrong meaning.
• Join fan-out. Guess a join path at the wrong grain and rows multiply. Sums inflate. Nothing errors out.
• Definition schism. Finance, Marketing and Support each mean something different by "active customer". The agent gets whichever one its tables happen to encode.

The pattern I'd use has three tiers. The raw lakehouse stays the store, with no direct agent access. Metric definitions live in versioned YAML (dbt Semantic Layer, Cube), reviewed in git. The agent asks for metrics by name, for example through the dbt MCP server's query_metrics, and the layer compiles the SQL. Turn off execute_sql and text_to_sql for that agent. dbt's own tool list warns that execute_sql doesn't block write statements.

In dbt's 2026 rerun, the semantic layer scored 100% on the 8 questions it could answer and declined the 3 multi-hop ones instead of guessing. A refusal beats a wrong answer.

It's not a cure, and I'm skeptical of parts of my own playbook. Consistent isn't correct: the agent can still pick the wrong metric, and a definition can be wrong or stale. Getting Finance, Sales and Product to agree on "churned" is political. One-off exploration doesn't fit a fixed metric list.

Before agents touch the warehouse, I'd start here:

1. Ask three developers to compute your main metric from raw tables. If they differ by even a couple of percent, an agent writing raw SQL won't do better.
2. Take broad raw table access away from agent service accounts. Curated views or semantic-layer APIs only.
3. Write your first five canonical metrics in YAML, in git, with required review.

Data volume isn't data readiness.

The full article has the diagrams, the benchmark breakdown, all sources and the complete four-step readiness audit: https://abivarma.medium.com/the-semantic-debt-trap-why-ai-cannot-query-your-raw-data-lakehouse-f3cdd1c48fa3