Context, Not Models: What Actually Made AI BI Reliable

You're in a planning meeting and someone says, "Can't we just let the LLM write the SQL?" Everyone looks at the data team. The honest answer is "yes, if we build the right things around it." Here's what those things are.

That answer has gotten more specific over the last couple of years. In 2024, Uber and LinkedIn published detailed write-ups of text-to-SQL systems running against warehouses with hundreds of thousands of datasets (Uber, in a limited release) and millions of tables (LinkedIn). Both teams said what worked and what didn't. Since then, newer models have raised the ceiling, and benchmark authors have started to measure what actually drives accuracy.

This article walks through that evolution from a data engineering and architecture point of view. It ends with the less glamorous question of whether AI can build the pipelines themselves. The time window starts in 2024, but I pull in sources through 2026 because that is where the strongest comparative evidence sits. I'll say so each time a source falls outside 2024.

One caution before we start. The numbers below come from different datasets, different question sets, and different definitions of "accuracy". I won't line them up in one table as if they were comparable, and you shouldn't either. Where a figure is vendor-run, user-reported, or comes from a small test, I say so.

🎯 The Thesis in One Page

Q: What is the actual claim here?

Two claims, one better supported than the other.

• Productivity went up, according to the people using these tools. Uber and LinkedIn both report user-side gains in writing queries, finding tables, and debugging. All of it is user-reported or usage data, not controlled measurement.
• My hypothesis: non-determinism is shrinking, and architecture is doing as much of the work as better models. The reliable results in the sources come from constraining what the model has to guess: curated context, semantic layers, retrieval over metadata, and step-by-step generation.

The second claim is the interesting one for architects. A raw LLM generating SQL against a raw schema is a probabilistic system. Add a semantic layer that compiles metric definitions into SQL, and the model's job changes from "write correct SQL" to "pick the right metric and dimensions". That second job has a much smaller space of wrong answers.

I couldn't find a controlled measurement of run-to-run variance across model generations. So this is a hypothesis the sources are consistent with, not a measured result.

Key insight: Among current frontier models, one 2026 paper found that adding context moved accuracy more than choosing between models (per its abstract). Newer models clearly helped too. In dbt Labs' benchmark, in-scope text-to-SQL accuracy rose from 26.9% in 2023 to 62.5% and 51.2% for two 2026 models. Both levers are real, and the second one is the one you control.

Roughly where each topic gets its evidence:

🗣️ From "Write SQL" to "Ask a Question": What AI BI Actually Is

Q: What do I mean by AI BI and natural interaction?

I mean an interface where a person asks a data question in plain language and gets back a query, a result, or both. The model sits between the person and the warehouse.

The idea is old. What changed in 2024 is that large companies wrote up their attempts in detail, and those write-ups list their limitations explicitly. Three of them anchor this article:

• Uber's QueryGPT (Uber Engineering, September 2024)
• LinkedIn's SQL Bot (LinkedIn Engineering, December 2024), with a follow-up paper in July 2025
• Snowflake's Cortex Analyst (Snowflake Engineering, August 2024), which is a vendor product and should be read as such

Q: Why did this get hard at enterprise scale when the demos looked easy?

Because a demo has a schema of ten tables, and an enterprise has far more. Uber's team said that the number of datasets (hundreds of thousands) prevents complete evaluation coverage. LinkedIn's data warehouse holds millions of tables. At that scale the first problem is not generating SQL. It is finding which table to use.

Uber's write-up gives some sense of the stakes. Uber's data platform handles about 1.2 million interactive queries a month, and the Operations organization contributes about 36% of them. Query authoring took about 10 minutes before QueryGPT and about 3 minutes with it. The limited release reached about 300 daily active users, and 78% of users said it reduced the time spent writing queries from scratch.

Those are useful numbers with an important caveat. The 10-to-3-minute figure and the 78% figure are user-reported or estimated, not controlled measurements. Treat them as a direction, not a benchmark.

Why a single prompt isn't enough

Snowflake's engineering post makes the case clearly. It argues that public benchmarks such as Spider and BIRD show 80–90%+ accuracy but fall short on real business use, and it names four gaps:

• Question complexity. Real business questions are messier than benchmark questions.
• Schema complexity. Production schemas are more complicated than benchmark schemas.
• Complex SQL features. CTEs and window functions show up constantly in real analytics.
• Missing business-metric semantics. "Revenue" or "active customer" is a definition, and a column name doesn't tell you what it is.

In Snowflake's own internal evaluation, GPT-4o with a single prompt scored 51%. The evaluation had 150 questions across sales, marketing, and finance, in three levels (filtering, aggregation, trend analysis), with multiple gold queries per question. Snowflake reports 90%+ accuracy for Cortex Analyst, which uses a semantic model.

Watch out: That is a vendor evaluating its own product on its own question set. The direction (context beats a bare prompt) matches what an independent paper reports in its abstract, but the specific numbers shouldn't be quoted as a general truth about the market.

Placeholder image — replace images/image-1.jpg with your generated image (keep the same filename). The full self-contained generation prompt is **Image 1* in 05_image_prompts.md.*

🏗️ Anatomy of a Production Text-to-SQL System

Q: If a single prompt isn't enough, what does a production system look like?

Both Uber and LinkedIn converged on multi-agent designs, where each agent handles one narrow step.

Uber's QueryGPT uses an Intent agent, a Table agent, and a Column Prune agent. The Column Prune agent exists for a very practical reason: to manage token usage on very wide tables. If a table has hundreds of columns, you cannot paste all of them into every prompt.

LinkedIn's SQL Bot, built inside its DARWIN platform, is also multi-agent and is backed by a knowledge graph. The follow-up paper describes three components: a knowledge graph, a text-to-SQL agent that retrieves context, generates queries, and corrects errors, and an interactive chatbot.

Roughly, the flow looks like this. It is my synthesis of the common shape, not a reproduction of either company's diagram, and the dotted feedback edge is a possible extension that neither source describes.

flowchart TD
    A[User question] --> B[Intent classification]
    B --> C[Question enhancement:<br/>add missing context]
    C --> D[Table retrieval<br/>over metadata and docs]
    D --> E[Column pruning<br/>for wide tables]
    E --> F[SQL generation<br/>incremental steps]
    F --> G{Executes?}
    G -- no --> H[Error-driven<br/>self-correction]
    H --> F
    G -- yes --> I[Result and generated SQL<br/>shown to the user]
    I --> J[User feedback and<br/>query logs]
    J -.->|possible extension| D

Q: What did these teams learn about each step?

A few lessons come straight from the write-ups, plus one that is my own reading:

• Uber reports that LLMs work well as specialized classifiers on narrow tasks.
• Uber also found that users' prompts lack enough context, which is why a "prompt enhancer" step is needed.
• LinkedIn reports that breaking complex questions into incremental steps outperformed monolithic generation.
• My read: table retrieval is the load-bearing step. If the wrong tables come back, everything downstream is wrong, no matter how good the SQL generator is. LinkedIn's finding that better table descriptions improved retrieval accuracy points the same way.

Trade-off: more agents, more moving parts

Splitting the pipeline into agents gives you narrow, testable steps. It also costs latency and more prompts to maintain, and errors compound across stages. Each stage has its own failure rate, so end-to-end accuracy is lower than the best stage.

Splitting the pipeline is still worth it if you evaluate each stage separately, which I come back to in the evaluation section.

🕸️ Context and Knowledge Graphs

Q: What is the "context" everyone keeps talking about?

It is everything the model needs to know that isn't in the question. For a data system, that means table and column descriptions, which tables are popular, how tables usually join, what the values in a categorical column look like, and what the business means by a given term.

LinkedIn's SQL Bot makes this concrete, and it is the best-documented example I found. Its knowledge graph has:

• Nodes for users, table clusters, tables, and fields
• Metadata from DataHub, including the top K values for categorical fields, partition keys, and a classification of fields into metrics, dimensions, and attributes
• Query logs that provide popularity signals and common joins
• Crowdsourced domain knowledge from people who know what the data means

Most of that list is plain metadata management: catalogs, usage statistics, documentation. The AI use case made all of that metadata valuable in a way that "please document your tables" never managed on its own.

Q: Does improving the documentation actually move the numbers?

LinkedIn reports that dataset certification, meaning better table descriptions, significantly improved retrieval accuracy. That is a statement about the quality of human-maintained metadata. The model didn't get smarter. The input got better.

For a data architect, this reframes a familiar chore. Table descriptions, column definitions, and ownership tags used to be hygiene. Now they are an input to a production system, and bad ones have a measurable cost.

Key insight: If you are deciding where to spend the first month of an AI BI initiative, the evidence here points to metadata quality and retrieval before model selection.

What I could not verify

An earlier paper I came across reports a large accuracy jump from knowledge graphs, but I haven't read it, so I'm leaving the number out. LinkedIn's follow-up paper includes ablation studies on its knowledge graph, but I only read the abstract page, so I can't tell you the per-component results. If you are making an architecture decision on this, read that paper in full.

Trade-off: a knowledge graph is a maintenance commitment

A knowledge graph that goes stale will confidently feed the model wrong context. Someone has to own ingestion from the catalog, refresh usage statistics, and handle the crowdsourced knowledge. LinkedIn's graph depends on DataHub metadata and query logs. If you don't have a catalog or query logs worth mining, building the graph first is a much bigger project than the diagrams suggest.

📐 Semantic Layers: Where Non-Determinism Gets Boxed In

Q: What is a semantic layer, and why does it matter for AI?

A semantic layer is a governed set of metric and dimension definitions that compiles a request into SQL. You define "revenue" once, with its filters and its grain. A consumer, human or model, asks for "revenue by month by region", and the layer produces the SQL.

For AI, the value is that the model no longer writes the SQL. It selects from a defined menu, and a deterministic compiler produces the query. That is the basis for my "non-deterministic to more deterministic" hypothesis.

An illustrative metric definition follows. It is not any specific product's exact syntax, and the names are invented.

# Illustrative only: a generic semantic model, not a specific product's syntax
semantic_model:
  name: orders
  table: analytics.fct_orders
  entities:
    - name: customer
      key: customer_id
  dimensions:
    - name: order_date
      type: time
      grain: [day, week, month]
    - name: region
      type: categorical
  measures:
    - name: net_revenue
      description: "Gross order value minus refunds, excluding test accounts"
      expr: "gross_amount - refund_amount"
      filter: "is_test_account = false"
      agg: sum

The lines that matter are the description and the filter. A model working from raw columns has to guess that test accounts should be excluded. A semantic layer states it once, and every query inherits it.

What the dbt Labs benchmark found

Q: Is there evidence that this actually helps?

Yes, with caveats. dbt Labs (Jason Ganz and Benoit Perigaud) published a "Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update" in April 2026. It reruns dbt's 2023 benchmark on current models, using the ACME Insurance dataset from data.world with 11 questions and 20 runs each. It compares four configurations:

1. Text-to-SQL
2. A minimal Semantic Layer
3. A modeled Semantic Layer
4. Text-to-SQL on the modeled data

The models tested were Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.3 Codex, GPT-5.2, and GPT-4 from November 2023 as the baseline.

The findings I'm comfortable quoting come in two separate sets. With modeled data:

For questions within the Semantic Layer's scope:

I couldn't pin down which of the four configurations the in-scope figures come from, so I've kept them in a separate table. Don't read across the two tables.

Two things stand out.

First, text-to-SQL improved a lot between 2023 and 2026 on the in-scope questions, going from 26.9% to 62.5% and 51.2% depending on the model. That is real progress from the models alone.

Second, the Semantic Layer reached 100% on in-scope questions for both 2026 models. The compile step plausibly removes most of the room for the model to be creative in the wrong place. Note that the Sonnet figure on modeled data is 98.2%, not 100%, so the model's choice of metrics is still not fully deterministic.

The benchmark's own caveats matter, and the authors state them:

• The entire schema was loaded as context, which "isn't practical for larger datasets".
• The Semantic Layer can only answer questions within its modeled scope. Some multi-hop questions fall outside it.
• The page says reasoning effort mattered little for Semantic Layer queries.

Add to that the structural caveat: this is a vendor benchmarking its own product on a small dataset of 11 questions. It is a useful signal, not proof.

Watch out: "100% on in-scope questions" depends entirely on how the scope is drawn. A semantic layer is deterministic about what it can answer. It does nothing for the questions it can't answer, and a user doesn't always know which kind they are asking.

An independent data point: a 4 KB document

Q: Does the same pattern show up outside a vendor's own benchmark?

An arXiv paper by Rumiantsau and Fokeev, submitted in April 2026, tests Claude Opus 4.7, Claude Sonnet 4.6, and GPT-5.4 on 100 questions against the Contoso retail dataset in ClickHouse. Each model runs twice: with the schema only, and with the schema plus a 4 KB semantic document.

Per the abstract (I could only read the abstract, not the full paper):

• The semantic document improved accuracy by "+17 to +23 percentage points across all three models."
• With the document, accuracy converged at 67.7–68.7%. Without it, accuracy was 45.5–50.5%.
• The authors state the document "accounts for essentially all of the significant variance; model choice within tier does not."

That last sentence is the strongest statement of my thesis in any source I found. Note the qualifier "within tier": these are three current frontier models from two vendors, and they landed within a point of each other once they had the same context. The paper's title also mentions hallucination results, which I couldn't extract, so I'm not making any hallucination claim from it.

Conceptually, a semantic document is a document, not a compiler. It gives the model better context, but it doesn't remove the model's freedom to guess. I'm not comparing its numbers with dbt's, because the datasets, question sets, and scoring differ.

Trade-off: coverage versus certainty

Free-form text-to-SQL can answer novel questions but fails by producing plausible, wrong SQL. A semantic layer only answers what it models but fails by saying "I can't answer that". A wrong-but-plausible query can quietly reach a dashboard, while a refusal gets noticed and gets someone to extend the model. For metrics that feed decisions, I'd take the refusal.

I found no documented incident where an AI-generated query led to a bad business decision. Uber's note that hallucinated tables and columns remain unresolved is the closest. So treat the dashboard scenario as a risk, not a track record.

🧠 Reasoning Models: What I Can and Can't Say

Q: Did reasoning models change data work?

I need to be careful here. I couldn't find a dedicated source on how reasoning models changed data work. What I have is a narrow signal: the dbt Labs benchmark found that reasoning effort mattered little for Semantic Layer queries.

That is at least consistent with the logic of the semantic layer. If the model's job is to choose among defined metrics, there is less to reason about. My guess is that reasoning matters more for open-ended, free-form SQL with multi-step logic, but that is a hypothesis, and no source I found tests it.

The 2023-to-2026 jump in text-to-SQL accuracy in the dbt benchmark suggests newer models are simply better at SQL. I can't attribute that to reasoning specifically, and I won't.

One thing worth testing on your own data, as an expectation I can't back with a source: as models get better at syntax, the remaining errors may shift from "the query failed" to "the query answered a slightly different question". If that holds, evaluation has to look at meaning, not just execution.

Trade-off: where to spend the effort

If reasoning effort matters little for constrained queries, then effort spent constraining the task (a semantic layer, curated context) may buy more than effort spent on heavier reasoning. That suggests, but doesn't prove, a priority order. The flip side is that a semantic layer has coverage limits, so open-ended questions still need free-form generation, and for those I have no evidence either way. Measure on your own questions before deciding.

📊 Evaluation: Measuring Something That Has Multiple Right Answers

Q: Why is evaluating text-to-SQL harder than it sounds?

Because "correct" is slippery. Uber's team lists the problems in its own limitations:

• Non-deterministic LLM behavior requires longer observation periods.
• Hundreds of thousands of datasets prevent complete evaluation coverage.
• Multiple valid SQL solutions exist for one question.
• Hallucinated tables and columns remain unresolved.

I'm paraphrasing Uber's post here, not quoting it.

Uber's evaluation framework measures intent accuracy, a table overlap score from 0 to 1, query execution success, and an LLM-judged similarity to a golden SQL query. Together those cover the classification, retrieval, execution, and output stages.

LinkedIn's benchmark and what it shows

LinkedIn's benchmark has 130+ questions across 10 product areas. About 60% of them have multiple valid answers. Its LLM-as-judge scores agreed with human evaluation 75% of the time.

Pause on that last number. A judge that agrees with humans three times out of four differs from the human call one time in four. That is fine for tracking trends, but it is not good enough to certify a single answer as correct.

Two numbers that look like the same thing and aren't

This is my favorite illustration of the evaluation problem, and the LinkedIn sources supply both halves.

• The December 2024 blog post reports that about 95% of users rated query accuracy "Passes" or above, and about 40% rated it "Very Good" or "Excellent".
• The July 2025 paper reports, per its abstract, that 53% of responses were evaluated as correct or close to correct on internal benchmarks, with over 300 weekly users.

These are not contradictory. They measure different things: the first is user satisfaction, the second is a benchmark rating. I only read the paper's abstract page, so I can't tell you how the 53% benchmark was set up.

The lesson is clear anyway: user satisfaction can be high while benchmark-measured correctness is moderate. One plausible reading is that a "passes" rating means the query got the user close enough to edit and run. That is a real kind of value, and it is not the same as being right.

A minimal evaluation harness

Below is a stripped-down sketch of a harness that respects the multiple-valid-answers problem. It compares result sets, not SQL text, and allows several gold queries per question. Snowflake describes using multiple gold queries per question; comparing result sets is my own addition. This is illustrative code, not any company's implementation.

# Illustrative sketch: compare results, not SQL strings.
from dataclasses import dataclass
import hashlib
import json


@dataclass
class EvalCase:
    question: str
    gold_queries: list[str]  # several valid answers per question
    ordered: bool = False    # set True for ranking / top-N questions


def result_fingerprint(rows: list[tuple], ordered: bool = False) -> str:
    # Order-insensitive by default so ORDER BY differences don't count as failures.
    # Float formatting can still cause false mismatches; round values before hashing if needed.
    canon = [json.dumps(r, default=str) for r in rows]
    if not ordered:
        canon = sorted(canon)
    return hashlib.sha256("\n".join(canon).encode()).hexdigest()


def evaluate(case: EvalCase, generated_sql: str, run_query) -> dict:
    try:
        gold_prints = {
            result_fingerprint(run_query(q), case.ordered) for q in case.gold_queries
        }
    except Exception as exc:
        # A broken gold query is a test bug, not a model failure.
        return {"executed": None, "match": None, "error": f"gold query failed: {exc}"}
    try:
        got = result_fingerprint(run_query(generated_sql), case.ordered)
    except Exception as exc:
        return {"executed": False, "match": False, "error": str(exc)}
    return {"executed": True, "match": got in gold_prints, "error": None}

It is deliberately small. In practice you'd add per-stage metrics (did retrieval return the right tables?), repeated runs to measure variance, and a human-reviewed sample to calibrate any LLM judge.

Q: What about measuring run-to-run variance?

This is a gap. Uber says non-determinism requires longer observation periods, and the dbt benchmark runs each question 20 times. But I couldn't find a controlled measurement of run-to-run variance across model generations, so my "less non-deterministic over time" hypothesis rests on the semantic-layer results and on Uber's note about the problem, not on a direct measurement. If you adopt any of this, measure your own variance. Run each test question multiple times and look at the spread, not just the mean.

Placeholder image — replace images/image-2.jpg with your generated image (keep the same filename). The full self-contained generation prompt is **Image 2* in 05_image_prompts.md.*

Trade-off: evaluation is a product you have to maintain

A gold set of real questions takes time to collect, and multiple valid gold queries per question take more. Schemas change, so the set goes stale. An LLM judge needs a human-reviewed sample to calibrate against, and that sample has to be refreshed too. LinkedIn's 75% agreement figure is a reminder that skipping the calibration leaves you with a cheap number you can't fully trust. The cost is real, but without it you are flying on user satisfaction alone.

⚙️ The Hard Part: Can AI Build the Pipelines?

Q: Everything so far is about querying data. What about data engineering proper?

This is where I have to be straight with you: the evidence is thinner and less flattering.

Almost all the strong sources I found are about AI BI and text-to-SQL for analytics. I couldn't find a firsthand team post-mortem on pipeline authoring, dbt model generation, orchestration, or self-healing pipelines. What I found was mostly listicles and vendor content, which I set aside.

The one solid source is academic: ELT-Bench, by Jin, Zhu, and Kang (arXiv, April 2025). It evaluates AI agents on building full ELT pipelines:

• 100 pipelines
• 835 source tables
• 203 data models
• Two agent frameworks (Spider-Agent and SWE-Agent) using six LLMs

The best configuration, Spider-Agent with Claude-3.7-Sonnet and extended thinking, "correctly generates only 3.9% of data models". It averaged $4.30 and 89.3 steps per pipeline.

Watch out: That is a snapshot of early-2025 tools and models, not the current state. Newer models may do better, and I didn't search for later results. Don't read 3.9% as "agents can't do this." Read it as "end-to-end pipeline generation was far behind SQL generation when this was measured."

Q: Why would building a pipeline be so much harder than answering a question?

My explanation is an interpretation, not something the paper states, and it has three parts:

• The task is wide. A pipeline touches extraction, loading, transformation, tests, and orchestration, and an error in an early step corrupts everything after it.
• Success is harder to check. A query returns a result you can compare, while a data model has to be right in grain, keys, and semantics across many tables.
• There is no gold context. The agent has to discover what the data means from raw source tables. That is the same missing-semantics problem Snowflake describes, but now there is no semantic model to lean on, because the pipeline is what creates one.

That's my reading of a single benchmark, so hold it loosely.

Where AI might help in data engineering today

Setting aside the benchmark, there are tasks where I'd expect an LLM assist to be a reasonable bet. None of the sources above measures these, so test them against your own stack:

• Drafting a first version of a SQL transformation that an engineer then reviews
• Explaining an unfamiliar query or model to a new teammate
• Writing documentation and column descriptions for human review (which feeds the context problem from earlier)
• Debugging a failing query from the error message

LinkedIn's data gives one hint about the last item. 80% of SQL Bot sessions use the "Fix with AI" debugging feature, which the post describes as needing minimal development effort. Usage is not the same as benefit, but a cheap-to-build feature that most sessions touch suggests assistance fits naturally right at the point of a concrete failure.

Trade-off: the review bottleneck

Faster drafting shifts work from writing to reviewing. If your team writes twice as much SQL but reviews the same way, review becomes the bottleneck, and reviewers see more plausible-looking code that is subtly wrong. The tooling that helps most here is not generation. It is tests, data contracts, and diff-based validation of outputs.

🧰 Where AI Got Attached: Patterns in the Sources

Q: How did the data tools themselves change?

I'll mark the limits of my sourcing up front: I couldn't find a source that documents how specific tools evolved in 2024, such as Databricks Genie, MCP-based data tools, or catalog changes. I'm not going to write a feature history I can't back up.

What the sources do show is a pattern in where AI got attached. Each point below says what it rests on:

1. Embedded in an existing platform. LinkedIn found that embedding SQL Bot in the DARWIN platform increased adoption 5–10x compared with a standalone tool. This is one company's experience, but it suggests where the assistant lives matters as much as how good it is.
2. Built on top of a catalog. LinkedIn pulls metadata from DataHub, so the catalog became the substrate for AI there. I only have this one firsthand production account in full.
3. Paired with a semantic or metric layer. Snowflake's Cortex Analyst relies on a semantic model, and dbt's benchmark is built around its Semantic Layer. That is two vendors arriving at a similar idea, which is a thinner basis than "vendors converged" would imply.
4. Evaluated with domain-specific benchmarks. Snowflake's critique of Spider and BIRD, and its own 150-question set, point toward your own questions on your own data.

So the shape of the evolution, from the evidence I have, is a move from "a model that writes SQL" toward "a system that retrieves metadata, consults definitions, generates in steps, and gets graded on your data."

Conceptual baseline                   Pattern in the sources (Uber, LinkedIn,
(not from a source)                   Snowflake; my synthesis)
-------------------                   ------------------------------------
question ──> LLM ──> SQL              question ──> intent ──> enhance
                                            ──> retrieve tables (catalog + graph)
                                            ──> prune columns
                                            ──> generate (stepwise)
                                            ──> self-correct
                                            ──> evaluate against your own gold set

If you're an architect choosing tools, that diagram suggests a shopping list that has little to do with the model:

• A catalog with good, current metadata
• A place to define metrics once
• A retrieval layer you can inspect and tune
• An evaluation harness you own

⚖️ Comparing the Approaches

Q: Which approach should I pick: free-form text-to-SQL, a semantic layer, or a knowledge-graph-backed system?

They are not mutually exclusive, because they solve different problems. dbt Labs' post has a diagram on when to use a Semantic Layer versus text-to-SQL, but I haven't seen its content. What follows is my own recommendation: use the semantic layer for questions you can predict, and text-to-SQL for exploration.

A pragmatic architecture combines them:

flowchart LR
    Q[Natural-language question] --> R{In semantic<br/>layer scope?}
    R -- yes --> S[Semantic layer<br/>compiles governed SQL]
    R -- no --> K[Knowledge-graph retrieval<br/>plus text-to-SQL]
    S --> V[Result with<br/>metric definition shown]
    K --> W[Result with generated SQL<br/>flagged as exploratory]

The routing step is the hard part, and I'm not aware of a source that evaluates it. The design intent is that governed answers and exploratory answers should look different to the user, so nobody mistakes a draft query for a certified metric.

Trade-off: don't pretend there's one right answer

If your company has a small set of KPIs that drive decisions, a semantic layer pays for itself quickly. If your problem is a sprawling warehouse where people can't find tables, retrieval and metadata matter more than metric definitions. If you have both problems, you need both. Be wary of any pitch that tells you one component solves everything.

🛠️ A Worked Example: Guardrails Around Generated SQL

Q: What does a defensive implementation look like in practice?

Below is an illustrative sketch of a wrapper that applies the lessons above. It checks that tables and columns exist (hallucinated tables and columns remain unresolved at Uber), filters out statements that don't look like queries, and attaches the provenance a human needs to review the answer. It is an example of the pattern, not production code.

# Illustrative sketch of guardrails around LLM-generated SQL.
import re

# A cheap first filter only. It does NOT enforce read-only access:
# "SELECT 1; DROP TABLE x" passes, and WITH can front data-modifying
# statements in some engines. Real enforcement is a read-only database
# role plus a proper SQL parser.
LOOKS_LIKE_QUERY = re.compile(r"^\s*(with|select)\b", re.IGNORECASE)


def validate_generated_sql(sql, known_schema, referenced_tables, referenced_columns):
    # known_schema: {"table_name": {"col_a", "col_b"}}
    problems = []
    if not LOOKS_LIKE_QUERY.match(sql):
        problems.append("Statement must start with SELECT or WITH.")
    for t in referenced_tables(sql):  # e.g. via a SQL parser
        if t not in known_schema:
            problems.append(f"Unknown table referenced: {t}")
    for t, c in referenced_columns(sql):  # (table, column) pairs, also via a parser
        if t in known_schema and c not in known_schema[t]:
            problems.append(f"Unknown column referenced: {t}.{c}")
    return problems


def answer(question, retrieve_tables, generate_sql, run_query,
           known_schema, referenced_tables, referenced_columns):
    tables = retrieve_tables(question)          # retrieval over catalog metadata
    sql = generate_sql(question, tables)        # LLM step
    problems = validate_generated_sql(sql, known_schema, referenced_tables, referenced_columns)
    if problems:
        # Feed errors back for one correction attempt, then give up loudly.
        sql = generate_sql(question, tables, feedback=problems)
        problems = validate_generated_sql(sql, known_schema, referenced_tables, referenced_columns)
        if problems:
            return {"status": "refused", "problems": problems}
    return {
        "status": "ok",
        "sql": sql,                     # always show the SQL
        "tables_used": tables,          # and where the context came from
        "rows": run_query(sql),         # run it under a read-only database role
    }

Three design choices are worth copying even if you change everything else:

• Show the SQL. Both the user and your evaluation process need it.
• Show the tables used. When an answer is wrong, the first question is almost always "did retrieval pick the right tables?"
• Refuse loudly. A visible failure beats a silent wrong answer.

Trade-off: latency and refusals

Validation adds work to every request, and a failed check triggers a second generation call, so slow answers get slower. Refusals also cost something with users: someone who gets "I can't answer that" three times may stop using the tool. You can tune this by loosening checks for exploratory answers and keeping them strict for anything labeled governed. Either way, pick the trade-off on purpose, because a wrong answer that arrives quickly is the more expensive failure.

🔮 Where I Think This Goes

Q: What would I stand behind as a prediction?

Based on the sources, and tied to the thesis I opened with:

• Productivity gains are plausible, especially for drafting, explaining, and debugging. The Uber and LinkedIn user-reported results point the same way, but they are not controlled measurements, and I wouldn't extrapolate them.
• Reliability will come mostly from architecture, if the hypothesis holds. The more the system narrows what the model must decide, through semantic layers, retrieval, and stepwise generation, the more deterministic the outcome should be. Rumiantsau and Fokeev's finding (per the abstract) that the context document accounts for essentially all the significant variance, while model choice within tier does not, is the strongest support.
• Evaluation will become a core data-platform capability, like data quality monitoring is now. Uber, LinkedIn, and Snowflake all describe the difficulty of measuring accuracy.
• Pipeline authoring is the open question. ELT-Bench showed a large gap in early 2025. Whether that gap has closed is something I couldn't establish, and I'd treat anyone claiming either extreme with suspicion until they show a benchmark.

What I'd not predict is a specific accuracy number or date. The sources show too much variation in definitions and datasets to extrapolate.

✅ Recap

The through-line of 2024 to 2026, in the evidence I could find:

• Text-to-SQL moved from demo to production or limited-release systems at Uber and LinkedIn, built as multi-agent systems with retrieval and self-correction.
• Context mattered a lot. Better table descriptions improved LinkedIn's retrieval, and one study reports (per its abstract) that a 4 KB semantic document improved accuracy by 17 to 23 points across three frontier models. Newer models helped too.
• Semantic layers narrow what the model can get wrong, at the cost of scope. In dbt Labs' small vendor benchmark, in-scope accuracy reached 100% for both 2026 models.
• Evaluation is hard, and the numbers can mislead in specific ways: user satisfaction is not correctness, and a single gold query penalizes valid answers.
• End-to-end pipeline building lagged badly in the one benchmark I found, though that was an early-2025 snapshot.

🚀 Next Steps

If you're a data engineer or architect, here is what I'd do in order. The specific numbers are my rule of thumb, not thresholds from any source.

1. Audit your metadata. Pick your 50 most-queried tables and check whether their descriptions would let a stranger choose the right one. Fix those first.
2. Define your top metrics once. Choose the five to ten KPIs that drive decisions and put them in a semantic layer or equivalent, with filters and grain stated explicitly.
3. Build your own gold set. Collect 100 or more real questions from your users, write multiple valid gold queries where needed, and compare result sets rather than SQL text.
4. Measure each stage. Track retrieval, generation, and execution separately, and run each question several times to see the variance.
5. Ship inside the tools people already use. LinkedIn's 5–10x adoption jump from embedding the assistant in an existing platform is a strong hint.
6. Make failures visible. Use the guardrails above, and prefer a clear refusal over a plausible guess.
7. Treat pipeline-authoring claims skeptically. Test any agent on your own pipelines, with review and tests in place, before trusting it with production.

💬 Over to You

Which part of your stack has changed the most since 2024: the interface people use to ask questions, the metadata underneath, or the way you evaluate results? I'd like to hear the numbers from your own setup in the comments.

Sources referenced

• dbt Labs (Jason Ganz, Benoit Perigaud), Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update (vendor benchmark of its own product)
• Rumiantsau and Fokeev, Semantic Layers for Reliable LLM-Powered Data Analytics (abstract only)
• Uber Engineering, QueryGPT
• LinkedIn Engineering, SQL Bot: Practical Text-to-SQL for Data Analytics
• Chen et al., Text-to-SQL for Enterprise Data Analytics (abstract only)
• Snowflake Engineering, Cortex Analyst: Evaluating Text-to-SQL Accuracy for Real-World BI (vendor-run evaluation)
• Jin, Zhu, and Kang, ELT-Bench