What one month of public benchmarks says about my text-to-SQL access layer

schemagate is an open-source layer for text-to-SQL that starts from one question: what is this caller allowed to see? It shows the model only the tables the current user may read, before any SQL is written, reads the permissions from your database's own grants, and refuses a query that touches anything else before it reaches the database.

This month I ran it on public benchmarks. Every number below comes from that benchmark's own evaluator, and the predictions are in the repo.

Access control

• RBAC Text-to-SQL (SIGMOD '27): 90.1 AC-F1, against 88.1 for the paper's best setup with the same model.
• Spider-ACL: 0.984 refusal F1 over all 19,624 question-and-role pairs; best published 0.933.
• 0 queries read data the role was not granted, out of 8,128 answered.

BIRD

• Dev set, all 1,534 questions: 77.1% execution accuracy with one model call per question. No candidate voting, no agent loop.
• Mini-Dev: 81.0 / 72.6 / 72.4 on SQLite / PostgreSQL / MySQL, above the current listed leaders. The leaderboard pull request is waiting on the maintainers.
• Hidden test set: submitted; the BIRD team runs it.

Spider 2.0

• SQLite questions (135): 71.9% with one call per question. The best published agent on the same subset reports 57.8% Pass@1.
• The BigQuery and Snowflake runs for a full leaderboard entry are under way.

Oracle

• UniQL, Oracle dialect, 1,534 questions: 64.5% on the first try, against a best published baseline of 49.6. Two prompt rules about Oracle syntax did most of it.

Adoption

About 10,000 downloads on PyPI so far (pypistats, mirrors excluded). Apache-2.0. Works with Oracle, PostgreSQL, SQL Server, MySQL and SQLite, as an MCP server, a LangChain retriever or a CLI.

pip install schemagate

What these numbers are not: the Mini-Dev rows are self-reported until BIRD merges them, single runs vary by about a point, and the Spider 2.0 figure is one third of that benchmark. The details and every caveat are on the benchmarks page.

GitHub: https://github.com/ashishsinha1602/schemagate · a star helps other people find it.