Where Business Meaning Lives

The schema underdetermines the question

A database schema is a description of storage. It names tables, columns, types and keys, and with respect to what the bytes are it is complete. A question is not asked in those terms. Someone asks for churn last quarter, or net revenue by region, or which accounts are at risk, and none of those phrases appears in the schema. A mapping from the second vocabulary to the first does exist, but it lives in analysts' heads, in dashboard definitions, in a wiki nobody has updated, and in the shape of queries people have already written. This literature is about what happens to that mapping when the translator is a language model.

For most of the field's history the difficulty was hidden by the benchmarks. Academic text-to-SQL datasets used small, clean, well-named schemas where a column called population really did mean population and where the question's words overlapped the schema's words. Systems trained on them looked competent. Then the benchmarks moved. KaggleDBQA kept the original column names from real uploaded datasets, and a state-of-the-art parser fell to 13.56 percent exact match. BIRD put 12,751 questions over 95 databases from 37 professional domains and named the missing ingredient in its own framing, external knowledge sitting between the question and the database contents. Spider 2.0 moved to enterprise warehouses where a single project routinely carries more than a thousand columns, and an o1-preview code agent solved 21.3 percent of those workflows after solving 91.2 percent of Spider 1.0.

The collapse is not a model deficiency in any interesting sense. The same models write correct SQL once they are told what the columns mean. What the collapse measures is that the question-to-SQL mapping is not a function of the schema. Two databases with identical structure and different business conventions require different queries for the same English sentence, and nothing in the schema distinguishes them. Every later theme here is a different answer to where that missing information should live and who is accountable for it.

The diagnosis predates the current generation of models. Baik, Jagadish and Li named the semantic gap in 2019 and proposed mining SQL query logs to close it, reasoning that the organization has already recorded the mapping in what its analysts wrote. Hold onto that proposal. The idea that the missing knowledge is recoverable from artifacts a company already produces returns three more times in this map, and it has still not been tested against a modern model.

Three later results sharpen the diagnosis. EntSQL holds dialect and schema scale fixed, every database is SQLite and the schemas hold 35 tables, and varies only the supply of business knowledge. The best of eight systems goes from 6.8 percent with the schema alone to 21.4 percent with the expert's evidence snippets, and on a 212-question sample a human expert handed the same evidence goes from 33.5 to 84.0. Biswal and Patel show the same thing from the workload side. Databricks users ask which reviews are positive, why sales dropped, and what the quarter-over-quarter trend is for the retail vertical over a table holding only account names, products and revenue. Text2SQL scored 17 percent on 80 such questions, and the SQL it returned parsed and ran. The oldest statement of the problem is Xiao's 2018 survey of ontology-based data access, written before language models were involved: Statoil's Slegge database held about 1,500 tables, a formation-pressure question needed a four-table join with two filters, and geologists were not expected to know how the schema was organized because the schema was organized for applications rather than for questions.

Practitioners running agents on warehouses in 2026 describe the failure in the same terms. Two agents connected to the same warehouse wrote valid SQL and disagreed about whether revenue includes pending invoices, and the slow part of every session was re-explaining the business logic. Every fix proposed in that thread moves the definition out of the prompt and into something the agent has to route through.

1. Towards a Theory of Natural Language Interfaces to Databases

   Popescu · 2003

   Synthesis

   Plain-language abstract PRECISE is a natural-language interface to databases built around a decidability result rather than around a parser. The authors define a class of questions they call semantically tractable, prove that their system maps every question in that class to the right SQL, and show that the system can tell, for any question it is given, whether the question is in the class. Questions outside it get a request for a paraphrase instead of an answer.

   Motivation The authors argue from user trust rather than from accuracy. A person will tolerate an interface that says it did not understand, and will stop trusting one that returns a plausible wrong answer, in the same way they would abandon a mouse that occasionally moved the pointer somewhere random. So the design target is a system that is never confidently wrong, which means the system needs a principled account of what it can and cannot attempt.

   Methodology A question is tokenized against the database's own elements: relations, attributes and values, with a lexicon supplying the word stems that match each one. A question is semantically tractable when some complete tokenization maps one-to-one onto a set of database elements with every attribute token paired to a compatible, syntactically attached value token. Matching is reduced to a max-flow problem, and syntactic attachment constraints come from the parse tree. Evaluation is on three of Ray Mooney's benchmark databases, restaurants, jobs and geography, with hand-written gold SQL, plus a comparison against Mooney's learning interface and Microsoft's English Query product.

   Results Theorem 1: given a lexicon and an attachment function, PRECISE is sound and complete for any semantically tractable question, and Corollary 1 adds that it rejects everything else as intractable. The tractable class turned out to be large: 97 percent of restaurant questions, 88 percent of jobs and 77.5 percent of geography. Inside the class the system made no mistakes on any of the three databases, and its precision advantage over Mooney's interface was significant for geography and jobs at p = 0.05. The rejections are diagnosed against the database, not the sentence: an unknown word, an unimplemented function, or a fact the database does not store, as in a question about the population density of cities when the schema records density only for states.

2. An End-to-end Neural Natural Language Interface for Databases

   Utama · 2018 arXiv

   Synthesis

   Builds a neural NLIDB that operates without hand-written rules or per-database annotation, using synthesized training data and a schema-aware model.

   Why it matters It is the cleanest early demonstration of what the schema alone supports, which is the baseline every knowledge-supplying method in the next theme is measured against.

3. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

   Yu · 2018 arXiv

   Synthesis

   Establishes cross-domain text-to-SQL as the task: models are tested on databases they never saw in training, over 200 databases with multiple tables and complex SQL including joins, nesting and set operations.

   Why it matters It set the target the rest of this literature aims at, and its schemas are clean and well-named enough that the semantic gap stays invisible, which is why later benchmarks had to be built.

4. A Comparative Survey of Recent Natural Language Interfaces for Databases

   Affolter · 2019 arXiv

   Synthesis

   Surveys 24 natural language interfaces to databases from 2010 onward and classifies them by translation technique, from keyword matching through grammar-based parsing to neural approaches.

   Why it matters Useful for what its taxonomy does not contain: there is no category for where business meaning is stored, which is the axis this map is organized on.

5. Bridging the Semantic Gap with SQL Query Logs in Natural Language Interfaces to Databases

   Baik · 2019 arXiv

   Synthesis

   Names the semantic gap between how users describe their information need and how the database stores it, and proposes mining the organization's SQL query log to recover the mapping automatically.

   Why it matters The diagnosis is pre-LLM and the proposed cure is the recurring idea of this map, that the missing knowledge already exists in artifacts the organization produced and has never been extracted with a modern model.

6. KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers

   Lee · 2021 arXiv

   Synthesis

   Plain-language abstract KaggleDBQA is a small text-to-SQL evaluation set built from real databases people uploaded to Kaggle, keeping their original abbreviated column names and unnormalized values, with questions written by people who had access to the data but not to the schema. It also runs the cleanest documentation ablation in this literature, measuring what a parser gains when the database's own documentation is supplied to it.

   Motivation The academic benchmarks of the time normalized their schemas and had question writers work from the schema, which means the questions were already phrased in the schema's vocabulary. The authors argue that this removes the specific difficulty of the deployed task, where a column is called something like pol_eff_dt and the person asking has never seen it.

   Methodology Eight real databases are taken from Kaggle with their original naming and values intact. Question writers see the data but not the schema, so their phrasing reflects how the domain is discussed rather than how the tables are named. The parser is then evaluated in three conditions: zero-shot, after in-domain fine-tuning, and with column descriptions drawn from each database's published documentation added to the input.

   Results A state-of-the-art parser scores 13.56% exact match zero-shot on 272 examples, rising to 17.96% with in-domain fine-tuning and 26.77% once documentation-derived column descriptions are supplied. Documentation is worth 8.81 points, about two thirds of the total lift and the largest single measured intervention in this corpus. It is also worth reading the other way: with the documentation supplied, the system still answers under 27% of the questions correctly, which bounds how much of the problem supplying meaning can solve on its own.

7. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs

   Li · 2023 arXiv

   Synthesis

   Plain-language abstract BIRD is a text-to-SQL benchmark built on databases big enough and messy enough to resemble the ones companies actually run. It holds 12,751 question and SQL pairs over 95 databases totalling 33.4 GB across 37 professional domains, and it ships an external knowledge note alongside each question so that the domain facts a human analyst would already know are part of the stated task rather than something the model is quietly expected to guess.

   Motivation By 2023 the leaderboards on the previous generation of benchmarks were close to saturated, and the reported numbers implied that natural-language querying was close to solved. The authors' position is that those benchmarks used small, clean, well-named academic schemas, and that the difficulty of the real task lives in exactly what those schemas removed: dirty and inconsistent stored values, hundreds of tables and columns per database, domain conventions that appear nowhere in the schema, and queries whose runtime matters because the data is large.

   Methodology The databases are collected from real sources across 37 domains rather than constructed, and the questions are written by annotators working with database experts. Each question carries an external knowledge annotation, which is the piece of domain understanding needed to map the question onto the schema. Alongside execution accuracy the benchmark scores a Valid Efficiency Score, so a query that returns the right rows by an unreasonably expensive path is distinguishable from one that does not.

   Results ChatGPT reaches 40.08% execution accuracy where human experts reach 92.96% on the same questions, a gap of roughly 53 points. The size and location of that gap is the paper's real contribution to this argument: the failures are not parse errors, they are the model attaching the wrong business meaning to a plausible-looking column. BIRD is also the first widely adopted benchmark to concede in its design that the schema alone is not a sufficient input to the task.

8. Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL

   Hong · 2024 arXiv

   Synthesis

   Organizes the LLM-era text-to-SQL literature by datasets, evaluation metrics, prompting methods and fine-tuning approaches, and identifies knowledge grounding as one of the open challenges.

   Why it matters Read alongside Affolter it shows the field's framing did not change when the models did: both surveys are organized by translation technique, five years apart.

9. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows

   Lei · 2024 arXiv

   Synthesis

   Plain-language abstract Spider 2.0 is a text-to-SQL benchmark built from real enterprise data workflows rather than from constructed question sets. Its 632 problems run against BigQuery, Snowflake and DuckDB databases that frequently exceed a thousand columns, require dialect-specific SQL, and often involve multi-step transformations and project code rather than a single query.

   Motivation Spider 1.0 was effectively solved, and the authors wanted to know whether that meant the task was solved or the benchmark had stopped resembling the task. Real enterprise work involves finding the relevant tables among thousands, reading dialect documentation, understanding transformations defined elsewhere in a codebase, and producing SQL that runs on a specific warehouse.

   Methodology Problems are drawn from real workflows on cloud warehouses. Solving one typically requires exploring database metadata, consulting dialect documentation, and reading project-level code, so context acquisition is part of the task rather than a preprocessing step performed for the model. Systems are evaluated as agents with access to the environment rather than as single-shot parsers.

   Results An o1-preview code agent solves 21.3%, against 91.2% on Spider 1.0 and 73.0% on BIRD. The 70-point drop from Spider 1.0 measures how much of the earlier score came from the benchmark's simplifications. For the argument in this map, the important structural point is that at this scale the question stops being whether to supply the model with business context and becomes how the system decides what context to go and look at.

10. LiveSQLBench: contamination-free text-to-SQL over industrial-scale databases with a hierarchical knowledge base

    BIRD team · 2025

    Synthesis

    Plain-language abstract LiveSQLBench is a text-to-SQL benchmark from the BIRD team that pairs industrial-scale PostgreSQL databases with a curated hierarchical knowledge base per database, and refreshes its task set over time to limit contamination. The knowledge base is a directed graph of business definitions in which definitions depend on other definitions, so answering a question can require composing a chain of them.

    Motivation Benchmarks that append business context to a prompt as prose treat meaning as a retrieval problem: find the right paragraph, paste it in. The observation behind LiveSQLBench is that real business definitions are compositional. Net revenue is defined in terms of gross revenue and returns, each of which is defined in terms of something else, and a system that retrieves one definition without its dependencies has retrieved a fragment.

    Methodology Each database ships with a knowledge base whose entries are typed and carry explicit dependency links to the entries they build on. Tasks are split into query and management categories, evaluated by executing generated SQL against the live database and comparing results under a normalization procedure. The task set is periodically refreshed so that models trained after a release are not evaluated on data they may have seen.

    Results Public leaderboard results as of 2025-05-28 put o3-mini at 47.78% and Claude Sonnet 4 at 42.59%. Independent measurement of the public Large-v1 release for this explorer finds 480 records over 18 databases, 971 tables and 17,749 columns, and a knowledge base of 1,090 entries (430 calculation, 462 domain, 198 value illustration) connected by 945 dependency edges, with 560 entries carrying at least one dependency, a longest chain of six edges, multi-hop structure present in all 18 databases, and no duplicate identifiers, dangling references, self-edges or cycles. That structure is what makes this the only public benchmark treating the business-knowledge layer as a first-class versioned artifact.

11. Text2SQL is Not Enough: Unifying AI and Databases with TAG

    Biswal · 2024 arXiv

    Synthesis

    Plain-language abstract Biswal and Patel asked whether text-to-SQL and retrieval-augmented generation cover the questions business users actually ask of their data. They defined table-augmented generation (TAG), in which a query is synthesized, executed in the database, and the returned rows are handed to a language model that writes the answer, and built an 80-question benchmark from BIRD where each question needs either world knowledge or semantic reasoning. Text2SQL, RAG and two stronger baselines answered at most 20 percent of the questions with exact match. Hand-written TAG pipelines answered 55 percent, at the lowest execution time of any method.

    Motivation Databricks customers and internal users asked which reviews are positive, why sales dropped in a period, and what the quarter over quarter trends are for the retail vertical, questions that need sentiment classification over text, aggregation across many rows, or knowledge of how the business defines its terms. Text2SQL serves only questions with a relational equivalent, and RAG serves only point lookups to a few records followed by one LM call that is error-prone at counting and math and degrades on long context. No benchmark had measured either paradigm on questions that require the language model's own knowledge or reasoning.

    Methodology The authors took match-based, comparison, ranking and aggregation queries from 5 BIRD databases (california_schools, debit_card_specializing, formula_1, codebase_community, european_football_2) and modified them so that 40 required parametric knowledge and 40 required reasoning, 20 per query type, with human-labeled answers. They ran Llama-3.1-70B-Instruct on vLLM across 8 A100 GPUs as the language model, SQLite3 for SQL, and an E5 base embedding model with a FAISS index retrieving 10 rows for RAG. Baselines were Text2SQL, RAG, retrieval with LM reranking, and Text2SQL whose retrieved rows were then fed to the LM; the TAG pipelines were hand-written in LOTUS with semantic filter, rank and aggregation operators. Accuracy was exact match for match-based, comparison and ranking queries; aggregation queries were judged qualitatively.

    Results Overall exact match was 17 percent for Text2SQL, 0 percent for RAG, 2 percent for retrieval with LM rerank, 13 percent for Text2SQL plus LM, and 55 percent for hand-written TAG. TAG scored 60 percent on match-based, 65 percent on comparison and 40 percent on ranking queries, and 53 percent on knowledge and 60 percent on reasoning questions. Text2SQL fell to 10 percent on ranking and on reasoning questions, Text2SQL plus LM hit context-length errors when many rows came back, and RAG answered no question correctly. Hand-written TAG averaged 2.94 seconds per query, up to 3.1 times faster than the other baselines, and on an aggregation question about races at Sepang it summarized every race from 1999 to 2017 where RAG listed 4 and Text2SQL plus LM used no database data at all.

12. EntSQL: A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge

    Liao · 2026 arXiv

    Synthesis

    Plain-language abstract The question is whether a model can write the SQL a business question needs when the definitions live in internal documents rather than the schema. Liao and colleagues collected 1,066 real enterprise questions across five business domains, paired each with a gold query, expert evidence snippets, and a long synthesized business document, then tested seven standalone models and Claude Code. With the schema alone the best system reached 6.8 percent execution accuracy; with the full document 15.9 percent; with only the concise evidence 21.4 percent. A human expert given the same evidence reached 84 percent on a sample, so the gap is in applying the rule, not only in finding it.

    Motivation Spider tests schema generalization, BIRD adds large databases and short external evidence, and Spider 2.0 adds industrial workflows with public documentation, but none covers private enterprise content: internal metric definitions, reporting conventions, fiscal-year rules, organization and product mappings, and management adjustments. A question such as comparing FY2024 and FY2025 revenue growth for the top 10 products cannot be answered from public knowledge. The authors built EntSQL so that each instance requires assimilating enterprise rules that are unavailable from public knowledge.

    Methodology Questions came from real user Q&A logs in treasury, finance, human resources, business management, and party building, keeping complex analytical queries with joins, subqueries, aggregation, and temporal comparison. Domain experts wrote the gold SQL, concise evidence snippets naming the key rules, and a synthesized domain document; sensitive entities were replaced through curated mappings with dimension-table anonymization propagated to fact tables, and Chinese originals were translated to English and reviewed. The result is 1,066 aligned examples over 15 database groups, 35 tables, and 1,489 columns, with difficulty set by gold SQL length. Eight systems (Claude Opus 4.6, Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro, Qwen 3.6 Max, Kimi K2.6, GLM 5.1, and Claude Code on Sonnet 4.6) ran at temperature 0 against SQLite in three settings: question only, question plus document, question plus evidence.

    Results Best average execution accuracy was 6.8 percent with the schema alone, 15.9 percent with the document, and 21.4 percent with evidence, and Claude Code led in all three. Best Hard accuracy without knowledge was 0.8 percent. Averaged over systems with evidence, finance stayed near 0 while human resources reached 30.4 percent and treasury 23.4; the document alone helped treasury and slightly hurt human resources. On 212 sampled examples the expert scored 33.5, 46.2, and 84.0 percent across the three settings against Claude Code's 6.1, 14.6, and 20.3. Of 982 Qwen 3.6 Max failures, 54.6 percent were wrong filters, 14.4 percent wrong scope, and 10.0 percent wrong aggregation; Chinese and English inputs differed by at most 0.9 points.

13. Do Agents Need Semantic Metadata? A Comparative Study in Agentic Data Retrieval

    Chen · 2026 arXiv

    Synthesis

    Plain-language abstract Chen, Halevy and Noy asked whether an agent looking for datasets does better when it searches structured schema.org metadata than when it searches the open web. They built 2 agents that differed only in the index they queried, ran both on 58 crowdsourced data-seeking questions, and graded every returned page for relevance, machine accessibility and page type with a validated Gemini 2.5 Pro autorater plus human review. Relevance was equal in the 2 conditions at about 60 percent Highly Relevant. The metadata-backed agent returned machine-readable, registry-hosted, fully FAIR-compliant datasets far more often, 46.4 percent against 28.0 percent, but answered 16 fewer questions because most data pages carry no annotation.

    Motivation Agents increasingly retrieve data on a user's behalf, and the open web mixes dataset registries with narratives, dashboards and portals that describe data without exposing it. Schema.org/Dataset markup was designed to make datasets findable and accessible, yet its value to an agentic retriever had not been measured against plain web search. Chen, Halevy and Noy framed the question through the FAIR principles and asked what the metadata buys at the point of use.

    Methodology Both agents ran on the Agent Development Kit with Gemini 2.5 Pro at temperature 0, returned at most 3 datasets as name and URL, and could answer that no relevant dataset was found. The Baseline Agent queried the Google Search index with data-seeking keywords appended; the Semantic Agent queried the Google Dataset Search index of about 90 million schema.org/Dataset records, kept after a classifier removed roughly 80 percent of annotated pages for invalid or misused markup. Queries were the 58 English keyword questions of the NTCIR-16 Data Search 2 IR subtask. Each result page was frozen as Markdown and rated by a Gemini 2.5 Pro autorater on a relevance scale of -1 to 2, an accessibility scale of 1 to 6, and 7 page types; 2 authors labelled an 11 percent gold set, on which the autorater reached kappa 0.73 to 0.78, and the 31 percent of pages the scraper could not fetch went to human raters.

    Results The baseline answered 56 of 58 queries with 164 datasets and the semantic agent 40 queries with 112. Highly Relevant rates were 60.4 and 60.7 percent. The semantic agent's results were machine-readable 71.4 percent of the time against 48.7 percent and sat on data registries 88.4 percent against 61.0 percent, with narrative pages down 86.6 percent and discovery portals eliminated. Fully FAIR-compliant precision rose from 28.0 to 46.4 percent (p < 0.01), a 65.7 percent relative gain, while results per answered query fell from 1.30 to 0.82 without reaching significance. The authors proposed a hybrid that queries the metadata layer first and falls back to open search on an empty answer.

14. I’m starting to think Text-to-SQL is the easy part of the problem, and context drift is the part that actually breaks things.

    Working-Chemical-337 · 2026

    Synthesis

    The most developed reply (u/CrownHim) argued that clean dbt models do not remove the problem because an agent with table-level access keeps full expressivity and can always find a new wrong way to compute revenue, at the wrong grain, through the wrong join, or with a filter nobody intended; clean models only make the wrong number look more believable. The proposed remedy was to shrink the agent's job from 'write SQL for revenue' to 'call the certified metric revenue_realized', so that intent maps to a metric name rather than to SQL, with the warning that a raw SQL passthrough added for flexibility moves the drift instead of removing it. u/roman-kir added that drift is not a failure mode but the only mode available when definitions have no structural home outside the prompt, that the boundary has to be complete to guarantee anything, and asked whether teams keep that boundary over time or let raw SQL paths creep back in, a question the thread did not answer.

    Why it matters Field evidence for the branch's premise from people running agents on warehouses: the failures described are definitional, not syntactic, and every fix proposed moves meaning out of the prompt and into something the agent must route through.

15. Ontology-Based Data Access: A Survey

    Xiao · 2018

    Synthesis

    The survey's motivating case is Statoil's Slegge database, with about 1500 tables and 1700 views; the main wellbore table has 38 columns, formation pressure for a wellbore needs a 4-table join with 2 filters, and stratigraphic information needs a join with 5 more tables. Geologists are not supposed to know how Slegge is organised, existing SQL templates are error-prone, and asking an IT expert can take days or even weeks. The authors state the general reason: databases are designed to serve applications, so their structure and meaning are obscure to most users, and the stored data is redundant, mixed with information only needed for company processes, and incomplete relative to the business domain.

    Why it matters Stated in 2018 with a real industrial schema, before any language model was involved: the schema is organised for applications, not for the question, and the fix proposed is a conceptual vocabulary rather than a better query writer.

Supplying meaning: what documentation buys

If the schema underdetermines the question, the obvious repair is to write the missing part down and hand it to the model. This theme is about how far that gets you, and it is the theme with the cleanest quantitative answer in the whole map.

Two measurements bracket it. KaggleDBQA ran the ablation in 2021: a zero-shot parser at 13.56 percent exact match, in-domain fine-tuning to 17.96, and then adding column descriptions taken from each database's own documentation to 26.77. Documentation contributed 8.81 points, roughly two thirds of the total lift, which is the single largest measured effect of any intervention in this literature. It also left the system at 26.77 percent. Five years later Rumiantsau and Fokeev ran a paired single-shot protocol over 100 questions on a retail schema in ClickHouse with three frontier models, and a four kilobyte hand-written markdown document describing measures, conventions and disambiguation rules moved accuracy by 17 to 23 points, with every cross-condition comparison significant below p equal to 0.01. Their sharpest result is not the lift. Within each condition the three models were statistically indistinguishable from one another, which says the document mattered more than the choice of frontier model.

Those two studies disagree about whether context is sufficient, and the disagreement is informative rather than a wash. Rumiantsau's 100 questions run over one curated star schema where a four kilobyte glossary can plausibly cover the whole semantic surface. KaggleDBQA's databases carry real-world encodings that documentation describes without resolving, and Spider 2.0's thousand-column warehouses have no document of any size that is the missing piece. Documentation closes the gap when the gap is vocabulary. It does not close the gap when the gap is structure or scale.

The 2026 work has mostly moved from whether to supply context to which context to supply, because at enterprise scale the context itself no longer fits. Schema linking became an evidence-acquisition problem under uncertainty, schema refinement became execution-grounded, and database context became something to compress. The most interesting dissent comes from GATE, which argues that the premise of this entire theme is wrong for expert domains: the specification is never finished, domain conventions are chronically under-documented, and the groundings should be discovered from execution feedback rather than declared in advance.

LiveSQLBench's hierarchical knowledge base is the structural version of the same idea. Rather than a flat glossary it is a directed graph of definitions that depend on other definitions, so a question can require composing a chain of them. That shifts the problem from retrieval to multi-hop composition, and it is the reason the next theme is about representation rather than about more text.

Four later measurements put numbers on the same trade. Sequeda, Allemang and Jacob gave GPT-4 the same 13 insurance tables as DDL and as an OWL ontology with R2RML mappings, and execution accuracy went from 16.7 percent to 54.2. The SQL condition scored 0 percent on every question that needed more than five tables; the graph condition scored 35.7 and 38.7 there. GROUND ran four pipelines on one model over 100 questions in a synthetic automotive warehouse. Approved metric and dimension definitions in the prompt took filter errors from 78 percent of questions to zero and grain errors from 15 percent to zero, at 2.9 times the tokens. What the definitions did not touch was which rows the user may see, and that is the next theme's business. EntSQL, built from real enterprise Q&A logs, found the concise expert evidence worth more than the long document it was extracted from, 21.4 against 15.9 percent, and found that even with the rule in hand the agent converted it to SQL at a quarter of the human expert's rate. Chen, Halevy and Noy split the credit further. On dataset retrieval, metadata bought precision about what a result is and whether a machine can use it, and no better match between question and dataset at all.

Practice has moved ahead of the measurements. Lyft's Metric Semantic Layer stores every Golden Metric as YAML with a plain-English description and definitive SQL, owned by two teams, and exposes it to agents through an MCP server. Replit's CEO describes a shared semantic layer over databases, conversations and docs. Neither reports a number. Both are the position this theme tests rather than evidence for it.

1. Bio-SODA: Enabling Natural Language Question Answering over Knowledge Graphs without Training Data

   Sima · 2021 arXiv

   Synthesis

   Answers natural-language questions over scientific knowledge graphs with no training data, ranking candidate interpretations using the graph's own structure and a generic ranking algorithm.

   Why it matters It is the precedent that a rich enough structural description of the domain can substitute for supervision entirely, which is the strongest form of the supplying-meaning claim.

2. KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers

   Lee · 2021 arXiv

   Synthesis

   Plain-language abstract KaggleDBQA is a small text-to-SQL evaluation set built from real databases people uploaded to Kaggle, keeping their original abbreviated column names and unnormalized values, with questions written by people who had access to the data but not to the schema. It also runs the cleanest documentation ablation in this literature, measuring what a parser gains when the database's own documentation is supplied to it.

   Motivation The academic benchmarks of the time normalized their schemas and had question writers work from the schema, which means the questions were already phrased in the schema's vocabulary. The authors argue that this removes the specific difficulty of the deployed task, where a column is called something like pol_eff_dt and the person asking has never seen it.

   Methodology Eight real databases are taken from Kaggle with their original naming and values intact. Question writers see the data but not the schema, so their phrasing reflects how the domain is discussed rather than how the tables are named. The parser is then evaluated in three conditions: zero-shot, after in-domain fine-tuning, and with column descriptions drawn from each database's published documentation added to the input.

   Results A state-of-the-art parser scores 13.56% exact match zero-shot on 272 examples, rising to 17.96% with in-domain fine-tuning and 26.77% once documentation-derived column descriptions are supplied. Documentation is worth 8.81 points, about two thirds of the total lift and the largest single measured intervention in this corpus. It is also worth reading the other way: with the documentation supplied, the system still answers under 27% of the questions correctly, which bounds how much of the problem supplying meaning can solve on its own.

3. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs

   Li · 2023 arXiv

   Synthesis

   Plain-language abstract BIRD is a text-to-SQL benchmark built on databases big enough and messy enough to resemble the ones companies actually run. It holds 12,751 question and SQL pairs over 95 databases totalling 33.4 GB across 37 professional domains, and it ships an external knowledge note alongside each question so that the domain facts a human analyst would already know are part of the stated task rather than something the model is quietly expected to guess.

   Motivation By 2023 the leaderboards on the previous generation of benchmarks were close to saturated, and the reported numbers implied that natural-language querying was close to solved. The authors' position is that those benchmarks used small, clean, well-named academic schemas, and that the difficulty of the real task lives in exactly what those schemas removed: dirty and inconsistent stored values, hundreds of tables and columns per database, domain conventions that appear nowhere in the schema, and queries whose runtime matters because the data is large.

   Methodology The databases are collected from real sources across 37 domains rather than constructed, and the questions are written by annotators working with database experts. Each question carries an external knowledge annotation, which is the piece of domain understanding needed to map the question onto the schema. Alongside execution accuracy the benchmark scores a Valid Efficiency Score, so a query that returns the right rows by an unreasonably expensive path is distinguishable from one that does not.

   Results ChatGPT reaches 40.08% execution accuracy where human experts reach 92.96% on the same questions, a gap of roughly 53 points. The size and location of that gap is the paper's real contribution to this argument: the failures are not parse errors, they are the model attaching the wrong business meaning to a plausible-looking column. BIRD is also the first widely adopted benchmark to concede in its design that the schema alone is not a sufficient input to the task.

4. DBCopilot: Natural Language Querying over Massive Databases via Schema Routing

   Wang · 2023 arXiv

   Synthesis

   Handles collections too large to fit any prompt by routing a question to the right databases and tables first, decoupling schema selection from SQL generation.

   Why it matters When the unit of scale is the number of databases rather than the number of columns, context assembly stops being the right frame and routing takes over.

5. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows

   Lei · 2024 arXiv

   Synthesis

   Plain-language abstract Spider 2.0 is a text-to-SQL benchmark built from real enterprise data workflows rather than from constructed question sets. Its 632 problems run against BigQuery, Snowflake and DuckDB databases that frequently exceed a thousand columns, require dialect-specific SQL, and often involve multi-step transformations and project code rather than a single query.

   Motivation Spider 1.0 was effectively solved, and the authors wanted to know whether that meant the task was solved or the benchmark had stopped resembling the task. Real enterprise work involves finding the relevant tables among thousands, reading dialect documentation, understanding transformations defined elsewhere in a codebase, and producing SQL that runs on a specific warehouse.

   Methodology Problems are drawn from real workflows on cloud warehouses. Solving one typically requires exploring database metadata, consulting dialect documentation, and reading project-level code, so context acquisition is part of the task rather than a preprocessing step performed for the model. Systems are evaluated as agents with access to the environment rather than as single-shot parsers.

   Results An o1-preview code agent solves 21.3%, against 91.2% on Spider 1.0 and 73.0% on BIRD. The 70-point drop from Spider 1.0 measures how much of the earlier score came from the benchmark's simplifications. For the argument in this map, the important structural point is that at this scale the question stops being whether to supply the model with business context and becomes how the system decides what context to go and look at.

6. LiveSQLBench: contamination-free text-to-SQL over industrial-scale databases with a hierarchical knowledge base

   BIRD team · 2025

   Synthesis

   Plain-language abstract LiveSQLBench is a text-to-SQL benchmark from the BIRD team that pairs industrial-scale PostgreSQL databases with a curated hierarchical knowledge base per database, and refreshes its task set over time to limit contamination. The knowledge base is a directed graph of business definitions in which definitions depend on other definitions, so answering a question can require composing a chain of them.

   Motivation Benchmarks that append business context to a prompt as prose treat meaning as a retrieval problem: find the right paragraph, paste it in. The observation behind LiveSQLBench is that real business definitions are compositional. Net revenue is defined in terms of gross revenue and returns, each of which is defined in terms of something else, and a system that retrieves one definition without its dependencies has retrieved a fragment.

   Methodology Each database ships with a knowledge base whose entries are typed and carry explicit dependency links to the entries they build on. Tasks are split into query and management categories, evaluated by executing generated SQL against the live database and comparing results under a normalization procedure. The task set is periodically refreshed so that models trained after a release are not evaluated on data they may have seen.

   Results Public leaderboard results as of 2025-05-28 put o3-mini at 47.78% and Claude Sonnet 4 at 42.59%. Independent measurement of the public Large-v1 release for this explorer finds 480 records over 18 databases, 971 tables and 17,749 columns, and a knowledge base of 1,090 entries (430 calculation, 462 domain, 198 value illustration) connected by 945 dependency edges, with 560 entries carrying at least one dependency, a longest chain of six edges, multi-hop structure present in all 18 databases, and no duplicate identifiers, dangling references, self-edges or cycles. That structure is what makes this the only public benchmark treating the business-knowledge layer as a first-class versioned artifact.

7. Bootstrapping Semantic Layer from Execution for Text-to-SQL

   Lee · 2026 arXiv

   Synthesis

   Argues that requiring a semantic layer to specify groundings in advance fails in expert domains where conventions are under-documented, and instead keeps grounding hypotheses open, executes the parts that are grounded, and accumulates execution-grounded memory.

   Why it matters It is the sharpest published dissent from the authored-semantic-model position, and the two camps have never been compared on a shared benchmark.

8. DPC: Training-Free Text-to-SQL Candidate Selection via Dual-Paradigm Consistency

   Li · 2026 arXiv

   Synthesis

   Selects among candidate SQL queries by checking consistency across two different generation paradigms, without additional training.

   Why it matters Consistency across paradigms is a cheap proxy for correctness when no semantic model exists, and it is the alternative that any enforcement argument has to beat.

9. Database Context Compression for Text-to-SQL on Real-World Large Databases

   Liu · 2026 arXiv

   Synthesis

   Compresses the database context supplied to the model so that large real-world schemas remain usable within a context budget.

   Why it matters It makes the cost side of the supplying-meaning argument explicit: more context is not free, and at enterprise scale the binding constraint is the budget rather than the availability of documentation.

10. EGREFINE: An Execution-Grounded Optimization Framework for Text-to-SQL Schema Refinement

    Wang · 2026 arXiv

    Synthesis

    Refines which schema elements are presented to the model using execution feedback rather than static similarity, treating schema selection as an optimization target.

    Why it matters Once the schema exceeds what fits in context, deciding what to show becomes the substantive design problem, and this is one of several 2026 answers to it.

11. EviLink: Multi-Path Schema Linking with Uncertainty-Guided Evidence Acquisition for Large-Scale Text-to-SQL

    Zheng · 2026 arXiv

    Synthesis

    Treats schema linking as evidence acquisition under uncertainty, following multiple candidate paths and gathering more evidence where the linking is least certain.

    Why it matters It reframes context supply as an active retrieval decision rather than a fixed prompt assembly, which is the direction the enterprise-scale problem forces.

12. Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models

    Rumiantsau · 2026 arXiv

    Synthesis

    Plain-language abstract A paired benchmark asking what a written description of a company's business conventions is worth to a text-to-SQL system, holding everything else fixed. One hundred natural-language questions are run over the Cleaned Contoso Retail dataset in ClickHouse, single-shot, against three frontier models, in a schema-only condition and a schema-plus-semantic-context condition.

    Motivation The semantic-layer argument is usually made either as a product claim or as an architectural preference, without an experiment that isolates the variable. The authors isolate the smallest version of it: not a compiled semantic layer, not a governed query interface, just a document that states what the measures mean, which conventions the business follows, and how to disambiguate the terms that are ambiguous.

    Methodology The semantic context is a roughly 4 KB hand-authored markdown description of measures, conventions and disambiguation rules for the schema. Questions are paired across conditions so each comparison is within-question, generation is single-shot with no retry or self-correction, and the same three frontier models run both conditions. Significance is reported per comparison.

    Results Schema only gives 45.5% to 50.5% across the three models; schema plus the 4 KB document gives 67.7% to 68.7%. Every cross-condition comparison is significant below p = 0.01. The more interesting finding is the within-condition one: the three frontier models are statistically indistinguishable from each other in both conditions, so on this workload the document matters and the choice of model does not. The evidence base is one schema, 100 questions and no peer review, which makes this a strong effect measured on a narrow base.

13. GROUND: Reducing Hallucinations in LLM-Based Enterprise Analytics Through Governed Semantic Definitions

    Sasidharan Pillai · 2026 arXiv

    Synthesis

    Plain-language abstract Can a language model answer reporting questions over a warehouse without inventing a metric, double counting, dropping a required filter, or showing a user rows they may not see? Sasidharan Pillai built GROUND, which puts approved metric definitions, join paths, grain rules, required filters, and the user's row-level predicate into the prompt and then validates the generated SQL against those rules before it runs, retrying on violations. Four pipelines were run on the same model over 100 synthetic questions; only GROUND had zero errors in every category, and the pipeline given exact definitions but no enforcement still leaked across tenants on 35 percent of questions. The result held on 271,718 real NHTSA complaints with independently written gold and across four models from three providers, at about 5 times the token cost of a bare prompt.

    Motivation Enterprise reporting depends on approved metrics, reporting grain, sanctioned join paths, required filters, and row-level security, none of which a physical schema states. A query can parse, execute, and return a clean number while using the wrong formula, counting at the wrong grain, omitting a filter, or ignoring who is asking; it looks authoritative and is silently wrong. Spider, BIRD, Spider 2.0, BEAVER, and EntSQL do not score these dimensions, and the closest prior study supplied a 4 KB semantic document as context and reported 17 to 23 point accuracy gains without separating supplying semantics from guaranteeing them.

    Methodology The synthetic warehouse has 6 dimension tables, 5 fact tables at different grains, 10 approved metrics each with a formula, base grain, required filter set, and allowed dimensions, approved join edges, and row-level predicates for four users; 100 questions (82 expecting SQL, 18 expecting abstention) carry gold generated deterministically from the semantic layer, and an oracle self-test scores that gold at 100 percent with no classifier flags. All four systems used claude-opus-4-8 with structured JSON output: raw CREATE TABLE text, keyword-retrieved column descriptions, approved definitions as context, or the full semantic layer plus the user's predicate with a validate-and-retry loop of up to 3 revisions. The validator checks schema existence, the metric's expression signature, approved join edges, COUNT(DISTINCT) at order grain, required filters, an un-widened security predicate, and cost signals such as a missing date filter. A replication used 271,718 NHTSA complaints with 8 metrics, manufacturer-scoped row-level security, and 40 hand-authored gold queries; an adversarial set of 40 questions ran across Claude Opus 4.8, Claude Sonnet 5, GPT-5.2, and Llama-3.3-70B with three runs each.

    Results On the synthetic set the bare-schema pipeline executed 95.1 percent of queries with 0.000 result accuracy, omitting required filters on 78 percent and violating row-level security on 78 percent; schema retrieval executed 64.6 percent and leaked on 53 percent; definitions as context reached 0.500 value-based accuracy with zero metric, grain, and filter errors but 35 percent security violations; GROUND executed 100 percent, scored 0.951 value-based and 0.793 strict, had zero errors in every category, and handled 94.4 percent of abstention cases. GROUND used 10,714 tokens per question against 2,140 for the bare prompt and added about 30 percent latency; 9 of 82 SQL questions triggered a retry. On NHTSA GROUND reproduced all 40 gold values while baselines violated manufacturer scope on 25 to 33 percent and dropped filters on 63 to 73 percent, and governed queries full-scanned 0 rows against 7,000 to 32,000 for the baselines. Across the four models the enforced filter and security checks stayed at 0.000 plus or minus 0.000, while Claude Opus 4.8 still invented a LAG-based year-over-year metric on all three adversarial runs and Llama-3.3-70B under GROUND fell to 0.63 execution and 0.25 value accuracy.

14. A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases

    Sequeda · 2023 arXiv

    Synthesis

    Plain-language abstract Sequeda, Allemang, and Jacob asked how accurately GPT-4 answers business questions over an enterprise SQL database, and how much a knowledge graph built from an ontology and mappings changes the answer. They built a benchmark from a 13-table subset of the OMG Property and Casualty insurance model with 43 questions spanning simple reporting to KPI aggregations over few or many tables. With a zero-shot prompt holding the DDL, GPT-4 produced a correct result 16.7 percent of the time; with the ontology and a SPARQL endpoint virtualizing the same tables, 54.2 percent. Questions needing more than 4 tables scored 0 percent in SQL and 35 to 39 percent over the graph.

    Motivation The authors saw existing text-to-SQL benchmarks such as Spider, WikiSQL, and KaggleDBQA as misaligned with enterprise use: their schemas are small where enterprise schemas run to hundreds of tables, they skip the reporting, metrics, and KPI questions that drive operational decisions, and they have no business-context layer of metadata, mappings, and ontologies. Without that layer, they argued, an LLM is disconnected from what the data means and produces hallucinated, unexplainable answers, and the size of the effect had not been measured.

    Methodology The schema is a 13-table excerpt of the 199-table, 243-foreign-key OMG P&C model, populated by hand with a couple of rows per table. The 43 questions are classed by question complexity (SELECT-FROM reporting versus aggregations and KPIs) and schema complexity (0 to 4 tables versus more than 4), giving four quadrants of 12, 11, 10, and 10 questions. The context layer is an OWL ontology of insurance concepts and R2RML mappings hosted on data.world, which translates SPARQL against the ontology into SQL over the tables. GPT-4 received a zero-shot prompt with either the DDL or the ontology, at temperature 0.3 with 2048 max tokens and a 60-second timeout counted as failure. Accuracy followed Spider's execution accuracy, aggregated per question as the fraction of repeated runs matching a human-written reference query, over sessions run between 20 September and 13 October 2023.

    Results Average execution accuracy was 16.7 percent for SQL and 54.2 percent for SPARQL over the knowledge graph, a 37.5 point gain. By quadrant, SQL scored 25.5 percent (low question, low schema), 37.4 percent (high question, low schema), and 0 percent in both high-schema quadrants, while the graph scored 71.1, 66.9, 35.7, and 38.7 percent. SQL failures were hallucinated column names, filter values, and joins; SPARQL failures were wrong or reversed paths through the ontology, with no hallucinated class or property observed. The authors concluded that context describing business semantics must be managed as a first-class asset, ideally in a data catalog on a knowledge graph architecture.

15. EntSQL: A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge

    Liao · 2026 arXiv

    Synthesis

    Plain-language abstract The question is whether a model can write the SQL a business question needs when the definitions live in internal documents rather than the schema. Liao and colleagues collected 1,066 real enterprise questions across five business domains, paired each with a gold query, expert evidence snippets, and a long synthesized business document, then tested seven standalone models and Claude Code. With the schema alone the best system reached 6.8 percent execution accuracy; with the full document 15.9 percent; with only the concise evidence 21.4 percent. A human expert given the same evidence reached 84 percent on a sample, so the gap is in applying the rule, not only in finding it.

    Motivation Spider tests schema generalization, BIRD adds large databases and short external evidence, and Spider 2.0 adds industrial workflows with public documentation, but none covers private enterprise content: internal metric definitions, reporting conventions, fiscal-year rules, organization and product mappings, and management adjustments. A question such as comparing FY2024 and FY2025 revenue growth for the top 10 products cannot be answered from public knowledge. The authors built EntSQL so that each instance requires assimilating enterprise rules that are unavailable from public knowledge.

    Methodology Questions came from real user Q&A logs in treasury, finance, human resources, business management, and party building, keeping complex analytical queries with joins, subqueries, aggregation, and temporal comparison. Domain experts wrote the gold SQL, concise evidence snippets naming the key rules, and a synthesized domain document; sensitive entities were replaced through curated mappings with dimension-table anonymization propagated to fact tables, and Chinese originals were translated to English and reviewed. The result is 1,066 aligned examples over 15 database groups, 35 tables, and 1,489 columns, with difficulty set by gold SQL length. Eight systems (Claude Opus 4.6, Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro, Qwen 3.6 Max, Kimi K2.6, GLM 5.1, and Claude Code on Sonnet 4.6) ran at temperature 0 against SQLite in three settings: question only, question plus document, question plus evidence.

    Results Best average execution accuracy was 6.8 percent with the schema alone, 15.9 percent with the document, and 21.4 percent with evidence, and Claude Code led in all three. Best Hard accuracy without knowledge was 0.8 percent. Averaged over systems with evidence, finance stayed near 0 while human resources reached 30.4 percent and treasury 23.4; the document alone helped treasury and slightly hurt human resources. On 212 sampled examples the expert scored 33.5, 46.2, and 84.0 percent across the three settings against Claude Code's 6.1, 14.6, and 20.3. Of 982 Qwen 3.6 Max failures, 54.6 percent were wrong filters, 14.4 percent wrong scope, and 10.0 percent wrong aggregation; Chinese and English inputs differed by at most 0.9 points.

16. Do Agents Need Semantic Metadata? A Comparative Study in Agentic Data Retrieval

    Chen · 2026 arXiv

    Synthesis

    Plain-language abstract Chen, Halevy and Noy asked whether an agent looking for datasets does better when it searches structured schema.org metadata than when it searches the open web. They built 2 agents that differed only in the index they queried, ran both on 58 crowdsourced data-seeking questions, and graded every returned page for relevance, machine accessibility and page type with a validated Gemini 2.5 Pro autorater plus human review. Relevance was equal in the 2 conditions at about 60 percent Highly Relevant. The metadata-backed agent returned machine-readable, registry-hosted, fully FAIR-compliant datasets far more often, 46.4 percent against 28.0 percent, but answered 16 fewer questions because most data pages carry no annotation.

    Motivation Agents increasingly retrieve data on a user's behalf, and the open web mixes dataset registries with narratives, dashboards and portals that describe data without exposing it. Schema.org/Dataset markup was designed to make datasets findable and accessible, yet its value to an agentic retriever had not been measured against plain web search. Chen, Halevy and Noy framed the question through the FAIR principles and asked what the metadata buys at the point of use.

    Methodology Both agents ran on the Agent Development Kit with Gemini 2.5 Pro at temperature 0, returned at most 3 datasets as name and URL, and could answer that no relevant dataset was found. The Baseline Agent queried the Google Search index with data-seeking keywords appended; the Semantic Agent queried the Google Dataset Search index of about 90 million schema.org/Dataset records, kept after a classifier removed roughly 80 percent of annotated pages for invalid or misused markup. Queries were the 58 English keyword questions of the NTCIR-16 Data Search 2 IR subtask. Each result page was frozen as Markdown and rated by a Gemini 2.5 Pro autorater on a relevance scale of -1 to 2, an accessibility scale of 1 to 6, and 7 page types; 2 authors labelled an 11 percent gold set, on which the autorater reached kappa 0.73 to 0.78, and the 31 percent of pages the scraper could not fetch went to human raters.

    Results The baseline answered 56 of 58 queries with 164 datasets and the semantic agent 40 queries with 112. Highly Relevant rates were 60.4 and 60.7 percent. The semantic agent's results were machine-readable 71.4 percent of the time against 48.7 percent and sat on data registries 88.4 percent against 61.0 percent, with narrative pages down 86.6 percent and discovery portals eliminated. Fully FAIR-compliant precision rose from 28.0 to 46.4 percent (p < 0.01), a 65.7 percent relative gain, while results per answered query fell from 1.30 to 0.82 without reaching significance. The authors proposed a hybrid that queries the metadata layer first and falls back to open search on an empty answer.

17. Metric Semantic Layer: How Lyft Governs and Scales Key Data Definitions

    Channe · 2026

    Synthesis

    Channe and Mirchandani describe the problem the Metric Semantic Layer (MSL) was built for: as Lyft grew, different teams used different definitions of the same metric, and with no centralized version control outdated definitions crept into decision-making. MSL is a Python package that stores, for every onboarded metric, a plain-English description and the definitive SQL as YAML with Jinja templates (an example defines rides_completed as 1.0*count_if(is_dropped_off) with a business_owner, an operational_owner, data_sources, and the allowed dimensions and time granularities). The definitions reach consumers through a get_sql() API, the Amundsen catalog, a self-service Metric UI, and an MCP server; the post says the YAML doubles as a knowledge base for agents and that the MCP answers natural-language questions about metrics with greater accuracy and fewer hallucinations, checked by guardrails that compare MCP results against ground truth and an LLM judge. No accuracy figures and no counts of metrics or teams are given.

    Why it matters The context an agent receives is a versioned definition read through a tool rather than prose in a prompt, and the same artifact feeds dashboards, Airflow orchestration and ML models, which is what keeps the meaning from diverging between consumers.

18. We built a self-driving & self-correcting shared semantic layer on top of our databases, conversations, and docs.

    Masad · 2026

    Synthesis

    Masad posted on 3 August 2026 that Replit had built what he called a self-driving and self-correcting shared semantic layer on top of its databases, conversations and docs, that everything in it is queryable and joinable regardless of source, and that anyone at Replit can now ask questions that previously needed a team of data scientists and weeks of work. The post quotes a Replit post saying that when an agent starts with shared context more people can get reliable answers, and links to a write-up of the internal truth layer. No method, numbers or evaluation are given.

    Why it matters A CEO's 1-paragraph statement of the context bet: put the meaning in one shared layer that spans structured and unstructured sources and let the agent start from it. It is a claim rather than a measurement and sits in the map as the position being tested.

Representation: the layer between language and SQL

Supplying meaning as text leaves the model doing the same job with better inputs. The alternative is to change the job: have the model produce something other than SQL, and compile that into SQL with a component that cannot make semantic mistakes. This is the oldest idea in the map and the one most often mistaken for the previous theme.

ATHENA made the argument in 2016 and it is worth reading for the argument rather than the numbers. The system translates a question into OQL, an ontology query language, and only then into SQL, and the stated reason is physical independence: the semantics of the question are decoupled from the physical layout of the tables. It reported 100 percent precision on two domains and 99 percent on a third. Read those numbers with the definitions attached, because ATHENA defines precision over the questions for which it produced any interpretation at all, and recall over the whole workload, at 87.2, 88.3 and 88.9 percent. The gap between the two is the questions it declined. Four years later ATHENA++ extended the ontology reasoning to nested and aggregate business queries, introduced FIBEN with 300 questions over 152 tables and an ontology derived from FIBO and FRO, and reported 88.33 percent on FIBEN and 78.89 percent on Spider development, against a best reported development accuracy of 70.6 percent at the time. A symbolic, ontology-driven system beating the neural leaderboard on the neural leaderboard's own benchmark is the most under-cited fact in this corpus.

The neural line rediscovered the same move from the other direction. IRNet emitted SemQL, an intermediate representation that removes the parts of SQL that have no counterpart in the question. NatSQL went further and stripped set operators and nested subqueries from the target. The decisive experiment is Herzig and colleagues in 2021: holding the model fixed and varying only the intermediate representation, a representation with stronger structural correspondence to the natural language gains 14.8 points on CFQ and 15.0 to 19.4 points on the template splits of three text-to-SQL datasets. The representation was the free variable all along.

The contemporary version has moved past SQL entirely. Semantic operators define a declarative algebra whose operators are themselves model calls, and NL2Pipe compiles a question into an executable pipeline of them rather than into a query, reporting up to 60 percent higher F1 on cross-source workloads. Palimpzest treats the same idea as a query optimization problem. What all of these share with ATHENA is the claim that the model should be asked for a smaller, better-typed object, and that the compiler downstream should own the rest.

The distinction that matters for the next theme: a representation constrains what the model can express, and it does so before generation begins. That is a different mechanism from constraining tokens during decoding, and the two behave differently under measurement.

The live disagreement in 2026 is what language the layer between natural language and SQL should speak. One camp compiles from an intermediate representation. Kim, Khoeurn and Yoon's agent writes a JSON object of metrics, filters and group_by that a deterministic engine compiles to dialect SQL, and it answers 94.15 percent of Spider2-snow against 31.26 for the best prior agent. The guarantee is stated narrowly: the compiler certifies physical names and joins, not the final query, and accuracy on the 18 native-Snowflake tasks was 55.6 percent. SemPlan ran the comparison with the model held fixed and found the compiled envelope worth 3.42 points over direct SQL, on a task where the best architecture was right a quarter of the time. Superset's SIP-182 reduces its 29-field query object to metrics, dimensions and filters, and the Open Semantic Interchange, incubating at Apache as Ossie since June 2026, is the attempt to make that object a file every layer can export.

The other camp keeps SQL. Rill's Metrics SQL restricts SQL rather than replacing it, no joins across metrics views, no SELECT star, measure filters only in HAVING, and compiles the restricted query to ClickHouse, DuckDB, Snowflake or Druid. SLayer defines a column once and lets the agent apply expressions to it. Omni's CTO states the position outright: a definition is relative to the database dialect it targets, and abstracting the dialect away is a trap. UniQL puts a price on the dialect an intermediate representation is meant to hide. The same 1,534 BIRD questions in 16 dialects, the best model at 54.63 percent on average and correct in all 16 on 20.14 percent of questions, with Oracle at 63.75 and Teradata at 37.74 for the same model. Translating at the edges is only as safe as its checker, and CrackSQL's checks are syntactic: the target parser accepts the snippet, the engine runs it, and neither certifies equivalence. The translation surveys say what falls out in the crossing. Nadjib Mami's inventory of more than 40 methods found joins, unions and aggregation among the least supported operations, which are the ones a metric needs.

Both camps descend from ontology-based data access. Xiao's 2018 survey itemizes what a compiled representation pays, rewritings exponential in size, mapping maintenance, and expressiveness limits past OWL 2 QL, and Calvanese's catalogue of mapping patterns hand-classified 1,559 production mapping assertions and found 44.7 percent data-driven, meaning the constraint held only in the rows and never in the schema. That is this map's central claim in another vocabulary.

A third line widens the representation past tables. Merrick's remark that long-form account-executive comments in Salesforce went from the most useless data for a BI tool to the most interesting has three measured forms. TAG composes exact computation in the database with row-wise model reasoning and reaches 55 percent on questions where Text2SQL scores 17 and RAG scores 0, with the pipelines hand-written. SUQL adds ANSWER and SUMMARY as typed SQL functions and returns an entity satisfying every constraint 93.8 percent of the time on Yelp against 57.0 for linearized retrieval. AnnoIndex extracts first into a materialized schema and confines the model to the residual predicate, at 0.87 F1 against 0.46 for vector RAG, and its ablations locate the ceiling: an attribute missing from the schema cannot be recovered at query time.

1. Constructing an Interactive Natural Language Interface for Relational Databases

   Li · 2014

   Synthesis

   Plain-language abstract NaLIR is a natural-language interface that does not try to be right on its own. It converts a question into a query tree, explains that tree back to the user in words, lets the user correct the interpretation, and only then compiles the confirmed tree into SQL with aggregation, nesting and joins.

   Motivation Li and Jagadish start from the position that a natural-language interface cannot be expected to be perfect, and that the real problem is what happens when it is wrong. A user handed a number has no way to tell whether the system understood the question, no way to verify the answer independently, and no guidance beyond rephrasing and hoping. Their fix is to move verification earlier, to a representation a person can actually read.

   Methodology Three components: a transformation from the dependency parse into a query tree, an interactive communicator that explains each ambiguous word or phrase and the reading of the whole sentence and offers up to five interpretations per ambiguity, and a translator that compiles the verified tree block by block into SQL. Evaluated with a user study on the Microsoft Academic Search database, 14 participants recruited with flyers, tasks drawn from 196 distinct query logics enumerated from the MAS website, with the faceted MAS web interface as the baseline and the system run both with and without the interactive step.

   Results With interaction the participants completed 88 of 98 query tasks; the per-difficulty splits were 34 of 34 simple, 34 of 34 medium and 20 of 30 hard, against 26 of 32, 23 of 34 and 15 of 32 without it, and 20 of 33, 18 of 32 and 18 of 33 for the MAS website. The finding that matters most is about detection rather than accuracy: of the 32 failures in the non-interactive condition the participants noticed 7. The other 25 wrong answers were accepted as correct by the person who had asked the question, and the authors note the undetected ones were mostly aggregates, which cannot be checked by inspecting the result.

2. ATHENA: An Ontology-Driven System for Natural Language Querying over Relational Data Stores

   Saha · 2016

   Synthesis

   Plain-language abstract ATHENA is a natural-language querying system that translates a question into a query over a domain ontology first, and only then compiles that ontology query into SQL against whatever physical schema the data happens to live in. The intermediate query language is OQL, and the ontology carries the business concepts and their relationships independently of how the tables are laid out.

   Motivation Systems that map language directly onto a schema inherit every accident of that schema: how it was normalized, what the columns were named, which denormalizations were done for performance. The authors' argument is that a question is asked in terms of domain concepts, so the translation target should be domain concepts, and the mapping from concepts to physical tables should be a separate, maintained artifact.

   Methodology The system performs evidence-based interpretation generation over the ontology, ranks candidate interpretations, and translates the selected one into OQL and then into SQL. It is evaluated on three domains, GEO, MAS and FIN, with precision measured over the questions for which it produced an interpretation and recall measured over the whole workload.

   Results ATHENA reports 100% precision on GEO and MAS and 99% on FIN, with recall of 87.2%, 88.3% and 88.9%. The gap between the two figures is the abstention rate: the system generated no interpretation at all for 11 of 108 finance questions. That is a 2016 system with refusal built into its architecture and visible in its reported metrics, long before the current literature named the behaviour. The durable argument is not the accuracy but the physical independence: an ontology query survives a schema migration, which is the property a semantic layer is actually bought for.

3. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers

   Wang · 2019 arXiv

   Synthesis

   Encodes the schema as a graph and uses relation-aware self-attention to jointly represent schema structure and its alignment to the question.

   Why it matters It makes explicit that binding language to schema elements is a separate problem from emitting syntax, which is the same split the semantic-layer architectures make at the system level.

4. Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation

   Guo · 2019 arXiv

   Synthesis

   Introduces SemQL, an intermediate representation that removes the parts of SQL with no counterpart in the natural-language question, and generates SemQL before deterministically inferring the SQL.

   Why it matters It is the neural rediscovery of ATHENA's move, arrived at independently and framed as a mismatch problem rather than as physical independence.

5. ATHENA++: Natural Language Querying for Complex Nested SQL Queries

   Sen · 2020

   Synthesis

   Plain-language abstract ATHENA++ extends ontology-driven natural-language querying to the nested and aggregate queries that business questions actually require, and introduces FIBEN, a financial benchmark of 300 natural-language queries mapping to 237 distinct complex SQL queries over 152 tables, with an ontology derived from the FIBO and FRO financial ontologies.

   Motivation The first generation of ontology-driven systems handled selection and projection well and nesting poorly, while the business questions that motivate these systems are largely comparative and aggregate: revenue by segment against the prior period, customers whose spend exceeds their cohort median. Existing benchmarks also lacked a domain where the ontology was a real published artifact rather than one constructed for the paper.

   Methodology The system detects the nested-query structure implied by a question and generates the corresponding nested OQL before compiling to SQL. FIBEN is built over a real financial schema of 152 tables with an ontology derived from published financial ontologies, and the system is also evaluated on the Spider development set for comparability with the neural parsing literature.

   Results 88.33% on FIBEN and 78.89% on Spider dev, against a best reported Spider dev accuracy of 70.6% at the time. A symbolic, ontology-mediated system beating the neural leaderboard on the neural leaderboard's own benchmark is the most under-cited result in this corpus, and it has never been re-run against a modern LLM baseline. Whether the result would survive that re-run is one of the open problems this explorer lists.

6. Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing

   Lin · 2020 arXiv

   Synthesis

   Augments the question-schema representation with anchor text drawn from the database's actual cell values, so linking can use the data and not only the column names.

   Why it matters The values carry meaning the schema omits, which is a cheap partial substitute for documentation and a useful control when arguing that a semantic layer was necessary.

7. Natural SQL: Making SQL Easier to Infer from Natural Language Specifications

   Gan · 2021 arXiv

   Synthesis

   Defines NatSQL, a target language that eliminates set operators and nested subqueries and reduces the number of schema items a model must predict, then compiles it back to executable SQL.

   Why it matters It quantifies the tradeoff the whole theme depends on: a smaller target is easier to hit and can express less, and this is one of the few papers that says out loud what it gave up.

8. Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations

   Herzig · 2021 arXiv

   Synthesis

   Plain-language abstract A controlled study of what an intermediate representation contributes to compositional generalization in semantic parsing. The model architecture is held fixed and only the representation the model generates is varied, comparing reversible and lossy designs by how closely their structure corresponds to the structure of the natural-language question.

   Motivation Text-to-SQL systems that generate an intermediate language rather than SQL directly had reported gains, but the gains were confounded with everything else those systems changed. Without holding the architecture fixed there is no way to tell whether the representation is doing the work or the surrounding system is.

   Methodology One pre-trained model is trained to generate several different intermediate representations of the same target queries. The representations vary along two axes: whether the mapping back to the original query is lossless, and how closely the representation's structure mirrors the natural language. Evaluation uses compositional splits, including CFQ and template splits of three text-to-SQL datasets, where the test set requires composing structures seen only separately in training.

   Results Gains of 14.8 points on CFQ and 15.0 to 19.4 points on template splits of three text-to-SQL datasets, from changing the representation alone. This is the experiment that makes representation a first-class variable in this literature rather than an implementation detail, and it is the reason changing what the model generates and constraining how it generates are treated as different interventions in this map rather than as two versions of one idea.

9. Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments

   Gu · 2022 arXiv

   Synthesis

   Proposes that the model rank candidate programs enumerated against the actual environment rather than generate them freely, so every candidate is executable and schema-valid by construction.

   Why it matters It is the limit case of this theme: the representation becomes a finite candidate set and the model's only job is judgment, which removes the fabrication surface entirely.

10. RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL

    Li · 2023 arXiv

    Synthesis

    Separates schema linking from SQL skeleton generation into distinct ranked stages so each can be optimized and inspected independently.

    Why it matters Decoupling is the recurring structural answer in this theme, and this paper is the clearest statement of it inside a single model rather than across a system.

11. Semantic Operators: A Declarative Model for Rich, AI-based Data Processing

    Patel · 2024 arXiv

    Synthesis

    Plain-language abstract Semantic operators extend the relational algebra with operators whose implementation is a model invocation, so a query plan can mix ordinary relational operations with semantic ones such as filtering by meaning or ranking by a described criterion, and the whole plan can be optimized as a plan.

    Motivation Analytical work over unstructured or partly unstructured data currently gets written as bespoke pipelines of prompts. The authors argue that this is where declarative query languages were before the relational model: every pipeline hand-tuned, no separation between what is asked and how it is executed, and no optimizer.

    Methodology The work defines a declarative model in which semantic operations are first-class algebraic operators with defined semantics, implements them in a system that plans and optimizes queries over them, and evaluates the resulting plans against hand-written pipelines on analytical tasks.

    Results The contribution most relevant to this map is not a benchmark number but a direction: it moves the target of generation off SQL entirely. If the model emits a smaller, better-typed object that a planner then executes, then the argument that generation should be constrained by changing the target language rather than by masking tokens is being made in its strongest form.

12. Palimpzest: Optimizing AI-Powered Analytics with Declarative Query Processing

    Liu · 2025

    Synthesis

    Treats analytics over unstructured and semi-structured data as declarative query processing, with an optimizer choosing among physical implementations of semantically equivalent plans.

    Why it matters It supplies the systems half of the semantic-operator argument: once the target is a plan rather than a query, the interesting work moves into the optimizer.

13. Bridge the Last-Mile Gap to Semantic Analytics: Compiling Natural-Language Queries into Semantic Operator Pipelines

    Dong · 2026 arXiv

    Synthesis

    Compiles a natural-language question into an executable semantic-operator pipeline in three phases, a query-data linker, a semantic planner and a code generator, reporting up to 60% higher F1 on complex cross-source workloads.

    Why it matters It is the contemporary competitor to the semantic layer, answering the same question with a compiled pipeline of model calls instead of governed SQL, and no benchmark contains both.

14. A Semantic-Layer-Mediated Agent for Natural Language to SQL over Heterogeneous Enterprise Databases

    Kim · 2026 arXiv

    Synthesis

    Plain-language abstract Enterprise warehouses have hundreds of tables with opaque names, and a model asked to write SQL directly against the raw schema fails on most Spider2 tasks. Kim, Khoeurn and Yoon built an agent that never sees the raw schema. It queries a curated semantic layer through a small JSON intermediate representation, the Semantic Model Query, which a deterministic engine compiles into dialect-correct SQL; the agent lifts physical names from that output and composes the final query itself. Driven by Gemini 3 Pro, the system scored 94.15 percent execution accuracy on the 547-task Spider2-snow benchmark, third on the official leaderboard, where schema-only pipelines score between 2.2 and 31.3 percent.

    Motivation Reported accuracies on Spider2 sit far below the near-saturated numbers on Spider and BIRD, which the authors read as proof that putting the schema in the prompt and asking for SQL does not transfer to production warehouses. They separate two sources of difficulty: grounding, meaning finding the right tables, columns and joins in a large, poorly self-describing schema, and composition, meaning assembling dialect-valid SQL once the blocks are known. Prompting and multi-agent methods such as DIN-SQL, DAIL-SQL, MAC-SQL and CHESS conflate the two in one free-form generation step, so a single wrong column name fails the whole query with no structured surface for recovery.

    Methodology Each physical table is wrapped as a YAML semantic model exposing dimensions, measures and metrics, each carrying a human-readable description and an expr field holding the exact physical column expression; joins are declared once in a per-database join graph. The agent runs a think-act loop with exactly one tool call per turn from three tools: getModelDataElements, convertSmqToSql, which returns SQL plus a five-row preview, and a terminal execute. Querying INFORMATION_SCHEMA is disallowed and no semantic model name may appear as a physical table in executed SQL. Evaluation used Spider2-snow, 547 tasks on Snowflake, with Gemini 3 Pro at temperature 0.1, extended thinking at the high setting, a 16,384-token output budget and a cap of 20 SMQ iterations per instance, scored by the official execution-result protocol.

    Results The system answered 515 of 547 instances, 94.15 percent, against leaderboard entries of 2.20 percent for DAIL-SQL with GPT-4o, 23.58 and 25.78 percent for Spider-Agent with o1-preview and Claude-4-Sonnet, and 31.26 percent for ReFoRCE. By origin class it scored 131 of 135 on SQLite-origin tasks (97.0 percent), 24 of 25 on GA4 (96.0), 350 of 369 on BigQuery-origin (94.9) and 10 of 18 on native Snowflake (55.6). Recurring failures were compiler gaps where a model lacked a needed join relation, composition errors in agent-authored nested time windows and ranking semantics, and columns the semantic layer did not yet expose. The authors state that disabling extended thinking substantially degrades accuracy, that the baselines lacked the per-database knowledge the semantic layer encodes, and that curating descriptions against the evaluation set carries a Goodhart-style risk of encoding expected answers.

15. CrackSQL: A Hybrid SQL Dialect Translation System Powered by Large Language Models

    Zhou · 2025 arXiv

    Synthesis

    Plain-language abstract A query written for one database will often not run on another, and the translators that fix this are either hand-maintained rule tables or a language model that invents functions the target does not have. Zhou, Gao, Zhou and Li built CrackSQL to combine the two. It parses the query into functional pieces, applies rules where they exist, sends only the pieces the target dialect's parser rejects to a language model along with retrieved documentation for the equivalent construct, and checks every result against the parser and the database. On Oracle-to-MySQL and Oracle-to-PostgreSQL translations of a modified BIRD set it introduced no errors in four of five error categories, while SQLGlot, jOOQ, SQLines and direct GPT-4o each did.

    Motivation Rule-based translators such as SQLGlot, jOOQ and SQLines require developers to identify equivalent syntax patterns and encode them by hand; the authors cite more than 2,000 unresolved issues in the jOOQ repository and over 200 contributors on SQLGlot as the maintenance cost. Direct LLM translation avoids the rule writing but hallucinates on long, complex queries, producing non-equivalent operations or functions that do not exist in the target dialect, and it ignores feedback such as database error messages. The demonstration extends the authors' research paper on the same system.

    Methodology Tree-based segmentation decomposes the source query along its syntax tree into functional operations such as a TIMESTAMPDIFF call; rules then normalize custom functions to standard forms and abstract away column and table names. A cross-dialect embedding model, trained with retrieval-enhanced contrastive learning over positive pairs (augmented specifications, same-function elements across dialects, equivalences from existing rule tools) and hard negatives, encodes each element's abstract syntax tree and its textual specification to find the matching target element. Translation runs local-to-global: operations flagged incompatible by the target BNF parser are translated by a hybrid of SQLGlot and the LLM, and operations that fail for lack of context are expanded to adjacent syntax elements until the assembled query parses and executes. Three modes ship, rule-only across 24 dialects, LLM-direct, and rule plus LLM, the latter two for PostgreSQL, MySQL and Oracle, through a web console, a PyPI package and a command line.

    Results On a modified BIRD benchmark, translating Oracle to MySQL and Oracle to PostgreSQL, CrackSQL produced zero errors in four of the five counted categories: syntax rules, functions and operators, keywords, and data types. Rule tools erred inconsistently by direction: jOOQ mistranslated Oracle date and time calculations for MySQL, and SQLGlot dropped or misused keywords when targeting PostgreSQL, which the authors trace to gaps in hand-coded mappings. Direct GPT-4o prompting raised the count of other errors, meaning renamed columns, data types borrowed from another dialect and clauses not present in the source. The demonstration reports error counts per category rather than an accuracy percentage.

16. Text2SQL is Not Enough: Unifying AI and Databases with TAG

    Biswal · 2024 arXiv

    Synthesis

    Plain-language abstract Biswal and Patel asked whether text-to-SQL and retrieval-augmented generation cover the questions business users actually ask of their data. They defined table-augmented generation (TAG), in which a query is synthesized, executed in the database, and the returned rows are handed to a language model that writes the answer, and built an 80-question benchmark from BIRD where each question needs either world knowledge or semantic reasoning. Text2SQL, RAG and two stronger baselines answered at most 20 percent of the questions with exact match. Hand-written TAG pipelines answered 55 percent, at the lowest execution time of any method.

    Motivation Databricks customers and internal users asked which reviews are positive, why sales dropped in a period, and what the quarter over quarter trends are for the retail vertical, questions that need sentiment classification over text, aggregation across many rows, or knowledge of how the business defines its terms. Text2SQL serves only questions with a relational equivalent, and RAG serves only point lookups to a few records followed by one LM call that is error-prone at counting and math and degrades on long context. No benchmark had measured either paradigm on questions that require the language model's own knowledge or reasoning.

    Methodology The authors took match-based, comparison, ranking and aggregation queries from 5 BIRD databases (california_schools, debit_card_specializing, formula_1, codebase_community, european_football_2) and modified them so that 40 required parametric knowledge and 40 required reasoning, 20 per query type, with human-labeled answers. They ran Llama-3.1-70B-Instruct on vLLM across 8 A100 GPUs as the language model, SQLite3 for SQL, and an E5 base embedding model with a FAISS index retrieving 10 rows for RAG. Baselines were Text2SQL, RAG, retrieval with LM reranking, and Text2SQL whose retrieved rows were then fed to the LM; the TAG pipelines were hand-written in LOTUS with semantic filter, rank and aggregation operators. Accuracy was exact match for match-based, comparison and ranking queries; aggregation queries were judged qualitatively.

    Results Overall exact match was 17 percent for Text2SQL, 0 percent for RAG, 2 percent for retrieval with LM rerank, 13 percent for Text2SQL plus LM, and 55 percent for hand-written TAG. TAG scored 60 percent on match-based, 65 percent on comparison and 40 percent on ranking queries, and 53 percent on knowledge and 60 percent on reasoning questions. Text2SQL fell to 10 percent on ranking and on reasoning questions, Text2SQL plus LM hit context-length errors when many rows came back, and RAG answered no question correctly. Hand-written TAG averaged 2.94 seconds per query, up to 3.1 times faster than the other baselines, and on an aggregation question about races at Sepang it summarized every race from 1999 to 2017 where RAG listed 4 and Text2SQL plus LM used no database data at all.

17. SUQL: Conversational Search over Structured and Unstructured Data with Large Language Models

    Liu · 2023 arXiv

    Synthesis

    Plain-language abstract Liu and Lam asked how a conversational agent can answer questions that need both a database's columns and its free-text fields, such as an Italian restaurant with a romantic atmosphere. They extended SQL with two primitives, ANSWER and SUMMARY, that call a language model on text values, built a few-shot LLM parser that translates user turns into this language, and wrote a compiler that uses dense retrieval and predicate ordering to keep LLM calls few. On HybridQA the few-shot system came within 8.9 exact-match points of a model trained on 62K examples. On a Yelp database of 1,828 restaurants the agent satisfied all user constraints 90.3 percent of the time in conversation, against 63.4 percent for a linearized-retrieval baseline.

    Motivation Patient records, financial databases and review sites hold structured columns beside free text, and crowdsourced restaurant questions showed more than 49 percent needed both. Prior chat systems routed each question to a structured or an unstructured module by a classifier, which cannot serve questions that need both, or linearized tables into text and retrieved by embedding, which gives up the comparisons and calculations SQL does well. Earlier hybrid question-answering systems worked only on small tables, dropped the expressiveness of structured queries, or supported limited compositions of the two.

    Methodology SUQL defines ANSWER(t, q), returning the answer to question q over text t, and SUMMARY(t) as sugar for ANSWER with a summary question; both return text usable anywhere SQL accepts text, so every HybridQA question type has a SUQL form. A two-stage in-context parser first decides whether the knowledge base is needed and then predicts the query, and the agent reports what it searched and says when nothing was found. The compiler answers ANSWER filters by retrieving candidates with precomputed dense embeddings and then verifying the clause with an LLM, aggregates similarity across multiple text constraints, lists enumerated values in the schema prompt when a type has at most 10 members and otherwise overloads equality with a zero-shot CLASSIFY, orders non-LLM predicates first in disjunctive normal form, and evaluates lazily under LIMIT. Experiments used GPT-4-1106-preview on HybridQA (about 70K questions over 13,000 Wikipedia tables) with fewer than 10 examples and up to 2 retries, and gpt-3.5-turbo-0613 on 1,828 Yelp restaurants from 4 cities with 100 crowdsourced questions and 96 turns across 20 conversations.

    Results The few-shot SUQL system scored 59.3 percent exact match and 68.3 F1 on the HybridQA development set and 59.0 EM and 68.4 F1 on test, 8.9 EM and 7.1 F1 below S3HQA and more than 10 points above HPROPRO's few-shot GPT-4 program prompting at 48.7 EM; under the substring metric of Sui et al. it scored 72.5 percent against their best 56.68. Of 72 sampled errors, 37.5 percent were format mismatches and 23.6 percent wrong or incomplete gold labels, so the authors estimated true accuracy near 84.2 percent. On Yelp, turn accuracy was 93.8 percent single-turn and 90.3 percent conversational, against 57.0 and 63.4 for linearization returning 1 result and 49.7 and 61.9 for 3 results. The agent returned nothing for 21 of 100 questions and 8 of 62 conversational queries, of which 7 and 2 truly had no answer, a false negative rate of 14 and 9 percent.

18. Structure then Query: Enabling Precise Analytical Queries over Unstructured Documents

    Lin · 2026 arXiv

    Synthesis

    Plain-language abstract Lin, Luo and Tang asked how to run precise analytical queries, with exact filters, joins and multi-step conditions, over collections of unstructured documents without paying for a language model on every document at every query. AnnoIndex induces a hierarchical schema from the corpus, extracts the attribute values once into a structured index, and compiles each question into a SQL-like plan that filters on the index first and calls extraction models in rising order of cost only on the survivors. On three corpora (legal decisions, Wikipedia pages, web pages) with 500 queries each it averaged 0.87 F1 against 0.80 for the best prior system and 0.73 for feeding whole documents to GPT-4o, while using 11 times fewer tokens than the direct LLM baseline. Values extracted at query time are written back, so repeated workloads get cheaper.

    Motivation Vector retrieval finds documents roughly about a topic but cannot enforce a condition such as birth year before 1985, so retrieve-then-extract pipelines hand the LLM noisy context, lose documents dropped early, and pay again on every query. Extract-everything approaches such as GraphRAG spend an LLM call on every document up front, and a knowledge graph is a relation explorer rather than an analytical engine: it cannot aggregate or join without an LLM reasoning over subgraphs, which stays error-prone even when the needed facts are present. Existing text-to-table systems (ZenDB, QUEST, Palimpzest, Lotus) still extract at query time and use fixed schemas that do not adapt across heterogeneous document clusters.

    Methodology SchemaLoop keeps a human-defined dataset-level schema, then has GPT-4o propose table-level and document-level schemas once per document group, deduplicates candidates by synonym and embedding similarity, extracts values with Mistral-7B on 5 sampled documents, and refines until extraction success rate stays above 0.6 and filtering efficiency above 0.3, for at most 5 rounds. The query engine parses a question layer by layer into schema-bound predicates plus residual EXTRACT predicates, evaluates regex, then small model, then LLM under a per-query budget, rewrites joins as IN filters, short-circuits on failure, and persists extracted values as virtual fields promoted to the schema after more than 10 references. Datasets were LCR (1,600 legal documents of over 6,000 tokens each), WikiText (219 Wikipedia pages across 10 domains) and SWDE (1,050 web pages), each with 500 queries split 20 percent simple filters, 30 percent conjunctive or disjunctive, 30 percent joins and 20 percent progressive reasoning, with ground truth extracted by GPT-4o and corrected by 8 graduate students. A tuple counted as correct only if all attribute values matched exactly; baselines were vector RAG, Graph RAG, ZenDB, Palimpzest, QUEST, Lotus, ClosedIE and direct GPT-4o.

    Results Average F1 was 0.87 in performance mode and 0.83 in economical mode, against 0.80 for QUEST, 0.73 for Lotus and for direct GPT-4o, 0.70 for Palimpzest, 0.67 for ZenDB, 0.60 for Graph RAG and 0.46 for vector RAG; on the legal corpus AnnoIndex reached 0.81 where Lotus scored 0.46, and on templated web pages every strong system exceeded 0.94. Amortized cost including offline indexing was 18.3K tokens per query in performance mode and 15.7K in economical mode, the offline build equalling about 1.5 online queries. Removing SchemaLoop in favour of a manual schema cut WikiText F1 from 0.87 to 0.72 and raised cost 72 percent; replacing the structured engine with vector retrieval over the same index cut F1 to 0.64. On three-way join and reasoning queries AnnoIndex scored 0.86 F1 against 0.74 for QUEST and 0.35 for the LLM baseline, and on legal conflict queries the plan narrowed 1,600 documents to 120 by index and to 35 by a small model before any LLM call, reaching 0.79 F1 where QUEST scored 0.61 under the same LLM budget.

19. SemPlan: Benchmarking Structured Semantic Planning for LLM-Based Queries over Enterprise Data

    Santos Teixeira · 2026 arXiv

    Synthesis

    Plain-language abstract SemPlan asks how much a language model should be allowed to write when it turns a business question into a query over enterprise data. Santos Teixeira built a 1,800-case bilingual benchmark over a synthetic commerce schema and ran 4 architectures over it with 1 model: free SQL behind guards, a typed tool agent, a structured semantic request the application compiles to SQL, and that request with conversational state. The compiled semantic request scored best on answer correctness at 25.67 percent, 3.42 points above direct SQL. Every architecture was wrong or refused on most cases, and the guards changed which kind of failure occurred more than they changed the total.

    Motivation Enterprise questions carry business semantics, governance rules and ambiguity that a plain text-to-SQL benchmark does not exercise, and existing evaluations rarely hold the model fixed while varying the architecture. Santos Teixeira wanted a fixed model, fixed prompts and fixed cases so that the only variable was how much of the query the model wrote. The benchmark also had to record refusals, clarifications and policy outcomes as first-class results rather than folding them into accuracy.

    Methodology The Northstar Commerce domain covers customers, products, orders, payments, expenses, budgets, suppliers, contracts and a calendar dimension behind governed read-only PostgreSQL views. Cases came from parameterized semantic templates over intents, metrics, dimensions, filters, temporal context and ambiguity, with gold plans executed deterministically; the 1,800 cases split into dev, validation, public test, hidden test, multi-turn and adversarial sets, of which 1,200 formed the frozen scientific subset. Architectures A1 to A4 all used gpt-5.6-luna at low reasoning effort with a 1,200-token output ceiling: A1 emitted SQL that had to pass parsing, AST allowlists, complexity limits and timeouts; A2 called typed analytics tools; A3 emitted a strict envelope of operation, metrics, dimensions, filters, time grain, sort and limit that deterministic code compiled to parameterized SQL; A4 added prior state and typed clarification outcomes. The run produced 4,800 primary records plus 1,200 stability records with no missing rows, analysed with McNemar tests and Holm-Bonferroni corrected paired risk differences.

    Results Answer correctness was 22.25 percent for A1, 22.58 for A2, 25.67 for A3 and 24.25 for A4; the A3 gains over A1 and A2 were 3.42 and 3.08 points with Holm p = 0.000110, and A4 trailed A3 by 1.42 points (p = 0.041). Direct SQL had the highest policy correctness at 43.67 percent but falsely refused 21.42 percent of cases; the other 3 refused under 2 percent and instead produced unsafe or invalid output on 60 to 64 percent. The compiled envelope cost USD 0.000512 per case against 0.000918 for direct SQL and repeated its outcomes on 98.67 percent of a 150-case subset run 3 times. Portuguese cases scored 4.8 to 11.8 points below English, and multi-turn state correctness peaked at 20 percent.

20. Mapping Patterns for Virtual Knowledge Graphs

    Calvanese · 2020 arXiv

    Synthesis

    Plain-language abstract A virtual knowledge graph exposes a relational database as an ontology through mappings, and writing those mappings by hand is where the work goes. Calvanese, Gal, Lanti, Montali, Mosca and Shraga asked whether the mappings practitioners write follow a small set of recurring shapes, and whether those shapes can be tied to the conceptual model the schema was built from. They defined a catalog of patterns, each fixing a conceptual model, a schema fragment, a mapping and an ontology fragment, and checked it against 1,559 mappings from 6 real deployments. 97.5 percent of the pattern applications fell inside the catalog, and close to half depended on constraints present in the data but absent from the schema.

    Motivation Ontology-based data access lets users query a database through an OWL 2 QL ontology while the data stays in place, but the mappings between schema and ontology are written case by case with little guidance. Existing bootstrappers such as the W3C Direct Mapping, BootOX and MIRROR generate mappings from the schema alone, and a single schema admits several conceptualizations, so schema-only rules cannot say which meaning was intended. The authors set out to name the patterns with the conceptual model made explicit.

    Methodology A VKG specification is a triple of an OWL 2 QL ontology, a set of mappings whose source is a SQL query and whose target is a set of atoms with R2RML-style templates, and a schema with keys, foreign keys and inclusion dependencies; the virtual ABox is never materialized and SPARQL is unfolded to SQL as in Ontop. A pattern is a quadruple (C, S, M, O). The schema-driven patterns cover entities, relationships, identifier alignment, merged relationships, weak entities, reified n-ary or attributed relationships, hierarchies and hierarchies with alignment; each has a data-driven twin where the constraint holds in the rows but not the DDL, and 3 clustering patterns map a column value to a class, an object or a data value. The evaluation hand-classified every mapping in BSBM, NPD, UOBM, Suedtirol OpenData, Open Data Hub and Cordis into pattern applications.

    Results The 1,559 mapping assertions resolved into 407 pattern applications. 52.8 percent of applications were schema-driven, 44.7 percent data-driven, and 2.5 percent (11 applications over 44 mappings) matched nothing in the catalog. The largest groups were schema entity with 81 applications over 614 mappings, schema merging with 104, data-driven merging with 63, and the denormalized merged 1-N relationship with 44 applications over 166 mappings. Whether the patterns preserve queries losslessly was left open.

21. The Query Translation Landscape: a Survey

    Nadjib Mami · 2019 arXiv

    Synthesis

    Plain-language abstract Nadjib Mami and colleagues surveyed the tools that translate a query written in one data language into another, covering SQL, SPARQL, Cypher, Gremlin, XPath, XQuery and MongoDB operations. They classified more than 40 methods published after 2003 on 8 criteria and summarized each as a Query Translation Identity Card. The taxonomy separates direct translation from translation through an intermediate language, through storage-scheme knowledge, through schema information, and through a mapping language such as R2RML. Coverage was uneven: selection and projection were nearly universal, joins, aggregation and updates were thin, and several language pairs had no translator at all.

    Motivation Data lives in relational, graph, document and XML stores, and a user or system that speaks one query language needs the others reachable without rewriting every application. Individual translators had been built for single pairs, but no survey mapped which pairs were covered, how completely, or how the translations were evaluated. The authors set out to draw that landscape and to identify a language fit to act as a universal front end.

    Methodology The survey covered works from 2003 onward across 7 query languages and grouped them by translation type, translation coverage, optimization strategy, one-to-one or one-to-many relationship, availability, adoption, evaluation, and metadata. Coverage was read from the article body, the evaluation section and the tool's web page where one existed, and recorded per operation in feature tables for SQL-to-X and SPARQL-to-X. Community factors were rated with stars for implemented, evaluated and available for download. Empirical performance across works was deliberately not compared.

    Results Among SQL-to-X translators, WHERE was supported by all, GROUP BY next most often, UNION and JOIN among the least, and INSERT, UPDATE and DELETE almost never; among SPARQL-to-X translators FILTER, UNION and OPTIONAL reached up to 60 percent of works and DISTINCT, LIMIT and ORDER BY about half. SQL-to-Document and Gremlin-to-SQL were weakly addressed for want of formal semantics, and SQL-to-Cypher, Gremlin-to-SPARQL, the XPath and XQuery pairings with Cypher and Gremlin, and Cypher-to-Document had no translator. Optimizations applied during translation were judged to have more room than post-generation rewriting because source statistics are available. No standardized query-translation benchmark existed, example queries were simpler than real workloads, and SQL and SPARQL were the candidates with the most outgoing translations.

22. Introducing Metrics SQL: A SQL-based semantic layer for humans and agents

    Bangarwa · 2026

    Synthesis

    Rill's Metrics SQL treats a metrics view as a table: a query names dimensions and measures, adds filters, ordering and limits, and contains no aggregation logic. Measures and dimensions are defined once in YAML with SQL expressions (revenue as sum(order_usd), order_volume as count(distinct order_id), a product_category dimension as a dictGet expression), and a parser, a query compiler and an executor rewrite each query into the SQL of ClickHouse, DuckDB, Snowflake or Druid, expanding measures to their aggregates, inferring GROUP BY from the selected dimensions, replacing the FROM clause with the underlying model, and turning literals into parameterized arguments. Bangarwa argues against metrics layers that invent a new query language (GraphQL, MDX, JSON over REST) and cites Julian Hyde's argument that metrics need first-class representation in SQL because SQL is the one language every tool and every model already speaks; the dialect is a restricted subset with no joins across metrics views, no SELECT *, and measure filters only in HAVING. An MCP server exposes the metrics views to Claude, and the post's claim for agents is that a question about revenue resolves to the same defined expression on every run instead of an expression inferred from raw tables, with no model fine-tuning required.

    Why it matters The intermediate representation here is SQL itself, restricted rather than extended, and the compilation target is the engine dialect. The roadmap (semantic pushdown, MEASURE as a native database keyword) makes the intended thinness explicit: the interface stays fixed and only the compilation target moves.

23. Semantic Layers in Apache Superset: SIP-182 and Apache Ossie

    Rusackas · 2026

    Synthesis

    Rusackas reports that Superset has carried a thin compile-time semantic layer, the dataset editor, since its Druid origins, and that every integration with an external semantic layer so far (Airbnb's Minerva, dbt MetricFlow, Snowflake, Data Junction) pretended the layer was a database: Superset's roughly 700-line SQL generator emitted SQL that the integration then parsed back into REST, GraphQL or a table-function call. SIP-182, written by Beto Dealmeida, makes a semantic layer a first-class connection: map_query_object() reduces Superset's 29-field QueryObject to a SemanticQuery of metrics, dimensions, filters, order, limit, offset and group limit, and a SemanticLayer interface exposes get_dimensions(), get_metrics(), get_compatible_metrics(), get_compatible_dimensions(), get_values(), get_table() and get_row_count(), returning an Arrow table; layers ship as .supx extensions, Snowflake Semantic Views and dbt MetricFlow are live on Preset, and the SEMANTIC_LAYERS flag is targeted at Superset 7.0. Under it sits Apache Ossie, the Open Semantic Interchange after acceptance into the Apache Incubator in June 2026: a declarative JSON or YAML spec of datasets, fields, relationships and metrics plus an ontology layer and an ai_context block of freeform instructions and worked examples, at v0.2.0 and marked draft, with over 100 commits from Snowflake, Dremio, Salesforce, Databricks and dbt Labs, 17 launch partners grown to more than 50 organizations, and 3 working groups (Metric Language, Catalog, Ontology). For agents the post's argument is 1 line: an agent pointed at a warehouse is guessing at what a column means, an agent pointed at a semantic layer is reading a definition someone wrote down, and Superset adds a usage signal (which metrics get charted, by whom, next to what) that the warehouse does not have.

    Why it matters The interchange file is the representation, and the claim is that once every layer exports the same YAML a consumer parses a document instead of emulating a database. The post also carries the governance answer to the Mullins critique: ASF incubation rather than a vendor consortium.

24. Open Semantic Interchange (OSI) Updates: Specification Now Live, New Working Group Members and More

    Klahr · 2026

    Synthesis

    Klahr and colleagues announced on 27 January 2026 that the first version of the OSI specification was live in an Apache 2 licensed repository at github.com/open-semantic-interchange/OSI. The post describes it as a vendor-neutral, extensible model for representing semantic layer constructs such as data sets, metrics, dimensions, relationships and contexts, in a form that can be interpreted consistently across tools, platforms and agentic applications, and calls it a starting point, not an endpoint. 8 organizations joined the working group (AtScale, Coalesce, Collate, Credible, Databricks, JetBrains, Lightdash, Qlik) alongside existing members that include Alation, Atlan, BlackRock, Collibra, Cube, DataHub, dbt Labs, Domo, Hex, Honeydew, Informatica, Instacart, Mistral AI, Omni, Preset, RelationalAI, Salesforce, Select Star, Sigma, Starburst Data, Strategy and ThoughtSpot. The post commits to a transition to neutral, foundation-led governance as the initiative matures and says nothing about what the specification covers beyond that list of constructs, or what it leaves out.

    Why it matters The standard's object is interchange of definitions, not execution: it fixes what a metric or a relationship is so that different tools read the same thing. The governance sentence is what the Mullins critique, published 12 days later, takes apart.

25. How Not to Run an Open Standards Initiative

    Mullins · 2026

    Synthesis

    Mullins accepts the problem OSI names (business logic rebuilt across tools, meaning trapped in proprietary formats) and then lists 9 anti-patterns in how the initiative ran, as of February 2026. The initiative was branded before an independent community existed; communication ran through corporate blogs, mainly Snowflake's, with an independent landing page he calls an inch deep; there was no public record of who decides, how disagreements are resolved, or what the change process is; the spec was not placed under a neutral foundation, with Snowflake keeping ownership; existing modeling systems and prior proposals such as AtScale's SML were ignored; and, in his reading, the specification is in practice dbt Labs' MetricFlow, with the canonical repository, issue tracker, roadmap and release process controlled by 1 vendor and the OSI repo showing contributors only from Snowflake. He adds that there was no migration story for existing semantic layers short of large rewrites and no incentive for anyone outside the originating companies to participate. His conclusion is that writing the schema is the easy part and that open standards are institutions first and specifications second.

    Why it matters Read against the Snowflake announcement and the Preset post, this is the same specification judged on governance rather than on the constructs it defines. The Apache incubation the Preset post reports for June 2026 answers Mullins' fourth and sixth points, 4 months after he wrote.

26. SLayer: An expressive, embeddable semantic layer for AI agents and humans

    MotleyAI · 2026

    Synthesis

    The README describes SLayer as an embeddable semantic layer, usable as a Python library or a standalone tool, that sits between agents and a database and handles read-only connectivity, SQL translation, common transformations and row-level security. Where a traditional BI semantic layer predefines each metric with its aggregation, SLayer defines a column such as revenue once and lets a query apply expressions to it: revenue:sum, revenue:avg, revenue:sum / *:count, time_shift(revenue:sum, -1, 'year'), with multi-stage queries for the rest. It is built around a search, inspect, query flow, with a search tool for discovery and a memory store for attaching business context, and it is reachable over MCP, REST, CLI, Python, Flight SQL or a Postgres-compatible SQL API, with importers for dbt, Cube and Ossie configs. The README's comparison of LLM-written SQL with the equivalent SLayer query for monthly revenue by store with month-over-month change is an image, not a measurement.

    Why it matters A dialect-native intermediate representation designed for agents: the expression grammar is richer than a fixed metric list because, the README argues, an agent gets asked for ratios, time shifts and alternative aggregations of the same column.

27. Why 99% of BI Tools Get Embedded Analytics Wrong And How Omni Fixed It ft. Chris Merrick

    Merrick · 2026

    Synthesis

    On representation the notes record 3 positions. Merrick says Omni does not invent new languages because it is not a language builder and a new language adds little value, and that a definition in Omni is relative to the database dialect it targets because Omni does not attempt to be a universal translator; the notes call abstracting away dialect differences a trap that creates more problems than it solves and name SQL as the common language instead. He also says semantics exist at every layer of the stack, from a database table upward, with more and different semantics at each level. The last item reaches past the warehouse: long-form account-executive comments in Salesforce, once the most useless data for a BI tool, are now the most interesting, and the stated goal is a system intelligent about the entire company, not just the tables in the warehouse, so that unstructured data alongside structured metrics can answer why questions.

    Why it matters The dialect-native side of the split, stated by a vendor: definitions stay in the target SQL and no intermediate language is introduced. It also widens the representation question to text fields that carry the reasons behind a number.

28. Ontology-Based Data Access: A Survey

    Xiao · 2018

    Synthesis

    Xiao and colleagues define OBDA as a specification P = (O, M, S): an ontology O that gives users a conceptual vocabulary, a source schema S, and a declarative mapping M whose assertions pair an SQL query over the source with ontology atoms whose arguments are built by IRI templates. A SPARQL query over the ontology is answered in 2 stages, rewritten against the ontology into a first-order query (the PerfectRef algorithm, a form of backward chaining) and then unfolded through the mapping into SQL that the source DBMS executes; with OWL 2 QL, built on DL-Lite, and R2RML mappings every conjunctive query is first-order rewritable, so certain answers have the same AC0 data complexity as ordinary query evaluation. The survey itemizes the costs: rewritings can be exponential in size, naive unfolding produces redundant self-joins (3 copies of the PRESSURE table in the Statoil example) that must be removed using integrity constraints or precomputed saturated T-mappings, and richer ontologies (EL, disjunction, temporal operators) lose first-order rewritability. Systems named are Mastro, Morph, Ontop, Stardog and Ultrawrap, with deployments at Statoil, Siemens and the Italian Ministry of Economy and Finance.

    Why it matters The formal ancestor of the semantic layer: meaning lives in an ontology plus a mapping, the user never touches the schema, and the guarantee is that answers are the certain answers under the ontology. The price the survey itemizes, rewriting size, mapping maintenance and expressiveness limits, is the price any compiled intermediate representation pays.

29. An Alternate Agentic AI Architecture (It's About the Data)

    Wenz · 2026 · 0 cites arXiv

    Synthesis

    Plain-language abstract Enterprise questions usually need data from several systems joined together, and the authors argue that letting a language model decide which tools to call and how to combine results is the wrong design. RUBICON instead has the user write a short query in a restricted notation, AQL, naming the tables and columns wanted and leaving only the filter condition in plain language. Wrappers turn each source into a table, a query processor joins them, and every intermediate result is visible. On seven questions over five sources, plain and tool-using LLM agents get none right; RUBICON gets all seven.

    Motivation Enterprise data is not in the pretraining corpus, lives in specialized stores that cannot be imported into a model, is governed by role- and user-specific access control that models ignore, and must be joined across sources. The authors' earlier work found text-to-SQL accuracy on real warehouses more than 50 percent below Spider and BIRD levels because of redundant schemas, institutional jargon and harder business queries, so an LLM-centric architecture built on text-to-SQL cannot be relied on. Copilots that ingest and index sources into one semantic search improve discoverability but cannot perform joins.

    Methodology AQL is FIND columns FROM table WHERE natural-language predicate, with aggregates, JOIN between two FIND blocks, schema introspection and SAVE, OUTPUT and DELETE. Source wrappers, assumed to be built by enterprise staff, translate the predicate to the native interface and expose each source as a logical relational view. Interactive mode executes one command at a time with the intermediate table shown; compiled mode runs a sequence as one plan through a relational-style optimizer. The benchmark has seven expert-written queries over Wikipedia, a 97-table anonymized university warehouse in SQLite, a research-lab website, Gmail and pretrained model knowledge, each requiring exactly two sources with three distractors. Baselines are single-shot GPT-5-mini, Gemini-3-flash-preview and Claude-Sonnet-4.6, and LangChain ReAct agents over the same models with all five sources as tools, each under natural-language and AQL-style prompts. Metrics are correctness, tokens, provider-reported cost, time to first token and tool-call count.

    Results All twelve LLM configurations score 0 of 7; RUBICON scores 7 of 7 with two tool calls per query. Failures are omitted sources, premature termination and incomplete joins rather than fabricated facts. ReAct input tokens average about 20,000 for GPT and over 469,000 for Gemini with AQL prompting, with first-token latency of minutes, while RUBICON uses 4,182 input tokens. AQL prompting improves which sources agents call but not how many calls they make. Two valid AQL plans for the Turing-or-Nobel query differ structurally in cost, award-first pushing a selective predicate into Wikipedia versus faculty-first scanning a page per professor, which the authors use to argue for a cost-based planner. The paper's benchmark is small and self-built; a hundred-query benchmark from the Munich transportation department is announced as pending permission.

Enforcement: constraint instead of advice

Documentation is advice. A representation narrows what the model is asked for. Enforcement is the third position: the system is built so that a whole class of wrong answer cannot be produced, whatever the model does.

The word covers two mechanisms that behave in opposite ways, and conflating them is the most common error in discussions of this literature. The first is constrained decoding. PICARD rejects tokens that cannot continue into a parseable, schema-valid query, and Synchromesh generalizes the idea with target-similarity retrieval and constrained semantic decoding. Both report gains. The second is a compiler between the model and the database, where the model chooses among modeled entities and a deterministic component emits the SQL. The dbt Semantic Layer benchmark is the sharpest published measurement of that arrangement: 11 questions run 20 times each over a 15-table insurance schema in third normal form, where claude-sonnet-4-6 goes from 90.0 percent writing SQL to 98.2 percent through the semantic layer, and gpt-5.3-codex from 84.1 to 100.0 percent.

Then Dang and colleagues in 2026 show that the first mechanism has a bias problem. Locally constrained decoding masks next tokens myopically, which produces biased sampling and degradation in performance, measured on function calling, keyword generation and SQL. Their remedy is to make the proposal distribution global rather than local. That result does not contradict PICARD; it explains the difference between PICARD and Herzig. Changing the target language changes what the model is asked to produce. Masking tokens leaves the target alone and distorts the distribution over what the model would otherwise have produced. Same word, two mechanisms, opposite sign.

The architectural statement of the second mechanism is Wu's, and it is a design argument rather than an experiment: a trusted deterministic kernel with a generative shell around it, holding one invariant, that a component which can fabricate may influence which question the system answers but never which value it returns. It carries a two-year production case study and reports no datasets and no accuracy numbers, and it should be read as a claim about system structure rather than as evidence about accuracy.

Read this theme against the next one before drawing conclusions. Every number quoted above is measured inside the boundary of what the enforcing system can express, and the interesting behaviour is at the edge of that boundary.

Two 2026 results reach the access-control question that earlier versions of this theme called absent. Fei, Jiang, Yang and Xiao gave 13 models a question, a role and the role's permissions in the prompt across 21,502 instances, and asked for SQL or a refusal. Violation rates, the share of instances where the model wrote SQL that needed a denied permission, ran from 2.55 percent for GPT-5 on Spider to 76.15 percent for a 7B model on LiveSQLBench, and exceeded over-refusal for every model on every dataset. Hiding the forbidden columns halved the reads and raised GPT-5-mini's violation rate from 12.49 to 26.72 percent, because the model wrote SQL over hallucinated columns instead of refusing. The authors' conclusion is the one this theme has been circling: the model's allow-or-deny decision sits above a deterministic check at the database and cannot replace it. GROUND measured the other half. Definitions in the prompt fixed formula, grain and filter errors and still returned rows outside the user's tenant on 35 percent of questions. Injecting the row-level predicate and rejecting any query that failed a security check took that to zero, at five times the tokens of the bare baseline.

Wang and colleagues supply the adversarial case. Across 3,075 tests and six systems, the structure that held was structure below the model. DB-GPT's sandbox kept 0.0 percent of hijacks from reaching the host, and Databricks Genie Code, scoped by Unity Catalog, blocked five of six hijack techniques at 0 percent. The same deployment returned attacker-preferred values on 80 percent of trust-bias payloads, where a false rule planted in a document beat the structured data, and lost to cross-query reconstruction 4 percent of the time, because policy is checked one query at a time and never tracks what a sequence of allowed queries adds up to.

The production layers fall on both sides of the line. Rill's Metrics SQL enforces row-level security in the executor and parameterizes every literal, so the policy sits below what the agent can reach. Lyft's layer governs by admission and ownership, two owning teams and dual approval for any change, and describes nothing that stops an agent from writing its own SQL against the warehouse. SLayer's read-only, row-filtered connection constrains what the agent can touch and not what it can compute. Sequeda's ontology is the older form of the same mechanism. An R2RML mapping that defines an Agent as a party with role code AG makes a wrong join impossible to express, and the failures that remained were path errors inside a structure the model could not leave. Merrick's curation, you curate the dataset the LLM is seeing, is a constraint on input rather than output. Lee's semantic integrity constraints sort the whole theme by mechanism: errors of form can be ruled out by grammar, errors of meaning still need a second model's judgement, and the reliability of that judgement is the open problem.

1. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models

   Scholak · 2021 arXiv

   Synthesis

   Plain-language abstract PICARD constrains a language model's SQL generation by incrementally parsing each candidate continuation and rejecting tokens that cannot extend into a parseable, schema-valid query. It runs at every decoding step, requires no change to the underlying model, and can be attached to an existing pre-trained parser.

   Motivation Sequence models generating SQL produce a large fraction of outputs that are simply invalid: unparseable, referring to columns that do not exist, or joining tables with no relationship. Fine-tuning reduces this but does not eliminate it, and the failures are mechanical rather than semantic, which suggests they should be handled mechanically.

   Methodology At each decoding step the candidate next tokens are checked by an incremental parser that maintains partial parse state and knows the schema. Continuations that cannot lead to a valid query are removed from consideration before sampling proceeds. The mechanism is applied on top of an unmodified pre-trained model, which is what makes the comparison against the same model without constraint clean.

   Results The system establishes constraint-as-masking as the reference technique for making generation respect a grammar and a schema, and it is the concrete method later work argues about. It matters here as a reference point rather than a conclusion: the mechanism it popularized is exactly the one the 2026 tractable-proposals work shows introduces sampling bias, which is why this map separates constraining the decoder from changing the target language.

2. Synchromesh: Reliable code generation from pre-trained language models

   Poesia · 2022 arXiv

   Synthesis

   Combines target-similarity example retrieval with constrained semantic decoding, so generated programs satisfy semantic as well as syntactic constraints of the target language.

   Why it matters It generalizes PICARD past syntax to semantic well-formedness, which is as close as decoding-time constraint gets to what a semantic layer enforces structurally.

3. Learning and Enforcing Context-Sensitive Control for LLMs

   Albinhassan · 2026 arXiv

   Synthesis

   Learns and enforces constraints that depend on generation history rather than on a fixed context-free grammar, extending what decoding-time enforcement can express.

   Why it matters It occupies the middle ground between advice and a compiler, which is where a governed analytics harness actually sits when the model still writes some of the query.

4. Mitigating Bias in Locally Constrained Decoding via Tractable Proposals

   Dang · 2026 arXiv

   Synthesis

   Plain-language abstract This paper shows that the standard way of enforcing constraints on language-model output, masking next tokens that violate the constraint, produces biased samples and measurable degradation in quality. It proposes using tensorized finite automata as global proposal distributions within sequential Monte Carlo instead, so the constraint is respected without distorting the distribution.

   Motivation Locally constrained decoding is applied as though it were free: the constraint is satisfied, and the model is otherwise unchanged. The authors point out that masking at each step is a myopic approximation of conditioning on the constraint over the whole sequence. The two are not equivalent, and the difference shows up as bias.

   Methodology The bias is characterized analytically, then the paper constructs a proposal distribution that accounts for the constraint globally rather than one token at a time, using tensorized finite automata, and plugs it into a sequential Monte Carlo sampler. Evaluation covers function calling, keyword-based generation and SQL generation.

   Results The finding relevant here is that on SQL generation, enforcing a constraint by masking degrades performance relative to the corrected sampler. This is the counter-evidence that makes this map's central distinction necessary. Changing what the model is asked to generate and masking what it is allowed to emit are opposite interventions that the literature calls by one name, and the evidence that constraints help comes from the first while the evidence that they hurt comes from the second.

5. Never the Number: Structural Abstention for AI Systems Whose Answers Are Consumed as Fact

   Wu · 2026 arXiv

   Synthesis

   Plain-language abstract An architectural argument for systems whose outputs are consumed as fact. A trusted deterministic kernel computes every value the system returns; a generative shell wraps it and may influence which question gets asked. The invariant is that a component capable of fabrication never determines a returned value, so requests the system cannot answer are unrepresentable rather than answered badly.

   Motivation Most work on when a system should decline treats it as an estimation problem: generate a candidate answer, estimate confidence, decline below a threshold. The author's position is that for outputs consumed as fact, a calibrated probability of being wrong is the wrong object. What is wanted is a construction in which the wrong answer cannot be produced at all.

   Methodology The paper is a design argument supported by a two-year production case study. It states the invariant, describes how the kernel and shell divide responsibility, and works through what the boundary implies for system behaviour. It reports no datasets and no accuracy metrics, which is a limitation the paper states rather than one a reader has to discover.

   Results The contribution is the distinction between structural and statistical abstention: structural abstention needs no confidence estimate because the unanswerable request has no representation in the system, in contrast to abstention that generates a candidate and then estimates whether to trust it. That distinction separates two literatures that otherwise look like one, and it also names the failure mode of the structural approach. Such a system cannot be miscalibrated, and it can be silently wrong about the scope of what it models.

6. Omni modeling layer documentation

   Omni · 2026

   Synthesis

   Public documentation for a shared semantic model with workbook-level extension, where dimensions, measures and relationships are defined once and queries are compiled from the model rather than written as raw SQL.

   Why it matters Included so the enforcement argument can be checked against a real production system's definition of a governed model rather than against the abstraction, and because the same product is the subject of the original benchmark work linked below.

7. Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update

   Ganz · 2026

   Synthesis

   Plain-language abstract A vendor benchmark comparing questions answered through dbt's semantic layer against the same questions answered by a model writing SQL directly. Eleven questions are run twenty times each over a fifteen-table insurance schema in third normal form, with two frontier models, and the results are reported separately for questions the semantic layer models and questions it does not.

   Motivation The semantic-layer argument needs a number, and the vendor with the semantic layer is the party motivated to produce one. Read as a designed experiment rather than as marketing, it is the only public measurement of what enforcement does when the same underlying data and the same models are used on both sides.

   Methodology Eleven questions, twenty repetitions each, one schema. Both conditions run against the same modeled data; the difference is whether the model calls the semantic layer's API or writes SQL against the tables. The authors also report a longitudinal comparison of the same question set from 2023 to 2026, and separately report performance on questions outside what the semantic layer models.

   Results Through the semantic layer, claude-sonnet-4-6 reaches 98.2% and gpt-5.3-codex 100.0%, against 90.0% and 84.1% writing SQL against the same data. Across the full question set from 2023 to 2026, text-to-SQL moved from 32.7% to 64.5% and the semantic layer from 60.5% to 72.7%. On questions the layer does not model, the layer scores 0.0% while text-to-SQL scores 70.0% and 100.0%, because the layer returns an error rather than a number. The same eleven-question experiment therefore produces both the highest and the lowest figures in this map, which is the clearest available demonstration that a single accuracy column cannot describe these systems.

8. Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models

   Rumiantsau · 2026 arXiv

   Synthesis

   Plain-language abstract A paired benchmark asking what a written description of a company's business conventions is worth to a text-to-SQL system, holding everything else fixed. One hundred natural-language questions are run over the Cleaned Contoso Retail dataset in ClickHouse, single-shot, against three frontier models, in a schema-only condition and a schema-plus-semantic-context condition.

   Motivation The semantic-layer argument is usually made either as a product claim or as an architectural preference, without an experiment that isolates the variable. The authors isolate the smallest version of it: not a compiled semantic layer, not a governed query interface, just a document that states what the measures mean, which conventions the business follows, and how to disambiguate the terms that are ambiguous.

   Methodology The semantic context is a roughly 4 KB hand-authored markdown description of measures, conventions and disambiguation rules for the schema. Questions are paired across conditions so each comparison is within-question, generation is single-shot with no retry or self-correction, and the same three frontier models run both conditions. Significance is reported per comparison.

   Results Schema only gives 45.5% to 50.5% across the three models; schema plus the 4 KB document gives 67.7% to 68.7%. Every cross-condition comparison is significant below p = 0.01. The more interesting finding is the within-condition one: the three frontier models are statistically indistinguishable from each other in both conditions, so on this workload the document matters and the choice of model does not. The evidence base is one schema, 100 questions and no peer review, which makes this a strong effect measured on a narrow base.

9. Benchmarking Text-to-SQL under Role-Based Access Control

   Fei · 2026 arXiv

   Synthesis

   Plain-language abstract Text-to-SQL benchmarks assume the person asking can see every table and column. Fei, Jiang, Yang, and Xiao asked what happens when the asker holds a role that can see only some of them. They built a pipeline that synthesizes plausible roles and column-and-operation permissions for Spider, BIRD, and LiveSQLBench, had four annotators validate them, and labeled each question-role pair allow or deny by parsing the gold SQL. Thirteen models were then scored on whether they refused when they should and wrote compliant SQL when they could. Models with strong unrestricted accuracy violated the policy on a large share of questions, and hiding columns, fine-tuning, and few-shot examples did not repair it.

   Motivation Production databases enforce role-based access control, so an unauthorized query fails at execution or is silently masked; a check at the database blocks it but cannot say whether the system should have refused earlier, missed a compliant alternative, or over-refuses. Existing benchmarks score execution accuracy under unrestricted access, so a query that is correct but reads a denied column counts as a success, which the authors call an RBAC-rejected success. The few prior efforts on access control used database- or table-level permissions and had no systematic construction pipeline or empirical study.

   Methodology An LLM inferred each database's application context from its schema, derived roles with responsibilities, and then assigned column-level permissions per operation; configurations were screened on denial rate (rejected below 5 or above 90 percent), coverage variance, pairwise Jaccard overlap above 0.8, and embedding similarity to the schema, then accepted by 3 of 4 annotators (28 of 53 passed first time, 16 after one regeneration, 9 were revised by hand). Each database also received 2 or 3 programmatically generated DataOperator roles with broad but incomplete access. SQLGlot extracted the (table, column, operation) triples each gold query needs, and an instance is allowed when they are a subset of the role's policy. The result is 53 databases, 399 tables, 3,353 columns, and 21,502 instances from 3,160 questions, with deny shares of 47, 65, and 76 percent on Spider, BIRD, and LiveSQLBench. Each model saw a fixed zero-shot template at temperature 0, one random role per question per pass, over 5 seeds, and was scored on violation rate, over-refusal rate, AC-F1, Safe-EX, and Safe-Deny.

   Results Violation rates exceeded over-refusal rates for every model on every dataset. On Spider GPT-5 violated 2.55 percent of instances with AC-F1 95.79 while Snowflake-Arctic-R1-7B violated 45.43 percent with AC-F1 70.14 despite higher execution accuracy; on BIRD Snowflake reached 63.77 percent and on LiveSQLBench 76.15 percent, where Claude Sonnet 4.5 was lowest at 12.40 percent. Compliance fell with query difficulty and with the number of joins, driven by violations rather than refusals. Restricting the prompt to the role's schema halved DeepSeek-V3.2-Coder's reads of known-forbidden columns but raised GPT-5-mini's violation rate from 12.49 to 26.72 percent as models wrote SQL over visible or invented columns instead of refusing. Fine-tuning on role-labeled Spider cut Llama3-SQLCoder-8B's violations from 46 to 4 percent at a 17-point Safe-EX cost, gained 7.7 and 13.0 AC-F1 on BIRD for the two tuned models, and lost 6.1 on LiveSQLBench for Qwen2.5-14B; a 298-instance row-level pilot showed DeepSeek-V3.2-Coder at 50.27 percent violations and GPT-5 at 2.15.

10. GROUND: Reducing Hallucinations in LLM-Based Enterprise Analytics Through Governed Semantic Definitions

    Sasidharan Pillai · 2026 arXiv

    Synthesis

    Plain-language abstract Can a language model answer reporting questions over a warehouse without inventing a metric, double counting, dropping a required filter, or showing a user rows they may not see? Sasidharan Pillai built GROUND, which puts approved metric definitions, join paths, grain rules, required filters, and the user's row-level predicate into the prompt and then validates the generated SQL against those rules before it runs, retrying on violations. Four pipelines were run on the same model over 100 synthetic questions; only GROUND had zero errors in every category, and the pipeline given exact definitions but no enforcement still leaked across tenants on 35 percent of questions. The result held on 271,718 real NHTSA complaints with independently written gold and across four models from three providers, at about 5 times the token cost of a bare prompt.

    Motivation Enterprise reporting depends on approved metrics, reporting grain, sanctioned join paths, required filters, and row-level security, none of which a physical schema states. A query can parse, execute, and return a clean number while using the wrong formula, counting at the wrong grain, omitting a filter, or ignoring who is asking; it looks authoritative and is silently wrong. Spider, BIRD, Spider 2.0, BEAVER, and EntSQL do not score these dimensions, and the closest prior study supplied a 4 KB semantic document as context and reported 17 to 23 point accuracy gains without separating supplying semantics from guaranteeing them.

    Methodology The synthetic warehouse has 6 dimension tables, 5 fact tables at different grains, 10 approved metrics each with a formula, base grain, required filter set, and allowed dimensions, approved join edges, and row-level predicates for four users; 100 questions (82 expecting SQL, 18 expecting abstention) carry gold generated deterministically from the semantic layer, and an oracle self-test scores that gold at 100 percent with no classifier flags. All four systems used claude-opus-4-8 with structured JSON output: raw CREATE TABLE text, keyword-retrieved column descriptions, approved definitions as context, or the full semantic layer plus the user's predicate with a validate-and-retry loop of up to 3 revisions. The validator checks schema existence, the metric's expression signature, approved join edges, COUNT(DISTINCT) at order grain, required filters, an un-widened security predicate, and cost signals such as a missing date filter. A replication used 271,718 NHTSA complaints with 8 metrics, manufacturer-scoped row-level security, and 40 hand-authored gold queries; an adversarial set of 40 questions ran across Claude Opus 4.8, Claude Sonnet 5, GPT-5.2, and Llama-3.3-70B with three runs each.

    Results On the synthetic set the bare-schema pipeline executed 95.1 percent of queries with 0.000 result accuracy, omitting required filters on 78 percent and violating row-level security on 78 percent; schema retrieval executed 64.6 percent and leaked on 53 percent; definitions as context reached 0.500 value-based accuracy with zero metric, grain, and filter errors but 35 percent security violations; GROUND executed 100 percent, scored 0.951 value-based and 0.793 strict, had zero errors in every category, and handled 94.4 percent of abstention cases. GROUND used 10,714 tokens per question against 2,140 for the bare prompt and added about 30 percent latency; 9 of 82 SQL questions triggered a retry. On NHTSA GROUND reproduced all 40 gold values while baselines violated manufacturer scope on 25 to 33 percent and dropped filters on 63 to 73 percent, and governed queries full-scanned 0 rows against 7,000 to 32,000 for the baselines. Across the four models the enforced filter and security checks stayed at 0.000 plus or minus 0.000, while Claude Opus 4.8 still invented a LAG-based year-over-year metric on all three adversarial runs and Llama-3.3-70B under GROUND fell to 0.63 execution and 0.25 value accuracy.

11. Semantic Integrity Constraints: Declarative Guardrails for AI-Augmented Data Processing Systems

    Lee · 2025 arXiv

    Synthesis

    Plain-language abstract Data systems now run LLM-backed operators that extract, summarize, or classify inside a query, and those operators can return wrong values. Lee and colleagues at Brown proposed semantic integrity constraints, declared in the same pipe-SQL query as the operators they check, so the system rather than the user decides how to enforce them. They defined six constraint classes, an enforcement plan that combines constrained decoding with validate-and-retry, and a detailed design for grounding constraints being implemented in the VectraFlow system. The paper is a vision statement and reports no measurements.

    Motivation Several systems have added semantic operators to database and data-flow engines, and the hallucinations those operators produce keep them out of healthcare, law, and finance. Existing guardrails (user-defined functions, LLM validators, and constrained-decoding libraries) are imperative, scattered, and unaware of operator semantics, which limits both usability and optimization. Traditional integrity constraints protect stored data from invalid modifications; the authors wanted the same declarative treatment for the outputs of semantic operators, and noted that only DocETL had made validation a first-class feature, with Python statements and prompts rather than a relational interface.

    Methodology Queries use pipe-SQL syntax with prompt string literals; an ASSERT operator can appear anywhere downstream and reference any prior attribute or operator, each semantic operator carries a retry threshold, and each assertion a failure mode of CONTINUE, DROP, or ABORT. The six classes (domain, inclusion and exclusion, grounding, soundness, relevance, assertions) were derived from a prior taxonomy and from use cases gathered in a collaboration with Rhode Island Hospital, illustrated with a sepsis-screening query over electronic health records. Enforcement pushes each constraint to the earliest point its inputs exist, appends it to the operator's prompt, and then either compiles it into a grammar for constrained decoding or checks it after generation, with reordering by cost and selectivity, fusion of checks, and recovery by feedback prompting, longer chain of thought, or a stronger model. Grounding constraints recurse through an attribute's lineage and select exact containment, a factual-consistency scorer, or an LLM validator depending on whether the operator is annotated extractive.

    Results No experiments are reported. The design claims are that domain constraints cover the majority of use cases in the cited survey and can be checked programmatically with a correctness guarantee, that grammar-compiled constraints need no post-check because decoding cannot violate them, that direct constrained decoding is far slower than grammar-based decoding, that fusing many LLM-based checks into one call is expected to degrade validator accuracy as input grows, and that specialized factual-consistency models such as MiniCheck reach GPT-4 accuracy at lower cost. The open problems named are adaptive enforcement driven by per-operator violation scores, verifying inputs against prompt attacks and unprotected data, and an enterprise-wide constraint store shared across queries.

12. Data Agents Under Attack: Vulnerabilities in LLM-Driven Analytical Systems

    Wang · 2026 arXiv

    Synthesis

    Plain-language abstract Data agents let a language model plan SQL queries, run Python, and read files over a company's databases. Wang and colleagues asked what new ways such an agent can be attacked when the database, not a web page, is the source of untrusted content. They defined 8 vulnerabilities across how the agent interprets inputs, executes tools, and applies policy, turned them into 14 attack techniques with 350 payloads grounded in real schemas, and ran 3,075 tests on four open-source agents and two cloud analytics services. No system blocked every attack, trust bias between conflicting sources was the most exploited weakness, and changing the underlying model shifted the exposure rather than removing it.

    Motivation Database security models the database as a closed SQL engine and studies injection, access control, and integrity, while LLM-agent security assumes the agent reasons over documents and web pages. A data agent couples the two: content in table cells, schema comments, and uploaded files becomes reasoning context, that reasoning becomes executable SQL and Python, and a session's policy must govern both transitions. The motivating example is a sequence of individually legitimate queries whose combination discloses a record no single query would, a channel neither line of prior work captures.

    Methodology The authors formalized a data agent as a model and state, a tool set, data assets, policies, and an execution environment, with an adversary who controls only prompts and uploaded data. From 8 vulnerabilities they derived 3 goals (hijack, mislead, drain), 7 tactics, and 14 techniques, then used a retrieval-augmented pipeline to instantiate 5 templates per technique on 5 enterprise databases from DAComp-DA, giving 350 frozen payloads over 27 databases. Targets were DataInterpreter, DB-GPT, and LAMBDA on DeepSeek-V3.2, DeepAnalyze on its own 8B model, and Databricks Genie Code and BigQuery's conversational agent under default governance in May 2026, plus DB-GPT rerun with Sonnet-4.6, GPT-5.2, and Gemini-3-flash. Hijack was scored by a five-level blast radius, mislead by relative error against the attacker's target value, and drain by token and time amplification against a benign run, with success defined as blast radius of at least 1, error of at most 10 percent, or amplification of at least 5.

    Results Every agent was exploitable on at least five of the eight vulnerabilities and every vulnerability appeared in at least three agents. Code injection was the strongest hijack, data-type trust bias the strongest mislead (100 percent on DeepAnalyze even at a 1 percent error cutoff), and low-value branch expansion the strongest drain; LAMBDA's time amplification on high-cost queries averaged 46.2 times, with 8 runs hitting the 1800 second cap. DB-GPT's Docker sandbox held every hijack below the host, while the other three open-source agents escaped to the host in 2.7 to 9.3 percent of cases. Databricks blocked five of six hijack techniques at 0 percent yet returned attacker-preferred values on 80 percent of trust-bias payloads and let 76 to 80 percent of search-space drain payloads through, and BigQuery showed 84 and 88 percent on the two trust-bias techniques. Swapping DB-GPT's backend among four frontier models suppressed different techniques for each model but none blocked the full taxonomy.

13. A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases

    Sequeda · 2023 arXiv

    Synthesis

    Plain-language abstract Sequeda, Allemang, and Jacob asked how accurately GPT-4 answers business questions over an enterprise SQL database, and how much a knowledge graph built from an ontology and mappings changes the answer. They built a benchmark from a 13-table subset of the OMG Property and Casualty insurance model with 43 questions spanning simple reporting to KPI aggregations over few or many tables. With a zero-shot prompt holding the DDL, GPT-4 produced a correct result 16.7 percent of the time; with the ontology and a SPARQL endpoint virtualizing the same tables, 54.2 percent. Questions needing more than 4 tables scored 0 percent in SQL and 35 to 39 percent over the graph.

    Motivation The authors saw existing text-to-SQL benchmarks such as Spider, WikiSQL, and KaggleDBQA as misaligned with enterprise use: their schemas are small where enterprise schemas run to hundreds of tables, they skip the reporting, metrics, and KPI questions that drive operational decisions, and they have no business-context layer of metadata, mappings, and ontologies. Without that layer, they argued, an LLM is disconnected from what the data means and produces hallucinated, unexplainable answers, and the size of the effect had not been measured.

    Methodology The schema is a 13-table excerpt of the 199-table, 243-foreign-key OMG P&C model, populated by hand with a couple of rows per table. The 43 questions are classed by question complexity (SELECT-FROM reporting versus aggregations and KPIs) and schema complexity (0 to 4 tables versus more than 4), giving four quadrants of 12, 11, 10, and 10 questions. The context layer is an OWL ontology of insurance concepts and R2RML mappings hosted on data.world, which translates SPARQL against the ontology into SQL over the tables. GPT-4 received a zero-shot prompt with either the DDL or the ontology, at temperature 0.3 with 2048 max tokens and a 60-second timeout counted as failure. Accuracy followed Spider's execution accuracy, aggregated per question as the fraction of repeated runs matching a human-written reference query, over sessions run between 20 September and 13 October 2023.

    Results Average execution accuracy was 16.7 percent for SQL and 54.2 percent for SPARQL over the knowledge graph, a 37.5 point gain. By quadrant, SQL scored 25.5 percent (low question, low schema), 37.4 percent (high question, low schema), and 0 percent in both high-schema quadrants, while the graph scored 71.1, 66.9, 35.7, and 38.7 percent. SQL failures were hallucinated column names, filter values, and joins; SPARQL failures were wrong or reversed paths through the ontology, with no hallucinated class or property observed. The authors concluded that context describing business semantics must be managed as a first-class asset, ideally in a data catalog on a knowledge graph architecture.

14. Introducing Metrics SQL: A SQL-based semantic layer for humans and agents

    Bangarwa · 2026

    Synthesis

    Enforcement in Metrics SQL rides on the same compiler. Row-level security and access rules are defined once on the metrics view and enforced on every query regardless of who or what is asking (a dashboard, an API call, or an agent through the MCP server), and every literal in a query becomes a parameterized argument, which the post presents as SQL injection safety regardless of where the query originates. The restrictions on the dialect are framed as design constraints rather than oversights: they keep the compilation model tractable and the security guarantees auditable.

    Why it matters A query-time constraint rather than a definition-time one: the agent can only express what the restricted dialect allows, and the access policy is applied below the level the agent can reach.

15. Metric Semantic Layer: How Lyft Governs and Scales Key Data Definitions

    Channe · 2026

    Synthesis

    Governance in MSL works by admission and ownership rather than by a query interface. Only Golden Metrics, those with at least 2 distinct use cases (Rides Completed is monitored on dashboards and used in operational ML models), are onboarded; anything too niche stays out. Every Golden Metric has a Business Owner (an analyst or data scientist team responsible for metric health, who supplies the written and SQL definitions, verifies them on a pre-defined cadence, and defines the allowed dimensions and grains) and an Operational Owner (an engineering team responsible for data health, on-call, data quality checks and backfills); owners are always teams, never individuals, and any change to a definition needs approval from both. Consumers call get_sql(metric_group_name, start_date, end_date, dimension_names, metric_names, time_granularity) and receive generated SQL, and the versioned package is deployed to dependent applications through automated code refactors so a change made once propagates.

    Why it matters Enforcement sits at the definition layer (what counts as a metric, who may change it, how a change propagates), and the post describes no mechanism that stops an agent from writing its own SQL against the warehouse.

16. SLayer: An expressive, embeddable semantic layer for AI agents and humans

    MotleyAI · 2026

    Synthesis

    What the README says SLayer enforces is narrower than what it defines. Database connections are read-only, row-level security is applied by the layer rather than left to the model, and a datasource password is read at initialization and is neither saved to disk nor exposed to Claude. The README describes no validation of metric definitions and no mechanism that prevents an agent from reaching the database by another path.

    Why it matters The constraint is on what the agent can touch (read-only, row-filtered), not on what it can compute; definition-level governance of the kind Lyft describes is left to whoever writes the layer.

17. Why 99% of BI Tools Get Embedded Analytics Wrong And How Omni Fixed It ft. Chris Merrick

    Merrick · 2026

    Synthesis

    The show notes for the 21 July 2026 episode quote Merrick: 'You gotta curate the dataset that the human or the LLM is seeing if you want it to be able to use it effectively', and describe the semantic layer as the place where that curation happens and where business terms such as 'uplift' are defined so that humans and agents work within guardrails. Merrick separates 2 modes of analytics, ad hoc exploratory analysis and the governed version, and the notes frame the design problem as combining exploratory AI-driven analysis with the consistent, repeatable metrics an organization depends on. The notes also list a feedback loop in which learnings from user conversations are captured automatically and fed back so that the analytics layer evolves with the business; no mechanism and no numbers are given.

    Why it matters The constraint is on the model's input rather than its output: the semantic layer decides what the LLM sees, and governed and exploratory questions are treated as different modes rather than one interface.

18. An Alternate Agentic AI Architecture (It's About the Data)

    Wenz · 2026 · 0 cites arXiv

    Synthesis

    Plain-language abstract Enterprise questions usually need data from several systems joined together, and the authors argue that letting a language model decide which tools to call and how to combine results is the wrong design. RUBICON instead has the user write a short query in a restricted notation, AQL, naming the tables and columns wanted and leaving only the filter condition in plain language. Wrappers turn each source into a table, a query processor joins them, and every intermediate result is visible. On seven questions over five sources, plain and tool-using LLM agents get none right; RUBICON gets all seven.

    Motivation Enterprise data is not in the pretraining corpus, lives in specialized stores that cannot be imported into a model, is governed by role- and user-specific access control that models ignore, and must be joined across sources. The authors' earlier work found text-to-SQL accuracy on real warehouses more than 50 percent below Spider and BIRD levels because of redundant schemas, institutional jargon and harder business queries, so an LLM-centric architecture built on text-to-SQL cannot be relied on. Copilots that ingest and index sources into one semantic search improve discoverability but cannot perform joins.

    Methodology AQL is FIND columns FROM table WHERE natural-language predicate, with aggregates, JOIN between two FIND blocks, schema introspection and SAVE, OUTPUT and DELETE. Source wrappers, assumed to be built by enterprise staff, translate the predicate to the native interface and expose each source as a logical relational view. Interactive mode executes one command at a time with the intermediate table shown; compiled mode runs a sequence as one plan through a relational-style optimizer. The benchmark has seven expert-written queries over Wikipedia, a 97-table anonymized university warehouse in SQLite, a research-lab website, Gmail and pretrained model knowledge, each requiring exactly two sources with three distractors. Baselines are single-shot GPT-5-mini, Gemini-3-flash-preview and Claude-Sonnet-4.6, and LangChain ReAct agents over the same models with all five sources as tools, each under natural-language and AQL-style prompts. Metrics are correctness, tokens, provider-reported cost, time to first token and tool-call count.

    Results All twelve LLM configurations score 0 of 7; RUBICON scores 7 of 7 with two tool calls per query. Failures are omitted sources, premature termination and incomplete joins rather than fabricated facts. ReAct input tokens average about 20,000 for GPT and over 469,000 for Gemini with AQL prompting, with first-token latency of minutes, while RUBICON uses 4,182 input tokens. AQL prompting improves which sources agents call but not how many calls they make. Two valid AQL plans for the Turing-or-Nobel query differ structurally in cost, award-first pushing a selective predicate into Wikipedia versus faculty-first scanning a page per professor, which the authors use to argue for a cost-based planner. The paper's benchmark is small and self-built; a hundred-query benchmark from the Munich transportation department is announced as pending permission.

The boundary: wrong, uncertain, or unwilling

The previous theme's numbers were measured inside the modeled boundary. This theme is about what the same systems do outside it, and it is where the argument of this map turns.

Take the dbt benchmark's other column. On questions that exceed what the semantic layer models, the layer scores 0.0 percent while unconstrained text-to-SQL scores 70.0 percent for one model and 100.0 percent for another. dbt states the reason plainly, that the layer returns an error rather than wrong data. That is the whole trade in one line, published by the party with the strongest incentive to bury it. ATHENA made the same trade twenty years earlier and measured it: no interpretation at all for 11 of 108 finance questions, which is exactly the distance between its 99 percent precision and its 88.9 percent recall. PRECISE made it thirteen years before ATHENA and proved it. Given a lexicon and an attachment function, PRECISE is sound and complete over a class of questions its authors call semantically tractable, it can decide for any question whether the question is in that class, and outside the class it asks for a paraphrase instead of guessing. The class covered 97 percent of restaurant questions, 88 percent of jobs and 77.5 percent of geography in Mooney's data, and inside it the system made no mistakes on any of the three databases.

So the two systems are not being compared on the same axis. Text-to-SQL maximizes the share of questions answered. An enforcing system maximizes the share of returned answers that are correct, and pays for it in coverage. Calling one more accurate than the other requires first deciding which of those two quantities the product is for, and that is a decision about consequences rather than about models. When an answer is pasted into a board deck, a confident wrong number costs more than a refusal. When an analyst is exploring, a refusal costs more than a number they will sanity-check anyway.

The literature has two distinct machineries for declining, and they are not interchangeable. Statistical abstention generates a candidate and then estimates whether to trust it: adaptive abstention at the schema-linking step with a human in the loop, confidence estimation over generated SQL, calibrated confidence for tabular question answering, reliability-shaped rewards. Structural abstention makes the unanswerable unrepresentable, so no confidence estimate is required, which is Wu's formulation and also what ATHENA and the dbt layer are doing without using the term. The first can be wrong about its own uncertainty. The second cannot, and instead is silently wrong about scope whenever the model is incomplete.

Underneath both sits ambiguity. Bhaskar and colleagues show that a large share of realistic questions have more than one defensible reading, which means a system with no way to say so is forced to guess, and a benchmark with one gold query per question will score the guess as a failure regardless of which reading it took.

One study measured why any of this is worth paying for. NaLIR asks the user to confirm a query tree before it executes anything. With that step, fourteen participants completed 88 of 98 tasks on the Microsoft Academic Search database. With it removed there were 32 failures and the participants caught 7, so 25 wrong answers were accepted as correct by the person who had asked the question, and Li and Jagadish note that the undetected ones were mostly aggregates, which cannot be checked by reading the number. That is the failure mode every enforcement mechanism in this map is built against, measured on people rather than argued from first principles.

Three 2026 results extend the outcome space. Fei's role-conditioned benchmark scores refusal as an output and finds a fourth cell the three-way split lacks: refused when it should have answered. Fine-tuning an 8B model on role-labeled Spider cut violations from 46 to 4 percent and cost 17 points of correct, compliant answers through over-refusal, and the gain did not transfer to LiveSQLBench. One model reasoned in its chain of thought that a request needed a forbidden UPDATE and then emitted a SELECT over the same table, a refusal reached and then dropped at decoding. SemPlan found the mirror image. Direct SQL behind parsing guards refused 21.42 percent of cases falsely, while the compiled architectures refused under 2 percent and produced unsafe or invalid output on 60 to 64 percent, so the guards moved the failures rather than removing them. SUQL reports the split for a hybrid system: 21 of 100 single-turn questions returned no answer, 7 of them warranted, a false-negative rate of 14 percent that is the price of a parser that declines rather than fabricates.

Wang's attacks are the adversarial form of the boundary. Outside its modeled scope the agent failed on unresolved precedence between sources of meaning, a false rule in a document beating the structured data, and the failures held across four base models and two production services. Merrick states a vendor's boundary in one line, if Omni becomes agentic middleware we will have failed, and puts identity inside it: business users log in with context about who they are, and that context is part of what the answer is allowed to use.

1. Towards a Theory of Natural Language Interfaces to Databases

   Popescu · 2003

   Synthesis

   Plain-language abstract PRECISE is a natural-language interface to databases built around a decidability result rather than around a parser. The authors define a class of questions they call semantically tractable, prove that their system maps every question in that class to the right SQL, and show that the system can tell, for any question it is given, whether the question is in the class. Questions outside it get a request for a paraphrase instead of an answer.

   Motivation The authors argue from user trust rather than from accuracy. A person will tolerate an interface that says it did not understand, and will stop trusting one that returns a plausible wrong answer, in the same way they would abandon a mouse that occasionally moved the pointer somewhere random. So the design target is a system that is never confidently wrong, which means the system needs a principled account of what it can and cannot attempt.

   Methodology A question is tokenized against the database's own elements: relations, attributes and values, with a lexicon supplying the word stems that match each one. A question is semantically tractable when some complete tokenization maps one-to-one onto a set of database elements with every attribute token paired to a compatible, syntactically attached value token. Matching is reduced to a max-flow problem, and syntactic attachment constraints come from the parse tree. Evaluation is on three of Ray Mooney's benchmark databases, restaurants, jobs and geography, with hand-written gold SQL, plus a comparison against Mooney's learning interface and Microsoft's English Query product.

   Results Theorem 1: given a lexicon and an attachment function, PRECISE is sound and complete for any semantically tractable question, and Corollary 1 adds that it rejects everything else as intractable. The tractable class turned out to be large: 97 percent of restaurant questions, 88 percent of jobs and 77.5 percent of geography. Inside the class the system made no mistakes on any of the three databases, and its precision advantage over Mooney's interface was significant for geography and jobs at p = 0.05. The rejections are diagnosed against the database, not the sentence: an unknown word, an unimplemented function, or a fact the database does not store, as in a question about the population density of cities when the schema records density only for states.

2. Constructing an Interactive Natural Language Interface for Relational Databases

   Li · 2014

   Synthesis

   Plain-language abstract NaLIR is a natural-language interface that does not try to be right on its own. It converts a question into a query tree, explains that tree back to the user in words, lets the user correct the interpretation, and only then compiles the confirmed tree into SQL with aggregation, nesting and joins.

   Motivation Li and Jagadish start from the position that a natural-language interface cannot be expected to be perfect, and that the real problem is what happens when it is wrong. A user handed a number has no way to tell whether the system understood the question, no way to verify the answer independently, and no guidance beyond rephrasing and hoping. Their fix is to move verification earlier, to a representation a person can actually read.

   Methodology Three components: a transformation from the dependency parse into a query tree, an interactive communicator that explains each ambiguous word or phrase and the reading of the whole sentence and offers up to five interpretations per ambiguity, and a translator that compiles the verified tree block by block into SQL. Evaluated with a user study on the Microsoft Academic Search database, 14 participants recruited with flyers, tasks drawn from 196 distinct query logics enumerated from the MAS website, with the faceted MAS web interface as the baseline and the system run both with and without the interactive step.

   Results With interaction the participants completed 88 of 98 query tasks; the per-difficulty splits were 34 of 34 simple, 34 of 34 medium and 20 of 30 hard, against 26 of 32, 23 of 34 and 15 of 32 without it, and 20 of 33, 18 of 32 and 18 of 33 for the MAS website. The finding that matters most is about detection rather than accuracy: of the 32 failures in the non-interactive condition the participants noticed 7. The other 25 wrong answers were accepted as correct by the person who had asked the question, and the authors note the undetected ones were mostly aggregates, which cannot be checked by inspecting the result.

3. ATHENA: An Ontology-Driven System for Natural Language Querying over Relational Data Stores

   Saha · 2016

   Synthesis

   Plain-language abstract ATHENA is a natural-language querying system that translates a question into a query over a domain ontology first, and only then compiles that ontology query into SQL against whatever physical schema the data happens to live in. The intermediate query language is OQL, and the ontology carries the business concepts and their relationships independently of how the tables are laid out.

   Motivation Systems that map language directly onto a schema inherit every accident of that schema: how it was normalized, what the columns were named, which denormalizations were done for performance. The authors' argument is that a question is asked in terms of domain concepts, so the translation target should be domain concepts, and the mapping from concepts to physical tables should be a separate, maintained artifact.

   Methodology The system performs evidence-based interpretation generation over the ontology, ranks candidate interpretations, and translates the selected one into OQL and then into SQL. It is evaluated on three domains, GEO, MAS and FIN, with precision measured over the questions for which it produced an interpretation and recall measured over the whole workload.

   Results ATHENA reports 100% precision on GEO and MAS and 99% on FIN, with recall of 87.2%, 88.3% and 88.9%. The gap between the two figures is the abstention rate: the system generated no interpretation at all for 11 of 108 finance questions. That is a 2016 system with refusal built into its architecture and visible in its reported metrics, long before the current literature named the behaviour. The durable argument is not the accuracy but the physical independence: an ontology query survives a schema migration, which is the property a semantic layer is actually bought for.

4. Duoquest: A Dual-Specification System for Expressive SQL Queries

   Baik · 2020 arXiv

   Synthesis

   Lets the user supply a partial query sketch alongside the natural-language question, turning an underspecified request into a constrained search over candidate queries.

   Why it matters Interaction is the resolution this literature keeps forgetting: a system that can ask rather than guess does not need to choose between a wrong answer and a refusal.

5. Benchmarking and Improving Text-to-SQL Generation under Ambiguity

   Bhaskar · 2023 arXiv

   Synthesis

   Constructs a benchmark where questions admit multiple valid SQL interpretations and measures how systems behave when there is no single correct query.

   Why it matters A benchmark with one gold query per question scores a defensible alternative reading as a failure, which means ambiguity contaminates every accuracy number in this map and is invisible in all of them.

6. Confidence Estimation for Text-to-SQL in Large Language Models

   Entezari Maleki · 2025 arXiv

   Synthesis

   Studies how well confidence signals from large language models predict whether a generated SQL query is correct.

   Why it matters Everything in the statistical abstention line depends on this question, and the answer determines whether a low-confidence flag is a usable product surface or a false comfort.

7. Reliable Text-to-SQL with Adaptive Abstention

   Chen · 2025 arXiv

   Synthesis

   Abstains when schema linking is insufficiently confident and routes those cases to a human, rather than emitting a query the system cannot justify. Published in PACMMOD as a SIGMOD 2025 paper.

   Why it matters It is the statistical counterpart to structural abstention: the candidate is generated first and the decision is whether to trust it, which means the abstention itself can be miscalibrated.

8. Boundary-Aware NL2SQL: Integrating Reliability through Hybrid Reward and Data Synthesis

   Tian · 2026 arXiv

   Synthesis

   Trains the boundary into the model with a hybrid reward and synthesized data, so declining is a learned behaviour rather than a post-hoc filter.

   Why it matters It is the third distinct mechanism for refusal in this theme, alongside structural impossibility and post-generation confidence, and the three have never been compared on one workload.

9. Calibrated Confidence Estimation for Tabular Question Answering

   Voss · 2026 arXiv

   Synthesis

   Targets calibration specifically, so that a stated confidence over a tabular answer corresponds to an actual frequency of correctness.

   Why it matters Calibration is the property that makes a confidence-gated analytics product defensible, and it is a different requirement from ranking answers by confidence.

10. Never the Number: Structural Abstention for AI Systems Whose Answers Are Consumed as Fact

    Wu · 2026 arXiv

    Synthesis

    Plain-language abstract An architectural argument for systems whose outputs are consumed as fact. A trusted deterministic kernel computes every value the system returns; a generative shell wraps it and may influence which question gets asked. The invariant is that a component capable of fabrication never determines a returned value, so requests the system cannot answer are unrepresentable rather than answered badly.

    Motivation Most work on when a system should decline treats it as an estimation problem: generate a candidate answer, estimate confidence, decline below a threshold. The author's position is that for outputs consumed as fact, a calibrated probability of being wrong is the wrong object. What is wanted is a construction in which the wrong answer cannot be produced at all.

    Methodology The paper is a design argument supported by a two-year production case study. It states the invariant, describes how the kernel and shell divide responsibility, and works through what the boundary implies for system behaviour. It reports no datasets and no accuracy metrics, which is a limitation the paper states rather than one a reader has to discover.

    Results The contribution is the distinction between structural and statistical abstention: structural abstention needs no confidence estimate because the unanswerable request has no representation in the system, in contrast to abstention that generates a candidate and then estimates whether to trust it. That distinction separates two literatures that otherwise look like one, and it also names the failure mode of the structural approach. Such a system cannot be miscalibrated, and it can be silently wrong about the scope of what it models.

11. Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update

    Ganz · 2026

    Synthesis

    Plain-language abstract A vendor benchmark comparing questions answered through dbt's semantic layer against the same questions answered by a model writing SQL directly. Eleven questions are run twenty times each over a fifteen-table insurance schema in third normal form, with two frontier models, and the results are reported separately for questions the semantic layer models and questions it does not.

    Motivation The semantic-layer argument needs a number, and the vendor with the semantic layer is the party motivated to produce one. Read as a designed experiment rather than as marketing, it is the only public measurement of what enforcement does when the same underlying data and the same models are used on both sides.

    Methodology Eleven questions, twenty repetitions each, one schema. Both conditions run against the same modeled data; the difference is whether the model calls the semantic layer's API or writes SQL against the tables. The authors also report a longitudinal comparison of the same question set from 2023 to 2026, and separately report performance on questions outside what the semantic layer models.

    Results Through the semantic layer, claude-sonnet-4-6 reaches 98.2% and gpt-5.3-codex 100.0%, against 90.0% and 84.1% writing SQL against the same data. Across the full question set from 2023 to 2026, text-to-SQL moved from 32.7% to 64.5% and the semantic layer from 60.5% to 72.7%. On questions the layer does not model, the layer scores 0.0% while text-to-SQL scores 70.0% and 100.0%, because the layer returns an error rather than a number. The same eleven-question experiment therefore produces both the highest and the lowest figures in this map, which is the clearest available demonstration that a single accuracy column cannot describe these systems.

12. TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis

    Shi · 2026 arXiv

    Synthesis

    Scores reliability and robustness for structured data analysis rather than accuracy alone, across perturbations and failure conditions.

    Why it matters It is the evaluation shape this theme needs, because a system that refuses well and a system that answers everything cannot be compared on a single accuracy column.

13. Benchmarking Text-to-SQL under Role-Based Access Control

    Fei · 2026 arXiv

    Synthesis

    Plain-language abstract Text-to-SQL benchmarks assume the person asking can see every table and column. Fei, Jiang, Yang, and Xiao asked what happens when the asker holds a role that can see only some of them. They built a pipeline that synthesizes plausible roles and column-and-operation permissions for Spider, BIRD, and LiveSQLBench, had four annotators validate them, and labeled each question-role pair allow or deny by parsing the gold SQL. Thirteen models were then scored on whether they refused when they should and wrote compliant SQL when they could. Models with strong unrestricted accuracy violated the policy on a large share of questions, and hiding columns, fine-tuning, and few-shot examples did not repair it.

    Motivation Production databases enforce role-based access control, so an unauthorized query fails at execution or is silently masked; a check at the database blocks it but cannot say whether the system should have refused earlier, missed a compliant alternative, or over-refuses. Existing benchmarks score execution accuracy under unrestricted access, so a query that is correct but reads a denied column counts as a success, which the authors call an RBAC-rejected success. The few prior efforts on access control used database- or table-level permissions and had no systematic construction pipeline or empirical study.

    Methodology An LLM inferred each database's application context from its schema, derived roles with responsibilities, and then assigned column-level permissions per operation; configurations were screened on denial rate (rejected below 5 or above 90 percent), coverage variance, pairwise Jaccard overlap above 0.8, and embedding similarity to the schema, then accepted by 3 of 4 annotators (28 of 53 passed first time, 16 after one regeneration, 9 were revised by hand). Each database also received 2 or 3 programmatically generated DataOperator roles with broad but incomplete access. SQLGlot extracted the (table, column, operation) triples each gold query needs, and an instance is allowed when they are a subset of the role's policy. The result is 53 databases, 399 tables, 3,353 columns, and 21,502 instances from 3,160 questions, with deny shares of 47, 65, and 76 percent on Spider, BIRD, and LiveSQLBench. Each model saw a fixed zero-shot template at temperature 0, one random role per question per pass, over 5 seeds, and was scored on violation rate, over-refusal rate, AC-F1, Safe-EX, and Safe-Deny.

    Results Violation rates exceeded over-refusal rates for every model on every dataset. On Spider GPT-5 violated 2.55 percent of instances with AC-F1 95.79 while Snowflake-Arctic-R1-7B violated 45.43 percent with AC-F1 70.14 despite higher execution accuracy; on BIRD Snowflake reached 63.77 percent and on LiveSQLBench 76.15 percent, where Claude Sonnet 4.5 was lowest at 12.40 percent. Compliance fell with query difficulty and with the number of joins, driven by violations rather than refusals. Restricting the prompt to the role's schema halved DeepSeek-V3.2-Coder's reads of known-forbidden columns but raised GPT-5-mini's violation rate from 12.49 to 26.72 percent as models wrote SQL over visible or invented columns instead of refusing. Fine-tuning on role-labeled Spider cut Llama3-SQLCoder-8B's violations from 46 to 4 percent at a 17-point Safe-EX cost, gained 7.7 and 13.0 AC-F1 on BIRD for the two tuned models, and lost 6.1 on LiveSQLBench for Qwen2.5-14B; a 298-instance row-level pilot showed DeepSeek-V3.2-Coder at 50.27 percent violations and GPT-5 at 2.15.

14. Data Agents Under Attack: Vulnerabilities in LLM-Driven Analytical Systems

    Wang · 2026 arXiv

    Synthesis

    Plain-language abstract Data agents let a language model plan SQL queries, run Python, and read files over a company's databases. Wang and colleagues asked what new ways such an agent can be attacked when the database, not a web page, is the source of untrusted content. They defined 8 vulnerabilities across how the agent interprets inputs, executes tools, and applies policy, turned them into 14 attack techniques with 350 payloads grounded in real schemas, and ran 3,075 tests on four open-source agents and two cloud analytics services. No system blocked every attack, trust bias between conflicting sources was the most exploited weakness, and changing the underlying model shifted the exposure rather than removing it.

    Motivation Database security models the database as a closed SQL engine and studies injection, access control, and integrity, while LLM-agent security assumes the agent reasons over documents and web pages. A data agent couples the two: content in table cells, schema comments, and uploaded files becomes reasoning context, that reasoning becomes executable SQL and Python, and a session's policy must govern both transitions. The motivating example is a sequence of individually legitimate queries whose combination discloses a record no single query would, a channel neither line of prior work captures.

    Methodology The authors formalized a data agent as a model and state, a tool set, data assets, policies, and an execution environment, with an adversary who controls only prompts and uploaded data. From 8 vulnerabilities they derived 3 goals (hijack, mislead, drain), 7 tactics, and 14 techniques, then used a retrieval-augmented pipeline to instantiate 5 templates per technique on 5 enterprise databases from DAComp-DA, giving 350 frozen payloads over 27 databases. Targets were DataInterpreter, DB-GPT, and LAMBDA on DeepSeek-V3.2, DeepAnalyze on its own 8B model, and Databricks Genie Code and BigQuery's conversational agent under default governance in May 2026, plus DB-GPT rerun with Sonnet-4.6, GPT-5.2, and Gemini-3-flash. Hijack was scored by a five-level blast radius, mislead by relative error against the attacker's target value, and drain by token and time amplification against a benign run, with success defined as blast radius of at least 1, error of at most 10 percent, or amplification of at least 5.

    Results Every agent was exploitable on at least five of the eight vulnerabilities and every vulnerability appeared in at least three agents. Code injection was the strongest hijack, data-type trust bias the strongest mislead (100 percent on DeepAnalyze even at a 1 percent error cutoff), and low-value branch expansion the strongest drain; LAMBDA's time amplification on high-cost queries averaged 46.2 times, with 8 runs hitting the 1800 second cap. DB-GPT's Docker sandbox held every hijack below the host, while the other three open-source agents escaped to the host in 2.7 to 9.3 percent of cases. Databricks blocked five of six hijack techniques at 0 percent yet returned attacker-preferred values on 80 percent of trust-bias payloads and let 76 to 80 percent of search-space drain payloads through, and BigQuery showed 84 and 88 percent on the two trust-bias techniques. Swapping DB-GPT's backend among four frontier models suppressed different techniques for each model but none blocked the full taxonomy.

15. SUQL: Conversational Search over Structured and Unstructured Data with Large Language Models

    Liu · 2023 arXiv

    Synthesis

    Plain-language abstract Liu and Lam asked how a conversational agent can answer questions that need both a database's columns and its free-text fields, such as an Italian restaurant with a romantic atmosphere. They extended SQL with two primitives, ANSWER and SUMMARY, that call a language model on text values, built a few-shot LLM parser that translates user turns into this language, and wrote a compiler that uses dense retrieval and predicate ordering to keep LLM calls few. On HybridQA the few-shot system came within 8.9 exact-match points of a model trained on 62K examples. On a Yelp database of 1,828 restaurants the agent satisfied all user constraints 90.3 percent of the time in conversation, against 63.4 percent for a linearized-retrieval baseline.

    Motivation Patient records, financial databases and review sites hold structured columns beside free text, and crowdsourced restaurant questions showed more than 49 percent needed both. Prior chat systems routed each question to a structured or an unstructured module by a classifier, which cannot serve questions that need both, or linearized tables into text and retrieved by embedding, which gives up the comparisons and calculations SQL does well. Earlier hybrid question-answering systems worked only on small tables, dropped the expressiveness of structured queries, or supported limited compositions of the two.

    Methodology SUQL defines ANSWER(t, q), returning the answer to question q over text t, and SUMMARY(t) as sugar for ANSWER with a summary question; both return text usable anywhere SQL accepts text, so every HybridQA question type has a SUQL form. A two-stage in-context parser first decides whether the knowledge base is needed and then predicts the query, and the agent reports what it searched and says when nothing was found. The compiler answers ANSWER filters by retrieving candidates with precomputed dense embeddings and then verifying the clause with an LLM, aggregates similarity across multiple text constraints, lists enumerated values in the schema prompt when a type has at most 10 members and otherwise overloads equality with a zero-shot CLASSIFY, orders non-LLM predicates first in disjunctive normal form, and evaluates lazily under LIMIT. Experiments used GPT-4-1106-preview on HybridQA (about 70K questions over 13,000 Wikipedia tables) with fewer than 10 examples and up to 2 retries, and gpt-3.5-turbo-0613 on 1,828 Yelp restaurants from 4 cities with 100 crowdsourced questions and 96 turns across 20 conversations.

    Results The few-shot SUQL system scored 59.3 percent exact match and 68.3 F1 on the HybridQA development set and 59.0 EM and 68.4 F1 on test, 8.9 EM and 7.1 F1 below S3HQA and more than 10 points above HPROPRO's few-shot GPT-4 program prompting at 48.7 EM; under the substring metric of Sui et al. it scored 72.5 percent against their best 56.68. Of 72 sampled errors, 37.5 percent were format mismatches and 23.6 percent wrong or incomplete gold labels, so the authors estimated true accuracy near 84.2 percent. On Yelp, turn accuracy was 93.8 percent single-turn and 90.3 percent conversational, against 57.0 and 63.4 for linearization returning 1 result and 49.7 and 61.9 for 3 results. The agent returned nothing for 21 of 100 questions and 8 of 62 conversational queries, of which 7 and 2 truly had no answer, a false negative rate of 14 and 9 percent.

16. Why 99% of BI Tools Get Embedded Analytics Wrong And How Omni Fixed It ft. Chris Merrick

    Merrick · 2026

    Synthesis

    Merrick is quoted: 'If Omni becomes agentic middleware, we will have failed', which the notes gloss as keeping human decision-makers at the center with AI as an accelerant rather than a replacement; the episode's framing question is whether the future of analytics is another middleware layer or getting closer to the business user. A second quoted line concerns identity: 'We have business users logging into this platform with a lot of context about who they are, and that gives us a heck of a lot of context to give them a great data experience', said in the setting the description calls embedded analytics and customer-facing data experiences.

    Why it matters A vendor's stated limit on its own scope: the system is meant to serve a person who is present and identified, not to sit as an autonomous layer between agents. Who is asking becomes part of the context the answer is allowed to use.

The harness became part of the system

Every result above is reported as a property of a model on a benchmark. It is a property of a model inside a scaffold, and the scaffold is rarely described in enough detail to reproduce.

Two 2026 studies measure this and appear to disagree. Vats and Golev ran two models across Goose, OpenCode and OpenHands-SDK on a stratified 50-task subset of Terminal-Bench Pro and found harness choice worth up to a 40-fold difference in tokens per solved task, while paired within-model pass-rate differences stayed between 0 and 8 points with bootstrap confidence intervals that include zero except at the largest gap. Starace pre-registered a comparison of ReAct, a Planner-Actor-Rater multi-agent scaffold, and a planner-then-executor across five models on GAIA levels 1 and 2, and found scaffold choice alone moving accuracy by as much as 28 points within a single model. The pre-registered prediction that stronger models would be less scaffold-sensitive was rejected in direction.

The two are compatible once you look at what was varied. Goose, OpenCode and OpenHands-SDK are three implementations of broadly the same tool-calling loop. ReAct, Planner-Actor-Rater and planner-then-executor are three different control structures. Terminal-Bench tasks put most of the work inside the model; GAIA tasks put much of it in deciding what to retrieve and when to stop. A defensible reading that neither paper states: harness implementation is mostly a cost variable, and harness architecture is an accuracy variable. That is testable, and nobody has tested it.

Harness-Bench arrives at the operational conclusion from 5,194 execution trajectories over 106 sandboxed tasks, finding substantial variation in completion, process quality, efficiency and failure behaviour across model-harness pairings, and arguing that capability should be reported at the model-harness configuration level rather than the model level. The audit literature says the field is not close to that: a pilot audit of twelve agent benchmark papers found their self-disclosure of harness configuration inconsistent enough that reproduction is often impossible.

For this map the consequence is specific. A semantic layer is a harness. It decides what the model sees, what it is allowed to emit, and what happens to the output. Comparing a governed analytics system against a raw text-to-SQL baseline is a scaffold comparison, and it inherits every attribution problem this theme documents. Any such comparison that does not hold the model fixed and vary one element at a time is measuring the sum of the differences.

The text-to-SQL analogue that earlier versions of this theme called missing now exists twice. Vyas and colleagues ran the same OpenHands framework and the same Claude Sonnet 4.5 with and without DIA's contract-and-verify scaffold and got 50.7 against 35.2 on LiveSQLBench, a 15.5-point harness effect on one model. The scaffold's residual failures sat where a composite metric was named without its formula, 41.6 percent on those questions against 58.9 for stated computations, and nearly all of them executed cleanly and returned a wrong answer. Tummalapenta and Addanki found two harness settings, a two-turn working-memory window and the reasoning configuration, moving multi-turn accuracy by more than any model choice. Stateless accuracy on memory-critical turns hit 0 percent by turn three for all five models, the window restored it to 87.6 to 100, and a mismatched reasoning default made one model generation look up to 33 points worse than its predecessor on one dataset. TwinBI holds the model fixed and changes the scaffold around a dashboard, raising exact match from 43.33 to 63.33 percent and halving the steps.

1. Evaluating the Text-to-SQL Capabilities of Large Language Models

   Rajkumar · 2022 arXiv

   Synthesis

   Evaluates Codex and GPT-3 on Spider without fine-tuning and shows that how the schema is serialized into the prompt materially changes the result.

   Why it matters The earliest text-to-SQL evidence that the scaffold is a variable, published four years before the scaffold papers, and routinely cited for the capability number rather than for the prompt-format finding.

2. Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model

   Jarmak · 2026 arXiv

   Synthesis

   Plain-language abstract A technical monograph synthesizing 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 original case records through a structured multivocal review, framing coding-agent reliability as a dependency chain across measurement/grading validity, containment/recovery engineering, retrieval/context, human review, and cost/allocation layers, and contributing a versioned catalog of 206 reliability records.

   Motivation Coding-agent reliability failures are often attributed to the model alone, but weaknesses in task construction, execution environments, retrieval, state management, verification, or observability can invalidate conclusions drawn about model quality — the system around the model, not just the model, needs systematic evaluation and engineering.

   Methodology A structured multivocal literature review combining 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 author-original case records, organized into a dependency chain across measurement and grading validity, containment and recovery engineering, retrieval and context management, human review, and cost/resource allocation, producing a versioned catalog of 206 reliability records (193 gated practices, 13 open research leads).

   Results Frames reliability as compounding across system layers such that a weakness at one layer (e.g. flawed grading) can invalidate what looks like a model-capability finding at another layer, and that improvements made at one layer often fail to propagate to end-to-end task outcomes — arguing for evaluating and engineering the full system around the model rather than the model in isolation. Note: authored by this site's own author (Stephanie Jarmak); flagged here as a conflict of interest rather than treated as an independent source.

3. From Question Answering to Task Completion: A Survey on Agent System and Harness Design

   Guo · 2026 arXiv

   Synthesis

   Surveys agent system and harness design as a subject in its own right rather than as an implementation detail of model evaluation.

   Why it matters The existence of the survey is the evidence: the harness became a design space with its own literature during 2026, which is recent enough that no text-to-SQL benchmark accounts for it.

4. Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

   Yao · 2026 arXiv

   Synthesis

   106 sandboxed offline tasks and 5,194 execution trajectories, finding substantial variation in completion, process quality, efficiency and failure behaviour across model-harness pairings.

   Why it matters Its operational conclusion, that capability should be reported at the model-harness configuration level, is the standard no text-to-SQL leaderboard in this map meets.

5. HarnessBridge: Learnable Bidirectional Controller for LLM Agent Harness

   Wang · 2026 arXiv

   Synthesis

   Makes the harness itself learnable, with a controller mediating between model and environment in both directions.

   Why it matters If the harness is a learned component then the model-harness pair is the unit of both capability and overfitting, which folds this theme into the improvement theme below.

6. Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures

   Rombaut · 2026 arXiv

   Synthesis

   Builds a taxonomy of coding agent architectures from their source code rather than from their papers.

   Why it matters It supplies the vocabulary the implementation-versus-architecture distinction needs, which is the hypothesis that reconciles the two scaffold studies in this theme.

7. Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety

   Gringras · 2026 arXiv

   Synthesis

   Shows that measured safety, not only measured capability, shifts with the evaluation scaffold.

   Why it matters It extends the attribution problem into the claims that carry the most weight, and the same logic applies to any governed analytics system that reports a hallucination rate.

8. Scaffold Effects on GAIA: A Controlled Comparison

   Starace · 2026 arXiv

   Synthesis

   Plain-language abstract A pre-registered comparison of three agent scaffold architectures across five models on GAIA validation levels 1 and 2, testing how much of measured agent accuracy is attributable to the control structure rather than the model, and whether stronger models are less sensitive to it.

   Motivation Scaffold and model are routinely reported together and attributed to the model. The pre-registered hypothesis was the intuitive one, that a stronger model needs less scaffolding and is therefore less affected by which scaffold it runs in, which if true would let the field keep reporting model-level results as capability.

   Methodology Three architecturally distinct scaffolds are compared: ReAct, a Planner-Actor-Rater multi-agent structure, and a planner-then-executor. Each of five models runs each scaffold on GAIA validation levels 1 and 2 with three attempts per question. The prediction about model strength and scaffold sensitivity is registered in advance of the runs.

   Results Scaffold choice alone moves measured accuracy by as much as 28 points within a single model, and the pre-registered prediction is rejected in direction: stronger models are not less scaffold-sensitive. The contrast with the small accuracy effects reported for three implementations of one tool-calling loop is the useful part. These are three different control structures, evaluated on tasks where the scaffold decides what gets retrieved, and that is the experimental difference that explains why two careful studies of harness effects disagree.

9. The Nuts and Bolts of Natural Language to SQL Translation: A Systematic Analysis of Model Pipeline Optimisations

   Klubicka · 2026 arXiv

   Synthesis

   Systematically varies the optimisations in a natural-language-to-SQL pipeline and finds their effects depend on each other rather than composing additively.

   Why it matters It is the text-to-SQL-specific version of the harness argument, and it explains why single-factor ablations in this literature disagree with each other so often.

10. The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation

    Vats · 2026 arXiv

    Synthesis

    Plain-language abstract A controlled study of how much of a coding agent's measured performance comes from the harness it runs in rather than the model. Two models are run across three agent harnesses on a stratified subset of a terminal-based benchmark, with paired analysis so each task is compared against itself across harnesses.

    Motivation Agent evaluations report a model name and a score, while the harness that decides what the model sees, how tools are called, and when the loop stops is treated as an uninteresting implementation detail. If the harness moves the score, then every published agent result is a measurement of a pair rather than a model.

    Methodology Qwen 3.6 Plus and MiniMax M2.5 are run in Goose, OpenCode and OpenHands-SDK on a stratified 50-task subset of Terminal-Bench Pro. Comparisons are paired within model and within task, with 95% bootstrap intervals over paired tasks, and token consumption per solved task is reported alongside pass rate.

    Results Harness choice produces up to a 40-fold difference in tokens per solved task, while paired within-model pass-rate differences stay between 0 and 8 points, with bootstrap intervals covering zero except at the largest gap. The reading that reconciles this with the much larger scaffold effects reported elsewhere is that these three harnesses are three implementations of broadly the same tool-calling loop. Implementation appears to be a cost variable; the accuracy variable is architecture, and this experiment holds architecture roughly fixed.

11. What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema

    Naser Moghadasi · 2026 arXiv

    Synthesis

    Audits twelve agent benchmark papers against an open scoring schema for what they disclose about their own evaluation conditions, and finds the disclosure inconsistent.

    Why it matters If harness configuration is a first-order variable and papers do not report it, then reproduction fails and cross-paper comparison is not licensed, which is the state most numbers in this map are in.

12. Data Intelligence Agents: Interpreting, Modeling, and Querying Enterprise Data via Autonomous Coding Agents

    Vyas · 2026 arXiv

    Synthesis

    Plain-language abstract Enterprise data work passes through lossy handoffs between the people who know what fields mean, the engineers who structure data, and the analysts who query it. Vyas and colleagues at C3 AI built DIA, in which a single sandboxed coding agent plays three roles (interpret raw files, build and validate a schema, write and execute SQL), keeps its outputs as reviewable artifacts, and records rules it learned from execution in a shared memory. The query agent, run with Claude Sonnet 4.5 and no fine-tuning, matched or beat the best published result on seven SQL benchmarks across four dialects, by 33 points on the conversational BIRD-Interact. Almost every remaining failure runs cleanly and returns the wrong answer, most often on questions that name a metric without defining it.

    Motivation Pipeline text-to-SQL systems chain modules tuned for one subtask; reinforcement-learned specialists lock to one benchmark and dialect; agentic explorers probe the database but forget everything between sessions; memory-equipped SQL agents keep one store and are evaluated narrowly. All emit text rather than executable, inspectable artifacts, and none handle the upstream understanding and schema-building work that decides whether SQL has anything sensible to run against. The authors frame the coding agent, not the language model, as the unit of abstraction.

    Methodology DIA runs one autonomous coding agent (OpenHands with Claude Sonnet 4.5) over a shared workspace as three agents: a Data Interpreter that profiles sources into schemas, value distributions, key candidates, and join paths; a Schema Creator that loads staging tables with provenance and validates row counts, column coverage, key integrity, and load rejects; and a Query Generator that declares an expected result shape, probes the schema by execution, generates SQL, and checks the result against its own declaration. Memory has three tiers: retrieved examples from the BIRD train split, session lessons confirmed on the current database, and cross-session rules promoted only when later questions bear them out, each re-verified by a live probe before use. Evaluation covered 4,187 instances on BIRD-Dev, BIRD-Critic, LiveSQLBench, BIRD-Interact, Spider2-Lite, Spider2-Snow, and Spider2-DBT across SQLite, PostgreSQL, Snowflake, and DuckDB, with o3 as the user simulator and per-benchmark adaptation limited to a standing instruction file.

    Results Scores against best prior: BIRD-Dev 77.7 versus 77.8, BIRD-Critic 64.2 versus 48.8, LiveSQLBench 50.7 versus 38.0, BIRD-Interact 55.7 versus 22.7, Spider2-Lite 71.3 versus 55.2, Spider2-Snow 69.5 versus 63.8, Spider2-DBT 37.5 versus 35.3. Modification tasks scored above pure queries (BIRD-Critic management 78.7 versus query 64.4) because a named target object can be validated by execution, while high-level questions naming a composite metric cost 15 to 17 points. Reasoning failures were the largest class on every benchmark and execution failures under 3 percent. In the BIRD-Interact development run the dominant failure was re-submitting near-identical SQL until the 15-turn cap without asking; a forced-clarification rule left none of the released run's 215 phase-1 failures with zero questions asked. Memory's effect on accuracy was examined only qualitatively; time per question ran from under a minute to roughly ten minutes and no cost breakdown is given.

13. Memory Architectures for Multi-Turn Text-to-SQL: A Benchmark and Empirical Study

    Tummalapenta · 2026 arXiv

    Synthesis

    Plain-language abstract Analysts ask follow-up questions that refer to earlier turns, yet text-to-SQL is mostly graded one question at a time. Tummalapenta and Addanki built EnterpriseMem-Bench, 300 templated sessions over banking, SEC filings, and sales data with each turn flagged for whether it needs prior context, and ran five frontier models under five memory setups. Without any memory every model fell to 0 percent on context-dependent turns by turn 3; a window of the last two turns recovered most of the loss. Adding retrieval of past sessions or written hints helped some model-and-dataset pairs and hurt others, and one model generation regressed sharply on financial filings.

    Motivation Spider and BIRD are single-turn, and accuracy above 70 percent on BIRD has fed a perception that text-to-SQL is solved for production. SParC, CoSQL, MMSQL, and BIRD-INTERACT are multi-turn but none treats memory architecture as a variable or labels which turns need prior context. The authors also note that GPT-5 models reason by default at the API while Claude models do not, so cross-provider comparisons without matched reasoning can misattribute a configuration effect to a capability gap.

    Methodology A 1,451-line Python script generates 300 sessions (1,400 turns) from live profiles of BIRD Financial (8 tables), SEC EDGAR annual data for 20 companies (1,897 records, 2009 to 2026), and Northwind (93 customers, 16,282 orders), with gold SQL as parameterized templates and no language model in construction. Tiers are single-turn (90 sessions), 3 to 5 turns (120), and 6 to 10 turns (90); 1,100 turns are memory-critical. Five conditions: stateless; working memory of 2 turns; of 5 turns; 5 turns plus top-3 episodic retrieval from a ChromaDB store shared across sessions; and that plus one semantic hint from six hand-written notes. Models were GPT-5 mini, GPT-5.2, Claude Sonnet 4.5, Sonnet 4.6, and Opus 4.6, with extended thinking enabled on Claude; 35,000 single-run evaluations, scored by execution accuracy and a six-way error taxonomy, with a fallback SQL per turn to detect cross-session contamination.

    Results Stateless aggregate accuracy was 15 to 19 percent and exactly 0 percent on memory-critical turns by turn 3 for every model, against 67.0 to 73.0 percent on first turns; a two-turn window lifted aggregate accuracy to 74.5 to 86.4 percent. Growing the window to 5 turns was small and usually negative; episodic retrieval ranged from +4.5 to -12.6 points, the semantic hint from +14.1 to -10.8, and the full stack net of the two-turn window from +12.6 to -15.9. Sonnet 4.6 trailed Sonnet 4.5 on SEC EDGAR by 17.2, 29.0, and 33.1 points across conditions while Opus 4.6 reached 99.2 percent, and all 250 failed Sonnet 4.6 EDGAR turns matched neither gold nor the contamination fallback. Enabling reasoning added 5.7 to 18.9 points for the Claude models and left wrong-result errors as the only remaining class; without reasoning, preamble errors had run at roughly 26, 38, and 45 percent for Sonnet 4.5, Sonnet 4.6, and Opus 4.6 in the stateless condition.

14. TwinBI: An Agentic Digital Twin for Efficient Augmented Interactions with Business Intelligence Dashboards

    Jang · 2026 arXiv

    Synthesis

    Plain-language abstract Jang and Li asked whether a business intelligence dashboard becomes easier for people and for agents to use when it is wrapped in a digital twin that knows the dashboard's state. TwinBI puts a Cube semantic layer, a multi-agent orchestrator and a unified interaction log around Apache Superset dashboards, so chat questions are answered against the same filters and tabs the user is looking at and every chart's SQL is exposed. A Playwright browser agent running gpt-5-mini answered 30 questions with and without the twin. Exact match rose from 43.33 to 63.33 percent and average steps fell from 16.47 to 6.90.

    Motivation Dashboards present pre-aggregated views, so answering a question that crosses views or grains means hovering, clicking and reading numbers off charts, which an LLM browser agent does slowly and unreliably. Chat assistants bolted onto BI tools do not know what the user has filtered or which tab is open, so their answers and the dashboard drift apart. Jang and Li wanted one authoritative record of interaction state that both the human interface and the agents read.

    Methodology TwinBI has 5 layers: a Streamlit chat with embedded Superset dashboards that logs tab navigation, series toggles, cross-filters and global filters; a FastAPI orchestration layer with Docs Retriever, Answer Composer, Schema Explorer, Chart Manager, Insight Seeker and Orchestrator agents over a unified interaction log; a Cube semantic layer declaring measures, dimensions, hierarchies and join paths and enforcing compatible grains; Superset; and DuckDB reached only through the semantic layer. The A/B benchmark used a retail sales dashboard with product, store and date dimensions and 30 queries in 5 families of 6, covering store and district ranking, premium products, quarter-over-quarter growth, cross-view aggregation and trap questions. The browser agent used gpt-5-mini with a 30-step budget; reference answers were resolved from the database, the cube API and the dashboard, checked for agreement, and verified by hand. A usability study ran 5 participants through 3 scenarios.

    Results Exact match went from 43.33 to 63.33 percent and partial credit from 48.33 to 70.83 percent. Average steps dropped from 16.47 to 6.90, timeouts from 40.00 to 10.00 percent, and invalid actions from 10.93 to 0.00 percent; the share of failures involving looped steps rose from 29.76 to 39.13 percent, and the remaining failures were repeated chat turns. On questions such as Q14 and Q17 the dashboard-only agent spent its full 30-step budget probing hover tooltips while TwinBI answered from 1 chat query grounded in dashboard state. The 5-participant study recorded task accuracy of 100, 73.33 and 100 percent across the 3 scenarios, agreement on feature ranking at Kendall's W = 0.62 (p < 0.01), and 3 of 5 participants used the /insights command, 2 with fully correct output.

15. HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

    Wu · 2026 · 0 cites arXiv

    Synthesis

    Plain-language abstract An agent's results depend on the software around the model, the harness, as much as on the model. HarnessDev asks whether models can build that software themselves. Six models each start from a seed that can parse a task but do nothing else, build a full harness, and then try to improve it from execution feedback. The harnesses are frozen and scored on held-out tasks in code, data analysis, writing and research, by task success and by tokens spent. Generated harnesses match or beat human-built ones for writing and machine-learning experiments, trail badly for code and search, often only work with the model that wrote them, and improve unreliably under feedback.

    Motivation Agent benchmarks fix a harness and report model performance inside it, so the harness is treated as experimental configuration rather than as an artifact to be developed. Yet with identical weights GPT-5 solves 35.2 percent of Terminal-Bench 2.1 inside Terminus 2 and 49.6 percent inside Codex CLI. The authors frame harness engineering as the third layer of forward-deployed-engineer work, after translating vague intent into objectives and constructing a feedback signal, and note that editing one's own harness differs from ordinary code editing because the change alters how the model observes, plans and recovers in every future task.

    Methodology A creator model works in a development environment (Claude Code, or Codex for GPT-5.5) from a weak seed with no loop, planner, context management, state, verifier or stopping rule, plus a specification and one to three development cases; the result is frozen and run by an executor model on hidden tasks. Creation covers SWE-bench Pro (731), Terminal-Bench 2.1 (89), MLE-bench (75), EQ-Bench3 (46) and BrowseComp (1,266), three independent harnesses per creator-benchmark pair, under Self-Eval (creator executes) and Unified-Eval (Gemini 3.1 Pro executes everything). Evolution starts from the creator's own code harness, exposes a 100-task SWE-Pro feedback set and all 89 Terminal-Bench tasks, budgets ten full evaluation pairs, and afterwards scores every frozen version on a disjoint 630-task SWE-Pro split never shown to the creator. Score paths are isolated from harness self-reports and every run's source and trajectory were audited for prohibited shortcuts, with none found.

    Results Under Self-Eval Opus 4.8 leads at 67.8 average against a human reference of 86.2; writing harnesses approach the reference and MLE-bench medal rates of 32.9 and 32.4 exceed the 24.0 reference, while search shows the largest gap. Fixing the executor reorders creators: an Opus SWE-Pro harness falls from 69.3 to 33.0 because it hard-codes a 120-step limit, and Qwen gains 17.6 points on BrowseComp. Edit size does not predict quality; Gemini adds the fewest lines (1,006) and scores highest on Terminal-Bench. Of 108 code-harness components, 18 never trigger and all concern state and memory; no checkpoint appears in 26,679 trajectories. In evolution, all five self-runtime lineages gain on held-out tasks (+1.43 to +4.44) but only Opus does under a fixed Gemini executor; of 64 revisions only two clear the roughly 4.75-point noise band, feedback and held-out scores agree in direction 53.1 percent of the time, and 2 of 9 declared final versions are held-out best. Of 169 new functions, 25 have no caller.

How text-to-SQL grades itself

Every claim above rests on a benchmark, and the benchmarks are in worse condition than the field's leaderboards suggest.

The result to start from is Jin, Choi, Zhu and Kang. A three-stage human-in-the-loop audit found annotation errors in 263 of 498 BIRD Mini-Dev examples, 52.8 percent, and 76 of 121 Spider 2.0-Snow examples, 62.8 percent. Earlier work had put the Mini-Dev figure at 36.1 percent. The errors are not typos: 57.79 percent of the BIRD cases are a mismatch between the question and the data or schema, 29.28 percent between the question's semantics and the gold query's logic, and 29.66 percent are questions ambiguous enough that more than one query is defensible. Re-evaluating the 16 open-source agents on the BIRD leaderboard against corrected gold changed relative execution accuracy by between minus 7 and plus 31 percent, moved 12 of 16 upward, and shifted ranks by as much as nine places. CHESS went from 62 to 81 percent and from seventh to a tie for first.

The number that should end the argument is the rank correlation. Rankings computed on the uncorrected subset track rankings on the full development set closely, Spearman 0.85 at p equal to 3.26e-5. Rankings computed on the corrected subset do not, Spearman 0.32 at p equal to 0.23. The subset and the full set agree about who is best only while both are wrong in the same way.

Contamination is the failure mode the field worries about instead, and it is the smaller one. SPENCE probes leakage with syntactic perturbations across Spider, SParC, CoSQL and BIRD and finds a clear temporal gradient, with Spider showing the strongest signal and BIRD showing minimal sensitivity and appearing largely uncontaminated. So the most widely used benchmark in the field is comparatively clean of leakage and more than half mis-annotated. Both statements are true, they are independent failure modes, and correcting annotations reshuffles the leaderboard far more than decontamination would.

The methodological repairs have been available for years and are unevenly adopted. Finegan-Dollak showed in 2018 that question splits measure template memorization and query splits do not. Zhong, Yu and Klein showed in 2020 that a single database instance cannot distinguish semantically different queries, and distilled test suites to fix it. de Vries and colleagues argued that the tasks themselves are not ecologically valid, in the sense that nobody outside the benchmark asks questions in that distribution. Any comparison in the earlier themes of this map that was scored by exact match on a single instance against a single gold query is carrying all four of these problems at once.

Three 2026 benchmarks change what is scored. Fei replaces execution accuracy with six outcomes, because a query that reads a denied column passes the standard scorer, and the reordering is large: a 7B model beat GPT-5 on Spider execution accuracy, 78.14 against 74.37, and trailed it on allow-or-deny F1 by 70.14 to 95.79. UniQL keeps the questions, schemas and data fixed and varies only the dialect, 24,544 gold queries across 16, with a metric stricter than BIRD's that fails correct rows in the wrong order. It finds that the stage at which a gold query was produced is itself a difficulty label: the best model scored 59.1 percent on queries a rule-based translator could produce and 14.6 on those that needed a newly summarized rule. SemPlan logs the failure type beside correctness across 6,000 records and shows guards shifting failures between categories. SUQL's error audit is the older warning. Of 72 sampled HybridQA failures, 61 percent were format mismatches or wrong gold, and only 38.8 percent were errors in the system.

1. Improving Text-to-SQL Evaluation Methodology

   Finegan-Dollak · 2018 arXiv

   Synthesis

   Shows that splitting by question rather than by query lets a model score well by recognizing SQL templates it has already seen, and rebuilds the standard datasets with query-based splits.

   Why it matters Eight years old, uncontested, and still not the default in every benchmark this map cites, which is the shape of most methodological progress here.

2. Semantic Evaluation for Text-to-SQL with Distilled Test Suites

   Zhong · 2020 arXiv

   Synthesis

   Plain-language abstract Execution accuracy is measured by running the generated query against one database instance and comparing results with the gold query's results. This paper shows that two semantically different queries frequently agree on any single instance, and constructs compact distilled test suites of database instances that distinguish them.

   Motivation Exact string or tree match penalizes correct queries written differently; execution on one instance rewards incorrect queries that happen to coincide. The field moved to the second because the first was clearly wrong, without measuring how wrong the second is.

   Methodology The authors characterize when queries are indistinguishable on an instance, then search for small sets of database instances that separate semantically distinct queries, distilling them so the suite stays cheap enough to run as an evaluation. Accuracy computed against the distilled suite is compared against single-instance execution accuracy.

   Results Single-instance execution accuracy is an upper bound on semantic correctness, and the paper provides the correction. Most of the literature surveyed in this map still reports the upper bound, which means published execution-accuracy comparisons are systematically generous by an amount nobody re-measures.

3. Towards Ecologically Valid Research on Language User Interfaces

   de Vries · 2020 arXiv

   Synthesis

   Argues that benchmark question distributions for language user interfaces are not produced by real users with real goals, so measured progress may not transfer to deployment.

   Why it matters It is the argument that a governed analytics evaluation has to answer directly, because a semantic layer is defined against the questions an organization actually asks and benchmarks are not.

4. NL2SQL-BUGs: A Benchmark for Detecting Semantic Errors in NL2SQL Translation

   Liu · 2025 arXiv

   Synthesis

   Builds a benchmark specifically for detecting semantic errors, queries that execute cleanly and answer a different question than the one asked.

   Why it matters This is the error class execution-match scoring is structurally blind to, and it is the class a business-meaning layer exists to prevent.

5. Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errors

   Shen · 2025 arXiv

   Synthesis

   Characterizes the errors real in-context-learning text-to-SQL systems make in deployment and builds detection and repair for them.

   Why it matters A deployment error taxonomy is the check on a benchmark taxonomy, and the two do not match: the failures that matter in production are semantic, not syntactic.

6. NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions

   Hou · 2026 arXiv

   Synthesis

   Separates the components of a natural-language-to-SQL solution so a score can be attributed to a stage rather than to the system as a whole.

   Why it matters Attribution by stage is the prerequisite for deciding whether reliability came from knowledge, representation, enforcement or the harness, which is the question this entire map is organized around.

7. Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards

   Jin · 2026 arXiv

   Synthesis

   Plain-language abstract A human-in-the-loop audit of the gold annotations in two widely used text-to-SQL benchmarks, followed by re-evaluation of published leaderboard systems against the corrected gold. The question is not whether models are accurate but whether the answer key is.

   Motivation Execution-based benchmarks are trusted because they execute: the query either returns the gold result or it does not. That trust is conditional on the gold query being a correct translation of the question, and nothing in the pipeline checks it. Earlier work had reported an error rate for one of these subsets, which suggested the problem was known but not sized.

   Methodology A three-stage human-in-the-loop audit reviews each example, distinguishing gold queries that mistranslate the question from those that are merely one of several valid readings. The corrected gold is then used to re-score sixteen open-source agents from the BIRD leaderboard, and rank correlation between the corrected subset and the full development set is computed against the correlation before correction.

   Results Annotation errors in 263 of 498 BIRD Mini-Dev examples (52.8%) and 76 of 121 Spider 2.0-Snow examples (62.8%), against 36.1% previously reported for Mini-Dev. Re-evaluating the sixteen agents changes relative execution accuracy by minus 7% to plus 31%, moves 12 of 16 upward, and shifts ranks by up to nine places; CHESS goes from 62% to 81% and from seventh to tied first. The finding that matters most is the rank correlation: uncorrected subset against full dev gives Spearman 0.85 at p = 3.26e-5, and corrected subset gives 0.32 at p = 0.23. The subset and the full set agreed about who was best only while both were wrong in the same way.

8. Rethinking coding agent benchmarks

   Jarmak · 2026

   Synthesis

   Practitioner argument that coding-agent benchmarks measure a task distribution and a retrieval setting that do not match enterprise deployment, and that stronger context retrieval changes what the score means.

   Why it matters The same objection applies with more force to text-to-SQL, where the retrieval setting is the semantic layer and no benchmark holds it fixed.

9. SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks

   Safarzadeh · 2026 arXiv

   Synthesis

   Probes leakage with syntactic perturbations across Spider, SParC, CoSQL and BIRD and measures rank sensitivity with Kendall's tau and bootstrap intervals, finding a clear temporal gradient with Spider strongest and BIRD showing minimal sensitivity and appearing largely uncontaminated.

   Why it matters Read against Jin it produces the theme's sharpest fact: the benchmark the field trusts is comparatively clean of leakage and more than half mis-annotated, and the community worries about the smaller failure mode.

10. Benchmarking Text-to-SQL under Role-Based Access Control

    Fei · 2026 arXiv

    Synthesis

    Plain-language abstract Text-to-SQL benchmarks assume the person asking can see every table and column. Fei, Jiang, Yang, and Xiao asked what happens when the asker holds a role that can see only some of them. They built a pipeline that synthesizes plausible roles and column-and-operation permissions for Spider, BIRD, and LiveSQLBench, had four annotators validate them, and labeled each question-role pair allow or deny by parsing the gold SQL. Thirteen models were then scored on whether they refused when they should and wrote compliant SQL when they could. Models with strong unrestricted accuracy violated the policy on a large share of questions, and hiding columns, fine-tuning, and few-shot examples did not repair it.

    Motivation Production databases enforce role-based access control, so an unauthorized query fails at execution or is silently masked; a check at the database blocks it but cannot say whether the system should have refused earlier, missed a compliant alternative, or over-refuses. Existing benchmarks score execution accuracy under unrestricted access, so a query that is correct but reads a denied column counts as a success, which the authors call an RBAC-rejected success. The few prior efforts on access control used database- or table-level permissions and had no systematic construction pipeline or empirical study.

    Methodology An LLM inferred each database's application context from its schema, derived roles with responsibilities, and then assigned column-level permissions per operation; configurations were screened on denial rate (rejected below 5 or above 90 percent), coverage variance, pairwise Jaccard overlap above 0.8, and embedding similarity to the schema, then accepted by 3 of 4 annotators (28 of 53 passed first time, 16 after one regeneration, 9 were revised by hand). Each database also received 2 or 3 programmatically generated DataOperator roles with broad but incomplete access. SQLGlot extracted the (table, column, operation) triples each gold query needs, and an instance is allowed when they are a subset of the role's policy. The result is 53 databases, 399 tables, 3,353 columns, and 21,502 instances from 3,160 questions, with deny shares of 47, 65, and 76 percent on Spider, BIRD, and LiveSQLBench. Each model saw a fixed zero-shot template at temperature 0, one random role per question per pass, over 5 seeds, and was scored on violation rate, over-refusal rate, AC-F1, Safe-EX, and Safe-Deny.

    Results Violation rates exceeded over-refusal rates for every model on every dataset. On Spider GPT-5 violated 2.55 percent of instances with AC-F1 95.79 while Snowflake-Arctic-R1-7B violated 45.43 percent with AC-F1 70.14 despite higher execution accuracy; on BIRD Snowflake reached 63.77 percent and on LiveSQLBench 76.15 percent, where Claude Sonnet 4.5 was lowest at 12.40 percent. Compliance fell with query difficulty and with the number of joins, driven by violations rather than refusals. Restricting the prompt to the role's schema halved DeepSeek-V3.2-Coder's reads of known-forbidden columns but raised GPT-5-mini's violation rate from 12.49 to 26.72 percent as models wrote SQL over visible or invented columns instead of refusing. Fine-tuning on role-labeled Spider cut Llama3-SQLCoder-8B's violations from 46 to 4 percent at a 17-point Safe-EX cost, gained 7.7 and 13.0 AC-F1 on BIRD for the two tuned models, and lost 6.1 on LiveSQLBench for Qwen2.5-14B; a 298-instance row-level pilot showed DeepSeek-V3.2-Coder at 50.27 percent violations and GPT-5 at 2.15.

11. UniQL: Towards Dialect-Universal Benchmarking for Text-to-SQL

    Gao · 2026 arXiv

    Synthesis

    Plain-language abstract Text-to-SQL benchmarks almost all run on SQLite, so nobody knew whether a model that answers a question there could answer it on PostgreSQL, Oracle or Teradata. Gao and colleagues took the 1,534 BIRD development questions and built human-verified, executable gold SQL for each in 16 dialects, 24,544 queries over the same schemas and data. Thirteen models were prompted zero-shot with the target dialect, schema and question. The best, Claude-4.5-Sonnet, averaged 54.63 percent, ranged from 63.75 percent on Oracle to 37.74 on Teradata, and answered only 20.14 percent of questions correctly in all 16 dialects.

    Motivation Spider and BIRD execute on SQLite only, and Spider 2.0 spans several systems but with system-specific tasks, so none of them can separate dialect variation from differences in schema, task or environment. The authors note that a query correct in one dialect may fail or return different results in another because of execution semantics, function libraries, ordering behaviour, implicit casting and aggregation rules. They argue that SQLite-centric evaluation may therefore overstate how universal current systems are.

    Methodology The BIRD databases were migrated to 15 target systems with type mapping and identifier normalization. Each SQLite gold query was first translated by SQLGlot and accepted only if it executed and matched the source result under a conservative protocol that preserves order for ORDER BY queries and duplicate multiplicity otherwise; failures went to GPT-5-mini with schema, dialect and an evolving rule set for up to three execution-feedback reflection rounds, then to Gemini-2.5-Pro rule summarization over the failure log for three rounds, then to two human annotators who resolved disagreements by consensus. Evaluation prompted five closed models (GPT-3.5-Turbo, GPT-5-mini, GPT-5.1-codex, Gemini-2.5-Pro, Claude-4.5-Sonnet) and eight open ones (Qwen3 from 1.7B to 32B, Llama-3 8B and 70B, DeepSeek-Coder-16B, DeepSeek-v4-flash) with one shared template, no fine-tuning or few-shot selection, and scored with the same strict execution metric. A post-hoc error analysis sorted every Claude-4.5-Sonnet failure into syntax/function, schema/reference, value/filter, SQL logic or other.

    Results Average execution accuracy was 54.63 percent for Claude-4.5-Sonnet, 52.10 for Gemini-2.5-Pro and 50.50 for GPT-5-mini; Qwen3 rose from 33.01 percent at 1.7B to 47.77 at 32B. Oracle was the easiest dialect on average at 48.92 percent, Teradata and Druid the hardest at 29.03 and 29.65. Claude-4.5-Sonnet fell from 60.8 percent on simple BIRD questions to 46.7 on moderate and 40.3 on challenging ones, and from 59.1 percent on queries SQLGlot could translate to 14.6 on queries that needed newly summarized rules. Its All-16 consistency was 20.14 percent and only 33.66 percent of its SQLite-correct questions held across the other 15 dialects; Gemini-2.5-Pro's figures were 9.19 and 15.38. Syntax and function errors were 56.40 percent of its Druid failures and 57.38 percent of its Teradata failures but under 5 percent on SQLite, MySQL, Presto and Trino, where value and filter errors dominated, and on Drill SQL logic errors were the largest class.

12. SUQL: Conversational Search over Structured and Unstructured Data with Large Language Models

    Liu · 2023 arXiv

    Synthesis

    Plain-language abstract Liu and Lam asked how a conversational agent can answer questions that need both a database's columns and its free-text fields, such as an Italian restaurant with a romantic atmosphere. They extended SQL with two primitives, ANSWER and SUMMARY, that call a language model on text values, built a few-shot LLM parser that translates user turns into this language, and wrote a compiler that uses dense retrieval and predicate ordering to keep LLM calls few. On HybridQA the few-shot system came within 8.9 exact-match points of a model trained on 62K examples. On a Yelp database of 1,828 restaurants the agent satisfied all user constraints 90.3 percent of the time in conversation, against 63.4 percent for a linearized-retrieval baseline.

    Motivation Patient records, financial databases and review sites hold structured columns beside free text, and crowdsourced restaurant questions showed more than 49 percent needed both. Prior chat systems routed each question to a structured or an unstructured module by a classifier, which cannot serve questions that need both, or linearized tables into text and retrieved by embedding, which gives up the comparisons and calculations SQL does well. Earlier hybrid question-answering systems worked only on small tables, dropped the expressiveness of structured queries, or supported limited compositions of the two.

    Methodology SUQL defines ANSWER(t, q), returning the answer to question q over text t, and SUMMARY(t) as sugar for ANSWER with a summary question; both return text usable anywhere SQL accepts text, so every HybridQA question type has a SUQL form. A two-stage in-context parser first decides whether the knowledge base is needed and then predicts the query, and the agent reports what it searched and says when nothing was found. The compiler answers ANSWER filters by retrieving candidates with precomputed dense embeddings and then verifying the clause with an LLM, aggregates similarity across multiple text constraints, lists enumerated values in the schema prompt when a type has at most 10 members and otherwise overloads equality with a zero-shot CLASSIFY, orders non-LLM predicates first in disjunctive normal form, and evaluates lazily under LIMIT. Experiments used GPT-4-1106-preview on HybridQA (about 70K questions over 13,000 Wikipedia tables) with fewer than 10 examples and up to 2 retries, and gpt-3.5-turbo-0613 on 1,828 Yelp restaurants from 4 cities with 100 crowdsourced questions and 96 turns across 20 conversations.

    Results The few-shot SUQL system scored 59.3 percent exact match and 68.3 F1 on the HybridQA development set and 59.0 EM and 68.4 F1 on test, 8.9 EM and 7.1 F1 below S3HQA and more than 10 points above HPROPRO's few-shot GPT-4 program prompting at 48.7 EM; under the substring metric of Sui et al. it scored 72.5 percent against their best 56.68. Of 72 sampled errors, 37.5 percent were format mismatches and 23.6 percent wrong or incomplete gold labels, so the authors estimated true accuracy near 84.2 percent. On Yelp, turn accuracy was 93.8 percent single-turn and 90.3 percent conversational, against 57.0 and 63.4 for linearization returning 1 result and 49.7 and 61.9 for 3 results. The agent returned nothing for 21 of 100 questions and 8 of 62 conversational queries, of which 7 and 2 truly had no answer, a false negative rate of 14 and 9 percent.

13. SemPlan: Benchmarking Structured Semantic Planning for LLM-Based Queries over Enterprise Data

    Santos Teixeira · 2026 arXiv

    Synthesis

    Plain-language abstract SemPlan asks how much a language model should be allowed to write when it turns a business question into a query over enterprise data. Santos Teixeira built a 1,800-case bilingual benchmark over a synthetic commerce schema and ran 4 architectures over it with 1 model: free SQL behind guards, a typed tool agent, a structured semantic request the application compiles to SQL, and that request with conversational state. The compiled semantic request scored best on answer correctness at 25.67 percent, 3.42 points above direct SQL. Every architecture was wrong or refused on most cases, and the guards changed which kind of failure occurred more than they changed the total.

    Motivation Enterprise questions carry business semantics, governance rules and ambiguity that a plain text-to-SQL benchmark does not exercise, and existing evaluations rarely hold the model fixed while varying the architecture. Santos Teixeira wanted a fixed model, fixed prompts and fixed cases so that the only variable was how much of the query the model wrote. The benchmark also had to record refusals, clarifications and policy outcomes as first-class results rather than folding them into accuracy.

    Methodology The Northstar Commerce domain covers customers, products, orders, payments, expenses, budgets, suppliers, contracts and a calendar dimension behind governed read-only PostgreSQL views. Cases came from parameterized semantic templates over intents, metrics, dimensions, filters, temporal context and ambiguity, with gold plans executed deterministically; the 1,800 cases split into dev, validation, public test, hidden test, multi-turn and adversarial sets, of which 1,200 formed the frozen scientific subset. Architectures A1 to A4 all used gpt-5.6-luna at low reasoning effort with a 1,200-token output ceiling: A1 emitted SQL that had to pass parsing, AST allowlists, complexity limits and timeouts; A2 called typed analytics tools; A3 emitted a strict envelope of operation, metrics, dimensions, filters, time grain, sort and limit that deterministic code compiled to parameterized SQL; A4 added prior state and typed clarification outcomes. The run produced 4,800 primary records plus 1,200 stability records with no missing rows, analysed with McNemar tests and Holm-Bonferroni corrected paired risk differences.

    Results Answer correctness was 22.25 percent for A1, 22.58 for A2, 25.67 for A3 and 24.25 for A4; the A3 gains over A1 and A2 were 3.42 and 3.08 points with Holm p = 0.000110, and A4 trailed A3 by 1.42 points (p = 0.041). Direct SQL had the highest policy correctness at 43.67 percent but falsely refused 21.42 percent of cases; the other 3 refused under 2 percent and instead produced unsafe or invalid output on 60 to 64 percent. The compiled envelope cost USD 0.000512 per case against 0.000918 for direct SQL and repeated its outcomes on 98.67 percent of a 150-case subset run 3 times. Portuguese cases scored 4.8 to 11.8 points below English, and multi-turn state correctness peaked at 20 percent.

What should generalize?

Cross-database generalization and cross-question generalization are different claims, and the field's headline numbers routinely conflate them.

Spider made the first one the goal: test on databases never seen in training. That framing is why the benchmark mattered, and it is also why the field spent years building parsers that exploited the lexical overlap between a question's words and the target schema's column names. Spider-Syn broke that by substituting synonyms for schema words in the questions, and accuracy fell. Pi and colleagues attacked from the other side, perturbing the table rather than the question. Dr.Spider generalized both into 17 perturbation test sets spanning question, database and SQL.

What that body of work establishes is that a system can generalize across databases while failing to generalize across ways of asking, because the two are held up by different mechanisms. Schema-word overlap gives cross-database transfer cheaply and gives nothing at all when the user's vocabulary differs from the schema's. This is the same finding as the first theme in this map, arrived at from the robustness direction: a system relying on lexical overlap has quietly assumed the schema is the business.

Shaw and colleagues add the third axis. Compositional generalization, the ability to handle novel combinations of familiar structures, and natural language variation are not the same requirement, and an approach that handles one may not handle the other. Saparina and Lapata take the constructive route, increasing natural language variation in training and measuring the resulting generalization gain.

For a governed analytics system the distinction is not academic. If a semantic model is authored for one warehouse and the system is evaluated on questions over that same warehouse, nothing has been shown about transfer to another company's data. Conversely, a system evaluated across many databases with the semantic layer authored per database has shown transfer of the method and nothing about the cost of authoring. Almost every evaluation in this literature, including the strongest ones, tests unseen questions rather than unseen databases, and states so only in a limitations section if at all.

UniQL adds a fourth axis the map had been folding into the first. Cross-database generalization, tested on unseen schemas, is not cross-dialect generalization, tested on the same schema in another SQL. On the same 1,534 questions and the same data, the best model was correct in all 16 dialects on 20.14 percent of questions, and of the questions it got right in SQLite only 33.66 percent survived the other 15. The authors read the ranking as familiarity rather than syntactic distance: Oracle was hard for their construction pipeline and easy for the models. For a semantic layer that compiles to dialect SQL, this is the axis the compiler is supposed to absorb, and the number is the size of what it absorbs.

1. Improving Text-to-SQL Evaluation Methodology

   Finegan-Dollak · 2018 arXiv

   Synthesis

   Separates memorizing a query template from composing a new one, which is a distinct axis from transferring to a new database.

   Why it matters Cross-database transfer and cross-template composition are routinely reported as one number, and this is the paper that showed they are not.

2. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

   Yu · 2018 arXiv

   Synthesis

   Defines generalization as transfer to unseen databases, which became the field's default meaning of the word.

   Why it matters Cross-database transfer and cross-phrasing transfer are different properties, and Spider's framing made the first one the one everybody optimized.

3. Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?

   Shaw · 2020 arXiv

   Synthesis

   Shows that handling novel combinations of familiar structures and handling varied phrasings are separate requirements, and that approaches strong on one can be weak on the other.

   Why it matters It is the paper that makes generalization a multi-axis question, which is what any claim about a semantic model transferring to a new domain has to be specific about.

4. Towards Robustness of Text-to-SQL Models against Synonym Substitution

   Gan · 2021 arXiv

   Synthesis

   Substitutes synonyms for schema-related words in the questions and observes a sharp accuracy drop, showing that parsers were relying on lexical overlap between question and column names.

   Why it matters It is the robustness-side statement of this map's first theme: a system leaning on lexical overlap has quietly assumed the schema is the business vocabulary.

5. Towards Generalizable and Robust Text-to-SQL Parsing

   Gao · 2022 arXiv

   Synthesis

   Treats generalization and robustness as explicit design targets in the parser rather than as properties measured after the fact.

   Why it matters It is where the two-axis distinction has to be made concrete, because a design cannot target both without saying which one it is trading away.

6. Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation

   Pi · 2022 arXiv

   Synthesis

   Perturbs the table rather than the question, using natural and realistic renamings, and finds the same fragility from the other direction.

   Why it matters Together with Spider-Syn it locates the fragility in the binding between language and schema rather than in either side alone.

7. Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness

   Chang · 2023 arXiv

   Synthesis

   Plain-language abstract Dr.Spider is a diagnostic benchmark that perturbs Spider along three axes, the question, the database and the SQL, producing seventeen perturbation test sets so that a robustness failure can be attributed to the specific kind of variation that caused it rather than reported as an aggregate drop.

   Motivation Robustness results in text-to-SQL were being reported as a single number on a single perturbed set, which tells a system builder that something is fragile but not what. Different perturbations stress different components: paraphrase stresses the language side, schema renaming stresses linking, SQL variation stresses the decoder.

   Methodology Seventeen test sets are constructed by applying controlled perturbations of each of the three kinds to Spider, keeping the underlying semantics fixed so that a correct system should be unaffected. Systems are evaluated per perturbation set, which turns robustness into a per-cause diagnosis.

   Results The suite is the closest thing this field has to a component-level diagnostic, and the per-perturbation breakdown is what makes it useful rather than the headline drop. It also sets up an absence: the semantic layer is now a load-bearing component in these systems, and there is no equivalent suite that perturbs it, no test for what happens when a metric is renamed, an entity split, or a measure left undefined.

8. Improving Generalization in Semantic Parsing by Increasing Natural Language Variation

   Saparina · 2024 arXiv

   Synthesis

   Increases the natural-language variation in training data as a direct intervention and measures the resulting generalization improvement.

   Why it matters The constructive counterpart to the perturbation suites: having established which axis is weak, this is what it costs to strengthen it.

9. UniQL: Towards Dialect-Universal Benchmarking for Text-to-SQL

   Gao · 2026 arXiv

   Synthesis

   Plain-language abstract Text-to-SQL benchmarks almost all run on SQLite, so nobody knew whether a model that answers a question there could answer it on PostgreSQL, Oracle or Teradata. Gao and colleagues took the 1,534 BIRD development questions and built human-verified, executable gold SQL for each in 16 dialects, 24,544 queries over the same schemas and data. Thirteen models were prompted zero-shot with the target dialect, schema and question. The best, Claude-4.5-Sonnet, averaged 54.63 percent, ranged from 63.75 percent on Oracle to 37.74 on Teradata, and answered only 20.14 percent of questions correctly in all 16 dialects.

   Motivation Spider and BIRD execute on SQLite only, and Spider 2.0 spans several systems but with system-specific tasks, so none of them can separate dialect variation from differences in schema, task or environment. The authors note that a query correct in one dialect may fail or return different results in another because of execution semantics, function libraries, ordering behaviour, implicit casting and aggregation rules. They argue that SQLite-centric evaluation may therefore overstate how universal current systems are.

   Methodology The BIRD databases were migrated to 15 target systems with type mapping and identifier normalization. Each SQLite gold query was first translated by SQLGlot and accepted only if it executed and matched the source result under a conservative protocol that preserves order for ORDER BY queries and duplicate multiplicity otherwise; failures went to GPT-5-mini with schema, dialect and an evolving rule set for up to three execution-feedback reflection rounds, then to Gemini-2.5-Pro rule summarization over the failure log for three rounds, then to two human annotators who resolved disagreements by consensus. Evaluation prompted five closed models (GPT-3.5-Turbo, GPT-5-mini, GPT-5.1-codex, Gemini-2.5-Pro, Claude-4.5-Sonnet) and eight open ones (Qwen3 from 1.7B to 32B, Llama-3 8B and 70B, DeepSeek-Coder-16B, DeepSeek-v4-flash) with one shared template, no fine-tuning or few-shot selection, and scored with the same strict execution metric. A post-hoc error analysis sorted every Claude-4.5-Sonnet failure into syntax/function, schema/reference, value/filter, SQL logic or other.

   Results Average execution accuracy was 54.63 percent for Claude-4.5-Sonnet, 52.10 for Gemini-2.5-Pro and 50.50 for GPT-5-mini; Qwen3 rose from 33.01 percent at 1.7B to 47.77 at 32B. Oracle was the easiest dialect on average at 48.92 percent, Teradata and Druid the hardest at 29.03 and 29.65. Claude-4.5-Sonnet fell from 60.8 percent on simple BIRD questions to 46.7 on moderate and 40.3 on challenging ones, and from 59.1 percent on queries SQLGlot could translate to 14.6 on queries that needed newly summarized rules. Its All-16 consistency was 20.14 percent and only 33.66 percent of its SQLite-correct questions held across the other 15 dialects; Gemini-2.5-Pro's figures were 9.19 and 15.38. Syntax and function errors were 56.40 percent of its Druid failures and 57.38 percent of its Teradata failures but under 5 percent on SQLite, MySQL, Presto and Trino, where value and filter errors dominated, and on Drill SQL logic errors were the largest class.

Improving the system without gaming the benchmark

The systems in this map get tuned rather than trained once and shipped. Prompts are rewritten, retrieval adjusted, semantic models extended, scaffolds rearranged, each change accepted or rejected on a development score. That loop is adaptive data analysis, and its statistics were settled in 2015.

Dwork, Feldman, Hardt, Pitassi, Reingold and Roth showed in Science that a holdout set reused across adaptively chosen analyses stops being a valid estimate of generalization, and that differentially private access to it buys back a quantified number of safe queries. Nearly every agent-optimization loop in this literature is an instance of the problem that paper solved, and nearly none of them cite it. The practical version is a metered holdout with a preregistered query budget, which is cheap to implement and almost never implemented.

The optimizers themselves have gotten good, which raises the stakes rather than lowering them. MIPRO jointly optimizes instructions and demonstrations across the modules of a language-model program and beats baselines on five of seven programs by as much as 13 points. GEPA replaces gradient-style search with reflective prompt evolution over a Pareto frontier of attempts and outperforms GRPO by 6 percent on average and up to 20 percent while using up to 35 times fewer rollouts, and beats MIPROv2 by more than 10 percent. A method that reaches a good configuration in few rollouts also reaches an overfitted one in few rollouts, and neither paper's evaluation is designed to tell those apart.

Reinforcement learning against execution reward has the same shape at a finer grain. Progressive rewards make the signal denser for text-to-SQL, which improves optimization and increases the number of effective queries against the same fixed evaluation data.

The defenses are known and mostly procedural: hold a sealed set that is scored once, preregister the primary endpoint before any results exist, meter the development holdout with a hard checkpoint budget, keep an append-only ledger of every accepted and rejected change, and forbid changes conditioned on specific evaluation instances. None of that is novel statistics. What is missing is any published demonstration that a metered holdout actually preserves its guarantee under a realistic agent-optimization loop, as opposed to being accepted on theory.

The feedback loop vendors now describe, learnings from user conversations captured and fed back so the layer evolves with the business, has one measured ancestor and several modern variants. Iyer and colleagues ran it in 2017: a parser deployed for three days, users marking answers correct or wrong, a hired annotator supplying gold for the wrong ones, retraining between stages, and judged-correct answers rising from 25 to 53.7 to 63.5 percent. The paper also measured what the loop ingests. Users marked 6.1 percent of correct queries wrong and 6.3 percent of wrong ones correct. Vyas's DIA stores what the agent learned from execution rather than what the user said, promotes a rule across sessions only when other databases bear it out, and re-probes the precondition before any stored rule changes an answer; the memory's effect is described and not measured. Tummalapenta measures memory directly and finds no layer positive for every model and dataset. Episodic retrieval across sessions moved accuracy between plus 4.5 and minus 12.6 points, and a short window over the current conversation carried nearly all of the gain. AnnoIndex learns from the query log without touching accuracy, writing extracted values back and cutting model calls 73 to 79 percent over five batches. The practitioner thread that names context drift as the part that breaks proposes the same loops, query logs as few-shot memory and benchmark-driven context refresh, and reports no before-and-after.

Every one of these is also an adaptive analysis, and the reusable-holdout problem applies to a feedback loop as much as to a prompt optimizer. A loop that writes conversational corrections into the semantic layer is tuning the layer on the questions people happened to ask.

1. The reusable holdout: Preserving validity in adaptive data analysis

   Dwork · 2015

   Synthesis

   Plain-language abstract A holdout set reused across a sequence of adaptively chosen analyses stops being a valid estimate of generalization, because each decision made after looking at the holdout leaks information from it. This paper quantifies the problem and shows that accessing the holdout through a differentially private mechanism restores validity for a bounded number of queries.

   Motivation Standard statistical guarantees assume the analysis is fixed before the data is seen. Actual practice is adaptive: a researcher looks at a result, changes the approach, looks again. The number of effective hypotheses tested is unbounded and untracked, and the holdout silently degrades into a training set.

   Methodology The setting is formalized as a sequence of adaptively chosen queries against a holdout. The authors show how validity fails, then construct a mechanism that answers queries about the holdout with noise calibrated for differential privacy, and prove that this preserves generalization guarantees for a number of queries that grows with the holdout size.

   Results The result is the theoretical foundation for any evaluation protocol with a metered holdout and a checkpoint budget. It is also the least cited relevant paper in this corpus: every development-set-driven agent optimization loop, where prompts are rewritten and scaffolds rearranged and each change accepted on a development score, is an instance of exactly this problem, and nearly none of that work cites it or bounds its adaptivity.

2. Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs

   Opsahl-Ong · 2024 arXiv

   Synthesis

   Jointly optimizes instructions and few-shot demonstrations across the stages of a multi-stage language-model program, beating baseline optimizers on five of seven programs with Llama-3-8B by as much as 13 points.

   Why it matters It establishes the multi-stage program as the unit of optimization, which is the same unit the harness theme argues is the unit of capability, and neither theme's evaluations account for the other.

3. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning

   Agrawal · 2025 arXiv

   Synthesis

   Plain-language abstract GEPA optimizes a compound language-model system by reflecting on its own execution traces in natural language, proposing prompt edits, and maintaining a Pareto frontier of candidate configurations rather than a single best-so-far. It is compared against reinforcement learning and against a prior prompt optimizer.

   Motivation Reinforcement learning on a compound system needs many rollouts, each of which is a full expensive execution, and it compresses the outcome of a rich trace into a scalar reward. The authors' argument is that the trace itself carries diagnostic information in language, and that a model can read it and propose a targeted change more efficiently than a gradient can find one.

   Methodology Execution traces from candidate configurations are fed back to a model that reflects on what went wrong and proposes prompt mutations. Candidates are kept on a Pareto frontier over tasks rather than collapsed to a single score, which preserves configurations that are good at different things. Comparison is against GRPO and against MIPROv2 across several tasks, with rollout counts reported.

   Results Outperforms GRPO by 6% on average and up to 20% with as much as 35 times fewer rollouts, and beats MIPROv2 by more than 10%. Accepted at ICLR 2026 as an oral. The caution belongs with the result: sample efficiency is symmetric, and an optimizer that reaches a good configuration in few rollouts reaches an overfitted one just as fast. The paper reports the optimization gain and not a held-out generalization gap alongside it, which is precisely the measurement the reusable-holdout literature says is needed.

4. Progress-SQL: Improving Reinforcement Learning for Text-to-SQL via Progressive Rewards

   Zhang · 2026 arXiv

   Synthesis

   Shapes a denser progressive reward for text-to-SQL reinforcement learning instead of relying on sparse execution correctness.

   Why it matters Denser reward means more effective queries against the same fixed evaluation data, which improves optimization and increases adaptive-overfitting exposure at the same time.

5. The Nuts and Bolts of Natural Language to SQL Translation: A Systematic Analysis of Model Pipeline Optimisations

   Klubicka · 2026 arXiv

   Synthesis

   A change validated in isolation can be a regression in combination, which is a route to development-set overfitting that stage-by-stage evaluation cannot detect.

   Why it matters Any improvement loop that accepts changes on individual ablations is accumulating interactions it never measured.

6. What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema

   Naser Moghadasi · 2026 arXiv

   Synthesis

   Provides a concrete scoring schema for evaluation-condition disclosure rather than a general call for more rigour.

   Why it matters A schema is enforceable at review time, which is the only mechanism in this theme that does not depend on authors voluntarily reporting how many development evaluations they consumed.

7. Structure then Query: Enabling Precise Analytical Queries over Unstructured Documents

   Lin · 2026 arXiv

   Synthesis

   Plain-language abstract Lin, Luo and Tang asked how to run precise analytical queries, with exact filters, joins and multi-step conditions, over collections of unstructured documents without paying for a language model on every document at every query. AnnoIndex induces a hierarchical schema from the corpus, extracts the attribute values once into a structured index, and compiles each question into a SQL-like plan that filters on the index first and calls extraction models in rising order of cost only on the survivors. On three corpora (legal decisions, Wikipedia pages, web pages) with 500 queries each it averaged 0.87 F1 against 0.80 for the best prior system and 0.73 for feeding whole documents to GPT-4o, while using 11 times fewer tokens than the direct LLM baseline. Values extracted at query time are written back, so repeated workloads get cheaper.

   Motivation Vector retrieval finds documents roughly about a topic but cannot enforce a condition such as birth year before 1985, so retrieve-then-extract pipelines hand the LLM noisy context, lose documents dropped early, and pay again on every query. Extract-everything approaches such as GraphRAG spend an LLM call on every document up front, and a knowledge graph is a relation explorer rather than an analytical engine: it cannot aggregate or join without an LLM reasoning over subgraphs, which stays error-prone even when the needed facts are present. Existing text-to-table systems (ZenDB, QUEST, Palimpzest, Lotus) still extract at query time and use fixed schemas that do not adapt across heterogeneous document clusters.

   Methodology SchemaLoop keeps a human-defined dataset-level schema, then has GPT-4o propose table-level and document-level schemas once per document group, deduplicates candidates by synonym and embedding similarity, extracts values with Mistral-7B on 5 sampled documents, and refines until extraction success rate stays above 0.6 and filtering efficiency above 0.3, for at most 5 rounds. The query engine parses a question layer by layer into schema-bound predicates plus residual EXTRACT predicates, evaluates regex, then small model, then LLM under a per-query budget, rewrites joins as IN filters, short-circuits on failure, and persists extracted values as virtual fields promoted to the schema after more than 10 references. Datasets were LCR (1,600 legal documents of over 6,000 tokens each), WikiText (219 Wikipedia pages across 10 domains) and SWDE (1,050 web pages), each with 500 queries split 20 percent simple filters, 30 percent conjunctive or disjunctive, 30 percent joins and 20 percent progressive reasoning, with ground truth extracted by GPT-4o and corrected by 8 graduate students. A tuple counted as correct only if all attribute values matched exactly; baselines were vector RAG, Graph RAG, ZenDB, Palimpzest, QUEST, Lotus, ClosedIE and direct GPT-4o.

   Results Average F1 was 0.87 in performance mode and 0.83 in economical mode, against 0.80 for QUEST, 0.73 for Lotus and for direct GPT-4o, 0.70 for Palimpzest, 0.67 for ZenDB, 0.60 for Graph RAG and 0.46 for vector RAG; on the legal corpus AnnoIndex reached 0.81 where Lotus scored 0.46, and on templated web pages every strong system exceeded 0.94. Amortized cost including offline indexing was 18.3K tokens per query in performance mode and 15.7K in economical mode, the offline build equalling about 1.5 online queries. Removing SchemaLoop in favour of a manual schema cut WikiText F1 from 0.87 to 0.72 and raised cost 72 percent; replacing the structured engine with vector retrieval over the same index cut F1 to 0.64. On three-way join and reasoning queries AnnoIndex scored 0.86 F1 against 0.74 for QUEST and 0.35 for the LLM baseline, and on legal conflict queries the plan narrowed 1,600 documents to 120 by index and to 35 by a small model before any LLM call, reaching 0.79 F1 where QUEST scored 0.61 under the same LLM budget.

8. Learning a Neural Semantic Parser from User Feedback

   Iyer · 2017 arXiv

   Synthesis

   Plain-language abstract The question is whether a natural-language interface to a database can be built for a new domain with almost no hand-written training data and then improve as people use it. Iyer and colleagues trained a neural sequence model to write SQL directly, seeded it with template-generated examples, put it online for an academic-paper database, and let users mark answers right or wrong; wrong ones were annotated by a hired SQL programmer and folded into retraining. Over three deployment rounds the share of user questions answered correctly rose from 25 percent to 63.5 percent. On the older Geo880 and ATIS benchmarks the same model reached 82.5 and 79.24 percent, close to systems that use specialised meaning representations.

   Motivation Earlier database interfaces either parsed to intermediate logical forms that lack SQL's expressivity and are understood by few annotators, or relied on hand-engineered rules and features that do not carry to a new domain. Crowdsourced paraphrasing of generated questions was one route to cheaper data, but a prior calendar-domain experiment found 48 percent of real user questions could not be generated that way. Iyer and colleagues wanted a loop in which real users supply the questions, a cheap binary judgment selects what needs labelling, and SQL-literate crowd workers supply the labels.

   Methodology The parser is an attention-based encoder-decoder with a bidirectional LSTM encoder, entity anonymization by type, and beam search of width 5. Initial data came from 22 schema-agnostic language-to-SQL templates instantiated over foreign-key join paths, expanded with PPDB paraphrases. The online experiment ran on an academic database for three stages of 10 new computer-science graduate students each, each issuing at least 10 questions and choosing among five feedback options; questions marked wrong went to one Upwork worker with SQL experience and database access. Simulated interactive runs on Geo880 and ATIS split the training data into batches and assumed perfect feedback. Benchmark accuracy was measured by executing predicted SQL and comparing results, on 600/280 and 4,473/497/448 splits.

   Results User-judged accuracy went from 25 percent in stage 1 (templates only) to 53.7 percent and 63.5 percent across the three stages. In the final stage 22.3 percent of generated queries failed to execute, 6.1 percent of correct queries were marked wrong, and 6.3 percent of incorrect queries were marked correct, mostly from users choosing Wrong instead of Incomplete on aggregation queries over incomplete data. On Geo880 the SQL model scored 82.5 percent and on ATIS 79.24 percent; paraphrase augmentation helped consistently while template augmentation helped only in the early interactive stages. In simulation, batches of 50 required annotation for 54.3 percent of Geo880 examples versus 70.2 percent for batches of 150. The released SCHOLAR dataset has 816 labelled utterances, on which the fully supervised parser reached 67 percent and 15 percent of predictions did not execute.

9. Data Intelligence Agents: Interpreting, Modeling, and Querying Enterprise Data via Autonomous Coding Agents

   Vyas · 2026 arXiv

   Synthesis

   Plain-language abstract Enterprise data work passes through lossy handoffs between the people who know what fields mean, the engineers who structure data, and the analysts who query it. Vyas and colleagues at C3 AI built DIA, in which a single sandboxed coding agent plays three roles (interpret raw files, build and validate a schema, write and execute SQL), keeps its outputs as reviewable artifacts, and records rules it learned from execution in a shared memory. The query agent, run with Claude Sonnet 4.5 and no fine-tuning, matched or beat the best published result on seven SQL benchmarks across four dialects, by 33 points on the conversational BIRD-Interact. Almost every remaining failure runs cleanly and returns the wrong answer, most often on questions that name a metric without defining it.

   Motivation Pipeline text-to-SQL systems chain modules tuned for one subtask; reinforcement-learned specialists lock to one benchmark and dialect; agentic explorers probe the database but forget everything between sessions; memory-equipped SQL agents keep one store and are evaluated narrowly. All emit text rather than executable, inspectable artifacts, and none handle the upstream understanding and schema-building work that decides whether SQL has anything sensible to run against. The authors frame the coding agent, not the language model, as the unit of abstraction.

   Methodology DIA runs one autonomous coding agent (OpenHands with Claude Sonnet 4.5) over a shared workspace as three agents: a Data Interpreter that profiles sources into schemas, value distributions, key candidates, and join paths; a Schema Creator that loads staging tables with provenance and validates row counts, column coverage, key integrity, and load rejects; and a Query Generator that declares an expected result shape, probes the schema by execution, generates SQL, and checks the result against its own declaration. Memory has three tiers: retrieved examples from the BIRD train split, session lessons confirmed on the current database, and cross-session rules promoted only when later questions bear them out, each re-verified by a live probe before use. Evaluation covered 4,187 instances on BIRD-Dev, BIRD-Critic, LiveSQLBench, BIRD-Interact, Spider2-Lite, Spider2-Snow, and Spider2-DBT across SQLite, PostgreSQL, Snowflake, and DuckDB, with o3 as the user simulator and per-benchmark adaptation limited to a standing instruction file.

   Results Scores against best prior: BIRD-Dev 77.7 versus 77.8, BIRD-Critic 64.2 versus 48.8, LiveSQLBench 50.7 versus 38.0, BIRD-Interact 55.7 versus 22.7, Spider2-Lite 71.3 versus 55.2, Spider2-Snow 69.5 versus 63.8, Spider2-DBT 37.5 versus 35.3. Modification tasks scored above pure queries (BIRD-Critic management 78.7 versus query 64.4) because a named target object can be validated by execution, while high-level questions naming a composite metric cost 15 to 17 points. Reasoning failures were the largest class on every benchmark and execution failures under 3 percent. In the BIRD-Interact development run the dominant failure was re-submitting near-identical SQL until the 15-turn cap without asking; a forced-clarification rule left none of the released run's 215 phase-1 failures with zero questions asked. Memory's effect on accuracy was examined only qualitatively; time per question ran from under a minute to roughly ten minutes and no cost breakdown is given.

10. Memory Architectures for Multi-Turn Text-to-SQL: A Benchmark and Empirical Study

    Tummalapenta · 2026 arXiv

    Synthesis

    Plain-language abstract Analysts ask follow-up questions that refer to earlier turns, yet text-to-SQL is mostly graded one question at a time. Tummalapenta and Addanki built EnterpriseMem-Bench, 300 templated sessions over banking, SEC filings, and sales data with each turn flagged for whether it needs prior context, and ran five frontier models under five memory setups. Without any memory every model fell to 0 percent on context-dependent turns by turn 3; a window of the last two turns recovered most of the loss. Adding retrieval of past sessions or written hints helped some model-and-dataset pairs and hurt others, and one model generation regressed sharply on financial filings.

    Motivation Spider and BIRD are single-turn, and accuracy above 70 percent on BIRD has fed a perception that text-to-SQL is solved for production. SParC, CoSQL, MMSQL, and BIRD-INTERACT are multi-turn but none treats memory architecture as a variable or labels which turns need prior context. The authors also note that GPT-5 models reason by default at the API while Claude models do not, so cross-provider comparisons without matched reasoning can misattribute a configuration effect to a capability gap.

    Methodology A 1,451-line Python script generates 300 sessions (1,400 turns) from live profiles of BIRD Financial (8 tables), SEC EDGAR annual data for 20 companies (1,897 records, 2009 to 2026), and Northwind (93 customers, 16,282 orders), with gold SQL as parameterized templates and no language model in construction. Tiers are single-turn (90 sessions), 3 to 5 turns (120), and 6 to 10 turns (90); 1,100 turns are memory-critical. Five conditions: stateless; working memory of 2 turns; of 5 turns; 5 turns plus top-3 episodic retrieval from a ChromaDB store shared across sessions; and that plus one semantic hint from six hand-written notes. Models were GPT-5 mini, GPT-5.2, Claude Sonnet 4.5, Sonnet 4.6, and Opus 4.6, with extended thinking enabled on Claude; 35,000 single-run evaluations, scored by execution accuracy and a six-way error taxonomy, with a fallback SQL per turn to detect cross-session contamination.

    Results Stateless aggregate accuracy was 15 to 19 percent and exactly 0 percent on memory-critical turns by turn 3 for every model, against 67.0 to 73.0 percent on first turns; a two-turn window lifted aggregate accuracy to 74.5 to 86.4 percent. Growing the window to 5 turns was small and usually negative; episodic retrieval ranged from +4.5 to -12.6 points, the semantic hint from +14.1 to -10.8, and the full stack net of the two-turn window from +12.6 to -15.9. Sonnet 4.6 trailed Sonnet 4.5 on SEC EDGAR by 17.2, 29.0, and 33.1 points across conditions while Opus 4.6 reached 99.2 percent, and all 250 failed Sonnet 4.6 EDGAR turns matched neither gold nor the contamination fallback. Enabling reasoning added 5.7 to 18.9 points for the Claude models and left wrong-result errors as the only remaining class; without reasoning, preamble errors had run at roughly 26, 38, and 45 percent for Sonnet 4.5, Sonnet 4.6, and Opus 4.6 in the stateless condition.

11. I’m starting to think Text-to-SQL is the easy part of the problem, and context drift is the part that actually breaks things.

    Working-Chemical-337 · 2026

    Synthesis

    The original poster (u/Working-Chemical-337, June 2026) reported that agents connected directly to a warehouse wrote syntactically fine SQL but disagreed on definitions: one agent took revenue to include pending invoices, another only realized cash, and the slow part of every session was re-explaining the business logic. The fixes proposed in the thread were the poster's plan to host definitions in Cube Core and expose only Certified Queries through an MCP server or REST API, so that context engineering happens in the modeling layer; u/funbike's suggestions to put COMMENT text on tables and columns into the schema dump and to keep a vector store of past correct queries, prefixing the 5 closest matches to each new prompt at a cost in tokens; and a Databricks-oriented reply recommending regular Genie benchmarks, pipelines that pull business semantics into the Genie space from benchmark feedback, and SQL snippets over text instructions. The thread had 5 upvotes and 10 comments, and no one reported a measured before-and-after.

    Why it matters The loop ideas here (query logs as few-shot memory, benchmark-driven context refresh) are the practitioner form of the improvement branch, and the poster's framing puts the loop's target on definitions rather than SQL syntax.

Original evidence: the Omni × LiveSQLBench benchmark

This section is our own work and is marked separately because it is at a different stage than everything above. As of 2026-09-01 the sealed evaluation is complete and frozen: 89 held-out questions over 16 databases, four conditions, three repetitions, 1,068 immutable attempts, both preregistered scorers reported together. The design, the original measurements of the benchmark artifact, and the frozen result are all reported here; a later development-only arm is marked as such.

The design decomposes the thing this map keeps treating as one variable. Four conditions run over the same sealed question set: raw schema with the model writing SQL directly; schema plus a searchable copy of the raw hierarchical knowledge base; a searchable export of the Omni semantic model; and that same model queried through the production governed harness. The first two steps isolate knowledge, the second isolates representation holding knowledge fixed, and the third isolates enforcement holding representation fixed. No published study separates those three, which is why the third theme of this map and the fourth are argued from different papers with different schemas and different models.

The population is the public LiveSQLBench Large-v1 release at a pinned dataset revision: 480 records, 332 of them Query tasks and 148 Management tasks, over 18 PostgreSQL databases. A deterministic split holds out 101 questions as a sealed set scored once, leaves 231 for development, and further divides development into 154 and 77 so the second half functions as a metered holdout with a hard cap of ten checkpoints. All 18 databases appear on both sides of the split, which means the design tests unseen questions and not unseen databases, and that is preregistered as a limitation rather than discovered afterward. The primary endpoint is the mean one-shot execution accuracy of the governed condition over three sealed repetitions, with the paired difference against the raw-schema condition as the primary comparison. There is no majority vote and no best-of-three.

The headline result is that business knowledge helped a great deal and almost none of that value survived compilation into an executable model. Under the official scorer, the raw-schema condition reaches 10.1 percent, adding the searchable knowledge base takes it to 22.1 percent, exporting that knowledge as a compiled semantic model drops it to 8.6 percent, and answering through the governed product gives 8.6 percent. The knowledge contrast is plus 12.0 points with a 95 percent interval of 5.6 to 18.7, and it survives the corrected-gold sensitivity scorer at plus 9.4 points. The compilation step gives back 13.5 points, and it does so before the governed runtime is involved at all. The proximate cause is coverage: 193 of the 1,090 knowledge definitions compiled, 17.7 percent, with 511 deferred because they crossed a grain the knowledge base never states, and a compiler that refuses to guess a join or an aggregation is doing the thing a semantic layer exists to do.

The design's central assumption was refuted by its own telemetry, which is the finding most worth carrying back to the fourth theme of this map. The governed condition was supposed to isolate semantic composition from agent-written SQL, and it did not: across six governed arms, 661 of 661 parseable attempts returned agent-authored SQL. That SQL does reference the deployed model, through field templating on 660 of 661 and taking the model's join scope on most attempts, but the metric logic is written by the agent every time, because the compiled topics define no measures. The governed rows therefore measure a governed vocabulary rather than a query compiler, and they are reported with that disclosure rather than repaired. A follow-up development arm, registered after the sealed aggregates were opened and confined to development data for that reason, redeployed the same product toward its own documentation, every table published and the full foreign-key join graph. Governed accuracy doubled on the identical frame, 13.2 against 6.6 percent at about two thirds the median token cost, and topic scoping rose from 69.6 to 98.5 percent, while the hand-written-aggregate rate did not move. View sparsity was not what stopped the governed path from composing metrics; the absent measures were.

The measurements that do exist are of the artifact, and several of them bear on this map's other themes. Across the 18 databases the hierarchical knowledge base holds 1,090 entries, split into 430 calculation entries, 462 domain entries and 198 value illustrations, connected by 945 dependency edges. 560 entries carry at least one dependency, 344 edges point at a target that itself has dependencies, the longest chain is six edges, and multi-hop structure is present in all 18 databases, with no duplicate identifiers, dangling references, self-edges or cycles. The schemas hold 971 tables and 17,749 columns. That is the shape of the knowledge object the second theme of this map argues about, measured rather than assumed.

Two findings are about the instrument rather than the systems. Reading the public evaluator at its pinned commit shows that the Query scoring path strips standalone DISTINCT while retaining DISTINCT ON regardless of what the task's own conditions declare, replaces ROUND with its first argument including in nested calls, normalizes datetimes to dates and discards time of day, recursively rounds floats to two places without consulting the task's declared precision, fails a comparison when both result sets are empty, and compares as sets when order is not required, which discards duplicate multiplicity. Separately, the public agent's knowledge service omits the type and children_knowledge fields from what it shows the agent, which erases exactly the dependency structure the benchmark was built to test. Both are reported here as measurements of a public artifact, and both are reasons the seventh theme of this map is not a digression.

1. LiveSQLBench: contamination-free text-to-SQL over industrial-scale databases with a hierarchical knowledge base

   BIRD team · 2025

   Synthesis

   Plain-language abstract LiveSQLBench is a text-to-SQL benchmark from the BIRD team that pairs industrial-scale PostgreSQL databases with a curated hierarchical knowledge base per database, and refreshes its task set over time to limit contamination. The knowledge base is a directed graph of business definitions in which definitions depend on other definitions, so answering a question can require composing a chain of them.

   Motivation Benchmarks that append business context to a prompt as prose treat meaning as a retrieval problem: find the right paragraph, paste it in. The observation behind LiveSQLBench is that real business definitions are compositional. Net revenue is defined in terms of gross revenue and returns, each of which is defined in terms of something else, and a system that retrieves one definition without its dependencies has retrieved a fragment.

   Methodology Each database ships with a knowledge base whose entries are typed and carry explicit dependency links to the entries they build on. Tasks are split into query and management categories, evaluated by executing generated SQL against the live database and comparing results under a normalization procedure. The task set is periodically refreshed so that models trained after a release are not evaluated on data they may have seen.

   Results Public leaderboard results as of 2025-05-28 put o3-mini at 47.78% and Claude Sonnet 4 at 42.59%. Independent measurement of the public Large-v1 release for this explorer finds 480 records over 18 databases, 971 tables and 17,749 columns, and a knowledge base of 1,090 entries (430 calculation, 462 domain, 198 value illustration) connected by 945 dependency edges, with 560 entries carrying at least one dependency, a longest chain of six edges, multi-hop structure present in all 18 databases, and no duplicate identifiers, dangling references, self-edges or cycles. That structure is what makes this the only public benchmark treating the business-knowledge layer as a first-class versioned artifact.

2. Omni modeling layer documentation

   Omni · 2026

   Synthesis

   The governed modeling layer used as the enforced condition, described here from its public documentation rather than from internal material.

   Why it matters The comparison is only interpretable against a public definition of what the modeling layer enforces.