The Data Problem Holding Back AI That Talks to Databases

The dream of simply asking a computer a question in plain English and receiving the right answer pulled from a database has driven decades of research in artificial intelligence. Natural language to SQL, often abbreviated NL2SQL, is the technology at the heart of that dream: it takes a sentence such as “Show all flight numbers with aircraft Airbus A340-330” and translates it into a formal SQL query that a relational database can execute. With the rise of large language models, these systems have made stunning progress on academic benchmarks. Yet a new survey published in the journal Vicinagearth argues that the research community has been staring at the wrong part of the problem. The real bottleneck, the authors contend, is not model architecture at all. It is data: how it is collected, structured, represented and used at every stage of the translation pipeline.

The survey, authored by Yuankai Fan, Qizhen Weng, Yin Chen and X. Sean Wang from the Institute of Artificial Intelligence at China Telecom and Fudan University, formally defines the task as learning a translation model that maps a natural language question and a database into an executable SQL query. That definition conceals a thorny reality. Unlike typical language processing tasks with fixed input and output spaces, NL2SQL sits at the intersection of messy, unstructured human language and rigidly structured database schemas. A single question can correspond to many valid SQL translations, a one-to-many mapping that makes evaluation and training genuinely difficult. The authors argue that this intersection makes the quality, diversity and contextualization of data central to success in a way that most model-centric research has systematically underestimated.

To understand why the problem is so hard, the survey catalogs the principal technical challenges. Natural language questions suffer from lexical ambiguity, where a word like “apple” could mean a fruit or a technology company, and syntactic ambiguity, where a sentence permits multiple grammatical interpretations. Questions are also frequently under-specified: someone asking to “meet at the station” has left crucial context implicit. On the database side, modern schemas are sprawling webs of tables, columns and foreign key relationships, and queries often demand multi-table joins whose conditions must be inferred correctly. Dirty data, including missing values, duplicates and inconsistencies, compounds the risk of erroneous results. Processing entire large databases as model input is impractical, so systems must selectively compress enormous amounts of structural and content information into a usable context window.

The field has traveled a long road to reach its current state. Early systems relied on hand-crafted rules and templates, which produced syntactically correct queries but collapsed when confronted with linguistically complex questions involving nested clauses, coreference or ellipsis, and required laborious manual updates for every new domain. Deep learning then brought sequence-to-sequence encoder-decoder models, with systems such as SQLNet framing translation as a slot-filling problem, TypeSQL injecting type information from knowledge graphs, and IRNet introducing intermediate representations that abstract away from raw SQL. Pre-trained language models like BERT and RoBERTa later raised accuracy further, though they still stumbled on outer joins and aggregations and degraded sharply across domains. The arrival of large language models, from the GPT series onward, has powered the current generation of systems, using prompt engineering for proprietary models or fine-tuning for open ones, with approaches such as DIN-SQL, DAIL-SQL and MAC-SQL achieving leading results.

The intellectual core of the survey is its taxonomy of five data types that together define the NL2SQL lifecycle. External knowledge includes domain-specific ontologies, knowledge graphs and common-sense resources, as well as knowledge derived from large language models themselves, which can fill gaps the database does not cover. Text corpora come in two flavors: annotated datasets pairing natural language questions with gold SQL queries, and raw unannotated text such as user query logs and documentation that supports pretraining. Database schemas provide the structural blueprint of tables, columns, data types and relationships, serving as the critical reference for mapping language onto the database. Database instances, the actual rows of stored records, ground the semantics and help resolve ambiguity. Finally, execution feedback, comprising query results and error messages, closes the loop by enabling error correction, verification and reinforcement learning.

These data types map onto four stages of the pipeline. During query understanding, external knowledge helps recognize user intent and entities, compensating for missing or implicit semantics in the input. Schema linking follows, identifying the relevant tables, columns and cell values, using classifiers, graph neural networks or prompting methods, with metadata such as column descriptions and foreign key constraints providing valuable semantic signals. SQL generation then treats the model as a translator, typically fed schema and instance data, often with few-shot examples for LLM-based systems, and supported by fine-tuning on large-scale text corpora. The final stage, query post-refinement, is where the authors see the greatest untapped potential. Most current methods lean almost exclusively on execution feedback, checking whether a query runs and returns plausible results, but this single signal cannot capture subtle semantic inconsistencies in query logic.

The survey also traces how benchmarks have evolved to stress-test these systems. Early datasets such as ATIS for flight information and Geo for United States geography were single-domain with simple queries. The field then shifted to cross-domain evaluation, led by WikiSQL, drawn from Wikipedia tables, and Spider, which introduced complex multi-table SQL across many domains along with extensions like Sparc and CoSql for contextual and conversational settings. Most recently, large-scale real-world benchmarks including BIRD, ScienceBenchmark and Spider 2.0 feature naturally occurring questions, authentic enterprise schemas and challenging structures such as nested queries and set operations, testing reasoning, schema linking and robustness to ambiguity in ways that matter for deployment.

Measuring success is itself a nuanced science. Exact match accuracy checks whether generated SQL is literally identical to the ground truth, but underestimates performance because the same intent can be expressed in many syntactically different ways. Execution accuracy compares the results of running the generated query against those of the reference query, yet risks overestimating correctness when different logic happens to produce identical outputs. Test-suite accuracy executes predictions on curated sets of randomly generated databases to probe semantic equivalence more rigorously, while the valid efficiency score adds a crucial practical dimension by measuring how efficiently correct queries run. In their comparative analysis of thirteen state-of-the-art systems on Spider and BIRD, the authors observe that LLM-based methods substantially outperform their pre-trained predecessors, and notably show stronger generalization on test sets than development sets, suggesting genuine robustness rather than benchmark overfitting.

Looking forward, the authors lay out a research agenda squarely centered on data. They propose dedicated query-rewriting modules that clarify ambiguous or context-dependent expressions before schema linking begins, borrowing proven techniques from dialogue systems to reduce error propagation. They call for richer post-refinement that integrates user interaction signals, semantic validation against schema constraints and cross-checking with alternative query formulations, rather than relying on execution feedback alone. Bridging NL2SQL with the broader natural language to code field could import multi-step reasoning, program synthesis, formal verification and human-in-the-loop debugging. Even structured noise, drawing on recent positive-incentive noise research, might be injected to improve resilience against ambiguous inputs. Beyond accuracy, real deployment demands efficiency and dialect compatibility, suggesting systems should exploit metadata such as indexes and statistics like column cardinality to generate queries optimized for the target engine. The message of the survey is ultimately a liberating one: the fastest path to databases anyone can talk to may run not through bigger models, but through smarter data.

Subject of Research: A data-centric survey of natural language to SQL translation systems, covering data types, benchmarks, evaluation metrics and future research directions.

Article Title: Rethinking data in NL2SQL: a survey of what we have and what we expect

Article References: Fan, Y., Weng, Q., Chen, Y., & Wang, X. S. (2025). Rethinking data in NL2SQL: a survey of what we have and what we expect. Vicinagearth, 2(1), Article 15. https://doi.org/10.1007/s44336-025-00026-9

Image Credits: AI Generated

DOI: 10.1007/s44336-025-00026-9

Keywords: NL2SQL, Text-to-SQL, large language models, databases, SQL generation, benchmarks, data-centric AI, schema linking, pre-trained language models, evaluation metrics, natural language processing, database interfaces