Outcome reward models improve LLM-based Text-to-SQL generation with GradeSQL

In an era when large language models are increasingly being asked to serve as intermediaries between everyday users and complex database systems, one of the most stubborn bottlenecks has been ensuring that the SQL queries these models generate are actually correct. Now, a team of researchers has unveiled GradeSQL, a framework that trains task-specific Outcome Reward Models (ORMs) to act as sophisticated judges of machine-generated database queries, consistently outperforming traditional verification methods on industry-standard benchmarks. The work, published in the Journal of Intelligent Information Systems, could reshape how intelligent information systems verify their own outputs at inference time.

The problem GradeSQL addresses is deceptively simple to state but notoriously difficult to solve. Text-to-SQL generation—the task of translating a natural language question like “Which hospitals in Boston treated more than 500 patients last year?” into an executable SQL query—has been a dream of computer scientists for more than five decades. The payoff is enormous: democratized database access for experts and non-experts alike, without requiring anyone to master query syntax. But while modern large language models have made impressive strides, they still stumble on queries involving multi-table joins, nested subqueries, and sophisticated aggregations. When a model generates a syntactically valid query that returns the wrong answer, the consequences in high-stakes database environments can range from merely confusing to genuinely harmful.

The dominant approach to filtering out bad queries has relied on what researchers call test-time inference strategies. Rather than retraining a model, these strategies generate multiple candidate outputs and then apply a heuristic to pick the best one. Two techniques have become standard: Best-of-N (BoN), which samples N candidate queries and selects the highest-scoring one according to some heuristic, and Majority Voting, which executes all candidates and chooses the output that appears most frequently. Prior empirical work on Text-to-SQL found that N=32 candidates represents an optimal trade-off between performance and computational cost. But both strategies share a fundamental weakness: they provide only coarse, discrete signals. Majority Voting depends on the frequency of execution results, while execution-based Best-of-N treats all executable queries that return non-empty result sets as equally plausible—discarding the rich semantic distinctions that separate a nearly correct query from a hopelessly wrong one.

GradeSQL’s central insight is borrowed from the reinforcement learning community, where Outcome Reward Models have proven their worth as verifiers of mathematical reasoning. Unlike Process Reward Models, which evaluate every intermediate reasoning step and require expensive step-level annotations, ORMs score only the final output. The concept was pioneered in multi-step mathematical reasoning, where a verifier assigns a probability of correctness to each candidate answer. But transplanting the idea into the Text-to-SQL domain demanded significant innovation, because a SQL verifier cannot simply check numeric equality. It must assess the semantic alignment between a natural language prompt, a structured database schema, and the resulting query execution—a far more nuanced judgment.

The GradeSQL framework unfolds in three carefully engineered stages. In the first stage, candidate generation, a generator LLM is prompted with a natural language question and its corresponding database schema to produce N candidate SQL queries using chain-of-thought reasoning, after which the reasoning trace is stripped away. In the second stage, data labeling, each candidate is executed against the target database and compared with the gold query’s result set. A candidate whose returned tuples exactly match the gold result set is labeled correct; one that executes successfully but returns a different result set is labeled a semantic mismatch; and one that crashes outright is discarded, since execution errors are trivially detectable. The crux of the framework lies in distinguishing semantically correct queries from those that are executable but wrong—a far subtler task.

The third stage is where the magic happens. The labeled candidates become training data for supervised fine-tuning, with the verification task cast as an autoregressive binary classification problem. The model receives an input sequence combining the database schema, the natural language question, and a candidate SQL query, and is trained to generate the token “Yes” if the query is semantically correct and “No” otherwise. Crucially, the researchers leverage the generator LLM’s latent ability to self-assess: they prompt the model to classify each candidate and extract the associated logits, which then serve as supervision signals during training. Fine-tuning is performed efficiently using Low-Rank Adaptation (LoRA) under a causal language modeling objective, minimizing the negative log-likelihood of the correct label. At inference time, the resulting ORM functions as a scoring function that maps any question-candidate pair to a probabilistic score between zero and one, inducing a ranked list from which the top-scoring candidate is selected.

To overcome the scarcity of reward-labeled data—a chronic obstacle that had stymied earlier attempts to apply ORMs to Text-to-SQL—the team built a scalable data synthesis pipeline that automates the generation and labeling of training candidates. This solved the cold-start problem that had left ORM-guided verification largely unexplored, and enabled the creation of task-specific reward models tailored to individual benchmark domains.

The evaluation was comprehensive, spanning multiple open-source LLM families and parameter sizes on the two most widely used Text-to-SQL benchmarks: BIRD and Spider. Spider, released in 2018, catalyzed the modern era of cross-domain schema generalization research, while BIRD poses real-world, Messy-data challenges that stress even state-of-the-art systems. The results were striking. ORM-guided Best-of-N achieved execution accuracy gains of up to 4.33 percentage points on BIRD and 2.10 points on Spider compared with execution-based Best-of-N, and gains of 2.91 points on BIRD and 0.93 points on Spider over Majority Voting. In a field where even minor gains represent substantial progress, these improvements are significant.

What makes ORMs particularly powerful is their ability to identify high-quality candidates that are underrepresented in the sample pool—a capability fundamentally absent from Majority Voting, which by design favors whatever output appears most often. ORMs can also recognize semantically equivalent queries that happen to differ syntactically, scoring two differently worded queries that return identical result sets as equally valid. This makes them a natural fit for re-ranking diverse candidate sets and for modular system architectures in which the generator and the verifier can be optimized independently. It also sidesteps the known failure mode of Best-of-N with coarse heuristics, which tends to over-prefer generic, high-probability outputs, and the vulnerability of Majority Voting when correct solutions are rare among the sampled candidates.

The research also situates itself within a broader trend of inference-time scaling, which allocates additional computational resources during inference rather than during training. The philosophy mirrors what has driven advances in mathematical reasoning: large language models can produce correct answers when given more inference time, rather than larger model sizes or additional training. Traditional decoding strategies, from greedy decoding and beam search to temperature scaling, top-k sampling, and nucleus sampling, operate only at the token level and cannot substantially improve the correctness of complete outputs, leaving systems vulnerable to hallucinations. Candidate-level verification with a trained reward model represents a fundamentally different lever—one that operates on the semantics of whole queries rather than the probabilities of individual tokens.

The team has released its codebase, synthesized datasets, and fine-tuned ORMs on GitHub and Hugging Face, an open approach intended to accelerate reproducibility and further research. The implications extend well beyond academic benchmarks. As enterprises increasingly deploy natural language interfaces over production databases—powering everything from customer support chatbots to internal analytics tools—the reliability of generated SQL becomes a first-order concern. A verification layer that provides continuous, probabilistic assessments of semantic quality, rather than binary pass-fail execution checks, offers a principled mechanism for deciding when to trust a machine-generated query and when to escalate to a human.

There remain open questions. The framework depends on the availability of gold queries for labeling during training, and its performance ceiling is presumably tied to the quality of the underlying generator’s self-assessment signals. The researchers also note that controlled ablations on prompt design, model scale, and training losses revealed nuances in how these factors interact, suggesting room for further optimization. Still, by demonstrating that task-specific Outcome Reward Models can consistently outperform both execution-based Best-of-N and Majority Voting, GradeSQL establishes a new baseline for test-time verification in Text-to-SQL—and points toward a future where language models not only write database queries, but grade their own work with genuine semantic understanding.