Source-linked AI summary
The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI Operators
Tarfah Alrashed, Fatma Ozcan, Per Jacobsson, Tal Neiman, Xianshun Chen
TL;DR
Text-to-SQL evaluation metrics assume deterministic execution, but AI operators produce flexible semantic outputs that make exact-result evaluation unreliable. The paper separates relational validation from AI-semantic evaluation, achieving state-of-the-art overall accuracy across BigQuery and ThalamusDB, up to 97.2%.
Problem
Existing Text-to-SQL evaluation struggles with AI-augmented queries because deterministic metrics do not account for flexible AI semantics.
Source: §1 · p. 2Method
The framework separates relational logic from AI components and evaluates them independently using database execution and an LLM autorater.
Source: §4 · p. 3, §5.2 · p. 5Results
Across BigQuery and ThalamusDB, the framework achieved state-of-the-art overall accuracy, reaching 97.2% on BigQuery.
Source: §5.2 · p. 5Takeaways & Limitations
The results indicate that evaluating relational logic separately from AI semantics can better assess AI-augmented Text-to-SQL than relying on exact execution results alone.
Source: §1 · p. 2, §5.2 · p. 5
Abstract
from arXiv · showhide
SQL has been augmented with AI operators, enabling modern data analytics platforms to derive insights from both structured and unstructured data. We observe that while current Text-to-SQL systems can successfully generate these AI-augmented queries, reliably evaluating their correctness remains a critical open challenge. Current metrics, which rely on exact query results and deterministic execution, systematically fail against the flexible, non-deterministic outputs of AI operators. In this paper, we formalize these unique evaluation failure modes and introduce a Multilayered Evaluation Framework that decouples deterministic database logic from flexible AI semantics. We test our approach across both industry (BigQuery) and academic (ThalamusDB) systems. We demonstrate that traditional Execution Accuracy severely penalizes valid queries, achieving as low as a 25% detection rate for correct translations. Furthermore, even a state-of-the-art LLM-based autorater falsely rejects 32% of accurate queries due to the complexity of judging both relational and AI components simultaneously. By validating the standard relational logic and the AI operations separately, our framework achieves state-of-the-art overall accuracy across both platforms (up to 97.2%), proposing a reliable standard for benchmarking AI-powered SQL generators.
1 Introduction
AI operators bring unstructured-data reasoning into SQL, motivating Text-to-SQL systems for complex hybrid queries. Existing evaluation assumes deterministic execution, while this paper proposes separating relational validation from semantic evaluation.
1 Introduction
- AI operators let SQL process unstructured content, and translating natural language into hybrid queries has emerged as a critical next step.Writing these complex queries manually can hinder adoption.Source: §1 · p. 1
- Current Text-to-SQL metrics assume deterministic execution over structured, tabular data, despite the shift toward AI-augmented SQL.Exact Match, Execution Accuracy, and Test Suite Accuracy were developed for that deterministic setting.Source: §1 · p. 1
- Prompt variations can change AI outputs and returned rows, causing rigid metrics to reject functionally equivalent queries.The paper argues that these cases require relaxed semantic correctness rather than strict equivalence.Source: §1 · p. 1
- LLM autoraters can struggle to verify relational constraints and fuzzy prompt semantics simultaneously, motivating decoupled evaluation.The paper also categorizes failure modes and outlines open challenges for future query builders.Source: §1 · p. 2
- The Multilayered Evaluation Framework separates deterministic relational logic from AI components, and its overall accuracy reaches up to 97.2% across BigQuery and ThalamusDB.EX detects as few as 25% of correct translations, while LLM autoraters falsely reject up to 32% of accurate queries.Source: §1 · p. 2
2 Background
Traditional relational databases cannot directly understand unstructured content, while AI operators bring semantic processing into the SQL interface. This enables queries that combine deterministic filtering with AI-based evaluation of text, images, or audio.
2 Background
- Traditional databases cannot comprehend unstructured data, and extracting insights from it has required external pipelines with data-movement overhead and architectural complexity.Source: §2 · p. 2
- AI operators embed semantic understanding in databases, allowing users to query unstructured data through SQL without external pipelines or task-specific fine-tuning.LLM inference introduces computational costs.Source: §2 · p. 2
- Platforms including BigQuery and ThalamusDB provide native AI functions for processing multimodal data, such as AI.IF and NLfilter.The passage also identifies vector search capabilities and other functions, including AI.SEARCH and AI.SCORE.Source: §2 · p. 2
- The example query combines a structured electric-fuel filter with AI evaluation of audio recordings for a dead battery.BigQuery uses AI.IF and ThalamusDB uses NLfilter for the same intent.Source: §2 · p. 2
3 The Breakdown of Current Evaluation
Execution Accuracy rejects valid AI-augmented queries when stochastic prompt interpretation produces different results, even when the generated output matches user intent. The section argues for evaluating semantic alignment rather than requiring exact result-set equivalence.
3 The Breakdown of Current Evaluation
- The generated query returned 11 positive reviews versus 9 in the benchmark, yet all 9 benchmark reviews appeared among the 11 and Execution Accuracy scored the valid translation 0%.Source: §3 · pp. 2–3
- Execution Accuracy assumes deterministic results, but slight prompt changes can shift an LLM’s decisions on subjective, borderline cases.It therefore falsely rejects the generated query when its count differs from the benchmark count.Source: §3 · p. 3
- The paper frames the needed standard as relaxed semantic correctness: accept queries that capture user intent despite small result variations from LLM sampling.Source: §3 · p. 3
4 Multilayered Evaluation Framework
The Multilayered Evaluation Framework separates a query’s relational structure from its AI component, executes both full and split queries, then uses an LLM autorater for independent semantic evaluation. This design aims to assess relational correctness and AI intent without conflating their distinct behaviors.
4 Multilayered Evaluation Framework
- The framework’s three phases are splitting queries, executing full and isolated components, and autorating with execution results and query context.The autorater receives the schema, natural-language question, queries, and execution results.Source: §4 · p. 3
- The SQL Splitter isolates relational structure from AI semantics and can be implemented with an AST parser or an LLM.The paper uses a few-shot LLM-based splitter for SemBench queries.Source: §4 · p. 3
- The split separates structural SQL from the AI component, enabling each part to be evaluated against its respective execution behavior.The example splits a query into a structural part and an AI part.Source: §4 · p. 3
- The autorater scores overall, relational, and AI similarity from 1 to 5, treating scores above 3 as correct and flagging the primary error source.Scores of 4 or 5 accept functionally equivalent or perfectly aligned queries; scores of 3 or below are rejected.Source: §4 · p. 3
5 Experiments
Experiments compare the Multilayered Evaluation Framework with execution-based and LLM-based baselines on AI-augmented SQL. The framework performs best overall across BigQuery and ThalamusDB, while the baselines miss valid translations or accept flawed ones.
5.1 Experimental Setup
- SemBench provides 55 queries across five real-world scenarios, and manually verified in-context-learning outputs were correct for 70% of BigQuery queries and 80% of ThalamusDB queries.The evaluation compared the proposed framework with EX and LLM-SQL-Solver.Source: §5.1 · pp. 3–4
5.2 Experimental Results
- 36.0% of valid BigQuery translations and 25.0% of valid ThalamusDB translations were correctly identified by EX, with overall accuracies of 52.8% and 26.7%, respectively.EX also accepted a query missing DISTINCT because the benchmark database had no duplicate values.Source: §5.2 · p. 4
- 91.7% True Positive Rate on ThalamusDB came with a 66.7% True Negative Rate for Explain & Compare, and adding execution data did not fix its errors.The baseline accepted UNION DISTINCT in place of UNION ALL because both returned the same results on this database.Source: §5.2 · p. 4
- 100.0% True Negative Rate on both systems for Miniature & Mull came with false rejection of around 32% of correct queries.Adding execution results further lowered overall accuracy, as real data conflicted with the prompt to invent counterexamples.Source: §5.2 · pp. 4–5
- 97.2% on BigQuery and 93.3% on ThalamusDB were the framework’s overall accuracies; it also captured 100% of correct BigQuery translations and achieved 100% ThalamusDB True Negative Rate.The framework separates relational logic from AI components, then evaluates them through database execution and independent LLM autorating.Source: Figure 1 · p. 4, Table 1 · p. 5, §5.2 · p. 5
6 Open Challenges in Text-to-SQL Generation with AI Operators
The paper identifies four challenges for generating AI-augmented SQL: choosing between structured and semantic processing, selecting among data modalities, combining database and external knowledge, and refusing unanswerable requests.
Structured vs. Semantic Misalignment
- Structured-versus-semantic misalignment requires choosing when structured filters suffice and when AI operators should interpret unstructured descriptions.The paper notes that relying on a room-count column can misclassify a house with three total rooms as having three bedrooms.Source: §6 · p. 5
Data Source and Modality Ambiguity
- Data-source and modality ambiguity requires selecting among descriptions, reviews, or images and deciding whether to apply sequential filters or a multimodal prompt.This is framed as a new form of semantic query optimization.Source: §6 · p. 5
The Open-World vs. Closed-World Divide
- Open-world versus closed-world queries must combine exact database filters with AI evaluation of subjective knowledge, such as vehicle reliability.The example pairs price and sedan filters with AI.IF.Source: §6 · p. 5
Forced Answering of Ambiguous Queries
- Forced answering generates costly AI queries when a request is invalid or incomplete, even when the database lacks the data to answer it.The paper identifies rejecting unanswerable queries before AI operators run as a requirement for real-world systems.Source: §6 · p. 5
7 Related Work
Related work situates AI-augmented SQL among LLM-powered data systems, specialized benchmarks, and LLM-as-a-Judge evaluation. The paper’s framework applies semantic judging to SQL by separating relational logic from AI components.
LLM-Powered Data Systems and AI-Augmented SQL
Evolution of Text-to-SQL Benchmarks
- Spider and BIRD support traditional Text-to-SQL evaluation, while SemBench [4] evaluates multimodal data systems’ cost, latency, and accuracy rather than natural-language translation.The passage calls for complex AI-operator queries and measures of relaxed semantic correctness in future generative-model benchmarks.Source: §7 · p. 6
LLM-as-a-Judge and Semantic Evaluation
- LLM-as-a-Judge methods assess semantic alignment rather than execution equivalence; this framework extends that approach by isolating relational logic from stochastic AI components.The passage cites prior LLM-as-a-Judge techniques.Source: §7 · p. 6
8 Conclusion & Future Work
The framework separates relational logic from AI components and achieves high accuracy on BigQuery and ThalamusDB. The conclusion also identifies a future benchmark aimed at query-generation failure modes.
8 Conclusion & Future Work
- 97.2% accuracy on BigQuery and 93.3% on ThalamusDB were achieved by the framework, while current generators produced valid queries 70–80% of the time.EX detected as few as 25% of correct translations, and existing LLM autoraters falsely rejected up to 32% of valid outputs.Source: §8 · p. 6
- The planned large-scale benchmark targets intelligent data routing, multimodal ambiguity, and safe refusal as failure modes for next-generation query engines.The passage presents it as future work.Source: §8 · p. 6
- The conclusion presents the framework as a way to evaluate AI-augmented queries while recognizing query generation as a distinct challenge.The paper’s preliminary analysis outlines four critical generation challenges.Source: §8 · p. 6