PaperlayerView exact revision on arXiv

Source-linked AI summary

TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification

Neelesh Kumar Shukla, Debasmita Panda, Srutanik Bhaduri, Aditya Banerjee, Viji Krishnamurthy

arXiv:2608.17795v1cs.CL

TL;DR

Reference-free Text-to-SQL verification lacks traceable evidence for diagnosing whether generated queries answer user questions. TraceSQL addresses this with a lightweight verifier using 67 explicit diagnostic features, achieving 66.47% F1 and 64.48% ROC-AUC versus 61.87% F1 and 58.26% ROC-AUC for GradeSQL-7B on held-out BIRD databases.

  • Problem

    Reference-free Text-to-SQL verification must assess generated queries without ground-truth SQL or reference execution results while supporting diagnosis of semantic, schema-grounding, and structural errors.

  • Method

    TraceSQL uses 67 explicit diagnostic features spanning question requirements, question–schema–SQL consistency, SQL structure, ambiguity, and intent alignment, retaining links to underlying evidence.

  • Results

    66.47% F1 and 64.48% ROC-AUC versus 61.87% F1 and 58.26% ROC-AUC for GradeSQL-7B were achieved on held-out BIRD development databases.

  • Takeaways & Limitations

    Reference-free SQL verification can be learned from structured diagnostic signals while preserving feature-level evidence for inspecting predictions and analyzing failures.

  • Takeaways & Limitations

    The reported training experiments use a 2,000-candidate sample, while scaling to the complete paired source is left for future work.

Abstract

from arXiv · show

Text-to-SQL systems are commonly evaluated using ground-truth SQL queries or reference execution results, but such supervision is unavailable at inference time in real-world deployments. This creates a critical verification problem: given only a user question, database context, and generated SQL, can a system estimate whether the generated query is likely to correctly answer the question? Recent approaches use LLMs as judge or specialized agents to inspect generated SQL, but their decisions can be difficult to trace. Outcome Reward Models (ORMs) address this by learning from execution-labeled candidate SQLs and assigning correctness scores to unseen queries, yet they still provide limited visibility into the signals behind each verification. To address this limitation, we propose TraceSQL, a lightweight and traceable verification model built on explicit diagnostic features. TraceSQL combines 67 features capturing question ambiguity, question requirements, question-schema-SQL consistency, SQL structure, and intent alignment. These signals remain available for examining which factors influence each prediction and for tracing decisions back to diagnostic evidence. On BIRD development databases, TraceSQL achieves 66.47% F1 and 64.48% ROC-AUC, compared with 61.87% F1 and 58.26% ROC-AUC for the GradeSQL-7B ORM baseline on the same generated-SQL evaluation. Feature attribution further shows that the model relies on both semantic grounding and deterministic SQL-structure signals. These results show that SQL verification can be performed with a lightweight learned model while retaining feature-level evidence for inspecting and diagnosing its predictions.

1 Introduction · 2 Related Work

TraceSQL addresses the deployment-time challenge of verifying generated SQL without reference queries or execution results by using explicit diagnostic features rather than only opaque correctness scores. It combines traceable evidence with learned verification and outperforms GradeSQL-7B on identical held-out BIRD candidate SQLs.

  • 1 Introduction: Deployment-time Text-to-SQL verification is necessary because generated SQL may contain semantic, schema-grounding, or structural errors without reference SQL or execution results.Benchmark evaluation can use ground-truth SQL or reference execution results, but deployment lacks these resources.
  • 1 Introduction: TraceSQL represents each question–candidate pair with 67 explicit features spanning five diagnostic families, including ambiguity, planning, question-schema-SQL analysis, SQL structure, and intent alignment.The representation is designed to preserve semantically meaningful diagnostic signals rather than relying only on a final verifier score.
  • 1 Introduction: TraceSQL provides interpretability through named features, explainability through feature-importance analysis, and traceability through links to the underlying diagnostic evidence.These visibility levels support examining which signals influence predictions and diagnosing candidate failures.
  • 1 Introduction: 66.47% F1 and 64.48% ROC-AUC are achieved by TraceSQL, compared with 61.87% F1 and 58.26% ROC-AUC for GradeSQL-7B on identical held-out BIRD generated-candidate SQLs.TraceSQL is trained on generated candidate SQLs from balanced BIRD ORM training data released with GradeSQL.
  • 2.1 Text-to-SQL Generation: Recent Text-to-SQL generation methods improve decomposition, schema linking, prompt construction, example selection, multi-agent refinement, or synthetic supervision, whereas TraceSQL verifies candidates after generation.TraceSQL focuses on preserving evidence behind the verification decision rather than improving SQL generation itself.
  • 2.2 Post-Generation Verification and Learned Verifiers: Post-generation verification includes self-correction guidelines, behavior comparison with an independently constructed execution path, and outcome-supervised reward models for ranking or scoring candidates.GradeSQL trains task-specific ORMs on execution-derived candidate labels and assigns continuous correctness scores at inference time.
  • 2.2 Post-Generation Verification and Learned Verifiers: TraceSQL differs from scalar-score verifiers by retaining a fixed set of diagnostic signals that can be analyzed individually and traced to their source evidence.This representation follows the use of execution-derived candidate labels while exposing more verification structure.
  • 2.3 Interpretable, Explainable, and Traceable Verification: TraceSQL’s transparency combines semantically defined input features, permutation-importance and SHAP attribution, and links from features to the diagnostic evidence from which they were derived.These mechanisms expose which signals influence predictions and provide measurable verification evidence.

3 Task Formulation · 4 Traceable Answerability Signals

TraceSQL estimates whether candidate SQL answers a question using only the question, database context, and candidate SQL, while retaining explicit diagnostic evidence for inspecting each prediction. Its 67 features combine ambiguity, requirements, question–schema–SQL consistency, SQL structure, and intent alignment signals derived through LLM probes, rule projection, and AST extraction.

  • 3 Task Formulation: TraceSQL verifies candidate SQL x against question q and database context s without using reference SQL or execution-match labels as inputs.Reference SQL and execution-match labels are used only to construct offline supervision.
  • 3 Task Formulation: The 67-dimensional representation captures question ambiguity, question requirements, question–schema–SQL consistency, SQL structure, and intent alignment.Feature values remain individually inspectable and retain diagnostic provenance for examining semantic, grounding, and structural influences.
  • 4.1 Diagnostic Evidence Generation: Three upstream modules generate retained evidence: the Ambiguity Detector, Pair Analyzer, and SQL Repair Module.Their outputs capture question ambiguity, question–schema–SQL consistency, and alignment between the user request and candidate SQL.
  • 4.1.1 Ambiguity Detector: The Ambiguity Detector outputs ambiguity status, a 0–100 ambiguity probability, and an explanation supporting its assessment.These outputs provide source evidence for downstream ambiguity features.
  • 4.1.2 Pair Analyzer: The Pair Analyzer applies rules for grounding, joins, identifiers, data types, temporal semantics, metrics, null and validity logic, and business-term mappings.It retains support classifications, scores, summaries, rule-level results, and findings describing grounding or consistency gaps.
  • 4.1.3 SQL Repair Module: The SQL Repair Module decomposes questions into structured intents, explains candidate SQL operations, and evaluates reference-free intent alignment.Its outputs include evaluation score, confidence, gate decision, intent-level assessments, and intervention information.
  • 4.2 Feature Extraction: Feature extraction uses fixed LLM probe sets for ambiguity, question planning, and intent alignment, while Pair Analysis statuses and SQL-AST properties are projected deterministically.SQL queries are parsed with SQLGlot to derive structural properties.
  • 4.2.2 Question-Planning Features: The feature families contain 10 ambiguity, 5 question-planning, 32 pair-analysis, 10 SQL-structure, and 10 intent-alignment features, totaling 67.SQL-structure features represent AST-detected properties such as aggregation, grouping, joins, filtering, ordering, limits, subqueries or CTEs, window functions, and DISTINCT; richer evidence is retained separately.

5 Research Questions

This section frames TraceSQL evaluation around four questions: verification effectiveness, diagnostic-signal contributions, cross-database generalization, and traceability to explicit evidence.

  • RQ1: RQ1 asks how effectively structured diagnostic evidence supports reference-free Text-to-SQL verification.
  • RQ2: RQ2 examines which diagnostic signals contribute most to TraceSQL’s learned verification decision.
  • RQ3: RQ3 evaluates how well TraceSQL generalizes across held-out databases.
  • RQ4: RQ4 investigates whether TraceSQL predictions can be traced back to explicit diagnostic evidence.

6 Data Preparation

TraceSQL training uses a deduplicated, balanced subset of the BIRD ORM corpus, preserving alternative candidates while representing all 69 databases. The final sample contains 2,000 candidates and is used for current experiments, while scaling to the full paired source remains future work.

  • Corpus construction: 30,686 candidate SQL queries span 69 databases and 2,642 unique question–database groups in the balanced BIRD ORM training corpus.Each group contains between 2 and 32 candidates, making row-level sampling potentially disproportionate.
  • Corpus construction: 19,592 candidates remain after removing exact duplicates across question, schema, SQL, data, label, and database ID.Database identifiers are assigned by matching normalized question-plus-evidence text to BIRD training records, while alternative candidates are preserved.
  • Balanced sampling: 2,642 balanced positive–negative pairs are formed by retaining one positive and one negative candidate per eligible question–database group.The paired source contains 5,284 candidates.
  • Balanced sampling: 2,000 candidates are selected as 1,000 complete pairs through database-stratified sampling, with all 69 training databases represented and reproducibility ensured by seed 42.The resulting training set contains 1,000 positive and 1,000 negative candidates, with remaining samples allocated proportionally across databases.
  • Training scope: 6.5% of the original ORM corpus and 37.9% of the 5,284-candidate paired source are represented by the final sample used for current TraceSQL training experiments.Scaling to the complete paired source is left for future work.

7 Experimental Setup · 8 Results

TraceSQL is evaluated with fixed LLM-backed diagnostic extraction, FLAML-selected modeling, and matched held-out BIRD verification tests. It outperforms GradeSQL-7B on generated SQL, combines structural and semantic signals, preserves feature-to-evidence provenance, and shows lower acceptance on ground-truth SQL.

  • 7.1 LLM Configuration for Diagnostic Evidence and Feature Extraction: GPT-4o powers ambiguity detection, SQL repair, and unified ambiguity, question-planning, and intent-alignment probes, while GPT-5.2 powers pair analysis.
  • 7.2 Training with FLAML: FLAML trains TraceSQL on 2,000 balanced candidates using all 67 diagnostic features, retaining 62 features after preprocessing for the fitted estimator.
  • 7.2 Training with FLAML: 4,228 trials within a 1,800-second search budget select a seven-tree Extra Trees classifier, with the same configuration and performance at 3,600 seconds.
  • 7.3 Baseline and External Evaluation: The saved 1,800-second model is evaluated without refitting on 11 disjoint BIRD development databases, with GradeSQL-7B tested on identical generated candidates and labels.
  • 8.1 Training Results: 0.6520 ROC-AUC is achieved after refitting on all 2,000 training candidates, alongside accuracy of 0.6085, precision of 0.5987, recall of 0.6580, and F1 of 0.6270.
  • 8.2 Which Diagnostic Signals Does the Model Use?: Feature attribution shows that the verifier relies on both deterministic SQL structure and semantic grounding, with nine features appearing in the top ten under both permutation importance and SHAP.
  • 8.3 Generated-SQL Verification and Cross-Database Generalization: 4.60 percentage points in F1 and 6.22 points in ROC-AUC separate TraceSQL’s stronger aggregate generated-SQL results from GradeSQL-7B on 1,521 candidates across 11 held-out databases.
  • 8.4 Traceability of Verification Decisions: TraceSQL connects each influential feature to a named probe, verification rule, SQL-structure property, or retained supporting evidence, providing a direct provenance path to diagnostic conditions.

9 Discussion

TraceSQL distinguishes itself from GradeSQL by exposing structured diagnostic evidence at inference time, combining question-level and candidate-level signals for reference-free verification. Its fixed, trace-linked features support inspection of influential predictions across SQL structure, schema grounding, and semantic validity.

  • TraceSQL exposes a structured diagnostic representation at inference time, whereas GradeSQL provides only a final correctness score.Both use execution-derived candidate supervision.
  • Verification depends jointly on question properties and candidate SQL, so TraceSQL explicitly captures ambiguity, planning requirements, grounding, SQL structure, and intent alignment.This supports reference-free verification when question clarity and candidate realization provide complementary evidence.
  • Feature analysis shows TraceSQL combines deterministic SQL-structure signals with grounding and semantic-validity signals.Examples include SQL DISTINCT Usage, SQL LIMIT Usage, SQL Aggregation Usage, Additional Grounding Check, Pattern-Based Business Meaning Support, and Data-Validity Rule Support.
  • Fixed semantic definitions and links to diagnostic records make influential predictions traceable to concrete evidence.The full diagnostic vocabulary remains available even when some signals are weak on the current BIRD distribution, supporting different database and query settings.

10 Future Work · 11 Conclusion · Supplementary Material

TraceSQL concludes that reference-free Text-to-SQL verification can combine traceable diagnostic evidence with learned prediction, while future work will test larger-scale training and integration with outcome reward modeling. Its reported results show improved held-out performance and inspectable reliance on semantic and SQL-structural signals.

  • 10 Future Work: Future work will scale training from 2,000 controlled candidates to the complete 5,284-candidate paired source while retaining the held-out 11-database evaluation.This will test whether additional supervision improves verification performance and whether diagnostic-signal importance remains stable as training expands.
  • 10 Future Work: Future work will incorporate the 67 TraceSQL features directly into a GradeSQL-style outcome reward model as additional structured evidence.The planned study will assess the effects of adding ambiguity, planning, grounding, SQL-structure, and intent-alignment signals on predictive performance.
  • 10 Future Work: The planned comparison will test whether TraceSQL is more useful as a standalone verifier or as an evidence layer within a larger learned verification model.This comparison concerns the representation’s role in verification.
  • 11 Conclusion: TraceSQL uses 67 diagnostic features spanning ambiguity, question requirements, question–schema–SQL consistency, SQL structure, and intent alignment.The representation retains its underlying diagnostic evidence for further inspection.
  • 11 Conclusion: 66.47% F1 and 64.48% ROC-AUC improve over GradeSQL-7B’s 61.87% and 58.26%, respectively, on held-out BIRD development databases.The comparison is reported for TraceSQL versus GradeSQL-7B.
  • 11 Conclusion: Feature analysis finds that verification draws on both deterministic SQL-structure signals and semantic grounding evidence.The results associate reliable verification with combining multiple forms of diagnostic information.
  • 11 Conclusion: TraceSQL remains inspectable because influential features connect to meaningful diagnostic conditions and the evidence from which they were derived.This supports inspection and failure analysis.

A FLAML Preprocessing Details … B.4 SQL-Structure Features

TraceSQL uses a fixed 67-feature candidate-level representation spanning five diagnostic families, with additional explanations and evidence retained separately for traceability. The feature definitions cover preprocessing, ambiguity, question planning, pair analysis, and deterministic SQL structure.

  • A FLAML Preprocessing Details: Five Pair Analysis features are absent after FLAML preprocessing and feature transformation.Their absence reflects preprocessing rather than an explicit feature-selection procedure.
  • B Complete Feature Definitions: 67 candidate-level features span ambiguity, question planning, pair analysis, deterministic SQL structure, and intent alignment.The families contain 10, 5, 32, 10, and 10 features, respectively.
  • B Complete Feature Definitions: Explanations, rationales, confidence, warnings, and SQL fragments are retained separately for traceability, not added as model inputs.The representation supplied to FLAML is fixed at 67 candidate-level features.
  • B.1 Ambiguity Features: 10 ambiguity features assess ambiguity once per question–database pair and share that assessment across candidate SQL queries.These features are summarized in Table 8.
  • B.2 Question-Planning Features: 5 question-planning features are included in the unified representation.They are summarized in Table 9.
  • B.3 Pair-Analysis Features: 32 pair-analysis features provide canonical rule-status inputs to the model.Rule scores and explanations remain diagnostic evidence rather than additional model features.
  • B.4 SQL-Structure Features: 10 binary SQL-structure indicators are derived by deterministically inspecting a SQLGlot-parsed abstract syntax tree.Parse failures remain diagnostic artifacts rather than being assigned a particular SQL structure.

B.5 Intent-Alignment Features … C.2 Coverage Across Diagnostic Feature Families

TraceSQL’s intent-alignment representation uses ten evaluator- and probe-based features, while feature-importance analyses show that highly ranked signals span multiple diagnostic families. The strongest signals combine deterministic SQL structure with semantic grounding and validity evidence, although family coverage is not a causal ablation.

  • B.5 Intent-Alignment Features: Ten intent-alignment features combine four direct evaluator fields with six probe-based alignment verdicts.These features are summarized in Table 13.
  • C Detailed Feature-Importance Analysis: The feature-importance analyses report global SHAP rankings and compare SHAP with permutation importance and native Extra Trees importance.The analyses characterize model reliance on the training/model-development population.
  • C.1 Global SHAP Analysis: 0.0199 is the mean absolute SHAP value for Additional Grounding Check, followed by SQL LIMIT Usage at 0.0147 and Pattern-Based Business Meaning Support at 0.0117.SQL DISTINCT Usage (0.0074), SQL Aggregation Usage (0.0073), and Data-Validity Rule Support (0.0070) form the next group.
  • C.1 Global SHAP Analysis: 0.0052, 0.0041, and 0.0034 are the mean absolute SHAP values for SQL Grouping Usage, Intent Alignment: Grouping, and Question Planner: Output Structure, respectively.The remaining top-fifteen features have smaller mean absolute contributions.
  • C.2 Coverage Across Diagnostic Feature Families: The leading features come from multiple TraceSQL diagnostic families rather than a single family.Table 15 counts family representation among the top ten under SHAP, permutation importance, and native Extra Trees feature importance.
  • C.2 Coverage Across Diagnostic Feature Families: SQL Structure and Pair Analysis account for most highly ranked features across all three methods, while Question Planning and Intent Alignment also contribute signals.The rankings should be interpreted as representation coverage, not feature-family ablation or causal contribution.
  • C.2 Coverage Across Diagnostic Feature Families: Semantic grounding and validity signals from Pair Analysis consistently appear among the strongest features, so the rankings do not reduce to SQL structure alone.This supports the presence of both deterministic SQL-structure properties and semantic grounding evidence among leading signals.

C.3 Cross-Method Feature-Importance Comparison

Cross-method comparison is based on feature ranks because permutation importance, SHAP, and native Extra Trees importance quantify different aspects of model reliance. The methods agree most on several semantic and SQL-structural signals, while ambiguity features are absent from their top ten.

  • Methodology: Feature-importance methods are compared by rank rather than raw value because they measure different aspects of model reliance.The comparison includes permutation importance, SHAP, and native Extra Trees feature importance.
  • Cross-method agreement: The strongest agreement covers Additional Grounding Check, SQL LIMIT Usage, Pattern-Based Business Meaning Support, SQL Aggregation Usage, and Data-Validity Rule Support.All five features rank near the top under SHAP, permutation importance, and native feature importance.
  • Cross-method agreement: SQL DISTINCT Usage ranks first under permutation importance and fourth under SHAP, but lower under native feature importance.This pattern indicates greater method sensitivity than the consistently high-ranked features.
  • Feature categories: The recurring high-ranked set spans SQL-structure and semantic diagnostic signals, whereas ambiguity features do not enter the top ten under any of the three methods.Ambiguity is assessed at the question–database level and shared across alternative candidate SQLs for the same question.

D Feature Correlation Analysis · E Representative Prediction Trace · F Additional Database-Level Results

The analysis identifies limited feature correlation, traces a representative false-positive decision to retained diagnostic evidence, and compares database-level acceptance rates while noting the generated-SQL setting remains primary. TraceSQL’s strongest dependencies and candidate-level evidence clarify how its predictions can be inspected.

  • D Feature Correlation Analysis: Pairwise correlation analysis is used to identify strongly related numeric TraceSQL signals that may contain overlapping information.
  • D Feature Correlation Analysis: Most numeric feature pairs show weak linear relationships, with notable dependencies between SQL ORDER BY Usage and SQL LIMIT Usage (r= 0.887).Intent Alignment Overall Score and Intent Alignment Confidence also correlate at r= 0.729; these descriptive training-population dependencies can redistribute predictive information across attribution methods.
  • E Representative Prediction Trace: A representative false-positive asks for the top three drivers and their points in the 2017 Chinese Grand Prix, while the candidate SQL receives a positive probability of 0.572 despite a negative execution-match label.The query filters the race, joins race, results, and driver tables, orders by points, and applies LIMIT 3.
  • E Representative Prediction Trace: Local attribution links influential model features to named SQL-structure properties, Pair Analyzer rule results, or probe-level diagnostic records retained during feature extraction.
  • F Additional Database-Level Results: Because all 1,534 ground-truth SQL instances are positive, accuracy equals recall, precision is 100%, ROC-AUC is undefined, and database-level acceptance rates are reported.Acceptance rate is equivalent to accuracy and recall in this setting.
  • F Additional Database-Level Results: TraceSQL has higher acceptance on California Schools, Codebase Community, Financial, Formula 1, and Thrombosis Prediction, whereas GradeSQL-7B is higher on the remaining six databases.
  • F Additional Database-Level Results: The generated-SQL setting remains the primary two-class verification evaluation.
Loading 2608.17795v1…