PaperlayerView exact revision on arXiv

Source-linked AI summary

PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems

Shuo Lin, Jinzhe Li, Chenlu Guo, Yanhan Li, Yuan Wu, Yi Chang

arXiv:2610.22158v1cs.CL

TL;DR

Text-to-SQL systems largely assume user queries are valid and answerable, while flawed premises can lead them to produce misleading SQL. This paper introduces PremiseSQL, a paired benchmark and evaluation framework for premise critique across prompting settings. Models often trust flawed premises by default; critique-oriented prompting improves flawed-query responses but also suppresses SQL generation on originals.

  • Problem

    Text-to-SQL research largely assumes queries are valid and answerable, despite questions that contain flawed premises and can elicit misleading SQL.

    Source: §1 · pp. 1–2

  • Method

    PremiseSQL pairs 1,800 constructed flawed queries with 1,800 originals across four premise-failure categories and evaluates intrinsic and prompt-induced behavior in three settings.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across prompting settings, models often trust flawed premises by default, while stronger critique instructions improve flawed-query detection and abstention but reduce SQL generation on originals.

    Source: §4.2 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    Strong SQL generation does not necessarily imply trustworthy premise-aware reasoning, and models continue to struggle with subtle and implicit premise flaws.

    Source: §5 · p. 8

    The benchmark focuses on English NL2SQL and taxonomy-driven constructed errors, and its partly LLM-based automatic judging may introduce evaluation bias.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that user queries are valid and answerable. As a result, models often generate executable SQL even for queries with flawed premises. We introduce premise critique ability in NL2SQL, namely the ability to recognize flawed queries and abstain from generating misleading SQL. To evaluate this capability, we propose PremiseSQL, a benchmark containing 1,800 flawed NL2SQL queries across four categories of premise failures, along with their 1,800 original queries. We further design an evaluation framework to measure intrinsic and prompt-induced premise-aware behavior across three prompting settings. Experiments on frontier LLMs show that current models are strongly premise-trusting by default and rarely critique flawed premises. Stronger critique-oriented prompting improves detection and abstention on flawed queries, but also increases false invalid judgments and suppresses SQL generation on originals. The code is available at https://github.com/vegeet/PremiseSQL.

1 Introduction

Text-to-SQL research largely assumes queries are valid and answerable, although real queries may contain flawed premises that lead models to generate executable but semantically invalid SQL. The paper defines premise critique as recognizing such flaws and abstaining from misleading SQL, then introduces PremiseSQL and an evaluation framework for this ability.

1 Introduction

  • Existing Text-to-SQL research largely assumes valid, answerable queries, but realistic questions can contain inconsistent conditions, false schema assumptions, missing premises, or requests beyond SQL and database capabilities.Models may silently add unsupported assumptions and produce executable SQL that is semantically invalid.Source: §1 · pp. 1–2
  • Premise critique combines validating a query against its database context with abstaining on flawed queries while still generating SQL for valid ones.It is distinct from SQL-generation accuracy: even SQL that coincidentally matches intent can fail to recognize an ill-posed question.Source: §1 · p. 2
  • PremiseSQL pairs 1,800 constructed flawed queries with 1,800 unmodified originals to evaluate critique while preserving SQL generation on valid queries.The benchmark is taxonomy-guided and evaluates both flawed-premise recognition and abstention.Source: §1 · p. 2
  • The evaluation framework separates intrinsic critique from prompt-induced behavior across three prompting settings, and experiments on 16 frontier LLMs reveal a shift toward rejecting valid queries under explicit critique instructions.The framework evaluates whether models critique flawed queries without suppressing SQL generation for originals.Source: §1 · p. 2

2 Related Work

Prior work studies unanswerable and ambiguous NL2SQL queries, as well as robustness and abstention in broader LLM settings. The paper positions PremiseSQL as a unified evaluation of diverse premise errors and premise-aware prompting.

2.1 Unanswerable and Ambiguous Queries in NL2SQL

  • Prior NL2SQL work covers intention classification, ambiguity and unanswerability, infeasible queries, and semantic translation errors, but lacks a fine-grained, comprehensive examination of diverse premise errors.PremiseSQL addresses this gap with a unified premise-error taxonomy.Source: §2.1 · p. 2

2.2 Robustness and False Premise Handling in LLMs

  • Beyond NL2SQL, robustness benchmarks study unanswerable questions and abstention; PremiseSQL adds diverse NL2SQL premise errors and examines prompting while paired originals distinguish critique from instruction-driven rejection.Its comparison with related benchmarks spans flawed-premise reasoning, abstention, SQL generation, fine-grained errors, and prompt-aware evaluation.Source: §2.2 · pp. 2–3, Table 2 · p. 3

3 Method

The method formalizes premise critique as validating a query against its database context and choosing whether to generate SQL or abstain. PremiseSQL organizes flawed queries into four categories, constructs verified query pairs, and evaluates behavior under three prompting settings.

3.1 Task Formulation

  • For flawed queries, premise critique requires a context-supported explanation of the flaw and abstention; for originals, models should generate SQL without attributing unsupported flaws.The benchmark’s reference label is determined by the query and database context, not by the prompting instructions.Source: §3.1 · pp. 3–4

3.2 Premise Taxonomy

  • The taxonomy distinguishes Expression Error, Premise Contradiction, Missing Premise, and Beyond Ability as four kinds of unanswerable or ill-posed queries.The categories cover unclear intent, logical or schema incompatibility, omitted requirements, and requests beyond SQL or database capabilities.Source: §3.2 · p. 4

3.3 Benchmark Construction

  • PremiseSQL pairs 1,800 taxonomy-guided flawed queries with their unmodified originals from Spider 1.0 and BIRD / BIRD-filtered.GPT-5.1 generates constrained perturbations, which undergo automatic evaluation by Gemini-2.5-Pro-Thinking followed by human review; the BIRD portion of E3.2 uses BIRD for gold schema annotations.Source: §3.3 · p. 4, Figure 2 · p. 4

3.4 Prompting Protocol

  • Base gives no warning about possible query flaws, Implicit asks models to assess validity, and Explicit directly asserts a flaw and requests critical examination.The Explicit assertion is supported for positive samples but false for negative samples.Source: §3.4 · p. 5

3.5 Evaluation Framework

  • Positive samples are evaluated by SQL abstention behavior, flaw detection, and type identification; negative samples are evaluated by SQL generation and validity judgment.SQL Behavior uses a three-level ordinal score from 0 to 2; GEN measures generation coverage, while VAL measures explicit valid judgments.Source: §3.5 · p. 5

4 Experiments

Across three prompting settings, the experiments find that current models do not reliably assess query premises: stronger critique instructions improve flawed-query scores but also increase false rejection of originals. The analyses show variation across models and error types, connect validity judgments to SQL behavior, and characterize how failures shift with prompting.

4.1 Experimental Setup

  • The evaluation covers 16 frontier LLMs, pairing each of 1,800 perturbed queries with its original across three prompting settings, for 172,800 responses.Models run at temperature 0 with default API configurations.Source: §4.1 · p. 5

4.2 Main Results

  • Under Base prompting, aggregate flawed-query SQL, DET, and TYPE scores are 0.177, 0.319, and 0.311, while original-query GEN is 0.988.Even the strongest Base detector, Claude Opus 5.5, detects only 0.492 of flaws.Source: §4.2 · pp. 5–6
  • Across Base, Implicit, and Explicit prompting, models do not reliably assess premises; explicit cues raise flawed-query scores but reduce SQL generation and validity judgments on originals.Table 3 reports SQL Behavior, Detection, and Type Identification on flawed queries, and SQL-generation rate and validity judgments on originals.Source: §4.2 · pp. 5–6, Table 3 · p. 6

4.2.1 Model Comparisons

  • Model gains vary by family and setting: newer or larger models show partial, setting-dependent improvements rather than a consistent rise in spontaneous premise critique.Under Implicit prompting, DeepSeek V4.1 Flash has higher original-query GEN than V3.2 (0.926 versus 0.455), but lower flawed-query DET (0.625 versus 0.898).Source: §4.2.1 · p. 6

4.2.2 Validity Judgments and Behavior

  • Models often align SQL generation with their validity judgment, so incorrect judgments can produce either omitted SQL on originals or SQL for flawed queries.Under Explicit prompting, original queries account for 41.37% of invalid judgments.Source: §4.2.2 · p. 7

4.2.3 Automatic Evaluation and Reliability

  • Automatic evaluation shows strong human agreement on sampled outputs, with mean κ scores for Claude Opus 4.6 of 0.9701, 0.9545, and 0.9072 for SQL Behavior, Detection, and Type Identification.The corresponding GPT-5.4 scores are 0.9567, 0.9355, and 0.8188.Source: §4.2.3 · p. 7

4.3 Error-Type Analysis

  • Under Implicit prompting, contradictions are easiest to detect, while ambiguity and unclear intent are hardest; performance is cue-dependent rather than robust across flaw types.Contradiction DET and TYPE exceed 0.91; ambiguity and unclear-intent DET are below 0.25 and TYPE below 0.14.Source: §4.3 · pp. 7–8, Table 4 · p. 7

4.4 From Error Types to Failure Behaviors

  • Across prompting settings, assumption injection is the dominant failure mechanism, while stronger instructions shift residual failures toward overinterpreting limited schema or example evidence.M01 falls from 81.03% under Base to 31.12% under Explicit prompting, while M03 rises from 1.46% to 58.06%.Source: §4.4 · p. 8, Figure 4 · p. 8

5 Conclusion

The paper introduces PremiseSQL to evaluate premise critique in NL2SQL and reports that models often trust flawed premises, even under explicit scrutiny. Its scope and evaluation have stated limits.

5 Conclusion

  • PremiseSQL pairs 1,800 flawed queries across nine fine-grained error types with 1,800 originals, and experiments find that models struggle to assess premises reliably, especially subtle flaws.The benchmark evaluates Base, Implicit, and Explicit prompting settings.Source: §5 · p. 8

Limitations

  • The benchmark focuses on English NL2SQL, uses taxonomy-driven constructed errors, and relies partly on LLM judges, limiting generalization and leaving possible evaluation bias.The authors note that it may not generalize to multilingual or real-world interactive database settings, and constructed errors may not capture naturally occurring mistakes.Source: Limitations · p. 9

A.1 Dataset Composition

This appendix describes PremiseSQL’s composition and construction, its prompting and evaluation procedures, and human verification of the automatic evaluations.

A.1 Dataset Composition

  • PremiseSQL contains 1,800 flawed queries across four major categories and nine fine-grained subcategories, paired with 1,800 originals; each subcategory has 200 positive samples.Source proportions from Spider 1.0 and BIRD / BIRD-filtered are not strictly identical.Source: §A.1 · p. 10, Table 5 · p. 10

A.2 Construction and Filtering

  • For eight perturbation-based subcategories, two-stage LLM verification and human review filter generated candidates; E3.2 is constructed separately and bypasses that filtering.Average rejection rates were 57.8% for LLM verification and 64.9% for human review.Source: §A.2 · pp. 10–11, Table 6 · p. 11

A.3 Generation and Review Prompts

  • Benchmark construction uses a generation stage to introduce one controlled premise flaw and a review stage to check candidates against category-specific validity criteria.The appendix illustrates the shared prompt structure with Ambiguity examples.Source: §A.3 · p. 11, §A.3.1 · p. 11, §A.3.2 · p. 11

B.1 Base, Implicit, and Explicit Prompts

  • Base, Implicit, and Explicit prompts share a schema-and-question input format while varying whether they ask for SQL generation, validity assessment, or scrutiny of an asserted flaw.The Explicit prompt asserts that the question is flawed; the Implicit prompt says it may contain errors.Source: §B.1 · pp. 11–12

B.2 Evaluation Rubrics

  • The evaluation rubric scores flawed-query SQL Behavior, Detection, and Type Identification, and scores original-query SQL generation and validity judgment.Type Identification is evaluated only when Detection is positive; VAL is available only under Implicit and Explicit prompting.Source: §B.2 · p. 12

B.4 Human Verification

  • Two graduate-level annotators independently reviewed 2,160 sampled outputs using the automatic evaluators’ rubrics and showed high agreement across the three metrics.Human Cohen’s κ was 0.9912 for SQL Behavior, 0.9632 for Detection, and 0.8759 for Type Identification.Source: §B.4 · p. 13

B.5 Robustness to Judge–Target Overlap

Robustness checks find that judge–target overlap does not materially change the aggregate results, leading models, or conclusions about prompting behavior.

B.5 Robustness to Judge–Target Overlap

  • Removing self-evaluation changes aggregate SQL, DET, and TYPE scores by less than 0.001 in every setting, while a family-disjoint check changes aggregate scores by at most 0.0016 and preserves the top-ranked model.Original and family-disjoint rankings have Spearman correlations from 0.944 to 1.000; GEN and VAL are unaffected because they come directly from structured model outputs.Source: §B.5 · p. 13

B.6 Paired Evaluation Design

The paired evaluation tests whether models respond differently to flawed queries while preserving correct validity judgments and SQL generation for originals. Separate measures distinguish flaw detection, explanation, abstention, validity judgment, and generation.

B.6 Paired Evaluation Design

  • Reliable critique should change responses to perturbed queries while preserving valid judgments and SQL generation for their originals.Base prompting usually answers both; stronger prompts increase abstention but Explicit prompting often suppresses original-query generation as well.Source: §B.6 · p. 13
  • Under Explicit prompting, LLaMA 4 Maverick has 0.974 GEN but only 0.363 VAL, while Qwen3.5-397B is low on both, exposing distinct patterns of original-query handling.Evaluating originals prevents either pattern from being mistaken for robust critique.Source: §B.6 · p. 13

B.7 Execution-Based Validation of SQL Generation

Execution-based validation examines whether models preserve SQL generation on original queries and how generation relates to execution accuracy. The results indicate that prompting-related accuracy declines are primarily associated with suppressed generation, not reduced correctness among generated SQL.

B.7 Execution-Based Validation of SQL Generation

  • GEN measures whether SQL generation is preserved on all 1,800 original queries; execution analysis uses 1,726 originals with reliably executable gold SQL.The other 74 originals are excluded only from execution analysis because their correctness cannot be determined reliably; EX treats abstention as incorrect, while EX | GEN measures accuracy among generated outputs.Source: §B.7 · p. 13, Table 8 · p. 14
  • The execution-based analysis calculates overall execution accuracy and accuracy conditional on SQL generation, with no SQL assigned execution equivalence of zero.The evaluation averages judgments from GPT-5.4 and Claude Opus 4.6.Source: §B.7 · pp. 13–14, Table 7 · p. 14
  • 0.990 Pearson’s r (p < .001) links GEN and EX across 48 model–setting combinations; from Implicit to Explicit, GEN changes correlate with EX changes at 0.992 (p < .001).Under Base prompting, GEN is near its upper bound, and its correlation with EX is weaker (r = 0.434, p = .093); EX | GEN changes only from 0.722 to 0.762 between Implicit and Explicit.Source: §B.7 · p. 14

B.8 Bootstrap Uncertainty Analysis

Paired bootstrap analyses support the DET–GEN trade-off and show that model comparisons depend on prompting setting; error-type differences also persist under query resampling. The mechanism analysis defines evidence-based response attributions, samples eligible failures, and reports consensus incidences.

B.8 Bootstrap Uncertainty Analysis

  • Explicit prompting increases DET and decreases GEN for every model; all 32 confidence intervals exclude zero and changes remain significant after Holm correction (padj < .004).The paired bootstrap resamples 1,800 aligned query instances; models and judges are treated as fixed, so intervals quantify uncertainty over benchmark queries.Source: §B.8 · p. 14, Table 9 · p. 15
  • GPT-6.1-Sol exceeds GPT-5.4 in Base DET by 0.156, but under Explicit prompting GPT-5.4 exceeds GPT-6.1-Sol by 0.050; the Base and Explicit differences are significant, while the Implicit difference is not.The Qwen3.5-35B DET advantage over 122B and 397B under Implicit prompting is significant, but the comparison is setting-specific; DeepSeek V4.1 Flash has higher GEN (+0.471) but lower DET (−0.274) than V3.2 under Implicit prompting.Source: §B.8 · pp. 14–15
  • Contradiction subcategories exceed expression-error subcategories by 0.695 in SQL Behavior, 0.730 in DET, and 0.807 in TYPE; all differences have p < .001.The results support the robustness of the finding that overt contradictions are easier than ambiguity and unclear intent; wider intervals for some missing-premise categories indicate greater query-level heterogeneity.Source: §B.8 · p. 15

C.1 Mechanism Definitions and Annotation Protocol

  • Attribution requires evidence-specific supported judgments from both judges, with deterministic checks validating quoted requirements, response transformations, and context evidence before labels are assigned.M01, M02, and M04 require exact requirement and response quotes; M03 requires an erroneous inference plus counterevidence from verified model-visible context.Source: §C.1 · p. 16
  • Mechanism labels are non-exclusive response-level attributions: M01 substitutes unsupported interpretations, M02 reinterprets contradictions, M03 misuses partial context, and M04 omits complete constraints.M01, M02, and M04 are mutually exclusive for the same atomic requirement under specified conditions; complete abstention alone is not M04, and ordinary SQL implementation defects are outside the taxonomy.Source: §C.1 · pp. 15–16

C.2 Eligibility and Sampling

  • The analysis samples 250 responses per dataset–model combination, yielding 8,000 observations; E3.2 is excluded because historical schema exposure cannot be recovered.Eligibility requires two primary evaluators to agree on all three fields, and sampling is stratified by prompting setting and source error category.Source: §C.2 · pp. 16–17
  • Across 8,000 sampled failures, strict two-judge consensus identifies M01 in 5,808 responses, M02 in 110, M03 in 860, and M04 in 69.Counts need not sum to the sample size because mechanisms are non-exclusive.Source: §C.2 · p. 17

C.3 Reliability and Human Validation

The paper reports agreement for human annotations and human–AI attributions, while noting that the covered mechanisms do not exhaust failure behavior. Analyses also find prompt-associated shifts in mechanism incidence and many-to-many links between source error categories and response mechanisms.

C.3 Reliability and Human Validation

  • Human annotators’ per-mechanism agreement has macro κ = .9434, and the human–AI macro agreement is .9212 for GPT-6.1-Sol and .8748 for Claude Opus 5.5.The human validation covers 300 responses; the study is assisted because the interface preselects a verdict when the attribution judges agree.Source: §C.3 · p. 17

C.4 Consensus Coverage

  • The four mechanisms are not an exhaustive partition: 714 of 8,000 sampled failures (8.93%) receive no mechanism label from either judge.Among those 714, 687 (96.22%) show SQL Behavior=0, Detection=0, and Type Identification=0, a recognition failure that does not identify the response mechanism.Source: §C.4 · p. 18

C.5 Prompt-Type, Benchmark-Type, and Error-Type Analyses

  • From Base to Explicit prompting, M01 declines while M03 rises on both BIRD and Spider; these percentages describe failure composition, not unconditional error probability.M01 falls from 72.10% to 25.30% on BIRD and from 88.99% to 39.45% on Spider, while M03 rises from 1.94% to 64.99% and 1.03% to 48.14%, respectively.Source: §C.5 · p. 18
  • M02 is concentrated in internal textual contradictions and M04 is most frequent for question–schema contradictions, while M01 and M03 dominate other source categories.The source taxonomy and response mechanisms map many-to-many: one source category can yield multiple mechanisms, and a mechanism can recur across categories.Source: §C.5 · p. 18

D Representative Premise Error Examples

Appendix D illustrates PremiseSQL error examples and distinguishes neighboring categories by the source of indeterminacy. Appendix E and the accompanying prompt figures show model responses to flawed queries and original-query controls.

D Representative Premise Error Examples

  • The taxonomy separates ambiguity, contradiction, missing-premise, and beyond-ability examples, with neighboring categories distinguished by whether indeterminacy comes from the schema, query, omitted context, semantics, or SQL’s limits.For example, E2.2 changes a query against an unchanged schema, E3.2 withholds required schema context, and E4.2 concerns external facts absent from the database.Source: D Representative Premise Error Examples · pp. 18–19, Table 17 · p. 25

E Behavioral Case Studies

  • Across flawed-query case studies, models variously miss flaws, acknowledge them but still produce unsupported SQL, or detect the issue and abstain.Examples include silently choosing a mapping for ambiguity, treating contradictory predicates as ordinary conditions, substituting a proxy for missing semantics, and replacing forecasting with a historical count.Source: E Behavioral Case Studies · p. 19, Figure 9 · p. 26, Figure 12 · p. 29
  • Original-query controls show mixed judgments and SQL behavior under Base, Implicit, and Explicit prompting, including valid SQL generation and false invalid judgments or abstentions.For instance, one Base-prompted model separates answer existence from query validity, while an Explicit-prompted model accepts a false-invalid framing and abstains.Source: E Behavioral Case Studies · p. 19, Figure 22 · p. 39, Figure 23 · p. 40

Prompt for Ambiguity-type Generation

  • Figure 5 presents a representative prompt for generating ambiguity-type instances.Source: Figure 5 · p. 20

Prompt for Ambiguity-type Review

  • The review materials include representative ambiguity-review and automatic-evaluation prompts, plus a failure-attribution prompt that appends instance-specific fields as structured input.The final decision rubric evaluates mechanisms independently, subject to same-target exclusivity rules.Source: Figure 6 · p. 21, Figure 7 · p. 22, Table 16 · p. 23

Related papers

Loading 2610.22158v1…