PaperlayerView exact revision on arXiv

Source-linked AI summary

Does a Governed Confidence Score Know When a Text-to-SQL Answer Is Right? An Exploratory Study on BIRD and a Pre-registered Confirmation on Spider

Raghu Venkat, Tricha Anjali

arXiv:2610.17256v1cs.AI

TL;DR

This paper asks whether a designed confidence score can identify correct text-to-SQL answers, where fluent, confident outputs may answer a different question from the one asked. It audits the score and tests a frozen, pre-registered agreement-first rule. The revised rule improves ranking and core-answer coverage on Spider as it did on BIRD, while the value of added checks depends on dataset and correctness definition.

  • Problem

    Fluent answers can use valid queries yet answer a different question, making it important to assess whether a confidence score distinguishes correct text-to-SQL answers.

    Source: §1.2 · p. 3

  • Method

    The paper audits a designed score and tests a frozen, pre-registered rule that uses agreement with the shown answer as its score and deterministic checks as vetoes and caps.

    Source: §12 · p. 18

  • Results

    The frozen agreement-first rule improved ranking and coverage of core-answer correctness on Spider as it had on BIRD, and agreement-based scores outperformed the designed score under every tested labelling protocol.

    Source: §12 · p. 18

  • Takeaways & Limitations

    The gain owes little to the choice of agreement statistic, while added checks cost unqualified coverage and their ranking benefit depends on dataset and correctness definition.

    Source: §9 · p. 16, §11 · p. 18

    Labels are model-judged, the two labellers differ by about ten accuracy points on the same answers, and the hand check is not yet done.

    Source: §11 · p. 17

Abstract

from arXiv · show

Conversational analytics lets people ask questions of records spread across many systems and get answers in plain language; it is useful only if each answer can be trusted and explained. A production system of this kind attaches a confidence score to every answer it writes over a database and uses it to show the answer, show it with a warning, or withhold it. The score's weights, caps and thresholds were set by design. This paper audits that design. On the 1,534 questions of BIRD dev, answered by GPT-5.1 with a benchmark-adapted production prompt and five samples per question, the designed score separates right from wrong answers weakly (AUROC 0.624, 95% CI 0.595-0.652) and withholds 35.7% of correct answers. Nearly all of its ranking signal comes from agreement among the samples, and most of it is lost where agreement is combined by a minimum with a checklist that drops to zero on any unsupported claim, including claims in commentary. We froze a rule in which agreement with the shown answer is the score and deterministic checks act as vetoes and caps, and pre-registered a test on Spider dev before any Spider answer existed. The primary hypothesis held: AUROC rose from 0.566 to 0.725 (paired difference +0.159, 95% CI +0.097 to +0.221), and correct answers withheld fell from 49.0% to 8.8% with PASS accuracy 95.7%. Under one matching policy the choice of agreement statistic matters little; the gain comes from how agreement enters the score. Whether added checks help depends on how correctness is defined, while they always cost unqualified coverage. A pre-registered empty-result check did not meet its effect-size criterion on Spider. Correctness here means the core answer; labels are model-judged, and a blinded two-person audit, not yet complete, will be reported in a revised version.

1 Introduction

Questions about records scattered across systems can require weeks of manual reconciliation and still produce wrong answers. This study audits a designed answer score and gate on text-to-SQL benchmarks, focusing on whether checks and traces help distinguish supported answers from confident misreadings.

1.1 The problem: one question, many places

  • Answering across systems can take weeks of reconciling tables, keys, units, and definitions, and the result can still be wrong.The paper illustrates the problem with chemical records split across transaction, inventory, safety, storage, and site systems.Source: §1 · p. 2

1.2 Conversational analytics helps, if it can be trusted

  • A fluent answer may use a valid query yet answer a different question, so the system needs traceable evidence and an answer-by-answer decision to show, warn, or withhold.The paper’s setting uses external checks on the answer and its trace, followed by a gate that names its reason.Source: §1.2 · p. 3, Figure 3 · p. 3

1.3 What can go wrong, and what was designed to catch it

  • The score was designed to catch failures including invalid tables or joins, stand-in values, unsupported written claims, incomplete answers, and inconsistent samples.A few failures force the score below the abstain line rather than being averaged away.Source: §1.3 · p. 4

1.4 This study

  • The study audits scoring and gating on structured text-to-SQL benchmarks, not document extraction, source reconciliation, or context-aware answer generation.The authors ask whether the score predicts correctness, whether checks catch planted failures, and whether each component earns its place.Source: §1.4 · p. 4
  • On BIRD, the useful signal came from sample agreement, and the authors then froze a revised rule for pre-registered testing on Spider, which did not inform its design.The paper also decomposes score components and their coverage costs, replicates on BIRD under the same harness, and measures labeller effects.Source: §1.4 · pp. 4–5
  • The contribution is an audit of a designed score built around execution-based consistency, rather than a new agreement statistic.The study reports the audit as measured, including components that did not work.Source: §1.4 · pp. 4–5

2 Related work

Related work treats agreement among sampled outputs as a black-box confidence signal, while also studying verification, selective prediction, calibration, and pre-registration. The paper places its audit within those lines of work.

Consistency as confidence

  • Sampling outputs and measuring agreement is an established confidence signal, including self-consistency, SelfCheckGPT, and semantic entropy.The designed L2 term uses semantic entropy with meaning fixed by execution.Source: §2 · p. 5

Confidence for text-to-SQL

  • For text-to-SQL, prior work found consistency-based methods strongest among black-box approaches and query execution useful; another study reported verification signals above consistency, with results not directly comparable [13].Those studies used different labels, models, and prompts.Source: §2 · p. 5

Selective prediction and calibration

  • The paper reports gate coverage and accuracy alongside AUROC, and defines calibration error following Guo et al. within selective classification.Selective prediction by answering or abstaining has also been studied under domain shift for question answering.Source: §2 · p. 5

Pre-registration

  • The authors pre-register because the rules under test were chosen using exploratory data, and fixing hypotheses and analysis before confirmatory data is uncommon in this literature.Source: §2 · p. 5

3 The score

The paper defines the sampled-answer setting, the designed weighted score and its checks, then proposes frozen revisions centered on agreement with the shown answer. Its analysis identifies a blind spot in entropy-based agreement and specifies how the gate applies checks.

3.1 Setting and notation

  • With K = 5 independent samples, sample 0 is the answer shown and scored, while the other samples contribute only to agreement.A label marks whether the scored answer is right.Source: §3.1 · p. 5

3.2 The designed score (bench-1.0)

  • The designed score averages measured terms with weights 0.30 for GROUNDED, 0.20 each for SCHEMA, SAFETY, and RETRIEVE, and 0.10 for COMPLETE, then applies caps and a three-way gate.On these benchmarks RETRIEVE is unmeasured, so weights renormalize over the remaining terms; the gate passes at IES ≥ 0.80, flags at 0.60 ≤ IES < 0.80, and abstains otherwise.Source: §3.2 · p. 6
  • GROUNDED checks tables, join-key overlap, and returned stand-in values, while SCHEMA caps uncited numbers and COMPLETE measures question-part coverage.A missing table, low-overlap join, or returned stand-in value can trigger a hard cap.Source: §3.2 · p. 6

3.3 What the entropy term cannot see

  • The proposition states that L2 is invariant to sample order, so the same cluster split receives the same value regardless of the scored sample’s position.The proof relies on cluster sizes being unchanged by reordering.Source: §3.3 · p. 7
  • L2 uses execution-based semantic entropy, but for K = 5 a four-to-one split yields L2 ≈ 0.69 whether the shown answer is in the majority or the minority.The term measures how settled samples are overall, not whether they settle on the answer being shown.Source: §3.2 · p. 6, §3.3 · p. 7

3.4 The frozen rules

  • The other frozen v2 rule replaces L2 with A, limits checklist and unsupported-claim penalties to issues about the answer rather than commentary, and caps answers with COMPLETE < 0.5 at 0.79.The frozen v3 rule adds an empty-result signal, with one signal capping at 0.79 and two or more at 0.59.Source: §3.4 · pp. 7–8
  • The frozen primary rule uses agreement with the shown answer, A, as the score, with deterministic checks acting as vetoes or caps.Missing tables, bad joins, and returned stand-ins are vetoes; uncited numbers, answer-related judge issues, low completeness, and thin evidence are caps. With five samples, four agreeing and no check firing passes; three flag; two or fewer abstain.Source: §3.4 · p. 7

4 Method

The study compares a production-prompted text-to-SQL system on BIRD and Spider, using model-judged answer correctness and standard discrimination and calibration metrics. BIRD analysis informed frozen rules that were then tested on Spider under a pre-registered plan.

4.1 Data

  • BIRD dev has 1,534 questions across 11 SQLite databases with evidence hints, while Spider dev has 1,034 questions across 20 databases without hints.Source: §4.1 · p. 8

4.2 System under test

  • GPT-5.1 answered with a lightly trimmed production prompt, database tools, and five sampled answers per question; GPT-5 mini judged without seeing references.Source: §4.2 · p. 8

4.3 Correctness labels

  • Identical-result-set grading marks 54.2% of BIRD answers correct but can reject acceptable answer forms and trust flawed reference SQL.The study instead uses GPT-5.4 to judge correctness against the question, hint, reference and generated SQL/results, and written answer; unclear or ambiguous cases are excluded.Source: §4.3 · p. 8
  • Correctness means the returned rows and reported figures answer the question; unsupported commentary is recorded separately, and model rulings can move results.Source: §4.3 · p. 8

4.4 Metrics

  • AUROC, paired-bootstrap AUROC differences, expected calibration error, and Brier score assess score performance; ties count one half in AUROC.Intervals use 2,000 answer-level bootstrap resamples; the exploratory fitted variant uses whole-database cross-fitting across 20 random splits.Source: §4.4 · pp. 8–9, Table 1 · p. 9

4.5 Order of work

  • BIRD exploration and rule selection preceded freezing and pre-registering hypotheses before Spider answers existed; the frozen bundle was then evaluated on Spider.The BIRD-derived rules are in-sample on the exploratory run; the harness was rebuilt and validated against stored records before confirmation and replication.Source: §4.5 · p. 9

5 Exploratory results on BIRD

On BIRD, the designed score gave limited information below its pass gate and withheld many correct answers. Agreement carried most of the ranking signal, while unsupported-claim handling degraded it; planted-error detection was stronger than detection of real errors.

5.1 E1: the designed score

  • 35.7% of correct answers and 43.7% of wrong answers were withheld by the designed score, despite 77.9% pass accuracy versus 72.9% overall accuracy.The hard cap assigned 534 answers a score of 0.59, and 70.4% of those answers were correct.Source: §5.1 · p. 9
  • 0.9–1.0 scores were right 84.4% of the time, versus 62.5% at 0.8–0.9 and 36.5% at 0.7–0.8, so the score was not a probability.Source: §5.1 · p. 10

5.2 E3: what each piece contributes

  • 0.721 AUROC for agreement alone contrasted with 0.572 when L2 was removed from SAFETY; other component removals or replacing the minimum with a mean changed AUROC by at most 0.007.On these clean BIRD databases, grounding terms were almost always 1 and carried no ranking information.Source: §5.2 · p. 10
  • Unsupported-claim handling was the largest observed degradation: L1 and the minimum could drive SAFETY to zero, and commentary claims made up 360 of 455 capped answers.The cap fired on 30.0% of answers, which were right 71.9% of the time; answers with commentary claims were right 75.6% of the time.Source: §5.2 · p. 10
  • Intent mismatch, bad joins, and wrong formulas corresponded to 32.5%, 46.8%, and 44.7% correctness, respectively, but each check fired on 5.5% of answers or fewer.Source: §5.2 · p. 10

5.3 E2: planted errors

  • The deterministic checks withheld every planted error of their own kind, while judge-based checks withheld 70–90%; the designed score withheld 43.7% of real wrong answers.All 480 unchanged controls reproduced their stored scores, but fault injection overstates real-world detection and should be reported beside it.Source: §5.3 · pp. 10–11

6 Pre-registered confirmation on Spider

The authors fixed the scorer, rules, thresholds, exclusions, and four hypotheses before generating Spider answers. On Spider, the primary comparison favored agreement-first, while the empty-result hypothesis failed and the agreement-only versus agreement-first ordering reversed one step.

6.1 What was fixed in advance

  • The scorer, tested rules, thresholds, truncation policy, exclusion rule, and four hypotheses were fixed before any Spider answer existed.The hypotheses specified a paired AUROC comparison, correct-answer withholding and pass accuracy, an empty-result effect, and an ordering among scores.Source: §6.1 · p. 11

6.2 Results

  • Of 1,034 Spider answers, 51 were excluded as unclear or question-ambiguous; the remaining 983 had 92.6% labelled accuracy and 73 labelled wrong answers.The test had fewer labelled wrong answers than the pre-registration’s 100-answer point below which intervals were considered too wide; the intervals were wide, and Spider was easier than BIRD for this model.Source: §6.2 · pp. 11–12
  • +0.159 was the Spider AUROC gain for agreement-first over the designed score; it withheld about a fifth as many correct answers without losing pass accuracy.The gain exceeded the in-sample BIRD effect (+0.106) and was close to the BIRD rerun (+0.165).Source: §6.2 · p. 11, Table 4 · p. 12
  • H3 failed: on Spider, empty final results were right 73.0% of the time, unlike the low right-answer rates on both BIRD runs.The authors did not establish why and reported the pre-registered failure without adjusting the rule.Source: §6.2 · p. 12
  • H4 reversed one step: agreement alone ranked above agreement-first on Spider, opposite to BIRD; the proposed explanation involving caps is untested.The authors suggest caps may lower some correct answers even when all five samples agree, but mark this explanation as untested.Source: §6.2 · p. 12
  • The answering, judging, and labelling prompts were rebuilt, while the frozen rule code, thresholds, truncation policy, and exclusion rule remained unchanged.The pre-registered prompt was unavailable, and evaluation ran at a different commit that carried the same bundle.Source: §6.2 · p. 12

7 Replication and labeller sensitivity

On the rebuilt harness, agreement-first improved AUROC over the designed score in the BIRD rerun. The comparison’s direction persisted under a rebuilt label set, while execution-match labels changed the relative ordering of agreement-based scores on Spider.

BIRD under the rebuilt harness

  • +0.165 was the agreement-first AUROC gain over the designed score on the BIRD rerun, from 0.589 to 0.755.The 95% CI was +0.129 to +0.204; agreement-first withheld 11.4% of correct answers, versus 47.5% for the designed score.Source: §7 · pp. 12–13, Table 6 · p. 13
  • The rebuilt BIRD harness had 54.4% execution match, close to 54.2% on 29 September, and 82.5% accuracy on 1,424 labelled answers.The authors describe the rerun as a check that rebuilding the harness did not change the BIRD picture.Source: §7 · pp. 12–13

Sensitivity to label version

  • Relabelling the September BIRD answers preserved the direction of the agreement-first gain, despite accuracy rising from 72.9% to 82.6%.The rebuilt labeller was more lenient, moving 114 answers from wrong to right and 18 the other way; agreement between two model-generated label sets is not independent validation.Source: §7 · p. 13

Under execution-match labels

  • Under execution-match labels, Spider AUROC was 0.551 for the designed score, 0.651 for agreement alone, and 0.669 for agreement-first.On the same 983 answers kept by the model-judged protocol, the respective AUROCs were 0.557, 0.639, and 0.663; the authors report the reversal in point estimates but no paired intervals for those differences.Source: §7 · p. 13

The hand check

  • The planned human audit will review random samples, all Spider answers with references judged wrong, and answers where scores make different gate decisions.Targeted groups will be reported separately from random samples, and adjudicated labels and every disagreement will be reported.Source: §7 · pp. 13–14

8 What the gain is made of

Most ranking loss occurs when the designed score takes the minimum of agreement and a checklist that can drop to zero on unsupported commentary. Agreement-first improves ranking, while added checks and threshold choices trade unqualified coverage against warnings or abstentions.

Where the designed score loses its ranking

  • The large ranking loss occurs at SAFETY = min(L1, L2), where an unsupported claim, including commentary, can make L1 zero and discard L2’s agreement signal.Hard caps also tie 437 Spider answers at exactly 0.59.Source: §8 · p. 14

The statistic under one matching policy

  • Under one matching policy, changing the agreement statistic moved AUROC by less than 0.02; the gain over the designed score came from avoiding the minimum with the checklist.The relaxed matching comparison is secondary because relaxed matching is not transitive and its entropy clusters are formed greedily in sample order.Source: §8 · p. 14

What the checks do on top of agreement

  • On Spider, checks removed 166 answers from pass—157 correct and 9 wrong—left wrong answers withheld unchanged, and withheld 46 more correct answers; pass accuracy changed by +0.2 points (95% CI -0.5 to +1.0).On the BIRD rerun, checks removed 264 answers from pass (46 wrong), raising pass accuracy by +2.4 points (+1.2 to +3.7); 7 wrong and 22 correct answers moved directly from pass under agreement.Source: §8 · pp. 14–15
  • The checks mainly turn passes into warnings and cost unqualified coverage; these benchmarks do not establish whether they help when tables are missing, joins are ambiguous, or codes stand in for values.The benchmarks therefore leave open the record conditions the checks were designed to address.Source: §8 · p. 15

Accuracy and coverage

  • Across most threshold settings, agreement alone and agreement-first had higher accuracy than the designed score; the three were close at the lowest BIRD coverages.The agreement share yields only a few operating points, and Figure 4 plots answer accuracy against the share shown as thresholds are lowered.Source: §8 · p. 15, Figure 4 · p. 15

Calibration

  • Agreement-based scores were better calibrated than the designed score in every evaluation, although none was designed as a probability.The paper recommends fitting a probability on a deployment’s own labelled answers if one is wanted.Source: §8 · p. 15, Table 9 · p. 16

9 Discussion

The designed score’s ranking signal largely comes from sample agreement, but combining agreement with a checklist can discard useful information. The discussion sets limits on what agreement and checks establish, then proposes testing approved context and reporting narrower deployment claims.

9 Discussion

  • On Spider, agreement-first improved ranking and withheld far fewer correct answers than the designed score at the same or better pass accuracy, while the preference for agreement over entropy was modest.The discussion says the minimum-combined checklist lost ranking signal and fired on commentary; it also cautions that designed score 0.80 does not mean about 80% accuracy.Source: §9 · p. 16
  • Checks caught every planted error of their kind, but on real benchmark answers they rarely caught wrong answers missed by agreement and always cost unqualified coverage.On Spider, checks moved 157 correct and 9 wrong answers out of pass and withheld no additional wrong answer; their ranking benefit depends on dataset and correctness definition.Source: §9 · p. 16, §11 · p. 18
  • Unsupported commentary appeared in 23.8% of BIRD answers, and capping it only when it concerned the answer raised pass share from 58.5% to 78.4% at 77.9% pass accuracy.The frozen rules record unsupported commentary without changing the gate.Source: §9 · p. 16
  • When all five samples agree on a wrong answer, agreement cannot help; the pre-registered empty-result signal also missed its effect-size criterion on Spider.The paper identifies this as an open problem and says the next section argues it is a problem of context.Source: §9 · p. 16

10 Next steps: context

  • Agreement cannot resolve an intended business definition absent from the evidence, even when every sample makes the same choice and deterministic checks pass.The staging example contrasts storage-reporting quantities with issue transactions interpreted through product composition.Source: §10 · p. 17
  • The proposed held-out test crosses context for answer generation and checking in four conditions, holding the answer and trace fixed when isolating the checker.It will report correctness, wrong answers passed, correct answers withheld, and fully agreed wrong answers; context rules are frozen before evaluation.Source: §10 · p. 17
  • Until that test, deployment guidance is to pass only with at least four of five matching results and no fired check, name every flag or abstention, and avoid presenting the score as probability.Per-customer threshold calibration would require examples that experts approve or correct in that customer’s terms.Source: §10 · p. 17

11 Limitations •

  • The evaluation is limited to core-answer correctness with model-judged labels; its exploratory analyses are not pre-registered, and the Spider test has 73 wrong answers versus 100 deemed adequate.It uses one model and two benchmarks outside the system’s own domain, with one run per benchmark beyond a BIRD rerun.Source: §11 · pp. 17–18

12 Conclusion

The conclusion reports that the designed score did not hold up as designed, while a frozen agreement-based rule improved ranking and core-answer coverage on Spider as it had on BIRD. It also records the study’s competing interests and makes the evaluation materials and frozen analysis reproducible.

12 Conclusion

  • The frozen, pre-registered agreement-first rule improved ranking and coverage of core-answer correctness on Spider as on BIRD, while added checks reduced unqualified coverage and the empty-result signal missed its Spider criterion.The improvement owed little to the agreement statistic; check benefits depended on dataset and correctness definition.Source: §12 · p. 18

Competing interests

  • The authors founded and work for the company developing the tested system; the scorer, rules, and pre-registration were fixed before confirmatory answers, and all results were reported.Source: Competing interests · p. 18

Reproducibility

  • The frozen scorer bundle, scripts, result files, labels, and pre-registration are released with commit history, and the BIRD evaluation was reproduced on a second host with identical reported scores.The production system prompt is not released, but its benchmark trimming specification records kept and removed line ranges and reasons.Source: Reproducibility · p. 19

A Prompt sections removed for the benchmark

The appendix documents the benchmark prompt trim and reference-scorer fixes, then uses a staging case to show how a valid query can return the wrong quantity when records encode different meanings.

A Prompt sections removed for the benchmark

  • The benchmark prompt retained core SQL, interpretation, verification, and response rules while removing tool and document features unavailable in the benchmarks.The trim is specified by line ranges and reasons, and the script stops if the source prompt changes.Source: A Prompt sections removed for the… · p. 20

B Planted errors

  • The planted-error set covers fake tables, weak joins, stand-in values, unsupported numbers, shifted dates, wrong aggregates or intent, and incomplete answers.Source: B Planted errors · p. 20

C Fixes to the reference scorer before the BIRD run

D A worked case from a staging implementation

  • The staging appendix describes generalized tables and simplified queries; its figures are session-derived, outside the study measurements, and not independently reconciled.Source: D A worked case from a staging… · p. 21

The records

  • Storage-reporting tables record signed transaction deltas by site and chemical, while issue records link transactions through inventory items and products to safety-data-sheet composition rows.Source: The records · p. 21

First answer: the wrong quantity

  • The model read “used” as a storage-reporting question and queried negative water deltas, returning annual pound totals from those tables.The query grouped the selected storage records by calendar year.Source: First answer: the wrong quantity · p. 21
  • The returned totals were the weight of products in which water was listed, not the weight of water in what was issued, despite valid tables and returned-cell numbers.Repeated runs made the same choice.Source: First answer: the wrong quantity · p. 21

Second attempt: no matching records and an explicit limitation

In the staging case, an issue-data query returned no rows, so the system withheld a quantity rather than reporting zero. An expert-approved usage definition then redirected the query, but the resulting total was not independently reconciled.

Second attempt: no matching records and an explicit limitation

  • With explicit instructions to use issue data, the query returned no rows, and the answer reported the checked tables and row counts, flagged a likely broken join path, and withheld an unsupported quantity.The CAS number existed under WATER, and the checked issue tables contained over nine million transactions.Source: Second attempt: no matching records… · p. 22

The approved definition and the answer under it

  • The expert-defined measure followed issue transactions through each product’s composition rows; under that definition, the fiscal-2024 query returned 2,618.94 pounds of water.The earlier calendar-year figure measured a different quantity and period, so the two totals are not directly comparable.Source: The approved definition and the answer… · p. 22

Other issues the staging sessions surfaced

  • The case also surfaced identifier variation, misleading table names, signed quantities, and unresolved site codes as practical interpretation hazards.For hexavalent chromium, which CAS numbers count was an organisational decision not stated by the schema.Source: Other issues the staging sessions… · pp. 22–23

What the case shows and what it does not

Related papers

Loading 2610.17256v1…