PaperlayerView exact revision on arXiv

Source-linked AI summary

Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline

Haowei Liu, Hsin-Tai Wu, Yi Fang

arXiv:2609.30290v1cs.CLcs.LG

TL;DR

Production text-to-SQL pipelines use final LLM judges despite limited measurement against human labels, while full human review does not scale. This paper audits a deployed judge, tests replacements and ensemble routing, and reports that Qwen is a low-cost replacement while strong-judge unanimity routing improves agreement at partial coverage.

  • Problem

    Production text-to-SQL pipelines use final LLM judges whose agreement with human labels is often unmeasured, while full human review does not scale.

    Source: §1 · p. 1

  • Method

    The authors audit a deployed judge against human gold, test replacement judges and prompt variants, and evaluate ensemble routing.

    Source: §1 · pp. 1–2

  • Results

    κ = 0.04 for production gpt-4o-mini; Qwen3.6-27B reaches κ = 0.72 at ∼1/300 of Claude Opus 4.7’s per-call cost, while 3-strong unanimity routing reaches κ = 0.79 at 89.7% auto-coverage.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The authors recommend periodic calibration of deployed LLM judges against domain-specific human gold and report that ensembling pays off only when each judge is well-calibrated.

    Source: §10 · p. 7

    The headline gold was produced by the two authors; independent re-labeling supports that gold, but over-flag and bug-detect rates remain co-defined with them.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Production text-to-SQL pipelines often end with an LLM-as-judge whose agreement with human annotators has never actually been measured. When we checked ours, the deployed gpt-4o-mini judge agreed with two-author gold at only Cohen's kappa = 0.04 on a disagreement-enriched set and 0.42 on a uniform-random spot-check, over-flagging 77.1% of the human-FAITHFUL cases in the enriched set. Most of its over-flags trace back to a single mechanism we call GRADE-HALLUCINATION. A self-hosted Qwen3.6-27B replacement (kappa = 0.72) lands in the same range as Claude Opus 4.7 (kappa = 0.71); the head-to-head is underpowered at n = 96, but for the deployment decision that hardly matters, since Qwen costs roughly 1/300 as much per call. Ensembling does not help for free. Pairing the weak judge with a stronger one degrades agreement, whereas three strong judges under unanimity routing reach kappa = 0.79 at 89.7% auto-coverage. Applied out-of-domain, the same audit recipe flags 25.5% of BIRD-financial's expert-authored gold SQLs as candidate gold-SQL issues under our annotation protocol. Code and pre-registration are at https://github.com/JamesL404/synca-audit.

1 Introduction

Industrial teams build synthetic text-to-SQL benchmarks for proprietary databases, using staged filtering that ends with a faithfulness judge; full human review does not scale. The paper audits that judge and reports replacement and routing results, alongside a concrete failure mechanism.

1 Introduction

  • SynCA generates candidate questions, compares independent SQL results, debates unresolved disagreements, and applies a final faithfulness judge; its output gates analyst-assist and evaluation tools.The pipeline produced 1,482 accepted question–SQL pairs and a faulty judge can affect product-visible acceptance rates and the in-house fine-tuning corpus.Source: §1 · p. 1
  • κ = 0.04 for production gpt-4o-mini; 77.1% of human-FAITHFUL cases were over-flagged in the enriched set, mainly through GRADE-HALLUCINATION.The judge treats some previewed result rows as invalid grades even though the question did not ask to exclude them.Source: §1 · p. 1, Figure 1 · p. 2
  • Three competent judges with unanimity routing reach κ = 0.79 at 89.7% auto-coverage, while pairing the weak production judge with a stronger one degrades agreement.The weak-plus-strong pairing reaches κ = 0.57 versus κ = 0.63 for the stronger judge.Source: §1 · p. 2
  • Qwen3.6-27B reaches κ = 0.72 versus Claude Opus 4.7 at κ = 0.71, at about 1/300 of Opus’s per-call cost; the n = 96 head-to-head is inconclusive.The authors report the comparison as not measurably worse at their sample size, while noting the head-to-head is underpowered.Source: §1 · pp. 1–2

2 Related Work

  • GRADE-HALLUCINATION is a triggered SQL-faithfulness instance of judges imposing rubric-absent criteria, and the no-preview prompt variant partly fixes it.The no-preview variant raises κ from 0.04 to 0.37, but no tested prompt closes the gap to a stronger judge.Source: §2 · p. 2, §1 · p. 2
  • The paper audits a deployed text-to-SQL evaluator rather than proposing a new one, and reports that judge ensembles help only when each member is well-calibrated.This qualifies earlier positive ensembling results, which largely concern long-form generation.Source: §2 · pp. 2–3

3 The SynCA Pipeline

SynCA preserves production schema characteristics in synthetic data and filters candidate question–SQL pairs through consensus, debate, and faithfulness review. Its human-gold sets serve distinct calibration and adjudication purposes.

3 The SynCA Pipeline

  • SynCA’s 604,146-row table is a synthetic derivative preserving production schema, value distributions, and data-quality artifacts while independently synthesizing every row.The released table contains no names, phone numbers, account IDs, or join-keys to non-synthetic tables.Source: §3 · p. 3
  • Preserved schema traps include text-encoded booleans, raw-byte usage, highly NULL interest columns, and legitimate E and z codes that can trigger grade hallucination.The production schema treats E and z as valid categorical values.Source: §3 · p. 3
  • The four stages are question generation, multi-model consensus on matching result rows, debate over unresolved cases, and a final faithfulness judge.The final judge assigns FAITHFUL, MINOR-ISSUE, or UNFAITHFUL verdicts using the schema, question, SQL, and executed-result preview.Source: §3 · p. 3

4 Human Calibration Set

  • The 234-set is disagreement-enriched for calibration, the 200-question random spot-check supports a population faithful-rate estimate, and the 96-case active sample supports only Qwen–Opus adjudication.The three human-gold sets are explicitly non-interchangeable.Source: §3 · p. 3
  • Two authors annotated 234 pairs drawn primarily from judge-flagged cases, intentionally concentrating labels where the pipeline could plausibly be contested; 148 were false-flag corrections.The sample contains 192 FAITHFUL and 42 UNFAITHFUL cases.Source: §4 · p. 3
  • An independent blind SQL-literate annotator agreed with author consensus at κ = 0.85 (95.7% raw), though author-only annotation remains a load-bearing limitation.The independent annotator re-labeled all 234 cases.Source: §4 · p. 4

5 Judge Calibration

The production judge’s calibration differs sharply between disagreement-enriched and random samples, and its dominant false-flag mechanism is only partly prompt-fixable. Qwen and Opus are not distinguishable at the reported head-to-head sample size.

5 Judge Calibration

  • κ = 0.04 on the disagreement-enriched 234-set and κ = 0.42 on the 200-question random spot-check for gpt-4o-mini; Qwen and Opus fall in the same range on both cuts.The enriched set is conditional on production-judge flagging; the random spot-check is closer to typical deployment, so either cut alone is misleading.Source: §5 · p. 4
  • The post-hoc binary grade-hallucination split has κ = 0.47 and 68% second-coder recall; the taxonomy was not pre-registered.The 66% author-labeled share survives recoding, and both upgraded judges fix over 94% of these cases.Source: §5 · p. 4
  • The four-variant prompt sweep flips gpt-4o-mini from over-flagging to under-flagging, but no tested variant closes the gap to Qwen’s κ = 0.72.No-preview raises κ from 0.04 to 0.37 by removing the result-row preview; it recovers 93/98 author-labeled grade-hallucination false flags.Source: §5 · p. 4, Table 3 · p. 5
  • At n = 96, Qwen matches 57 strict-disagreement cases and Opus 39, but the head-to-head is statistically inconclusive (p = 0.082; κ-difference 95% CI [−0.198, +0.276]).Rejecting equality at α = 0.05 with power 0.8 for a true ∆κ = 0.10 would require n ≈ 380.Source: §5 · p. 5

6 Ensembling Cheap Judges

Ensembling improves selective agreement only when the judges are competent: weak-plus-strong pairings degrade, while three-strong unanimity routing abstains on disagreements. Cross-domain audits find candidate issues in expert gold and show that judge behavior shifts by domain and schema.

6 Ensembling Cheap Judges

  • A weak-plus-strong pair degrades agreement: AND gives κ = 0.572 versus gpt-5.4-mini alone at κ = 0.631, while OR reaches κ = 0.067.On 154 disagreement cases, gpt-5.4-mini is correct 142 times against 12, indicating that the weak judge’s noise dominates this pairing.Source: §6 · p. 5
  • κ = 0.788 at 89.7% auto-coverage for three-strong unanimity routing; the full-coverage majority rule reaches κ = 0.731.Routing gains the remaining 0.057 by declining to decide 10.3% of disagreement cases, so its selective-prediction κ is not comparable to a full-coverage score.Source: §6 · p. 5, Table 4 · p. 6

7 Cross-Domain Replication: BIRD and Spider

  • 21.7–25.5% of BIRD-financial’s 106 expert gold SQLs were classified as candidate gold issues under the authors’ protocol, not as a verdict that experts erred.Of 65 judge-flagged cases, 27 were bugs, 16 ambiguous questions, and 22 over-flags; an independent SQL-literate annotator confirmed 23 of the 27 bug labels.Source: §7 · pp. 5–6
  • Qwen catches 85% of 27 identified BIRD bugs at 7.5% over-flag, while gpt-5.4-mini catches 74% and more than doubles the over-flag rate.Rich-schema enrichment reduces gpt-5.4-mini’s over-flag rate from 17.0% to 11.3%, but reduces Qwen’s bug detection from 85% to 78%.Source: §7 · p. 6
  • Across four Spider databases, Opus has the lowest or tied-lowest over-flag rate in every cell; rich-schema helps gpt-5.4-mini but harms gpt-4o-mini on English schemas.The reported aggregate sparse-to-rich rates are 1.8→0.6% for Opus, 9.8→6.1% for gpt-5.4-mini, and 11.0→15.9% for gpt-4o-mini.Source: §7 · p. 6

8 Cost and Released Benchmark

The paper contrasts a low-cost Qwen replacement for routine traffic with a more expensive three-judge routing setup for higher-stakes use. It also reports the released benchmark’s operational and reweighted faithfulness estimates, validation, and artifact availability.

8 Cost and Released Benchmark

  • Qwen-only costs $0.10 per 1,000 questions versus $0.40 for gpt-4o-mini and about 1/300 of Opus’s $30.The paper presents this as the single-judge operating point once domain calibration is established.Source: §8 · p. 6
  • Three-strong unanimity routing costs $31 per 1,000 questions and reaches κ = 0.79 at about 90% auto-coverage, routing the remaining cases to humans.The paper frames this as a higher-precision option for high-stakes runs, evaluation harnesses, and training-corpus curation.Source: §8 · p. 6
  • 84.3% of SynCA-final is FAITHFUL by Qwen’s raw operational rate; the reweighted estimate for benchmark-quality comparisons is about 90.6%.The benchmark contains 1,249 FAITHFUL, 170 UNFAITHFUL, 53 MINOR-ISSUE, and 10 ERROR records.Source: §8 · p. 6
  • Qwen’s binary accuracy on the 200-question random spot-check is 98.0% (196/200), with κ = 0.935.The two authors’ pre-consensus agreement on the spot-check was κ = 0.87.Source: §8 · p. 7
  • On the dual-judged supplementary set, Qwen and Opus have 95.5% raw agreement (1,185/1,241) and κ = 0.728 [0.655, 0.791].The audit-pipeline code, pre-registration, prompts, decoding settings, and analysis scripts are available; datasets are deferred until post-publication for anonymization.Source: §8 · p. 7

9 Discussion

The production judge audit led to self-hosted Qwen for routine traffic and three-strong unanimity routing for high-stakes runs. The authors argue that audit methods transfer across domains, but judge over-flag rates do not, and note limits in the study’s scope and annotation.

9 Discussion

  • The audit recipe transfers across domains, but per-judge over-flag rates shift with schema and value-vocabulary, so judge calibration is deployment-specific.The authors re-audit against fresh domain-specific gold when the schema or model snapshot changes.Source: §9 · p. 7

10 Conclusion

  • κ = 0.79 at 89.7% auto-coverage is the reported result for three strong judges under unanimity routing; Qwen now handles routine traffic and this routing covers high-stakes runs.The n = 96 Qwen–Opus head-to-head remains inconclusive, but the authors say cost determines the deployment decision.Source: §10 · p. 7, §9 · p. 7

Limitations

  • The study covers one English-language telecom deployment over SQLite; cross-domain checks help, but do not remove the conditionality of which judge is best.Qwen’s Spider evaluation is restricted to concert_singer because of endpoint outages.Source: Limitations · p. 8
  • The headline gold was annotated by the two authors, while an independent re-labeling supports that it is not idiosyncratic to their pair.Over-flag and bug-detection rates nevertheless remain co-defined with the authors; no judge was fine-tuned on the 234-set.Source: Limitations · p. 8

A Extended discussion

The extended discussion defines calibration as agreement with humans on domain-specific gold, measured for a judge–prompt–domain combination. It also describes why three competent judges are preferred to adding a weak fourth, and records model-access and pipeline details.

A Extended discussion

  • Calibration here means agreement with humans on domain-specific gold, and is a property of the judge, prompt, and domain jointly rather than the judge alone.The same gpt-4o-mini model reaches κ = 0.04 with one prompt and κ = 0.37 with another; the authors call re-auditing on each new domain or prompt unavoidable.Source: A Extended discussion · p. 9
  • Adding gpt-4o-mini as a fourth unanimity judge drops auto-coverage from 89.7% to 29%, because it often disagrees with the three strong judges without enough gain in auto-decided κ to justify added review.The authors conclude that decorrelated competent judges help, while adding more weak judges does not.Source: A Extended discussion · p. 9
  • gpt-5.4-mini’s over-flag rate shifts from 4.7% to 17.0%, Qwen’s from 3.7% to 7.5%, and Opus’s from 1.6% to 7.5% across domains.The authors relate these shifts to schema and value-vocabulary.Source: A Extended discussion · p. 9

B Model versions and access details

  • OpenAI model snapshots are pinned, but Anthropic and Google do not expose dated identifiers, so their closed-judge κ values may drift even though the audit recipe is reproducible.Qwen3.6-27B-Instruct used public HuggingFace weights served via vLLM on a single A100 80GB.Source: B Model versions and access details · p. 10

C Pipeline statistics

  • SynCA contains 1,598 final records, including 1,482 accepted and 116 rejected; 885 accepted 3/3 consensus and 604 reached 2/3.The consensus stage uses gpt-4o, claude-sonnet-4.6, and gemini-2.5-flash with value-only row matching, followed by up to two debate rounds with o1 as meta-judge.Source: C Pipeline statistics · p. 10

D Cost breakdown and benchmark distribution

The appendix reports judge and routing costs, the accepted benchmark’s verdict distribution, calibration-set composition, and details of the false-flag taxonomy. It also qualifies that taxonomy as illustrative rather than robustly separated into categories.

D Cost breakdown and benchmark distribution

  • Qwen3.6-27B costs $0.10 per 1,000 questions, versus $30 for Claude Opus 4.7; three-strong unanimity routing costs $31 plus $15–30 of human labor.At an assumed $1.50/GPU-hour, Qwen rises to $0.50 per 1,000 questions and remains about 60× cheaper than Opus.Source: D Cost breakdown and benchmark… · p. 10
  • Of the 1,482 accepted records, 1,249 (84.3%) are FAITHFUL, 170 (11.5%) UNFAITHFUL, 53 (3.6%) MINOR-ISSUE, and 10 (0.7%) ERROR.The verdicts comprise 1,248 Qwen-judged records and 234 two-author human gold records.Source: D Cost breakdown and benchmark… · p. 11

E Calibration set stratification

  • The 234-set includes 148 false-flag corrections (63%), with 52 easy, 115 medium, and 67 hard questions; 124 cases were 2-of-3 consensus and 110 were 3-of-3.These cases were primarily drawn from production-judge flags, with a balance of accepted cases.Source: E Calibration set stratification · p. 11

F Failure-mode taxonomy: full categories and regex examples

  • Grade hallucination accounts for 66% of false flags in the keyword-pattern categorization, and both upgraded judges fix more than 94% of those cases.The six overlapping categories cover 78% of cases; the four minority subcategories are treated as illustrative groupings because second-coder labels do not separate them reliably.Source: F Failure-mode taxonomy: full… · p. 11

G H5 schema/preview ablation on gpt-5.4-mini

The supplied results examine schema and preview ablations, prompt variants, SQL-feature stratification, a pre-registered head-to-head, and learned routing. They show specific prompt and feature effects while finding the learned router’s small apparent advantage over unanimity routing to be within noise.

G H5 schema/preview ablation on gpt-5.4-mini

  • Removing schema from gpt-5.4-mini’s prompt drops κ by 0.20, while removing the executed-result preview drops it by about 0.05.The preview adds little calibration signal but triggers grade hallucination on the weaker judge.Source: G H5 schema/preview ablation on… · p. 11

H Prompt-ablation: variant prompts

  • The four gpt-4o-mini prompt variants all flip the failure mode from over-flagging to under-flagging, and none closes the gap to Qwen’s κ = 0.72.The no-preview variant removes the result-row preview; the variants also include chain-of-thought, rubric, and strict prompts.Source: H Prompt-ablation: variant prompts · p. 11

I SQL-feature stratification

J Active-sampled adjudication: pre-registration details

  • The 96-case Qwen–Opus head-to-head yields a κ-difference 95% CI of [−0.198, +0.276], which excludes neither direction beyond an absolute ∆κ = 0.20 effect.The pre-registered sample drew 182 Qwen-flagged and 200 random Qwen-FAITHFUL questions from a 382-question pool.Source: J Active-sampled adjudication:… · p. 11

K Learned routing as a negative control

  • The learned router reaches 91.0% coverage at κ = 0.790, only ∆κ = +0.002 and ∆cov = +1.3pp over unanimity routing’s 89.7% / 0.788, a difference within Monte-Carlo noise.On the 96-set it collapses to negative κ at every threshold; the authors read unanimity routing as close to this hypothesis class’s empirical Pareto frontier.Source: K Learned routing as a negative control · p. 12

L BIRD failure-mode patterns and recovered-vs-not-recovered

BIRD-financial reveals a recurring verification-demanding over-flag pattern, while Spider results and the uniform-random spot-check show domain-specific behavior and uncertainty in Qwen’s per-class estimates.

L BIRD failure-mode patterns and recovered-vs-not-recovered

  • All 7 recovered gpt-5.4-mini cases were verification-demanding, while 11 remained interpretive; its corrected over-flag rate was 17.0%, ∼3.6× SynCA-conditional.The schema could not address the 11 interpretive demands that stayed flagged.Source: L BIRD failure-mode patterns and… · p. 12

M Spider concert_singer per-judge results

  • Qwen’s sparse over-flag was ∼0 on concert_singer, while Opus had the lowest or tied-lowest rate in every (DB, schema) cell and an aggregate at or below 2% in both schemas.Qwen’s four rich-schema false positives on concert_singer were defensible strict interpretations, not grade-hallucination cases.Source: M Spider concert_singer per-judge… · p. 12, Table 6 · p. 12

N Reconciliation: 84.3% vs 90.6% in full

  • ≈90.6%: stratified reweighting of Qwen’s verdict distribution using per-class precision yields 0.853 × 1.0 + 0.043 × 1.0 + 0.104 × 0.10.The passage gives the equivalent calculation as 84.3 + 3.6 + 1.2 ≈ 90.6.Source: N Reconciliation: 84.3% vs 90.6% in full · p. 12

O 200-question spot-check per-class results

  • All four Qwen disagreements were UNFAITHFUL calls on human-FAITHFUL cases, making it mildly conservative; the 100% cells remain imprecise, with the MINOR-ISSUE interval reaching 83.9%.Table 7 reports per-class precision with Wilson 95% confidence intervals.Source: O 200-question spot-check per-class… · p. 12, Table 7 · p. 12

P Opus prompt-variation robustness check

Opus and Qwen prompt variants remain statistically overlapping on their tested sets, while optimization exposed a weak-pairing regression and a three-judge routing result.

P Opus prompt-variation robustness check

  • Opus κ across prompts: baseline 0.706, CoT 0.790, rubric 0.686, no-preview 0.638, and strict 0.731; all BCa 95% CIs heavily overlap.The variants were run against the same human gold under deterministic decoding.Source: P Opus prompt-variation robustness check · p. 13

Q Qwen prompt-variation robustness check

  • All four Qwen prompt-variation CIs overlap, and its verdicts are stable across prompt choice on the 234-set.The table caption reports BCa confidence intervals.Source: Table 8 · p. 13

R Optimization trajectory

  • The optimization trajectory’s key transitions were a two-judge regression to κ = 0.57 at run 2 and routing enabled by a third decorrelated judge.Figure 2 traces the best κ across seven inner-loop runs.Source: R Optimization trajectory · p. 13
  • gpt-4o-mini produced 148 false flags, while Qwen had the fewest errors of both types against human gold.Figure 3 presents the confusion matrices with FAITHFUL as the positive class.Source: Figure 3 · p. 13
  • Grade hallucination accounted for 66% of over-flag failures, and both upgraded judges fixed nearly every category.The figure describes the over-flag failure-mode taxonomy.Source: Figure 4 · p. 13
  • The learned router’s best point was 91.0% coverage / 0.790 κ, within Monte-Carlo noise of unanimity routing at 89.7% / 0.788.The comparison uses five-fold cross-validation on the 234-set for the learned router.Source: Figure 5 · p. 13
Loading 2609.30290v1…