PaperlayerView exact revision on arXiv

Source-linked AI summary

ModularSQL: A Runtime Guardrail for the Multiplicity Blind Spot in Text-to-SQL

Tianxin Zhou, Ruixi Lin

arXiv:2609.29573v1cs.CL

TL;DR

Set-based execution accuracy can collapse duplicate rows, so benchmark-correct Text-to-SQL may still have unsafe multiplicity structure. The paper measures this gap with Multiset-EX and introduces ModularSQL, a post-selection runtime guardrail. On DeepEye-SQL with Qwen3-Coder, it improves Multiset-EX while preserving Set-EX, supporting multiplicity-aware guardrails as a practical complement to existing pipelines.

  • Problem

    Set-based execution accuracy collapses duplicate rows, leaving multiplicity-sensitive execution errors unmeasured even when queries pass benchmark evaluation.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces Multiset-EX as a multiplicity-preserving diagnostic and ModularSQL as a post-selection guardrail that probes results and intervenes on flagged queries.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    65.86% to 67.75% (+1.89 pp) Multiset-EX improvement, with Set-EX preserved at 72.06%, is reported for ModularSQL atop DeepEye-SQL with Qwen3-Coder.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Benchmark execution accuracy does not necessarily imply execution-safe SQL, and lightweight multiplicity-aware guardrails are a practical complement to existing Text-to-SQL pipelines and evaluation practices.

    Source: §6 · p. 8

    The full system, including candidate-based LLM rescue, is evaluated only on the DeepEye-SQL reproduction; cross-pipeline transfer tests candidate-free components on one SQL per question.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Text-to-SQL systems are increasingly deployed on production databases, where queries that pass benchmark evaluation can still produce results that distort downstream workflows. Standard set-based execution accuracy (Set-EX) collapses duplicate rows and can therefore miss multiplicity errors, including missing DISTINCT, inflated aggregates, and Cartesian-style join explosions. We call this the Multiplicity Blind Spot (MBS) and introduce Multiset-EX, a multiplicity-preserving evaluation criterion that exposes such failures. Across released DeepEye-SQL artifacts from three backbones (Qwen2.5-Coder-32B, Qwen3-Coder-30B-A3B, and Gemma-3-27B) on executable BIRD-Dev N=1532, we find a consistent 5.81--6.79 pp gap between Set-EX and Multiset-EX. The gap is not specific to DeepEye-SQL: it persists on released DAIL-SQL+GPT-4 (5.22 pp) and BIRD GPT-3.5-turbo (3.39 pp) predictions. We further introduce ModularSQL, a lightweight post-selection runtime guardrail that probes executed results for multiplicity anomalies and applies deterministic patches or low-cost LLM rescue only to flagged queries. Integrated with DeepEye-SQL using Qwen3-Coder, ModularSQL preserves Set-EX at 72.06% while improving Multiset-EX from 65.86% to 67.75% (+1.89 pp). It flags 77 high-risk anomalies, while adding only $0.0076 in total LLM cost and 120 ms amortized latency per query. Cross-pipeline evaluation shows that the candidate-free detector and deterministic patches also transfer to independently released prediction sets. Overall, these results show that benchmark accuracy does not necessarily imply execution-safe SQL, and that lightweight, multiplicity-aware runtime guardrails can narrow this gap with modest computational overhead.

1 Introduction

Set-based execution accuracy can overlook multiplicity errors that make benchmark-correct SQL unsafe for production analytics. ModularSQL adds a post-selection guardrail, improving multiplicity-aware accuracy while preserving Set-EX.

1 Introduction

  • Duplicate tuples introduced or removed by SQL can distort aggregates, dashboards, and downstream analytical workflows despite benchmark correctness.Examples include missing DISTINCT, erroneous joins, and Cartesian-style explosions.Source: §1 · pp. 1–2
  • The Multiplicity Blind Spot (MBS) occurs when set-based evaluation discards duplicate counts, allowing queries that pass Set-EX to fail multiplicity-preserving comparison.Multiset-EX is introduced as a complementary diagnostic criterion, not a replacement for official benchmark metrics.Source: §1 · p. 2
  • 5.81–6.79 pp gaps between Set-EX and Multiset-EX appear across three DeepEye-SQL backbones and persist on DAIL-SQL+GPT-4 (5.22 pp) and BIRD GPT-3.5-turbo (3.39 pp).The DeepEye-SQL results use the executable BIRD-Dev subset (N = 1,532).Source: §1 · p. 2
  • 72.06% Set-EX is preserved while Multiset-EX rises from 65.86% to 67.75% (+1.89 pp) under ModularSQL atop DeepEye-SQL with Qwen3-Coder.The system flags 77 high-risk anomalies while adding $0.0076 in total LLM cost.Source: §1 · p. 2

2 Background and Related Work

Text-to-SQL pipelines use execution feedback for validity and repair, but their feedback may not monitor output multiplicity. The section distinguishes set-based evaluation from multiplicity-preserving comparison and situates the Multiplicity Blind Spot within that distinction.

2.1 Text-to-SQL Pipelines and Execution Feedback

  • DeepEye-SQL’s candidate generation, revision, and selection stages use execution feedback, but validity-oriented signals do not explicitly monitor output duplicates or join-amplified row counts.The workflow includes 12 candidate queries per question, iterative repair, and tournament selection.Source: §2.1 · pp. 2–3
  • ModularSQL complements execution-guided repair and multi-candidate refinement by targeting the residual 6.20 pp Multiset-EX gap after DeepEye-SQL revision and selection.It closes 30.5% of that gap post hoc while preserving Set-EX; the paper does not evaluate an additional full repair round.Source: §2.1 · p. 3

2.2 From Set-EX to Multiplicity-Preserving Evaluation

  • BIRD’s set-collapsing protocol creates the Multiplicity Blind Spot, whereas Spider’s test-suite evaluator compares multisets; the issue is protocol-specific, not universal to Text-to-SQL evaluation.Earlier critiques focused on semantic equivalence, an orthogonal concern.Source: §2.2 · p. 3
  • SQL’s duplicate-preserving execution can affect aggregation operators such as COUNT and SUM, unlike classical relational algebra’s set treatment of relations.Dayal et al. (1982) formalize duplicate-elimination control in an extended algebra.Source: §2.2 · p. 3
  • Set-EX compares result rows as sets, while Multiset-EX preserves duplicate counts and ignores row order, making Multiset-EX at least as strict as Set-EX.The implication is one-way: Multiset-EX = 1 implies Set-EX = 1, but not necessarily the reverse.Source: §2.2 · p. 3
  • The Multiplicity Blind Spot comprises queries that pass Set-EX but fail Multiset-EX because set projection removes the distinguishing duplicate information.The blindness belongs to the evaluator by construction; its incidence depends on the pipeline, and Multiset-EX is a complementary diagnostic rather than a replacement.Source: §2.2 · p. 3

3 ModularSQL Methodology

ModularSQL adds a post-selection guardrail that probes executed queries, leaves unflagged queries unchanged, and routes anomalies to targeted LLM or deterministic interventions.

3 ModularSQL Methodology

  • ModularSQL operates after S7 selects a query, probing its execution for multiplicity anomalies and intervening only on flagged queries.Unaffected queries pass through unchanged; flagged queries can receive deterministic patches or low-cost LLM rescue.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Runtime Probe Design

  • The runtime probe flags execution errors, empty results, or duplication ratios at or above 0.80.The ratio measures redundant copies among result rows; empty results are a conservative trigger, and semantic errors with normal-looking outputs are out of scope.Source: §3.1 · p. 4

3.2 Deterministic Patches

  • At duplication ratios ≥0.80, ModularSQL injects DISTINCT; for existing DISTINCT, it tests removal and commits when the resulting ratio is ≤0.10.The thresholds are asymmetric; with LIMIT, DISTINCT injection can alter even set-semantics results, though no patch-eligible query in the reproduction contained LIMIT.Source: §3.2 · pp. 4–5

3.3 Budget-Aware LLM Rescue

  • On 77 of 1,532 queries, the gated rescue added approximately $0.0076 total and used an LLM to select among upstream candidates using question, schema, and execution previews.If no candidate is selected, the original S7 prediction is retained.Source: §3.3 · p. 5

4 Experiments

Across the evaluated BIRD-Dev prediction sets, set-based evaluation consistently misses multiplicity-sensitive errors. ModularSQL improves Multiset-EX while preserving Set-EX in the DeepEye-SQL reproduction, with selective runtime interventions and limited overhead.

4.1 Experimental Setup

  • The evaluation uses 1,532 executable BIRD-Dev queries, excluding two gold queries that exceed the 5 s timeout.The main reproduction uses Qwen3-Coder-30B-A3B; cross-pipeline evaluation uses released DAIL-SQL+GPT-4 and BIRD GPT-3.5-turbo predictions.Source: §4.1 · p. 5

4.2 Cross-Backbone Evidence of MBS

  • 5.81–6.79 pp separates Set-EX and Multiset-EX across four DeepEye-SQL measurements on executable BIRD-Dev (N = 1,532).The measurements span three released backbone artifacts and the Qwen3-Coder reproduction.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Cross-Pipeline Transfer

  • The candidate-free detector and deterministic patches transfer to two released prediction sets, where Multiset-EX gains +10 and +8 queries without changing Set-EX.The candidate-based LLM rescue is not evaluated because those sets provide only one SQL per question.Source: §4.3 · p. 6, Table 2 · p. 6

4.4 Main Results and Ablations

  • 72.06% Set-EX is preserved while ModularSQL raises Multiset-EX from 65.86% to 67.75% (+1.89 pp) on the DeepEye-SQL reproduction.The two deterministic patches account for 24 of the 29 net Multiset-EX rescues; the LLM rescue contributes the remaining five.Source: §4.4 · p. 6, Table 3 · p. 6

4.5 Analysis of Runtime Interventions

  • Among 77 probe-flagged queries, 70 trigger on duplication ratio; LLM rescue raises Multiset-EX from 12/77 (15.6%) to 29/77 (37.7%) versus raw S7, adding approximately $0.0076 total LLM cost.End-to-end, rescue contributes net +5 Multiset-EX queries; the largest difficulty-stratum gain is +4.17 pp in the challenging stratum.Source: §4.5 · p. 7

4.6 Detector Diagnostics

  • The deployed trigger has precision 0.844 on DeepEye-SQL, 0.945 on DAIL-SQL+GPT-4, and 0.991 on GPT-3.5-turbo, while recall for MBS failures is 0.35–0.43.Semantic errors with normal-looking outputs remain invisible to runtime probing; the guardrail adds 120 ms per query amortized across the workload.Source: §4.6 · p. 7

5 Discussion

ModularSQL is presented as an incremental runtime layer that complements existing Text-to-SQL pipelines. The discussion pairs preserved benchmark Set-EX with a residual multiplicity gap and recommends Multiset-EX as a complementary diagnostic.

Deployment profile

  • The guardrail preserves Set-EX, probes only 5% of queries, and adds under one cent of LLM cost and 120 ms amortized latency per query in the reproduction.The authors say teams can adopt it without retraining the backbone or modifying upstream stages.Source: §5 · p. 8

Evaluation implications

  • The observed 30.5% reduction leaves a residual 4.31 pp gap, which the authors say is not exhausted by duplicate-elimination patterns alone.They propose pairing benchmark Set-EX with Multiset-EX to surface multiplicity errors relevant to downstream applications.Source: §5 · p. 8

6 Conclusion

The conclusion presents ModularSQL as a post-selection guardrail that partially mitigates multiplicity failures while preserving Set-EX. Its candidate-free components transfer to two other released prediction sets, with less than one cent of LLM cost and 120 ms amortized latency per query.

Conclusion

  • ModularSQL preserves the 72.06% Set-EX baseline and closes 30.5% of the reproduced MBS gap on the Qwen3-Coder DeepEye-SQL reproduction.Candidate-free components transfer unchanged to two additional pipelines’ released predictions.Source: §6 · p. 8

Limitations

The paper identifies limits in pipeline coverage, benchmark and engine coverage, patch scope, rescue prompting, threshold calibration, and direct repair comparisons.

Limitations

  • The full ModularSQL system, including candidate-based LLM rescue, is evaluated only on the DeepEye-SQL reproduction; other released sets lack candidate pools for testing rescue.The candidate-free detector and patches transfer to DAIL-SQL+GPT-4 and BIRD GPT-3.5-turbo predictions.Source: Limitations · p. 8
  • Evaluation covers only BIRD-Dev on SQLite, not Spider, BIRD-Test, or production engines such as PostgreSQL or BigQuery.Source: Limitations · p. 8
  • Deterministic patches cover DISTINCT injection and removal, leaving GROUP BY semantics, nested COUNT/SUM expressions, and JOIN topology rewrites out of scope.The authors say closing the residual 4.31 pp gap likely requires extending the patch family to those operators.Source: Limitations · p. 8
  • The rescue prompt does not direct the LLM to prefer candidates with fewer spurious duplicates or provide duplication-ratio statistics.Providing each candidate’s duplication ratio is described as a possible low-cost extension.Source: Limitations · p. 8
  • The P1 threshold τ = 0.80 was calibrated on DeepEye-SQL P1-fire cases; other artifact families or query distributions may need retuning.The reported sensitivity sweep has a plateau over τ ∈ [0.55, 0.80].Source: Limitations · p. 8
  • The study does not compare an additional repair or regeneration round on flagged queries, leaving direct comparison with execution-guided repair for future work.Source: Limitations · p. 8

A P1 Threshold Sensitivity

The appendix examines threshold choices for DISTINCT injection and removal, the rescue prompt, and safeguards on P2 rewrites. P1 has a broad useful threshold plateau, whereas P2 is actionable only at very low thresholds.

A P1 Threshold Sensitivity

  • τ = 0.80 sits atop a P1 threshold plateau from 0.55 to 0.80, with net at least +20 throughout and held-out net +17 across nine databases.Table 5 reports +22 on the 87-case threshold-selection subset; the end-to-end gain is +21 because the refined rule also triggers on one case outside that subset, causing a PASS-to-FAIL regression.Source: A P1 Threshold Sensitivity · p. 9, Table 5 · p. 9

B P2 Threshold Sensitivity and Asymmetric Signal

  • P2 fires five times at τ = 0.10, fixing three cases with no regressions; unlike P1, it is actionable only in the extreme tail, τ ≤ 0.20.Above that point, its fix-break tradeoff degrades sharply; the authors attribute the asymmetry to unnecessary DISTINCTs reflecting query-intent misclassification rather than mechanical row amplification.Source: B P2 Threshold Sensitivity and… · p. 10, Table 6 · p. 10

C LLM Rescue Prompt

  • The rescue is a single judge-style LLM query selecting from 12 candidate SQL queries using execution previews, without multiplicity-aware instructions or duplication-ratio statistics.The prompt uses temperature 0.0 and a 64-token limit; invalid selections fall back to the original S7 selection.Source: C LLM Rescue Prompt · p. 10

D P2 Implementation Safeguards

E Intervention-Order Ablation: v1 vs v2

The intervention-order ablation finds that rescue-before-patching outperforms allowing rescue to overwrite already-patched SQL. The nine-query difference isolates the cost of that overwrite ordering.

E Intervention-Order Ablation: v1 vs v2

  • The two intervention orderings differ by nine Multiset-EX queries: v1’s rescue on patched SQL loses four correct queries, while v2 adds five beyond patches alone.In v1, the rescue overwrites a patched fix with a worse candidate in four cases; v2 rescues the original S7 selection before patches refine it.Source: E Intervention-Order Ablation: v1 vs v2 · pp. 10–11, Table 7 · p. 11

F By-Difficulty Per-Stage Breakdown

The appendix breaks down intervention-order differences by difficulty, checks two implementations of multiset equality, and details cross-pipeline detector and patch behavior.

F By-Difficulty Per-Stage Breakdown

G Multiset Equality Implementation Check

H Cross-Pipeline Details

  • P1 fires on 56 of DeepEye-SQL’s 70 Cartesian-flagged queries; the 14 excluded cases contain LIMIT, and the patch has non-negative Multiset-EX net across all three systems.The smaller eligible pools are 49 for DAIL-SQL and 11 for GPT-3.5-turbo; neither has a Set-EX regression.Source: H Cross-Pipeline Details · p. 11
  • The candidate-free P1 guardrail preserves Set-EX and improves Multiset-EX by 0.65 pp and 0.52 pp on the two cross-pipeline systems.Across systems, the trigger distribution varies: execution errors or empty results total seven for DeepEye-SQL versus 514 for GPT-3.5-turbo, while Cartesian explosions range from 41 to 74.Source: H Cross-Pipeline Details · p. 11, Table 10 · p. 12

I Detector Diagnostics Tables

The appendix expands detector diagnostics with overall and per-trigger precision, MBS recall, and results across three evaluated systems. The results show a precision–recall trade-off and a boundary for runtime-only detection.

Loading 2609.29573v1…