PaperlayerView exact revision on arXiv

Source-linked AI summary

BudgetSchemaBench: A Budget-Swept Diagnostic for Schema Context in Text-to-SQL

Chen Shen

arXiv:2610.00092v1cs.CLcs.AIcs.DB

TL;DR

Agents working with large structured-data catalogs must choose schema under a token budget, while existing evaluations do not jointly sweep that budget and isolate schema representation. BudgetSchemaBench provides an execution-grounded diagnostic with mechanically derived labels and controlled schema-context conditions. Execution accuracy rises with budget under coverage-limited retrieval, and predictions can still name removed tables when required schema is absent.

  • Problem

    No public diagnostic fixes a large catalog, sweeps an explicit schema-context budget, and measures downstream execution accuracy while isolating schema representation.

    Source: §1 · p. 1

  • Method

    BudgetSchemaBench uses gold-SQL-derived relevance labels to evaluate three schema representations across four budgets and three table-availability conditions, with retrieval rankings fixed across representations.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    Execution accuracy rises with budget under coverage-limited retrieval; with gold tables removed, 94.6% of correct predictions name a removed gold table exactly.

    Source: §4 · p. 6, §6 · p. 9

  • Takeaways & Limitations

    When retrieval is coverage-limited, execution accuracy is more sensitive to schema budget than to the tested serializations.

    Source: §6 · p. 9

    The evaluation uses a BIRD snapshot chosen as a tractable pilot, and larger catalogs remain future work.

    Source: §3 · p. 4

Abstract

from arXiv · show

Data agents over structured sources must fit database schema into the model's context window. Large catalogs can span many databases and thousands of columns, so cost constraints may require choosing between table coverage and serialization detail well before the context window is full. We introduce BudgetSchemaBench, an execution-grounded diagnostic for this setting. Its construction derives relevance labels mechanically from gold SQL, without human- or LLM-authored ground truth. Using a pooled 80-database catalog, we sweep four schema-context budgets and compare three representations while keeping each retriever's table ranking fixed. A source-namespace check rejects queries that obtain the correct result from the wrong database. The evaluation covers three conditions: end-to-end retrieval; frozen-gold, in which the required tables are guaranteed; and a probe that removes those tables. For the primary solver with raw serialization, raising the budget from 2.5% to 50% of the catalog improves execution accuracy on 1,279 held-out questions by 18 percentage points under lexical retrieval but only 3 under dense retrieval; the dense retriever already finds most required tables at the smallest budget. When the required tables are removed, 94.6% of correct predictions name one of them exactly, consistent with reconstruction of absent schema from parametric knowledge. For the two main solvers in the frozen-gold condition, the three representations differ by at most 2 percentage points, and the widest paired 95% confidence interval bounds the difference within +/-4 points. We observe the same qualitative patterns with one reasoning model from a different family. When retrieval is coverage-limited, execution accuracy is more sensitive to the schema budget than to the tested serializations. The diagnostic and the code used to construct and evaluate it are publicly available.

1 Introduction

Large structured-data catalogs force agents to choose which schema to include under a token budget, but existing work lacks a shared diagnostic for measuring downstream execution accuracy across budgets while isolating representation. BudgetSchemaBench provides such a diagnostic and methodology, with a held-out evaluation examining how results respond to budget, table availability, and representation.

1 Introduction

  • At realistic scale, catalogs spanning many databases and thousands of columns can exceed practical schema-context budgets, requiring agents to select and serialize schema.The paper frames this as a distinct typed, hierarchical, abstractable “context beyond the window” regime.Source: §1 · p. 1
  • Existing approaches address parts of the problem, but no public diagnostic combines a fixed large catalog, a swept schema budget, execution accuracy, and representation isolation.Schema linking, union retrieval, and prompt compression each address parts of the setting.Source: §1 · p. 1
  • BudgetSchemaBench contributes a reusable execution-grounded diagnostic and construction methodology, then evaluates how it responds to budget, table availability, and representation changes.The evaluation uses a held-out complement of N=1,279 for end-to-end and frozen-gold conditions, with the remove-needed-table probe evaluated separately.Source: §1 · pp. 1–2

2 The missing evaluation object

Prior work covers large-catalog retrieval, controlled context-length evaluation, schema representation, and schema selection, but the supplied studies do not combine a swept schema budget with representation isolation and execution-grounded accuracy. Table 1 positions BudgetSchemaBench as combining these evaluation elements with a source-namespace contract and a remove-needed-table probe.

2 The missing evaluation object

  • Large-catalog retrieval work reports retrieval recall or evaluates retriever methods, while controlled context-length studies evaluate prose rather than executable relational catalogs.The cited work does not measure execution accuracy across a swept schema budget while isolating representation.Source: §2 · p. 2
  • Schema-representation studies vary format within full pipelines or use representation diversity for ensembling, rather than jointly controlling budget and isolating representation in execution-grounded accuracy.Large-catalog realism benchmarks increase schema difficulty, scale, and cost but do not sweep a controlled budget axis or isolate representation.Source: §2 · p. 2
  • Prior work on enrichment and irrelevant-schema tolerance assumes the full catalog fits or perfect recall, unlike the budget-binding setting where some schema must be omitted.The paper identifies coverage as the binding variable for evaluating schema representations in this setting.Source: §2 · p. 2
  • Table 1 presents BudgetSchemaBench as combining a fixed large catalog, a controlled swept budget, and execution-grounded representation isolation, alongside namespace validation and a remove-needed-table probe.The paper describes this combination as, to its knowledge, the first of its kind.Source: Table 1 · p. 3

3 BudgetSchemaBench

BudgetSchemaBench builds a pooled BIRD catalog and evaluates schema context under explicit token budgets, using mechanically derived relevance labels and controlled retrieval, serialization, and execution checks. Its protocol compares end-to-end retrieval, guaranteed gold-table, and removed-gold-table conditions.

3 BudgetSchemaBench

  • The public release includes build scripts, label schema, namespace evaluator, serializers, prompts, token logs, validation reports, and a one-command reproduction procedure.The artifact is released under a BSD 3-Clause license with a content-hashed manifest and pinned dependencies.Source: §3 · p. 3
  • Four absolute token caps span 2.5%, 10%, 25%, and 50% of the rich catalog; tables are packed whole in rank order, skipping those that do not fit.The catalog pools BIRD dev and train databases (80 total); the 1,279-question held-out complement comes from BIRD-dev’s 11 databases, and larger catalogs remain future work.Source: §3 · pp. 3–4
  • Name-only lexical and dense rankers keep table rankings identical across representations, so richer descriptions cannot improve retrieval scores or rankings.Representation-aware retrieval is outside the study’s scope.Source: §3 · p. 4
  • Relevance labels come from parsing gold SQL for relevant tables, columns, and roles, with a parser-coverage gate and no human- or LLM-authored ground truth.The parser-coverage gate excludes 0% of questions on the substrate after a CTE-aware extractor fix.Source: §3 · p. 4
  • The source-namespace contract counts a prediction as correct only when it matches the result and reads tables from the gold database namespace.The primary EX metric adds lineage to BIRD’s result-equivalence metric; the lineage check changed the classification of 0.3% of otherwise result-correct sampled predictions.Source: §3 · pp. 4–5
  • The calibration-set error study found roughly a third of pooled solver errors arose from projection or format mismatches or errors in the gold SQL, rather than schema understanding.These errors affect all representations equally, while widening confidence intervals and depressing absolute EX.Source: §3 · p. 5
  • The protocol tests end-to-end retrieval, a frozen-gold condition that guarantees required tables, and a probe that removes those tables.Frozen-gold compares net utility under a fixed budget because longer serializations leave room for fewer non-gold distractors; a fixed-table-set control remains future work.Source: §3 · p. 5
  • Raw, enriched, and hybrid representations are evaluated using EX brackets, table recall, token consumption, and collision-aware strata.Role-stratified column recall and a collision-strata breakdown remain future work.Source: §3 · p. 5

4 Construction validation and held-out-complement evaluation

Construction checks support the held-out evaluation, which shows that schema-budget gains are larger under lexical than dense retrieval when table coverage is limited. In frozen-gold tests, serialization differences are small, while correct predictions often name removed gold tables exactly.

Construction validation

  • Construction checks found no label-parser exclusions, gold invariance for 1,532/1,532 executable rewritten queries, and lexical recall rising from 0.75 to 1.0 across budgets.The catalog exceeds the largest tested budget, so no tested budget admits the full schema.Source: §4 · p. 5

Held-out-complement evaluation

  • The independently audited held-out complement contains 1,279 questions for end-to-end and frozen-gold evaluation; the removed-table probe covers 1,236 predictions.The full held-out complement is disjoint from development.Source: §4 · pp. 5–6
  • For gpt-5.4-mini, the large-minus-tiny EX gain is +0.18 under lexical retrieval versus +0.03 under dense retrieval.The evaluation uses raw serialization on the full held-out complement (N=1,279).Source: Table 3 · p. 6, §4 · p. 6
  • With gold tables removed, 94.6% (1,169/1,236) of correct predictions named a removed gold table exactly, consistent with reconstruction of absent schema.The authors interpret this as consistent with parametric memory, while noting exact-name overlap does not distinguish it from generic schema-name priors.Source: §4 · p. 6
  • In frozen-gold tests, raw, enriched, and hybrid serialization differ by at most |∆|=0.02, with the widest paired 95% CI bounding effects within ±0.04 EX.No contrast survives Holm correction; the authors report a bounded null at this scale, not established equivalence.Source: §4 · p. 6, Table 4 · p. 7
  • DeepSeek-V4-Flash reproduces the qualitative patterns, with large-minus-tiny changes of +0.18 under lexical and +0.06 under dense retrieval.This is one off-family replication, not evidence of broad cross-model generality.Source: §4 · pp. 7–8, Table 5 · p. 8

5 Limitations

The evaluation’s conclusions are bounded by its model and benchmark coverage, leakage audit, and retrieval-isolated design. It reports a bounded null for representation differences rather than claiming equivalence.

5 Limitations

  • Broader model and benchmark coverage remains future work; the evaluation includes one off-family reasoning model and a gpt-5.5 frontier slice of N=30.The collision-strata breakdown also remains future work.Source: §5 · p. 8
  • 94.6% of correct remove-gold predictions name removed gold tables exactly, but a small value-leakage residual remains and broader audit controls are future work.The authors interpret the probe’s nonzero performance as a likely leakage level.Source: §5 · p. 8
  • The widest paired 95% confidence interval is ±0.04 EX, exceeding the preregistered δ=0.03 margin, so the paper reports a bounded null without claiming equivalence.The authors do not report a formal two-one-sided-tests equivalence verdict.Source: §5 · p. 8
  • The database-clustered bootstrap covers only 11 source databases, and the primary solver’s single temperature-1 decode leaves decode-sampling variance unmodeled.The question-level and database-clustered budget analyses agree.Source: §5 · p. 8
  • Representation-independent rankers isolate representation effects on generation, so the study does not test representation-aware retrieval or enrichment when it can change retrieved content.The derivation-first labeling method also requires gold SQL, and its labels inherit non-minimal joins in source queries.Source: §5 · p. 8
  • The dense retriever’s near-1.0 table recall at the tightest budget may understate schema-selection difficulty; paraphrased queries and harder-to-disambiguate catalogs remain future work.Source: §5 · p. 8

6 Conclusion

BudgetSchemaBench is a validated, execution-grounded diagnostic for schema context under a budget, built with relevance labels derived from gold SQL. Its evaluation finds budget sensitivity under coverage-limited retrieval, schema-memorization-floor behavior when needed tables are removed, and bounded differences among tested representations when retrieval is isolated.

6 Conclusion

  • BudgetSchemaBench provides an executable diagnostic and curation methodology with relevance labels derived without human- or LLM-authored ground truth.The artifact is intended to help benchmark builders and method developers measure where schema-context budget is spent.Source: §6 · p. 9
  • Across its evaluation questions, execution accuracy rises with schema budget under coverage-limited retrieval, removed-table predictions are consistent with schema reconstruction, and tested representations differ within bounded limits in frozen-gold.The paper makes no claim that any tested retriever or representation is best.Source: §6 · p. 9
  • When retrieval is coverage-limited, execution accuracy is more sensitive to schema budget than to the tested serializations.Source: §6 · p. 9

A Unprefixed-catalog robustness

The unprefixed-catalog robustness check preserves the budget-dependent pattern, including a significant lexical gain, while dense retrieval remains near saturation. Removing database prefixes lowers EX by a small, uniform margin.

A Unprefixed-catalog robustness

  • The robustness sweep changes prompt names to bare table names, adding source-agnostic numeric suffixes for 146 colliding tables, while keeping retrieval and scoring in the union namespace.Predictions are translated back to the union namespace before the unchanged scorer.Source: A Unprefixed-catalog robustness · p. 11
  • Removing the <db> signal lowers EX by ∼0.03–0.05 across cells, but the budget effect does not depend on the namespace prefix.Source: A Unprefixed-catalog robustness · p. 11
  • Unprefixed lexical EX gain is +0.157 [0.132, 0.182] vs. prefixed +0.178; dense gain is +0.025 [0.003, 0.046] vs. +0.034.The unprefixed lexical large-minus-tiny confidence interval excludes zero; table recall is identical by construction.Source: Table 6 · p. 11, A Unprefixed-catalog robustness · p. 11
Loading 2610.00092v1…