PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Harness: End-to-End Optimization of Context Artifacts for Enterprise Text-to-SQL

Kate Gwimm, Carson Eisenach

arXiv:2608.22830v1cs.AIcs.LG

TL;DR

Enterprise Text-to-SQL must select evidence from complex production schemas, but the relative value of optimizing context artifacts versus the retrieval harness is unclear. This paper recovers query DAGs from production SQL, distills historical usage into reusable SQL reference cards, and compares graph and retrieval interventions. Context-artifact optimization yields larger internal AST gains than harness optimization, while public BEAVER results are more mixed and directional.

  • Problem

    Enterprise Text-to-SQL requires selecting evidence from thousands of tables and intermediate views because no model can ingest a full enterprise catalog at once.

  • Method

    The paper recovers query DAGs from production SQL and optimizes a distillation function that converts historical query profiles into reusable SQL reference-card artifacts.

  • Results

    Context-artifact optimization produces larger internal AST gains than harness optimization, with approximately 12% versus 3% for Sonnet and 25% versus 12% for Qwen.

  • Takeaways & Limitations

    Retrieved knowledge-base context is especially valuable when added to a full oracle graph, supporting offline improvement of context before inference.

  • Takeaways & Limitations

    The internal benchmark relies mainly on structural proxies and LLM judgment, while execution accuracy is available only for a smaller executable cohort.

Abstract

from arXiv · show

Deploying LLMs for enterprise Text-to-SQL is bottlenecked less by the model than by what context reaches it: business logic spans thousands of tables, and no model can ingest a full catalog at once. We argue that the most effective place to intervene is therefore the \emph{knowledge-base context} the model consumes, and that this context should be \emph{constructed} from historical usage rather than tuned for as a fixed input. Using a query-DAG decomposition--the same family of intermediates that enterprise benchmarks like BEAVER annotate, here recovered from production SQL--we compare the value of oracle query graphs versus retrieved knowledge-base context. In this ablation, retrieved knowledge-base context provides the largest marginal improvement when added to the full oracle graph. Building on this, we optimize a distillation procedure that turns historical query profiles into reusable SQL reference cards. On a benchmark of 5176 production queries from a major online retailer, optimizing these context artifacts yields larger gains (${\sim}12$--$25\%$ AST similarity) than optimizing the retrieval harness (${\sim}3$--$12\%$). On the public BEAVER benchmark, which lacks the production-usage signals available in our internal setting, the picture is more mixed: table cards alone perform about the same as raw historical SQL. The best optimized variant retrieves both cards and raw SQL, scoring $9.00\%$ versus $6.33\%$ (p-value $0.12$) for the comparable baseline on a held-out $N{=}300$ subset, using retrieved context and harness changes but no agentic loop.

1 Introduction

Enterprise Text-to-SQL is bottlenecked by selecting useful context from complex production schemas, so this work optimizes reusable context artifacts constructed from historical SQL usage.

  • Approach: The paper constructs reusable SQL reference-card artifacts from raw production traces instead of treating context artifacts as fixed inputs.This complements work that optimizes already-built knowledge bases or accumulates artifacts with hand-designed update rules.
  • Problem: Enterprise business logic spans thousands of tables and intermediate views, while accuracy degrades with longer inputs despite large nominal context windows.Academic benchmarks often use substantially simpler schemas than production data lakes.
  • Problem: The central bottleneck is deciding which evidence to surface for a query because no model can ingest a full enterprise catalog at once.
  • Approach: Query-DAG decomposition makes subtasks such as table identification independently measurable and supports ablations of table linkage, output schemas, and graph structure.The DAG supervision is recovered from production SQL rather than hand-labeled.
  • Findings: On the internal benchmark, context-artifact optimization produced larger relative AST gains than prompt/tool harness optimization for both Sonnet and Qwen.The reported comparisons are approximately 12% versus 3% for Sonnet and 25% versus 12% for Qwen.

2 Query-DAG Supervision Framework

The framework represents production SQL as query DAGs with verifiable intermediate structure, then compares oracle graph fidelity with retrieved knowledge-base context as distinct optimization levers.

  • Query-DAG representation: Each production query is parsed into a DAG whose nodes are CTEs or subqueries, whose edges encode dataflow, and whose annotations include descriptions and output schemas.Input schemas record source tables or upstream node outputs read by each node.
  • Query-DAG representation: Production query graphs average approximately 7 intermediate steps and reference approximately 5 source tables, making them more complex than typical academic schemas.
  • Query-DAG supervision: DAG intermediates can be checked against ground truth without executing full queries, enabling nested controls that add linkage, output schemas, and full graph structure.The levels are I1 NL-only, I2 plus input linkage, I3 plus output schemas, and I4 plus full graph inputs.
  • Evaluation: The benchmark scores AST and string similarity with complexity-weighted subquery comparisons, plus DAG-edge linkage, LLM-judge semantic similarity, and execution accuracy on a subsample.
  • Diagnosis: +0.093 AST similarity for Sonnet and +0.092 for Qwen resulted from moving from linkage to the full oracle graph, while retrieved context remained competitive with weaker oracle views.For Sonnet, NL+RAG scored 0.278 versus 0.248 for I2 and nearly matched I4 at 0.341.
  • Diagnosis: 0.241 AST similarity for Sonnet and 0.267 for Qwen were the largest jumps in the table when retrieval was added to the full graph.The corresponding changes were 0.341→0.582 for Sonnet and 0.291→0.558 for Qwen.

3 Optimizing Context Artifacts

The paper treats retrieved knowledge-base content as an optimization target rather than a fixed input, separately from the retrieval and generation harness. It distills historical query traces into reusable table artifacts and optimizes their construction through held-out evaluation and targeted failure feedback.

  • Retrieved knowledge-base context provides the largest marginal gain, including when added to the full oracle query graph.This motivates optimizing retrieved content itself rather than treating it as fixed.
  • The agent exposes three optimization surfaces: knowledge base K, harness H, and instruction prompt P.The knowledge base contains historical table queries and synthesized table summaries; leakage is prevented by excluding the evaluated query.
  • For each table, selector f filters historical queries into evidence, and summarizer LLM produces a reusable table artifact Kt.The optimized summaries augment the knowledge base and improve the evidence available to the agent.
  • The method uses production SQL as its distillation source internally, while BEAVER uses benchmark SQL, schema metadata, and intermediate annotations as a weaker public proxy.The method does not require proprietary traces, but its strongest setting uses a real production corpus.
  • The search loop mutates one surface, rebuilds candidate artifacts, scores held-out examples, and accepts improvements or reverts them.Candidate artifacts are rebuilt only for relevant tables during search, while reported results use real retrieval end-to-end.
  • Failure feedback attributes incorrect predictions to the selector, summarizer prompt, or harness, enabling targeted subsequent mutations.The diagnoser examines failed examples, produced artifacts, injected context, and selected evidence, then aggregates missing-information categories.

4 Empirics

The experiments compare harness and knowledge-base context optimization on internal production queries and BEAVER, finding larger internal gains from optimized SQL reference cards but mixed external transfer.

  • Internal production benchmark: 5176 production queries and two models are evaluated with real retrieval, comparing baseline and optimized harnesses crossed with baseline and optimized knowledge-base context.Internal similarity metrics use the production benchmark, while execution accuracy is reported on a separate executable cohort of n=102.
  • Harness optimization: +3% and +12% AST similarity result from harness optimization for Sonnet and Qwen, respectively, alongside table-selection recall gains of +4% and +10%.Accepted changes primarily modify retrieval through semantic and keyword searches over queries and table documentation.
  • Context-artifact optimization: ∼12% and ∼25% relative AST-similarity gains result from optimized SQL reference cards for Sonnet and Qwen under the baseline harness.The cards combine verbatim SQL fragments, join recipes, filter templates, and example CTEs, with roughly 60% of tokens devoted to concrete SQL examples.
  • Combined optimization: Combined optimization roughly matches but does not exceed the better single surface, with content-only configurations showing larger execution-accuracy point estimates on the n=102 cohort.Execution estimates are directional because confidence intervals overlap within each model block.
  • Remaining limitations: Persistent schema- and instance-linking failures remain after optimization, which mainly reduces retrieval-phase errors.The broader study is limited by structural-proxy scoring internally, an underpowered N=300 public subset, weak performance on deeply compositional queries, and a single-call retrieval-only regime.
  • BEAVER validation: 9.00% versus 6.33% execution accuracy is obtained by optimized cards+raw context versus the comparable pre-optimization harness on held-out BEAVER questions.The comparison uses the same 300 questions, generator, one generation call, and no agentic loop; p=0.12 makes it directional.
  • BEAVER validation: 6.67% versus 6.33% shows cards alone performing similarly to raw-query retrieval, while cards+raw achieves the best score; BEAVER lacks production-usage signals.The authors characterize this transfer check as encouraging but non-decisive.

A Extended related work

Related work spans schema-aware Text-to-SQL, historical workload signal, learnable prompts and harnesses, enterprise-oriented benchmarks, and concerns about imperfect evaluation.

  • Schema-aware Text-to-SQL: Schema-aware parsers make schema structure and schema linking explicit, while later systems separate schema linking from SQL-skeleton prediction and move more structure into prompting.Examples include RAT-SQL, IRNet, and RESDSQL.
  • Historical workload signal: Historical workload methods exploit prior queries because they reveal common join paths and schema semantics absent from table names alone.Related systems also construct reusable knowledge bases or store prior execution traces as structured semantic memories.
  • Learnable pipelines: Prompt, program, and harness optimization work treats language-model pipelines as learnable computational objects.The cited approaches optimize instructions, demonstrations, or compound-agent programs using search or natural-language feedback.
  • Enterprise benchmarks: Spider, BIRD, Spider 2.0, and BEAVER progressively move evaluation toward larger schemas, external knowledge, realistic workflows, and enterprise use cases.The internal benchmark targets a similar production-data-lake regime using private data.
  • Evaluation: Execution accuracy can penalize semantically valid alternatives, reward structurally wrong queries, and change model rankings, especially for long enterprise queries.The paper therefore combines structural and judge-based proxies internally with execution-graded BEAVER as an external check.

B Benchmark construction details

The internal benchmark is constructed from validated production query profiles, canonicalized into query DAGs, and annotated to support measurable context and graph ablations.

  • Profile selection: 5176 queries are selected from an approximately 100K-profile corpus using business-domain relevance, latest-version retention, and successful execution within three years.The profiles come from an enterprise workload-orchestration system and contain production SQL plus usage metadata.
  • Canonicalization: Canonicalization normalizes SQL identifiers and aliases, standardizes temporary-table creation, and flattens nested subqueries before structural comparison.These transformations are intended to produce uniform representations without changing query semantics.
  • Query-profile DAG: The query DAG uses source tables as inputs, temporary tables and CTEs as internal nodes, the final SELECT as sink, and column-level lineage as edges.This graph supplies the verifiable intermediates and fidelity levels used in the study.
  • Annotation: Qwen Coder 3-30B generates natural-language descriptions for DAG nodes and intents, while the original human-authored SQL remains the gold target.Only natural-language labels are model-generated.
  • Graph-structure ablations: The DAG enables ablations over intent and output schema, table linkage, node descriptions, node output schemas, and full inter-node linkages.These inputs correspond to progressively richer graph information for diagnosing context requirements.

C Confidence intervals for the internal benchmark

Confidence intervals quantify uncertainty for internal benchmark cells, with larger effective samples for structural metrics than for execution accuracy.

  • Interval construction: 95% bootstrap percentile intervals are computed over per-task scores using 2000 resamples with seed 42 for continuous internal metrics.The metrics include table-selection recall and precision plus three end-to-end similarities.
  • Sample sizes: Effective per-cell sample sizes are 488–517 for table selection, 509–517 for end-to-end metrics, and 102 for execution accuracy.Table 4 labels the context baseline as handcrafted documentation and the optimized condition as SQL reference cards.
  • Interpretation: [0.461, 0.520] versus [0.521, 0.579] are the Sonnet AST intervals for baseline documentation and optimized cards, respectively, while EX intervals overlap within model blocks.The EX ordering is therefore consistent with structural metrics but cannot independently establish the ranking.

D Harness and prompt optimization: search space and accepted mutations

Harness optimization searches retrieval tools and prompts while holding the search procedure fixed. Accepted mutations broaden evidence gathering and adjust generation and table-selection instructions.

  • The harness search restricts mutations to retrieval tools H and instructions P within the same autoresearch loop used for artifact search.The search procedure is unchanged; only the optimized surface differs.
  • The best retrieval configuration searches dense and keyword indices over profiles and documentation instead of using one semantic search.The baseline uses k=10 semantic retrieval, whereas the optimized configuration increases independent evidence per call.
  • Prompt edits emphasize request coverage and prohibit guessing columns absent from retrieved evidence.
  • Table-selection prompts shift from precision-only selection toward recall-weighted selection of grounded candidates, retaining uncertain plausible source tables.These edits are reported to explain table-selection recall gains.

E LLM-judge versus AST-similarity correlation

The LLM judge and AST similarity move together on the internal benchmark. The judge preserves configuration ordering while recognizing semantically equivalent SQL that differs syntactically.

  • Judge score and AST similarity move together, supporting their joint use as semantic and structural evaluation signals.
  • The judge credits semantically equivalent queries that differ syntactically, complementing structural similarity measures.

F Our few-shot baseline versus the published BEAVER baseline

The reproduced BEAVER few-shot baseline scores 6.33% on a 300-question draw, compared descriptively with the authors’ 8.8% result from a different sample.

  • 6.33% (19/300) is the reproduced BEAVER few-shot baseline on the study’s 300-question stratified draw.
  • 8.8% is the benchmark authors’ reported result for the same generator family in setting 0 without oracle tables.
  • Because the question samples differ, the comparison is descriptive rather than paired.The published 8.8% lies within the reproduced result’s 95% confidence interval [4.1, 9.7].

G Failure analysis

Optimization primarily reduces retrieval and table-selection failures, while residual errors persist in schema linking, joins, filters, aggregations, and deeply compositional queries.

  • Residual difficulty concentrates in column-, join-, and composition-level structure after optimized cards improve table selection.
  • Internal benchmark: structural error categories: Optimized cards reduce wrong_tables and partial_tables failures, shifting errors toward near_correct predictions.
  • Findings: 45%→29% of failures is the reduction in Sonnet’s R1 insufficient-information errors, while execution accuracy rises from 25% to 30%.The change is consistent with optimized summaries surfacing correct table identifiers.
  • Findings: Schema linking remains the modal generation error at ∼42–44%, while instance linking rises from 41% to 52% in QC3N’s smaller residual failure set.
  • BEAVER: difficulty by query compositionality: BEAVER accuracy reaches 48% for single-level queries but collapses to ∼1.5% for deeply compositional nested-CTE queries across context conditions.The cited analysis identifies query structure rather than injected context as the limiting factor.

H Example context artifacts

The paper contrasts raw retrieved SQL with distilled table cards as reusable context artifacts for Text-to-SQL. The optimized card combines table purpose, join structure, filter patterns, representative SQL, and correctness rules, while raw SQL preserves an example query verbatim.

  • Aggregated table card (distilled): The distilled card summarizes historical queries into a reusable fixed structure containing purpose, join recipes, filters, representative SQL, and correctness rules.Its join recipes include cardinality annotations, while its filters record observed idioms and value domains.
  • Common Join Recipes + Cardinality: The card records that room-to-building joins are one-to-many and require aggregation rather than DISTINCT when grouping.It also documents a three-way rooms-to-building-to-address join pattern.
  • Frequent Filter Idioms + Value Domains: The card captures recurring filters such as BUILDING_TYPE = 'ACADEMIC', SITE = 'MIT', numeric access levels, and HAVING thresholds.These patterns are tied to observed example counts and value domains.
  • Representative Full-SQL Examples: The representative SQL examples show statistical aggregation by building, including area range, variance, standard deviation, filtering, grouping, and ordering.The examples also include a three-way join and safe division using NULLIF(AVG(r.AREA),0).
  • Correctness Rules: The correctness rules require NULLIF for coefficient-of-variation division, aggregation of FCLT_ROOMS when joined, grouping on building identifiers, and specified type conventions.They also state that COUNT(r.FCLT_ROOM_KEY) and COUNT(*) are interchangeable in the documented setting.
  • Raw retrieved query (no distillation): Raw retrieved SQL skips distillation and injects another question's gold query verbatim; Table 3 reports performance matching the distilled card within noise.Both raw queries and cards are mined from other questions' gold SQL under the stated leakage discipline, with identifiers and literals reproduced verbatim.
Loading 2608.22830v1…