Source-linked AI summary
HarnessSQL: Harness-Native Training for SQL Agents in Realistic Database Environments
Haolin Yang, Jipeng Zhang, Jian Xie, Shuaishuai Gong, Sirui Han, Yike Guo
TL;DR
Static SQL training does not capture the multi-turn interaction required by database agents working with live databases. HARNESSSQL trains agents inside an executable SQL harness using verified trajectories and execution-reward RL. It improves compact-model execution accuracy on Spider 2.0-SQLite and transfers to out-of-distribution interactive benchmarks.
Problem
Static question-to-query training contrasts with realistic database tasks that require multi-turn schema inspection, query execution, error diagnosis, and revision.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
The harness is part of the behavior distribution that training should reproduce, rather than merely an inference-time wrapper.
Source: §8 · p. 10The study focuses primarily on SQLite, while realistic benchmarks also use dialects such as PostgreSQL, Snowflake, and BigQuery.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Text-to-SQL models are commonly trained to map questions directly to static queries, whereas real-world database agents operate through stateful, multi-turn interaction with live databases -- inspecting schemas, executing probe queries, diagnosing errors, and revising hypotheses. This creates a critical train-deploy mismatch, as the execution harness that mediates this interaction is introduced only at inference time. To bridge this gap, we propose HarnessSQL, a harness-native post-training framework that preserves the full interaction structure throughout both supervised fine-tuning and reinforcement learning. HarnessSQL builds isolated, executable database environments paired with hidden execution oracles, rolls out teachers directly inside the target SQL harness, and retains only verified trajectories for full-sequence SFT, followed by execution-reward RL. Across Spider 2.0-SQLite, HarnessSQL dramatically boosts the execution accuracy of compact models, raising Qwen3-8B from 15.5% to 45.2% and Qwen3-14B from 22.2% to 54.8%, while transferring effectively to out-of-distribution interactive benchmarks such as BIRD-Interact and LiveSQLBench. Our findings demonstrate that training database agents directly within their execution harness is essential for mastering complex, long-horizon database workflows.
1 INTRODUCTION
Interactive SQL agents must explore live databases, use execution feedback, and revise queries, but conventional training often targets static SQL. HARNESSSQL aligns post-training with the runtime harness and substantially improves compact-model results.
1 INTRODUCTION
- Realistic database tasks require agents to inspect tables and schemas, execute intermediate queries, diagnose errors, and revise solutions over multiple turns.The needed information may not fit in a static prompt, motivating interactive database reasoning.Source: §1 · p. 2
- Static question-to-query training mismatches deployment, where agents must explore, execute, revise, and terminate through a concrete database harness.The paper frames interactive Text-to-SQL as long-horizon policy learning grounded in an executable harness.Source: §1 · p. 2
- HARNESSSQL combines executable database sandboxes, an SQL-specific harness, and verified full-interaction trajectories for SFT followed by execution-reward RL.The training targets exploration, execution, recovery, and submission, not only correct SQL output.Source: §1 · p. 2
- Execution accuracy on Spider 2.0-SQLite rises from 15.5% to 45.2% for Qwen3-8B and from 22.2% to 54.8% for Qwen3-14B after SFT and RL.The learned policies also transfer to BIRD-Interact Mini and LiveSQLBench Base-Lite.Source: §1 · p. 2
2 PROBLEM SETTING
The paper models interactive Text-to-SQL as a partially observed process mediated by a harness, then contrasts this with static SQL training that omits interaction trajectories and runtime feedback.
2.1 SQL AGENTS AS PARTIALLY OBSERVED POLICIES
- Interactive Text-to-SQL is modeled as a partially observed Markov decision process in which a task includes the question, database, harness, and hidden oracle.The agent acts from dialogue history and harness observations rather than directly observing the full database state.Source: §2.1 · p. 3
- Task success combines denotational execution equivalence against the hidden verifier with compliance to harness protocols and the step budget.Protocol checks include read-only constraints, valid action schemas, avoiding redundant queries, and termination within the allowed steps.Source: §2.1 · p. 3
2.2 THE TRAIN–DEPLOY MISMATCH
- Conventional post-training optimizes a golden query from a prompt and static schema, while reasoning-oriented methods still terminate without environment interaction.This leaves intermediate interaction states outside the training objective.Source: §2.2 · p. 3
- Static supervision omits live execution transitions and runtime observations, leaving agents without trained procedures for exploration, feedback handling, and termination.Harness-native training instead optimizes policies over trajectories collected in the deployment harness using execution-grounded rewards.Source: §2.2 · p. 3
3 METHODOLOGY
HARNESSSQL couples executable database environments and a specialized interaction contract with verified trajectory supervision, followed by reinforcement learning in the same harness.
3 METHODOLOGY
- The methodology couples hermetic sandbox construction, an SQL-specific interaction contract, verified trajectory synthesis, full-sequence SFT, and execution-reward RL.SFT and RL use the same harness environment and interaction protocol.Source: §3 · p. 4, §3.3 · p. 5
3.1 EXECUTABLE SQL ENVIRONMENT CONSTRUCTION
- Task synthesis extracts live-engine schemas, sample values, and valid join paths, then derives natural-language prompts and execution oracles from sampled executable SQL.Difficulty controls guide query sampling.Source: §3.1 · p. 4
3.2 SQL-SPECIFIC HARNESS ADAPTATION
- The specialized harness restricts agents to table listing, schema inspection, bounded read-only query execution, and terminal submission.Unrelated shell, filesystem, and code-runtime tools are disabled; middleware bounds observations and addresses repeated calls and long episodes.Source: §3.2 · pp. 4–5
3.3 HARNESS-NATIVE POST-TRAINING
- Teachers generate demonstrations inside the target harness without oracle access; only trajectories passing denotational verification and protocol checks are retained for full-trajectory SFT.The collection contains 2,512 verified episodes across 79 databases, and harness observations are masked from the loss.Source: §3.3 · p. 5
- RL continues in the identical harness, using hidden-verifier execution rewards and policy gradients across generated assistant segments.The reward is sparse and terminal, and the method preserves tool semantics, observation boundaries, and termination protocols across training stages.Source: §3.3 · p. 5
4 EXPERIMENTAL SETUP
Experiments train Qwen3-8B and Qwen3-14B with harness-native SFT and RL, then evaluate execution accuracy on one in-domain and two out-of-distribution interactive benchmarks.
4 EXPERIMENTAL SETUP
- Training data includes 2,512 verified multi-turn SFT trajectories from 79 databases, plus 2,700 synthesized question–SQL prompts and 100 validation instances for RL.The databases originate from Spider 2.0-Lite and Spider 2.0-DBT, with DuckDB databases ported to SQLite.Source: §4 · p. 5
- Qwen3-8B and Qwen3-14B are trained sequentially with harness-native full-trajectory SFT and execution-reward RL, and evaluated using the four-tool DSH-SQL contract.Tasks run in fresh isolated processes with read-only bindings and 60-second execution deadlines; missing explicit submission counts as failure.Source: §4 · p. 5
- The evaluation covers Spider 2.0-SQLite, BIRD-Interact Mini, and LiveSQLBench Base-Lite SQLite, including in-domain and out-of-distribution settings.BIRD-Interact Mini uses the A-Interact protocol with a GPT-4o user simulator.Source: §4 · p. 5
- Baselines span general-purpose models, SQL-specialized models, and a multi-agent system, each evaluated with its native inference protocol.The protocol avoids degradation from forcing baseline models into harness interactions.Source: §4 · p. 6
5 EXPERIMENTAL RESULTS
HarnessSQL’s compact agents outperform larger open-loop baselines in-domain and transfer to out-of-distribution interactive benchmarks. Ablations examine post-training stages, harness alignment, trajectory organization, context length, and RL algorithms.
5.1 MAIN RESULTS
- Qwen3-8B reaches 45.2% execution accuracy and Qwen3-14B reaches 54.8% on Spider 2.0-SQLite, exceeding 32B baselines at 14.8%–16.3%.Table 1 compares execution accuracy across in-domain and out-of-distribution benchmarks; the reported in-domain comparisons show compact harness-native models ahead of the cited larger open-loop models.Source: Table 1 · p. 6, §5.1 · p. 6
- On BI-mini, HARNESSSQL scores 9.0% (8B) and 11.3% (14B); on LSB-SQLite, it scores 24.8% and 28.5%, respectively.The paper reports these results above existing baselines on both out-of-distribution benchmarks, including 7.0%–8.9% for evaluated 32B baselines on LSB-SQLite.Source: Table 1 · p. 6, §5.1 · p. 6
5.2 EFFECT OF POST-TRAINING
- SFT raises 8B from 15.5% to 30.4% and 14B from 22.2% to 37.8%; post-SFT RL reaches 45.2% and 54.8%, versus 20.0% for direct 8B RL.The reported total gains over base models are 29.7 and 32.6 points for 8B and 14B, respectively; the authors describe verified trajectory supervision as important for task grounding and subsequent RL gains.Source: §5.2 · pp. 6–7
5.3 ABLATION STUDIES
- With DSH-SQL inference fixed, harness-native training scores 30.4% versus 9.6% for non-harness training on S2-SQLite.Applying DSH-SQL to the base model raises S2-SQLite accuracy from 2.2% to 15.5%; non-harness training improves non-harness inference from 2.2% to 14.1% but transfers poorly to DSH-SQL at 9.6%.Source: Table 3 · p. 7, §5.3 · p. 7
- Full-trajectory SFT outperforms turn-split training across all three benchmarks.In the full-trajectory setting, assistant actions are jointly supervised in one episode sequence while environment observations are masked from loss.Source: Figure 3 · p. 7, §5.3 · pp. 7–8
- Increasing the shared SFT and RL context limit from 8K to 32K raises S2-SQLite accuracy from 27.4% to 45.2%, with similar upward trends on BI-mini and LSB-SQLite.The authors suggest that truncating earlier interactions can remove execution evidence and intermediate reasoning needed for later decisions.Source: Figure 4 · p. 8, §5.3 · p. 8
- Under matched training and rollout budgets, DAPO and GSPO outperform GRPO across in-domain and out-of-distribution suites; DAPO scores 45.2% versus GRPO’s 40.7% on Spider2.0-SQLite.The relatively small DAPO–GSPO difference suggests the gains are not tied to a single RL algorithm.Source: Table 4 · p. 8, §5.3 · p. 8
- Trajectory-level analysis finds improved protocol adherence, error recovery, and valid termination after harness-native post-training.The authors offer these behavioral measures as evidence that the model learns the intended interaction policy, beyond terminal accuracy.Source: §5.3 · p. 8
6 RELATED WORK
Related work spans realistic interactive Text-to-SQL benchmarks, harness-based agent systems, and synthetic multi-step supervision. HarnessSQL combines these threads by making the deployment harness part of learning as well as evaluation.
LLMs for Text-to-SQL
- Text-to-SQL research spans Spider and BIRD to realistic settings such as Spider2.0 (Lei et al., 2025) and BIRD-Interact (Huo et al., 2025).Recent approaches use synthetic supervision, execution-based optimization, retrieval, database exploration, execution feedback, and iterative refinement.Source: §6 · p. 8
Harnesses and tool-using agents
- Harness research treats agent behavior as a property of the model–harness system; HARNESSSQL extends this perspective to learning by including the deployment harness in task generation, teacher rollout, post-training, and evaluation.The related work cites ReAct (Yao et al., 2023) and Harness-Bench (Yao et al., 2026a) among studies of agent interfaces and harness variation.Source: §6 · p. 9
Synthetic instruction and agent trajectory data
- Synthetic supervision has progressed from instruction-response pairs to complete agent trajectories, with verification used to improve reliability.HARNESSSQL grounds tasks in executable environments, rolls out teachers through the target harness, and retains trajectories after outcome verification.Source: §6 · p. 9
7 LIMITATIONS AND ONGOING WORK
The study focuses primarily on SQLite, while realistic SQL benchmarks also use other dialects. The authors describe extending harness-native synthesis to dialect-specific environments as ongoing work.
7 LIMITATIONS AND ONGOING WORK
- The study focuses primarily on SQLite, while realistic benchmarks also involve PostgreSQL, Snowflake, and BigQuery.The authors identify this as a current limitation and state that they are extending harness-native synthesis to dialect-specific execution environments.Source: §7 · p. 9
- The authors attribute part of the limitation to smaller models’ weaker and less consistent initial capabilities across non-SQLite dialects.They state that long-horizon interaction compounds the challenge because dialect-specific syntax, clients, metadata commands, and execution behavior enter the agent trajectory.Source: §7 · p. 9
- Ongoing work aims to extend harness-native synthesis to dialect-specific execution environments and close the performance gap between SQLite and other practically important dialects.Source: §7 · p. 9
8 CONCLUSION
HarnessSQL’s conclusion attributes closing the train–deploy gap to aligning the executable environment, interaction protocol, and trajectory-level supervision. It reports consistent gains across tested model scales and benchmarks, while trajectory-synthesis ablations highlight the importance of teacher quality and a specialized harness.
8 CONCLUSION
- HarnessSQL aligns authentic database states and runtime feedback, a task-specific interaction protocol, and complete-trajectory SFT followed by execution-reward RL.Its implementation uses isolated SQL sandboxes with hidden executable oracles and a minimal four-tool harness.Source: §8 · p. 9
- Qwen3-8B improves from 15.5% to 45.2% and Qwen3-14B from 22.2% to 54.8% on Spider 2.0-SQLite; gains also transfer to two OOD interactive benchmarks.The conclusion frames the harness as part of the behavior distribution training should reproduce, not merely an inference-time wrapper.Source: §8 · pp. 9–10
A DSH-SQL HARNESS COMPOSITION AND LIFECYCLE
- DSH-SQL is a profile-level composition: its SQL profile, environment adapter, and data and evaluation pipeline are additions to DSH’s generic agent loop and session substrate.The generic plugin lifecycle, agent loop, and session substrate come from DSH.Source: A DSH-SQL HARNESS COMPOSITION AND… · p. 13
B.1 TRAJECTORY SYNTHESIS CONFIGURATION
- Weak Qwen3.5-9B teacher supervision degrades Qwen3-8B below its unfine-tuned base, underscoring the importance of high-capacity teachers for viable trajectory synthesis.The ablation fine-tunes Qwen3-8B on a fixed 1,000-example synthesized subset.Source: §B.1 · pp. 14–15, Table 7 · p. 14
- DSH-SQL outperforms Full DSH in zero-shot base inference and post-training, with the specialized harness narrowing the action space and reducing interface noise.The reported zero-shot comparison is 15.5% vs. 13.3%.Source: Table 7 · p. 14, §B.1 · p. 15
B.2 INTERACTION BEHAVIOR ANALYSIS
The analysis reports that SFT improves protocol adherence and recovery behavior, while DAPO further consolidates valid interaction and successful termination. It also describes the execution-reward objective, the interaction-rich verified SFT corpus, the effect of retaining benign failures, and the scope of the leakage audit.
C.1 TRAINING CONFIGURATION
- Both RL runs allow a 32,768-token DSH session and at most 4,096 newly generated tokens per model call; the trainer’s token cap is not an additional context-window limit.The Qwen3-8B and Qwen3-14B RL runs start from the 32K SFT checkpoint.Source: §C.1 · p. 15
C.2 POLICY OPTIMIZATION OBJECTIVES
- The effective checkpoint objective uses clipped, group-normalized execution rewards without a KL term, awarding terminal success for a matching execution result and zero for execution failures or wrong outputs.There is no partial credit for syntactic validity, tool-call count, or intermediate SQL.Source: §C.2 · pp. 15, 17
B.2 INTERACTION BEHAVIOR ANALYSIS
- Invalid calls fall from 67.4% to 17.4% after SFT and to 0.6% after DAPO; exactly-one-valid-submit trajectories rise from 75.7% to 93.5%.Strict protocol violations fall from 83.2% to 26.7% after SFT and 11.5% after DAPO; SQL execution errors decline more modestly, from 9.0% to 8.1%.Source: §B.2 · p. 15, Table 8 · p. 16
D SFT CORPUS STATISTICS
- The final SFT corpus contains 2,512 successful trajectories from 79 database instances, averaging 11.8K tokens, 16 agent turns, 17.0 tool calls, and 13.9 SQL executions per trajectory.77.71% of trajectories revise a query after an execution failure, capturing iterative execution, feedback, and revision.Source: D SFT CORPUS STATISTICS · pp. 17–18
- For the evaluated models, training on original, unpruned trajectories achieves 30.4% execution accuracy, while training on sanitized trajectories achieves 28.2%, a drop of 2.2 percentage points.The authors say retaining benign execution failures and subsequent recovery turns provides valuable supervised signal for error recovery and environment self-correction.Source: D SFT CORPUS STATISTICS · p. 18
E DATA INTEGRITY AND LEAKAGE AUDIT FOR SHARED-DATABASE EVALUATION
- The leakage audit compares synthetic tasks from 30 Spider 2.0-Lite SQLite databases with the official 135-task test set across lexical, semantic, and execution-output dimensions.The evaluation uses stable database schemas while varying queries, and the authors audit whether test tasks appeared in or leaked into the synthetic corpus.Source: E DATA INTEGRITY AND LEAKAGE AUDIT FOR… · p. 18
E.1 EXACT MATCHING AND NEAR-DUPLICATE ANALYSIS
The audit finds no exact lexical overlap, low-to-moderate semantic similarity without any test query reaching the stated high-similarity threshold, and no matching execution outputs under the most permissive fingerprint criterion.
E.1 EXACT MATCHING AND NEAR-DUPLICATE ANALYSIS
- Across all 30 byte-identical database instances, the 135 test queries have zero overlap in task IDs, exact or normalized question strings, and substring containment.Nearest-neighbor checks also report a median token sequence ratio of 0.0913 and median token-set Jaccard similarity of 0.0971.Source: §E.1 · pp. 18–19, Table 12 · p. 19
E.2 SEMANTIC OVERLAP ANALYSIS
- Median document cosine similarity is 0.4965, and zero test queries reach 0.75 document or max-chunk cosine similarity against same-database candidates.The single pair above 0.70 reflects shared cricket terminology but distinct task objectives.Source: §E.2 · p. 20, Table 14 · p. 20
E.3 EXECUTION RESULT FINGERPRINTING
- Execution fingerprinting finds strictly zero matches under its most permissive criterion, ignoring column names and row order and comparing raw result values only.The authors summarize zero intersection across task identifiers, query phrasing, semantic embeddings, and execution outputs.Source: §E.3 · p. 20
F CROSS-HARNESS EVALUATION
Baselines transferred to DSH-SQL without harness-specific training consistently lose execution accuracy. The authors therefore compare existing systems under their native protocols and reserve the shared DSH-SQL protocol for controlled training ablations.
F CROSS-HARNESS EVALUATION
- The cross-harness comparison keeps benchmark instances and execution-based evaluation unchanged, but evaluates representative baselines both natively and under DSH-SQL without harness-specific adaptation.The baselines are trained for different interaction paradigms: direct SQL generation or multi-turn tool use with execution feedback.Source: F CROSS-HARNESS EVALUATION · p. 21
- Under DSH-SQL, OmniSQL-32B falls from 14.8% to 1.5% and Arctic-Text2SQL-7B from 15.6% to 0% on S2-SQLite.Even Qwen2.5-Coder-32B declines substantially under the unfamiliar interaction protocol.Source: F CROSS-HARNESS EVALUATION · p. 21
- The authors use native-protocol results for existing baselines to avoid conflating model capability with harness compatibility; controlled ablations instead share DSH-SQL inference.The reported cross-harness results indicate that interactive-harness access alone does not guarantee better performance.Source: F CROSS-HARNESS EVALUATION · p. 21
G EVALUATION BENCHMARKS
The evaluation spans three SQLite benchmarks designed to test complex in-domain querying, interactive clarification, and zero-shot transfer. The accompanying sections specify data exports, a bounded SQL-difficulty curriculum, representative synthesized tasks, verifier checks, and trajectory-generation diagnostics.
G EVALUATION BENCHMARKS
- The three SQLite benchmarks test complex in-domain querying, dialogue-driven clarification, and transfer to continually updated real-world tasks.S2-SQLite has 135 database instances; BI-mini uses the A-Interact protocol, and LSB-SQLite tests generalization to unseen database topologies and dialect-specific edge cases.Source: G EVALUATION BENCHMARKS · p. 21
H DATA RECORD AND EXPORT FORMAT
- Training exports preserve full task and trajectory records while removing hidden oracle fields from model-visible context and masking loss on system, user, and tool-observation tokens.Records include the message stream, tool arguments, observations, final SQL, execution result, verifier outcome, and selection method.Source: H DATA RECORD AND EXPORT FORMAT · p. 22
I.1 CONTROLLABLE STRUCTURAL DIFFICULTY
- A structural score and bounded difficulty bands control synthesized SQL complexity without treating syntax as a predictor of model success.The curriculum can be adjusted through band mixtures or bounds; candidates outside the requested interval receive feature-level feedback and bounded repair.Source: §I.1 · p. 22
I.2 REPRESENTATIVE SYNTHESIZED TASK RECORDS
- The retained core and stretch examples vary in SQL composition depth while following the same catalog-grounded, execute-and-verify construction protocol.The core example has Dstr = 15.732; the stretch example has Dstr = 20.824, 302 SQL tokens, and two measured advanced families.Source: §I.2 · pp. 23–24
J VERIFIER CHECKLIST
- The verifier checks read-only SQL safety, schema validity, live execution, result quality, instruction-program consistency, and trajectory-protocol compliance.Source: J VERIFIER CHECKLIST · p. 24
K GENERATION DIAGNOSTICS
- Table 16 reports verified trajectories retained after three teacher rollouts per task.Source: Table 16 · p. 24