PaperlayerView exact revision on arXiv

Source-linked AI summary

HybridResearcher: Bridging the Structured-Unstructured Boundary for Deep Research Agents

Soyoung Yoon, Peiran Xu, Han Wang, Ruofan Wu, Xiaodong Yu, Feng Yan, Mohit Bansal, Debiao Li, Seung-Won Hwang, Yuxiong He, Zhewei Yao

arXiv:2610.24899v2cs.AIcs.CL

TL;DR

Agents can gather evidence from databases or the web, but hybrid questions require carrying evidence between structured and unstructured sources, and current training approaches do not teach that bridge. HYBRIDRESEARCHER builds gated cross-source trajectories from validated single-source tasks and trains with supervised fine-tuning followed by reinforcement learning. It improves HybridDeepResearch performance over its base and a much larger model while also improving held-out single-source benchmarks.

  • Problem

    Existing training recipes mostly target one source, and no prior training framework, data, or trained model addresses agents that must bridge structured and unstructured sources in one reasoning chain.

    Source: §1 · p. 2

  • Method

    HYBRIDRESEARCHER extends validated text-to-SQL or web-search seeds with a missing source around a bridge entity, retains examples only when gates verify both sources are necessary, and trains with supervised fine-tuning followed by reinforcement learning.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    30.00 points pass@8: HYBRIDRESEARCHER-4B improves over Qwen3.5-4B on HybridDeepResearch, outperforms Qwen3.5-397B-A17B, and also improves on single-source SQL and search benchmarks.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The reported results show that a compact model can be trained for hybrid reasoning while improving, rather than losing, performance on single-source benchmarks.

    Source: §1 · p. 3, §7 · p. 9

    On S2SQL, the required bridge entity fails to surface in 22.2% of rollouts, and 82.0% of RL’s wrong answers cite time budget, effort spent, or a guess.

    Source: §6.1 · p. 8

Abstract

from arXiv · show

While LLM agents are increasingly capable of gathering and synthesizing evidence from a single homogeneous source, such as structured tables or unstructured data, real-world analytical questions frequently span heterogeneous sources. For example, answering one enterprise question may require both a result computed over an internal database and evidence found on the web. Recent evaluations on benchmarks like HybridDeepResearch show that even frontier agents struggle when evidence must cross this boundary. While these benchmarks have exposed the limitation, methodologies to explicitly teach cross-source reasoning remain absent. To address this, we introduce HybridResearcher, a 4B agent trained to bridge evidence across the structured and unstructured boundary. To build it, we synthesize cross-source bridge trajectories by starting with validated single-source tasks-such as existing text-to-SQL or web search queries-and fabricating the missing source around a bridge entity drawn from the seed's own output. We retain a sample only when strict construction gates confirm that neither side alone suffices, ensuring the data requires true cross-source interdependence. Finally, we train on this curated dataset using supervised fine-tuning followed by reinforcement learning. Evaluated on the existing HybridDeepResearch benchmark, HybridResearcher improves over our base model, Qwen3.5-4B, by 30.0 points pass@8 and surpasses the far larger Qwen3.5-397B-A17B by 8.3 points, while also improving over its base model on established, held-out single-source benchmarks (BrowseComp and BrowseComp-Plus), demonstrating that this complex cross-source reasoning can be taught directly to small models without triggering catastrophic forgetting.

1 INTRODUCTION

Agents have advanced on database and web tasks in isolation, but real-world questions often require evidence to cross the structured–unstructured boundary. HYBRIDRESEARCHER trains a small model on gated cross-source trajectories, improving HybridDeepResearch performance while also improving held-out single-source results.

1 INTRODUCTION

  • Training recipes for database or web tasks usually use one source, while real-world analytical questions can require evidence from both structured and unstructured sources.HybridDeepResearch shows that even frontier agents struggle when evidence must cross that boundary, and the authors report no prior training framework, data, or trained model for this setting.Source: §1 · pp. 1–2
  • Single-source training does not teach the bridge, and correctness-filtered hybrid trajectories can still bypass substantive SQL execution.The authors report that 79.5% of teacher trajectories accepted by standard correctness filters never execute a substantive SQL query.Source: §1 · p. 2
  • HYBRIDRESEARCHER extends validated text-to-SQL and web-search tasks across the source boundary, retaining examples only when construction gates verify that neither source alone suffices.The curated data is used for supervised fine-tuning and reinforcement learning.Source: §1 · p. 2
  • 30.0 points pass@8: HYBRIDRESEARCHER-4B improves over Qwen3.5-4B on HybridDeepResearch and surpasses Qwen3.5-397B-A17B-FP8 by 8.3 points.It also outperforms its base model on held-out Spider 2.0-lite, BrowseComp, and BrowseComp-Plus.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work has advanced agents for single-source database and web tasks, and research on heterogeneous-source questions has expanded. The paper distinguishes its approach by training agents to carry evidence across sources through multi-hop SQL and web search.

Single-Source Agents

  • Database and web agents have progressed on single-source tasks, but the paper trains an agent to carry evidence found in one source into the other.The cited database work includes realistic workflows and query revision, while web systems tackle long-horizon questions.Source: §2 · p. 3

Reasoning across Heterogeneous Sources

  • Earlier hybrid question answering often supplies or retrieves evidence in one step, while recent benchmarks evaluate heterogeneous-source agents but few works train them.DSL-R1 uses structured filters and vector search in one query language over a closed collection; Wu et al. (2026a) train on simpler SQL-plus-Wikipedia questions with hint-guided reinforcement learning.Source: §2 · p. 3
  • This paper presents a training recipe for harder multi-hop questions combining SQL execution with long-horizon web search.The authors contrast this with prior work that does not require carrying a result from one source to another.Source: §2 · p. 3

3 METHODOLOGY: SYNTHESIZING CROSS-SOURCE TRAJECTORIES

The methodology builds hybrid trajectories from validated single-source seeds because single-source training does not teach cross-source interdependence and naive hybrid synthesis invites shortcuts. It constructs SQL-originated and search-originated tasks, using gates to ensure each source is load-bearing.

3.1 TWO OBSTACLES TO TRAINING HYBRID AGENTS

  • On HybridDeepResearch’s 120-question hard subset requiring SQL and web search, SQL-only fine-tuning reaches 31.67 pass@8 versus base 30.83, while union and search-only score 29.17 and 15.83.Single-source trajectories never use one tool’s output as the other tool’s input.Source: §3.1 · pp. 3–4
  • Correct answers from naive hybrid synthesis can still bypass the source boundary when the model recalls search facts or the question reveals the bridge entity.Bridge entities from real databases may be famous, allowing parametric knowledge to override retrieved evidence.Source: §3.1 · p. 4

3.2 SQL-ORIGINATED TRAJECTORIES

  • For SQL-originated SQL2S, an output cell absent from the question becomes the bridge; coined facts and near-miss documents require search to identify the unique hub.For S2SQL, an execution-tested SQL literal in the question is hidden and replaced by intersecting search clues; the answer remains the unchanged SQL output.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 SEARCH-ORIGINATED TRAJECTORIES

  • Search-originated tasks add a synthetic SQL component to a hard-search seed, with the bridge taken from either the question for SQL2S or the answer for S2SQL.SQL2S requires querying the database to find the question’s entity; S2SQL requires resolving the web answer before using it as a SQL parameter.Source: §3.3 · p. 5
  • The search-originated pipeline plans executable SQL patterns, removes schema shortcuts, builds a populated database, and masks the bridge in the merged prompt.Verification checks that SQL2S bridges do not appear in the merged question and S2SQL bridges do not leak into the SQL result table; the database must require fresh structured reasoning.Source: §3.3 · p. 5, Figure 3 · p. 5

4 TRAINING RECIPE

The training recipe generates candidate trajectories for supervised fine-tuning and curates unsolved hybrid tasks for reinforcement learning. It uses filtering, hard gates, and majority-vote judging to target training data and stabilize reward evaluation.

4.1 SUPERVISED FINE-TUNING

  • SFT trajectories combine directly used training data with judge-correct examples generated by running the agent harness against a local corpus and databases.The trajectories are generated using Qwen3.5-397B-A17B-FP8 or DeepSeek-V4-Flash.Source: §4.1 · p. 6

4.2 REINFORCEMENT LEARNING

  • The RL candidate pool starts with 21K synthesized hybrid tasks unsolved during SFT and is filtered to 0.7K high-signal QA pairs.Filtering removes tasks that are consistently solved or never solved, ambiguous or duplicate QA pairs, and imbalances across source and task direction.Source: §4.2 · p. 6
  • RL uses CISPO with hard gates that terminate and zero-reward rollouts for invalid or degenerate tool use.Triggers include exhausting the turn budget without a tool call, malformed arguments, nonexistent tools, invalid final answers, and repeated queries.Source: §4.2 · p. 6
  • A five-vote LLM-judge majority determines the binary pass/fail reward for SQL2S and S2SQL.The ensemble is intended to stabilize optimization where a single judge may be unreliable on nuanced semantic matches or equivalent SQL solutions.Source: §4.2 · p. 6

5 EVALUATION

The evaluation measures hybrid reasoning and retained single-source skills on held-out benchmarks. Hybrid SFT and RL improve cross-source performance, while the trained 4B model also improves over its base model on search and SQL tasks.

5 EVALUATION

  • Evaluation uses 120 hard-subset HybridDeepResearch questions, plus held-out BrowseComp, BrowseComp-Plus, and Spider 2.0-lite sets that do not appear in training.BrowseComp uses the live web, BrowseComp-Plus a fixed 100,195-document corpus, and Spider 2.0-lite its public subset; results are reported as avg@8 and pass@8 unless specified otherwise.Source: §5 · pp. 6–7
  • A five-vote Qwen3.8-27B judge evaluates hybrid and search-only answers, while Spider 2.0-lite uses official execution accuracy.The benchmark table covers hybrid training progression and larger-model comparisons across Pass@8 and Avg@8; the single-source table covers search benchmarks and SQL execution accuracy.Source: §5 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7

5.1 RESULTS

  • 30.83% overall pass@8 for the Qwen3.5-4B base rises to 59.17% with hybrid SFT; RL raises overall avg@8 from 23.75% to 27.19%.The final 4B agent reaches 27.19% overall avg@8, surpassing Qwen3.5-397B-A17B-FP8 at 26.77% avg@8 and 52.50% pass@8; the final agent also approaches Claude Sonnet 4.6 (65.83% pass@8) and GLM-5.2-FP4 (63.33% pass@8).Source: Table 1 · p. 7, §5.1 · p. 7
  • Hybrid training improves held-out single-source performance: BrowseComp pass@8 rises from 54.03% to 61.37%, BrowseComp-Plus from 60.84% to 69.76%, and Spider 2.0-lite execution accuracy from 59.26% to 65.19%.Hybrid RL largely preserves these gains and raises BrowseComp-Plus to 73.13% pass@8.Source: Table 2 · p. 7, §5.1 · p. 7

6 ANALYSIS

The analysis traces how SFT, RL, data filtering, judge voting, and hard gating affect tool use and hybrid-task performance. It also identifies retrieval and early stopping as remaining bottlenecks.

6.1 BEHAVIORAL SHIFTS: HOW SFT AND RL TRANSFORM TOOL USE

  • SFT nearly halves web searches while increasing page visits; SQL2S accuracy when the gold answer is present rises from 12.9% to 18.5%.The base model makes 38.6 searches per rollout but averages only 0.29 page visits per search.Source: §6.1 · p. 8

6 ANALYSIS

  • After RL, 48.8% of queries use quotes, operator use doubles, and exact repeats drop by 63%; SQL2S accuracy when gold answers are surfaced rises from 18.5% to 27.2%.RL maintains similar or lower turn counts and token usage than the SFT checkpoint; Table 4 compares search queries and efficiency for SFT and RL across 960 rollouts each.Source: §6.1 · p. 8, Table 4 · p. 8

6.1 BEHAVIORAL SHIFTS: HOW SFT AND RL TRANSFORM TOOL USE

  • The required bridge entity fails to surface in 22.2% of S2SQL rollouts, and 82.0% of RL wrong answers cite time pressure, effort spent, or guessing.The authors identify searching more when uncertain and refraining from guessing as possible next directions.Source: §6.1 · p. 8

6.2 DATA QUALITY OUTWEIGHS SCALE IN RL OPTIMIZATION

  • The curated RL dataset reaches 27.2% overall avg@8 versus 21.0% for the full 21K candidate pool; 71% of curated rollout groups provide reward contrast, compared with 38% uncurated.The comparison uses the identical training recipe.Source: §6.2 · p. 8

6.4 HARD-GATING YIELDS CLEANER TRAINING SIGNAL AND EFFICIENT TOOL-USE

  • Removing hard gating lowers avg@8 from 27.2% to 25.6%; with gating, average tool calls fall from 76 to 55, repeated-call frequency from 49% to 11%, and invalid-answer failures from 10.7% to 0.0%.Without gating, trajectory token length and tool-call counts rise during training, while they remain stable with gating.Source: Figure 4 · p. 9, §6.4 · p. 9

6.3 JUDGE MAJORITY-VOTING HELPS IMPROVE REWARD RELIABILITY

  • A five-vote judge raises overall avg@8 from 25.2% to 27.2%; single-judge disagreement with consensus is 8.2% overall, including 12.2% for S2SQL and 1.2% for SQL2S.The authors attribute noisy rewards to this judge instability, particularly for S2SQL.Source: §6.3 · p. 9

7 CONCLUSION

HYBRIDRESEARCHER-4B is trained on gated cross-source tasks and improves on hybrid and single-source benchmarks. The paper also reports analyses of the training stages and outlines possible extensions to other heterogeneous sources.

7 CONCLUSION

  • 30.00 points pass@8 over its base model on HybridDeepResearch, with performance also exceeding Qwen3.5-397B-A17B and improving on single-source SQL and search benchmarks.The reported analysis also finds no catastrophic forgetting.Source: §7 · p. 9
  • Training extends validated text-to-SQL and deep-search tasks across sources, retains instances that require both sources, and applies supervised fine-tuning followed by reinforcement learning.The conclusion describes the cross-source task as carrying a result from one source into the other.Source: §7 · p. 9
  • Fine-tuning teaches the agent to read retrieved material rather than issue many searches, while reinforcement learning benefits more from a small curated task set than a larger unfiltered pool.Source: §7 · p. 9

A OVERVIEW

  • The appendix covers training data, construction pipelines, retrieval and agent environments, evaluation, failures, and reinforcement-learning analyses; the conclusion proposes extending the recipe to other source types.Examples named include internal documents, code repositories, and domain-specific databases.Source: §7 · p. 9, A OVERVIEW · p. 17

B DIRECT TRAINING DATA

Direct training data combines HyDRA items with deduplicated text-to-SQL examples from nine public datasets. The appendix also identifies the SQL-originated construction subsets and their seed families.

B DIRECT TRAINING DATA

  • The direct-training set contains 10,184 HyDRA items and 25,747 deduplicated text-to-SQL examples from nine public datasets.These provide direct supervision for hybrid and SQL capabilities, respectively.Source: B DIRECT TRAINING DATA · p. 17
  • For datasets other than SynSQL, questions are retained only when at least one sampled trajectory is correct; SynSQL uses a random subset.Source: B DIRECT TRAINING DATA · p. 17
  • The direct-training table uses Qwen3.5 tokens with 128k packing.Source: Table 5 · p. 17

C SQL-ORIGINATED CONSTRUCTION DETAILS

C.1 BUILDING SQL2S: THE BRIDGE IS AN OUTPUT CELL

SQL2S builds cross-source tasks from SQL results linked to fabricated web evidence, with verification gates intended to require both sources. Its construction also filters bridge entities for reliable bridging and grading and expands the seed pool with filtered public deep-research questions through a distinct entity-mining path.

C.1 BUILDING SQL2S: THE BRIDGE IS AN OUTPUT CELL

  • SQL2S selects an unnamed gold-output cell as its bridge, builds coined facts and intersecting hubs, then applies deterministic and model-based gates to enforce evidence dependence.Each coined fact receives a gold document and typed hard negatives; six deterministic and five model-based gates test evidence sufficiency and plausibility. For SQL-ORIG-PRIM, an anti-shortcut pass adds authoritative prior decoys, homonyms, and namesake or misspelling negatives.Source: §C.1 · pp. 18–19

C.2 BUILDING S2SQL: THE ANCHOR IS A WHERE-CLAUSE LITERAL

  • S2SQL anchors on a name-bearing WHERE-clause literal that changes the executed row set, is hidden from the output, and supports at least three clues.Each coined near-miss satisfies all but one clue, and items carry about 43 hard negatives; the seed question follows the clues so the answer remains the result table.Source: §C.2 · pp. 19–20

C.3 CORPUS AND INGESTION

  • The retrieval corpus contains 1,548,257 SQL-ORIG-SYN documents, about 250,000 SQL-ORIG-PRIM documents, and 591,664 SQL-ORIG-S2SQL-SYN documents.Fabricated documents are inserted alongside real encyclopedic and web text in the retrieval service used at evaluation.Source: §C.3 · p. 20

C.4 WORKED EXAMPLES

  • Worked examples show SQL2S resolving coined multi-hop affiliations, a primary-split puzzle overriding the Google Photos prior, and S2SQL recovering Wilson before rerunning the seed query.The primary-split example answers Jivoka Networks using the claim attributed to the 2025 Mobile App Industry Report; the S2SQL trajectory resolves Wilson from three clues and obtains the seed query’s result table.Source: §C.4 · pp. 21–23

D.1 SEED POOL AND CONSTRUCTION

  • Search-originated construction collects multi-hop questions with short verified answers from six public deep-research question sets and removes malformed, duplicate, and unverified QA pairs.Source: §D.1 · p. 23
  • 14,233 of 22,536 candidate seeds remain after a Knowledge Graph gate retains database-storable, web-confirmable named entities, excluding most numeric and yes/no answers.Four-digit years are retained, and remaining strings pass when the top Knowledge Graph result scores above 50; the answer anchors S2SQL questions and remains SQL2S’s final answer for reliable grading.Source: §D.1 · p. 24
  • SQL2S mines candidate entities from each seed’s search question and adds 24,285 filtered seeds from Orbit (Thakur et al., 2026), ToolMind (Yang et al., 2025), and LoHoSearch (Zhao et al., 2026).Each seed is grounded once and follows a single construction chain; the added sets are used only for SQL2S, and Table 11 reports remaining data size after each stage.Source: §D.1 · p. 24

D.2 SHIPPED DATABASES.

The shipped databases are generally small, centered on bridge entities, and may involve multiple joined tables. Worked examples show how the bridge links a database result to web-search evidence in both construction directions.

D.2 SHIPPED DATABASES.

  • The 35,698 SQL2S and 14,179 S2SQL databases typically contain one table with a few hundred peer rows, while over a third and a quarter, respectively, join multiple tables.Source: §D.2 · p. 24

D.3 WORKED EXAMPLES

  • In the SQL2S example, the search-seed bridge “Bust!” becomes the SQL answer, and the agent uses the ranked episode result to resolve the web clues and answer Rob Mayes.The SQL result is Bust! with score 7.62; the trajectory reports 21 searches and 7 pages.Source: §D.3 · p. 25
  • In the S2SQL example, web clues identify French Guiana, which the agent then uses to retrieve its database rank and GDP-per-capita value.The reported result is French Guiana, 30.0, rank 2.Source: §D.3 · p. 26

E SFT DATA ABLATION

This block reports an SFT data ablation and details the local corpus, hybrid harness, and evaluation setup. Adding search-originated data, then SQL-originated data, raises overall pass@8.

E SFT DATA ABLATION

  • 54.17 pass@8 after adding search-originated data, then 59.17 after adding SQL-originated data; the second addition raises S2SQL from 52.50 to 65.00.The ablation adds each data family in turn, starting from hydra and nine raw text-to-SQL sources.Source: E SFT DATA ABLATION · p. 26

F LOCAL CORPUS

  • A 21M-document local corpus reproduces query-dependent search snippets using two-stage hybrid retrieval, reranking, and page-gated full-text visits.Documents are split into non-overlapping chunks of up to 500 characters; the top k ≤20 documents are returned, one result per page.Source: F LOCAL CORPUS · p. 27

G HYBRID AGENT HARNESS

  • The native tool-calling harness serves teacher collection, RL rollouts, and evaluation in the same interaction format used during training.Tool observations are appended turn by turn until the agent commits an answer or exhausts its budget.Source: G HYBRID AGENT HARNESS · p. 27

G.1 TOOL INTERFACE

  • The agent must discover both sources through tools; SQL results bridge to web research in SQL2S, while web-resolved entities bridge to SQL in S2SQL.The harness exposes the same tools at each turn without prescribing the order.Source: §G.1 · p. 27

G.2 EPISODE EXECUTION

  • Tool failures return as observations, and episodes end on a successful final answer, a turn without a tool call, or exhausted turn or wall-clock budgets.The final answer is executable SQL for S2SQL and concise text for SQL2S and other text-answer tasks.Source: §G.2 · p. 27

G.3 AGENT HARNESS COMPARISON

  • The hybrid harness achieves the highest overall pass@8 across all three evaluated backbones compared with smolagents and MiroFlow.The improvement is most pronounced for Qwen3.5-397B-A17B-FP8 and smaller for GLM-5.2.Source: Table 15 · p. 28, §G.3 · p. 28

H EVALUATION DETAILS

  • Evaluation details cover live-web BrowseComp, fixed-corpus BrowseComp-Plus, inference settings, Spider 2.0-lite, and judge prompts.BrowseComp-Plus uses the hybrid retrieval backend over full documents without a separate chunk index or local reranker.Source: H EVALUATION DETAILS · p. 28, §H.1 · p. 28, §H.2 · p. 29

H.4 SPIDER 2.0-LITE EVALUATION

This section evaluates SQL generation on Spider 2.0-lite’s local SQLite instances using a constrained agent harness and execution-based grading, with judge prompts distinguishing semantic errors from presentational differences. It also analyzes outcomes and answer justifications across base, SFT, and RL models on a 120-question hard subset, including 35 questions none of the models solve.

H.4 SPIDER 2.0-LITE EVALUATION

  • Spider 2.0-lite evaluation uses all 135 local SQLite instances across 30 databases, with the database schema and any external-knowledge document provided to the agent.The subset is determined by database engine, not difficulty or results; the harness removes web access and provides SQL execution and final-answer tools.Source: §H.4 · pp. 29–30
  • Submitted queries are graded by official execution accuracy against acceptable gold results, ignoring row order and allowing extra columns; four 600-second timeouts remained unfinished after one hour.Only the query submitted through final answer is graded; queries run during interaction are not. Numeric values use 0.01 absolute tolerance, and non-executing queries are incorrect.Source: §H.4 · p. 30

H.5 LLM JUDGE PROMPTS

  • For S2SQL table judging, semantic equivalence centers on the requested values and scope, while presentation differences are tolerated; contradictory, incomplete, or differently scoped answers are incorrect.The prompt treats column names, table shape, row order when unrequested, and nonessential columns as presentational, but requires key metrics, entities, groupings, and averages to match.Source: §H.5 · p. 31
  • For SQL2S and Parallel answers, the judge accepts semantically equivalent answers that include the gold information without contradiction, but rejects incorrect, incomplete, or merely hedged answers.Extra context is permitted when it does not replace or contradict the committed core answer.Source: §H.5 · pp. 31–32

I FAILURE ANALYSIS OF THE BASE, SFT, AND RL MODELS

  • The analysis covers 2,880 rollouts across the base, SFT, and RL models, with 35 of 120 questions unsolved by all three.Each model receives identical prompts, tool schemas, and decoding parameters; rollouts without a final answer call count as wrong. The 35 universally unsolved questions comprise 9 S2SQL, 21 SQL2S, and 5 Parallel questions.Source: I FAILURE ANALYSIS OF THE BASE, SFT,… · p. 34

I.1 OUTCOME OF EVERY ROLLOUT

The rollout analysis traces where evidence is surfaced and used across S2SQL, SQL2S, and Parallel tasks, then examines answer commits flagged for time, effort, or guessing. The trained models’ remaining errors include retrieval and cross-source use failures, while flagged commits are more often wrong.

I.1 OUTCOME OF EVERY ROLLOUT

  • Outcome labels use ordered first-match categories, and evidence tracing tracks bridge and gold-answer strings through tool results, reasoning, and final output.The string-matching rates are lower bounds because paraphrases and entities encoded by ID do not match.Source: §I.1 · p. 34, Figure 5 · p. 35
  • 35.0% of SFT and 33.1% of RL S2SQL rollouts use the wrong bridge entity, while 29.7% and 31.3% build wrong SQL around the right entity.Malformed SQL is rare, at at most 4.1% of rollouts; the figure categorizes outcomes for every rollout.Source: §I.1 · p. 34, Figure 5 · p. 35
  • 52.2% of SFT and 52.8% of RL SQL2S rollouts have a gold answer that never surfaces; across categories, it never appears in 59.4% and 54.1%.When the gold answer never appears in search results or visited pages, accuracy is at most 2.1%.Source: §I.1 · p. 34, Figure 5 · p. 35
  • 18.4% of SFT and 19.7% of RL Parallel rollouts contain the gold answer in both sources but submit an answer in only one candidate set; 17.5% and 20.3% submit an answer in neither.When the submitted answer satisfies only one side, SQL is the matching side more often than web.Source: §I.1 · pp. 34–35, Figure 5 · p. 35
  • 50.8% of base-model rollouts end without an answer; 43.2% end in tool-free reasoning that exhausts the 32K-token limit.Wrong answers also repeat across samples; split judge votes are rare among wrong rollouts.Source: §I.1 · p. 35

I.2 COMMITS JUSTIFIED BY TIME, EFFORT, OR A GUESS

  • 82.0% of RL wrong commits are flagged, versus 48.7% of correct commits; within-question flagged commits are 24.3 points less often correct.The flag detects rationales citing time or turn budgets, effort already spent, or a guess; the figure reports flagged and unflagged accuracy comparisons.Source: §I.2 · pp. 35–36, Figure 6 · p. 36
  • Flagged submissions remain negatively associated with correctness after controls, and a longer turn budget alone is unlikely to help; the authors suggest evidence-triggered verification.The controlled gap is −10.0 [−16.2, −3.1] points for RL and −7.9 [−13.2, −1.8] points for SFT; RL does not detectably change the gap versus SFT.Source: §I.2 · p. 36, Table 17 · p. 37

J.1 RL TRAINING PARAMETERS

  • RL uses GRPO/CISPO with 32 prompts per step and group size 5; difficulty filtering excludes 0% and 100% solve-rate groups and removes tasks solved once or failed three consecutive groups.Rollouts use up to 32k completion tokens per turn and a maximum of 100 turns.Source: §J.1 · p. 38

J.2 JUDGE VOTING ANALYSIS

The judge-voting analysis presents cases where five-vote majority judging reaches the correct judgment after a single-vote error. The examples include semantic ambiguity and SQL result-level comparisons.

J.2 JUDGE VOTING ANALYSIS

  • The representative examples include semantic ambiguity and non-trivial variations in SQL result tables.Source: §J.2 · p. 38, Table 18 · p. 38
  • Five-vote majority judging reaches the correct judgment in representative cases where single-vote judging gives an incorrect reward.The examples are selected to illustrate challenging cases rather than provide aggregate rates.Source: Table 18 · p. 38, §J.2 · p. 38
  • The SQL example requires reasoning about result-level semantics rather than exact table matching.The caption identifies this as a representative case of a single-vote error corrected by majority judging.Source: Table 18 · p. 38

Related papers

Loading 2610.24899v2…