PaperlayerView exact revision on arXiv

Source-linked AI summary

FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents

Jia Deng, Yimeng Chen, Xiaoqing Xiang, Ziyang Zeng, Shuo Tang, Wayne Xin Zhao, Feng Chang, Chuan Hao, Yuan Wei, Ran Tao, Bryan Dai, Ji-Rong Wen

arXiv:2606.12087v1cs.CL

TL;DR

Deep-search training data needs to require evidence acquisition, but structural complexity alone may not ensure the intended search remains necessary. FORT controls shortcut risks in synthesis, and FORT-Searcher achieves the best overall performance among comparable-size open-source search agents on challenging benchmarks.

  • Problem

    Existing search data often lacks long-horizon evidence acquisition, and structural complexity alone does not establish that a question requires the intended search process.

    Source: §1 · p. 2

  • Method

    FORT operationalizes a shortcut-aware difficulty framework by controlling shortcut risks across entity selection, evidence graph construction, question formulation, and adversarial refinement.

    Source: §1 · p. 2

  • Results

    FORT-Searcher achieves the best overall performance among comparable-size open-source agents on challenging benchmarks, while FORT induces longer pre-answer search and reduces shortcut patterns.

    Source: §1 · p. 3, §6 · p. 22

  • Takeaways & Limitations

    Deep-search supervision should be judged not only by intended structural complexity but by whether evidence acquisition remains necessary during actual agent search.

    Source: §6 · p. 21

    The reported scope centers on web-only trajectories; tool-augmented search and more complex search-grounded tasks are left for future work.

    Source: §6 · p. 22

Abstract

from arXiv · show

Training deep search agents requires verifiable questions whose answers remain unavailable until sufficient evidence has been acquired through search. Existing synthesis methods often increase apparent difficulty by enriching graph structures, but structural complexity alone does not guarantee realized search difficulty: the intended search process can collapse through a cheaper identifying route. We formalize this gap with a shortcut-aware difficulty framework and identify four actionable shortcut risks: evidence co-coverage, single-clue selectivity, exposed constants, and prior-knowledge binding. To diagnose their realized effects, we use trajectory signatures including solving cost, answer hit time, and prior-shortcut rate. Guided by this framework, we introduce FORT, a Framework of Shortcut-Resistant Training-Data Synthesis. FORT constructs shortcut-resistant training data by controlling shortcut risks across entity selection, evidence graph construction, question formulation, and adversarial refinement. Experiments show that FORT induces longer pre-answer search and fewer shortcut patterns than existing open-source deep search datasets. Using the resulting trajectories, we train FORT-Searcher with supervised fine-tuning (SFT) only, and it achieves the best overall performance among comparable-size open-source search agents on challenging deep search benchmarks. Relevant resources will be made available at https://github.com/RUCAIBox/FORT-Searcher.

1. Introduction

FORT frames deep-search difficulty around whether questions force intended evidence acquisition in real search trajectories, then controls shortcut risks; FORT-Searcher achieves the best overall comparable-size open-source benchmark performance with SFT alone.

Motivation

  • Enriching intended task structures such as hop count, graph shape, hierarchical constraints, or evidence dispersion does not ensure agents perform the intended long-horizon evidence acquisition.Agents may bypass intended evidence gathering using environmental evidence or prior knowledge, limiting incentives for long-horizon search, planning, and synthesis.Source: §1 · p. 2

Shortcut-aware difficulty framework

  • The framework treats realized difficulty as depending on the cheapest identifying route and diagnoses evidence co-coverage, single-clue selectivity, exposed constants, and prior-knowledge binding as shortcut risks.It monitors realized effects through solving cost, answer hit time, and prior-shortcut rate.Source: §1 · p. 2

FORT synthesis framework

  • FORT controls shortcut risks across entity selection, evidence-graph construction, question formulation, and adversarial refinement.It selects long-tail entities, diversifies and expands evidence, uses weak-in-isolation facts that identify in combination, withholds intermediate names, fuzzes constants, and tests drafts with a strong search agent to repair shortcut-prone or unclear cases.Source: §1 · pp. 2–3

Results

  • With SFT alone, FORT-Searcher achieves the best overall performance among comparable-size open-source agents on BrowseComp, BrowseComp-ZH, xbench-DeepSearch, and Seal-0; FORT also induces longer pre-answer search and reduces shortcut patterns.The analyses compare FORT's search trajectories with those from existing open-source deep search datasets.Source: §1 · p. 3

2. Difficulty Framework

The framework separates task-imposed search cost from solver-specific shortcuts, then uses trajectory signatures to diagnose whether multi-constraint retrieval actually requires substantial pre-answer evidence acquisition.

2.1. Problem Formulation

  • A task is a set of candidate answers, question constraints, and a retrieval interface; it is well-posed when the full constraint set uniquely determines the answer.The agent issues retrieval queries, observes evidence, and may continue searching or answer; trajectory length counts retrieval queries.Source: §2.1 · pp. 3–4, §2 · p. 3

2.2. Difficulty Analysis

  • Evidence dispersion and dependency depth set a lower bound through the cheapest identifying route, while selectivity shapes exploration and prior knowledge can reduce a specific solver’s realized cost.A single source covering multiple clues can reduce evidence-acquisition effort; dependency depth reflects serial steps needed to make queries executable. Selectivity governs candidate breadth, while prior utility accounts for solver-side cost reduction.Source: §2.2 · pp. 4–6

2.3. Shortcut Patterns

  • The framework distinguishes route-level shortcuts—single-clue selectivity, evidence co-coverage, and exposed constants—from solver-level prior-knowledge binding.The first three can shorten the cheapest identifying route; prior binding instead lowers a particular solver’s realized cost by allowing an answer before retrieved evidence supports it.Source: §2.3 · pp. 6–7

2.4. Trajectory Signatures

  • Realized difficulty is diagnosed using solving cost, answer hit time, and prior-shortcut rate, which together distinguish long search from early answer exposure or answer-before-evidence behavior.Answer hit time tracks when the answer first appears in retrieved evidence or visible model output; a large gap between solving cost and hit time can indicate post-hit verification or detours.Source: §2.4 · pp. 7–8
  • OpenSeeker’s bΩ = 84.7, T_hit = 9.3, and b_pprior = 31.9 show that long trajectories can coexist with early answer exposure and frequent prior binding.REDSearcher has stronger trajectory signatures than other open-source datasets, yet its answer hit time remains much earlier than its total solving cost; the section argues that search length alone does not ensure search-heavy supervision.Source: §2.4 · p. 8

3. Methodology

FORT treats shortcut-resistant question synthesis as an inverse construction problem: it controls identifying routes and solver-side prior binding before questions reach a solver. It uses adversarially refined search trajectories to train FORT-Searcher with supervised fine-tuning and a context-managed inference protocol.

3.1. FORT: Shortcut-Resistant Data Synthesis

  • FORT combines long-tail root selection, evidence-graph construction, question formulation, and adversarial refinement to reduce route-level shortcuts and solver-side prior binding.Its internal graph organizes entities, verified facts, evidence sources, and dependencies before subgraphs are rendered as natural-language questions.Source: §3.1 · pp. 8–9

3.1.1. Graph Initialization

  • Graph initialization favors rare, evidence-supported root entities and less linearly exposed seed structures to limit prior binding and avoid revealing downstream constants.FORT uses a pre-mined cycle seed when possible, otherwise falling back to a single-node seed.Source: §3.1.1 · pp. 9–10

3.1.2. Graph Construction

  • Graph construction collects facts from diverse sources, derives multi-fact clues, verifies consistency, and favors generic facts so evidence is dispersed and individual clues are less identifying.Expanding the deepest unprocessed node first preserves referenced-entity chains that can become serial dependencies in the question.Source: §3.1.2 · pp. 10–12

3.1.3. Question Formulation

  • Question formulation selects jointly identifying but individually generic clues, withholds intermediate entity names, and fuzzes exposed values to prevent premature retrieval while preserving the gold answer.Fuzzing is selected according to each value’s semantic type, distinctiveness, and role; it is intended to avoid ambiguity or unverifiability.Source: §3.1.3 · pp. 12–13

3.1.4. Adversarial Refinement

  • Adversarial refinement accepts drafts only when a strong search agent answers correctly after sufficient retrieval turns and late answer discovery, without unsupported prior binding.Fast or prior-bound trajectories trigger shortcut repairs; unsolved drafts are narrowed or clarified so questions remain solvable but search-heavy.Source: §3.1.4 · pp. 13–14

3.2. Agent Training and Inference

  • FORT-Searcher is trained with supervised fine-tuning on FORT-generated trajectories to acquire evidence over multiple turns, while inference retains tool results within a rollout and restarts after the turn limit.This links data-side shortcut resistance to model behavior that delays answering until evidence is acquired.Source: §3.2 · p. 14

4. Experiment

FORT-Searcher achieves the highest overall score among comparable-size open-source agents, while context management yields its largest gains on BrowseComp and BrowseComp-ZH. The experiments evaluate five deep-search benchmarks and use comparable-size open-source agents as the primary fair-comparison group.

4.1. Experimental Setup

  • Evaluation spans five deep-search benchmarks, blocks Hugging Face domains, and treats comparable-size open-source agents as the primary fair-comparison group.The benchmarks cover long-tail entity identification, broader real-world search tasks, and reasoning with noisy or conflicting evidence; baselines include proprietary, larger open-source, and comparable-size open-source agents.Source: §4.1 · pp. 15–16

4.2. Main Results

  • With SFT only and about 3B active parameters, FORT-Searcher scores 75.0 on BrowseComp-ZH, the best open-source result listed, and matches or exceeds most larger open-source agents on both xbench benchmarks.On BrowseComp, it scores 72.2, outperforming DeepSeek-V3.2 (67.6), Step 3.5 Flash (69.0), LongCat-Flash-Thinking-2601 (67.6), Hy3-preview (67.1), and Qwen3.5-122B-A10B (63.8).Source: §4.2 · p. 16
  • FORT-Searcher scores 66.2 overall, beating comparable-size MiroThinker-1.7-mini by 1.6 points and Qwen3.5-35B-A3B by 6.3, ranking first on three benchmarks and tying xbench-DeepSearch-2510.The authors interpret the results as evidence that FORT’s shortcut-resistant synthesis improves deep-search behavior under SFT-only training.Source: §4.2 · pp. 16–17, Table 6 · p. 17

4.3. Context Management

  • Context management raises BrowseComp from 55.9 to 72.2 (+16.3) and BrowseComp-ZH from 62.1 to 75.0 (+12.9), versus gains of 0.7, 3.1, and 2.3 elsewhere.Clearing interaction history and restarting from the original question gives the model another search path; the authors say this matters more when limited rollout budgets encourage inefficient search.Source: §4.3 · p. 17, Table 7 · p. 18

5. Further Analysis

Across training-data comparisons, ablations, refinement, and trajectory diagnostics, the findings indicate that realized difficulty depends on extended pre-answer search and fewer shortcut opportunities, not trajectory length alone. The section evaluates these effects through training results and realized-search behavior.

5.1. Training-Data Difficulty Analysis

  • At similar bΩ = 140.0, FORT training data yields the best results, with 𝑇hit = 47.0 and b𝑝prior = 11.4, beyond moderate gains from longer open-source trajectories.The comparison suggests that useful training difficulty involves later answer exposure and reduced prior binding, not just longer trajectories.Source: Table 8 · p. 18, §5.1 · p. 18

5.2.1. Shortcut-Resistance Ablation

  • Cumulative ablation makes questions easier: accuracy rises from 29.0 to 81.6, bΩ falls from 141.9 to 43.7, and 𝑇hit shifts from 46.5 to 11.8.Removing fuzzing produces the largest drop in difficulty in this ablation order; b𝑝prior also increases from 11.4 to 22.3.Source: §5.2.1 · p. 18, Table 9 · p. 19

5.2.2. Adversarial Refinement

  • For shortcut-prone drafts, refinement raises bΩ from 33.9 to 82.7, delays 𝑇hit from 12.4 to 31.4, and lowers b𝑝prior from 17.0 to 12.0.Initially unsolved drafts become solvable after refinement, with bΩ = 123.0 and 𝑇hit = 50.2; refinement thus calibrates both types of drafts.Source: Table 10 · p. 19, §5.2.2 · p. 19

5.3. Dataset Difficulty Comparison

  • Against REDSearcher, FORT raises bΩ from 92.1 to 141.0 and 𝑇hit from 18.7 to 46.9 under the same diagnostic setting.Its b𝑝prior remains comparable to REDSearcher, so the higher solving cost is not driven by more explicit prior-bound behavior.Source: Table 11 · p. 20, §5.3 · p. 20

5.4. Trajectory-Level Proxies for Difficulty Factors

  • Because exact theoretical quantities were generally not enumerable or directly observable, GPT-5.5 inspected 200 successful question–trajectory pairs from each of open-source deep-search data and FORT as realized proxies, not direct estimates.Source: §5.4 · p. 20
  • Compared with open-source data, FORT moves all trajectory-level proxies in the desired direction: higher retrieval effort and evidence dispersion, less clue selectivity, costlier pre-answer dependencies, and fewer prior-bound answers.These are operational proxies for realized effects, not exact estimates of the theoretical difficulty quantities.Source: Table 12 · p. 21, §5.4 · p. 21

6. Conclusion

FORT-Searcher uses FORT’s shortcut-resistant supervision to train a deep search agent that achieves the best overall performance among comparable-size open-source search agents. The conclusion argues that deep search data should be judged by whether actual search requires the intended evidence acquisition, not merely by structural complexity.

Conclusion

  • Apparent structural complexity may not create realized search difficulty when evidence co-coverage, single-clue selectivity, exposed constants, or prior-knowledge binding enable cheap solutions.The authors argue that high-quality supervision should be judged by whether the intended evidence-acquisition process remains necessary during actual search.Source: §6 · p. 21
  • FORT controls shortcut risks through entity selection, evidence graph construction, question formulation, and adversarial refinement, producing trajectories focused on answer discovery rather than early exposure, post-hit verification, or parametric priors.FORT-Searcher is trained on these trajectories with supervised fine-tuning alone.Source: §6 · pp. 21–22
  • With supervised fine-tuning alone, FORT-Searcher achieves the best overall performance among comparable-size open-source search agents.The authors leave integrating reinforcement learning with FORT trajectories to future work.Source: §6 · pp. 21–22
  • The authors call for more efficient tool-augmented search and complex search-grounded tasks requiring heterogeneous evidence integration, conflict resolution, and well-supported decisions under uncertainty.They suggest shortcut-aware data construction and trajectory-level diagnostics could help train and evaluate these agents.Source: §6 · p. 22

A. Notation Summary · B. Formal Details for the Difficulty Framework

The formal details define a pure-posterior policy through restrictions on executable queries, evidence-based termination, and problem-specific prior knowledge. They also define valid evidence-acquisition routes for identifying subsets.

A. Notation Summary

  • Table 13 summarizes the main notation used for the shortcut-aware difficulty framework and trajectory diagnostics.Source: Table 13 · p. 26

B.1. Pure-Posterior Policy Class

  • The pure-posterior policy class Πpost is defined by three restrictions.Source: §B.1 · p. 26
  • At each history, a policy may issue only executable queries, using constants available from the question or previously retrieved evidence.The passage defines K(hₜ) as those available constants and Const(θ) as the constants needed to instantiate query θ.Source: §B.1 · p. 27
  • A policy may use basic reading and composition over retrieved evidence, but not question-specific entity knowledge or a memorized target profile.Source: §B.1 · p. 27

B.2. Valid Evidence-Acquisition Routes

  • A valid evidence-acquisition route for target y★ and identifying subset P under Σ is an ordered sequence of queries.Source: §B.2 · p. 27
  • Every query in a valid route must be executable when issued, and the accumulated evidence must verify that y★ satisfies every constraint in P.Source: §B.2 · p. 27

C. Proofs for the Difficulty Framework · C.1. Proof of the Structural Lower Bound

The proof establishes a structural lower bound on post-policy difficulty: successful trajectories must acquire executable evidence that verifies an identifying subset, so their cost is bounded below by the corresponding evidence-route cost.

C.1. Proof of the Structural Lower Bound

  • A successful trajectory under π ∈ Πpost can terminate with y★ only after accumulated evidence verifies an identifying subset P ∈ Iq.This follows from the no-guessing restriction in Appendix B.1.Source: §C.1 · p. 27
  • Every query in a successful trajectory must be executable when issued.This follows from the executable-query restriction.Source: §C.1 · p. 27
  • Together, the verification and executability conditions make the successful trajectory a valid evidence-acquisition route for (y★, P) under Σ.Source: §C.1 · p. 27
  • The trajectory length is at least QΣ(P), the cost of the corresponding evidence-acquisition route.Source: §C.1 · p. 27
  • The proof carries the route-cost lower bound through expectation over pure-posterior trajectories and the infimum over Πpost.This yields the stated difficulty bound.Source: §C.1 · p. 27
  • Dpost(q) ≥ Q★ for the infimum expected cost over pure-posterior trajectories.The proof obtains this bound by taking expectation over pure-posterior trajectories and then taking the infimum over Πpost.Source: §C.1 · p. 27

C.2. Proof of the Component Lower Bound

The proof establishes the component lower bound by relating the shortest valid route for an identifying subset to evidence-acquisition and dependency-depth constraints. It clarifies that s(P) does not directly lower-bound QΣ(P), but affects whether a small subset is identifying.

C.2. Proof of the Component Lower Bound

  • For an identifying subset P, the shortest valid evidence-acquisition route is lower-bounded by the minimum verification steps and its longest dependency chain, yielding the component bound in Eq. (12).The proof treats the route as a feasible sequence for defining M_ev(P), then combines the route-length and dependency-depth bounds.Source: §C.2 · p. 28
  • For fixed P, s(P) does not directly lower-bound QΣ(P); instead, it affects whether a small subset becomes identifying and enters the minimization defining Q★.Source: §C.2 · p. 28

C.3. Collapse of the Structural Lower Bound

The structural lower bound collapses when an identifying subset can be verified by one initially executable query: that query yields a one-step valid route. The proof also states that verifying evidence requires at least one retrieval step.

C.3. Collapse of the Structural Lower Bound

  • A single initially executable query that verifies an identifying subset yields a valid one-step route and QΣ(P) = 1.The proof notes that a retrieval step is required to verify evidence, so route cost cannot be below one.Source: §C.3 · p. 28

D. Shortcut Diagnostic Cases

The appendix presents representative trajectory-level examples from open-source data showing how apparently multi-constraint questions can be solved through cheaper identifying routes or solver-side prior binding.

D. Shortcut Diagnostic Cases

Related papers

Loading 2606.12087v1…