PaperlayerView exact revision on arXiv

Source-linked AI summary

FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents

Jia Deng, Yimeng Chen, Xiaoqing Xiang, Ziyang Zeng, Shuo Tang, Wayne Xin Zhao, Feng Chang, Chuan Hao, Yuan Wei, Ran Tao, Bryan Dai, Ji-Rong Wen

arXiv:2606.12087v1cs.CL

TL;DR

Deep-search training data can appear structurally complex yet remain easy through shortcut routes. FORT synthesizes shortcut-resistant data and trains FORT-Searcher, which achieves the best overall performance among comparable-size open-source search agents.

  • Problem

    Existing retrieval datasets often do not require long-horizon evidence acquisition, while structural complexity alone may not produce realized deep-search difficulty.

  • Method

    FORT controls shortcut risks across entity selection, evidence-graph construction, question formulation, and adversarial refinement to synthesize shortcut-resistant training data.

  • Results

    With supervised fine-tuning alone, FORT-Searcher achieves the best overall performance among comparable-size open-source search agents on challenging benchmarks.

  • Takeaways & Limitations

    Shortcut-resistant supervision can improve deep-search behavior by keeping answer discovery necessary during actual agent search.

  • Takeaways & Limitations

    Trajectory measurements are not direct estimates of the framework’s theoretical quantities.

Abstract

from arXiv · show

Training deep search agents requires verifiable questions whose answers remain unavailable until sufficient evidence has been acquired through search. Existing synthesis methods often increase apparent difficulty by enriching graph structures, but structural complexity alone does not guarantee realized search difficulty: the intended search process can collapse through a cheaper identifying route. We formalize this gap with a shortcut-aware difficulty framework and identify four actionable shortcut risks: evidence co-coverage, single-clue selectivity, exposed constants, and prior-knowledge binding. To diagnose their realized effects, we use trajectory signatures including solving cost, answer hit time, and prior-shortcut rate. Guided by this framework, we introduce FORT, a Framework of Shortcut-Resistant Training-Data Synthesis. FORT constructs shortcut-resistant training data by controlling shortcut risks across entity selection, evidence graph construction, question formulation, and adversarial refinement. Experiments show that FORT induces longer pre-answer search and fewer shortcut patterns than existing open-source deep search datasets. Using the resulting trajectories, we train FORT-Searcher with supervised fine-tuning (SFT) only, and it achieves the best overall performance among comparable-size open-source search agents on challenging deep search benchmarks. Relevant resources will be made available at https://github.com/RUCAIBox/FORT-Searcher.

1. Introduction

The introduction argues that deep-search training requires verifiable questions demanding long-horizon evidence acquisition, because structural complexity alone may not prevent cheaper identifying shortcuts. It presents a shortcut-aware framework and FORT, which controls shortcut risks during synthesis and yields stronger search-agent supervision.

  • Motivation: Existing traditional retrieval datasets often fail to require the long-horizon evidence acquisition needed to train effective search agents.Such data can support supervised fine-tuning or reinforcement learning, but does not match deep-search requirements.
  • Problem: Enriching hop count, graph shape, hierarchical constraints, evidence dispersion, or treewidth increases intended complexity but may not increase realized search difficulty.The intended question-to-answer structure can diverge from the actual retrieval process available to the solver.
  • Framework: The shortcut-aware framework defines difficulty through both intended task complexity and the cheapest identifying route available in the realized retrieval environment.It diagnoses evidence co-coverage, single-clue selectivity, exposed constants, and prior-knowledge binding as actionable shortcut risks.
  • FORT: FORT controls shortcut risks across entity selection, evidence-graph construction, question formulation, and adversarial refinement.Its procedures include selecting long-tail root entities, using heterogeneous sources, constructing non-verbatim derived facts, and repairing shortcut-prone or ambiguous drafts.
  • Results: With supervised fine-tuning alone, FORT-Searcher achieves the best overall performance among comparable-size open-source agents on challenging deep-search benchmarks.Analyses also find longer pre-answer search and fewer shortcut patterns than existing open-source deep-search datasets.

2. Difficulty Framework

The difficulty framework separates task-imposed search cost from solver-specific shortcutting. It models the task-side floor through the cheapest executable evidence route that identifies the answer, while diagnosing additional exploration and realized-trajectory behavior through selectivity, evidence dispersion, dependency depth, and answer availability.

  • Difficulty sources: Task difficulty has objective and solver-specific sources: constraint-and-interface structure imposes retrieval cost, while prior knowledge or unsupported early commitment can reduce realized cost.The framework derives a lower bound for no-prior, evidence-only solvers and allows concrete solvers to bypass part of the search.
  • Task-side cost: The pure-posterior floor is the minimum shortest valid evidence-acquisition route over all constraint subsets that uniquely identify the gold answer.Valid routes must verify the answer, respect query executability, and use only constants available from the question or earlier evidence.
  • Objective factors: Selectivity measures remaining candidate breadth and governs exploration beyond the cheapest route, while also determining which identifying subsets enter the route minimization.A selective clue can uniquely identify the answer without itself lower-bounding route length.
  • Objective factors: Evidence dispersion counts distinct retrievals needed to verify a clue subset, and query executability can only increase that cost, yielding Q_Σ(P) ≥ M_ev(P).When one page states multiple required facts, the verification may need only one retrieval.
  • Objective factors: Dependency depth measures the longest serial chain of queries requiring intermediate facts, and exposing an intermediate constant can shorten that chain.For example, discovering an advisor’s name can be required before searching for the advisor’s other students.
  • Realized search: Trajectory length alone is insufficient to establish discovery difficulty, because the answer may appear early while later retrievals serve verification or detours.Answer availability is therefore measured by the first retrieval step whose observation contains the answer.

3. Methodology

FORT is an inverse construction framework that synthesizes questions and trajectories while reducing route-level shortcuts and solver-side prior binding. It uses an evidence graph and a four-stage pipeline, then trains FORT-Searcher with SFT-only and context-managed inference.

  • Framework objective: FORT constructs shortcut-resistant questions by making cheap identifying routes less available and solver-side prior binding less likely.The framework targets both route-level cost collapse and solver-side cost reduction from prior binding.
  • Methodology overview: The methodology has two components: shortcut-resistant question and trajectory synthesis, followed by SFT-only training of FORT-Searcher with context-managed inference.This separation connects data-side shortcut reduction with agent training.
  • Evidence graph: FORT uses an internal evidence graph whose entity nodes, verified-fact edges, evidence sources, and dependencies organize construction before the final question is verbalized.The graph is a construction workspace rather than the object solved by the final agent.
  • FORT pipeline: The four-stage pipeline initializes a long-tail root, expands a heterogeneous evidence graph, formulates questions, and applies adversarial refinement to control shortcut risks.Graph initialization reduces prior binding and premature constants, while graph construction disperses evidence and avoids overly identifying clues.
  • Graph initialization: Root selection prefers rare, externally supported entities, especially those without an English Wikipedia page, while cycle seeds reduce exposed constants by avoiding purely linear structures.Entities lacking sufficient external evidence are discarded to preserve solvability.
  • Graph construction: Evidence expansion targets evidence co-coverage and single-clue selectivity through diverse sources, derived facts, verification, and deepest-first expansion within depth and node budgets.The deepest-first policy preserves multi-step referenced-entity chains for serial dependencies in the final question.

4. Experiment

FORT-Searcher is evaluated across five challenging deep-search benchmarks against proprietary and open-source agents, including a comparable-size group for fair comparison. It achieves the strongest overall comparable-size open-source performance, while context management substantially improves results on BrowseComp and BrowseComp-ZH.

  • Benchmarks: Evaluation covers five benchmarks: BrowseComp, BrowseComp-ZH, xbench-DeepSearch-2505, xbench-DeepSearch-2510, and Seal-0.BrowseComp and BrowseComp-ZH focus on long-tail entity identification from indirect and range-based constraints, while the xbench tasks cover broader real-world deep search.
  • Main results: FORT-Searcher achieves the highest comparable-size open-source overall score of 66.2, outperforming MiroThinker-1.7-mini by 1.6 points and Qwen3.5-35B-A3B by 6.3 points.It ranks first on BrowseComp, BrowseComp-ZH, and xbench-DeepSearch-2505, and ties the best result on xbench-DeepSearch-2510.
  • Main results: Despite activating about 3B parameters and using SFT only, FORT-Searcher outperforms several larger open-source agents on BrowseComp and achieves 75.0 on BrowseComp-ZH, the best listed open-source result.The compared BrowseComp agents include DeepSeek-V3.2, Step 3.5 Flash, LongCat-Flash-Thinking-2601, Hy3-preview, and Qwen3.5-122B-A10B.
  • Context management: Context management raises BrowseComp from 55.9 to 72.2, an absolute improvement of 16.3 points, and BrowseComp-ZH from 62.1 to 75.0, an improvement of 12.9 points.Improvements are smaller on xbench-DeepSearch-2505, xbench-DeepSearch-2510, and Seal-0: 0.7, 3.1, and 2.3 points, respectively.

5. Further Analysis

Further analyses show that realized search difficulty depends on delayed answer exposure and limited shortcut access, not trajectory length alone. FORT’s components and refinement consistently increase pre-answer search effort while aligning trajectory-level proxies with the intended difficulty factors.

  • Trajectory-signature training difficulty: At matched average solving cost, FORT data achieves stronger downstream results by delaying answer exposure and reducing prior-shortcut rates.Open-source data rising from 40.0 to 140.0 turns brings only moderate gains, whereas FORT at bΩ = 140.0 has 𝑇hit = 47.0 and b𝑝prior = 11.4.
  • Component ablation: Cumulative ablation makes questions easier as shortcut-resistant components are removed, with accuracy increasing from 29.0 to 81.6 while difficulty signatures deteriorate.Across the ablation, bΩ decreases from 141.9 to 43.7, 𝑇hit shifts from 46.5 to 11.8, and b𝑝prior increases from 11.4 to 22.3; removing fuzzing causes the largest difficulty drop in this order.
  • Adversarial refinement: Adversarial refinement suppresses shortcut-prone routes and rescues initially unsolved drafts while retaining substantial search difficulty.For shortcut-prone drafts, bΩ rises from 33.9 to 82.7, 𝑇hit is delayed from 12.4 to 31.4, and b𝑝prior decreases from 17.0 to 12.0; initially unsolved drafts become solvable after refinement.
  • Dataset trajectory diagnosis: Under the same diagnostic setting, FORT produces the highest solving cost and latest answer exposure among compared datasets, extending pre-answer search rather than post-hit detours.Against REDSearcher, bΩ increases from 92.1 to 141.0 and 𝑇hit from 18.7 to 46.9, while b𝑝prior remains comparable.
  • Trajectory-level diagnosis: Trajectory-level proxies indicate that FORT increases retrieval effort, clue dispersion, dependency-chain cost, and resistance to prior binding while reducing individual-clue selectivity.These measurements are operational proxies rather than direct estimates of the theoretical quantities, based on annotated successful question–trajectory pairs.

6. Conclusion

FORT-Searcher introduces shortcut-resistant search supervision through FORT, which controls shortcut risks so structural complexity translates into realized search difficulty. Trained with supervised fine-tuning alone, it improves deep search behavior while motivating future work on reinforcement learning, richer tools, and harder search-grounded tasks.

  • Contribution: FORT-Searcher is a deep search agent trained with FORT, a framework for generating shortcut-resistant, search-heavy supervision.FORT is presented as the paper’s central contribution.
  • Motivation: Structural complexity alone may not create realized search difficulty because solvers can exploit evidence co-coverage, single-clue selectivity, exposed constants, or prior-knowledge binding.Questions with many clues, long latent chains, or dense evidence graphs may still be solved cheaply through these shortcuts.
  • Method: FORT controls shortcut risks across entity selection, evidence graph construction, question formulation, and adversarial refinement.The resulting trajectories emphasize answer discovery rather than early answer exposure, post-hit verification, or parametric priors.
  • Empirical result: Using supervised fine-tuning alone, FORT-Searcher achieves the best overall performance among comparable-size open-source search agents.The result indicates that shortcut-resistant supervision can improve deep search behavior.
  • Future work: Future deep search agents should pursue efficient tool-augmented search and complex tasks requiring heterogeneous evidence integration, conflict resolution, and supported decisions under uncertainty.The paper also leaves reinforcement learning with FORT trajectories for future work.

A. Notation Summary · B. Formal Details for the Difficulty Framework

The section introduces notation for shortcut-aware difficulty and trajectory diagnostics, then formally defines admissible policies and valid evidence-acquisition routes. These definitions require executable queries, evidence-based termination, and no problem-specific prior knowledge.

  • A. Notation Summary: The notation summary specifies symbols used throughout the shortcut-aware difficulty framework and trajectory diagnostics.It provides the main notation needed to describe difficulty and search trajectories.
  • B.1. Pure-Posterior Policy Class: A pure-posterior policy belongs to Πpost only if it satisfies three formal restrictions.The restrictions define the admissible policy class for the framework.
  • B.1. Pure-Posterior Policy Class: At each history h_t, a policy may issue only executable queries whose required constants are available.K(h_t) contains constants from the question and previously retrieved evidence, while Const(θ) lists constants needed to instantiate θ.
  • B.1. Pure-Posterior Policy Class: A policy may terminate with answer x only after accumulated evidence E verifies an identifying subset for x.The formal condition requires some P ⊆ C_q to be verified by the retrieved evidence.
  • B.1. Pure-Posterior Policy Class: A pure-posterior policy cannot use entity-level prior knowledge specific to the current question.It may perform basic reading and composition over retrieved evidence, but cannot name the target from memory or use a memorized profile.
  • B.2. Valid Evidence-Acquisition Routes: For an identifying subset P ∈ I_q, a valid route for (y★, P) under Σ is an ordered sequence of queries.The route formalization describes how evidence is acquired for a target and identifying subset.
  • B.2. Valid Evidence-Acquisition Routes: Executing each query θ_j returns Retrieve_Σ(θ_j), and route validity requires every query to be executable when issued.Executability is evaluated against the history containing the question and all evidence retrieved before θ_j.
  • B.2. Valid Evidence-Acquisition Routes: A route is valid only when its accumulated evidence verifies that y★ satisfies all constraints in P.This condition connects evidence acquisition to the identifying subset required for termination.

C. Proofs for the Difficulty Framework · C.1. Proof of the Structural Lower Bound

The structural lower-bound proof shows that every successful pure-posterior trajectory must follow a valid evidence-acquisition route for an identifying subset. Consequently, its expected optimal difficulty is bounded below by the minimum structural route cost.

  • C.1. Proof of the Structural Lower Bound: Any successful trajectory under π∈Πpost can terminate with y★ only after accumulated evidence verifies an identifying subset P∈I_q.This follows from the no-guessing restriction.
  • C.1. Proof of the Structural Lower Bound: The executable-query restriction ensures that every query in the trajectory is executable when issued.
  • C.1. Proof of the Structural Lower Bound: Therefore, each successful trajectory realizes a valid evidence-acquisition route for (y★, P) under Σ.
  • C.1. Proof of the Structural Lower Bound: The trajectory length is at least Q_Σ(P), the structural cost of its verified identifying subset.
  • C.1. Proof of the Structural Lower Bound: Taking expectations over pure-posterior trajectories preserves the resulting lower-bound relationship for D^post(q).
  • C.1. Proof of the Structural Lower Bound: Taking the infimum over Πpost yields D^post(q) ≥ Q★.

C.2. Proof of the Component Lower Bound

This subsection proves Eq. (12) by analyzing a shortest valid evidence-acquisition route for an identifying subset. It shows that evidence-step count and dependency depth constrain route length, whereas s(P) affects identifying-subset eligibility indirectly.

  • Route-based lower bound: Proposition 3 analyzes any identifying subset P ∈ I_q using a shortest valid evidence-acquisition route for (y★, P) under Σ.The proof uses the definitions of QΣ(P), M_ev(P), and dep(P) from Section 2.2.
  • Route-based lower bound: Because the route verifies all constraints in P, its number of acquisition steps is at least the minimum verification cost M_ev(P).The route is feasible for the optimization defining M_ev(P), so the minimum cannot exceed its length.
  • Route-based lower bound: The route length is also at least depth(θ★), because its longest dependency chain is contained within the route.Dependency depth is defined by the longest chain among the route’s queries.
  • Scope of the bound: The resulting bound applies to M_ev and dep as route-length constraints for a fixed identifying subset, not directly to QΣ(P).Equations (29) and (33) are combined to obtain the final implication.
  • Scope of the bound: In contrast, s(P) determines whether a small subset becomes identifying and therefore enters the minimization defining Q★.Thus, s(P) influences the optimization indirectly rather than serving as a direct lower bound on QΣ(P).

C.3. Collapse of the Structural Lower Bound

The structural lower bound collapses when an identifying subset can be verified by one initially executable query. Such a shortcut yields a valid one-step route, while evidence verification requires at least one retrieval step.

  • An identifying subset P ∈ I_q that is verifiable by a single initially executable query triggers the collapse of the structural lower bound.
  • The single query forms a valid route θ = (θ_1) for (y★, P), so Q_Σ(P) = 1.
  • Because at least one retrieval step is required to verify evidence, the one-step route establishes the minimum route cost.

D. Shortcut Diagnostic Cases

The appendix presents four trajectory-level diagnostic cases showing how apparently multi-constraint questions can be solved through cheaper identifying routes or solver-side prior binding. The cases correspond to evidence co-coverage, single-clue selectivity, exposed constants, and prior-knowledge binding shortcuts.

  • Diagnostic overview: The appendix frames four representative trajectory-level cases around the shortcut risks of evidence co-coverage, single-clue selectivity, exposed constants, and prior-knowledge binding.These examples are sampled from open-source data and illustrate cheaper identifying routes or solver-side prior binding.
  • Case 1: Evidence Co-coverage Shortcut: The evidence co-coverage case asks for a corporate entity linked to Pacific-region holdings, Apia ventures, and a hospitality entrepreneur.Its constraints span formal establishment, commercial lineage, location, and the founder’s career.
  • Case 2: Single-clue Selectivity Shortcut: The single-clue selectivity case defines a mid-twentieth-century year through four converging threads involving an international convention, a UN agency, and a later UN-related organization.The clues also connect the organization’s institutional history to the first human-crafted object reaching another celestial body.
  • Case 3: Exposed-constant Shortcut: The exposed-constant case identifies a forestry-union regional leader through an ambiguous acronym, a late-1980s role, and associations with a British Columbia tourism-region city.The city’s name derives from an Indigenous word for “grizzly bear,” and the question continues into provincial labor legislation.
  • Case 4: Prior-knowledge Binding Shortcut: The prior-knowledge binding case asks for a legendary queen using a coastal town, a telegraph-era mechanical marvel, a Roman conqueror, and a naval battle as cues.The historical chain runs from the conqueror’s crossing of a boundary river to conflicts ending in the queen’s demise.
Loading 2606.12087v1…