LexiHorizon: Stabilizing Reinforcement Learning for Long-Horizon Deep Search

1 INTRODUCTION

• Query-sensitive retrieval can miss relevant evidence because of differences in entity names, aliases, terminology, or keyword combinations, requiring repeated query reformulation and producing long trajectories.Long trajectories may exhaust context before sufficient evidence is collected or a final answer is produced, while accumulated tool outputs can interfere with later query formulation.Source: §1 · pp. 1–2
• LEXIHORIZON expands the trajectory context budget, keeps complete reasoning history while retaining only recent tool observations in full, and gives a bounded search-effort bonus only to trajectories with nonzero answer reward.The policy is optimized with GSPO over the resulting trajectories to learn evidence selection and query adaptation.Source: §1 · p. 2
• Across XBench, WebWalkerQA, and BrowseComp-ZH, LEXIHORIZON-9B achieves maximum absolute improvements of 8.7 percentage points over Qwen3.5-9B and 23.8 percentage points over MiroThinker-1.7-mini.The model consistently outperforms both comparison models across all three benchmarks.Source: §1 · p. 2
• The paper frames long-horizon search reinforcement learning as a stability problem involving premature context exhaustion, interference from accumulated retrieval content, and weak incentives for sustained exploration.Its proposed framework combines expanded context, reasoning-preserving context management, and outcome-gated reward to address these coupled challenges.Source: §1 · p. 2

2 RELATED WORK

• Proprietary deep-search systems retrieve from heterogeneous sources and synthesize research reports, but their models, training procedures, and extended-trajectory mechanisms remain largely inaccessible.Source: §2 · p. 2
• Open research agents explore question decomposition, parallel search, search-augmented reasoning, and iterative search-and-draft workflows, while related work builds training data and trajectories for broader research workflows.Source: §2 · p. 2
• Reinforcement learning research progresses from outcome-based search behavior to denser process rewards and realistic web interaction, while longer trajectories raise optimization-stability and context-management challenges.The section states that scaling to substantially longer search trajectories while maintaining stable optimization under real retrieval feedback remains insufficiently understood.Source: §2 · p. 3

3.1 OVERVIEW AND PROBLEM FORMULATION

• The agent follows ReAct, interleaving reasoning with tool calls or a final answer, while query reformulation is learned jointly with search planning under finite interaction and context budgets.The formulation considers lexically sensitive retrieval, where semantically similar queries can return substantially different results.Source: §3.1 · p. 3

3.2 LONG-HORIZON SEARCH TRAINING

• LEXIHORIZON pairs an extended context budget with a reasoning-preserving context window and a bounded search-effort bonus gated on nonzero answer reward.Trajectory reset is an auxiliary mechanism for pathological long-tail trajectories.Source: §3.2 · p. 4

3.2.1 REASONING-PRESERVING CONTEXT WINDOW

• The window keeps reasoning and tool actions intact, retains full observations from the K most recent interactions, and replaces earlier raw observations with markers.This preserves information already incorporated into reasoning while reducing redundant historical retrieval content in the active context.Source: §3.2.1 · p. 4

3.2.2 OUTCOME-GATED SEARCH-EFFORT REWARD

• The search-effort bonus scales with answer reward, saturates after 30 tool calls, and is paired with format and repetition penalties.Zero-answer-reward trajectories receive no positive tool bonus; penalties address malformed or repetitive trajectories.Source: §3.2.2 · pp. 4–5

3.2.3 TRAINING OBJECTIVE

• Training uses live tool-interaction rollouts and GSPO, computes group-normalized advantages, and applies policy loss only to policy-generated tokens.Tool observations provide conditioning context but are excluded from the loss.Source: §3.2.3 · pp. 5–6
• Benchmark accuracy is averaged over eight independently sampled trajectories per question, while reliability and efficiency are reported under matched evaluation configurations.Policy length excludes tool-observation tokens; throughput is measured in policy-generated tokens per second.Source: Table 1 · p. 6, Table 2 · p. 6

4.1 EXPERIMENTAL SETUP

• The evaluation compares Qwen3.5-9B-based LEXIHORIZON-9B with its base model, a larger model, and a specialized search agent on XBench, WebWalkerQA, and BrowseComp-ZH.All models use the same search engine, tools, interaction budget, decoding configuration, and answer evaluator.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• LEXIHORIZON-9B ranks first or second across three benchmarks, improving accuracy over MiroThinker-1.7-mini by 23.8, 4.6, and 6.5 percentage points on XBench, WebWalkerQA, and BrowseComp-ZH, respectively.It also outperforms Qwen3.5-27B on XBench and BrowseComp-ZH, though that larger model scores highest on WebWalkerQA.Source: §4.2 · pp. 6–7
• Compared with Qwen3.5-9B, LEXIHORIZON-9B reduces tool-call failures from 12.80% to 2.00%, raises valid answers from 87.60% to 99.88%, and reduces policy-generated tokens by 38.4%.Under the same hardware allocation and evaluation configuration, it reaches 1,069 policy-generated tokens per second and 0.169 completed trajectories per second.Source: §4.2 · p. 7

4.3 ABLATION STUDIES

• Reducing the context budget from 128K to 64K lowers the valid-answer rate from 99.88% to 66.97%, while removing the context window also reduces accuracy across all three benchmarks.The ablations compare the 128K windowed full configuration with variants that remove either the extended horizon or the window, holding other settings fixed.Source: §4.3 · p. 7

4.3.1 EFFECT OF THE SEARCH HORIZON

• Under 64K, persistent length clipping precedes sharp declines in response length and tool use; 128K sustains longer responses and gradually increasing, stabilizing tool use.The 64K policy's reward and actor entropy also decline near zero, whereas 128K reward generally improves without the same abrupt entropy decline.Source: §4.3.1 · p. 7, Figure 1 · p. 8, Figure 2 · p. 8

4.3.2 EFFECT OF THE TOOL CONTEXT WINDOW

• The window retains recent tool observations in full while preserving complete reasoning and action history, limiting accumulation of verbose historical observations.The reported analysis suggests this reduces interference and is consistent with more sustained tool interaction and a shorter response-length upper tail.Source: §4.3.2 · pp. 8–9
• The context window activates in more than 80% of evaluation cases and reduces overall active context by 79.1%; reasoning retains relevant evidence in 44 of 55 evidence-bearing removed observations.All 11 retention failures occur in trajectories with more than 40 tool calls.Source: §4.3.2 · p. 8, Table 4 · p. 9
• With the window, test-time tool calls rise from 29.5 to 54.9 while the 95th-percentile active context falls from 120,818.9 to 28,324.5 tokens.During training, window-enabled reward rises steadily while reward without the window remains largely flat; the two configurations use the same outcome-gated search-effort reward.Source: §4.3.2 · pp. 8–9, Table 5 · p. 9

• Across XBench, WebWalkerQA, and BrowseComp-ZH, the resulting 9B policy consistently outperforms its base model and MiroThinker-1.7-mini.Controlled ablations indicate that context capacity and context management provide complementary benefits for long search trajectories.Source: §5 · p. 9
• LEXIHORIZON combines an extended context budget, a reasoning-preserving context window, and an outcome-gated search-effort reward.Source: §5 · p. 9
• The findings underscore balancing context capacity and composition to support deeper search without a proportional increase in active context length.Source: §5 · p. 9

• The context-window activation metric records whether the window has been activated for each rollout.Source: §A.1 · p. 12
• The corresponding window activation rate is defined over a set of N trajectories.Source: §A.1 · p. 12

• The answer-correctness judge receives the original question, reference answer, and extracted final answer.Source: §A.2 · p. 12
• The judge assigns an ordinal score from {0, 0.25, 0.5, 0.75, 1} based on semantic agreement between final and reference answers.Source: §A.2 · p. 12
• The prompt treats evaluation fields as untrusted data and tells the judge to ignore instructions embedded in model answers.Source: §A.2 · p. 12, Figure 4 · p. 13

• Repeated query reformulation after retrieval failures can produce excessively long trajectories, especially when the agent cannot tell whether failure reflects its search hypothesis or query wording.Source: §A.3.1 · p. 13
• A length-triggered reset mechanism is introduced to handle pathological long-tail trajectories.Source: §A.3.1 · p. 13
• Discarded trajectory segments contribute to rollout cost but are excluded from the active trajectory used for policy optimization.Source: §A.3.1 · p. 14
• Reset abandons the current search path, whereas the context window controls retained information within an active trajectory.Source: §A.3.1 · p. 14

A.3.2 EXPERIMENT

• Reset improves BrowseComp-ZH by 1.9 percentage points, leaves WebWalkerQA unchanged, and decreases XBench by 2.3 percentage points.The experiment uses an active trajectory threshold of 32,768 response tokens and allows at most three resets per rollout.Source: Table 6 · p. 14, §A.3.2 · p. 14
• With reset disabled during evaluation for both policies, the reset-trained policy produces a lighter upper tail, with extremely long responses less frequent.This difference suggests reset affects learned search behavior rather than merely truncating evaluation trajectories.Source: §A.3.2 · p. 14, Figure 5 · p. 14
• Trajectory reset reduces average active response length throughout training by restarting trajectories that exceed the predefined threshold.This limits the response content retained for policy optimization.Source: §A.3.2 · p. 15
• Under the same hardware and batch settings, reset training requires less wall-clock time per token, despite generation costs for discarded segments.Reset stops an unproductive active path from continuing indefinitely and reduces observed rollout-generation cost per token.Source: §A.3.2 · p. 15