PaperlayerView exact revision on arXiv

Source-linked AI summary

AREX: Towards a Recursively Self-Improving Agent for Deep Research

Shuqi Lu, Chaofan Li, Kun Luo, Zhang Zhang, Hui Wang, Hongwang Xiao, Lei Xiong, Jiahao Wang, Sen Wang, Xiyan Jiang, Wanli Li, Yuyang Hu, Hongjin Qian, Bingyu Yan, Jianlyu Chen, Ziyi Xia, Yingxia Shao, Kang Liu, Zhicheng Dou, Di He, Chaozhuo Li, Qiwei Ye, Zhongyuan Wang, Zheng Liu

arXiv:2607.21461v3cs.AI

TL;DR

Deep research requires satisfying multiple coupled constraints, while candidate discovery is costly even when verification can be broken into simpler checks. AREX recursively refines provisional answers through constraint-wise audits and targeted research, and achieves consistent gains over comparable-scale baselines across deep-research, wide-search, reasoning, and tool-use benchmarks.

  • Problem

    Deep research must satisfy multiple coupled constraints, making candidate discovery costly even though verification can often be decomposed into simpler constraint-wise checks.

    Source: §1 · p. 2

  • Method

    AREX alternates evidence-gathering research with constraint-wise audits that direct targeted follow-up, while preserving verified progress in a compact improvement state.

    Source: §1 · pp. 2–3

  • Results

    AREX demonstrates consistent gains over comparable-scale baselines across deep-research, wide-search, reasoning, and tool-use benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors highlight the importance of reinforcing decision-critical steps rather than treating all intermediate actions uniformly in long-horizon trajectories.

    Source: §6 · p. 14

    AREX task synthesis assumes answers cannot be inferred directly from queries, each constraint is verifiable from evidence, and joint constraints uniquely identify the answer.

    Source: §3.1 · p. 7

Abstract

from arXiv · show

Deep research requires agents to find answers that jointly satisfy multiple constraints. Discovering such answers is costly, whereas verifying a candidate can often be decomposed into tractable constraint-wise checks. This discovery--verification asymmetry suggests that a research agent should do more than simply search longer: it should recursively improve its current answer by verifying intermediate results and using the partially verified state to guide subsequent refinement. We introduce AREX, a family of Recursively Self-Improving (RSI) deep research agents. AREX alternates between an inner research loop that gathers evidence and constructs a provisional answer, and an outer self-improvement loop that audits the answer constraint-wise, identifies unresolved claims, and launches targeted follow-up research. To sustain RSI over long horizons, AREX learns an autonomous context-update tool that compresses growing interaction history into a compact improvement state preserving verified evidence and unresolved constraints, without relying on an external model. We train AREX on verified synthetic tasks and high-quality trajectories through agentic mid-training and long-horizon reinforcement learning. To mitigate sparse final rewards during long horizon learning, we emphasize key steps where decisive evidence is acquired or erroneous research directions are corrected. We instantiate a dense 4B model and a 122B-A10B Mixture-of-Experts model. Across BrowseComp, WideSearch, DeepSearchQA, Humanity's Last Exam (HLE), and other reasoning and tool-use benchmarks, AREX substantially outperforms comparable-scale baselines and remains competitive with models using substantially more activated parameters.

1 Introduction

AREX reframes multi-constraint deep research as recursive refinement: constraint-wise verification turns a provisional answer into a partially verified state that guides the next research round. It combines this process with autonomous context management and long-horizon training.

1 Introduction

  • Because finding answers that satisfy coupled constraints is costly while checking candidates can be decomposed into simpler constraint-wise checks, verification can guide targeted research rather than merely filter completed answers.The partially verified state preserves supported claims, isolates unresolved or conflicting evidence, and defines the next research problem.Source: §1 · pp. 2–3
  • AREX alternates an inner loop that gathers evidence and builds a provisional answer with an outer loop that audits constraints, targets weak claims, and uses belief estimates to govern continuation and termination.Verified progress carries across rounds, and sufficiently supported claims allow the process to terminate.Source: §1 · pp. 2–3
  • AREX autonomously uses a learned context-update tool to compress interaction history into an objective-aligned state preserving verified evidence and citations, constraint status, information gaps, and the next research plan.The tool is produced by AREX itself rather than an external model, and is designed to keep the compressed state aligned with evolving beliefs and actions.Source: §1 · pp. 2–3
  • AREX training combines verified synthetic tasks and high-quality trajectories with supervised capability acquisition, agentic mid-training, and long-horizon reinforcement learning, emphasizing steps that acquire evidence, resolve contradictions, or repair mistaken directions.This critical-step exposure addresses sparse credit assignment because final rewards alone do not identify which intermediate actions produced progress.Source: §1 · pp. 2–3
  • AREX’s dense 4B Turbo and 122B-A10B Base variants show consistent gains over comparable-scale baselines across deep-research, wide-search, reasoning, and tool-use benchmarks.Base has 122B total parameters and 10B activated parameters; the evaluated benchmarks include BrowseComp, WideSearch, DeepSearchQA, Humanity’s Last Exam (with Tools), GAIA, and xbench-DeepSearch-2510.Source: §1 · pp. 2–3

2 Recursive Self-Improvement

AREX uses nested research and self-improvement loops to build a provisional answer, audit its confidence and unresolved constraints, then accept, refine, or restart. It supports long-horizon research by autonomously refreshing research context and externalizing structured answers for evaluation.

2.1 Overall Framework

  • AREX alternates an inner loop that gathers evidence and constructs a provisional answer with an outer loop that accepts sufficiently confident answers or refines or restarts weaker trajectories.The outer loop preserves useful findings for targeted refinement and restarts from the original problem when the trajectory is noisy or uninformative.Source: §2.1 · p. 3

2.2 Inner Research Loop

  • The inner loop searches, integrates evidence, and adapts its next steps to unresolved constraints, contradictions, and candidate invalidations until the objective is investigated or further search is unlikely to help.It can prioritize stronger, more direct, or more temporally relevant evidence, and broaden or reformulate the search when no plausible candidate remains.Source: §2.2 · p. 4

2.2.1 Autonomous Context Updating

  • AREX’s autonomous update_context tool refreshes long research trajectories into compact states that preserve verified findings, provenance, candidates, unresolved constraints, validity concerns, and next steps.It removes redundant observations and obsolete conclusions or plans; the model decides when to invoke it, supporting reuse of progress across recursive rounds.Source: §2.2.1 · p. 5

2.2.2 Structured Answer Externalization

  • The finish interface externalizes a provisional answer, its supporting evidence and document identifiers, and an answer-level confidence score for outer-loop evaluation.The confidence score estimates completeness, consistency, provenance, and temporal validity; finishing ends the current inner loop, not the recursive process.Source: §2.2.2 · p. 5

2.3 Outer Self-Improvement Loop

  • The outer loop accepts answers meeting its confidence threshold; otherwise, it preserves useful findings and targets remaining issues for refinement, or discards an unrecoverable trajectory and restarts from the original problem.After each round it repeats this decision up to a predefined maximum, returning the highest-confidence answer if none meets the threshold.Source: §2.3 · p. 6

3 Training Data Construction

AREX’s training data combines verifiable research tasks with quality-controlled teacher trajectories collected in the same tool environment. The pipeline is designed to teach iterative evidence gathering, answer construction, and adaptive research.

3.1 Recursive Research Task Synthesis

  • Tasks span browse-intensive, reasoning-intensive, and scientific-literature research, with transformed constraints requiring multi-hop investigation and jointly identifying a unique, evidence-verifiable answer.Experts define task templates and instances are generated from real-world sources; automated checks and independent rollouts remove ambiguous, shallow, or unsolvable tasks.Source: §3.1 · pp. 6–7

3.2 Teacher Trajectory Collection and Quality Control

  • For each task, teacher models use AREX’s research environment, and their trajectories record actions, tool calls, observations, intermediate analysis, and structured answers.These traces expose intermediate behaviors needed for long-horizon research, unlike standard input-output supervision.Source: §3.2 · p. 7
  • Trajectories are retained only when investigation adapts to evidence and the final answer is reconstructable from collected evidence, with supporting evidence and confidence recorded.Filtering removes ineffective exploration, ignored observations, unsupported or contradictory claims, and unjustified assumptions; the resulting data supports evidence acquisition, research-state maintenance, answer construction, and adaptive continuation.Source: §3.2 · p. 7

4 Training Pipeline

AREX progressively builds research and reasoning capabilities, then focuses supervised training and reinforcement learning on verified, decision-critical steps rather than treating all trajectory actions alike.

4.1 Multi-stage Agentic Mid-training

  • Mid-training first develops browsing and evidence-acquisition skills, then expert reasoning, before mixed-capability consolidation replays informative research steps and adds harder tasks.Consolidation also trains verification-driven transitions that preserve verified evidence, identify unresolved conditions, and formulate targeted follow-up research.Source: §4.1 · p. 8
  • Rule-based detectors identify key steps such as evidence discovery, rejection of incorrect hypotheses, and context updates; annotations require grounded evidence and a verified final answer.Annotations are created offline for tasks with verifiable reference answers and are not used at inference.Source: §4.1 · pp. 8–9
  • After full-trajectory training, detected key steps still have substantially higher loss, and adding supervision only to these steps further improves long-horizon deep-research performance.The selected step’s preceding context is preserved, while loss is applied only to that step.Source: §4.1 · p. 9

4.2 Step-aware Reinforcement Learning

  • Step-aware reinforcement learning normalizes policy ratios by step length and averages over steps before trajectories, limiting domination by longer trajectories.This addresses the coarse credit assignment of sequence-level advantages in long-horizon tool-use tasks.Source: §4.2 · pp. 9–10
  • AREX adds a bounded bonus to annotated key steps only in successful trajectories, while keeping final-answer correctness as the primary reinforcement-learning objective.The shaping signal is a targeted auxiliary preference for identifiable, decision-critical steps, not a general solution to intermediate-action credit assignment.Source: §4.2 · p. 10

5 Experiments

AREX performs strongly across diverse search-augmented reasoning benchmarks, while controlled analyses examine how context updating, recursive refinement, and training choices contribute to its performance. The experiments also characterize how ACU is used and what information it preserves.

5.1 Experimental Setup

  • The experiments cover six benchmarks spanning deep research, agentic task completion, broad-coverage retrieval, and high-level reasoning, using a unified search-agent interface.WideSearch uses Item-F1, DeepSearchQA uses F1, and the other benchmarks use accuracy; episodes allow up to 300 inner-loop turns and 5 outer-loop operations.Source: §5.1 · pp. 10–11

5.2 Overall Performance

  • AREX-Base achieves the best reported WideSearch-en score, remains competitive on BrowseComp and DeepSearchQA, and AREX-Turbo exceeds Qwen3.5-35B on five of six benchmarks.AREX-Base also surpasses DeepSeek-V4-Pro on DeepSearchQA, WideSearch-en, and text-only HLE, and exceeds Kimi-K2.6 on GAIA and WideSearch-en.Source: Table 1 · p. 11, §5.2 · p. 11

5.3 Inference Framework Analysis

  • In BrowseComp, AREX invokes update_context in 80.3% of cases, usually proactively at a mean context size of 25,721 tokens; 66.9% of calls follow search-strategy revision.ACU preserves unresolved constraints and next-step plans along with verified findings and rejected candidates, consolidating the trajectory to redirect effort toward unresolved constraints.Source: §5.3 · p. 12
  • On BrowseComp, ACU raises accuracy from 59.6 to 71.4, while the outer loop adds 10.2 points without ACU and 11.1 points with ACU.The full system reaches 82.5 accuracy, 22.9 points above the configuration with neither ACU nor the outer loop.Source: Table 3 · p. 13, §5.3 · p. 13
  • Correct final outputs fall in the 90–100 confidence bin in 89.3% of cases without ACU and 95.9% with ACU, while 61.0% and 55.2% of errors fall below 60.This separation supports using answer-level confidence to guide whether to continue recursive refinement.Source: Figure 3 · p. 13, §5.3 · p. 13

5.4 Ablation Studies

  • Progressive multi-round capability training first establishes browse-intensive tool use and then adds reasoning-intensive trajectories, rather than mixing both distributions from the outset.The BrowseComp ablation shows a reduction from 82.5 to 77.5 accuracy when progressive training is replaced with direct mixed training.Source: §5.4 · p. 13
  • After full-trajectory mid-training, evidence discovery, path rejection and redirection, and key context-update steps have average losses of 0.277, 0.298, and 0.300, above ordinary steps at 0.232.The higher losses indicate that decision-critical intermediate steps remain comparatively underlearned.Source: Figure 4 · p. 14, §5.4 · p. 14
  • On BrowseComp, direct mixed training lowers accuracy from 82.5 to 77.5, random-step replay lowers it to 74.1, and standard GRPO lowers it to 79.4.These matched ablations test progressive multi-round training, key-step focused supervision, and step-aware reinforcement learning, respectively.Source: §5.4 · pp. 13–14, Table 4 · p. 14

6 Conclusion

AREX refines provisional deep-research answers through verification-guided investigation, autonomous context updating, and step-aware training. It performs strongly across several benchmark areas, while the training analysis points to future work on estimating step utility and assigning fine-grained signals.

6 Conclusion

  • AREX turns provisional answers into partially verified research states, preserving supported evidence and isolating unresolved claims for targeted investigation.It combines verification-guided refinement with autonomous context updating.Source: §6 · p. 14
  • AREX achieves strong performance across deep-search, wide-search, agentic-reasoning, and tool-use benchmarks.The conclusion gives no benchmark-specific values or comparisons.Source: §6 · p. 14
  • The preliminary training analysis highlights reinforcing decision-critical steps in long-horizon trajectories rather than treating all intermediate actions uniformly.Future work will investigate more autonomous step-utility estimation and fine-grained training signals.Source: §6 · p. 14

A Related Work

AREX extends tool-augmented deep research by using constraint-wise verification to direct successive research rounds, while managing long-horizon research state through learned context updates and key-step training.

Tool-Augmented Deep Research

Verification as Recursive Research Control

  • AREX makes constraint-wise verification the transition between research rounds, preserving supported claims, isolating unresolved conditions, and surfacing contradictory evidence to guide targeted follow-up research.This differs from using verification only as a final acceptance filter or local action critic; prior work also uses verification for outcome ranking, process supervision, reward modeling, test-time search, and critique (Lightman et al., 2024; Wang et al., 2024; Zeng et al., 2025b; Team et al., 2026).Source: A Related Work · p. 18

Research-State Management and Long-Horizon Training

B Preliminary Exploration under a Simplified Setting

In a simplified setting, trajectory self-distillation improved BrowseComp performance, but the experiment was exploratory and does not establish that the gain carries over to the full AREX recipe.

B Preliminary Exploration under a Simplified Setting

  • The early configuration omitted ACU, the outer self-improvement loop, key-step supervision, the complete multi-stage data mixture, and full test-time scaling, so its absolute BrowseComp scores are not directly comparable with complete AREX-Base or original-backbone results.The comparison measures relative differences only within the same controlled setting.Source: B Preliminary Exploration under a… · p. 18
  • The experiment compared direct training on original browse-intensive trajectories with training a fresh base-model copy on accepted trajectories regenerated by an intermediate browse-trained agent.The comparison used the same problems and matched training and inference configurations for both variants.Source: B Preliminary Exploration under a… · p. 18
  • Under the simplified setting, trajectory self-distillation improves BrowseComp from 52.3 to 57.1, a gain of 4.8 points.Both configurations use the 122B-A10B backbone without ACU, the outer self-improvement loop, or key-step supervision.Source: B Preliminary Exploration under a… · p. 18, Table 5 · p. 19
  • The study does not isolate the improvement’s source or show that it persists in the full recipe, and self-generated trajectories may inherit or amplify teacher biases and failure modes.Trajectory self-distillation is presented as a promising direction for future iterations, not a component of the final system.Source: B Preliminary Exploration under a… · p. 19

Related papers

Loading 2607.21461v3…