Source-linked AI summary
AREX: Towards a Recursively Self-Improving Agent for Deep Research
Shuqi Lu, Chaofan Li, Kun Luo, Zhang Zhang, Hui Wang, Hongwang Xiao, Lei Xiong, Jiahao Wang, Sen Wang, Xiyan Jiang, Wanli Li, Yuyang Hu, Hongjin Qian, Bingyu Yan, Jianlyu Chen, Ziyi Xia, Yingxia Shao, Kang Liu, Zhicheng Dou, Di He, Chaozhuo Li, Qiwei Ye, Zhongyuan Wang, Zheng Liu
TL;DR
Deep research agents must find answers satisfying multiple coupled constraints, yet simply extending search does not ensure systematic progress. AREX recursively verifies provisional answers to target unresolved claims and achieves consistent gains over comparable-scale baselines across deep-research, wide-search, reasoning, and tool-use benchmarks.
Problem
Deep research requires discovering candidates, integrating distributed evidence, and verifying answers against multiple coupled constraints, while longer single-trajectory search does not ensure systematic progress.
Method
AREX alternates research and constraint-wise self-improvement loops, using verification, belief estimates, and learned context updating to target unresolved claims over long horizons.
Results
AREX demonstrates consistent gains over comparable-scale baselines across deep-research, wide-search, reasoning, and tool-use benchmarks.
Takeaways & Limitations
AREX supports recursively converting partially verified solutions into better-targeted research problems as a basis for effective deep-research agents.
Takeaways & Limitations
Key-step annotations are constructed offline for verifiable training tasks and are not used during evaluation or inference.
Abstract
from arXiv · showhide
Deep research requires agents to find answers that jointly satisfy multiple constraints. Discovering such answers is costly, whereas verifying a candidate can often be decomposed into tractable constraint-wise checks. This discovery--verification asymmetry suggests that a research agent should do more than simply search longer: it should recursively improve its current answer by verifying intermediate results and using the partially verified state to guide subsequent refinement. We introduce AREX, a family of Recursively Self-Improving (RSI) deep research agents. AREX alternates between an inner research loop that gathers evidence and constructs a provisional answer, and an outer self-improvement loop that audits the answer constraint-wise, identifies unresolved claims, and launches targeted follow-up research. To sustain RSI over long horizons, AREX learns an autonomous context-update tool that compresses growing interaction history into a compact improvement state preserving verified evidence and unresolved constraints, without relying on an external model. We train AREX on verified synthetic tasks and high-quality trajectories through agentic mid-training and long-horizon reinforcement learning. To mitigate sparse final rewards during long horizon learning, we emphasize key steps where decisive evidence is acquired or erroneous research directions are corrected. We instantiate a dense 4B model and a 122B-A10B Mixture-of-Experts model. Across BrowseComp, WideSearch, DeepSearchQA, Humanity's Last Exam (HLE), and other reasoning and tool-use benchmarks, AREX substantially outperforms comparable-scale baselines and remains competitive with models using substantially more activated parameters.
AREX Team
The AREX team is affiliated with the Beijing Academy of Artificial Intelligence (BAAI), and the paper includes a figure presenting AREX’s benchmark performance.
- The AREX team is affiliated with the Beijing Academy of Artificial Intelligence (BAAI).
- Figure 1 presents the benchmark performance of AREX.
1 Introduction
AREX addresses the discovery–verification asymmetry in multi-constraint deep research by recursively turning partially verified answers into targeted research problems. Its framework combines constraint-wise auditing, learned context updates, multi-stage long-horizon training, and dense and MoE model variants that achieve consistent gains across research and tool-use benchmarks.
- Challenge: Deep research requires discovering candidates, integrating distributed or conflicting evidence, and verifying that multiple coupled constraints are simultaneously satisfied.The difficulty arises from both locating relevant evidence and establishing support for every required condition.
- Motivation: AREX exploits the discovery–verification asymmetry by using simpler constraint-wise checks to identify supported, unresolved, and weakly supported claims for subsequent research.The approach treats candidate verification as guidance for navigating a costly, sparsely informative discovery space.
- AREX framework: AREX alternates an inner research loop that gathers evidence and builds provisional answers with an outer loop that audits constraints and redirects research.Belief estimates govern targeted continuation and evidence-aware termination.
- Long-horizon state: A learned context-update tool preserves a compact improvement state containing actionable research information as long-horizon interaction histories accumulate.The state addresses the tension between distraction from retaining full histories and evidence loss from indiscriminate truncation.
- Training: AREX is trained through supervised capability acquisition, agentic mid-training, and long-horizon reinforcement learning on verified synthetic tasks and high-quality trajectories.Critical-interval exposure improves credit assignment by emphasizing decisive evidence acquisition and corrections to erroneous research directions.
- Evaluation: The system includes a dense 4B model and a 122B-A10B MoE variant, demonstrating consistent gains over comparable-scale baselines across deep-research, wide-search, reasoning, and tool-use benchmarks.The evaluated benchmarks include BrowseComp, WideSearch, DeepSearchQA, Humanity’s Last Exam with Tools, GAIA, and xbench-DeepSearch.
2 Recursive Self-Improvement
AREX performs deep research through nested inner research and outer self-improvement loops. It maintains a compact research state that preserves verified progress and unresolved constraints, enabling targeted refinement or restart across long-horizon investigations.
- Recursive architecture: AREX uses a bi-level recursive process in which an inner loop researches and constructs a provisional answer, while an outer loop evaluates and improves it.The system instantiates Qwen3.5-4B for AREX-Turbo and Qwen3.5-122B-A10B for AREX-Base.
- Outer self-improvement loop: The outer loop accepts high-confidence answers, targets recoverable trajectories for refinement, and restarts noisy or unrecoverable investigations from the original problem.Preserved information and unresolved issues form the next research objective, allowing subsequent rounds to reuse useful progress while focusing on uncertainty or incompleteness.
- Inner research loop: The inner loop invokes search or browsing tools, integrates evidence, tracks unresolved constraints, and adapts subsequent actions when findings support, contradict, or invalidate current candidates.Targeted objectives can verify unsupported constraints, resolve conflicting evidence, check temporal validity, or explore alternatives.
- Autonomous context updating: AREX’s update_context tool refreshes research state by preserving verified findings, source identifiers, candidates, unresolved constraints, validity concerns, rejected candidates, and the next-step plan while removing obsolete information.The model autonomously invokes the tool when meaningful subproblems are resolved, candidates are eliminated, conflicting evidence is reconciled, or plans change.
3 Training Data Construction
AREX’s training data combines synthesized, verifiable deep-research tasks with quality-controlled teacher trajectories. The construction process emphasizes iterative investigation, constraint verification, evidence grounding, and adaptive research-state maintenance.
- Pipeline: The dataset pipeline has two stages: recursive research task synthesis and teacher trajectory collection with quality control.It is designed to provide challenging tasks, multi-step investigations, and evidence-grounded solutions.
- Recursive research task synthesis: Task synthesis targets browse-intensive, reasoning-intensive, and scientific-literature problems requiring multi-source integration, verification, and adaptive planning.Human-defined templates specify answer formats, sources, reasoning requirements, and verification criteria, while instances draw on web pages, academic literature, knowledge bases, and repositories.
- Recursive research task synthesis: Each task uses jointly constraining objectives that require multi-hop research, cannot be inferred directly from the query, are evidence-verifiable, and uniquely identify the answer.Automated checks and independent research rollouts remove ambiguous, inconsistent, insufficiently evidenced, shallow-retrieval, or effectively unsolvable tasks.
- Teacher trajectory collection: Teacher trajectories record actions, tool calls, observations, intermediate analysis, and structured answers while exposing behaviors needed for long-horizon research.Teachers operate in the same tools and environment as AREX.
- Teacher trajectory collection: Quality control retains trajectories with iterative investigation, coherent state maintenance, evidence-based revision, valid tool use, and reconstructable final answers.It removes guessing, ignored observations, unsupported or contradictory claims, unjustified assumptions, and answers produced before sufficient evidence.
4 Training Pipeline
AREX trains long-horizon research through progressive capability development, mixed-capability consolidation, and selective emphasis on difficult research decisions. Its reinforcement-learning design uses step-aware credit assignment, including key-step bonuses gated by valid trajectory outcomes.
- Progressive Multi-round Capability Training: AREX progressively trains browse-intensive tool use and evidence acquisition before expert reasoning, then consolidates capabilities to reduce interference.The pipeline uses browse-intensive multi-round trajectories first, followed by expert-level reasoning and mixed-capability replay.
- Key-step Focused Mixed-Capability Consolidation: Key-step supervision targets evidence discovery, hypothesis redirection, and context updates rather than routine trajectory transitions.Candidate steps are detected with high-precision rules and retain full preceding context while applying loss only to the selected step.
- Key-step Focused Mixed-Capability Consolidation: Detected key steps have substantially higher loss than ordinary steps, and adding focused supervision after full-trajectory training further improves long-horizon research performance.Annotations are retained only for trajectories passing final-answer verification and are grounded in externally verifiable evidence.
- Step-aware Group Policy Optimization: Step-aware policy optimization replaces coarse sequence-level credit assignment with turn-level shaped advantages for heterogeneous long-horizon trajectories.All steps share the group-relative outcome advantage, while annotated key steps receive an auxiliary bonus; hierarchical averaging balances steps within trajectories and trajectories within rollout groups.
- Step-aware Group Policy Optimization: The key-step bonus is applied only when the trajectory-level result is valid, preventing reward for spurious intermediate behavior.The bonus uses a key-step indicator derived from the Section 4.1 annotations.
5 Experiments
AREX is evaluated across diverse search-augmented reasoning and agentic task-completion benchmarks, where it shows strong performance. Ablations further identify autonomous context updating, recursive self-improvement, staged training, key-step supervision, and step-aware reinforcement learning as important contributors.
- Evaluation: AREX is evaluated on six benchmarks spanning deep research, agentic task completion, broad-coverage retrieval and synthesis, and other search-augmented reasoning regimes.The evaluation uses a unified long-horizon interface with search, visit, update context, and finish tools, allowing up to 300 inner-loop turns and 5 outer-loop operations.
- Overall results: AREX-Base demonstrates a strong capability-to-parameter trade-off with 10B active parameters and consistently improves over the Qwen3.5 backbone family, including Qwen3.5-397B.It remains competitive with leading open-source and proprietary research agents across all evaluations.
- Autonomous Context Updating: 80.3% of BrowseComp cases invoke update_context, typically at 25,721 mean active-context tokens, while search-strategy revision triggers 66.9% of calls.Only 0.01% of updates occur at or above the 128K-token limit, indicating proactive state refreshing rather than predominantly limit-driven compression.
- Mechanism analysis: ACU raises matched single-round BrowseComp accuracy from 59.6 to 71.4, while enabling the outer loop raises accuracy from 71.4 to 82.5 when ACU is enabled.ACU preserves unresolved constraints, next-step plans, verified findings, rejected candidates, and source identifiers for subsequent recursive research.
- Answer-Level Confidence Score: 89.3% of correct outputs without ACU and 95.9% with ACU fall in the 90–100 confidence bin, while 61.0% and 55.2% of errors, respectively, fall below 60.The confidence score therefore separates many correct and incorrect final outputs, supporting confidence-based decisions in the outer loop.
- Training ablations: Key-step focused supervision is decisive: equal-budget random-step replay reduces BrowseComp accuracy from 82.5 to 74.1, while standard GRPO reduces it from 82.5 to 79.4.Progressive multi-round capability training also outperforms direct mixed training, 82.5 versus 77.5; annotated evidence discovery, path rejection and redirection, and context-update steps have losses of 0.277, 0.298, and 0.300 versus 0.232 for ordinary steps.
6 Conclusion
AREX is a recursively self-improving deep-research agent that converts provisional answers into partially verified states, preserving supported evidence and isolating unresolved claims for targeted investigation. With autonomous context updating and step-aware training, it achieves strong performance across diverse search, reasoning, and tool-use settings.
- AREX is introduced as a recursively self-improving agent for deep research.
- AREX exploits discovery–verification asymmetry by transforming provisional answers into partially verified research states.These states preserve supported evidence while isolating unresolved claims for subsequent targeted investigation.
- Autonomous context updating and step-aware training support AREX’s self-improvement process.
- AREX achieves strong performance across deep search, wide search, agentic reasoning, and tool-use benchmarks.
A Related Work
Related work frames deep research as tool-augmented, multi-constraint problem solving, highlights verification’s role in guiding search, and studies memory for sustaining long-horizon research.
- Tool-Augmented Deep Research: Deep-research agents use tools, web environments, and external knowledge to locate sparse evidence, compare conflicting sources, and satisfy coupled constraints.This extends language-model agents beyond static response generation toward iterative problem solving.
- Verification as Recursive Research Control: Prior verification methods rank outcomes, supervise processes, model step-level rewards, or guide test-time search and critique intermediate agent decisions.These approaches exploit the asymmetry between generating a correct solution and verifying a proposed one.
- Research-State Management and Long-Horizon Training: Research-state management studies hierarchical memory, virtual context management, learned memory states, periodic summarization, and explicit memory-editing actions for long trajectories.These trajectories may contain evidence, failed searches, speculative hypotheses, and evolving plans.
B Preliminary Exploration under a Simplified Setting
In a simplified setting, trajectory self-distillation raised BrowseComp performance, suggesting intermediate-agent trajectories can improve supervision. The study treats this as a promising future direction, not part of final AREX-Base, because the source of improvement and its persistence under the complete recipe remain unestablished.
- Scope and limitations: The preliminary result is not directly comparable with complete AREX-Base or original-backbone scores because the configuration omitted several final-system components.It used a reduced tool set and lacked ACU, outer self-improvement, key-step supervision, the complete multi-stage data mixture, and full test-time scaling.
- Trajectory self-distillation: The experiment compared direct training on original browse-intensive trajectories with self-distillation using trajectories generated by an intermediate browse-trained agent.The variants shared the 122B-A10B initialization, problem distribution, and matched training and inference configurations.
- Trajectory self-distillation: 4.8 points: Trajectory self-distillation improved BrowseComp from 52.3 to 57.1 under the simplified setting.Both configurations used the 122B-A10B backbone without ACU, the outer self-improvement loop, or key-step supervision.
- Interpretation: The improvement may reflect better alignment with the target policy’s action distribution, more consistent search decisions, or cleaner intermediate reasoning structure.These are proposed explanations rather than isolated causal findings.
- Scope and limitations: The study does not establish the improvement’s source or persistence under the complete recipe, and self-generated trajectories may inherit or amplify teacher biases and failure modes.Accordingly, trajectory self-distillation is presented as a promising future direction rather than a final-system component.
Contributions
The paper credits core contributors, participants, and advisors, with equal contribution among the first listed authors and Zheng Liu as project leader.
- Core Contributors: Core contributors include Shuqi Lu, Chaofan Li, Kun Luo, Zhang Zhang, Hui Wang, Hongwang Xiao, and Zheng Liu.Shuqi Lu, Chaofan Li, and Kun Luo contributed equally; Zheng Liu led the project.
- Participants: Participants include Lei Xiong, Jiahao Wang, Sen Wang, Xiyan Jiang, Wanli Li, Yuyang Hu, Hongjin Qian, Bingyu Yan, Jianlyu Chen, and Ziyi Xia.
- Advisors: Advisors include Yingxia Shao, Kang Liu, Zhicheng Dou, Di He, Chaozhuo Li, Qiwei Ye, and Zhongyuan Wang.