Source-linked AI summary
Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning
Peng Xia, Kaide Zeng, Jiaqi Liu, Can Qin, Fang Wu, Yiyang Zhou, Caiming Xiong, Huaxiu Yao
TL;DR
RL-based LLM agents depend heavily on human-curated data, while existing self-evolution methods are constrained by model capabilities and single-round interactions. Agent0 addresses these limits with tool-integrated, multi-step co-evolution between curriculum and executor agents initialized from one base LLM. It reports 18% and 24% improvements in mathematical and general reasoning, respectively, while progressively generating harder curricula without external data.
Problem
RL-based agent training depends heavily on massive, high-quality human-curated datasets, while existing self-evolution methods face capability and single-round interaction constraints.
Method
Agent0 co-evolves curriculum and executor agents from the same base LLM, using tools, progressively challenging tasks, self-consistency filtering, and ambiguity-aware policy optimization.
Results
18% mathematical reasoning and 24% general reasoning improvements are reported across ten benchmarks, with Agent0 also outperforming compared baseline methods.
Takeaways & Limitations
Agent0 provides a data-free pathway for evolving LLM agents through a virtuous cycle in which tool-equipped executor improvement drives progressively harder curricula.
Abstract
from arXiv · showhide
Large Language Model (LLM) Agents, often trained with Reinforcement Learning (RL), are constrained by a dependency on human-curated data, limiting scalability and tethering AI to human knowledge. Existing self-evolution frameworks offer an alternative but are typically restricted by the model's inherent capabilities and single-round interactions, hindering the development of complex curricula involving tool use or dynamic reasoning. We introduce Agent0, a fully autonomous framework that evolves high-performing agents without external data through multi-step co-evolution and seamless tool integration. Agent0 establishes a symbiotic competition between two agents initialized from the same base LLM: a curriculum agent that proposes increasingly challenging frontier tasks, and an executor agent that learns to solve them. We integrate external tools to enhance the executor's problem-solving capacity; this improvement, in turn, pressures the curriculum agent to construct more complex, tool-aware tasks. Through this iterative process, Agent0 establishes a self-reinforcing cycle that continuously produces high-quality curricula. Empirically, Agent0 substantially boosts reasoning capabilities, improving the Qwen3-8B-Base model by 18% on mathematical reasoning and 24% on general reasoning benchmarks. Code is available at https://github.com/aiming-lab/Agent0.
1. Introduction
Agent0 addresses the scalability limits of human-curated RL data and the stagnation of existing self-evolution methods through autonomous, tool-integrated co-evolution. Its curriculum and executor agents jointly produce progressively harder tasks and improve reasoning capabilities from scratch.
- Human-curated datasets remain central to RL methods, creating a severe scalability bottleneck for training complex reasoning agents.
- Existing self-evolution methods are limited by the model’s inherent capabilities and usually support only single-round interactions.These constraints make generated tasks rarely exceed the model’s current complexity, causing learning stagnation.
- Agent0 initializes curriculum and executor agents from a base LLM and co-evolves them through tool-integrated reinforcement learning without external data.The curriculum agent proposes frontier tasks, while the executor learns to solve filtered challenging problems.
- The curriculum agent uses executor uncertainty and tool-use frequency as reward signals to generate tasks that challenge current capabilities.The executor is trained concurrently on challenging tasks generated by the curriculum agent.
- 18% mathematical reasoning and 24% general reasoning improvements are reported across ten benchmarks for Agent0.The paper attributes these gains to a co-evolutionary loop that generates progressively complex tasks and improves executor capability.
2. Preliminaries
The preliminaries formulate language-model agents as policies optimized by reinforcement learning and introduce the co-evolution loop’s task-generation and executor-training stages. The loop combines reward-based curriculum evolution with self-consistency filtering and ambiguity-aware optimization.
- An LLM agent is modeled as a policy πθ that generates responses autoregressively and is optimized to maximize expected reward.
- GRPO avoids a critic by computing normalized advantages from relative rewards among multiple sampled responses for each prompt.
- The policy update uses a PPO-style clipped loss with importance sampling, normalized advantages, and KL regularization for training stability.The importance ratio compares the current policy with the previous-iteration reference policy.
- Curriculum evolution trains the curriculum policy to maximize rewards based on executor uncertainty, tool use, and repetition penalties.
- Executor evolution filters tasks by self-consistency to form a challenging dataset, then trains the executor with ambiguity-aware policy optimization using majority-vote pseudo-labels.
3. The Agent0 Framework
Agent0 co-evolves a curriculum agent that generates frontier tasks with an executor agent that solves them, using tool-integrated, multi-turn reasoning and ambiguity-aware optimization. Iteratively filtering challenging tasks and updating both agents creates progressively more demanding curricula while addressing noisy pseudo-labels and restricted exploration.
- Framework Overview: Agent0 initializes curriculum and executor agents from the same base LLM and trains them through iterative symbiotic competition.The curriculum agent proposes appropriately challenging tasks, while the executor agent learns to solve them.
- Tool-Integrated Rollouts: Tool integration and multi-turn rollouts strengthen the evolution loop by enabling tool-based curricula and context-rich conversational tasks.Generation pauses for tool calls during each trajectory, allowing tool responses to participate in the reasoning process.
- Curriculum Evolution: The curriculum agent is trained with RL to generate tasks near the executor’s capability frontier, using uncertainty and tool-use signals in its composite reward.Its uncertainty reward peaks when self-consistency is 0.5, while tasks that are too easy or too hard are penalized.
- Executor Evolution: After freezing the trained curriculum agent, Agent0 filters generated tasks by self-consistency and trains the executor on tasks that are neither too easy nor too hard.The executor’s objective is to maximize success on the curriculum agent’s generated tasks.
- Ambiguity-Dynamic Policy Optimization: ADPO downweights low-consistency training signals to reduce overfitting to unreliable majority-vote pseudo-labels.Its advantage scaling factor increases with self-consistency, reducing the influence of ambiguous samples.
- Ambiguity-Dynamic Policy Optimization: ADPO also dynamically modulates the trust region because static clipping can suppress low-probability tokens and restrict new reasoning paths on ambiguous tasks.The method makes the upper clipping bound a decreasing function of self-consistency.
4. Experiments
Agent0 is evaluated against self-evolving and tool-integrated baselines across mathematical and general-domain reasoning benchmarks, with analyses probing co-evolution, component contributions, iteration, and task evolution. It consistently improves across iterations, outperforms compared methods, and generates increasingly difficult, tool-using tasks while supporting hybrid reasoning.
- Experimental setup: Agent0 is evaluated on mathematical and general-domain reasoning suites using accuracy metrics across AMC, Minerva, MATH, GSM8K, Olympiad-Bench, AIME25, AIME24, SuperGPQA, MMLU-Pro, and BBEH.Greedy-decoding pass@1 is reported except for AMC and AIME benchmarks, which use mean@32.
- Main results: Agent0 significantly outperforms all compared baselines in both mathematics and general-domain reasoning.On Qwen3-8B-Base, it surpasses R-Zero by 6.4%, Absolute Zero by 10.6%, and Socratic-Zero by 3.7%.
- Main results: Agent0 achieves the highest overall average score on Qwen3-8B general-domain reasoning tasks, indicating transfer of cultivated multi-step reasoning abilities.The reported comparison is against other approaches, especially data-free methods.
- Ablation study: Removing curriculum-agent training reduces performance by 9.3%, while removing the tool reward reduces it by 7.2%.The ablation also reports a 1.9% drop with standard GRPO and highlights multi-turn reasoning for complex mathematical reasoning.
- Co-evolution analysis: On Qwen3-8B-Base, average math performance rises from 55.1 at Iter 1 to 56.5 at Iter 2 and 58.2 at Iter 3.General-domain reasoning improves by an average of 2% per iteration compared with the previous one.
- Tool integration: Strategic tool integration outperforms merely providing a tool because the curriculum explicitly rewards complex tasks requiring tool use.The executor also uses tools alongside multi-step reasoning, producing the reported co-evolutionary gains.
- Task evolution: The curriculum generates progressively harder tasks, while the executor increasingly combines natural-language reasoning with Python code execution.The qualitative analysis progresses from basic geometry in Iter 1 to complex constraint-satisfaction tasks in Iter 3.
5. Related Work
Related work covers self-evolution from zero data and tool-integrated reasoning, emphasizing scalability, stability, generalization, and complexity in multi-turn interactions.
- Self-evolution enables LLMs to generate their own training data, ranging from dual-agent setups to fully autonomous frameworks.
- Tool-integrated reasoning applies reinforcement learning to enhance LLM tool use, but many approaches depend on domain-specific data or supervised fine-tuning.
- Recent tool-integrated reasoning methods address multi-turn challenges through stability guarantees, cross-domain transfer, long-horizon planning, memory management, and interaction efficiency.
6. Conclusion
Agent0 uses two co-evolving agents and a code interpreter to improve reasoning without human-curated data. The resulting framework is presented as a scalable pathway for evolving capable agents.
- Agent0 co-evolves a curriculum agent and an executor agent without relying on human-curated data.
- Integrating a code interpreter creates a cycle in which executor improvements drive the curriculum agent to generate progressively harder tasks.
- The experiments report significant enhancements in the reasoning abilities of base LLMs.
Executor Agent Training
Executor-agent training combines tool-enabled multi-turn interaction, distributed execution infrastructure, and comparisons against base and self-evolving baselines.
- Executor and curriculum prompts, together with judging prompts, are specified in Tables 6–8.
- The executor uses a Python-based code-execution sandbox for verification and algorithmic reasoning.The system combines a multi-turn interaction protocol with a distributed execution orchestrator.
- The multi-turn protocol halts generation at executable Python code, runs it in an isolated sandbox, and feeds execution output back into the conversation.
- Distributed execution uses isolated worker nodes, round-robin scheduling, and asynchronous calls to manage parallel candidate generation.
- The evaluation compares the base model, zero-shot tool access, zero-data self-evolution, tool-aware self-play, multi-turn self-play, and proprietary-model-assisted baselines.
C. Evaluation Benchmarks
The evaluation spans mathematical, scientific, general reasoning, and difficult symbolic or algorithmic benchmarks. Increasing curriculum-generation interactions from 1 to 4 turns improves executor performance across domains.
- Mathematical benchmarks: The benchmark suite includes AMC, Minerva, MATH, GSM8K, Olympiad-Bench, AIME24, and AIME25 for mathematical reasoning.
- General reasoning benchmarks: SuperGPQA, MMLU-Pro, and BBEH evaluate difficult graduate-level, enhanced general-knowledge, symbolic, logical, and algorithmic reasoning.
- Turn-length analysis: 3.4%: the 4-turn curriculum-generation setting improves overall executor performance over the single-turn baseline.The reported gains are 3% on mathematical benchmarks and 2.6% on general-domain tasks.
- Turn-length analysis: Four-turn curriculum generation produces tasks with longer context dependencies and progressive difficulty, requiring more extended logical consistency and reasoning.
D.2. Detailed Results
The detailed results report complete three-iteration performance across individual mathematical and general reasoning benchmarks, while an ablation examines how interaction turns affect performance.
- Detailed benchmark results: Complete three-iteration results are reported across individual mathematical and general reasoning benchmarks.Table 10 covers mathematical reasoning, while Table 11 covers general-domain reasoning.
- Interaction-turn ablation: Increasing interaction turns from 1 to 4 leads to consistent performance gains across all domains.
- Detailed benchmark results: Table 10 highlights each model’s peak performance during training.
E. Case Analysis
The case analysis presents representative curriculum questions across three iterations and reports a clear progression from straightforward queries to highly complex, multi-step problems requiring deep reasoning.
- Iteration 1–2: Representative questions are shown for Curriculum Agent outputs from Iteration 1 through Iteration 3.The examples are organized across Tables 12 through 20.
- Iteration 1: Iteration 1 examples contain relatively straightforward queries.
- Iteration 3: Iteration 3 tasks evolve into highly complex, multi-step problems requiring deep reasoning.One example involves operations on coordinates of points on a sphere satisfying x^2 + y^2 + z^2 = 1.