PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentEvolver: Towards Efficient Self-Evolving Agent System

Yunpeng Zhai, Shuchang Tao, Cheng Chen, Anni Zou, Ziqian Chen, Qingxu Fu, Shinji Mai, Li Yu, Jiaji Deng, Zouying Cao, Zhaoyang Liu, Bolin Ding, Jingren Zhou

arXiv:2511.10395v1cs.LGcs.AIcs.CL

TL;DR

LLM-agent training is costly and inefficient because it depends on handcrafted tasks, random exploration, and weak sample utilization. AgentEvolver replaces this dependence with an LLM-guided self-evolving framework that generates tasks, reuses experience, and assigns fine-grained credit; the paper reports more efficient exploration, better sample utilization, and faster adaptation than traditional RL-based baselines.

  • Problem

    RL-driven agent development requires costly handcrafted task construction and suffers from inefficient exploration and poor sample utilization, particularly in novel environments.

  • Method

    AgentEvolver uses self-questioning for autonomous task generation, self-navigating for experience-guided exploration, and self-attributing for fine-grained credit assignment.

  • Results

    The paper reports more efficient exploration, better sample utilization, and faster adaptation compared with traditional RL-based baselines.

  • Takeaways & Limitations

    The integrated framework supports scalable, cost-effective, and continual improvement of tool-augmented agent capabilities.

  • Takeaways & Limitations

    The interaction sandbox excludes predefined rewards and task objectives, so the agent must autonomously generate learning signals and objectives.

Abstract

from arXiv · show

Autonomous agents powered by large language models (LLMs) have the potential to significantly enhance human productivity by reasoning, using tools, and executing complex tasks in diverse environments. However, current approaches to developing such agents remain costly and inefficient, as they typically require manually constructed task datasets and reinforcement learning (RL) pipelines with extensive random exploration. These limitations lead to prohibitively high data-construction costs, low exploration efficiency, and poor sample utilization. To address these challenges, we present AgentEvolver, a self-evolving agent system that leverages the semantic understanding and reasoning capabilities of LLMs to drive autonomous agent learning. AgentEvolver introduces three synergistic mechanisms: (i) self-questioning, which enables curiosity-driven task generation in novel environments, reducing dependence on handcrafted datasets; (ii) self-navigating, which improves exploration efficiency through experience reuse and hybrid policy guidance; and (iii) self-attributing, which enhances sample efficiency by assigning differentiated rewards to trajectory states and actions based on their contribution. By integrating these mechanisms into a unified framework, AgentEvolver enables scalable, cost-effective, and continual improvement of agent capabilities. Preliminary experiments indicate that AgentEvolver achieves more efficient exploration, better sample utilization, and faster adaptation compared to traditional RL-based baselines.

BFCL v3

Figure 1 compares AgentEvolver with baseline models on AppWorld and BFCL-v3, highlighting higher performance with fewer parameters.

  • AgentEvolver achieves superior results on the AppWorld and BFCL-v3 benchmarks while using substantially fewer parameters than larger baseline models.

1 Introduction

RL-based agent development is costly and sample-inefficient, especially in novel environments. AgentEvolver addresses these challenges through an LLM-guided self-evolution loop built around three synergistic mechanisms.

  • RL-driven agent development faces expensive task construction and severe sample inefficiency in novel environments.Long-horizon, tool-augmented agents make existing exploration strategies often ineffective.
  • AgentEvolver uses LLM capabilities to guide task generation, exploration, and performance refinement through direct environment interaction.The framework establishes a self-training loop for continual competence improvement.
  • Self-questioning generates tasks by probing the environment’s state-action space and discovering functional boundaries.This reduces dependence on costly handcrafted datasets.
  • Self-navigating reuses and generalizes past experiences through hybrid policy learning and trajectory guidance for more targeted exploration.
  • Self-attributing assigns fine-grained rewards to intermediate states and actions according to their respective contributions along long trajectories.This replaces uniform outcome attribution used in typical GRPO methods.
  • Together, the mechanisms address task scarcity, inefficient exploration, and low sample utilization within an integrated self-evolution framework.

2 Problem Formulation

The paper formulates self-evolution in an interaction sandbox without predefined tasks or rewards, requiring the agent to synthesize proxy tasks and rewards. The goal is to optimize a policy that performs well on an unknown target task distribution.

  • Environment Formulation: The interaction sandbox specifies accessible states, executable actions, and transition dynamics but provides no predefined reward function or task objective.This distinguishes the setting from standard RL environments.
  • Target Task Space and Oracle Reward: The agent is evaluated on an unknown target task distribution, where each task corresponds to a desired goal state in the environment.
  • Target Task Space and Oracle Reward: For each target task, a ground-truth reward measures action utility relative to the goal, and the agent learns a goal-conditioned policy maximizing expected return.
  • Proxy Objective of Self-Evolution: Because target tasks and rewards are unknown, the agent must synthesize proxy training tasks and reward signals through formal generation mechanisms.
  • Proxy Objective of Self-Evolution: Self-questioning maps environment exploration to candidate tasks, producing a training distribution that emphasizes appropriate difficulty and diversity.
  • Proxy Objective of Self-Evolution: Self-attributing infers proxy rewards that provide finer-grained credit assignment than sparse environment rewards without human intervention.
  • Proxy Objective of Self-Evolution: Self-navigating guides the policy toward high-learning-value trajectories to address efficiency bottlenecks in multi-turn environments.

3 Self-Questioning

Self-questioning converts exploration in an unknown environment into curated, preference-aligned training tasks with reference solutions and proxy rewards. It combines environment-profile-guided curiosity, adaptive synthesis, feasibility verification, and automated assessment to support task–agent co-evolution.

  • Motivation: New environments often lack diverse, high-quality training tasks, making manual construction costly and limiting robust agent learning.The stated challenges include absent training data, expensive multi-step task construction, restricted diversity, and skewed distributions.
  • Overview: Self-questioning transforms a sandbox into exploration trajectories and then user-preferred tasks to bootstrap self-evolution.The module instantiates proxy task generation and synthetic rewards through an exploration-generation pipeline.
  • Curiosity-Guided Exploration: Environment profiles summarize entities, attributes, and supported operations, guiding curiosity toward relevant states while reducing aimless exploration.The exploration policy uses profiles as part of the initial state and explores diverse actions and states.
  • Curiosity-Guided Exploration: A high-temperature LLM performs stochastic exploration, using breadth-first coverage initially and depth-first investigation of promising trajectories afterward.The strategy balances broad action-state coverage with focused exploration while seeking unconventional interaction patterns and hidden states.
  • Adaptive Task Synthesis: Adaptive task synthesis combines explored trajectories with user preferences to generate candidate queries and reference solutions aligned with desired difficulty and style.Raw trajectories are distilled into interaction components before an LLM synthesizes preference-constrained tasks.
  • Task Curation and Reward: Exploration-derived action-observation traces provide proxy ground-truth solutions, while a reference-based LLM judge assesses completion quality without predefined rewards.The extracted solution contains the key steps required to solve each synthesized task, and the judge serves as a general fallback proxy reward.

4 Self-Navigating

Self-navigating reuses structured experiences to guide agent exploration while retaining unguided rollouts for flexibility. It incorporates these experiences into policy optimization through stripping and selective boosting, balancing exploitation with exploration and reducing over-reliance on textual cues.

  • Motivation: Self-navigating reuses knowledge from prior trajectories to make exploration more targeted and efficient in complex environments.The mechanism is motivated by the redundancy and slow convergence of repetitive random trial-and-error.
  • Experience Acquisition: Experiences are structured natural-language units with a When to use retrieval trigger and Content containing instructions, precautions, or recommended strategies.Their natural-language representation supports interpretability and embedding-based retrieval.
  • Experience Acquisition: Experience acquisition builds a high-quality repository offline and retrieves task-specific guidance online during rollout or inference.Pool construction processes trajectories from proxy tasks, while retrieval dynamically supplies relevant context.
  • Experience-mixed Rollout: Experience-mixed rollout interleaves vanilla and experience-guided trajectories, using their mixture to balance prior knowledge with unconstrained exploration.The balance is controlled by η: higher values favor prior experience, while lower values promote exploration.
  • Experience Incorporation: Experience incorporation removes retrieved tokens before optimization and selectively boosts positive-advantage experience-guided updates.Stripping prevents memorization of external content, while selective boosting preserves favorable experience-conditioned learning signals despite clipping.

5 Self-Attributing

Self-attributing replaces sparse trajectory-level credit with LLM-based, step-wise attribution that distinguishes action contributions. It combines process-quality and outcome signals through separate normalization before GRPO optimization.

  • Motivation and mechanism: Self-attributing uses an LLM to assess each action’s contribution, replacing uniform trajectory-level credit with contribution-based attribution.The mechanism implements the proxy reward function and shifts credit assignment from temporal propagation to retrospective contribution assessment.
  • Process quality attribution: The process-quality signal evaluates completed trajectories holistically and assigns each action a binary GOOD or BAD label.The full task context, intermediate steps, and final outcome are provided in one prompt so the evaluator can capture inter-step dependencies.
  • Attribution reward construction: GOOD and BAD labels become attribution rewards of +1 and −1, yielding an interpretable directional signal before normalization.The formulation avoids arbitrary scalar magnitudes and supports integration with outcome-based rewards.
  • Attribution reward construction: Trajectory-level standardization computes normalization statistics over per-trajectory average attribution rewards, giving trajectories equal weight regardless of length.This avoids allowing longer trajectories to disproportionately influence the mean and standard deviation.
  • Composite reward and advantage: Attribution Outcome Integration separately normalizes process and terminal-outcome signals, then combines them into a step-level composite reward.The outcome reward is applied only at the terminal step, while α controls the relative contribution of attribution feedback.
  • Composite reward and advantage: The resulting advantage is the undiscounted cumulative future composite reward, giving equal weight to all subsequent trajectory steps.This corresponds to the special case of discounted return with γ = 1.

6 Framework and Infrastructure

AgentEvolver implements its three learning mechanisms as a modular, service-oriented loop from task synthesis through model optimization. Its infrastructure separates agent logic, runtime services, context management, and interchangeable algorithmic components.

  • Training framework: The training framework organizes self-evolution into a dataflow from environments to model updates, using hierarchical rollouts and modular components.The design emphasizes extensibility and decoupling across the end-to-end training system.
  • Overall architecture and data flow: A Master Orchestrator drives task synthesis, trajectory rollout, experience summarization, and sample construction with model optimization.Task synthesis uses self-questioning to generate candidate tasks, followed by parallel task executions and downstream training preparation.
  • Hierarchical rollout execution: Rollouts use service, rollout-worker, and master layers to support parallel multi-turn interaction and coordinated scheduling.The service layer manages environment and LLM workers, rollout workers collect trajectories, and the master coordinates workers and termination.
  • Modularity and extensibility: Agent logic is decoupled from environment and LLM services through a context manager, allowing diverse workflows without changing the service layer.The same infrastructure can host ReAct, planner–executor, and tool-using agents.
  • Context management: The Context Manager addresses the trade-off between temporally consistent causal rollouts and editable but computationally expensive step-independent rollouts.It provides a common interface for causal and non-causal interaction modes.
  • Context management: The Live Context Timeline stores mutable working context, while the Timeline Snapshot Recorder preserves immutable episode snapshots and merges overlapping timelines.Together they support efficient causal rollouts and flexible reasoning-oriented rollouts.
  • Context templates: The four context templates share the LCT–TSR mechanism but span a spectrum from rigid causality to autonomous memory regulation.The Basic Causal Template prioritizes efficiency, whereas the Self-Context Managing Template supports dynamic memory control.
  • Environment infrastructure: The Environment Service decouples Gym-compatible environment execution into remote, concurrent, and isolated service instances.This replaces tight coupling between traditional Gym environments and the training loop.

7 Experiments

The experiments evaluate AgentEvolver on two sparse-reward, tool-augmented long-horizon benchmarks using Qwen2.5 policy models and Vanilla GRPO as the baseline. The study includes component ablations and implementation analyses.

  • Benchmarks and evaluation: AgentEvolver is evaluated on AppWorld and BFCL v3, both of which require multi-step API or tool interactions under sparse terminal rewards.Both benchmarks run through the Environment Service with isolated actors and a unified HTTP interface.
  • Benchmarks and evaluation: AppWorld uses Task Goal Completion, while BFCL v3 multi-turn correctness requires both final-state and response-based checks.The evaluation definitions follow the respective official benchmark criteria.
  • Models and baseline: Experiments use Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct policy models, compared against Vanilla GRPO using only sparse final-outcome rewards.The comparison isolates the contribution of AgentEvolver’s additional mechanisms.
  • Implementation details: Self-attributing uses Qwen-Max for joint single-shot trajectory evaluation, maps attribution labels to signed unit scores, and normalizes them before fusion.Near-zero-variance normalization cases are neutralized for stability.

7.2 Main Results

Across AppWorld and BFCL v3, the complete AgentEvolver improves both average and best-of-eight performance for 7B and 14B models. Ablations further show gains from synthetic task generation, experience-guided exploration, and fine-grained attribution.

  • Main results: 29.4% overall avg@8 and 36.1% overall best@8 gains occur for the 7B model, while the 14B model gains 27.8% and 30.3%, respectively.For 14B, AppWorld gains are +30.7% avg@8 and +38.0% best@8; BFCL v3 gains are +24.9% and +22.6%.
  • Component contributions: +Questioning raises avg@8 from 15.8% to 36.1% for 7B and from 29.8% to 52.3% for 14B.The authors attribute this initial gain to autonomously generated diverse training tasks.
  • Component contributions: Adding self-navigating increases avg@8 to 39.8% for 7B and 54.1% for 14B.The reported improvement is associated with more efficient exploration after self-questioning.
  • Component contributions: Adding self-attributing increases avg@8 to 41.3% for 7B and 56.4% for 14B.The ablation study links this gain to fine-grained credit assignment.
  • Synthetic data: Synthetic self-questioning data substantially improves AppWorld and BFCL performance over zero-shot agents and approaches training with original target-distribution data.Combining synthetic and original data surpasses training on original data alone, according to the authors.
  • Synthetic data: 40.3% performance is achieved with 100 synthetic samples, with performance improving as the data volume increases.The analysis attributes the trend to the diversity of synthetic data.
  • Cross-domain generalization: Qwen2.5-14B transferred from AppWorld to BFCL shows only a 4.3% performance drop.This cross-domain result supports retained utility of synthetic-data-trained capabilities across the two benchmarks.
  • LLM Judge analysis: Adding scoring principles and a reference solution substantially improves the LLM Judge, with performance closely matching human-curated-data training.The reference solution supplies environment-specific criteria for checking essential steps.

7.4 Results of Self-Navigating

Self-navigating improves agent exploration by reusing experience, with implicit learning internalizing useful behavior more effectively than retrieval alone. Moderate relaxation of the GRPO clipping threshold balances rapid early learning with stable long-term optimization.

  • Experience-guided navigation: Retrieved experiences raise trajectory quality without additional training, yielding average gains of ↑5.4% avg@4 and ↑6.7% best@4.On AppWorld and BFCL, experience stabilizes rollouts and raises the performance ceiling.
  • Implicit vs. explicit experience learning: Implicit experience learning surpasses ICL-only by ↑34.2% in avg@4 and ↑28.1% in best@4, while exceeding vanilla RL by ↑7.9% and ↑8.5%.The policy internalizes experience and performs without retrieval at inference.
  • Implicit vs. explicit experience learning: Removing selective boosting drops performance below the RL baseline by ↓0.5% in avg@4 and ↓2.9% in best@4.The ablation supports selective boosting as necessary for incorporating experience signals effectively.
  • Short-term vs. long-term optimization: ˆϵhigh = 0.6 provides the best balance between early learning speed and long-term performance.Higher values accelerate learning within the first 20 steps but harm performance up to 80 steps through overfitting to biased trajectories.

7.5 Results of Self-Attributing

Self-attributing improves performance and sample efficiency by combining step-wise attribution rewards with terminal outcome rewards. Its gains reflect richer credit assignment, while moderate attribution weights best balance early convergence and final performance.

  • Effectiveness: 47.6% avg@8 on 7B and 62.0% avg@8 with 75.7% best@8 on 14B make Attributing the best-performing configuration across settings.The method substantially exceeds the corresponding Qwen2.5 baselines.
  • Effectiveness: Combining attribution and outcome rewards outperforms either channel alone across both model scales.Attribution supplies intermediate guidance, while terminal outcomes ground learning in task objectives.
  • Sample efficiency: 40 steps versus 90 on AppWorld and 20 versus 60 on BFCL v3 reduce the steps needed to reach 90% of baseline final performance by 55% and 67%.Attributing also raises AUC from 41.03 to 46.26 on AppWorld and from 55.78 to 61.02 on BFCL v3.
  • Sample efficiency: Step-wise attribution rewards provide dense feedback alongside sparse terminal outcomes, reducing variance and redundant exploration.The composite reward gives each trajectory richer gradient information at decision points.
  • Attribution-weight trade-off: α=0.30 reaches 45% at 20 steps versus 28% for the baseline, but declines to 43% by training end.α=0.10 and α=0.20 converge to approximately 59%, while α=0.05 achieves comparable final performance more conservatively.
  • Attribution-weight trade-off: The optimal attribution-weight range α∈[0.10, 0.20] balances dense process guidance with outcome-based supervision.Excessive attribution can overfit heuristic labels, whereas too little attribution slows early progress.

7.6 Effectiveness of Context-Managing Templates

Context-management templates materially affect adaptation across diverse tasks. SCMT performs best overall, especially on long-horizon TGC@8, by dynamically reducing redundant context.

  • Template comparison: SCMT achieves the best overall AppWorld performance, particularly on the long-horizon TGC@8 metric.It dynamically compresses or discards redundant context segments when many available tool APIs are irrelevant.
  • Template comparison: RAT ranks second, associating explicit “thinkbefore-act” steps with stronger reasoning in complex multi-tool environments.SWT performs moderately because periodic summarization can omit dependencies, while BCT remains robust but lacks adaptive memory.

8 Conclusion and Next Steps

AgentEvolver shifts agent learning from fixed human-engineered pipelines to LLM-guided self-improvement through self-questioning, self-navigating, and self-attributing. The paper also presents reusable infrastructure and identifies broader environments, larger models, and LLM-level self-evolution as next steps.

  • Conclusion: AgentEvolver combines self-questioning, self-navigating, and self-attributing to address task scarcity, inefficient exploration, and low sample utilization.The mechanisms follow the flow from environments to tasks, tasks to trajectories, and trajectories to policy.
  • Conclusion: The supporting infrastructure standardizes environment interfaces, integrates with RL stacks such as veRL, and supports modular extension.This combines algorithmic design with engineering infrastructure for adaptive, tool-augmented agents.
  • Next steps: Challenge-oriented applications will target higher-complexity environments such as multi-API enterprise workflows, safety-critical tool chains, and long-horizon interleaved objectives.The proposed direction co-designs environments and self-questioning curricula.
  • Next steps: Scaling to larger models will test how self-questioning, experience reuse, and self-attribution change with model capacity.The stated questions concern task quality, exploration redundancy, and credit-assignment accuracy.
  • Next steps: LLM-level self-evolving aims to unify task generation, environment navigation, outcome attribution, and policy updates within one model or tightly coupled model family.The direction pursues a tighter end-to-end self-improvement loop.
Loading 2511.10395v1…