Source-linked AI summary
CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution
Shidong Yang, Ziyu Ma, Tongwen Huang, Yiming Hu, Yong Wang, Xiangxiang Chu
TL;DR
Current agent RL relies on costly, static human-curated demonstrations that inadequately cover interaction patterns and limit generalization. CoEvolve jointly evolves the agent and its training distribution through feedback-guided task synthesis and environment validation, improving performance across benchmarks and model backbones.
Problem
Current agent RL relies on costly human-curated trajectories that provide static interaction snapshots, limiting exploration and generalization beyond observed distributions.
Method
CoEvolve uses interaction-driven feedback to identify failure-prone patterns, synthesize targeted tasks, validate them in the environment, and integrate them into training without human supervision.
Results
CoEvolve improves average performance by 19.43%, 15.58%, and 18.14% on AppWorld and BFCL across Qwen2.5-7B, Qwen3-4B, and Qwen3-30B-A3B, respectively.
Takeaways & Limitations
The framework supports mutual evolution between agents and their data distributions through feedback-guided task synthesis, with experiments validating its effectiveness and efficiency.
Takeaways & Limitations
The extracted feedback depends on the current policy and may be noisy or incomplete when the agent’s behavior is immature early in training.
Abstract
from arXiv · showhide
Reinforcement learning for LLM agents is typically conducted on a static data distribution, which fails to adapt to the agent's evolving behavior and leads to poor coverage of complex environment interactions. To address these challenges, we propose CoEvolve, an agent-data mutual evolution framework that enables LLM agents to improve through closed-loop, interaction-driven training. Specifically, CoEvolve extracts feedback signals such as forgetting and uncertainty from rollout trajectories to identify failure-prone interaction patterns, and utilizes them to guide LLM-based task synthesis. The synthesized tasks are validated through environment interaction and utilized to update the data distribution, enabling joint adaptation of the agent and its data. Extensive experiments on AppWorld and BFCL across Qwen2.5-7B, Qwen3-4B, and Qwen3-30B-A3B demonstrate consistent and significant improvements over strong base models, yielding absolute gains of 19.43%, 15.58%, and 18.14%, respectively.
1 Introduction
Existing agent training relies on costly, static data that poorly covers evolving interaction patterns. CoEvolve closes this gap by using feedback-driven task synthesis and joint agent-data evolution without human supervision.
- Human-curated trajectories are expensive to collect and represent static interaction snapshots, limiting coverage of long-tail behaviors.A single trajectory may require several minutes or more of expert effort.
- Static synthetic data reduces annotation needs but remains open-loop, shallow, and unable to target the agent’s evolving weaknesses.Its exploration is guided by the LLM’s world knowledge without feedback from actual agent performance.
- CoEvolve synthesizes and validates tasks targeting current weaknesses, allowing the agent and its training distribution to evolve jointly through interaction.Validated tasks are integrated into training through a closed loop.
- 19.43%, 15.58%, and 18.14% average-performance gains are reported for Qwen2.5-7B, Qwen3-4B, and Qwen3-30B-A3B, respectively, across AppWorld and BFCL.The evaluation uses two benchmarks and three backbone models.
- CoEvolve alternates agent optimization with feedback-driven updates to the training distribution without human supervision.The framework uses signals such as forgetting signals to identify failure-prone patterns and guide task discovery.
- CoEvolve incorporates feedback signals such as forgetting signals into LLM-based environment exploration rather than relying on unguided random exploration.
2 Related Work
Prior LLM-agent methods increasingly explore autonomously or use structured curricula, but trajectory generation remains largely open-loop. CoEvolve instead uses environment feedback to drive on-demand discovery of executable queries and states.
- The framework extracts weakness signals, uses them for LLM-guided re-exploration, and validates discovered interactions as executable training tasks.
- Recent LLM-agent extensions add autonomous exploration or structured curricula, but trajectory generation remains loosely coupled to evolving failure modes.
- CoEvolve closes the loop by using environment feedback to synthesize trajectories on demand and continuously adapt the training distribution.
- Unlike fixed-query trajectory refinement, CoEvolve drives renewed environment interaction to discover new executable queries and states.Its data evolution is not limited to generating variants around seed tasks.
3 Method
CoEvolve jointly updates the agent and its synthetic-task distribution through interaction-driven feedback. It extracts behavioral weakness signals, guides LLM re-exploration and task abstraction, validates synthesized tasks, and adds them to subsequent training iterations.
- Training on Synthetic Tasks: CoEvolve alternates agent optimization with updates to an executable synthetic-task set, allowing the training distribution to evolve with the agent.The initial task set comes from unguided LLM exploration; newly synthesized and validated tasks are appended during training.
- Training and Signal Extraction: Training uses GRPO over sampled trajectories with clipped policy optimization and KL regularization against a fixed reference policy.The objective uses group-relative advantages, an importance ratio, and a weighted KL term.
- Signal Extraction: Rollout analysis identifies three complementary weakness patterns: forgetting, boundary instability, and rare action patterns.All activated signal types are retained because they capture complementary weaknesses.
- Signal Extraction: Forgetting signals detect tasks previously solved but now failed, while boundary signals detect both successful and failed trajectories under one policy.Rare signals target low-frequency action patterns that recur across trajectories, indicating systematic underexploration.
- Signal-Guided Environment Re-exploration: Signal-annotated trajectories are converted into structured failure contexts that condition multi-round, multi-step LLM re-exploration.The resulting action-observation triplets are grouped by task for abstraction and validation.
- Task Abstraction and Validation: The LLM abstracts grouped interaction evidence into task specifications, then environment execution determines whether synthesized tasks enter the updated training set.Successful executions are accepted, and failed executions with positive environment reward are also retained; other tasks are discarded.
4 Experiments
CoEvolve is evaluated across AppWorld and BFCL-V3 using multiple Qwen backbones, with experiments covering overall performance, training dynamics, ablations, signal distributions, diversity, transfer, and efficiency. Results show consistent gains, stable closed-loop optimization, complementary feedback signals, and diverse synthesized tasks.
- Main results: 19.4 average-score points are gained on Qwen2.5-7B, while Qwen3-4B gains 15.6 points over their baselines.The reported gains are measured across the evaluated AppWorld and BFCL-V3 results.
- Main results: +23.21 / +21.43 TGC/SGC gains occur on AppWorld’s challenge split for Qwen3-30B-A3B, while BFCL-V3 improves by +48.0 for Qwen2.5-7B-Instruct.On Qwen3-30B-A3B’s normal split, the corresponding gains are +11.75 / +10.79; Qwen3-4B-Instruct gains +36.5 on BFCL-V3.
- Main results: 63.00 on BFCL-V3 lets CoEvolve-trained Qwen3-4B surpass GPT-4 at 54.00 and Gemini-2.5-Flash at 41.50.The comparison is reported alongside evidence that CoEvolve improves over GRPO across all three backbones.
- Training dynamics: 0.21 → 0.35 performance growth accompanies a baseline decline from 0.29 to 0.23, while detected signals fall from 269 to 204 during training.Signal-driven task pass rate rises from 0.71 to 0.85 and stabilizes at 0.80; generated tasks expand into previously underrepresented regions.
- Ablation study: 49.36 is the best average score in the Qwen3-4B ablation, exceeding static synthetic data at 43.29 and random exploration at 45.43.Feedback-guided generation also improves AppWorld from 30.36 to 35.71 and BFCL-V3 from 60.50 to 63.00 relative to random exploration.
- Feedback signals: 51.4% of AppWorld signals and 45.5% of BFCL signals are boundary signals, while removing forgetting signals reduces performance from 49.36 to 45.18.Boundary and rare signals also contribute gains of approximately 1.6–1.9 points, supporting a complementary signal set.
- Data diversity and behavior: Most synthesized tasks have moderate similarity to validation examples, commonly 0.4–0.7, with few near 1.0, indicating non-near-duplicate task generation.Cross-comparison results also report retention and recovery of correct and previously failed cases on BFCL and AppWorld.
5 Conclusion
CoEvolve enables mutual evolution between the agent and its data distribution by using interaction feedback during policy optimization to guide task synthesis. Experiments on AppWorld and BFCL validate its effectiveness and efficiency.
- CoEvolve enables mutual evolution between the agent and its data distribution.
- Feedback signals extracted during policy optimization guide task synthesis and progressively adapt the agent’s capabilities and training data.
- Experiments on AppWorld and BFCL validate CoEvolve’s effectiveness and efficiency.
Limitations
CoEvolve’s feedback-driven co-evolution relies on a limited set of signals and on the agent’s current interaction trajectories. Early-training signals may be noisy or incomplete, while safety-critical deployment may require additional oversight and controls.
- The current feedback set covers only a subset of potentially informative signals, including forgetting, boundary, and rare signals.
- Because signals derive from the current policy’s trajectories, immature early-stage behavior may produce noisy or incomplete feedback.
- Adversarial or safety-critical settings may require human oversight, policy constraints, continuous auditing, and explicit safety filters.
A.1 Dataset
The evaluation covers AppWorld and BFCL, with appendix-only transfer experiments on ALFWorld and WebShop. These environments span digital-service APIs, function calling, web shopping, and household tasks.
- AppWorld: AppWorld simulates multi-step digital-service interactions across calendar, email, music, and social applications through Python API calls.
- AppWorld: AppWorld reports Task Goal Completion for individual tasks and Scenario Goal Completion for completing all tasks within a scenario.
- BFCL: BFCL v3 Multi-turn evaluates function and tool calling using strict multi-turn accuracy across correct functions and valid arguments.
- WebShop: WebShop simulates e-commerce interactions using search and click actions, scored by attribute matching between the selected product and request.
- ALFWorld: ALFWorld is a text-only, partially observable household environment requiring long-horizon navigation and object-manipulation tasks.
A.2 Implementation Details
Training uses VeRL with GRPO across Qwen instruction-tuned backbones and specified hardware, episode limits, task initialization, feedback cadence, and exploration models. Comparisons include closed-source, open-source, and backbone baselines.
- Training uses the VeRL framework with GRPO for reinforcement learning.
- Qwen2.5-7B-Instruct and Qwen3-4B-Instruct use one machine with 8× NVIDIA H20 GPUs, while Qwen3-30B-A3B-Instruct uses two such machines.
- Episodes are capped at 30 environment steps for AppWorld and BFCL and 15 steps for WebShop and ALFWorld, with excess treated as failure.
- Unless otherwise specified, training starts with 100 synthetic tasks, runs for 120 steps, regenerates feedback data every 10 steps, and uses Qwen3-Max for exploration.
- Baselines include closed-source and open-source LLMs plus backbone models evaluated with and without CoEvolve.
A.3 Additional Experiments and Analyses
Additional analyses show that CoEvolve transfers across interactive environments, depends on validated task synthesis and feedback-guided evolution, and benefits from balancing task relevance with diversity.
- Cross-environment transfer: CoEvolve consistently outperforms adaptive data-generation baselines across ALFWorld, BFCL, AppWorld, and WebShop.Under Qwen3-4B-Instruct, it beats Curriculum Learning and ReST on ALFWorld and WebShop, and Reflexion and Curriculum Learning on BFCL and AppWorld.
- Task validation: Removing task validation drops BFCL performance from 63.00 to 58.50 and AppWorld performance from 35.71 to 27.38.The authors attribute this degradation to synthesized tasks that are noisy or misaligned with environment dynamics.
- Exploration model and feedback: Feedback improves performance over matched synthesis-only baselines, while stronger exploration models raise the attainable ceiling.The controlled study compares each external exploration model with and without feedback-guided data evolution.
- Similarity-controlled synthesis: Mixed similarity settings perform best, indicating that balancing relevance and diversity is more effective than concentrating synthesis within one similarity range.Mean similarity is 38.43% for low, 53.28% for medium, and 64.73% for high bins.
- Hyperparameter robustness: CoEvolve remains reasonably robust beyond the reported hyperparameter range, with extreme settings trading off data quality against update cadence.These settings do not change the overall conclusion.
A.4 Analysis of Interaction Turns.
Synthesized BFCL trajectories shift toward longer interaction sequences and heavier tails than original trajectories, reflecting greater multi-step complexity while remaining a distributional observation.
- Interaction-turn distributions: Synthesized BFCL tasks shift toward higher interaction-step counts and a heavier tail relative to original data.The shift indicates more frequent longer interaction sequences and multi-step dependencies.
- Interpretation boundary: The analysis describes underrepresented interaction patterns without making claims beyond the observed data distribution.This limits the interpretation of the turn-distribution comparison.
A.5 Diversity and Relevance Analysis.
The analyses evaluate synthesized-task diversity and relevance, describe the feedback-loop prompt modules, and show that synthetic examples are more compositional and longer-horizon than original tasks.
- Metrics: Diversity uses Self-Redundancy@k, where lower cosine similarity to nearest-neighbor task intents indicates less redundancy and higher diversity.The metric is computed from sentence embeddings of synthesized task intents.
- Metrics: Relevance uses Relative Energy Distance, where lower EDrel indicates closer alignment between generated and target task-intent distributions.The target distribution may comprise human-annotated intents or a predefined target distribution.
- Metric trends: Across training steps, mean SR is 21.44% and mean EDrel is 0.95%, with SR showing greater variance than EDrel.SR variance is 17.30%, while EDrel variance is 0.73%.
- Synthetic sample analysis: Across BFCL and AppWorld, synthetic examples require more rounds, stronger cross-step dependencies, intermediate reasoning, and explicit correctness verification than mostly linear originals.Examples include BFCL file-content verification and AppWorld queue-versus-liked-library filtering before iterative actions.
- Feedback-loop prompts: Signal-conditioned prompts target forgetting, rare events, and boundary cases, while summarization extracts failure causes, objectives, focus patterns, and do-not-repeat constraints.These representations connect rollout traces to downstream exploration prompts.
- Task processing: Validation verifies task executability and abstraction converts validated traces into concise training specifications.The prompt templates operationalize these two downstream stages of task generation.