Source-linked AI summary
Complementary RL: Towards Efficient Experience-Driven Agent Learning
Dilxat Muhtar, Jiashun Liu, Wei Gao, Weixun Wang, Shaopan Xiong, Ju Huang, Siran Yang, Wenbo Su, Jiamang Wang, Ling Pan, Bo Zheng
TL;DR
Outcome-based RL for LLM agents is sample-inefficient because sparse rewards do not explain trajectory success or failure, while static experience can become misaligned with the improving actor. Complementary RL jointly evolves an outcome-trained policy actor and an experience extractor optimized by the utility of distilled experiences. The method improves performance over outcome-based baselines, with a reported 10% gain in single-task scenarios and robust multi-task scalability.
Problem
Outcome-based RL for LLM agents is sample-inefficient because sparse rewards do not explain trajectory success or failure, while historical experience may remain static or fail to coevolve with the actor.
Method
Complementary RL jointly trains a policy actor with outcome-based rewards and an experience extractor whose objective reflects whether distilled experiences facilitate actor success.
Results
Complementary RL outperforms outcome-based agentic RL baselines, achieving a 10% performance improvement in single-task scenarios and robust scalability in multi-task settings.
Takeaways & Limitations
The paper presents co-evolving experience extraction and policy learning as a paradigm for efficient experience-driven agent learning.
Takeaways & Limitations
Actor-Critic integration can collapse later in MiniHack Room training, and the authors leave investigation of hyperparameters and per-step versus periodic self-distillation to future work.
Abstract
from arXiv · showhide
Reinforcement Learning (RL) has emerged as a powerful paradigm for training LLM-based agents, yet remains limited by low sample efficiency, stemming not only from sparse outcome feedback but also from the agent's inability to leverage prior experience across episodes. While augmenting agents with historical experience offers a promising remedy, existing approaches suffer from a critical weakness: the experience distilled from history is either stored statically or fail to coevolve with the improving actor, causing a progressive misalignment between the experience and the actor's evolving capability that diminishes its utility over the course of training. Inspired by complementary learning systems in neuroscience, we present Complementary RL to achieve seamless co-evolution of an experience extractor and a policy actor within the RL optimization loop. Specifically, the actor is optimized via sparse outcome-based rewards, while the experience extractor is optimized according to whether its distilled experiences demonstrably contribute to the actor's success, thereby evolving its experience management strategy in lockstep with the actor's growing capabilities. Empirically, Complementary RL outperforms outcome-based agentic RL baselines that do not learn from experience, achieving 10% performance improvement in single-task scenarios and exhibits robust scalability in multi-task settings. These results establish Complementary RL as a paradigm for efficient experience-driven agent learning.
1 Introduction
Outcome-based RL for LLM agents is sample-inefficient because sparse rewards leave useful rollout information underused. Complementary RL addresses this by co-evolving the policy actor and experience extractor during training.
- Sparse outcome rewards provide little guidance about why multi-turn trajectories succeed or fail, leaving effective behaviors and failure patterns underused.
- Historical experience can improve rollout-data utilization, but self-reflection may become unreliable when the base model is weak or tasks are complex.
- Complementary RL jointly optimizes an actor and an experience extractor that maintains an evolving experience bank.
- The extractor is optimized by whether its distilled experiences facilitate actor success, while the actor learns from outcome-based rewards.
- The framework includes structured experience addition, refinement, and merging operations to resolve conflicts and redundancies.
- The authors also develop a fully asynchronous training framework with a centralized MemoryManager for scalable experience management.
2 Methodology
Complementary RL augments outcome-based actor training with distilled experience maintained by a jointly optimized extractor. Its actor and extractor are trained through an asynchronous infrastructure that retrieves, distills, and updates experience during learning.
- Problem formulation: The actor operates in an interactive MDP, receives a task goal, and is optimized for binary episode success.Episodes terminate upon task completion or a maximum step limit.
- Learning from experience: Structured experience distilled from past trajectories is stored in an experience bank and retrieved to guide later actor episodes.The experience bank is introduced to exploit behavioral information that outcome-only trajectory optimization leaves unused.
- Experience extractor: The experience extractor distills entries from completed trajectories, while each entry receives a reward based on the outcome of the trajectory it guided.Experience-reward pairs are accumulated for extractor optimization, with repeated retrieved entries treated as shared training samples.
- Actor optimization: The actor uses both experience-guided and experience-free rollouts, with separate condition-wise advantage normalization to preserve distinct learning signals.The two subgroups are balanced and their clipped surrogate losses use subgroup-specific means and standard deviations.
- Training infrastructure: Complementary RL trains the actor and extractor asynchronously, using a centralized ExperienceManager to coordinate shared-bank retrieval, consolidation, and distillation.The primary loop collects and optimizes actor rollouts while a background track distills trajectories and maintains the experience bank.
- Training infrastructure: Periodic experience processing and producer-consumer distillation support scalable maintenance of the shared experience bank across parallel environments.Completed trajectories are queued for background extraction, while periodic merging addresses redundant or conflicting entries.
4 Experiments
Experiments show that Complementary RL improves performance across single-task and multi-task settings, while ablations indicate that co-evolution and experience retrieval are important to its gains. Additional analyses examine extractor capacity, self-distillation, rollout latency, and task scaling.
- Single-Task Training: Complementary RL consistently outperforms the baseline across all four single-task environments, including a 1.3× performance margin on MiniHack Room and ALFWorld and a +3.0% gain on SWE-Bench.It also shows better training stability on MiniHack Room and ALFWorld, and faster improvement on SWE-Bench.
- Single-Task Training: Complementary RL achieves higher success rates while requiring 1.5× fewer actions on MiniHack Room and 2× fewer actions on ALFWorld.On SWE-Bench, more actions correspond to fully completing tasks and achieving a higher success rate rather than submitting prematurely.
- Multi-Task Training: In multi-task training, Complementary RL outperforms the baseline by +7% with test-time experience and +2% without it, while Static Online Exp. fails to surpass the baseline.The authors attribute Static Online Exp.’s weakness to distributional misalignment from a fixed extractor and report marginal actor improvement when only the extractor is optimized.
- Effect of Experience Extractor Capacity: A larger experience extractor yields a +5% average improvement across tasks in multi-task training.This analysis compares Qwen3-30B-A3B-Instruct-2507 with the default Qwen3-4B-Thinking-2507 extractor.
- Complementary RL with Self-Distillation: Self-distillation initially improves Complementary RL on MiniHack Room but collapses later in training.The authors suggest suboptimal hyperparameters or periodic rather than every-step self-distillation as possible explanations and leave further investigation to future work.
- Task Scaling: Complementary RL consistently outperforms the baseline as task mixtures scale, achieving +6.6% on three tasks and +8.1% on six tasks.The results are presented as evidence that its performance gains scale robustly with the number of tasks.
5 Related Works
Related work uses stored or retrieved experience and experience-conditioned or decoupled policy optimization, but these approaches do not provide the same co-evolutionary training of actor and extractor described here.
- Prior approaches store historical trajectories or workflows and retrieve them at inference time, but cannot guarantee the quality or relevance of the resulting experience.
- Some methods include experience as context during policy-gradient optimization, while others decouple experience-conditioned rollout collection from policy optimization or reproduce successful trajectories without experience context.
- Complementary RL differs by jointly co-evolving the policy actor and experience extractor within a unified training framework.
6 Conclusion
The paper presents Complementary RL as a unified algorithm and infrastructure co-design framework for accumulating and leveraging experience throughout RL training.
- Complementary RL jointly trains the policy actor and experience extractor in an asynchronous dual-loop rather than treating experience construction and management as static.The actor’s capabilities reshape extractor behavior, while improved extractor outputs accelerate actor learning.
A.1 Action Efficiency Under Multi-Task Training
During multi-task RL training, Complementary RL achieves superior action efficiency alongside higher success rates.
- Complementary RL consistently achieves superior action efficiency alongside higher success rates in the multi-task setting.Figure 10 reports the average number of actions per task during multi-task RL training.
A.2 Per-Task Performance with Stronger Experience Extractor
Per-task multi-task results compare two experience extractor sizes, with the larger 30B-A3B extractor providing greater benefit across all tasks.
- The 30B-A3B experience extractor consistently yields greater benefit than the 4B extractor across all tasks.
A.3 Search Time Throught Training
Average search time increases with larger rollout batch sizes, but the maximum observed search time remains around 1 second.
- Around 1 second is the maximum observed search time, despite increases with larger rollout batch sizes.Tuning query batch size, maximum waiting time, and parallel search workers may reduce search time further.
- Search time is measured across all environments and training steps for different rollout batch sizes.
A.4 Training Curves for Task Scaling Experiments
The task scaling experiments report training curves for Complementary RL and a baseline across different task mixture settings.
- Figure 13 provides the training curves for the task scaling experiments introduced in Section 4.3.
B Implementation Tricks
The paper addresses instability and inefficiency in experience-extractor training with retrieval diversification, count-aware reweighting, and selective actor criticism. It also reports implementation choices and auxiliary-reward findings that shape the final Complementary RL design.
- Retrieval stabilization: Retrieval diversification oversamples candidate experiences and penalizes frequent or recently retrieved entries before selecting the final top-K.The scoring function combines relevance with historical retrieval count and a recency penalty.
- Extractor optimization: Count-aware advantage reweighting discounts experiences according to cumulative training count and recency, while cooldown suppresses repeated short-window updates.The method targets overfitting and instability caused by repeatedly optimizing the same experience.
- Experience quality: Retrieved experiences can confuse the actor when they are stale or imprecise, especially when they conflict with the actor’s current capability.These failure modes motivate mechanisms that let the actor evaluate retrieved experience before consuming it.
- Actor-Critic analysis: Actor-Critic can improve early MiniHack Room success rates, but blocking rollout collection adds non-trivial latency, so it is not used by default.The authors recommend it when final performance is more important than rollout efficiency.
- Parameterization: The implementation uses separate actor and extractor parameters because their distinct objectives can impose conflicting gradient directions and destabilize shared-parameter training.The shared-parameter design did not guarantee stable training despite extensive experimentation.
- Reward design: The authors exclude auxiliary perplexity-reduction and relative-reward strategies after empirical comparisons failed to yield consistent improvement.The perplexity bonus was based on entropy reduction after injecting retrieved experience, while relative reward compared experience-guided outcomes with an experience-free subgroup.
C.2 Training Configuration
The experiments use standardized training settings across single-task and multi-task configurations, with representative distilled experiences reported for several environments. Multi-task training also produces experience described as transferable across tasks.
- General configuration: The general setup uses 4 parallel search workers, 4 embedding workers, Qwen3-Embedding-0.6B, and a constant AdamW learning rate of 1 × 10−6.Neither policy uses KL regularization.
- General configuration: Experiments use rollout batches of 128, group size K = 8, clip ratio ϵ = 0.2, and 145 training steps unless otherwise specified.The maximum interaction turns are 30, with sequence limits of 32,768 tokens for πθ and 65,536 for πϕ.
- Multi-task training: Multi-task training uses a total rollout batch size of 384, with each task contributing 128 examples, and evaluates 3-task and 6-task combinations.The reported 3-task set is MiniHack Room, WebShop, and ALFWorld; the 6-task set adds MiniHack Maze, MiniHack KeyRoom, and Sokoban.
- Single-task training: Single-task studies include MiniHack Room, WebShop, ALFWorld, and SWE-Bench, with representative distilled experiences presented for each environment.The WebShop and ALFWorld configurations use different rollout batches, training steps, and interaction limits.
- Experience analysis: The experience extractor distills universal experience transferable across tasks, illustrated through representative examples in Table 6.The corresponding table is identified as universal distilled experience from multi-task training.