Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agents

Motivation

• Experience-driven agents reuse prior interactions to adapt at deployment, but trained LLMs are largely static and lack continual adaptation to novel tasks.Existing approaches’ effectiveness is bounded by the model’s ability to extract and leverage experience, which relies on in-context learning and abstract reasoning.Source: §1 · pp. 1–2

Results

• Evolving-RL-trained skills transfer to other policies, while self-extracted skills can degrade performance versus no injection; training also improves the policy without test-time skill injection.Without injection, the trained policy reaches 81.1% success on ALFWorld unseen tasks and 28.05% overall action accuracy on Mind2Web, versus GRPO baselines of 33.7% and 22.83%, respectively.Source: Figure 1 · p. 2, §1 · p. 3

Research gap

• Prior RL methods improve experience utilization alone, while relying on external models or hand-crafted filters for extraction quality.Noisy experience can introduce variance and ambiguity, leading the policy to learn to ignore the provided experience.Source: §1 · p. 2

Approach

• Evolving-RL evaluates candidate textual skills on multiple retrieved tasks, using downstream feedback to co-optimize the extractor and solver.Evaluation trajectories are also reused to refine skill utilization.Source: §1 · p. 2

Results

• Evolving-RL raises ALFWorld unseen-task success from 44.6% (GRPO [19] baseline) to 88.6%, and Mind2Web overall action accuracy from 22.73% to 30.87%.Source: §1 · pp. 2–3

2.1 Experience-Driven Self-Evolving Agent

• Prior self-evolving-agent work explores experience representations, extraction, and memory management, but generally depends on the base model’s existing ability to extract and use experience.The cited work includes trajectories, workflows, executable skills, and strategic principles as representations.Source: §2.1 · p. 3

2.2 Experience-Augmented Reinforcement Learning

• Recent experience-augmented RL methods either use task-specific reflections that are difficult to transfer or reusable skill libraries whose extraction relies on manual filters or stronger external models.Evolving-RL instead evaluates transferable textual skills on retrieved downstream tasks and uses the feedback to co-train extractor and solver in one RL loop.Source: §2.2 · p. 3
• A concurrent co-optimization approach [15] uses separate models and single-task validation, whereas Evolving-RL evaluates for generalizability and trains both capabilities in one model.The authors state that their approach resolves the underlying instability; Figure 2 depicts online extraction followed by downstream evaluation that supplies signals to both roles.Source: §2.2 · p. 3, Figure 2 · p. 4

3.1 Framework Overview

• The shared policy acts as extractor and solver: it distills a source interaction into a procedural skill, then produces an action trajectory for a new task conditioned on that skill.Each iteration solves a source task, generates candidate skills, and evaluates skill-conditioned rollouts on retrieved tasks.Source: §3.1 · p. 4

3.2 Extractor Training

• Candidate skills are evaluated on the same retrieved tasks, selected using task-description similarity, with the source task also included in evaluation.Using identical downstream tasks makes candidate comparisons fair; embeddings and cosine similarity determine retrieval.Source: §3.2 · p. 5
• Each skill’s reward is its mean performance across retrieved tasks, aligning extraction with transfer utility; negative entropy regularization constrains extraction-policy entropy.The paper notes that noisy evaluation and the diversity of valid skill representations can otherwise drive entropy growth and training collapse.Source: §3.2 · p. 5

3.3 Solver Training

• For each downstream task, skill-conditioned trajectories form one comparison group, and within-group reward normalization trains the solver to exploit useful skills while resisting misleading ones.The task semantics and objective are shared across the group, making the trajectories comparable despite their different injected skills.Source: §3.3 · pp. 5–6

3.4 Joint Objective and Co-Evolution Dynamics

• The solver reuses all skill-conditioned evaluation trajectories, while the final objective combines extractor and solver losses over the same parameters.A gradient step improving extraction also updates solver weights; the solver encounters informative and noisy skills, building robustness under test-time-like conditions.Source: §3.3 · p. 6, §3.4 · p. 6

4.1 Experimental Setup

• Experiments use Qwen2.5-7B-Instruct and average results over five runs, with ALFWorld split by task type and Mind2Web evaluated across three standard settings.ALFWorld training uses seen tasks, while test-time skill collection may use the unseen training subset; Mind2Web permits trajectory generation and skill extraction on its evaluation set.Source: §4.1 · p. 6

4.2 Main Results

• Evolving-RL leads both benchmarks: ALFWorld reaches 96.0 overall success with skills, while Mind2Web action-accuracy gains over GRPO span all three settings.On ALFWorld unseen tasks, success rises from 44.6 with GRPO (w/ skills) to 88.6; Mind2Web relative action-accuracy improvements over GRPO are 45.9%, 42.8%, and 29.4% on cross-task, cross-website, and cross-domain settings, respectively.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 7

4.3 Ablations and Analysis

• Solver-only training improves ALFWorld generalization by 73.2% relative to GRPO, while extractor-only training improves seen tasks but virtually not unseen tasks.The solver-only result is consistent with injected skills acting as structured auxiliary supervision; extractor-only overfitting underscores the need for co-evolution.Source: Table 3 · p. 8, §4.3 · p. 8
• Relevant skills distinctly improve performance, whereas irrelevant skills perform comparably to no injected skills, indicating gains depend on skill content rather than prompt presence.Irrelevant skills are constructed from questions ranked least relevant by the retrieval function.Source: Figure 3 · p. 8, §4.3 · p. 9
• Evolving-RL skills transfer across policies: when supplied to the base model, overall success rises from 45.5 to 60.4.The reported comparison also finds larger gains from Evolving-RL skills than from the evaluation policy’s self-extracted skills.Source: §4.3 · p. 9, Table 4
• Cross-policy transfer suggests Evolving-RL improves skill extraction quality rather than creating a private communication protocol between extractor and solver.The paper points to an Appendix B case study for further illustration.Source: §4.3 · p. 9

5 Conclusion

• Evolving-RL models experience extraction and utilization as a unified process in which evaluation rewards and skill-conditioned trajectories jointly train a shared policy.The framework aligns experience rewards with practical utility and reuses trajectories collected during evaluation.Source: §5 · p. 9
• Models trained with Evolving-RL consistently outperform GRPO in both in-domain performance and out-of-domain generalization.The conclusion attributes gains to explicit skill conditioning at inference and internalized reusable procedural patterns.Source: §5 · p. 9
• The framework uses relatively straightforward skill-management and retrieval strategies, leaving more sophisticated experience evolution for future work.The authors identify post-deployment adaptation as an important setting for those mechanisms.Source: §5 · p. 9
• Evaluation error can inject training noise and may cause instability in some cases; reducing that noise and improving evaluation robustness remain future work.Source: §5 · pp. 9–10

A.1 Reliability of skill evaluation

• Each candidate skill is evaluated with one rollout per retrieved task, producing unbiased estimates of its average reward across those tasks.Source: §A.1 · p. 13
• With moderately large K, independent bounded rollout rewards yield an approximately Gaussian estimated gap, even when task-specific reward distributions differ.Source: §A.1 · p. 13
• Skill-ranking reliability increases with a larger true performance gap and lower variance in the estimated gap.The appendix states this after deriving an approximate probability that finite-sample evaluation preserves the correct ordering.Source: §A.1 · p. 13
• Stability-controlled co-evolution keeps KL and entropy losses bounded and produces smoother reward improvement than naive co-evolution.Source: Figure 4 · p. 14
• For Bernoulli rewards, the analysis reduces to the binary-reward derivation used for ALFWorld.Source: §A.1 · p. 14

A.2 Training Stability

• Small reward gaps between malformed and well-formed skills can make empirical evaluation unreliable, allowing malformed skills to receive positive extractor-side advantages by chance.Source: §A.2 · p. 14
• Chance-positive advantages can reinforce rare tokens, grow extractor entropy, and transfer noise to the solver, potentially destabilizing or collapsing training.Source: §A.2 · p. 14
• The method filters skills with abnormal characters and applies negative entropy regularization to constrain extraction-policy entropy; together, these controls improve co-evolution stability.Source: §A.2 · p. 14
• In the case study, the Evolving-RL-trained model extracts a coherent procedure, while the base model misorders steps and omits a critical action.The base-model skill can substantially reduce solver success when injected.Source: Figure 5 · p. 15
• An LLM-based naturalness assessment produced stabilizing effects similar to the rule-based correction.Source: §A.2 · pp. 14–15
• The authors attribute instability to evaluation-noise feedback and unconstrained extraction entropy, rather than parameter interference, and report that controls allow joint optimization in one policy.Source: §A.2 · p. 15

B Case Study

• For “put the vase in the safe,” the trained model’s skill correctly guides locating and picking up the object, finding the container, and placing the object inside.Source: B Case Study · p. 15
• The base model’s skill puts container-finding before moving to the object and omits picking it up, flaws that can substantially reduce solver success when the skill is injected.Source: B Case Study · p. 15

• The section describes the environment configuration and training implementation in detail.Source: C Implementation Details · p. 16

• ALFWorld samples are complete interaction trajectories, with observations retained as context but masked from policy optimization loss.The context includes prior observations and executed actions; successful trajectories receive reward 10 and unsuccessful ones reward 0.Source: §C.1 · p. 16
• Mind2Web samples contain the current filtered observation and prior actions, but omit past observations because of context-length constraints.Each sample is a single interaction step rather than a full trajectory.Source: §C.1 · p. 16
• Mind2Web terminates a trajectory after an incorrect action because the dataset supplies only the next state on the ground-truth trajectory.Action accuracy is the proportion of correct predictions over executed actions.Source: §C.1 · p. 16
• Mind2Web solver training uses turn-wise GRPO with a reward of 1 for a correct action and 0 otherwise.The extractor reward sums step-level rewards into a trajectory-level reward, then averages trajectory rewards over K retrieved downstream tasks.Source: §C.1 · p. 16
• Solver training uses fine-grained step-level supervision, while extractor evaluation tracks the overall quality of procedural experience distilled from complete trajectories.Source: §C.1 · p. 16
• Table 5 summarizes the main training hyperparameters for ALFWorld and Mind2Web.Source: Table 5 · p. 17

• All experiments start from Qwen2.5-7B-Instruct and use AdamW with a constant learning rate of 1 × 10−6, weight decay 0.1, β1 = 0.9, and β2 = 0.98.Training uses GRPO-style policy optimization with low-variance KL regularization and clipped policy updates.Source: §C.2 · p. 17
• Each rollout samples 16 source tasks, while similar-task embeddings are precomputed offline using Qwen3-Embedding-4B.Runs use 8 NVIDIA H800 GPUs and take approximately 10 hours on ALFWorld and 17 hours on Mind2Web.Source: §C.2 · p. 17
• Training runs for 75 steps on ALFWorld and 150 steps on Mind2Web; GRPO uses the same step counts and matches solver samples per update.Each GRPO update uses B × N × K samples.Source: §C.2 · p. 17
• The number of candidate skills N sets the extractor-side comparison set, while K controls how broadly each skill is evaluated for transferability.The weights λe and λs balance skill extraction and utilization during joint optimization.Source: §C.2 · p. 17