Source-linked AI summary
Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning
Yaorui Shi, Yuxin Chen, Zhengxi Lu, Yuchun Miao, Shugui Liu, Qi GU, Xunliang Cai, Xiang Wang, An Zhang
TL;DR
Persistent skill-augmented agents must jointly select, use, and distill skills, but prior methods optimize these capabilities separately or with different reward sources. Skill1 trains one policy to co-evolve all three using a shared task-outcome signal, achieving a 97.5% success rate on ALFWorld and the best performance across methods on WebShop.
Problem
Prior skill-based agents optimize skill selection, utilization, and distillation incompletely or with separate reward sources rather than toward a shared objective.
Method
Skill1 trains a single policy to co-evolve skill selection, utilization, and distillation using one task-outcome signal decomposed into trend and variation credits.
Results
97.5% success rate on ALFWorld, with Skill1 surpassing all baseline skill-augmented agents and achieving the best performance across methods on WebShop.
Takeaways & Limitations
The results support jointly evolving selection, utilization, and distillation through shared credit assignment because the three capabilities improve together and depend on one another.
Takeaways & Limitations
Evaluation covers only two text-based environments, and the skill library is capped at 5,000 entries, leaving broader generalization and scalability unexplored.
Abstract
from arXiv · showhide
A persistent skill library allows language model agents to reuse successful strategies across tasks. Maintaining such a library requires three coupled capabilities. The agent selects a relevant skill, utilizes it during execution, and distills new skills from experience. Existing methods optimize these capabilities in isolation or with separate reward sources, resulting in partial and conflicting evolution. We propose Skill1, a framework that trains a single policy to co-evolve skill selection, utilization, and distillation toward a shared task-outcome objective. The policy generates a query to search the skill library, re-ranks candidates to select one, solves the task conditioned on it, and distills a new skill from the trajectory. All learning derives from a single task-outcome signal. Its low-frequency trend credits selection and its high-frequency variation credits distillation. Experiments on ALFWorld and WebShop show that Skill1 outperforms prior skill-based and reinforcement learning baselines. Training dynamics confirm the co-evolution of the three capabilities, and ablations show that removing any credit signal degrades the evolution.
1 Introduction
Skill1 unifies skill selection, utilization, and distillation by training a single policy toward a shared task-outcome objective. A single-signal credit-assignment scheme enables their co-evolution and improves performance on ALFWorld and WebShop.
- Motivation: Skill-augmented agents require jointly evolving skill selection, utilization, and distillation, but prior methods optimize only subsets or use separate reward sources.Unoptimized capabilities create bottlenecks, such as effective skill use paired with sub-optimal skill routing.
- Method: Skill1 trains a single policy across selection, utilization, and distillation with a shared task-outcome signal.The policy queries the skill library, re-ranks candidates, solves the task conditioned on the selected skill, and distills a new skill from experience.
- Credit Assignment: Skill1 decomposes outcome signal r(τ) into low-frequency trends for selection and high-frequency variation for distillation, while the outcome directly rewards utilization.The trend is the moving average of outcomes associated with each skill and reflects skill utility.
- Results: 97.5% success rate on ALFWorld surpassed all other baseline skill-augmented agents, while training dynamics showed simultaneous improvement across selection, utilization, and library quality.Removing any single stage’s credit-assignment signal degraded all three capabilities.
2 Preliminary: LLM Agent with Skill Library
The paper models a skill-augmented LLM agent as a POMDP with a persistent skill library. The agent selects and uses skills during execution, then distills new skills, with §3 jointly optimizing these stages from task outcomes.
- Task formulation: The agent operates in a POMDP whose state contains the task instruction, environment state, and persistent skill library.Observations provide a partial view of the task, environment, and skill selected from the library via a frozen encoder.
- Skill library interaction: The workflow comprises skill selection, skill-conditioned multi-turn execution, and trajectory-based distillation of a reusable skill.The policy generates a query, re-ranks retrieved candidates, interacts with the environment, and reflects on the trajectory to create a new skill.
- Skills for LLM agents: Each skill pairs a natural-language action strategy with a scenario description specifying when it applies.The library continuously expands as the agent explores the environment, and execution is conditioned on the selected strategy.
- Joint optimization: All three stages are optimized jointly through a single policy, with every learning signal derived from the task outcome r(τ).The policy πθ can be optimized with reinforcement-learning algorithms such as GRPO.
3 Method
Skill1 trains one policy to jointly select, use, and distill skills toward a shared task-outcome objective. It assigns capability-specific credits from the outcome and optimizes all generation segments together.
- Workflow: Skill1 executes skill selection, utilization, and distillation sequentially within each trajectory, receiving a binary terminal reward r(τ) ∈{0, 1}.The trajectory contains a query, selected skill, interaction sequence, and newly distilled skill.
- Workflow: The policy generates a library-search query, re-ranks top-K semantically retrieved candidates, and conditions environment interaction on the top-ranked skill.Both query generation and re-ranking are produced by πθ, making selection directly optimizable through policy gradients.
- Workflow: After each rollout, the policy distills a reusable strategy and applicability description; the new skill enters the library only when r(τ) = 1.At capacity |B| = Nmax, the lowest-retirement-score skill is removed.
- Credit assignment: Selection uses task-outcome gradients for queries and a per-skill exponential-moving-average utility trend to reward re-ranking that agrees with long-term skill quality.All retrieved candidates are updated, and the best retrieved utility provides the library baseline.
- Credit assignment: Distillation is credited by outcome variation relative to the library’s highest retrieved utility: positive variation favors novel improvements, while negative variation discourages redundancy.This variation approximates whether a newly distilled skill would improve future performance, which is unavailable during training.
- Joint optimization: GRPO jointly optimizes query, re-ranking, action, and distilled-skill segments in one update, combining utilization, re-ranking, and distillation objectives as J(θ) = J util(θ) + λ1 J rerank(θ) + λ2 J distill(θ).GRPO normalizes rewards within each task’s G rollouts into group-relative advantages.
4 Experiments
Skill1 achieves the strongest reported performance on ALFWorld and WebShop by jointly optimizing skill selection, utilization, and distillation. Ablations and training analyses show that these capabilities and their credit signals mutually reinforce performance, skill quality, and library coverage, while distillation controls computational cost.
- Main results: Skill1 reaches 97.5% average success rate on ALFWorld, surpassing RetroAgent by 2.6 points and ranking first on five of six task types; it also performs best on WebShop.Clean is the sole ALFWorld exception, where Skill1 is slightly below RetroAgent, with a statistically insignificant gap.
- Main results: Skill1 surpasses GiGPO, the strongest RL-only method, by 6.7 points, with largest gains on Look and Pick2, where reusable skills support multi-procedure composition.GiGPO absorbs strategies implicitly into parameters rather than explicitly reusing them across tasks.
- Ablations: Removing the library drops average success from 97.5% to 80.9%, removing distillation reduces it by 5.1 points, and removing selection reduces it by 5.7 points.Heat and Pick2 are especially affected because they require routing to and composing reusable multi-step sub-procedures; without distillation, the library stores raw trajectories instead of condensed strategies.
- Ablations: Setting λ1=0 or λ2=0 reduces performance by 3.5 and 2.6 points, while removing both yields 90.2%, demonstrating complementary auxiliary objectives.The signals also benefit utilization beyond their direct targets, so both are needed for full co-evolution.
- Training dynamics: Selection precision reaches 0.95 by step 20, followed by utilization and distillation reaching 0.8 by step 60; removing credit signals slows all three capabilities.Full Skill1’s task-skill similarity rises from 0.51 to 0.60, while top-ranked skill utility reaches 0.91 by step 85 and both ablations lag by approximately 0.10.
- Efficiency and library evolution: Skill1 runs at 387 to 494s per step, roughly 1.3 to 1.7 times slower than library-free GRPO, while distillation prevents raw-trajectory growth from making the system 69% slower by step 60.Skill1’s broader, more uniform skill usage contrasts with ablations that concentrate usage on a few popular skills; its library remains capped at 5,000 skills.
5 Conclusion and Limitations · A Related Work
Skill1 unifies skill selection, utilization, and distillation under a shared task-outcome objective, using signal decomposition for per-capability credit assignment and showing gains across ALFWorld and WebShop. The paper also identifies limits in environmental coverage and skill-library scalability, while situating the framework among reinforcement learning, experience reuse, and persistent skill-library research.
- 5 Conclusion and Limitations: Skill1 trains a single policy to co-evolve skill selection, utilization, and distillation toward a shared task-outcome objective.The framework derives per-capability credit assignment by decomposing the task signal into low-frequency trend and high-frequency variation, without auxiliary rewards.
- 5 Conclusion and Limitations: Experiments on ALFWorld and WebShop show consistent gains over prior skill-based and reinforcement learning baselines.Ablations confirm that the three capabilities evolve in a coupled manner.
- 5 Conclusion and Limitations: Skill1’s evaluation covers only two representative text-based agent environments, leaving generalization to deep search and visual-observation settings unexplored.The limitation explicitly concerns environment coverage.
- 5 Conclusion and Limitations: The skill library is capped at 5,000 entries, which may become a bottleneck as task diversity grows.The paper identifies more sophisticated eviction or hierarchical organization as an open scalability concern.
- A Related Work: Related reinforcement learning work for LLM agents includes GRPO, anchor-state grouping, dynamic sampling with asymmetric clipping, and methods for long-horizon multi-turn challenges.The cited multi-turn approaches use hierarchical value functions, leave-one-out advantage estimation, MCTS-guided search, and exploration-based trajectory optimization.
- A Related Work: Experience-reuse research improves RL sample efficiency through structured past experience and supports continuous adaptation through memories of interaction histories or distilled lessons.RetroAgent combines intrinsic progress rewards, language-based lesson extraction, and utility-aware selection.
- A Related Work: Persistent skill-library research studies selection through embedding selectors, heuristic scoring, learned routing, and policy log-probability ranking.The passage also identifies persistent skill libraries as a growing direction for LLM agents and begins discussing utilization approaches.
B Algorithm Details
Skill1 uses Group Relative Policy Optimization (GRPO) to optimize the policy without a separate value network, estimating advantages relative to rollouts sampled from the same task. Its clipped surrogate objective is controlled by clipping, KL-regularization, and rollout-length terms.
- Optimization: GRPO eliminates the need for a separate value network by computing advantages relative to a group of rollouts sampled from the same task.For each task d, the method samples a group of G rollouts.
- Optimization: The GRPO objective maximizes a clipped surrogate based on per-token importance ratios.The passage introduces the per-token importance ratio before defining the clipped surrogate objective.
- Optimization: The clipping ratio ϵ, KL coefficient β, reference policy πref, and rollout token count |τi| define the objective’s control terms.β controls KL regularization toward πref, while |τi| denotes the number of tokens in rollout i.
C Implementation Details
Skill1 uses a distributed VeRL training setup with fixed auxiliary-objective weights and converges in approximately 100–150 steps. Baselines are reproduced or sourced under matched model, budget, and data-split conditions, with additional hyperparameters documented in tables.
- Training infrastructure: Skill1 trains on 8 NVIDIA H800-80GB GPUs with VeRL, FSDP, BFloat16 precision, and vLLM tensor parallelism of 4.Training converges in approximately 100–150 steps, requiring roughly 30 hours on ALFWorld.
- Training infrastructure: The auxiliary objective weights remain fixed at λ1 = λ2 = 0.3 throughout all experiments unless otherwise specified.
- Baseline reproduction: Baseline comparisons use the same Qwen2.5-7B-Instruct base model, 150-epoch training budgets, and train/test splits for all reinforcement-learning baselines.RetroAgent is reproduced with its official implementation, while SkillRL, EvolveR, Mem0, SimpleMem, and GiGPO use results from their respective papers.
- Hyperparameters: Shared, per-environment, and skill-library hyperparameters are documented in Tables 4, 5, and 6, respectively.
D Statistical Analysis
The analysis uses three independent random seeds, reports mean ± standard deviation, and evaluates significance with Welch’s t-test. Random seeds capture variability from initialization, rollout ordering, and skill-library evolution.
- Statistical Analysis: All methods use 3 independent random seeds and report mean ± standard deviation (1- σ).The random seed is identified as the primary source of variability.
- Statistical Analysis: Random seeds affect parameter initialization, rollout sampling order, and the skill library evolution trajectory.These factors explain the reported variability across runs.
- Statistical Analysis: Statistical significance is assessed using SciPy’s ttest_ind with equal_var=False, corresponding to Welch’s t-test.The test is configured without assuming equal variances.
D.1 Full Performance Breakdown
This section reports per-task-type ALFWorld success rates for RetroAgent, selected as the strongest baseline and evaluated with three independent seeds to estimate variance.
- D.1 Full Performance Breakdown: RetroAgent is selected as the strongest baseline and evaluated under identical conditions with 3 independent seeds.The three seeds provide variance estimates for the per-task-type success rates reported in Figure 7.
D.2 Analysis · E Broader Impacts
Skill1 significantly outperforms RetroAgent on aggregate performance with lower variance across seeds, while autonomous skill accumulation offers efficiency benefits but introduces unpredictability, harmful-action, and injection risks requiring oversight. The framework therefore combines promising agent improvement with deployment-safety concerns.
- D.2 Analysis: 97.5±0.6 versus 94.9±0.9 on ALF All shows Skill1 significantly outperforming RetroAgent.Welch’s t-test gives t = 4.06, df = 3.40, p = 0.021 (< 0.05), ruling out seed variance as the explanation for the gain.
- D.2 Analysis: Per-task significance is strongest on Heat, Cool, and Look, with p = 0.004, p = 0.005, and p = 0.020, respectively.These are the tasks where RetroAgent struggles most.
- D.2 Analysis: Skill1’s overall standard deviation is 0.6 versus RetroAgent’s 0.9, indicating more stable convergence across seeds.The unified evolution of selection, utilization, and distillation reduces sensitivity to initialization.
- D.2 Analysis: Skill1 consistently outperforms RetroAgent on five of six task types and the average score.The per-task success rates are reported as mean ± standard deviation over three seeds.
- E Broader Impacts: Autonomous acquisition and reuse of behavioral skills can reduce manual engineering effort and enable more sample-efficient learning in interactive environments.These benefits arise from training LLM agents to acquire and reuse skills through reinforcement learning.
- E Broader Impacts: Autonomous skill accumulation may produce emergent behaviors that are difficult to predict or audit.This risk is associated with agents that autonomously accumulate skills.
- E Broader Impacts: In high-stakes deployments, an unconstrained skill library could encode harmful action sequences and introduce new injection risks.The passage recommends human-in-the-loop oversight and constraining the action space in safe domains.
F Case Studies · G Prompt Templates
The case studies show Skill1 transferring failure-avoidance and error-correction knowledge through its co-evolved skill library, while the prompt templates operationalize selection, utilization, and distillation for ALFWorld and WebShop. Together, these sections connect qualitative behavior to the framework’s concrete prompting stages.
- F Case Studies: Skill1’s ALFWorld case studies illustrate two transfer mechanisms: failure avoidance in appliance selection and error correction in object targeting.Both cases compare Skill1 with RetroAgent on the same test task and attribute qualitatively different behavior to unified optimization of selection, utilization, and distillation.
- F Case Studies: Skill1 avoided the stoveburner failure by using the retrieved microwave strategy and completed the execution in 6 actions, described as optimal.Its reasoning explicitly identified the microwave as the most suitable option based on past reflections.
- F Case Studies: In the object-targeting task, Skill1 addressed multiple alarmclock instances by using an origin skill that warned against selecting the wrong object.The test task examined the alarmclock with the desklamp, and targeting the wrong instance caused failure.
- F Case Studies: Skill1’s reasoning first searched the dresser for both the alarmclock and desklamp, completing the execution trace in 3 actions, described as minimal.The retrieved skill had Utility: 0.997 and Used 112 times.
- G Prompt Templates: Algorithm 1’s prompt templates comprise query generation, candidate re-ranking, environment interaction conditioned on a selected skill, and trajectory-based skill distillation.The selection stage generates q to retrieve candidates from B, ranks BK, and selects the top skill z; utilization uses z.strat, while distillation produces snew from τ.
- G.1 ALFWorld: The ALFWorld templates prompt one-sentence experience queries, usefulness rankings in <rank> tags, skill-conditioned action selection, and JSON evaluation of task outcomes and lessons.The utilization prompt includes past reflections, recent observations and actions, the current step, and admissible actions; evaluation asks for task success, action, navigation, and general-use descriptions.
- G.2 WebShop: The WebShop templates mirror this pipeline with shopping-specific queries and rankings, skill-conditioned stepwise action reasoning, and JSON evaluation of search, filtering, selection, and purchase.The action prompt requires reasoning inside <think> tags before choosing an admissible action inside <action> tags, and the evaluator records general-use lessons.