Source-linked AI summary
SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, Huaxiu Yao
TL;DR
LLM agents need better ways to learn from experience because raw-trajectory memories are redundant and noisy. SkillRL distills experiences into hierarchical skills and recursively evolves the skill library with the policy, achieving state-of-the-art results across diverse benchmarks with reported improvements over strong baselines.
Problem
LLM agents often operate in isolation, while raw-trajectory memories are redundant and noisy, limiting extraction of reusable behavioral patterns.
Method
SkillRL distills successful and failed experiences into a hierarchical SkillBank with adaptive retrieval, then recursively evolves the library alongside the policy during reinforcement learning.
Results
15.3% improvements across ALFWorld, WebShop, and seven search-augmented benchmarks yield state-of-the-art performance against memory-based agent-tuning baselines.
Takeaways & Limitations
SkillRL demonstrates that abstracting experience into compact, reusable skills can support capable, sample-efficient agents using substantially less context than memory-based approaches.
Takeaways & Limitations
The policy is trained subject to the context-length constraint |c| ≤Lmax.
Abstract
from arXiv · showhide
Large Language Model (LLM) agents have shown stunning results in complex tasks, yet they often operate in isolation, failing to learn from past experiences. Existing memory-based methods primarily store raw trajectories, which are often redundant and noise-heavy. This prevents agents from extracting high-level, reusable behavioral patterns that are essential for generalization. In this paper, we propose SkillRL, a framework that bridges the gap between raw experience and policy improvement through automatic skill discovery and recursive evolution. Our approach introduces an experience-based distillation mechanism to build a hierarchical skill library SkillBank, an adaptive retrieval strategy for general and task-specific heuristics, and a recursive evolution mechanism that allows the skill library to co-evolve with the agent's policy during reinforcement learning. These innovations significantly reduce the token footprint while enhancing reasoning utility. Experimental results on ALFWorld, WebShop and seven search-augmented tasks demonstrate that SkillRL achieves state-of-the-art performance, outperforming strong baselines over 15.3% and maintaining robustness as task complexity increases. Code is available at this https://github.com/aiming-lab/SkillRL.
1. Introduction
LLM agents struggle to learn from past successes and failures because raw-trajectory memories are noisy and redundant. SkillRL addresses this gap by distilling experience into reusable skills and recursively evolving the skill library with the policy.
- Motivation: LLM agents often operate episodically, limiting their ability to learn from past successes or failures and transfer knowledge across tasks.
- Motivation: Raw-trajectory memory is lengthy, redundant, and noisy, making critical information difficult to extract and potentially degrading performance.
- Approach: SkillRL uses experience-based distillation to preserve successful episodes as demonstrations and synthesize failures into concise lessons.
- Approach: SkillRL builds a hierarchical SkillBank containing general and task-specific skills for adaptive retrieval during decision-making.
- Approach: SkillRL recursively analyzes failure modes during reinforcement learning so the skill library and policy co-evolve as task complexity increases.
- Results: 15.3% improvements across ALFWorld, WebShop, and seven search-augmented benchmarks establish state-of-the-art performance against memory-based agent-tuning baselines.
2. Preliminaries
The paper models an LLM agent as a policy acting in a natural-language task environment, with skills and demonstrations supplied as additional context. Training seeks to maximize expected return under a context-length constraint, while GRPO optimizes responses using relative rewards without a critic.
- LLM Agents: An LLM agent observes states, selects actions, receives rewards, and forms trajectories while acting in an interactive environment.
- LLM Agents: The policy conditions on observation history, task description, and additional context such as skills or demonstrations.
- LLM Agents: The learning objective maximizes expected return subject to the context-length constraint |c| ≤Lmax.
- Group Relative Policy Optimization: GRPO avoids critic training by using intra-group relative rewards to optimize the policy.
- Group Relative Policy Optimization: GRPO computes normalized advantages and applies a PPO-style clipped objective using sampled responses and their rewards.
3. SKILLRL
SKILLRL converts interaction trajectories into a hierarchical skill library and uses retrieved skills during reinforcement learning. Its recursive evolution mechanism updates the library from validation failures while the policy improves.
- 3. SKILLRL: SKILLRL transforms redundant trajectories into concise, actionable skills and organizes them in a hierarchical SKILLBANK for retrieval.The framework includes experience-based distillation and separates general from task-specific expertise.
- 3.1. Experience-based Skill Distillation: Successful trajectories yield strategic patterns, while failed trajectories are converted into concise lessons covering failure points, flawed actions, corrections, and prevention principles.Both trajectory outcomes are retained because failures expose boundary conditions that successes alone may not reveal.
- 3.2. Hierarchical Skill Library (SKILLBANK) Construction: General skills provide universal strategic guidance, whereas task-specific skills encode heuristics for particular task categories and are retrieved by semantic similarity.General skills are always included; task-specific retrieval uses a similarity threshold and a top-K limit.
- 3.1. Experience-based Skill Distillation: 10–20× token compression is achieved compared with raw trajectories while preserving or enhancing the utility of the original experience.The reduced representation enables experiential knowledge to fit within limited context windows.
- 3.3. Recursive Skill Evolution: Recursive evolution expands or refines SKILLBANK using diverse failed trajectories from task categories whose validation accuracy falls below a threshold.The teacher identifies uncovered failure patterns, proposes new skills, and refines ineffective existing skills.
- 3.3. Recursive Skill Evolution: Cold-start supervised fine-tuning teaches the base model to retrieve, interpret, and apply skills before reinforcement learning begins.The resulting policy initializes RL and serves as the KL-regularization reference policy.
4. Experiments
Experiments evaluate SKILLRL across ALFWorld, WebShop, and seven search-augmented QA benchmarks, comparing it with competitive agents and analyzing its components and training dynamics. SKILLRL consistently outperforms baselines, while hierarchical skills, distilled experience, and recursive evolution improve performance and efficiency.
- Experimental Setup: SKILLRL is evaluated on ALFWorld, WebShop, and seven search-augmented QA tasks against closed-source, prompt-based, vanilla RL, and memory-augmented RL baselines.The experiments also examine component contributions, skill-library growth, and convergence speed.
- Main Results: SKILLRL achieves 89.9% success on ALFWorld and 72.7% on WebShop, outperforming the best prompt-based baselines.These results indicate stronger use of experience than in-context retrieval from verbose trajectories.
- Main Results: 12.3% absolute improvement over GRPO on ALFWorld, from 77.6% to 89.9%, is attributed to skill augmentation rather than optimizer choice.On Cool and Pick2, SKILLRL exceeds GRPO by 23.0% and 22.8%, respectively.
- Main Results: SKILLRL reaches 54.7% on ALFWorld and 37.5% on WebShop with Mem0+GRPO, while outperforming this hybrid by about 35.2 percentage points in absolute success rate.The comparison supports high-level skill abstraction and a co-evolving library over simple trajectory storage or prompt-based retrieval.
- Main Results: SKILLRL exceeds GPT-4o by 41.9% and Gemini-2.5-Pro by 29.6% on ALFWorld despite using Qwen2.5-7B-Instruct.The result is presented as evidence that structured experiential knowledge can compensate for model-scale differences.
- Main Results: 47.1% average score on search-augmented QA surpasses Search-R1 at 38.5% and EvolveR at 43.1%.SKILLRL also surpasses EvolveR by 19.4% on Bamboogle and remains competitive on out-of-distribution TriviaQA and 2Wiki.
- Ablation Studies: Removing hierarchical structure decreases performance by 13.1% on ALFWorld and 11.3% on WebShop, while replacing the skill library with raw trajectories causes degradation of up to 25%.These ablations support the contribution of universal strategic guidance and distilled skills.
- Efficiency and Evolution: SKILLRL averages fewer than 1,300 prompt tokens versus approximately 1,450 for raw memory, reducing context length by about 10.3%.With recursive evolution, SKILLRL exceeds 80% success within 60 training steps, while the baseline needs approximately 90 steps to reach a lower peak.
5. Related Work
Prior agent systems use reasoning, reflection, orchestration, and external memory, but raw trajectories remain redundant and token-heavy. SkillRL instead illustrates adaptive retrieval and integration of general and task-specific skills during reasoning.
- LLM Agents: LLM-agent frameworks such as ReAct, Reflexion, AutoGen, and CAMEL support reasoning, reflection, orchestration, and tool integration.These approaches include interleaved reasoning and acting, verbal reinforcement from failures, and general-purpose multi-agent capabilities.
- Memory Mechanisms in Agents: External-memory methods commonly use static retrieval or raw trajectories as few-shot references, but those trajectories are token-heavy and redundant.Memory architectures address finite context windows and agents’ inability to learn from experience, while raw storage limits efficiency.
- Evolution of Agentic Skills and Reinforcement Learning: Agentic skills are compact, reusable strategies increasingly connected to continual learning and reinforcement learning.Applying reinforcement learning to agentic skills remains challenging because of sparse rewards and long horizons.
- SkillRL: SkillRL case studies show adaptive retrieval and integration of General Skills and Task-Specific Skills during WebShop and ALFWorld reasoning.The examples associate hierarchical skill use with precise and efficient task execution.
6. Conclusion
The supplied material details prompt templates for agent execution, skill discovery, and synthetic trajectory generation. These templates structure reasoning, action selection, failure analysis, and skill-conditioned demonstrations across ALFWorld and WebShop.
- Online Inference: Online inference prompts provide task descriptions, interaction history, current observations, admissible actions, and retrieved experiences before action selection.ALFWorld and WebShop prompts require step-by-step reasoning enclosed in <think> tags followed by an admissible action in <action> tags.
- Online Inference: WebShop execution prompts instruct agents to reason about which admissible action best advances the shopping goal.Actions include search[query], click[element], and buy now.
- Skill Discovery: Skill-discovery prompts ask a teacher model to analyze successful and failed trajectories and distill reusable skills into structured JSON.The prompts cover initial skill distillation for WebShop and ALFWorld, plus dynamic discovery of new skills from failures.
- Skill Discovery: Dynamic failure analysis requests 1-3 new actionable skills with unique identifiers, titles, principles, and application conditions.The requested skill identifiers follow the pattern “dyn 001”, “dyn 002”, and so on.
- Skill Discovery: WebShop distillation requests 10-15 general skills covering search, product selection, configuration, constraint verification, navigation, and price handling.ALFWorld distillation requests 8-12 concise, actionable, transferable, and failure-aware skills focused on navigation, manipulation, state tracking, recovery, and containers.
- Synthetic Trajectory Generation: Synthetic-trajectory prompts condition teacher-generated ALFWorld and WebShop demonstrations on relevant skills and require complete successful trajectories.The prompts emphasize realistic actions, planning, systematic exploration, search strategies, and product evaluation.
B.1. Hyperparameters
The supplied appendix identifies Table 4 as the hyperparameter table for SKILLRL.
- Hyperparameters: Table 4 presents the hyperparameters used for SKILLRL.The supplied passage provides the table’s label but no individual parameter values.
- Hyperparameters: The passage does not report performance results or compute requirements for the hyperparameters table.Its stated role is limited to identifying Table 4.
B.2. Compute Resources
Experiments used 8 NVIDIA H100 80GB GPUs, with approximately 30 hours of wall-clock time per experiment. The listed stages include trajectory collection, skill distillation, cold-start SFT, and RL training.
- Compute Resources: 8 NVIDIA H100 80GB GPUs were used for all experiments.The compute setup is reported as a cluster configuration.
- Training Times: 3 hours were allocated to trajectory collection.This is the first listed training stage.
- Training Times: 0.5 hours were allocated to skill distillation.The distillation stage is listed separately from trajectory collection and model training.
- Training Times: 2 hour was allocated to cold-start SFT.The supplied passage reports the duration using the singular form “hour.”
- Training Times: 24 hours were allocated to RL training.RL training is the longest individually listed stage.
- Training Times: Approximately 30 hours of wall-clock time were required per experiment.This is the reported total across the experiment workflow.
C. Illustration of Skill Library
The section illustrates SkillRL’s distilled skill library through WebShop navigation skills and categorized failure cases. These examples emphasize reusable strategies for product search, constraint satisfaction, and diagnosing common shopping mistakes.
- The catalog includes general exploration, state-management, and goal-tracking principles that provide guidance transferable across task categories.
- Table 7 presents distilled WebShop skills as strategic patterns for large-scale product search and constraint satisfaction.
- Table 8 categorizes common failures in web-based shopping tasks for structured error analysis.
D. Additional Cases
The additional cases show skill-guided agents completing WebShop purchases and household tasks by combining constraint verification, early mismatch handling, phase ordering, target-first sequencing, and spatial co-location.
- WebShop Footwear: The footwear case purchased black, size-10, rubber-sole, slip-resistant work shoes for $38.99, satisfying the stated constraints.The agent searched with key attributes, verified the description and selected variant, then confirmed the updated price before purchase.
- Cross-case pattern: Across the cases, the demonstrated skills emphasize verifying constraints, aborting mismatched candidates, ordering actions by phase, and using spatial or object-location priors.The cases explicitly associate successful outcomes with verification, abort-fast behavior, decomposition, sequence control, object pairing, and spatial priors.
- Electronics: The speaker case handled a missing black-color option by abandoning the first candidate and selecting a waterproof black speaker priced at $19.99.The agent verified the mismatch early, retried with the next candidate, and confirmed black availability, IPX5 waterproofing, and the price cap.
- Cleaning: The lettuce task succeeded in 7 steps through a locate-clean-place plan, using fridge and sink location priors before immediate countertop placement.The trajectory opened the fridge, took the lettuce, cleaned it at the sink, and placed it on the countertop.
- Heating: The egg task succeeded in 8 steps by securing the exact target first, opening the microwave before heating, and placing the heated egg directly on the countertop.The agent systematically searched surfaces, took the egg, controlled the appliance sequence, and avoided detours after heating.
- Looking: The pencil-and-desklamp task achieved the optimal 3-step solution by prioritizing a desk where the target and tool were co-located.The agent took the pencil first and toggled the lamp once to satisfy the under-light condition.