Source-linked AI summary
Reinforcement Learning for Self-Improving Agent with Skill Library
Jiongxiao Wang, Qiaojing Yan, Yawei Wang, Yijun Tian, Soumya Smruti Mishra, Zhichao Xu, Megha Gandhi, Panpan Xu, Lin Lee Cheong
TL;DR
LLM-based agents struggle to improve continuously in new environments, while prompt-driven skill libraries can limit skill quality and adaptability. The paper introduces SAGE, which integrates skill libraries into RL through Sequential Rollout and Skill-integrated Reward. On AppWorld, SAGE outperforms baselines in accuracy and efficiency, including 8.9% higher Scenario Goal Completion, 26% fewer interaction steps, and 59% fewer generated tokens.
Problem
Existing skill library approaches rely primarily on prompting, which limits skill quality and adaptability, while continuous improvement in new environments remains challenging.
Method
SAGE extends GRPO with Sequential Rollout across similar-task chains and Skill-integrated Reward for skill generation and utilization.
Results
8.9% higher Scenario Goal Completion, 26% fewer interaction steps, and 59% fewer generated tokens were achieved on AppWorld versus baseline training with GRPO.
Takeaways & Limitations
SAGE enables skill library agents to outperform baselines in both performance and efficiency on AppWorld.
Takeaways & Limitations
Evaluation was conducted exclusively on AppWorld, and different scenarios may require different agent designs.
Abstract
from arXiv · showhide
Large Language Model (LLM)-based agents have demonstrated remarkable capabilities in complex reasoning and multi-turn interactions but struggle to continuously improve and adapt when deployed in new environments. One promising approach is implementing skill libraries that allow agents to learn, validate, and apply new skills. However, current skill library approaches rely primarily on LLM prompting, making consistent skill library implementation challenging. To overcome these challenges, we propose a Reinforcement Learning (RL)-based approach to enhance agents' self-improvement capabilities with a skill library. Specifically, we introduce Skill Augmented GRPO for self-Evolution (SAGE), a novel RL framework that systematically incorporates skills into learning. The framework's key component, Sequential Rollout, iteratively deploys agents across a chain of similar tasks for each rollout. As agents navigate through the task chain, skills generated from previous tasks accumulate in the library and become available for subsequent tasks. Additionally, the framework enhances skill generation and utilization through a Skill-integrated Reward that complements the original outcome-based rewards. Experimental results on AppWorld demonstrate that SAGE, when applied to supervised-finetuned model with expert experience, achieves 8.9% higher Scenario Goal Completion while requiring 26% fewer interaction steps and generating 59% fewer tokens, substantially outperforming existing approaches in both accuracy and efficiency.
1 Introduction
LLM-based agents can perform complex interactive tasks but struggle to improve continuously in new environments. The paper addresses this gap with an RL-trained skill library agent and evaluates SAGE on AppWorld.
- RL-trained agents still face limitations in continuous self-improvement, especially when deployed in new environments.
- Prompt-based skill libraries are constrained by base-model instruction following, limiting skill quality and adaptability.
- SAGE extends GRPO with Sequential Rollout and Skill-integrated Reward for skill library agents.Sequential Rollout preserves skills across chains of similar tasks, while Skill-integrated Reward combines outcome-based and skill-quality or utilization rewards.
- SAGE trains agents across chains of similar tasks so skills generated earlier remain available for later tasks.
- AppWorld evaluates transfer across three similar tasks per scenario using Scenario Goal Completion.The benchmark involves API documentation lookup, API calls, and logical programming constructs.
- 72.0% Task Goal Completion and 60.7% Scenario Goal Completion were achieved on Test Normal with 12.1 interaction steps and 1,475 generated tokens.These results are reported for SAGE applied to Qwen2.5-32B-Instruct.
2 Related Work
Related work develops LLM agents for interactive environments and applies RL to improve their behavior. However, continuous self-improvement after deployment, especially in new environments, remains challenging.
- LLM-agent research has developed reason-then-act pipelines and other frameworks for interacting with external environments.
- RL has supported agent self-improvement through rollout experiences, but continuous improvement after deployment remains difficult.The challenge is especially pronounced in new environments.
3 Method
SAGE extends GRPO for skill-library agents by combining sequential task-chain rollouts with rewards that encourage skill generation and reuse. Its agent unifies task solving and skill generation, allowing skills learned earlier in a chain to support later tasks.
- 3.1 Skill Library Agent: Unified task solving and skill generation address the context-length and learning-consistency limitations of defining skills only after complete task trajectories.Existing approaches may extend long-horizon contexts and separate execution from skill generation, creating inconsistency for RL.
- 3.1 Skill Library Agent: The skill-library agent performs online learning by retrieving existing skills, using them, and generating new skill functions during task execution.Skills are generated as code functions and called to process tasks rather than invoking multiple APIs directly.
- 3 Method: SAGE extends GRPO with Sequential Rollout and Skill-integrated Reward for training skill-library agents.The framework is designed to optimize both task outcomes and the development and use of reusable skills.
- 3.2.2 Sequential Rollout: Sequential Rollout processes a chain of tasks so skills generated in earlier tasks accumulate in the library and can be used by subsequent tasks.The paper uses two-example task chains for simplicity because longer chains would significantly increase training costs.
- 3.2.3 Skill-integrated Reward: Skill-integrated Reward adds incentives for generating a skill in the first task and using it in the second, alongside verifiable outcome rewards.For a two-task chain, the first task’s generated skills are directly available to the second task, and successful skill use contributes to the rewards.
- 3.2.4 SAGE: SAGE optimizes outputs across task chains with advantages computed from Skill-integrated Rewards, while each chain position uses its corresponding skill library.The first library is empty, and later-task generations can use skills accumulated from earlier tasks; environment observations are masked when counting generated outputs.
4 Experiments
Experiments on AppWorld evaluate SAGE across task performance, skill usage, retrieval, reward design, and initialization. SAGE improves scenario-level completion and efficiency, while ablations show that skill use, retrieval choices, reward design, and expert-data initialization materially affect outcomes.
- Experimental Setup: AppWorld contains 750 tasks across 250 scenarios, with three similar tasks per scenario, making it suitable for Sequential Rollout.Scenario Goal Completion measures the proportion of scenarios in which all three tasks succeed.
- Main Results: SAGE improves Scenario Goal Completion by 8.9% over baseline GRPO on the test normal dataset while reducing generated tokens by 59%.The reported gains are attributed to transferring and reusing skills across related tasks, with lower interaction steps also reported.
- Main Results: SAGE ultimately outperforms prompting-based, SFT, and baseline GRPO approaches after reinforcement learning further improves the SFT-trained model.Prompting-based skill-library use performs worse than the training-free ReAct baseline, while SFT improves performance but does not surpass GRPO without a skill library.
- Skill Usage Analysis: SAGE increases both Skill Usage Rate and Success Skill Usage Rate relative to earlier stages, whereas the Base Model generates more skills but uses them less effectively.The SFT model surpasses the Base Model specifically in Success Skill Usage Rate.
- Ablation Studies: Using skills improves SGC, average steps, and average tokens for all evaluated agent models compared with empty skill libraries, although Skill Library Agent TGC declines.The reported TGC decline may reflect limited skill-utilization proficiency and inappropriate skill applications causing task failures.
- Ablation Studies: Retrieval method selection significantly affects performance, while Skill-integrated Reward achieves superior TGC and SGC compared with alternative reward designs.Skill-integrated Reward yields the lowest average steps but relatively high average tokens, reflecting a steps-versus-generation trade-off.
- Ablation Studies: SAGE initialized with SFT on expert experience data significantly outperforms other initialization methods, while direct Base Model training performs notably poorly.Among methods without extra data, RL Warm-Up performs better than Self-Distillation.
5 Conclusion
The paper introduces SAGE, an RL framework that combines GRPO with Sequential Rollout and Skill-integrated Reward for self-improving agents with skill libraries. On AppWorld, SAGE improves both agent performance and efficiency over baselines.
- SAGE incorporates GRPO with Sequential Rollout and Skill-integrated Reward for self-improving agents with skill libraries.
- On AppWorld, SAGE enables skill library agents to significantly outperform baselines in both performance and efficiency.
Limitations
The evaluation is limited to the AppWorld dataset, and the paper notes that different scenarios may require different agent designs. The interaction examples contrast direct code generation with defining and calling a function.
- Experiments are conducted exclusively on the AppWorld dataset.The authors chose AppWorld because its simulated environment more closely resembles real-world application scenarios.
- Different scenarios may require different agent designs even when applying similar skill library approaches.The authors plan to extend evaluation to other tool-using agent datasets.
- The baseline directly generates code, whereas the skill library agent defines a function and then calls it to process the task.
B SAGE with Longer Task Chain in Sequential Rollout
SAGE’s Sequential Rollout processes chains of related tasks so skills accumulate across tasks, while Skill-integrated Reward credits successful skill generation and usage. Longer chains did not necessarily improve performance and increased computational costs.
- Sequential Rollout: Sequential Rollout processes tasks from q1 to qK while accumulating generated skills for subsequent tasks.The general format samples K examples and collects trajectories by sequentially processing the task chain.
- Reward boundaries: The first task begins with an empty skill library, while the final task’s generated skills cannot be verified by subsequent tasks.
- Skill-integrated Reward: Skill-integrated Reward supplements outcome rewards with rewards for successful skill generation and effective skill usage.Skill generation is rewarded when later use contributes to task success; skill usage is rewarded when acquired skills help complete the current task.
- Optimization: The current policy is optimized with a skill-library-integrated GRPO objective using advantages computed from Skill-integrated Reward.
- Longer task chains: Longer task chains do not necessarily improve performance and substantially increase computational costs as the number of tasks grows.The paper attributes this to reward distribution imbalance and potentially larger gradient variance, ultimately choosing a two-example chain.
C AppWorld Dataset
AppWorld evaluates tool-using agents on diverse everyday digital tasks in a simulated environment with scenario-level structure that supports Sequential Rollout. The paper uses this benchmark with expert-experience fine-tuning and specified rollout constraints.
- Environment: Agents complete everyday digital tasks by consulting API documentation and executing API calls through generated code across nine simulated applications.
- Dataset: AppWorld contains 750 challenging tasks across Train, Dev, Test-Normal, and Test-Challenge splits.Test-Challenge includes unfamiliar Amazon and Gmail APIs absent from the other splits, evaluating generalization to unfamiliar APIs.
- Evaluation: AppWorld’s manually written programs evaluate final environment states and produce completion rates from 0 to 1 as outcome-based rewards.Stage-based tests accommodate multiple solution paths.
- Evaluation: Scenario Goal Completion measures the success rate of scenarios in which all three similar tasks are completed successfully.The scenario-based structure comprises 250 scenarios, each containing three tasks with similar instructions under different simulated users.
- Training setup: The training framework uses strictly on-policy reinforcement learning and full-parameter fine-tuning, with 288 rollouts per training step.Rollouts use 36 sampled examples with G = 8 agents per group.
- Training setup: Rollouts allow up to 40 interaction turns and 1,500 output tokens per turn, with early stopping under specified collection conditions.
- Expert experience: Expert-experience data is collected through rejection sampling and yields 1,129 valid examples for full-parameter supervised fine-tuning.
G Training Details of SAGE
SAGE training uses scenario-based task chains to support Sequential Rollout, with training configured around AppWorld scenarios and rollout batches. Training progress is tracked through SAGE’s training curve and SGC/TGC scores.
- Dataset and sampling: AppWorld training samples two tasks from selected scenarios to construct the task chains required by Sequential Rollout.The sampling strategy first selects scenarios, then samples tasks within each selected scenario.
- Training settings: The mini-batch size for policy-gradient updates is aligned with the rollout batch size of 384.Most training parameters remain consistent with baseline GRPO, while the mini-batch size is changed for the Sequential Rollout setup.
- Rollout settings: 384 rollouts are obtained per training step using 24 scenarios, 48 sampled tasks, and group size G = 8.The 48 tasks result from sampling two tasks from each of the 24 training scenarios.
- Training progress: Figure 4 presents the training curve of SAGE.The figure is identified as the training curve for SAGE.
- Training progress: Figure 5 reports SGC and TGC scores on the development set at every five training steps.The caption specifies the evaluation metrics, split, and reporting interval.
H Task Execution Examples with Different Models
Execution examples compare baseline GRPO with the Skill Library Agent, SFT, and SAGE under comparable scenarios. They show progressively fewer basic errors and more efficient skill-based execution, with SAGE reducing redundant API interactions.
- Comparison setup: The examples compare baseline GRPO, the Skill Library Agent, SFT, and SAGE using summarized trajectories from AppWorld tasks.Two examples are presented for each agent model under the same scenario.
- Action analysis: Figure 6 explains action categories including code execution failures and anti-patterns that lead to task failures.The caption defines code execution failures and anti-patterns as the figure’s analyzed action categories.
- Skill Library Agent: The untrained Skill Library Agent often needs multiple attempts to define executable functions and may still produce errors, simulated data, or repetitive behavior.These examples indicate poor task execution before training despite access to a skill library.
- SFT: SFT reduces interaction steps and errors in skill generation and usage, but the model still struggles to complete tasks successfully.Expert-experience patterns minimize basic errors without resolving successful task completion.
- SAGE: SAGE improves efficiency over baseline GRPO by eliminating step-by-step API calls and reducing redundant actions in similar tasks.The cited execution analysis attributes the efficiency improvement to skill-based execution patterns.
I.2 Further Analysis
Further analysis compares retrieval strategies and reward designs for skill-library agents. Query N-gram approximates scenario-aware retrieval, while broader semantic retrieval improves efficiency but can weaken accuracy across scenarios.
- Retrieval methods: Query N-gram performs most similarly to the ideal Same Scenario retrieval method because tasks within scenarios share query structure.Some same-scenario tasks even use identical queries under different simulated users.
- Retrieval methods: Query Embedding yields slightly lower TGC and SGC but fewer average steps and tokens than the scenario-focused approach.Semantic similarity can retrieve structurally different queries, broadening skill usage across scenarios.
- Retrieval methods: Broader skill usage can reduce interaction steps and tokens without improving accuracy because cross-scenario skill adaptation is difficult.The analysis links the efficiency–accuracy trade-off to the difficulty of adapting skills across scenarios.
- Retrieval methods: Skill Embedding performs worse than the other retrieval approaches, possibly because useful tool functions are difficult to retrieve from queries.The analysis suggests skill- or tool-specific retrievers as future work.
- Reward designs: Outcome-based Reward uses task completion rates as rewards, whereas Chain-based Reward adds a bonus when all tasks in the chain succeed.Outcome-based rewards are formulated from r1 and r2, while Chain-based Reward additionally tracks complete-chain success.
J.2 Skill Library Usage Analysis
The skill-usage analysis examines alternative reward designs and initialization settings alongside execution examples. Skill-integrated Reward increases skill-use frequency, while skill usage trades interaction steps against token cost and SFT initialization can constrain later training behavior.
- Reward comparison: Figure 7 compares skill-usage patterns under different reward designs using ratios relative to Skill-integrated Reward.The figure annotates numerical values for the relative performance metrics.
- Reward comparison: Skill-integrated Reward produces substantially more skill-usage behavior, while successful skill application is achievable across reward designs.The analysis distinguishes skill-use frequency from the broader ability to apply skills successfully.
- Efficiency trade-off: Outcome-based and Chain-based Rewards use smaller skill libraries and fewer skills, resulting in more average steps but fewer average tokens than Skill-integrated Reward.Skill usage typically reduces interaction steps, whereas generating skill functions generally requires more tokens.
- Reward comparison: Chain-based Reward shows lower skill-usage behavior than Outcome-based Reward, possibly because early successful completion reinforces patterns that exclude skill usage.The passage presents this as a possible explanation rather than a definitive causal finding.
- Initialization analysis: SFT-initialized baseline GRPO can generate fewer tokens, but the model may remain constrained by patterns acquired during SFT.The comparison includes original baseline GRPO, SFT-initialized baseline GRPO, and SAGE, with only SAGE using the skill library during evaluation.
- Execution examples: Figures 10–13 present task-execution examples for baseline GRPO, the Skill Library Agent, SFT, and SAGE.The examples are used to compare models across stages of the proposed approach.