Source-linked AI summary
SkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolution
Zhiyuan Yao, Yuxin Chen, Zhengxi Lu, Zishan Xu, Yueqing Sun, Yifu Guo, Yuquan Lu, Zhengzhou Cai, Kangning Zhang, Zhuowen Han, Zi-Han Wang, Ziang Ye, Qi Gu, Xunliang Cai, Weiwen Liu, Yongliang Shen
TL;DR
LLM agents often treat related tasks independently, limiting reuse of experience and transferable skills. SkillRise trains one policy to solve tasks and curate an evolving skill document across task sequences, achieving the strongest overall performance across three benchmarks.
Problem
Standard agentic RL treats tasks independently, while existing skill-learning methods entangle extraction, retrieval, and execution, leaving cross-task skill reuse difficult to evaluate.
Method
SkillRise uses one policy to solve related tasks, curate an evolving skill document, and apply decoupled credit assignment across task-solving and downstream skill transfer.
Results
Across ALFWorld, WebShop, and ScienceWorld, SkillRise achieves the strongest overall Pass@1 performance, beating the strongest baseline by 2.3, 7.1, and 8.5 percentage points, respectively.
Takeaways & Limitations
SkillRise learns transferable cross-task self-improvement, with performance increasing across longer related-task sequences and strong effectiveness relative to multi-stage skill-learning pipelines.
Takeaways & Limitations
The formulation assumes task-family metadata to construct related-task sequences, and its generality beyond three text-based benchmarks with verifiable rewards remains unestablished.
Abstract
from arXiv · showhide
Large language model agents often encounter related yet distinct tasks that share reusable solution patterns. Yet standard agentic reinforcement learning treats tasks as independent episodes, while existing approaches to skill learning either focus on repeated attempts of one task or use pipelines with multiple stages that entangle extraction, retrieval, and execution. We introduce SkillRise, a unified reinforcement learning framework for learning skills across tasks. SkillRise organizes related instances into progressively challenging sequences and uses a single policy to alternate between task solving and curating an evolving skill document passed directly to the next task. Decoupled credit assignment across tasks supervises solving with the current task outcome and curation with discounted downstream outcomes. Experiments on ALFWorld, WebShop, and ScienceWorld show that SkillRise achieves the strongest Pass@1 performance among the compared methods, with gains over the strongest baseline ranging from 2.3 to 8.5 percentage points. Although trained across distinct tasks, its learned curation policy remains effective for repeated attempts on the same task. Further analysis reveals scaling at test time across tasks: performance improves with longer sequences of related tasks even when each task is attempted only once. This trend suggests that SkillRise reuses transferable skills across tasks rather than benefiting from repeated sampling of the same task. SkillRise further retains strong performance while substantially reducing the runtime overhead of skill learning pipelines with multiple stages. Together, these results provide a simple and efficient training paradigm for LLM agents to extract, refine, and reuse transferable skills across tasks.
1 INTRODUCTION
SkillRise addresses the limitations of independent-episode reinforcement learning and multi-stage skill pipelines with an end-to-end framework for transferring skills across related but distinct tasks. It uses one policy to solve tasks, curate an evolving skill document, and support later-task performance through cross-task credit assignment.
- Standard agentic RL treats tasks as independent episodes, discarding interaction experience despite recurring solution patterns across related tasks.
- Existing skill-learning approaches either repeatedly attempt one task or maintain external skill banks, making knowledge instance-specific or coupling extraction, retrieval, and execution.
- SkillRise organizes similar yet distinct instances into progressively harder sequences so earlier-task experience can support later tasks.
- A single policy alternates between solving each task and curating an evolving skill document that is directly passed to the next task.
- Across ALFWorld, WebShop, and ScienceWorld, SkillRise outperforms prompting and standard reinforcement-learning baselines while demonstrating skill transfer, within-task generalization, and cross-task test-time scaling.
2 PRELIMINARIES
The preliminaries model an LLM agent as a policy interacting with an environment to solve sampled task instructions and evaluate completed trajectories with verifiable rewards. Standard reinforcement learning maximizes expected task reward independently for each task, motivating a later extension to related task sequences connected by an evolving skill state.
- Agent–environment interaction: An LLM agent samples a task instruction x from task distribution D and selects actions from A based on observations and interaction history.The policy is denoted πθ, and the history is ht = (o0, a0, …, at−1, ot).
- Trajectory evaluation: An episode yields a trajectory τ and terminates when the task is completed or the interaction budget is exhausted.The trajectory is evaluated by a verifiable outcome reward R(τ).
- Independent-task objective: The standard reinforcement learning objective maximizes expected task reward under the task distribution and policy-induced trajectories.This objective treats each task independently.
- Independent-task objective: The framework is subsequently extended to sequences of related tasks connected through an evolving skill state.This extension addresses the independent-task formulation introduced by the standard objective.
3 METHOD
SkillRise sequences related task instances by increasing difficulty and uses one policy to alternate between solving tasks and curating a transferable skill document. Separate learning signals evaluate immediate task performance and downstream skill transfer while jointly optimizing both roles.
- Sequential rollout: A single policy alternates between solving each task with the current document and curating a revised document for the next task.The document begins empty, and only the revised document—not earlier trajectories—is passed forward.
- Sequence construction: SkillRise orders related instances into progressively challenging sequences so skills from earlier tasks can guide later tasks and later performance measures transfer.Sequences contain distinct instances from the same task family that share interaction routines despite differing entities and goals.
- Skill curation: During curation, the policy preserves useful skills, consolidates successful procedures and failure modes, and removes instance-specific details.The evolving document is the sole information channel across tasks, with its utility reflected in subsequent-task performance.
- Credit assignment: Decoupled credit assignment supervises solving with the current task reward and curation with discounted outcomes from later tasks.The cross-task discount factor γ ∈[0, 1] places more credit on nearby tasks directly affected by the revised document.
- Policy optimization: Role-aware group-relative optimization separates solving and curation baselines while jointly optimizing responses through shared policy parameters.Phase-level advantages apply to all policy-generated tokens in the corresponding phase, and clipping controls the update range.
4 EXPERIMENTS
Across ALFWorld, WebShop, and ScienceWorld, SkillRise achieves the strongest overall Pass@1 performance and generalizes to within-task adaptation. Its advantages are evaluated against prompting, task-independent RL, and Meta-RL baselines under matched training-play budgets.
- Baselines: The experiments compare SkillRise with Zero-shot, ReAct, Reflexion, PPO, RLOO, GRPO, GiGPO, and LaMer.LaMer is a Meta-RL baseline for repeated attempts of the same task, while the other listed methods provide prompting or task-independent RL comparisons.
- Implementation Details: All trainable methods process 384 task plays per update, aligning SkillRise’s 16 sequences of 3 tasks and 8 trials with the comparison methods’ rollout budgets.SkillRise uses K = 3 tasks and N = 8 independent trials per sequence.
- Overall Performance: SkillRise achieves the strongest overall Pass@1 performance on all three benchmarks.It scores 85.9% on ALFWorld, 84.4% on WebShop, and 54.6% on ScienceWorld.
- Overall Performance: 2.3, 7.1, and 8.5 percentage points are SkillRise’s gains over GiGPO on ALFWorld, WebShop, and ScienceWorld, respectively.On ALFWorld, SkillRise also ranks first or second in five of six task families.
- Generalization to Within-Task Adaptation: SkillRise achieves the best Pass@2 and Pass@3 on all three benchmarks when retrying held-out tasks with the evolving skill document.This evaluates whether skills curated across distinct training tasks support within-task adaptation.
5 ANALYSIS
SkillRise benefits from longer cross-task context and larger backbones, while ablations show robustness to discounting and the importance of skill curation. Its compact end-to-end design also matches RetroAgent’s ALFWorld success rate while exceeding SkillRL.
- Cross-task test-time scaling: SkillRise’s ALFWorld performance rises monotonically from 83.6% at K = 2 to 87.5% at K = 6 when each task is attempted once.The same 128 held-out tasks are partitioned into related sequences, with the skill document carried forward and updated.
- Model scaling: At Qwen3-1.7B and Qwen3-4B, SkillRise achieves 78.1% and 85.9% Pass@1, exceeding the strongest baseline by 3.1 and 6.2 points, respectively.The advantage is present at the smaller scale and becomes more pronounced with the larger backbone.
- Model scaling: SkillRise gains 7.8 points from 1.7B to 4B, compared with 4.7 points for GRPO and 3.3 points for LaMer.The larger improvement is attributed to capacity supporting task solving, cross-task regularity identification, and reusable skill-document curation.
- Ablations: Across γ ∈ {0.3, 0.4, 0.6, 0.7}, variants follow aligned training trajectories and converge within approximately one percentage point.This indicates that SkillRise’s gains do not depend on carefully tuning the cross-task discount factor.
- Ablations: SkillRise separates from no-curation after sufficient cross-task interactions and maintains the strongest training score thereafter, confirming the importance of skill curation.The no-curation variant uses the same K = 3 sequences and environment interaction budget but prevents inter-task skill transfer.
- Pipeline efficiency: SkillRise achieves an 85.9% average success rate on ALFWorld, matching RetroAgent and outperforming SkillRL by 12.5 percentage points.Its single policy solves tasks, curates one sequence-local skill document, and uses it on the next task without an external teacher, separate memory maintenance, or retrieval module.
6 RELATED WORK
Related work frames SkillRise within agentic reinforcement learning and experience-based skill learning for LLM agents. Prior research addresses long-horizon credit assignment and exploration, while skill-learning methods transform interaction histories into reusable knowledge through concrete or abstract representations.
- Reinforcement Learning for LLM Agents: Agentic reinforcement learning extends reinforcement learning from single-turn generation to multi-turn decision making, where long horizons and delayed outcomes make exploration and credit assignment harder.
- Reinforcement Learning for LLM Agents: Existing agentic RL methods develop hierarchical, step-level, or milestone-guided credit assignment and improve exploration through autonomous interaction and online curricula.
- Experience and Skill Learning in LLM Agents: Experience-based self-evolving agents transform past interaction histories into reusable knowledge, using trajectories, examples, verbal reflections, guidelines, workflows, or reasoning strategies.
7 CONCLUSION
SkillRise is an end-to-end reinforcement learning framework for learning skills across related but distinct tasks. It uses ordered task sequences, alternates solving with skill-document curation, and decouples credit assignment by temporal role.
- SkillRise provides an end-to-end reinforcement learning framework for cross-task skill learning.
- The framework organizes related yet distinct tasks into ordered sequences for cross-task skill learning.
- A single policy alternates between solving tasks and curating an evolving skill document.
- Decoupled credit assignment uses current-task outcomes for solving and downstream outcomes for skill curation.
8 LIMITATION
SkillRise is limited by its reliance on task-family metadata, evaluation only up to 4B-parameter models, and testing on three text-based benchmarks with verifiable outcome rewards.
- Limitations: The formulation assumes task-family metadata, while automatically discovering related instances in open-ended task streams remains future work.This limits sequence construction to settings where task-family relationships are already available.
- Limitations: Experiments evaluate models up to 4B parameters because of computational constraints, leaving larger-scale evaluation for future work.The reported model-scale conclusions therefore do not extend to models above 4B parameters.
- Limitations: Experiments cover three text-based agent benchmarks and use verifiable outcome rewards.The passage identifies this benchmark and reward scope as an additional limitation.
A PROMPTS
SkillRise uses runtime-filled prompts to alternate task solving with skill-document curation across related tasks. Its curation prompts revise prior skills using the latest trajectory and outcome, with task-family-specific formulations across ALFWorld, WebShop, and ScienceWorld.
- Prompt initialization: At each task-sequence start, SkillRise initializes the skill document with an explicit empty-document message.The document is passed through runtime-filled user-message templates.
- Task-solving prompts: Skill-solving prompts are provided separately for ALFWorld, WebShop, and ScienceWorld.These prompts are shown in Figures 5–7.
- Skill-curation prompts: Curation prompts instruct the agent to revise an old skill document using the most recent task trajectory and outcome to improve later tasks in the same family.The formulation is instantiated for household tasks, same-category shopping tasks, and ScienceWorld task families.
- Example of skill refinement: After a failure placing one of two required objects, curation records a repeat-for-each-instance workflow rule, and the next task successfully places both instances of a different object.The rule extends the workflow to repeat locating, taking, and placing steps for each target instance.
B CASE STUDY
The case studies show SkillRise transferring failure-derived lessons across tasks and incrementally revising a reusable skill document after successful task execution. In ALFWorld rollouts, the evolving document—not earlier trajectories—is carried between tasks.
- Case-study setup: In ALFWorld rollouts with sequence length K = 3, each rollout starts with an empty skill document and carries only the updated document between tasks.Displayed tasks are numbered from 1, and earlier-task trajectories are not forwarded.
- Case 1: distilling a lesson from failure: After failing to place both target instances, the agent curates a general instruction to repeat the procedure for each instance, enabling success on a different object in the next task.The first task fails after placing only one spraybottle; the next task places both peppershaker instances successfully.
- Case 2: incremental skill evolution: A successful cellphone-placement task produces skill v1 containing a five-step workflow for navigating, identifying, picking up, navigating, and placing the target object.The workflow is expressed using generic object and receptacle terms.
- Case 2: incremental skill evolution: After succeeding on a task requiring a closed microwave, curation adds an “open if closed” condition to the existing placement workflow, producing skill v2.The case documents the revision itself rather than downstream use of the new condition.