PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning

Junhao Shen, Teng Zhang, Xiaoyan Zhao, Hong Cheng

arXiv:2605.10923v2cs.LGcs.CL

TL;DR

Existing skill-based agentic RL methods typically assume external skills either accumulate indefinitely or disappear through internalization, leaving the evolution of the active skill set under limited capacity unresolved. SLIM dynamically retains, retires, or expands skills during policy learning, outperforming the best baselines by an average of 7.1% points across ALFWorld and SearchQA.

  • Problem

    Existing methods assume external skills should either keep accumulating or eventually vanish, leaving how the active skill set should evolve under finite capacity and uneven contributions unresolved.

  • Method

    SLIM jointly optimizes policy learning and a task-conditioned active skill set using leave-one-skill-out contribution estimates to retain, retire, or expand skills.

  • Results

    7.1% points average improvement over the best baselines across ALFWorld and SearchQA, with SLIM achieving the strongest overall performance.

  • Takeaways & Limitations

    The learned endpoint is non-monotonic and compact but non-empty, indicating that some capabilities are internalized while others remain externally useful.

  • Takeaways & Limitations

    SLIM assumes a hierarchical external skill library and finite parametric capacity when allocating capabilities among active, internalized, and inactive sets.

Abstract

from arXiv · show

Large language model agents increasingly rely on external skills to solve complex tasks, where skills act as modular units that extend their capabilities beyond what parametric memory alone supports. Existing methods assume external skills either accumulate as persistent guidance or internalized into the policy, eventually leading to zero-skill inference. We argue this assumption is overly restrictive, since with limited parametric capacity and uneven marginal contribution across skills, the optimal active skill set is non-monotonic, task- and stage-dependent. In this work, we propose SLIM, a framework of dynamic Skill LIfecycle Management for agentic reinforcement learning (RL), which treats the active external skill set as a dynamic optimization variable jointly updated with policy learning. Specifically, SLIM estimates each active skill's marginal external contribution through leave-one-skill-out validation, then applies three lifecycle operations: retaining high-value skills, retiring skills whose contribution becomes negligible after sufficient exposure, and expanding the skill bank when persistent failures reveal missing capability coverage. Experiments show that SLIM outperforms the best baselines by an average of 7.1% points across ALFWorld and SearchQA. Results further indicate that policy learning and external skill retention are not mutually exclusive: some skills are absorbed into the policy, while others continue to provide external value, supporting SLIM as a more general paradigm for skill-based agentic RL.

1 Introduction

SLIM reframes skill-based agentic RL as dynamic management of an active external skill set rather than monotonic skill accumulation or elimination. It retains, retires, or expands skills using marginal-contribution estimates and outperforms baselines across ALFWorld and SearchQA.

  • 1 Introduction: Existing methods either continuously expand persistent skill banks or remove temporary skills toward zero-skill inference.These paradigms respectively treat skills as ongoing augmentation or scaffolds whose benefits are transferred into model parameters.
  • 1 Introduction: Finite parametric storage makes external skills useful for preserving narrow, low-frequency, or long-tail procedures that may be costly to encode.The passage also notes that retaining too many skills creates a competing concern, motivating selective management.
  • 1 Introduction: SLIM treats the task-conditioned active skill set as a dynamic optimization variable and estimates each active skill’s marginal contribution through leave-one-skill-out validation.It retrieves hierarchical skills from the active pool and couples these signals with reinforcement-learning training.
  • 1 Introduction: SLIM uses marginal-contribution estimates to retain valuable skills, retire negligible ones, and expand the skill bank when missing capability coverage is revealed.This lifecycle management replaces the assumption that the external skill set must monotonically grow or vanish.
  • 1 Introduction: 7.1% points is SLIM’s average improvement over the best baselines across ALFWorld and SearchQA.The comparison includes GRPO, SkillRL, and Skill0.

2 Related Work

Prior work establishes LLM agents, agentic reinforcement learning, and external skills as complementary foundations for reliable, compositional behavior. SLIM extends this line by focusing on how the active external skill set should evolve during RL training.

  • Large Language Model Agents: LLM agents use tools, APIs, and embodied interfaces to plan, act, and interact with external environments as sequential decision makers.Progress in tool use, web navigation, computer use, and long-horizon task completion highlights the importance of structured action spaces and external scaffolding.
  • Large Language Model Agents: External memory and skill support improve LLM-agent robustness and compositionality, while this work studies active skill-set evolution during RL training.The related-work framing positions SLIM within research on external scaffolding and skill support, with an emphasis on changing the active set over training.
  • Agentic Reinforcement Learning: Agentic reinforcement learning supports LLM-agent posttraining when interaction, exploration, and delayed credit assignment are required.Recent methods combine policy optimization with structured rewards, preference signals, or group-relative objectives to improve reasoning and action quality.
  • Skill-Based Agents: Skill-based agents organize reusable behavior through external skill banks, reusable prompt modules, and distilled procedural guidance.Related methods keep skills as persistent augmentation, eliminate them toward zero-skill inference, or co-evolve decision and skill-bank agents from rollouts.

3 Preliminaries

The paper models agents as history-conditioned policies interacting with environments over sequential trajectories, and formulates skill use through a dynamically selected active subset of a hierarchical external skill bank. GRPO provides the reinforcement-learning optimizer, while capacity and overhead constraints motivate managing skills beyond simple accumulation or full internalization.

  • LLM Agent: An LLM agent is a policy πθ that maps task and interaction history ht to actions over a trajectory τ of observations and actions.In text-only environments, observations and actions are token sequences generated autoregressively by a causal language model.
  • Group Relative Policy Optimization: GRPO samples a group of trajectories from the behavior policy, assigns scalar outcome rewards, and uses group-relative advantages for policy optimization.Because rewards are outcome-level, the same advantage is applied across all action-generation steps within each trajectory.
  • Skill Bank and Problem Setting: The hierarchical skill bank contains general and task-specific pools, while the agent accesses an active subset At ⊆ S through a skill-conditioned policy.At audit step t, selected external skill s conditions the policy alongside the interaction history.
  • Skill Bank and Problem Setting: Skill allocation balances the overhead of extra active skills against finite policy capacity, allowing removed skills to become internalized, noisy, or obsolete.SLIM retrieves task-conditioned visible skills, estimates marginal contributions with leave-one-skill-out validation, and updates policy and skills through retain–retire–expand operations.

4 Method: SLIM

SLIM makes the active external skill set a dynamic optimization variable by restricting task-conditioned candidates, estimating each skill’s marginal external contribution, and alternating policy updates with lifecycle management. It retains valuable skills, retires redundant skills after sufficient exposure, and expands coverage when persistent failures reveal missing capability.

  • 4.1 Task-conditioned skill selection: SLIM converts global skill selection into task-conditioned candidate selection, reducing the combinatorial active-set search over the full skill bank.It retrieves task-specific skills from the currently active pool, so lifecycle decisions directly determine capabilities exposed to later rollouts.
  • 4.2 Marginal external contribution: SLIM estimates each audited skill’s marginal external contribution through leave-one-skill-out validation on the validation tasks whose rollouts use that skill.The estimate compares performance with and without the skill on its routed validation subset, avoiding exhaustive skill-combination enumeration.
  • 4.2 Marginal external contribution: Lifecycle management uses an exponential moving average of marginal contribution, interpreting positive values as ongoing external benefit and near-zero or negative values as possible absorption, redundancy, or harm.The estimate is local to the current policy, active set, and routing behavior rather than a global attribution across all skill subsets.
  • 4.3 Alternating optimization: SLIM alternates GRPO policy optimization with the active set fixed and skill lifecycle management with the policy fixed, separating continuous policy updates from discrete set operations.Policy learning may reduce reliance on external skills, but MEC determines whether that reliance has actually disappeared.
  • 4.3 Skill lifecycle management: The lifecycle rules retain skills with clearly positive smoothed MEC, retire negligible contributors after stable low contribution and sufficient exposure, and expand the bank after persistent routed failures.The rules use exposure and low-contribution streaks to protect infrequently used skills, while expansion targets task regions inadequately covered by current skills.

5 Implementation

SLIM alternates GRPO policy updates with periodic lifecycle audits that evaluate active skills and apply retain, retire, or expand operations. Its implementation uses bounded auditing, task-specific retrieval, and outcome-level rewards, with K = 3 retrieval and audits every d = 10 GRPO steps.

  • Algorithm: Each GRPO step retrieves active skills, performs skill-conditioned rollouts, and updates the policy.The loop follows the three SLIM components during policy training.
  • Algorithm: Audits run every d = 10 GRPO steps and estimate marginal external contribution before retain, retire, or expand operations.Audits record routed skills, outcomes, and failures, then update lifecycle statistics and apply the lifecycle rules.
  • Algorithm: SLIM limits audit cost by selecting audited skills under a bounded budget, including the top-M skills by recent routed usage.Each audited skill is evaluated with leave-one-skill-out validation using Eq. (4).
  • Algorithm: Up to B task-specific skills are created from routed failure buckets during the lifecycle loop.Expansion is driven by routed failures identified during validation.
  • Training and Inference Settings: Training uses Qwen3-Embedding-0.6B with K = 3 and τemb = 0.45 for task-specific retrieval, alongside GRPO outcome-level rewards.Invalid-action penalties are applied during trajectory collection, while policy-side KL loss and KL-in-reward regularization are disabled in the main SLIM runs.

6 Experiment

Experiments on ALFWorld and SearchQA show that SLIM improves performance over prompt-, memory-, and RL-based baselines by dynamically retaining, retiring, and expanding skills. Ablations and lifecycle analyses indicate that contribution-aware, non-monotonic skill management—not accumulation, fixed budgets, or random perturbations—drives the gains.

  • Overall Comparison: SLIM† reaches 87.5 on ALFWorld, outperforming SkillRL† by 12.5 points and exceeding GRPO, GRPO†, and Skill0.The comparison uses Qwen3-4B across ALFWorld and SearchQA against prompt-based, agent/memory-based, and RL-based baselines.
  • Detailed Analysis: SLIM† reaches 91.4 on Clean and 88.5 on Cool, while SLIM or SLIM† is best or near-best across most SearchQA subsets.The largest ALFWorld gains concentrate on procedural state-transformation tasks, whereas SearchQA improvements are smaller but broadly distributed.
  • Training Dynamics: SLIM’s active set expands from 38 to 46, fluctuates through alternating expansion and retirement, and stabilizes at 21 skills.No-skill performance rises from 29.7% to 84.4%, while with-skill performance peaks at 93.8% and remains 90.6% at the end.
  • Lifecycle Components: Removing retirement lowers ALFWorld success from 87.5 to 73.4, while removing expansion reaches 78.9, showing that pruning and coverage expansion are complementary.Without retirement, expansion degenerates toward accumulation; without expansion, pruning cannot repair under-covered task regions.
  • Ablations: Random Audit obtains only 68.8, and Fixed Active Set Size reaches 75.6, showing that contribution-aware selection matters beyond perturbation or prompt-budget control.Fixed Active Set Size remains 11.9 points below SLIM, indicating that which skills remain active matters, not only their number.
  • Lifecycle Analysis: Lifecycle decisions follow marginal external contribution rather than selection frequency: disabling pic_002 and gen_011 causes 0.062 and 0.080 drops, while gen_004 causes a 0.284 drop.The diagnostic probe identifies internalized skills only analytically, using frequent selection, near-zero MEC, and a small disabling-related validation drop.

7 Conclusion and Future Work

SLIM presents dynamic skill lifecycle management for agentic reinforcement learning, treating the active external skill set as an optimization variable updated alongside policy learning. This avoids assuming indefinite skill accumulation or forcing zero-skill inference.

  • Conclusion and Future Work: SLIM frames external skill lifecycle management as a dynamic optimization problem in agentic reinforcement learning.The active external skill set is updated jointly with policy learning.
  • Conclusion and Future Work: The framework rejects both indefinite accumulation of external skills and forced convergence to zero-skill inference.Its learned endpoint can avoid persistent full accumulation as well as forced zero-skill inference.
  • Conclusion and Future Work: SLIM evaluates dynamic external-skill management across ALFWorld and SearchQA.The supplied conclusion introduces these two evaluation settings but does not report the corresponding results in the passage.

A Theoretical Analysis

The theoretical analysis gives local, conservative sufficient conditions for SLIM’s lifecycle decisions rather than proving global optimization. Under bounded retrieval, validation, and operating-cost assumptions, retaining, retiring, and expanding skills can preserve or improve the local objective, while patience and margins limit erroneous decisions.

  • Scope: The analysis is local and explains conservative lifecycle heuristics under stated operating assumptions, rather than proving global optimization of the full objective.The objective includes continuous policy variables, a discrete active set, black-box cost, and latent capacity constraints.
  • Retrieval: Task-conditioned retrieval misses each locally useful active skill with probability at most δret, and misses at least one such skill with probability at most |Lt(x)|δret.This follows from the bounded miss-probability assumption and a union bound.
  • Validation: The smoothed leave-one-skill-out estimate is an εval-accurate proxy for true marginal external contribution with probability at least 1 −δval.EMA smoothing, sufficient exposure, and patience or minimum-exposure conditions prevent decisions based on a single noisy audit.
  • Lifecycle decisions: Under bounded single-skill operating costs, retaining or expanding skills does not decrease the local objective under sufficient margins, while retiring improves it only when recovered saved cost exceeds the required margin.The retirement guarantee is explicitly conservative because the saved-cost quantity is not directly observed by the algorithm.
  • False-retirement control: Patience exponentially reduces false retirement probability, and an externally necessary active skill with contribution at least τretire + εval cannot be retired by the retirement rule.These protections apply to currently audited active skills and do not guarantee immunity for previously retired inactive skills.

B Implementation Details … B.3 Additional Baselines Setup

The implementation uses Qwen3-4B with benchmark-specific GRPO, retrieval, validation, and lifecycle-audit settings, while baselines are standardized under shared environment protocols. Additional comparisons include proprietary models, memory and optimizer variants on ALFWorld, and retrieval, search-agent, RL, supervised, and reject-sampling methods on SearchQA.

  • B.1 SLIM Setup: SLIM uses Qwen3-4B, separates training, validation, and test splits, and applies benchmark-specific task counts, prompt limits, and response limits.ALFWorld uses 16 training tasks per update and 32 validation tasks per pass, with maximum prompt and response lengths of 4096 and 512.
  • B.1 SLIM Setup: SLIM trains with GRPO without a warmup or cold-start stage, using learning rate 10^-6 and benchmark-specific rollout, episode, history, and batching settings.ALFWorld uses 8 rollouts, 50 environment steps, and 50 history turns; SearchQA uses 4 rollouts, 4 environment steps, and 4 history turns.
  • B.1 SLIM Setup: SLIM retrieves task-specific skills from the active pool for the detected task type, ranking candidates by cosine similarity between Qwen3-Embedding-0.6B embeddings.The query is the current task description, while skill keys concatenate title, description or principle, when_to_apply, body, tags, and task type.
  • B.1 SLIM Setup: Lifecycle audits run after GRPO validation, record routed skills and failures, and apply bounded leave-one-skill-out auditing with exposure and patience safeguards.Audited skills use smoothed marginal contributions with EMA coefficient 0.9; retirement uses τretire = 0.001, patience p = 3, and minimum exposure nmin = 30 for ALFWorld and nmin = 20 for SearchQA.
  • B.2 Baselines Setup: Prompt, memory, reasoning, and skill baselines are adapted to shared environment parsers, success metrics, and evaluation protocols, with bounded retrieval or memory where applicable.Reflexion, ExpeL, and Mem0 use 200-item memory caps and retrieve at most three items; Skill0 progressively reduces visible skills, while SkillRL persistently retrieves them.
  • B.2 Baselines Setup: The GRPO baseline matches SLIM’s optimizer, environment stack, and reward definition but excludes external context and lifecycle updates, while SkillRL retains a reference-policy KL loss.SkillRL uses KL coefficients 0.01 for ALFWorld and 0.001 for SearchQA; KL-in-reward remains disabled.
  • B.3 Additional Baselines Setup: Additional ALFWorld comparisons include GPT-4o, Gemini-2.5-Pro, SimpleMem, RLOO, and MemRL under adapted or matched protocols.RLOO samples K = 8 rollouts and changes only the advantage estimator relative to GRPO; MemRL updates episodic memory without updating model parameters.
  • B.3 Additional Baselines Setup: Additional SearchQA comparisons cover one-shot RAG, inference-time Search-o1, RL-with-search Search-R1, SFT, and Reject Sampling.RAG inserts the top three retrieved passages once; Search-o1 allows 4 search rounds; Search-R1 uses top-3 passages and maximum action budget 4; SFT and Reject Sampling replace outcome-reward RL with supervised objectives.

C Evaluation Setup · D Additional Experimental Results

The evaluation uses controlled ALFWorld and SearchQA protocols with explicit data partitions, standardized success metrics, fixed final-test behavior, and fairness controls. Additional experiments examine lifecycle transfer, active-skill utility, initialization robustness, gain robustness, benchmark-specific baselines, and audit overhead.

  • C Evaluation Setup: ALFWorld evaluation reports overall and task-type success rates across Pick, Look, Clean, Heat, Cool, and Pick2 household tasks.ALFWorld is a long-horizon text-interaction benchmark requiring admissible text actions.
  • C Evaluation Setup: Both benchmarks use explicit training, development, and final-evaluation partitions, with ALFWorld containing 16, 64, and 128 tasks respectively.SearchQA training data comprise 169,615 questions, with 4,000 development examples balanced by skill type.
  • C Evaluation Setup: Success rate is the primary metric, defined through terminal task success in ALFWorld and final-answer correctness in SearchQA.ALFWorld overall success averages binary episode indicators, while task-type rates group episodes by environment category.
  • C Evaluation Setup: Robustness diagnostics include category results, lifecycle trajectories, ablations, audit overhead, and transfer or initialization sensitivity analyses.The appendix states that these analyses address variance concerns beyond the single-run main training curves.
  • C Evaluation Setup: Final evaluation disables training-time adaptation and fixes SLIM’s active skill set before test rollouts.Test-time retain, retire, expand, lifecycle audit, and skill-creation operations are disabled.
  • C Evaluation Setup: Main comparisons control the Qwen3-4B backbone, checkpoint, environment, rewards, action format, success metric, and no-cold-start-SFT setting.These controls aim to isolate online RL and lifecycle behavior from supervised pre-adaptation and implementation differences.
  • D Additional Experimental Results: Cross-source and transfer evaluations use held-out SearchQA sources and compare no external skill, the initial bank, and SLIM’s fixed final active skills.The transfer evaluation performs no lifecycle update during evaluation.
  • D Additional Experimental Results: Supplementary experiments test lifecycle transfer across SearchQA task families, final active-skill utility, initial-bank robustness, gain robustness, benchmark-specific baselines, and audit overhead.These questions are identified as supplementary analyses supporting the main claims.

D.1 Cross-Task Generalization on SearchQA · D.2 Transfer of the Final Active Skill Bank · D.3 Sensitivity to Skill Initialization

SLIM generalizes across SearchQA source splits and transfers reusable procedural guidance to policies outside lifecycle training. Its lifecycle operations remain robust across skill-bank initializations, repairing weak or noisy banks while preserving an advantage for informative initialization.

  • D.1 Cross-Task Generalization on SearchQA: SLIM achieves the best train-source and held-out-source averages on SearchQA, raising the held-out average from 34.6 for Skill0 to 36.9.The gains are broad rather than concentrated in one dataset.
  • D.2 Transfer of the Final Active Skill Bank: 24.4 and 27.8 points: final SLIM skills improve ALFWorld zero-shot and few-shot policies over no-skill prompting.These policies did not participate in lifecycle training.
  • D.2 Transfer of the Final Active Skill Bank: 2.5 and 2.8 points: final SLIM skills improve ALFWorld zero-shot and few-shot policies over the initial skill bank.The learned active set captures reusable procedural guidance rather than only providing direct transfer gains.
  • D.3 Sensitivity to Skill Initialization: SLIM remains robust to imperfect initial banks because expansion repairs missing coverage while retirement filters noisy or low-value skills.The original initial bank still gives the highest final score, so lifecycle management complements rather than replaces skill initialization.
  • D.3 Sensitivity to Skill Initialization: 76.4%: starting from an empty skill bank, SLIM creates 26 skills during training but remains 11.1 points below the original setting.With only 25% of the original skills, SLIM reaches 81.2% and expands 29 new skills.
  • D.3 Sensitivity to Skill Initialization: 85.6%: with 30% of original skills corrupted and 30% extra mismatched skills injected, SLIM finishes only 1.9 points below the original setting.The controller retires 46 skills and expands 33 new ones, actively removing harmful knowledge and repairing coverage.
  • D.3 Sensitivity to Skill Initialization: The original skill bank gives the best result, while SLIM retires 33 skills and expands 16 new ones, leaving a compact active set of 21 skills.This shows that performance gains do not come from static reuse of the initial skills.

D.4 Robustness of SLIM Performance … F Skill Bank Details

Across robustness checks, expanded baselines, audit-cost analysis, prompt specifications, and skill-bank details, SLIM’s improvements remain reliable while its lifecycle machinery incurs bounded overhead and supports dynamic skill expansion.

  • D.4 Robustness of SLIM Performance: All bootstrap confidence intervals remain above zero, with positive lower bounds against Skill0 and SkillRL† on ALFWorld and tight, statistically reliable gains on SearchQA.ALFWorld intervals are wider because its test set contains 128 episodes, whereas SearchQA’s larger test set produces tighter intervals despite smaller absolute gains.
  • D.5 Expanded Results: SLIM remains competitive with closed-source, memory-based, and RL-based ALFWorld baselines under the same environment and evaluation protocol.The expanded comparison strengthens the conclusion that lifecycle management improves procedural agent training beyond a narrow baseline set.
  • D.5 Expanded Results: SLIM’s SearchQA gains persist against retrieval-augmented, supervised, rejection-sampling, and RL-style baselines, rather than relying solely on stronger prompting or search interfaces.The expanded comparisons cover alternative approaches to improving search-augmented question-answering behavior.
  • D.6 Audit Overhead Comparison: SLIM’s periodic, capped leave-one-skill-out audits cost more than ordinary validation but keep lifecycle cost bounded by K rather than the full skill-bank size.SkillRL uses lighter validation but may accumulate a growing skill bank, while Skill0 is cheaper and primarily models progressive withdrawal.
  • E Prompts: The appendix specifies prompt templates for environment interaction, skill insertion, baseline adapters, and SLIM’s skill-expansion procedure.Templates cover skill-conditioned and no-skill rollouts, retrieved general and task-specific skill insertion, prompt-based and memory-style baselines, and creation of standalone SKILL.md artifacts from routed failures.
  • F Skill Bank Details: The skill-bank appendix summarizes hierarchical general and task-specific skills for ALFWorld and SearchQA, including trigger conditions and concise procedural content.Dynamically expanded skills enter the corresponding task-specific pool and become eligible for later retrieval and lifecycle auditing.

G Limitations · H Broader Impacts

SLIM’s limitations concern local attribution, domain-sensitive lifecycle tuning, and lifecycle auditing. Its broader-impact perspective makes capability allocation between model parameters and modular skills explicit, with implications for controllability, auditability, sample efficiency, and interpretability.

  • G Limitations: SLIM’s marginal contribution estimate is local, single-skill leave-one-out attribution conditioned on the current policy, routing behavior, and active set.It is not a global Shapley-style attribution and does not capture high-order interactions among skills.
  • G Limitations: Lifecycle thresholds and audit budgets require validation tuning, so substantially different domains may require additional calibration.
  • H Broader Impacts: SLIM studies how external capabilities should be allocated between model parameters and modular skill artifacts during agentic RL.
  • H Broader Impacts: Explicit capability allocation may improve controllability, auditability, and sample efficiency while clarifying which behaviors are internalized or preserved externally.
  • H Broader Impacts: ALFWorld and SearchQA use distinct no-skill and skill-conditioned rollout prompts, with skill-conditioned prompts incorporating retrieved memories.The supplied figures identify these prompt variants for both environments.
  • H Broader Impacts: SLIM’s skill insertion format groups active general skills and retrieves task-specific skills by task type and semantic similarity.
Loading 2605.10923v2…