Source-linked AI summary
Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning
Jiapeng Zhu, Jianxiang Yu, Yibo Zhao, Chengcheng Han, Qi Gu, Xunliang Cai, Xiang Li, Weining Qian
TL;DR
Existing agentic RL methods treat general and task-specific skills uniformly, forcing a tradeoff between context-heavy externalization and risky internalization. Skill0.5 routes tasks by difficulty to internalize general skills while enforcing task-specific skill use, outperforming the strongest skill-augmented baseline by +2.2% ID and +8.5% OOD across ALFWorld and WebShop.
Problem
Existing methods treat general and task-specific skills uniformly, creating tradeoffs between prohibitive context overhead and overfitting or knowledge conflicts.
Method
Skill0.5 uses a difficulty-aware router and tier-specific optimization to internalize general skills while enforcing task-specific skill utilization.
Results
+2.2% ID and +8.5% OOD over the strongest skill-augmented baseline across ALFWorld and WebShop.
Takeaways & Limitations
Skill0.5 yields moderate in-distribution gains and substantially improves generalization to unseen out-of-distribution tasks.
Takeaways & Limitations
The framework has been validated only in text-based interactive environments, with broader domains and longer-horizon settings left for future work.
Abstract
from arXiv · showhide
Equipping large language models with explicit skills has emerged as a promising paradigm for enabling autonomous agents to solve complex tasks. Agent skills can be inherently divided into general skills for broad cognitive transfer and task-specific skills for dynamic execution. However, existing skill-based reinforcement learning (RL) methods typically force a rigid choice between full externalization, which incurs prohibitive context overhead, and full internalization, which risks overfitting and knowledge conflicts. To address this dilemma, we propose Skill0.5, a novel agentic RL framework that explicitly differentiates skill treatments by combining general skill internalization with task-specific skill utilization. Driven by a dynamic, difficulty-aware router, Skill0.5 streams tasks into distinct mastery tiers to apply tailored optimization strategies: it internalizes general skills via privileged distillation to build a cognitive foundation for hard tasks, while using diagnostic probing on easy tasks to penalize shortcuts and enforce specific skill utilization. Experiments on ALFWorld and WebShop demonstrate that Skill0.5 outperforms both memory-based and skill-based RL baselines, yielding performance improvements across both in-distribution and out-of-distribution scenarios.
1 Introduction
Skill0.5 addresses the limitations of uniformly treating general and task-specific skills by combining general-skill internalization with task-specific skill utilization through difficulty-aware routing. Evaluations on ALFWorld and WebShop show gains over the strongest skill-augmented baseline in both ID and OOD settings.
- Motivation: Agent skills package procedural knowledge into modular, reusable textual directives that are dynamically retrieved and injected into prompts to guide complex workflows.Skills codify standard operating procedures and heuristics for intricate tasks.
- Limitations: Full externalization can overwhelm in-context learning with lengthy prompts, while unfamiliar external skills may conflict with internalized patterns and cause execution hallucinations.These limitations are especially relevant to long-horizon tasks and dynamic real-world environments.
- Skill taxonomy: General skills provide domain-agnostic strategies, whereas task-specific skills encode granular rules and are dynamically updated; treating both uniformly creates a critical dilemma.The paper motivates differentiated treatment because general skills are lengthy while task-specific skills are susceptible to retrieval noise.
- Method and results: +2.2% (ID) and +8.5% (OOD) across ALFWorld and WebShop: Skill0.5 outperforms the strongest skill-augmented baseline.The framework uses a difficulty-aware router to stream tasks into three tiers and jointly optimizes decoupled general and specific skills based on real-time task mastery.
2 Preliminary
This section formalizes text-based agent interaction as a POMDP with language actions and prompt-injected skill context, and distinguishes general skills from retrieved task-specific skills. It also defines evaluation across observed and unseen domains, where general skills remain globally available and specific skills are retrieved from the relevant pool.
- Agent and Environment: The agent receives partial textual observations, generates free-form language actions from its history, instruction, and context, and induces POMDP state transitions.The environment is modeled as M = (Senv, A, O, T, Ω, Rstep, γ), with observations emitted after each transition.
- Objective: For outcome-based tasks, training maximizes expected success under a sparse binary terminal reward, with procedural skills injected as runtime prompt context.The objective is maxθ E[R(τ)] where R(τ) ∈ {0, 1}.
- Skill Bank: The skill bank separates universally applicable general skills from fine-grained task-domain skills, while retrieving relevant specific skills through Top-K cosine-similarity matching.General skills can be fully provided, whereas an embedding model selects the task-relevant subset Kt(x) from the specific-skill pool.
- Evaluation Setup: The deployment protocol partitions domains and specific skills into ID and OOD sets, trains only on ID tasks and skills, and evaluates retrieval separately on observed and unseen pools.General skills SG remain globally accessible, while OOD tasks and paired specific skills remain unobserved during training.
- Evaluation Setup: At evaluation, the framework uses ID validation tasks with retrieval over Sid and OOD tasks with retrieval over the previously unseen Sood, while pursuing general-skill internalization and specific-skill utilization.This design exposes accessible skills according to each method’s principles and targets generalization through plug-and-play specific skills.
3 Method
Skill0.5 dynamically routes tasks by real-time mastery and applies tier-specific objectives, combining foundational reasoning internalization with task-specific skill utilization. Its unified RL procedure uses privileged distillation for hard tasks, GRPO for medium tasks, and counterfactual utilization optimization for easy tasks.
- Phase-1: Difficulty-Aware Routing: Skill0.5 uses a two-phase paradigm that estimates difficulty from empirical pass rates, routes tasks into Hard, Medium, and Easy tiers, and reuses routing rollouts for optimization.The router assigns Hard when p_i = 0, Medium when 0 < p_i ≤ η_t, and Easy when p_i > η_t, with η_t computed from a cross-step sliding average.
- Phase-2: Hard-Tier Internalization: Hard tasks receive privileged general skills to generate successful teacher trajectories, whose reasoning steps are distilled into the standard-prompt student using token-level Jensen-Shannon Divergence.The student learns to reproduce the teacher’s behavior without explicitly conditioning on the general-skill set during inference.
- Phase-2: Medium-Tier Optimization: Medium tasks reuse Phase-1 trajectories and apply standard GRPO to improve success through reward-driven trial and error while reinforcing active use of specific skills.This tier targets tasks that avoid complete cold-start failure but remain below the dynamic threshold η_t.
- Phase-2: Easy-Tier Utilization Diagnosis: Easy tasks are probed without retrieved skills to expose shortcut learning, and their utilization gain u_i = p_i − p_i^none determines an auxiliary task-level advantage.The no-skill trajectories serve only as a counterfactual anchor and do not enter policy-gradient computation.
- Phase-2: Easy-Tier Utilization Optimization: When a task’s utilization gain falls below the recent-window anchor, the negative auxiliary advantage suppresses gradients for actions that bypass specific skills within the GRPO objective.The composite advantage combines task-level utilization with trajectory-level quality, penalizing low-reliance optimization.
- Overall Objective: Skill0.5 jointly optimizes foundational reasoning internalization and task-specific guidance utilization through tier-specific losses aggregated in one RL framework.The full procedure is summarized in Algorithm 1.
4 Experiments
Experiments on ALFWorld and WebShop show that Skill0.5 outperforms prompting, memory-based, and skill-augmented baselines, especially on OOD splits. Training dynamics and ablations attribute these gains to difficulty-aware internalization for hard tasks and anti-shortcut skill utilization for easy tasks.
- Overall performance: Skill0.5 improves over SkillRL by +2.3% ID and +13.2% OOD on ALFWorld, and +2.1% and +3.9% on WebShop.These gains establish the strongest reported performance across the evaluated settings.
- Baseline comparison: Prompting and memory-based methods trail Skill0.5 by over 45% on ALFWorld and 28% on WebShop on average.SimpleMem+GRPO reaches 47.2% on ALFWorld OOD, whereas Mem0+GRPO reaches 17.0%.
- Training dynamics: Privileged distillation resolves early zero-gradient training caused by hard-task reward collapse, enabling faster initial optimization.The difficulty-aware router triggers distillation as a surrogate gradient source when reward variance and advantages vanish.
- Training dynamics: Anti-shortcut diagnostic probing sustains training and ID growth while producing consistent, non-decaying OOD improvements as easy tasks dominate later training.This behavior indicates utilization of novel skills rather than overfitting through skill bypasses.
- Ablation study: Ablations show that internalization is necessary for capability, while utilization further maximizes OOD adaptability.Utilize-Only suffers catastrophic ID and OOD decline, whereas Internalize-Only has a moderate performance drop.
- Failure analysis: Qualitative analysis identifies distinct baseline failures: SkillRL has contextual interference, SKILL0 has parametric knowledge conflict, and SLIM has behavioral collapse.These mechanisms describe how ID habits, internalized templates, or degeneration impair ALFWorld OOD behavior.
5 Conclusion
Skill0.5 addresses the skill-treatment dilemma by jointly optimizing decoupled general and task-specific skills instead of choosing between full externalization and indiscriminate internalization. Experiments on ALFWorld and WebShop show moderate in-distribution gains and substantially stronger generalization to unseen out-of-distribution tasks.
- Framework: Skill0.5 resolves the skill-treatment dilemma through a unified framework that jointly optimizes decoupled general and task-specific skills.The framework explicitly differentiates how the two skill types are treated.
- Framework: The framework replaces the rigid choice between full context externalization and indiscriminate internalization with differentiated skill treatment.This directly targets the limitation identified in existing reinforcement-learning methods for large-language-model agents.
- Results: Experiments on ALFWorld and WebShop found that Skill0.5 significantly outperformed prompt-, memory-, and skill-based baselines.The reported gains were moderate on in-distribution tasks and substantial for generalization across practical deployments involving unseen out-of-distribution tasks.
Limitations
Skill0.5 is validated on text-based interactive environments, while future work will test its differentiated skill treatment in broader and more demanding settings.
- Skill0.5 is currently validated on text-based interactive environments.
- Future work will extend Skill0.5 to code generation, multi-modal environments, open-ended web navigation, longer horizons, and larger action spaces.
Ethical Considerations · A Related Work
The paper reports no direct ethical risks, as its experiments remain in controlled simulated environments without human interaction, sensitive content, or new data collection. It nevertheless acknowledges broader societal implications associated with advancing autonomous-agent capabilities.
- Ethical Considerations: The research identifies no direct ethical risks from improving LLM-agent reasoning and generalization in ALFWorld and WebShop.The work is limited to simulated environments.
- Ethical Considerations: All experiments use controlled, sandboxed settings without real-world deployment or interaction with human users.This limits the study’s direct exposure to real-world stakeholders.
- Ethical Considerations: The publicly available benchmarks contain no personally identifiable information or sensitive content.The evaluation data therefore avoids those categories of potentially sensitive information.
- Ethical Considerations: The method introduces no new data-collection procedures, and its training data comes from synthetic environment interactions.Training relies on interactions generated within the simulated environments.
- Ethical Considerations: The paper acknowledges that advances in autonomous-agent capabilities carry broader societal implications.This acknowledgment extends beyond the immediate controlled experimental setting.
A.1 Skill-Augmented Agentic Training · A.2 Agentic Reinforcement Learning
Skill-augmented agent training has progressed from training-free in-context skills toward learned skill handling, but joint internalization and utilization remains underexplored. Skill0.5 addresses this gap by decoupling general and task-specific skills, while agentic RL builds on GRPO to tackle multi-turn training challenges.
- A.1 Skill-Augmented Agentic Training: Early agent-skill research primarily used skills as training-free in-context augmentations.
- A.1 Skill-Augmented Agentic Training: Recent work trains agents to harness skills as agentic tasks grow more complex and exceed model capacities.
- A.1 Skill-Augmented Agentic Training: One emerging paradigm favors full skill externalization rather than internalizing skills within the model.
- A.1 Skill-Augmented Agentic Training: Joint skill internalization and utilization remains underexplored, while SLIM dynamically retains skills for use or retires them after internalization.
- A.1 Skill-Augmented Agentic Training: Skill0.5 decouples general and task-specific skills, jointly optimizing foundational internalization and adaptive utilization for authentic OOD settings.
- A.2 Agentic Reinforcement Learning: Group Relative Policy Optimization has become a core backbone for training large language models as interactive agents.
- A.2 Agentic Reinforcement Learning: Recent agentic RL advances build on GRPO to address temporal credit assignment, long-horizon optimization, and training stability in multi-turn environments.
B WebShop Domain Split Statistics. · C Pseudo Code
WebShop goals are split into balanced ID and held-out OOD domains, while Skill0.5 routes training examples by empirical difficulty into tier-specific optimization procedures. The pseudocode combines general and specific skills with distinct hard-, medium-, and easy-tier objectives.
- B WebShop Domain Split Statistics.: 12,087 human-annotated WebShop goals are partitioned into seven keyword-defined domains, with four ID domains and three OOD domains.The OOD categories are reserved for OOD evaluation only.
- B WebShop Domain Split Statistics.: FPS based on sentence embeddings down-samples the heavily imbalanced Other category, which initially contains over 60% of all goals.The training Other category is reduced from approximately 6,600 goals.
- B WebShop Domain Split Statistics.: The ID training set contains 3,320 goals, while ID validation contains 454 goals across Apparel, Electronics, Footwear, and Other.Training counts are Apparel 776, Electronics 938, Footwear 606, and Other 1,000; ID validation counts are Apparel 113 and Electronics 152, with remaining counts reported separately.
- B WebShop Domain Split Statistics.: The OOD validation set contains 207 goals across Accessories, Beauty & Health, and Home Decor.The domain counts are Accessories 80, Beauty & Health 54, and Home Decor 73; training uses goal indices 1,500+ while validation merges original test and development splits.
- C Pseudo Code: Algorithm 1 requires policy πθ, general skills SG, and specific skills SS, then samples each batch from the ID training distribution.Each training step initializes the loss and processes examples through routing before policy updates.
- C Pseudo Code: Difficulty-aware routing evaluates each example’s empirical pass rate pi and computes a dynamic routing threshold ηt.The routing stage is applied to every xi in the sampled batch.
- C Pseudo Code: Hard examples use privileged general-skill guidance and token-level JSD, medium examples use standard GRPO, and easy examples use diagnostic trajectories.The hard tier is selected when pi == 0, the medium tier when pi ≤ηt, and the easy tier otherwise.
- C Pseudo Code: After tier-specific losses are accumulated, Skill0.5 updates policy parameters θ using ∇θL for each training step.The pseudocode completes the per-example loop before applying the gradient update.
D Implementation Details
Skill0.5 uses differentiated inference prompts, exposing only task-specific skills because general skills are internalized. Its implementation uses Qwen2.5-7B-Instruct with GRPO, task-specific retrieval, sliding-window routing, and top-64-token privileged distillation.
- Inference Protocol: Inference prompts follow each baseline’s design: SkillRL retrieves general plus ID/OOD-specific skills, SKILL0 uses none on ID and OOD-specific skills on OOD, and SLIM swaps ID-specific skills for OOD ones.SLIM uses a trained active set containing general and ID-specific skills for ID testing.
- Training and Implementation Details: The implementation uses Qwen2.5-7B-Instruct with GRPO, group size G = 8, learning rate 1 × 10^-6, four H800 GPUs, and batch size 16 tasks per iteration.Training spans 120 steps for ALFWorld and 150 steps for WebShop.
- Training and Implementation Details: Maximum interaction horizons are 30 steps for ALFWorld and 15 steps for WebShop.These horizons are specified alongside the training schedules for the two environments.
- Training and Implementation Details: Task-specific skills are retrieved with Qwen3-Embedding-0.6B at capacity K = 3, while routing and utilization tracking uses a sliding window W = 5.Privileged distillation optimizes token-level JSD over the top-64 tokens.
E Case Study · F Detailed Hyperparameters
The ALFWorld OOD case study shows that uniform skill treatment causes distinct interference, conflict, and degradation failures, whereas Skill0.5 separates internalized general reasoning from externally utilized task-specific skills. The accompanying hyperparameter section provides the detailed training configuration and trajectory evidence for these comparisons.
- E Case Study: Across representative OOD failures, Skill0.5 consistently succeeds with significantly fewer steps than the skill-augmented baselines.Table 4 summarizes the failure mechanisms, while Figure 4 compares the detailed trajectories.
- E Case Study: SkillRL suffers contextual interference when general principles and OOD skills compete within an approximately 1,617-token context.OOD-specific skills occupy only approximately 12% of the context, yet the agent executes an action contradicting both the task goal and injected skill.
- E Case Study: SKILL0 exhibits parametric knowledge conflict by following internalized ID procedures despite an explicit OOD instruction to switch on the lamp.The agent moves the bowl and repeatedly attempts to take the desklamp as a portable tool.
- E Case Study: Skill0.5 internalizes domain-agnostic general skills while keeping task-specific skills external, enabling faithful compliance with novel OOD skills.Contrastive utilization training builds a read-instruction-to-execute capability.
- E Case Study: SLIM exhibits behavioral collapse after retiring general skills, with 66.7% retired by step 50 and catastrophic degradation by step 120.It initially succeeds at step 85, but later shows task hallucination and think-action decoupling after 70 additional training steps without the retired constraints.
- E Case Study: The unified diagnosis attributes OOD failures to context-level interference, parameter-level conflict, and irreversible temporal degradation under full externalization, full internalization, and indiscriminate retirement.Skill0.5 is designed to eliminate all three failure modes through differentiated skill treatment.
- F Detailed Hyperparameters: In the SkillRL trajectory, the agent heats the potato, reaches the fridge, then incorrectly cools it, following an ID habit instead of the injected OOD skill.The injected skill requires navigating straight to the destination and placing the object once after heating.