PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning

Jiapeng Zhu, Jianxiang Yu, Yibo Zhao, Chengcheng Han, Qi Gu, Xunliang Cai, Xiang Li, Weining Qian

arXiv:2605.28424v1cs.CL

TL;DR

Existing methods treat general and task-specific skills uniformly, despite the context overhead of externalizing general skills and overfitting and knowledge-conflict risks of internalizing specific skills. Skill0.5 routes tasks by mastery tier, internalizing general skills on hard tasks, maximizing success on medium tasks, and enforcing specific-skill use on easy tasks. Across ALFWorld, Skill0.5 outperformed the strongest skill-augmented baseline by +2.2% (ID) and +8.5% (OOD); it also significantly outperformed prompt-, memory-, and skill-based baselines.

  • Problem

    Existing methods treat general and task-specific skills uniformly, despite the context overhead of externalizing general skills and overfitting and knowledge-conflict risks of internalizing specific skills.

    Source: §1 · p. 2

  • Method

    Skill0.5 routes tasks by mastery tier, internalizing general skills on hard tasks, maximizing success on medium tasks, and enforcing specific-skill use on easy tasks.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across ALFWorld, Skill0.5 outperformed the strongest skill-augmented baseline by +2.2% (ID) and +8.5% (OOD); it also significantly outperformed prompt-, memory-, and skill-based baselines.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The authors conclude that differentiated skill treatment yields moderate improvements on ID tasks and substantially enhances generalization to unseen OOD tasks.

    Source: §5 · p. 8

    The framework has been validated in text-based interactive environments; extending it to more complex domains and longer horizons remains future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Equipping large language models with explicit skills has emerged as a promising paradigm for enabling autonomous agents to solve complex tasks. Agent skills can be inherently divided into general skills for broad cognitive transfer and task-specific skills for dynamic execution. However, existing skill-based reinforcement learning (RL) methods typically force a rigid choice between full externalization, which incurs prohibitive context overhead, and full internalization, which risks overfitting and knowledge conflicts. To address this dilemma, we propose Skill0.5, a novel agentic RL framework that explicitly differentiates skill treatments by combining general skill internalization with task-specific skill utilization. Driven by a dynamic, difficulty-aware router, Skill0.5 streams tasks into distinct mastery tiers to apply tailored optimization strategies: it internalizes general skills via privileged distillation to build a cognitive foundation for hard tasks, while using diagnostic probing on easy tasks to penalize shortcuts and enforce specific skill utilization. Experiments on ALFWorld and WebShop demonstrate that Skill0.5 outperforms both memory-based and skill-based RL baselines, yielding performance improvements across both in-distribution and out-of-distribution scenarios.

1 Introduction

Skill0.5 distinguishes general skills from task-specific skills, internalizing the former while dynamically utilizing the latter through difficulty-aware, tier-specific optimization. It reports gains over the strongest skill-augmented baseline across ALFWorld and WebShop in both ID and OOD settings.

Introduction

  • Agent skills package procedural knowledge into reusable textual directives that are retrieved and injected into prompts to guide complex workflows.The introduction frames skills as a way to help agents master operational logic in real-world tasks.Source: §1 · p. 1
  • Differentiated treatment addresses the context overhead of externalizing lengthy general skills and the overfitting and knowledge-conflict risks of internalizing volatile task-specific skills.General skills provide a context-efficient cognitive foundation, while dynamically utilized task-specific skills support adaptability in OOD scenarios.Source: §1 · pp. 1–2
  • Skill0.5 beats the strongest skill-augmented baseline by +2.2% (ID) and +8.5% (OOD) across ALFWorld and WebShop.The evaluation covers both in-distribution and out-of-distribution settings.Source: §1 · p. 2
  • A difficulty-aware router assigns hard tasks to general-skill distillation, medium tasks to standard RL, and easy tasks to diagnostic probing for specific-skill use.Hard tasks internalize general skills through privileged distillation; easy-task probing enforces faithful utilization of task-specific skills.Source: §1 · p. 2

2 Preliminary

The paper formulates text-based agent interaction as a POMDP and supplies skills as runtime context. It distinguishes broadly applicable general skills from task-specific skills and evaluates general-skill internalization alongside specific-skill use on ID and unseen OOD tasks.

2.1 Task Formulation

  • The agent observes partial textual states, selects natural-language actions, and is optimized for expected success under sparse binary terminal rewards.Actions change the environment state and produce the next observation; each task instruction and interaction history condition the policy.Source: §2.1 · p. 2

2.2 Skill Bank and Runtime Context

  • General skills provide strategic heuristics, while task-specific skills encode domain rules and are retrieved by top-K cosine similarity to the task instruction.The retrieved subset of specific skills and the general skill pool form candidate runtime guidance.Source: §2.2 · pp. 2–3
  • Skill0.5 uses task-difficulty-tailored runtime context during training and relies on retrieved specific skills alone at inference.The passage contrasts this with full externalization in SkillRL (Xia et al., 2026a), full internalization in SKILL0 (Lu et al., 2026b), and SLIM’s active skill subset (Shen et al., 2026).Source: §2.2 · p. 3

2.3 ID and OOD Settings

  • Training exposes ID tasks with their paired specific skills while keeping OOD tasks and skills unseen; evaluation tests ID validation and OOD tasks using their respective skill pools.General skills remain globally accessible, and the stated approach internalizes their strategic essence during ID training while using plug-and-play specific skills at evaluation.Source: §2.3 · p. 3

3 Method

Skill0.5 jointly internalizes general reasoning and promotes use of task-specific skills by routing tasks according to real-time mastery and applying tier-specific objectives. It builds foundational heuristics on hard tasks, reinforces success on medium tasks, and counters skill-bypassing shortcuts on easy tasks.

3 Method

  • Skill0.5 routes tasks into hard, medium, and easy tiers using empirical pass rates, then applies mutually exclusive optimization objectives tailored to each tier.Hard tasks have zero pass rate; medium tasks have a positive rate at or below a sliding-window threshold, and easy tasks exceed it.Source: §3 · p. 4, §3.1 · p. 4, §3.2.3 · p. 5

3.1 Phase-1: Difficulty-Aware Routing

  • Phase-1 estimates difficulty from empirical pass rates on the standard prompt with retrieved specific skills, and reuses its probing trajectories for Phase-2 optimization.The easy/medium threshold is a cross-step sliding-window average of batch mean pass rates, making it more robust to limited batch task counts.Source: §3.1 · p. 4

3.2 Phase-2: Tier-Tailored Optimization

  • Phase-2 targets objectives using Phase-1 real-time mastery: hard tasks get G privileged-context teacher re-attempts, while medium tasks below router threshold ηt reuse G Phase-1 routing rollouts with standard GRPO to maximize success.Source: §3.2 · p. 4, §3.2.1 · p. 4, §3.2.2 · p. 5

3.2.1 Hard Tasks: Internalization via Privileged Distillation

  • For hard tasks, privileged prompts add general heuristics to elicit successful teacher traces, which are distilled into a student conditioned only on the standard prompt.Successful trajectories form a golden set, and token-level Jensen-Shannon Divergence aligns the student’s reasoning with the teacher’s, internalizing heuristics without conditioning on them at inference.Source: §3.2.1 · p. 4

3.2.2 Medium Tasks: Capability Reinforcement

  • For medium tasks, standard GRPO on reused Phase-1 rollouts maximizes success rates through reward-driven trial and error, reinforcing active use of specific skills.This tier targets tasks that have passed the cold-start stage but retain substantial room for capability optimization.Source: §3.2.2 · p. 5

3.2.3 Easy Tasks: Anti-Shortcut Utilization

  • Easy tasks risk shortcut learning that bypasses retrieved skills and harms OOD generalization, so a no-skill probe measures utilization gain and low gain suppresses task-level optimization.The probe is a counterfactual anchor, not part of the policy gradient; utilization gain compares the original pass rate with the no-skill pass rate, and below-window-average gain penalizes shortcut behavior.Source: §3.2.3 · p. 5

4 Experiments

Across ALFWorld and WebShop, Skill0.5 combines general-skill internalization with task-specific skill utilization to improve performance over skill-augmented RL baselines, especially on OOD tasks. Training dynamics, ablations, and trajectory analyses support the need for both skill treatments.

4.1 Experimental Setup

  • ALFWorld uses {Pick, Cool, Clean} as ID and {Look, Heat, Pick2} as OOD; WebShop separates product categories into ID and OOD domains.WebShop’s OOD categories have attribute vocabularies and matching heuristics absent from training.Source: §4.1 · p. 6

4.2 Main Results

  • Skill0.5 improves over SkillRL by +2.3% (ID) and +13.2% (OOD) on ALFWorld, and +2.1% and +3.9% on WebShop.The WebShop gains are on the respective ID and OOD splits.Source: §4.2 · p. 7
  • Prompt-based methods trail Skill0.5 by over 45% on ALFWorld and 28% on WebShop on average, while memory-augmented methods remain below skill-augmented approaches.On ALFWorld OOD, SimpleMem+GRPO reaches 47.2%, whereas Mem0+GRPO reaches 17.0%; the paper attributes memory limits to context noise rather than reusable procedural abstractions.Source: §4.2 · p. 7

4.3 Training Dynamics

  • Early privileged distillation supplies gradient signal when hard tasks yield zero reward variance, while easy-task diagnostic probing counters shortcuts and sustains OOD gains.The method shows faster initial ascent, followed by continued OOD improvement alongside training and ID validation growth.Source: §4.3 · p. 7
  • SkillRL’s validation and OOD performance declines despite training gains, SKILL0’s pure internalization limits OOD performance, and SLIM oscillates after retiring general skills.The reported failure modes are shortcut overfitting, rigidity with novel task-skill pairs, and disrupted reasoning from premature skill retirement, respectively.Source: §4.3 · p. 7

4.4 Ablation Study

  • The ALFWorld ablation shows internalization provides the reasoning foundation, while utilization improves peak OOD adaptability; either treatment alone underperforms the joint design.Utilize-Only lacks basic competence and its contrastive objective becomes futile, whereas Internalize-Only retains a baseline but lacks grounding in novel OOD guidance.Source: §4.4 · pp. 7–8, Table 2 · p. 8

4.5 Case Study

  • In ALFWorld OOD trajectories, SkillRL shows contextual interference, SKILL0 parametric knowledge conflict, and SLIM behavioral collapse, while Skill0.5 executes novel skills using internalized general reasoning.The case analysis links each baseline’s failure to its skill treatment: habitual ID associations, incompatible internalized templates, or premature retirement of general skills.Source: §4.5 · p. 8

5 Conclusion

Skill0.5 addresses the skill-treatment dilemma by combining general-skill internalization with task-specific skill utilization through a difficulty-aware router. Experiments on ALFWorld and WebShop show improvements over prompt-, memory-, and skill-based baselines, including stronger generalization to unseen OOD tasks.

5 Conclusion

  • A dynamic, difficulty-aware router applies privileged distillation to internalize general skills for hard tasks and contrastive advantage probing to enforce task-specific skill use on easy tasks.These tier-specific strategies jointly optimize decoupled general and task-specific skills.Source: §5 · p. 8
  • Skill0.5 resolves the rigid choice between full context externalization and indiscriminate internalization by explicitly differentiating how general and task-specific skills are treated.Source: §5 · p. 8
  • Skill0.5 outperformed prompt-, memory-, and skill-based baselines on ALFWorld and WebShop, with moderate ID improvements and substantially stronger generalization to unseen OOD tasks.The conclusion presents these results as evidence of improved generalization across practical deployment settings involving unseen tasks.Source: §5 · p. 8

Limitations

Skill0.5 is validated on text-based interactive environments, while extension to more complex domains and settings remains future work.

Limitations

  • The authors plan to extend Skill0.5 to code generation, multimodal environments, open-ended web navigation, longer horizons, and larger action spaces.They describe differentiated skill treatment as broadly applicable, but these extensions are planned future work.Source: Limitations · p. 9

Ethical Considerations · A Related Work

The authors identify no direct ethical risks from their work on LLM-agent reasoning and generalization in simulated environments, while acknowledging broader societal implications of autonomous-agent advances.

Ethical Considerations

  • The work studies LLM-agent reasoning and generalization in simulated ALFWorld and WebShop environments and identifies no direct ethical risks.Source: Ethical Considerations · p. 9
  • Experiments use sandboxed settings, public benchmarks without personal or sensitive data, and synthetic interactions; the authors assess misuse risk in these domains as minimal.They also acknowledge broader societal implications of advances in autonomous-agent capabilities.Source: Ethical Considerations · p. 9

A.1 Skill-Augmented Agentic Training · A.2 Agentic Reinforcement Learning

The paper contrasts externalized and internalized skill training, then motivates separating general-skill internalization from task-specific skill use. It also motivates difficulty-aware auxiliary objectives because difficulty-agnostic RL can stall on very hard or very easy tasks.

A.1 Skill-Augmented Agentic Training

  • Skill training has shifted from training-free in-context augmentation toward training agents to harness skills through full externalization or full internalization.The cited early work used skills as in-context augmentations, while later approaches train agents to use them.Source: §A.1 · p. 12
  • SkillRL (Xia et al., 2026a) externalizes guidance in a hierarchical skill bank, keeping general and task-specific skills in context during training and inference.This paradigm persistently maintains guidance within the contextual window.Source: §A.1 · p. 12
  • SKILL0 (Lu et al., 2026b) progressively withdraws contextual skills through a dynamic curriculum until guidance is assimilated into model parameters.This full-internalization paradigm aims to eliminate runtime context overhead.Source: §A.1 · p. 12
  • Skill0.5 decouples general and task-specific skills, internalizing foundational reasoning while adaptively using external guidance for out-of-distribution tasks.This is presented in contrast to SLIM, which treats skills uniformly and risks retaining a skill set incompatible with OOD tasks involving unseen specific skills.Source: §A.1 · p. 12

A.2 Agentic Reinforcement Learning

  • Difficulty-agnostic GRPO can stall when reward variance collapses on impossibly hard or overly simple tasks, motivating difficulty-aware auxiliary objectives.The paper associates these cases with exploration stagnation and saturation-induced mode collapse, respectively.Source: §A.2 · p. 12
  • GRPO (Shao et al., 2024) is a core backbone for interactive-agent training, with extensions addressing credit assignment, long-horizon optimization, and degenerate action cycles.Despite these advances, the GRPO objective remains difficulty-agnostic.Source: §A.2 · p. 12

B WebShop Domain Split Statistics. · C Pseudo Code

The appendix specifies WebShop’s ID/OOD domain split and summarizes Skill0.5’s tiered training loop, which routes tasks by empirical pass rate and applies tier-specific optimization.

B WebShop Domain Split Statistics.

  • WebShop’s 12,087 human-annotated goals are classified into seven domains: four ID categories for training and ID evaluation, and three OOD categories held out for OOD evaluation.The Other category initially accounts for over 60% of goals, so farthest-point sampling based on sentence embeddings downsamples it to a scale comparable with other domains.Source: B WebShop Domain Split Statistics · p. 12
  • The ID training set contains 3,320 goals—776 Apparel, 938 Electronics, 606 Footwear, and 1,000 Other—and ID validation contains 454 goals.Other is downsampled via FPS from approximately 6,600 training goals; ID validation counts are Apparel 113, Electronics 152, Footwear 89, and Other 100, downsampled from approximately 940.Source: B WebShop Domain Split Statistics · pp. 12–13
  • The OOD validation set contains 207 goals: 80 Accessories, 54 Beauty & Health, and 73 Home Decor; validation merges original test and development splits for per-domain coverage.Training uses goal indices 1,500+ in the original ordering; the merged validation pool draws from original test indices 0–499 and development indices 500–1,499.Source: B WebShop Domain Split Statistics · p. 13

C Pseudo Code

  • Skill0.5 routes each training batch using empirical pass rates and a dynamic threshold, then applies different optimization paths to hard, medium, and easy tasks.The pseudocode defines hard tasks by a zero pass rate and medium tasks by a pass rate at or below the threshold; it samples diagnostic trajectories for easy tasks.Source: C Pseudo Code · p. 13
  • The loop initializes the loss for each sampled training batch, accumulates tier-specific losses, and updates policy parameters using the loss gradient.The supplied pseudocode also indicates diagnostic trajectory sampling for the easy tier, but the corresponding loss operation is truncated.Source: C Pseudo Code · p. 13
  • The hard tier uses privileged guidance from general skills and token-level JSD, while the medium tier applies standard GRPO to a sampled trajectory.The pseudocode names the hard-tier guidance as cpriv and combines general skills with Kt(xi); it does not fully define those terms in the supplied passages.Source: C Pseudo Code · p. 13

D Implementation Details

The implementation specifies matched test-time skill-context protocols and the model, optimization, training, retrieval, routing, and distillation settings used for ALFWorld and WebShop.

Inference Protocol

  • At inference, Skill0.5 uses only task-specific skills, while SkillRL, SKILL0, and SLIM receive contexts according to their designs and the ID/OOD setting.SkillRL receives general and ID- or OOD-specific skills; SKILL0 receives no context on ID tasks and only OOD-specific skills on OOD tasks. SLIM uses its final active skill set for ID testing and replaces ID-specific skills with OOD ones for OOD evaluation; Skill0.5 receives only ID- or OOD-specific skills because general skills are internalized.Source: D Implementation Details · p. 13

Training and Implementation Details

  • Training used Qwen2.5-7B-Instruct with GRPO (G = 8, learning rate 1 × 10−6) on 4 H800 GPUs and batches of 16 tasks.Training ran for 120 steps on ALFWorld and 150 on WebShop, with maximum interaction horizons of 30 and 15 steps, respectively.Source: D Implementation Details · p. 13
  • Task-specific skills were retrieved with Qwen3-Embedding-0.6B at K = 3, while routing tracking used W = 5 and privileged distillation optimized token-level JSD over the top 64 tokens.The sliding window tracked routing threshold and utilization gain; the distillation setup follows Ding (2026).Source: D Implementation Details · p. 13

E Case Study · F Detailed Hyperparameters

The case studies show how uniform skill treatments can cause distinct OOD failures, while Skill0.5 separates permanent internalization of general skills from faithful use of task-specific instructions. The accompanying materials identify ALFWorld OOD trajectories and detailed training hyperparameters.

E Case Study

  • Skill0.5’s ALFWorld cases contrast with skill-augmented baselines to illustrate why differentiated skill treatment is essential.Table 4 summarizes failure mechanisms, and Figure 4 provides detailed trajectory comparisons.Source: E Case Study · p. 13, Table 4 · p. 14
  • SkillRL’s ∼1,617-token context lets an ID-trained “fridge →cool” association override an OOD placement instruction, whereas Skill0.5 uses ∼214 tokens of OOD guidance and completes the task in 7 steps.The SkillRL context contains general principles, common mistakes, and OOD-specific skills, with OOD-specific skills occupying ∼12% of the total.Source: E Case Study · p. 13
  • SLIM retires “Systematic Exploration” at step 5 and 7 other general skills by step 50; by step 120, the case shows task hallucination, think-action decoupling, and cross-task collapse.Skill0.5 permanently embeds general skills via JSD distillation, avoiding retirement-driven cognitive erosion.Source: E Case Study · p. 14
  • The cases associate full externalization with context interference, full internalization with parametric conflict, and indiscriminate skill retirement with temporal degradation; Skill0.5 is designed to avoid all three.It permanently internalizes domain-agnostic general skills while keeping dynamic task-specific skills external for faithful utilization.Source: Table 4 · p. 14, E Case Study · p. 15

F Detailed Hyperparameters

  • Figure 4 presents ALFWorld OOD trajectory comparisons, with its caption attributing Skill0.5’s success to internalized general reasoning and faithful use of novel OOD skills; Table 5 lists detailed training hyperparameters.The supplied Table 5 passage provides no individual hyperparameter values.Source: Table 5 · p. 15, Figure 4 · p. 16

Related papers

Loading 2605.28424v1…