Source-linked AI summary
SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization
Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
TL;DR
Inference-time skill augmentation is limited by retrieval noise, token overhead, and the fact that models follow rather than internalize supplied skills. SKILL0 uses in-context reinforcement learning with an adaptive curriculum that progressively removes training-time skill context, achieving stronger benchmark performance while using fewer than 0.5k tokens per step.
Problem
Inference-time skill augmentation can add irrelevant guidance and token overhead while leaving models dependent on skills they execute rather than internalize.
Method
SKILL0 supplies grouped skills during reinforcement-learning rollouts, evaluates their on-policy helpfulness, and progressively removes them through Dynamic Curriculum until inference is skill-free.
Results
+9.7% for ALFWorld, +6.6% for Search-QA, and +10.1% for WebShop over the standard RL baseline, with fewer than 0.5k tokens per step.
Takeaways & Limitations
SKILL0 supports skill internalization as an alternative to inference-time skill augmentation for autonomous LLM agents.
Takeaways & Limitations
SKILL0 relies on the quality of the initial SkillBank, and its offline relevance-driven grouping requires re-partitioning for new task domains.
Abstract
from arXiv · showhide
Agent skills, structured packages of procedural knowledge and executable resources that agents dynamically load at inference time, have become a reliable mechanism for augmenting LLM agents. Yet inference-time skill augmentation is fundamentally limited: retrieval noise introduces irrelevant guidance, injected skill content imposes substantial token overhead, and the model never truly acquires the knowledge it merely follows. We ask whether skills can instead be internalized into model parameters, enabling zero-shot autonomous behavior without any runtime skill retrieval. We introduce SKILL0, an in-context reinforcement learning framework designed for skill internalization. SKILL0 introduces a training-time curriculum that begins with full skill context and progressively withdraws it. Skills are grouped offline by category and rendered with interaction history into a compact visual context, teaching he model tool invocation and multi-turn task completion. A Dynamic Curriculum then evaluates each skill file's on-policy helpfulness, retaining only those from which the current policy still benefits within a linearly decaying budget, until the agent operates in a fully zero-shot setting. Extensive agentic experiments demonstrate that SKILL0 achieves substantial improvements over the standard RL baseline (+9.7\% for ALFWorld, +6.6\% for Search-QA, and+10.1\% for WebShop), while maintaining a highly efficient context of fewer than 0.5k tokens per step. Our code is available at https://github.com/ZJU-REAL/SkillZero.
1 Introduction
SKILL0 addresses the limits of inference-time skill augmentation by training agents to internalize skills and operate autonomously without runtime skill retrieval. Its curriculum supplies skills during training, then adaptively withdraws them as the policy gains competence.
- Takeaway: “Skills at training, zero at inference” summarizes SKILL0’s intended transition from context-dependent execution to autonomous behavior.The framework begins with skill-enhanced training rollouts and ends with no runtime skill context.
- Motivation: Inference-time skill augmentation can introduce retrieval noise, incur compounding token overhead, and leave models executing rather than learning skills.These limitations motivate transferring skill knowledge from context into model parameters.
- Contribution: SKILL0 formulates skill internalization as an explicit reinforcement-learning objective for zero-shot autonomous behavior.The framework moves agents from inference-time skill dependence toward competence encoded in model parameters.
- Method: In-Context Reinforcement Learning provides structured skill guidance during training rollouts and removes it entirely at inference.This design combines guided training with a skill-free deployment setting.
- Method: Dynamic Curriculum retains each skill only while the current policy benefits from it, replacing a rigid withdrawal schedule with adaptive annealing.Skills are evaluated by on-policy helpfulness and progressively discarded until the skill budget reaches zero.
2 Related Work
Related work situates SKILL0 within agentic reinforcement learning and the evolution from raw trajectory replay toward structured skills used as episodic memory. The paper’s overview further distinguishes skill grouping, skill-enhanced reinforcement learning, and dynamic curriculum learning.
- Agentic Reinforcement Learning: Reinforcement learning for LLMs has become a post-training approach for equipping agents with robust decision-making capabilities.This work is framed within broader progress toward autonomous agents in dynamic environments.
- SKILL0 Overview: SKILL0’s overview organizes the approach around relevance-driven skill grouping, an in-context reinforcement-learning loop, and dynamic curriculum learning.These components describe how skills are selected, used during training, and progressively withdrawn.
- Skill-Based Agent Memory: Skills emerged as reusable, abstracted, and structured behavioral primitives that address the length, redundancy, and noise of raw trajectories.They can be consulted by agents as a form of episodic memory during decision making.
3 Method: SKILL0
SKILL0 trains agents to internalize reusable skills by combining compact visual context with an adaptive curriculum that progressively removes external guidance. The method organizes skills by task relevance, evaluates their on-policy helpfulness, and reduces the active skill budget toward zero.
- Agent Loop: SKILL0 models agent automation as sequential action generation from task instructions, observations, and interaction history.The policy samples actions in an environment until task completion or a maximum step threshold.
- Skill Management: The hierarchical SkillBank separates general strategic principles from task-specific knowledge and stores related skills in categorized files.Task-specific skills include specialized action sequences and preconditions, while files are selected as units during training.
- Context Rendering: SKILL0 renders interaction history and selected skills as compact RGB images whose visual representations are fed to the policy to reduce token overhead.The policy can generate the compression ratio alongside the task action at each step.
- In-Context Reinforcement Learning: In-Context Reinforcement Learning provides skill guidance during training while progressively internalizing it into model parameters, eliminating explicit retrieval at inference.The training objective uses sampled trajectories and rewards that combine task success with compression efficiency.
- Adaptive Curriculum Learning: A linear skill-budget decay limits context changes and transitions the policy smoothly to a self-reliant state with no active skills.The curriculum constrains reductions in the active skill set across progressive stages.
- Adaptive Curriculum Learning: The curriculum first groups validation subtasks by skill relevance, then ranks skill files by on-policy helpfulness and selects an active subset within a decreasing budget.Helpfulness compares validation performance with and without each skill after every d training steps.
4 Experiment
SKILL0 is evaluated across ALFWorld, Search-QA, and WebShop against prompting and reinforcement-learning baselines, with additional analyses of efficiency, training dynamics, and curriculum design. It achieves strong skill-free performance while progressively internalizing skills and using compact context.
- Experiment Setup: SKILL0 is evaluated on ALFWorld, Search-QA, and WebShop against in-context prompting and RL-based baselines.The setup includes broader comparisons with memory-augmented, retrieval-based, search-based, and closed-source systems.
- Main Results: 87.9 on ALFWorld, 40.8 on Search-QA, and 78.6 score with 66.4% accuracy on WebShop are achieved by skill-free SKILL0 (3B).These results outperform AgentOCR by +9.7, +6.6, and +10.1 (accuracy), respectively.
- Main Results: 89.8 on ALFWorld, 44.4 on Search-QA, and 85.1/74.2 score/accuracy on WebShop are achieved by skill-free SKILL0 (7B).On WebShop, SKILL0 (7B) surpasses AgentOCR by +6.5 in score and +14.9 in accuracy.
- Token Efficiency: 0.38k, 0.18k, and 0.49k tokens per step are used by 3B SKILL0 on ALFWorld, Search-QA, and WebShop, respectively.The paper attributes the low context cost to visual context modeling and skill internalization.
- Training Dynamics: SKILL0 maintains higher reward curves than AgentOCR on both 3B and 7B backbones during reinforcement-learning optimization.Validation with skills improves faster initially, while skill-free validation gradually catches up, revealing the reported internalization trend.
- Ablations: +1.6% transfer is achieved when prompts are removed with the helpfulness-driven Filter, Rank, and Select curriculum, whereas random selection causes ∆= −13.7%.Using all skills up to the budget drops performance by 2.7%, while static low budgets limit early exploration and produce unstable learning.
5 Conclusion
SKILL0 internalizes agent skills into model parameters through dynamic curriculum training, eliminating external skill reliance at inference. It improves over RL baselines across evaluated tasks while using fewer than 0.5k tokens per step, but depends on SkillBank quality and domain-specific regrouping.
- SKILL0 internalizes agent skills directly into model parameters via a Dynamic Curriculum, eliminating external skill reliance at inference.
- +9.7, +6.6, and +10.1 improvements over RL baselines are reported for ALFWorld, Search-QA, and WebShop, respectively.
- Fewer than 0.5k tokens per step accompany the reported improvements across the evaluated benchmarks.
- SKILL0 relies on the quality of the initial SkillBank, and offline relevance-driven skill grouping requires re-partitioning for new task domains.
A Theoretical Analysis
The theoretical analysis introduces stage-wise conditions for the SKILL0 curriculum, including a condition on the number of stages.
- Each curriculum stage s ∈{1, . . . , NS} is analyzed under a stated condition given by Eq. 14.
A.1 Stability Analysis
The stability analysis connects gradual skill-budget reduction with reliable advantage estimation and policy improvement under bounded skill influence and approximate submodularity assumptions. It also explains how internalized skills become automatically excluded as their marginal helpfulness vanishes.
- Stability Analysis: Abrupt skill-context shifts can degrade the signal-to-noise ratio of the advantage estimator, motivating analysis of the linear budget schedule.The schedule uniformly bounds the number of skills removed at each stage transition.
- Stability Analysis: The linear budget schedule removes at most ⌈N/(NS − 1)⌉ skills at each stage transition.
- Stability Analysis: Assuming each skill file has bounded influence on expected reward, the stage transition introduces non-stationarity and corresponding advantage bias.The non-stationarity magnitude is given as ⌈N/(NS − 1)⌉δr in Eq. 10.
- Stability Analysis: The estimator remains reliable under stated conditions involving n(s), while GRPO policy improvement is analyzed using a bound on ϵ = maxa |Aπθs (a)|.
- Skill Selection: The skill-selection analysis defines marginal helpfulness as Δk(S) and selects positive-helpfulness skills greedily in decreasing order up to budget M(s).The objective J(S; πθ) is analyzed under α-approximate submodularity.
- Skill Selection: As the policy internalizes a skill, its helpfulness score approaches zero, so the positivity filter automatically excludes it from the active curriculum.This produces a self-paced curriculum in which M(s) serves as an upper bound.
B More Training Dynamics
Training dynamics show that skill context accelerates early performance on ALFWorld and Search-QA, while skill-free performance catches up as optimization proceeds. These dynamics align with progressive skill internalization.
- More Training Dynamics: With skill context, SKILL0 achieves faster early-stage performance improvement across ALFWorld and Search-QA subtasks.
- More Training Dynamics: Without skill context, initial performance is lower but gradually catches up toward the end of optimization.
- More Training Dynamics: The training dynamics are consistent with progressive annealing driving task-relevant knowledge into model parameters.
C Ablation Details
The paper provides detailed ablation results in Table 5 as supplementary analysis for Figure 7 and Table 3.
- Table 5 reports detailed results of the curriculum ablations.
D More Comparisons
Extended comparisons show that SKILL0 outperforms a broad range of memory-augmented, retrieval-based, search-based, and closed-source baselines across ALFWorld and Search-QA.
- 87.9 (3B) and 89.8 (7B) average success rates on ALFWorld outperform memory-augmented and closed-source baselines.Compared methods include ExpeL, SimpleMem, Mem0, MemRL, GPT-4o, and Gemini-2.5-Pro.
- 40.8 (3B) and 44.4 (7B) average scores on Search-QA surpass retrieval-augmented and search-based methods.Compared methods include RAG, Search-R1, ZeroSearch, and EvolveR.
- 63.7/66.9 on Bamboogle demonstrates particularly strong performance on an out-of-domain multi-hop dataset.The passage characterizes this result as highlighting robust generalization to unseen reasoning tasks without domain-specific adaptation.
E Implementation Details
The implementation constructs compact visual contexts from rendered prompts, task descriptions, skill context, and interaction history across ALFWorld, Search-based QA, and WebShop.
- SKILL0 renders benchmark-specific visual contexts with controlled font, spacing, width, and semantic color-coding configurations.The stated settings use monospace text, 1.2 line spacing, and benchmark-specific font sizes and maximum widths.
- Search-based QA: Table 7 reports results on Search-based QA, distinguishing in-domain and out-of-domain evaluations.
- Table 8 and Table 9 show representative structured procedural skills stored in SkillBank across the task categories, including WebShop.
- Figures 11–13 provide the prompt templates used for ALFWorld, Search-based QA, and WebShop.
- Search-based QA: Search-based QA prompts combine skill context, the task question, step count, and an image containing the full interaction history.Past queries and results are represented inside <search>...</search> and <information>...</information> tags.