PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillOS: Learning Skill Curation for Self-Evolving Agents

Siru Ouyang, Jun Yan, Yanfei Chen, Rujun Han, Zifeng Wang, Bhavana Dalvi Mishra, Rui Meng, Chun-Liang Li, Yizhu Jiao, Kaiwen Zha, Maohao Shen, Vishy Tirumalashetty, George Lee, Jiawei Han, Tomas Pfister, Chen-Yu Lee

arXiv:2605.06614v1cs.AIcs.CL

TL;DR

Streaming agents need to learn from past interactions, but existing skill-curation methods are manual, heuristic, or short-horizon. SkillOS trains a skill curator with experience-driven RL and grouped task streams, consistently improving effectiveness and efficiency across tasks and executors.

  • Problem

    Existing skill-curation methods are manual or fixed-rule approaches that lack scalable adaptation and downstream performance feedback for streaming agents.

  • Method

    SkillOS uses experience-driven RL to train a curator that updates an external SkillRepo while a frozen executor applies its skills across grouped task streams.

  • Results

    +9.8% relative performance and −6.0% interaction steps versus the strongest baseline, with gains across tasks and executor backbones.

  • Takeaways & Limitations

    Trained, modular skill curation can help agents self-evolve from experience while generalizing across diverse tasks and LLM executors.

  • Takeaways & Limitations

    SkillOS-base can produce generic skill recipes without numbers, formulas, or examples.

Abstract

from arXiv · show

LLM-based agents are increasingly deployed to handle streaming tasks, yet they often remain one-off problem solvers that fail to learn from past interactions. Reusable skills distilled from experience provide a natural substrate for self-evolution, where high-quality skill curation serves as the key bottleneck. Existing approaches either rely on manual skill curation, prescribe heuristic skill operations, or train for short-horizon skill operations. However, they still struggle to learn complex long-term curation policies from indirect and delayed feedback. To tackle this challenge, we propose SkillOS, an experience-driven RL training recipe for learning skill curation in self-evolving agents. SkillOS pairs a frozen agent executor that retrieves and applies skills with a trainable skill curator that updates an external SkillRepo from accumulated experience. To provide learning signals for curation, we design composite rewards and train on grouped task streams based on skill-relevant task dependencies, where earlier trajectories update the SkillRepo, and later related tasks evaluate these updates. Across multi-turn agentic tasks and single-turn reasoning tasks, SkillOS consistently outperforms memory-free and strong memory-based baselines in both effectiveness and efficiency, with the learned skill curator generalizing across different executor backbones and task domains. Further analyses show that the learned curator produces more targeted skill use, while the skills in SkillRepo evolve into more richly structured Markdown files that encode higher-level meta-skills over time.

1. Introduction

SkillOS addresses the limits of one-off agents and existing skill-curation methods by learning long-term curation policies from experience. It combines a frozen executor with a trainable curator and evaluates reusable skills through related task streams.

  • Motivation: Streaming agents need self-evolution because one-off execution prevents them from building long-term proficiency across sequential tasks.Agents should continually improve rather than repeatedly start from scratch.
  • Motivation: Reusable skills provide procedural memory through a loop of skill retrieval, task execution, and collection updates.This closed-loop workflow uses resulting trajectories to update the skill collection.
  • Limitations: Manual and heuristic curation do not scale or adapt because they require substantial human expertise or rely on fixed rules without downstream feedback.These limitations prevent curation from matching the executor’s actual needs across diverse tasks.
  • SkillOS: SkillOS groups related tasks and uses composite rewards so earlier curation decisions are evaluated by their long-term utility on later tasks.This addresses prior RL approaches’ focus on skill use or short task streams, which limits learning signals for reusable skills and complex updates or deletions.
  • SkillOS: SkillOS pairs a frozen Agent Executor with a trainable Skill Curator that edits a Markdown-based SkillRepo through insert, update, and delete operations.The executor retrieves skills to act, while the curator manages the repository from resulting experiences.
  • Results: Up to +9.8% relative performance improvement and −6.0% fewer interaction steps distinguish SkillOS from the strongest baseline across multi-turn agentic and single-turn reasoning tasks.The curator also generalizes across executors and tasks, including performance improvements with the Gemini-2.5-Pro executor.

2. Related Work

Related work develops self-evolving agents through procedural memory, experience representations, and reinforcement learning for memory management. These approaches range from replayable case-based memories to editable higher-level knowledge and learned memory operations.

  • Memory for Self-Evolving Agents: Procedural memory encodes interaction histories into reusable and retrievable representations, forming a central mechanism for self-evolving agents.
  • Memory for Self-Evolving Agents: Case-based methods store raw trajectories or abstracted query–response pairs for direct replay or reuse as in-context exemplars.Another line abstracts experiences into higher-level knowledge that is editable, auditable, and composable.
  • Learning Memory and Skill Curation with RL: Reinforcement learning for memory-enabled LLM agents includes long-context management with predefined operations and learned memory tool-calls.These represent growing directions in training agents to utilize and manage memory.

3. Methodology

SkillOS separates execution from skill curation: a frozen executor applies retrieved skills while a trainable curator updates an external SkillRepo from experience. Training uses related task groups and composite rewards so downstream task performance and intermediate curation quality supervise longer-horizon skill evolution.

  • SkillOS Architecture: A frozen executor retrieves relevant skills with BM25 and solves each streaming task, while the curator observes trajectories and correctness to update SkillRepo.The curator applies insert_skill, update_skill, and delete_skill operations, creating a closed loop across sequential tasks.
  • SkillOS Architecture: Skills are stored as SKILL.md Markdown files combining YAML metadata with executable instructions, workflows, constraints, and reusable heuristics.The YAML frontmatter specifies the skill name and when it should be used.
  • Reward and Optimization: The curator is optimized with GRPO using four signals: downstream task outcomes, valid function calls, concise repository updates, and judged content quality.The compression signal discourages verbatim trajectory copying, while the content-quality signal uses an external Qwen3-32B judge.
  • Grouped Training: Training groups related tasks by skill-relevant attributes, then evaluates updates from earlier tasks on subsequent dependent tasks.This grouped formulation exposes longer skill-evolution trajectories and denser feedback than short-horizon transfer.
  • Reward and Optimization: The compression reward encourages distilling reusable skills rather than storing raw trajectories by rewarding concise repository updates.Repository and curator-context token lengths determine the compression signal.
  • Reward and Optimization: GRPO samples independent full-group rollouts and applies a clipped surrogate objective, assigning each rollout’s advantage uniformly across curation-operation tokens.The objective discards the KL term to encourage policy exploration.

4. Experiments

SkillOS is evaluated on multi-turn agentic and single-turn reasoning tasks for effectiveness and efficiency against memory-free, memory-based, and internal baselines. It consistently improves performance across benchmarks and transfers across frozen executors and task domains.

  • Datasets: Experiments cover ALFWorld and WebShop agentic tasks plus single-turn reasoning tasks, testing SkillOS across multiple task settings.The agentic environments involve household-task completion and online product search and purchase.
  • Evaluation Configurations: Evaluation measures success rate or accuracy for effectiveness and execution steps or tokens for efficiency.Comparisons include No Memory, ReasoningBank, MemP, SkillOS-base, and SkillOS-gemini.
  • Main Results: SkillOS achieves strong performance gains across all three benchmarks, improving agentic and reasoning outcomes through reusable skill curation.Agentic skills capture action ordering, exploration, recovery, and environment constraints, while reasoning skills encode decomposition, constraint formulation, and verification patterns.
  • Executor Transfer: SkillOS consistently improves frozen executors spanning Qwen3-8B, Qwen3-32B, and Gemini-2.5-Pro, including executors not seen during training.Training uses Qwen3-8B as the executor, while evaluation additionally tests other executor backbones.
  • Cross-Task Generalization: SkillOS transfers across task domains, with most off-diagonal evaluations improving over baselines and demonstrating reusable skills beyond task-specific heuristics.Same-task training and testing often yields the strongest gains, while reasoning-trained curators also transfer partially to other tasks.

5. Analysis

The analysis shows that auxiliary rewards improve SkillOS performance, while RL evolves the curator from repository population toward more actionable skill refinement. Evolved skills improve task solving through broader but more precise utilization, using fewer skills per example.

  • Ablation Studies: 58.6 success rate after removing the content-quality reward, down from 61.2.Removing the compression reward also causes a smaller but consistent performance drop.
  • Behaviors of Skill Curator: Insert_skill overwhelmingly dominates at the beginning of training as the curator populates the repository with newly distilled knowledge.The analyzed operations are insert_skill, update_skill, and delete_skill.
  • Skill Evolution Dynamics: RL shifts skill additions from verbose generic guidance toward actionable failure-handling logic and conditional branches.These structures specify when to deviate from the default workflow, improving execution-oriented refinement.
  • Role of Curated Skills in Task Success: SkillOS invokes skills on all evaluation examples, achieves higher success, uses a larger fraction of curated skills, and selects fewer skills per example than the baseline.The results indicate that gains come from more precise skill selection rather than from providing more skill context.

6. Conclusion · Contents of Appendix

SkillOS learns modular skill curation through reinforcement learning, separating curation from execution and optimizing updates by their downstream effects on future tasks. Across benchmarks and LLM backbones, it improves performance and efficiency while surpassing frontier models’ zero-shot curation ability.

  • 6. Conclusion: SkillOS is an RL training recipe for learning skill curation in self-evolving agents.
  • 6. Conclusion: The conclusion identifies modularity, downstream evaluation, and broad benchmark gains as central properties of SkillOS.
  • 6. Conclusion: Decoupling the skill curator from the agent executor enables modular curation without retraining the underlying executor.
  • 6. Conclusion: Grouped task streams and executor-grounded rewards optimize curation decisions according to their downstream impact on future tasks.
  • 6. Conclusion: SkillOS consistently improves both performance and efficiency across diverse benchmarks and LLM backbones.
  • 6. Conclusion: Trained skill curation can outperform frontier models’ zero-shot curation ability.

A. Prompts … B.3. Experiment Setup

The appendix specifies SkillOS’s prompt templates, reward-related judging procedures, hyperparameters, and two-stage task-grouping pipeline. The setup curates reusable skills from execution experience and constructs related, progressively challenging task groups through latent attributes, dependency-aware similarity, and curriculum constraints.

  • A. Prompts: SkillOS provides full prompt templates for its framework phases and supplies the skill curator with a role focused on converting execution experiences into reusable skills.The curator prompt includes task descriptions, past skills, agent trajectories, and task results as input data.
  • A.2. Prompt for Agent Executor: Frozen agent-executor prompts provide the current task, prior interactions, and retrieved skills, explicitly requiring chain-of-thought reasoning across agentic and reasoning tasks.ALFWorld and WebShop follow GiGPO’s environment and prompt setting, while reasoning prompts request step-by-step answers in boxed form.
  • A.3. Prompt Used During Training: A memory-analyst prompt evaluates skill quality using abstraction, reusability, and actionability criteria for generated Markdown skill contents.An external judge assigns the corresponding reward score for skill contents.
  • A.4. Prompt for LLM-as-a-Judge to Obtain Correctness Signals: SkillOS obtains self-judged correctness signals through an LLM-as-a-judge using the corresponding frozen executor backbone for ALFWorld, reasoning, and WebShop.The correctness-signal prompts are presented for self-evolution.
  • B.1. Hyperparameters: Table 4 reports the hyperparameter choices used during SkillOS training and inference for different tasks.The appendix explicitly presents these settings as the experiment setup.
  • B.2. Grouping Training Instances: The grouping pipeline has two stages: an LLM annotator assigns latent attributes, then retrieval, filtering, and ranking assemble related task groups using phrase-level similarity.DeepMath-103K supplies difficulty scores for curriculum grouping, while multi-turn benchmarks use default task-type partitions such as ALFWorld’s six task types.
  • B.2.1. Stage 1: Latent Attribute Annotation: Each task’s latent attribute tuple contains phrase lists for topics, skills, mathematical concepts, heuristics, and pitfalls, using standardized, question-independent terminology.Phrases are short, and the annotator is instructed to use as few phrases per dimension as possible.
  • B.2.2. Stage 2: Group Construction: Stage 2 grows distinct task groups from seed tasks by accepting pairs that share foundations and reasoning, avoid redundancy, introduce new concepts or skills, and follow curriculum direction.Soft-Jaccard similarity uses exact matches plus embedding-based one-to-one phrase matching; dependency-field indexing, fallback pools, randomized curriculum modes, and progression or similarity-band conditions support scalable, nondegenerate grouping.

B.3.1. Datasets … C. Additional Analyses

The paper evaluates SkillOS across interactive, shopping, and demanding reasoning benchmarks, comparing learned curation against memory-free, heuristic, and untuned alternatives. Effectiveness and efficiency are measured under controlled retrieval, execution, and decoding settings.

  • B.3.1. Datasets: ALFWorld contains six household task types across 120 rooms, with 3,553 training tasks and 140 valid_seen test tasks.Agents follow textual goals using high-level commands such as go to, take, open, heat, and put.
  • B.3.1. Datasets: WebShop comprises 1.18 million Amazon products and 12,087 instructions, partitioned into 10,587 training, 1,000 development, and 500 test instructions.Agents perform grounded shopping through search[query] and click[button] actions.
  • B.3.1. Datasets: Reasoning evaluation uses DeepMath-103K, AIME24, AIME25, and GPQA-Diamond, spanning difficult mathematics and specialist-authored science questions.DeepMath-103K includes approximately 103K problems, while AIME24 and AIME25 contain 30 problems each and GPQA-Diamond contains 198.
  • B.3.2. Baselines: All baselines use the same frozen Agent Executor, task suites, retrieval budgets, and decoding settings to isolate memory-mechanism contributions.The comparison includes five representative baselines spanning memory-free agents, memory-augmented methods, and internal SkillOS variants.
  • B.3.2. Baselines: The baselines distinguish no memory, heuristic experience distillation, heuristic procedural-memory management, and prompting-based SkillOS variants without RL fine-tuning.SkillOS-base uses the open-source backbone without RL, whereas SkillOS-gemini uses Gemini-2.5-Pro without GRPO optimization.
  • B.3.3. Evaluation Metrics: Evaluation covers task effectiveness and action efficiency using Success Rate, WebShop Score, Number of Steps, and reasoning Accuracy.Success Rate applies to ALFWorld and WebShop; WebShop Score provides dense partial-match rewards, while Steps measures actions per episode and Accuracy uses exact-match answers.
  • B.3.3. Evaluation Metrics: Each configuration runs with three independent random seeds, reports the mean with one standard deviation, and uses official held-out evaluation splits.Methods share top-k BM25 retrieval, maximum step budgets, frozen executors, and backbone-specific decoding temperatures.

C.1. Results on Gemini-3.1-Flash-Lite

SkillOS generalizes to the newer Gemini-3.1-Flash-Lite executor on ALFWorld, achieving stronger average effectiveness and efficiency than the evaluated baselines. Its learned curation policy also outperforms prompting-only and heuristic alternatives across subsets.

  • Evaluation setting: SkillOS is evaluated on ALFWorld with Gemini-3.1-Flash-Lite as the frozen executor across six subsets.The experiment tests whether SkillOS gains generalize to a newer model family.
  • Overall results: 73.1% average success rate surpasses ReasoningBank at 66.0% by +7.1 points and No Memory at 61.2% by +11.9 points, with 15.5 steps versus 18.5.SkillOS achieves the highest average success rate while requiring the fewest interaction steps.
  • Curation policy: SkillOS reaches 73.1%, ahead of SkillOS-gemini at 71.2% and SkillOS-base at 63.6%, showing task-level curator learning matters more than curator scaling or prompting alone.SkillOS-base remains barely above No Memory, while the stronger SkillOS-gemini curator still trails SkillOS.
  • Subset analysis: SkillOS wins four of six subsets, including Look at 84.6% versus 71.8% and Cool at 68.0% versus 48.0%, while MemP reaches only 58.6%.SkillOS-gemini wins Pick and ReasoningBank wins Heat, but SkillOS remains competitive on both.

C.2. Case Studies

SkillOS curates task-specific, compositional skills that encode reusable recovery workflows and branching reasoning paths. Compared with baseline and memory-free behavior, these skills are more specific, usable, and effective in guiding successful task completion.

  • Curated Skills for Different Tasks: For agentic tasks, SkillOS abstracts failure recovery into a reusable workflow and explicitly references existing skills, demonstrating compositional curation.The workflow proceeds from exhaustive search to confirming unavailability, identifying a substitute, and proceeding with it.
  • Curated Skills for Different Tasks: For reasoning tasks, SkillOS encodes branching-out reasoning by representing multiple solution paths for inradius–circumradius–semiperimeter relations.The skill relates the target distance to either the inradius or circumradius pathways.
  • How SkillOS Curates Better Skills Compared to Baselines: Compared with SkillOS-base’s generic recipe, SkillOS provides explicit constraints, equation setup, and a worked example tailored to the target sub-problem.The baseline omits explicit formulas, constraints, and examples, while SkillOS produces a concrete counting framework.
  • How Curated Skills Help to Solve Tasks Successfully: In the desklamp task, SkillOS completes the task by retrieving a strategy for interpreting “under” a lamp, whereas the memory-free baseline exhausts its step budget searching irrelevant containers.The agent first locates and picks up the CD near the desk, then inspects the correct location around the desklamp.

D. Limitations

SkillOS’s current design has limitations in retrieval, skill representation, and executor adaptation. Its simple keyword retrieval and single-file skills preserve tractability and modularity but constrain retrieval relevance and skill expressiveness, while freezing the executor limits adaptation to executor-specific needs.

  • Retrieval Mechanism: SkillOS relies on simple keyword-based retrieval, such as BM25, which may retrieve less relevant skills than dense, hybrid, or learned retrievers.The choice isolates skill curation, updating, and organization as the study’s primary focus.
  • Simplified Skill Representation: Each skill is represented as one Markdown file with YAML frontmatter and a Markdown body, simplifying the curator’s action space but discarding supporting scripts and external resource files.These discarded affordances could encapsulate executable procedures rather than purely declarative knowledge.
  • Frozen Agent Executor: Keeping the agent executor πL frozen isolates skill curation and supports modularity, but the curator can influence behavior only through content written into SkillRepo.The passage identifies potential miscalibration between curated skills and executor idiosyncrasies as a downside.

E. Future Research Directions

Future work centers on making SkillOS more adaptive, structured, and collaborative. Proposed directions include agentic retrieval, hierarchical skill composition, and shared memory across multiple agents.

  • Agentic Search over Experiential Memory: SkillOS could replace fixed top-k BM25 retrieval with agentic search that selects and injects relevant skill fragments at each decision step.This addresses retrieval challenges as SkillRepo grows across thousands of tasks and domains.
  • Hierarchical and Compositional Skills: SkillRepo could evolve beyond flat Markdown entries by supporting hierarchical decomposition, linking, composition, and abstraction of skills.The proposed structure reflects competence in which high-level procedures invoke lower-level sub-skills and primitive operations.
  • Multi-Agent and Shared Memory: Future multi-agent extensions must address conflicting curation decisions, shared-skill credit assignment, specialization, and cross-agent transfer.SkillOS currently treats memory as a single agent’s private artifact, while realistic deployments may involve parallel agents.
Loading 2605.06614v1…