PaperlayerView exact revision on arXiv

Source-linked AI summary

CoSkill: Joint Reinforcement Learning of Reasoning and Meta-Skill Agents for Hierarchical Skill Evolution

Jinyuan Feng, Dongmin Li, Yiqun Chen, Yang Gao, Xing Chen, Huimu Wang, Zhiqiang Pu

arXiv:2609.04865v1cs.AI

TL;DR

Existing skill-evolution approaches either separate skill updates from policy optimization or use fixed meta-skill workflows, limiting skill–policy co-adaptation. CoSkill makes the meta-skill a learnable agent jointly trained with a Reasoning Agent over a hierarchical skill library, and it outperforms prior baselines on ALFWorld and WebShop while improving efficiency.

  • Problem

    Existing methods decouple skill evolution from policy optimization or use fixed meta-skill workflows, limiting flexible skill evolution and co-adaptation with the reasoning agent.

  • Method

    CoSkill jointly trains a learnable Meta-Skill Agent and a Reasoning Agent sharing one backbone over a hierarchical skill library, using task and child step skills with outcome-aligned feedback.

  • Results

    98.4% average success on ALFWorld (+3.5 pp over RetroAgent) and 90.6% success on WebShop (+6.2 pp over the strongest baselines), with superior early-stage sample and wall-clock efficiency.

  • Takeaways & Limitations

    Skill evolution can be treated as a learnable process co-optimized with reasoning rather than externally orchestrated.

Abstract

from arXiv · show

Skill libraries improve the sample efficiency of agentic reinforcement learning (RL) by enabling large language model (LLM) agents to reuse procedural knowledge. Yet existing paradigms exhibit structural shortcomings: they either decouple skill evolution from policy optimization or instantiate meta-skills as fixed workflows. Both treat skills as passive objects to be managed, limiting the flexible evolution of skills and their co-adaptation with the reasoning agent. To address the limitations, we propose CoSkill, a unified multi-agent RL framework that recasts the static meta-skill workflow as a learnable Meta-Skill Agent and jointly trains it with a Reasoning Agent over a hierarchical skill library. By modeling the Reasoning and Meta-Skill Agents as a cooperative team sharing a single backbone, CoSkill enables end-to-end co-adaptation: the Reasoning Agent conditions its actions on a retrieved task skill and step skills selected from its child set, while its task performance guides the Meta-Skill Agent in refining those step skills. Experiments on ALFWorld and WebShop show that CoSkill substantially outperforms prior skill-based and RL baselines, achieving success rates of 98.4% and 90.6%, respectively (+3.5 and +6.2 pp). As shown in Figure 1, CoSkill achieves superior early-stage sample efficiency, asymptotic performance, and wall-clock efficiency. Our code is available at https://github.com/jinyuan-cookie/CoSkill.

1 INTRODUCTION

Existing skill-evolution methods either decouple library updates from policy learning, optimize library operations only coarsely, or rely on fixed workflows that cannot co-adapt with reasoning policies. CoSkill instead jointly trains Reasoning and Meta-Skill Agents over a hierarchical library for fine-grained skill–policy co-evolution.

  • Skill libraries improve long-horizon agentic RL by letting LLM agents reuse procedural knowledge, but existing methods struggle to coordinate skill optimization with reasoning-policy updates.
  • External orchestration delegates skill generation, revision, and maintenance outside policy learning, risking stale, redundant, or policy-misaligned libraries.
  • RL-based library management adapts lifecycle decisions using downstream rewards but treats each skill as atomic, leaving internal procedures unoptimized.
  • Fixed meta-skill workflows optimize individual procedures while freezing the reasoning agent, preventing co-adaptation between skill updates and an evolving policy.
  • CoSkill jointly trains Reasoning and Meta-Skill Agents sharing one LLM backbone, using retrieved task and child step skills plus outcome feedback for fine-grained co-evolution.

2 PRELIMINARIES

The agentic RL setting models an LLM policy solving natural-language tasks through sequential observation, action, and reward interactions. CoSkill augments this setting by hierarchically retrieving task and step skills and jointly training a learnable Meta-Skill Agent with the Reasoning Agent.

  • An LLM policy πθ solves natural-language tasks x sampled from D through sequential interaction with an environment.
  • At each step, the agent receives a textual observation, takes an action from A, and retrieves relevant skills from library B using the task, history, and current observation.
  • The reasoning policy is trained to maximize expected discounted return under the skill library B.
  • CoSkill hierarchically retrieves task and step skills while jointly training a learnable Meta-Skill Agent with the Reasoning Agent for skill–policy co-evolution.

3 METHOD

CoSkill jointly trains a Reasoning Agent and a learnable Meta-Skill Agent over a hierarchical skill library, using shared parameters and delayed verification to co-adapt reasoning and skill edits.

  • Framework overview: CoSkill jointly trains Reasoning and Meta-Skill Agents with shared parameters over a hierarchical skill library.The Meta-Skill Agent replaces a fixed workflow and proposes structured edits to step skills.
  • Hierarchical skill library: The library retrieves an episode-level task skill globally, then selects observation-dependent step skills exclusively from that task skill’s children.This preserves global guidance while restricting local procedures to a task-relevant subtree.
  • Shared-parameter interaction: The shared MSMDP lets the Meta-Skill Agent observe reasoning transitions and propose edits while preserving the baseline trajectory through staged updates.Edits are maintained in a private bundle until the task attempt ends and verification determines whether they can be promoted.
  • Skill verification reward: Post-edit reward measures mean improvement across verification attempts relative to the original bundle, with baseline subtraction controlling task difficulty and aggregation reducing rollout variance.Only valid, nontrivial edits that improve verification success become promotion candidates, and at most the top Kprom versions per task group are promoted.
  • Joint optimization: GiGPO jointly updates the shared actor using role-specific advantages, comparing reasoning returns separately from skill-edit improvements under matched task and skill conditions.The resulting update lets reasoning exploit evolving procedures while skill editing is trained against value confirmed by subsequent execution.

4 EXPERIMENTS

CoSkill is evaluated on long-horizon ALFWorld and WebShop tasks, where it outperforms prompting, reinforcement-learning, and skill-augmented baselines. Ablations and learning-dynamics analyses attribute its gains to joint optimization, hierarchical skills, frequent coordination, and environment-aligned Meta-Skill Agent updates.

  • Experimental Setup: Experiments cover ALFWorld and WebShop, comparing CoSkill with closed-source LLM, prompting, experience-based, RL, and skill-augmented baselines.ALFWorld reports task-type and macro-average success rates; WebShop reports task score and success rate.
  • Overall Performance: 98.4% ALFWorld average success and 90.6% WebShop success rank CoSkill first, exceeding strongest baselines by 3.5 and 6.2 percentage points.CoSkill also reaches 95.9 WebShop task score and 100% success on five of six ALFWorld task types.
  • Multi-Agent Co-Training: CoSkill improves over skill-free GiGPO by 7.6 pp on ALFWorld and 17.8 pp on WebShop, while surpassing SkillRL and D2Skill variants across both metrics.The comparisons use a shared Qwen2.5-7B backbone for CoSkill, whereas D2Skill uses external skill editors.
  • Fine-Grained Step-Skill Optimization: Compared with Skill1, CoSkill improves ALFWorld average success by 4.7 pp and WebShop success by 15.6 pp through fine-grained step-skill optimization.Skill1 treats each skill as an atomic library entry, whereas CoSkill assigns execution feedback to the active child step skill.
  • Ablation Study: 70.31% success at Step 20 exceeds all ablations by 7.81–14.06 pp, while removing Meta-Skill Agent RL lowers Step-20 success to 57.81% and final convergence to 92.19%.The ablation retains the same editor and verification mechanism, linking the gap to RL-based alignment of skill edits with downstream feedback.
  • Ablation Study: Hierarchical organization and frequent co-training improve learning stability: flattening reduces Step-20 success by 14.06 pp, while sparse updates trail CoSkill by up to 25.00 pp at Step 60.The hierarchy preserves task-level guidance while child step skills provide stage-specific guidance; frequent updates avoid stale cross-role feedback.
  • Meta-Skill Agent Learning Dynamics: With RL, editing converges toward 97–99% UPDATE, cumulative promotions fall 47.6% to 387, retrieval reaches approximately 100%, and empty bundles decline to 0.2%.Without RL, UPDATE falls to 54%, DELETE rises to 36%, promotions reach 739, retrieval deteriorates, and empty bundles reach 12.4%.

5 CONCLUSION

The paper concludes that multi-agent co-training can improve efficiency and performance in skill-augmented agentic RL, while positioning skill evolution as a process to co-optimize with reasoning. It also reports code availability and auxiliary AI use and ethics statements.

  • Multi-agent co-training jointly improves sample efficiency, asymptotic performance, and training efficiency in skill-augmented agentic RL.
  • Execution-feedback optimization yields more selective, stable, and policy-adaptive skill evolution than relying solely on powerful closed-source LLMs.
  • The paper positions skill evolution as a learnable process that should be co-optimized with reasoning rather than externally orchestrated.
  • Code is available at the project’s GitHub repository.

A RELATED WORK

Prior work differs mainly in which component receives the learning signal: the reasoning policy, the skill-library lifecycle, or individual skill content. These paradigms progressively move optimization toward skills but retain limitations in coordination or adaptability.

  • Across these paradigms, the learning signal targets either the reasoning policy, the library lifecycle, or individual skill content.
  • Externally Orchestrated Skill Evolution: Externally orchestrated methods treat skills as external knowledge and do not jointly train the policies that write and use them.
  • RL-Optimized Skill-Library Management: RL-optimized library management uses downstream rewards to control skill generation, selection, reuse, distillation, and retention.
  • Meta-Skill-Driven Skill Optimization: Meta-skill-driven methods revise procedural artifacts through automatic text-space optimization and skill-centered content revision.

B OFFLINE CONSTRUCTION OF THE HIERARCHICAL SKILL LIBRARY

Offline construction converts task trajectories into hierarchical task and step skills using environment-specific prompts, outcome-based evidence selection, and strict grounding and insertion rules. The resulting bundles support later meta-skill editing during interaction.

  • The initial libraries contain 300 task skills and 1,625 linked step skills for ALFWorld, and 300 task skills and 1,406 linked step skills for WebShop.
  • The fixed external LLM collects eight rollouts per task group using task, recent interaction history, observations, and admissible actions.
  • Trajectory groups provide successful and failed evidence, from which a reflection agent generates one end-to-end task skill and one to eight state-conditioned step skills.
  • Generated step skills must cite a listed trajectory and source turn, use grounded observations as retrieval keys, and describe corrected decisions for failures.
  • Only valid JSON bundles containing one task skill and at least one grounded child step skill enter the library; duplicates and invalid skills are discarded.
  • During baseline attempts, the Meta-Skill Agent proposes INSERT, UPDATE, DELETE, or KEEP edits using action, feedback, history, and active-skill context.

D ALGORITHM PSEUDOCODE

The pseudocode separates offline hierarchical-library initialization from the online CoSkill loop. Training stages skill edits, verifies their effects, promotes effective bundle versions, and jointly updates the shared policy.

  • Algorithm 1: Offline initialization: Offline initialization collects labeled expert trajectories, generates task skills with grounded child step skills, verifies source turns, and discards invalid or duplicate children.
  • Algorithm 1: Offline initialization: A valid task bundle enters the initial library only when its task skill and at least one grounded child step skill remain after filtering.
  • Algorithm 2: Core CoSkill loop: The online loop retrieves top-1 task and child step skills, runs the Reasoning Agent, stages Meta-Skill edits, verifies them, and promotes effective candidates.
  • Algorithm 2: Core CoSkill loop: CoSkill jointly updates the shared policy with role-specific advantages using clipped PPO and KL regularization, while updating utilities and pruning stale bundles.
  • Algorithm 2: Core CoSkill loop: The Meta-Skill edit sequence receives terminal delayed reward ∆skill_i, and only edits whose verification success exceeds the baseline become promotion candidates.

E BASELINE DETAILS

The evaluation compares prompting, memory, policy-optimization, skill-augmented, and closed-source agents under task-specific benchmark protocols. Baselines differ in whether they use persistent skills, external memory, policy updates, and skill evolution.

  • Baseline categories: Table 1 organizes baselines into prompting and experience-based agents, RL-trained agents without skills, RL-trained skill-augmented agents, and closed-source LLM agents.The comparison retains each method’s original prompts, memory modules, and training objectives.
  • Evaluation protocol: ALFWorld reports task-type success rates and macro averages, whereas WebShop reports task score and success rate.D2Skill-based Qwen2.5-7B evaluations train for 160 steps, validate every five steps on 128 held-out tasks, and freeze the skill bank during validation.
  • Prompting and experience-based agents: Prompting and experience-based baselines use unmodified or instructed backbones without task-specific RL updates or persistent external skill libraries.ReAct adds textual reasoning traces, while Reflexion, Mem0, and ExpeL use feedback, memory, or distilled insights without updating the base policy.
  • RL-trained agents without skills: RL-trained baselines without skills optimize policies using PPO, RLOO, GRPO, or GiGPO without persistent skill stores.These methods isolate policy optimization without persistent skills.
  • RL-trained skill-augmented agents: RL-trained skill-augmented baselines evolve, retrieve, compress, or co-evolve reusable memories and skills alongside policy learning.Examples include EvolveR, Mem0, SimpleMem, D2Skill, Skill1, SkillRL, and RetroAgent.

F IMPLEMENTATION DETAILS

CoSkill uses a shared actor backbone and jointly optimizes its Reasoning and Meta-Skill Agents, with environment-specific training configurations reported separately. The implementation table highlights differences between ALFWorld and WebShop in schedules, context limits, and data splits.

  • Joint optimization: Both agents share the same actor backbone and are jointly optimized in each global update.Environment-specific rollout and context limits are reported separately.
  • Environment configurations: Table 3 summarizes environment-specific training schedules, context limits, and data splits for ALFWorld and WebShop.The two environment columns highlight these implementation differences.
  • Training cost: Training CoSkill for 160 steps on ALFWorld with eight mainstream GPUs took approximately 66 hours, or roughly 526 GPU-hours.This reports the wall-clock and aggregate GPU cost of the ALFWorld training run.

G ADDITIONAL ABLATION RESULTS

The ablation study separates the effects of Meta-Skill Agent reinforcement learning, hierarchical retrieval, and update coordination. Variants retain the full model’s backbone, tasks, rollout protocol, and evaluation procedure unless otherwise stated.

  • Full model: CoSkill jointly optimizes the Reasoning and Meta-Skill Agents over the complete hierarchical skill library.This is the full-model configuration used as the reference variant.
  • Meta-Skill Agent RL: Removing Meta-Skill Agent RL preserves its edit actions and verification but eliminates reinforcement learning for edit trajectories.The Reasoning Agent remains trainable in this ablation.
  • Hierarchical library: Removing the hierarchical skill library disables task-skill/step-skill hierarchy and parent-conditioned step-skill retrieval.The remaining reasoning and skill-editing pipeline is retained.
  • Update coordination: Alternating Updates = 10 and Alternating Updates = 20 switch between reasoning and meta-skill optimization after 10 or 20 optimization steps.These variants isolate less frequent coordination between the two learning processes.
  • Training curves: Figure 6 plots smoothed ALFWorld success-rate trends for CoSkill and four ablations, with selected checkpoint values reported in Table 2.A horizontal dashed line marks the 80% success-rate threshold.

H CASE STUDIES

The case studies show how Meta-Skill Agent RL and verification support constraint-preserving, branch-complete, and reusable skill revisions. Without Meta-Skill Agent RL, locally plausible observations can trigger globally harmful edits such as relaxing constraints, deleting search branches, or removing temporarily inactive procedures.

  • Scope: The case studies are mechanistic diagnoses rather than independent statistical replicates, and stored skill text is reproduced verbatim.This limits how broadly the case-study findings should be generalized.
  • Preserving a task-critical cleaning constraint: In the cleaning case, Meta-Skill RL routes the tomato to the sink and preserves the delayed cleaning constraint, whereas the no-RL editor takes a countertop detour.The RL-trained editor sharpens the trigger while the no-RL editor makes a global procedural change inconsistent with the task constraint.
  • Preserving a task-critical cleaning constraint: Post-edit verification promotes the constraint-preserving revision and rejects the alternative because it shows no success improvement.The promoted skill retains the sinkbasin constraint, while the rejected edit is not written to the persistent library.
  • Completing a missing search branch: In the missing-search case, RL preserves a reusable branch that treats absent eggs as evidence for continued search rather than global irrelevance.The no-RL editor confuses a false current applicability condition with deletion of the broader procedure.
  • Completing a missing search branch: Verification promotes the branch-complete bundle with success rate 1.0, while the no-RL editor ends with an empty step bundle that is rejected.The trace attributes this difference to outcome-aligned editing and verification.
  • Retaining reusable procedures under repeated subgoals: In the repeated-subgoal case, RL retains a high-utility deposit procedure that is temporarily inactive after the first soapbar but remains needed for the second.The no-RL editor deletes the step as apparently redundant, whereas RL retains the reusable five-procedure hierarchy.
Loading 2609.04865v1…