PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillForge: Evolving Verifiable Skills for Reinforcement Learning Agents

Shidong Yang, Ziyu Ma, Tongwen Huang, Xucong Wang, Renda Li, Yiming Hu, Yong Wang, Xiangxiang Chu

arXiv:2608.24747v1cs.CL

TL;DR

RL-trained agents often fail to accumulate reusable knowledge, and append-only skill banks do not verify whether stored skills remain effective. SkillForge explicitly tracks skill invocation, verifies and refines skills through interaction, and induces new skills through multiple pathways. Across ALFWorld, WebShop, and AppWorld, it consistently outperforms SKILLRL, including a reported 6.3% average improvement.

  • Problem

    RL-trained agents remain episodic, while append-only skill banks lack verification of whether stored skills remain effective.

  • Method

    SkillForge explicitly records skill calls so RL optimizes actions and invocations, while evidence-based verification and multi-pathway induction evolve the skill bank.

  • Results

    SkillForge consistently outperforms SKILLRL across ALFWorld, WebShop, and AppWorld, improving performance by 6.3% on average.

  • Takeaways & Limitations

    Continuously verifying and refining skills supports more adaptive, skill-driven reinforcement-learning agents in open-ended environments.

  • Takeaways & Limitations

    SkillForge depends on the teacher LLM’s capability, may incur retrieval overhead as the bank grows, and adds tokens that can increase inference cost.

Abstract

from arXiv · show

Large language model (LLM) agents are trained with reinforcement learning (RL) for complex decision-making tasks. However, most RL-trained agents remain episodic and cannot accumulate reusable knowledge across episodes. Recent skill-based approaches, such as SkillRL, attempt to address this issue by extracting skills from raw trajectories, but treat the skill bank as an append-only repository without verifying whether stored skills remain effective. In this paper, we propose SkillForge, a framework for continuous skill evolution that enables skills to be verified and refined through environment interaction. By making skill usage explicit during agent interaction, RL can directly optimize both environment actions and skill invocation decisions. SkillForge further introduces evidence-based skill verification and multi-pathway skill induction, allowing the skill bank to continuously grow while maintaining its quality. Extensive experiments on ALFWorld, WebShop, and AppWorld show that SkillForge consistently outperforms SkillRL, demonstrating the effectiveness of continuously verified skills in training stronger LLM agents.

1 Introduction

RL-trained agents often remain episodic, while existing skill-based methods add reusable skills without verifying their ongoing usefulness. SkillForge makes skill use explicit and continuously verifies, refines, and expands the skill bank, outperforming SKILLRL across three benchmarks.

  • RL-trained agents often rediscover effective behaviors because they do not retain reusable knowledge across episodes.
  • Existing memory methods store long, redundant, noisy trajectories, whereas skill-based methods distill experience into compact, reusable decision principles.
  • Append-only skill banks lack observable usage, effectiveness attribution, and quality control for stored skills.
  • SkillForge enables explicit skill invocation, allowing RL to jointly optimize environment actions and skill-calling decisions.
  • Evidence-based verification reviews underperforming skills, while multi-pathway induction adds skills from successful trajectories, failed attempts, and contrastive outcome analysis.
  • 6.3% average improvement over SKILLRL was observed across ALFWorld, WebShop, and AppWorld without cold-start initialization.

2 Related Work

Related work frames LLM agents as systems for reasoning, acting, interaction, collaboration, and tool use, while skill learning offers a structured form of experience abstraction. Skill learning in open-ended agent environments remains underexplored.

  • LLM Agents: LLM-agent frameworks support interleaved reasoning, self-reflection, multi-agent collaboration, and tool use.
  • SkillForge Framework: SkillForge retrieves skills into a compact catalog, supports explicit calling during rollout, and evolves the skill bank from interaction trajectories.
  • Skill Learning for Agents: Skill learning abstracts experience into reusable skills and connects to continual learning and reinforcement learning.
  • Skill Learning for Agents: Skill learning in open-ended agent environments remains underexplored.

3 Method

SkillForge maintains a continuously verifiable skill bank alongside the policy during RL training. It retrieves and explicitly invokes skills, jointly optimizes actions and invocations, and evolves skills through induction, verification, and refinement.

  • Framework overview: SkillForge maintains a continuously verifiable skill bank alongside the agent’s policy during RL training.The framework combines skill representation, retrieval, explicit calling, policy optimization, and skill-bank updates.
  • Skill representation: Each skill is a structured unit with a callable title, intent, decision principle, applicability conditions, category, and revision status.The full skill content is accessed after explicit invocation, keeping the skill representation self-contained.
  • Skill retrieval: Embedding retrieval selects top-K relevant skills, whose titles and one-line intents form a compact catalog appended to the system prompt.Full principles and applicability conditions are revealed only after explicit skill calling.
  • Explicit skill calling: During rollout, the agent emits an optional structured skill-call tag alongside environment actions, making invocations discrete, observable, and attributable.The framework resolves the called name against the catalog and returns the skill’s intent, principle, and applicability conditions in the next observation.
  • Policy optimization: Because skill-call tags are generated tokens, GRPO jointly optimizes environment actions and skill-invocation decisions.Trajectories are scored by binary outcome rewards and optimized using group-relative advantages.
  • Multi-pathway induction: SkillForge induces new skills through extraction, refinement, or contrastive analysis, then deduplicates them before adding them to the bank.The pathway depends on whether successful and failed trajectory abstractions are available.
  • Evaluation: Table 1 reports performance on ALFWorld, WebShop, and AppWorld using success rate, score, TGC, and SGC metrics.The table reports ALFWorld subtask and overall success rates, WebShop score and success rate, and AppWorld Task Goal Completion and Scenario Goal Completion.
  • Evidence-based verification: Skill verification tracks per-skill usage and outcomes, prioritizing heavily used low-success skills for LLM-based keep-or-revise decisions.Revision rewrites the principle and applicability conditions; periodic updates make the revised bank available to subsequent rollouts.

4 Experiments

Experiments evaluate SkillForge across ALFWorld, WebShop, and AppWorld, showing consistent gains, component contributions, skill-bank evolution, transferability, and training advantages.

  • Experimental Setup: SkillForge is evaluated on ALFWorld, WebShop, and AppWorld using task-appropriate success, score, TGC, and SGC metrics.ALFWorld reports overall success rate, WebShop reports average score and success rate, and AppWorld reports TGC and SGC.
  • Main Results: 93.6 on ALFWorld and 89.8/83.0 on WebShop show SkillForge improving over GRPO by +16.0 and +10.5 score (+16.9 success), respectively.These results use Qwen2.5-7B.
  • Baseline Comparison: SkillForge improves over SkillRL by +3.7 on ALFWorld and +10.3 WebShop success, while AppWorld TGC/SGC rises from 19.0/5.36 to 23.8/14.3.The AppWorld SGC increase is described as nearly tripling.
  • Scaling Across Models: Qwen3-4B reaches 87.9 ALFWorld and 84.0 WebShop success, while Qwen3-30B-A3B reaches 94.3 ALFWorld and 59.5 AppWorld TGC.The results indicate strong performance across model scales.
  • Ablations: Removing explicit calling or the skill bank causes ALFWorld performance to drop from 87.9 to 77.9 and 79.3, while deduplication reduces AppWorld TGC from 44.6 to 38.7.Effectiveness tracking and LLM reflexion also improve performance, with effectiveness tracking contributing more substantially.
  • Skill Bank Evolution: Active skills grow from 44 to 90 on ALFWorld and from 44 to 86 on AppWorld as general and task-specific skills expand and unreliable skills are revised.The skill bank therefore grows steadily while remaining controlled.
  • Transferability: Later-stage skill banks outperform earlier ones on ALFWorld, 32.9 versus 27.9, and AppWorld, 31.5 versus 27.4, when transferred without further training.Qwen3-4B step-80 skills reach 31.5 on AppWorld versus 30.4 for Qwen3-30B-A3B’s self-evolved bank.
  • Training Dynamics: SkillForge initially lags behind the no-explicit-calling variant but then achieves faster convergence and higher final performance as induction and verification compound.The embedding analysis likewise shows newly induced skills expanding into previously unexplored regions over training.

5 Conclusion

SkillForge evolves skills continuously by making their use explicit, verifying them through environment interaction, and supporting evidence-based induction. Across ALFWorld, WebShop, and AppWorld, it consistently outperforms SkillRL without SFT initialization.

  • 5 Conclusion: SkillForge makes skill usage explicit so reinforcement learning can optimize environment actions and skill invocation decisions.It also introduces evidence-based skill verification and multi-pathway skill induction.
  • 5 Conclusion: SkillForge maintains skill quality as its skill bank grows through continuous verification and refinement.
  • 5 Conclusion: SkillForge consistently outperforms SKILLRL on ALFWorld, WebShop, and AppWorld without requiring an SFT initialization stage.

Limitations

SkillForge’s skill evolution pipeline depends on an external teacher LLM and may incur growing retrieval and inference costs as skills and explicit invocations accumulate.

  • Limitations: Skill induction and verification depend on an external teacher LLM, so skill-bank quality depends on the teacher model’s capability.
  • Limitations: Continuous skill induction may grow the skill bank over long training runs, introducing additional retrieval overhead.
  • Limitations: Explicit skill calling adds interaction tokens, potentially increasing prompt length and inference cost in large-scale deployments.

A.1 Dataset

The evaluation uses ALFWorld for long-horizon household tasks, WebShop for interactive e-commerce, and AppWorld for multi-step digital-service interactions.

  • A.1 Dataset: AppWorld simulates digital-service interactions through Python APIs across calendaring, email, music streaming, and social platforms.It reports Task Goal Completion and Scenario Goal Completion, measuring per-task capability and consistency across three scenario variants.
  • A.1 Dataset: ALFWorld evaluates textual agents on long-horizon household tasks in partially observable rooms.Success requires completing the full goal, with per-subtask and overall average success rates reported.
  • A.1 Dataset: WebShop simulates e-commerce requests using search[query] and click[element] actions.Evaluation uses the attribute-matching score between the selected product and the user’s request.

A.2 Baselines

The baselines span closed-source LLMs and prompt-based or memory-based agents that rely on in-context prompting or external memory without parameter updates.

  • A.2 Baselines: The comparison includes GPT-4o and Gemini-2.5-Pro as closed-source LLM baselines representing general-purpose reasoning capabilities.
  • A.2 Baselines: Prompt-based and memory-based baselines include ReAct, Reflexion, Mem0, and SimpleMem.These methods rely on in-context prompting or external memory without parameter updates.

A.3 Implementation Details

Implementation uses GRPO with VeRL, Qwen instruction models, bounded rollouts, and embedding-based retrieval with fixed skill-selection and verification settings.

  • Training setup: GRPO training uses the VeRL framework.The detailed hyperparameters are summarized in Table 4.
  • Training setup: Qwen2.5-7B-Instruct and Qwen3-4B-Instruct run on one 8× H20-GPU node, while Qwen3-30B-A3B-Instruct uses two such nodes.Tensor parallelism is 1 on one node and 2 across two nodes, respectively.
  • Rollout settings: Rollouts are truncated at 15 environment steps for WebShop and ALFWorld and 30 for AppWorld, with longer trajectories counted as failures.
  • Skill retrieval and verification: Each episode retrieves the top 6 general skills and top 6 task-specific skills using embedding-based retrieval.Evidence-based verification uses EMA smoothing α = 0.1 and usage half-life h = 20.

A.4 Training Procedure for SKILLFORGE

SKILLFORGE alternates skill-augmented rollouts with policy optimization while maintaining skill statistics and periodically inducing, verifying, and revising skills from successful and failed trajectories.

  • Training loop: Training alternates between skill-augmented rollouts and policy optimization.Retrieved task-relevant skills condition rollouts, and collected trajectories update policy parameters via Eq. (5).
  • Training loop: At each rollout step, the agent retrieves a small relevant skill subset and records skill-calling events for online effectiveness statistics.These statistics support subsequent skill-bank maintenance.
  • Skill-bank maintenance: Every I steps, trajectories are abstracted and partitioned into successful T + and unsuccessful T − sets for skill-bank maintenance.The algorithm also updates per-skill statistics from calling events and deduplicates newly induced skills before insertion.
  • Skill induction: An expert LLM induces candidate skills from both successful and unsuccessful trajectories, capturing effective behaviors and fixes for failure modes.New candidates are deduplicated before insertion to control redundancy.
  • Verification and refinement: Flagged skills enter reflexion, where the expert keeps them when evidence is insufficient or noisy and revises them when stale or misleading.The updated bank is immediately used in later rollouts, creating a continuous generate-test-refine cycle.

A.5 Formal Skill Definition and Representation

SKILLFORGE defines skills as callable, environment-verifiable decision units and contrasts this explicit verification design with SkillRL across conceptual, teacher-sensitivity, efficiency, and case-study evidence.

  • A.5 Formal Skill Definition and Representation: A SKILLFORGE skill is a callable, environment-verifiable decision unit distilled from agent-environment interaction.Its structured representation includes a title, intent, principle, applicability, category, and status.
  • A.5 Formal Skill Definition and Representation: Explicit <skill_call> events make usage observable and link invocations to outcomes for per-skill success statistics, confidence scoring, and revision decisions.The agent initially sees only callable names and applicability triggers, then accesses full skill content on invocation.
  • A.6 Detailed Comparison with SkillRL: SkillForge records explicit skill calls for skill-level credit assignment, whereas SkillRL injects retrieved skills as prompt context without revealing which skill was relied upon.SkillRL also requires SFT initialization, while SkillForge uses RL-only training without SFT initialization.
  • A.6 Detailed Comparison with SkillRL: +3.7 on ALFWorld and +10.3 on WebShop success rate, while AppWorld TGC/SGC rises from 19.0/5.36 to 23.8/14.3 versus SkillRL.These results use the same Qwen2.5-7B-Instruct backbone.
  • A.7 Teacher Sensitivity Analysis: SkillForge improves over no-teacher GRPO on all three benchmarks with a self-teacher, and Qwen3-Max further raises performance on all benchmarks.The evidence indicates complementary contributions from framework design and teacher quality.
  • A.8 Efficiency Analysis: Total skill-related overhead remains below 10% across benchmarks; AppWorld adds 0.4h while the score rises from 34.5 to 44.6.SkillForge is faster than GRPO on ALFWorld and WebShop, and the authors associate the efficiency with reduced redundant exploration and shorter rollouts.
  • A.9 Case Study: Skill Calling in Action: Case studies show skills supporting procedural sequencing in ALFWorld, data-access scoping in AppWorld, and adaptive search strategy in WebShop.In ALFWorld, Open Then Heat prevents attempting to heat an object before opening the microwave.
Loading 2608.24747v1…