PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Dual-Granularity Skill Bank for Agentic RL

Songjun Tu, Chengdong Xu, Qichao Zhang, Yaocheng Zhang, Xiangyuan Lan, Linjing Li, Dong Li, Dongbin Zhao

arXiv:2603.28716v2cs.AI

TL;DR

Existing agentic RL skill methods emphasize trajectory-level guidance and lack principled evolving memory. D2Skill combines task- and step-level skills with joint policy-bank training and dynamic maintenance, consistently improving over strong baselines across interactive and search-augmented QA benchmarks.

  • Problem

    Existing skill-based agentic RL methods mainly provide trajectory-level guidance, while long-horizon learning also needs reusable knowledge for local errors and evolving skill memory.

  • Method

    D2Skill co-evolves task and step skills with the policy using paired baseline and skill-injected rollouts, hindsight utility estimation, reflection, retrieval, and pruning.

  • Results

    D2Skill consistently improves over strong baselines across interactive benchmarks and search-augmented QA, with ablations confirming the importance of dual-granularity modeling and dynamic bank maintenance.

  • Takeaways & Limitations

    Reusable experience benefits agentic RL when organized for both high-level guidance and fine-grained correction and maintained according to utility.

  • Takeaways & Limitations

    The study covers a small set of benchmark environments, and its strongest settings rely on external reflector models.

Abstract

from arXiv · show

Agentic RL can benefit substantially from reusable experience, yet existing skill-based methods mainly extract trajectory-level guidance and often lack principled mechanisms for maintaining an evolving skill memory. We propose D2Skill, a dynamic dual-granularity skill bank for agentic RL that organizes reusable experience into task skills for high-level guidance and step skills for fine-grained decision support and error correction. D2Skill jointly trains the policy and skill bank through paired baseline and skill-injected rollouts under the same policy, using their performance gap to derive hindsight utility signals for both skill updating and policy optimization. Built entirely from training-time experience, the skill bank is continuously expanded through reflection and maintained with utility-aware retrieval and pruning. Experiments on ALFWorld, WebShop, and Search-Augmented QA tasks show that D2Skill substantially improves performance over skill-free baselines across models of different scales. Further ablations and analyses show that both dual-granularity skill modeling and dynamic skill maintenance are critical to these gains, while the learned skills exhibit higher utility, transfer across evaluation settings, and introduce only modest training overhead.

1 Introduction

D2Skill addresses limitations in agentic RL by organizing reusable experience into task and step skills while dynamically managing the skill bank. It jointly trains policy and memory using hindsight utility and outperforms strong baselines across benchmark settings.

  • Agentic RL involves long-horizon, partially observable decision-making with sparse rewards, large action spaces, and difficult credit assignment.
  • Existing skill methods often emphasize trajectory-level reflection, leaving fine-grained interaction errors insufficiently addressed.
  • D2Skill jointly trains the policy and skill bank using paired skill-injected and non-injected rollouts to estimate hindsight skill utility.
  • D2Skill consistently outperforms strong skill-free and skill-augmented baselines on ALFWORLD and WEBSHOP and generalizes favorably to SEARCH-AUGMENTED QA.
  • D2Skill organizes reusable experience into task skills for high-level guidance and step skills for local interaction support.
  • The skill bank expands through reflection and uses utility-aware retrieval and pruning to retain more effective and efficient memory.

3. We evaluate D2Skill on ALFWORLD,

The evaluation covers ALFWORLD, WEBSHOP, and SEARCH-AUGMENTED QA using QWEN2.5-7B-INSTRUCT and QWEN3-4B-INSTRUCT. D2Skill consistently outperforms skill-free baselines, with ablations supporting its core design.

  • A representative ALFWORLD task requires cooling a potato and placing it in a garbagecan.
  • The evaluation includes ALFWORLD, WEBSHOP, and SEARCH-AUGMENTED QA tasks.
  • Experiments use QWEN2.5-7B-INSTRUCT and QWEN3-4B-INSTRUCT.
  • D2Skill consistently outperforms skill-free baselines across the evaluated settings.

2 Motivation

D2Skill is motivated by two complementary failure modes: global planning and local recovery. Task-level guidance alone cannot reliably correct local errors, while unmanaged skill memories can become redundant or ineffective.

  • 2.1 Local Errors Need Fine-Grained Skills: Task-level skills provide global guidance but may not correct local errors after an otherwise reasonable plan reaches a concrete decision point.
  • 2.1 Local Errors Need Fine-Grained Skills: Step skills provide observation-conditioned corrective guidance, such as switching locations after an empty-container search instead of repeating the failure.
  • Figure 3 examines training curves and compares skill-bank dynamics with and without management using average utility and retrieval statistics.
  • 2.1 Local Errors Need Fine-Grained Skills: The two skill granularities correspond to global planning and local recovery, the two dominant failure modes identified for agentic RL.
  • Static skill-bank reuse can waste retrieval and prompt budget because generated skills may be overly specific, redundant, or no longer useful.

3 Method

D2Skill jointly trains an agentic RL policy with a dynamic skill bank that provides task-level guidance and step-level error correction. It derives hindsight signals from paired baseline and skill-injected rollouts, while reflection, retrieval, utility updates, and pruning maintain the bank.

  • RL Training with Skill Injection: D2Skill pairs baseline and skill-injected rollouts under the same policy, using their performance gap for policy optimization and skill utility estimation.The rollout group is evenly divided into baseline and skill groups, with skill injection applied only to the latter.
  • Reflection-Driven Skill Generation: The skill bank stores task skills for high-level guidance and step skills for local error correction, generated by reflection on representative trajectories.Reflection is triggered for low-performing task groups and can produce at most one task skill and one step skill per group.
  • Skill Utility Updates: Task skills receive a shared task-level hindsight signal, whereas retrieved step skills receive trajectory-specific credit and update their utilities separately.Each skill utility is maintained with an exponential moving average; step-skill credit depends on the trajectory in which the skill appears.
  • Hindsight Optimization: Hindsight intrinsic rewards measure skill-group performance gain over the baseline and are added to skill-injected trajectory returns for policy optimization.The shaping strength is controlled by λ, and the additional reward is applied at the end of each skill-injected trajectory.
  • Skill Retrieval: Two-stage retrieval matches task or step query keys by cosine similarity, then ranks candidates using similarity and a utility-based exploration bonus.Only skills above the minimum similarity threshold are retained, and the top-k candidates are injected into the policy context.
  • Bank Management: Periodic utility-based pruning limits each skill pool, removes the lowest-scoring skills beyond capacity, and protects newly created skills during an evaluation period.Skills created within the protection window are excluded from eviction.

4 Experiments

D2Skill is evaluated across interactive and search-augmented QA benchmarks, model scales, training regimes, and component ablations. It consistently improves performance, with dual-granularity skills and dynamic skill management contributing to the gains while adding modest overhead.

  • Main Performance: D2Skill consistently outperforms skill-free baselines across ALFWORLD, WEBSHOP, and SEARCH-AUGMENTED QA.The evaluation compares standard RL and prior memory-, skill-, and search-augmented methods across three agentic settings.
  • Main Performance: 90.6 overall success on ALFWORLD and 84.4 success on WEBSHOP are reached by the strongest D2Skill variants with Qwen2.5-7B-Instruct.For Qwen3-4B-Instruct-2507, ALFWORLD overall success rises from 53.9 with GRPO to 72.7.
  • Main Performance: 44.9 average accuracy is achieved on SEARCH-AUGMENTED QA with Qwen2.5-7B-Instruct, while Qwen3-4B-Instruct-2507 improves from 36.0 with Search-R1 to 42.8.The gains are especially clear on multi-hop datasets such as HOTPOTQA and 2WIKI.
  • Main Performance: 82.8 overall success on ALFWORLD and 81.3 success on WEBSHOP are reached in the Self setting without closed-source reflectors.The results suggest the core framework gain does not depend on privileged closed-source supervision, although stronger external reflectors can further improve diagnosis and abstraction.
  • Main Performance: 95.3 ALFWORLD performance after 120 steps exceeds GRPO under the same budget, while 92.2 after 40 steps is already close to GRPO trained for 120 steps.D2Skill also remains better than GRPO on WEBSHOP under the same training budget and works with teacher-initialized SFT policies.
  • Ablation Study: Removing either task skills or step skills reduces performance, while removing skill management causes the larger degradation in the ablation study.Removing the baseline group or utility estimation produces smaller but clear drops, consistent with roles in credit assignment, skill valuation, optimization, and retrieval quality.
  • Additional Analysis: Utility-aware skill management yields higher-average-utility banks and retrieved skills, and D2Skill-trained policies remain competitive with or outperform GRPO without a skill bank at evaluation.Cross-bank evaluation also shows gains from Gemini-3-Flash-generated skills, while the self-generated bank is most effective.
  • Additional Analysis: 25.6 wall-clock training hours keeps D2Skill close to GRPO at 20.8 hours and below SkillRL at 49.2 hours on ALFWORLD with Qwen3-4B-Instruct-2507.Figure 6 indicates D2Skill reaches strong evaluation performance about 1.7× faster than SkillRL in practice.

5 Related Works

D2Skill extends memory-based agent evolution by valuing and maintaining reusable skills during RL, rather than only accumulating textual memories. It builds on external-memory and skill abstractions for guiding long-horizon decisions.

  • Prior work organizes accumulated experience into reusable lessons, strategies, workflows, and skills for agent adaptation beyond parameter updates.
  • D2Skill focuses on RL-time skill valuation and maintenance through paired skill/base rollouts rather than only accumulating reusable textual memories.
  • External memory complements policy optimization by storing and retrieving useful successful or failed experiences to guide future decisions.

6 Conclusion

D2Skill integrates task-level guidance, step-level correction, and utility-aware skill management in a joint training loop. Across interactive benchmarks and search-augmented QA, it improves over strong baselines, with ablations supporting both design components.

  • D2Skill integrates task-level guidance, step-level correction, and utility-aware skill management within a joint training loop.
  • Across interactive benchmarks and search-augmented QA, D2Skill consistently improves over strong baselines.
  • Ablations and analyses confirm the importance of both dual-granularity skill modeling and dynamic bank maintenance.
  • The study is limited to a small set of benchmark environments, and its strongest settings continue to rely on external reflector models.

Limitations

The study identifies limitations in skill construction, benchmark coverage, and skill-quality or retrieval robustness. These constraints leave stability under weaker supervision and performance in more open-ended environments for future work.

  • Skill construction quality depends on the supervision source and training setup used during reflection and initialization.
  • The evaluated benchmarks cover representative interactive and search-based settings but do not capture the full diversity and noise of open-ended real-world environments.
  • Generated-skill quality and utility estimates depend on the underlying policy and reflector, while noisy, overly specific, or outdated skills may affect retrieval and memory management.

Ethics Statement

The paper models agentic RL in long-horizon environments with textual, partially observable interaction contexts. Its framework uses a persistent skill bank to augment policy inputs and optimize decisions under skill-augmented observations.

  • The effective context combines the task specification with the most recent observation–action history, and the policy selects actions from this context.
  • Agentic RL is modeled as a history-augmented partially observable decision process because the policy receives a fixed-window textual context rather than the latent environment state.
  • A persistent skill bank retrieves relevant language guidance and adds it to the current context before policy action selection.
  • Under skill-augmented observations, the policy is optimized with the same objective as standard RL.

B Overall Training Algorithm

Algorithm 1 integrates paired rollouts, dual-granularity retrieval, hindsight-based optimization, and reflection-driven skill updates into one D2Skill training loop.

  • Algorithm 1 unifies paired rollout construction, dual-granularity skill retrieval, hindsight-based optimization, and reflection-driven skill updating.These components are integrated within a single training procedure.
  • The training loop jointly coordinates policy learning with ongoing skill-bank operations.
  • D2Skill’s procedure is presented as the detailed operational specification for the framework.

C.1 Benchmarks and Evaluation Protocol

The evaluation covers ALFWorld, WebShop, and Search-Augmented QA with held-out validation or test evaluation, multiple baselines, and skill-aware prompt variants. Implementation details specify training schedules, model initialization, environment constraints, and reflection-based skill construction.

  • Benchmarks and Evaluation Protocol: D2Skill is evaluated on ALFWorld, WebShop, and Search-Augmented QA using task-specific success, score, and test-performance measures.ALFWorld reports subtask and overall success rates; WebShop reports average score and success rate.
  • Benchmarks and Evaluation Protocol: Interactive benchmarks use 160 training steps, validation every 5 steps on 128 tasks, and best performance with the skill bank frozen during evaluation.Search-Augmented QA uses 200 training steps and test-set evaluation.
  • Baselines: Comparisons include origin models, skill-free GRPO, memory-augmented GRPO, SkillRL, and reference-only direct rollouts from strong closed-source LLMs.Search-Augmented QA additionally uses search-agent baselines including Search-R1, ZeroSearch, StepSearch, EvolveR, and Dr.Zero.
  • Prompts: Skill-augmented prompts add retrieved task-level and step-level experiences to otherwise aligned environment-facing benchmark prompts.Both baseline and skill-augmented layouts include task information, recent history, observations, and admissible actions.
  • Skill Construction: Reflection uses failed trajectories and, when available, successful trajectories to produce structured step-level and task-level skills.The reflector identifies the first error step and outputs one reflection at each granularity.
  • Related Methods: D2Skill treats memory as a non-parametric complement to policy optimization and uses skills as reusable abstractions for long-horizon tasks.Its dual-granularity design separates task-level planning guidance from step-level decision support.

D.4 Comparison with Contemporaneous Work

D2Skill differs from contemporaneous experience-driven RL methods in its skill granularity, retrieval frequency, and training-time skill management. Compared with SkillRL, it provides both task-level and step-level guidance without relying on privileged validation information for skill construction.

  • Comparison with Contemporaneous Work: RetroAgent and Complementary RL use self-evolving experience to improve agentic RL but rely on more elaborate retrospection and extraction pipelines.The paper notes that these pipelines may increase system complexity and prompt dependence.
  • Comparison with Contemporaneous Work: SkillRL distinguishes task types rather than skill granularities, retrieves task-level guidance once per task, and reuses it throughout the trajectory.
  • Comparison with Contemporaneous Work: D2Skill retrieves task skills and step skills at each interaction step, combining high-level guidance with fine-grained decision support.Its two-granularity design separates task-level planning from step-level action support.
  • Comparison with Contemporaneous Work: D2Skill generates and manages skills during training rather than relying on privileged validation information for skill construction.
Loading 2603.28716v2…