ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL

• Agentic RL policies lack systematic accumulation of transferable problem-solving strategies across tasks, motivating reusable modular skills.Source: §1 · p. 1
• Existing approaches adopt experiential knowledge without principled testing, risking conflict between externally produced skills and the evolving policy.Source: §1 · pp. 1–2
• RESKILL uses GRPO groups to compare competing skill versions under the same task and policy while Thompson Sampling balances exploration and exploitation as learning evolves.Each rollout supports policy optimization, failure diagnosis, skill revision, and bandit-based version evaluation without additional rollout budget.Source: §1 · p. 2
• RESKILL automatically creates, tests, refines, and prunes skills, outperforming memory- and skill-based baselines with the largest gains on unseen and hard tasks.The co-evolved policy can adapt to new domains through skill creation alone at test time.Source: §1 · p. 2

• An LLM agent interacts with an environment over multiple turns, conditionally loading triggered skills while maximizing expected trajectory reward.The policy generates actions from the interaction history, active skills, and tool engine.Source: §2 · p. 3
• GRPO samples multiple trajectories for each task from the current policy and optimizes using group-relative advantages.Source: §2 · p. 3
• All rollouts in a group share the task and policy but differ stochastically, providing a controlled setting for comparing skill versions.Source: §2 · p. 3

3 Reconciling Skill Creation with Policy Learning

• RESKILL integrates within-group skill testing, RL-in-the-loop skill creation, and Thompson Sampling into the GRPO training loop.The framework is organized around training with skill testing, candidate creation from experience, and reward-guided version evolution.Source: §3 · p. 3, Figure 2 · p. 4

3.1 RL Training with Within-Group Skill Testing

• Each rollout receives a skill-version assignment and is generated under the corresponding skill bank, enabling controlled comparison because group members share task and policy conditions.Varying skills within a group also builds the policy’s capacity to reconcile with new skills.Source: §3.1 · p. 3
• Every rollout simultaneously supplies GRPO learning signals, experience for skill creation, and posterior updates for accepting or rejecting skill versions.The total rollout count remains identical to standard GRPO.Source: §3.1 · p. 3

3.2 RL-in-the-Loop Skill Creation

• The skill creator replaces human examples with a continuously refreshed reservoir of labeled success-failure trajectories from ongoing RL training.The reservoir evolves with the policy and supports diagnosis, while conditional triggers load only situationally relevant skills.Source: §3.2 · p. 4
• Assertion-based grading estimates failure prevalence across the reservoir, and history-informed revision proposes ADD, MODIFY, and DELETE operations with trigger-coverage validation.Source: §3.2 · pp. 4–5

3.3 RL-Guided Skill Evolution with Thompson Sampling

• Thompson Sampling with adaptive discounting allocates rollouts between new and old versions while reducing the influence of stale evidence as the policy evolves.The memory parameter M is selected from completed evaluation cycles by sequential predictive-likelihood maximization, avoiding manual tuning.Source: §3.3 · p. 5
• Version assignments use posterior estimates of whether the new skill outperforms the old one, preserve minimum exploration, and shift allocation toward the better-performing version.At cycle boundaries, the candidate is accepted when its posterior expected success exceeds the old version’s; otherwise the system reverts.Source: §3.3 · pp. 5–6

4.1 Setup

• Experiments span embodied reasoning, agentic search, science discovery, coding, and web shopping across seen and unseen task configurations and multiple model scales.The evaluation compares RESKILL with base, memory-evolution, skill-only, and RL-based baselines.Source: §4.1 · p. 6

4.2 Main Results

• RESKILL outperforms the strongest memory and skill-augmented RL baseline by 5.9%/6.9% on ALFWorld and 2.5%/2.3% on Search at 4B/8B scales.It also exceeds vanilla GRPO by 3.0–14.5% across the evaluated benchmarks.Source: §4.2 · p. 8
• RESKILL achieves 95.3% on unseen ALFWorld tasks, a 12.7% gap over SKILLRL’s 82.6%, compared with a 1.2% seen-task margin.On Search, its unseen-split improvement over SKILLRL is about 2.1× larger than on seen splits.Source: §4.2 · p. 8
• Across ScienceWorld, InterCode-SQL, and WANDS, RESKILL’s advantage widens on harder or out-of-domain tasks, including 48.8% versus 26.9% on ScienceWorld OOD tasks.It also reaches 63.6% versus 55.3% on InterCode-SQL Extrahard and 86.6% versus 82.1% on WANDS 3-opt.Source: Figure 3 · p. 7, §4.2 · p. 8

4.3 Framework Analysis

• Untested skill injection collapses during ALFWorld training, reaching 62.7% and 45.3% on two benchmarks, while uniform allocation trails RESKILL at 46.0% versus 47.6% on Search.RESKILL tests competing versions under training conditions through Thompson Sampling, avoiding policy damage and wasted rollouts.Source: §4.3 · p. 8
• Replacing Claude 4.5 Sonnet with Qwen3-4B-Instruct for skill creation causes only a 4.8% ALFWorld and 0.5% Search drop while still outperforming existing RL baselines.This supports reconciled skill-policy co-evolution as the primary source of gains rather than creator capacity.Source: §4.3 · p. 9
• Joint co-evolution, adaptive weighting, and assertion-based diagnosis each improve performance, while conditional triggers keep context compact and general triggers raise training context length to 1.29× GRPO.Sequential co-evolution, stale or insufficient weighting, and per-episode diagnosis all underperform the full system.Source: §4.3 · p. 9
• With the ALFWorld policy frozen, RESKILL rapidly adapts its skill bank to ScienceWorld at test time, whereas baselines fail to evolve for the new domain.Its lifecycle builds foundational skills early, then prunes internalized skills and refines responses to subtler failures.Source: §4.3 · p. 9, Figure 5 · p. 9, §4.4 · p. 10

Prompt and Skill Optimization

• Prompt and skill optimization methods create or improve instructions and reusable workflows, but most operate offline or cannot integrate skill creation directly into reinforcement learning.Anthropic’s Skill Creator adds controlled blind testing but does not reconcile creation with policy training.Source: §5 · p. 10

Memory and Skill-Based Reinforcement Learning

• The paper positions RESKILL as integrating skill evolution with policy training rather than adopting externally produced knowledge without principled testing.This distinction targets potential conflict between skills and the evolving policy.Source: §5 · p. 10
• Memory- and skill-based RL methods evolve memories, principles, skills, or instruction candidates, yet treat skill creation and policy learning as separate processes.RESKILL instead uses within-group testing and reward-based selection to close this gap.Source: §5 · p. 10

• RESKILL embeds assertion-driven skill revision, within-group skill testing, and adaptive Thompson Sampling into the GRPO training loop at zero additional rollout cost.Source: §6 · pp. 10–11
• Across five benchmark domains, RESKILL consistently outperforms existing skill- and memory-based RL methods, with the largest gains on unseen tasks.Source: §6 · p. 11

A Limitations and Future Work

• Evolution frequency is a per-domain hyperparameter: infrequent evolution can cause fixed-context adaptation, later distribution shift, and training instability.Adaptive evolution scheduling is proposed as future work.Source: A Limitations and Future Work · p. 15
• Trigger-based skill loading avoids extra inference steps but gives the model less control over when skills are applied than explicit tool calls.Batched or amortized invocation is suggested to preserve rollout efficiency.Source: A Limitations and Future Work · p. 15
• Adding skills to context increases cognitive load, causing some smaller models to emit skill names as commands or over-apply skills when they are inapplicable.The paper suggests format-aware reward shaping or separating skill invocation from action generation.Source: A Limitations and Future Work · p. 15

B.1 Training Details and Hyperparameters

• All benchmarks use the same learning rate and KL loss type, while batch sizes and context lengths differ with environment characteristics.ScienceWorld uses longer contexts, while Search uses larger batches because its episodes are shorter.Source: §B.1 · p. 15

• For fair comparison, methods requiring language-model-generated skills, memories, or instructions use the same Claude Sonnet 4.5 API.Source: §B.2 · p. 16
• The experiments use ReAct and Claude Sonnet 4.5 as untrained reference models under identical evaluation protocols.Source: §B.2 · p. 16
• The baseline suite spans base models, memory-evolution methods, skill-only retrieval, and RL-based methods including GRPO, MEMRL (+RL), and EVOLVER.Source: §B.2 · p. 16
• MEMRL is evaluated both without policy updates and as MEMRL (+RL), where memory utility updates online while GRPO concurrently trains the policy.Source: §B.2 · p. 16
• Skill-only comparisons use SKILLRL’s fixed skill bank with BM25, HippoRAG, or full long-context loading, reporting the best strategy as “Best†.”Source: §B.2 · p. 16
• SKILLRL distills skills with a teacher LLM and permanently adds skills proposed from unaddressed failure patterns after validation epochs.Source: §B.2 · p. 17

B.3 Benchmark Details

• The evaluation spans ALFWorld, Agentic Search, WANDS, ScienceWorld, and InterCode-SQL, with seen/unseen or in-/out-of-distribution splits and task-specific success or accuracy metrics.The benchmarks vary in action spaces, horizons, reward definitions, training data, and held-out evaluation settings.Source: §B.3 · pp. 17–18

C.1 Pseudocode

• ReSkill alternates within-group new-versus-old skill rollouts, GRPO policy updates, reservoir collection, posterior evaluation, and periodic skill creation and acceptance.The algorithm samples skill versions using Beta posteriors, updates the policy from rollout rewards, and accepts a new version when its expected performance exceeds the old version’s.Source: §C.1 · p. 18

C.3 Skill Creator Pipeline: Full Details

• The skill creator contrasts diverse success and failure episodes across skill versions, then uses rule-based assertions and diagnosis to identify and cluster evolving failure modes.Assertions track action patterns, ordering, repetition, trajectory length, and environment states, while the diagnoser can add, modify, or delete assertions.Source: §C.3.1 · pp. 18–19, §C.3.2 · p. 19

C.3.3 Skill Recommendation and Authoring

• The recommender proposes ADD, MODIFY, or DELETE operations based on failure clusters, skill diagnostics, lifecycle stage, and evidence about contradictory or internalized guidance.The author converts recommendations into concise skill content with examples and general, beginning, or action-pattern triggers.Source: §C.3.3 · pp. 19–20

C.4.1 Trigger Types

• Deterministic triggers route skills by episode-wide, initialization, or previous-action conditions, while when_to_use descriptions provide a second in-context relevance filter.Action-pattern triggers use regular expressions, allowing guidance to activate only after matching behavior such as a search.Source: §C.4.1 · p. 20, §C.4.2 · p. 20

C.5 Adaptive Staleness Correction: Details

• Adaptive discounting reduces stale evidence as policy behavior changes, using wt = 1/(1 + nt/M) so memory shortens with larger allocation batches and lengthens with smaller ones.The memory parameter M controls the stability-adaptation tradeoff, while predictive likelihood estimates M online from completed A/B tests using grid search.Source: §C.5.1 · pp. 21–22, §C.5.2 · pp. 22–23

C.5.3 Empirical Validation

• After an approximately 50-step warm-up, online discount estimates closely track final estimates, confirming rapid convergence without separate calibration.Figure 8 compares incrementally estimated weights with weights computed from all data through step 250.Source: §C.5.3 · p. 23
• Thompson Sampling’s allocation creates anti-correlated discounting: the favored arm forgets more, while the under-sampled arm receives more memory.The new skill dominates allocation around step 30, whereas the old skill dominates around step 175, illustrating adaptive retention.Source: §C.5.3 · p. 23

C.6 Full Baseline Results

• Full non-RL baseline results are reported in Table 4 as mean±std over three random seeds, expanding Table 1’s per-cell best-value summaries.BM25, HIPPORAG, and LONG-CONTEXT use SKILLRL’s oracle skill bank, while REME and MEMRL evolve memories or procedures without policy training.Source: §C.6 · p. 23, Table 4 · p. 24
• Figure 9 evaluates sensitivity to evolution frequency K_e and skill-bank size on ALFWorld and Search, using K_e=5 and bank size 8 by default.The supplied passage identifies the evaluation scope and defaults but does not report the plotted numerical outcomes.Source: Figure 9 · p. 24

C.7 Sensitivity Analysis

• 89.8% at K_e=5 falls to 85.0% at K_e=20 on ALFWorld, while Search remains stable at 47.1–47.6% across settings.On ALFWorld, bank size also peaks at 8 with 89.8%.Source: §C.7 · p. 24
• Performance depends on the evolution-to-convergence ratio: limited cycles constrain refinement, whereas Thompson Sampling tests revisions before full adoption.ALFWorld provides six cycles at K_e=20, while Search provides twelve.Source: §C.7 · pp. 24–25

C.8.1 Contrastive Analyzer

• The contrastive analyzer compares same-task successes and failures to extract behavioral insights, failure points, and skill impacts.Its outputs include structured snippets, patterns, and confidence.Source: §C.8.1 · p. 25

C.8.2 Assertion Diagnoser

• The assertion diagnoser evolves rule-based checks, quantifies failures, and clusters related insights using pass rates and active skills.It returns assertion operations, diagnoses, and representative insight groups.Source: §C.8.2 · p. 25

C.8.3 Skill Recommender

• The skill recommender proposes evidence-linked add, modify, or delete operations, while the author writes bounded content and trigger conditions.The author limits skill content to 500 characters and specifies trigger types or regex patterns.Source: §C.8.3 · p. 26, §C.8.4 · p. 26

C.8.5 Agent Runtime Prompt (Skill Injection)

• At runtime, triggered skills are injected into the agent prompt, where matching when_to_use guidance directs the next action.The prompt includes task description, memory context, and triggered skills before the action-format instruction.Source: §C.8.5 · pp. 26–27

D Extended Related Work

• RESKILL extends prior memory, distillation, tool-use, and bandit approaches by adapting skills during RL and testing competing versions within on-policy training.The related-work discussion positions within-group testing and adaptive discounting as the direct gap addressed by RESKILL.Source: D Extended Related Work · p. 27

E Evolved Skill Examples

• Evolved examples span embodied reasoning, search, science, shopping, and SQL, including process-before-placement and verify-entity-role-match skills.These examples demonstrate conditional triggers and domain-specific action guidance.Source: E Evolved Skill Examples · p. 27

E.1 ALFWorld

• The ALFWorld process-before-placement skill triggers after taking an object and completes required transformations before final placement.Its example contrasts cooling bread before placement with skipping the transformation.Source: §E.1 · pp. 27–28

E.2 Search

• Search skills require targeted retrieval, entity-role verification, and multi-hop decomposition before producing an answer.Examples distinguish the requested role from nearby entities and preserve intermediate-to-final search dependencies.Source: §E.2 · p. 28

E.3 ScienceWorld

• ScienceWorld skills stop repeated failed moves and convert focus results into immediate object placement decisions.The triggers respond to focus actions or four failed move attempts.Source: §E.3 · p. 29

E.4 WANDS

• WANDS skills commit to matching products after search and select required options before buying.The procedures discourage repeated searching and premature purchase actions.Source: §E.4 · p. 29

E.5 InterCode-SQL

• InterCode-SQL skills enforce filtering and use wildcard matching for descriptive text, while handling NULL cases in maximum-value subqueries.The examples contrast LIKE-based matching with exact equality and include filtered nested queries.Source: §E.5 · p. 30

F.1 Search

• Search case studies show that skills guide evidence retrieval, role verification, and multi-step confirmation before answering.The traces include Deniece Williams verification and confirmation of Andy Biersack’s film and writers.Source: §F.1 · pp. 30–31

F.2 ALFWorld

• ALFWorld case studies show look-verify-retry after failed actions and immediate action after navigation completes.The examples connect environment-state verification and navigation-triggered behavior to successful recovery.Source: §F.2 · p. 31

Skill-Driven Object Naming

• A newly created object-naming skill fires immediately and produces a minimum-step solution, contrasting with pre-skill episodes that spent 20+ steps manipulating the wrong object.The skill arose after agents confused “glass jar” with “glass cup.”Source: §F.3 · p. 32

F.3 SciWorld

• The SciWorld examples illustrate both the benefits of conditional skill guidance and the risk that models confuse reasoning-level skill use with executable actions.The cases include successful object naming, commitment, and recovery alongside invalid skill-name actions.Source: §F.3 · pp. 32–33

Commitment After Observation

• A commitment-threshold skill prevents repetitive focus actions by helping the agent decide after sufficient observation.Although three focuses are conservative, the model interprets the threshold adaptively to avoid observation loops.Source: §F.3 · p. 32

Recovery via Workshop Navigation

• A recovery skill redirects the agent after repeated failed moves to the workshop, encoding the location of the colored boxes.The agent stops retrying from the wrong locations and navigates to the workshop before placing the object.Source: §F.3 · p. 32

The Meta-Bug: Skill Name as Environment Action

• A major failure occurs when the agent emits a skill name as an environment command instead of using its guidance in reasoning.This meta-bug motivates creating a no-skill-invocation revision.Source: §F.3 · p. 33

F.4 WANDS (Web Shopping)

• Three foundational WANDS skills fire sequentially to commit to a matching product, select required options, and complete the purchase.This four-step workflow serves as a scaffold for later specialized skills.Source: §F.4 · p. 33
• Five WANDS skills compose into a branching conditional state machine, with require-color-selection both blocking premature purchase and enabling an escape when the requested color is unavailable.The branching behavior depends on whether the requested color appears among available options.Source: §F.4 · p. 34

F.5 InterCode-SQL

• The verified Asia query returns 51 countries after schema inspection establishes that Asia is a valid Continent value.The result is reported as consistent with the verified schema.Source: §F.5 · p. 34
• InterCode-SQL uses verify-column-semantics to confirm that Continent contains human-readable names before constructing a filter query.The model first inspects distinct values, confirming that Asia exists as a stored value.Source: §F.5 · p. 34
• Fuzzy-text-matching fails when the requested entity is absent: repeated loosening of LIKE conditions eventually returns unrelated rows.The model over-applies the skill instead of recognizing that the entity may not exist in the database.Source: §F.5 · p. 35