PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillScaffold: Scaffolding Agentic Reinforcement Learning with Policy-Adaptive Skill Credit Assignment and Selective Injection

Wenlin Zhong, Hao Wang, Kun Yuan, Fan Yang, Jinluan Yang, Dian Jin, Xu Tan, Yiquan Wu, Kun Kuang

arXiv:2610.18807v1cs.LGcs.CL

TL;DR

Skill-based agentic RL must decide when and which skill to use, but rollout-based estimates are costly and unstable, and skill benefits vary with candidate and policy competence. SkillScaffold estimates utility from outcome-verified trajectories to select or withhold skills, and improves over strong baselines across four benchmarks under skill-free inference.

  • Problem

    Existing skill-based RL approaches may assume generated skills help, use indirect and delayed feedback, or rely on costly, unstable candidate-specific rollouts, although skill benefits vary by candidate and current policy.

    Source: §1 · p. 2, §3.2 · p. 4

  • Method

    SkillScaffold compares outcome-aware likelihoods on existing trajectories to estimate policy-relative skill utility, then uses it for candidate selection, abstention, and Generator learning without candidate-specific environment rollouts.

    Source: §1 · p. 3, §4.3 · p. 6

  • Results

    Across four benchmarks, SkillScaffold beats strong agentic-RL and skill-based RL baselines; its WebShop gain over the strongest skill-based RL result is 3.3 points.

    Source: §5.2 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    SkillScaffold selectively uses skills during training while encouraging their internalization, allowing deployment without the Generator or skill pathway.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

Agentic reinforcement learning (RL) enables large language models (LLMs) to learn from multi-turn interactions with verifiable outcomes. Textual skills can improve exploration by distilling prior interactions into reusable guidance. However, existing skill-based RL approaches often either assume skill injection is beneficial or rely on indirect feedback or rollout-intensive evaluation to assess its utility. Our pilot studies show that rollout-based utility estimation for skill credit assignment is costly and unstable under limited sampling, while skill benefits vary across candidates and depend on the current policy's competence on each query. To address these challenges, we introduce SkillScaffold, a policy-adaptive training scaffold that combines low-cost skill-utility estimation with selective skill injection. By reusing outcome-verified training trajectories, SkillScaffold generates candidate skills and estimates their utility through outcome-aware likelihood comparisons, without candidate-specific environment rollouts. The resulting utility guides whether and which skill to inject, retaining skill-free execution when no candidate is estimated to help. Joint training enables the Generator to produce skills tailored to the evolving Executor. Scaffolded RL encourages skill internalization for skill-free inference. Experiments on ALFWorld, WebShop, ScienceWorld, and $\tau^2$-Bench demonstrate gains over strong agentic-RL and skill-based RL baselines across both short- and long-horizon tasks. Ablations and further analyses support the importance of utility-based selection and abstention. Our code is available at https://github.com/Vaidurya00/SkillScaffold.

1 Introduction

Textual skills can provide reusable guidance for agentic RL, but existing approaches may assume skills help, use delayed feedback, or require costly, unstable rollout-based utility estimates. SkillScaffold estimates policy-relative utility from existing trajectories to select or withhold skills, and improves over strong baselines across four benchmarks under skill-free inference.

1 Introduction

  • Textual skills distill prior interactions into reusable procedural guidance, and prior work has explored shared-policy skill generation and skill-free inference through internalization.The paper builds on these directions by evaluating and selectively using guidance for the current policy and query.Source: §1 · pp. 1–2
  • Skill utility varies across candidates and with the current policy’s ability to solve each query, while rollout-based estimates are costly and unstable under limited sampling.These challenges motivate candidate ranking and abstention rather than assuming every skill helps.Source: §1 · pp. 2–3
  • SkillScaffold estimates candidate utility through outcome-aware likelihood comparisons on existing trajectories, guiding skill selection, abstention, and Generator learning without candidate-specific environment rollouts.A shared policy generates candidates and executes tasks; the highest-utility skill is injected only when its estimated utility is positive.Source: §1 · pp. 2–3
  • Across ALFWorld, WebShop, ScienceWorld, and τ2-Bench, experiments show improvements over strong agentic-RL and skill-based RL baselines under skill-free inference.Ablations and further analyses support the importance of utility-based selection and abstention.Source: §1 · pp. 2–3

2 Preliminaries

The preliminaries describe an LLM policy interacting with an environment over multiple turns, with trajectories receiving verifier-assigned outcome rewards. They also introduce GRPO and define the policy-update notation used in the paper.

2 Preliminaries

  • The policy interacts with an environment over multiple turns through a tool engine, producing an action-observation trajectory conditioned on a task and optional skill.The passage introduces the trajectory notation but does not supply its full displayed expression.Source: §2 · p. 3
  • A rule-based Outcome Verifier assigns each trajectory a reward given the task, and the Executor is optimized to maximize expected environment return.Skill-free execution is represented by an empty skill.Source: §2 · p. 3
  • GRPO updates the policy using group-relative advantages; its notation includes token-level importance ratios, group-normalized trajectory advantages, a clipping threshold, and a KL-penalty coefficient.The passage identifies these terms but does not provide the full update expression.Source: §2 · p. 3

3 Motivating Observations

Rollout-based skill-utility estimation is costly and remains unstable under limited sampling. Skill effects also vary among candidates for the same query and depend on the current policy’s skill-free competence, motivating query- and policy-relative utility that can rank candidates or abstain.

3.1 Rollout-Based Skill-Utility Estimation Is Unstable and Costly

  • Estimating each candidate condition with K episodes and a shared skill-free baseline requires (M+1)K additional environment rollouts per query.Online RL must decide whether and which skill to inject while these candidate-specific estimates require additional interaction.Source: §3.1 · pp. 3–4
  • 23% at K=1 to 16% at K=8: independent rollout-based utility estimates still disagree in sign under sparse outcome rewards.The disagreement measure counts cases where both estimates are non-zero and have opposite signs.Source: §3.1 · p. 3
  • 32% to 55%: increasing the additional rollout budget eight-fold, from 5 to 40 per query, raises the share of non-tied candidate groups only modestly.Nearly half of the groups remain unresolved at the higher budget.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Skill Utility Is Query- and Policy-Dependent

  • 28 of 100 queries contain both positive- and negative-effect candidates, so candidate skills can differ in utility even within the same query.The pilot compares each query’s best and worst observed candidate relative to skill-free execution.Source: §3.2 · p. 4, Figure 3 · p. 4
  • Average skill gains are positive when only a minority of skill-free probes succeed, but become negative when most or all succeed.At higher competence, even the best observed candidate provides little or no additional benefit.Source: §3.2 · p. 4, Figure 3 · p. 4

3 Motivating Observations

  • Skill credit assignment therefore needs query- and policy-relative utility to identify helpful candidates and abstain when none adds positive value.The downstream measurements in these motivating observations diagnose the problem; they are not training signals for SkillScaffold.Source: §3.2 · p. 4

4 SkillScaffold

SkillScaffold uses a shared policy to scout without skills, generate and score candidate skills, and inject only a highest-utility positive candidate. Joint optimization supports skill-free deployment.

4 SkillScaffold

  • SkillScaffold uses one shared policy as Executor and Generator, scoring candidates and injecting only the highest-utility positive skill into fresh Executor rollouts.Environment rewards train the Executor, while skill utility trains the Generator.Source: §4 · p. 4

4.1 Skill-Free Rollout

  • Skill-free Executor trajectories, called Scouts, receive verifier-assigned success or failure outcomes.The resulting group can be all-success, all-failure, or mixed.Source: §4.1 · p. 5

4.2 Outcome-Anchored Skill

  • Group-based generation draws reusable decisions, recurring errors, or contrasts from outcome-labeled Scouts; anchor-based generation instead uses one successful Scout and skips generation if none succeeds.Source: §4.2 · p. 5

4.3 Skill Utility Estimate & Select

  • Skill utility compares the current policy’s likelihood of fixed Scout behavior with and without a candidate, using outcomes to reward successful behavior and suppress failed behavior.For mixed groups, success-support and failure-suppression terms receive equal total weight.Source: §4.3 · pp. 5–6
  • Anchor-based utility scores candidates on one successful Scout, whereas group-based utility also incorporates failure evidence.The resulting policy-relative signal supports candidate ranking and abstention.Source: §4.3 · p. 6

4.4 Adaptive Skill Injection Rollout

  • The system selects the highest-utility valid candidate only when its utility is positive; otherwise, it does not inject a skill.Ranking finds the most compatible candidate, while the sign test determines whether to inject any skill.Source: §4.3 · p. 6

4.5 Joint Policy Optimization

  • Environment rewards train the Executor and utility-derived rewards train the Generator through shared parameters; deployment removes the skill pathway.Both skill-free Scouts and selectively scaffolded Post-Scouts update the shared policy.Source: §4.5 · pp. 6–7

5 Experiments

Across four interactive benchmarks, SkillScaffold outperforms strong agentic-RL and skill-based RL baselines, including under skill-free inference. Experiments also test the utility signal, selective injection, training-time exploration, and skill internalization.

5.1 Experimental Setup

  • The evaluation covers short-horizon ALFWorld, WebShop, and ScienceWorld, plus long-horizon 𝜏2-Bench; baselines span prompting, standard RL, and skill-based RL.Main runs use Qwen3-8B.Source: §5.1 · p. 7

5.2 Main Results

  • SkillScaffold beats the strongest skill-based RL results on every benchmark, gaining 3.3 points on WebShop, 3.0 on ALFWorld, and 6.3 on Telecom.Table 2 covers ScienceWorld ID/OOD and the three 𝜏2-Bench domains.Source: §5.2 · pp. 7–8, Table 2 · p. 8
  • Without skills at inference, SkillScaffold reaches 45.1 on WebShop versus 41.8 for Evolving-RL with skills and leads competing results on both ID and OOD splits.Reported ID/OOD results are 46.6/42.0 on WebShop and 96.2/48.8 on ScienceWorld.Source: §5.2 · p. 8

5.3 Ablation Studies

  • Removing Generator learning, utility-guided selection, positive gating, or skill injection reduces ALFWorld/WebShop results, supporting each component's role in the tested system.Reported ablations include 89.6/39.5 without Generator learning, 79.9/39.2 with random selection, 85.8/39.6 without the positive gate, and 90.3/41.5 without Executor-rollout injection.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 Validating Skill Utility for Credit Assignment

  • Measured gain over skill-free execution rises with estimated utility, supporting utility's use for candidate ranking and abstention.The validation protocol uses fixed Scout trajectories to estimate utility and reserves separate downstream rollouts for evaluation.Source: Figure 5 · p. 8, §5.4 · p. 9

5.5 SkillScaffold Improves Training-Time Exploration

  • Post-Scout success improves 21.1 points for all-wrong groups and 5.4 for mixed groups, while changing little for all-correct groups (−2.3 points).Injection is nearly universal for all-wrong groups, selective for mixed groups, and rare for all-correct groups.Source: Figure 6 · p. 9, §5.5 · p. 9

5.6 Skill Internalization into Model Parameters

  • Later skill-free success improves on fixed early skill-assisted cohorts by 19.7 points on WebShop, 29.6 on ALFWorld, 65.0 on ScienceWorld, and 16.6 on 𝜏2-Bench.The Executor receives no skill text at later evaluation; intervals exclude zero on the latter three benchmarks.Source: §5.6 · p. 9

6 Conclusion

SkillScaffold reuses outcome-verified trajectories to estimate policy-relative skill utility, selectively scaffolds training, and improves benchmark performance under skill-free inference. The conclusion also reports skill-free gains on fixed cohorts and notes the paper's reproducibility and AI-use statements.

6 Conclusion

  • Across diverse short- and long-horizon benchmarks, SkillScaffold improves over strong agentic-RL and skill-based RL baselines under fully skill-free inference, using outcome-verified trajectories to estimate policy-relative skill utility.Source: §6 · p. 9
  • The method estimates policy-relative utility from outcome-verified trajectories, uses it for selection, abstention, and Generator learning, and removes the skill pathway at deployment.The conclusion reports gains over strong agentic-RL and skill-based RL baselines across short- and long-horizon benchmarks.Source: §6 · p. 9

Reproducibility Statement

AI Use Statement

  • Generative AI assisted with language editing, LaTeX formatting, and code navigation, while authors reviewed and verified the final text and code.Source: AI Use Statement · p. 10

A.1 Benchmark and Data Details

The appendix details the benchmark data and configurations, then describes how training evidence is constructed and used to generate candidate skills. It distinguishes skill-free Scout evidence from subsequent skill-conditioned execution.

A.1 Benchmark and Data Details

  • The appendix specifies training and evaluation data across household, shopping, scientific-reasoning, and long-horizon dialogue/tool benchmarks.ALFWorld uses 3,553 training tasks and 134 valid-unseen tasks; WebShop partitions goals into ID and OOD domains; ScienceWorld uses electricity tasks with ID/OOD episodes; 𝜏2-Bench uses 300 RL-pool entries per domain.Source: §A.1 · p. 12

A.2 Benchmark Configurations

  • ALFWorld, WebShop, and ScienceWorld start from base Qwen3-8B, while 𝜏2-Bench RL methods use the same domain-adapted SFT checkpoint.Source: §A.2 · p. 12, Table 5 · p. 13

A.3 End-to-End Training Procedure

  • Environment rewards supervise the Executor and detached policy-relative utility supervises the Generator, with both objectives updating shared policy parameters.Source: §A.3 · p. 13

A.4 Evidence Construction and Candidate Generation

  • Group-based evidence contrasts successful and failed Scout decisions or emphasizes reusable successful decisions; anchor-based evidence uses the first verified successful Scout.If no Scout succeeds in anchor-based training, candidate generation and Generator optimization are skipped and Executor execution remains skill-free. Each record yields independently sampled candidates that must provide reusable guidance rather than task-specific answers; invalid candidates are not selected.Source: §A.4 · p. 13

A.5 Diagnostic Evaluation Protocols

The appendix describes fixed-trajectory utility diagnostics and a fixed-cohort protocol for measuring skill-free strategy acquisition. It also provides expanded 𝜏2-Bench results and training-curve comparisons across benchmarks.

A.5 Diagnostic Evaluation Protocols

  • Utility diagnostics estimate candidate utility from fixed Scout trajectories and use separate downstream rollouts to measure gains over skill-free execution; strategy-acquisition recovery compares later and early skill-free success on a frozen cohort.The cohort contains queries benefiting from skill-assisted exploration at an early checkpoint, and confidence intervals use a query-level bootstrap.Source: §A.5 · pp. 13–14
  • The training procedure scores fixed Scout tokens in skill and no-skill contexts, selects the highest-utility valid candidate only when its utility is positive, and returns a skill-free policy.Source: §A.5 · p. 14

B.1 Full 𝜏2-Bench Pass@𝑘Results

  • Table 6 expands 𝜏2-Bench comparisons to Pass@4 and Pass@8, with averages calculated across Airline, Retail, and Telecom.Source: Table 6 · p. 14, §B.1 · p. 14

B.2 Training Dynamics Across Benchmarks

  • Training curves cover WebShop, ALFWorld, and each 𝜏2-Bench domain; the largest late-training separation is on Telecom, with Airline and Retail also retaining later advantages.The appendix also reports injection rates for WebShop and ScienceWorld and positive-utility selection on 𝜏2-Bench.Source: Figure 7 · p. 15, §B.2 · p. 15, Figure 8 · p. 15

B.3 Skill Injection Rates Across Benchmarks

SkillScaffold activates its training-time scaffold selectively, with injection rates that vary across benchmarks.

B.3 Skill Injection Rates Across Benchmarks

  • SkillScaffold injects skills for 12.1% of WebShop queries, 21.3% of ScienceWorld queries, and 39.3% of τ2-Bench queries on average.Always-on SkillRL supplies a skill to every query.Source: §B.3 · p. 15

B.4 Cross-Benchmark Post-Scout Exploration Dynamics

Across benchmarks, SkillScaffold concentrates positive-utility skills on queries where they support exploration and reduces their use as the Scout policy becomes competent. The theoretical analysis interprets utility through likelihood comparisons and balances evidence from mixed Scout groups.

B.4 Cross-Benchmark Post-Scout Exploration Dynamics

  • Positive-utility skills concentrate on all-wrong and mixed groups, where Post-Scout rollouts recover successful behavior, and become rare once all Scouts succeed.The analysis describes this pattern across benchmarks.Source: §B.4 · p. 16

C.1 Log-Likelihood-Ratio Interpretation

  • A positive d(s, τ) means conditioning on skill s increases the geometric mean token likelihood assigned to the observed Executor behavior.This is the log-likelihood-ratio interpretation of the policy-relative skill utility.Source: §C.1 · pp. 16–17

C.2 Selection Robustness under Utility Error

The analysis gives robustness guarantees for utility-based selection, relates utility thresholds to downstream gain under a calibration assumption, and characterizes the computation required.

C.2 Selection Robustness under Utility Error

  • Ranking by estimated utility incurs at most 2ε proxy regret, while abstention does not overlook a candidate whose proxy advantage exceeds the estimation-error scale.These guarantees are stated for utilities under a uniform estimation-error bound.Source: §C.2 · p. 18

C.3 Connection to Downstream Return

  • A positive utility threshold yields a guaranteed downstream-gain margin under the local surrogate calibration assumption, with higher thresholds trading off against that margin.The experiments use t = 0, while the gate also supports a tunable positive margin.Source: §C.3 · p. 18

C.4 Interaction and Computation Complexity

  • SkillScaffold avoids candidate-specific environment trajectories and reuses Scout trajectories, requiring O(MN) additional model scoring work for utility computation.With N = M = L = 4, each update uses eight Executor-group environment trajectories and twenty reusable trajectory-context scores.Source: §C.4 · pp. 18–19

D Skill-Guided Execution Examples

  • The appendix introduces skill-generation prompts and benchmark-specific skill-guided execution examples, including abridged prompts for ALFWorld and ScienceWorld.The supplied passage describes these materials but does not report their task outcomes.Source: D Skill-Guided Execution Examples · p. 20

D.1 Qualitative Case of Skill-Free Strategy Acquisition

The appendix illustrates scaffolded exploration and skill generation through task-specific examples, including a case where a strategy later appears in skill-free execution.

D.1 Qualitative Case of Skill-Free Strategy Acquisition

  • At a later checkpoint, the skill-free ALFWorld policy reproduces the clean-before-place routine first introduced by a selected positive-utility skill.The skill enables all Post-Scouts to complete the task, while early skill-free Scouts omit cleaning.Source: §D.1 · p. 20, Figure 13 · p. 20

1. Skill generation prompt (abridged template)

  • The abridged ALFWorld prompt instructs the Generator to ground corrections in observed evidence and avoid presenting an unobserved complete solution as verified.Its output is a concise skill card linking observable conditions to actions, checks, or recovery steps.Source: §1 · pp. 20–21

D.2 ALFWorld: Clean Before Placement

  • The ALFWorld skill preserves confirmed progress and recommends inspecting hidden contents, using observed identifiers, and correcting action preconditions before retrying.The example task requires finding, cleaning, and placing a soapbar; feedback is checked after each action.Source: §1 · p. 21, # Skill:… · pp. 21–22

D.3 WebShop: Verify the Required Option Before Purchase

  • The WebShop prompt asks the Skill Extractor to derive one reusable decision mechanism from a successful trajectory, while warning that success does not make every action useful or optimal.The example skill checks whether a required option is available before purchase.Source: §1 · p. 23, # Skill:… · p. 23

D.4 ScienceWorld: Complete the Circuit Before Classifying

  • The ScienceWorld prompt warns against treating an absent signal as a negative material test until the experimental setup is complete and verified.The selected skill checks equipment and circuit connections before interpreting the bulb’s signal or classifying the material.Source: §1 · p. 25, §2 · pp. 25–26

# Skill: Exchange item for same product type in delivered order

  • The retail skill checks delivery and exchange eligibility, confirms the variant and refund method, and requires confirmation before proceeding with an exchange.It stops or escalates when a request exceeds the exchange scope.Source: # Skill: Exchange item for same… · pp. 27–28

Related papers

Loading 2610.18807v1…