PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillCycle: Co-Evolving Agent Policies and Skill Banks

Ling Li, Qiuyu Shen, Zheng Jiang, Qinwei Ma, Yuxuan Liu, Zhidong Deng

arXiv:2610.09430v1cs.CV

TL;DR

As an agent internalizes skills, its remaining guidance can become redundant, misleading, or insufficient, yet withdrawing guidance does not determine how it should change. SkillCycle co-evolves policy and skill bank through internalization and verified revision. Without inference-time skills, its 3B model reaches 74.74% success and an 88.37 score on WebShop, with relative gains over SOTA.

  • Problem

    Internalization changes the guidance an agent needs, but withdrawing guidance does not determine how skill content or applicability should evolve.

    Source: §1 · p. 1

  • Method

    SkillCycle alternates policy learning with a fixed bank and rule revision with a frozen policy, using token-level feedback and outcomes to guide edits verified in paired environment comparisons.

    Source: §1 · p. 1, §2.3 · p. 4, §3.4 · p. 8

  • Results

    Without inference-time skills, the 3B model achieves 74.74% success and an 88.37 score on WebShop, with relative gains of 0.73% and 3.96% over SOTA, respectively.

    Source: §4 · p. 9

  • Takeaways & Limitations

    Experiments on ALFWorld and WebShop show continual revision improves skill-free performance over static banks and a single update.

    Source: §4 · p. 9

    The study leaves open when revision is most useful and focuses on individual rules and applicability, not skill composition, hierarchy, or cross-task transfer.

    Source: LIMITATIONS AND FUTURE WORK · p. 9

Abstract

from arXiv · show

Internalizing external skills changes a language agent's capabilities and, with them, the value of its remaining guidance: rules can become redundant, misleading, or insufficient for newly encountered decisions. This creates a coupled problem of learning from skills and adapting the skills that supervise further learning. We introduce SkillCycle, a framework for co-evolving agent policies and skill banks through a feedback loop between skill internalization and rule revision. Our central contribution is to give distillation feedback a second role: token-level contextual differences help locate rules for inspection, while interaction outcomes guide edits to their content and applicability. SkillCycle alternates between two phases: policy learning with a fixed skill bank and router, and rule revision with a frozen policy. Candidate edits undergo rule-level and whole-bank environment comparisons before they guide the next learning cycle. On WebShop, SkillCycle with a 3B model achieves a success rate of 74.74% and a score of 88.37 without inference-time skill inputs, representing relative improvements of 0.73% and 3.96% over the state-of-the-art (SOTA) model, respectively. In Cycle 3 ablations on ALFWorld and WebShop, SkillCycle's no-skill success rates improve by 10.18% and 18.11% relative to a static skill bank, and by 2.41% and 2.50% relative to a single bank update, respectively. These results show that continually revising skill guidance as the agent's capabilities change helps transform external skills into policy capabilities that require no skill inputs at inference. We will release code, configurations, skill banks, and evaluation protocols.

1 INTRODUCTION

Skill internalization changes what guidance an agent needs, but withdrawing skills does not determine how their content or applicability should evolve. SkillCycle uses feedback from internalization to revise skill guidance for subsequent learning without inference-time skills.

1 INTRODUCTION

  • As capabilities change, previously useful rules can become redundant, misleading, or insufficient for newly encountered decisions.Withdrawing guidance addresses continued reliance on it but does not determine how its content or applicability should evolve.Source: §1 · p. 1
  • Token-level contextual differences locate decisions to inspect, rule-removal rescoring identifies influential instructions, and interaction outcomes guide revisions.The approach uses task outcomes to distinguish mastery from guidance that helps neither policy.Source: §1 · p. 1, Figure 1 · p. 2
  • SkillCycle alternates skill internalization with rule revision, adapting supervision to the evolving policy for inference without skill inputs.Distillation feedback both supervises policy learning and provides diagnostic evidence for improving skill guidance.Source: §1 · p. 1, Figure 1 · p. 2
  • Candidate edits are tested with the policy fixed in paired environment comparisons at both rule and whole-bank levels before acceptance.These checks assess local benefits and effects on the assembled guidance.Source: Figure 1 · p. 2

2 METHOD: SKILLCYCLE

SkillCycle treats the skill bank as editable supervision for a policy that acts without skill context. It alternates learning with a fixed bank and revising that bank with a frozen policy, using routed guidance, diagnostic evidence, and environment comparisons.

2.1 PROBLEM SETUP

  • SkillCycle alternates internalization with revision: each learning segment uses a fixed bank, then proposed edits are tested with the policy fixed.The committed bank supplies revised guidance for the next learning cycle.Source: §2 · p. 2, §2.1 · p. 3

2.2 SKILL INTERNALIZATION

  • The router selects applicable guidance from observable states, prioritizes critical-event skills, falls back to general workflows, and abstains on unresolved ties.The Student acts without skills; routing declarations can change between cycles while routing code stays fixed.Source: §2.2 · p. 3
  • A same-weight Teacher rescores no-skill Student responses with routed skills, retaining signed token-level differences alongside environment feedback for policy learning.Positive and negative signals indicate whether skill context raises or lowers the sampled token’s probability; outcomes provide task-level feedback.Source: §2.2 · pp. 3–4

2.3 EVIDENCE-GUIDED SKILL REVISION

  • With the policy frozen, token-level diagnostics and interaction outcomes identify rules for inspection and guide KEEP, REWRITE, DROP, or ADD proposals.Action-token likelihood changes prioritize diagnosis, while invalid actions and successful terminal actions provide directional labels; failed decisions without a matching rule can motivate additions.Source: §2.3 · p. 4
  • Rewrites must produce a nontrivial likelihood change and a nonnegative mean outcome-aligned effect on labeled actions, though individual regressions can remain.This screening is followed by environment verification.Source: §2.3 · p. 4

2.4 VERIFIED BANK UPDATES

  • Candidate edits first face targeted paired acceptance tests, then a whole-bank comparison that checks assembled revisions for routing interactions or conflicting instructions.The policy is fixed during testing; failed whole-bank tests restore the previous bank and routing declarations.Source: §2.4 · pp. 4–5

3 EXPERIMENTS

The experiments evaluate no-skill policy performance, bank-revision schedules, rule-level verification, and inference-time routing across ALFWorld and WebShop. They compare final benchmark results with prior methods and test when revised skills help.

3.1 BENCHMARKS AND EVALUATION

  • Evaluation uses three seeds on fixed 128-task panels; WebShop uses a train-side development panel, with official-test evaluation outstanding.The study uses Qwen3-1.7B and Qwen2.5-3B on ALFWorld’s six task families and a 1,000-product WebShop subset.Source: §3.1 · p. 5

3.2 MAIN RESULTS

  • The 3B ALFWorld Student reaches 82.53% macro SR, while its Clean and Pick2 gains are 21.16 and 31.28 pp.The 1.7B Student gains 8.11 pp in ALFWorld macro SR and 4.95 pp in WebShop SR across Cycles 0–3.Source: §3.2 · p. 5
  • The no-skill 3B Student reaches WebShop Score 88.37 and SR 74.74%, topping listed baselines including OPID (85.00/74.20%).Evaluation omits skill inputs.Source: §3.2 · p. 5, Table 1 · p. 6

3.3 ABLATIONS AND LEARNING DYNAMICS

  • At Cycle 3, SKILLCYCLE exceeds Static by 4.95/11.46 pp on ALFWorld/WebShop, and iterative revision adds 1.26/1.82 pp beyond one refresh.The comparison uses matched budgets; ALFWorld improvements vary across task families, with Look and Clean remaining lower than both bank-update controls.Source: §3.3 · p. 7
  • With policy weights frozen, bank revision raises ALFWorld macro SR from 49.43% to 52.45%; WebShop reaches 68.23% and then 67.97%.These results measure guidance at fixed weights rather than no-skill learning, and successive WebShop gains are nonmonotonic.Source: §3.3 · pp. 7–8, Table 3 · p. 8
  • Student SR rises from 48.34% to 55.72% on ALFWorld across Cycles 1–4 and from 69.53% to 74.74% on WebShop across Cycles 1–3.Later WebShop records plateau near 74.74%; shaded regions add training only for SKILLCYCLE, with comparisons using Cycle 3.Source: Figure 4 · p. 8

3.4 VERIFICATION OF SKILL REVISIONS

  • Two of ten revision proposals pass verification across three seeds, with pooled net gains of two and four successes.The cool-search rewrite is rejected despite three gains because it causes six losses; accepted edits can still harm individual trajectories.Source: §3.4 · p. 8, Table 4 · p. 9

3.5 SKILL ROUTING ON API MODELS

  • Across four API models without parameter updates, Skill + Router improves over No Skill by 2.16–7.13 pp on ALFWorld and 1.30–17.05 pp on WebShop.Routing improves over Static General Skill in all eight model–benchmark comparisons.Source: §3.5 · p. 9, Figure 5 · p. 9

4 CONCLUSION

SkillCycle co-evolves agent policies and skill banks through internalization and verified revision. Its no-skill results improve over static and single-update banks, while the 3B WebShop model exceeds SOTA.

4 CONCLUSION

  • 74.74% success and an 88.37 score without inference-time skills give the 3B WebShop model relative gains of 0.73% and 3.96% over SOTA, respectively.These results support a reciprocal process in which policies internalize useful guidance and policy feedback helps maintain skills for subsequent training.Source: §4 · p. 9
  • SkillCycle co-evolves agent policies and skill banks through skill internalization and verified revision.Distillation feedback improves the policy and identifies skills for revision; environment verification selects bank updates for later learning cycles.Source: §4 · p. 9
  • On ALFWorld and WebShop, continual skill-bank revision improves skill-free performance over static banks and a single update; without inference-time skills, the 3B model reaches 74.74% success and 88.37 score on WebShop.Source: §4 · p. 9

LIMITATIONS AND FUTURE WORK

The paper leaves open how skill guidance becomes lasting policy capability and when revision is most useful. It also identifies evaluation boundaries and research directions beyond individual rules and their applicability.

LIMITATIONS AND FUTURE WORK

  • The study leaves open how skill benefits translate into lasting policy capabilities and when revision is most useful.Future work could combine causal analysis with learning dynamics to guide adaptive revision schedules.Source: LIMITATIONS AND FUTURE WORK · p. 9

ETHICS STATEMENT

  • Evaluation covers household simulation and benchmark shopping, while physical or live-commerce deployment requires safeguards for authorization, privacy, and unintended actions.The verification gates test task performance under a fixed policy, not general safety.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The planned release links results to checkpoint and protocol identities and includes training code, configurations, skill/router versions, task manifests, seeds, acceptance settings, outcomes, and aggregation scripts.The stated purpose is to make results traceable to their checkpoints and protocols.Source: REPRODUCIBILITY STATEMENT · p. 10

A RELATED WORK

  • SkillCycle differs from related context-maintenance and distillation work by revising persistent procedural guidance as the policy changes, with environment acceptance gates before edits become supervision.Its research question is whether internalization feedback can improve persistent guidance and subsequent no-skill learning.Source: A RELATED WORK · pp. 14–15

B NOTATION

  • The notation specifies task instructions, environment state and observations, interaction history, the skill bank and router, and training evidence and acceptance gates.Source: B NOTATION · p. 15

C INTERACTION PROTOCOL

  • The interaction protocol models partially observable tasks: the agent acts from exposed observations, and an environment-specific interface parses its generated response into an action.The agent has no direct access to the latent state; only the parsed action affects the environment.Source: C INTERACTION PROTOCOL · pp. 15–16

C.1 CONTEXTUAL POLICY VIEWS AND PERFORMANCE

  • The same-parameter Teacher and Student differ in whether the selected skill is included in context, and evaluation measures both skill-conditioned utility and the no-skill Student target.The no-skill router always abstains for the no-skill evaluation.Source: §C.1 · p. 16

D POLICY INITIALIZATION

  • Cycle 0 is the outcome-only base policy before SkillCycle updates; benchmark trajectories start from their respective checkpoints, with the 3B trajectory beginning at Cycle 1.The reported 3B gain is measured from that checkpoint, and matched initializations apply within each controlled ablation.Source: D POLICY INITIALIZATION · p. 16

E TRAINING PROCEDURE AND IMPLEMENTATION DETAILS

The implementation combines PPO-style policy updates with token-level distillation and verified demonstrations, then uses environment outcomes to form advantages and screen alternatives.

E.1 DIFFERENTIABLE OBJECTIVE AND TOKEN REDUCTIONS

  • Positive token-level gaps increase the sampled token probability and negative gaps decrease it; this signed loss is not a full-vocabulary KL divergence.The Teacher and Student scores are detached, and prompt, observation, skill-context, and padding tokens receive no direct loss.Source: §E.1 · p. 17
  • Demonstration rows are excluded from PPO, entropy, and reference regularization, while WebShop configurations differ in which verified-response tokens receive supervision.The 1.7B configuration selects only the alternative action span; the 3B configuration selects all valid tokens in the verified alternative response.Source: §E.1 · p. 17

E.2 ENVIRONMENT ADVANTAGES AND VERIFIED ALTERNATIVES

  • WebShop selects the first varying coordinate among success, score, and progress to compute a centered environment advantage; if all coordinates are constant, the advantage is zero.This prioritizes success when it separates rollouts, using denser signals only when preceding coordinates do not.Source: §E.2 · p. 17
  • Alternative actions are accepted when same-history branches improve success or progress; tied outcomes instead require a two-sided probability-gap condition.Proposals must be valid, admissible, and different from the original; replay or execution errors are rejected.Source: §E.2 · p. 18

E.3 TRAINING CONFIGURATIONS

The configurations specify alternative Cycle 3 optimization setups and implementation profiles, while bank-update ablations hold optimization fixed to isolate revision schedule.

E.3 TRAINING CONFIGURATIONS

  • Table 5 specifies auxiliary OPD with verified demonstrations, verified demonstrations alone, and OPD advantage injection; bank-update ablations vary only the revision schedule.Source: Table 5 · p. 18, §E.3 · p. 18
  • For WebShop 3B, OPD selects demonstration candidates and supports rule diagnosis, while parameter updates use PPO and verified demonstration supervision without an auxiliary OPD term.Its profile uses microbatch 8 per GPU and four GPUs; the 1.7B profile uses microbatch 4 per GPU and two GPUs.Source: §E.3 · p. 18
  • The release will include run-specific task manifests, acceptance configurations, independent-training seed records, and interval-generation scripts for documenting comparisons.Source: §E.3 · p. 19

F COMPLETE TRAINING PROCEDURE

SkillCycle alternates policy learning under a fixed skill bank with rule revision under a frozen policy. Observable routing, evidence-driven proposals, and paired environment checks determine which guidance changes proceed to the next cycle.

F COMPLETE TRAINING PROCEDURE

  • Each segment trains with the previous bank fixed, then freezes the policy to test proposed edits; the final segment omits revision.The no-skill Student generates training trajectories, and each segment ends with a no-skill evaluation.Source: F COMPLETE TRAINING PROCEDURE · p. 19

G.1 OBSERVABLE CRITICAL-FIRST SKILL ROUTING

  • Critical-event skills take priority only when observable eligibility checks pass; unresolved critical ties trigger abstention, while general workflows are fallback guidance.Eligibility is recomputed at each decision, and repeated bindings are suppressed until confirmed progress.Source: §G.1 · pp. 19–20

G.2 EVIDENCE-DRIVEN RULE AND CANDIDATE SCREENING

  • High disagreement selects decisions for inspection, but rule-removal likelihood influence is not task-return improvement; observed outcomes guide proposed edits.Proposals can keep, rewrite, drop, or add rules; likelihood filters apply to text rewrites, while all mutations require environment verification.Source: §G.2 · pp. 20–21

G.3 PAIRED VERIFICATION AND SKILL BANK UPDATE

  • Each valid edit is tested against the current bank with the policy frozen, then a whole-bank comparison checks interactions before commitment.Likelihood scores alone cannot authorize commitment; rejection restores both bank and router to their prior state.Source: §G.3 · p. 21

H ALFWORLD CANDIDATE SCREENING

  • ALFWorld rewrite screening uses labeled outcomes when available and an auxiliary action-effect score for unknown actions; screened edits still require confirmation.Candidates outside the confirmation budget are deferred, not accepted, and these filters do not universally screen additions or metadata-only edits.Source: H ALFWORLD CANDIDATE SCREENING · pp. 21–22

I VERIFICATION PROTOCOL AND RULE-LEVEL RESULTS

The verification protocol compares bank interventions under matched conditions with the policy fixed. Separate gates govern individual edits and whole-bank promotion, with task-specific panels and cycle-indexed acceptance criteria.

I VERIFICATION PROTOCOL AND RULE-LEVEL RESULTS

  • Verification compares each proposed intervention and the whole bank with policy, task/reset identities, decoding seeds, and environment interface held fixed.Paired success differences report gains and losses separately, and routing edits may change both selection and downstream behavior.Source: I VERIFICATION PROTOCOL AND RULE-LEVEL… · p. 22

I.1 NUMERICAL ACCEPTANCE PROTOCOLS

  • WebShop bank gates use train-side development panels, with cycle-specific acceptance criteria and attribute-group guards; official test tasks do not guide acceptance.The Cycle 2–3 panel aggregates 128 tasks across three decoding seeds, or 384 episodes per arm.Source: §I.1 · p. 22
  • In the ALFWorld 3B operation test, candidate-removal controls estimate each operation’s conditional contribution within the tested bank; interactions require a separate whole-bank comparison, not summing these effects.The sealed 128-task panel is excluded from operation tests, and the gates do not require statistical significance.Source: §I.1 · p. 23

I.2 RULE-LEVEL VERIFICATION SCOPE

  • Rule-level tests estimate conditional contributions within the tested bank, not no-skill Student improvement or a substitute for whole-bank acceptance.The generic invalid-action rewrite covers five families and 240 pairs; each other operation covers one family and 48 pairs.Source: §I.2 · p. 23

J.1 CONTROLLED EXPERIMENTAL PROTOCOL

The controlled ablations compare bank-update schedules under shared training and evaluation settings, while separating skill-conditioned guidance from no-skill Student performance. The paper also distinguishes benchmark results from development-panel ablations and reports a rejected whole-bank candidate.

J.1 CONTROLLED EXPERIMENTAL PROTOCOL

  • The bank-update ablation compares continued outcome-based RL, a fixed bank, one refresh, and iterative revision at the common Cycle 3 endpoint.The compared models are Qwen3-1.7B on ALFWorld and Qwen2.5-3B on WebShop.Source: §J.1 · p. 23
  • Conditions share initialization, task interface, optimization budget, initial guidance, revision resources, and evaluation settings, while Student evaluation omits skill context.Equal budgets mean comparable resource allowances, not identical trajectories or wall-clock time.Source: §J.1 · p. 23
  • Benchmark tables report distinct experiment series and checkpoints; the reported development-panel results are not official-test estimates.The WebShop bank-update ablation extends to Cycles 4–5, separately from the benchmark series.Source: §J.1 · pp. 24–25
  • A WebShop Cycle 3 candidate matched its parent at 295/384 successes but scored 89.7775 versus 89.8519, so the gate retained the parent bank.The comparison is a whole-bank decision, distinct from published GiGPO results and the paper’s local GiGPO checkpoint evaluation.Source: §J.1 · p. 25

K ADDITIONAL ABLATION RESULTS

The ablations test iterative revision against static and one-refresh controls, then separately examine frozen-policy revisions and task-level guidance. Case studies illustrate that local rule influence can conflict with paired episode outcomes and that retirement can leave text archived but inactive.

K ADDITIONAL ABLATION RESULTS

  • At Cycle 3, iterative revision improves no-skill Student SR over Static by 4.95/11.46 pp and over Refresh once by 1.26/1.82 pp on ALFWorld/WebShop.The cited tables report ALFWorld Student and WebShop ablation results; the comparisons use the common Cycle 3 endpoint.Source: Table 10 · p. 27, K ADDITIONAL ABLATION RESULTS · p. 28, Table 12 · p. 28
  • In the separate frozen-policy WebShop series, revisions 1 and 2 reach 68.23% and 67.97%, versus 57.55% at revision 0.The revision-0 skill-access configuration is unspecified, so its contrast with revision 1 is not attributed solely to editing an existing bank.Source: K ADDITIONAL ABLATION RESULTS · p. 28
  • Teacher guidance varies by task: it adds 13.06 points on Cool at Cycle 3, while Pick2 is nearly tied then and favors the Student at Cycle 4.These contextual gaps do not identify a causal rule or establish that it should be retired.Source: K ADDITIONAL ABLATION RESULTS · p. 28

L.1 RETIRING A PICK2 RULE

  • The Pick2 retirement case reports 15 control successes versus nine treatment successes across three seeds, despite positive local action-token influence in some examples.The review attributes the conflict to repeated take-back or stalling behavior and favors dropping the rule because observable facts cannot isolate its benefit.Source: §L.1 · p. 28, §1 · p. 30

3. Rule before and after the accepted update

  • The accepted update deactivates the Pick2 rule but retains its text in the archive, with no replacement rule introduced.When no active rule remains for the family, the router abstains.Source: §3 · p. 30

L.2 OBSERVABLE ROUTING AND AGENT ACTIONS

The routing cases show that eligibility can narrow guidance to the rule relevant to the current state, while routed guidance remains advisory. Routing resolves failures in Case A and the first two seeds of Case B, but does not reduce action count in every trajectory.

L.2 OBSERVABLE ROUTING AND AGENT ACTIONS

  • In Case A, repeated examination with no progress or target object leaves search as the only eligible rule, despite its lower priority.Source: §L.2 · p. 30
  • In Case B, confirmed cooling and a closed microwave leave opening as the sole eligible rule; router candidates guide rather than enforce actions.Source: §L.2 · p. 30
  • Routing resolves Case A failures and the first two Case B seeds; in the third, both conditions succeed but routing raises actions from 13 to 15.These case outcomes are descriptive, and complete checkpoint and prompt identities are unavailable.Source: §L.2 · p. 30, Table 13 · p. 30
  • Local rule retirement is distinct from whole-bank commitment: the v11 bank failed whole-bank acceptance, while a later combined revision passed its own criteria.Source: §L.2 · p. 30
Loading 2610.09430v1…