PaperlayerView exact revision on arXiv

Source-linked AI summary

UniSkill: Learning Actor-Aligned Skill Proposals for an Evolving Policy

Yifei Lu, Cheng Liu, Dianzhi Yu, Hui Xiang, Ji Zhang, Yuanchu Xiao, Rong Liang

arXiv:2610.10164v1cs.AI

TL;DR

Jointly learning an actor and skill proposer makes it difficult to distinguish a skill’s benefit from intervening actor improvement, while direct candidate-skill evaluation requires extra rollouts. UniSkill uses a shared policy and contrastive action feedback on previously collected trajectories, with regularization to preserve edit exploration. It reports strong performance on ALFWorld and WebShop and continued effectiveness with a smaller backbone.

  • Problem

    During joint training, later skill reuse may conflate a skill’s benefit with actor improvement, while directly evaluating each proposed skill requires additional environment rollouts.

    Source: §1 · p. 2

  • Method

    UniSkill uses a shared policy to act and propose skill edits, training the proposer with contrastive action feedback on fixed successful and failed trajectories and support regularization for edit exploration.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    UniSkill demonstrates strong performance among compared methods on ALFWorld and WebShop and remains effective with a smaller backbone.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments support policy–skill co-evolution with stable joint training across model sizes.

    Source: §1 · p. 2, §6 · p. 10

    The alignment reward does not assess proposal format, operation appropriateness, or whether skill content is grounded in the source trajectory, and may favor trajectory-specific detail over reusable guidance.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Large language model agents can improve across tasks by retaining reusable skills distilled from prior interactions. Recent work jointly optimizes task execution and skill extraction, enabling the policy and skillbank to co-evolve. However, as the actor continues learning, rewarding skill proposals through their reuse in subsequent training steps may conflate skill benefits with actor improvement, while directly testing each proposed skill requires costly additional actor rollouts. In this paper, we introduce UniSkill, which uses a shared policy to interact with the environment and propose skillbank edits (Add, Update, or No Edit) from the resulting trajectories. Specifically, the actor learns from environment rewards, while contrastive action feedback guides skill proposal learning. This feedback provides an actor-alignment signal by measuring how replacing the retrieved skill with a proposed skill changes the current actor's action log-likelihood gap between previously collected successful and failed trajectories from the same task, thereby avoiding new rollouts for each proposal. Since proposal-level feedback may suppress an otherwise appropriate edit operation when the proposed skill content scores poorly, we further apply skill-edit support regularization to preserve exploration. Empirically, UniSkill achieves strong performance, reaching 98.4% success on ALFWorld and 84.7% on WebShop while maintaining stable joint training. Further ALFWorld experiments show that UniSkill remains effective when the shared policy uses a smaller backbone. Our implementation is available at https://github.com/LimOkii/UniSKill.

1 INTRODUCTION

UniSkill addresses how to evaluate proposed skills for the current actor during joint training, when later reuse can confound skill benefits with actor improvement. It trains a shared policy to act and propose skillbank edits using contrastive action feedback and skill-edit support regularization.

1 INTRODUCTION

  • Existing proposal-learning signals use task-outcome prediction or extraction-trajectory rewards, but do not assess how the current actor behaves with a proposed skill.Evaluating skills through subsequent task performance is more direct, but the actor may have changed before reuse.Source: §1 · p. 2
  • UniSkill uses a shared policy to act with retrieved skills and propose Add, Update, or No Edit operations from completed trajectories.The actor learns from environment rewards, while the proposer receives contrastive action feedback and rewards for operation appropriateness.Source: §1 · p. 2
  • Skill-edit support regularization preserves exploration when negative content feedback might suppress an otherwise appropriate edit operation.Skill edits that pass both skill-critic and actor-alignment checks update the skillbank for subsequent interactions.Source: §1 · p. 2
  • Contrastive action feedback compares the current actor’s likelihood gap on same-task successful and failed trajectories when using the proposed versus retrieved skill.The trajectories are already collected, avoiding additional environment rollouts for each proposal.Source: §1 · p. 2
  • UniSkill performs strongly among compared methods on ALFWorld and WebShop and remains effective with a smaller backbone.Further ALFWorld experiments show later validation-success gains while the skillbank continues to add and refine skills.Source: §1 · p. 2

2 RELATED WORK

Prior work stores reusable experience as reflections, programs, workflows, or skills, with some methods updating external memory and others coupling reuse with policy optimization. Work on credit assignment evaluates generated skills through task outcomes or later reuse, whereas UniSkill compares skill-conditioned action likelihoods on fixed trajectories.

2.1 EXPERIENCE REUSE AND SKILL-AUGMENTED AGENTS

  • Prior experience-reuse methods store reflections, insights, programs, workflows, or skills, with approaches ranging from prompted memory updates to policy optimization and skill-graph evolution.The section frames evaluation and credit assignment for co-evolving policies and stored experience as an open question.Source: §2.1 · p. 3

2.2 CREDIT ASSIGNMENT FOR CO-EVOLVING POLICIES AND SKILLS

  • Skill-generation feedback has used outcome-prediction accuracy, task returns, task success, later skill reuse, or comparisons across skillbank revisions.MASA finds skill effectiveness varies across model backbones; Evolving-RL tests candidate skills before policy updates using additional skill-conditioned rollouts.Source: §2.2 · p. 3
  • UniSkill holds the current actor and recorded trajectories fixed, comparing action likelihoods under retrieved and proposed skills without additional evaluation rollouts.This contrasts with evaluating a skill after later reuse, when the actor may already have been updated.Source: §2.2 · p. 3

3 METHODOLOGY

UniSkill uses one shared policy to act with retrieved skills and propose skillbank edits from completed trajectories. Actor learning uses environment rewards, while skill proposals receive contrastive action feedback, critic checks, and support regularization before eligible edits are applied.

3.1 PRELIMINARIES

  • The shared policy acts using a retrieved skill and proposes NO EDIT, ADD, or UPDATE from a completed trajectory paired with an opposite-outcome trajectory from the same task.ADD proposes a separate reusable skill; UPDATE revises the retrieved skill when its guidance needs correction, refinement, or extension.Source: §3.1 · pp. 3–4

3.2 SKILL-AUGMENTED ACTOR LEARNING

  • The actor is trained with GRPO using environment rewards, with group-relative advantages normalized by the reward standard deviation and a KL regularization term toward a reference policy.The rollout policy generates groups of trajectories for each task conditioned on the same retrieved skill.Source: §3.2 · p. 4

3.3 ACTOR-ALIGNED SKILL PROPOSAL LEARNING

  • Contrastive feedback replaces the retrieved skill in fixed successful and failed reference contexts, measuring how the candidate changes their action log-likelihood gap under the current actor.The pre-update policy, recorded histories, and action tokens remain fixed; positive Ralign means the gap widens relative to the retrieved skill.Source: §3.3 · p. 5
  • The critic checks edit appropriateness and whether ADD or UPDATE content is trajectory-supported, while supported content receives clipped alignment feedback and NO EDIT receives neutral content reward.The critic judges validity rather than estimating utility for the current actor.Source: §3.3 · p. 5
  • Skill-Edit Support Regularization penalizes available edit operations whose normalized probabilities fall below a threshold, discouraging premature suppression without enforcing a uniform operation distribution.UPDATE is available only when the source trajectory has a retrieved skill; NO EDIT and ADD are always available.Source: §3.3 · p. 6

3.4 JOINT POLICY–SKILL CO-EVOLUTION

  • Eligible ADD and UPDATE proposals must pass both critic checks and satisfy Δ+ > 0 and Ralign > 0; the highest-Ralign eligible proposal is applied for each UPDATE target.Actor rollouts and proposals come from the same pre-update policy and skillbank snapshot, and the shared policy is jointly optimized.Source: §3.4 · p. 6

4 EXPERIMENTS

UniSkill is evaluated on ALFWorld and WebShop, with additional experiments examining training dynamics, model size, generalization, and skill-critic sensitivity. It achieves strong benchmark results and maintains stable joint training, including with a smaller shared backbone.

4.1 EXPERIMENTAL SETUP

  • ALFWorld covers text-based household navigation and manipulation, while WebShop tests searching, inspecting, and purchasing products; evaluation reports success rates and normalized WebShop scores.Source: §4.1 · p. 6

4.2 PERFORMANCE ON ALFWORLD AND WEBSHOP

  • UniSkill exceeds Evolving-RL by 5.4 pp in ALFWorld and sustains stable joint training over 250 steps, while skillbank growth slows after early expansion.Source: §4.2 · p. 7

4.4 ADDITIONAL EXPERIMENTS

  • Additional experiments examine out-of-distribution generalization and critic sensitivity; Figure 3 compares per-step time for UniSkill-7B and Evolving-RL-7B.Source: §4.4 · p. 7, Figure 3 · p. 7

4.3 PERFORMANCE ACROSS MODEL SIZES

  • With Qwen2.5-3B-Instruct as the shared backbone, UniSkill-3B continues improving late, exceeds 90% success, and finishes above GRPO-7B.Source: Figure 4 · p. 7, §4.3 · p. 7

4.2 PERFORMANCE ON ALFWORLD AND WEBSHOP

  • 98.4% ALFWorld success; WebShop task score of 90.5 and 84.7% success; UniSkill leads GRPO/GiGPO by 20.8/7.6 pp and SkillRL by 12.0 pp on success.Source: §4.2 · p. 7, Table 1 · p. 8

5 ANALYSIS

The analysis tests joint actor and skill-proposer learning, actor-alignment feedback, and skill-edit support regularization. Its findings cover retrieval-disabled performance, how Ralign tracks observed gains, and how regularization affects edit exploration and validation success.

Q1: How Does Joint Actor and Skill Proposer Learning Contribute to Performance?

  • 98.4% success with UniSkill’s evolved skillbank; without retrieval, it retains 96.1%, versus 84.4% for Actor Only and the no-Ralign variant, and 34.4% for Skill Proposer Only.The table reports ALFWorld ablations with and without skill retrieval at evaluation.Source: Table 2 · p. 8, §5 · p. 8

Q2: Does Ralign Capture Actor Alignment?

  • ρ = 0.312 and 0.367 linked Ralign positively with success-rate gains at steps 25 and 75, but 24.5% and 15.6% of positive-score proposals had negative observed gains.Mean gains for positive-score proposals were 8.4 and 7.2 percentage points, versus −3.4 and −0.5 for negative-score proposals.Source: Figure 5 · p. 9, §5 · p. 9

Q3: Does Support Regularization Sustain Skill-Edit Exploration?

  • Support regularization kept all three edit operations sampled with changing proportions and showed faster early improvement and higher later validation success; without it, one run collapsed to ADD.The figure reports one run per setting for edit distributions and three independent runs per setting for validation success.Source: Figure 6 · p. 9, §5 · p. 9

6 CONCLUSION

UniSkill jointly learns task execution and skill proposals with a shared policy, supporting actor–skillbank co-evolution and stable training across the reported benchmarks.

6 CONCLUSION

  • UniSkill jointly learns task execution and skill proposals with a shared policy, enabling the actor and skillbank to co-evolve.The conclusion reports strong task performance and stable joint training on ALFWorld and WebShop, and effectiveness with a smaller shared-policy backbone in further ALFWorld experiments.Source: §6 · p. 10

ETHICS STATEMENT

The reported evaluation uses publicly available ALFWorld and WebShop benchmark environments.

ETHICS STATEMENT

  • The study evaluates publicly available ALFWorld and WebShop benchmarks, without human-subject experiments, personal-data collection, real-world physical actions, or purchases.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement maps the paper’s training, evaluation, ablation, and configuration materials and says implementation and configuration files are planned for release.

REPRODUCIBILITY STATEMENT

  • Section 3 specifies the training objectives, contrastive action feedback, and skillbank update rules; Appendix C summarizes training in Algorithm 1.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Section 4 describes benchmarks, evaluation metrics, and baseline sources; Appendix B reports training and generation configurations, the ALFWorld protocol, and choices of η and pmin.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Section 5 describes ablations and statistical analyses, Appendix D.3 details query selection and proposal sampling, and the authors plan to release implementation and configuration files.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The appendix documents author responsibility and describes task-specific prompts for proposing skill edits, alongside a critic that evaluates edit operations and skill content separately.

AI USE STATEMENT

  • The authors led the research, implementation, experiments, and manuscript preparation, and take full responsibility for the paper.Generative AI tools assisted with literature review, methodological and experimental discussions, translation, editing, and formatting.Source: AI USE STATEMENT · p. 10

A.1 ALFWORLD PROMPTS

  • ALFWorld proposals compare a completed episode with an opposite-outcome trajectory, then select No Edit, Add, or Update and provide reusable, trajectory-supported skill text when needed.Update is available only when a skill was retrieved; No Edit uses NONE in the skill block.Source: §A.1 · p. 13

A.2 WEBSHOP PROMPTS

  • WebShop proposals likewise compare opposite-outcome episodes and choose an edit, with guidance required to be concrete, reusable across shopping tasks, and not copied from existing skills.No Edit applies when no concrete reusable guidance is revealed or the retrieved skill already contains it.Source: §A.2 · p. 14

A.3 SKILL CRITIC PROMPT

  • The skill critic judges whether the operation fits the trajectory separately from whether proposed content is valid, accepting imperfect but reusable, actionable guidance unless it is clearly defective.When Add and Update are genuinely ambiguous, either operation is accepted; operation judgments ignore candidate wording and content quality.Source: §A.3 · p. 15

B IMPLEMENTATION DETAILS

Implementation details specify a shared actor–proposer policy, evaluation protocols, and thresholds used for contrastive feedback and skill-edit support.

B IMPLEMENTATION DETAILS

  • The actor and skill proposer share a Qwen2.5-7B-Instruct policy and AdamW optimizer; each run starts with an empty skillbank.A frozen DeepSeek-V4-Pro critic and Qwen3-Embedding-0.6B top-1 retrieval are used by default.Source: B IMPLEMENTATION DETAILS · p. 15
  • ALFWorld results use the valid seen split, with the valid unseen split reserved for out-of-distribution evaluation and means plus sample standard deviations across three runs.Each run's trained policy and skillbank remain fixed during evaluation.Source: B IMPLEMENTATION DETAILS · p. 15
  • Extended Evolving-RL training becomes increasingly unstable beyond 100 steps and eventually collapses, consistent with the authors’ public clarification.The reproduction's collapse during extended training is noted alongside Figure 1.Source: B IMPLEMENTATION DETAILS · p. 15
  • The feedback threshold η = 0.05 preserves most raw Ralign scores while limiting larger positive and negative feedback.The choice is based on the Ralign distribution in a preliminary ALFWorld run.Source: B IMPLEMENTATION DETAILS · p. 16, Table 4 · p. 16
  • With group size G = 8, pmin = 0.1 exceeds the heuristic p ≈ 0.083 bound but softly constrains operation probabilities rather than guaranteeing every operation appears.The reference is a 50% minimum chance of at least one occurrence under a simplified independent-draw model.Source: B IMPLEMENTATION DETAILS · p. 16

C TRAINING ALGORITHM

This block describes UniSkill’s training procedure and reports evaluations, skillbank evolution, and fixed-actor case studies across the supplied sections.

C TRAINING ALGORITHM

  • Each training step collects actor rollouts, generates proposals, validates them, computes contrastive feedback, updates the shared policy, and applies accepted edits to the skillbank.The proposer objective combines proposal rewards with skill-edit support regularization.Source: C TRAINING ALGORITHM · p. 17

D.1 OUT-OF-DISTRIBUTION GENERALIZATION

  • 98.2% mean overall success with retrieval versus 96.6% without it on ALFWorld’s valid unseen split, a gain of 1.6 percentage points.Both settings use the same trained policy, and retrieval draws on that run’s learned skillbank.Source: Table 5 · p. 17, §D.1 · p. 17

D.2 SENSITIVITY TO THE SKILL CRITIC

  • UniSkill’s WebShop performance is comparable across the tested skill critics, suggesting its performance is not tied to the default critic among those models.The alternatives have single-run results, while DeepSeek-V4-Pro has three-run mean and standard deviation; Figure 7 shows similar validation-success trajectories.Source: §D.2 · p. 17, Table 6 · p. 17, Figure 7 · p. 18

D.3 ACTOR-ALIGNMENT EVALUATION

  • The actor-alignment evaluation compares proposed and original skills on the same query and fixed checkpoint, using 32 rollouts per condition and plotting success-rate differences against Ralign.At each of steps 25 and 75, 100 eligible proposals are sampled without conditioning on rollout gains.Source: §D.3 · p. 18

D.4 ADDITIONAL SKILL-EDIT DYNAMICS

  • Without support regularization, two additional runs each concentrate on one edit operation, but one favors No Edit and the other Add.The authors report this alongside Figure 6 as evidence that the collapse is not specific to Add or a single run.Source: §D.4 · p. 18, Figure 8 · p. 18

E.1.1 ADDING A MISSING SEARCH LOCATION

  • After an accepted kettle-search update was committed, the same task instance later retrieved it; success rose from 6/8 to 8/8 while mean interactions fell from 28.25 to 16.75.The update added shelves, the microwave, and stove burners as fallback search locations.Source: §E.1.1 · p. 19

E.1.2 EVOLVING A SHARED LAMP-SEARCH SKILL

  • An updated lamp-search skill was retrieved for later tasks with different target objects, and both the statue and pencil cases achieved 8/8 rollout success.The final update had Ralign = 0.11585, with its training contribution clipped to 0.05.Source: §E.1.2 · p. 20

E.2.1 WHEN ALIGNMENT AND ROLLOUT UTILITY AGREE

  • Retrieved versus proposed cabinet-opening skills yielded 7/32 versus 21/32 successes; no skill versus candidate lamp guidance yielded 26/32 versus 32/32, while newspaper candidate guidance reduced success from 17/32 to 4/32.Source: §E.2.1 · pp. 21–22

E.2.2 WHEN ALIGNMENT AND ROLLOUT UTILITY DISAGREE

These cases examine proposals with positive alignment but negative fixed-actor rollout gains, distinguishing admission safeguards, actor–skill mismatch, and omitted action ordering.

E.2.2 WHEN ALIGNMENT AND ROLLOUT UTILITY DISAGREE

  • In the kettle-search case, Ralign was positive at 0.01067 but Δ+ was −0.00449, so the proposal failed the skillbank admission criteria.The candidate broadened the search, but the kettle was already in a location prioritized by the retrieved skill.Source: §E.2.2 · p. 23
  • For two kettles, the multi-object procedure passed skillbank admission, but fixed-actor success decreased from 30/32 to 24/32 as three candidate rollouts left two placements incomplete and five placed kettles in different cabinets.All eight candidate failures reached the 50-interaction limit, while mean length increased from 21.78 to 34.03 interactions. The interpretation attributes the negative gain to the fixed actor not consistently following complete guidance, disrupting the shorter routine used without a skill.Source: §E.2.2 · p. 24
  • For two remotes, positive alignment accompanied success falling from 16/32 to 6/32: the candidate left placement-before-search implicit, and ALFWorld permits carrying only one object at a time.The actor sometimes began searching for the second remote while still holding the first.Source: §E.2.2 · p. 24
  • Positive-score proposals with negative rollout gains fell from 13/53 (24.5%) at step 25 to 10/64 (15.6%) at step 75; the Δ+ > 0 criterion excluded four step-25 cases.At step 25, nine of the 100 evaluated proposals met both Δ+ > 0 and Ralign > 0.Source: §E.2.2 · p. 24

Related papers

Loading 2610.10164v1…