PaperlayerView exact revision on arXiv

Source-linked AI summary

AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization

Huizu Lin, Chengkai Huang, Tianqi Gao, Tao Huang, Daijiao Liu, Tongxin Li, Xiaoyan Sun, Lina Yao

arXiv:2608.21292v1cs.AI

TL;DR

Existing skill-based agent methods rarely model the lifecycle from learning skills to using them selectively, and trajectory-level routing cannot distinguish helpful from distracting actions. AUSO unifies this lifecycle with progressive reinforcement learning and action-level skill-sensitive optimization, consistently improving performance and out-of-distribution generalization across ALFWorld, WebShop, and SearchQA.

  • Problem

    Existing methods use external skills, fully internalize them, or route objectives by noisy trajectory-level outcomes that treat actions uniformly despite differing skill benefits.

  • Method

    AUSO progressively combines teacher guidance, outcome-based reinforcement learning, and JSD-based action-level skill-sensitive optimization to adapt skill use.

  • Results

    AUSO consistently outperforms competitive baselines across ALFWorld, WebShop, and SearchQA, including in-distribution and out-of-distribution settings.

  • Takeaways & Limitations

    AUSO transforms external skill guidance into transferable policy knowledge while emphasizing decisions that benefit from skills and suppressing unhelpful guidance.

  • Takeaways & Limitations

    Training interacts only with in-distribution tasks while evaluation distinguishes in-distribution and out-of-distribution domains.

Abstract

from arXiv · show

Skills play different roles as an agent's policy evolves: they should first provide learnable knowledge, then support capability formation, and finally be invoked only when they improve individual decisions. Existing methods rarely model this lifecycle. They either keep skills outside the model, fully internalize them, or select among internalization and utilization objectives through noisy task-level success rates. Such designs fragment training and assign uniform importance to actions within the same trajectory, even though skill guidance may help some decisions while distracting others. To solve these problems, we introduce AUSO (Action-level Unified Skill Optimization), which unifies skill learning and skill use through a progressive, action-aware optimization process. At the beginning of training, AUSO jointly learns from teacher guidance and environmental outcomes, enabling the policy to acquire foundational skills without losing task-oriented feedback. It subsequently emphasizes outcome-based policy optimization to consolidate autonomous problem-solving ability. As the policy matures, AUSO evaluates each sampled action under both skill-conditioned and skill-free contexts. The resulting action-level information signal is coupled with the trajectory outcome advantage, allowing beneficial skill-sensitive actions to receive stronger updates and harmful ones to be suppressed. Therefore, skills gradually transition from an external source of supervision into decision knowledge whose utilization is adapted to its action-level benefit, while reinforcement learning remains the shared backbone across all stages. Experiments on ALFWorld, WebShop, and SearchQA show that AUSO consistently improves agent performance and out-of-distribution generalization over competitive baselines.

1 Introduction

Interactive agents use reusable skills to support long-horizon decisions, but existing methods struggle to determine when skills should be externalized, internalized, or selectively used. AUSO addresses these gaps with progressive, action-aware optimization that adapts skill influence to individual decisions.

  • Motivation: Interactive LLM agents interleave reasoning, tool use, environment interaction, and feedback-driven plan revision in complex long-horizon tasks.These settings include web shopping, embodied household execution, and information access.
  • Motivation: Skills encode reusable procedural knowledge, strategies, tool-use patterns, and environment-specific guidance for complex interactive task solving.They can provide transferable priors through retrieval, composition, injection, or learnable skill-conditioned behaviors.
  • Existing Methods: Existing approaches externalize skills, internalize them into model parameters, or combine these roles through hybrid training designs.External libraries preserve interpretability and modularity but introduce context overhead and retrieval noise.
  • Optimization Gap: Trajectory-level rewards cannot directly identify which actions benefit from skill conditioning and which are distracted, creating an action-level credit-assignment challenge.Action-level signals are more precise but harder to obtain than broad trajectory-level routing.
  • AUSO: AUSO progressively learns from teacher guidance and outcomes, consolidates autonomous policy optimization, and then weights actions by skill-sensitive information coupled with trajectory advantage.The framework uses JSD to measure how skill guidance changes action distributions and suppresses harmful skill-sensitive updates.

2 Related Work

Related work improves interactive agents through reasoning, tools, memory, reflection, reinforcement learning, and reusable skills. It also studies skill retrieval, composition, benchmarking, and internalization, highlighting unresolved challenges in deciding when and how skills should be used.

  • Interactive Agents: Prompting, tool augmentation, and broad agent benchmarks support complex decision making across web, shopping, embodied, software, and general evaluation settings.Examples include chain-of-thought, ReAct, Tree-of-Thoughts, API calls, search, and calculation.
  • Agent Improvement: Memory, reflection, experience reuse, and reinforcement learning improve agents through feedback, reusable programs, and interaction-based post-training.Reflexion uses verbal feedback without parameter updates, while Voyager constructs reusable programs and curriculum-driven exploration.
  • Agent Skills: Agent skills represent reusable procedures such as executable primitives, task plans, natural-language strategies, heuristics, tool-use patterns, and stored procedures.Their representations vary between embodied systems and LLM-agent skill libraries.
  • Skill Use: Skill retrieval and composition research shows that agents must decide which skills to incorporate, how many to select, and in what order.Skill-usage benchmarking further examines degradation under realistic retrieval and reuse conditions.
  • Skill Internalization: Skill internalization methods absorb skills into parameters or latent representations to reduce reliance on explicit prompts at inference time.Related work also studies continual experience internalization and test-time skill evolution, emphasizing the importance of granularity and timing.

3 Method

AUSO treats skill learning as a progressive reinforcement-learning lifecycle, moving from teacher-guided internalization to autonomous exploration and action-level skill utilization. It uses action-level skill signals to weight updates according to how skill guidance affects decisions, while retaining GRPO as the optimization backbone.

  • Problem Formulation: AUSO models interactive agents as policies that generate actions from history states, optionally conditioned on reusable procedural skills and trained with trajectory-level outcomes.The formulation distinguishes skill-conditioned and skill-free policies while tasks receive rewards such as success or task score.
  • Unified Skill Lifecycle: AUSO replaces hard trajectory-level routing with a continuous lifecycle: internalize useful skills, explore autonomously through GRPO, then adapt skill use per action.This addresses fuzzy routing boundaries and uniform treatment of actions within a trajectory.
  • Early-stage Teacher-guided Skill Internalization: When all sampled trajectories fail and GRPO provides nearly no advantage signal, AUSO applies teacher-guided action-distribution supervision to identify actions requiring correction.The added supervision is restricted to no-signal task groups.
  • Early-stage Teacher-guided Skill Internalization: During early internalization, a skill-conditioned teacher guides a skill-free student through detached teacher distributions, transferring general skills into policy parameters without additional environment rollouts.The teacher uses general skills, while optimization updates only the student.
  • Early-stage Teacher-guided Skill Internalization: AUSO measures action-level teacher–student information gain with Jensen-Shannon divergence, aggregates token discrepancies within each action, and normalizes them across valid actions.Action-span aggregation prevents longer textual actions or reasoning tokens from receiving disproportionately large supervision weights.
  • Unified Skill Lifecycle: AUSO gradually ramps up and then anneals teacher supervision, while later stages activate action-level skill utilization and modulate outcome-driven updates by each action’s skill benefit.Clipping limits teacher-guided reweighting so high-discrepancy actions receive stronger correction without dominating the overall gradient.

4 Experiments

AUSO is evaluated across embodied, web-based, and search-oriented agent benchmarks, using ID/OOD performance where applicable and ablations of its optimization components. It consistently improves performance over baselines, with especially strong gains on unseen tasks.

  • Datasets and evaluation: AUSO is evaluated on ALFWorld, WebShop, and SearchQA, covering embodied decision-making, web interaction, and knowledge-intensive reasoning.ALFWorld and WebShop use ID/OOD evaluation, while SearchQA reports task success without an ID/OOD split.
  • Main results: AUSO achieves the strongest overall performance across WebShop, ALFWorld, and SearchQA, improving under both ID and OOD settings.The reported improvements extend beyond seen tasks to unseen-task generalization.
  • Main results: WebShop averages reach 49.7 ID and 51.2 OOD, improving over Skill0.5 by 9.3 and 10.6 points, respectively.The gains are particularly pronounced for Electronics and Other under ID and Access and Beauty under OOD.
  • Main results: ALFWorld averages reach 94.3 ID and 67.9 OOD, surpassing Skill0.5 by 1.2 and 9.4 points, with Pick2 increasing from 33.3 to 54.2.The OOD improvement is especially substantial on challenging unseen tasks.
  • Main results: SearchQA reaches a 47.5 overall average, exceeding SkillRL, Skill1, and Skill0.5, while achieving the best scores on TriviaQA and HotpotQA.AUSO also shows stronger validation performance than Skill0.5 during training, particularly on multi-hop tasks.
  • Ablation studies: Removing any AUSO component degrades performance; removing action-level internalization reduces ALFWorld from 94.3/67.9 to 85.1/54.7 in ID/OOD settings.A 2:5:3 allocation across internalization, exploration, and utilization performs best, while K=4 reaches 94.3 ID and 67.9 OOD success rates.

5 Conclusion

AUSO unifies skill internalization and utilization through a progressive, action-level reinforcement-learning framework. Experiments show consistent improvements over competitive baselines and stronger out-of-distribution generalization.

  • AUSO unifies skill internalization and utilization through an information-gain-based, action-level optimization framework for LLM agents.Skills progressively move from external guidance to decision knowledge used according to action-level benefit.
  • AUSO progressively learns from external skill guidance, consolidates autonomous decision-making through reinforcement learning, and uses action-level signals to adjust skill utilization.The framework emphasizes decisions that benefit from skills and suppresses decisions where guidance is unhelpful.
  • Across ALFWorld, WebShop, and SearchQA, AUSO consistently outperforms competitive baselines and achieves stronger out-of-distribution generalization.Ablations also verify contributions from progressive scheduling, action-level optimization, and the uncertainty-based gate.

A.1 Analysis of the Global Uncertainty Gate

The global uncertainty gate scales action-level credit assignment according to the balance of successful and failed rollouts. It improves reliability when outcomes provide contrast, preserves the original optimization direction, and redistributes rather than changes average task-level update scale.

  • The success–failure information contrast becomes unreliable when the rollout success rate approaches 0 or 1.One outcome group then becomes absent or severely underrepresented.
  • The action-level modulation preserves the sign of the original GRPO advantage and cannot introduce an unsupported optimization direction.The modulation changes update strength without reversing the trajectory-level reward signal.
  • Within each task, centered action-level modulation leaves the average optimization scale unchanged while redistributing credit across actions.The gate controls reliability according to success–failure contrast, whereas the modulation reallocates credit within trajectories.

A.2 A Unified View of Internalization and Utilization

AUSO uses one counterfactual action-level information operator across skill internalization and utilization, changing how the signal enters optimization rather than how it is measured. GRPO remains the shared backbone while skill information is progressively introduced, reduced, or used to refine action updates.

  • Unified signal: The same operator serves direct teacher supervision during internalization and action-level advantage modulation during utilization.For internalization it measures teacher–student discrepancy; for utilization it measures dependence on task-specific skill conditioning.
  • Progressive curriculum: AUSO progressively changes skill-information contributions instead of routing individual tasks into discrete optimization regimes.The internalization coefficient decreases while utilization activates, with curriculum proportions controlling transition timing.
  • Progressive curriculum: GRPO remains the persistent optimization backbone as skill information evolves from internalization through autonomous exploration to utilization.Utilization is a continuous, bounded refinement of GRPO rather than an independent competing objective.
  • Forms of unification: AUSO unifies the signal, optimization, and backbone: both stages use one counterfactual JSD operator, update one policy, and reduce continuously to standard GRPO when auxiliary information is absent or unreliable.This design avoids combining unrelated auxiliary losses heuristically.
  • Action-level information operator: AUSO compares skill-conditioned and skill-free executions of the same policy at each visited state to measure action-level skill information.The two counterfactual contexts avoid comparing separately generated trajectories, and token-level divergences are averaged over each environment action.
  • Action-level information operator: JSD provides a symmetric, bounded measure of how skill conditioning changes the action distribution.Its bounded scale supports stable action-level modulation, unlike potentially unbounded one-sided KL divergence.
  • Interpretation: The action-level information score measures skill influence, not task-success probability; reinforcement learning determines whether skill-induced changes are useful.A large score indicates a substantial policy change under skill conditioning but does not by itself establish benefit.

B Experimental Setup

AUSO is evaluated on ALFWorld, WebShop, and SearchQA using ID-only training and separate ID/OOD evaluation. The setup reports task-level and aggregate performance while fixing the skill banks during training.

  • Benchmarks and protocol: Experiments cover ALFWorld, WebShop, and SearchQA, with policy optimization restricted to in-distribution training tasks.Out-of-distribution tasks are reserved for validation and final evaluation, and ID/OOD performance is reported separately.
  • ALFWorld: ALFWorld uses three ID task types for training and three disjoint OOD task types for evaluation, with separate ID and OOD skill banks.The interaction horizon is 30 actions, and macro-averaged success rates are reported within each domain.
  • WebShop: WebShop partitions 12,087 goals into four ID and three OOD product domains, using 3,320 ID training goals, 454 ID evaluation goals, and 207 OOD evaluation goals.Training uses only ID goals, while evaluation reports per-domain and ID/OOD averages.
  • SearchQA: SearchQA trains on NQ and HotpotQA and evaluates seven datasets, including five OOD datasets excluded from policy optimization.The protocol reports per-dataset accuracy and ID/OOD averages, with at most four search actions per episode.
  • Training analysis: Training curves compare AUSO with GRPO on ALFWorld using success rate, reward, policy entropy, and episode length.The figure caption states that AUSO achieves higher success and reward with lower entropy and shorter episodes.
  • Implementation: All benchmarks initialize from Qwen2.5-7B-Instruct and use GRPO with learning rate 1×10^-6, group size G=8, one policy epoch, and 150 optimization steps.Skill banks remain fixed during training to evaluate policy learning and OOD skill utilization without online corpus changes.

C.1 Training Dynamics of AUSO on ALFWorld

AUSO outperforms Skill0.5 under the same training budget and acquires successful behaviors earlier and more effectively than standard GRPO. Its optimization also yields lower policy entropy and shorter episodes during later training.

  • Training-budget comparison: Under an identical 150-step budget, AUSO maintains stronger ID and OOD performance than Skill0.5, ruling out extra training steps as the primary improvement source.The comparison instead supports more effective and robust skill-conditioned behavior learning.
  • Comparison with GRPO: AUSO achieves higher training success rates and mean rewards than GRPO throughout most of optimization.The reported dynamics indicate that successful task behaviors are acquired earlier and reinforced more effectively.
  • Comparison with GRPO: During middle and later training, AUSO exhibits lower policy entropy and progressively shorter episode lengths than GRPO.The passage specifically associates reduced policy entropy with the observed optimization behavior.

C.2 Case Study: AUSO is More Sample-Efficient than Skill0.5

Case studies across SearchQA and ALFWorld show AUSO completing tasks with concise, goal-directed action sequences, while Skill0.5 often performs redundant exploration or retrieval. These examples illustrate more economical interaction trajectories and successful OOD behavior for AUSO.

  • SearchQA: AUSO retrieves the answer-bearing passage, extracts “Manchester,” and terminates after two steps on the SearchQA query.The trajectory uses a minimal answer span once sufficient evidence is available.
  • SearchQA: Skill0.5 retrieves the same evidence but issues redundant searches, then produces an overlong exact-match answer trajectory.The additional searches introduce irrelevant retrieval before the final answer.
  • ALFWorld OOD: Skill0.5 fails its OOD ALFWorld task after exhausting 25 steps while repeatedly exploring checked locations and failing to find the target lettuce.The trajectory revisits the fridge, countertop, and cabinet without establishing reliable progress.
  • ALFWorld OOD: AUSO completes the OOD ALFWorld task “put a spraybottle in garbagecan” in four steps through locate–acquire–navigate–place actions.Each action makes explicit progress toward task completion.
  • Cross-task takeaway: Across these cases, AUSO produces more economical and goal-directed trajectories than Skill0.5 on SearchQA and ALFWorld.The examples connect concise evidence use with compact environment interaction.
Loading 2608.21292v1…