PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills

Kolby Nottingham, Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Sameer Singh, Peter Clark, Roy Fox

arXiv:2402.03244v2cs.LGcs.CL

TL;DR

LLM actors need a practical way to use environment rewards for continual improvement in sequential decision-making settings. SSO constructs transferable skills from high-reward subtrajectories, supplies them in context, and prunes skills that later perform poorly. It achieves state-of-the-art ScienceWorld results and outperforms baselines by 40% on a custom NetHack task, while exceeding the previous ScienceWorld state of the art by 35%.

  • Problem

    Continual improvement of LLM actors using environment rewards is difficult, especially when direct policy-gradient finetuning is impractical or impossible for black-box models.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    SSO extracts common high-reward subtrajectories, generates abstract subgoals and instructions as transferable skills, retrieves them in context, and prunes skills with poor subsequent rewards.

    Source: §1 · p. 2, §3 · p. 4, §4.1.1 · p. 4

  • Results

    SSO achieves state-of-the-art performance in ScienceWorld, outperforming CLIN by an average of 35%, and outperforms baselines by 40% on a custom NetHack task.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SSO provides an in-context policy optimization method that rapidly adapts and transfers skill sets across sequential decision-making tasks.

    Source: §7 · p. 8

    SSO’s similarity metric may be less effective with distracting state information or low-level actions, and best performance requires intermediate subgoal rewards.

    Source: §7 · p. 8

Abstract

from arXiv · show

Large language models (LLMs) have recently been used for sequential decision making in interactive environments. However, leveraging environment reward signals for continual LLM actor improvement is not straightforward. We propose Skill Set Optimization (SSO) for improving LLM actor performance through constructing and refining sets of transferable skills. SSO constructs skills by extracting common subtrajectories with high rewards and generating subgoals and instructions to represent each skill. These skills are provided to the LLM actor in-context to reinforce behaviors with high rewards. Then, SSO further refines the skill set by pruning skills that do not continue to result in high rewards. We evaluate our method in the classic videogame NetHack and the text environment ScienceWorld to demonstrate SSO's ability to optimize a set of skills and perform in-context policy improvement. SSO outperforms baselines by 40% in our custom NetHack task and outperforms the previous state-of-the-art in ScienceWorld by 35%.

1. Introduction

SSO addresses the difficulty of improving black-box or expensive-to-finetune LLM actors in sequential environments by constructing and refining transferable skills for in-context policy improvement. It achieves state-of-the-art ScienceWorld performance and improves over baselines on a custom NetHack task.

Motivation

  • Sequential environments make naive in-context learning difficult because long trajectories and complex credit assignment limit generalization and scalability.Direct RL policy-gradient finetuning is often impractical for contemporary or closed-source LLMs.Source: §1 · p. 1

Contribution

  • SSO constructs skills from similar, high-reward subtrajectories and represents them with abstract subgoals and instructions for in-context use.It retrieves skills using initial states and refines them by filtering skills that fail to produce high future rewards.Source: §1 · p. 2, Figure 2 · p. 2

Results

  • SSO outperforms the previous top-performing CLIN agent by an average of 35% on ScienceWorld.This result is part of SSO’s evaluation across ScienceWorld and NetHack.Source: §1 · p. 2
  • SSO outperforms baselines by 40% on the authors’ custom NetHack task.The analysis and ablations indicate continual extraction of increasingly helpful skills that maximize reinforcement signals.Source: §1 · p. 2

2. Related Work

Prior work improves LLM behavior through prompting, memories, reflection, and task insights, but generally lacks scalable continual evaluation and transfer-aware policy improvement. SSO is positioned against these approaches by using environment feedback to construct and refine skills.

2.1. In-Context Learning

  • In-context learning avoids expensive finetuning, but prompt-editing methods often require continual downstream evaluation that is impractical for delayed-reward sequential tasks.These methods typically preserve interpretability by editing existing prompts.Source: §2.1 · p. 2

2.2. LLMs for Sequential Decision Making

  • LLM research has emphasized prompting and evaluation for planning and decision making, while continual learning remains difficult because finetuning is expensive and sequential tasks have long, noisy credit-assignment paths.The paper applies in-context learning to rapid adaptation and task generalization.Source: §2.2 · p. 3

2.3. LLMs with Environment Feedback

  • Reflexion, Voyager, DEPS, and ADAPT use failures to prompt retries or improvements, but most do not learn from successful completions or support task transfer.Voyager is identified as the exception regarding successful-completion learning.Source: §2.3 · p. 3
  • Voyager, ExpeL, and CLIN build memory-like information, but they differ from SSO in execution format, use of partial successes, performance-aware construction, or continual environment-based refinement.SSO compares directly with Reflexion, CLIN, and few-shot trajectories while using SSO alone for policy improvement.Source: §2.3 · p. 3

3. In-Context Policy Improvement

The paper frames in-context policy improvement as changing text inputs to an LLM actor rather than its parameters, addressing the cost and impracticality of gradient-based optimization. It introduces compact skills that preserve sequential information while improving transferability and context efficiency.

3. In-Context Policy Improvement

  • In-context policy improvement treats text inputs as policy parameters, avoiding gradient-based optimization that is costly for LLMs or impossible for black-box models.The goal is to identify text inputs that increase the trajectory-reward objective.Source: §3 · p. 3
  • Few-shot successful trajectories can create long, redundant, and poorly transferable contexts for sequential decision making.This motivates a modular memory structure rather than directly presenting complete trajectories.Source: §3 · p. 3
  • A skill combines an initial state for retrieval, a natural-language subgoal from its final state, and instructions derived from its action sequence.This representation keeps sequential information brief and modular.Source: §3 · p. 4
  • Skill-based contexts were on average 5x shorter than few-shot trajectory contexts in the comparison setup.Both actors used the same 30 trajectories, with three retrieved skills or three selected trajectories supplied at each step.Source: §3 · p. 4

4. Skill Set Optimization

SSO iteratively rolls out the current actor, constructs skills from repeated high-reward subtrajectories, and removes skills that fail under later environment feedback. Its construction pipeline combines extraction, scoring, non-overlapping sampling, and LLM-generated subgoals and instructions.

4. Skill Set Optimization

  • Each SSO iteration rolls out the current actor, constructs new skills, filters poorly performing skills, and uses the updated set in the next iteration.Relevant skills are retrieved from their initial-state similarity to the current environment state.Source: §4 · p. 4

4.1.1. EXTRACT

  • Multiple similar subtrajectories make skills more abstract and transferable while repeated behavior signals useful subgoals.Similarity is estimated from average cosine similarity of state and action embeddings.Source: §4.1.1 · p. 4
  • SSO restricts extraction to subtrajectory pairs from different trajectories and experiments with lengths from 2 to 5 steps.For each new trajectory, it finds the most similar subtrajectory in each previous completed trajectory.Source: §4.1.1 · p. 4

4.1.2. SCORE

  • Candidate subtrajectory pairs are scored using similarity, discounted future reward, and length, with experimental weights w1 = 1, w2 = 0.1, and w3 = 0.01.The weighting prioritizes similarity first, reward second, and length third.Source: §4.1.2 · p. 4

4.1.3. SAMPLE

  • Beam search selects a non-overlapping skill set that prioritizes unique, similar, high-reward, multi-step pairs while retaining pairs sampled in earlier iterations.The non-overlap constraint addresses substantial overlap among candidate skills.Source: §4.1.3 · p. 5

4.1.4. GENERATE

  • The generative LLM is used only to summarize paired subtrajectories into a numbered instruction list and a target observation representing skill success.The authors avoid using generative LLMs for extraction and scoring because they found that approach less reliable and more costly.Source: §4.1.4 · p. 5

4.2. Skill Set Refinement

  • Skill refinement records discounted future return after self-reported skill execution and removes a skill when its observed value is at or below threshold ϵ, set to zero in experiments.The actor correctly self-reports skill execution about 70% of the time.Source: §4.2 · p. 5

5. Experimental Setup

The evaluation tests SSO's adaptation and transfer in ScienceWorld and its continual skill learning in a challenging custom NetHack task. Both settings compare SSO with established LLM-actor baselines under randomized or repeated-task protocols.

5.1. Science World

  • ScienceWorld evaluates adaptation on individual variants and transfer from 10 training variants to held-out test variants across 18 task classes.Adaptation permits five attempts per test variant, while transfer uses 30 episodes before testing.Source: §5.1 · p. 5
  • ScienceWorld tasks vary their substances and environment setups while providing intermediate subtask rewards and a final successful-task score of 100.The evaluation therefore tests adaptation across task variants rather than a single fixed procedure.Source: §5.1 · p. 5
  • ScienceWorld compares SSO with GPT-4-based ReAct, Reflexion, and the previous state-of-the-art CLIN agent.ReAct adds reasoning before actions, Reflexion reflects on failed attempts, and CLIN learns transition information from past experience.Source: §5.1 · p. 5

5.2. NetHack

  • The custom NetHack task requires exploration, key-and-door interaction, levitation, and safe lava crossing using low-level character actions.Randomized starting locations and alternative levitation items prevent memorized paths and require different activation actions.Source: §5.2 · p. 6
  • NetHack compares SSO with ReAct and Reflexion over 30 iterations, evaluating every 10 iterations with frozen skill or reflection sets.CLIN is omitted because it was designed specifically for ScienceWorld, and NetHack uses GPT-4-Turbo for cost reasons.Source: §5.2 · p. 6

6. Experimental Results

SSO delivers performance gains in ScienceWorld and NetHack while continually expanding and refining its skill set. Skill analysis links this improvement to increasingly useful skills, and ablations show that refinement, reward scoring, and similarity-based extraction each contribute.

6. Experimental Results

  • SSO provides large performance gains in both ScienceWorld and NetHack, achieving state-of-the-art ScienceWorld adaptation and transfer and superior continual learning against NetHack baselines.ScienceWorld lacks standard errors because of task scale and GPT-4 cost, while NetHack error bars report standard error across five runs.Source: §6 · p. 6, Figure 5 · p. 7

6.1. Skill Analysis

  • SSO continuously expands its skill set while executed skills per trajectory increase, indicating that learned skills become more useful during training.The skill-set growth rate decreases as experience coverage increases; average extracted subtrajectory lengths are 2.64 in ScienceWorld and 2.95 in NetHack.Source: §6.1 · p. 7, Figure 5 · p. 7
  • SSO skills help the actor overcome unintuitive action requirements, including focusing on objects before melting in ScienceWorld and applying a key to unlock a NetHack door.These skills let the actor pass bottlenecks and continue exploring instead of repeating invalid actions.Source: §6.1 · p. 7, Table 2 · p. 7
  • Skill refinement prunes most skills shortly after creation, while more recently created skills tend to be used longer and later-required actions appear later in training.Figure 6 tracks when iteration-specific skills are executed and when corresponding actions first appear.Source: §6.1 · p. 7, Figure 6 · p. 8

6.2. Ablations

  • Each ablation lowers SSO performance but generally remains above baseline, with refinement slightly more impactful than reward weighting and similarity-based extraction.Similarity-based extraction still contributes by automatically selecting start and end states for high-reward subtrajectories.Source: §6.2 · p. 8, Figure 7 · p. 8

7. Discussion & Conclusion

SSO is an in-context policy optimization method that builds, evaluates, and refines transferable skills for LLM actors. It achieves state-of-the-art ScienceWorld results and outperforms all tested NetHack baselines, while depending on similarity metrics and, for best performance, intermediate subgoal rewards.

  • SSO achieves state-of-the-art results in ScienceWorld and outperforms all experimented baselines on the custom NetHack task.Source: §7 · p. 8
  • Each iteration adds commonly executed high-reward skills and filters poorly performing skills using ongoing environment feedback.Source: §7 · p. 8
  • SSO supports rapid adaptation and transfer between tasks through abstract skills, skill retrieval, and intermediate-reward-based subgoals.Source: §7 · p. 8
  • SSO’s current similarity metric may be less effective with distracting state information or low-level actions.Source: §7 · p. 8
  • Although SSO can operate without intermediate rewards, best performance requires task designers to provide intermediate subgoal rewards.Source: §7 · p. 8
  • SSO does not independently exploit negative environment feedback beyond skill-set refinement, but Reflexion can be added for that capability [Shinn et al., 2023].Source: §7 · p. 8
  • The authors position SSO as a general-purpose step toward reliable in-context policy optimization for LLM actors.Source: §7 · p. 8

Impact Statement

The work aims to help deployed LLMs continually adapt and transfer knowledge between tasks without retraining. It may improve LLM applications but does not alter their fundamental limitations.

  • The work targets continual adaptation and knowledge transfer between tasks without retraining.Source: Impact Statement · p. 9
  • The authors state that this direction may improve many state-of-the-art LLM applications without altering their fundamental limitations.Source: Impact Statement · p. 9

A. Models and Hyperparameters

Experiments use GPT-4 models and text-embedding-ada-002 across ScienceWorld and NetHack. Hyperparameters were robust in these experiments, but skill granularity may require adjustment in other domains.

B. Prompts and Examples

The appendix specifies prompts for LLM action selection and skill generation. The actor selects subgoals and actions from state information, while skill generation abstracts successful trajectory examples into reusable instructions and target observations.

B. Prompts and Examples

  • The appendix states that prompts are shared across domains except for task, state, and admissible-action descriptions, with example skills and subtrajectories provided.Source: B. Prompts and Examples · p. 11

B.1. LLM Actor Prompt

  • The LLM actor may pursue or ignore presented subgoals while interacting with a text-based game.Source: §B.1 · p. 11
  • Given the current state, the actor reflects, plans, selects a subgoal verbatim or none, and outputs the next admissible action.Source: §B.1 · p. 11
  • The actor prompt structures responses around prior-action effects, task plans, current subgoals, and next actions, followed by reusable skill instructions.Source: §B.1 · pp. 11–12

B.2. Skill Generation Prompt

  • Skill generation treats a skill as instructions plus a target state and uses environment observations to assess later execution success.Source: §B.2 · pp. 12–13
  • The generator compares example state-action subtrajectories and summarizes their commonalities before producing a skill name.Source: §B.2 · p. 12
  • The generator produces generic, example-valid numbered instructions constrained by the available action templates.Source: §B.2 · p. 12

C. SSO Code

The SSO code constructs skills from similar subtrajectories, scores and samples them, then generates and refines skills using subsequent trajectory rewards. It trains an agent by retrieving skills for each state, executing actions, and recording trajectories.

Training loop

  • SSO trains by retrieving relevant skills for the current state, selecting an action, stepping through the environment, and appending each transition to the trajectory.The loop resets the environment, retrieves skills, acts, receives the next state and reward, and records the transition until completion.Source: C. SSO Code · p. 14

Skill construction

  • The construction procedure extracts similar subtrajectory sets from recent trajectories and scores them using similarity, reward, and length with weights [1, .1, .01].The implementation uses the most recent N trajectories and extracts subtrajectories bounded by the specified minimum and maximum lengths.Source: C. SSO Code · p. 14
  • High-scoring sampled subtrajectory sets are converted into new skills, which are added to the existing skill set.Sampling avoids overlapping sets before skill generation, and the resulting skills update the stored collection.Source: C. SSO Code · p. 14

Skill refinement

  • Skill refinement computes discounted future rewards for executed skills and removes skills whose observed values are at or below epsilon.The refinement loop traverses each trajectory and uses gamma=.9 with epsilon=0 in the shown function signature.Source: C. SSO Code · p. 14

Examples

  • ScienceWorld examples show a learned movement skill transferring from art-studio or greenhouse trajectories into a kitchen state, while NetHack examples show skills representing acquisition of the Master Key of Thievery.The tables identify top learned skills and the subtrajectories used to generate the corresponding ScienceWorld and NetHack skills.Source: Table 4 · p. 15, C. SSO Code · p. 16, Table 6
Loading 2402.03244v2…