PaperlayerView exact revision on arXiv

Source-linked AI summary

MaestroMotif: Skill Design from Artificial Intelligence Feedback

Martin Klissarov, Mikael Henaff, Roberta Raileanu, Shagun Sodhani, Pascal Vincent, Amy Zhang, Pierre-Luc Bacon, Doina Precup, Marlos C. Machado, Pierluca D'Oro

arXiv:2412.08542v1cs.AIcs.CLcs.LG

TL;DR

Natural-language skill descriptions could provide an accessible way to inject human decision-making knowledge, but existing LLM-based skill design can require substantial specialized effort. MaestroMotif uses LLM feedback, code generation, and reinforcement learning to create and combine skills. In NLE, it succeeds on complex tasks without further training where even task-trained approaches struggle.

  • Problem

    Existing LLM-based frameworks for combining skills often require specialized human effort to design low-level skills, limiting their applicability and generality.

    Source: §1 · p. 2

  • Method

    MaestroMotif uses LLM feedback to turn natural-language skill descriptions into rewards, generates skill-boundary and skill-selection code, and trains skill policies with reinforcement learning.

    Source: §1 · p. 2

  • Results

    MaestroMotif succeeds without further training on complex NLE navigation, interaction, and composite tasks where even approaches trained for those tasks struggle.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach uses LLMs to address two recurring labor-intensive needs in agent design: programming control policies and designing reward functions.

    Source: §6 · p. 11

    The behaviors MaestroMotif can express are limited by its available skills, whose composition may not represent a satisfactory policy for a task.

    Source: §6 · p. 11

Abstract

from arXiv · show

Describing skills in natural language has the potential to provide an accessible way to inject human knowledge about decision-making into an AI system. We present MaestroMotif, a method for AI-assisted skill design, which yields high-performing and adaptable agents. MaestroMotif leverages the capabilities of Large Language Models (LLMs) to effectively create and reuse skills. It first uses an LLM's feedback to automatically design rewards corresponding to each skill, starting from their natural language description. Then, it employs an LLM's code generation abilities, together with reinforcement learning, for training the skills and combining them to implement complex behaviors specified in language. We evaluate MaestroMotif using a suite of complex tasks in the NetHack Learning Environment (NLE), demonstrating that it surpasses existing approaches in both performance and usability.

1 INTRODUCTION

The introduction frames natural-language skill descriptions as a way to communicate high-level knowledge without prescribing low-level actions, while noting that current skill design often demands specialist labor. MaestroMotif uses LLM feedback, code generation, and reinforcement learning to create and combine skills; in NLE, it succeeds on complex tasks without further training where task-trained approaches struggle.

1 INTRODUCTION

  • MaestroMotif largely outperforms existing methods zero-shot across NLE task categories, including methods trained on each task.The introduction reports success on complex navigation, interaction, and composite tasks where even task-trained approaches struggle.Source: Figure 1 · p. 1, §1 · p. 2
  • Prior LLM-controlled skill-design frameworks require technical knowledge and substantial specialized human labor, limiting their applicability and generality.They may require collecting skill-specific data, developing heuristics, or manually engineering rewards.Source: §1 · p. 2
  • AI-assisted skill design pairs human-provided task knowledge with automation of the lower-level, time-consuming parts of skill design.The paper presents this as a human-AI collaboration paradigm.Source: §1 · p. 2
  • MaestroMotif turns natural-language skill descriptions into skill-specific rewards using LLM feedback, then uses generated code and reinforcement learning to train and combine the skills.The LLM generates initiation and termination functions and a policy over skills; each skill policy is trained to maximize its corresponding reward.Source: §1 · p. 2
  • MaestroMotif shifts policy production from low-level actions to high-level skills, where an LLM can generate code policies to combine them.The paper argues that programming-language features can express sophisticated behaviors that may be hard to learn with neural networks.Source: §1 · p. 2

2 BACKGROUND

The background formalizes language-conditioned decision-making and skills as options that can be selected in sequence. It also frames AI-assisted skill design as translating natural-language skill prompts into options.

2 BACKGROUND

  • A language-conditioned MDP includes states, actions, language task specifications, rewards, transitions, a discount factor, and an initial-state distribution.Its reward function maps states and task specifications to reward values.Source: §2 · p. 2
  • A deterministic Markovian skill is an option with an initiation function, an intra-option policy, and a termination function; a policy over options selects the next option when one terminates.The skill-design problem is framed as constructing options that can be composed by such a policy.Source: §2 · p. 2
  • The illustrated MaestroMotif process turns skill descriptions into rewards using LLM preferences, generates skill boundaries and a train-time policy as code, and trains each skill policy with reinforcement learning.When a skill terminates, the policy over skills chooses another available skill.Source: Figure 2 · p. 3
  • AI-assisted skill design maps natural-language prompts describing high-level skills into options, and the approach generalizes to partially observable settings.The experiments use memory-conditioned policies.Source: §2 · p. 3

3 METHOD

MaestroMotif combines reinforcement learning from LLM feedback with LLM-generated code to design skills from language descriptions and compose them for zero-shot control. Its method proceeds from skill reward design and activation rules through training and deployment, with a NetHack implementation using preference annotations and skill-conditioned policies.

3.1 AI-ASSISTED SKILL DESIGN WITH MAESTROMOTIF

  • MaestroMotif uses LLM preferences over sampled interaction pairs to fit a separate reward function for each natural-language skill description.Motif distills the preferences into skill-specific rewards by minimizing negative log-likelihood with a Bradley–Terry model.Source: §3.1 · pp. 3–4
  • An LLM translates each skill specification into initiation and termination code, defining when that skill can start and when its execution should stop.Source: §3.1 · p. 4
  • A designer-specified training policy interleaves skills so they learn under more deployment-relevant state distributions and avoid redundant learning; RL then trains each selected skill on its corresponding reward until termination.The training policy selects among skills allowed by their initiation functions, and each skill policy is trained using call-and-return execution.Source: §3.1 · p. 4

3.2 ZERO-SHOT CONTROL WITH MAESTROMOTIF

  • For deployment, an LLM turns a task description and template into code for a policy over skills, which selects trained skills without further training.The policy sends the selected skill index to the skill network; initiation and termination functions determine availability and when execution ends.Source: §3.2 · pp. 4–5, Figure 3 · p. 4

3.3 MAESTROMOTIF ON NETHACK

  • The NetHack implementation defines five skills—Discoverer, Descender, Ascender, Merchant, and Worshipper—for exploration, stairway finding, and context-dependent interaction.The Descender and Ascender pair reflects the bottleneck-options concept.Source: §3.3 · p. 5
  • For skill preferences, the implementation adds dungeon and experience levels and 100-step state differences to message observations, giving the LLM compressed trajectory history.It uses Llama 3.1 70B to annotate preference pairs from observations collected by a Motif baseline and unannotated human gameplay.Source: §3.3 · p. 5
  • Llama 3.1 405B generates Python initiation, termination, and skill-selection code, refined with unit tests; a shared PPO-trained network uses a one-hot skill identifier.The shared network lets skills share an environment representation while reducing potential negative effects from a multi-head architecture.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments assess MaestroMotif on complex early-game NetHack tasks and report strong task performance. Further analyses examine LLM code-generator scale, skill architecture, and training schedule.

4 EXPERIMENTS

  • Figure 4 depicts labeled early-game areas and entities; the benchmark covers navigation, interaction, and composite tasks, comparing language-promptable methods with task-trained and privileged-reward baselines.Navigation and composite tasks use success rate; interaction tasks use collected objects. Non-zero-shot methods require several GPU-days of task-specific training.Source: Figure 4 · p. 6, §4 · p. 6, §4.1 · pp. 6–7

4.1 PERFORMANCE EVALUATION

  • MaestroMotif outperforms all baselines in navigation and interaction tasks, despite methods trained specifically for each task.Table 1 summarizes the comparison, including a PPO baseline with privileged reward information.Source: Table 1 · p. 7, §4.1 · p. 7

4.2 COMPARISON TO SCORE MAXIMIZATION

  • Score-maximizing approaches can collect substantial score yet rarely exhibit benchmark behaviors, and their objective does not align with NetHack community preferences.Figure 5 compares MaestroMotif with score-maximizing approaches in interaction and navigation tasks.Source: Figure 5 · p. 8, §4.2 · p. 9

4.1 PERFORMANCE EVALUATION

  • For composite tasks, MaestroMotif is reported as the first approach competitive at this complexity while learning through the lowest-level action space.The tasks involve long-term dependencies; the code policy can compose skills using sophisticated logic, memory, or higher-level time abstraction.Source: Table 2 · p. 7, §4.1 · p. 8

4.3 ALGORITHM ANALYSIS

  • One-hot conditioning on the active skill is crucial for learning skill policies; separate policy heads collapse in performance, possibly from gradient interference.The gradient-interference explanation is presented as a hypothesis.Source: §4.3 · p. 9
  • Figure 7 and the accompanying analysis show a trend of improved task performance with larger LLM code generators.The LLM generates code policies over skills for zero-shot adaptation.Source: Figure 7 · p. 9, §4.3 · p. 9
  • Synchronous skill learning induces an emergent curriculum: simpler skills become useful before more complex, context-dependent skills.When learned in isolation, a majority of skills do not maximize their own rewards; the training-time skill policy creates the situations in which advanced skills can be initiated.Source: §4.3 · pp. 9–10

5 RELATED WORK

MaestroMotif connects LLM-based hierarchical control and code-policy methods while differing in how it obtains and trains skills. It also uses natural language to connect intuitive skill descriptions with low-level execution.

LLM-based hierarchical control methods

  • Unlike SayCan and PaLM-E (Ahn et al., 2022; Driess et al., 2023), MaestroMotif trains skills from intrinsic rewards synthesized from observational data and language descriptions, rather than human teleoperation datasets and hand-coded rewards.Voyager (Wang et al., 2024) assumes access to low-level control primitives.Source: §5 · p. 10

Hierarchical reinforcement learning

  • Where many hierarchical reinforcement-learning methods learn representations, MaestroMotif uses natural language for compositional, interpretable skills and links intuitive concepts to sensorimotor execution through LLM feedback.The passage connects some NetHack skills to earlier bottleneck and in-betweenness ideas (Iba, 1989; McGovern & Barto, 2001; Menache et al., 2002; S¸ims¸ek & Barto, 2004).Source: §5 · p. 10

HRL approaches with code policies

  • Among code-policy HRL approaches, MaestroMotif uses LLMs to generate rewards, initiation/termination functions, and policies over skills, simplifying human interaction with the system.Related approaches include concurrent hierarchical Q-learning, policy sketches (Andreas et al., 2017), and program-guided agents.Source: §5 · p. 10

6 DISCUSSION

MaestroMotif uses limited prompting to connect high-level planning with low-level sensorimotor control in NetHack. Its expressible behaviors remain bounded by the skills available for composition.

6 DISCUSSION

  • MaestroMotif links high-level planning and low-level sensorimotor control in NetHack using a limited number of prompts.The authors describe it as achieving untapped levels of controllability for sequential decision-making in this setting.Source: §6 · p. 10
  • LLMs help address two recurring labor-intensive needs in agent design: manually programming control policies and manually designing reward functions.Source: §6 · p. 11
  • The behaviors MaestroMotif can express depend on its available skills, so designers should consider the target behaviors when designing that skill set.A satisfactory task policy may not be representable through a given set of skills.Source: §6 · p. 11

A.1 SKILL REWARDS

The skill-reward procedure elicits preferences by comparing gameplay descriptions against a skill-specific preference, then extracts the selected description from the LLM response.

A.1 SKILL REWARDS

  • For each skill, the reward prompt asks the LLM to compare two gameplay descriptions against a skill-specific preference.The prompt first asks the model to discuss its knowledge of NetHack and the game's goal, then requests an analysis grounded in the descriptions and NetHack knowledge.Source: §A.1 · p. 16
  • The prompt template is customized with different skill modifiers and uses chain-of-thought prompting before requesting annotations.The authors say this follows closely the strategy of Klissarov et al. (2024) and cite Wei et al. (2022) for chain-of-thought prompting.Source: §A.1 · p. 16
  • A regular expression extracts labels 1, 2, or none; when extraction fails, a follow-up prompt requests the desired format.The reported response rate with Llama 3.1 is around 98% for most prompt configurations.Source: §A.1 · p. 16

A.2 POLICY OVER SKILLS

MaestroMotif uses an LLM to generate Python code for policies over skills, with refinement guided by unit-test execution traces. The described policy-generation process also defines skill selection and produces code for task strategies.

A.2 POLICY OVER SKILLS

  • The LLM generates code policies over skills from a natural-language skill set and desired strategy, with self-refinement using unit-test traces for up to three iterations.The model judges whether the trace satisfies the strategy and, if not, revises its code; the unit test is itself generated by the LLM.Source: §A.2 · pp. 16–17
  • The generated exploration policy selects skills and defines methods for navigating between the Dungeons of Doom and the Gnomish Mines.The authors also show a policy for the composite Discovery Hunger task.Source: §A.2 · p. 17
  • The policy can select among Discoverer, Descender, Ascender, Merchant, and Worshipper skills, whose described behaviors include exploring, changing dungeon levels, and interacting with shopkeepers or altars.Source: §A.2 · pp. 17–18
  • When the LLM adds attributes to a policy, it is queried to generate code that gathers their values from the NLE.Source: §A.2 · p. 18
  • The average generated policy over skills contains 9030 tokens, while initiation and termination functions average 810 tokens.The authors report throughput up to 969 tokens/second and generation costs as low as $3/1M tokens, making a policy take largely less than a minute and cost a few cents.Source: §A.2 · p. 18

A.3 INITIATION AND TERMINATION

The LLM defines skill initiation and termination functions as well as skill policies. These functions determine when skills can be selected and when their execution ends.

A.3 INITIATION AND TERMINATION

  • The LLM defines initiation and termination functions for the skills, alongside their policies.Source: §A.3 · p. 19
  • A termination function indicates when a skill should finish, while an initiation function indicates when the high-level policy can select it.Source: §A.3 · p. 19
  • The authors omit unit-test self-refinement for initiation and termination functions because they are significantly simpler than the high-level policy.Source: §A.3 · p. 19

A.4 CODE REFINEMENT

Self-generated unit-test refinement improves LLM Coder results for the 405b Llama model, while smaller models show no significant difference.

A.4 CODE REFINEMENT

  • The section labels the example as a unit-test execution trace and identifies it as output from a test written by the LLM.Source: §A.4 · p. 20
  • The example unit-test trace alternates between discoverer and descender as dungeon depth increases, then switches to ascender as depth decreases.The trace covers 20 turns and reaches depth 5 before ascending.Source: §A.4 · p. 20
  • Self-generated unit-test refinement improves results with the 405b Llama model, but shows no significant difference for smaller models.Figure 9 presents the comparison.Source: §A.4 · p. 19, Figure 9 · p. 20

A.5 ENVIRONMENT AND METHOD DETAILS

The implementation modifies NetHack actions and trains skill rewards with an LLM-based intrinsic reward, including a short-window count transformation.

A.5 ENVIRONMENT AND METHOD DETAILS

  • Agents can eat or quaff any item at random by issuing the command associated with key y; all trained agents use these conditions, except behavior-cloning agents in Figure 5.The environment also includes pray, cast, and enhance actions.Source: §A.5 · p. 20
  • The LLM-generated unit test simulates how the environment reacts to the select-skill method, and the prompt restricts it to writing the test only.The prompt says not to create new classes, functions, or imports.Source: §A.5 · p. 21
  • The example skill set comprises discoverer, descender, ascender, merchant, and worshipper, covering exploration, stairway navigation, shopkeeper interaction, and altar interaction.Source: §A.5 · p. 21
  • The reward transformation divides by an observation count over the last 20 steps, enabling short-memory counting instead of full-episode counting.The authors say this opens the transformation to environments where full-course counting is difficult in practical settings.Source: §A.5 · pp. 21–22
  • LLM-based rewards are trained for 20 epochs at a 1 × 10^-5 learning rate, with count exponent 3 and skill-specific reward thresholds.Thresholds use the 85th empirical reward quantile except for Discoverer, which uses the 95th; Motif and Embedding Similarity use the 50th. MaestroMotif uses only the LLM intrinsic reward, without Motif’s additive environment-reward hyperparameter α.Source: §A.5 · p. 22

A.6 BENCHMARK DESIGN AND MOTIVATION

This section describes a benchmark built around recognizable NetHack progress landmarks and useful inventory interactions, alongside supplementary analyses of skill design and selection, planning, and prompting. It also details prompts for skill behavior, including initiation and termination conditions for Merchant and Worshipper.

A.6 BENCHMARK DESIGN AND MOTIVATION

  • The benchmark targets Gnomish Mines, Minetown, and Delphi landmarks, plus interactions with altars and shopkeepers that help identify items.Delphi is included because reaching it is necessary for completing NetHack, whereas interacting with the Oracle is not; altar use reveals an item’s blessed/cursed/uncursed status, and shopkeeper offers can identify item types.Source: §A.6 · pp. 22–23

A.7 HIERARCHICAL ARCHITECTURE

  • Skill policies use one network conditioned on a one-hot skill index; a multi-head alternative collapsed in performance, attributed to interference between skill gradients.Skills activate at different frequencies, with Discoverer activated almost 50 times more often than Worshipper.Source: §A.7 · p. 23, Figure 10 · p. 23

A.8 ADDITIONAL ABLATIONS

  • Without player statistics or diffStats, preference-elicitation performance falls to almost 30% of full potential, while Dungeons and Data episodes help achieve full performance.The authors attribute the need for diffStats to NetHack observations being non-Markovian and say the dataset adds coverage of possible interactions and observations.Source: §A.8 · pp. 23–24, Figure 11 · p. 24

A.9 CONSIDERATIONS FOR THE SKILL SELECTION

  • Because skill-selection policies and initiation and termination functions remain fixed during learning, an emerging curriculum develops easier skills before harder ones; online adaptation could improve efficiency and robustness.The current high-level policy is checked with a self-generated unit test, which generally avoids particular failure modes but is not foolproof.Source: §A.9 · p. 24

A.10 CONNECTIONS TO THE PLANNING LITERATURE

  • MaestroMotif’s skill sequencing instantiates the options formalism and connects to symbolic planning abstractions, including PDDL and Hoare logic.The authors identify further investigation of connections between options and symbolic representations as promising, particularly for LLMs.Source: §A.10 · pp. 24–25

A.11 ADDITIONAL PROMPTING EXPERIMENTS

  • Providing MaestroMotif’s prior knowledge to LLM Policy and Motif through equivalent prompting does not improve their performance.The authors suggest the challenge is learning how and when to use that information from context.Source: Table 3 · p. 25, §A.11 · p. 25
  • The skill descriptions assign exploration and combat to Discoverer, stair travel to Descender and Ascender, selling to Merchant, and item identification at altars to Worshipper.Source: §A.11 · p. 29
  • Merchant and Worshipper should terminate whenever any of their preconditions is true.Source: §A.11 · p. 29
  • The initiation prompt asks the LLM to define preconditions only for Merchant and Worshipper, identifying relevant entities through ASCII characters, encodings, and colors from a color map.Source: §A.11 · p. 29
Loading 2412.08542v1…