Source-linked AI summary
Motif: Intrinsic Motivation from Artificial Intelligence Feedback
Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, Mikael Henaff
TL;DR
Agents in rich environments must evaluate actions without prior knowledge, while LLM common sense is difficult to connect to low-level decision-making. Motif distills LLM preferences over event captions into intrinsic rewards for RL agents. On NetHack, intrinsic-reward-only training exceeds direct score training, while combined rewards improve performance further; reported behaviors are human-aligned and prompt-steerable.
Problem
Agents entering rich environments without prior knowledge must learn what is good or useful through interaction, while harnessing LLM common sense for low-level decision-making is challenging.
Source: §1 · pp. 1–2Method
Results
Motif’s intrinsic reward alone yields higher NetHack scores than direct score training, combined rewards improve scores further, and reported behaviors are human-aligned and prompt-steerable.
Source: §1 · p. 2, §4.1 · p. 5Takeaways & Limitations
Motif uses event captions to bridge LLM knowledge and decision-making, and the approach can generalize to environments where such captioning is available.
Source: §6 · p. 9Motif assumes access to an observation dataset with sufficient coverage, although its generating policies need not have any particular proficiency.
Source: §2 · p. 2
Abstract
from arXiv · showhide
Exploring rich environments and evaluating one's actions without prior knowledge is immensely challenging. In this paper, we propose Motif, a general method to interface such prior knowledge from a Large Language Model (LLM) with an agent. Motif is based on the idea of grounding LLMs for decision-making without requiring them to interact with the environment: it elicits preferences from an LLM over pairs of captions to construct an intrinsic reward, which is then used to train agents with reinforcement learning. We evaluate Motif's performance and behavior on the challenging, open-ended and procedurally-generated NetHack game. Surprisingly, by only learning to maximize its intrinsic reward, Motif achieves a higher game score than an algorithm directly trained to maximize the score itself. When combining Motif's intrinsic reward with the environment reward, our method significantly outperforms existing approaches and makes progress on tasks where no advancements have ever been made without demonstrations. Finally, we show that Motif mostly generates intuitive human-aligned behaviors which can be steered easily through prompt modifications, while scaling well with the LLM size and the amount of information given in the prompt.
1 INTRODUCTION
Motif uses LLM preferences over event captions to derive intrinsic rewards, bridging high-level knowledge and agents’ low-level decision-making. On NetHack, it outperforms direct score training in some settings and produces behaviors with reported alignment, scalability, and steerability properties.
1 INTRODUCTION
- Motif addresses the challenge that agents entering unfamiliar environments must learn what is useful through interaction, while humans may apply common sense such as picking up a key to open a door.Manually providing such knowledge for each task does not scale.Source: §1 · p. 1
- Motif’s intrinsic reward alone yields higher NetHack scores than direct score training, and combining intrinsic and environment rewards improves scores further.The paper reports this as progress on a relatively dense-reward task.Source: Figure 1 · p. 1, §1 · p. 2
- Motif derives intrinsic rewards from LLM preferences over pairs of event captions, then uses reinforcement learning to train agents in rich observation and action spaces.The LLM evaluates coarse event descriptions rather than selecting low-level actions.Source: §1 · p. 2
- Motif’s intrinsic reward typically produces behaviors more aligned with human NetHack gameplay, and prompt changes can steer agents toward semantically diverse policies.The paper also reports anticipatory rewards and favorable scaling with LLM size and prompt information.Source: §1 · p. 2
2 BACKGROUND
The background frames sequential decision-making as learning a policy to maximize discounted reward from observations, which may include textual captions. It assumes an observation dataset with sufficient coverage, but no required proficiency level for its generating policies.
2 BACKGROUND
- A POMDP formalizes sequential decision-making through states, actions, observations, transitions, rewards, and a discount factor, with a policy maximizing expected discounted cumulative reward.Observations may include a textual caption.Source: §2 · p. 2
- Motif assumes an observation dataset with sufficient coverage, but does not require its generating policies to have any particular proficiency.Such datasets may be easier to obtain than records containing actions or rewards.Source: §2 · p. 2
- The depicted Motif workflow annotates caption pairs with LLM preferences, distills those preferences into an observation-based reward, and uses it for interactive RL training.RL may use the environment reward alongside the learned reward.Source: Figure 2 · p. 3
3 METHOD
Motif turns LLM preferences over pairs of captions into an intrinsic reward, then uses that reward to train an agent with reinforcement learning. On NetHack, the method uses gameplay observations and message-based rewards, with count normalization and thresholding to shape the reward for learning.
3 METHOD
- Motif trains a preference-based reward from LLM judgments of caption pairs, then gives that reward to an RL agent alone or combined with environment reward.The LLM can abstain when uncertain; the reward model is trained with cross-entropy on preference pairs.Source: §3 · p. 3
3.1 LEARNING FROM ARTIFICIAL INTELLIGENCE FEEDBACK ON NETHACK
- NetHack messages serve as captions; Motif builds its observation dataset from episodes collected across training checkpoints and seeds using CDGPT5.The dataset includes 100 episodes at every 100 million learning steps, repeated for 10 seeds; CDGPT5 is trained for 1 billion steps to maximize score.Source: §3.1 · p. 4
- Motif uses 70-billion-parameter Llama 2 as annotator, prompting it to summarize NetHack knowledge and analyze message pairs before choosing a preference.Pairs with unparseable labels are retried once and then removed; identical messages receive the abstention label automatically.Source: §3.1 · p. 4
- The message-based reward is thresholded and divided by an episode count term, helping reduce noisy preferences and encourage diverse outcomes.The count is how often a message has appeared during the episode; the threshold suppresses low reward values.Source: §3.1 · p. 4
- Figure 3 reports that Motif is sample-efficient on sparse-reward tasks and makes progress where no baseline learns useful behaviors.The experiments compare Motif with extrinsic-reward-only agents and agents combining extrinsic reward with RND.Source: Figure 3 · p. 5
4 EXPERIMENTS
Motif improves performance across NetHack score and sparse-reward tasks, while producing distinct survival-oriented and prompt-steerable behaviors. Its reward also scales with LLM size and can combine with task rewards in ways that produce unexpected behavior.
4.1 PERFORMANCE ON THE NETHACK LEARNING ENVIRONMENT
- Motif’s intrinsic reward alone yields more score than baselines that directly maximize score, and combining intrinsic and extrinsic rewards further improves final performance and sample efficiency.The authors describe this as the first deep-RL result in which intrinsic reward alone outperforms environment-reward training on a relatively dense-reward complex task.Source: §4.1 · p. 5
- On staircase levels 3 and 4 and the oracle task, Motif makes progress with intrinsic reward and more with combined rewards, while extrinsic-only and RND agents have zero success.Combined-reward Motif reaches about 30% success on the oracle task, where prior work reported no meaningful progress without demonstrations.Source: §4.1 · p. 5
4.2 BEHAVIOR AND ALIGNMENT ANALYSIS
- Intrinsic-only policies favor survival and exploration, with less pet killing and anticipatory rewards that ease credit assignment; combined policies pair these skills with dungeon progress.The combined agent trades a shorter lifespan for better combat skills, deeper descent, and more gold and score.Source: §4.2 · p. 6, Figure 4 · p. 6
- On the oracle task, combined rewards can produce a hallucination-based reward hack rather than actual descent, illustrating misalignment by composition.The agent survives to find and eat a yellow mold, then avoids aggression while hallucinating an apparent oracle; it also performs on a modified task requiring success without hallucination, though with lower success.Source: §4.2 · pp. 6–7, Figure 5 · p. 7, §4.3 · p. 7
4.3 SENSITIVITY TO LLM SIZE AND PROMPT
- Larger Llama 2 annotators yield higher success rates when Motif trains agents on staircase level 3 with combined rewards.The tested annotators were Llama 2 7b, 13b, and 70b.Source: §4.3 · p. 7, Figure 6 · p. 8
- Motif performs with a zero-knowledge prompt, while adding task information in one sentence significantly boosts score-task performance.The zero-knowledge setting tests whether the LLM can infer goals and desirable behavior without game-specific prompt information.Source: §4.3 · p. 8, Figure 6 · p. 8
- Semantically similar prompt rewording can change oracle behavior from hallucination-based reward hacking to descending the dungeon, despite no significant score-task performance difference.The paper identifies the interaction of prompt sensitivity and RL training as a subject for future safety research.Source: §4.3 · p. 8, Figure 6 · p. 8
4.4 STEERING TOWARDS DIVERSE BEHAVIORS VIA PROMPTING
- Prompt modifiers steer agents toward collecting gold, descending, or fighting, with more than twice baseline effectiveness for gold collection or combat.The three agents’ tailored metrics are gold collected, dungeon level reached, and monsters killed, respectively.Source: §4.4 · pp. 8–9, Table 1 · p. 9
5 RELATED WORK
Motif builds on preference learning and LLM feedback while distinguishing its use of event-caption preferences from prior reward-design approaches. It also differs from methods that generate goals or use LLMs directly as environment agents.
5 RELATED WORK
- Preference learning has a long history in sequential decision-making, while human and AI feedback have driven alignment advances in NLP.The cited work includes Thomaz et al. (2006), Knox & Stone (2009), Ouyang et al. (2022), and Bai et al. (2022).Source: §5 · p. 9
- Kwon et al. (2022) use LLMs to design rewards from complete trajectories with game states and actions, whereas Motif uses event-caption preferences as intrinsic motivation in long-horizon open-ended settings.Motif draws on LLM preferences about current events and their anticipated future developments.Source: §5 · p. 9
- Du et al. (2023) generate goals and reward cosine similarity between goal descriptions and observation captions; Motif instead elicits preferences over current events.Source: §5 · p. 9
- Methods that let LLMs act directly in environments require grounding them in both observation and action spaces.The passage cites Yao et al. (2022), Wang et al. (2023), and Carta et al. (2023).Source: §5 · p. 9
6 CONCLUSIONS
Motif derives intrinsic rewards from LLM preferences over event captions and uses them to train agents, performing strongly in NetHack with and without environment rewards. The paper also examines behavior and alignment, and relates its approach to prior work and prompting choices.
6 CONCLUSIONS
- Motif performs strongly in NetHack both without and with an external environment reward, while its behavior, alignment, scalability, sensitivity, and steerability are also analyzed.The authors describe it as a first step toward harnessing LLM common sense and domain knowledge for competent agents.Source: §6 · p. 9
- Motif uses event captions as its interface and can generalize to environments where a captioning mechanism is available.The authors suggest that more capable LLMs or prompting techniques may increase control competence, while better multimodal LLMs could remove the need for captions.Source: §6 · p. 9
A ADDITIONAL RELATED WORK
- Unlike LLM-as-policy approaches for short-horizon text environments, Motif extracts preferences into a reward rather than requiring the LLM to generate behavior directly.The authors say this can support exploration and anticipatory rewards, beyond prior reward-extraction methods limited to task specification.Source: A ADDITIONAL RELATED WORK · p. 19
- Prior work has used LLMs to provide episode-level feedback, but the cited approach assumes full state and action information, which the authors say is impractical for long-horizon tasks.The authors also connect Motif’s anticipatory rewards to earlier work on rewards from human preferences.Source: A ADDITIONAL RELATED WORK · p. 19
- Unlike intrinsic-motivation methods that encourage novelty, Motif’s LLM-derived rewards incorporate exploratory information and future rewards; unlike other diversity approaches, it steers behavior through language prompts.The authors describe its rewards as decision-aware and its annotation process as related to teacher-based exploration.Source: A ADDITIONAL RELATED WORK · p. 20
B PROMPTS AND ANNOTATION PROCESS
- Motif elicits preferences over pairs of gameplay messages after prompting the LLM to describe its NetHack knowledge, analyze the messages, and choose the more promising one or neither.When the response format fails, the conversation continues with an explicit format request; the overall response rate is around 90% for most prompt configurations.Source: B PROMPTS AND ANNOTATION PROCESS · pp. 21–22
C LLM OUTPUTS
The appendix probes whether Llama 2 knows enough about NetHack to evaluate gameplay messages. It reports that Llama 2 demonstrated game knowledge, unlike FLAN-T5 XXL, and illustrates how Llama 2 compares informative and uninformative messages.
C LLM OUTPUTS
- Llama 2 demonstrated knowledge of NetHack’s dynamics, while FLAN-T5 XXL falsely described it as a first-person shooter.The authors suggest fine-tuning an LLM as a possible direction for applying Motif to domains it does not know.Source: C LLM OUTPUTS · pp. 22–23
- When comparing a staircase-and-helm message with “It’s solid stone,” Llama 2 preferred the former as more likely to indicate game progress.Its reasoning associated the staircase with descending and the helm with potentially improved combat ability.Source: C LLM OUTPUTS · p. 23
- When both messages were uninformative—an empty message and “It’s solid stone”—Llama 2 recognized neither as better.The authors report that it could decline to choose between them.Source: C LLM OUTPUTS · p. 23
D ANALYZING MOTIF’S INTRINSIC REWARD
The reward analysis finds that Motif favors both exploratory actions and events associated with game progress, while assigning low rewards to several undesirable behaviors. The authors note that thresholding removes this low-reward tail to reduce noise, at the cost of semantics.
D ANALYZING MOTIF’S INTRINSIC REWARD
- Llama 2’s preference for a staircase message over “It’s solid stone” illustrates how its reward can anticipate progress from an event caption.The authors describe this as using game knowledge and predictive capabilities.Source: D ANALYZING MOTIF’S INTRINSIC REWARD · p. 24
- Motif’s highest-reward messages include opening doors, finding hidden passages, moving boulders, descending stairs, fighting monsters, and seeing potentially useful objects.The authors note a general bias toward longer messages that may reduce reward-function quality.Source: D ANALYZING MOTIF’S INTRINSIC REWARD · pp. 23–24
- The lowest-reward messages include mistimed actions, attacks on non-playing characters, and damage to a pet kitten.Thresholding discards the tail’s effect, reducing noise but losing semantics in the reward function.Source: D ANALYZING MOTIF’S INTRINSIC REWARD · p. 24
E DIVERSE BEHAVIORS THROUGH DIVERSE PREFERENCES
The appendix examines how prompt changes steer the messages rewarded by Motif. It compares each steered agent’s held-out message rankings with the zero-knowledge prompt baseline and describes distinct preferences for gold, exploration, and combat.
E DIVERSE BEHAVIORS THROUGH DIVERSE PREFERENCES
- The Gold Collector favors gold-related messages and also rewards door interactions, which the authors interpret as anticipatory rewards for possible gold beyond the door.Messages are compared by their ranking shifts against the zero-knowledge prompt baseline to avoid confounding from reward-scale differences.Source: E DIVERSE BEHAVIORS THROUGH DIVERSE… · pp. 24–25
- The Descender shows an even stronger preference for messages indicating exploration, including door interactions, recovering sight, and swapping places with a pet.The LLM interprets swapping places with the pet as a sign of swift movement.Source: E DIVERSE BEHAVIORS THROUGH DIVERSE… · p. 25
- The Monster Slayer favors combat-related messages, but its prompt can also lead it to reward attacking a pet.The authors attribute this to the prompt prescribing combat as the goal regardless of the target.Source: E DIVERSE BEHAVIORS THROUGH DIVERSE… · p. 25
F ADDITIONAL BASELINES
The appendix compares Motif with additional intrinsic-motivation baselines across score, staircase, and oracle tasks. It also reports an unsuccessful LLM-as-policy attempt and explains the baseline choices.
F ADDITIONAL BASELINES
- The comparison includes E3B, NovelD, Motif without LLM feedback, and BCO; BCO was omitted because it typically does not outperform its data-generating agent.E3B uses inverse-dynamics features for episodic elliptical bonuses, while NovelD uses RND-inspired novelty and episodic counts.Source: F ADDITIONAL BASELINES · p. 27
- The LLM-as-policy attempt made no progress, though the authors note that more prompt engineering might improve it; Motif instead uses simple prompts that non-experts can compose.The authors describe avoiding involved prompt optimization as part of Motif’s appeal.Source: F ADDITIONAL BASELINES · p. 27
- Motif outperforms existing intrinsic-motivation approaches on score; on staircase, some baselines help only with extrinsic reward, while none make meaningful oracle progress.Figure 7 compares these tasks in intrinsic-only and intrinsic-plus-extrinsic settings.Source: Figure 7 · p. 28, F ADDITIONAL BASELINES · p. 28
G EXPERIMENTAL DETAILS
The experimental details describe environment adjustments, reward-model training, and evaluation procedures. Additional experiments test Motif on an unhackable oracle task and examine pet-killing behavior.
G EXPERIMENTAL DETAILS
- The NetHack environment was modified so the agent could eat inventory items via the y action; intrinsic-only runs also continued interaction after reaching the goal, while rewarding it only once.The eating change addresses the environment’s inability to eat most inventory items through the usual action.Source: G EXPERIMENTAL DETAILS · p. 29
- Reward-model training used the Elliptical Bonus message encoder, an 80/20 train-validation split, 10 epochs, and a 1 × 10^-5 learning rate.The validation loss stabilized around 10 epochs.Source: G EXPERIMENTAL DETAILS · p. 29
- The reward was normalized by mean and standard deviation, then divided by episodic counts and thresholded using count exponent 3 and the empirical reward’s 50th quantile.Results were averaged over 10 seeds with standard deviations for Section 4.1, and 5 seeds with 95% confidence intervals for Section 4.4.Source: G EXPERIMENTAL DETAILS · pp. 29–30
H.1 ADDITIONAL EXPERIMENTS
- In the oracle-sober task, Motif with intrinsic and extrinsic rewards still solves the extremely sparse task, but at a lower success rate than before.The task requires standing by the oracle without hallucinating; the figure reports performance on this unhackable version.Source: Figure 8 · p. 30, §H.1 · p. 30
- Motif’s intrinsic-reward agent kills its pet 33.4% ± 25.14% of the time, versus 99.4% ± 0.63% for the extrinsic-reward agent, despite the latter’s score incentive to kill it.The authors characterize the intrinsic-reward behavior as more intuitive for most humans.Source: §H.1 · p. 30
H.2 ABLATIONS
The ablations examine count normalization, reward weighting, LLM size, and prompt wording. They identify count normalization as critical, while reporting robustness to several other choices.
H.2 ABLATIONS
- With no count normalization (β = 0), Motif can loop while seeking armor or weapons that the score task’s action set cannot interact with.The figure caption identifies count-based normalization as critical, while describing other hyperparameter choices as generally robust.Source: §H.2 · p. 30, Figure 9 · p. 31
- Performance is best when the extrinsic-reward coefficient is around 0.1, in the same range as the intrinsic coefficient; increasing it tends to reduce performance toward the extrinsic-only agent.The authors vary the extrinsic coefficient in the score task.Source: §H.2 · p. 30
- On the oracle task, the 70B Llama 2 model finds the oracle more consistently than the 13B and 7B models.Figure 10 reports the model-size sensitivity experiments.Source: Figure 10 · p. 31, §H.2 · p. 31
- In the more open-ended score task, rewording the prompt produces no significant performance difference with intrinsic reward alone or combined with extrinsic reward.The figure caption describes score-task performance as reasonably robust to prompt rewording in both settings.Source: Figure 10 · p. 31, §H.2 · p. 31
H.3 IMPACT OF DATASET DIVERSITY AND PERFORMANCE LEVEL
The authors vary the return distribution of the policies that generated the preference dataset to test its performance level and diversity. Motif tolerates substantial restrictions before performance degrades sharply.
H.3 IMPACT OF DATASET DIVERSITY AND PERFORMANCE LEVEL
- Dataset diversity is varied by removing observation pairs containing episodes in either return tail, using symmetric percentile cutoffs from 5% to 35% on the low end and 95% to 65% on the high end.Figure 12 shows the progressively reduced score distributions.Source: Figure 12 · p. 32, §H.3 · p. 33
- The base dataset contains 500,000 preference pairs, and its episode-score distribution is diverse due to variation across RL seeds and procedurally generated episodes.Figure 11 shows a long right tail, including occasional high scores associated with lucky dungeon configurations.Source: §H.3 · pp. 31–32, Figure 11 · p. 32
- Maximum dataset performance is reduced by discarding pairs containing observations from episodes above return thresholds of 1300, 1000, 700, or 400.Figure 13 depicts the resulting score distributions; the text reports that this filtering progressively reduces the performance of the policies that collected the dataset.Source: Figure 13 · p. 32, §H.3 · p. 33
- Motif has only minimal performance drops when dataset returns are capped at 700 or diversity is reduced to η = 25%, leaving around 100,000 samples; performance degrades completely only under more extreme removals.The reported severe conditions are η = 35% or a maximum score of 400, which leave only a few tens of thousands of pairs.Source: Figure 14 · p. 33, §H.3 · p. 33