Source-linked AI summary
Guiding Pretraining in Reinforcement Learning with Large Language Models
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, Jacob Andreas
TL;DR
RL exploration is difficult without dense rewards, while novelty-based methods can pursue irrelevant behaviors in large environments. ELLM uses pretrained language models to suggest goals from current state descriptions and rewards their achievement, producing broader common-sense behavior coverage and usually matching or improving downstream performance.
Problem
Dense reward functions are costly to engineer, and novelty-based exploration can emphasize irrelevant discoveries in large environments.
Method
ELLM prompts a pretrained language model with the agent’s current state description, then rewards semantic similarity between suggested goals and achieved transitions.
Results
ELLM-trained agents achieve better coverage of useful behaviors during pretraining and usually match or improve baseline performance across downstream tasks.
Takeaways & Limitations
Pretrained language-model priors can bias reinforcement-learning exploration toward common-sense, plausibly useful behaviors without direct human intervention.
Takeaways & Limitations
ELLM is limited by prompt sensitivity, missing domain knowledge, required captioning, and the time and cost of querying large language models.
Abstract
from arXiv · showhide
Reinforcement learning algorithms typically struggle in the absence of a dense, well-shaped reward function. Intrinsically motivated exploration methods address this limitation by rewarding agents for visiting novel states or transitions, but these methods offer limited benefits in large environments where most discovered novelty is irrelevant for downstream tasks. We describe a method that uses background knowledge from text corpora to shape exploration. This method, called ELLM (Exploring with LLMs) rewards an agent for achieving goals suggested by a language model prompted with a description of the agent's current state. By leveraging large-scale language model pretraining, ELLM guides agents toward human-meaningful and plausibly useful behaviors without requiring a human in the loop. We evaluate ELLM in the Crafter game environment and the Housekeep robotic simulator, showing that ELLM-trained agents have better coverage of common-sense behaviors during pretraining and usually match or improve performance on a range of downstream tasks. Code available at https://github.com/yuqingd/ellm.
1. Introduction
ELLM addresses the difficulty of exploring without dense rewards by using pretrained language models to suggest context-sensitive, common-sense goals. It biases exploration toward human-meaningful behaviors and reports better pretraining coverage with competitive downstream performance.
- Hand-defining dense reward functions requires substantial engineering effort except in simple cases.
- The method biases exploration toward goals that are diverse, context-sensitive, and human-meaningful without direct human intervention.
- Novelty-based intrinsic rewards can prioritize irrelevant phenomena, such as noisy TVs or moving leaves, rather than useful behaviors.
- ELLM queries language models with the agent’s current context and rewards accomplishment of suggested goals.
- ELLM-trained agents show better coverage of useful behaviors during pretraining and outperform or match baselines on downstream tasks.
2. Background and Related Work
Intrinsic motivation helps agents explore complex outcome spaces, but undirected novelty can favor irrelevant behaviors. ELLM instead uses language and pretrained models to guide exploration toward plausible, useful goals.
- Sparse rewards and complex action spaces make undirected exploration unlikely to discover long, specific action sequences.
- Intrinsic motivation methods explore outcomes rather than actions, including state diversity in knowledge-based methods and skill diversity in competence-based methods.
- Maximizing outcome diversity can drive agents toward irrelevant behaviors when environments contain effectively infinite novelty.
- Language-based goals are easy for non-experts to express, can be abstract, and may support better generalization through compositionality.
- Unlike related approaches, ELLM uses pretrained LLMs to constrain task-agnostic exploration without pretrained low-level policies, demonstrations, or task-specific prompts.
3. Structuring Exploration with LLM Priors
ELLM structures competence-based exploration around language-model-generated goals that are diverse, feasible, common-sense sensitive, and context sensitive. It matches goals to observed transitions using language embeddings and supports open-ended or closed-form goal generation.
- Problem Description: The partially observed environment is modeled with states, actions, observations, transition dynamics, rewards, and a discount factor.
- Problem Description: Competence-based intrinsic motivation samples goals and trains a goal-conditioned policy using goal-specific intrinsic rewards.
- Goal-based Exploration Desiderata: Exploration goals should be diverse, common-sense sensitive, and feasible in the current environment configuration.
- Goal Generation with LLMs: ELLM uses autoregressive language models to generate goals and masked language models to represent them as vectors.
- Goal Generation with LLMs: At each timestep, the LLM receives available actions and a caption of the current observation to produce context-sensitive goals.
- Goal Generation with LLMs: ELLM supports open-ended goal generation and closed-form yes/no evaluation, with the latter suited to large but delimitable goal spaces.
- Rewarding LLM Goals (Rint): Rewards are based on semantic similarity between transition captions and generated goals, using cosine similarity from a pretrained SentenceBERT encoder.
- Training Settings: The method filters goals already achieved in the episode and can train either goal-conditioned or goal-free policies.
4. Experiments
Experiments evaluate whether prompted language models generate useful exploratory goals and whether ELLM improves exploration and downstream performance in Crafter and Housekeep. Across these environments, ELLM generally achieves stronger pretraining coverage and matches or outperforms comparison methods, with performance varying by task and training strategy.
- Experimental setup: The experiments test whether pretrained LLMs generate diverse, commonsense, context-sensitive goals and whether these goals improve downstream performance.The evaluation covers Crafter and Housekeep, comparing ELLM with intrinsic-motivation baselines, an oracle, and ELLM ablations.
- Experimental setup: ELLM uses prompted goal suggestions, filters previously achieved goals, computes suggestion-achievement rewards, and updates an RL agent from the resulting experience.The open-ended Crafter implementation prompts Codex with available actions and a current-state description, using generated text as suggested goals.
- Crafter: 64.9% of suggested Crafter goals are context-sensitive, sensible, and achievable, while 13.6% violate context-sensitivity and 16.4% violate common sense.Another 5% are disallowed by Crafter’s physics, although most of those remain context- and commonsense-sensitive.
- Crafter: ELLM unlocks about 6 Crafter achievements per pretraining episode versus 9 for the ground-truth Oracle and fewer than 3 for APT, RND, and Novelty.This supports the finding that common-sensical goals improve exploration beyond methods focused only on generating novel behaviors.
- Crafter: Goal-conditioned ELLM outperforms unconditioned ELLM, while goal conditioning provides no significant pretraining advantage.Downstream performance varies widely by task and seed; goal-conditioned ELLM is best on average and ELLM is the only method with nonzero performance across all tasks.
- Housekeep: In Housekeep, ELLM yields higher pretraining success in three of four tasks and matches or outperforms baselines during downstream training.LLM mismatch-identification accuracy exceeds 87%, while match-identification accuracy ranges from 50–90%; inaccuracies affect pretraining success.
5. Conclusions and Discussion
ELLM biases exploration toward common-sense, plausibly useful goals and is less suitable when goals cannot be expressed naturally in language or state captions. Its effectiveness is also limited by prompt sensitivity, missing domain knowledge, captioning requirements, and the cost of querying large models.
- ELLM concentrates exploration on common-sensical goals rather than undirected novelty, helping in environments with many possible but few plausibly useful behaviors.
- ELLM is less helpful when goal-based exploration is limited, human common-sense is irrelevant or linguistically inexpressible, or state information lacks a natural-language representation.
- LLM performance is sensitive to prompt choice, and missing domain knowledge can produce false negatives that permanently block learning key skills.In Crafter, the LLM never suggests creating wood pickaxes.
- ELLM requires state and transition captions, whose training may be less efficient than collecting demonstrations or hard-coding rewards in some environments.
- Suggestion quality improves with model size, but regularly querying massive LLMs may be time- and cost-prohibitive in some reinforcement-learning environments.
- ELLM-like approaches could extend beyond text by using generative models to suggest plausible visual goals or goals in other state representations.
A. Crafter Pretraining Ablation
Figure 8 evaluates achievement coverage after pretraining across methods using median, interquartile mean, and mean statistics over repeated trials and seeds. Textual observations are distinguished from vision-only variants.
- Achievement coverage is reported using the median, interquartile mean, and mean over 10 evaluation trials, each averaged across 10 episodes and 5 seeds.The figure reports 50 evaluation points per statistic.
- Opaque bars represent methods using textual observations in addition to visual observations, while dashed lines show the gap relative to less-opaque vision-only variants.
- Figure 8 reports the number of ground-truth achievements unlocked per episode at the end of pretraining for each method listed in Table 1.
B. Crafter Downstream Training
Crafter downstream evaluation tests pretrained exploration policies on tasks ranging from familiar short behaviors to long prerequisite chains and sequential or compositional skills. The experiments also vary action-space difficulty and prompt design.
- Downstream tasks: Seven downstream Crafter tasks plus the Crafter game reward are used for fine-tuning pretrained policies.
- Downstream tasks: Place Crafting Table and Attack Cow are relatively easy tasks because most agents commonly encounter them during pretraining.
- Downstream tasks: Make Wood Sword requires multiple crafting prerequisites, so many agents rarely or never achieve it during pretraining.
- Downstream tasks: Mine Stone requires a long sequence of gathering, crafting, exploration, and mining, and fully sparse rewards are replaced with semi-sparse subtask rewards.
- Downstream tasks: Deforestation and Gardening test whether goal conditioning directs agents toward repeated or sequential behaviors that novelty-driven pretraining may not reinforce.
- Downstream tasks: Plant Row is challenging because the state captioner has no concept of a “row,” preventing an ELLM agent from learning it zero-shot during pretraining.
- Action-space design: The modified Crafter action space expands from 17 dimensions to 260 possible verb-plus-noun actions, including arbitrary combinations such as drinking grass.
- Prompt analysis: The prompt’s listed actions account for 6 / 10 (60%) of good actions received by the ELLM agent and 6 / 21 (28.6%) of all rewarded actions received.
F. Housekeep Tasks
The Housekeep setup scopes RL pretraining to object-placement tasks, using language descriptions and LLM-suggested receptacle judgments to guide agents.
- Housekeep pretraining focuses on the first four tasks, with five misplaced objects per task, rather than the benchmark’s full mapping and planning setup.
- The task prompt asks whether a household item should be stored in or on a specified receptacle.Examples include rejecting a dirty spoon on a chair and accepting a mixing bowl in a dishwasher or a clean sock in a drawer.
- Agents use DQN with double Q-learning, dueling networks, and multi-step learning.
- Policies process 84 × 84 images through the Nature Atari CNN, while text- or goal-conditioned variants add SBERT language embeddings.
- Housekeep captions use semantic sensors to describe visible objects, receptacles, held objects, and their placements.A learned captioner is trained from trajectories using frozen CLIP visual embeddings and semantic-state differences.
- The learned captioner can introduce false-negative and false-positive rewards, making caption quality a direct limitation of the reward computation.The supplied analysis reports an average false-negative rate of 11%, with chop grass reaching 100%.
K. Crafter LLM Analysis
The Crafter analysis examines whether LLM-generated reward labels correspond to the actions agents actually perform. It finds substantial reward confusion across achievements.
- Crafter reward labels are dominated by good actions at 66.5% and bad actions at 32.4%.Impossible actions cannot be achieved, while context-insensitive suggestions may still succeed when the relevant object is present.
- The analysis includes examples of LLM suggestions in Crafter and a table of LLM classification accuracy for each Housekeep task.
- Figure 12 reports the probability of detecting each column achievement when the row achievement was truly unlocked.The estimates use expert PPO trajectories, with 27 to 100 datapoints per row.
L. Novelty Bonus Ablation
Removing ELLM’s novelty bonus causes agents to repeat a small set of easy goals instead of exploring diverse behaviors.
- Without the novelty bonus, agents repeat a small set of easy goals and fail to explore diversely.The ablation repeatedly rewards achievement of the same goal to isolate the novelty bias.
M. Analysis of Downstream Training Approaches
The paper compares finetuning with guided exploration after pretraining. Guided exploration is more reliable in Crafter, while both approaches work for ELLM in Housekeep because its pretraining and finetuning tasks are similar.
- The study compares finetuning pretrained exploration-policy weights with training a new policy from scratch while using the pretrained policy for ε-greedy exploration.
- In Housekeep, both downstream-training methods are effective for ELLM.
- In Crafter, finetuning performs poorly across ELLM, baselines, and oracles, and is less reliable than guided exploration.The authors associate this with reward-density and reward-magnitude mismatch, which can cause useful behaviors to be unlearned.
- Housekeep shows only minor performance dips at finetuning because its ELLM pretraining and human-directed finetuning tasks are similar.RND and APT have greater pretrain-finetune mismatch and perform comparatively better with guided exploration.
- Figure 14 presents mean ± standard deviation success rates across seven Crafter downstream tasks over five seeds.Each run finetunes the pretrained agent with a learning rate of 2e −5.
- NovelD performs similarly to other prior-free intrinsic-motivation methods during pretraining.
O. Code and Compute
The paper describes its code release, LLM API setup, caching strategy, and hardware, while warning that bias-related risks require further study before real-world deployment.
- Code is released under the MIT license, with Crafter and Housekeep retaining their respective licenses.
- The experiments use OpenAI APIs with Codex and Davinci after smaller GPT-3 models degraded performance.Davinci costs $0.02/1000 tokens, while Codex is free to use.
- Caching substantially reduces API queries; without it, one 5-million-step run would spend 27 hours querying the API.Each API query takes .02 seconds, before additional rate-limit delays.
- Experiments run on NVIDIA TITAN Xp and GeForce RTX 2080 Ti GPUs, with 2-3 seeds per GPU.
- Using LLMs as reward functions requires mitigating harmful biases, especially before deploying ELLM in open-ended real-world learning.Suggested mitigations include filtering generations, adding prompting guidelines, or constraining goal spaces with a closed-form variant.