PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling

Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, Roy Fox

arXiv:2301.12050v2cs.LGcs.CL

TL;DR

RL agents often learn without prior world knowledge, although high-level subgoals and transitions could support planning and exploration. DECKARD uses few-shot LLMs to hypothesize an Abstract World Model, then learns modular subgoal policies while verifying and correcting that model through Minecraft experience. The approach improves sample efficiency by an order of magnitude and remains robust to LLM decomposition errors.

  • Problem

    RL agents typically learn tabula rasa, while sparse-reward Minecraft crafting requires knowledge of complex item dependencies for efficient exploration.

    Source: §1 · p. 1

  • Method

    DECKARD prompts an LLM to hypothesize a subgoal-dependency AWM, then alternates Dream-phase branch selection with Wake-phase modular policy learning and environment-based verification.

    Source: §1 · pp. 1–2, §4.2.1 · p. 4

  • Results

    DECKARD improves sample efficiency by an order of magnitude and remains robust to errors in the LLM’s task decomposition.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Environment-grounded verification lets RL agents use noisy, large-scale LLM knowledge for exploration without relying on its predictions being fully correct.

    Source: §7 · p. 8

    DECKARD requires language-grounded environments, assumes state abstractions can be specified, and considers only deterministic AWM transitions.

    Source: §7 · p. 8

Abstract

from arXiv · show

Reinforcement learning (RL) agents typically learn tabula rasa, without prior knowledge of the world. However, if initialized with knowledge of high-level subgoals and transitions between subgoals, RL agents could utilize this Abstract World Model (AWM) for planning and exploration. We propose using few-shot large language models (LLMs) to hypothesize an AWM, that will be verified through world experience, to improve sample efficiency of RL agents. Our DECKARD agent applies LLM-guided exploration to item crafting in Minecraft in two phases: (1) the Dream phase where the agent uses an LLM to decompose a task into a sequence of subgoals, the hypothesized AWM; and (2) the Wake phase where the agent learns a modular policy for each subgoal and verifies or corrects the hypothesized AWM. Our method of hypothesizing an AWM with LLMs and then verifying the AWM based on agent experience not only increases sample efficiency over contemporary methods by an order of magnitude but is also robust to and corrects errors in the LLM, successfully blending noisy internet-scale information from LLMs with knowledge grounded in environment dynamics.

1. Introduction

DECKARD addresses sparse-reward Minecraft crafting by using an LLM to hypothesize subgoal dependencies, then grounding those dependencies through modular RL and environment experience. This LLM-guided exploration improves sample efficiency while remaining robust to decomposition errors.

  • RL commonly trains tabula rasa, despite evidence that sequential decision-making systems can exploit prior task knowledge.Source: §1 · p. 1
  • Minecraft crafting poses difficult sparse-reward exploration because items form a deep technology tree with complex dependencies and high branching factors.Source: §1 · p. 1
  • DECKARD hypothesizes an Abstract World Model by prompting an LLM to decompose tasks into subgoals, then alternates Dream exploration with Wake-phase modular policy learning and verification.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • 12x sample-efficiency improvement over ablated DECKARD without LLM guidance is reported when exploring toward specific tasks.The paper also reports that unguided exploration takes over twice as long for most items.Source: §1 · p. 2
  • DECKARD consistently outperforms baselines as errors are introduced into the LLM’s task decomposition.Source: §1 · p. 2
  • The method uses environment dynamics to ground noisy LLM knowledge, enabling RL agents to exploit large-scale prior information for exploration.Source: §1 · p. 2

2. Related Work

Related work connects language-assisted decision making, interaction-grounded world models, modular RL, and Minecraft’s sparse-reward crafting challenge. DECKARD differs by using LLMs to predict and verify exploration dynamics rather than directly generate action plans.

2.1. Language-Assisted Decision Making

  • Prior language-assisted agents commonly use textual knowledge, web queries, or pretrained models to support decision making.Source: §2.1 · p. 2
  • Out-of-the-box LLM action plans are often ungrounded in environment dynamics, motivating methods that ground language or structure generated plans.Source: §2.1 · p. 2
  • DECKARD uses LLMs for exploration by predicting and verifying environment dynamics through experience, which the authors identify as novel.Source: §2.1 · p. 3

2.2. Language Grounded in Interaction

  • Prior interaction-grounding work learns or grounds textual world models through environment interaction; DECKARD instead obtains its model few-shot from an LLM and verifies it during interaction.Source: §2.2 · p. 3

2.3. Modularity in RL

  • Modular RL composes independent policies to facilitate training and generalization, and prior work applies modular policies to exploration in Zork and Minecraft.Source: §2.3 · p. 3

2.4. Minecraft

  • Minecraft crafting remains challenging because sparse rewards, scarce items, and deep technology-tree dependencies make exploration difficult.Source: §2.4 · p. 3
  • Because existing Minecraft agents often simplify environment parameters, DECKARD focuses its evaluation on the benefits of LLM-guided exploration.Source: §2.4 · p. 3

3. Background

The background frames DECKARD as modular RL over compositional subgoals and LLM prompting for structured world knowledge. Subgoal modules chain through a dependency DAG, while prompted code-like output supplies candidate item dependencies.

3.1. Modular Reinforcement Learning

  • DECKARD assigns each discovered Minecraft subgoal an independently trained policy module and learns a dependency DAG for transitioning between subgoals.Source: §3.1 · p. 3
  • A sequence of chained subgoal modules can navigate connected portions of the discovered DAG and reach arbitrary connected goal states.Source: §3.1 · p. 3

3.2. Large Language Models

  • LLMs use prompt engineering to adapt pretrained language-model knowledge to downstream tasks, including decision-making and action planning.Source: §3.2 · pp. 3–4
  • The algorithm initializes an LLM-hypothesized graph, visit counts, and verified-node set before alternating branch sampling with Wake-phase execution.Source: §3.2 · p. 4
  • DECKARD prompts OpenAI’s Codex to generate a Python dictionary of Minecraft item dependencies, which is mapped into a graph of items and dependencies.Source: §3.2 · p. 4

4. DECKARD

DECKARD represents subgoal dependencies as a DAG, updates that model from experience, and uses LLM predictions to guide which frontier branches to explore. Dream selects candidate branches, while Wake executes policies, verifies nodes, and expands the model.

4.1. Abstract World Model

  • The AWM is a DAG over textual state abstractions, with directed edges representing dependencies between subgoals; experiments use the agent’s inventory as the abstraction.Source: §4.1 · p. 4
  • DECKARD updates the AWM by adding experienced transitions and recovers paths whose subgoal policies are executed toward previously experienced targets.Source: §4.1 · p. 4

4.2. LLM Guidance

  • Without LLM guidance, the agent can learn modular subgoal policies but must construct the AWM tabula-rasa.Source: §4.2 · p. 4
  • LLM guidance predicts candidate edges, while environment interaction verifies or corrects them and tracks verified nodes.Source: §4.2 · p. 4

4.2.1. DREAM PHASE

  • Dream computes a verified frontier, prunes it to branches on the predicted path when possible, and samples a branch for Wake exploration.Source: §4.2.1 · p. 4

4.2.2. WAKE PHASE

  • Wake executes the sampled subgoal sequence, updates the AWM with experience, and adds successful new nodes to the verified set.Source: §4.2.2 · pp. 4–5
  • LLM initialization reduces unnecessary policy finetuning outside the predicted target path, while failed recipes trigger additional subgoal policies for exploration.Source: §4.2.2 · p. 5

5. Experiment Setup

DECKARD is evaluated on Minecraft crafting and exploration using an LLM-generated Abstract World Model, modular subgoal policies, and environment-specific observations and actions. The experiments test AWM prediction, finetuning, environment interaction, and exploration with and without LLM guidance.

5. Experiment Setup

  • DECKARD maps Minecraft inventory items to AWM subgoals and learns modular policies that compose to collect and craft arbitrary technology-tree items.The evaluation covers crafting tasks requiring ingredient collection and target-item crafting, plus open-ended exploration.Source: §5 · p. 5, §5.4 · p. 5

5.1. Predicting the Abstract World Model

  • Codex predicts recipes, tool requirements, and workbench requirements for 391 Minecraft items, producing the hypothesized AWM before training or inference.Table 1 evaluates predictions for item type, workstation requirements, ingredient correctness, and exact recipe matches across all items and the 37-item tool technology tree.Source: §5.1 · p. 5, Table 1 · p. 5

5.2. Subgoal Finetuning

  • Each subgoal module finetunes lightweight adapters with an RL objective, using VPT as the pretrained starting policy rather than training policies from scratch.Adapters support separate modular policies while retaining the shared VPT model parameters in memory.Source: §5.2 · p. 5

5.3. Environment Details

  • The MineDojo setup uses pixel observations and a large multi-discrete action space, while inventory quantities determine transitions between subgoal policies.Crafting subgoals use single craft actions, whereas collectable-item subgoals require finetuned navigation or collection policies.Source: §5.3 · p. 5

5.4. Experiments

  • Open-ended experiments compare verified-AWM growth for DECKARD with and without LLM guidance and a non-finetuned VPT baseline, alongside goal-driven crafting evaluations.The study also tests robustness to artificial errors in the hypothesized AWM.Source: §5.4 · p. 6, Figure 2 · p. 6

6. Experiment Results

Across open-ended exploration, targeted crafting, and robustness tests, DECKARD uses LLM-guided AWM structure to focus exploration, improve crafting efficiency, and withstand decomposition errors.

6.1. Open-Ended Exploration

  • DECKARD’s open-ended exploration compares AWM growth against unguided DECKARD and a VPT baseline while intrinsically attempting new craftable items.Its frontier is smaller than the full AWM, increasing the likelihood of sampling useful items, with prioritization stopped after c0 failed attempts.Source: §6.1 · p. 6
  • DECKARD’s LLM-guided exploration avoids dead ends by sampling useful frontier nodes, whereas unguided exploration takes 2x longer to learn wooden-pickaxe-based cobblestone mining.The guided agent also corrects an erroneous prediction that glass requires no crafting or tools.Source: §6.1 · p. 6, Figure 3 · p. 6

6.2. Crafting Tasks

  • DECKARD’s success rates are evaluated against pretrained VPT and VPT-a across logs, pickaxes, cobblestone, furnace, sand, and glass under random-world-seed limits.The evaluation allows 1,000 steps for collectable items and 5,000 for other items, with training capped at 6 million steps.Source: §6.2 · p. 7, Figure 4 · p. 7
  • DECKARD outperforms direct reinforcement-learning training on item tasks and reaches technology-tree items where the baseline completely fails.The task set includes logs, wooden and stone pickaxes, cobblestone, furnace, sand, and glass.Source: §6.2 · p. 7
  • LLM guidance yields an order-of-magnitude sample-efficiency improvement by training only subgoal policies for predicted prerequisites rather than about fifteen collectable items.For stone-pickaxe crafting, guided DECKARD trains policies for needed items such as logs and cobblestone.Source: §6.2 · p. 7, Figure 5 · p. 7
  • Compared with prior Minecraft agents, DECKARD achieves sample efficiency equal to or better than existing methods for crafting a stone pickaxe despite their differing shortcuts and interfaces.The comparison includes methods using expert demonstrations, simplified action spaces, or intermediate crafting rewards.Source: §6.2 · p. 7, Table 2 · p. 8

6.3. Robustness

  • DECKARD remains robust when the hypothesized AWM contains inserted or deleted recipe edges, outperforming unguided DECKARD even under large LLM-output errors.Ingredient-quantity errors were most common, yet DECKARD often compensated with surplus ingredients.Source: §6.3 · p. 8, Figure 6 · p. 8

7. Discussion & Conclusion

DECKARD combines an LLM-hypothesized Abstract World Model with environment-grounded modular reinforcement learning, while its prompting and parsing pipeline turns Minecraft knowledge into subgoal dependencies. The system alternates between Dream-based exploration and Wake-based policy learning, correcting predicted dependencies through experience.

7. Discussion & Conclusion

  • DECKARD alternates Dream-phase subgoal selection with Wake-phase modular policy learning to ground noisy LLM world knowledge in environment dynamics.The AWM represents transitions between subgoals in a directed acyclic graph, and agent experience verifies or corrects the hypothesis.Source: §7 · p. 8
  • DECKARD’s broader contribution is a general method for alternating predicted frontier expansion with subgoal execution to learn compositional modular policies.The approach uses environment dynamics to ground noisy LLM knowledge for sequential decision making.Source: §7 · p. 8

A.1. Prompting Details

  • The prompting pipeline asks Codex to generate a nested Python dictionary of Minecraft item requirements and recipes for all 391 items.Entries encode crafting-table and furnace requirements, required collection tools, and ingredient quantities.Source: §A.1 · p. 12

A.2. Parsing Details

  • The generated dictionary treats zero-length recipes as collectible items without AWM parents, producing parsable entries for all but one Minecraft item.The unparsed exception was brown mushroom block.Source: §A.2 · p. 12
  • Parsing normalizes several predicted identifiers, including plank, wood, their variants, and cane, to align common Minecraft names with MineDojo identifiers.These predicted names are common Minecraft identifiers but do not always match MineDojo’s identifiers.Source: §A.2 · p. 13
  • The pipeline removes circular dependencies from the hypothesized AWM before using it for decision making.It removes selected edges involving crafting tables, furnaces, tools, and mutually dependent recipe items; four circular cases were identified.Source: §A.2 · p. 13

A.3. Additional Results

The additional-results materials document the Minecraft technology-tree evaluation and report that few-shot Codex prompting can produce structured knowledge for decision making. The supplied passages provide the evaluation scope and qualitative finding but no additional table values.

A.3. Additional Results

  • The evaluation covers 37 Minecraft items from the tool technology tree.Source: Table 3 · p. 13
  • Table 4 is intended to report additional Codex metrics for predicting the Minecraft Abstract World Model.The supplied passage identifies the table’s scope but does not provide its metric values.Source: Table 4 · p. 13
  • Few-shot prompting with Codex shows that LLMs can generate structured knowledge for decision making, although the predictions are not perfect.The authors therefore treat the predictions as hypotheses verified through environment interactions.Source: §A.3 · p. 13

B. Subgoal Finetuning

DECKARD’s subgoal policies are built by fine-tuning VPT with adapter-based reinforcement learning, while training on nearby-target seeds and evaluating on random seeds tests generalization. The supplied materials emphasize the fine-tuning setup rather than overall performance values.

B. Subgoal Finetuning

  • Subgoal policies are trained on seeds where target items are nearby and evaluated on random Minecraft world seeds to measure generalization.Cobblestone is described as the most ubiquitous target and sand as the least, based on generalization to random seeds.Source: Figure 7 · p. 14
  • The supplied materials include a dedicated table of DECKARD subgoal-finetuning hyperparameters.Source: Table 5 · p. 14

B.1. VPT Finetuning

  • DECKARD fine-tunes VPT with PPO by updating transformer adapters and the value head while leaving the rest of the agent unchanged.Each transformer layer receives two residual adapter modules with a 16x hidden-state reduction.Source: §B.1 · p. 14
  • A KL loss to the non-finetuned VPT policy helps prevent catastrophic forgetting early in training, with its coefficient decaying over time.The experiments reaffirmed the importance of this term even though most VPT weights remain fixed.Source: §B.1 · p. 14

B.2. MineClip Reward

MineClip supplies shaped rewards for subgoal-policy fine-tuning, while the Minecraft setup controls initialization, episode length, and world-seed variation. For ambiguous AWM transitions, the implementation retains the first experienced edge as the presumed simplest path.

B.2. MineClip Reward

  • MineClip reward shaping subtracts 21 from the alignment score, scales it by 0.005, and smooths it over 50 steps.Non-zero alignment reward is provided only when the episode reaches a new maximum, with an additional +1 for adding the target item to inventory.Source: §B.2 · p. 14

B.3. Minecraft Settings

  • Subgoal-policy training initializes agents with items from parent nodes, terminates episodes after 1,000 steps, and generates a new world.Source: §B.3 · p. 15
  • Training uses one world seed per subgoal policy and evaluates on random seeds because poor training seeds can strand agents far from targets, underwater, or on islands.The authors find that VPT can generalize to random seeds after training on a single seed.Source: §B.3 · p. 15

C.1. Disambiguating the World Model

  • When multiple AWM transitions exist, the implementation keeps the first experienced edge between nodes as an assumed simplest path.Minecraft may offer alternatives such as mining coal or burning wood in a furnace.Source: §C.1 · p. 15

C.2. Additional Results

Figure 8 shows that the ground truth AWM enables earlier glass crafting and supports crafting glass bottles, which other methods did not reach.

  • The ground truth AWM agent learns to craft glass sooner during open-ended exploration.Source: Figure 8 · p. 15
  • The ground truth AWM agent also learns to craft glass bottles.Source: Figure 8 · p. 15
  • None of the other methods reaches glass-bottle crafting in this comparison.Source: Figure 8 · p. 15
Loading 2301.12050v2…