PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft

Hao Li, Xue Yang, Zhaokai Wang, Xizhou Zhu, Jie Zhou, Yu Qiao, Xiaogang Wang, Hongsheng Li, Lewei Lu, Jifeng Dai

arXiv:2312.09238v2cs.AIcs.CLcs.CVcs.LG

TL;DR

Minecraft’s sparse rewards and complex decision space make learning difficult for reinforcement-learning agents. Auto MC-Reward uses LLMs to design, verify, and refine dense rewards based on agent trajectories; experiments report better results than sparse and existing dense rewards on diamond and plains tasks.

  • Problem

    Minecraft’s sparse rewards and large decision space challenge reinforcement learning, and effective learning still requires billions of environmental interactions.

    Source: §1 · p. 2

  • Method

    Auto MC-Reward uses LLMs to design and verify dense reward functions, then analyze agent trajectories to suggest reward refinements.

    Source: §3 · p. 3

  • Results

    Across diamond exploration and plains tasks, Auto MC-Reward outperformed sparse and existing dense rewards; diamond acquisition reached 36.5% success using raw information.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments support automated dense reward design for improving learning on complex Minecraft tasks with sparse rewards.

    Source: §5 · p. 8

    Trajectory analysis is limited to the last 32 frames because of LLM context length, hindering analysis of long-term failures.

    Source: §5 · p. 8

Abstract

from arXiv · show

Many reinforcement learning environments (e.g., Minecraft) provide only sparse rewards that indicate task completion or failure with binary values. The challenge in exploration efficiency in such environments makes it difficult for reinforcement-learning-based agents to learn complex tasks. To address this, this paper introduces an advanced learning system, named Auto MC-Reward, that leverages Large Language Models (LLMs) to automatically design dense reward functions, thereby enhancing the learning efficiency. Auto MC-Reward consists of three important components: Reward Designer, Reward Critic, and Trajectory Analyzer. Given the environment information and task descriptions, the Reward Designer first design the reward function by coding an executable Python function with predefined observation inputs. Then, our Reward Critic will be responsible for verifying the code, checking whether the code is self-consistent and free of syntax and semantic errors. Further, the Trajectory Analyzer summarizes possible failure causes and provides refinement suggestions according to collected trajectories. In the next round, Reward Designer will further refine and iterate the dense reward function based on feedback. Experiments demonstrate a significant improvement in the success rate and learning efficiency of our agents in complex tasks in Minecraft, such as obtaining diamond with the efficient ability to avoid lava, and efficiently explore trees and animals that are sparse in the plains biome.

1. Introduction

Auto MC-Reward uses LLMs to design, verify, and iteratively refine dense rewards for Minecraft tasks. The paper reports improved task performance, including diamond acquisition using only raw information.

Challenge

  • Minecraft’s sparse rewards and huge decision space make RL difficult, requiring billions of interactions despite behavior-cloning pretraining [3].Minecraft is used for RL research [16].Source: §1 · p. 2

Prior work

  • Existing dense rewards for sparse tasks [2] [27] [35] [37] often fail to make significant progress on complex, long-horizon Minecraft tasks, even after behavior cloning.The paper illustrates this difficulty with underground diamond exploration.Source: §1 · p. 2

Motivation

  • LLMs can summarize historical actions and success or failure signals to design and refine auxiliary rewards, helping agents overcome Minecraft’s sparse-reward challenge.The paper motivates this approach by drawing on human practice-based learning, such as avoiding lava after an accidental burning death.Source: §1 · p. 2

Method

  • Its Reward Designer creates task-specific dense rewards, the Reward Critic self-verifies them, and the Trajectory Analyzer summarizes agent trajectories to guide reward refinement.The Designer uses environment and task descriptions; trajectory feedback addresses potential biases or oversights in the LLM’s understanding.Source: §1 · p. 2

Results

  • Auto MC-Reward significantly outperforms sparse and existing dense rewards on diamond and plains tasks, learns lava avoidance, and reaches 36.5% diamond success using raw information.The benchmarks include underground diamond exploration and approaching trees and animals in the plains biome.Source: §1 · p. 2

2. Related Work

Prior Minecraft-agent research emphasizes learning algorithms and high-level planning, while Auto MC-Reward uses LLMs to produce explainable, code-form dense rewards that can be refined from agent experience. The paper distinguishes this approach from existing reward-design methods for sparse-reward tasks and robotics.

Related Work

  • Minecraft-agent work spans hierarchical RL [30] [42], curriculum learning [23], imitation learning [1], foundation models [3], world models [17], and LLM high-level planning [19] [46] [51] [53].Auto MC-Reward addresses reward design, which the paper describes as orthogonal to prior work focused mainly on RL algorithms or high-level planning.Source: §2 · p. 2
  • Auto MC-Reward generates explainable reward functions from task descriptions and refines them using agent experience, unlike prior Minecraft-agent work focused mainly on RL algorithms or high-level planning.The authors frame automated dense-reward design as complementary to those lines of work.Source: §2 · pp. 2–3
  • Handcrafted dense rewards require task-specific trial and error, while auxiliary methods [6] [20] [37], self-imitation [35], and goal-conditioned RL [2] [24] [27] [45] have uncertain applicability to complex Minecraft.Pre-trained reward models [10] [11] [14] [31] [32] assign values to intermediate states, but are black-box, difficult to improve from agent experience, and not guaranteed to generalize to new tasks.Source: §2 · pp. 2–3
  • Evolutionary reward search [15] is typically constrained to task-specific templates, while LLM-based reward assignment [7] [11] [25] [29] is black-box and computationally costly; Auto MC-Reward generates white-box code.The LLM-based approaches described prompt models with trajectories and task descriptions, generating rewards on the fly.Source: §2 · p. 3
  • Unlike robotics reward coders, Auto MC-Reward targets uncertain, diverse Minecraft scenarios: L2R [50] needs templates, while Text2Reward [49] and EUREKA [33] require complete environment descriptions and human feedback.The paper says these requirements are unavailable in open worlds and motivates more precise, thorough reward design for Minecraft.Source: §2 · p. 3

3. Method

Auto MC-Reward iteratively designs executable dense rewards, checks them for errors, and refines them using failures observed in agent trajectories. Its components divide reward coding, code review, and trajectory analysis while using predefined observations and constrained reward scales.

3. Method

  • Reward Designer proposes executable Python rewards, Reward Critic checks them, and Trajectory Analyzer uses agent trajectories to guide subsequent revisions.A reward that passes review is used to train agents; feedback from trajectory analysis informs the next reward update.Source: §3 · p. 3

3.1. Reward Designer

  • Reward Designer generates executable reward code from task descriptions, game information, and requirements, and receives agent-performance analysis when updating it.Its predefined observations include nearby block distances across current and previous steps, inventory changes, health, and past locations.Source: §3.1 · p. 3
  • A multi-step memory dictionary lets reward functions retain information across steps, such as tree distances that reward approaching a tree relative to the prior step.The memory begins empty and the reward function can store data for later use.Source: §3.1 · p. 3
  • Reward Designer writes design thoughts as code comments before implementation, helping initialize and update rewards and giving Reward Critic context to assess code rationality.This mechanism is similar to Chain of Thought (CoT) [48], and generated comments can explain checks such as whether lava appeared in the previous step.Source: §3.1 · p. 3
  • Scale constraints separate sparse goal or death signals from dense intermediate signals, with sparse values limited to {1, 0, −1} and dense values to {0.1, 0, −0.1}.The final reward is R = sgn(sparse)∗1+sgn(dense)∗0.1, keeping values within a preset range so the LLM focuses on reward scenarios rather than tuning magnitudes.Source: §3.1 · pp. 3–4

3.2. Reward Critic

  • Reward Critic checks syntax and whether code matches its stated design, task requirements, and game information; failed reviews return critique for revision, for up to 3 rounds.Runtime tracebacks can also be sent to Reward Designer for modification when errors arise during environment interaction.Source: §3.2 · p. 4

3.3. Trajectory Analyzer

  • Trajectory Analyzer summarizes the last L frames of failed trajectories from K interactions and suggests failure causes and key considerations for the next reward revision.It separately analyzes trajectory data and guides Reward Designer, which need not process complex trajectories or know the analyzer’s details.Source: §3.3 · pp. 4–5
  • In diamond search, trajectory-driven revisions addressed reward-gaming movement, lava deaths, and excessive lava avoidance that disrupted exploration.The revisions encouraged unvisited locations, increased penalties for lava, then constrained camera movement so the agent could turn left or right while continuing to search.Source: Figure 2 · p. 4, §3.3 · p. 5

4. Experiment

Experiments in MineDojo evaluate Auto MC-Reward on exploration and diamond-collection tasks, with results showing improved task success and ablations highlighting the value of iterative reward refinement. The study also reports the task setups and training details used for these comparisons.

4.1. Environment Setup

  • The experiments use MineDojo [14] harvest mode and include diamond exploration, approaching trees or animals in plains, and obtaining diamonds through a multi-stage tech tree.Diamond exploration starts underground with an iron pickaxe and a 60,000-step limit; the tree and animal tasks have 2,000-step limits, and the diamond-ore success criterion is approaching visible ore within 2 distance units.Source: §4.1 · p. 5

4.2. Implementation Details

  • The LLM components use GPT-4 [36] at temperature 0.3, with prompts supplying task and game information, trajectory statistics, and reward-function inputs and requirements.LLMs are used once per agent training rather than per action, so their computation overhead is described as negligible.Source: §4.2 · pp. 5–6

4.3. Main Results

  • Auto MC-Reward achieves 73.4% success with 972 average steps for approaching trees, and 56.3% cow-approaching success versus 12.6% with sparse reward.For cows, the passage describes this as 43.7% higher than sparse reward; MineCLIP also performs strongly but requires image-text similarity calculations throughout training.Source: Table 1 · p. 7, §4.3 · pp. 7–8
  • Auto MC-Reward raises diamond-ore exploration success from the imitation model’s 38.9% to 45.2%, with 70% lava-escape success.Sparse-reward RL achieves 0.5% success and 1.5% death rate, but its average moving distance is only 16.8; MineCLIP, curiosity, and self-imitation do not significantly improve success.Source: Table 1 · p. 7, §4.3 · p. 7
  • Auto MC-Reward reaches 36.5% diamond-collection success in forest, 7.7% higher than imitation learning, and 28.1% in plains, where imitation learning completes no tasks.The method optimizes obtaining log, cobblestone, iron ore, and diamond; the authors report success without Lidar or Voxel information during inference.Source: Table 2 · p. 7, §4.3 · p. 8

4.4. Ablation studies

  • Adding the Reward Critic raises diamond-exploration success from 35.1% to 40.5%, while the Trajectory Analyzer raises it from 40.5% to 45.2%.The Critic reduces syntax and semantic errors, while the Analyzer feeds failure causes back to the Designer; its lava-risk feedback improves survival and moving distance.Source: Table 3 · p. 7, §4.4 · p. 8
  • The Reward Designer restores diamond-exploration ability under dense reward after sparse-reward RL fails to explore diamonds normally.The Critic example describes how an unchecked penalty using “magma” instead of “lava” can cause semantic errors and insufficient lava avoidance, reflected in a 2.1% success-rate difference (43.1% vs. 45.2%).Source: Table 3 · p. 7, §4.4 · p. 8

5. Conclusion

Auto MC-Reward uses LLMs to automate dense reward design for Minecraft, with experiments demonstrating improvement on complex tasks. The conclusion identifies limited trajectory length as a constraint on analyzing long-term failures.

5. Conclusion

  • Auto MC-Reward uses LLMs to generate dense rewards and demonstrated improved performance in complex Minecraft tasks.The framework is intended to address learning challenges caused by sparse rewards and complex environments.Source: §5 · p. 8
  • Reward Designer, Reward Critic, and Trajectory Analyzer respectively design, verify, and analyze the reward function.Source: §5 · p. 8
  • Trajectory analysis is limited to the last 32 frames, hindering analysis of long-term failures such as unexplored areas or circling around lava.Source: §5 · p. 8

A. Experiments on More Minecraft Tasks · A.1. Environment Setup

The experiments extend evaluation to approaching animals and attacking a cow in the plains biome. The setup specifies success conditions and step limits for approaching animals, and a step limit for the cow task.

A. Experiments on More Minecraft Tasks

  • The experiments test the reward-design mechanism on additional Minecraft tasks, including approaching animals and attacking a cow.The approaching task includes pigs and sheep; the passage also names cow as the attacking task.Source: §A.1 · p. 11

A.1. Environment Setup

  • Approaching an animal in the plains biome succeeds when it is visible and less than 2 distance units away.The task concerns moving animals, including small animals such as chickens.Source: §A.1 · p. 11
  • The approaching task is challenging because animals move constantly and can be small, such as chickens.Source: §A.1 · p. 11
  • The approaching-animals task is limited to 2,000 steps.Source: §A.1 · p. 11
  • The cow-attacking task takes place in the plains biome.The supplied passage does not complete its description of the task’s success condition.Source: §A.1 · p. 11
  • The cow-attacking task is limited to 4,000 steps.Source: §A.1 · p. 11

A.2. Results

Auto MC-Reward outperformed imitation learning on animal-approaching and cow-attacking tasks, while video review suggested many successful cases involved luck rather than intentional approaches.

A.2. Results

  • On animal-approaching, absent from imitation learning’s training data, Auto MC-Reward achieved significantly higher success in fewer steps than its zero-shot baseline, and also clearly outperformed it on cow-attacking.The paper reports that video review found most successful cases appeared due to good luck, not intentional target approach.Source: §A.2 · p. 11, Table I · p. 11
  • Video review suggested most successful cases resulted from good luck rather than an intentional effort to approach the target.The authors also report supplementary demo videos in which the trained agent appears aware of completing the corresponding tasks.Source: §A.2 · p. 11

B. Experiments of scalability on MiniGrid Environment

Preliminary MiniGrid LavaCrossing experiments indicate that Auto MC-Reward outperforms the original sparse reward, suggesting effectiveness beyond Minecraft.

B. Experiments of scalability on MiniGrid Environment

  • Auto MC-Reward outperforms the original sparse reward on MiniGrid LavaCrossing, indicating effectiveness in a new environment.The performance comparison is shown in Figure I.Source: B. Experiments of scalability on… · p. 11, Figure I · p. 11
  • The experiments use LavaCrossingS9N1 and LavaCrossingS9N3, both with 9 × 9 maps and respectively one and three lava streams.Agents must reach the opposite-corner goal while avoiding deadly lava rivers.Source: B. Experiments of scalability on… · p. 11
  • Prompts for game information, task description, and input parameters are adapted to MiniGrid, while reward-function requirements and format remain unchanged.The cross-lava task is from the MiniGrid environment [8].Source: B. Experiments of scalability on… · p. 11

C. Experiments with Different LLMs · D. Foundation Model Imitation Learning

The LLM comparison reports that Mistral Medium outperformed GPT-4 in testing lava-avoidance reward design. The framework assigns code comprehension and generation to the Designer and Critic, and JSON-like data interpretation to the Analyzer.

C. Experiments with Different LLMs

D.1. Pretraining Data · D.2. Action Space

The appendix describes imitation-learning pretraining data and the action space used for four Minecraft tasks. GITM-generated diamond-mining trajectories supply recorded observations, and the imitation-learning setup uses 14 selected actions.

D.1. Pretraining Data

  • The paper uses imitation learning to pre-train foundation models, motivated by RL’s expensive sampling for hard-exploration tasks such as mining diamond.The approach follows VPT [3].Source: §D.1 · p. 12
  • GITM continuously performs Diamond Mining while the authors record frame-level RGB, actions, inventory, compass data such as y_level, and structured action plans.Source: §D.1 · p. 12
  • The collected records include action indices, biomes, deaths, nearby-object names and distances, inventory, and task names.Source: §D.1 · p. 13
  • The authors collect approximately 20.5 million forest-biome and 5.3 million plains-biome images, then use 9 million and 2 million images for training and validation in forest and plains biomes, respectively.The selected data total about 153 hours of game video at a 20 Hz control frequency.Source: §D.1 · p. 13

D.2. Action Space

  • The paper selects 14 actions for imitation learning across four tasks: obtaining a log, cobblestone, iron ore, or diamond.Source: §D.2 · p. 13

D.3. Training Details

The training pipeline builds supervised foundation models for four Diamond mining tasks, then initializes imitation-learning networks from a VPT foundation model. These models provide basic behaviors that the proposed methods further improve and extend to new tasks.

D.3. Training Details

  • Fully supervised foundation models were trained for log, cobblestone, iron ore, and diamond with Impala CNN [13] and Transformer [44], varying biome, temporal frames, and conditioning.Variants used forest or plains, 16 or 128 temporal frames, and optional goal and y_level embeddings; training used 32 A800 GPUs.Source: §D.3 · p. 13
  • Imitation-learning networks were initialized from the VPT foundation model to speed training, with hyperparameters listed in Table IV.Source: §D.3 · p. 14, Table IV
  • Foundation models, especially the goal-conditioned model, showed satisfactory basic behaviors; proposed methods further improved behaviors and enabled tasks such as approaching or killing animals.One cited behavior improvement is avoiding lava while exploring for diamond.Source: §D.3 · p. 14

E. Reinforcement Learning Training

The authors train with PPO and GAE, adding a pretrained-policy KL loss and trajectory-return-based reward normalization to support RL training across tasks.

E. Reinforcement Learning Training

  • PPO with GAE, implemented in TorchRL, is used for training; the main hyperparameters are listed in Table V [39, 38, 5].The supplied table passage identifies the hyperparameter table but does not provide its values.Source: E. Reinforcement Learning Training · p. 15, Table V · p. 15
  • An auxiliary KL-divergence loss against the frozen pretrained policy mitigates catastrophic forgetting and overly aggressive updates, preserving abilities such as digging horizontal tunnels [3].Its default loss weight is α = 0.125.Source: E. Reinforcement Learning Training · p. 15
  • Rewards are normalized based on trajectory returns to constrain return scales across tasks, using a momentum parameter β set to 0.7 by default.The passages define r as reward and R as trajectory returns.Source: E. Reinforcement Learning Training · p. 15

F. Full Text Prompts · G. Examples of LLM-Designed Reward Functions

Appendix F details the prompts, task descriptions, inputs, and requirements used to design and repair Minecraft rewards, verify reward functions, and analyze failed trajectories. Appendix G illustrates LLM-designed dense and sparse rewards across four Minecraft tasks.

F. Full Text Prompts

  • Appendix F provides all LLM input prompts for the method.Reward Designer prompts cover function initialization, updating, and handling critic failure and execution error.Source: F. Full Text Prompts · p. 16, §F.2 · p. 17

F.1. Task Description

  • The task-description prompt states that each task specifies its objective, initial status, success criteria, and procedure without supplying prior game-strategy knowledge.The diamond task places the agent underground at y level 11 with an iron pickaxe, and has it explore horizontally, face a diamond, and approach within 2 meters.Source: §F.1 · p. 16
  • The wood task is to find and approach wood within 1 meter in the plains, where wood is sparse and may require a long search.The agent starts on the ground and is instructed to find, face, and approach the wood.Source: §F.1 · p. 16
  • The plains-biome cow tasks ask the agent to approach a cow within 2 meters or to find, approach, and kill one.Both begin with the agent on the ground; the kill task succeeds when the cow is killed.Source: §F.1 · p. 16

F.2. Reward Designer

  • After testing, Reward Designer receives failure and inefficiency analysis with suggestions and can modify or replace the current reward function.The update prompt also asks it to consider balancing reward and penalty values.Source: §F.2 · pp. 17–19
  • Reward Designer inputs include current and previous nearby-block distances and angles, inventory changes, health, position history, and persistent GLOBAL_DATA.The prompt defines these as structured observations, including inventory changes as signed counts and GLOBAL_DATA as a dictionary for information retained between steps.Source: §F.2 · p. 17
  • The initial reward prompt requires dense and sparse functions, combining their signs as np.sign(sparse(...)) * 1 + np.sign(dense(...)) * 0.1.Sparse reward signals goal achievement or heavy punishment, while dense reward provides intermediate guidance.Source: §F.2 · pp. 17–18
  • When verification fails, the prompt gives the reasoning and critique and asks Reward Designer to revise the function, allowing a completely different function if necessary.It also requests concise thoughts on the revision that do not directly copy the critique.Source: §F.2 · p. 20
  • The Reward Designer receives nearby-block observations, inventory changes, health, position history, and persistent GLOBAL_DATA as inputs to its Python reward function.The prompt describes these as structured inputs, including relative block distances and angles, signed inventory changes, and a dictionary for retaining information between steps.Source: §F.2 · pp. 20–21
  • For execution errors, the Reward Designer is asked to modify the supplied reward function, briefly explain its error analysis and revision thoughts in comments, and follow the required code format.The prompt supplies the task description, current function, and error traceback; it requires no text outside the code block.Source: §F.2 · pp. 21–22
  • The Reward Designer prompt requires separate dense and sparse functions, combining their signs as 1× the sparse sign plus 0.1× the dense sign.Sparse reward marks goal achievement or heavy punishment; dense reward supplies intermediate guidance, and the return magnitudes do not matter.Source: §F.2 · pp. 21–22

F.3. Reward Critic

  • The Reward Critic checks compliance with reward requirements, consistency between comments and implementation, code format, and the supplied game-state information.It is explicitly told not to judge whether the reward function meets the task description.Source: §F.3 · pp. 23–24
  • The Critic must return reasoning and a success decision, with critique when the function fails, as parseable JSON and no extra explanation.The requested JSON fields are reasoning, success, and critique.Source: §F.3 · p. 24

F.4. Trajectory Analyzer

  • The Trajectory Analyzer uses failed-test actions, rewards, and observations to identify possible failure or inefficiency causes and suggest game strategies.Its prompt frames the analysis around an RL agent trained and tested in Minecraft for a specified task.Source: §F.4 · p. 25
  • The Analyzer’s game context specifies available movement, turning, jumping, and attack actions, a 2-meter attack range, and typically 60 stationary attacks to break a block.Its context also gives a 64-block visibility limit and notes that block detection is constrained by the field of view.Source: §F.4 · p. 25
  • The Trajectory Analyzer receives failed trajectories with histories truncated to the last 32 steps, while aggregate statistics include successful and failed trajectories and the success rate.Trajectory records include rewards, actions, locations, inventory changes, health, nearby blocks, the block underfoot, and whether the agent died.Source: §F.4 · pp. 25–26
  • The analyzer is asked to concisely explain failure and inefficiency causes and suggest strategies, including possible failure cases beyond the task description.Its analysis should draw on test results.Source: §F.4 · p. 26
  • An alternative supplies GPT-4V video captions to the trajectory-analysis prompt, enabling the LLM to assess possible failure causes from the described sequence.The caption prompt requests a description of the full sequence rather than each image separately.Source: §F.4 · p. 26

G. Examples of LLM-Designed Reward Functions

  • The attacking-cow function accepts damage-dealt and kill-entity inputs, rewards approaching and damaging a cow densely, and gives a sparse reward for killing one.The extra inputs record damage caused and counts of entities killed.Source: G. Examples of LLM-Designed Reward… · pp. 27, 30
  • The diamond example rewards progress toward diamond and penalizes hazards, poor pitch, and health loss, while sparse rewards mark reaching diamond or dying.The function’s stated dense-reward basis is distance to diamond and health, with a threat-detection mechanism to avoid unnecessary deaths.Source: G. Examples of LLM-Designed Reward… · p. 27
  • Across the four examples, the returned reward combines the signed sparse reward at weight 1 with the signed dense reward at weight 0.1.Source: G. Examples of LLM-Designed Reward… · pp. 27–30
  • For approaching wood in the plains, the example rewards reducing distance and maintaining health, discourages circling or standing still, and gives a sparse bonus upon reaching the wood.The objective is to approach the nearest wood block.Source: G. Examples of LLM-Designed Reward… · p. 28
  • The cow-approach example prioritizes getting closer to a cow and encouraging movement, penalizes health loss, omits inventory change, and rewards successful approach.The stated sparse bonus condition is having a cow within distance 2.Source: G. Examples of LLM-Designed Reward… · p. 29

H. Example of Full Interactions with LLM

The examples trace Auto MC-Reward’s interaction sequence for designing, critiquing, and refining a diamond-seeking reward function. They show code checks and trajectory analysis informing revisions that combine progress and safety signals with sparse goal and death rewards.

H. Example of Full Interactions with LLM

  • The example task starts at y=11 with an iron pickaxe and seeks to approach a diamond within 2 meters while avoiding death.The agent is to explore horizontally, face the diamond, and approach it.Source: H. Example of Full Interactions with LLM · p. 31
  • The reward gives +1 when diamond distance decreases, −1 when it increases, and a negative penalty for health loss; it gives a large positive reward below 1 meter from diamond and a large negative reward at death.Source: H. Example of Full Interactions with LLM · pp. 32–33
  • The critic is instructed to check reward requirements, consistency between comments and implementation, and function format, then provide reasoning and critique if the reward fails.It is told to consider the game-state information but not to judge whether the function satisfies the task description.Source: H. Example of Full Interactions with LLM · pp. 34–35
  • The example’s sparse reward gives a positive reward for reaching within one meter of a diamond and a negative reward on death.The design instructions distinguish sparse goal or heavy-punishment signals from dense intermediate guidance and combine their signs with weights 1 and 0.1, respectively.Source: H. Example of Full Interactions with LLM · pp. 35–36
  • The critic rejected the initial reward because GLOBAL_DATA["health"] was accessed before initialization, and recommended initializing the key.The designer was then asked to revise the reward using that feedback.Source: H. Example of Full Interactions with LLM · p. 36
  • The revised dense reward checks whether "health" is in GLOBAL_DATA before comparing health, then stores the current health value.It also rewards getting closer to the diamond and penalizes moving farther away.Source: H. Example of Full Interactions with LLM · p. 37

Critic output

  • The critic accepted a reward that rewarded getting closer to diamond and penalized health loss, with sparse rewards for reaching diamond or dying.Its response said the function met the requirements and had no critique.Source: H. Example of Full Interactions with LLM · pp. 40, 46

Analyzer input

  • The analyzer was asked to diagnose failed diamond-search trials, where success meant approaching within 2 meters while exploring horizontally underground with an iron pickaxe.The supplied trajectories contained only failed cases, while the accompanying statistics covered both successful and failed trajectories.Source: H. Example of Full Interactions with LLM · pp. 40–41

Test results

Analyzer output and designer input

  • The analyzer said implementing its suggestions and improving training could increase diamond-finding success, and the designer was then asked to modify the reward function.The revised request retained distance progress and health loss as dense signals, with diamond approach and death as sparse outcomes; the final reward combines their signs.Source: H. Example of Full Interactions with LLM · pp. 45–47

Analysis of Possible Reasons of Failure and Inefficiency

  • A failed trajectory showed the agent dying to lava without avoiding it, prompting a recommendation to detect hazards and prioritize safety.The suggested hazards included lava and deep holes.Source: H. Example of Full Interactions with LLM · p. 47

Suggestion on Game Strategy

Critic Output

  • The critic approved a revised reward function combining diamond progress, lava and health penalties, and sparse rewards for reaching diamond or dying.The sparse reward triggers when diamond is within 1 meter and penalizes death; the final function combines the signs of sparse and dense rewards.Source: H. Example of Full Interactions with LLM · p. 51

I. Example of GPT-4V as a Captioner

This example shows GPT-4V turning failed-mining frames into a sequence description that the Trajectory Analyzer uses to infer a likely lava-related failure and suggest safer diamond-mining strategies.

I. Example of GPT-4V as a Captioner

  • GPT-4V narrates the mining sequence: the player breaks stone, reveals lava, retreats, and cautiously assesses the hazard.The captioner describes the player as remaining near the lava, possibly trying to contain it with cobblestone.Source: §I · p. 52
  • The Analyzer identifies the sudden lava flow and the player’s difficulty escaping or containing it as the likely cause of death.It says the player’s proximity to lava and ineffective management may have led to sustained damage.Source: §I · pp. 53–54
  • The suggested response is to carry a water bucket, keep blocks ready to block lava, and mine from a safer distance.The Analyzer recommends water to neutralize lava and cobblestone or dirt barriers to block its flow.Source: §I · p. 54
  • The Analyzer also recommends checking for lava cues, listening for bubbling, and planning an escape route before mining.It suggests practicing lava-management techniques, including diverting flows or using water to solidify lava.Source: §I · p. 54
  • The paper presents these strategies as ways to reduce lava-related deaths and improve success in finding valuable resources.This is the Analyzer’s stated expected benefit of following its recommendations.Source: §I · p. 54
Loading 2312.09238v2…