VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraft

1 Introduction

• Minecraft LLM agents often depend on environmental APIs for perception and action, which may hinder generalization and constrain autonomy.The authors identify visual-only reasoning and low-level mouse-and-keyboard actions as a more human-like alternative.Source: §1 · p. 1
• Visual policies and large-scale domain-specific finetuning can limit LLM decision-making or incur prohibitive development costs.Global visual input can add irrelevant noise and destabilize mouse-and-keyboard predictions.Source: §1 · pp. 1–2
• VistaWise finetunes only its object detector, using fewer than 500 annotated gameplay frames to incorporate domain knowledge into visual perception.The detector identifies visual entities while minimizing irrelevant information and noise.Source: §1 · p. 2
• A cross-modal knowledge graph combines detected visual information with textual dependencies, while retrieval-based pooling extracts task-related graph information.The agent also uses a desktop skill library based on human mouse-and-keyboard behavior to control Minecraft directly.Source: §1 · p. 2
• VistaWise achieves a 33% success rate in obtaining diamonds, surpassing the previous state-of-the-art rate of 25%.The paper reports that it outperforms other non-API-based baselines in complex tasks.Source: §1 · p. 2

2.1 Agents in Minecraft

• Visual-policy Minecraft agents face challenges in systematic exploration, interpretability, and generalization, while API-based LLM agents may not generalize to environments without such APIs.Integrated LLM-and-vision approaches seek visual-only action but can require substantial data collection and training.Source: §2.1 · pp. 2–3

2.2 LLMs with Knowledge Graph

• Knowledge graphs can supply structured, up-to-date domain knowledge to LLMs, but effectiveness depends on retrieving relevant, reliable information.Prior approaches convert retrieved graph knowledge into textual prompts or use LLMs to enhance reasoning over graphs.Source: §2.2 · p. 3

3.1 Overview

• The framework combines text- and vision-based graph construction and retrieval with a desktop skill library and memory stack around an LLM.At each timestep, the agent uses the prompt, retrieved information, memory, and skills to act in Minecraft.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Cross-modal Information Integration

• Online textual knowledge is structured as entity nodes and relationship edges, then linked to real-time visual attributes in a vision-text graph.The visual attributes include spatial and conditional context associated with corresponding semantic entities.Source: §3.2 · pp. 3, 5
• Environmental entities provide screen coordinates and bounding boxes, while inventory entities provide item-icon coordinates; bounding-box size supports coarse interaction-range estimates.The method uses empirical width and height thresholds to estimate whether the agent is within interaction range.Source: §3.2 · p. 4

3.3 Graph-based Information Retrieval

• Retrieval-based pooling retains Player-to-Target paths, then matches task-prompt entities and visually attributed entities before textualizing the pooled graph for the LLM.This targets redundant information while supplying task-relevant dependencies and real-time visual context.Source: Figure 3 · p. 5, §3.3 · p. 5

3.4 Desktop-level Skill Library

• The desktop skill library combines low-level mouse-and-keyboard operations into skills, letting the LLM generate input parameters from visual information without simulators or environment-specific APIs.The library supports direct control of the Minecraft desktop client.Source: §3.4 · pp. 5–6

3.5 Memory Stack

• A LIFO memory stack prioritizes recent decisions and allows history to be recalled from newest to earlier decisions with controllable recall steps.The agent pushes its latest decision at each timestep.Source: §3.5 · p. 6

3.6 Agent Workflow Modeling

• At each timestep, the LLM policy predicts an executable skill and its parameters from the synthetic prompt, pooled graph, decision history, and skill library.The workflow then executes the action in Minecraft before observing the next timestep.Source: §3.6 · p. 6

4.1 Dataset and Implementation

• The Minecraft detector dataset contains 471 annotated frames with 3,304 instances across 23 visual-entity types.The model is finetuned for 150 epochs using a 9:1 train-test split; experiments otherwise use GPT-4o and 15 repeated trials unless specified.Source: §4.1 · p. 6

4.2 Comparison against other methods

• VistaWise achieves state-of-the-art performance on “obtain diamond” and most sequential sub-goals despite substantially smaller training data and VRAM requirements.The comparison covers non-API-based baselines; VistaWise uses 471 video frames and 24 GB GPU VRAM, versus baselines using at least 160M frames with 192 GB VRAM or 990M text tokens with 64 GB VRAM.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 Ablation Study

• Entity names alone preserve entity-dependency understanding without crafting failures, while omitting other KG attributes reduces injected information.Success-rate variation between the two ablation groups is attributed to random in-game resource distribution.Source: §4.3 · p. 7, Table 2 · p. 7
• Path pooling followed by entity pooling is the optimal retrieval strategy, outperforming similarity-based retrieval on FPR and FNR.Entity pooling first disrupts the node-edge structure and may lose critical paths; the figure defines lower FPR and FNR as better retrieval.Source: §4.3 · p. 7, Figure 4 · p. 7
• Without accurate object-detected entities, MLLM visual capability alone is inadequate for complex tasks; extra MLLM visual input adds no significant performance benefit when detection is used.The authors state that object detection can replace MLLM visual capability and enhance spatial perception.Source: §4.3 · p. 7, Table 3 · p. 8
• The memory stack is crucial for complex tasks, CoT underpins decision-making, and the KG improves crafting stability.Table 4 reports success rates for ablations of these components.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 Inference Costs

• 30.6% and 41.0% fewer tokens are consumed for “obtain diamond” when full entity attributes and MLLM visual capabilities are omitted, respectively, without significant performance degradation.Figure 5 reports token consumption across goals and variants.Source: §4.4 · p. 8, Figure 5 · p. 8
• ∼$1.28 is VistaWise’s cost for 160 GPT-4o iterations, versus Voyager’s ∼$25 under the same conditions, a 94.9% cost reduction.The paper presents this comparison as evidence of VistaWise’s cost-effectiveness.Source: §4.4 · p. 8

• VistaWise combines retrieved textual dependencies and a dedicated object detector to reduce the need for extensive domain-specific training data.Source: §5 · p. 8
• The framework represents visual information and textual dependencies in a cross-modal KG and retrieves task-relevant information through pooling.Source: §5 · p. 8
• A desktop-level skill library controls the Minecraft client directly through mouse-and-keyboard inputs.Source: §5 · p. 8

• Each action iteration waits for LLM inference and, when applicable, a server response, affecting real-time performance.Source: Limitations · p. 8
• Pausing the virtual environment synchronizes decisions with its timescale but increases task completion time.Source: Limitations · p. 8

• Table 5 provides explanations for icons mentioned in the main paper.Source: Table 5 · p. 13, A The Explanation of Icons · p. 13

B Dataset

• 3,304 visual entities are annotated across 471 gameplay-video frames.The entity details and annotation counts are provided in Table 6.Source: B Dataset · p. 13
• Entity types include trunk, plank_icon, and log_icon, which have higher counts because they appear with different textures.The text gives oak and birch as examples of those textures.Source: B Dataset · p. 13
• Table 6 presents the defined visual entity types and their annotation counts across the 471 game-screen frames.Source: Table 6 · p. 13

C Game Environment

• The virtual environment is the Vanilla 1.11.2 Minecraft desktop client.Source: C Game Environment · p. 13
• Table 7 details the specific game-world settings used for the environment.Source: C Game Environment · p. 13

D Skill Library

• Table 8 gives examples of desktop-level skill-library functions and their explanations.Source: D Skill Library · p. 13
• The skill library is constructed semiautomatically from manually designed templates and requirements passed to an LLM for function generation.Human effort includes defining function names, inputs, and descriptions, then refining generated functions when needed.Source: D Skill Library · p. 13
• Using GPT-4o mini, the authors generated all skill functions, with fewer than half needing minor manual adjustments.Source: D Skill Library · p. 14

E Knowledge Graph

• An LLM with web access summarizes selected websites into entities and dependencies, then refines and merges them into a NetworkX-compatible knowledge graph.Sites are processed individually because of context-length limits, and their outputs are merged into a draft before refinement.Source: E Knowledge Graph · p. 14
• The graph restricts relations to nine allowed categories and receives a brief manual review to remove redundant or irrelevant nodes.Human involvement also includes selecting source websites and defining the allowed relation types.Source: E Knowledge Graph · p. 14

F Prompt Template

• Table 9 presents the general prompt template used for action prediction.The paper reports its action-prediction prompt template in Table 9.Source: F Prompt Template · p. 14, Table 9 · p. 16
• Table 8 provides examples from the desktop-level skill library.Source: Table 8 · p. 15