Source-linked AI summary
MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft
Tianjie Ju, Yueqing Sun, Zheng Wu, Wei Zhang, Yaqi Huo, Xi Su, Qi Gu, Xunliang Cai, Gongshen Liu, Zhuosheng Zhang
TL;DR
Whether MLLM agents can sustain exploration in dynamic open worlds remains unclear, because existing benchmarks often use short-horizon tasks or Minecraft-specific mechanics. MineExplorer filters such confounds and evaluates implicit multi-hop exploration, finding that strong agents handle many single-hop tasks but degrade sharply when coordinating hidden prerequisites over longer trajectories.
Problem
Whether MLLM agents can sustain multi-step exploration in open worlds remains unclear, while Minecraft-specific mechanics can confound evaluation of general reasoning.
Method
MineExplorer filters Minecraft-dependent atomic tasks, evaluates perception, reasoning, and action, and composes retained tasks into implicit multi-hop prerequisite structures.
Results
Advanced MLLM agents often handle single-hop tasks but degrade sharply when coordinating hidden prerequisites across longer trajectories, showing that open-world exploration remains challenging.
Takeaways & Limitations
MineExplorer provides a testbed for studying MLLM agents beyond short-horizon execution toward sustained open-world exploration.
Takeaways & Limitations
The benchmark is bounded by Minecraft and cannot fully cover the diversity of physical situations encountered in broader embodied worlds.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have shown strong capabilities in perception, reasoning, and action generation. However, their ability to sustain exploration in dynamic open worlds remains unclear. Existing embodied and game-based benchmarks often compress interaction into short-horizon tasks or entangle success with domain-specific game mechanics. In this paper, we introduce MineExplorer benchmark for evaluating open-world exploration capabilities of MLLM agents in Minecraft. We first filter atomic tasks whose solutions rely heavily on Minecraft-specific knowledge to better reflect general open-world reasoning. Then we organize the benchmark around a ReAct-style capability formulation and compose atomic tasks into implicit multi-hop tasks. To further construct reliable instances, MineExplorer uses a multi-agent synthesis workflow that jointly designs task graphs, sandbox scenes, and rule-based milestone evaluators. Human evaluation shows that the multi-agent synthesis workflow produces significantly more reliable instances than a single-agent baseline. Experiments with advanced MLLM agents show that open-world exploration remains challenging, as strong models can handle many single-hop tasks but degrade sharply when hidden prerequisites must be coordinated over longer trajectories. Further analysis finds that task difficulty tracks agent completion, and larger models or thinking modes do not consistently translate into better performance. Code and dataset are available at https://github.com/Jometeorie/MineExplorer.
1 Introduction
MineExplorer benchmarks general open-world exploration by filtering Minecraft-specific knowledge, evaluating ReAct-style perception, reasoning, and action, and composing atomic tasks into multi-hop instances. Its construction is reliable, yet advanced MLLM agents still struggle with long-horizon prerequisite coordination.
- Motivation: Existing embodied benchmarks use constrained scenes or short horizons, limiting controlled evaluation of sustained exploration across changing states.Minecraft offers a scalable environment, but can introduce confounding game-specific knowledge.
- Benchmark design: MineExplorer removes atomic tasks dependent mainly on Minecraft-specific priors and evaluates perception, reasoning, and action through a ReAct-based capability formulation.Retained tasks are composed into implicit multi-hop tasks for open-world exploration evaluation.
- Benchmark construction: 1,497 knowledge-controlled atomic tasks were retained from 3,382 Minecraft tasks, alongside 813 human-validated composite instances spanning one-hop to four-hop settings.The benchmark uses dynamic environments and rule-based milestone checks to evaluate progress.
- Benchmark construction: Around 30% higher overall validity was achieved by the multi-agent workflow than by the single-agent baseline under the same generation setting.The workflow includes task selection, scene design, milestone evaluation, Minecraft expertise, validation, and iterative adjustment after rendering.
- Findings: Strong models often handle single-hop tasks but drop sharply when coordinating hidden prerequisites across longer trajectories, while recognizing what is present is easier than deciding what comes next.Larger models and thinking modes do not automatically improve open-world exploration, which remains far from solved.
2 Benchmark Construction
MineExplorer constructs an open-world exploration benchmark by filtering Minecraft-specific tasks, organizing the remainder around perception, reasoning, and action, and composing them into implicit multi-hop instances. A multi-agent workflow jointly designs tasks, scenes, milestones, and validation to produce structurally reliable benchmark instances.
- Pipeline: The benchmark pipeline filters atomic tasks for general open-world reasoning, maps them to capabilities, and synthesizes composite instances with latent dependencies and milestone checkers.This three-stage construction progressively turns Minecraft into a testbed while reducing bias from game-specific conventions.
- Task filtering: Tasks are screened with a Minecraft-rule reference sheet and an LLM judge to distinguish general world knowledge from Minecraft-specific mechanics.The filtering targets tasks whose success depends primarily on general open-world knowledge rather than game conventions.
- Capability formulation: The capability taxonomy follows ReAct and represents exploration through perception, reasoning, and executable action, with each atomic task assigned a binary capability vector.The action space includes moving, jumping, collecting, placing, crafting, and attacking.
- Implicit multi-hop tasks: Composite tasks hide prerequisite nodes in a dependency graph, requiring agents to infer unstated subgoals from the environment and satisfy rule-based milestone checkers.Difficulty increases with more diverse capabilities, hidden prerequisites, or deeper causal dependencies.
- Multi-agent synthesis: A multi-agent group-chat workflow specializes task selection, scene construction, commonsense checking, milestone design, and structural validation under orchestrator control.The validator checks DAG validity and alignment between milestone rules and atomic tasks, while agents revise outputs during a debate phase.
3 Benchmark Overview
MineExplorer filters Minecraft-specific atomic tasks and constructs 1- to 4-hop benchmark instances through a validated multi-agent workflow. The benchmark spans perception, reasoning, and action while providing progressively harder exploration scenarios.
- Data Statistics: After filtering, MineExplorer retains 1,497 tasks from 3,382 atomic tasks to reduce reliance on Minecraft-specific priors.Claude-Opus-4.6 audits the atomic task pool and removes tasks whose completion depends on Minecraft-specific knowledge.
- Capability Coverage: MineExplorer covers perception, reasoning, and action, supporting evaluation of their integration during open-world exploration.The capability distribution provides broad coverage across the three dimensions.
- Data Statistics: The benchmark generates instances from 1-hop to 4-hop settings, producing 1,039 instances through the multi-agent workflow.A task selector chooses k compatible tasks from 10 randomly sampled candidates, where k equals the target hop number.
- Data Statistics: Human evaluation shows that the multi-agent pipeline consistently outperforms the single-agent baseline in valid rates and overall quality across hidden dependency depths.Human-rejected instances are removed before further evaluation.
- Task Difficulty Distribution: Difficulty gradually shifts upward from 1-hop to 4-hop tasks while retaining within-group variation, yielding a graded exploration spectrum.This distribution supports evaluation across different levels of open-world exploration difficulty.
4 Experiments
MineExplorer evaluates state-of-the-art multimodal agents on open-world exploration using task, milestone, and capability-level metrics. Results show that agents handle many single-hop tasks but struggle with hidden prerequisites, long-horizon coordination, and maintaining plans as the world changes.
- Models: The evaluation covers state-of-the-art MLLMs from Anthropic, OpenAI, Google, Qwen, and other model families, excluding smaller-scale and text-only models.The benchmark requires multimodal reasoning and challenging open-world interaction.
- Main results: Agents perform substantially better on single-hop tasks than on multi-hop tasks requiring hidden prerequisites to be inferred and coordinated across longer trajectories.Claude-Opus-4.6 and Gemini-3.1-Pro-Preview are strongest overall, yet even the best models fail on most multi-hop tasks.
- Capability analysis: Perception scores generally exceed reasoning scores, while action scores usually fall between them, with the gap widening when agents must prioritize subgoals and adapt to changing environments.Agents can recognize visible objects but struggle to convert observations into coherent strategies.
- Capability analysis: Larger models and thinking-mode variants do not consistently improve behavior, indicating that scale and explicit reasoning help only when coupled with visual grounding and state synchronization.Larger Qwen variants do not consistently outperform smaller ones, and thinking variants do not reliably beat instruction-tuned counterparts.
- Difficulty analysis: TSR decreases consistently with task difficulty, and most models lose around 50% TSR from low-difficulty to high-difficulty instances.Difficulty accumulates capability requirements along latent dependency graphs, so harder tasks require more coordinated capabilities.
- Efficiency analysis: Most solvable tasks finish early, whereas unsolved tasks remain incomplete through the maximum horizon, showing that current agents are mainly effective on short-horizon exploration.Stronger models often use more steps on completed tasks because they solve additional medium-horizon tasks.
5 Related Work
Prior work evaluates MLLM agents in embodied environments, game sandboxes, and Minecraft, with increasing attention to long-horizon reasoning, open-ended tasks, and autonomous exploration. These efforts include unified suites, automatic evaluation, memory augmentation, and open-world skill benchmarks.
- Interactive embodied environments: Embodied benchmarks study MLLM planning and interaction in complex 3D environments, including MuEP, EmbodiedBench, and MM-Escape.EmbodiedBench spans four environments and reports modest average performance even for the strongest model.
- Game-sandbox benchmarks: Game-sandbox benchmarks provide controlled evaluation of game playing and long-horizon agentic reasoning across diverse environments and difficulty levels.Examples include lmgame-Bench, which unifies various games, and VideoGameBench, which evaluates MLLMs in 10 real environments.
- Minecraft benchmarks: Minecraft benchmarks exploit the platform’s controllability and open-ended combinatorial world to evaluate instruction following, automatic assessment, and multimodal spatial planning.Representative studies specify tasks through gameplay videos, introduce automatic evaluation for open-ended agents, and assess multimodal spatial planning.
- Advanced Minecraft agents: Recent Minecraft work evaluates increasingly capable agents through memory-augmented task suites, visual-temporal context, open-world skill libraries, and autonomous-exploration benchmarks.One memory-augmented agent completes over 200 Minecraft tasks, while other work improves spatially grounded interaction or benchmarks autonomous exploration.
6 Conclusion
MINEEXPLORER evaluates whether MLLM agents can sustain exploration in open-world environments beyond short-horizon execution. It filters Minecraft-prior-dependent tasks, composes implicit multi-hop tasks, and uses validated multi-agent synthesis to improve benchmark reliability.
- MINEEXPLORER evaluates whether MLLM agents can sustain exploration in open-world environments.
- The benchmark filters tasks relying on Minecraft priors and composes implicit multi-hop tasks requiring perception, reasoning, and action across hidden prerequisites.
- Its multi-agent synthesis workflow produces task graphs, sandbox scenes, and rule-based milestones, with human validation showing clear advantages over single-agent construction.
- Evaluation of advanced MLLM agents shows that open-world exploration remains challenging, motivating a testbed beyond short-horizon execution.
Limitations
MineExplorer reduces Minecraft-specific confounding through a controlled testbed, but its evaluation remains bounded by Minecraft’s limited coverage of broader embodied situations. Future work should extend this direction when controllable embodied sandboxes become available.
- Environment scope: MineExplorer remains limited by the Minecraft environment, which cannot fully represent the diversity of physical situations in broader embodied worlds.The benchmark reduces Minecraft-specific knowledge confounds and provides a controlled testbed, but Minecraft’s sandbox is not fully representative of general embodied settings.
- Future work: Future work should extend open-world exploration evaluation once controllable embodied sandboxes become available.
Ethical Considerations … A.3 Action
The paper reports no foreseeable ethical risks because MineExplorer uses simulated sandbox tasks without private data, real-world deployment, or human-affecting actions. Its capability taxonomy organizes benchmark requirements across perception, reasoning, and action, with capabilities marked essential only when their removal makes completion impossible.
- Ethical Considerations: The benchmark study uses knowledge-controlled open-world exploration tasks in Minecraft and evaluates agent progress through rule-based milestones.The human evaluation assesses instance validity and quality.
- Ethical Considerations: Because tasks run in simulated sandboxes without private data, sensitive attributes, real-world deployment, or human-affecting actions, the authors identify no foreseeable ethical risk.Human evaluation is limited to assessing validity and quality.
- A Capability Taxonomy: The taxonomy marks a capability as required only when removing it would make successful benchmark completion impossible.It is designed to describe the minimum capabilities needed for each instance.
- A.1 Perception: Perception capabilities cover spatial, temporal, entity, state, and inventory information needed to interpret environments, track changes, identify entities, monitor status, and inspect carried items.These include navigation and relative-position judgments, before-and-after differences, interactive entities, health or durability, and item counts or tools.
- A.2 Reasoning: Reasoning capabilities include commonsense, causal, and relational inference about physical relations, action outcomes, and relations among objects, entities, and locations.Relational examples include near, inside, above, connected to, blocked by, and different from.
- A.3 Action: Action capabilities include moving, jumping, collecting, and placing, covering locomotion, vertical movement, obtaining objects, and construction.Collection includes mining, breaking, harvesting, gathering drops, and picking up items.
- A.3 Action: The action taxonomy also includes crafting and attacking for item transformation and combat-oriented execution.Crafting produces new items from existing materials, while attacking involves combat or hunting entities.
B Details of Human Evaluation
Human evaluation uses a web interface to jointly assess benchmark-scene quality, task structure, milestone validity, and agent execution. Annotators follow standardized procedures, compare manual and automatic judgments, and verify uncertain cases.
- Evaluation procedure: Annotators review shuffled instances by reading task descriptions, watching Claude-Opus-4.6 execution videos, and inspecting dependency graphs.The interface supports joint assessment of scene reasonableness and agent execution reliability.
- Scene and task assessment: Scene quality is rated on a five-point scale based on task relevance and whether the environment supports fair evaluation.Explicit criteria for each rating level are provided to improve annotation consistency.
- Scene and task assessment: Annotators check task completeness, hidden prerequisites, and dependency-graph ordering, then judge milestone completion and milestone-rule quality.Manual milestone judgments are compared with automatic detections to assess semantic correctness and implementation reliability.
- Agent execution assessment: Overall execution quality is rated to examine consistency between rule-based metrics and human-verified task completion.Annotators followed tutorials, and uncertain cases underwent second-round verification before labels were finalized.
C Reliability of Minecraft-Specific Knowledge Evaluation
Human validation indicates that Claude-Opus-4.6 reliably distinguishes Minecraft-specific knowledge from general open-world knowledge when filtering atomic tasks.
- Reliability evaluation: 86.8% overall agreement was achieved between human annotations and Claude-Opus-4.6 judgments on Minecraft-specific knowledge dependence.Both false-positive and false-negative cases accounted for less than 10%.
D Reliability of Milestone Check · E Fine-grained Results on Multi-hop Tasks
The benchmark’s automated milestone evaluators agree with Claude-Opus-4.6 human annotations at 86.8%, and inconsistent instances are removed. Multi-hop performance is further broken down by task depth into 2-hop, 3-hop, and 4-hop subsets using Table 3’s metrics.
- D Reliability of Milestone Check: The reliability analysis compares Claude-Opus-4.6 human annotations with automated milestone detection.The comparison is reported in Figure 9.
- D Reliability of Milestone Check: The milestone detector is rule-based rather than human-operated.Its outputs are evaluated against Claude-Opus-4.6 human annotations.
- D Reliability of Milestone Check: 86.8% overall agreement is observed between Claude-Opus-4.6 human annotations and the rule-based milestone detector.This agreement supports the reliability of the automated milestone-checking process.
- D Reliability of Milestone Check: The multi-agent workflow produces reliable milestone evaluators according to the reported agreement.The reported agreement is 86.8%.
- D Reliability of Milestone Check: Instances with inconsistent annotations are removed from the final benchmark.This filtering follows the milestone-evaluator reliability check.
- E Fine-grained Results on Multi-hop Tasks: Table 6 breaks down model performance on the multi-hop subset.The section reports fine-grained results separately by task depth.
- E Fine-grained Results on Multi-hop Tasks: The multi-hop breakdown distinguishes 2-hop, 3-hop, and 4-hop tasks.These categories organize the reported performance results by hop count.
- E Fine-grained Results on Multi-hop Tasks: The 2-hop, 3-hop, and 4-hop results use the same metrics as Table 3.The passage specifies metric consistency but does not provide the corresponding values.
F Failure Mode Analysis of MINEEXPLORER
Navigation failure is the dominant error for Claude-Opus-4.6, while resource gathering, action execution, and goal misidentification also contribute substantially. These failures span perception, action, and reasoning, revealing broad weaknesses in current MLLM agents’ open-world exploration.
- Failure modes: Resource gathering failure is a non-negligible source of unsolved milestones.
- Failure modes: Action execution failure and goal misidentification each account for about 10% of errors.
- Failure modes: The three major failure types correspond to perception, action, and reasoning, respectively.The mapping is navigation to perception, action execution to action, and goal misidentification to reasoning.
G Stability Analysis … I.4 Multi-Agent Benchmark Construction
MineExplorer is evaluated as reproducible across repeated runs, while example trajectories show that exploration can succeed through sustained localization or fail after ineffective collection and lost direction. Its benchmark-construction workflow filters Minecraft-specific tasks, annotates minimal capabilities, and uses coordinated agents to design dependency graphs, scenes, and checkable milestones.
- G Stability Analysis: Repeated evaluations of Claude-Opus-4.6 and LLaMA-3.2-90B-Vision-Instruct show acceptable TSR variance across difficulty levels over three independent runs.The reported standard deviations suggest stochasticity does not substantially change observed performance trends.
- H Example trajectories of MINEEXPLORER: Claude-Opus-4.6 succeeds when it identifies blue concrete powder, explores the surrounding area, and mines nearby brown concrete powder at around 22 seconds.The successful trajectory demonstrates sustained exploration and completion of a sequential spatial task.
- H Example trajectories of MINEEXPLORER: A coal-to-helmet task fails after the agent does not collect coal, leaves the area, and loses direction during subsequent exploration.The trajectory illustrates how an early collection failure and later navigation breakdown can jointly prevent task completion.
- I.1 Minecraft-Specific Knowledge Elicitation: Task filtering asks whether a never-played Minecraft user can solve each task using only real-world common sense, retaining tasks whose logic has general-world analogies.Mining ore, fighting creatures, hunting animals, and building are treated as not requiring domain knowledge, whereas recipes and portal construction are game-specific.
- I.2 Capability Set Annotation: Capability annotation assigns only the minimum necessary abilities across perception, reasoning, and action dimensions.The dimensions include five perception capabilities, three reasoning capabilities, and six action capabilities, with examples covering spatial understanding, causal inference, movement, collection, placement, crafting, and attack.
- I.4.1 Task Selector Agent: The Task Selector Agent composes coherent multi-hop sequences into an acyclic dependency graph whose edges explain why prerequisite outputs are needed.It selects atomic tasks, incorporates critiques, and sends scene requirements onward; the graph may contain parallel branches that later converge.
- I.4.2 Scene Designer Agent: The Scene Designer Agent builds a compact, sequentially traversable Minecraft scene with relative-coordinate commands and previews it through sandbox exploration before finalizing.The workflow requires submitting all agreed commands, examining returned frames, reporting issues, and documenting verified screenshot observations rather than fabricating them.
- I.4.3 Milestone Agent: The Milestone Designer Agent creates at least one rule-based, programmatically checkable criterion per task, preferring object state and semantically appropriate evaluators over unnecessary position constraints.The evaluator uses inventory, event counters, and spatial or entity boxes; position-near-with-facing is preferred for find, locate, and observe tasks.