Mine Odyssey: Benchmarking Spatial Agentic Intelligence in the Wild

1 INTRODUCTION

• Existing benchmarks often focus on household or urban settings, leaving agents insufficiently tested across varied spatial structures and traversal requirements.The settings differ in scale, connectivity, access constraints, and elevation, while agents must maintain spatial information and revise plans during execution.Source: §1 · p. 2
• MINE ODYSSEY comprises 180 tasks across 30 Minecraft reconstructions in 20 countries and regions on five continents, including 20 outdoor and 10 indoor environments.First-person movement, collision, opening doors, and climbing stairs and ladders make the reconstructed places interactive.Source: §1 · p. 2, Figure 2 · p. 3
• Tasks use ordered sequences of recognizable waypoints, requiring agents to find feasible paths and reach each destination in order.The authors select named places such as landmarks, palace rooms, stadium facilities, and building entrances rather than arbitrary map points.Source: §1 · p. 3
• The agent framework supports programmable control, intermediate feedback, and interruption of ongoing commands for closed-loop interaction.Isolated execution and independent arrival checks enable automated, parallel evaluation.Source: §1 · p. 3
• 85.6% task success for GPT-6 Astra exceeds Claude Opus 5.5’s 73.9% and DeepSeek-V4.1-Flash’s 23.9%.The Astra–Opus gap is larger indoors than outdoors, and more than half of failed attempts do not reach the first destination.Source: §1 · p. 3

2.1 ENVIRONMENTS

• The 30 maps span 20 countries and regions, with 20 outdoor and 10 indoor environments featuring varied terrain, access constraints, and vertical circulation.Tasks may involve stairs, ladders, gates, or buttons; versioned world saves provide consistent starting map states.Source: §2.1 · p. 4
• The catalog contains 1,059 waypoints, each matched to a reachable standing position on the correct floor, while precise arrival coordinates remain evaluator-only.Each waypoint links a human-readable place name to a three-dimensional position; agents can see its map marker and a first-person marker within 32 blocks.Source: §2.1 · p. 4
• The 180 tasks specify a starting waypoint and orientation plus 2–12 ordered destinations, with routes manually traversed to verify reachability.Candidate instructions are selected and refined for semantic coherence, route plausibility, and diversity; tasks vary starting points, distances, elevation changes, and interactions.Source: §2.1 · p. 4

2.2 VERIFIER.

• An independent evaluator samples player position at 1 Hz and credits only the next required waypoint when its arrival condition is met.Out-of-order visits do not advance progress, and episodes can fail through death, leaving the game, or going out of bounds.Source: §2.2 · pp. 4–5

2.3 AGENT FRAMEWORK

• Agents control the game through API and keyboard–mouse tools, receiving screenshots, map views, and textual command feedback.The asynchronous observation–action loop lets agents continue, stop, or replace commands after intermediate observations.Source: §2.3 · pp. 5–6
• Safeguards block coordinate disclosure, waypoint teleportation, and Baritone navigation, while sandbox isolation excludes hidden task data and separates runs.The sandbox also disables external network access while retaining approved AgentBridge APIs.Source: §2.3 · p. 6

3.1 SETUP

• The evaluation covers eight proprietary and open-weight models, each limited to 500 agent steps and six hours per task.It reports SR, CC, path length, SPL, and average turns; SR measures ordered completion, while CC measures the fraction of ordered intermediate stops reached.Source: §3.1 · pp. 6–7

3.2 MAIN RESULTS

• 85.6% SR for GPT-6 Astra leads the 180-task benchmark, ahead of Claude Opus 5.5 at 73.9% and DeepSeek-V4.1-Flash at 23.9%.Astra’s lead over Opus 5.5 widens from 8.8 percentage points outdoors to 18.2 indoors; the indoor and outdoor subsets differ in layouts and route composition.Source: Table 1 · p. 7, §3.2 · p. 7
• Opus 5.5 reaches 86.1% CC and 50.35% SPL, 2.9 and 3.35 percentage points below Astra despite an 11.7-point SR gap.High intermediate-stop coverage need not translate into full completion; these are full-cohort averages, including failures.Source: §3.2 · p. 7

4.1 ABLATIONS

• HUD raises success rate by 6.1 points for DeepSeek and 4.0 for GLM; route guidance slightly raises CC but lowers DeepSeek’s success rate by 2.8 points.Multi-view raises DeepSeek’s success rate by 2.2 points while lowering SPL by 0.3 points; midnight lowers success rate by 1.7 points for both models.Source: Table 2 · p. 8, §4.1 · p. 8

4.2 PROLONGED LOCAL STAYS.

• Gemini spends 51.7% of turns in local stays, while Astra averages 0.17 episodes and 2.7% of turns in such stays.Stays are defined as at least ten consecutive turns within one block of a fixed 3D anchor; Gemini averages 7.09 episodes per task.Source: Figure 5 · p. 8, §4.2 · p. 8

4.3 CASE STUDIES

• At the White House, Astra detours while DeepSeek revisits rooms; at the Plaza Hotel, Astra returns downstairs to the target in 12.6 minutes, while Grok remains upstairs until the step limit at 135.7 minutes.The displayed White House detour takes 2.4 minutes.Source: Figure 6 · p. 9, §4.3 · p. 9
• At Hagia Sophia, Astra reveals and ascends a side-branch stair route, whereas Opus 5 leaves without exploring it.Both agents receive nearly matching views and see the same door ahead; the stair opening is visible on Astra’s right.Source: Figure 7 · p. 9, §4.3 · p. 10

5 CONCLUSION

• Mine Odyssey covers 180 spatial tasks across 30 Minecraft reconstructions, and its evaluations identify challenges in grounding spatial information, discovering traversable connections, and recovering from unsuccessful routes.GPT-6 Astra achieves 85.6% success, versus 73.9% for Claude Opus 5.5 and 23.9% for DeepSeek-V4.1-Flash.Source: §5 · p. 10

A.1 SPATIAL INTELLIGENCE EVALUATION

• Spatial intelligence evaluation has expanded from image relations to multi-view and video integration, perspective-taking, map understanding, and interactive spatial tasks; this work studies goal-directed navigation within that line.Source: §A.1 · p. 16

A.2 EMBODIED NAVIGATION EVALUATION

• Embodied-navigation benchmarks cover single and successive goals in indoor and outdoor settings, while Mine Odyssey broadens coverage to real-world reconstructions and agent-written navigation programs.Its persistent workspace lets agents compose control routines, process feedback, and retain task information across decisions.Source: §A.2 · p. 16

A.3 INTERACTIVE EVALUATION OF GENERALIST AGENTS

• Interactive evaluations include realistic web and desktop workflows, sustained execution, and extended game planning; Mine Odyssey adds programmable spatial interaction analyzed through logs, trajectories, and recorded waypoint visits.Source: §A.3 · p. 16

B MAP COLLECTION AND ENVIRONMENT DIVERSITY

• The maps range from districts, parks, hillsides, and waterfronts to palaces, religious buildings, hotels, stadiums, and ships, supporting district routing, entrance-finding, and movement between floors.Stairs, ladders, doors, and gates add local traversal and interaction contexts.Source: B MAP COLLECTION AND ENVIRONMENT… · p. 17

C.1 DEFAULT SETTINGS.

• Figure 8 shows all 30 environments; Table 4 summarizes main-comparison defaults, while controlled variants modify designated settings.The route-guidance condition displays a planned route while leaving movement under agent control.Source: Table 4 · p. 17, §C.1 · pp. 17–18, Figure 8 · p. 18

C.2 OBSERVATION AND INTERACTION.

• Agents receive a first-person view and may use the world map, but cannot access exact target coordinates or evaluator state; flight and block modification are restricted.The minimap, navigation hints, automated navigation, and waypoint teleportation are disabled by default; the world continues running during inference.Source: §C.2 · p. 19

C.3 SPL AND REFERENCE PATH CONSTRUCTION

• SPL combines task success with agent path length relative to a reference route through required stops; failed tasks score zero and each contribution is capped at one.Reference paths use A* on an approximate movement graph supporting walking, steps, ladders, bounded falls, and operable doors; several movement types are excluded.Source: §C.3 · p. 19

D.1 RESOURCE ACCOUNTING

• Resource-use means cover all 180 tasks, including failures, and turn counts exclude automatic history-summary responses.Figures 9 and 10 report estimated monetary cost and output-token usage.Source: §D.1 · p. 19

D.2 PERFORMANCE AND RESOURCE USE

• At $1.11 per task, DeepSeek-V4.1-Flash reaches 23.9% SR versus GLM-5.3-Flash’s 15.6% at $1.20; Gemini 3.8 Flash costs $5.13 for a 2.8-point SR gain.The estimated cost comparison is shown in Figure 9.Source: §D.2 · p. 19, Figure 9 · p. 20
• 85.6% SR with 10.3K output tokens per task makes GPT-6 Astra the highest-SR, lowest-output model; DeepSeek-V4.1-Flash uses 618.0K tokens for 23.9% SR.GPT-5.6 Sol reaches 43.9% SR with 68.5K output tokens; output volume alone does not explain the completion ranking.Source: §D.2 · pp. 19–20, Figure 10 · p. 20

D.3 REASONING EFFORT

• 35.0% to 50.6% SR accompanies Claude Opus 5’s low-to-max effort increase, while output use rises from 73.2K to 249.9K tokens per task, about 3.4×.Estimated cost rises from $10.47 to $19.21 per task for a 15.6-point SR gain; mean turns increase by 11.0%.Source: Figure 10 · p. 20, §D.3 · p. 20
• 28.3K additional output tokens yield a 4.4-point SR gain from low to medium effort, versus 105.1K tokens for a 6.1-point gain from high to max.The comparisons average over all tasks and do not hold successful-task membership fixed.Source: Figure 10 · p. 20, §D.3 · p. 20

E FAILURE STAGES

• 70.0% of tasks reach Gemini’s first stop, versus 67.8% for Sol, but Gemini fails afterward on 43.3% of all tasks compared with Sol’s 23.9%.Reaching a stop is based on the evaluator’s ordered-arrival record; failure afterward can include runs that visit every stop but lack an accepted completion claim.Source: E FAILURE STAGES · p. 21, Figure 11 · p. 21
• Reaching the first stop is not equivalent to success: full completion additionally requires an accepted claim, even after all stops are visited.The first stop is the first intermediate stop, or the destination for tasks without intermediate stops.Source: E FAILURE STAGES · p. 21
• 48.3–55.6% of all tasks end before the first stop for DeepSeek, Grok, and GLM.Figure 11 separates failure before the first stop, failure after reaching it, and successful completion.Source: Figure 11 · p. 21, E FAILURE STAGES · p. 21

F INTERACTION STATISTICS

• Across 400,571 execution requests, category percentages are averaged equally across tasks, and a request may count in multiple categories.Figure 12 shows information-access and command-structure categories; percentages need not sum to 100%.Source: Figure 12 · p. 21, F INTERACTION STATISTICS · p. 22
• Over 90% of requests combine commands for Astra, Opus 5, DeepSeek, and GLM, compared with 25.6% for Gemini.Gemini has the highest explicit image-processing share; Opus 5 and Gemini use loops more often than Astra.Source: Figure 12 · p. 21, F INTERACTION STATISTICS · p. 22
• Astra and Grok adjust viewing direction in about 59% of requests, while Astra toggles the map more frequently.Gemini often stays in one place while checking maps, operating menus, and processing screenshots without resuming movement.Source: Figure 12 · p. 21, F INTERACTION STATISTICS · p. 22

• Gemini converts player-to-marker pixel offsets into estimated coordinates to orient and move toward U20; the scale is uncalibrated and arrival is unverified.Source: G CASE STUDIES · p. 22, Figure 13 · p. 22
• Astra packages four 1.3-second forward movements and turns into a loop, completing one stairwell circuit from Y40 to Y48 and three circuits through Y72.Source: G CASE STUDIES · pp. 22–23, Figure 14 · p. 23
• Agents use programming to turn visual map information into explicit coordinates and individual movement commands into reusable action sequences.Source: G CASE STUDIES · p. 23