PaperlayerView exact revision on arXiv

Source-linked AI summary

ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop

Yining Hong, Jiageng Liu, Han Yin, Manling Li, Leonidas Guibas, Li Fei-Fei, Jiajun Wu, Yejin Choi

arXiv:2605.18746v2cs.CVcs.AIcs.CLcs.LGcs.RO

TL;DR

Embodied spatial intelligence requires agents to choose actions that reveal information unavailable from passive observations. ESI-BENCH evaluates this perception-action loop across diverse spatial tasks, finding that active exploration improves performance while action selection, belief revision, and confidence calibration remain difficult.

  • Problem

    Prior spatial-intelligence benchmarks largely assume passive oracle observations, leaving agents’ ability to select informative actions and resolve hidden spatial properties under-evaluated.

  • Method

    ESI-BENCH benchmarks embodied spatial intelligence across 10 categories and 29 subcategories, requiring agents to acquire evidence through perception, locomotion, and manipulation.

  • Results

    Active exploration improves over passive observation, while action selection, belief revision, and confidence calibration remain major model weaknesses.

  • Takeaways & Limitations

    Closing the perception-action loop is a useful testbed for diagnosing action-selection and epistemic-calibration failures in embodied spatial agents.

  • Takeaways & Limitations

    The benchmark relies on synthetically generated OmniGibson scenes and GPT-4o-assisted task construction, requiring audits for generation artifacts and shortcut biases.

Abstract

from arXiv · show

Spatial intelligence unfolds through a perception-action loop: agents act to acquire observations, and reason about how observations vary as a function of action. Rather than passively processing what is seen, they actively uncover what is unseen - occluded structure, dynamics, containment, and functionality that cannot be resolved from passive sensing alone. We move beyond prior formulations of spatial intelligence that assume oracle observations by recasting the observer as an actor. We introduce ESI-BENCH, a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson, grounded in Spelke's core knowledge systems. Agents must decide what abilities to deploy - perception, locomotion, and manipulation - and how to sequence them to actively accumulate task-relevant evidence. We conduct extensive experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts, with agents spontaneously discovering emergent spatial strategies without explicit instructions, while random multi-view often adds noise rather than signal despite consuming far more images. Most failures stem not from weak perception but from action blindness: poor action choices lead to poor observations, which in turn drive cascading errors. While explicit 3D grounding stabilizes reasoning on depth-sensitive tasks, imperfect 3D representation proves more harmful than 2D baselines by distorting spatial relations. Human studies further reveal that unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality, exposing a metacognitive gap that neither better perception nor more embodied interaction alone can close.

1 Introduction

ESI-BENCH reframes spatial intelligence as an embodied perception–action loop in which agents actively choose abilities and observations to resolve what passive sensing cannot. Experiments show that action selection, evidence-seeking, and epistemic calibration are central bottlenecks beyond perception alone.

  • Spatial intelligence requires agents to know how observations change with action and which actions elicit informative sensing, especially for occluded or otherwise unseen properties.The introduction characterizes perception as perceptually guided action rather than passive sensing.
  • The benchmark contrasts passive single-view, passive random multi-view, and active exploration, using a ground-truth oracle to separate perception errors from action errors.This evaluation setup directly tests whether agents choose effective embodied abilities and observation sequences.
  • ESI-BENCH evaluates embodied spatial competence across 10 task categories, 29 subcategories, and 3,081 task instances grounded in Spelke’s core knowledge systems.Its benchmark design targets questions that cannot be answered from passive observation alone.
  • Action blindness dominates perceptual blindness for most tasks: suboptimal actions produce uninformative views, triggering worse subsequent actions and unrecoverable cascading errors within the step budget.Oracle viewpoints yield largely successful models, although some tasks retain a hard perceptual ceiling.
  • Explicit 3D grounding stabilizes depth-sensitive reasoning, but imperfect reconstructions can mislead more than 2D baselines by distorting fine-grained spatial relations.The introduction also identifies epistemic calibration as a critical gap in human studies.

2 Related Works

Prior spatial benchmarks largely evaluate reasoning over fixed observations, while embodied benchmarks assess navigation, interaction, and question answering. ESI-Bench instead evaluates whether models can actively choose the observations needed for spatial reasoning.

  • Benchmarks for Spatial Reasoning: Most spatial benchmarks assume fixed observations, spanning single-image relations, visual perception, fine-grained 3D reasoning, tool-augmented agents, and egocentric video.Examples include VSR, BLINK, 3DSRBench, SpatialScore, and VSI-Bench.
  • Benchmarks for Spatial Reasoning: VILASR combines textual reasoning with visual drawing, improving reasoning over provided inputs rather than testing observation selection.ESI-Bench tests whether models can choose the observations they need.
  • Embodied Evaluation and Active Perception: EmbodiedBench and EmbodiedEval measure navigation, interaction, and question answering, while OpenEQA and EXPRESS-Bench study embodied question answering.These benchmarks represent a separate line evaluating models as embodied agents.

3 ESI-BENCH

ESI-Bench is a 3,081-instance benchmark spanning 10 task categories and 29 subcategories, designed to evaluate embodied spatial reasoning through interactive 3D scenes and egocentric actions. Tasks formalize scene state, agent pose, question, answer, action space, observations, and scene transitions, with generated trajectories and human verification supporting reproducible evaluation.

  • Task definition: Each task is defined by a 3D scene, initial agent pose, natural-language spatial question, and ground-truth answer within an environment governed by actions, egocentric observations, and scene transitions.The formal task tuple is (S, p0, q, y∗), and the environment is E = ⟨S, A, O, T⟩.
  • Task construction: GPT-4o selects task-relevant objects and configurations, determines agent and object initial positions, and generates the optimal action trajectory for resolving each task.The selected objects and their spatial arrangement define the task, while the answer is derived from the resulting scene state.
  • Verification and audit: Human annotators verify physical correctness, answerability, and non-triviality, and find GPT-4o-generated tasks high-quality, limited in shortcut bias, and similarly difficult to human-generated tasks.Verification uses rendered per-step observations and metadata to ensure tasks require genuine spatial uncertainty rather than visual bias or prior knowledge.
  • Benchmark scope: ESI-Bench comprises 3,081 task instances across 10 categories and 29 subcategories, with each category requiring distinct combinations of embodied abilities.The benchmark is built on BEHAVIOR-1K within OmniGibson and covers interactive 3D environments.

4 Experiments

Experiments compare passive, active, and oracle observation paradigms across 2D, 3D-augmented, and human agents. Active exploration improves performance through emergent strategies, but action blindness, imperfect 3D reconstruction, and poor epistemic calibration remain major failure sources.

  • Experimental setup: The benchmark evaluates passive single-view, passive multi-view, active exploration, and oracle paradigms using 2D vision-language and 3D-augmented models.Passive multi-view supplies 30 randomly sampled trajectory views, matching the maximum active trajectory budget Tmax = 30.
  • Active exploration: Agents spontaneously discover spatial strategies, improving View Hallucination from 39.9% to 68.1% and Rigid Containment from 47.5% to 67.5%.Strategies include moving behind objects, repositioning top-down, picking objects up, and pouring them out; passive multi-view often adds noise rather than signal.
  • Action selection: GPT-5 improves from 42.5% under active exploration to 95.0% with ground-truth trajectory views on Rigid Containment, indicating action selection is usually the bottleneck.On Physical Contact, GPT-5 improves from 64.2% to 90.0%; Geometric Configuration remains limited to 26.0% under ground-truth passive views.
  • Action selection: Active-to-oracle gaps reach 43.4 points on Counting w Occlusion and 45.0 points on Structural Enclosure as bad actions produce bad views and cascading unrecoverable errors.Some tasks instead reach hard perceptual limits that action strategies cannot overcome.
  • 3D grounding: Ground-Truth 3D+Gemini reaches 70.8% versus 27.5% for Gemini 3.1 on Geometric Configuration, but imperfect reconstruction can be more harmful than 2D baselines.On Counting w Occlusion, Ground-Truth 3D+Gemini reaches 33.3% versus 3.3%, a 30.0-point gain.
  • Epistemic calibration: Models can match fixed-observation human performance yet commit prematurely, seek confirmation, and fail to revise beliefs when contradicted.Under ground-truth trajectories, Gemini reaches 88.4% on Partial Occlusion compared with 87.4% for humans.

5 Conclusion · A Contribution Statement · B Broader Impacts & Limitations

ESI-BENCH evaluates embodied spatial intelligence by requiring agents to acquire evidence through perception, locomotion, and manipulation, revealing gaps in action selection, belief revision, and confidence calibration. Its diagnostic value extends to improving the safety and reliability of embodied systems in real-world settings.

  • 5 Conclusion: ESI-BENCH spans 10 categories and 29 subcategories requiring agents to close the perception-action loop through active evidence acquisition.The benchmark combines perception, locomotion, and manipulation rather than relying on pre-given observations.
  • 5 Conclusion: Active exploration improves over passive observation, while models continue to struggle with action selection, belief revision, and confidence calibration.These failures motivate ESI-BENCH as a testbed for studying agents that actively close the perception-action loop.
  • A Contribution Statement: Yining Hong contributed the idea, Perceptual Grounding code, multiple benchmark task implementations, paper writing, figure drawing, code structuring, author coordination, and the website.The listed tasks included Physical Structure, Physical Dynamics, Spatial Relations, Metric Comparison, Agent Observation and Action Sequencing.
  • A Contribution Statement: Jiageng Liu wrote code for Deformable, Specular Reflection, Cognitive Mapping, Enumerative Perception, and Unobserved Change, and handled file generation, code cleaning, and the demo video.These responsibilities are reported in the contribution statement.
  • A Contribution Statement: Han Yin contributed to the related works section and drew some original material.The supplied contribution passage is truncated after this description.
  • B Broader Impacts & Limitations: The benchmark’s primary societal benefit is diagnostic, exposing systematic failures in epistemic calibration and action selection.These failure patterns provide concrete targets for improving embodied-system safety and reliability.
  • B Broader Impacts & Limitations: Improved embodied-system safety and reliability could benefit deployments in homes, hospitals, warehouses, and assistive robotics.The broader-impacts passage frames these environments as settings where the benchmark’s diagnostic targets may matter.

C Per-Category Task Description … H.2 Verification Scores for GPT-4o-Generated Tasks

The appendix specifies ESI-BENCH’s taxonomy, human-study procedures, qualitative examples, and validation pipeline. GPT-4o proposals are simulator-filtered and human-verified for correctness, answerability, and non-triviality, with lower pass rates concentrated in interaction- and hidden-state-heavy categories.

  • C Per-Category Task Description: ESI-BENCH defines 10 task categories spanning 29 subcategories, organized around Spelke’s four core knowledge systems.The taxonomy covers object representation, layout and geometry, number representation, and agents and goal-directed actions.
  • D Per-Category Task Distribution: Figure 6 details the distribution of subcategories within each of ESI-BENCH’s 10 task categories.
  • E Human Study Protocol: The human study recruited 28 participants who explored OmniGibson scenes through locomotion, perception, and manipulation actions before answering tasks.Participants completed tasks from at least 6 categories, and researchers recorded action sequences, frames, answers, confidence, and response timing.
  • F More Benchmark Examples: Additional benchmark examples are organized by the four core knowledge systems: object representation, layout and geometry, number representation, and agents and goal-directed actions.
  • G More Qualitative Examples: Qualitative examples show active agents using top-down views, manipulation, alternative viewpoints, and interaction to resolve hidden objects and compare sizes.Mirror angles can cause persistent misjudgment, while densely packed objects can merge across viewpoints and produce undercounting.
  • H Human Verification and Generator Bias Analysis: GPT-4o serves as a proposal engine, while final tasks are instantiated, simulator-filtered, and verified using rendered observations, simulator metadata, and human judgment.The appendix also audits linguistic and object-category artifacts with shortcut baselines, diversity measures, and matched accuracy comparisons.
  • H.1 Human Verification Protocol: Each candidate is independently checked by 3 annotators for correctness, answerability, and non-triviality, and retained only if it passes all three axes.Correctness concerns simulator consistency, answerability requires uniquely obtainable evidence through available actions, and non-triviality requires active evidence acquisition.
  • H.2 Verification Scores for GPT-4o-Generated Tasks: Verification scores are generally high, but Physical Dynamics, Enumerative Perception, Temporal Understanding, and Action Sequencing have lower overall pass rates than Cognitive Mapping and Metric Comparison.These differences reflect greater demands for physical control, occlusion handling, object interaction, or temporal dependencies.

H.3 Bias Audit for GPT-4o-Generated Tasks · H.4 Comparison with Human-Generated Tasks

The bias audit finds that GPT-4o-generated tasks are diverse and that shortcut baselines underperform visual and active exploration. Matched human comparisons show broadly similar model accuracy, indicating no substantial systematic difficulty bias from task generation.

  • H.3 Bias Audit for GPT-4o-Generated Tasks: ESI-BENCH audits linguistic and object-category artifacts using question-only and metadata-only shortcut baselines.The audit removes visual observations, positions, action history, and simulator state to test whether wording or object categories reveal answers.
  • H.3 Bias Audit for GPT-4o-Generated Tasks: 36.6% question-only accuracy and 39.3% metadata-only accuracy remain below 42.5% passive single-view and 56.9% active exploration.This ordering indicates that language and object-category priors do not explain benchmark performance, while action-guided evidence acquisition adds value.
  • H.3 Bias Audit for GPT-4o-Generated Tasks: GPT-4o-generated tasks are not dominated by a small set of labels, objects, or spatial templates.Answer distributions have average normalized entropy 0.92, while object diversity reflects BEHAVIOR-1K inventories, random 200-category subsampling, and category-specific filtering.
  • H.4 Comparison with Human-Generated Tasks: Human annotators authored matched tasks under the same category definitions, action space, and simulator scene context as GPT-4o.The comparison evaluates representative agents on subsets with the same category distribution.
  • H.4 Comparison with Human-Generated Tasks: Model accuracies are broadly similar across GPT-4o-generated and human-generated subsets, with average gaps within a few percentage points.The matched comparison tests whether GPT-4o-generated tasks are systematically easier or harder than human-authored tasks.
  • H.4 Comparison with Human-Generated Tasks: Simulator-filtered GPT-4o task proposals achieve high human verification scores and are generally correct, answerable, and non-trivial.Human verification is presented as evidence supporting the quality of the generated task proposals.
  • H.4 Comparison with Human-Generated Tasks: Together, the results indicate that benchmark performance cannot be explained by linguistic or object-category shortcuts, and GPT-4o-generated tasks have similar difficulty to human-generated tasks.Question-only and metadata-only baselines remain below passive single-view and far below active exploration, while cross-source model accuracies remain close.

I Diverse Emergent Behaviors in Active Exploration

Active exploration produces diverse, self-developed behavioral routes for resolving the same spatial uncertainty, rather than following a single prescribed policy. Strategy clustering shows that these routes reflect task-relevant evidence acquisition beyond simply receiving more images, although diversity does not guarantee human-like epistemic control.

  • I Diverse Emergent Behaviors in Active Exploration: Agents develop multiple ways to resolve the same spatial uncertainty without category-specific policies, prescribed trajectories, or step-by-step instructions.These behaviors emerge from the interaction among the question, current egocentric observation, and embodied action space.
  • I Diverse Emergent Behaviors in Active Exploration: Human annotation groups successful active trajectories into distinct behavioral clusters without predefined strategy labels.The number of clusters measures how many qualitatively different routes agents use to acquire evidence within each representative subcategory.
  • I Diverse Emergent Behaviors in Active Exploration: Emergent diversity varies substantially across subcategories, with agents using different viewpoints, interactions, and action orders to reach answers.This indicates that active exploration can generate several self-developed behavioral patterns for the same underlying spatial problem.
  • I Diverse Emergent Behaviors in Active Exploration: Strategy clustering indicates that active-exploration gains are not explained solely by receiving more images, because agents acquire task-relevant evidence through multiple behavioral routes.However, behavioral diversity alone does not ensure human-like epistemic control; models may still commit prematurely or fail to revise beliefs.

J Failure Analysis of VGGT-Based 3D Augmentation

VGGT-based 3D augmentation can hurt fine-grained spatial reasoning when reconstruction and scene-graph errors corrupt symbolic inputs. The failures reflect unreliable reconstructed structure—not an inherent lack of value in accurate 3D representations.

  • Performance degradation: VGGT + Gemini underperforms the 2D Gemini baseline on several fine-grained spatial tasks, most strikingly in Counting w Occlusion.The degradation is attributed to errors introduced during 3D reconstruction and scene-graph construction.
  • Recurring error modes: Object duplication is the dominant failure mode, as partial or depth-ambiguous observations become repeated fragments or instances that over-count objects.This is especially harmful for Enumerative Perception tasks, whose answers depend directly on object-instance counts.
  • Recurring error modes: Object hallucination and corrupted spatial relations further make reconstructed scene graphs misleading for downstream reasoning.These recurring errors follow the dominant duplication problem in VGGT-based augmentation.
  • Interpretation: Ground-truth 3D representations improve several depth-sensitive tasks, whereas noisy reconstructed scene graphs can be more misleading than 2D baselines.The key limitation is the mismatch between ESI-BENCH’s precision requirements and current scene-graph reliability.

K Operationalizing the Metacognitive Gap

The section operationalizes the human–MLLM metacognitive gap through trajectory-level measures of evidence sufficiency, falsification seeking, and belief revision. These metrics complement accuracy by assessing whether agents gather adequate evidence, seek disconfirming observations, and update beliefs when new evidence arrives.

  • Metacognitive gap: Humans seek falsifying viewpoints, delay commitment under ambiguity, and revise beliefs when contradicted, whereas models often stop early with high confidence.This behavioral contrast motivates operationalizing the gap with trajectory-level measures.
  • Operational measures: Evidence sufficiency measures whether agents gather diverse viewpoints and observations that directly reveal task-relevant evidence.Viewpoint diversity and task-relevant evidence acquisition provide practical approximations of sufficiency.
  • Operational measures: Falsification seeking measures whether agents choose actions that could disconfirm their current hypothesis rather than merely confirm it.Contrastive evidence includes observations that disambiguate competing hypotheses or reveal evidence against the initial hypothesis.
  • Operational measures: Belief revision rate measures the percentage of trajectories in which the predicted answer changes after additional observations.Together with view diversity and contrastive view rate, it is computed on active trajectories.
  • Human–model comparison: Humans acquire more diverse and contrastive evidence and revise beliefs more frequently than current MLLMs, whose falsification-seeking and belief-revision rates are lower.These measures complement accuracy by capturing whether agents recognize insufficient evidence and acquire better evidence.

L Analysis of Passive Multi-View Evaluation

Passive multi-view input often fails to outperform passive single-view input and can degrade performance, showing that unstructured observation accumulation does not substitute for active exploration. The analysis attributes this weakness to random view selection, image overload, and absent action-linked temporal grounding rather than multi-view reasoning itself.

  • Findings: Passive multi-view input often fails to improve over passive single-view input and can degrade performance in some task categories.The baseline uses 30 randomly sampled views to match the active exploration step budget, but is not an optimal multi-view reasoning system.
  • Failure sources: 30 randomly sampled views may miss diagnostic regions while adding irrelevant images, making unstructured observation accumulation ineffective.The analysis also identifies image overload when only a few views contain task-relevant evidence.
  • Failure sources: Passive views are unordered and disconnected from the agent’s actions, limiting inference about how observations relate to spatial movement.This lack of temporal grounding is identified as one reason passive multi-view input may underperform.
  • Interpretation: The results do not show that multi-view reasoning is inherently unhelpful; stronger baselines could learn view selection, rank views, or align them spatially.The central limitation is unstructured view accumulation, not the use of multiple views itself.

M Why a Synthetic Simulator Benchmark? · N Active Agent Prompting and Action Selection · O Why High-Level Actions?

ESI-BENCH uses a controlled simulator and high-level action space to diagnose whether agents actively acquire evidence and close the perception-action loop, while isolating failures from real-world sensing and low-level control. Its active-agent protocol requires selecting one evidence-seeking action at a time from the current observation, history, and action vocabulary.

  • M Why a Synthetic Simulator Benchmark?: ESI-BENCH uses simulation to make hidden spatial and physical variables exactly verifiable for controlled diagnosis.These include containment, contact, fill level, occlusion, room connectivity, object count, and unobserved state changes.
  • M Why a Synthetic Simulator Benchmark?: OmniGibson and BEHAVIOR-1K scenes systematically vary occlusion, viewpoint ambiguity, object placement, interaction, and action requirements while controlling difficulty.This separates evidence acquisition from uncontrolled sensor noise, calibration, and deployment constraints.
  • M Why a Synthetic Simulator Benchmark?: The simulator is a controlled testbed rather than a claim of direct real-world transfer, because real-world noise, actuation, reconstruction, and variability could obscure perception-action failures.The central question is whether current MLLMs actively select informative observations.
  • N Active Agent Prompting and Action Selection: At each timestep, the active MLLM receives the task, egocentric observation, action-observation history, and full action vocabulary, then selects exactly one next action.The action executes in OmniGibson and the loop continues until answer(answer, confidence) or Tmax = 30 steps.
  • N Active Agent Prompting and Action Selection: The prompt directs agents to explore only when actions gather evidence, delay answering until sufficient evidence, and return a brief reason plus one valid action.Available actions include movement, turning, object manipulation, filling, pouring, and answering with confidence.
  • N Active Agent Prompting and Action Selection: Invalid, malformed, or multiple-action outputs trigger one reprompt; persistent invalidity counts the step as invalid and leaves the agent in place.Manipulation object names are matched to visible or previously observed object identifiers.
  • O Why High-Level Actions?: High-level discrete actions preserve choices about movement, inspection, manipulation, and answering while reducing robot-specific control confounds.This makes failures more diagnostic of evidence acquisition, action selection, belief updating, or spatial reasoning than controller performance.
  • O Why High-Level Actions?: Low-level control is intentionally factored out as an orthogonal, complementary direction involving continuous commands, imperfect actuation, and execution-error recovery.ESI-BENCH therefore focuses on higher-level spatial intelligence rather than visuomotor control.

P Step Budget Ablation … Q.5 Physical Dynamics: Stacking and Stability

The benchmark uses targeted active exploration, with performance saturating around a 30-step budget, and constructs physically grounded tasks through controlled object selection, placement, simulation, and diagnostic viewpoints. Its physical-dynamics tasks derive labels from settling outcomes, geometric checks, and stability criteria across transparency, liquid volume, deformability, inclined-plane behavior, and stacking.

  • P Step Budget Ablation: Performance rises rapidly from 5 to 15 steps, saturates around 30, and slightly decreases after 40, motivating Tmax = 30 for active exploration.Longer budgets can add computational cost, redundant views, conflicting evidence, and excessive shifts from task-relevant exploration.
  • Q Per-Category Task Construction Details: Each task pipeline selects relevant objects, creates physically valid configurations, renders fixed egocentric views, and records ground-truth labels after physics settles.Scenes use a single room, a 200-category BEHAVIOR-1K subsample, and task-diagnostic initial viewpoints.
  • Q.1 Physical Structure: Material Transparency: Transparency tasks sample containers and smaller objects, place positives inside or settle them with containment checks, maximize initial occlusion, and label whether the object is inside.Object size is constrained by the container’s narrowest cross-section.
  • Q.2 Physical Structure: Liquid Volume: Liquid-volume tasks select container pairs differing by at least 200 particles, simulate complete transfer or overflow, and label the container with larger offline-measured capacity.Capacities are measured through a particle-based water-dip simulation and offline analysis.
  • Q.3 Physical Structure: Deformable: Deformable tasks cover a settled target object with cloth and ask for its semantic category among four shuffled choices containing one target and three distractors.Targets are portable objects selected from a GPT-4o-generated pool, while cloth assets are constrained in size.
  • Q.4 Physical Dynamics: Inclined Plane: Inclined-plane tasks randomize object mass, slope angle, and friction, then classify the object as stable, sliding, or falling from displacement and height thresholds after 30 simulation steps.Slope angles range from 10° to 45°, while the simulation uses 0.03 m XY displacement and 0.05 m vertical-drop thresholds.
  • Q.5 Physical Dynamics: Stacking and Stability: Stacking tasks select 2 or 3 solid objects, normalize their XY footprints, test every permutation after 60 settling steps, and derive labels from stable stacking orders.Stability uses on-top geometry and uprightness checks, including a world-up dot product above 0.9, corresponding to roughly 26° from vertical.

Q.6 Metric Comparison: Spatial Distance … Q.28 Cognitive Mapping : Long-Term Navigation

The benchmark constructs embodied spatial-intelligence tasks by placing objects, containers, agents, and rooms in deliberately ambiguous configurations that require perception, reasoning, and navigation. Across Q.6–Q.28, ground truth is derived from geometric relations, temporal states, visibility, containment, room connectivity, and shortest-path structure.

  • Q.10 Physical Structure: Rigid Containment; Q.11 Action Sequencing: Action Order Inference; Q.12 Temporal Understanding: Agent Observation: Q.10–Q.12 test physical structure, action ordering, and temporal observation through containment assignments, hierarchical placement sequences, and objects encountered along navigation paths.Labels specify object-to-container assignments, the candidate required at each action step, or the count and categories of hidden objects passed by the navigating robot.
  • Q.13 Temporal Understanding: Unobserved Changes; Q.14 Perceptual Grounding: Partial Occlusion; Q.15 Perceptual Grounding: View Hallucination: Q.13–Q.15 probe unobserved changes and perceptual ambiguity using phase-based box changes, partial occlusion, and viewpoints selected to maximize category confusion.Ground truth records change presence or type, the changed box, the occluded object category, or the true category against confusable alternatives.
  • Q.16 Specular Reflection : Reflection Authorization; Q.17 Specular Reflection : Spatial Relationship; Q.18 Specular Reflection : Scene Correspondence: Q.16–Q.18 use geometric mirror setups to test reflection authorization, real-object distance comparison, and correspondence between reflected and real-image positions.Labels respectively identify whether an object is seen through the mirror, which object is closer in 3D Euclidean distance, and whether the target appears left, middle, or right.
  • Q.25 Cognitive Mapping : Connectivity; Q.26 Cognitive Mapping : Traversable Passage: Q.25–Q.26 derive cognitive-map connectivity and traversable-passage labels from shortest paths projected onto room sequences.Connectivity is Yes when a valid shortest path exists, while passage questions depend on whether the queried via-room occurs between source and target.
  • Q.27 Cognitive Mapping : Regional Boundary; Q.28 Cognitive Mapping : Long-Term Navigation: Q.27–Q.28 extend cognitive mapping to regional boundaries and long-term navigation across the farthest connected room pairs.Labels assign objects to regions or compare regional proximity, and navigation answers follow projected room sequences with turn choices computed from heading changes.
Loading 2605.18746v2…