PaperlayerView exact revision on arXiv

Source-linked AI summary

Representing the Unseen: Rethinking Visual Imagination for Spatial Reasoning

Zengqi Zhao, Zun Wang, Shoubin Yu, Mohit Bansal, Yue Zhang

arXiv:2610.37437v1cs.CVcs.AIcs.CL

TL;DR

RGB is the default interface for imagined states, although it is not necessarily the best representation for spatial reasoning. IMAGIREP compares representations of the same states, and GEARS adaptively selects imagined evidence; results show complementary benefits across representations and improved performance with GEARS over other spatial imagination methods.

  • Problem

    RGB is commonly used because it suits visual generation, but existing methods do not establish that it best supports spatial reasoning, and prior benchmarks do not control access to the same underlying states across representations.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    IMAGIREP compares oracle representations of the same imagined states across two settings and seven question categories, while GEARS adaptively decides when to use imagined evidence and which representation to select.

    Source: §1 · pp. 1–2

  • Results

    Different representations provide complementary benefits across models and questions, with none consistently best; GEARS outperforms other spatial imagination methods, while a substantial gap remains versus oracle-assisted results.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Effective spatial imagination depends not only on view accuracy but also on representing spatial information in ways reasoners can use.

    Source: §1 · p. 2, §6 · p. 8

    IMAGIREP covers static indoor scenes and selected tasks; estimated-representation analysis uses few world models and does not evaluate all representations in both settings.

    Source: §7 · p. 9

Abstract

from arXiv · show

Visual world models increasingly support spatial reasoning by generating RGB views of unobserved states. However, RGB has become the default interface largely because it is well suited for visual generation, rather than necessarily being the most effective representation for downstream reasoning. Different spatial questions may benefit from different representations of the same imagined state: top-down geometry can facilitate global distance comparisons, while panoramic views provide broader context for reasoning about open space. To isolate the role of representation in spatial imagination, we introduce ImagiRep, a simulator-grounded benchmark of 922 questions spanning seven question types across two settings, Viewpoint Imagination and Cross-view Integration, providing oracle RGB and six alternative representations (e.g., depth, panoramic views, and top-down geometry) of the same unobserved state. This controlled comparison reveals that even accurate imagined RGB does not guarantee accurate spatial reasoning, and different questions benefit from different representations. Similar trends hold for representations derived from world-model-estimated states. Notably, estimated RGB alone can mislead spatial reasoning, while alternative representations retain complementary benefits. Building on these findings, we introduce GEARS, a lightweight test-time strategy for adaptive representation selection, outperforming existing spatial imagination methods and showing that reasoning about unseen states requires more than generating RGB views.

1 Introduction

Spatial imagination systems commonly expose imagined states as RGB, although different spatial questions may benefit from different representations. The paper introduces controlled comparisons and GEARS, finding complementary representation benefits and improved test-time reasoning.

1 Introduction

  • IMAGIREP holds the underlying spatial state and question fixed while comparing representations across seven question categories and two settings.The settings are Viewpoint Imagination and Cross-view Integration.Source: §1 · pp. 1–2
  • Across GPT-6.1-sol and Gemini 3.7 Flash, imagined-view accuracy does not ensure reasoning usefulness, and no single representation consistently performs best.Representations derived from estimated states can still improve reasoning even when estimated RGB alone degrades performance.Source: §1 · p. 2
  • 68.4% to 58.2%: adding oracle endpoint RGB reduces GPT-6.1-sol accuracy on Farthest Object questions, while top-down geometry raises it to 75.5%.The example illustrates that how spatial information is presented matters.Source: §1 · p. 2
  • 61.4% accuracy on Viewpoint Imagination and 77.7% on Cross-view Integration; GEARS outperforms other spatial imagination methods.A substantial gap remains compared with oracle-assisted results.Source: §1 · pp. 2–3
  • GEARS checks agreement between predictions using observed evidence and estimated RGB, then adaptively selects observed evidence or estimated representations when they disagree.It is a lightweight test-time strategy for choosing when to use imagined evidence and which representation to select.Source: §1 · p. 2
  • GEARS remains substantially behind oracle-assisted results, leaving more accurate state estimation and usable spatial information as open needs.The paper identifies both estimation accuracy and the usefulness of information to reasoners as areas needing improvement.Source: §1 · p. 3

2 Related Work

Prior spatial reasoning benchmarks cover diverse abilities, but do not systematically isolate how alternative representations of the same unobserved state affect reasoning. This work compares representations under oracle and estimated imagination and explores adaptive selection.

2 Related Work

  • Prior benchmarks assess diverse spatial abilities but do not systematically compare representations of the same unobserved state, leaving representation effects difficult to isolate from state-estimation errors.The cited benchmark examples include perspective changes, object relations, and imagined movements (Ray et al., 2025; Yang et al., 2026b; Wang et al., 2026b).Source: §2 · p. 3
  • IMAGIREP provides multiple oracle representations of one simulator-grounded target state, holding the spatial state and question fixed for controlled comparison.Source: §2 · p. 3
  • Existing world-model methods primarily rely on imagined RGB, while this work compares oracle and estimated representations and explores adaptive test-time selection.Prior methods include AVIC (Yu et al., 2026) and Astra (Zhu et al., 2026); the passage notes that world models can also generate depth, surface normals, and camera geometry (Aether Team et al., 2025).Source: §2 · p. 3

3 IMAGIREP: A Controlled Benchmark for Spatial Imagination

IMAGIREP is designed to isolate representation utility by holding the target spatial state and question fixed while varying the imagined evidence. It spans viewpoint changes and cross-view integration, with multiple representations rendered from shared scene geometry and camera information.

3.1 Benchmark Overview and Design Principles

  • IMAGIREP addresses the lack of controlled representation comparisons by using Matterport3D geometry and camera poses to vary representation while keeping the scene and question fixed.Its design principles cover unobserved states, diverse spatial tasks, and controlled representation comparisons.Source: §3.1 · pp. 3–4

3.2 Spatial Imagination Tasks

  • Viewpoint Imagination tests spatial updates after instructed movement, while Cross-view Integration tests combining information distributed across partial observations.The six viewpoint question types cover direction, distance, and visibility; cross-view tasks include nearest/farthest-object comparisons and spatial ordering.Source: §3.2 · p. 4

3.3 Representations of Imagined States

  • The benchmark renders oracle RGB and six additional representations from the same target state, including depth, surface normals, panorama, top-down geometry, spatial text, and video.Depth and surface normals use the target viewpoint; panoramas and top-down maps provide broader spatial context, while spatial text and video encode movement information.Source: §3.3 · p. 5

3.4 Dataset Statistics and Quality Control

  • 922 human-annotated questions span 622 Viewpoint Imagination questions and 300 Cross-view Integration questions, with manual review of inferability and quality checks.The questions use four candidate answers; the benchmark also checks rendering quality, camera-pose alignment, and annotation consistency.Source: §3.4 · p. 5

4 Evaluating Imagined Evidence for Spatial Reasoning

The evaluation compares spatial reasoning with oracle and estimated imagined evidence, first testing endpoint RGB, then alternative representations, and finally estimated imagination.

4.1 Experimental Setup

  • The setup compares two multimodal models and six oracle representations while holding questions, answer options, and the underlying spatial state fixed.It also evaluates estimated representations; accuracy is the percentage of correctly answered questions.Source: §4.1 · pp. 5–6

4.4 Do Representation Benefits Persist under Imperfect Imagination?

  • Despite imperfect imagination, six alternative representations provide complementary evidence, yielding potential error-recovery scores of 61.7–65.6%; selecting the best representation per question type raises this to 67.2%.For Cross-view Integration, estimated RGB improves the score from 68.0% to 76.0%, and estimated depth further increases it to 82.0%.Source: §4.4 · p. 8

4.2 Does Oracle Imagination Improve Spatial Reasoning?

  • Oracle endpoint RGB adds around 10% accuracy across models and settings, but errors remain and GPT-6.1-sol falls from 68.4% to 58.2% on Farthest Object.Observed views improve accuracy by more than 20 absolute percentage points over question-only performance; Viewpoint Imagination is harder than Cross-view Integration, and oracle RGB gains vary by question type.Source: §4.2 · p. 6

4.3 Can Alternative Representations Improve Reasoning Beyond RGB?

  • On GPT-6.1-sol, panorama scores 79.3% versus RGB’s 66.1% on Viewpoint Imagination, while depth scores 84.7% versus 75.7% on Cross-view Integration.Benefits also vary by question: top-down geometry raises Farthest Object accuracy from 58.2% with RGB to 75.5%, panorama raises Open-space Direction from 57.1% to 73.6%, and depth raises Cross-view Nearest Object from 74.1% to 84.0%.Source: §4.3 · p. 7

5 Our Method: Gated Evidence and Adaptive Representation Selection (GEARS)

GEARS adaptively decides when to use imagined evidence and which representation to provide. It achieves the highest accuracy on most question types against existing spatial-imagination methods.

Motivation

  • GEARS addresses two practical challenges: estimated RGB may not help, and different representations provide complementary spatial information.The approach is a lightweight test-time strategy for choosing whether to use imagined evidence and which representation to supply.Source: §5 · p. 8

Gating and adaptive selection

  • GEARS keeps the answer when predictions with and without estimated endpoint RGB agree, then routes disagreement cases to evidence suited to the question.It considers all six alternative representations for Viewpoint Imagination and estimated depth for Cross-view Integration.Source: §5 · p. 8

Comparison with existing methods

  • GEARS scores 61.4% on Viewpoint Imagination and 77.7% on Cross-view Integration, leading on most question types against MindJourney and AVIC.All methods use GPT-6.1-sol as the final reasoner and are evaluated on the same benchmark questions.Source: Table 5 · p. 8, §5 · p. 8

6 Conclusion

The paper introduces IMAGIREP to study how representations of imagined spatial states affect reasoning. Its findings motivate GEARS and a focus on representations suited to reasoning about unseen states.

Benchmark

  • IMAGIREP is a controlled benchmark for studying how representations of imagined spatial states affect reasoning.The conclusion frames this as a study of representation choice for spatial reasoning over unseen states.Source: §6 · p. 8

Findings

  • Accurate imagined RGB does not guarantee effective reasoning, while alternative representations offer complementary benefits even under imperfect imagination.The conclusion reports that GEARS improves test-time spatial reasoning.Source: §6 · p. 8

Implication

  • The conclusion calls for world models to provide spatial representations that support reasoning about the unseen, not merely accurate views.Source: §6 · p. 8

7 Limitations

The study’s findings are bounded by its scene and task coverage, limited world-model analysis, and remaining distance from oracle-assisted results.

Limitations

  • IMAGIREP covers static indoor scenes and selected tasks; dynamic environments, complex interactions, and broader world-model and representation coverage remain unexplored.GEARS still has a substantial gap relative to oracle-assisted results, leaving reliable estimation and selection of imagined evidence as open challenges.Source: §7 · p. 9

A.1 Details of IMAGIREP

The appendix documents Viewpoint Imagination annotation and trajectory characteristics, alongside the shared camera geometry used for Cross-view Integration.

A.1 Details of IMAGIREP

  • Viewpoint Imagination trajectories contain one to three moves, have a 4.2 m median length, and carry one to six questions each.257 questions (41%) ask observers to turn back toward the start; 52 trajectories include a turn of at least 45°.Source: §A.1 · p. 11
  • Cross-view Integration groups render all observed views from one camera position at a 15° downward pitch, with adjacent views 98.5° apart.Source: §A.1 · p. 11
  • Annotators review simulator-rendered trajectory views, write a shared movement description, then create a question, four answer options, and its correct answer.For look-back questions, the endpoint evidence is a view facing the start.Source: §A.1 · p. 11

A.2 Results on Oracle Representations with Gemini 3.7 Flash

With Gemini 3.7 Flash, the best added representation varies by setting and question type, while hindsight selection reaches higher accuracy than fixed additions.

A.2 Results on Oracle Representations with Gemini 3.7 Flash

  • 73.0% vs. 63.7% with endpoint RGB alone on Viewpoint Imagination, and 85.3% vs. 78.7% on Cross-view Integration, are the best additions: top-down geometry and video, respectively.Source: §A.2 · p. 11
  • 74.4% accuracy is reached on Viewpoint Imagination by adding the best representation per question type.Source: §A.2 · p. 11
  • 82.5% on Viewpoint Imagination and 89.0% on Cross-view Integration are reached by choosing the best representation for each question in hindsight.Source: §A.2 · p. 11

A.3 Implementation Details on Estimation

These appendix sections describe how estimated representations are generated, how GEARS routes evidence, and the implementation and licensing details of prompts and artifacts.

A.3 Implementation Details on Estimation

  • For Viewpoint Imagination, HY-World 2.0 builds a scene from the initial image, then plans a camera path from the movement instruction and renders endpoint representations.Planned endpoints deviate from ground truth by a median of 1.4 m in position and 14° in heading.Source: §A.3 · p. 11
  • For Cross-view Integration, estimated poses determine a target view or trajectory; SEVA synthesizes endpoint RGB, and MoGe-2 estimates depth, the only additional representation considered.Source: §A.3 · pp. 11–12

A.4 Implementation Details of GEARS and Baselines

  • GEARS compares answers from observed views alone and with estimated endpoint RGB, returning agreement or routing conflicting cases to evidence selection before answering.Source: §A.4 · pp. 12–13
  • On Viewpoint Imagination, gating and routing each improve over No Gate, and their combination performs best; MindJourney and AVIC are comparison baselines.Source: §A.4 · p. 13, Table 10 · p. 13

A.5 Prompts of GEARS

A.7 Artifact Licenses and Usage Rights

  • Matterport3D use is limited to non-commercial academic research, and its original dataset is not redistributed; third-party models and services retain their respective usage terms.Source: §A.7 · p. 15
Loading 2610.37437v1…