PaperlayerView exact revision on arXiv

Source-linked AI summary

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Jaewoo Jung, Hyeonseo Yu, Honggyu An, Jisang Han, Mungyeom Kim, Minkyeong Jeon, Heeseong Shin, Wonjun Moon, Federico Tombari, Daniel Barath, Marc Pollefeys, Seungryong Kim, Sunghwan Hong

arXiv:2609.38177v1cs.CVcs.CL

TL;DR

Multi-view 3D reasoning remains difficult for MLLMs despite approaches that add pixel-level geometry or 3D foundation-model features. Imagine3D-LLM reconstructs a compact Gaussian scene representation before answering, and outperforms prior approaches across spatial reasoning and 3D understanding benchmarks. Its training scope is indoor scenes, and larger outdoor scenes may need more representation capacity.

  • Problem

    MLLMs struggle to integrate evidence across viewpoints into coherent 3D understanding, and existing approaches still leave a substantial gap to human reasoning.

    Source: §1 · pp. 1–2

  • Method

    Imagine3D-LLM appends learnable Gaussian summary tokens to image tokens, decodes them into a compact 3D representation, and jointly trains reconstruction and language modeling.

    Source: §1 · p. 2

  • Results

    Imagine3D-LLM significantly outperforms prior approaches across challenging spatial reasoning benchmarks, with gains across seven spatial reasoning and 3D scene understanding benchmarks.

    Source: §1 · p. 2, §5 · p. 11

  • Takeaways & Limitations

    The results suggest that abstract scene reconstruction is an effective inductive bias for building 3D-aware MLLMs.

    Source: §1 · p. 2

    Training uses indoor scenes, and the fixed Gaussian budget may be insufficient for larger, more complex outdoor scenes.

    Source: E Limitations · p. 25

Abstract

from arXiv · show

Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists. In this work, we revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, we introduce Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D representation of the scene and conditions its answer on this representation. Concretely, we append a small set of learnable summary tokens after the image tokens, decode them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and train jointly with the standard next-token prediction objective. Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM's underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model. As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.

1 Introduction

Imagine3D-LLM reconstructs a compact 3D scene representation before answering, inspired by human spatial reasoning that uses common objects and coarse spatial relationships across views. It improves performance across seven spatial reasoning and 3D scene understanding benchmarks, while reconstruction also makes the LLM’s image features more 3D-aware.

1 Introduction

  • Human spatial reasoning is described as identifying common objects across views and using them to infer coarse spatial relationships rather than maintaining pixel-perfect correspondences or dense depth maps.This motivates an object-level, approximate scene representation.Source: §1 · p. 2
  • The method appends learnable Gaussian summary tokens to image tokens and decodes them into 3D Gaussian Splatting primitives that represent portions of the scene.The tokens are jointly trained using photometric reconstruction and next-token prediction losses.Source: §1 · p. 2
  • Fewer Gaussian summary tokens than image tokens create an information bottleneck that encourages recurring objects across views to share tokens in a unified 3D layout.The design also aims to prevent direct prediction of image-aligned Gaussians from image features.Source: §1 · p. 2
  • Imagine3D-LLM achieves significant gains across seven spatial reasoning and 3D scene understanding benchmarks, while reconstruction makes image features more 3D-aware.The analysis reports sharper attention to corresponding image features and more consistent semantic encoding of the same object across views.Source: §1 · p. 2

2 Related Works

Prior 3D scene understanding methods often depend on task-specific architectures or per-scene optimization, while multi-view MLLM work injects pixel-level geometry or external 3D features. Imagine3D-LLM instead brings compact Gaussian scene representations into an MLLM for generalist language-based reasoning.

3D Scene Understanding

  • Many 3D scene understanding pipelines distill 2D foundation-model features into scene representations, but often require task-specific architectures and per-scene optimization.Their language understanding is bounded by the compositional capacity of the underlying 2D embedding model.Source: §2 · p. 2

3D Multimodal Large Language Models (MLLMs)

  • Multi-view 3D MLLMs inject pixel-level 3D signals or fuse image features with geometry-aware features from 3D foundation models, yet a substantial gap to human-level benchmark performance remains.Earlier approaches also used explicit 3D inputs, including lifted 2D features and dedicated point-cloud encoders.Source: §2 · p. 3

Compact Scene Representations

  • Compact scene representations draw on coarse abstractions and geometric primitives; Imagine3D-LLM decodes 3D Gaussians from learnable tokens within an MLLM.The LLM conditions its reasoning on the resulting compact scene representation.Source: §2 · p. 3

3 Methodology

Imagine3D-LLM extends a multi-view MLLM with compact Gaussian summary tokens that reconstruct a scene and condition language generation. The method combines language modeling, photometric reconstruction, and teacher distillation, and analyzes the resulting image features and token groupings.

3.1 Preliminaries: Multi-view MLLMs

  • A multi-view MLLM encodes each input frame into visual tokens, concatenates them with instruction tokens, and autoregressively generates text from the visual and prior text tokens.The model comprises a vision encoder, vision-language projector, and pretrained language model.Source: §3.1 · p. 3

3.2 Imagine3D-LLM

  • Imagine3D-LLM inserts a compact set of learnable Gaussian summary tokens between image and text tokens, then decodes their intermediate-layer states into a 3D Gaussian scene representation.The overview describes joint language modeling, rendered-view photometric reconstruction, and pretrained-teacher distillation.Source: Figure 1 · p. 4, §3.2 · p. 4

3.2.1 Mental Reconstruction via Gaussian Summary Tokens

  • The token bottleneck merges overlapping content across views into shared summaries, whose decoded Gaussians are trained through rendered-view reconstruction alongside language modeling.Tokens attend to visual evidence; decoding at a middle LLM layer leaves later layers for text tokens to use the scene summary.Source: §3.2.1 · pp. 4–5

3.2.2 Accelerating Convergence via Distillation from a Compact Gaussian Teacher

  • Because photometric supervision alone converges slowly, Imagine3D-LLM distills a compact Gaussian teacher’s representations and Gaussian attributes into its summary tokens and decoded scene.The teacher is ZipSplat, and the token-level target guides the LLM toward representations its Gaussian head can decode.Source: §3.2.2 · pp. 5–6

3.3 Analysis

  • Joint reconstruction training is associated with sharper cross-view image-feature correspondences and more semantically organized features; K-means clusters of summary tokens localize recurring objects without clustering supervision.The feature analysis reports consistent representations across viewpoints, while the Gaussian visualization shows object-level grouping.Source: Figure 2 · p. 6, §3.3 · pp. 6–7, Figure 3 · p. 7

4 Experiments

Imagine3D-LLM performs strongly across ScanNet-based benchmarks and SPAR-Bench, while controlled comparisons and ablations examine its reconstruction objective and design choices.

4 Experiments

  • Evaluation covers seven benchmarks: six ScanNet-based tasks and SPAR-Bench, which measures reasoning across three cognitive levels.The experiments use task-specific metrics, including exact match, captioning and grounding metrics, and SPAR-Bench accuracy by level and overall mean.Source: §4 · p. 7

4.1 Quantitative Comparison

  • Imagine3D-LLM significantly improves across ScanNet-based benchmarks, outperforming the strongest prior 3D LMM, Ross3D, on almost all benchmarks.The benchmarks cover question answering, dense captioning, and visual grounding.Source: §4.1 · p. 8, Table 1 · p. 8, Table 2 · p. 8
  • Imagine3D-LLM achieves state-of-the-art SPAR-Bench performance across every cognitive level, beating 3DThinker-7B by 5.2 points overall.It also surpasses the largest 72B general-purpose 2D LMM by over 29 points despite using a 7B backbone.Source: §4.1 · p. 9, Table 3 · p. 9

4.2 Ablation Studies

  • With identical backbone, training data, and schedule, Imagine3D-LLM improves over standard visual instruction tuning on every benchmark.Its medium-level SPAR-Bench cross-view score rises by 15.5 points, from 51.5 to 67.0.Source: §4.2 · p. 9, Table 4 · p. 9
  • Decoding Gaussian tokens from layer 14 consistently performs best among the tested LLM layers 7, 14, and 21.The authors identify layer 14 as balancing scene information in the Gaussian tokens with remaining layers for language reasoning.Source: §4.2 · p. 9, Table 5 · p. 9
  • Without distillation, reconstruction-only training performs substantially worse initially, but nearly matches the full one-epoch model after four epochs.The authors report that large reconstruction-loss gradients leave language modeling under-optimized; distillation accelerates convergence and makes training practical.Source: §4.2 · p. 10, Table 6 · p. 10
  • Teacher tokens alone and distillation alone perform on par with the baseline, whereas reconstruction makes Gaussian summary tokens aggregate visual evidence.The authors attribute the contrast to reconstruction actively shaping attention and representations, including cross-view correspondence and object-level grouping.Source: §4.2 · p. 10, Table 7 · p. 10
  • Too few Gaussian summary tokens limit capacity, while too many weaken the bottleneck that encourages object-level grouping.The tested totals were 1296, 2592, and 5184 tokens; both extremes underperform the chosen setting.Source: §4.2 · p. 10, Table 8 · p. 10

5 Conclusion

Imagine3D-LLM reconstructs a compact 3D scene representation before answering and outperforms prior approaches across challenging spatial reasoning benchmarks. Its reconstruction objective also reshapes image features into more spatially aligned and semantically structured representations.

5 Conclusion

  • Imagine3D-LLM lets a multi-image MLLM explicitly reconstruct a 3D scene before answering, using learnable Gaussian summary tokens decoded into a compact 3DGS representation.The tokens are appended to image tokens, and the framework is jointly trained with the standard language-modeling objective.Source: §5 · p. 11
  • Across challenging spatial reasoning benchmarks, Imagine3D-LLM significantly outperforms prior approaches based on explicit pixel-level geometry or geometry-foundation features.Source: §5 · p. 11
  • The reconstruction objective propagates beyond summary tokens, reshaping image features into more spatially aligned and semantically structured representations.Source: §5 · p. 11

A Additional Implementation Details

The implementation uses multi-view inputs from ScanNet-based datasets and SPAR, a frozen compact-Gaussian teacher, and specified training and architecture settings. The accompanying analysis tracks cross-view correspondence during training.

A Additional Implementation Details

  • ScanNet-based datasets use 32 uniformly sampled frames per scene, while SPAR supplies 2, 3, or 32 views per scene; SPAR-7M is mixed into training.Inputs are 384 × 384; the SPAR subset contains 104K samples.Source: A Additional Implementation Details · p. 17
  • The frozen compact-Gaussian teacher uses 2,592 tokens and is used only during training; its framework can be swapped for other token-level feed-forward 3DGS systems.The implementation builds on ZipSplat’s open-source codebase.Source: A Additional Implementation Details · p. 17
  • Training uses AdamW with global batch size 64, peak learning rates of 1e−5 for the LLM and 2e−6 for the vision encoder, and reconstruction and distillation weights of 3.0 and 1.0.The Gaussian decoder is a three-layer MLP that specifies 32 Gaussians per summary token; spherical harmonics of degree 2 model view-dependent color.Source: A Additional Implementation Details · p. 17
  • Compared systems encode scenes as point clouds, object-centric representations, lifted 3D voxel grids, or multi-view video sequences.The passage lists the compared methods associated with each encoding approach.Source: A Additional Implementation Details · p. 17
  • Correspondence scores for image-token representations in LLM layers 11–18 rise throughout training, with pronounced gains near Gaussian decoding layer 14.The scores are reported on SQA3D test samples, with shaded regions showing variance across samples.Source: Figure 4 · p. 18

B Additional Correspondence Analysis

Cross-view correspondence improves in middle LLM layers during training, while visualizations show image tokens attending to matching object regions and Gaussian summary tokens clustering common objects.

B Additional Correspondence Analysis

  • Cross-view correspondence scores improve substantially during training in middle LLM layers 11–18, surrounding the Gaussian decoding layer; the authors attribute propagation to the compact bottleneck aggregating shared 3D regions across views.Higher scores measure stronger feature consistency for image tokens describing the same 3D region, and the improvement emerges without direct correspondence optimization.Source: B Additional Correspondence Analysis · p. 18

C.1 Additional visualization of attention between images

  • Image tokens reliably focus on matching regions of the same object across different views in the attention visualizations.The passage describes this pattern across diverse scenes.Source: §C.1 · p. 18

C.2 Additional visualization of clustered Gaussian

  • Common objects cluster into similar Gaussian summary tokens without explicit supervision in the attention visualizations.Source: §C.2 · p. 18

C.3 Additional quantitative evaluation of semantic feature structuring

An unsupervised segmentation probe evaluates whether reconstruction training structures image features semantically. The model’s image features outperform the baseline and approach the DINOv3 reference on mIoU.

C.3 Additional quantitative evaluation of semantic feature structuring

  • The probe follows PCA and cross-view correspondence analyses, then clusters image features jointly across 32 SQA3D views and measures mIoU against semantic masks.K-means uses the number of NYU40 classes present in each scene; clusters receive labels by majority vote before evaluation.Source: §C.3 · pp. 18–19
  • 35.43 mIoU for the model’s image features exceeds the baseline’s 26.54 and closes much of the gap to DINOv3’s 37.62 under the probe.The table caption identifies this as an unsupervised semantic-segmentation probe using K-means clusters and ScanNet labels.Source: §C.3 · p. 19, Table 9 · p. 19
  • The image features are not directly supervised with semantic labels; the reported structuring emerges as a byproduct of reconstruction training.Source: §C.3 · p. 19

C.4 Additional quantitative evaluation of reconstruction quality

Imagine3D-LLM's reconstructions are somewhat below the compact Gaussian teacher's quality, but reconstruction serves as an inductive bias for scene understanding rather than a high-fidelity rendering objective.

C.4 Additional quantitative evaluation of reconstruction quality

  • Reconstruction quality on SQA3D test scenes is somewhat below the compact Gaussian teacher's because the LLM is jointly optimized for language and reconstruction, trained for fewer iterations, and not designed for rendering fidelity.The comparison is reported in Tab. 10.Source: §C.4 · p. 19
  • The reconstruction objective is intended to teach scene structure, not to serve as a novel-view-synthesis system.Improving 3D reasoning, rather than high-fidelity rendering, is the primary goal.Source: §C.4 · p. 19
  • Rendering fidelity is not the primary design target because the model's main objective is improving 3D reasoning.The authors frame reconstruction quality as secondary to the model's reasoning goal.Source: §C.4 · p. 19

C.5 Additional analysis between Gaussian summary token and multi-view images

Gaussian summary tokens aggregate evidence about the same objects across views, producing object-centric representations without explicit clustering supervision. The image features also show stronger cross-view correspondences and more consistent object-level structure.

C.5 Additional analysis between Gaussian summary token and multi-view images

  • Each Gaussian summary token attends to regions of the same object across multiple views, as limited tokens aggregate repeated object occurrences into a shared representation.This object-centric behavior emerges without explicit clustering supervision.Source: §C.5 · p. 19, Figure 6 · p. 21, Figure 7 · p. 22
  • The model's image features exhibit stronger cross-view correspondences than the baseline, with corresponding objects such as chairs and desks encoded in consistent PCA colors.The PCA visualizations also show cleaner, more semantically structured features.Source: Figure 5 · p. 20

C.6 Computation cost analysis.

The compact Gaussian representation keeps reconstruction training and inference costs manageable relative to pixel-aligned estimation and external-feature-fusion alternatives. Its visualizations show coarse but spatially coherent layouts, while related methods differ in their use of external features, per-scene optimization, or output format.

C.6 Computation cost analysis.

  • 40 GB to 44 GB peak per-GPU VRAM is a 10% training increase; Imagine3D-LLM decodes about 83K Gaussians, less than 1.8% of the pixel-aligned count.The compact budget keeps rasterization, the teacher pass, and gradient buffers comparatively lightweight; the teacher is discarded at inference.Source: §C.6 · p. 23, Table 11 · p. 23
  • 20.43 GB and 176.9 ms per sample are below VLM3R-7B's 25.29 GB and 344 ms, though above the baseline's 17.09 GB and 112.1 ms.The comparison is on the SPAR evaluation set; VLM3R-7B runs external CUT3R on input frames, while Imagine3D-LLM uses no external 3D model at inference.Source: §C.6 · p. 23

C.7 Visualization of estimated Gaussians.

  • With M = 2592 summary tokens, rendered views recover overall scene layout and rough placement of major objects, trading pixel-level fidelity for compact object-level abstraction.The visualization is described as coarse rather than photorealistic.Source: §C.7 · p. 24, Figure 8 · p. 24

D Additional Related Works

  • Related approaches differ in representation and setting: SplatTalk lifts frozen 3DGS features, LangSplat/FMGS/M3 optimize per scene, and SpatialLM outputs fixed-vocabulary boxes.Imagine3D-LLM instead targets feed-forward, free-form natural-language understanding of novel scenes.Source: D Additional Related Works · p. 25
  • Imagine3D-LLM requires summary tokens to reconstruct scenes, unlike 3DThinker’s alignment to frozen VGGT features, and outperforms 3DThinker-7B by 5.2 points on SPAR-Bench overall.The reported margins are larger on the medium- and high-level reasoning splits.Source: D Additional Related Works · p. 25

E Limitations

Training efficiency remains a limitation, and the experiments cover indoor scenes with a fixed representation budget that may not suffice for larger, more complex outdoor scenes.

E Limitations

  • Teacher-free training can learn Gaussian summary tokens, but requires more steps to reach comparable downstream performance.The pretrained compact Gaussian teacher accelerates convergence but is not necessary to the framework.Source: E Limitations · p. 25
  • Training is limited to indoor scenes, and the fixed Gaussian budget could be insufficient for larger, more complex outdoor scenes.The authors identify broader indoor/outdoor coverage and dynamically adapting token count as future directions.Source: E Limitations · p. 25

Related papers

Loading 2609.38177v1…