PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Generalization Capacities of MLLMs for Spatial Intelligence

Gongjie Zhang, Wenhao Li, Quanhao Qian, Jiuniu Wang, Deli Zhao, Shijian Lu, Ran Xu

arXiv:2603.06704v1cs.CVcs.LG

TL;DR

RGB-only MLLMs can confuse scene geometry with camera perspective, creating an ambiguity that undermines cross-camera generalization. The paper introduces a Camera-Aware MLLM framework combining camera-conditioned tokens, geometric augmentation, and distilled 3D priors; experiments report robust performance where camera-agnostic baselines fail on unseen cameras.

  • Problem

    Omitting camera intrinsics creates an irresolvable geometric ambiguity that causes camera-agnostic MLLMs to fail under changed image geometry and camera conditions.

  • Method

    The Camera-Aware MLLM framework injects camera parameters into visual-token representations, varies them through geometric augmentation, and distills priors from a 3D vision foundation model.

  • Results

    Camera-aware models maintain robust cross-camera performance, whereas camera-agnostic baselines fail catastrophically on out-of-distribution cameras.

  • Takeaways & Limitations

    Explicit camera awareness is crucial for reliable and generalizable spatial intelligence in MLLMs.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) that directly process RGB inputs for tasks like 3D localization and navigation have shown remarkable potential. However, we argue that these RGB-only approaches are fundamentally flawed in their ability to generalize across cameras. By ignoring camera parameters, they entangle an object's physical properties with the camera's perspective, creating an irresolvable ambiguity. We show this leads MLLMs to overfit to the training camera distribution, rather than learning true and generalizable 3D geometric principles. To address this, we propose Camera-Aware MLLM framework for spatial MLLMs. It learns generalizable spatial reasoning by: (i) injecting camera intrinsics via a dense embedding that conditions each visual token; (ii) introducing a camera-aware data augmentation strategy that synthetically varies camera parameters, forcing the model to disentangle camera properties from scene content; and (iii) distilling geometric priors from a 3D vision foundation model. Extensive experiments demonstrate that camera-aware MLLMs substantially outperform their naive counterparts, particularly in cross-camera generalization tests on spatially-grounded tasks, indicating that camera-awareness is not only beneficial but also a prerequisite for robust and generalizable spatial intelligence in MLLMs.

1 INTRODUCTION

RGB-only MLLMs face an inherent geometric ambiguity when camera intrinsics are omitted, limiting cross-camera generalization. The Camera-Aware MLLM framework addresses this with camera-conditioned visual tokens, camera-aware augmentation, and distilled geometric priors, and experiments report robust cross-camera performance.

  • Problem: RGB-only spatial reasoning is ambiguous because different combinations of object size, depth, and camera parameters can produce the same image.The ambiguity prevents reliable 3D interpretation when intrinsics are unknown.
  • Method: The Camera-Aware MLLM framework conditions every visual token on camera rays derived from intrinsic parameters.This explicitly connects image pixels to 3D space.
  • Method: Camera-aware augmentation synthetically varies intrinsics and applies corresponding geometric transformations to separate camera properties from scene content.The strategy is designed to expose the model to diverse camera configurations during training.
  • Method: The framework distills geometric priors from a pretrained 3D vision foundation model trained on millions of RGB-depth pairs across diverse cameras.The resulting inference pipeline remains RGB-only.
  • Results: Experiments find that camera-agnostic baselines fail catastrophically on out-of-distribution cameras, whereas camera-aware models maintain robust performance.The reported evaluation targets cross-camera generalization in spatial MLLMs.
  • Conclusion: The paper presents camera-awareness as a prerequisite for generalizable spatial intelligence in MLLMs.Its stated contribution combines analysis of the ambiguity, a camera-aware framework, and experiments supporting this conclusion.

2 RELATED WORK

MLLM spatial research has progressed from explicit 3D representations toward scalable RGB-only processing, while monocular depth work shows that camera intrinsics are central to cross-camera generalization. This paper positions camera-aware spatial reasoning as an extension of that insight.

  • MLLMs: MLLMs support vision-language tasks by aligning visual encoders with large language models.Applications include object grounding and image captioning.
  • MLLMs for Spatial Intelligence: Spatial MLLM research includes both point-cloud methods and RGB-only approaches motivated by the scarcity of large-scale 3D data.RGB-only models process images or videos directly for spatial concepts and offer greater data scalability.
  • MLLMs for Spatial Intelligence: Mixed-source training can reduce ScanNet validation performance, suggesting that conflicting camera-source geometry harms generalization.The cited table caption frames the result as a robustness and generalization failure.
  • Evaluation Notation: ScanNet-val x0.8 denotes ScanNet validation images resized by a factor of 0.8.This is an evaluation preprocessing notation.
  • Monocular Metric Depth Estimation: Monocular metric depth estimation addresses single-image metric depth and has historically struggled with scale ambiguity and cross-camera generalization.Metric3D and UniDepth improve generalization by canonicalizing inputs or conditioning networks on camera intrinsics.

3 BRITTLENESS OF CAMERA-AGNOSTIC MLLMS IN SPATIAL REASONING

Camera-agnostic MLLMs struggle with spatially grounded reasoning because omitted camera intrinsics create geometric ambiguity and camera-specific shortcuts. Empirical tests show failures on mixed-camera data and simple image transformations, motivating explicit camera awareness for robust generalization.

  • 3.1 THE CHALLENGE OF SPATIALLY-GROUNDED TASKS: Spatially-grounded tasks require quantitative 3D understanding anchored to a coordinate frame, unlike qualitative relational tasks.
  • 3.2 EMPIRICAL EVIDENCE OF GENERALIZATION FAILURE: Mixed-source training reduces ScanNet validation performance because distinct camera-intrinsic distributions provide conflicting geometric signals.The analysis describes these distributions as clustered and multimodal across datasets, preventing a coherent assumed focal length.
  • 3.2 EMPIRICAL EVIDENCE OF GENERALIZATION FAILURE: Camera-agnostic MLLMs fail under simple resizing, producing systematic shifts and severely offset 3D localization predictions.The failure is attributed to overfitting to the resolution and camera geometry present during training.
  • 3.3 ANALYSIS: THE UNRESOLVED 3D GEOMETRIC AMBIGUITY: The pinhole model creates focal-depth and size-depth ambiguities because different combinations of focal length, object size, and depth can yield identical projections.Without intrinsic parameters, a model cannot distinguish camera properties from scene properties; deviations from an assumed focal length corrupt depth and scale estimates.
  • 3.3 ANALYSIS: THE UNRESOLVED 3D GEOMETRIC AMBIGUITY: Resizing by factor s scales focal lengths and principal-point coordinates, so models trained at one camera geometry can infer depth errors inversely related to s.This mechanism explains both the performance collapse under resizing and the systematic localization offsets.
  • 3.3 ANALYSIS: THE UNRESOLVED 3D GEOMETRIC AMBIGUITY: The failures stem from missing camera intrinsics rather than model scale or architecture, making explicit camera awareness necessary for generalizable spatial intelligence.

4 CAMERA-AWARE MLLM FRAMEWORK

The Camera-Aware MLLM framework resolves RGB-only geometric ambiguity by conditioning visual tokens on camera geometry and enriching them with 3D priors. It also expands camera diversity during training and supports inputs with unknown intrinsics.

  • Camera-Aware Architecture: The Geometry-Aware Visual Encoder enriches visual tokens with appearance, camera-intrinsic context, and geometric information before multimodal reasoning.Camera rays are computed for token locations and fused with visual features through a dense camera embedding.
  • Camera Ray Embedding: Dense camera ray embeddings condition each visual token on its line of sight and global focal lengths derived from camera intrinsics.The resulting embedding is fused with visual features by element-wise addition.
  • Geometric Augmentation: Camera-aware geometric augmentation synthetically varies intrinsics through image scaling and principal-point shifting while updating images and parameters consistently.These transformations expose the model to altered camera geometries during training.
  • Geometric Prior Distillation: Geometric prior distillation uses a frozen UniDepth v2 model to derive dense 3D information from RGB-depth knowledge and add it to visual features.UniDepth v2 was pretrained on over 10M RGB-depth pairs across diverse cameras.
  • Geometric Prior Distillation: Geometric prior distillation also enables training and evaluation on images without known camera parameters because UniDepth estimates intrinsics from images.This extends the framework to large 2D datasets lacking camera-intrinsic annotations.

5 EXPERIMENTS

Experiments evaluate cross-camera robustness, standard spatial reasoning, and component contributions. The camera-aware model remains robust under altered intrinsics, performs strongly across benchmarks, and benefits most from combining camera awareness with diverse training data.

  • Cross-Camera Evaluation: Cross-camera evaluations cover single-frame 3D detection, video 3D detection, and video 3D visual grounding against task-finetuned baselines.The experiments compare against Qwen2.5-VL and VG-LLM.
  • Cross-Camera Evaluation: Under resized camera parameters, the camera-aware model maintains consistent accuracy while camera-agnostic baselines degrade substantially.The result is reported across all spatially grounded tasks and is attributed to reduced overfitting to training-camera geometry.
  • Cross-Camera Evaluation: The framework remains robust when camera intrinsics are unavailable, consistently and significantly outperforming camera-agnostic baselines in that setting.Geometric prior distillation enables on-the-fly intrinsic estimation.
  • Benchmark Evaluation: The model achieves the highest performance on SPAR-Bench and state-of-the-art performance on general RGB-only spatial reasoning benchmarks.These results cover settings with precise camera parameters and settings where intrinsics are not provided.
  • Ablation Study: Ablations show that camera-aware architecture and diverse training data are individually insufficient, whereas their combination produces substantial generalization gains.The ablation evaluates single-frame 3D detection on a synthetically zoomed-in test set.
  • Qualitative Evaluation: Qualitative comparisons show a significant localization-accuracy gap on zoomed-in ScanNet images and in-the-wild TUM-RGBD images.The compared camera-agnostic baselines use identical training data.

6 CONCLUSION

The paper concludes that camera-agnostic MLLMs are fundamentally limited by geometric ambiguity and overfit to training-camera properties. Its camera-aware framework addresses this through camera parameters, geometric augmentation, and 3D priors.

  • Conclusion: Camera-agnostic MLLMs fail to generalize because ignoring camera intrinsics creates geometric ambiguity and encourages overfitting to training-camera properties.The conclusion frames this as a limitation of pixel-only spatial reasoning.
  • Conclusion: The proposed framework combines camera-parameter injection, geometric augmentation, and 3D-prior distillation to support generalizable spatial reasoning.The paper argues that robust 3D understanding requires reasoning about the geometric principles underlying pixels.

A DETAILED PROOF OF GEOMETRIC AMBIGUITY

The proof shows that camera intrinsics determine the pixel-to-ray mapping, creating ambiguities between camera parameters, object size, and depth in monocular RGB images. These ambiguities persist beyond the idealized fronto-parallel case as a metric identifiability limitation.

  • Camera projection model: Under the ideal pinhole-camera model, a 3D world point is transformed into camera coordinates before projection, with rotation R, translation t, and projective scale s.The analysis assumes lens distortion and skew are ignored because including them does not change the conclusions.
  • Coupled-scaling invariance: Projected height follows hproj = fyH/Z, producing focal–depth and size–depth tradeoffs that preserve the same image measurement.Increasing focal length while proportionally increasing depth, or scaling physical size and depth together, leaves projected height unchanged.
  • Coupled-scaling invariance: A nearby small object can therefore be observationally indistinguishable from a larger distant object, while depth changes can resemble focal-length changes.The exact pixel-wise invariance is derived for fronto-parallel segments, but metric depth and size remain unidentifiable in general monocular RGB with unknown intrinsics.
  • Image resampling as intrinsic transformation: Rescaling pixel coordinates rescales the intrinsic matrix, while cropping shifts the principal point after rescaling.The transformation is represented by K′ = diag(σx, σy, 1) K, with crop offsets measured after rescaling.
  • Per-pixel rays and the role of intrinsics: The back-projected ray field depends on focal lengths and principal-point coordinates, so unknown intrinsics allow similar images to represent different ray bundles.This ambiguity encourages camera-specific shortcuts rather than geometry that transfers across cameras.

B IMPLEMENTATION DETAILS

The implementation builds on VG-LLM-4B, adds distilled 3D geometric priors, and trains on mixed datasets covering spatially grounded and spatial reasoning tasks. The paper also specifies category-conditioned prompts for 3D video object detection.

  • Model and training: The model adopts VG-LLM-4B as its baseline and distills geometric priors from UniDepth v2 to enhance geometric understanding.VG-LLM-4B combines Qwen2.5-VL-3B with VGGT-1B for 3D geometry feature extraction.
  • Training data: Training uses mixed datasets spanning indoor 3D scenes, object-centric data, referring expressions, video, and spatial reasoning tasks.The listed sources include ScanNet, ARKitScenes, Matterport3D, 3RScan, SUN RGB-D, Objectron, ScanRefer, Scan2Cap, and LLaVA-Video-178k.
  • 3D video object detection: The 3D video object detection prompt requests category-specific 3D bounding boxes in a JSON list.Each box contains center coordinates, dimensions, and yaw, pitch, and roll values.
  • 3D video object detection: Categories are explicitly supplied to reduce hallucinated detections of objects absent from the scene.The paper gives this category conditioning as the reason for the prompt design.

C LLM USAGE DISCLOSURE

The authors disclose using LLMs as assistive tools for language polishing, writing, and coding-related support. They state that generated content was reviewed against the authors’ ideas and results and was not autonomously produced.

  • Use of LLMs: LLMs assisted with manuscript language, readability, writing tasks, and coding-related aspects.The disclosure characterizes LLMs as general-purpose assistive tools during preparation.
  • Author verification: The authors state that LLM-generated content was based on their ideas and results, reviewed and verified, and not used to autonomously complete the paper.They also state that the content accurately reflects the intended meaning.
Loading 2603.06704v1…