PaperlayerView exact revision on arXiv

Source-linked AI summary

N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models

Yuxin Wang, Lei Ke, Boqiang Zhang, Tianyuan Qu, Hanxun Yu, Zhenpeng Huang, Meng Yu, Dan Xu, Dong Yu

arXiv:2512.16561v1cs.CV

TL;DR

Current VLMs lack intrinsic 3D object perception for understanding spatial relationships and depth cues. N3D-VLM combines native 3D localization with explicit 3D-aware reasoning and scalable lifted training data, achieving strong performance in both 3D grounding and spatial reasoning.

  • Problem

    Current VLMs largely lack the object-level 3D perception needed to reason about spatial configurations and physical environments.

  • Method

    N3D-VLM unifies native 3D detection and grounding with chain-of-thought spatial reasoning, supported by data that lifts 2D annotations into 3D.

  • Results

    N3D-VLM consistently surpasses existing methods in both 3D object localization and spatial reasoning, with the highest accuracy across three spatial QA benchmarks.

  • Takeaways & Limitations

    Explicit 3D grounding provides a structured basis for accurate localization and interpretable spatial reasoning within one vision-language model.

Abstract

from arXiv · show

While current multimodal models can answer questions based on 2D images, they lack intrinsic 3D object perception, limiting their ability to comprehend spatial relationships and depth cues in 3D scenes. In this work, we propose N3D-VLM, a novel unified framework that seamlessly integrates native 3D object perception with 3D-aware visual reasoning, enabling both precise 3D grounding and interpretable spatial understanding. Unlike conventional end-to-end models that directly predict answers from RGB/RGB-D inputs, our approach equips the model with native 3D object perception capabilities, enabling it to directly localize objects in 3D space based on textual descriptions. Building upon accurate 3D object localization, the model further performs explicit reasoning in 3D, achieving more interpretable and structured spatial understanding. To support robust training for these capabilities, we develop a scalable data construction pipeline that leverages depth estimation to lift large-scale 2D annotations into 3D space, significantly increasing the diversity and coverage for 3D object grounding data, yielding over six times larger than the largest existing single-image 3D detection dataset. Moreover, the pipeline generates spatial question-answering datasets that target chain-of-thought (CoT) reasoning in 3D, facilitating joint training for both 3D object localization and 3D spatial reasoning. Experimental results demonstrate that our unified framework not only achieves state-of-the-art performance on 3D grounding tasks, but also consistently surpasses existing methods in 3D spatial reasoning in vision-language model.

1. Introduction

Current VLMs largely lack the 3D object perception needed for reliable spatial reasoning, motivating N3D-VLM’s unified native 3D grounding and explicit reasoning framework. Its scalable data pipeline addresses limited 3D training diversity by lifting 2D annotations into 3D and generating localization and reasoning data.

  • Motivation and approach: Current VLMs largely lack the object-level 3D perception required to infer spatial configurations and reason about physical environments.The paper frames robust 3D spatial ability as requiring perception, grounding, and reasoning from visual inputs.
  • Motivation and approach: N3D-VLM unifies 3D detection, grounding, and chain-of-thought reasoning, using explicit 3D bounding boxes to support interpretable spatial understanding.The model reasons over detected 3D objects, including distances from 3D coordinates and relative sizes from bounding-box dimensions.
  • Motivation and approach: Existing 3D datasets are limited in diversity, scale, and category coverage, whereas large-scale 2D detection datasets offer richer scene and class diversity.This mismatch creates a central data challenge for general 3D object perception.
  • Motivation and approach: The data pipeline uses depth estimation to lift large-scale 2D annotations into 3D and constructs spatial QA data for chain-of-thought supervision.It also addresses consistent scale and camera geometry with shared depth estimation and depth-aware positional encoding.
  • Results: N3D-VLM consistently surpasses existing methods on object localization and spatial reasoning metrics, achieving state-of-the-art effectiveness and generalization.The authors describe this as the framework’s overall empirical outcome across both capabilities.

2. Related Work

Related 3D VLM approaches use point clouds, video, RGB/RGB-D inputs, or auxiliary perception modules for spatial understanding and grounding. However, many rely on external tools or predefined object information rather than directly producing comprehensive 3D object localizations.

  • 3D spatial understanding: Recent approaches extend general-purpose VLMs with point clouds, video, or RGB/RGB-D inputs for 3D captioning, question answering, and spatial understanding.Examples include object-marked point-cloud views, bird’s-eye-view images, and broader spatial-question support.
  • 3D visual grounding: Existing 3D visual grounding methods often rely on external segmentation, multi-view matching, or known object positions instead of directly predicting 3D bounding boxes.The paper contrasts these dependencies with its explicit 3D object localization approach.

3. The Proposed Framework: N3D-VLM

N3D-VLM constructs 3D localization and reasoning data by lifting 2D annotations with depth, then combines explicit 3D representations with depth-aware visual encoding and staged training. Its benchmark and pipeline support detection, grounding, and interpretable spatial reasoning across diverse categories and question types.

  • 3D Data Construction: The framework lifts 2D annotations into a diverse 3D repository, then derives datasets for 3D detection, grounding, and spatial reasoning QA.The repository is used for both training and evaluation.
  • 3D Data Construction: 2D boxes, segmentation masks, and monocular depth are combined into camera-space point clouds and filtered 3D bounding boxes.Rule-based filters remove outlier points and implausible boxes caused by invalid depth values.
  • 3D Data Construction: Each 3D box is encoded as bbox(id, class, u, v, z, sx, sy, sz), combining category identity, projected center, depth, and spatial dimensions.Known camera intrinsics make the projected center and camera-space coordinates interconvertible.
  • 3D Data Construction: The pipeline generates grounding questions for uniquely identifiable objects, repeated categories, referring expressions, and rendered box references.These strategies produce diverse QA pairs for training 3D object localization.
  • 3D Data Construction: Spatial reasoning data uses predefined geometric templates and deterministic reasoning chains derived from 3D bounding boxes, such as clock-direction relations.The reasoning is rephrased with an LLM while retaining explicit geometric intermediate steps.
  • Model Architecture and Training: Depth-aware encoding back-projects RGB-D pixels into 3D, applies sinusoidal coordinate encoding, and adds the resulting embedding to image features before language-model prediction.The model is trained in two stages: 3D localization first, followed by grounding-based spatial reasoning.
  • N3D-Bench: N3D-Bench broadens spatial evaluation with 264 object categories, multi-object relations, viewpoint-shifted questions, and explicit chain-of-thought reasoning.The benchmark is constructed from the same 3D spatial data-generation pipeline.

4. Experiments

Across spatial reasoning and 3D grounding benchmarks, N3D-VLM consistently outperforms the compared baselines, while ablations support its depth-aware design and explicit grounding–reasoning decomposition.

  • 4.2. Main Results: N3D-VLM achieves the highest accuracy across all three spatial reasoning benchmarks and outperforms Qwen3-VL on numerical questions.It reaches 92.1% and 78.0% on the two numerical tasks discussed, versus Qwen3-VL’s 36.3% and 40.7%.
  • 4.2. Main Results: N3D-VLM achieves the best projected and aligned 3D bounding-box results against Qwen3-VL, demonstrating stronger 3D grounding.The comparison covers projected 2D metrics and aligned 3D bounding-box evaluation.
  • 4.3. Ablation Study on Model Design: Scaling training data from 340K to 1.7M samples substantially improves 3D detection performance.The larger-scale variant supports the effectiveness of the 3D data-generation pipeline.
  • 4.3. Ablation Study on Model Design: Adding depth input raises F1 from 9.4 to 12.8, while pixel-space center prediction outperforms direct camera-space prediction.The ablation attributes these gains to depth cues and alignment between pixel-space representations and the base model’s 2D pretraining.
  • 4.4. 3D Grounding Helps Spatial Reasoning: Providing intermediate 3D grounding improves Qwen3-VL, whereas end-to-end QA training underperforms the explicitly decomposed grounding-and-reasoning model.These experiments compare grounding-assisted inference and direct QA training under the same architecture.

5. Conclusion

N3D-VLM unifies native 3D grounding with explicit spatial reasoning, supported by scalable 3D data construction and reasoning datasets. Experiments show strong performance in both 3D grounding and spatial reasoning.

  • 5. Conclusion: N3D-VLM bridges native 3D object perception and explicit 3D-aware reasoning within one unified framework.The framework is designed to provide accurate localization and structured spatial understanding.
  • 5. Conclusion: A scalable pipeline projects 2D annotations into 3D space and creates explicit reasoning datasets to support training.The resulting data supports both 3D grounding and spatial reasoning.
  • 5. Conclusion: Extensive experiments report strong performance among existing vision-language models on both 3D grounding and 3D spatial reasoning.

Supplementary Material

The supplementary material adds experiments, a video demonstration, and a description of the N3D-Bench data distribution.

  • Supplementary Material: The supplement provides additional experiments in Section A and a video demo with more qualitative results in Section B.
  • Supplementary Material: Section C illustrates the data distribution of the proposed N3D-Bench.

A.1. 3D Grounding Comparison

Qualitative comparisons show N3D-VLM localizing objects more accurately than SpatialLM and Qwen3-VL across indoor and outdoor scenes, while SpatialReasoner lacks comprehensive 3D box output.

  • A.1. 3D Grounding Comparison: N3D-VLM more accurately localizes indoor objects than SpatialLM and Qwen3-VL in qualitative comparisons.Examples include pillows and washing machines, where baselines miss objects or produce inaccurate spatial predictions.
  • A.1. 3D Grounding Comparison: N3D-VLM outperforms Qwen3-VL in diverse outdoor scenes, while SpatialLM fails because it is limited to predefined indoor categories.
  • A.1. 3D Grounding Comparison: SpatialReasoner predicts 3D object centers but does not support explicit 3D bounding-box output, limiting comprehensive spatial perception.Its prompted center predictions were also inconsistent and often did not follow a coherent 3D coordinate format.

A.2. 3D Spatial Reasoning Comparison

N3D-VLM outperforms several vision-language baselines in qualitative 3D spatial reasoning comparisons by grounding answers in explicit 3D object locations. It also consistently surpasses Qwen2.5VL across question types on spatial reasoning benchmarks.

  • A.2. 3D Spatial Reasoning Comparison: N3D-VLM outperforms GPT-4o and Qwen3-VL-8B in qualitative examples involving viewpoint changes and visual spatial cues.The baselines fail because of incorrect viewpoint reasoning or reliance on commonsense priors rather than the observed scene.
  • A.2. 3D Spatial Reasoning Comparison: N3D-VLM outperforms SpatialRGPT and SpatialReasoner on relative-distance and depth-comparison questions by interpreting prompts and using explicit 3D boxes.SpatialRGPT gives incorrect or misunderstood answers, while SpatialReasoner may calculate absolute distances for depth-ordering questions.
  • A.2. 3D Spatial Reasoning Comparison: N3D-VLM accurately answers spatial questions by first localizing 3D bounding boxes and then reasoning over the grounded results.This grounding-based process supports reasoning about viewpoint changes, relative distances, and depth comparisons.
  • A.2. 3D Spatial Reasoning Comparison: N3D-VLM consistently outperforms Qwen2.5VL across different spatial question types on the reported benchmarks.The comparison is presented as evidence that native 3D grounding improves understanding of varied spatial questions.

A.3. Failure Cases

The reported failures reveal two limitations of N3D-VLM’s native 3D grounding: confusing reflections with objects and missing objects in dense scenes.

  • A.3. Failure Cases: N3D-VLM mistakes a duck’s reflection on water for a real object, indicating difficulty with specular reflections.The authors suggest improving reflection understanding as a direction for strengthening 3D grounding.
  • A.3. Failure Cases: N3D-VLM detects 30 jellyfish with accurate 3D bounding boxes but still misses several objects in the dense scene.The authors identify improved 3D grounding as a way to potentially boost overall spatial-understanding performance.

C.1. Distribution Summary

The supplied material describes N3D-Bench as a diverse spatial-reasoning benchmark spanning many object categories and question types, alongside qualitative and quantitative comparison materials.

  • C.1. Distribution Summary: The benchmark materials compare N3D-VLM qualitatively with SpatialLM and Qwen3-VL-8B for indoor and outdoor 3D grounding.Green boxes denote ground-truth 3D boxes and red boxes denote predictions.
  • C.1. Distribution Summary: N3D-Bench contains 2,000 questions across 11 major categories, including relative-distance and depth comparisons among three or more objects.It also varies viewpoint phrasing for relations such as left and right.
  • C.1. Distribution Summary: N3D-Bench includes 264 indoor and outdoor object classes derived from Objects365.The object-class distribution is illustrated in Fig. 10.
  • C.1. Distribution Summary: Additional qualitative comparisons evaluate N3D-VLM against GPT-4o, Qwen3-VL-8B, SpatialRGPT, and SpatialReasoner for 3D spatial reasoning.The supplied passages describe baseline failures but do not provide numerical cell values.
  • C.1. Distribution Summary: Tables 7 and 8 report detailed accuracy comparisons on N3D-Bench and SpatialRGPT-Bench, respectively.Their captions identify highest- and second-highest-accuracy highlighting, but the supplied text does not expose the table values.
Loading 2512.16561v1…