PaperlayerView exact revision on arXiv

Source-linked AI summary

3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark

Wufei Ma, Haoyu Chen, Guofeng Zhang, Yu-Cheng Chou, Jieneng Chen, Celso M de Melo, Alan Yuille

arXiv:2412.07825v4cs.CV

TL;DR

Existing LMMs have been less studied for 3D spatial reasoning on diverse natural images, despite its relevance to understanding 3D scenes and downstream applications. 3DSRBench addresses this gap with a manually annotated, robust benchmark and viewpoint-paired evaluation, revealing broad weaknesses and degraded performance on uncommon viewpoints.

  • Problem

    LMMs lack comprehensive evaluation of 3D spatial reasoning on diverse natural images, limiting evidence about their 3D awareness and downstream relevance.

  • Method

    3DSRBench manually annotates 2,100 natural-image VQA pairs and 672 paired synthetic-viewpoint pairs, using balanced data and robust evaluation strategies.

  • Results

    Benchmarking shows limitations across 3D awareness dimensions and degraded generalization from common to uncommon camera 6D viewpoints.

  • Takeaways & Limitations

    The benchmark provides diagnostic findings for improving LMM visual encoders and reasoning abilities in applications involving 3D scenes.

  • Takeaways & Limitations

    Current LMMs use implicit 3D scene representations and face severe challenges on less-represented uncommon camera viewpoints.

Abstract

from arXiv · show

3D spatial reasoning is the ability to analyze and interpret the positions, orientations, and spatial relationships of objects within the 3D space. This allows models to develop a comprehensive understanding of the 3D scene, enabling their applicability to a broader range of areas, such as autonomous navigation, robotics, and AR/VR. While large multi-modal models (LMMs) have achieved remarkable progress in a wide range of image and video understanding tasks, their capabilities to perform 3D spatial reasoning on diverse natural images are less studied. In this work we present the first comprehensive 3D spatial reasoning benchmark, 3DSRBench, with 2,772 manually annotated visual question-answer pairs across 12 question types. We conduct robust and thorough evaluation of 3D spatial reasoning abilities by balancing data distribution and adopting a novel FlipEval strategy. To further study the robustness of 3D spatial reasoning w.r.t. camera 3D viewpoints, our 3DSRBench includes two subsets with 3D spatial reasoning questions on paired images with common and uncommon viewpoints. We benchmark a wide range of open-sourced and proprietary LMMs, uncovering their limitations in various aspects of 3D awareness, such as height, orientation, location, and multi-object reasoning, as well as their degraded performance on images from uncommon 6D viewpoints. Our 3DSRBench provide valuable findings and insights about future development of LMMs with strong spatial reasoning abilities. Our project page is available at https://3dsrbench.github.io/.

1. Introduction

3D spatial reasoning remains less studied than other LMM abilities, despite its importance for understanding object relationships and downstream interaction. 3DSRBench addresses this gap with manually annotated natural-image questions, balanced evaluation, paired viewpoints, and broad model analysis.

  • Motivation: Spatial reasoning supports high-level reasoning and interaction, but prior benchmarks often captured only viewer-relative 2D relationships from object bounding boxes.These benchmarks neglected relationships in 3D world space or from an object’s perspective.
  • Motivation: Synthetic spatial-reasoning datasets have domain gaps, while real annotated datasets remain restricted to domains such as indoor rooms and self-driving scenes.Rule-based VQA generation also limits diversity and the range of represented entities.
  • Benchmark contribution: 3DSRBench contains 2,772 questions across 12 types, manually annotated on diverse natural images and open-vocabulary entities.The benchmark covers rigid objects, humans, animals, and implicit concepts such as logos and billboard arrows.
  • Benchmark contribution: The benchmark covers height, location, orientation, and multi-object reasoning, requiring combinations of 3D locations, ground planes, camera extrinsics, and object poses.These categories target distinct forms of 3D awareness and reasoning.
  • Robust evaluation: 3DSRBench evaluates viewpoint robustness using paired common and uncommon camera 6D viewpoints, which are especially relevant to embodied AI and robotics.Uncommon viewpoints can differ substantially from typical eye-level image perspectives.
  • Robust evaluation: Its design balances data distributions, avoids shortcuts, and uses FlipEval to reduce left/right biases in paired visual question-answer evaluation.The authors also benchmark diverse LMMs, visual encoders, language-model scaling, and failure modes.

2. Related Works

Related work moves from viewer-relative 2D spatial relationships toward 3D awareness and reasoning, but existing datasets remain limited by synthetic domain gaps, narrow object categories, or proxy tasks.

  • Spatial reasoning: Early spatial-reasoning datasets focused on viewer-relative left/right relationships derivable from 2D object bounding boxes.They did not directly capture 3D world-space or object-perspective relationships.
  • Spatial reasoning: Synthetic datasets provide 3D ground truth but exhibit a domain gap from natural images, leaving generalization to real-image settings unclear.Later datasets improve image quality but remain constrained by limited rigid-object categories and rule-based VQA generation.
  • 3D awareness: Prior studies of visual-foundation-model 3D awareness commonly use proxy tasks such as part correspondence, pose estimation, and linear probing.3DSRBench instead evaluates LMMs directly on fundamental 3D spatial relationship questions.
  • 3D awareness: These 3D-awareness findings are intended to inform downstream areas involving 3D scenes, including automatic navigation and robotic manipulation.The connection is presented as a future research direction based on better 3D spatial reasoning evaluation.

3. 3DSRBench

3DSRBench combines diverse 3D question types with balanced, shortcut-resistant data and robust evaluation procedures. Its real and synthetic splits test both general 3D reasoning and sensitivity to camera viewpoints.

  • Design: The benchmark uses open-vocabulary entities, avoids shortcuts and trivial comparisons, and balances answer distributions with complementary image pairs.These choices aim to make evaluation more robust across diverse real-world objects.
  • Question types: 3DSRBench is organized around four question types: height, location, orientation, and multi-object reasoning.The types target progressively varied combinations of 3D awareness and spatial relationships.
  • Question types: Location questions combine 2D grounding with depth to assess nearness, camera distance, and above-underneath relationships.They include three subtypes involving relative distance, camera proximity, and vertical arrangement.
  • Question types: Orientation and multi-object questions require reasoning about 3D object orientations, relative sides, facing directions, and distances among several objects.These tasks extend beyond viewer-relative 2D relationships and simpler single-object spatial concepts.
  • Benchmark splits: The real split contains 2,100 MS-COCO questions, while two synthetic splits contain 672 HSSD questions paired across common and uncommon camera 6D viewpoints.Common viewpoints approximate typical human photography; uncommon viewpoints include large pitch or roll and unusual camera heights.
  • Evaluation: CircularEval tests every multiple-choice question under multiple answer-orderings and requires all passes to be correct.This reduces ordering bias and the influence of random guessing.
  • Evaluation: FlipEval horizontally flips paired images so invariant answers remain stable while left/right answers change, reducing systematic spatial biases.It also reduces random-guessing effects and enriches the image distribution.

4. Experiments

The experiments benchmark LMMs on 3DSRBench, examine visual encoder and language-model scaling, and test robustness to uncommon camera viewpoints. Results show limited 3D spatial reasoning, benefits from some encoder and model scaling choices, and substantial degradation under uncommon viewpoints.

  • Experimental setup: The study benchmarks open-source and proprietary LMMs on real-image and synthetic splits, including common and uncommon camera viewpoints.The evaluation uses 3DSRBench-real and paired synthetic splits to assess standard reasoning and viewpoint robustness.
  • Experimental setup: The evaluation combines balanced and augmented questions with FlipEval, exact or LLM-assisted answer matching, and CircularEval across choice orderings.Rule-based augmentation produces 5,250 real-image and 1,692 synthetic-image questions.
  • 3D spatial reasoning results: State-of-the-art LMMs achieve limited 3D spatial reasoning performance and remain far behind human-level performance.The benchmark comparison reports this pattern across open-source and proprietary models.
  • 3D spatial reasoning results: Mixed visual encoders improve overall reasoning, with DINOv2 helping orientation and multi-object questions and MAE or SAM improving height questions.Adding a spatial vision aggregator raises the mixed-encoder result from 37.2% to 37.8%.
  • 3D spatial reasoning results: Larger language models improve 3D spatial reasoning, but models above 70B parameters are computationally inefficient for common robotics and embodied-AI systems.A 72B language model with a 6B vision encoder still falls more than 40% below human-level performance.
  • Robustness to uncommon camera viewpoints: All evaluated LMMs show significantly degraded accuracy when generalizing from common to uncommon viewpoints, including drops of 13.5% for GPT-4o and 32.2% for Gemini-1.5-Pro.These paired images depict the same scene from different camera viewpoints.

5. Conclusions

The paper introduces 3DSRBench to evaluate LMMs’ 3D spatial reasoning across diverse question types, natural and synthetic images, and camera viewpoints. It benchmarks model capabilities, studies visual encoder and language-model scaling, and reports findings for developing robust 3D reasoning systems.

  • 3DSRBench contains 2,772 manually annotated visual question-answer pairs covering height, location, orientation, and multi-object reasoning.The benchmark includes 2,100 natural-image pairs and 672 synthetic multi-view pairs.
  • The benchmark evaluates open-sourced and proprietary LMMs on diverse 3D spatial reasoning capabilities and robustness to uncommon camera viewpoints.
  • The study examines how visual encoder designs and language-model scaling affect LMM 3D spatial reasoning.
  • Experimental results provide findings and insights for developing LMMs and selecting models for downstream applications requiring robust 3D spatial reasoning.

A. 3DSRBench Data Card

The data card describes a manually annotated benchmark built through a two-stage quality-control process, with natural and synthetic image subsets and documented ethical and licensing practices.

  • Ten annotators created 2,772 unique visual question-answer pairs across 12 question types using a two-stage annotation pipeline.Low-quality or distribution-imbalanced samples were rejected and replaced when necessary.
  • The dataset includes 2,100 questions from MS-COCO natural images and 674 questions from HSSD-rendered synthetic images.
  • Testing augmentation and FlipEval expand the benchmark to 5,250 real-image questions and 1,692 synthetic-image questions.The augmented questions are further evaluated with CircularEval.
  • The study received IRB approval, described risks to annotators, and paid annotators fairly according to institutional requirements.
  • The dataset is released under the CC-BY-4.0 license.

B. Baseline Models

The baseline models span proprietary and open-sourced LMMs, including LLaVA-v1.5 and Cambrian-1, alongside controlled experiments with alternative visual encoders and connectors.

  • The study evaluates proprietary LMMs including QWen-VL, Claude, Gemini, and GPT-4o, with additional Gemini-2.0-Flash variants using 3D bounding boxes or thinking-model evaluation.
  • LLaVA-v1.5-7B combines a Vicuna-v1.5 language model with a CLIP-ViT-L vision encoder, an MLP connector, and increased image resolution.
  • Cambrian-1 uses vision-centric designs including an advanced connector, spatial vision aggregator, and visual-instruction tuning data.
  • The encoder study extends LLaVA-v1.5 with mixed encoders such as DINOv2, MAE, and SAM, and compares visual connector designs.

C. Qualitative Examples of 3DSRBench

The supplementary material presents two example questions for each of the benchmark’s 12 question types, organized across height, location, orientation, and multi-object reasoning.

  • Two example questions are provided for each of the 12 question types.
  • The examples are distributed across height and location, orientation, and multi-object reasoning figures.

D. Qualitative Examples of Common and Uncommon Viewpoints

3DSRBench-synthetic renders the same scene and objects from common and uncommon viewpoints to provide qualitative multi-view examples.

  • D. Qualitative Examples of Common and Uncommon Viewpoints: 3DSRBench-synthetic uses multi-view images that render the same scene and objects from different viewpoints.The examples compare common and uncommon viewpoints.
  • D. Qualitative Examples of Common and Uncommon Viewpoints: The qualitative examples place common viewpoints on the left and uncommon viewpoints on the right.
  • D. Qualitative Examples of Common and Uncommon Viewpoints: These paired renderings support comparison of 3D spatial reasoning across camera viewpoints.

E. Failure Cases

The paper illustrates benchmark question types and documents failure cases in which GPT-4o and Gemini 2.0 Flash thinking do not reliably perform multi-dimensional 3D spatial reasoning.

  • E. Failure Cases: Figures 6–8 provide example questions covering height, location, orientation, and multi-object reasoning.These examples span the benchmark’s illustrated question categories.
  • E. Failure Cases: Figure 9 shows the same scenes and objects rendered from common and uncommon viewpoints.Common viewpoints appear on the left and uncommon viewpoints on the right.
  • E. Failure Cases: GPT-4o sometimes relies on visual cues and, in other examples, fails while attempting 3D spatial reasoning from the scene.These failures are presented in Figure 10.
  • E. Failure Cases: Gemini 2.0 Flash thinking decomposes the question and derives visual cues, but still fails when reasoning over multiple 3D information such as depth and orientation.This failure case is presented in Figure 11.
Loading 2412.07825v4…