PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

Anubhav Khanal, Prabigya Acharya, Roshni Poudel, Sujan Kapali, Bigyan Bhatta, Pramish Paudel, Francois Rameau, Danda Pani Paudel

arXiv:2609.16233v1cs.CVcs.AI

TL;DR

Existing 3D benchmarks often omit rich visual cues, hierarchical scene structure, or multi-step reasoning. SceneBench combines photorealistic Gaussian Splatting scenes with hierarchical annotations and tasks for spatial and multi-step reasoning; evaluations report that current methods struggle on the benchmark.

  • Problem

    Existing benchmarks often lack persistent 3D structure, rich visual cues, hierarchical scene organization, or tasks testing multi-step reasoning.

    Source: §1 · pp. 1–2

  • Method

    SceneBench provides a hierarchical, tree-based representation of 3D scenes and benchmarks spatial and multi-step reasoning in photorealistic scenes.

    Source: §1 · p. 2

  • Results

    Evaluations show that existing methods struggle on SceneBench, with performance bottlenecks particularly evident as hierarchical complexity increases.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    SceneBench provides a realistic testbed for diagnosing model capabilities in photorealistic 3D scene understanding.

    Source: §6 · p. 8

Abstract

from arXiv · show

Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning. Progress is hindered by limitations in current benchmarks. First, 3D datasets often rely on point clouds that capture geometry but discard rich visual features like texture, text, and materials. Second, annotations treat objects in isolation while ignoring real-world hierarchical organization (scenes, rooms, functional areas, object groups). Third, evaluation tasks focus narrowly on basic recognition rather than multi-step spatial reasoning. In this context, we introduce SceneBench, a benchmark of 966 photorealistic 3D scenes reconstructed with Gaussian Splatting and densely annotated with hierarchical semantics spanning scenes, rooms, functional areas, object groups, and individual objects. These annotations are produced through a human-in-the-loop pipeline combining vision-language models with roughly 1,500 human-hours of iterative refinement and verification, producing over 183K annotated nodes with textual descriptions and 3D bounding boxes. Building on this representation, we define three evaluation tasks: Existence-Based Questions probing object attributes, Spatial Intelligence Questions covering counting, size comparison, distance, and directional relations, and Grounded Question-Reasoning-Answer (QRA) triplets requiring multi-step reasoning across semantic levels. Experiments with state-of-the-art vision-language models show that while models perform well on basic recognition tasks (e.g., up to 85% accuracy for detection), performance drops substantially on hierarchical and compositional reasoning (e.g., down to 60% for counting), revealing limitations not captured by existing benchmarks. SceneBench provides a realistic testbed for developing and evaluating models capable of fine-grained spatial reasoning in photorealistic 3D environments.

1. Introduction

SceneBench addresses gaps in 3D benchmarks that omit visual detail, hierarchical scene organization, or multi-step reasoning. It combines photorealistic Gaussian Splatting scenes with hierarchical annotations and tasks spanning spatial and higher-level reasoning.

1. Introduction

  • Existing benchmarks often use 2D imagery or geometry-only point clouds, flatten scene semantics to object categories, and emphasize recognition or localization over multi-step reasoning.These limitations leave benchmarks short of capturing both visual detail and the organization of real-world scenes.Source: §1 · pp. 1–2
  • SceneBench is a benchmark for immersive and hierarchical 3D vision-language understanding, comprising 966 photorealistic 3D scenes with 183K text-annotated nodes arranged hierarchically and semantically spanning scenes, rooms, functional areas, object groups, and individual objects.Source: §1 · p. 2
  • The benchmark evaluates spatial reasoning and advanced multi-step reasoning alongside simpler 3D localization.Its task suite includes Existence, Spatial Intelligence, and Grounded Question-Reasoning-Answer tasks.Source: §1 · p. 2, Figure 1 · p. 2, §2 · p. 3

2. Related Work

  • Prior work includes point-cloud and egocentric-query benchmarks, multi-view and compositional tasks, cognitive-mapping evaluations, and reasoning frameworks, but lacks a combination of photorealistic 3D and densely grounded structured QA.Existing 3D Gaussian Splatting datasets are also described as largely limited to flat object structures, while explicit multi-level hierarchies in datasets remain limited to mesh or point-cloud settings.Source: §2 · p. 3

3. Dataset

SceneBench represents 3D Gaussian Splatting scenes with a tree of grounded textual descriptions and builds its annotations through a human-verified pipeline. The pipeline generates object descriptions and then organizes objects into spatial and semantic hierarchies.

3. Dataset

  • The tree-based representation uses object leaves and inner nodes for object hierarchies, pairing each inner node with a 3D-grounded textual description.An object mentioned within another node’s description is grounded as a constituent part of that inner node.Source: §3 · p. 3
  • The 966-scene corpus combines 584 real-world ScanNet++ scenes and 382 synthetic InteriorGS scenes, all represented as 3D Gaussian Splats.ScanNet++ splats come from SceneSplat; InteriorGS scenes are natively available as Gaussian Splats.Source: §3 · pp. 3–4
  • A human-in-the-loop pipeline, refined and verified across approximately 1,500 human-hours, produces detailed descriptions and hierarchical scene structure.It comprises object-description generation and object-hierarchy generation.Source: §3 · p. 4, Figure 2 · p. 4
  • For object descriptions, Gemini 3.0 receives semantic labels, counts per label, and scene video with overlaid instance IDs; input video is captured for ScanNet++ and rendered for InteriorGS.Large InteriorGS scenes may be partitioned by room metadata or manually divided, while dense object clusters receive multi-view images to address mapping errors.Source: §3 · p. 4
  • Hierarchy generation starts with structural relations, then adds functional areas through human verification and iterative VLM refinement, and revises parent descriptions to reference their child nodes.The hierarchy runs from a scene-level root to individual-object leaves.Source: §3 · pp. 4–5

4. Benchmarking Tasks

The benchmark covers attribute-existence questions, five spatial tasks, and multi-step reasoning questions grounded in scene hierarchies. Its generated questions are checked through a user study that contrasts clean and perturbed samples.

4.1. Existence-Based Questions

  • Existence questions ask whether an object has a visual attribute, with positive attributes unique to an instance and negative attributes absent from every instance of its category.Annotators verify the generated test questions.Source: §4.1 · p. 5

4.2. Spatial Intelligence Questions

  • Spatial Intelligence questions cover size, count, absolute and relative distance, and relative direction, using metadata-derived ground truth and attribute-based referring expressions to distinguish same-category instances.For counting, questions can target subsets sharing a visual attribute, such as red shoes rather than shoes generally.Source: §4.2 · p. 5

4.3. Reasoning-Based Questions

  • Reasoning questions are QRA triplets generated from scene video and hierarchical descriptions, with steps moving from scene overview and area inspection to target identification, neighboring observations, fine details, inference, and synthesis.The paper classifies these questions into twelve reasoning types and generates one per category for each scene where possible.Source: §4.3 · p. 5, Figure 3 · p. 6

4.4. Quality and Statistics

  • A user study assessed the accuracy and relevance of 100 questions across 10 scenes on a five-point scale, and users rated clean samples higher than perturbed samples.The perturbed samples serve as a quality-control measure.Source: §4.4 · p. 6

5. Experiments

The experiments evaluate vision-language models across existence, spatial intelligence, and complex reasoning on real-world and synthetic 3D scenes. Results vary by task: dense video performs strongly overall, while geometry-aware fine-tuning improves spatial reasoning.

5. Experiments

  • The evaluation uses 766 scenes and 26,311 questions for training, with 200 scenes and 7,213 questions reserved for testing.The 966 scenes span real-world ScanNet++ reconstructions and synthetic InteriorGS Gaussian splats.Source: §5 · p. 6, §5.3 · p. 7

5.1. Evaluation Metrics

  • Existence uses precision, recall, F1, and accuracy; spatial tasks use ACC or MRA, while reasoning uses text-similarity metrics and LLM-as-a-judge ratings.Figure 4 breaks down existence, reasoning, and spatial performance categories.Source: §5.1 · pp. 6–7, Figure 4 · p. 7

5.2. Methods

  • The baselines include proprietary and open-source VLMs, video and panorama inputs, CoT prompting, and specialized CoV, Think3D, and GeoThinker frameworks.GeoThinker is also fine-tuned on the SceneBench data.Source: §5.2 · p. 7

5.3. Benchmark Results

  • Video-based VLMs achieve strong existence results, with Video (CoT) best across primary metrics; CoV outperforms Think3D, and SceneBench fine-tuning boosts GeoThinker.Panoramic images perform slightly worse than video for existence tasks.Source: §5.3 · pp. 7–8
  • VLMs perform strongly on counting, but CoT tends to degrade spatial results; fine-tuned GeoThinker surpasses dense-video VLMs on spatial queries.Geometry-aware methods suit fine-grained distance and direction queries, while video evaluation requires many frames.Source: §5.3 · p. 8
  • Gemini Video (Basic) is best overall on advanced reasoning, except for Narrative and Contextual Synthesis, where CoV performs better.Fine-tuning GeoThinker on SceneBench increases its advanced-reasoning performance.Source: §5.3 · p. 8

6. Conclusion

SceneBench combines photorealistic Gaussian Splatting scenes with hierarchical annotations to evaluate 3D vision-language understanding. The evaluation finds bottlenecks as hierarchical complexity increases, especially in reasoning across semantic levels and combining visual details with spatial relationships.

6. Conclusion

  • SceneBench provides 966 Gaussian Splatting scenes with hierarchical annotations spanning scenes, rooms, functional areas, and individual objects.Its evaluation reveals performance bottlenecks as hierarchical complexity increases, including in cross-level reasoning and integrating fine visual details with spatial relationships.Source: §6 · p. 8
  • The authors present SceneBench as a realistic testbed for diagnosing model capabilities and guiding development of vision-language systems for photorealistic 3D environments.They hope it will support development and evaluation of systems with stronger spatial intelligence.Source: §6 · pp. 8–9

7. Overview

The supplement collects additional data-generation details, question-set statistics, benchmark results, human-evaluation interface information, and qualitative examples.

7. Overview

  • The supplement covers data and question generation, question-set statistics, additional quantitative results, the human-evaluation interface, and qualitative results.Additional results include reasoning text metrics, per-attribute and per-hierarchy breakdowns, extra GeoThinker frames, and average frame usage.Source: §7 · p. 12

8. Data Generation Process

The data-generation pipeline builds navigable scene videos, hierarchical object descriptions, and question sets spanning visual attributes, spatial intelligence, and multi-step reasoning. Human review and grounding checks refine the generated annotations and questions.

8.1. Video Generation Process

  • A* paths from occupancy-map waypoints are smoothed with curvature-aware B-splines, then rendered at floor, mid-level, and flying heights for scene coverage.On sampled scenes, automatically generated videos were judged comparable in quality for annotation to manually recorded camera videos.Source: §8.1 · p. 12, Figure 5 · p. 13

8.2. Dataset Generation Process

  • The human-in-the-loop pipeline inspects model outputs, revises prompts, and reruns stages until outputs are satisfactory.Object descriptions are generated from videos with overlaid instance IDs and object lists, then checked and corrected against the video.Source: §8.2 · p. 12, §8.2.1 · p. 12

8.2.2. Hierarchy Generation

  • Objects are grouped using labels, bounding-box proximity, and spatial relationships, then VLMs generate subregion and scene hierarchies with grounded descriptions.Annotators split visually distinct same-label groups where possible, and descriptions are checked for mentions of immediate child nodes.Source: §8.2.2 · pp. 12–13

8.2.3. Existence Based Questions

  • Existence questions pair a present attribute with a plausible but visually absent distractor, requiring inspection of high-fidelity visual details.Shape, size, and object state are excluded; test questions are checked and corrected by annotators.Source: §8.2.3 · p. 13

8.2.4. Spatial Intelligence Based Questions

  • Spatial-intelligence questions use object referring expressions and 3D bounding boxes to ask about counts, size, absolute and relative distances, and relative direction.Ground-truth dimensions and distances come directly from the source datasets’ object bounding boxes; annotators review the test questions.Source: §8.2.4 · p. 13

8.2.5. Reasoning Based Questions

  • Reasoning questions span 12 categories, with a VLM generating questions from the scene video and full hierarchy JSON.The categories range from narrative synthesis to fine-grained physical reasoning.Source: §8.2.5 · p. 14

8.2.6. Question Attribute and Level

  • Each reasoning question includes an observation-led chain from scene overview to object details and inference, with instance IDs anchoring each step.Question statistics map visual features to questions and assign a hierarchy level using the deepest referenced object.Source: §8.2.5 · p. 14, §8.2.6 · p. 14

9. Data Statistics

The benchmark combines training and held-out questions from real-world and synthetic scenes, with reasoning forming the largest question category. Question composition tracks visual features, including questions that refer to multiple features.

Question counts

  • The benchmark contains 33,524 question–answer pairs—7,213 test and 26,311 train—from 966 scenes, split into 766 training and 200 held-out test scenes.The scenes comprise 584 ScanNet++ real-world scenes and 382 InteriorGS synthetic scenes; reasoning is the largest category, with 11,279 questions.Source: §9 · p. 14, Table 6 · p. 14

Reasoning questions

  • Reasoning questions cover 12 subcategories sampled near-uniformly, with one question per category per scene wherever possible.The subcategory counts are provided separately for test, train, and total questions.Source: Table 7 · p. 14, §9 · p. 14

Visual-feature composition

  • Existence questions probe one visual feature, with color and material more frequent than texture, depicted content, text, and lighting; spatial questions use multiple objects and attributes.Questions naming multiple features are counted in each relevant category, so a question may appear in more than one row.Source: §9 · p. 14, Table 8 · p. 14, Table 9 · p. 15
  • The Other category in the visual–spatial question breakdown primarily contains non-visual properties such as object positions.Source: §9 · p. 15

10. Additional Results

Additional analyses compare reasoning-answer text similarity, performance across visual attributes and hierarchy depth, the effect of more frames on GeoThinker, and per-question frame use. GeoThinker leads text-similarity metrics, while deeper hierarchy levels remain challenging for most models.

10.1. Reasoning Text-Similarity Metrics

  • GeoThinker (Finetuned) has the highest BLEU-1, CIDEr, METEOR, and ROUGE-L scores on both datasets, while Gemini leads LLM-as-a-judge evaluation.Gemini scores lower on text-based metrics, likely because its longer step-by-step answers are penalized by n-gram matching.Source: §10.1 · p. 15, Table 11 · p. 15

10.2. Granular Performance Breakdown

  • Video-based VLMs perform consistently across visual attributes, but most models struggle beyond hierarchy levels 4–5; Video (CoT) and fine-tuned GeoThinker retain leaf-level accuracy.Source: §10.2 · p. 15, Figure 6 · p. 16

10.3. Results of GeoThinker with more frames

10.4. Frame Usage Across Methods

  • Video (Basic) uses 165–330 frames per ScanNet++ question, whereas CoV averages 18.6 on ScanNet++ and 20.34 on InteriorGS.GeoThinker uses a fixed budget of 16, 32, or 64 frames; CoV combines 16 initial frames, one BEV frame, and a few adaptive frames.Source: §10.4 · p. 15

11. Human Evaluation Interface

The user-evaluation interface presents questions alongside interactive Gaussian Splatting scenes, labels, and task-specific bounding-box highlights. A small study found higher ratings for clean than perturbed questions, especially for relative-distance and relative-direction tasks.

Interface and feedback

  • The interface loads a selected scene and sampled questions, then overlays labels and 3D boxes with colors that vary by question type.Reasoning questions receive no automated highlighting, so users explore the scene to check the displayed logic.Source: §11 · pp. 16–17

Question distribution

  • The user study sampled 100 questions from 10 scenes, including perturbed samples.Ratings are recorded on a 1-to-5 scale in each user’s local profile.Source: §11 · p. 17, Table 16 · p. 17

Sample perturbation

  • Perturbations flip existence answers, alter counts or attributes, change size and distance values by factors of 5.0 to 10.0, or introduce logically flawed reasoning questions.Source: §11 · p. 17

User study results

  • Seven users rated clean samples higher than perturbed samples, with the clearest difference for relative distance and relative direction.The gap was smaller for absolute distance and object size, which users generally found harder because they require guessing.Source: §11 · p. 17

12. Qualitative Results

The qualitative figures present examples spanning existence, spatial-intelligence, and reasoning questions. They also show how the evaluation interface supports human inspection of answers and task-specific scene evidence.

12. Qualitative Results

  • Figure 20 presents existence-question examples, while Figures 15–19 present examples for spatial-intelligence questions.These figures show qualitative results across the task categories, without specifying comparative outcomes in the supplied text.Source: §12 · p. 17
  • Figures 21–22 show one example for each of 12 reasoning task types and compare Gemini Video Basic, CoV, GeoThinker, and GeoThinker-finetuned.Long Gemini answers are shortened with ellipses for visual clarity.Source: §12 · p. 17
  • The hierarchy-generation prompt treats object descriptions and locations as the primary source, uses scene video for visual context, and treats a programmatic hierarchy as supplementary guidance.The reasoning-question prompt takes a scene video and hierarchical JSON whose branches descend from the scene to individual objects.Source: §12 · pp. 20, 24
  • The human-evaluation interface displays sampled QA pairs with ground-truth answers and a 1–5 rating scale; reasoning views provide a sidebar process without automated highlighting.For reasoning tasks, annotators can explore the 3D scene to verify the logic.Source: Figure 23 · p. 30, Figure 27 · p. 31
  • For existence and direction queries, the interface marks targets or same-class context in green or yellow and distinguishes origin, target, and selected objects by color.Absent existence targets are contextualized by highlighting other instances of the same class in yellow.Source: Figure 24 · p. 30
  • Counting views mark qualifying instances green and nonqualifying same-class instances yellow; size and absolute-distance views mark all referenced objects green.Source: Figure 25 · p. 30, Figure 26 · p. 31

Related papers

Loading 2609.16233v1…