PaperlayerView exact revision on arXiv

Source-linked AI summary

Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence

Yuanyuan Gao, Hao Li, Yifei Liu, Xinhao Ji, Yuning Gong, Yuanjun Liao, Fangfu Liu, Manyuan Zhang, Yuchen Yang, Dan Xu, Xue Yang, Huaxi Huang, Hongjie Zhang, Ziwei Liu, Xiao Sun, Dingwen Zhang, Zhihang Zhong

arXiv:2603.07660v1cs.CV

TL;DR

Spatial intelligence research is constrained by scarce, narrowly annotated 3D data and costly curation pipelines. Holi-Spatial automatically converts raw videos into high-fidelity 3D geometry and holistic semantic supervision, producing Holi-Spatial-4M. The dataset improves curation quality and yields consistent gains in 3D grounding and spatial reasoning when used to fine-tune VLMs.

  • Problem

    Existing spatial-supervision methods depend on small manually annotated 3D datasets or feed-forward single-image models, limiting scalability and semantic coverage.

  • Method

    Holi-Spatial combines 3DGS geometric optimization, open-vocabulary image perception, and scene-level lifting and refinement to annotate raw videos automatically.

  • Results

    Fine-tuning VLMs on Holi-Spatial-4M consistently improves 3D grounding and spatial reasoning across ScanNet, ScanNet++, and DL3DV; ScanNet++ 3D grounding gains 15% AP50.

  • Takeaways & Limitations

    Holi-Spatial-4M provides large-scale, multi-level spatial supervision for training and evaluating holistic 3D spatial intelligence.

  • Takeaways & Limitations

    The pipeline can be computationally expensive and may degrade on limited-view, blurred, heavily occluded, or dynamic videos; foundation-model labeling may inherit biases or errors.

Abstract

from arXiv · show

The pursuit of spatial intelligence fundamentally relies on access to large-scale, fine-grained 3D data. However, existing approaches predominantly construct spatial understanding benchmarks by generating question-answer (QA) pairs from a limited number of manually annotated datasets, rather than systematically annotating new large-scale 3D scenes from raw web data. As a result, their scalability is severely constrained, and model performance is further hindered by domain gaps inherent in these narrowly curated datasets. In this work, we propose Holi-Spatial, the first fully automated, large-scale, spatially-aware multimodal dataset, constructed from raw video inputs without human intervention, using the proposed data curation pipeline. Holi-Spatial supports multi-level spatial supervision, ranging from geometrically accurate 3D Gaussian Splatting (3DGS) reconstructions with rendered depth maps to object-level and relational semantic annotations, together with corresponding spatial Question-Answer (QA) pairs. Following a principled and systematic pipeline, we further construct Holi-Spatial-4M, the first large-scale, high-quality 3D semantic dataset, containing 12K optimized 3DGS scenes, 1.3M 2D masks, 320K 3D bounding boxes, 320K instance captions, 1.2M 3D grounding instances, and 1.2M spatial QA pairs spanning diverse geometric, relational, and semantic reasoning tasks. Holi-Spatial demonstrates exceptional performance in data curation quality, significantly outperforming existing feed-forward and per-scene optimized methods on datasets such as ScanNet, ScanNet++, and DL3DV. Furthermore, fine-tuning Vision-Language Models (VLMs) on spatial reasoning tasks using this dataset has also led to substantial improvements in model performance.

1. Introduction

Holi-Spatial addresses the scarcity and limited coverage of spatial data by converting raw videos into high-fidelity 3D geometry and holistic semantic annotations without explicit 3D sensors or human labeling. Its curation pipeline and resulting Holi-Spatial-4M dataset improve annotation quality and downstream spatial-model performance.

  • Motivation: Existing spatial datasets rely on limited manual 3D annotations or feed-forward single-image models, constraining scalability and semantic coverage.ScanNet, for example, provides only 50 labeled classes.
  • Contribution: Holi-Spatial systematically composes recent AI tools into a fully automated spatial annotation engine operating on raw video streams.The framework aims to avoid explicit 3D sensors and human-in-the-loop labeling.
  • Method: The pipeline combines geometric optimization, image-level perception, and scene-level lifting and refinement to produce multi-level spatial supervision.Its outputs include 3DGS geometry, depth, masks, boxes, captions, grounding annotations, and spatial QA.
  • Evaluation: On ScanNet++, the curation pipeline improves multi-view depth estimation by up to 0.5 F1 and boosts 3D detection AP50 by 64%.The evaluation compares Holi-Spatial with model-based and 3DGS-based optimization baselines across ScanNet, ScanNet++, and DL3DV-10K.
  • Dataset: Holi-Spatial-4M contains 12K optimized 3DGS scenes and millions of masks, grounding instances, and spatial QA pairs.The dataset also includes 320K 3D bounding boxes and 320K instance captions.
  • Results: Fine-tuning Qwen3-VL on Holi-Spatial-4M yields a 15% AP50 improvement for ScanNet++ 3D grounding and a 7.9% accuracy increase on MMSI-Bench.The annotations also exhibit higher granularity and improved boundaries compared with official ScanNet labels.

2. Related Work

Existing spatial-intelligence approaches face scalability bottlenecks from human-annotated 3D data, manually curated scans, or costly per-scene optimization. Holi-Spatial instead frames spatial-data curation as a non-human pipeline that converts raw videos into annotated high-fidelity 3D scenes.

  • Related Work: Recent spatial datasets remain scarce relative to billion-scale 2D image resources, limiting the diversity available for spatial intelligence.The passage contrasts spatial scene scarcity with datasets such as LAION-5B.
  • Related Work: 3D-native spatial LMMs consume explicit 3D observations, while 2D-centric spatial LMMs scale datasets and training recipes for spatial perception and reasoning.These represent two major approaches to improving spatial intelligence in multimodal models.
  • Related Work: Optimization-driven 3DGS methods support open-vocabulary grounding within individually optimized scenes.Their scene-specific optimization distinguishes them from feed-forward approaches.
  • Limitations of Existing Paradigms: Human-annotated approaches incur 3D acquisition and annotation overhead, whereas optimization-driven 3DGS requires time-consuming and unstable per-scene training or finetuning.These costs create fundamental bottlenecks for large-scale deployment.
  • Holi-Spatial: Holi-Spatial reframes spatial-data curation as a scalable, non-human pipeline that converts raw videos into annotated high-fidelity 3D scenes.This framing targets automatic generation of large-scale spatial-understanding data.

3. Method

Holi-Spatial builds spatial annotations through geometric optimization, image-level perception, and scene-level refinement. The pipeline lifts segmented 2D observations into reliable 3D instances, consolidates and validates them, then generates dense semantic annotations and spatial QA.

  • Pipeline Overview: The pipeline has three stages: geometric optimization, image-level perception, and scene-level refinement.These stages respectively distill 3D structure, extract and lift object labels, and refine instances into dense spatial annotations.
  • Geometric Optimization: Geometric optimization combines camera estimation, monocular depth initialization, and 3DGS optimization with multi-view depth consistency to suppress floaters.The resulting representation is intended to provide clean, physically aligned geometry for downstream 3D bounding-box generation.
  • Image-level Perception: Mask erosion and mesh-guided depth filtering remove boundary errors and depth outliers before estimating initial 3D oriented bounding boxes.The lifting equation back-projects each masked pixel using rendered depth and camera intrinsics.
  • Scene-level Refinement: Scene-level refinement merges same-category proposals with sufficient 3D overlap, floor-aligns their OBBs, and preserves the most reliable observation for later captioning.The merge threshold is τ_merge = 0.2, and the retained source image is associated with the maximum confidence score.
  • Scene-level Refinement: A tri-level confidence rule filters consolidated instances, while a VLM agent reassesses proposals in the verify band before low-confidence cases are discarded.The high and low thresholds are 0.9 and 0.8, respectively; validated instances receive fine-grained captions and templated spatial QA pairs.

4. Dataset

Holi-Spatial-4M is a large-scale, multi-spatial-modal dataset built from diverse raw videos, combining millions of annotations across geometry, semantics, grounding, and spatial reasoning.

  • Dataset introduction: Holi-Spatial-4M is constructed from raw video streams using the proposed automated curation pipeline.The dataset is presented as a large-scale, multi-spatial-modal resource.
  • Data Composition and Scale: 12,000+ optimized 3DGS scenes and 4 million+ spatial annotations form the dataset’s core scale.Annotations include 1.3M 2D instance masks, 1.2M 3D grounding pairs, 320K 3D bounding boxes, and 320K instance captions.
  • Open-Vocabulary Diversity: Open-vocabulary annotation covers fine-grained indoor objects beyond traditional closed-category datasets.The dataset uses VLM knowledge to annotate a broad array of object categories.
  • Spatial Question-Answering Pairs: 1.25M spatial QA pairs are organized into camera-centric and object-centric tasks for holistic 3D reasoning.Tasks include camera rotation, movement direction, object-to-object distance, global/local direction, and size measurement.
  • Evaluation coverage: The dataset evaluates 3D detection, 2D segmentation, and depth estimation using AP@25, AP@50, IoU, and F1-score metrics.Higher values are better, and unavailable metrics are marked with an em dash.

5. Experiment

Experiments evaluate Holi-Spatial’s geometry, segmentation, detection, spatial reasoning, grounding, and refinement components on established benchmarks. The results show stronger multimodal predictions and benefits from geometric refinement, confidence filtering, and VLM-based training.

  • Settings: Evaluation uses 10 manually annotated scenes from each of ScanNet, ScanNet++, and DL3DV-10K, with official depth maps for depth assessment.The selected scenes provide evaluation ground truth for 2D masks and 3D bounding boxes.
  • Framework Evaluation: 0.89 Depth F1-score on ScanNet++ exceeds M3-Spatial’s 0.39, while multi-view projection produces fewer ghosting artifacts and floaters.The evaluation compares Holi-Spatial with 3DGS-based baselines and other depth methods.
  • Framework Evaluation: 0.64 IoU exceeds SA2VA’s 0.25 for 2D segmentation, with sharper boundaries and more complete masks under occlusion.The reported improvement is attributed to combining image-level semantic reasoning with multi-view geometric information.
  • Framework Evaluation: 81.06 AP25 on ScanNet++ exceeds LLaVA-3D’s 12.2 AP25, with tighter 3D boxes, more recovered objects, and more accurate labels.Qualitative comparisons also report tighter boxes and more correct categories than SceneScript, LLaVA-3D, and SpatialLM.
  • VLM Finetuning Evaluation: Fine-tuning Qwen3-VL models on Holi-Spatial-4M improves spatial reasoning and achieves 27.98 AP50 for 3D grounding, 14.48 AP above the strongest baseline.Spatial reasoning is evaluated on MMSI-Bench and MindCube, while grounding uses 1.2M grounding pairs.
  • Ablation: GS-refined depth outperforms direct DA3 depth, while confidence filtering raises precision from 0.35 to 0.67 but lowers recall from 0.74 to 0.69.The refinement stage addresses multi-view ghosting, false positives, and difficult instances discarded by confidence filtering.

6. Conclusion

Holi-Spatial converts raw videos into high-fidelity 3D geometry and holistic annotations through automated geometric, semantic, and scene-level processing. Its released dataset supports improvements in 3D grounding and spatial reasoning after VLM fine-tuning.

  • Conclusion: Holi-Spatial combines 3DGS optimization, open-vocabulary perception, and scene-level lifting to produce multi-level spatial supervision.Supervision includes rendered depth, 2D masks, 3D boxes, instance captions, and spatial QA.
  • Conclusion: 12K optimized 3DGS scenes and millions of masks, boxes, captions, grounding instances, and QA pairs comprise Holi-Spatial-4M.The conclusion reports the dataset’s scale across multiple annotation types.
  • Conclusion: Fine-tuning VLMs on Holi-Spatial-4M consistently improves 3D grounding and spatial reasoning on ScanNet, ScanNet++, and DL3DV.The reported improvements span the listed benchmark datasets and both task areas.

Impact Statement

Holi-Spatial enables large-scale spatial dataset construction from raw video, but its pipeline remains computationally demanding and sensitive to difficult video conditions and foundation-model errors.

  • Impact: The pipeline converts raw video streams into high-fidelity 3D geometry and holistic spatial annotations for large-scale dataset construction.The approach is described as a fully automated data curation pipeline.
  • Limitations: Per-scene optimization and multiple upstream components can be computationally expensive, especially for limited viewpoints, motion blur, occlusion, or dynamic objects.These conditions may degrade pipeline performance.
  • Limitations: Open-vocabulary labeling may inherit foundation-model biases or errors, motivating stronger verification and uncertainty estimation.The paper identifies robust verification and uncertainty estimation as future directions.
  • Responsible Use: Publicly available data avoids collecting new sensitive personal data, but the technology could be misused for privacy-invasive reconstruction of personal spaces.The paper recommends consent, data governance, and appropriate safeguards.

A. Why 3D Multi-view Merges

Reliable 3D multi-view merging is necessary because image-level instance predictions can fragment objects, while inaccurate depth can cause different objects to merge incorrectly. Holi-Spatial addresses these issues through 3D geometric clustering and refined depth constraints.

  • Failure Modes: SAM3 can fragment one object into multiple image-level instances under occlusion, making direct adoption of its predictions unreliable.The paper gives a complete bed detected as two separate beds as an example.
  • Failure Modes: Depth accuracy is crucial for 3D clustering because ghosting and aliasing can spatially entangle different objects and cause false merges.The multi-view geometric constraint is used to reduce these depth artifacts.
  • Refinement: Holi-Spatial’s 3D grounding predictions show multi-view consistent localization across diverse indoor scenes.The examples are presented as evidence of robust detection across varied scenes.
  • Refinement: 3D geometric clustering merges separated SAM3 segments into unified object instances after occlusions fragment them.Figure 12 presents this as the effect of geometric clustering on image-level predictions.

B. QA Examples

Holi-Spatial curates spatial QA examples covering camera motion, camera-object relations, object relations, and object measurements. Training on these QA pairs improves spatial reasoning across MindCube and MMSI-Bench, especially for perspective-changing and egocentric-imagination questions.

  • QA Coverage: The curated QA set includes ten spatial question types spanning camera motion, camera-object relations, object relations, and object measurement.Examples include rotation, movement direction and distance, global and local directions, and object-object distance.
  • QA Results: The largest reported improvements concern perspective-changing and egocentric-imagination questions.The passage identifies these task types as especially enhanced after QA training.
  • Curation Tools: Agent verification and 2D object-grounding prompt templates support the curation workflow.The templates are provided in Figures 13 and 14.
  • Curation Tools: Additional 3D detection visualizations show predictions across diverse indoor scenes.The figure emphasizes multi-view consistent localization and robust detection.

D. Detail of Scene-Level Refinement Algorithm

The scene-level refinement algorithm lifts 2D open-vocabulary masks into 3D candidates, merges same-label observations using 3D IoU, and validates merged instances through confidence gating and VLM arbitration. It produces a validated 3D instance set for the scene.

  • Algorithm Overview: The refinement algorithm operates in three stages to lift 2D open-vocabulary predictions into a globally consistent 3D representation.The stages are lifting, multi-view merging, and confidence-based verification.
  • Lift 2D Instances to 3D Candidates: Frame-level SAM3 masks are projected into 3D using depth maps and camera intrinsics, producing localized point clouds and 3D bounding boxes.Each mask is converted into a per-label 3D candidate.
  • Multi-view Merging: Candidates sharing a label are spatially merged when their 3D Intersection-over-Union exceeds a predefined threshold τ_iou.This resolves redundancies across views within each label.
  • Confidence Gating and VLM Verification: High-confidence instances are accepted, low-confidence noise is discarded, and ambiguous groups are verified by a VLM using a cropped canonical view.The thresholds satisfy τ_low < τ_high.
  • Output: The pipeline filters spurious detections and yields a robust, validated set of 3D scene instances.The algorithm output is the validated instance set O, with each item containing a label, bounding box, canonical frame, and mask.
  • Spatial QA: The paper provides curated examples of ten spatial QA types, including camera, object-relation, and measurement questions.These examples are part of the broader Holi-Spatial annotation pipeline.
  • Multi-view Merging: The algorithm groups candidate instances into label-specific sets before adding them to the global instance collection.Each group consists of candidates associated with the same label.
Loading 2603.07660v1…