PaperlayerView exact revision on arXiv

Source-linked AI summary

Holi-Spatial 360: Generative Reconstruction from Sparse Panoramas for Spatial Rendering and Intelligence

Yuanyuan Gao, Zhirui Liu, Yuning Gong, Yifei Liu, Yifan Zhan, Kuixuan Jiao, Xiaolong Zhou, Yifan Yang, Hongjie Zhang, Xue Yang, Dan Xu, Zhihang Zhong

arXiv:2610.19502v1cs.CV

TL;DR

Sparse panoramas of multi-room homes offer a potential source of diverse environments, but their limited overlap and unobserved regions make reliable reconstruction difficult. Holi-Spatial 360 combines correspondence-guided 3DGS reconstruction with adapted image and video priors to build annotated scenes from panoramas. The paper reports improved reconstruction and annotation quality and benefits to VLM spatial reasoning.

  • Problem

    Spatial intelligence needs photorealistic, semantically grounded environments, but sparse panoramas have limited overlap and unobserved regions that make multi-room geometry difficult to reconstruct reliably.

    Source: §1 · p. 2

  • Method

    Holi-Spatial 360 uses feed-forward reconstruction priors to optimize cross-view-consistent 3DGS, then applies adapted image and video priors to restore and complete views while grounding them in observed panoramas.

    Source: §1 · p. 2

  • Results

    Experiments report improved reconstruction and annotation quality under sparse panoramic inputs and show that Holi360 benefits VLM spatial reasoning.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Holi360 provides over 1K multi-room scenes, 100K instance annotations, and 2M spatial question-answer pairs.

    Source: §1 · p. 2

    Release is limited to permitted materials, and reconstructed geometry and annotations may contain errors or biases; they should not support occupant identification, home surveillance, or safety-critical decisions.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Spatial intelligence requires scalable 3D environments that are both photorealistic and semantically grounded, yet existing indoor datasets are dominated by room-scale captures and costly LiDAR acquisition. Online real-estate platforms offer abundant panoramas of diverse multi-room homes, but sparse viewpoints and texture-poor regions make reconstruction severely under-constrained. We introduce Holi-Spatial 360, a generative reconstruction and data curation framework for converting these panoramas into annotated multi-room 3D environments. Because reconstruction quality bounds subsequent generation, our Correspondence-Guided Geometric Optimization converts feed-forward reconstruction priors into reliable cross-view constraints and builds a coherent 3D Gaussian Splatting representation as 3D spatial memory. Adapted single-image and video priors then restore appearance and complete continuous trajectories while preserving observed content. The reconstructed scenes support arbitrary RGB-D rendering and an automated curation pipeline that combines end-to-end 3D grounding with VLM-based verification. Holi360 contains over 1K multi-room 3DGS scenes, more than 100K instance annotations, and 2M spatial question-answer pairs. We also introduce the Holi360 benchmark to evaluate the spatial understanding of VLMs in real-world, multi-room indoor environments. Experiments demonstrate superior reconstruction and curation quality under sparse panoramic inputs, while fine-tuning VLMs on Holi360 substantially improves spatial reasoning capabilities.

1 INTRODUCTION

Scalable spatial intelligence needs photorealistic, semantically grounded multi-room environments, but existing datasets and sparse panoramas make them costly and difficult to reconstruct. Holi-Spatial 360 turns online panoramas into annotated 3D environments through correspondence-guided reconstruction and generative completion; the resulting Holi360 dataset supports spatial reasoning research.

Motivation

  • Existing indoor datasets are dominated by room-scale captures or require dedicated capture, expert annotation, or dense video trajectories, leaving diverse real-world multi-room environments costly to obtain.The paper identifies this as a gap in environments with both high-quality reconstruction and semantic supervision.Source: §1 · p. 2
  • Sparse real-estate panoramas offer diverse multi-room homes, but wide viewpoint gaps, texture-poor surfaces, and unobserved regions make geometry unreliable and generated views prone to drift.Generative restoration alone does not establish reliable geometry from sparse, multi-room panoramas.Source: §1 · p. 2

Approach

  • Holi-Spatial 360 combines feed-forward reconstruction priors and cross-view constraints to optimize coherent 3DGS spatial memory, then uses image and video priors to restore and complete grounded views.The pipeline uses only RGB panoramas, without depth capture or human-in-the-loop labeling.Source: §1 · p. 2

Dataset and results

  • Holi360 contains over 1K multi-room 3DGS scenes, more than 100K verified 3D instances, and 2M spatial question-answer pairs, with reconstruction and annotation using only RGB panoramas.RealSee3D LiDAR and semantic maps are reserved for evaluation.Source: §1 · p. 2
  • Experiments report improved rendering, geometry, and grounding, while fine-tuning VLMs on Holi360 boosts spatial reasoning.Source: §1 · p. 2

2 RELATED WORK

Prior work expands spatial supervision over existing assets or refines sparse-view reconstructions with generative models, but leaves gaps in environment diversity and initial geometry. Holi-Spatial 360 instead scales the pool of real-world multi-room homes and globally optimizes a unified 3DGS before using generation to fill missing views.

2.1 SCALABLE INDOOR ENVIRONMENTS FOR SPATIAL INTELLIGENCE

  • Existing annotation efforts enrich supervision over largely fixed 3D asset pools, whereas Holi-Spatial 360 scales the pool itself by reconstructing diverse real-world multi-room homes from online panoramas.Source: §2.1 · p. 3

2.2 GENERATIVE RECONSTRUCTION FROM SPARSE VIEWS

  • Generative reconstruction methods refine 3D representations, but their main advances center on the generator and leave initial reconstruction largely unchanged; Holi-Spatial 360 globally optimizes unified 3DGS and uses generation to densify missing evidence.The approach aims to preserve scene identity along continuous trajectories.Source: §2.2 · p. 3

3 METHOD

The method stabilizes sparse-view geometry with fixed cross-view correspondences, adapts image and video priors using panorama-derived supervision, then densifies trajectories and curates grounded spatial annotations.

3.1 CORRESPONDENCE-GUIDED GEOMETRIC OPTIMIZATION

  • Nearby co-visible panorama pairs processed independently with DA3 provide dense, reliable correspondence anchors that are fixed during initial 3DGS optimization.The fixed references stabilize geometric supervision in texture-poor regions.Source: §3.1 · pp. 3–4

3.2 IN-THE-WILD GENERATIVE PRIOR ADAPTATION

  • Held-out panoramas provide RGB supervision for DiFix restoration and fixed-center rotational clips for Wan-VACE video restoration, conditioned on rendered RGB-D and nearby reference views.For adaptation, the guide image is the original 3DGS rendering; depth remains geometric conditioning during trajectory densification.Source: §3.2 · pp. 4–5

3.3 GENERATIVE TRAJECTORY DENSIFICATION

  • Trajectory densification uses two stages: DiFix-restored intermediate views update Θ0 to Θ1, then Wan-VACE-refined sequences produce the final Θ2.After densification, captured and generated views jointly provide multi-view supervision; rendered depth conditions restoration but is not treated as pseudo-ground-truth geometry.Source: §3.3 · pp. 5–6

3.4 AUTOMATED SPATIAL DATA CURATION

  • The curation pipeline builds scene-specific object vocabularies, grounds instances in posed RGB-D views, verifies predictions geometrically and semantically, then creates spatial QA pairs.VLM-generated appearance descriptions are combined with geometric attributes and relations from verified 3D boxes.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments assess novel-view rendering, geometric reconstruction, 3D grounding, and VLM spatial reasoning. Results include stronger geometry and grounding metrics, alongside gains from Holi360 fine-tuning.

4.1 OUT-OF-DISTRIBUTION RENDERING QUALITY

  • On held-out household views, the method outperforms reconstruction-only and generative baselines; anchored 3DGS better preserves observed structure than the illustrated alternatives.Along interpolated trajectories, geometry-anchored restoration maintains scene identity across views despite artifacts or misaligned structures from comparison methods.Source: §4.1 · pp. 7–8

4.2 GEOMETRIC RECONSTRUCTION

  • At the 2 cm threshold, ours attains F-score 0.2617 versus 0.2207 for DA3 on 10 RealSee3D scenes, using the same DA3 initialization for all methods.Figure 6 also shows more complete and coherent novel-view depth than 3DGS, 2DGS, and PGSR over 0.2–10.0 m.Source: Table 2 · p. 8, §4.2 · p. 8, Figure 6 · p. 8

4.3 3D GROUNDING AND VLM FINE-TUNING

  • In qualitative 3D grounding comparisons on real-world scenes, verified boxes align more closely with target objects across diverse categories than the compared methods, including pseudo ground truth.Source: Figure 7 · p. 9
  • Fine-tuning Qwen3.5-4B on Holi360 raises classification macro accuracy from 46.77% to 85.25% and numerical Acc@0.5m from 37.59% to 76.34%.Its MAE falls from 0.9767 m to 0.3898 m; zero-shot numerical Acc@0.5m ranges from 28.95% to 49.46%.Source: §4.3 · p. 8, Table 4 · p. 9

5 ABLATIONS

The ablations examine rendering and geometry, generative restoration, and grounding curation. They report gains from correspondence guidance, generated views, and combining curation steps.

Grounding curation

  • F1@0.25 rises from 16.81 to 19.42 when cross-view merging and verification are combined.The grounding-curation ablation is on five RealSee scenes.Source: Table 6 · p. 9, §5 · p. 10

Rendering and geometry

  • Correspondence guidance raises F@2cm from 0.1991 to 0.3297 by stabilizing sparse-view geometry; generated views further increase PSNR from 19.64 to 21.30 and F@2cm to 0.3529.Source: §5 · p. 9

Generative restoration

  • Combining fine-tuned DiFix and Wan-VACE produces cleaner, more structurally coherent views, though each model retains a distinct failure mode.DiFix can introduce unnatural structures around a cabinet; Wan-VACE can produce a spurious protrusion under the influence of input floaters. Figure 8 compares DiFix and Wan variants, including versions without depth input.Source: §5 · p. 9, Figure 8 · p. 9

6 CONCLUSION

Holi-Spatial 360 turns sparse panoramas of multi-room homes into 3D environments and spatial annotations. Its experiments report improved reconstruction and annotation quality, with benefits for VLM spatial reasoning.

  • Holi-Spatial 360 is an automated pipeline that converts sparse multi-room-home panoramas into high-fidelity 3D environments and holistic spatial annotations.Source: §6 · p. 10
  • Holi360 contains over 1K scenes, 100K instance annotations, and 2M spatial question-answer pairs.Source: §6 · p. 10
  • Experiments show improved reconstruction and annotation quality under sparse panoramic inputs, and demonstrate benefits to VLM spatial reasoning.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT

The statement locates the reconstruction and curation pipeline in Section 3 and summarizes where supporting details and experiments are reported.

ETHICS STATEMENT

The authors restrict release to materials they are permitted to share and screen out identifying details. They also caution against using the reconstructed data for identification, surveillance, or safety-critical decisions.

ETHICS STATEMENT

  • Release is limited to materials the authors are permitted to share, since public accessibility does not itself grant redistribution permission.Source: ETHICS STATEMENT · p. 10
  • Faces, addresses, and personal documents are screened out as identifying details.Source: ETHICS STATEMENT · p. 10
  • The authors warn that reconstructed geometry and generated annotations may contain errors or biases and should not support surveillance or safety-critical decisions.Source: ETHICS STATEMENT · p. 10

A IMPLEMENTATION DETAILS

The implementation uses a scene-disjoint evaluation split and a shared DA3 initialization for reconstruction comparisons. RealSee evaluation is bounded by sparse LiDAR geometry and inconsistent semantic labels, while illumination differences can depress PSNR.

A IMPLEMENTATION DETAILS

  • 30 of approximately 1,000 collected scenes are reserved for testing, split into three 10-scene batches, with no scene overlap across training and evaluation sets.Source: A IMPLEMENTATION DETAILS · p. 14
  • Exposure and illumination inconsistencies may leave scene structure correct but create color discrepancies that lower PSNR across methods.Source: A IMPLEMENTATION DETAILS · p. 14
  • RealSee is described as the only publicly available real-world multi-room dataset with both LiDAR measurements and semantic annotations, but its LiDAR coverage is sparse and its category taxonomy is not unified.Source: A IMPLEMENTATION DETAILS · p. 14
  • Table 7 compares the number of forward–backward-consistent matches below 1 px across input-view counts.Source: Table 7 · p. 15
  • Sparse depth measurements and inconsistent label granularity can introduce inaccuracies when projecting annotations into 3D ground-truth boxes.Source: A IMPLEMENTATION DETAILS · p. 15
  • All reconstruction methods, including the proposed method, use DA3 initialization for a fair comparison.Source: A IMPLEMENTATION DETAILS · p. 15

B ABLATION VISUALIZATION

The ablation indicates that correspondence guidance supports stable geometry under sparse views, while unconstrained optimization can overfit RGB observations and produce floaters. It also highlights failure modes in grounding that the post-processing pipeline addresses.

B ABLATION VISUALIZATION

  • Without correspondence guidance, RGB-driven optimization weakens multi-view constraints, overfits observations, and produces more floaters under sparse views.The guided design adds constraints against geometrically plausible but semantically incorrect matches, which can persist despite low reprojection error.Source: B ABLATION VISUALIZATION · p. 15
  • Pairwise matching errors can be reinforced during optimization, and a low reprojection error does not rule out a semantically incorrect correspondence.Source: B ABLATION VISUALIZATION · p. 15
  • Boxer can duplicate detections of one object or misclassify objects, and the post-processing pipeline is designed to improve accuracy and robustness.Source: B ABLATION VISUALIZATION · p. 15

C ADDITIONAL VISUALIZATIONS AND QA TEMPLATES

The supplementary visualizations show generated views from test scenes and QA examples spanning ordinary RGB understanding and panorama-based spatial understanding. Additional figures depict ablations and reference, 3DGS, and trajectory views.

C ADDITIONAL VISUALIZATIONS AND QA TEMPLATES

Loading 2610.19502v1…