Source-linked AI summary
PanOVOcc: Panoramic Embodied Open-Vocabulary Occupancy Mapping with Long-term Spatial Voxel Memory
Di Kuang, Mengfei Duan, Yuhang Wang, Weixing Peng, Kailun Yang
TL;DR
Perspective-based occupancy systems have limited spatial coverage, while panoramic methods mainly predict local volumes from individual observations. PanOVOcc combines panoramic SLAM, open-vocabulary perception, and long-term voxel memory to build global language-queryable maps; it improves geometric and semantic occupancy accuracy across its synthetic and real-world benchmarks.
Problem
Perspective-based systems provide limited spatial coverage, and existing panoramic methods primarily predict local volumes from single observations rather than persistent maps.
Source: §I · p. 1Method
PanOVOcc integrates panoramic SLAM, open-vocabulary segmentation, and online voxel fusion to accumulate geometric and semantic evidence in a persistent, language-queryable global map.
Source: §I · p. 2, B. Overview of PanOVOcc · p. 3Results
Takeaways & Limitations
The findings highlight the value of combining omnidirectional sensing with persistent spatial memory for open-vocabulary embodied scene understanding.
Source: §V · p. 7
Abstract
from arXiv · showhide
Persistent semantic occupancy mapping is essential for embodied scene understanding. However, perspective-based systems provide limited spatial coverage, while existing panoramic methods primarily predict local volumes from single observations. We introduce PanOVOcc, a training-free framework for persistent open-vocabulary semantic occupancy mapping from panoramic sequences. PanOVOcc unifies panoramic SLAM, open-vocabulary perception, and long-term spatial voxel memory within an online architecture, continuously integrating geometric and semantic evidence into a global, language-queryable map. To facilitate systematic evaluation of this setting, we establish Pan-Replica and Pan-Holo360D, two benchmarks pairing continuous panoramic RGB-D sequences with scene-level semantic occupancy ground truth across synthetic and real-world scenes. Compared with the strongest evaluated baseline for each metric, PanOVOcc improves occupancy IoU and semantic mIoU by absolute +20.03 and +7.06 on Pan-Replica, and by +43.26 and +20.16 on Pan-Holo360D, respectively. The source code and the established benchmarks will be available at https://github.com/bakereet/PanOVOcc.
I. INTRODUCTION
PanOVOcc addresses the gap between limited-coverage perspective mapping and panoramic methods that predict only local volumes, by building persistent, language-queryable global maps from panoramic sequences. The work also introduces two benchmarks and reports gains across synthetic and real-world settings.
I. INTRODUCTION
- Perspective-based online systems see a narrow sector at a time, while panoramic methods mainly predict bounded local volumes from single observations rather than persistent maps.Panoramic observations offer denser spatial evidence around corners and through narrow passages.Source: §I · p. 1
- PanOVOcc combines panoramic SLAM, open-vocabulary perception, and voxel mapping to incrementally build a global, language-queryable scene representation.The map accumulates geometric and language-aligned semantic evidence as the agent explores.Source: §I · p. 2
- Pan-Replica and Pan-Holo360D pair continuous panoramic RGB-D sequences with scene-level semantic occupancy annotations across synthetic and real-world scenes.Pan-Replica offers controlled synthetic scenes with high-fidelity geometry and dense annotations; Pan-Holo360D covers larger interconnected rooms and more realistic sensing variation.Source: §I · p. 2
- PanOVOcc improves ceiling and floor IoU on Pan-Holo360D by 37.97 and 36.23, respectively, and outperforms FreeOcc on several additional open-vocabulary categories.Qualitative comparisons also show more complete and coherent geometry in large multi-room scenes.Source: §I · p. 2
- PanOVOcc is presented as a training-free framework for persistent global open-vocabulary semantic occupancy mapping from panoramic sequences.The framework combines omnidirectional sensing with long-term spatial memory for embodied exploration.Source: §I · p. 2
- Its persistent fusion scheme combines panoramic surface and free-space evidence with reliability-weighted multi-view semantics and reversible observation-level updates under pose refinement.Global consistency is challenging because sequential observations contain cross-view inconsistencies and redundancy, and individual errors can accumulate.Source: §I · p. 2
II. RELATED WORK
Prior panoramic scene-understanding work spans image semantics, 3D reconstruction, and voxel-level occupancy, but persistent mapping from long-horizon panoramic sequences remains underexplored. Embodied open-vocabulary occupancy work adds progressive mapping and language-aligned features, while still largely relying on perspective cameras or predefined categories.
A. Panoramic Semantic Scene Understanding
- Panoramic research has progressed from image-space semantics to 3D reconstruction and voxel-level occupancy, but persistent semantic mapping over long-horizon panoramic sequences remains underexplored.Image-space work addresses equirectangular distortion and open-vocabulary recognition; later work covers depth, structure, 3D semantics, and scene reconstruction.Source: A. Panoramic Semantic Scene… · p. 2
B. Embodied Open-Vocabulary Occupancy Mapping
- Embodied occupancy methods progressively update global maps, while open-vocabulary 3D methods add language-aligned features; existing approaches largely use fixed categories or perspective cameras.Perspective systems require more viewpoints and fragmented-view integration for complete scene context; PanOVOcc instead maps panoramic sequences into a persistent, language-queryable voxel map.Source: B. Embodied Open-Vocabulary Occupancy… · p. 2
III. METHOD
PanOVOcc incrementally builds a persistent global semantic occupancy map from panoramic RGB-D observations and an open-vocabulary category set. Its online pipeline combines panoramic pose estimation, semantic priors, and voxel-based fusion, including geometry-aware evidence and reversible observation updates.
A. Problem Formulation
- PanOVOcc incrementally constructs a persistent 3D semantic occupancy map from panoramic RGB-D observations and an open-vocabulary category set as an agent explores an unknown environment.The map is globally defined and tracks each voxel as occupied, free, or unobserved, alongside accumulated semantic evidence.Source: A. Problem Formulation · pp. 2–3
B. Overview of PanOVOcc
- Panoramic SLAM estimates global camera poses, open-vocabulary segmentation supplies pixel-wise semantic priors, and online fusion integrates geometric and semantic evidence into shared voxel memory.The memory supports reversible observation-level corrections and produces a globally aligned, language-queryable map without task-specific 3D training.Source: B. Overview of PanOVOcc · p. 3
C. Panoramic Geometric Mapping
- An anisotropic kernel aligned with spherical viewing geometry models panoramic measurements, while surface support is aggregated per observation and active voxel centers are projected to incorporate free-space evidence.Multiple pixels from one panorama can increase surface support but contribute at most one view count.Source: C. Panoramic Geometric Mapping · p. 3
- Reversible updates retain each observation’s geometric contribution so it can be removed and recomputed when pose refinement changes its spatial association.Raw additive statistics preserve provenance and allow these operations before nonlinear occupancy decisions.Source: C. Panoramic Geometric Mapping · p. 3
D. Geometry-Aware Semantic Consolidation
Geometry-aware consolidation regularizes uncertain semantic labels using local geometric and appearance cues, while persistent updates reconcile per-frame evidence after pose revisions.
D. Geometry-Aware Semantic Consolidation
- Pixel logits are softmaxed into category distributions, with each measurement weighted by confidence plus latitude- and range-dependent reliability.Source: D. Geometry-Aware Semantic Consolidation · pp. 3–4
- Ambiguous voxel unaries are regularized on a sparse six-neighbor graph over occupied voxels, using signed-PSVF, normal, tangent-plane, and color affinities.Source: D. Geometry-Aware Semantic Consolidation · p. 4
- On this graph, anchored diffusion starts from the initial semantic distributions q(0)=p_i.Source: D. Geometry-Aware Semantic Consolidation · p. 4
- Pose revisions subtract a frame's prior semantic delta before reintegrating it; labels are assigned only to occupied voxels, with unsupported occupied voxels remaining unknown.A bounded nearest-evidence transfer supplies labels where direct semantic support is absent.Source: D. Geometry-Aware Semantic Consolidation · p. 4
IV. EXPERIMENTS
The experiments assess panoramic sequential mapping, transfer from synthetic to real-world sequences without retraining, and factors affecting system performance.
IV. EXPERIMENTS
- The evaluation asks how panoramic sequences support scene-level occupancy mapping compared with existing methods.Source: §IV · p. 4
- A second question tests whether PanOVOcc generalizes from controlled synthetic to real-world panoramic sequences without retraining.Source: §IV · p. 4
- A third question examines how geometric and semantic components, including upstream prediction quality, govern performance.Source: §IV · p. 4
- On Pan-Replica, perspective methods receive twelve directional views per panoramic observation, providing approximately 360° horizontal coverage.Source: Table II · p. 5
- Pan-Holo360D includes scene-level semantic occupancy ground truth for seven real-world indoor scenes selected from Holo360D [18].Source: Figure 3 · p. 5
A. Benchmark
The benchmarks pair continuous panoramic RGB-D observations with scene-level occupancy evaluation across synthetic and real-world indoor settings.
A. Benchmark
- Pan-Replica and Pan-Holo360D provide panoramic evaluation sets, with synthetic evaluation and real-world generalization assessed respectively.Source: A. Benchmark · p. 5
- Pan-Replica adapts ReplicaOcc by rendering panoramic RGB-D observations from Replica assets while retaining camera trajectories and occupancy ground truth.The rendered sequences retain the camera centers and temporal sampling of their perspective counterparts.Source: A. Benchmark · p. 5
- Pan-Holo360D uses seven Holo360D indoor scenes with long, continuous panoramic RGB-D sequences, aligned poses, and realistic sensing noise and appearance variation.Source: A. Benchmark · p. 5
- Because Holo360D lacks suitable semantic annotations, labels from representative panoramic frames are lifted to voxels, consolidated across views, and manually inspected and corrected.The selected sequences span house-scale environments with multiple interconnected rooms.Source: A. Benchmark · p. 5
B. Experimental Setup
Evaluation uses occupancy IoU and semantic mIoU on two benchmarks, with shared-category comparisons and adapted perspective baselines.
B. Experimental Setup
- Experiments use one NVIDIA RTX 4090 GPU and an occupancy-mapping voxel resolution of 0.08m.Source: B. Experimental Setup · p. 5
- The main comparison uses occupancy IoU and semantic mIoU over eight shared categories; fine-grained annotations are retained for open-vocabulary visualization and supplementary evaluation.The shared categories are ceiling, floor, wall, window, chair, bed, sofa, and table.Source: B. Experimental Setup · p. 5
- Each panoramic frame is projected into twelve perspective views across four azimuths and three elevation levels, then processed as orientation-consistent sequences.Voxel predictions are combined by union, with semantic conflicts resolved through class voting for dense outputs or maximum class confidence for sparse outputs.Source: B. Experimental Setup · p. 5
C. Quantitative Comparison.
PanOVOcc improves semantic occupancy mapping on both benchmarks, while panoramic observations also accumulate occupied-scene coverage faster than perspective observations on a matched trajectory.
C. Quantitative Comparison.
- On Pan-Replica, PanOVOcc improves occupancy IoU by 20.03 points and mIoU by 7.06 points over FreeOcc.It reaches an occupancy IoU of 75.63 and a semantic mIoU of 36.99.Source: C. Quantitative Comparison · pp. 5–6
- On Pan-Holo360D, PanOVOcc improves occupancy IoU by 45.61 points and semantic mIoU by 24.71 points over FreeOcc.The evaluation includes eight shared categories and four additional open-vocabulary categories.Source: C. Quantitative Comparison · p. 6, Table III · p. 6
- PanOVOcc's gains are particularly pronounced on large structural categories, while chair and bed accuracy remains lower than several baselines.It also outperforms FreeOcc on several additional open-vocabulary categories.Source: C. Quantitative Comparison · p. 6
- At N=110, panoramic observations cover 94.29% of ground-truth occupied voxels, versus 67.84% for perspective observations.At the first camera location, coverage is 45.15% for panoramic and 8.24% for perspective observations.Source: C. Quantitative Comparison · p. 6
D. Ablation Studies
Ablations show complementary contributions from PanOVOcc's fusion, semantic consolidation, persistent-update components, and upstream pose estimation; the reported scores are protocol-specific.
D. Ablation Studies
- GSC raises semantic mIoU from 31.51 to 32.09 without changing occupancy IoU, while RPU further improves both metrics.The components are described as supporting geometric fusion, semantic consolidation, and persistent map maintenance, respectively.Source: D. Ablation Studies · p. 6
- Component and upstream-module effects are assessed within their matched ablation protocols, not by comparing their absolute scores with the main experiment.The ablations use different pose-update, map-maintenance, or replay settings from the default online pipeline.Source: D. Ablation Studies · pp. 6–7
- Removing PSVF produces the largest component-ablation drop, reducing IoU to 53.02 and mIoU to 23.51.The component ablation is summarized in Table IV.Source: D. Ablation Studies · p. 6, Table IV · p. 7
- ODGS-SLAM has the highest geometric IoU among estimated-pose frontends, with gains over OpenVSLAM and its dense fork on both benchmarks.The gains are 17.63 and 15.59 IoU points on Pan-Replica, and 7.29 and 23.97 points on Pan-Holo360D, respectively.Source: D. Ablation Studies · p. 7
- Replacing the semantic predictor leaves geometric IoU unchanged, separating the frontend's geometric-consistency role from semantic occupancy quality.The study uses controlled final-pose replay with fixed mapping frames and voxel coordinates.Source: D. Ablation Studies · p. 7
E. Qualitative Analysis
Qualitative comparisons show clearer geometry in PanOVOcc maps than in existing methods, especially in larger real-world multi-room examples.
E. Qualitative Analysis
- On Pan-Replica, PanOVOcc reconstructs clearer room boundaries and furniture layouts, while existing methods show occupancy noise and geometric distortion.Figure 5 presents qualitative comparisons on Pan-Replica and Pan-Holo360D.Source: E. Qualitative Analysis · p. 7
- In larger Pan-Holo360D examples, PanOVOcc better preserves sofa and floor layout and room partitions where several baselines produce fragmented or distorted layouts.The examples include a connecting area in a multi-room scene.Source: E. Qualitative Analysis · p. 7
- Preserved semantic targets and spatial context could support target localization and approach planning, while coherent room layouts could facilitate cross-room navigation.Source: E. Qualitative Analysis · p. 7
V. CONCLUSION
PanOVOcc combines panoramic SLAM, open-vocabulary perception, and long-term voxel memory to build continuously updated, language-queryable global maps; qualitative comparisons show stronger geometry, including at larger scale.
V. CONCLUSION
- PanOVOcc is a training-free framework for persistent open-vocabulary semantic occupancy mapping from native panoramic RGB-D sequences.It extends panoramic mapping from isolated local predictions to continuously updated, language-queryable global maps.Source: §V · p. 7
- The framework integrates panoramic SLAM and open-vocabulary perception with long-term spatial voxel memory.Experiments show improved geometric and semantic occupancy accuracy over evaluated baselines, especially on major structural categories.Source: §V · p. 7
EmbodiedOcc EmbodiedOcc++ VEOcc GPOcc FreeOcc Ours GT
- Qualitative comparisons report more accurate geometry than EmbodiedOcc [1], EmbodiedOcc++ [4], VEOcc [5], GPOcc [39], and FreeOcc [2], including in larger environments.The figure removes top-layer voxels to reduce occlusion.Source: Figure 5 · p. 8