PaperlayerView exact revision on arXiv

Source-linked AI summary

Spheriverse: 3D Scene Understanding from Spherical Observations in the Wild

Fei Teng, Sheng Wu, Mengfei Duan, Guoqiang Zhao, Junhui Ma, Kai Luo, Siyu Li, Hao Shi, Zhiyong Li, Kailun Yang

arXiv:2609.09012v2cs.CVcs.ROeess.IV

TL;DR

Spherical observations encode visual information angularly while 3D environments use Cartesian coordinates, leaving a cross-space representation gap for scene understanding. The paper introduces Spheriverse and benchmarks spherical 3D tasks, and proposes SphereOcc for occupancy prediction. SphereOcc reaches 13.91% mIoU and 24.65% GeoIoU, ranking first across all five scene categories.

  • Problem

    Spherical observations encode visual information angularly while 3D environments use Cartesian coordinates, complicating geometric correspondence and semantic evidence aggregation.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces Spheriverse and unified benchmarks for three spherical 3D tasks, and proposes SphereOcc, which combines spherical geometry with semantic evidence retrieval for occupancy prediction.

    Source: §7 · p. 14

  • Results

    13.91% mIoU and 24.65% GeoIoU: SphereOcc improves the best prior results by 1.70 and 2.10 percentage points, respectively, and ranks first across all five scene categories.

    Source: §1 · p. 2, §7 · p. 14

  • Takeaways & Limitations

    Spheriverse, its benchmarks, and SphereOcc provide a unified foundation for studying 3D perception from spherical observations.

    Source: §7 · p. 14

    SphereOcc omits spherical elevation from its geometric cue because elevation is determined by retained height and range and is treated as redundant.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

Spherical observations provide global visual context for 3D scene understanding. However, visual information is encoded in an angular domain, whereas the physical world is represented in Cartesian coordinates. This cross-space representation gap complicates geometric correspondence and semantic evidence aggregation. To delve into this challenge, we introduce Spheriverse, comprising 64,400 temporally aligned spherical image-LiDAR pairs organized into 644 sequences. The dataset spans diverse scenes, illumination, and weather conditions, with fine-grained semantic classes. We further establish benchmarks for semantic occupancy prediction, semantic mapping, and 3D object detection, evaluating 30+ methods through overall and scene-wise comparisons. For dense prediction, we propose SphereOcc, an occupancy framework that couples spherical geometry modeling with semantic evidence retrieval. Cartesian-Spherical Representation Remodeling (CSRR) incorporates spherical range-azimuth geometry into Cartesian voxel features through region-wise modulation. Spherical Evidence Re-querying (SER) then conditions queries on voxel content and range-height-azimuth geometry to adaptively retrieve relevant semantic evidence from source spherical image features. SphereOcc achieves 13.91% mIoU and 24.65% GeoIoU, yielding relative improvements of 13.9% and 9.3% over the respective best-performing methods, TPVFormer and SurroundOcc. It also ranks first in both metrics across all five scene categories, with consistent advantages across the evaluated spatial partitions and reduced fields of view. The established benchmark and source code will be available at https://feit-feiteng.github.io/Spheriverse.

1 INTRODUCTION

Spherical observations offer broad visual context, but their angular representation does not align directly with Cartesian voxels, complicating geometric correspondence and semantic evidence aggregation. The paper introduces Spheriverse and benchmarks for spherical 3D understanding, then proposes SphereOcc for semantic occupancy prediction.

1 INTRODUCTION

  • Spherical images encode observations angularly, while 3D environments and voxel representations use Cartesian coordinates, so image-space proximity does not preserve metric neighborhoods directly.The non-uniform mapping can distort object appearance, geometry, and spatial relationships.Source: §1 · pp. 1–2, Figure 2 · p. 2
  • Spheriverse contains 644 real-world sequences across 13 regions, with varied times, weather, environments, fine-grained 3D annotations, and high-quality camera and LiDAR sensing.The cameras provide 360° horizontal and 136.7° vertical coverage, alongside a 128-beam LiDAR.Source: §1 · p. 2
  • The benchmark evaluates more than 30 methods across semantic occupancy prediction, semantic mapping, and 3D object detection, with overall and scene-wise analyses.The benchmark reports existing approaches under spherical observations.Source: §1 · p. 2
  • SphereOcc integrates spherical geometry and semantic evidence into voxel representations through a geometry–semantics coupled cross-space mechanism.The introduction identifies semantic occupancy prediction as its dense 3D modeling task.Source: §1 · p. 2
  • 13.91% mIoU and 24.65% GeoIoU: SphereOcc surpasses the strongest prior results by 1.70 and 2.10 percentage points, respectively, and ranks first across all five scene categories.Region-wise and reduced-field-of-view evaluations also show consistent performance across spatial variations and observation coverage.Source: §1 · p. 2

2 RELATED WORK

Prior work studies spherical vision and Cartesian 3D perception largely in separate settings. The related work reviews spherical scene understanding, occupancy, mapping, and detection, identifying spherical 3D spatial understanding as an open evaluation setting.

2 RELATED WORK

  • The related work organizes prior studies into spherical scene understanding, voxel-based scene understanding, and semantic mapping and 3D object detection.It focuses on voxel-based scene understanding because of its completeness in scene representation.Source: §2 · p. 3

2.1 Spherical Scene Understanding

  • Spherical scene understanding spans segmentation, open-vocabulary recognition, optical flow, 2D detection and tracking, and depth estimation, while lifting semantic representations into Cartesian 3D remains largely unexplored.Existing spherical datasets often use multi-camera systems with constrained or discontinuous coverage; annular cameras have limited vertical fields of view, and Matterport3D mainly covers static indoor scenes.Source: §2.1 · p. 3

2.2 Semantic Occupancy Prediction

  • Camera-based occupancy methods use projection, learnable queries, or Gaussian primitives, but do not explicitly address spherical-to-Cartesian representation gaps; annular-camera occupancy is limited to approximately 45° vertical coverage.LiDAR-assisted methods provide geometric cues but add sensor payload, power, and onboard computation.Source: §2.2 · p. 4

2.3 Semantic Mapping and 3D Object Detection

  • Semantic mapping methods use rasterized grids or vectorized structures, while 3D detection methods project image features or aggregate them with queries; these approaches are designed for perspective imaging.The paper establishes semantic mapping and 3D detection benchmarks under spherical inputs to characterize existing methods in that setting.Source: §2.3 · p. 4

3 SPHERIVERSE DATASET

Spheriverse is a real-world spherical image–LiDAR dataset with diverse annotated driving scenes, organized into 64,400 synchronized pairs across 644 sequences. Its statistics describe temporal coverage and spatial variation in semantic class composition.

3 SPHERIVERSE DATASET

  • Spheriverse covers 644 sequences from 13 geographically and visually diverse regions, using a spherical camera and 128-beam LiDAR for dense observations.The dataset includes rich semantic annotations for spherical 3D perception.Source: §3 · p. 4

3.1 Data Acquisition and Processing

  • Sequences are grouped into five major and thirteen fine-grained scene categories, with manual semantic labeling of point clouds.The categories reflect traffic characteristics, environmental context, and geometric structure.Source: §3.1 · p. 4, Figure 3 · p. 5

3.2 Dataset Statistics

  • 64,400 synchronized image–LiDAR pairs form 644 sequences, each spanning 20 seconds at 5 Hz.The benchmark subset follows temporal alignment and data filtering of 89,674 raw spherical images.Source: §3.2 · p. 5
  • The dataset spans night, dawn, daytime, and evening, with daytime comprising 48.29% and evening 29.97% of samples.Night and dawn account for 11.34% and 10.40%, respectively.Source: §3.2 · p. 5
  • Its three-level semantic hierarchy covers traffic participants, built structures, and natural environments, with class proportions varying across azimuth and range.Vehicles decrease from 15.78% to 0.60% across radial bins, while vegetation increases from 22.74% to 47.95%.Source: §3.2 · pp. 5–6, Figure 4 · p. 6

4 SPHEREOCC: PROPOSED METHOD

SphereOcc addresses the cross-space gap between spherical observations and Cartesian voxels by coupling spherical geometry with semantic evidence. Its CSRR remodels voxel features with range–azimuth geometry, while SER retrieves relevant evidence from spherical image features.

4.1 Problem Formulation

  • Occupancy prediction recovers a voxelized 3D semantic representation from an ERP image, with voxels representing local units of real-world space.The paper frames SphereOcc as a response to the gap between angular observations and metric voxels.Source: §4.1 · p. 6, §4 · p. 6

4.2 Overview of SphereOcc

  • SphereOcc lifts multi-scale spherical image features into Cartesian voxels, then applies CSRR and SER before decoding occupancy predictions.The four decoder outputs are supervised, but only the high-resolution output is used at inference.Source: Figure 5 · p. 7, §4.2 · p. 7

4.3 Cartesian-Spherical Representation Remodeling

  • CSRR embeds sphere-aligned range–azimuth relations into Cartesian voxel features, using horizontal range and azimuth as geometric cues.Spherical elevation is omitted because it is determined by retained height and horizontal range.Source: §4.3 · p. 7
  • CSRR uses a spatially gated channel residual: regional importance controls where updates apply, while a learnable channel residual controls the adjustment.The residual is initialized to zero, preserving the original voxel features at initialization.Source: §4.3 · p. 8

4.4 Spherical Evidence Re-querying

  • SER addresses the missing adaptive semantic correspondence by re-querying source spherical features for relevant evidence for each voxel feature.It conditions queries on voxel content and range-height-azimuth geometry, predicting learned sampling offsets and attention weights.Source: Figure 5 · p. 7, §4.4 · pp. 8–9

5 BENCHMARK EXPERIMENTS

The benchmarks compare spherical perception methods under unified evaluation, covering semantic occupancy, semantic mapping, and 3D object detection. SphereOcc leads occupancy results, while the strongest mapping and detection methods vary by task and scene.

5.1.1 Evaluation Metrics and Implementation Details

  • All methods follow a unified protocol with standardized training schedule, input size, and voxelization while retaining their core representations and task-specific losses.Occupancy is assessed with class-wise IoU, mIoU, and GeoIoU; semantic mapping uses mIoU, and 3D detection uses mAP and NDS.Source: §5 · pp. 9–10, §5.1.1 · p. 10, §5.2 · p. 11

5.2 Semantic Mapping and 3D Object Detection

  • 22.66% mIoU makes OneBEV the top semantic-mapping method, 1.95 percentage points ahead of HDMapNet, and it ranks first in all five scene subsets.Its largest advantage is in structurally constrained scenes: 23.15% versus 20.28%.Source: Table 4 · p. 10, §5.2 · p. 12

5.1.1 Evaluation Metrics and Implementation Details

  • 13.91% mIoU and 24.65% GeoIoU make SphereOcc the overall occupancy leader, with first place in both metrics across all five scene subsets.It also achieves the best class-wise IoU in seven of nine categories; Person and Bike improve over SurroundOcc despite not ranking first.Source: §5.1.1 · p. 10

5.2 Semantic Mapping and 3D Object Detection

  • SparseBEV leads overall 3D detection with 0.1689 mAP and 0.1221 NDS, while PD-BEV leads in functional scenes.SparseBEV leads in expressway, rural, structurally constrained, and urban scenes; detection performance varies substantially across the five scene categories.Source: Table 5 · p. 11, §5.2 · p. 12

6 ABLATION STUDIES

SphereOcc’s components, region-wise results, and reduced-field-of-view evaluations examine how its occupancy performance holds across spatial partitions and incomplete observations. The dataset also includes challenging conditions that aggregate statistics do not fully characterize.

6.1 Main Components

  • 13.91% mIoU and 24.65% GeoIoU are reached after adding SER to the InternImage-T-plus-CSRR baseline, with SER providing the largest incremental improvement.With the same ResNet backbone, adding CSRR and SER raises mIoU from 12.05% to 13.17% and GeoIoU from 22.55% to 23.85%.Source: §6.1 · pp. 12–13

6.2 Angular and Vertical Region Analysis.

  • SphereOcc achieves the highest mIoU and GeoIoU in every horizontal azimuth interval and vertical voxel-height partition.Gains are particularly clear at 135°–180° and 180°–225° azimuth intervals and in the upper-middle and lower-middle height layers.Source: Table 7 · p. 13, §6.2 · p. 13

6.3 Robustness under Reduced Spherical FoVs

  • 8.98% mIoU and 17.42% GeoIoU at 120° horizontal FoV exceed the corresponding second-best results of 8.34% and 17.09%.With vertical FoV reduced from 136.70° to 76.70°, mIoU changes from 13.91% to 13.80% and GeoIoU from 24.65% to 24.48%.Source: Table 8 · p. 13, §6.3 · p. 14

6.4 Representative Challenging Scenarios in Dataset

  • Spheriverse includes camouflage, uncommon rural content, complex spatial layouts, low illumination, exposure variation, and lens contamination, which can occur together.These scenarios create semantic ambiguity and spatially non-uniform image degradation beyond conditions captured by aggregate statistics.Source: Figure 8 · p. 14, §6.4 · p. 14

7 CONCLUSION

Spheriverse and its benchmarks support study of spherical 3D perception, while SphereOcc addresses the gap between angular observations and Cartesian voxels. SphereOcc reports leading occupancy results across scene categories, spatial partitions, and reduced fields of view.

B.2 Scene Taxonomy

  • The unified benchmarks cover semantic occupancy prediction, semantic mapping, and 3D object detection, with overall and scene-wise comparisons of more than 30 methods.The scene taxonomy groups thirteen fine-grained categories into five major categories.Source: §7 · p. 14, §B.2 · p. 17

7 CONCLUSION

  • 13.91% mIoU and 24.65% GeoIoU are SphereOcc’s occupancy results, with first place in both metrics across all five scene categories.The conclusion reports advantages across spatial partitions and reduced fields of view, supporting robustness to directional variation and incomplete observations.Source: §7 · p. 14

APPENDIX A MOTIVATION

  • SphereOcc uses CSRR to embed spherical range–azimuth geometry in Cartesian voxel features and SER to retrieve semantic evidence using voxel states and spherical position.The conclusion presents Spheriverse, its benchmarks, and SphereOcc as a unified foundation for studying 3D perception from spherical observations.Source: §7 · p. 14, Appendix A · p. 17

7 CONCLUSION

  • Future work will connect spherical observations and dense 3D representations to safety-critical decisions, language-grounded spatial reasoning, and trajectory planning.The stated direction is to evaluate perception from 3D scene understanding through physical action.Source: §7 · p. 14

B.1 Acquisition and Scene-Level Annotation

  • Spheriverse contains 64,400 synchronized image–LiDAR pairs in 644 sequences, collected across 13 diverse regions and varied scene, time-of-day, and weather conditions.The dataset has a three-level semantic annotation hierarchy.Source: §7 · p. 14, §B.1 · p. 17

B.3 Sequence-Level Embedding Analysis

The appendix evaluates whether images remain visually consistent within sequences and differ across scene categories, then documents the dataset splits, annotations, evaluation settings, metrics, and SphereOcc’s geometry-driven design.

B.3 Sequence-Level Embedding Analysis

  • Intra-sequence image distances range from 3.8 to 6.9, while inter-category distances range from 18.3 to 28.6, so even the closest category pair exceeds all reported within-sequence distances.Rural and Structural have the largest inter-category distance (28.6); Functional and Expressway have the smallest (18.3).Source: §B.3 · p. 18

B.4 Data Splits

  • The data split contains 457 training, 134 evaluation, and 53 test sequences, with 100 temporally continuous frames per sequence.The evaluation split is used for benchmark results in the main paper.Source: §B.4 · p. 19, Table 10 · p. 19

B.5 Semantic Annotation Hierarchy

  • The semantic hierarchy maps fine-grained attributes to intermediate and unified classes; occupancy evaluation uses nine unified classes.The taxonomy also distinguishes detailed categories such as construction vehicles, temporary buildings, curbs, and gravel.Source: §B.5 · p. 19

C.1 Training and Evaluation Settings

  • Baselines use standardized image resolution, training schedule, and output discretization while retaining their core representations and task-specific losses.Occupancy uses a 200 × 200 × 16 voxel grid at 0.5 m resolution; mapping uses a 200 × 200 grid.Source: §C.1 · p. 19

C.2 Metrics

  • Occupancy metrics include class-wise IoU, mIoU, and GeoIoU, while mapping uses seven classes and occupancy uses nine classes for mIoU.GeoIoU measures class-agnostic overlap between predicted and ground-truth occupied voxel sets.Source: §C.2 · p. 19

C.3.1 Clarification of Motivation.

  • CSRR and SER explicitly encode spherical geometry in voxel features and queries, contrasting with initial lifting’s use of projected image coordinates as reference anchors.The figure visualizes CSRR feature changes at fine, middle, and coarse scales; its circular direction denotes azimuth and its radial direction denotes horizontal distance.Source: §C.3.1 · pp. 20–21, Figure 10 · p. 21

C.3.2 Ablation for CSRR

The CSRR visualization shows range-dependent changes to voxel features, with stronger responses farther from the ego vehicle. These changes reflect geometry-conditioned remodeling that enriches Cartesian voxel features with spherical cues.

C.3.2 Ablation for CSRR

  • The top-down response map uses azimuth around the ego vehicle and radial distance from its vertical axis to locate voxel-feature changes.The circular direction denotes 0°–360° azimuth, and the radial direction denotes horizontal distance ρ = √(x² + y²).Source: §C.3.2 · p. 21
  • CSRR responses strengthen from near to far regions, showing that voxel-feature remodeling varies with range rather than applying a uniform update.The response map averages feature-change magnitudes over 16 vertical voxel layers.Source: §C.3.2 · p. 21
  • The observed remodeling jointly encodes Cartesian coordinates (x,y,z) and spherical cues (sin θ, cos θ,ρ) in voxel features.This geometry-conditioned update adapts voxel representations to spherical observation geometry.Source: §C.3.2 · p. 21

C.3.3 Ablation for SER

SER re-queries source image features using a spherical geometric descriptor, which improves occupancy metrics over Cartesian XYZ re-querying. Combined with CSRR, it contributes complementary gains in semantic evidence retrieval and voxel geometry remodeling.

C.3.3 Ablation for SER

  • Replacing Cartesian XYZ with the spherical descriptor raises mIoU from 12.38% to 12.49% and GeoIoU from 23.09% to 23.16%.The comparison uses the same SurroundOcc baseline.Source: §C.3.3 · p. 21
  • With CSRR added, SER reaches 13.17% mIoU and 23.85% GeoIoU, indicating complementary effects from re-querying and voxel remodeling.CSRR enriches lifted voxel features with range–azimuth geometry; SER retrieves additional semantic cues from source image features.Source: §C.3.3 · p. 21
  • The spherical descriptor uses range, height, and azimuth information as a more suitable geometric prior for spherical feature retrieval than Cartesian coordinates alone.The descriptor is [ρℓ,n, zℓ,n, sin θℓ,n, cos θℓ,n].Source: §C.3.3 · p. 21

D.1 Robustness under Visual Degradations

SphereOcc remains ahead of compared baselines on clean and synthetically corrupted spherical observations. Spheriverse also offers broader vertical coverage and richer sensing and scene annotations than panoramic annular datasets.

D.1 Robustness under Visual Degradations

  • The robustness evaluation applies Gaussian noise, large raindrops, sunlight/lens flare, and motion blur to validation panoramas, using identical corrupted observations for all methods.The perturbations include both local visibility degradation and global appearance shifts.Source: §D.1 · p. 21
  • Fig. 11 presents one representative spherical image per column, with rows showing original input, Gaussian noise, sunlight/lens flare, raindrops, and motion blur.Source: Figure 11 · p. 22

D.2 Comparison of Panoramic Annular Datasets

  • Spheriverse provides 136.7° vertical field of view, compared with 45° for previous panoramic annular works, capturing broader ground-level, surrounding, and elevated regions.The red dashed box in Fig. 12 marks the annular dataset’s perceptual range mapped onto a Spheriverse panorama.Source: Figure 12 · p. 22, §D.2 · p. 23

D.1 Robustness under Visual Degradations

  • On clean images, SphereOcc raises mIoU from 12.21% to 13.91% and GeoIoU from 22.55% to 24.65%.It also outperforms the best baseline under each tested corruption; under motion blur, it improves mIoU from 6.48% to 7.71% and GeoIoU from 15.35% to 17.14%.Source: §D.1 · p. 22, Table 13 · p. 23

D.2 Comparison of Panoramic Annular Datasets

  • Spheriverse combines 33 annotated classes across 13 fine-grained scene types with high-quality spherical cameras and a 128-beam LiDAR.The dataset comparison also notes clearer imagery under diverse illumination and dense geometric supervision.Source: §D.2 · p. 23

D.3 Efficiency Analysis

CSRR and SER add few parameters while improving semantic occupancy prediction. With a backbone replacement, the complete model also reduces parameters relative to the original baseline.

D.3 Efficiency Analysis

  • CSRR and SER add 0.001M and 0.327M parameters, respectively, increasing the InternImage-T configuration’s parameter count by approximately 0.45%.Source: §D.3 · p. 23
  • With these modules, mIoU rises from 12.14% to 13.91% and GeoIoU from 23.35% to 24.65%.The passage characterizes these gains as improved semantic occupancy prediction with small parameter overhead.Source: §D.3 · p. 23
  • With the backbone replacement, the complete model uses 18.43% fewer parameters than the original baseline while gaining 1.86 percentage points in mIoU and 2.10 percentage points in GeoIoU.Source: §D.3 · p. 23

D.4 More Qualitative Results

Additional visualizations illustrate SphereOcc’s occupancy predictions on rare structures and contaminated scenes, while mapping and detection examples expose remaining challenges. Appendix E describes potential applications of spherical perception in embodied systems, mapping, and spatial services.

D.4 More Qualitative Results

  • SphereOcc preserves an elevated bridge’s occupancy geometry and produces a more faithful layout in a contaminated nighttime scene than compared methods.The compared methods tend to label the bridge as Pillar or over-predict Vegetation or Vehicle in the nighttime example.Source: §D.4 · p. 23
  • Nighttime glare weakens far-field texture and boundary cues, and existing methods produce fragmented, truncated, or locally overextended road predictions.These errors are especially pronounced along viewing directions dominated by intense glare.Source: §D.4 · p. 24
  • Spherical 3D detection examples show remaining missed detections, absolute position errors, and orientation deviations.The examples compare ground truth and representative method predictions across enlarged regions of diverse traffic scenes.Source: §D.4 · pp. 24–25

APPENDIX E SOCIAL IMPACT

  • Spherical observations cover front, side, and rear regions and provide surrounding behavioral cues relevant to social interaction and safety-critical perception.The text notes examples including pedestrians approaching from the side or rear, merging traffic, blind spots, and roadside obstacles.Source: Appendix E · pp. 25–26
  • Spherical perception is described as useful for cross-view localization, relocalization, map construction, navigation, inspection, and city-scale spatial understanding.The paper also presents Spheriverse as a resource for future spherical spatial-intelligence tasks beyond occupancy prediction.Source: Appendix E · p. 26
  • Spherical perception may provide broader environmental context for low-altitude autonomy, UAV inspection, and aerial–ground collaborative perception.The text describes potential use in spatial modeling, object discovery, and path understanding in complex scenes.Source: Appendix E · p. 26

Related papers

Loading 2609.09012v2…