PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentation

1. Introduction

• Existing perception tasks focus on specific scene aspects, while PanoOcc targets a unified representation integrating object detection and semantic segmentation.Detection emphasizes foreground objects, BEV semantic segmentation predicts only a BEV map, and vector map construction focuses on static road structure.Source: §1 · p. 2
• Camera-based panoptic segmentation uses multi-view images to produce dense panoptic occupancy predictions throughout the scene.This differs from sparse LiDAR panoptic prediction and from camera detection and segmentation methods that use separate models.Source: Figure 1 · p. 1, §1 · p. 2
• Directly using BEV features can omit shape and height details; occupancy represents open-set and irregular objects, special road structures, and scene semantics.Prior occupancy methods can be memory-inefficient when lifting 2D features to 3D, and tri-plane representations may not capture complete 3D information or instance-level distinctions.Source: §1 · p. 2
• PanoOcc uses voxel queries to learn a unified occupancy representation from multi-frame, multi-view images in a coarse-to-fine scheme.An occupancy sparsify module progressively prunes occupancy during upsampling to improve memory efficiency.Source: §1 · p. 2
• PanoOcc achieves state-of-the-art camera-based semantic and panoptic segmentation on nuScenes and shows promising performance on Occ3D dense occupancy prediction.The method is presented as extendable to dense occupancy prediction.Source: §1 · p. 2

Camera-based 3D Perception

• Camera-based 3D perception methods use image-to-BEV lifting or learnable queries, but compressing information onto the BEV plane may sacrifice integral 3D scene structure.Voxel representation is presented as better suited to holistic 3D understanding and tasks such as semantic and panoptic segmentation.Source: §2 · p. 2

3D Occupancy Prediction

• Camera occupancy prediction includes sparse methods supervised by LiDAR and dense methods related to semantic scene completion, with benchmarks including OpenOccupancy and Occ3D.Examples include TPVFormer for tri-perspective occupancy, MonoScene for dense occupancy from a monocular image, and VoxFormer for a two-stage voxel-query framework.Source: §2 · p. 3

LiDAR Panoptic Segmentation

• LiDAR panoptic segmentation unifies semantic segmentation and object detection, while camera-based panoptic segmentation targets dense voxel segmentation rather than sparse point predictions.Recent LiDAR methods build on semantic segmentation networks to address limitations in fine-grained 3D feature representations.Source: §2 · p. 3

3D Scene Reconstruction and Representation

• Explicit scene reconstruction uses multi-view geometry but can be computationally expensive and struggle with dynamic or interacting scenes; implicit representations learn continuous scene functions.The passage describes implicit representations as more computation-efficient and capable of representing hidden or occluded regions.Source: §2 · p. 3

3.1. Problem Setup

• Camera-based 3D panoptic segmentation predicts a dense voxel volume with semantic labels and instance IDs from current multi-view images and previous frames.Semantic occupancy prediction is a subproblem that predicts voxel semantics, including whether each voxel is empty or occupied.Source: §3.1 · p. 3

3.2. Overall Architecture

• PanoOcc processes multi-frame, multi-view image features into coarse occupancy, upsamples them, then uses separate detection and segmentation heads and a refinement module to assign thing-voxel instance IDs.The refinement module uses 3D detection results to refine thing-class predictions.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Voxel Queries

• Voxel queries are learnable parameters shaped to the 3D grid, with each query responsible for one voxel cell and augmented by learnable 3D positional embeddings.Each voxel cell corresponds to a real-world region sized by the grid's spatial dimensions.Source: §3.3 · p. 4

3.4. Occupancy Encoder

• The occupancy encoder uses deformable voxel attention to combine image features with voxel queries, then aligns historical voxel queries in 3D and fuses them with current queries.Historical features are aligned by interpolation sampling and fused with current queries using residual 3D convolution.Source: §3.4 · pp. 4–5

3.5. Occupancy Decoder

• Coarse-to-fine 3D deconvolutions recover high-resolution occupancy features, while optional occupancy sparsification prunes empty positions and uses sparse convolutions during upsampling.The method prunes after upsampling to maintain spatial sparsity.Source: §3.5 · p. 5

3.6. Multi-task Training

• Multi-task training selects foreground voxel features for detection and predicts segmentation over occupancy features; detection-guided refinement assigns instance IDs and improves segmentation consistency.The segmentation loss combines focal loss over all voxels and Lovasz loss for voxels containing LiDAR points; the detection loss uses focal classification and L1 box regression. The refinement module assigns thing-class predictions within high-confidence boxes and ignores excessive instance overlaps.Source: §3.6 · pp. 5–6

4.1. Datasets

• nuScenes supplies 1,000 scenes for detection and segmentation, while Occ3D-nuScenes provides 700 training and 150 validation scenes for dense occupancy evaluation.nuScenes samples include images from six cameras; Occ3D occupancy labels use 17 semantic categories.Source: §4.1 · p. 6

4.3. Main Results

• 70.7 mIoU puts PanoOcc-Base 1.8 points above TPVFormer-Base; PanoOcc-Large-T reaches 74.5 mIoU, PanoOcc posts 62.1 PQ, and occupancy prediction sets a new state of the art.The panoptic comparison is against LiDAR-based methods; the paper attributes its remaining performance gap to lower detection performance.Source: Table 1 · p. 7, Table 2 · p. 7, §4.3 · p. 8

4.2. Experimental Settings

• The experiments use four-frame temporal fusion, 24-epoch training, object-level detection labels, and sparse LiDAR point labels to supervise voxel segmentation.The study evaluates detection with mAP and NDS, semantic segmentation with mIoU, and panoptic segmentation with PQ.Source: §4.2 · pp. 7–8

4.4. Ablation

• Ablations favor height-aware voxel queries, coarse-to-fine over direct high-resolution querying, voxel self-attention and temporal fusion, joint detection-segmentation, and Lovasz loss.The coarse-to-fine design uses nearly half the memory of direct 200x200x8 queries; removing Lovasz lowers mIoU from 65.6 to 59.6.Source: §4.4 · pp. 8–10

4.5. Discussion

• Coarse-to-fine voxels use less memory and infer faster than TPVFormer despite detection; temporal fusion raises motorcycle and trailer mIoU by 11.7 and 8.2.The paper reports that motorcycle and trailer categories are typically affected by occlusion.Source: §4.5 · p. 11, Table 13 · p. 11, Table 10 · p. 11

4.6. Visualization

• Qualitative visualizations show nuScenes semantic and panoptic voxel predictions at 200x200x32 and Occ3D dense occupancy predictions at 200x200x16.The nuScenes predictions are assigned to LiDAR points.Source: Figure 4 · p. 10, Figure 5 · p. 10, §4.6 · p. 11

• PanoOcc frames camera-based 3D panoptic segmentation around a unified occupancy representation for comprehensive scene understanding.Source: §5 · p. 11
• Voxel queries incorporate multi-frame and multi-view image information in a coarse-to-fine scheme for occupancy learning.Source: §5 · p. 11
• Experiments on nuScenes and Occ3D-nuScenes demonstrate PanoOcc’s effectiveness and its potential to advance holistic 3D scene understanding.Source: §5 · p. 11