Source-linked AI summary
SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction
Pin Tang, Zhongdao Wang, Guoqing Wang, Jilai Zheng, Xiangxuan Ren, Bailan Feng, Chao Ma
TL;DR
Dense 3D representations are costly, while BEV and TPV projections lose information relevant to semantic occupancy. SparseOcc uses a sparse latent representation with sparse completion, a feature pyramid, and a sparse transformer head. It reports a 74.9% FLOPs reduction versus C-CONet alongside higher semantic occupancy mIoU.
Problem
Dense 3D latent spaces are computationally costly, while efficient BEV and TPV projections lose geometry information relevant to fine-grained semantic occupancy prediction.
Source: §1 · pp. 1–2Method
SparseOcc uses COO-format sparse 3D layers, including a latent diffuser, a feature pyramid with sparse interpolation, and a sparse transformer head.
Source: §1 · p. 2Results
SparseOcc reduces FLOPs by 74.9% versus C-CONet and improves semantic occupancy accuracy from 12.8% to 14.1% mIoU.
Source: §1 · p. 2, §4.2 · p. 7Takeaways & Limitations
The paper attributes part of the accuracy improvement to sparse representation avoiding hallucinations on empty voxels.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Vision-based perception for autonomous driving requires an explicit modeling of a 3D space, where 2D latent representations are mapped and subsequent 3D operators are applied. However, operating on dense latent spaces introduces a cubic time and space complexity, which limits scalability in terms of perception range or spatial resolution. Existing approaches compress the dense representation using projections like Bird's Eye View (BEV) or Tri-Perspective View (TPV). Although efficient, these projections result in information loss, especially for tasks like semantic occupancy prediction. To address this, we propose SparseOcc, an efficient occupancy network inspired by sparse point cloud processing. It utilizes a lossless sparse latent representation with three key innovations. Firstly, a 3D sparse diffuser performs latent completion using spatially decomposed 3D sparse convolutional kernels. Secondly, a feature pyramid and sparse interpolation enhance scales with information from others. Finally, the transformer head is redesigned as a sparse variant. SparseOcc achieves a remarkable 74.9% reduction on FLOPs over the dense baseline. Interestingly, it also improves accuracy, from 12.8% to 14.1% mIOU, which in part can be attributed to the sparse representation's ability to avoid hallucinations on empty voxels.
1. Introduction
Vision-based semantic occupancy prediction needs a 3D representation, but dense volumes are inefficient and BEV/TPV projections lose scene information. SparseOcc uses sparse 3D components and reports lower computation alongside improved occupancy accuracy.
1. Introduction
- Dense representations are redundant when much of the 3D space is empty, while BEV and TPV projections lose geometry information or degrade perception accuracy.The passage estimates that approximately 67% of the considered 3D space is empty.Source: §1 · p. 2
- SparseOcc stores 3D latent features in COO format and uses sparse layers, including a diffuser, feature pyramid, and transformer head.The diffuser propagates non-empty features to adjacent empty regions; sparse interpolation transfers information across scales.Source: §1 · p. 2
- SparseOcc reduces FLOPs by 59.8% to 74.9% and memory usage by 31.6% to 40.9% versus existing dense- or TPV-representation approaches.The reported benchmark is nuScenes-Occupancy.Source: §1 · p. 2
- SparseOcc surpasses state-of-the-art C-CONet, raising semantic occupancy accuracy from 12.8% to 14.1% mIoU; on nuScenes-Occupancy, it reduces FLOPs of existing dense- or TPV-representation approaches by 59.8% to 74.9%.The sparse representation naturally avoids hallucinations on empty voxels.Source: §1 · p. 2
2. Related Work
Existing 3D scene representations trade computational efficiency against the fine-grained geometry needed for semantic occupancy. Sparse scene completion instead diffuses features using sparse operators.
2.1. 3D Scene Representation
- Dense 3D operators have cubic time and space complexity, while BEV and TPV representations trade efficiency against fine-grained scene geometry and occupancy performance.BEV compresses 3D features to a plane; TPV uses three perpendicular planes but still has difficulty capturing complex driving scenes.Source: §2.1 · p. 2
- The paper seeks an effective and efficient sparse representation for semantic occupancy prediction.Source: §2.1 · p. 2
2.2. Scene Completion in Occupancy Prediction
- Sparse latent diffusion offers an alternative to prior completion methods that stack dense 3D operators to propagate features into surrounding empty regions.Lifted 3D representations cover only the initial intersection surfaces between sensor rays and objects.Source: §2.2 · p. 3
3. Proposed Approach
SparseOcc converts lifted image features into a COO sparse 3D latent space and builds sparse completion, multiscale fusion, and occupancy-decoding blocks on it. The design avoids redundant computation on empty voxels while completing the scene and predicting semantic occupancy.
3.1. Preliminary: Sparse Representation
- Gathering non-empty voxels into COO form lets subsequent operations avoid redundant computation on empty voxels.The sparse tensor stores each voxel's coordinates and features.Source: §3.1 · p. 3
3.2. Sparse Latent Diffuser
- One-layer sparse completion diffuses features, while submanifold aggregation preserves active sites; orthogonal kernel decomposition supports directional diffusion.Decomposed kernels reduce complexity from O(k^3) to O(3k^2) or O(4k^2), although the actual cost is not reduced for k = 3.Source: §3.2 · pp. 3–4
3.3. Sparse Feature Pyramid
- A downsampled sparse pyramid expands receptive field; weighted interpolation fuses scales, enriching semantics and supplying high-resolution features with completion from denser coarse scales.The last two scales omit the height dimension and use 2D completion and contextual aggregation blocks.Source: §3.3 · pp. 4–5
3.4. Sparse Transformer Head
- The sparse head filters predicted-empty voxels, represents all empty space with one learnable token, and decodes occupied masks only over retained voxels before scatter reconstruction.Mask prediction and reconstruction have complexity O(N_lN_qC + HWZ).Source: §3.4 · p. 5
3.5. Objective Function
- The objective sums Hungarian-matched mask and class losses, LSS depth supervision, and segmentation supervision for non-empty voxels.The segmentation loss supervises coarse binary classification on non-empty voxels.Source: §3.5 · p. 5
4. Experiments
Experiments evaluate SparseOcc on nuScenes-Occupancy and SemanticKITTI, comparing benchmark performance, qualitative predictions, and design ablations. The results show strong nuScenes-Occupancy performance and efficiency, comparable SemanticKITTI performance, and ablation evidence about the sparse completion, decoder, head, and input-size choices.
4.2. Benchmark Performance
- On nuScenes-Occupancy, SparseOcc beats 3D dense C-CONet by 1.7% geometry IoU and 1.3% semantic mIoU, reducing FLOPs 74.9%; on SemanticKITTI, it achieves comparable performance to OccFormer at approximately 44.2% of its FLOPs.Source: §4.2 · p. 7
4.1. Experimental Setup
- Evaluation covers nuScenes-Occupancy and SemanticKITTI, reporting geometry IoU, semantic mIoU, and efficiency measures including FLOPs.The setup also describes each dataset’s validation split and the image encoders used.Source: §4.1 · pp. 5–6
4.3. Qualitative Evaluation
- In nuScenes-Occupancy scenes, SparseOcc better completes flat roads, captures vegetation structure, and reconstructs distant cars than C-CONet.Figure 4 presents predictions alongside input images and ground truth.Source: §4.3 · p. 7, Figure 4 · p. 8
4.4. Ablation Studies
- Ablations favor decomposed orthogonal kernels over regular 3×3×3 kernels and interpolation-and-summation decoding over FPN3D.Sparse completion remains satisfactory without its completion block, and stacking more convolution blocks does not improve performance. The sparse voxel decoder exceeds FPN3D by 2.1% IoU and 3.3% mIoU, with slightly more memory; six-layer MSDeformAttn performs better, but reduced-layer variants lag while using more GPU memory and FLOPs.Source: §4.4 · p. 7, Table 4 · p. 7
- The sparse transformer head trades off geometry supervision and mask prediction, while higher image resolution raises mIoU but can reduce SparseOcc’s IoU through empty-voxel hallucination.The linear head reaches 36.8 geometry IoU; disabling the completion block in the final diffuser layer can relieve the higher-resolution hallucination.Source: §4.4 · pp. 7–8, Table 5 · p. 7, Table 6 · p. 8
5. Conclusion
SparseOcc uses a pure sparse representation and sparse components for 3D semantic occupancy prediction. It reports state-of-the-art results on nuScenes-Occupancy and comparable performance on SemanticKITTI.
5. Conclusion
- SparseOcc applies a pure sparse representation to 3D scene description for semantic occupancy prediction.The conclusion describes the work as exploring the possibility of pure sparse representation for 3D scene description.Source: §5 · p. 8
- Decomposed orthogonal kernels diffuse non-empty features into adjacent empty regions, while stacked diffusers and downsampling form a coarse-to-fine feature pyramid fused by interpolation and summation.A sparse transformer head queries multi-scale sparse features to generate occupancy predictions.Source: §5 · p. 8
- SparseOcc reports state-of-the-art results on nuScenes-Occupancy and comparable performance on SemanticKITTI.The authors conclude that these results demonstrate effectiveness and efficiency.Source: §5 · p. 8