Source-linked AI summary
S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Clouds
Ran Cheng, Christopher Agia, Yuan Ren, Xinhai Li, Liu Bingbing
TL;DR
Semantic scene completion must infer dense structure and semantics from increasingly sparse LiDAR scans, but dense methods struggle in large outdoor environments. S3CNet addresses this with sparse 2D and 3D networks, geometric features, and multi-view fusion, achieving state-of-the-art SemanticKITTI results and strong 2D-task performance.
Problem
Existing semantic scene completion methods perform well mainly on small indoor scenes with dense inputs but struggle with large, sparse outdoor LiDAR scenes and their memory demands.
Method
S3CNet uses sparse 2D and 3D convolutional networks with geometric features, a geometric-aware segmentation loss, and multi-view fusion to complete scenes from a single LiDAR scan.
Results
S3CNet achieves a 29.5% mean IoU on the SemanticKITTI test set, a +23.9% improvement over the previous leading method, while its 2D variant outmatches several adapted LiDAR segmentation baselines.
Takeaways & Limitations
The method reconstructs large outdoor scenes and predicts voxel-wise semantic labels from a single LiDAR scan, with a complementary 2D variant for semantic scene completion.
Abstract
from arXiv · showhide
With the increasing reliance of self-driving and similar robotic systems on robust 3D vision, the processing of LiDAR scans with deep convolutional neural networks has become a trend in academia and industry alike. Prior attempts on the challenging Semantic Scene Completion task - which entails the inference of dense 3D structure and associated semantic labels from "sparse" representations - have been, to a degree, successful in small indoor scenes when provided with dense point clouds or dense depth maps often fused with semantic segmentation maps from RGB images. However, the performance of these systems drop drastically when applied to large outdoor scenes characterized by dynamic and exponentially sparser conditions. Likewise, processing of the entire sparse volume becomes infeasible due to memory limitations and workarounds introduce computational inefficiency as practitioners are forced to divide the overall volume into multiple equal segments and infer on each individually, rendering real-time performance impossible. In this work, we formulate a method that subsumes the sparsity of large-scale environments and present S3CNet, a sparse convolution based neural network that predicts the semantically completed scene from a single, unified LiDAR point cloud. We show that our proposed method outperforms all counterparts on the 3D task, achieving state-of-the art results on the SemanticKITTI benchmark. Furthermore, we propose a 2D variant of S3CNet with a multi-view fusion strategy to complement our 3D network, providing robustness to occlusions and extreme sparsity in distant regions. We conduct experiments for the 2D semantic scene completion task and compare the results of our sparse 2D network against several leading LiDAR segmentation models adapted for bird's eye view segmentation on two open-source datasets.
1 Introduction
Semantic scene completion extends 3D scene understanding from labeling observed LiDAR points to inferring missing structure and semantics. Existing dense methods struggle with large, sparse outdoor scans, motivating S3CNet’s sparse geometric encoding and complementary 2D formulation.
- Semantic scene completion predicts both missing 3D structure and semantic labels from sparse LiDAR representations.It supports 3D semantic map construction and robust scene understanding for autonomous driving.
- Jointly learning segmentation and completion exploits the relationship between object semantics and underlying geometry.Semantic context can help infer occluded shapes, while inferred shapes can provide priors for semantic classes.
- Dense 2D and 3D methods that work in small indoor scenes struggle with the sparsity, scale, and efficiency demands of outdoor LiDAR.Large outdoor scans contain fewer geometric and semantic descriptors, making dense processing difficult.
- S3CNet uses point-wise normals, flipped TSDF spatial encoding, sparse tensors, and a geometric-inspired segmentation loss to complete LiDAR scenes.The method uses local surface convexity and free, occupied, and occluded-space cues to guide completion.
2 Related Works
Prior work spans volumetric reconstruction, point-cloud segmentation, semantic scene completion, and multi-view fusion. Dense approaches are effective mainly in small indoor settings, while large outdoor scenes motivate sparse representations and cross-view information.
- Volume Reconstruction: Volume reconstruction methods use symmetry, plane fitting, mesh fitting, bounding boxes, or grid-octree CNNs for partial geometric data.These approaches are often limited by scene complexity, model-library diversity, retrieval cost, local-geometry neglect, or object-level scope.
- Semantic Scene Completion: Mainstream semantic scene completion methods rely on dense depth maps or point clouds and dense 3D networks for small indoor scenes.Representative systems transform dense depth into volumetric TSDF inputs, with extensions incorporating RGB semantic segmentation.
- Multi-view Fusion: Multi-view fusion methods combine semantic and geometric information across image and 3D viewpoints.Prior work lifts image-space semantic labels into 3D or uses completed 3D geometry as a prior for image segmentation.
3 Method
S3CNet completes large outdoor LiDAR scenes using sparse 2D and 3D representations, geometric features, and shared sparse-convolution components. Its 2D branch and fusion modules complement 3D completion under occlusion and extreme sparsity.
- System Overview: The system converts one LiDAR scan into memory-efficient sparse 2D and 3D tensors, then processes them with corresponding S3CNet networks.Dynamic voxel fusion uses the predicted 2D bird’s-eye-view semantic map to densify the reconstructed scene and offset 3D memory demands.
- Spatial Feature Engineering: Each sparse 2D pillar encodes normalized mean, minimum, and maximum heights, intensities, and point density across the x-y plane.The representation approximates the point distribution as non-empty BEV pillars.
- Spatial Feature Engineering: The 3D representation combines voxelized point-cloud features with normals extracted from a dynamically depth-completed spherical range image and a memory-efficient flipped TSDF encoding.The normal feature is reversely assigned to 3D points to support spatial feature extraction from sparse scans.
- Sparse Network Design: Sparse convolutions represent only non-empty coordinates and support arbitrary input-output coordinates, kernel shapes, and dimensionalities.The same sparse-convolution, transposed-convolution, and pooling components are used for the 2D and 3D networks with different coordinate dimensions.
- Sparse Network Design: The 3D network uses encode, decode, dilation, and spatial-propagation blocks with squeeze re-weighting and context aggregation modules.The context aggregation modules capture large-receptive-field context, while sparse transposed convolutions are used in decoding.
- Loss Functions: Training combines 2D focal and weighted cross-entropy losses with pruning losses, while 3D training balances voxel occupancy completion against geometric-aware segmentation using λ=0.35.The geometric-aware loss uses local anisotropy, volumetric gradients, and entropy to promote local consistency while emphasizing structured class boundaries.
4 Experiments
Experiments evaluate S3CNet on SemanticKITTI and nuScenes, combining benchmark comparisons, qualitative inspection, and component ablations. The results show strong 3D benchmark performance and competitive 2D segmentation with comparable completion results.
- Qualitative Results: Qualitative results show detailed predictions for small objects and challenging scenes with heavy front-facing occlusion.
- S3CNet achieves 29.5% mean IoU on the SemanticKITTI test set, improving 23.9% over the previous leading method.
- The 2D variant outmatches adapted LiDAR segmentation baselines on semantic segmentation while obtaining comparable completion results on SemanticKITTI and nuScenes.
- Ablation study: Removing CAM or SR reduces mean IoU, while removing SR causes a larger completion-IoU drop because SR affects decoder blocks.
- Ablation study: Removing spatial features substantially reduces performance by eliminating geometric and spatial priors used for completion and free-versus-occluded-space distinction.
- Data augmentation: Training uses random cropping, dropout, translation, and task-specific rotations to improve robustness of the 2D and 3D networks.
5 Conclusion
The paper presents S3CNet for efficient semantic completion of large outdoor scenes from single LiDAR scans. Its 3D and 2D variants achieve strong benchmark performance, while future work targets real-time speed and improved spatial feature encodings.
- S3CNet reconstructs large outdoor scenes and predicts voxel-wise semantic labels from a single LiDAR scan.
- The geometric-aware sparse tensor loss promotes class consistency in homogeneous regions and structured multi-class predictions near object boundaries.
- S3CNet achieves state-of-the-art SemanticKITTI results, while its 2D variant outperforms adapted LiDAR segmentation baselines on two large-scale datasets.
- Future work includes extending the sparse tensor method toward real-time speeds and investigating additional spatial feature encodings.
6 Appendix A: Extra Qualitative Results
Additional qualitative results illustrate S3CNet’s completion and semantic segmentation across static and dynamic scenes. The predictions also cover road surfaces and identify several object categories in 2D results projected into 3D.
- S3CNet detects moving buses and bicyclists despite labels being learned from static scenes, indicating generalization to dynamic scenes.
- The model captures both completion and semantic segmentation characteristics across additional SemanticKITTI training scenes.
- NuScenes 2D predictions cover most road surfaces and precisely detect vehicles, pole-like objects, and pedestrians.
7 Appendix B: Experiment Configurations
Appendix B documents the training configurations and qualitative visualizations for the 2D and 3D semantic scene completion experiments. It also reports that S3CNet achieved the best validation mean IoU among the evaluated 3D competitors.
- 2D experiments: Table 4 specifies training configurations for 2D semantic scene completion models on SemanticKITTI and NuScenes.The configurations cover adapted LiDAR segmentation baselines and S3CNet-related experiments.
- Training controls: All 2D experiments use identical input data and augmentation configurations, while model and optimization settings vary.Changes include the model, loss function, optimizer, scheduler, and supporting hyperparameters.
- 3D experiments: S3CNet achieved the best validation mean IoU among the evaluated 3D competitors on sequence 08.The models were trained on SemanticKITTI for 50 epochs before validation comparison.
- 2D experiments: The 2D experiments include qualitative ground-truth and prediction visualizations for SemanticKITTI and NuScenes.Figures 9 and 10 present qualitative results for the two datasets.