Source-linked AI summary
LMSCNet: Lightweight Multiscale 3D Semantic Completion
Luis Roldão, Raoul de Charette, Anne Verroust-Blondet
TL;DR
Sparse 3D sensing and the cubic memory cost of voxel-based 3D CNNs make full-resolution semantic completion difficult. LMSCNet uses lightweight 2D convolutions with 3D segmentation heads and multiscale inference; on SemanticKITTI, it matches semantic-completion performance while outperforming the state of the art on completion.
Problem
Sparse 3D sensors and voxel-based 3D CNNs make full-resolution semantic completion computationally difficult, limiting real-time deployment.
Method
LMSCNet processes voxel grids with lightweight 2D convolutions, maps features to 3D through segmentation heads, and supports multiscale completion.
Results
On SemanticKITTI, LMSCNet performs on par with other methods on semantic completion and outperforms the state of the art on completion.
Takeaways & Limitations
The multiscale pipeline enables coarser inference at very high speed while preserving performance despite the loss of 3D spatial connectivity from 2D convolutions.
Takeaways & Limitations
SemanticKITTI provides no visual results for its test set, so TS3D baselines using additional modalities were omitted because retraining was complex.
Abstract
from arXiv · showhide
We introduce a new approach for multiscale 3Dsemantic scene completion from voxelized sparse 3D LiDAR scans. As opposed to the literature, we use a 2D UNet backbone with comprehensive multiscale skip connections to enhance feature flow, along with 3D segmentation heads. On the SemanticKITTI benchmark, our method performs on par on semantic completion and better on occupancy completion than all other published methods -- while being significantly lighter and faster. As such it provides a great performance/speed trade-off for mobile-robotics applications. The ablation studies demonstrate our method is robust to lower density inputs, and that it enables very high speed semantic completion at the coarsest level. Our code is available at https://github.com/cv-rits/LMSCNet.
1. Introduction
LMSCNet addresses the computational burden of 3D semantic scene completion by using lightweight 2D convolutions with 3D segmentation heads and multiscale outputs. The method targets efficient deployment while reporting strong SemanticKITTI performance.
- 3D semantic scene completion is difficult because sparse sensing, limited field of view, and occlusions prevent comprehensive scene understanding.The task infers both completion and semantic labels for the whole scene.
- 3D CNN voxel processing is computationally heavy because memory requirements grow cubically with voxel resolution.This has limited prior methods to lower predicted resolutions and restricted real-time deployment.
- LMSCNet processes voxel grids with considerably lighter 2D convolutions and maps features to 3D using segmentation heads.The approach uses no additional modalities.
- Multiscale outputs preserve efficient feature flow and enable very fast inference at coarser levels.The architecture combines a 2D backbone with 3D segmentation heads for multiscale completion.
- On SemanticKITTI, LMSCNet achieves state-of-the-art performance while improving completion performance over other published methods.The paper presents a lightweight architecture with significantly fewer parameters.
2. Related Works
Prior work uses diverse representations and architectures for 3D completion and semantic scene completion, but voxel methods face memory costs and richer modalities increase complexity. LMSCNet instead uses only voxelized occupancy data with lightweight multiscale processing.
- Voxel representations incur substantial memory requirements because many voxels are empty, motivating optimized structures and sparse convolutions.
- 3D completion methods have used primitive surfaces, TSDFs, continuous energy minimization, and learning-based approaches for missing or occluded regions.
- Semantic scene completion combines scene completion and semantic segmentation, initially using end-to-end 3D CNNs and later incorporating RGB or spherical projections.
- Additional modalities and spherical projections can improve performance but increase network complexity and inference time.
- LMSCNet differs by using only 3D voxelized occupancy data, avoiding preprocessing while adding lightweight multiscale capability.
3. Method
LMSCNet performs dense 3D semantic completion with a lightweight multiscale UNet that combines 2D backbone convolutions and 3D segmentation heads. Its design reduces computation while producing outputs at multiple resolutions for coarse, fast inference.
- Problem formulation: LMSCNet predicts semantic labels for every voxel in a dense 3D scene representation from a sparse voxel grid.The label set includes N semantic classes, a free-voxel class, and an unknown class during training.
- 2D backbone: The architecture uses a four-level encoder-decoder UNet with 2D convolutions along X,Y and the height dimension treated as a feature dimension.Pooling reduces spatial resolution by two at each level, while skip connections preserve information flow.
- 3D segmentation head: 3D segmentation heads recover the third spatial dimension and output class-wise probabilities using dense and dilated convolutions.The heads use Atrous Spatial Pyramid Pooling with dilation rates [1, 2, 3] to fuse information across receptive fields.
- Multiscale completion: Attaching a 3D segmentation head after every UNet level produces multiscale completion at relative resolutions 1/2^l for l ∈ {0, 1, 2, 3}.Coarser ground truths are obtained by majority-vote pooling from the original-resolution ground truth.
- Training strategy: Training uses cross-entropy at each scale with class weighting based on inverse class frequency and combines the level losses with per-level weights.The implementation uses α_l = 1 for every level and trains end-to-end from sparse voxel inputs and semi-dense labeled grids.
- Design choices: The method avoids TSDF input variants and retains an occupancy-grid representation to preserve training and inference efficiency.The authors report little improvement from encoding the unknown class, while ray-casting preprocessing would add substantial time.
4. Experiments
Experiments on SemanticKITTI show that LMSCNet delivers competitive semantic completion and leading occupancy completion with a lightweight, multiscale architecture that supports fast coarse inference and remains robust to sparse inputs.
- Benchmark setup: SemanticKITTI evaluation compares LMSCNet with SSCNet, TS3D, TS3D+DNet, TS3D+DNet+SATNet, and SSCNet-full on official benchmark data.The official evaluation uses full-size ground truth, while SSCNet-full denotes SSCNet upsampled to input resolution.
- Semantic Scene Completion: LMSCNet ranks second on semantic completion mIoU and first on completion IoU with a comfortable margin.TS3D+DNet+SATNet is slightly better on mIoU, whereas LMSCNet outperforms all compared methods on IoU.
- Multiscale performance: Multiscale inference maintains good performance across resolutions and improves at coarser levels, while multiscale UNet raises completion by 0.68% and semantic completion by 0.56% over Vanilla UNet.The single-scale variant performs slightly better at full size but loses multiscale capacity.
- Architectures comparison: 0.35M parameters and 72.6G FLOPs make full-resolution LMSCNet substantially lighter than the compared networks.LMSCNet is at least an order of magnitude faster than TS3D baselines, although SSCNet is twice as fast in the reported comparison.
- Architectures comparison: 372 FPS is reached at the coarsest multiscale setting, reported as 6x faster than SSCNet and 300x faster than TS3D+DNet+SATNet.Coarser inference removes unnecessary network components and supports high-speed coarse scene analysis for mobile robotics.
- Ablation studies: With 8 input LiDAR layers at 2.10% density, LMSCNet still retrieves the general scene outline, although lower density degrades performance especially far from the sensor.The study also reports sharp labeling on 32-layer nuScenes inputs despite training on 64-layer SemanticKITTI data.
5. Conclusion
LMSCNet combines 2D and 3D convolutions to remain lightweight while supporting inference at multiple scales. On SemanticKITTI, it matches other methods for semantic completion, outperforms the state of the art for completion, and supports fast, lower-density inference.
- LMSCNet mixes 2D and 3D convolutions to preserve a lightweight architecture while enabling inference at multiple scales.
- LMSCNet performs on par with other methods on semantic completion with a lighter architecture and faster inference speed.
- LMSCNet outperforms the state of the art on completion.
- The method remains robust to much lower input density and enables lower-resolution scene completion at very high speed.
- The authors conjecture that higher variance in all directions would increase the impact of lost 3D spatial connectivity from 2D convolutions.
Performance in SemanticKITTI [1] (64 layers)
The section presents additional qualitative semantic-completion results for LMSCNet on SemanticKITTI and nuScenes across different scene viewpoints. NuScenes ground-truth images are omitted because point-wise semantic labels are unavailable.
- Additional qualitative 3D semantic-completion results are shown for LMSCNet on SemanticKITTI and nuScenes.
- Each pair of rows represents one scene viewed from different viewpoints.
- NuScenes ground-truth images are not shown because point-wise semantic labels are unavailable.
A.1. Baselines implementations
The appendix details implementation choices for the baselines used in the main comparison, including additional TS3D variants and a full-resolution SSCNet configuration.
- The appendix provides additional implementation details for the baselines listed in the main article’s Table 1.
- TS3D comparisons include variants using RGB, LiDAR intensity, projected semantic labels, or a modified SATNet backbone.
- SSCNet-full restores the original input dimensions by applying a 4×4×4 transposed convolution to the network’s last layer.
A.2. Architecture comparison
LMSCNet offers the best speed-performance trade-off among the compared architectures, while its coarser-scale inferences provide especially low latency with high performance.
- LMSCNet keeps the best speed-performance trade-off among the compared architectures.
- SSCNet-full is faster than LMSCNet at the original scale, but has slightly lower and noisier performance.
- TS3D+DNet+RangeNet achieves slightly higher performance but requires considerably more inference time and parameters.
- LMSCNet’s coarser-scale inferences combine considerably lower inference times with high performances.
B. Qualitative results
Qualitative results show smoother, less noisy reconstructions across SemanticKITTI and nuScenes, while coarse-scale variants retain fast inference with good performance.
- Coarse-scale LMSCNet variants preserve good mIoU while achieving fast inference, especially at the 1:x scales.Figure 10 plots network speed in FPS against performance in mIoU, with FPS shown on a log scale.
- LMSCNet produces smoother, less noisy reconstructions than SSCNet-full in nuScenes examples.
- The network correctly reconstructs vehicles despite dynamic-object accumulation in SemanticKITTI ground truth.The passage attributes this to the abundance of parked vehicles in the dataset.