Source-linked AI summary
SCPNet: Semantic Scene Completion on Point Cloud
Zhaoyang Xia, Youquan Liu, Xin Li, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao
TL;DR
Semantic scene completion must infer scenes from sparse, incomplete observations despite diverse object scales and noisy labels for moving objects. SCPNet redesigns the completion network, distills relational knowledge from a multi-frame teacher, and rectifies labels; it ranks first on SemanticKITTI and reports strong results on SemanticPOSS and semantic segmentation.
Problem
Outdoor semantic scene completion is difficult because inputs are sparse and incomplete, objects vary in scale, and moving objects introduce label noise.
Source: §1 · p. 1Method
SCPNet combines a no-downsampling, multi-scale completion network with Dense-to-Sparse Knowledge Distillation and panoptic-label-based rectification of dynamic-object traces.
Source: §1 · pp. 1–2Results
7.2 mIoU above S3CNet, SCPNet ranks 1st on SemanticKITTI, outperforms other completion algorithms on SemanticPOSS, and achieves superior SemanticKITTI segmentation performance.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Training deep models for semantic scene completion (SSC) is challenging due to the sparse and incomplete input, a large quantity of objects of diverse scales as well as the inherent label noise for moving objects. To address the above-mentioned problems, we propose the following three solutions: 1) Redesigning the completion sub-network. We design a novel completion sub-network, which consists of several Multi-Path Blocks (MPBs) to aggregate multi-scale features and is free from the lossy downsampling operations. 2) Distilling rich knowledge from the multi-frame model. We design a novel knowledge distillation objective, dubbed Dense-to-Sparse Knowledge Distillation (DSKD). It transfers the dense, relation-based semantic knowledge from the multi-frame teacher to the single-frame student, significantly improving the representation learning of the single-frame model. 3) Completion label rectification. We propose a simple yet effective label rectification strategy, which uses off-the-shelf panoptic segmentation labels to remove the traces of dynamic objects in completion labels, greatly improving the performance of deep models especially for those moving objects. Extensive experiments are conducted in two public SSC benchmarks, i.e., SemanticKITTI and SemanticPOSS. Our SCPNet ranks 1st on SemanticKITTI semantic scene completion challenge and surpasses the competitive S3CNet by 7.2 mIoU. SCPNet also outperforms previous completion algorithms on the SemanticPOSS dataset. Besides, our method also achieves competitive results on SemanticKITTI semantic segmentation tasks, showing that knowledge learned in the scene completion is beneficial to the segmentation task.
1. Introduction
Outdoor semantic scene completion is challenged by sparse observations, varied object scales, and noisy moving-object labels. SCPNet redesigns completion, distills multi-frame knowledge, and rectifies labels; it ranks first on SemanticKITTI and reports gains on SemanticPOSS and semantic segmentation.
1. Introduction
- Outdoor SSC faces sparse, incomplete input, diverse object scales, moving-object label noise, and weak completion on small distant objects and crowded scenes.Prior methods rely heavily on voxelwise completion labels, whose dynamic-object traces can hamper learning.Source: §1 · p. 1
- SCPNet processes raw voxel features without lossy downsampling and uses Multi-Path Blocks to aggregate multi-scale context for objects of diverse sizes.The completion-first design is intended to address misclassification of small objects and crowded scenes.Source: §1 · p. 1
- Dense-to-Sparse Knowledge Distillation aligns sparse teacher and student features by index and matches pairwise similarity maps to transfer relational knowledge.The multi-frame teacher provides knowledge to the single-scan student.Source: §1 · pp. 1–2
- Completion-label rectification uses off-the-shelf panoptic segmentation labels to remove dynamic-object traces, producing more reliable labels and improving completion on moving objects.The strategy targets long traces in the original completion labels.Source: §1 · p. 2
- 7.2 mIoU higher than S3CNet, SCPNet ranks 1st on SemanticKITTI and also outperforms other completion algorithms on SemanticPOSS and achieves superior SemanticKITTI segmentation performance.The paper also reports that knowledge learned in completion benefits the segmentation task.Source: §1 · p. 2
2. Related Work
Outdoor semantic scene completion addresses sparse, large-scale point clouds with varying density. SCPNet combines multi-scale completion blocks, teacher–student distillation, and a completion-first design without lossy downsampling.
Semantic scene completion
- Earlier outdoor completion methods include LMSCNet’s mixed 2D/3D backbone, S3CNet’s joint completion and dynamic voxel fusion, and JS3C-Net’s segmentation-linked completion with point-voxel refinement.Outdoor point clouds are sparse, large-scale, and variably dense, unlike indoor point clouds.Source: §2 · pp. 2–3
Knowledge distillation
- SCPNet uses a multi-frame teacher and single-frame student with the same architecture, transferring dense semantic knowledge through distillation.Each network voxelizes point features, completes voxel features, then uses a segmentation sub-network adapted from Cylinder3D to produce voxelwise output.Source: Figure 2 · p. 3
Semantic scene completion
- SCPNet avoids lossy downsampling and uses Multi-Path Blocks to aggregate multi-scale features for objects of varying sizes.It transfers dense, relation-based semantic knowledge from a multi-frame model to a single-frame model, and is presented as the first use of knowledge distillation for semantic scene completion.Source: §2 · p. 3
3. Methodology
SCPNet predicts voxelwise geometry and semantics from sparse point clouds using completion and segmentation sub-networks. It redesigns completion, distills relational knowledge from a multi-frame teacher, rectifies dynamic-object labels, and combines segmentation, completion, and distillation losses.
3.1. Framework overview
- SCPNet assigns each voxel an empty or semantic-class label, using completion followed by segmentation to produce voxelwise output.The completion module processes raw voxel features, and the segmentation module is based on Cylinder3D with cubic rather than cylindrical partitioning and without its point-refinement module.Source: §3 · p. 3, §3.1 · p. 3, §3.2 · p. 4
3.2. Redesigning the Completion Sub-network
- The completion sub-network preserves sparsity, avoids downsampling and upsampling, and aggregates multi-scale features through 3×3×3, 5×5×5, and 7×7×7 convolution blocks.Removing convolution biases and batch normalization maintains sparse features; MPBs feed dense completed features onward as extracted non-empty sparse voxels for segmentation.Source: §3.2 · p. 4, Figure 3 · p. 4
3.3. Distilling Multi-frame Knowledge
- A multi-frame teacher supplies denser input, while DSKD aligns teacher and student voxel indices and matches pairwise feature similarities for the shared single-frame voxels.The student and teacher relational maps are compared using a squared-error loss averaged over the student’s voxel pairs.Source: §3.3 · pp. 4–5
3.4. Completion Label Rectification
- Completion labels concatenate segmentation labels from consecutive frames, but panoptic instance bounds filter voxels outside dynamic-object bounds to remove long object traces.The authors report that rectification makes completion labels more accurate.Source: §3.4 · pp. 5–6
3.5. Overall objective
- The overall objective sums cross-entropy, Lovasz-softmax, and DSKD losses, with coefficients α and β balancing the latter two terms.The Lovasz-softmax term carries the paper’s citation.Source: §3.5 · p. 6
4. Experiments
Experiments evaluate SCPNet on SemanticKITTI and SemanticPOSS, report competitive segmentation transfer, and test the effects of label rectification, the completion sub-network, DSKD, and downsampling.
4. Experiments
- Experiments use SemanticKITTI and SemanticPOSS; evaluation includes per-class IoU and all-class mIoU, with mIoU the major SSC metric.SemanticKITTI uses 19 classes and SemanticPOSS uses 11 classes for evaluation.Source: §4 · p. 6
4.1. Results
- On SemanticKITTI, SCPNet significantly outperforms other scene-completion algorithms in mIoU, scoring 7.2 mIoU above S3CNet; it also achieves superior performance on the SemanticPOSS validation set.Table 2 reports SemanticPOSS validation results using completion labels processed by the proposed rectification algorithm; bold indicates the best column result among all single-frame methods.Source: §4.1 · p. 6, Table 2 · p. 7
- Cylinder3D initialized with SCPNet completion-task weights improves SemanticKITTI segmentation by 2.6 mIoU over the original Cylinder3D.The paper reports that completion-learned knowledge also benefits segmentation.Source: §4.1 · p. 7, Table 3 · p. 7
4.2. Ablation studies
- On SemanticKITTI validation, label rectification improves IoU by 8.1 for car, 24.5 for person, 26.1 for bicyclist, and 17.6 for motorcyclist.Table 5 evaluates SCPNet with and without completion-label rectification.Source: §4.2 · p. 7, Table 5 · p. 8
- Adding the completion sub-network to JS3CNet improves its mIoU by 6.8 on SemanticKITTI validation.Table 6(a) reports the completion sub-network comparison.Source: §4.2 · p. 8, Table 6 · p. 8
- On SemanticKITTI validation, DSKD improves SCPNet by 2.8 mIoU over the version without DSKD.The comparison is reported in Table 6(b).Source: §4.2 · p. 8
- Adding downsampling reduces SCPNet from 37.2 mIoU to 33.1 mIoU on SemanticKITTI validation.The comparison is reported in Table 6(c).Source: §4.2 · p. 8
5. Conclusion
SCPNet combines completion-network redesign, dense-to-sparse knowledge distillation, and completion-label rectification, with strong completion performance on two benchmarks and benefits for semantic segmentation.
5. Conclusion
- SCPNet addresses SSC challenges through network redesign, dense-to-sparse knowledge distillation, and completion-label rectification, and achieves superior completion performance on SemanticKITTI and SemanticPOSS.Knowledge learned on completion also benefits semantic segmentation.Source: §5 · p. 8
A. Ablation studies
Appendix ablations examine the DSKD loss coefficient, class-level DSKD gains, and the effect of adding downsampling.
A. Ablation studies
- Changing the DSKD loss coefficient from 1,000 to 4,000 makes SCPNet completion performance improve and then decline; the authors set β to 3,000 for best performance.The coefficient sweep is reported in Table 7.Source: A. Ablation studies · p. 10
- DSKD improves IoU by more than 3 on motorcycle, truck, person, and bicyclist.The class-level comparison is reported in Table 8.Source: A. Ablation studies · p. 10
- Adding downsampling substantially degrades completion, especially for truck, other-vehicle, other-ground, and traffic-sign.The comparison is reported in Table 9.Source: A. Ablation studies · p. 10
B. Elaborated implementation details
The appendix discusses range mismatch and sparsity in the completion network, plus transfer, computational overhead, and runtime comparisons.
B. Elaborated implementation details
- Using mismatched point-cloud and completion-label ranges creates many empty voxels and significantly hampers completion, so the authors use the completion-label range.The appendix also notes that convolution biases and Batch Normalization can make empty voxel features nonzero and break sparsity.Source: B. Elaborated implementation details · p. 10
- Tables 7–9 report the DSKD loss-coefficient, DSKD-loss, and downsampling comparisons, respectively.Table 10 is identified as a training- and inference-time comparison using A100.Source: Table 7 · p. 11, Table 8 · p. 11, Table 9 · p. 11
- On SemanticKITTI validation, DSKD improves JS3CNet from 24.0 to 26.2 mIoU, while label rectification improves it from 24.0 to 26.8 mIoU.The completion sub-network adds around 24.4 ms overhead; SCPNet has comparable training and inference time to JS3CNet while achieving better completion performance.Source: B. Elaborated implementation details · p. 11
C. Qualitative results
Qualitative comparisons show that DSKD improves single-frame completion on parking areas and small objects, while downsampling and upsampling distort object shapes.
C. Qualitative results
- With DSKD, single-frame SCPNet achieves better completion and segmentation than SCPNet without DSKD by distilling dense and relation-based information from the multi-frame teacher; completion improves on parking areas and small objects.Figure 7 compares both SCPNet variants with SCPNet-4Frames and ground truth on the SemanticKITTI [1] validation set.Source: C. Qualitative results · p. 11, Figure 7 · p. 12
- Downsampling and upsampling cause over-dilation and shape distortion in the objects marked by red ellipses.Figure 8 compares SCPNet with downsampling, without downsampling, SCPNet-4Frames, and ground truth on SemanticKITTI validation data.Source: C. Qualitative results · p. 11, Figure 8 · p. 12