Source-linked AI summary
Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation
Yuenan Hou, Xinge Zhu, Yuexin Ma, Chen Change Loy, Yikang Li
TL;DR
LiDAR segmentation models are computationally heavy, and existing distillation methods provide marginal gains because point clouds are sparse, random, and variably dense. PVD combines point- and voxel-level output and affinity distillation with supervoxel partitioning and difficulty-aware sampling. Across nuScenes and SemanticKITTI, it consistently outperforms prior distillation approaches, achieving approximately 75% MACs reduction and 2× speedup on Cylinder3D with very minor performance degradation.
Problem
Existing distillation methods provide marginal gains for LiDAR semantic segmentation because point clouds exhibit sparsity, randomness, and varying density, while segmentation models are costly to deploy.
Method
PVD transfers point- and voxel-level output and affinity knowledge, using supervoxel partitioning and difficulty-aware sampling to improve efficient structural distillation.
Results
PVD consistently outperforms previous distillation approaches across Cylinder3D, SPVNAS, and MinkowskiNet on nuScenes and SemanticKITTI; on Cylinder3D it achieves approximately 75% MACs reduction and 2× speedup with very minor performance degradation.
Takeaways & Limitations
PVD provides a strong baseline for compressing cumbersome 3D segmentation models, indicating substantial redundancy in these models.
Takeaways & Limitations
The number of input points differs across point clouds, an assumption relevant to processing point and voxel features.
Abstract
from arXiv · showhide
This article addresses the problem of distilling knowledge from a large teacher model to a slim student network for LiDAR semantic segmentation. Directly employing previous distillation approaches yields inferior results due to the intrinsic challenges of point cloud, i.e., sparsity, randomness and varying density. To tackle the aforementioned problems, we propose the Point-to-Voxel Knowledge Distillation (PVD), which transfers the hidden knowledge from both point level and voxel level. Specifically, we first leverage both the pointwise and voxelwise output distillation to complement the sparse supervision signals. Then, to better exploit the structural information, we divide the whole point cloud into several supervoxels and design a difficulty-aware sampling strategy to more frequently sample supervoxels containing less-frequent classes and faraway objects. On these supervoxels, we propose inter-point and inter-voxel affinity distillation, where the similarity information between points and voxels can help the student model better capture the structural information of the surrounding environment. We conduct extensive experiments on two popular LiDAR segmentation benchmarks, i.e., nuScenes and SemanticKITTI. On both benchmarks, our PVD consistently outperforms previous distillation approaches by a large margin on three representative backbones, i.e., Cylinder3D, SPVNAS and MinkowskiNet. Notably, on the challenging nuScenes and SemanticKITTI datasets, our method can achieve roughly 75% MACs reduction and 2x speedup on the competitive Cylinder3D model and rank 1st on the SemanticKITTI leaderboard among all published algorithms. Our code is available at https://github.com/cardwing/Codes-for-PVKD.
1. Introduction
LiDAR segmentation models are difficult to deploy on resource-constrained autonomous vehicles, while existing distillation methods provide marginal gains on point clouds. PVD addresses point-cloud sparsity, randomness, and varying density by combining point- and voxel-level distillation with efficient, difficulty-aware affinity sampling.
- LiDAR segmentation models deliver strong benchmark performance but require heavy computation and storage, hindering deployment on resource-constrained devices.
- Previous distillation algorithms provide marginal gains on LiDAR segmentation because point clouds are sparse, random, and variably dense.
- PVD distills pointwise and voxelwise probabilistic outputs, combining fine-grained point information with richer voxel-level environmental clues.
- PVD partitions point clouds into supervoxels and samples K supervoxels for affinity distillation, making structural knowledge transfer tractable.
- PVD produces more accurate predictions for minority classes and faraway objects than baseline distillation on the SemanticKITTI validation set.
- Difficulty-aware sampling more frequently selects supervoxels containing minority classes and faraway objects, emphasizing hard cases.
- PVD consistently outperforms previous distillation approaches by a large margin across Cylinder3D, SPVNAS, and MinkowskiNet on nuScenes and SemanticKITTI.
- Approximately 75% MACs reduction and 2× speedup are achieved on Cylinder3D with very minor performance degradation.
2. Related Work
Prior work developed direct LiDAR segmentation models and transferred distillation ideas from other settings, but straightforward distillation remains ineffective for point clouds. PVD is presented as an approach tailored to transferring point- and voxel-level knowledge.
- PointNet and its variants process point clouds directly but are extremely slow on large-scale outdoor point clouds.
- Previous distillation methods that perform well on 2D segmentation provide marginal gains when directly deployed on LiDAR segmentation.
- The paper presents PVD as the first application of knowledge distillation to LiDAR semantic segmentation, transferring point-level and voxel-level knowledge.
3. Methodology
PVD distills LiDAR segmentation knowledge through pointwise and voxelwise outputs plus affinity relationships within difficulty-sampled supervoxels. It builds on Cylinder3D’s point-to-voxel pipeline and addresses sparse supervision, structural information, computational cost, and uneven point distributions.
- Framework overview: Cylinder3D extracts point features, voxelizes them by aggregation, processes voxel features with asymmetric 3D convolutions, and refines pointwise predictions.The pipeline produces voxelwise output before pointwise refinement and final classification.
- Point-to-voxel output distillation: PVD distills both pointwise and voxelwise outputs because point predictions provide fine-grained information while voxel predictions provide coarser, richer environmental clues.Both outputs use KL-divergence losses, with voxel labels assigned by majority encoding.
- Point-to-voxel affinity distillation: Affinity distillation supplements output distillation by transferring structural information from pairwise similarities among point and voxel features.Inter-point affinity is computed within sampled supervoxels from pairwise feature similarities, and the student matches the teacher’s affinity matrices.
- Point-to-voxel affinity distillation: Supervoxel partitioning makes affinity distillation tractable by replacing the intractable whole-cloud similarity matrix with affinity computation on K sampled supervoxels.Whole-cloud affinity matrices can contain over ten billion elements because input clouds may have hundreds of thousands of points.
- Difficulty-aware sampling: Difficulty-aware sampling favors supervoxels containing minority classes and faraway objects, while fixed feature counts handle variable point-cloud density and size.Features are randomly reduced for oversized groups and padded with zeros when undersized.
- Affinity-map analysis: PVD produces affinity maps closer to the teacher’s, pulling same-class features together and pushing different-class features apart compared with channel distillation.The reported visualization supports improved transfer of structural knowledge from teacher to student.
4. Experiments
Experiments on nuScenes and SemanticKITTI evaluate PVD across LiDAR segmentation models, compression settings, and component choices. PVD matches strong baselines while reducing computation, and ablations examine its distillation and sampling mechanisms.
- Experimental setup: Experiments use nuScenes and SemanticKITTI benchmarks to evaluate LiDAR segmentation and model compression.nuScenes includes 16 semantic classes after class merging and filtering; the implementation uses Cylinder3D as the primary backbone.
- Results: PVD significantly outperforms previous distillation algorithms on both SemanticKITTI and nuScenes, with a performance gap larger than 1.8 over the strongest competing KD method.The comparison covers quantitative results on the SemanticKITTI test set and nuScenes validation set.
- Ablation studies: Combining point-to-voxel output distillation with affinity distillation produces the largest performance gains in component ablations.The ablation also reports that voxel-level distillation contributes more than point-level distillation, while affinity distillation contributes more than output distillation.
- Ablation studies: Supervoxel size affects PVD because very small supervoxels limit affinity learning, whereas large ones reduce learning efficiency.The study controls the number of sampled supervoxels at 4 when examining this factor.
- Ablation studies: Difficulty-aware sampling outperforms distance-aware, category-aware, and random sampling strategies.The result supports jointly emphasizing distant objects and rare classes during affinity distillation.
5. Conclusion
The paper presents PVD for compressing LiDAR semantic segmentation models through point-to-voxel output and affinity distillation. Across two benchmarks and three backbones, PVD outperforms baseline distillation algorithms and indicates substantial redundancy in 3D segmentation models.
- Conclusion: PVD combines point-to-voxel output distillation with affinity distillation, using supervoxel partitioning and difficulty-aware sampling to improve affinity-learning efficiency.The method is tailored for LiDAR semantic segmentation model compression.
- Conclusion: PVD significantly outperforms baseline distillation algorithms when distilling Cylinder3D, SPVNAS, and MinkowskiNet on two LiDAR segmentation benchmarks.The conclusion characterizes the results as evidence of substantial redundancy in 3D segmentation models.
A. Quantitative results
PVD consistently improves compression results over previous distillation algorithms on SPVNAS and MinkowskiNet. It nearly closes the performance gap between the original and pruned networks.
- Quantitative results: PVD consistently outperforms previous distillation algorithms on SPVNAS and MinkowskiNet by a large margin.The results are reported on the SemanticKITTI validation set.
- Quantitative results: 2.6 mIoU points separate PVD from SKD on SPVNAS.For both evaluated models, PVD can almost mitigate the performance gap between the original network and the pruned model.
B. Ablation studies
Ablations analyze loss weighting and component contributions in PVD. The results favor stronger inter-voxel affinity weighting and indicate that voxel-based distillation has a larger performance impact.
- Loss coefficients: The inter-voxel affinity distillation loss should receive a larger coefficient than the other distillation losses for the best distillation effect.This conclusion comes from comparisons of loss-coefficient configurations in Table 7.
- Loss coefficients: Swapping the point-based and voxel-based loss coefficients deteriorates performance, indicating that voxel-based distillation guides point-based distillation.The voxel-based loss is reported as more important in this comparison.
- Component influence: The voxel-based loss term has a larger impact on final performance than the other evaluated PVD components.The paper attributes this potential advantage to voxel representations aggregating information from all points within a voxel and providing richer structural information.
C. Elaborated implementation details
The implementation uses pretrained weights, standardized training protocols, and GPU latency measurement; FLOPs are estimated from average sparse-convolution kernel-map sizes.
- Training protocol: All distillation methods load pretrained weights, shortening overall training to three days for fair comparison.Training a single model from scratch may take more than a week; all methods use the same acceleration strategy.
- Evaluation setup: Latency is recorded on a single NVIDIA Tesla PG503-216 GV100 GPU.
- Method naming: The multi-scan competition names the method PV-KD.
- Training protocol: SPVNAS and MinkowskiNet follow the training protocol of their open-sourced codes, while finetuning retrains for 10 additional epochs at learning rate 2e-4.
- Computation measurement: Sparse-convolution FLOPs are computed using estimated average kernel-map sizes for each layer.The estimate accounts for nonzero positions and varying input point-cloud patterns.
D. Qualitative results
The qualitative comparison evaluates teacher, ground truth, SKD, and PVD predictions, highlighting PVD's improvements on difficult classes, distant objects, and affinity structure.
- Prediction comparison: PVD produces smaller prediction errors than SKD for minority classes such as person and bicycle.
- Prediction comparison: PVD yields more accurate predictions than SKD for faraway objects, including the highlighted car.
- Affinity comparison: PVD has lower inter-class similarity and higher intra-class similarity than SKD in the visual affinity comparison.
- Related configuration: Table 9 reports the loss coefficients used by different distillation methods.
- Visualization: The qualitative figure compares Student, SKD, PVD, Teacher, and GT outputs.
- Affinity comparison: The inter-voxel affinity ground truth assigns similarity 1 within classes and 0 across classes.