Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion

Xu Yan; Jiantao Gao; Jie Li; Ruimao Zhang; Zhen Li; Rui Huang; Shuguang Cui

Source-linked AI summary

Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion

Xu Yan, Jiantao Gao, Jie Li, Ruimao Zhang, Zhen Li, Rui Huang, Shuguang Cui

arXiv:2012.03762v1Dec 7, 2020cs.CV

TL;DR

Sparse single-sweep LiDAR segmentation is difficult because sparsity and missing geometric details impair accurate prediction. JS3C-Net learns contextual shape priors through semantic scene completion and point-voxel interaction, achieving state-of-the-art results in segmentation and scene completion on two benchmarks.

Problem
Sparse single-sweep LiDAR point clouds remain challenging to segment accurately because sparsity causes missing shapes and uneven sampling across categories.
Method
JS3C-Net jointly learns segmentation and semantic scene completion from merged consecutive frames, using contextual shape priors and a shape-aware point-voxel interaction module.
Results
JS3C-Net achieves state-of-the-art results in semantic segmentation and scene completion on SemanticKITTI and SemanticPOSS, outperforming previous methods by a large margin.
Takeaways & Limitations
Contextual shape priors and interactive feature fusion provide an effective approach for understanding sparse LiDAR scenes with severe local detail loss.

Abstract

from arXiv · show

LiDAR point cloud analysis is a core task for 3D computer vision, especially for autonomous driving. However, due to the severe sparsity and noise interference in the single sweep LiDAR point cloud, the accurate semantic segmentation is non-trivial to achieve. In this paper, we propose a novel sparse LiDAR point cloud semantic segmentation framework assisted by learned contextual shape priors. In practice, an initial semantic segmentation (SS) of a single sweep point cloud can be achieved by any appealing network and then flows into the semantic scene completion (SSC) module as the input. By merging multiple frames in the LiDAR sequence as supervision, the optimized SSC module has learned the contextual shape priors from sequential LiDAR data, completing the sparse single sweep point cloud to the dense one. Thus, it inherently improves SS optimization through fully end-to-end training. Besides, a Point-Voxel Interaction (PVI) module is proposed to further enhance the knowledge fusion between SS and SSC tasks, i.e., promoting the interaction of incomplete local geometry of point cloud and complete voxel-wise global structure. Furthermore, the auxiliary SSC and PVI modules can be discarded during inference without extra burden for SS. Extensive experiments confirm that our JS3C-Net achieves superior performance on both SemanticKITTI and SemanticPOSS benchmarks, i.e., 4% and 3% improvement correspondingly.

Introduction

Sparse single-sweep LiDAR point clouds suffer from missing shapes and uneven sampling, making semantic segmentation difficult despite strong results on complete data. JS3C-Net addresses this by learning shape priors through auxiliary scene completion and improving feature fusion between segmentation and completion.

Motivation: Adjacent LiDAR frames provide richer shape information that can make objects such as partially observed trucks easier to segment.The paper illustrates this by comparing a sparse per-sweep truck with its completion using neighboring frames.
Problem: Sparse single-sweep LiDAR point clouds exhibit missing shapes and uneven sampling, making semantic segmentation substantially more difficult than segmentation on complete data.Sparsity increases with reflection distance and affects different categories unevenly.
Method: JS3C-Net merges dozens of consecutive LiDAR frames to create complete point-cloud supervision for semantic scene completion without extra annotation.The optimized SSC module learns shape priors from these merged-frame annotations.
Method: The framework uses auxiliary scene completion to enhance sparse single-sweep semantic segmentation and introduces point-voxel interaction for feature fusion between the two tasks.The PVI module promotes interaction between point-cloud local geometry and voxel-wise global structure.
Contributions: The auxiliary components are designed in cascaded and disposable manners to balance segmentation performance and effectiveness.The paper presents JS3C-Net as the first method to enhance sparse single-sweep LiDAR semantic segmentation through auxiliary scene completion.

Related Work

Related work organizes point-cloud processing into projection-, voxel-, and point-based strategies, while semantic scene completion and multi-task learning provide complementary frameworks for incomplete-scene understanding and knowledge transfer.

Point clouds are challenging because they are sparse and disordered, unlike 2D images with regular grids.
Projection-based methods map point clouds onto 2D pixels, enabling conventional CNNs to process plane- or spherical-projected LiDAR data.
Semantic Scene Completion: Semantic scene completion produces a complete 3D voxel representation from incomplete input and can jointly predict scene completion and semantic labels.
Multi-task Learning: Multi-task learning transfers knowledge between tasks to improve learning efficiency and prediction accuracy, including semantic and instance segmentation with designed fusion modules.

Method

JS3C-Net jointly improves sparse single-frame segmentation by combining an initial segmentation network with semantic scene completion and shape-aware point-voxel interaction. The auxiliary completion and interaction modules provide contextual shape information during training but can be removed at inference.

Pipeline: The pipeline first obtains point-wise semantic segmentation and shape embeddings, then feeds the segmentation probabilities into semantic scene completion.The SSC module voxelizes the input and uses convolutional processing with skip connections and dense upsampling to generate a complete volume.
Semantic Scene Completion: The SSC module introduces contextual shape priors learned from the entire LiDAR sequence by predicting completion results from SparseConv semantic probabilities.The segmentation backbone stores and convolves only non-empty voxels, while using point coordinates as inputs.
Point-Voxel Interaction: The PVI module fuses incomplete point-cloud geometry with coarse complete voxels through voxel-center selection, Euclidean k-nearest-neighbor queries, and graph convolution.Stacked graph-convolution layers learn spatial and semantic relations before producing fine-grained completion through residual refinement.
Point-Voxel Interaction: Feature interaction lets sparse point-cloud features predict complete-voxel semantics and transfers complete geometric details back to segmentation through back propagation.PVI also enhances object-probability prediction for corresponding voxels by using spatial and semantic relationships.
Joint Training: Uncertainty weighting uses trainable parameters σ1 and σ2 to automatically balance weighted cross-entropy losses for segmentation and completion during end-to-end training.Segmentation gradients update only the segmentation-network parameters Wseg, while both task losses update the joint network parameters.
Inference: The segmentation backbone is replaceable, and the SSC and PVI modules can be discarded during inference without adding computing burden for segmentation.This design supports real-time applications while retaining the joint-learning framework during training.

Experiments

Experiments on SemanticKITTI and SemanticPOSS show that JS3C-Net improves semantic segmentation, semantic scene completion, and scene completion through joint learning and PVI-based interaction. Ablations and complexity analysis further support its effectiveness and inference efficiency.

Benchmark Evaluation: JS3C-Net is evaluated on SemanticKITTI and SemanticPOSS for LiDAR semantic segmentation.SemanticKITTI contains 43,552 annotated scans across 21 sequences and 19 valid classes.
Semantic Segmentation: JS3C-Net surpasses existing SemanticKITTI segmentation methods in mIoU, especially on geometrically sparse small objects.Qualitative results also show improvements over KPConv in fine details and small objects.
Semantic Segmentation: On SemanticPOSS, JS3C-Net achieves larger improvements over its baseline on challenging data with remarkable small objects.Table 2 compares the method against state-of-the-art projection-based, point-based, and voxel-based methods.
Scene Completion: JS3C-Net achieves state-of-the-art results on both SC and SSC tasks through semantic guidance, joint learning, and PVI interaction.Its SSC module generates more faithful geometric details, with results 6% higher than previous state-of-the-art methods.
Ablation Study: 66.1% SS IoU and 55.0% SC IoU are achieved by joint learning, improving over the separately trained baseline’s 63.1% and 51.1%.UMTL adds 0.4% on SS and 1.1% on SC, while PVI produces the best results on both tasks.
Complexity Analysis: JS3C-Net uses 1/5 the model size and 1/3 the inference time of KPConv, while disposable SSC and PVI modules preserve segmentation-backbone speed.The extra SSC and PVI operations can be ignored during semantic-segmentation inference.

Conclusion

The paper presents JS3C-Net, a single-sweep LiDAR semantic segmentation framework that uses contextual shape priors from semantic scene completion. It combines sophisticated pipelines, interactive modules, and a reasonable loss function to achieve state-of-the-art results on semantic segmentation and scene completion.

Framework: JS3C-Net is proposed as a single-sweep LiDAR point cloud semantic segmentation framework using contextual shape priors from semantic scene completion.The framework is named JS3C-Net.
Design: The model exploits sophisticated pipelines, interactive modules, and a reasonable loss function.
Results: JS3C-Net achieves state-of-the-art results on both semantic segmentation and scene completion tasks, outperforming previous methods by a large margin.

Concrete Experimental Design

JS3C-Net combines semantic segmentation, semantic scene completion, and point-voxel interaction modules, with specified sparse-convolution, decoder, and graph-learning configurations. Experiments compare segmentation against official or officially implemented results and report semantic scene completion scores on SemanticKITTI and SemanticPOSS.

Model Components: JS3C-Net consists of a semantic segmentation network, an SSC decoder, and a PVI module.These are the framework’s three main components.
Segmentation Network: The segmentation backbone uses seven two-layer 3D sparse-convolution blocks with channels increasing from 16 to 112 and pooling after each block.The channel sequence is 16, 32, 48, 64, 80, 96, 112.
SSC and PVI Configuration: The SSC decoder has five basic blocks with 32 channels each, while PVI uses a three-layer 32-channel MLP and one GCN layer.The PVI graph-based edge learning configuration sets l = 1.
Compared Methods: SemanticKITTI comparisons use official benchmark results, whereas SemanticPOSS comparisons use official results for PointNet++ and SequeezeSegV2 and official-code implementations for RandLA-Net and KPConv.The passage describes the evaluation sources for the compared methods.
Semantic Scene Completion Results: 67.5 SemanticKITTI and 60.2 SemanticPOSS are reported with moving objects, while 67.7 SemanticKITTI and 60.2 SemanticPOSS are reported without moving objects.These values are presented as benchmark results for the two moving-object settings.

Additional Discussion

The discussion justifies semantic scene completion for joint learning by showing that scene completion alone biases segmentation, while semantic scene completion transfers category-level shape priors with aligned outputs. Ablations further show limited impact from moving objects, weaker results when using only historical frames, and no added inference burden from discardable auxiliary components.

Scene completion versus semantic scene completion: Jointly learning scene completion and semantic segmentation worsens segmentation relative to directly using the segmentation network, indicating that scene completion introduces bias.Scene completion can improve its own result, but the segmentation result becomes worse than model A.
Scene completion versus semantic scene completion: Because the cascaded SS and SSC outputs are spatially aligned, the SSC decoder can complete category shapes and backpropagate shape priors to segmentation with less noise.The paper contrasts this category-focused completion with completing the entire scene.
Moving-object analysis: 0.2% improvement in semantic segmentation is obtained when moving objects are removed from semantic scene completion, so they have limited influence because they occupy a small dataset proportion.Moving objects remain in training data to ensure fair comparison with other methods.
Historical-frame comparison: Using only attention or feature aggregation over historical frames cannot learn complete shape priors better, while historical-frame ground truth still outperforms the reimplemented baseline.Historical-only supervision misses shape details of newly incoming objects, whereas incorporating future frames further improves results.
Inference cost: Unlike the compared historical-frame method, the proposed auxiliary components are fully discardable and therefore do not add computational burden to segmentation during inference.The comparison explicitly refers to latency in the table.

More Visualization

This section presents additional visualization cases on SemanticPOSS and selected semantic scene completion ground truths. It also contrasts reconstructed results with and without moving objects.

More Visualization: Figure 2 shows additional visualization cases on the SemanticPOSS dataset.The passage identifies these as more cases on SemanticPOSS.
More Visualization: Figure 1 presents selected ground truths for semantic scene completion.These examples are described as selected ground truths for SSC.
More Visualization: The reconstructed semantic scene completion results are illustrated with and without moving objects.Figure 1 uses left and right views to show the two conditions.