PaperlayerView exact revision on arXiv

Source-linked AI summary

SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving

Tiago Cortinhal, George Tzelepis, Eren Erdal Aksoy

arXiv:2003.03653v4cs.CVcs.LG

TL;DR

Real-time, fine-grained semantic segmentation of full 3D LiDAR scans remains less developed than camera-based segmentation, despite its importance for autonomous-vehicle scene understanding. SalsaNext extends SalsaNet with multiscale context, architectural and loss changes, and Bayesian uncertainty estimation; on SemanticKITTI, it achieves 59.5% mean IoU, ranks first, and estimates both data- and model-based uncertainty.

  • Problem

    Real-time semantic segmentation of full 3D LiDAR scans is important for autonomous-vehicle scene understanding, but relatively few approaches address 3D LiDAR compared with camera images.

  • Method

    SalsaNext extends SalsaNet with a multiscale residual dilated context module, pixel-shuffle decoding, weighted cross-entropy plus Lovasz-Softmax loss, and Bayesian treatment for point-wise uncertainty estimation.

  • Results

    59.5% mean IoU, +3.6% over the previous state-of-the-art method, with SalsaNext ranking first on the Semantic-KITTI leaderboard.

  • Takeaways & Limitations

    SalsaNext combines real-time full-scan segmentation with epistemic and aleatoric uncertainty estimates, distinguishing model limitations from sensor observation noise.

Abstract

from arXiv · show

In this paper, we introduce SalsaNext for the uncertainty-aware semantic segmentation of a full 3D LiDAR point cloud in real-time. SalsaNext is the next version of SalsaNet [1] which has an encoder-decoder architecture where the encoder unit has a set of ResNet blocks and the decoder part combines upsampled features from the residual blocks. In contrast to SalsaNet, we introduce a new context module, replace the ResNet encoder blocks with a new residual dilated convolution stack with gradually increasing receptive fields and add the pixel-shuffle layer in the decoder. Additionally, we switch from stride convolution to average pooling and also apply central dropout treatment. To directly optimize the Jaccard index, we further combine the weighted cross-entropy loss with Lovasz-Softmax loss [2]. We finally inject a Bayesian treatment to compute the epistemic and aleatoric uncertainties for each point in the cloud. We provide a thorough quantitative evaluation on the Semantic-KITTI dataset [3], which demonstrates that the proposed SalsaNext outperforms other state-of-the-art semantic segmentation networks and ranks first on the Semantic-KITTI leaderboard. We also release our source code https://github.com/TiagoCortinhal/SalsaNext.

I. INTRODUCTION

SalsaNext targets real-time, uncertainty-aware semantic segmentation of full 3D LiDAR scans for autonomous driving. It combines architectural, loss, and Bayesian changes, and reports higher accuracy with fewer parameters and less computation time than prior networks.

  • Motivation: Real-time semantic segmentation provides fine-grained scene understanding needed for autonomous vehicles.Reliable uncertainty estimates can support downstream decision-making, manoeuvre planning, and emergency braking.
  • Motivation: LiDAR segmentation is challenging because point clouds are sparse, unstructured, and non-uniformly sampled.Compared with camera imagery, LiDAR offers a wider field of view and more accurate distance measurements.
  • Contributions: SalsaNext extends SalsaNet with a context module, residual dilated convolutions, pixel shuffle, average pooling, modified dropout, and a combined weighted cross-entropy and Lovasz-Softmax loss.The design aggregates receptive fields at multiple scales, reduces upsampling artifacts, lightens downsampling, emphasizes basic features, and directly optimizes the Jaccard index.
  • Contributions: Bayesian treatment transforms SalsaNet into a stochastic model that estimates epistemic and aleatoric uncertainty for each 3D LiDAR point.The output contains point-wise classification scores together with uncertainty measures.
  • Results: SemanticKITTI experiments show significantly higher pixel-wise segmentation accuracy with fewer parameters and less computation time than other state-of-the-art networks.SalsaNext also ranks first on the Semantic-KITTI leaderboard.

II. RELATED WORK

The related-work section surveys semantic segmentation of 3D point clouds and Bayesian neural networks for uncertainty estimation.

  • Related work: Recent 3D point-cloud segmentation methods are reviewed before discussing Bayesian neural networks for uncertainty estimation.The section provides a brief literature review of both topics.
  • Related work: The review covers semantic segmentation of 3D point-cloud data.
  • Related work: The review also covers literature on Bayesian neural networks for uncertainty estimation.

A. Semantic Segmentation of 3D Point Clouds

3D LiDAR segmentation methods differ in architecture and point-cloud representation, with point-wise and projection-based approaches forming the main categories. The literature also motivates uncertainty estimation and this paper’s LiDAR-only scope.

  • Architectures: Encoder-decoder, fully convolutional, and multi-branch networks are mainstream architectures for encoding and fusing multilevel features.SalsaNext follows an encoder-decoder design based on its performance in prior state-of-the-art methods.
  • Representations: Point-wise and projection-based rendering are the two common representations for semantic segmentation of unstructured, unordered 3D LiDAR points.Point-wise methods process raw points, whereas projection-based methods transform them into formats such as voxels, multi-view surfaces, lattices, or rasterized images.
  • Scope: The study focuses on LiDAR-only semantic segmentation and excludes multimodal methods that fuse LiDAR with camera data.
  • Uncertainty estimation: Prior 3D LiDAR segmentation approaches generally lack uncertainty measures such as prediction confidence scores.Bayesian neural networks estimate uncertainty through distributions over weights and distinguish aleatoric data noise from epistemic model uncertainty.

III. METHOD

SalsaNext represents the full 3D LiDAR scan as a dense spherical Range View image, preserving geometric and intensity information for convolutional processing.

  • LiDAR Point Cloud Representation: The method projects the unstructured 3D LiDAR point cloud onto a spherical surface to create a dense, compact Range View image.This representation enables standard convolution operations.
  • LiDAR Point Cloud Representation: Each raw point (x, y, z) is mapped to image coordinates (u, v) in the 2D Range View image.
  • LiDAR Point Cloud Representation: The projection uses the point range r, image height h, image width w, and sensor vertical field of view f.Range is defined from the 3D coordinates, while f combines the downward and upward vertical fields of view.
  • LiDAR Point Cloud Representation: The full 360° field of view is projected, storing coordinates, intensity, and range index in separate channels.The resulting input has dimensions [w × h × 5].

B. Network Architecture

SalsaNext extends SalsaNet with context-aware dilated convolutions, efficient upsampling, selective dropout, and average pooling within an encoder-decoder architecture.

  • Network Architecture: SalsaNext takes a Range View projection as input and builds on SalsaNet’s encoder-decoder design with residual feature fusion through skip connections.
  • Contextual Module: The contextual module fuses larger and smaller receptive fields using residual dilated convolutions with 1×1 and 3×3 kernels.
  • Dilated Convolution: The encoder replaces SalsaNet’s ResNet blocks with dilated convolutions having effective receptive fields of 3, 5, and 7.Outputs are concatenated, passed through a 1×1 convolution, and combined with a residual connection.
  • Pixel-Shuffle Layer: Pixel-shuffle layers replace computationally expensive transpose convolutions by rearranging channel features into spatially upsampled feature maps.The upsampled outputs are concatenated with skip connections before decoder dilated-convolution blocks.
  • Central Encoder-Decoder Dropout: Dropout is concentrated in central encoder-decoder layers, while average pooling replaces strided convolution for downsampling.The architecture applies leaky-ReLU, batch normalization, dropout, and a final soft-max classifier.

C. Uncertainty Estimation

SalsaNext estimates aleatoric uncertainty by propagating modeled sensor noise and epistemic uncertainty through Bayesian dropout sampling during inference.

  • Aleatoric Uncertainty: Aleatoric uncertainty may be homoscedastic or heteroscedastic; distant LiDAR points and object edges can exhibit input-dependent uncertainty.
  • Heteroscedastic Aleatoric Uncertainty: Assumed Density Filtering propagates a known LiDAR noise distribution through a probabilistic network to produce predictions with aleatoric uncertainties.The noisy input is modeled with a normal distribution N(x, v), and the forward pass outputs μ with σA.
  • Epistemic Uncertainty: Epistemic uncertainty is approximated by treating dropout as a distribution over network weights rather than analytically evaluating the intractable posterior.
  • Epistemic Uncertainty: The Bayesian optimization objective combines the likelihood term with KL-divergence regularization over variational parameters.The KL term regularizes the approximate weight posterior, while the likelihood depends on network outputs and ground-truth labels.
  • Epistemic Uncertainty: Monte Carlo inference estimates epistemic uncertainty by running n trials and averaging the variance of the predicted outputs.
  • Uncertainty Estimation: The dropout rate is selected by grid search to minimize the KL divergence, while total uncertainty sums aleatoric and epistemic components.

D. Loss Function

SalsaNext addresses class imbalance with weighted cross-entropy and directly targets the IoU metric by adding Lovász-Softmax loss.

  • Loss Function: Rare classes such as bicycles and traffic signs can bias training toward frequent vehicle classes and reduce segmentation performance.
  • Weighted Cross-Entropy: Weighted cross-entropy uses inverse square-root class frequency to increase the response to underrepresented classes.
  • Lovász-Softmax Loss: Lovász-Softmax loss is added to maximize intersection-over-union, or the Jaccard index, despite IoU being discrete and non-differentiable.
  • Total Loss: The total SalsaNext loss is the linear combination L = Lwce + Lls.

E. Optimizer And Regularization

SalsaNext training uses stochastic gradient descent with weight decay, momentum, spatial dropout, and independent geometric point-cloud augmentations to reduce overfitting.

  • Stochastic gradient descent starts at a learning rate of 0.01, decays it by 0.01 each epoch, and uses L2 penalty λ = 0.0001 with momentum 0.9.
  • The batch size is 24, and spatial dropout probability is fixed at 0.2.
  • Training augmentation independently applies random rotation or translation, y-axis flipping, and point dropping with probability 0.5 each.

F. Post-processing

Projection-based LiDAR segmentation can lose information during discretization and back-projection, so SalsaNext applies kNN post-processing during inference to refine point labels.

  • Projection-based representations lose information through discretization and blurry convolutional responses.
  • Multiple LiDAR points can map to one image pixel, causing misclassification especially at object edges and in background shadows.
  • The kNN post-processing selects closest neighbors within a window around each corresponding image pixel to address back-projection issues.It operates on every LiDAR point and uses range distances because the window is small.
  • The post-processing is applied only during inference and does not affect learning.
  • Evaluation uses SemanticKITTI, which contains over 43K annotated full 3D LiDAR scans divided into training, validation, and test splits.Over 21K scans are used for training, sequence 08 is used for validation, and sequences 11–21 form the test split.

A. Evaluation Metric

The evaluation uses mean intersection-over-union, while quantitative and qualitative analyses examine accuracy, uncertainty, and class frequency on SemanticKITTI.

  • Mean intersection-over-union, or the Jaccard Index, averages class-wise IoU values over all classes.For each class, IoU is the cardinality of predicted-and-labelled points divided by the cardinality of their union.
  • 59.5% mean IoU is achieved on SemanticKITTI, exceeding the previous state-of-the-art by +3.6%.SalsaNext performs best in 9 of 19 categories and remains comparable in several others.
  • Epistemic uncertainty is higher for rare classes with fewer test-set points, including motorcyclist and motorcycle.The reported relationship is shown between uncertainty and class point count in log scale.
  • Qualitative results display predicted segments alongside projected epistemic and aleatoric uncertainties, with lighter colors indicating greater uncertainty.The camera images are used only for visualization and were not used to train SalsaNext.

D. Ablation Study

Ablations show that SalsaNext’s architectural, loss, post-processing, and uncertainty components improve accuracy while maintaining a compact real-time model.

  • Each SalsaNext contribution yields a distinct accuracy improvement over the original SalsaNet.
  • The kNN post-processing step increases accuracy by around 2%.
  • The dilated convolution stack increases parameters, while pixel-shuffle layers substantially reduce that increase in the decoder.
  • Combining weighted cross-entropy with Lovász-Softmax produces the largest accuracy increment by directly optimizing the Jaccard index.
  • SalsaNext reaches 59.5% accuracy with only 2.2% or 0.15M extra parameters over SalsaNet.
  • SalsaNext runs at 24 Hz without uncertainty computation and uses 7× fewer parameters than RangeNet++.Runtime measurements use the full SemanticKITTI dataset on one NVIDIA Quadro RTX 6000 card.
Loading 2003.03653v4…