PaperlayerView exact revision on arXiv

Source-linked AI summary

RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving

Angelika Ando, Spyros Gidaris, Andrei Bursuc, Gilles Puy, Alexandre Boulch, Renaud Marlet

arXiv:2301.10222v2cs.CVcs.AIcs.LG

TL;DR

Projection-based LiDAR segmentation has relied mainly on 2D CNNs, raising whether vision transformers can improve this setting despite sparse data and limited LiDAR annotations. RangeViT adapts an image-pretrained ViT with convolutional tokenization and decoder refinement, achieving state-of-the-art results among projection-based methods on nuScenes and SemanticKITTI.

  • Problem

    ViT-based segmentation for outdoor LiDAR remains challenging because ViTs require extensive training data while large annotated LiDAR datasets are costly and uncommon.

  • Method

    RangeViT uses range projection, image-pretrained ViT weights, a convolutional stem, and a convolutional decoder with a skip connection for LiDAR segmentation.

  • Results

    RangeViT achieves state-of-the-art performance among projection-based LiDAR segmentation methods on nuScenes and SemanticKITTI.

  • Takeaways & Limitations

    ViT image representations can be repurposed for LiDAR segmentation despite the domain gap when tokenization and prediction refinement are adapted.

  • Takeaways & Limitations

    Performance remains difficult for sparse, imbalanced classes such as bicycles and pedestrians, and future work identifies LiDAR tokenization as a key improvement area.

Abstract

from arXiv · show

Casting semantic segmentation of outdoor LiDAR point clouds as a 2D problem, e.g., via range projection, is an effective and popular approach. These projection-based methods usually benefit from fast computations and, when combined with techniques which use other point cloud representations, achieve state-of-the-art results. Today, projection-based methods leverage 2D CNNs but recent advances in computer vision show that vision transformers (ViTs) have achieved state-of-the-art results in many image-based benchmarks. In this work, we question if projection-based methods for 3D semantic segmentation can benefit from these latest improvements on ViTs. We answer positively but only after combining them with three key ingredients: (a) ViTs are notoriously hard to train and require a lot of training data to learn powerful representations. By preserving the same backbone architecture as for RGB images, we can exploit the knowledge from long training on large image collections that are much cheaper to acquire and annotate than point clouds. We reach our best results with pre-trained ViTs on large image datasets. (b) We compensate ViTs' lack of inductive bias by substituting a tailored convolutional stem for the classical linear embedding layer. (c) We refine pixel-wise predictions with a convolutional decoder and a skip connection from the convolutional stem to combine low-level but fine-grained features of the the convolutional stem with the high-level but coarse predictions of the ViT encoder. With these ingredients, we show that our method, called RangeViT, outperforms existing projection-based methods on nuScenes and SemanticKITTI. The code is available at https://github.com/valeoai/rangevit.

1. Introduction

RangeViT adapts vision transformers to projection-based LiDAR segmentation, addressing limited LiDAR training data and ViTs’ weak inductive bias. It combines image pretraining, convolutional tokenization, and decoder refinement to improve projection-based segmentation.

  • Motivation: Projection-based LiDAR segmentation converts point clouds into 2D representations processed with image-oriented networks.This approach supports semantic perception of outdoor environments independently of lighting conditions.
  • Approach: RangeViT repurposes a plain ViT encoder for projected LiDAR data and maps decoded pixel predictions back to the 3D point cloud.The method extracts patch-based tokens from a range projection, then decodes them into pixel-wise labels.
  • Approach: A multi-layer convolutional stem replaces the classical linear embedding layer to compensate for ViTs’ lack of inductive bias.This is one of three ingredients identified as necessary for peak performance.
  • Approach: A lightweight convolutional decoder and skip connection combine fine-grained stem features with coarse, high-level ViT representations.The decoder refines pixel-wise predictions before they are projected back to 3D points.
  • Approach: Image-pretrained ViTs improve LiDAR segmentation despite the substantial difference between natural images and range-projection images.Large image datasets are easier to acquire, annotate, and store than large LiDAR datasets.
  • Outcome: RangeViT achieves state-of-the-art results among projection-based LiDAR segmentation methods.The paper presents this as a simple projection-based approach that transfers ViT models from the RGB image domain.

2. Related work

Prior work mainly processes projected outdoor LiDAR with 2D CNNs, while transformer-based point-cloud methods have focused largely on indoor scenes. Outdoor LiDAR ViT segmentation remains difficult because of computational scale, sparse data, and limited training resources.

  • 2D Methods: Outdoor point-cloud methods commonly project data into range, perspective, or BEV images and process the result with 2D CNNs.Examples include range-projection networks using U-Net-like architectures and BEV methods using Ring CNNs.
  • Point-Based Methods: Direct raw-point methods developed for indoor scenes are difficult to adapt to outdoor scenes because large point quantities create computational difficulties.The related-work discussion contrasts indoor PointNet-inspired processing with the scale of outdoor point clouds.
  • Transformer Methods: Recent transformer-based point-cloud segmentation research has primarily targeted indoor scenes.These methods include Point-BERT, Point Transformer, and Bridged Transformer.
  • Research Gap: Outdoor LiDAR semantic segmentation with a ViT-based architecture had not yet been published according to the paper.The paper identifies outdoor LiDAR as a challenging setting for transformer-based semantic segmentation.
  • Transfer Learning: ViTs require vast training datasets, whereas large annotated LiDAR datasets are less common because annotation is costly and time-consuming.This motivates transfer learning from image-pretrained models to point clouds.
  • Transfer Learning: Existing image-to-point-cloud transfer methods adapt CNNs or transformer components through kernel inflation, teacher-student learning, specialized tokenizers, or representation distillation.The cited approaches span outdoor and indoor point-cloud settings.

3. RangeViT

RangeViT projects LiDAR point clouds into range images, processes them with a convolutional stem, ViT encoder, and decoder, then refines point-wise predictions in 3D. Its design replaces standard ViT tokenization and uses convolutional features to recover fine spatial information.

  • Range projection: Range projection converts each LiDAR point cloud into an H × W range image with five features: range, coordinates, and intensity.If multiple points map to one pixel, the feature with the smallest range is retained; empty pixels receive zeros.
  • Convolutional stem: The convolutional stem replaces linear patch embedding with nonlinear residual context modules that capture short-range dependencies and produce pixel-wise features.Average pooling then reduces the feature map to patch resolution before a 1×1 convolution produces ViT-compatible tokens.
  • ViT encoder: The ViT encoder processes visual and classification tokens augmented with positional embeddings, after which the classification token is removed to retain patch representations.The encoder transforms the input sequence through L transformer blocks into deep patch representations.
  • Decoder: The decoder reshapes patch representations into a 2D map, applies a 1×1 convolution and Pixel Shuffle, and restores features to the original range-image resolution.A single skip connection from the convolutional stem supplies fine-grained features alongside the decoder output.
  • 3D refiner: The 3D refiner projects decoded features back to the original points and applies KPConv with point coordinates to produce geometry-aware point features.This learned refinement replaces inference-time K-NN or CRF post-processing used to correct projection-related segmentation errors.
  • Training and inference: Training combines multi-class focal loss with Lovász-softmax loss, while inference averages overlapping sliding-window decoder features before 3D refinement.The focal loss addresses sample hardness and class imbalance, whereas Lovász-softmax is designed to optimize mIoU.

4. Experiments

Experiments evaluate RangeViT on nuScenes and SemanticKITTI using mIoU, ablate architectural and tokenization choices, and compare image-pretrained initialization and fine-tuning strategies. The results show benefits from convolutional components, rectangular small patches, image pre-training, selective fine-tuning, and strong performance against projection-based methods.

  • Experimental setup: Experiments use nuScenes and SemanticKITTI, with mean Intersection over Union (mIoU) as the evaluation metric.nuScenes provides 28,130 training and 6,019 validation scans; SemanticKITTI provides 19,130 training and 4,071 validation scans.
  • Architecture ablations: 69.82 mIoU results from adding the convolutional stem to a linear-stem, linear-decoder model, improving over 65.52.The convolutional stem supplies a nonlinear component for producing input token features.
  • Patch-size ablations: 2 × 8 patches perform best among the tested tokenizations, as smaller patches provide finer information for thin objects and boundaries.The convolutional stem produces pixel-wise features before local average pooling reduces them to patch tokens.
  • Image pre-training: Image-pretrained ViTs outperform random initialization despite the domain gap, improving mIoU by 2.4 points with ImageNet21k and 2.8 points with Cityscapes.Pre-training also accelerates convergence on the nuScenes validation set.
  • Fine-tuning strategies: Keeping attention layers frozen produces the best fine-tuning result, suggesting pretrained attention is already able to generalize to range images.The authors attribute the greater impact of FFN fine-tuning partly to differences between LiDAR and RGB data and easier FFN optimization.
  • Final comparison: RangeViT achieves higher mIoU than prior 2D projection-based methods on both datasets and narrows the gap with Cylinder3D.Class-wise performance is often best or second best, though sparse and imbalanced classes such as bicycles and pedestrians remain difficult.

5. Conclusion

RangeViT demonstrates that pre-trained ViTs can be repurposed for projection-based LiDAR segmentation despite the domain gap between RGB and range images. The authors identify tokenization as a key remaining opportunity for improvement.

  • The method leverages ViT models pre-trained on large image datasets despite the substantial RGB-to-range domain gap.
  • Pre-trained ViTs can be successfully used for LiDAR segmentation without changing the original transformer backbone.The approach adapts tokenization and preprocessing and adds a convolutional decoder.
  • RangeViT reaches state-of-the-art performance among 2D projection methods for LiDAR segmentation.
  • Future improvements could target LiDAR tokenization through random patch sizes, learned token extraction, or tokenizing raw 3D data instead of 2D projections.

A. Additional visualizations

The supplementary visualizations examine RangeViT predictions on nuScenes validation point clouds, including both accurate and inaccurate examples.

  • Figures 6 and 7 visualize RangeViT segmentation accuracy on nuScenes validation point clouds.
  • Figure 8 presents examples of both correct and incorrect RangeViT predictions.

B. Model parameter count analysis

The supplementary ablation compares a deeper linear-stem, linear-decoder model with the full RangeViT model while keeping their parameter counts similar.

  • Model (e) uses a linear stem, linear decoder, and L = 14 transformer layers instead of L = 12.
  • Model (e) and full RangeViT have a similar number of parameters, isolating architectural effects from parameter-count differences.

C. Computation cost comparison

RangeViT has moderate parameter and inference costs relative to the compared LiDAR segmentation methods on nuScenes. Its 25 ms inference time is close to SalsaNext and below Cylinder3D under the stated hardware and processing conditions.

  • Table 10 reports parameter counts and inference times on the nuScenes validation set.
  • RangeViT has 27.1M parameters, versus 6.73M for SalsaNext, 55.9M for Cylinder3D, and 213.2M for KPRNet.
  • 25 ms inference time for RangeViT is close to SalsaNext with K-NN post-processing at 28 ms and below Cylinder3D at 49 ms.The comparison uses the same GeForce RTX 2080 GPU on the nuScenes validation set.
  • SalsaNext requires 15 ms without K-NN post-processing, compared with 25 ms for RangeViT.

D. Additional ablation analysis

The ablations examine crop size, convolutional stem and decoder design, and the classification token. Crop size has little effect, while removing the class token reduces mIoU.

  • Impact of crop size: The fixed crop covers the entire vertical FOV and one fifth of the horizontal FOV, avoiding whole-image self-attention while retaining identifiable objects.The crop spans 67.5 degrees horizontally and corresponds to the view of a single nuScenes camera.
  • Impact of crop size: Crop size has a small impact on performance across the tested configurations.The authors suggest that retuning the learning rate and training duration could reduce the observed gaps.
  • Convolutional stem and UpConv decoder: The convolutional stem and UpConv decoder preserve feature-map dimensions around their non-pooling operations through appropriate padding.Figure 5 presents the overall architecture and detailed stem and decoder components; convolutions are marked by kernel size and batch-normalization layers by BN.
  • Role of the classification token: Removing the classification token lowers mIoU from 75.21% to 74.64% with Cityscapes pre-training and from 72.37% to 72.24% with random initialization.The token interacts with patch embeddings but is removed from the encoder output used for segmentation.

E. Additional implementation details

The implementation details specify the convolutional stem and decoder dimensions, the 3D refiner configuration, alternative backbones and range projection, and qualitative visualization conventions.

  • Convolutional stem and UpConv decoder: The stem reduces H × W to (H/PH) × (W/PW) using average pooling with kernel (PH + 1) × (PW + 1), stride PH × PW, and corresponding padding.Convolutions before pooling and after Pixel Shuffle preserve spatial dimensions through appropriate padding.
  • Convolutional stem and UpConv decoder: The stem uses 32 channels in its first three residual blocks, Dh channels in the fourth, and changes channels from C = 5 at the first convolutions.The final stem convolution uses D output channels, as specified in the implementation description.
  • 3D Refiner: The 3D Refiner uses a KPConv layer with Dh input and output channels, a 15-point kernel, and influence radius 1.2.
  • Alternative backbone: Replacing ViT-S with ResNet-50 keeps inference time at 25 ms but increases parameters from 25.2M to 35.3M.The RN50-based model uses compatible stem and decoder channel dimensions and has comparable FLOPs.
  • SemanticKITTI range projection: SemanticKITTI range images are generated by unfolding LiDAR scans in sensor-capture order rather than using the spherical projection from the main paper.
  • Qualitative visualizations: NuScenes qualitative figures compare ground truth and predictions, while accuracy views mark correct predictions blue and incorrect predictions red.Figure 8 additionally uses half-blue, half-red circles for predictions that are mostly correct except for a few points.
Loading 2301.10222v2…