PaperlayerView exact revision on arXiv

Source-linked AI summary

Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data

Corentin Sautier, Gilles Puy, Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, Renaud Marlet

arXiv:2203.16258v1cs.CVcs.LG

TL;DR

Annotated Lidar data is costly, and existing self-supervised 3D methods are not tailored to sparse autonomous-driving point clouds. SLidR distills self-supervised image representations into 3D networks using superpixel-based region matching without annotations, and its representations transfer well across semantic segmentation and object detection while surpassing related methods.

  • Problem

    Annotated Lidar data is tedious and costly, while many existing 3D self-supervised methods are adapted to dense indoor point clouds rather than sparse autonomous-driving data.

  • Method

    SLidR distills self-supervised image representations into a 3D network by aligning pooled image and point features from corresponding superpixel-defined regions.

  • Results

    SLidR produces 3D representations that transfer across semantic segmentation and object detection tasks and surpass related state-of-the-art methods across evaluation settings.

  • Takeaways & Limitations

    Superpixel-driven image-to-Lidar distillation provides object-aware 3D representations suited to multiple autonomous-driving perception tasks without requiring image or point-cloud annotations.

  • Takeaways & Limitations

    The method may be impaired in low-light conditions when computed superpixels form irrelevant object segments.

Abstract

from arXiv · show

Segmenting or detecting objects in sparse Lidar point clouds are two important tasks in autonomous driving to allow a vehicle to act safely in its 3D environment. The best performing methods in 3D semantic segmentation or object detection rely on a large amount of annotated data. Yet annotating 3D Lidar data for these tasks is tedious and costly. In this context, we propose a self-supervised pre-training method for 3D perception models that is tailored to autonomous driving data. Specifically, we leverage the availability of synchronized and calibrated image and Lidar sensors in autonomous driving setups for distilling self-supervised pre-trained image representations into 3D models. Hence, our method does not require any point cloud nor image annotations. The key ingredient of our method is the use of superpixels which are used to pool 3D point features and 2D pixel features in visually similar regions. We then train a 3D network on the self-supervised task of matching these pooled point features with the corresponding pooled image pixel features. The advantages of contrasting regions obtained by superpixels are that: (1) grouping together pixels and points of visually coherent regions leads to a more meaningful contrastive task that produces features well adapted to 3D semantic segmentation and 3D object detection; (2) all the different regions have the same weight in the contrastive loss regardless of the number of 3D points sampled in these regions; (3) it mitigates the noise produced by incorrect matching of points and pixels due to occlusions between the different sensors. Extensive experiments on autonomous driving datasets demonstrate the ability of our image-to-Lidar distillation strategy to produce 3D representations that transfer well on semantic segmentation and object detection tasks.

1. Introduction

SLidR addresses the cost and sparsity challenges of autonomous-driving Lidar by distilling self-supervised image representations into 3D networks without annotations. It uses superpixels to align pooled image and point features, and outperforms prior self-supervised and 2D-to-3D pre-training methods across evaluation settings.

  • Motivation: Annotated Lidar data is costly, while existing 3D self-supervised methods are mainly designed for dense indoor point clouds and struggle with sparse outdoor data.Autonomous-driving vehicles provide synchronized, calibrated cameras and Lidar that are easier to acquire than to annotate.
  • Approach: SLidR distills self-supervised image representations into a 3D network using synchronized and calibrated camera–Lidar data without image or point-cloud annotations.The approach targets downstream semantic segmentation and object detection in autonomous-driving Lidar data.
  • Approach: Superpixels pool visually similar image pixels and back-projected 3D points, then a contrastive loss aligns the corresponding pooled features.This region-level formulation is intended to produce object-aware 3D representations while reducing matching noise from sensor occlusions.
  • Approach: The method includes a high-resolution image-feature upsampling architecture designed to avoid degenerate solutions during distillation.The paper identifies this architecture and the superpixel-to-superpoint contrastive loss as core technical contributions.
  • Results: SLidR surpasses state-of-the-art 3D self-supervised pre-training and prior 2D-to-3D distillation methods in all reported evaluation settings.The study evaluates semantic segmentation on nuScenes and SemanticKITTI and object detection on KITTI against adapted strong baselines.

2. Related Works

The related work situates SLidR among self-supervised learning, 3D representation learning, knowledge distillation, and 2D-to-3D distillation. It emphasizes the challenge of applying methods developed on curated indoor or image data to raw, sparse autonomous-driving data.

  • Self-supervision: Self-supervised learning pre-trains representations with annotation-free tasks so they can transfer to downstream tasks with limited labeled data.The reviewed image methods include contrastive instance discrimination, feature reconstruction, and clustering-style approaches.
  • 3D self-supervision: Most reviewed 3D self-supervised methods target single objects or use reconstruction, transformation prediction, clustering, or contrastive feature objectives.This literature differs in task scope and pretext design from autonomous-driving scene representation learning.
  • Autonomous-driving data: Autonomous-driving data is raw, redundant, and imbalanced, making self-supervision both challenging and useful for reducing continuous annotation demands.The paper addresses these conditions by leveraging multimodal camera and Lidar information.
  • Knowledge distillation: Knowledge distillation transfers information from a trained teacher network to a student by matching outputs or intermediate features.The paper places its method within unsupervised and semi-supervised teacher–student representation learning.
  • 2D-to-3D distillation: Prior 2D-to-3D distillation contrasts pixels with points, whereas SLidR contrasts image regions and 3D point regions defined by superpixels.The paper reports that its adapted comparison method performs worse empirically on autonomous-driving data.

3. Our approach

SLidR distills self-supervised image representations into 3D Lidar networks using synchronized, calibrated sensors and superpixel-to-superpoint contrastive learning. It pools corresponding image and point features into visually coherent regions, then matches them while reducing sampling and sensor-misalignment effects.

  • Image-to-Lidar self-supervised distillation: SLidR leverages aligned Lidar and image data to learn 3D representations without human annotations.The method uses a self-supervised image network as a teacher for a 3D Lidar network.
  • Image-to-Lidar self-supervised distillation: Known sensor poses project Lidar points into camera images, enabling corresponding 2D pixels and 3D points to be grouped.The projection mapping returns the corresponding pixel index or indicates that a point is not visible in a camera.
  • Superpixel-driven contrastive distillation loss: SLIC superpixels group locally similar pixels, while their projected point sets form corresponding superpoints for region-level matching.Superpixels are used as pooling masks for image features and projected point features.
  • Superpixel-driven contrastive distillation loss: The contrastive loss jointly trains the 3D backbone and projection heads to make each superpoint feature closer to its corresponding superpixel feature than to other regions.Features are average-pooled and projected into a shared F-dimensional space before contrastive training.
  • Superpixel-driven contrastive distillation loss: Superpixel pooling encourages object-aware local coherence, equalizes region weights despite point-density variation, and reduces false negatives and spurious sensor matches.Averaging features mitigates approximate point-pixel correspondences caused by occlusions and motion.
  • The image projection head: The image teacher uses a self-supervised MoCov2 ResNet-50, while a pixel-wise projection and fixed upsampling recover high-resolution features for distillation.The proposed projection head uses a 1 × 1 convolution, bilinear upsampling by 4 in each spatial direction, and ℓ2-normalization; dilated convolutions preserve encoder resolution.

4. Experiments

Experiments evaluate SLidR’s pre-training with adapted baselines across semantic segmentation and object detection, including few-shot and cross-dataset settings. The results show gains from the superpixel-driven loss, stronger transfer than competing pre-training methods, and locally coherent object-aware features, alongside low-light and feature-collision limitations.

  • Baselines: The evaluation adapts PPKT, PointContrast, and DepthContrast baselines to autonomous-driving data and uses matched training setups where possible.Baselines share the nuScenes split, augmentations, voxel-based backbone, and cylindrical-coordinate voxels; DepthContrast uses four GPUs because this significantly improves performance.
  • Ablation Study: A 4.5-point improvement results when the superpixel-driven contrastive loss is combined with dilated convolutions in the mini-val ablation.Dilated convolutions alone provide a 1.9-point gain, while the combined configuration yields the larger improvement.
  • Semantic Segmentation: Image-to-Lidar distillation methods outperform DepthContrast and PointContrast, while SLidR achieves better semantic segmentation performance, especially under linear probing.The comparison covers nuScenes and SemanticKITTI using linear probing and few-shot end-to-end fine-tuning protocols.
  • Annotation Efficiency: 8.7 points is the largest improvement over random initialization when annotation availability decreases in nuScenes semantic segmentation.The improvement is 0.4 point with 100% of annotations and increases to up to 8.7 points as the available annotation percentage decreases.
  • Feature Analysis: Feature similarity maps show locally coherent SLidR features within objects, including trucks and traffic lights, indicating object distinction without fine-tuning.Additional maps report similar coherence for trees, roads, vehicles, and bikes.

5. Conclusion

SLidR is a self-supervised image-to-Lidar distillation method for synchronized camera and Lidar data. Its superpixel-based representations transfer across multiple tasks and datasets, surpassing related state-of-the-art methods.

  • SLidR distills image representations into 3D models using synchronized Lidar and camera data.
  • Superpixels produce object-aware point representations suited to semantic segmentation and object detection.
  • SLidR representations transfer and generalize across multiple tasks and datasets.
  • SLidR surpasses related state-of-the-art methods.

A.1. Visual Inspection

Visual inspections show that SLidR features generally associate query points with corresponding objects across points and image pixels, while image-feature maps exhibit boundary-related spurious correlations. The method retains performance with self-supervised image pretraining and benefits slightly from transformer backbones.

  • Query points on trees, roads, vehicles, or bikes mostly correlate with pixels and points from the corresponding object category.
  • Image feature maps show more spurious correlations near object boundaries, potentially due to reduced resolution and imprecise ResNet-50 features.
  • The model classifies several vegetation, car, and pedestrian points across a sequence without backbone fine-tuning, despite some misclassifications.
  • Self-supervised MoCov2 pretraining causes no performance loss compared with supervised ImageNet pretraining for SLidR on nuScenes semantic segmentation.
  • A self-supervised transformer image backbone yields slightly higher performance than ResNet-50 and is compatible with SLidR.

A.3. Choice of the Superpixels Method

The supplementary analysis evaluates superpixel algorithms and counts, finding that parameter selection matters while SLidR is relatively insensitive to the algorithm once parameters are properly set.

  • SLidR uses SLIC with 150 superpixels per image for the main-paper results.
  • Felzenszwalb’s method uses parameters yielding at most 143 superpixels per nuScenes training image.
  • SLidR is less sensitive to the superpixel algorithm, such as FH versus SLIC, once its parameters are set correctly.
  • With 1% of available annotations, SLidR achieves the highest mIoU and ranks first on 9 nuScenes classes and 11 SemanticKITTI classes.

B. Data Augmentations

Training applies separate strong augmentations to point clouds and images while preserving sufficient point-pixel correspondences. Point clouds undergo geometric transformations and cuboid dropping, whereas images are flipped and crop-resized.

  • Point-cloud and image augmentations require updating point-pixel correspondences appropriately.
  • Point clouds receive random z-axis rotations, independent x/y flips, and random axis-aligned cuboid point dropping.
  • Dropped cuboids must preserve at least 1024 point-pixel pairs.
  • Images are horizontally flipped half the time and crop-resized to 416 × 224 with constraints on crop area and aspect ratio.
  • Random crops must preserve at least 1024 or 75% of pixel-point pairs.

C. Baselines’ Implementations Details

The baselines are adapted or retrained for autonomous-driving data with implementation choices designed to support fair comparison. PPKT† uses direct point-pixel sampling, whereas SLidR avoids this sampling through superpixel-based matching.

  • PPKT†: PPKT† adapts PPKT from indoor RGB-D data to autonomous driving because no publicly released code is available.The adaptation is referred to as PPKT†.
  • PPKT†: PPKT† shares the data augmentations, voxel-based 3D backbone, and cylindrical coordinate voxels used by the proposed method.
  • PPKT†: PPKT† uses one pixel per superpixel, strided image-backbone convolutions, and bilinear upsampling from 1/32 resolution.
  • PPKT†: Unlike SLidR, PPKT† randomly samples point-pixel pairs because evaluating all possible pairs is intractable; SLidR avoids this through superpixels.The sampled set can include point-pixel pairs from all scenes in a training batch.
  • Baseline differences: PPKT and PPKT† differ in coordinate systems, correspondence construction, and augmentation choices for indoor RGB-D versus autonomous-driving data.The differences include Cartesian versus cylindrical coordinates, direct versus projection-based correspondences, and distinct augmentation omissions.
  • PointContrast†: PointContrast† is retrained on nuScenes using registered keyframe pairs and nearest-point matches within 10 cm.The matching points are obtained after registration with ground-truth Lidar poses.

D. Additional Training Details

The study standardizes downstream training procedures across methods while tuning learning rates by dataset and method. It also defines a proportional nuScenes mini-validation split and notes the computational trade-offs of self-supervision.

  • Downstream training: All methods use a pointwise linear classification head trained for 50 epochs with a learning rate of 0.05.
  • Downstream training: SemanticKITTI and nuScenes fine-tuning each run for 100 epochs, with batch sizes of 10 and 16, respectively.
  • Downstream training: Learning rates for the classification head and backbone are optimized separately for each method and dataset, except for randomly initialized backbones.
  • Limited-label training: For varying annotation budgets, networks are fine-tuned for 100 epochs with 1% labeled data and 50 epochs for other percentages.
  • nuScenes split: The nuScenes training set contains 700 scenes: 310 in Singapore and 390 in Boston; the mini-validation split contains 100 scenes in the same proportions.The mini-validation split contains 44 Singapore scenes and 56 Boston scenes.
  • Computational considerations: Self-supervision can increase computational and environmental costs during pre-training, while pre-trained models can reduce downstream training time.The authors distribute their pre-trained models.
Loading 2203.16258v1…