PaperlayerView exact revision on arXiv

Source-linked AI summary

UniPAD: A Universal Pre-training Paradigm for Autonomous Driving

Honghui Yang, Sha Zhang, Di Huang, Xiaoyang Wu, Haoyi Zhu, Tong He, Shixiang Tang, Hengshuang Zhao, Qibo Qiu, Binbin Lin, Xiaofei He, Wanli Ouyang

arXiv:2310.08370v2cs.CV

TL;DR

Adapting self-supervised pre-training from 2D images to sparse, irregular 3D point clouds remains challenging. UniPAD uses masked inputs and differentiable volumetric rendering across modalities, and reports improvements across nuScenes detection and segmentation tasks.

  • Problem

    Extending 2D self-supervised approaches to point clouds is challenging because the data are sparse and vary with sensor placement and occlusion.

    Source: §1 · p. 1

  • Method

    UniPAD masks LiDAR points or multi-view images, encodes each modality, and uses volumetric neural rendering to pre-train representations.

    Source: §3 · p. 3

  • Results

    UniPAD reports gains across nuScenes 3D-backbone detection and segmentation and camera-based detection, with 79.4 mIoU for segmentation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The rendering-decoder approach is adaptable to both 2D and 3D frameworks, and the authors suggest cross-domain representation learning could draw on advances in either domain.

    Source: §5 · p. 8

    Explicitly converting point and image features into volumetric representations increases memory usage as voxel resolution rises.

    Source: §5 · p. 8

Abstract

from arXiv · show

In the context of autonomous driving, the significance of effective feature learning is widely acknowledged. While conventional 3D self-supervised pre-training methods have shown widespread success, most methods follow the ideas originally designed for 2D images. In this paper, we present UniPAD, a novel self-supervised learning paradigm applying 3D volumetric differentiable rendering. UniPAD implicitly encodes 3D space, facilitating the reconstruction of continuous 3D shape structures and the intricate appearance characteristics of their 2D projections. The flexibility of our method enables seamless integration into both 2D and 3D frameworks, enabling a more holistic comprehension of the scenes. We manifest the feasibility and effectiveness of UniPAD by conducting extensive experiments on various downstream 3D tasks. Our method significantly improves lidar-, camera-, and lidar-camera-based baseline by 9.1, 7.7, and 6.9 NDS, respectively. Notably, our pre-training pipeline achieves 73.2 NDS for 3D object detection and 79.4 mIoU for 3D semantic segmentation on the nuScenes validation set, achieving state-of-the-art results in comparison with previous methods. The code will be available at https://github.com/Nightmare-n/UniPAD.

1. Introduction

UniPAD addresses the difficulty of transferring 2D self-supervised pre-training ideas to sparse, irregular 3D point clouds. It uses differentiable volumetric rendering for pre-training across modalities and reports gains on nuScenes detection and segmentation tasks.

1. Introduction

  • Extending 2D self-supervised methods to point clouds is challenging because 3D data are sparse and vary with sensor placement and occlusion.Contrastive methods face sample-selection sensitivity and latency, while coarse voxelwise supervision may limit representation capability.Source: §1 · p. 1
  • On nuScenes, 3D-backbone pre-training improves detection by 9.1 NDS and segmentation by 6.1 mIoU; camera detection improves by 7.7 NDS, and segmentation reaches 79.4 mIoU.Multimodal detection reaches 73.2 NDS.Source: Figure 1 · p. 1, §1 · p. 2
  • UniPAD masks point clouds and uses differentiable 3D neural rendering to reconstruct missing geometry in projected depth images, providing continuous supervision without complex positive/negative assignments.The approach also extends to 2D backbones and uses memory-efficient ray sampling to reduce training costs and memory consumption.Source: §1 · p. 2
  • The authors report surpassing six pre-training strategies across seven backbones and two perception tasks, and identify their work as the first exploration of 3D differentiable rendering for autonomous-driving self-supervision.Their nuScenes experiments include 3D object detection and semantic segmentation.Source: §1 · p. 2

2. Related Work

Prior work develops self-supervised representations for point clouds and images, while neural rendering has been applied to perception and simulation in autonomous driving. UniPAD repurposes neural rendering for universal pre-training rather than novel-view synthesis.

2. Related Work

  • Point-cloud pre-training includes contrastive, masked-autoencoding, and other strategies, while image pre-training commonly uses contrastive or masked-patch reconstruction methods.ImageNet-pretrained models are also used for image-related autonomous-driving tasks, with depth estimation and monocular 3D detection serving as additional pre-training techniques.Source: §2 · p. 2
  • Neural rendering in autonomous driving supports perception tasks such as segmentation and detection, as well as simulation and LiDAR data generation.Prior simulation work includes scene modeling and strategies for synthesizing large-scale driving scenarios.Source: §2 · p. 2
  • Unlike prior neural-rendering work aimed at novel-view synthesis, UniPAD uses NeRF for universal pre-training across image-based and point-based models.Its framework combines modality-specific encoders with a volumetric rendering decoder.Source: §2 · pp. 2–3, Figure 2 · p. 3

3. Methodology

UniPAD uses modality-specific encoders and a volumetric rendering decoder to pre-train masked LiDAR and multi-view image inputs. Its shared 3D volume supports rendered color and depth supervision across modalities.

3. Methodology

  • UniPAD combines modality-specific encoders with a volumetric rendering decoder, training masked LiDAR or multi-view inputs to match rendered 2D projections and learn continuous geometry or appearance.The framework uses ray sampling to keep rendering memory-efficient.Source: §3 · p. 3

3.1. Modal-specific Encoder

  • The modality-specific encoders extract hierarchical point-cloud or image features, aggregate multi-scale information with FPNs, and use block-wise masking; image features are computed only at visible locations.Masked regions are zero-padded after encoding and combined with visible features to form dense feature maps.Source: §3.1 · p. 3

3.2. Unified 3D Volumetric Representation

  • A shared 3D voxel space preserves information that image-plane projection or bird’s-eye-view merging would omit; image features are projected and interpolated into it, while point features retain height.The image volume uses projected voxel coordinates to retrieve features and scaling factors through bilinear and trilinear interpolation.Source: §3.2 · pp. 3–4

3.3. Neural Rendering Decoder

  • The rendering decoder samples rays through volumetric features, represents geometry with an implicit SDF field, predicts color with an MLP, and applies color and depth losses to rendered outputs.Depth rendering also supplies supervision for image features.Source: §3.3 · pp. 4–5
  • Rendering all S×H×W rays is computationally costly, so dilation, random, and LiDAR depth-aware sampling reduce the ray set; depth-aware sampling selects projected pixels below a depth threshold.The sampling strategies are illustrated in Figure 4.Source: §3.3 · p. 5, Figure 4 · p. 5

4. Experiments

Experiments on nuScenes evaluate UniPAD across detection and segmentation, modalities, pre-training comparisons, backbone scales, and design choices. The reported results show gains over baselines and pre-training alternatives, with ablations examining masking, rendering, sampling, and feature projection.

4.1. Datasets and Evaluation Metrics; 4.2. Implementation Details

  • On nuScenes, experiments use 700 training, 150 validation, and 150 test scenes; the implementation trains models for 12 pre-training epochs with specified image and point masking ratios.The implementation uses 0.3 image masking and 0.8 point masking; evaluation includes NDS, mAP, and mIoU.Source: §4.1 · p. 5, §4.2 · p. 5

4.3. Comparison with State-of-the-Art Methods

  • Without test-time augmentation or model ensemble, UniPAD-L, UniPAD-C, and UniPAD-M achieve 70.6, 47.4, and 73.2 NDS, respectively, reaching the level of existing state-of-the-art methods.Against SpUNet implemented by Pointcept as baseline, UniPAD improves by 6.1 mIoU and achieves state-of-the-art performance on the validation set; it achieves 81.1 mIoU on the test set.Source: §4.3 · p. 6

4.4. Comparisons with Pre-training Methods.

  • UniPAD shows superior knowledge transfer to image-based pre-training alternatives and achieves the best NDS among the compared point-based methods, with slightly lower mAP than the contrast-based method.The point-based comparison notes that UniPAD avoids complex positive-negative sample assignments.Source: §4.4 · p. 7, Table 3 · p. 7, Table 4 · p. 7

4.5. Effectiveness on Various Backbones.

  • Across view transformations, UniPAD yields 5.2–6.3 NDS gains; modality and ConvNeXt-scale experiments also report consistent improvements.The ConvNeXt variants improve by +6.0–7.7 NDS and +8.2–10.3 mAP.Source: Table 5 · p. 7, §4.5 · p. 7, Table 6 · p. 7
  • UniPAD improves UVTR-L, UVTR-C, and UVTR-M by 9.1, 7.7, and 6.9 NDS, respectively.The reported comparison covers point, camera, and fusion modalities.Source: Table 6 · p. 7, §4.5 · p. 7

4.6. Ablation Studies

  • A 0.3 camera masking ratio is optimal; the authors attribute the lower ratio than prior MAE methods to the greater complexity of rendering images from volumes.For point modality, the authors use a 0.8 mask ratio, citing spatial redundancy in point clouds.Source: §4.6 · p. 7, Table 8 · p. 8
  • NeuS rendering improves NDS by 0.4 over UniSurf and 0.1 over VolSDF; decoder depth matters, while decoder width has relatively little impact.The default decoder dimension is 32 for efficiency.Source: §4.6 · pp. 7–8, Table 10 · p. 8, Table 11 · p. 8
  • Depth-aware ray sampling improves NDS by 0.4 over random sampling and 1.0 over dilation sampling; removing feature projection or randomly initializing FPN and view-transformation weights reduces performance.Removing projection from pre-training and fine-tuning lowers NDS by 1.8 and 2.7, respectively; random initialization lowers NDS by 0.2 for FPN and 0.7 for view transformation.Source: §4.6 · p. 8, Table 12 · p. 8, Table 13 · p. 8

5. Conclusion

UniPAD uses a unified NeRF rendering decoder to support pre-training across 2D and 3D frameworks and a range of 3D perception tasks. Its volumetric representations have a memory cost that increases with voxel resolution.

5. Conclusion

  • UniPAD uses NeRF as a unified rendering decoder and can be integrated into both 2D and 3D frameworks for 3D perception tasks.The approach is described as adaptable across domains, with potential to support representation learning through advances in either domain.Source: §5 · p. 8
  • UniPAD’s cross-domain adaptability could support adding semantic knowledge to point clouds through supervision from 2D models such as SAM.Source: §5 · p. 8
  • Explicitly converting point and image features into volumetric representations increases memory use as voxel resolution rises.Source: §5 · p. 8
Loading 2310.08370v2…