PaperlayerView exact revision on arXiv

Source-linked AI summary

Transfer Learning from Synthetic to Real LiDAR Point Cloud for Semantic Segmentation

Aoran Xiao, Jiaxing Huang, Dayan Guan, Fangneng Zhan, Shijian Lu

arXiv:2107.05399v2cs.CV

TL;DR

The paper targets the shortage of large-scale annotated synthetic LiDAR data and effective 3D synthetic-to-real transfer methods. It introduces SynLiDAR and PCT, which separately translate appearance and sparsity, and reports consistent improvements across three transfer-learning setups.

  • Problem

    3D synthetic-to-real LiDAR segmentation is limited by scarce large-scale synthetic datasets and transfer methods suited to point clouds.

  • Method

    The paper creates SynLiDAR and designs PCT to translate synthetic point clouds by separately addressing appearance and sparsity differences.

  • Results

    SynLiDAR and PCT consistently improve point-cloud segmentation across data augmentation, semi-supervised domain adaptation, and unsupervised domain adaptation.

  • Takeaways & Limitations

    SynLiDAR provides a large annotated source for 3D transfer research, while PCT reduces synthetic-to-real gaps in the input data.

Abstract

from arXiv · show

Knowledge transfer from synthetic to real data has been widely studied to mitigate data annotation constraints in various computer vision tasks such as semantic segmentation. However, the study focused on 2D images and its counterpart in 3D point clouds segmentation lags far behind due to the lack of large-scale synthetic datasets and effective transfer methods. We address this issue by collecting SynLiDAR, a large-scale synthetic LiDAR dataset that contains point-wise annotated point clouds with accurate geometric shapes and comprehensive semantic classes. SynLiDAR was collected from multiple virtual environments with rich scenes and layouts which consists of over 19 billion points of 32 semantic classes. In addition, we design PCT, a novel point cloud translator that effectively mitigates the gap between synthetic and real point clouds. Specifically, we decompose the synthetic-to-real gap into an appearance component and a sparsity component and handle them separately which improves the point cloud translation greatly. We conducted extensive experiments over three transfer learning setups including data augmentation, semi-supervised domain adaptation and unsupervised domain adaptation. Extensive experiments show that SynLiDAR provides a high-quality data source for studying 3D transfer and the proposed PCT achieves superior point cloud translation consistently across the three setups. SynLiDAR project page: \url{https://github.com/xiaoaoran/SynLiDAR}

1 Introduction

The paper addresses limited annotated real LiDAR data and the lack of effective 3D synthetic-to-real transfer by introducing SynLiDAR and PCT. SynLiDAR supplies large-scale annotated synthetic data, while PCT separately handles appearance and sparsity gaps.

  • Real LiDAR segmentation needs extensive point-wise annotations, but existing datasets remain small because labeling and collecting 3D data are time-consuming.
  • SynLiDAR provides over 19 billion point-wise annotated points across 32 semantic classes from multiple virtual environments.
  • PCT decomposes the synthetic-to-real domain gap into appearance and sparsity components and processes them with separate translation modules.
  • PCT is presented as the first translation method for LiDAR point clouds in natural scenes.
  • The paper evaluates synthetic-to-real transfer through data augmentation, semi-supervised domain adaptation, and unsupervised domain adaptation.

2 Related Works

Related work covers LiDAR segmentation, real and synthetic point-cloud datasets, transfer learning, and domain translation. The paper positions SynLiDAR and PCT against limited real annotations, insufficient prior synthetic data, and translation methods developed mainly for other settings.

  • LiDAR Point Cloud Segmentation: LiDAR segmentation methods include projection-based, point-based, sparse-convolutional, and customized 3D convolutional approaches.
  • LiDAR Sequential Point Cloud Datasets: Existing real LiDAR datasets have limited sizes because point-wise semantic labeling is prohibitively time-consuming.
  • Synthetic LiDAR Point Cloud Datasets: Earlier GTA-V-based synthetic point-cloud datasets used inaccurate 3D meshes and only two object classes, limiting fine-grained segmentation research.
  • Transfer Learning of Point Cloud: The paper studies data augmentation, semi-supervised domain adaptation, and unsupervised domain adaptation as distinct point-cloud transfer-learning settings.
  • Domain Translation: Prior domain-translation work largely targets 2D images or 3D objects, whereas LiDAR-scene translation remains comparatively underexplored.

3 The SynLiDAR Dataset

SynLiDAR is constructed from realistic virtual scenes and accurate object models, yielding extensive point-wise annotations across diverse semantic classes. PCT complements the dataset by translating synthetic clouds toward real appearance and sparsity.

  • Dataset Construction: SynLiDAR uses multiple realistic outdoor virtual scenes, physically accurate object models, and automatically collected coordinates and point-wise annotations.
  • Dataset Construction: SynLiDAR includes predicted intensity values learned from real LiDAR point clouds to address the difficulty of simulating LiDAR intensity.
  • Dataset Statistics: SynLiDAR contains 13 sequences, 198,396 scans, 19 billion points, and 32 annotated semantic classes.
  • Dataset Statistics: SynLiDAR is described as an ideal transfer-learning source because it exceeds existing datasets in point numbers and semantic classes.
  • Point Cloud Translation: PCT separates translation into appearance reconstruction and sparsity translation, then fuses the learned sparsity with the reconstructed point cloud.

4 Point Cloud Translation

PCT addresses the synthetic-to-real gap in LiDAR point clouds by separating appearance and sparsity differences, then translating synthetic data through specialized modules. The translated data improves semantic segmentation when combined with real training data in SemanticKITTI and SemanticPOSS.

  • Synthetic-to-real gaps arise from differences in continuous-scene appearance and LiDAR sampling sparsity, motivating separate treatment of both components.The paper frames point clouds as discrete samplings of a continuous geometric world.
  • PCT translates synthetic point clouds toward real appearance and sparsity by up-sampling inputs, applying dedicated generators, and fusing the translated representations.ATM handles appearance, while STM projects point clouds into images to learn real sparsity information before projecting it back into 3D.
  • ATM uses adversarial 3D generation after up-sampling synthetic and real point clouds to reduce the influence of domain-specific sparsity during appearance translation.A geometry-preserving EMD term is included to keep the translated representation consistent with the synthetic geometry.
  • Combining SynLiDAR with SemanticKITTI or SemanticPOSS improves segmentation, while adding PCT-translated SynLiDAR further improves the results.The reported data-augmentation experiments cover both SemanticKITTI and SemanticPOSS.
  • STM learns real sparsity in projected depth images, uses geometry consistency, and guides 3D point dropout before assigning labels from neighboring original points.This design addresses the inability of standard Chamfer or EMD supervision to capture LiDAR sparsity directly.

5 Experiments

Experiments evaluate SynLiDAR and PCT across data augmentation, semi-supervised domain adaptation, and unsupervised domain adaptation on SemanticKITTI and SemanticPOSS. SynLiDAR improves segmentation, while PCT-translated data further reduces the synthetic-to-real gap across these setups.

  • Experimental setup: Experiments use SemanticKITTI and SemanticPOSS with MinkowskiNet, evaluating data augmentation, SSDA, and UDA.SemanticKITTI and SemanticPOSS are the two real-world targets, and mean Intersection over Union (mIoU) is used for evaluation.
  • Data augmentation: SynLiDAR improves mIoU by 2.2% on SemanticKITTI and 2.6% on SemanticPOSS when augmenting real point-cloud training data.These results indicate that SynLiDAR provides a high-quality synthetic source for transfer learning.
  • Data augmentation: PCT-translated SynLiDAR improves augmentation mIoU by 2.2% on SemanticKITTI and 2.6% on SemanticPOSS.The reported gains demonstrate PCT's effectiveness in mitigating the domain gap for these datasets.
  • Data augmentation: SynLiDAR augmentation saves up to 40% of SemanticPOSS without sacrificing segmentation performance, while PCT-translated data further improves results.The improvement holds across different proportions of SemanticPOSS data.
  • Semi-supervised domain adaptation: In 1-shot SSDA, PCT-translated SynLiDAR improves mIoU by 2.2% on SemanticKITTI and 4.5% on SemanticPOSS.Combining PCT with APE reaches 27.0% and 31.2% on the two datasets, respectively.
  • Unsupervised domain adaptation: In UDA, PCT-translated SynLiDAR improves mIoU by 3.5% on SemanticKITTI and 2.8% on SemanticPOSS.PCT complements ST, with their combination reaching 28.9% and 29.6%, respectively.

6 Conclusion

The paper presents SynLiDAR and PCT for synthetic-to-real transfer in LiDAR segmentation. SynLiDAR supplies large-scale annotated synthetic data, while PCT translates it toward real appearance and sparsity, improving segmentation across three transfer setups.

  • Conclusion: SynLiDAR contains 19 billion points with point-wise annotations for 32 semantic classes.The dataset supports fine-grained synthetic-to-real transfer learning for LiDAR point-cloud segmentation.
  • Conclusion: PCT translates synthetic point clouds to have similar appearance and sparsity as real point clouds.The method addresses two components of the synthetic-to-real gap.
  • Conclusion: Across data augmentation, semi-supervised domain adaptation, and unsupervised domain adaptation, PCT-translated data consistently further improves point-cloud segmentation.The conclusion reports consistent gains across all three transfer-learning setups.
Loading 2107.05399v2…