PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds

Siyuan Huang, Yichen Xie, Song-Chun Zhu, Yixin Zhu

arXiv:2109.00179v1cs.CV

TL;DR

3D scene understanding lacks practical, generalizable pre-trained models because labeled point-cloud data is supervision-starved and existing methods have limited generalization evidence. STRL learns invariant representations from augmented, temporally correlated point-cloud frames, achieving strong cross-task transfer and improved representation quality from combined spatio-temporal cues.

  • Problem

    Supervised training struggles to produce practical and generalizable 3D point-cloud pre-trained models because 3D data is supervision-starved.

  • Method

    STRL uses positive pairs of temporally correlated, spatially augmented point clouds and online-to-target prediction to learn invariant representations.

  • Results

    STRL outperforms prior unsupervised methods and transfers learned representations across 3D shape classification, object detection, and semantic segmentation tasks.

  • Takeaways & Limitations

    Combining spatial and temporal cues improves learned representations, while pre-training can generalize across synthetic, indoor, and outdoor domains.

  • Takeaways & Limitations

    Prior generalization evidence was limited across domains and tasks, and the paper identifies bridging domain gaps across jointly trained unlabeled data as future work.

Abstract

from arXiv · show

To date, various 3D scene understanding tasks still lack practical and generalizable pre-trained models, primarily due to the intricate nature of 3D scene understanding tasks and their immense variations introduced by camera views, lighting, occlusions, etc. In this paper, we tackle this challenge by introducing a spatio-temporal representation learning (STRL) framework, capable of learning from unlabeled 3D point clouds in a self-supervised fashion. Inspired by how infants learn from visual data in the wild, we explore the rich spatio-temporal cues derived from the 3D data. Specifically, STRL takes two temporally-correlated frames from a 3D point cloud sequence as the input, transforms it with the spatial data augmentation, and learns the invariant representation self-supervisedly. To corroborate the efficacy of STRL, we conduct extensive experiments on three types (synthetic, indoor, and outdoor) of datasets. Experimental results demonstrate that, compared with supervised learning methods, the learned self-supervised representation facilitates various models to attain comparable or even better performances while capable of generalizing pre-trained models to downstream tasks, including 3D shape classification, 3D object detection, and 3D semantic segmentation. Moreover, the spatio-temporal contextual cues embedded in 3D point clouds significantly improve the learned representations.

1. Introduction

STRL addresses the shortage of practical, generalizable 3D point-cloud pre-training by learning invariant representations from unlabeled spatio-temporal data. It transfers these representations across several downstream tasks and domains, with spatial and temporal cues improving performance.

  • 3D point clouds are widely used for scene understanding, but their annotation is substantially more difficult than labeling conventional 2D images.
  • Simplicity: Existing self-supervised point-cloud methods emphasize spatial reconstruction, motivating simpler designs that also exploit temporal contextual cues.
  • Method: STRL learns invariant representations from two temporally correlated point-cloud inputs using spatial augmentation and online-to-target prediction.
  • Results: 90.9% 3D shape classification accuracy is achieved on ModelNet40 with linear evaluation, alongside improvements in semi-supervised learning and downstream transfer.
  • Results: 3% accuracy improvement results from combining spatial and temporal cues rather than using either augmentation type alone.
  • Generalizability: ShapeNet pre-training can achieve comparable or better performance on downstream tasks involving complex physical-world data.

2. Related Work

Related work covers point-cloud representation learning and unsupervised learning, contrasting spatially aware architectures with generative and discriminative approaches. STRL extends BYOL-style learning to point clouds and targets broader downstream generalization.

  • Point clouds are unordered sets of vectors, creating representation-learning challenges that require permutation-invariant processing.
  • PointNet uses max pooling to form a global feature vector, while later methods introduce hierarchy and convolution-based structures.
  • Generative unsupervised methods reconstruct inputs or model latent distributions, whereas discriminative methods learn from generated labels or contrastive mechanisms.
  • Following BYOL, STRL excludes negative pairs and uses a moving-average target network to learn stable, invariant representations.
  • Earlier point-cloud self-supervised methods mainly demonstrated shape-classification performance on synthetic datasets rather than higher-level natural-scene tasks.

3. Spatio-temporal Representation Learning

STRL constructs temporal point-cloud sequences from natural sensor data or synthetic transformations, then trains online and target networks to match representations of augmented correlated frames. The learned backbone is retained for downstream use.

  • 3.1. Building Temporal Sequence of Point Clouds: The method builds point-cloud sequences from natural RGB-D observations or synthetic transformations to represent temporal viewpoint changes.
  • Natural Sequence: Natural sequences backproject depth images using camera extrinsic and intrinsic parameters to obtain point clouds in world coordinates.
  • Synthetic Sequence: Synthetic sequences consecutively rotate, translate, and scale an original point cloud, with each sampled transformation emulating a temporal view change.
  • 3.2. Representation Learning: The online network predicts the target representation through a predictor, while the target parameters follow an exponential moving average of the online parameters.
  • 3.2. Representation Learning: STRL samples two frames, applies separate spatial augmentations, and feeds one to the online network and the other to the target network.
  • 3.2. Representation Learning: The objective symmetrizes normalized online-to-target prediction errors in both directions, and only the online backbone is retained after training.

4. Implementation Details

STRL uses stochastic spatial and temporal transformations to generate training inputs, then optimizes online and target networks with a self-supervised loss. Training includes point-cloud sampling, normalization, augmentation probabilities, and cosine-scheduled optimization.

  • Temporal and spatial transformations: Random rotation, translation, and scaling construct synthetic temporal transformations for point-cloud sequence generation.Rotation angles are sampled within 15°, translation within 10% of point-cloud dimensions, and scaling factors from [0.8, 1.25].
  • Temporal and spatial transformations: Spatial augmentation changes local geometry through random cropping and random cutout to improve spatial-structure representation.Cropping retains a randomly sized cuboid patch, while cutout removes a random 3D cuboid.
  • Optimization pipeline: The training algorithm samples temporal-correlated point-cloud batches, applies spatial augmentations, projects inputs, computes cross-network losses, and updates the online and target networks.The total loss combines two directional terms, followed by optimization of the online encoder and predictor and target-network updating.
  • Input processing: Point clouds are down-sampled to the encoder input dimension and normalized to a unit sphere for synthetic training.Down-sampling randomly selects the necessary number of 3D points.
  • Augmentation sampling: Each sampled transformation is applied with probability 0.5, while cropping and cutout specifically use probability 0.5 because they alter spatial structure more evidently.The figure illustrates stochastic spatial augmentation and temporal sequence generation.
  • Optimization pipeline: Training uses LARS with cosine learning-rate decay, a 10-epoch warm-up, and an exponential-moving-average target parameter increasing from 0.996 toward 1.Most pre-trained models use batch sizes from 64 to 256 across 8 TITAN RTX GPUs; projector and predictor modules are MLPs.

5. Experiment

The experiment section organizes evaluation into pre-training, downstream-task assessment, and analyses of model modules and parameters. It also includes additional analytic experiments and discussions of open problems.

  • Experimental organization: Experiments first pre-train STRL on various data sources, then evaluate the resulting models on downstream tasks.The section structure separates pre-training in Sect. 5.1 from downstream evaluation in Sect. 5.2.
  • Experimental organization: The study analyzes the effects of different model modules and parameters through ablative and analytic experiments.These analyses are presented in Sect. 5.3 alongside discussions of open problems.
  • Experimental organization: The experimental program spans pre-training, downstream evaluation, and additional discussion of model behavior and open problems.

5.1. Pre-training

STRL pre-trains on synthetic shapes and natural indoor or outdoor scenes by constructing temporally related point-cloud pairs, then applying spatial augmentation. Natural sequences use correlated frames from RGB-D or lidar data, while ShapeNet uses synthetic transformations.

  • Pre-training strategy: STRL constructs point-cloud sequences and samples input pairs to learn spatio-temporal invariance across synthetic and natural data.Synthetic shapes and natural indoor/outdoor scenes use different sequence-generation and pair-sampling strategies.
  • Synthetic shapes: ShapeNet pre-training uses 57,448 synthetic objects from 55 categories, augmented into temporally correlated views before spatial augmentation.Two views are generated with the temporal transformations defined in Eq. (2).
  • Natural indoor and outdoor scenes: Natural RGB-D sequences provide temporally correlated frames captured from smoothly changing camera poses, enabling temporal-invariance learning from adjacent frames.Depth images are converted into point clouds using camera geometry.
  • Natural indoor and outdoor scenes: ScanNet pre-training uses 1,513 reconstructed meshes from 707 unique scenes and roughly 25 thousand keyframes after sub-sampling every 100 frames.Two random frames are sampled within fixed-length sliding windows and back-projected into a shared world coordinate.
  • Natural indoor and outdoor scenes: KITTI pre-training uses point-cloud sequences from over 100 scenes across six categories, recorded at roughly 10 FPS with about 120,000 points per frame.Keyframes are sub-sampled and paired within sliding windows similarly to ScanNet.
  • Pre-training strategy: Synthetic temporal transformations and spatial augmentation are applied to natural-scene point-cloud pairs to increase data diversity.

5.2. Downstream Tasks

STRL representations are evaluated across shape classification, indoor detection and segmentation, and outdoor detection. Pre-training generally improves downstream performance, including under limited labels and cross-dataset transfer.

  • Shape Understanding: STRL outperforms state-of-the-art unsupervised and self-supervised methods on ModelNet40 linear shape classification.
  • Shape Understanding: Up to 0.9% higher final classification accuracy is achieved after supervised fine-tuning on ModelNet40.The result is comparable to the state-of-the-art supervised learning method.
  • Shape Understanding: 2.1% and 1.6% performance gains occur when 1% and 20% of ModelNet40 training samples are available, respectively.The gains indicate stronger performance with limited labeled data.
  • Shape Understanding: t-SNE embeddings separate most ModelNet10 categories, except visually similar dressers and night stands.The visualization uses PointNet and DGCNN models pre-trained on ShapeNet.
  • Indoor Scene Understanding: 1.2 mAP improvement over training VoteNet from scratch and 0.7 mAP over a state-of-the-art self-supervised method are reported on SUN RGB-D detection.The ScanNet-pre-trained representation transfers to a different dataset through fine-tuning.
  • Outdoor Scene Understanding: STRL-pre-trained models outperform models trained from scratch across all three KITTI detection categories, with marked elevation for cyclists.Frozen-backbone results remain comparable to training from scratch.

5.3. Analytic Experiments and Discussions

The analytic experiments examine STRL's cross-domain transfer, temporal and spatial transformations, data efficiency, and robustness. Results show strong transfer behavior, sensitivity to augmentation choices, limited dependence on sampling density, and stable performance with simple augmentations.

  • Generalizability: ScanNet vs ShapeNet Pre-training: Pre-training on natural ScanNet scenes successfully transfers to synthetic shape classification with comparable performance.The cross-domain experiments compare ScanNet and ShapeNet pre-training for generalizability across domains.
  • Generalizability: ScanNet vs ShapeNet Pre-training: ShapeNet pre-training outperforms ScanNet pre-training for SUN RGB-D object detection in the reported opposite observation.The authors suggest cleaner spatial structures, encoder capacity, and ScanNet's 707-scene scope as possible explanations.
  • Spatial Data Augmentation: Random crop improves performance, whereas random cutout hurts it by breaking structural continuity needed for neighbor-based point-wise feature aggregation.This result comes from disabling different spatial augmentations in the ablation study.
  • Data Efficiency: Using 25,000 ScanNet frames reduces ModelNet40 performance by only about 0.5% versus training on the full 0.4 million-frame set.The finding applies to both linear evaluation and fine-tuning; the authors hypothesize that diversity matters more than sampling density.
  • Robustness: STRL robustly achieves about 85% accuracy on ModelNet40 linear classification using simple augmentations.The authors contrast this limited augmentation sensitivity with reported ImageNet effects of up to 10% and hypothesize that point-cloud down-sampling introduces useful structural noise.

6. Conclusion

The paper concludes that STRL provides a simple spatio-temporal self-supervised framework for learning 3D point-cloud representations and transferring them across downstream scene-understanding tasks. It identifies holistic 3D scene understanding and joint training across unlabeled domains as future directions.

  • Conclusion: STRL uses a simple structure to learn 3D point-cloud representations without labels.The conclusion characterizes the framework as spatio-temporal self-supervised learning.
  • Conclusion: The learned representations show promising transfer to various downstream 3D scene-understanding tasks.The conclusion states transfer as the central demonstrated outcome without specifying individual task metrics.
  • Conclusion: Future work includes extending STRL to holistic 3D scene understanding and bridging domain gaps by jointly training on unlabeled data from multiple domains.These directions are stated as extensions rather than established results.
Loading 2109.00179v1…