PaperlayerView exact revision on arXiv

Source-linked AI summary

PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection

Anthony Chen, Kevin Zhang, Renrui Zhang, Zihan Wang, Yuheng Lu, Yandong Guo, Shanghang Zhang

arXiv:2303.08129v1cs.CVcs.AI

TL;DR

MAE pre-training has largely focused on single modalities, while interactive learning from point clouds and RGB images remains challenging. PiMAE uses complementary masking, a shared-decoder two-branch MAE, and cross-modal reconstruction; experiments report improvements across evaluated downstream methods and tasks.

  • Problem

    Existing MAE pre-training objectives are limited to single modalities, and directly combining paired 2D pixels and 3D points can yield marginal or degraded performance.

    Source: §1 · pp. 1–2

  • Method

    PiMAE pre-trains paired point clouds and images with complementary mask alignment, a shared decoder for mask-token interaction, and cross-modal reconstruction.

    Source: §1 · p. 2, §3.5 · p. 5

  • Results

    PiMAE improves state-of-the-art methods across evaluated downstream tasks, including 2D and 3D detection.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments and ablations indicate PiMAE has potential as an interactive multimodal learning pipeline for point-cloud and image features.

    Source: §5 · p. 8

    Uniform masking can project onto semantically weaker masked image features, which may negatively influence the ablation study.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

Masked Autoencoders learn strong visual representations and achieve state-of-the-art results in several independent modalities, yet very few works have addressed their capabilities in multi-modality settings. In this work, we focus on point cloud and RGB image data, two modalities that are often presented together in the real world, and explore their meaningful interactions. To improve upon the cross-modal synergy in existing works, we propose PiMAE, a self-supervised pre-training framework that promotes 3D and 2D interaction through three aspects. Specifically, we first notice the importance of masking strategies between the two sources and utilize a projection module to complementarily align the mask and visible tokens of the two modalities. Then, we utilize a well-crafted two-branch MAE pipeline with a novel shared decoder to promote cross-modality interaction in the mask tokens. Finally, we design a unique cross-modal reconstruction module to enhance representation learning for both modalities. Through extensive experiments performed on large-scale RGB-D scene understanding benchmarks (SUN RGB-D and ScannetV2), we discover it is nontrivial to interactively learn point-image features, where we greatly improve multiple 3D detectors, 2D detectors, and few-shot classifiers by 2.9%, 6.7%, and 2.4%, respectively. Code is available at https://github.com/BLVLab/PiMAE.

1. Introduction

The introduction frames interactive learning from paired point clouds and RGB images as a challenge: existing MAE objectives focus on single modalities, while direct fusion can yield marginal or degraded performance. PiMAE combines complementary masking, a shared decoder, and cross-modal reconstruction, and improves evaluated downstream methods.

1. Introduction

  • Directly combining paired 2D pixels and 3D points can yield marginal or degraded performance, despite their complementary scene perspectives.The introduction identifies designing a model that interacts with both modalities as difficult.Source: §1 · p. 1
  • Existing MAE pre-training objectives address only a single modality, leaving interactive point-cloud and RGB learning underexplored.The introduction also notes that prior multi-modal approaches have not shown promising results bridging 3D and 2D data.Source: §1 · p. 2
  • PiMAE pre-trains paired point and image inputs in a two-branch MAE framework designed to increase interaction between their features.The method learns embeddings individually for the two modalities before applying three interaction-focused design features.Source: §1 · p. 2
  • The three interaction features are projected token alignment for complementary masking, a shared decoder for mask-token interaction, and cross-modal reconstruction.The reconstruction module uses image features to encourage point-cloud features to encode image-level understanding.Source: §1 · p. 2
  • PiMAE improves state-of-the-art methods across evaluated downstream tasks, including 2D and 3D detection and classification.The evaluation includes RGB-D scene datasets and multiple 2D detection and classification datasets.Source: §1 · p. 2

2. Related Work

Related work covers 3D detection, point-cloud/image joint representation learning, and masked autoencoders. The paper positions PiMAE as an MAE-based approach to interactive RGB and point-cloud learning amid limitations in prior methods.

3D Object Detectors

  • Transformer-based 3D detectors include end-to-end point-cloud detection and Group-Free-3D, which fuses object features across stages.The paper draws on both projection-based and attention-based detector designs when developing its multi-modal framework.Source: §2 · p. 2

Point Cloud and Image Joint Representation Learning

  • Point-cloud/image joint-learning methods use cross-modal modules across tasks, while contrastive self-supervision pairs RGB and point clouds for downstream detection.The paper says contrastive methods can suffer sampling bias in negative sampling and augmentation, making them impractical in real-world scenarios.Source: §2 · pp. 2–3

Masked Autoencoders (MAE)

  • MAE predicts pixels from highly masked images, and later work applies reconstruction to point-cloud tokens and aligns multiple modalities.The paper describes MultiMAE and I2P-MAE as related efforts, while arguing earlier methods do not easily incorporate RGB inputs and bring only trivial performance gain.Source: §2 · p. 3
  • PiMAE is presented as an MAE approach aligning RGB images with point clouds, unlike prior objectives described as single-modality or limited in RGB integration.The paper calls this the pioneering work aligning RGB images with point clouds through MAE pre-training.Source: §2 · p. 3

3. Methods

PiMAE pre-trains point clouds and images jointly, aligning their masks and combining modality-specific learning with cross-modal interaction. Its pipeline uses separate and shared autoencoder stages, followed by cross-modal reconstruction.

3.1. Pipeline Overview

  • PiMAE embeds and randomly masks point tokens, aligns image masks to them, and follows separate, shared, then separate autoencoder stages.The pipeline concludes with cross-modal reconstruction to strengthen point-cloud representation learning.Source: §3.1 · p. 3, Figure 3 · p. 4

3.2. Token Projection and Alignment

  • Visible point tokens are projected onto the image plane, and their corresponding image patches are masked to create complement alignment.The authors argue this makes visible tokens more semantically abundant than under uniform masking.Source: §3.2 · pp. 3–4, Figure 2 · p. 3

3.3. Encoding Phase

  • Separate modality-specific encoders extract image and point features, then a shared encoder fuses their visible-token representations.Aligned tokens are used for interaction because they reveal similar information in both modalities.Source: §3.3 · p. 4

3.4. Decoding Phase

  • A shared decoder processes visible features and mask tokens from both modalities before specialized decoders reconstruct image and point-cloud outputs.The shared stage lets mask tokens engage in cross-modal fusion, while separate decoders address modality-specific reconstruction.Source: §3.4 · pp. 4–5

3.5. Cross-modal Reconstruction

  • Point tokens predict corresponding visible image features in a cross-modal reconstruction stage, alongside separate point-cloud and image reconstruction losses.The method avoids visible point tokens for this stage because complement masking pairs them with masked image features.Source: §3.5 · p. 5

4. Experiments

PiMAE is evaluated across 3D and 2D detection, few-shot classification, and ablations of its multimodal design. Its pre-training improves the reported downstream baselines, while the ablations examine the roles of cross-modal masking, reconstruction, and joint training.

4.1. Implementation Details

  • PiMAE is pre-trained on SUN RGB-D for 400 epochs with a 60% masking ratio and no input augmentation.The point branch reduces 20,000 sampled points to 2,048 before its ViT backbone; the image branch uses 16 × 16 patches. During fine-tuning, each task uses its modality-specific encoder and the shared encoder.Source: §4.1 · p. 5

4.2. Results on Downstream Tasks

  • PiMAE improves indoor 3D detection baselines on ScanNetV2 and SUN RGB-D, and also improves MonoDETR on the outdoor KITTI benchmark.For indoor detection, gains are reported for 3DETR and GroupFree3D across datasets and criteria; the KITTI result is presented as evidence of generalization beyond the indoor pre-training data.Source: §4.2 · p. 6, Table 1 · p. 6, Table 3 · p. 7
  • PiMAE significantly improves DETR by replacing its vanilla transformer backbone for ScanNetV2 2D detection, and surpasses CrossPoint in few-shot image classification.The comparison reports top-1 accuracy on CIFAR-FS, FC100, and miniImageNet test sets under 5-way 1-shot and 5-way 5-shot settings.Source: §4.2 · pp. 6–7, Table 4 · p. 8

4.3. Ablation Study

  • Complement masking outperforms random masking in 3D detection, while uniform masking aligns the modalities without improving use of 2D information.The attention visualization reports higher attention values on foreground objects; it displays query-point attention to visible image tokens and the projected query locations.Source: §4.3 · pp. 7–8, Figure 5 · p. 7, Table 5 · p. 8
  • Adding feature-level cross-modal reconstruction targets brings further 3D detection gains and is described as encoding 2D knowledge into the feature extractor.The ablation compares reconstruction targets including coordinates, features, and pixels.Source: §4.3 · p. 7, Table 6 · p. 8
  • Single-branch pre-training shows a critical performance drop relative to PiMAE’s double-branch pre-training on 3D detection and few-shot classification.The compared tasks are SUN RGB-D 3D detection and CIFAR-FS few-shot classification.Source: §4.3 · p. 8, Table 7 · p. 8
  • At 10% of labels, PiMAE-based 3DETR exceeds 3DETR by 17.4% AP25, and PiMAE outperforms the baseline at every tested label level.The models are evaluated on the full SUN RGB-D validation set after training with label amounts ranging from 1% to 100%.Source: §4.3 · p. 8, Figure 6 · p. 8

5. Conclusion

The conclusion presents PiMAE as an effective, interactive learning pipeline for point clouds and images. It attributes this design to aligned masking, a shared decoder, and cross-modal reconstruction, and reports improvements across multiple baselines and tasks.

5. Conclusion

  • PiMAE is presented as an effective multimodal pipeline for feature extraction from point clouds and images.The conclusion reports improvements across multiple baselines and tasks.Source: §5 · p. 8
  • Its three interaction components are aligned point-cloud and image mask patterns, a shared decoder for both modalities’ mask tokens, and cross-modal reconstruction.The conclusion says cross-modal reconstruction enhances learned semantics.Source: §5 · p. 8
  • Experiments and ablations on datasets of both modalities show improvements across multiple baselines and tasks.Source: §5 · p. 8

A. Supplementary Implementation Details

This supplementary section outlines its sequence: dataset descriptions, baseline-model details, and experiment configurations.

A. Supplementary Implementation Details

A.1. Datasets and metric

The supplementary section describes the evaluated datasets and detection metrics. It includes indoor 3D, outdoor 3D, 2D detection, and few-shot classification settings.

A.1. Datasets and metric

  • SUN RGB-D contains 10,335 RGB-D images and 37 categories; its test-set 3D detection is evaluated with mAP at IoU thresholds 0.25 and 0.5.Depth images are converted to point clouds using the provided camera poses, with standard 5,285/5,050 training and testing splits.Source: §A.1 · p. 12
  • ScanNetV2 has 1,513 indoor scenes and 18 object classes; detection metrics are 3D mAP at IoU 0.25 and 0.5 and 2D AP at 0.5, 0.75, and 1.0.KITTI evaluation uses car-category 3D-box AP at IoU 0.7 across easy, moderate, and hard difficulty levels.Source: §A.1 · p. 12
  • CIFAR-FS, FC100, and miniImageNet evaluate image features using top-1 accuracy in 5-way 1-shot and 5-way 5-shot settings.Source: §A.1 · p. 12

A.2. Baseline Approaches

The evaluation fine-tunes PiMAE with three state-of-the-art 3D detectors; the section also describes the detection baselines used across 3D, 2D, and monocular 3D tasks.

A.2. Baseline Approaches

  • PiMAE is evaluated by fine-tuning three state-of-the-art 3D object detectors, with baseline implementations following their released code.Source: §A.2 · p. 12
  • 3DETR uses an attention-based backbone that preserves vanilla Transformer blocks without requiring a finely crafted 3D detection backbone.Source: §A.2 · p. 12
  • Group-Free 3D uses object query locations and ensembling, applying attention across all point-cloud points rather than grouping points locally for each candidate.Source: §A.2 · p. 12
  • DETR detects 2D objects with Transformer-based object queries and bipartite matching, while MonoDETR predicts 3D annotations from image-inherent depth without depth supervision, anchors, or NMS.Source: §A.2 · p. 12

A.3. Pre-training Details

PiMAE uses a compact Transformer architecture and paired point-cloud and image inputs for pre-training. Fine-tuning adapts its pretrained feature extractors to several detector pipelines with reduced encoder learning rates in specified cases.

A.3. Pre-training Details

  • PiMAE uses three Transformer blocks in specific and shared encoders, two in specific decoders, and one in shared decoders.Encoder blocks have 256 hidden dimensions and four attention heads; decoder blocks use 192 dimensions and three heads.Source: §A.3 · p. 12
  • Pre-training samples 2,048 SUN RGB-D points per scene and resizes images to 256×352 pixels, patchified at size 16.Source: §A.3 · p. 12

A.4. Fine-tuning Protocol on SUN RGB-D, ScanNetV2, and KITTI

  • For GroupFree3D fine-tuning, the encoder learning rate is lowered to 3e-5 to preserve the pre-trained prior; for detection on ScanNetV2, it is lowered to 6e-5.Source: §A.4 · pp. 12–13
  • For 3DETR, the six-block PiMAE-pretrained encoder is fine-tuned with a 1e-5 learning rate while the original setup is otherwise retained.Source: §A.4 · p. 13
  • DETR replaces its vanilla encoder with PiMAE image and joint encoders, while MonoDETR replaces its depth encoder with PiMAE’s three-layer 3D feature extractor.Both use the stated original training configurations apart from DETR’s reduced encoder learning rate of 1e-5.Source: §A.4 · p. 13

B. Additional Ablation Study

Additional ablations examine the shared-decoder architecture and masking ratio. They report a performance benefit for shared decoding and the strongest latent features at a 60% masking ratio.

Ablation study on Pipeline Architecture

  • The shared-decoder design enhances 3DETR performance by accounting for cross-modal influence among masked tokens.The decoder first disentangles encoded features, after which task-specific decoders complete reconstruction.Source: B. Additional Ablation Study · p. 13
  • The reported ablation attributes improved feature extraction to interactions among masked tokens in the shared decoder.Source: B. Additional Ablation Study · p. 13

Ablation Study on Masking Ratio

C. Additional Visualization

The additional visualizations show simultaneous image and point-cloud reconstruction, cross-modal attention patterns, and qualitative object detections. The attention examples compare PiMAE’s final design with variants that omit proposed components.

C. Additional Visualization

Related papers

Loading 2303.08129v1…