PaperlayerView exact revision on arXiv

Source-linked AI summary

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Ziyi Wang, Xumin Yu, Yongming Rao, Jie Zhou, Jiwen Lu

arXiv:2208.02812v2cs.CVcs.AIcs.LG

TL;DR

Limited and costly 3D data make it difficult to pre-train fundamental point-cloud models. P2P transfers pre-trained image knowledge to point clouds by projecting and coloring them for frozen image models, and reports competitive classification performance and part-segmentation potential.

  • Problem

    Pre-training fundamental 3D models is difficult because point-cloud data are limited, inconvenient, and costly to collect.

    Source: §1 · p. 2

  • Method

    P2P projects point-cloud geometry into image layouts, predicts colors for the projected features, and uses the resulting images with frozen pre-trained image models.

    Source: §1 · p. 2, §3.3.1 · p. 5, §3.3.2 · p. 5

  • Results

    P2P achieves results comparable to the best object-classification methods on ModelNet40 and ScanObjectNN with much fewer trainable parameters, and demonstrates potential for ShapeNetPart segmentation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    P2P results support the feasibility of transferring pre-trained image knowledge to point-cloud tasks, with larger image models associated with higher classification performance.

    Source: §4.2.1 · p. 6, §5 · p. 10

    P2P may have difficulty with geometry-dependent tasks such as completion, reconstruction, or upsampling, and its overall training parameters and FLOPs remain large when the image model is large.

    Source: §4.4 · p. 10

Abstract

from arXiv · show

Nowadays, pre-training big models on large-scale datasets has become a crucial topic in deep learning. The pre-trained models with high representation ability and transferability achieve a great success and dominate many downstream tasks in natural language processing and 2D vision. However, it is non-trivial to promote such a pretraining-tuning paradigm to the 3D vision, given the limited training data that are relatively inconvenient to collect. In this paper, we provide a new perspective of leveraging pre-trained 2D knowledge in 3D domain to tackle this problem, tuning pre-trained image models with the novel Point-to-Pixel prompting for point cloud analysis at a minor parameter cost. Following the principle of prompting engineering, we transform point clouds into colorful images with geometry-preserved projection and geometry-aware coloring to adapt to pre-trained image models, whose weights are kept frozen during the end-to-end optimization of point cloud analysis tasks. We conduct extensive experiments to demonstrate that cooperating with our proposed Point-to-Pixel Prompting, better pre-trained image model will lead to consistently better performance in 3D vision. Enjoying prosperous development from image pre-training field, our method attains 89.3% accuracy on the hardest setting of ScanObjectNN, surpassing conventional point cloud models with much fewer trainable parameters. Our framework also exhibits very competitive performance on ModelNet classification and ShapeNet Part Segmentation. Code is available at https://github.com/wangzy22/P2P.

1 Introduction

The data demands of 3D pre-training make it difficult to apply the pretraining-and-tuning paradigm to point clouds. P2P instead adapts frozen pre-trained image models to point cloud tasks and reports competitive classification results with fewer trainable parameters.

1 Introduction

  • 3D data such as LiDAR point clouds are inconvenient and costly to collect, leaving pre-training fundamental 3D models with limited data an open question.ShapeNet contains only 50 thousand synthetic samples; balancing numerous trainable parameters against limited data may cause insufficient optimization or overfitting.Source: §1 · p. 2
  • P2P transfers pre-trained image-model knowledge to point clouds by converting them into colorful images with geometry-preserved projection and geometry-aware coloring.The image-model weights are frozen, while the conversion and task-specific heads are optimized end to end.Source: §1 · p. 2
  • P2P achieves results comparable to the best object-classification methods on synthetic ModelNet40 and real-world ScanObjectNN with much fewer trainable parameters.The paper also reports potential for part segmentation on ShapeNetPart.Source: §1 · p. 2

2 Related Work

Image pre-training offers diverse supervised, weakly supervised, and unsupervised approaches, while 3D pre-training remains constrained by data limitations. Prompt tuning adapts downstream tasks at low cost; this paper applies prompting to transfer image-model knowledge to point cloud analysis.

2.1 Visual Pre-training

  • Image pre-training spans supervised methods, weakly supervised methods that use fewer annotations, and unsupervised methods that introduce no task-related bias and offer higher transferability.In contrast, 3D pre-training uses self-supervised pretext tasks, but data limitations remain an obstacle.Source: §2.1 · p. 3

2.2 Prompt Tuning

  • Prompt tuning adapts downstream tasks with limited annotated data to the pre-training task at minimum cost, exploiting pre-trained knowledge for downstream problems.Continuous prompting methods have recently become mainstream for their flexibility and high performance.Source: §2.2 · p. 3

2.3 Object-level Point Cloud Analysis

  • Prior visual prompting work includes L2P for continual learning and VPT for continuous prompt tuning, but the authors report no previous prompting approach for tuning pre-trained image models for point cloud analysis.Point cloud analysis also uses voxel-based and point-based representations, while projection-based methods use multi-view images; MVTN renders images online and regresses an optimal view, unlike P2P's single-view training pattern without explicit multi-view fusion.Source: §2.2 · p. 3, §2.3 · p. 3

3 Approach

P2P encodes point-cloud geometry, projects point features into an image layout, and colors that projection for a frozen pre-trained image model. Task-specific heads then support point-cloud classification and part segmentation.

3.1 Overview

  • P2P combines a geometry encoder, Point-to-Pixel Prompting, a pre-trained image model, and a task-specific head in an end-to-end framework.The frozen image-model weights guide the learnable coloring module indirectly through downstream-task objectives.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Point Cloud Feature Encoding

  • A lightweight DGCNN encodes local geometric features by locating each point's k-nearest neighbors and encoding relative positions to its neighbors.The resulting per-point features have dimension C.Source: §3.2 · p. 4

3.3 Point-to-Pixel Prompting

  • Point-to-Pixel Prompting rearranges 3D geometric features according to projection correspondences and colors the projected image to transfer image-model color knowledge to colorless point clouds.The resulting images are intended to be more distinguishable to the pre-trained image model.Source: §3.3 · p. 4

3.3.1 Geometry-Preserved Projection

  • The projection maps point coordinates to pixels after a view rotation, then sums features of all points landing in each pixel.Summation retains spatial density information and avoids discarding points through depth-based occlusion; training uses a single view without explicit multi-view feature aggregation.Source: §3.3.1 · p. 5

3.3.2 Geometry-Aware Coloring

  • A lightweight 2D network predicts RGB colors from projected pixel features, enriching geometry-only features with color information for the frozen image model.Convolutions smooth the sparse projected features so the resulting images are more recognizable to the image model.Source: §3.3.2 · p. 5

3.4 Optimization on Downstream Tasks

  • Task-specific heads use the image model's features: classification applies a linear classifier to the class token, while segmentation upsamples image tokens and predicts per-pixel labels.Segmentation projects 3D labels into 2D and uses a multi-label loss where points from multiple classes share a pixel; inference fuses predictions from multiple projection views.Source: §3.4 · pp. 5–6

4 Experiments

The experiments evaluate P2P on point-cloud classification and part segmentation, then examine prompt design, tuning strategies, and image-model pre-training choices.

4.1 Datasets and Experiment Settings

  • Experiments cover synthetic ModelNet40 and ShapeNetPart plus perturbed real-world ScanObjectNN (PB-T50-RS), with 40-view majority voting at inference.Training uses random rotations that keep objects upright; the image-model weights are frozen except for normalization layers.Source: §4.1 · p. 6

4.2.1 Results

  • P2P outperforms traditional 3D pre-training and reaches state-of-the-art ScanObjectNN accuracy; larger versions of the same image model improve classification.The authors also report relatively stronger performance on real-world ScanObjectNN than on synthetic ModelNet.Source: §4.2.1 · pp. 6–7, Table 2 · p. 8

4.2.2 Ablation Studies

  • P2P is the best low-cost transfer choice over fine-tuning and VPT; colorful images outperform bypassed-image and binary-image variants.The bypassed-image variant adds trainable parameters, while the binary-image variant uses fewer parameters but performs far worse.Source: §4.2.2 · p. 8
  • Normalization-only tuning is most suitable; supervised pre-training fares relatively better, while stronger ImageNet linear-probe unsupervised models also transfer better to 3D.The authors attribute the tuning result to retaining 2D information while adapting to point-cloud data, and report that 3D data are insufficient to train a large ViT from scratch.Source: §4.2.2 · pp. 8–10

4.3 Part Segmentation

  • On ShapeNetPart, P2P beats classical point-based methods for part segmentation, using ConvNeXt with SemanticFPN or larger ConvNeXt with UPerNet.The authors leave more advanced segmentation heads and supervision strategies for future work.Source: §4.3 · p. 10, Table 4 · p. 10

4.4 Limitations

  • P2P may struggle with geometry-dependent tasks such as completion, reconstruction, and upsampling; large image models also retain high total training parameters and FLOPs.These are identified as limitations for future work despite the framework's relatively few trainable parameters.Source: §4.4 · p. 10

5 Conclusion

P2P tunes pre-trained image models for point-cloud analysis at low trainable-parameter cost, adapting image-domain knowledge to 3D tasks. Experiments on object classification and part segmentation demonstrate the framework’s potential.

5 Conclusion

  • P2P tunes pre-trained image models for point-cloud analysis at low trainable-parameter cost.It preserves point-cloud geometry in projected images while transferring color information from the pre-trained image model to colorless point clouds.Source: §5 · p. 10
  • P2P mitigates the point-cloud field’s data-starvation problem, which has been an obstacle for massive 3D pre-training research.Source: §5 · p. 10
  • Experiments on object classification and part segmentation demonstrate the framework’s superiority and potential.The conclusion reports competitive performance compared with state-of-the-art point-based methods.Source: §5 · p. 10

A.1 Experiments on Different Pre-trained Image Models

Experiments evaluate Point-to-Pixel Prompting with differently scaled pre-trained image models on ModelNet40 and ScanObjectNN. Larger models of the same type yield higher point-cloud classification performance.

A.1 Experiments on Different Pre-trained Image Models

  • Experiments use pre-trained ConvNeXt, Vision Transformer, and Swin Transformer models on ModelNet40 and ScanObjectNN.The image models are pre-trained on ImageNet-22k; the experiments report trainable-parameter counts and classification accuracy.Source: §A.1 · p. 11
  • Larger scales of the same image model yield higher classification performance on point-cloud tasks.The authors report this trend as consistent with observations in image classification.Source: §A.1 · p. 11
  • Table 5 reports ImageNet classification accuracy, trainable parameters, and ModelNet40 and ScanObjectNN accuracy for migrated image models.Source: Table 5 · p. 11

A.2 Ablation Studies on Test View Choices

The test-view ablation selects 40 views for majority voting on ModelNet40, using evenly divided ranges of two rotation angles.

A.2 Ablation Studies on Test View Choices

  • The ablation chooses 10 θ values and 4 φ values, producing 40 views for majority voting during inference, based on ModelNet40 results with an ImageNet-1k-pretrained ViT image model.Source: §A.2 · p. 11
  • At inference, θ and φ are each evenly divided into segments and combined to form multiple views for majority voting.Source: §A.2 · p. 11
  • During training, θ is randomly selected from [−π, π] and φ from [−0.4π, −0.2π] to keep objects upright in images.Source: §A.2 · p. 11

A.3 Ablation Studies on Projection Pooling Strategy

Projection pooling choices have different effects across classification and segmentation: summation performs better for classification, while the three strategies perform equally for segmentation. Feature visualizations show discriminative features from projected colorful images.

A.3 Ablation Studies on Projection Pooling Strategy

  • The pooling ablation evaluates ModelNet40 classification accuracy and ShapeNetPart instance average IoU.It uses ViT-B pre-trained on ImageNet-1k for classification and ConvNeXt-B with SemanticFPN for segmentation.Source: §A.3 · pp. 11–12, Table 7 · p. 12
  • Summation outperforms max-pooling and mean-pooling in classification; max-pooling drops geometric information and mean-pooling neglects point density.Source: §A.3 · p. 12
  • Max-pooling, mean-pooling, and summation produce the same part-segmentation performance.The authors suggest multi-hot 2D labels may provide extra geometrical guidance that makes up for pooling-strategy differences.Source: §A.3 · p. 12

A.4 Visualization of Feature Distributions

  • Feature distributions show that P2P enables the pre-trained image model to extract discriminative features from projected colorful images for point-cloud analysis.Source: §A.4 · p. 12

B Network Architecture

The architecture converts point clouds into images through geometry-preserved projection and feature-based coloring, while a geometry encoder extracts point features. The appendix also provides architecture and experiment settings and visualizes feature distributions with t-SNE.

B.1 Point-to-Pixel Prompting

  • The geometry encoder embeds point coordinates into 8-dimensional features, aggregates information from 32 nearest neighbors, and outputs 64-dimensional point features.It uses a one-layer DGCNN edge convolution, followed by a 1×1 convolution and max pooling over each point’s neighbors.Source: §B.1 · p. 12
  • The geometry-preserved projection module fits the point cloud to a 224×224 image using grid sizes derived from the cloud’s coordinate range.The grid dimensions are gh = H/xr and gw = W/xr.Source: §B.1 · p. 12
  • The coloring module smooths pixel-level features with ResNet-style blocks and predicts the image’s RGB channels.It uses 3×3 convolutions followed by a final 1×1 convolution.Source: §B.1 · p. 12
  • Table 8 presents the framework’s architecture and experimental settings, with Cemb denoting the pre-trained image feature embedding dimension.The appendix also includes a classification-model architecture illustration and a t-SNE visualization of feature distributions.Source: Table 8 · p. 13, §B.1 · p. 13, Figure 4 · p. 13

C Implementation Details

Implementation details vary between classification and part segmentation; ModelNet40 uses 4096 points to produce smoother projected images.

C Implementation Details

  • ModelNet40 uses 4096 points because fewer points can create sparse, discontinuous projected pixels that are less similar to real 2D images.The classification and part-segmentation architectures differ, and Table 8 provides their implementation and experimental settings.Source: C Implementation Details · p. 13

Related papers

Loading 2208.02812v2…