PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training

Xiaoyang Wu, Zhuotao Tian, Xin Wen, Bohao Peng, Xihui Liu, Kaicheng Yu, Hengshuang Zhao

arXiv:2308.09718v2cs.CV

TL;DR

3D vision has not fully benefited from large-scale training data because existing representation-learning frameworks struggle to process larger point-cloud data efficiently and current 3D datasets are often limited in scale. PPT is a multi-dataset 3D representation-learning framework using Prompt-driven Normalization and Language-guided Categorical Alignment, and it supports supervised and unsupervised pre-training. PPT overcomes negative transfer and achieves state-of-the-art results on indoor and outdoor benchmarks with shared-weight models, while outperforming other pre-training techniques in representation quality.

  • Problem

    3D vision has not fully benefited from large-scale training data because existing representation-learning frameworks struggle to process larger point-cloud data efficiently and current 3D datasets are often limited in scale.

    Source: §1 · p. 1

  • Method

    PPT is a multi-dataset 3D representation-learning framework using Prompt-driven Normalization and Language-guided Categorical Alignment, and it supports supervised and unsupervised pre-training.

    Source: §1 · p. 2

  • Results

    PPT overcomes negative transfer and achieves state-of-the-art results on indoor and outdoor benchmarks with shared-weight models, while outperforming other pre-training techniques in representation quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper’s experiments support multi-dataset synergistic training as an approach to large-scale 3D representation learning using a unified framework.

    Source: §5 · p. 8

    The study’s module designs remain limited in scope and may be sub-optimal; further work could test more advanced unsupervised pre-training and prompting techniques.

    Source: §5 · p. 8

Abstract

from arXiv · show

The rapid advancement of deep learning models often attributes to their ability to leverage massive training data. In contrast, such privilege has not yet fully benefited 3D deep learning, mainly due to the limited availability of large-scale 3D datasets. Merging multiple available data sources and letting them collaboratively train a single model is a potential solution. However, due to the large domain gap between 3D point cloud datasets, such mixed supervision could adversely affect the model's performance and lead to degenerated performance (i.e., negative transfer) compared to single-dataset training. In view of this challenge, we introduce Point Prompt Training (PPT), a novel framework for multi-dataset synergistic learning in the context of 3D representation learning that supports multiple pre-training paradigms. Based on this framework, we propose Prompt-driven Normalization, which adapts the model to different datasets with domain-specific prompts and Language-guided Categorical Alignment that decently unifies the multiple-dataset label spaces by leveraging the relationship between label text. Extensive experiments verify that PPT can overcome the negative transfer associated with synergistic learning and produce generalizable representations. Notably, it achieves state-of-the-art performance on each dataset using a single weight-shared model with supervised multi-dataset training. Moreover, when served as a pre-training framework, it outperforms other pre-training approaches regarding representation quality and attains remarkable state-of-the-art performance across over ten diverse downstream tasks spanning both indoor and outdoor 3D scenarios.

1. Introduction

3D representation learning has not fully benefited from large-scale data, and combining datasets can cause negative transfer because their domains differ. Point Prompt Training (PPT) addresses this challenge with domain-specific normalization and language-guided label alignment, achieving strong results across indoor and outdoor benchmarks and pre-training tasks.

1. Introduction

  • PPT is a universal framework for multi-dataset 3D representation learning that supports supervised and unsupervised pre-training, with direct evaluation or transfer learning.Unlike cited 2D prompt-learning counterparts, it tackles pre-training directly.Source: §1 · p. 2
  • Prompt-driven Normalization adapts to dataset-specific variation, while Language-guided Categorical Alignment unifies label spaces using relationships between category text.These are the framework’s adopted strategies for domain prompts and aligned categorical supervision.Source: §1 · p. 2
  • PPT overcomes negative transfer and produces generalizable representations, achieving state-of-the-art performance on ScanNet and S3DIS with one shared-weight model trained on multiple indoor datasets.It also achieves state-of-the-art results on SemanticKITTI, nuScenes, and Waymo using a shared-weight model trained on diverse outdoor datasets.Source: §1 · p. 2
  • The framework also outperforms other pre-training techniques in representation quality and performs strongly across indoor and outdoor downstream tasks.The introduction describes results on an array of tasks spanning both scenarios.Source: §1 · p. 2
  • The paper frames multi-dataset synergistic training as a route toward large-scale 3D representation learning and addresses the negative transfer found in naive baselines.Its unified framework combines Prompt-driven Normalization and Language-guided Categorical Alignment.Source: §1 · p. 2

2. Multi-dataset Synergistic Training

The section formulates supervised multi-dataset training as learning one model across datasets, with unsupervised objectives also possible. Its pilot study finds that directly merging indoor point-cloud datasets can cause negative transfer.

2.1. Problem Setup

  • Supervised synergistic training seeks one model that minimizes cumulative sample-wise loss across multiple datasets; replacing the supervised loss with an unsupervised objective also permits unsupervised learning.The dataset formulation uses data-label pairs for each dataset.Source: §2.1 · p. 2
  • The setup targets scene-level semantic segmentation on ScanNet, S3DIS, and Structured3D, spanning real-world and synthetic data with differing scale, scene variety, and point-cloud density.The evaluation considers joint supervised training with direct evaluation, as well as supervised or unsupervised pre-training followed by target-dataset fine-tuning.Source: §2.1 · pp. 2–3

2.2. Pilot Study: Uncovering the Negative Transfer

  • The pilot revisits limited gains from earlier unsupervised pre-training on ScanNet and Arikitscene despite using three times more data, then investigates direct supervised joint training.The cited earlier work is MSC [108].Source: §2.2 · p. 3
  • Directly merging indoor datasets for segmentation produces negative transfer between datasets, exposing a challenge for collaborative 3D learning.The pilot evaluates pairwise dataset combinations and a combination of all datasets on each related individual dataset.Source: §2.2 · p. 3

3. Point Prompt Training

Point Prompt Training (PPT) is designed for multi-dataset 3D representation learning, using domain-specific prompts and categorical alignment to address domain differences and inconsistent supervised label spaces. Its approach differs from conventional prompt tuning, which adapts pretrained models to downstream tasks.

3. Point Prompt Training

  • PPT adapts one model to different dataset contexts with learnable domain-specific prompts and trains it alongside a categorical-alignment process for supervised multi-dataset learning.The framework is designed to mitigate negative transfer in multi-dataset training.Source: §3 · p. 3

3.1. Learning with Domain Prompting

  • Conventional prompt tuning adapts a large-scale pretrained model to downstream tasks, while domain prompting is introduced because 3D perception lacks such a pretrained model and must fit multiple datasets during training.Domain prompts are trained cooperatively with the backbone rather than merely fine-tuned on a pretrained model.Source: §3.1 · pp. 3–4
  • Each interested dataset receives a learnable d-dimensional prompt, and these prompts are intended to help the backbone discover dataset distribution differences and learn representations shared across datasets.The collection of prompts is denoted C = {c_i ∈ R^d | i ∈ N, 1 ≤ i ≤ n}.Source: §3.1 · p. 4
  • Prompt-driven Normalization is the selected domain adapter: it conditions normalization scale and shift on the domain prompt, while normalization statistics are computed independently for each dataset.The paper also investigates direct injection and cross-attention adapter alternatives.Source: §3.1 · pp. 4–5, Figure 2 · p. 4
  • To stabilize joint training, the method zero-initializes PDNorm’s γ(c) and β(c) parameters and starts prompt-related parameters at a smaller learning rate than the backbone.The stated aim is to prioritize the backbone early and avoid unstable training from randomly initialized prompts.Source: §3.1 · p. 5

3.2. Categorical Alignment

  • Supervised multi-dataset training must address inconsistent label spaces; separate dataset heads add redundant parameters, while a shared head with dataset-specific losses is another considered option.The shared-head option projects into a unified label space but keeps loss computation constrained to each dataset’s label space.Source: §3.2 · p. 5
  • Language-guided Categorical Alignment aligns point representations with category-language embeddings from a pretrained text encoder, using InfoNCE negatives restricted to each dataset’s category space.It uses label meaning to avoid treating categories as independent and uncorrelated.Source: §3.2 · p. 5

4. Experiments

The experiments ablate PPT’s prompt, alignment, and data choices, then compare it with prior methods across indoor and outdoor segmentation, instance segmentation, and data-efficient evaluation.

4.1. Ablation Study

  • Prompt-driven normalization outperforms direct injection and cross attention; zero initialization and a 0.1 domain-prompt learning rate also benefit training.All prompt-adapter designs outperform the no-adapter baseline in the ScanNet 20-category ablation.Source: §4.1 · pp. 5–6
  • Prompt adapters help more in deeper backbone stages, while a feature dimension of 256 is sufficient and different dimensions yield similar results.The authors associate deeper features with high-level semantics where domain gaps have negative effects.Source: §4.1 · p. 6
  • A unionized segmentation head aligns dataset supervision better than separate heads, language guidance adds gains, and InfoNCE avoids the mode collapse observed with simple L2 loss.Augmenting class names into sentences did not show effectiveness.Source: §4.1 · p. 6
  • PPT is relatively robust to dataset sampling ratios and delivers consistent benefits across joint-training data sources despite domain gaps.The tested sources differ in data source, sparsity, complexity, and scale.Source: §4.1 · p. 6

4.2. Results Comparision

  • Indoor results surpass prior state-of-the-art methods across benchmarks, including with a single shared-weight model.The comparisons cover unsupervised pre-training integrated with MSC, supervised joint training, and supervised pre-training.Source: Table 3 · p. 7, §4.2 · p. 7
  • On indoor semantic segmentation, unsupervised pre-training integrated with MSC improves ScanNet200 Val by 1.6 points and S3DIS Area5 mIoU by 1.8 points.The results also surpass previous state-of-the-art methods.Source: §4.2 · p. 7, Table 3 · p. 7
  • Outdoor joint training improves all tasks over scratch, including 7.1 points on SemanticKITTI Validation; fine-tuning beats SphereFormer by 5.0 points mIoU there.PPT also reports multiple superior results across the SemanticKITTI, nuScenes, and Waymo benchmarks.Source: §4.2 · pp. 7–8, Table 4 · p. 7
  • On ScanNet and ScanNet200 instance segmentation, PPT exceeds MSC by 2.4 and 2.6 points in mAP@50, respectively, and outperforms prior pre-training methods in data-efficient settings.The instance results use fine-tuning with PointGroup; the data-efficient comparison follows the ScanNet benchmark.Source: §4.2 · p. 8, Table 5 · p. 8, Table 6 · p. 8

5. Conclusion and Discussion

PPT addresses negative transfer in 3D multi-dataset training with a unified framework and reports consistent performance gains. The appendix details alternative domain-prompt and categorical-alignment designs, while the discussion identifies scope for further work.

5. Conclusion and Discussion

  • PPT combines Prompt-driven Normalization and Language-guided Categorical Alignment to address negative transfer in multi-dataset 3D representation learning.The conclusion reports consistent and significant performance gains.Source: §5 · p. 8
  • The current study is limited in scope; future work could explore more advanced unsupervised pre-training, prompting techniques, indoor–outdoor training, and multi-task scene understanding.The present formulation considers only one pre-training task.Source: §5 · p. 8

Appendix

  • The appendix provides an overview and visual guide to detailed implementation material for PPT.Source: Appendix · p. 9

A. Related Work

  • Prior 3D representation learning often uses single datasets, while naive merging can be suboptimal; this work targets generalized representations under supervised and unsupervised training.The paper uses voxel-based SparseUNet for efficient large-scale training.Source: A. Related Work · p. 9

B.1. Domain Prompt Adapter

  • Alternative domain adapters include direct injection at each block and cross-attention between point queries and domain-prompt embeddings.Both designs use zero-initialized projections; cross-attention adds its projected output to the point embedding.Source: §B.1 · p. 10
  • Prompt-driven Normalization replaces normalization layers and applies prompt-derived scale-shift vectors to normalized feature embeddings.The prompt is projected to the scale-shift vector through a zero-initialized linear layer.Source: §B.1 · p. 10

B.2. Categorical Alignment

  • Categorical alignment alternatives include dataset-specific prediction heads, a unified head with domain-restricted loss spaces, and language-guided point–text alignment.The language-guided option uses CLIP text embeddings and InfoNCE alignment, then computes category cross-entropy.Source: §B.2 · pp. 10–11

C.1. Experimental Settings

The appendix describes indoor datasets used for joint training, the sampling procedure, and the SparseUNet backbone configuration.

Data

  • Joint training uses ScanNet v2, S3DIS, and Structured3D, with dataset-specific scene counts, category sets, and sampling or annotation details.Structured3D is synthetic; its 40 categories are filtered to 25 for perception.Source: §C.1 · p. 11

Training

  • Each joint-training iteration samples a batched point cloud from a single dataset, with sampling ratios set by each dataset’s best-performance iteration requirement.Total iterations equal the sum of those per-dataset requirements.Source: §C.1 · p. 11

Backbone

  • PPT is validated with SparseUNet, selected for speed and memory efficiency; the primary results use the 39-million-parameter SpUNet-S.A 412-million-parameter SpUNet-L is also explored.Source: §C.1 · p. 12

C.2. Additional Pilot Study

An additional pilot study tests whether changing dataset sampling ratios resolves negative transfer in naive joint training.

Additional Pilot Study

  • The pilot study’s naive pairwise and all-dataset merges used a 4:2:1 sampling ratio for Structured3D, ScanNet, and S3DIS, respectively.Source: §C.2 · p. 12
  • Balanced 1:1:1 sampling slightly alleviated performance degradation, but significant negative transfer remained in the vanilla setting.The compared representative ratios were 4:2:1 and 1:1:1.Source: §C.2 · p. 12

C.3. Additional Results

The appendix reports S3DIS six-fold cross-validation results and supplementary error-bar results from repeated runs.

S3DIS 6-fold semantic segmentation

  • 12.7% improvement in S3DIS six-fold semantic segmentation established a new state-of-the-art result for Point Prompt Training.Each fold withholds one area and trains with the remaining S3DIS data plus ScanNet and Structured3D.Source: §C.3 · p. 12

Error bar-supplemented results

  • Supplementary full semantic-segmentation results include error bars from five independent runs, but ScanNet test mIoU mean and standard deviation are unavailable.Multiple ScanNet test submissions are not allowed.Source: §C.3 · p. 12

C.4. Additional Ablation Study

The ablations examine backbone scaling, prompt sharing, and prediction-head choice. The comparison appendix provides context for segmentation results across prior work.

C.4. Additional Ablation Study

  • With PPT, scaling to a larger backbone mitigates the overfitting seen with ScanNet-only training and improves performance.Without PPT, increasing model capacity significantly overfits; with PPT, a larger-scale data source mitigates this issue.Source: §C.4 · p. 12
  • With PPT, SpUNet-L has a more favorable loss curve than SpUNet-S, while without PPT the opposite trend appears.Curves show average training and validation losses calculated at the end of each of 100 epochs.Source: §C.4 · p. 12
  • Expanding convolution-based model depth and dimension significantly increases parameters; transformer-based methods are better suited to capacity expansion but face speed and memory limits.The authors identify improving transformer-backbone efficiency after scaling as future work.Source: §C.4 · pp. 12–13
  • Block-wise independent domain prompts decline in performance, which the authors attribute to added complexity and overfitting.This aligns with the reported degradation from scaling up prompt dimensions.Source: §C.4 · p. 13
  • LCA also enhances performance as a prediction head in standard single-dataset segmentation tasks.It uses scaled similarity between point embeddings and category-text embeddings as predicted logits.Source: §C.4 · pp. 13–14

D. Additional Comparision

  • The additional comparison table tracks prior-work publication years and marks results from scratch with ◦ and pretrained models with •.It expands the main paper’s combined semantic-segmentation results tables.Source: D. Additional Comparision · p. 14

D.1. Indoor Semantic Segmentation

The indoor semantic-segmentation comparison covers ScanNet v2 and S3DIS, with dataset-specific sampling and evaluation setups.

D.1. Indoor Semantic Segmentation

  • ScanNet v2 comprises 1,513 room scans reconstructed from RGB-D frames, divided into 1,201 training scenes and 312 validation scenes; input point clouds are sampled from reconstructed-mesh vertices.Source: §D.1 · p. 14
  • S3DIS includes 271 rooms across six areas from three buildings, with area 5 held out for testing and six-fold cross-validation.Its mesh-surface points are annotated into 13 categories.Source: §D.1 · p. 14
  • Mean class-wise intersection over union (mIoU) is the primary evaluation metric for indoor semantic segmentation.The comparison covers pre-training technologies and backbones on ScanNet v2 and S3DIS.Source: §D.1 · p. 14

D.2. Outdoor Semantic Segmentation

The outdoor semantic-segmentation comparison covers SemanticKITTI and nuScenes, using mIoU as its primary evaluation metric.

D.2. Outdoor Semantic Segmentation

  • SemanticKITTI has 22 sequences, with 19 for training and three for testing, and 28 semantic classes for richly annotated LiDAR scans.The dataset is derived from the KITTI Vision Benchmark Suite and includes varied driving scenarios.Source: §D.2 · p. 14
  • nuScenes comprises 1,000 diverse urban driving scenes from Boston and Singapore.The paper extends its pre-training technology and backbone comparison to this dataset.Source: §D.2 · p. 14
  • Mean class-wise intersection over union (mIoU) is the primary evaluation metric for outdoor semantic segmentation.The comparison covers SemanticKITTI and nuScenes.Source: §D.2 · pp. 14–15

Related papers

Loading 2308.09718v2…