PaperlayerView exact revision on arXiv

Source-linked AI summary

Uni3D: Exploring Unified 3D Representation at Scale

Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, Xinlong Wang

arXiv:2310.06773v1cs.CVcs.CL

TL;DR

Scalable 3D representation learning remains relatively unexplored despite progress in scaling language and 2D vision models. Uni3D adapts a 2D-initialized ViT and image-text alignment to 3D, scales to one billion parameters, and transfers across diverse tasks while enabling cross-modal retrieval and painting.

  • Problem

    Scalable 3D representation learning with broad transfer across objects, scenes, and task scenarios remains relatively unexplored.

  • Method

    Uni3D end-to-end trains a 2D-initialized ViT-based 3D encoder to align point-cloud features with frozen image-text aligned model features.

  • Results

    Uni3D scales to one billion parameters and transfers well across zero-shot, few-shot, open-world understanding, part segmentation, and other 3D tasks.

  • Takeaways & Limitations

    Uni3D provides a unified direction for applying 2D priors, multimodal alignment, and scaling strategies to large-scale 3D representation learning.

Abstract

from arXiv · show

Scaling up representations for images or text has been extensively investigated in the past few years and has led to revolutions in learning vision and language. However, scalable representation for 3D objects and scenes is relatively unexplored. In this work, we present Uni3D, a 3D foundation model to explore the unified 3D representation at scale. Uni3D uses a 2D initialized ViT end-to-end pretrained to align the 3D point cloud features with the image-text aligned features. Via the simple architecture and pretext task, Uni3D can leverage abundant 2D pretrained models as initialization and image-text aligned models as the target, unlocking the great potential of 2D models and scaling-up strategies to the 3D world. We efficiently scale up Uni3D to one billion parameters, and set new records on a broad range of 3D tasks, such as zero-shot classification, few-shot classification, open-world understanding and part segmentation. We show that the strong Uni3D representation also enables applications such as 3D painting and retrieval in the wild. We believe that Uni3D provides a new direction for exploring both scaling up and efficiency of the representation in 3D domain.

1 INTRODUCTION

Uni3D addresses the relatively unexplored challenge of scalable 3D representation learning by adapting 2D pretrained models and image-text alignment to 3D. Scaled to one billion parameters, it transfers across diverse 3D tasks and supports cross-modal applications.

  • Motivation: Scalable 3D representation learning remains challenging because prior work mainly uses limited parameters, data, and task scenarios.The paper frames transfer in the wild as relatively unexplored.
  • Approach: Uni3D uses a simple framework that initializes a 2D ViT-based 3D encoder and aligns point-cloud features with image-text aligned features.This design leverages pretrained 2D models as initialization and image-text aligned models as targets.
  • Scaling: Uni3D systematically scales model size from 6M to 1B parameters, 2D initialization, and image-text target models from 150M to 5B parameters.The study observes continuous performance improvements as each component scales.
  • Results: 88.2% zero-shot classification accuracy on ModelNet demonstrates transfer that performs on par with some supervised methods.The paper also reports state-of-the-art results on open-world understanding and part segmentation.
  • Applications: Uni3D transfers across varied downstream tasks and scenarios and enables point-cloud painting plus text/image-based 3D shape retrieval.The authors present these applications as evidence of the learned representation’s breadth.

2 RELATED WORK

Prior 3D representation work explores architectures, self-supervised objectives, and multimodal alignment, but generally remains limited in scale. Uni3D is positioned as a unified framework for scaling 3D pretraining with 2D and language priors.

  • 3D Representation Learning: Earlier point-cloud methods study architectures and 3D-specific pretext tasks, typically using limited datasets such as ShapeNet.These methods do not investigate multimodal representation transfer from 2D vision or language to 3D.
  • Multimodal 3D Learning: Recent multimodal methods align text, image, and point-cloud features through contrastive learning.The cited approaches build on the success of CLIP-style visual concept learning from raw text.
  • Scalable 3D Pretraining: OpenShape and ULIP2 explore scalable 3D pretraining, but remain limited by small backbones or difficulty scaling to larger sizes.Uni3D targets a unified and scalable alternative.
  • Foundation Models: Foundation-model research has established model and data scaling as a route to unified representations in language and 2D vision.The paper transfers this scaling motivation to 3D representation learning.

3 METHOD

Uni3D combines a ViT-equivalent 3D backbone, scalable model sizes, pretrained cross-modal initialization, and multimodal contrastive alignment. It trains on large collections of 3D shapes, rendered images, and texts to produce transferable 3D representations.

  • Unified 3D Representation: Uni3D uses a vanilla Transformer structurally equivalent to ViT, replacing ViT’s patch embedding with a point tokenizer.The tokenizer groups points into local patches using farthest point sampling and k-nearest neighbors.
  • Scaling Up Uni3D: The framework scales Transformer backbones from Tiny (6 M) through giant (1B) and observes continuous performance improvements with model size.The scaling strategy follows established ViT model-size progression.
  • Scaling Up Uni3D: Uni3D trains a one-billion-parameter model on nearly one million 3D shapes paired with 10 million images and 70 million texts.Training uses a multimodal alignment objective.
  • Initializing Uni3D: Larger 3D backbones can overfit and converge poorly, while limited 3D pretraining data makes task-specific initialization expensive and weak.These constraints motivate importing pretrained priors from other modalities.
  • Initializing Uni3D: Uni3D flexibly initializes its 3D backbone with Transformer-based 2D or cross-modal models trained on billions of images and texts.Candidate priors include DINO, EVA, CLIP, and EVA-CLIP.
  • Training Data: Uni3D uses an ensembled dataset of Objaverse, ShapeNet, 3D-FUTURE, and ABO, sampling colored points and rendering images from multiple views.The setup follows OpenShape’s triplet construction for consistent comparison.
  • Multi-Modal Alignment: The training objective aligns point-cloud features with frozen CLIP image and text features for corresponding 3D shape triplets.Only the point encoder is learnable, while image and text encoders remain frozen.
  • Teacher Scaling: Different CLIP teachers can be substituted and scaled, with the largest EVA-CLIP-E achieving the best reported performance.The ablation identifies teacher capability and scale as important factors for Uni3D performance.

4 EXPERIMENT

Experiments evaluate Uni3D across zero-shot and few-shot recognition, real-world understanding, segmentation, retrieval, painting, and scaling ablations. Across these settings, the results support broad transfer, cross-modal use, and benefits from larger models and pretrained components.

  • 4.1 ZERO-SHOT SHAPE CLASSIFICATION: Uni3D significantly outperforms prior methods in zero-shot shape classification across multiple benchmarks and training settings.Evaluations cover ModelNet, ScanObjNN, and Objaverse-LVIS, including ensembled settings with and without LVIS shapes.
  • 4.2 FEW-SHOT LINEAR PROBING: Uni3D significantly outperforms competing representations in few-shot linear probing across all Objaverse-LVIS label budgets.The evaluation freezes Uni3D and trains only a linear classifier with 1, 2, 4, 8, or 16 labeled samples per class.
  • 4.3 OPEN-WORLD UNDERSTANDING: Uni3D achieves the best zero-shot recognition performance on ScanNet despite training only on synthetic data, unlike several baselines trained with real-world triplets.ScanNet contains reconstructed meshes of real-world scenes, and the task recognizes each instance category without using category labels during training.
  • 4.4 OPEN-VOCABULARY / FEW-SHOT PART SEGMENTATION: With 1 or 2 samples per class, Uni3D outperforms PointBERT by +13.3%/+9.8% in ShapeNetPart few-shot part segmentation.Uni3D remains comparable in overall mIoU even when comparative methods use 10% or 20% of the training set.
  • 4.5 APPLICATIONS: Uni3D supports prompt-driven point-cloud painting and image- or text-query 3D retrieval through its learned multimodal representations.Painting optimizes point-cloud RGB values against CLIP text features, while retrieval uses cosine similarity and kNN over a large 3D dataset.
  • 4.7 ABLATION STUDY: Performance improves continuously as Uni3D scales model size, 2D initialization, and CLIP teacher size; EVA provides the best initialization and EVA-CLIP-E the best teacher.The ablations also show that freezing the pretrained ViT can fail to provide strong 3D understanding without fine-tuning.

5 CONCLUSION

Uni3D scales a unified 3D representation model to one billion parameters by aligning point-cloud features with image-text-aligned features. It achieves state-of-the-art performance across multiple 3D understanding tasks and is intended to support applications in the 3D community.

  • Uni3D scales a unified 3D representation model to one billion parameters.It uses a vanilla transformer structurally equivalent to ViT, enabling established 2D and NLP scaling strategies.
  • Uni3D is trained on about one million 3D point clouds, 10 million images, and 70 million texts.Training aligns 3D point-cloud features with image-text-aligned features.
  • Uni3D achieves state-of-the-art performance in zero-shot and few-shot classification, open-world understanding, and part segmentation.
  • Uni3D can serve as a 3D foundation model for applications in the 3D community.

A TRAINING DETAILS

Training freezes the CLIP image and text encoders while optimizing the 3D encoder with cross-modal contrastive loss. The procedure combines cosine learning-rate decay, stochastic-depth regularization, and random point-token masking.

  • The CLIP text and image encoders remain frozen while the 3D encoder is trained with cross-modal contrastive loss.
  • The optimizer is Adam with a peak learning rate of 1e-3 followed by cosine learning-rate decay.
  • Stochastic depth is used as regularization to improve training stability.
  • FLIP randomly masks 50% of point tokens during training to reduce time.

B PART SETMENTATION DETAILS

Uni3D converts global point-cloud–text alignment into local point–text alignment for 3D dense prediction. In one-shot part segmentation, its visual results are more accurate than competing results.

  • Uni3D converts global point-cloud–text alignment into local point–text alignment for 3D dense prediction.The approach targets detailed local 3D visual semantics learned through object-level pretraining.
  • In the one-shot part segmentation setting, Uni3D produces more accurate segmentation results in visual comparisons.

C MORE VISUALIZATION OF CROSS-MODAL RETRIEVAL

The paper visualizes additional 3D shapes retrieved from real-world images and demonstrates retrieval from input texts. These visualizations cover both image-to-3D and text-to-3D retrieval.

  • Uni3D retrieves additional 3D shapes from one or multiple real-world images.
  • Uni3D also retrieves 3D shapes from input texts.

D MORE ANALYSIS ON INITIALIZING UNI3D

Uni3D’s initialization analysis shows that strong 2D priors are important for cross-modal contrastive training. Without EVA initialization, optimization performs substantially worse when only image or text features are available.

  • EVA is the best-performing 2D initialization model for Uni3D.
  • EVA initialization improves and stabilizes Uni3D’s cross-modal representation learning.The analysis attributes this to general patterns learned by EVA.
  • 20.7 vs. 40.1 under image-only contrastive learning and 12.4 vs. 26.3 under text-only learning show the cost of removing EVA initialization.

E FEW-SHOT RESULTS

The few-shot evaluation uses Objaverse-LVIS to measure linear-probing performance across increasingly labeled training regimes. Results are visualized in Figure 4 and quantified in Table 9.

  • Few-shot linear probing is conducted on the Objaverse-LVIS dataset.
  • The comparison appears in Figure 4, with detailed quantitative results provided in Table 9.
Loading 2310.06773v1…