PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding

Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, Hao Su

arXiv:2305.10764v2cs.CV

TL;DR

OpenShape addresses limited 3D training scale and poor generalization to unseen categories by learning scalable joint representations across text, images, and point clouds. It ensembles 876k shapes, enriches noisy text, scales 3D backbones, and applies hard-negative mining; it reaches 46.8% on 1,156-category Objaverse-LVIS and enables CLIP-based cross-modal applications.

  • Problem

    Existing data-driven 3D methods are limited by training-dataset scale and generalize poorly to unseen shape categories.

  • Method

    OpenShape learns multimodal joint representations by aligning 3D shapes with language and images, using ensembled datasets, text filtering and enrichment, scaled backbones, and hard-negative mining.

  • Results

    OpenShape outperforms previous zero-shot methods across ModelNet40 and Objaverse-LVIS, achieving 46.8% accuracy on the 1,156-category Objaverse-LVIS benchmark and supporting CLIP-based cross-modal applications.

  • Takeaways & Limitations

    The learned representations capture broad semantic and visual concepts and can be integrated with off-the-shelf CLIP-based models for point-cloud captioning and point-cloud-conditioned image generation.

  • Takeaways & Limitations

    Rendering-based alternatives suffer from occlusion, information loss during projection, and additional latency from rendering and multiple CLIP inferences.

Abstract

from arXiv · show

We introduce OpenShape, a method for learning multi-modal joint representations of text, image, and point clouds. We adopt the commonly used multi-modal contrastive learning framework for representation alignment, but with a specific focus on scaling up 3D representations to enable open-world 3D shape understanding. To achieve this, we scale up training data by ensembling multiple 3D datasets and propose several strategies to automatically filter and enrich noisy text descriptions. We also explore and compare strategies for scaling 3D backbone networks and introduce a novel hard negative mining module for more efficient training. We evaluate OpenShape on zero-shot 3D classification benchmarks and demonstrate its superior capabilities for open-world recognition. Specifically, OpenShape achieves a zero-shot accuracy of 46.8% on the 1,156-category Objaverse-LVIS benchmark, compared to less than 10% for existing methods. OpenShape also achieves an accuracy of 85.3% on ModelNet40, outperforming previous zero-shot baseline methods by 20% and performing on par with some fully-supervised methods. Furthermore, we show that our learned embeddings encode a wide range of visual and semantic concepts (e.g., subcategories, color, shape, style) and facilitate fine-grained text-3D and image-3D interactions. Due to their alignment with CLIP embeddings, our learned shape representations can also be integrated with off-the-shelf CLIP-based models for various applications, such as point cloud captioning and point cloud-conditioned image generation.

1 Introduction

OpenShape addresses limited 3D training scale and poor generalization to unseen categories by scaling multimodal representation learning and improving noisy training descriptions. It achieves strong zero-shot recognition and supports semantic and visual shape interactions through CLIP-aligned embeddings.

  • Existing 3D approaches are constrained by limited training datasets and generalize poorly to unseen shape categories.
  • Rendering-based use of CLIP enables zero-shot 3D classification but suffers from occlusion, information loss, and extra inference latency.
  • OpenShape combines four public 3D datasets into 876k shapes and automatically filters and enriches noisy text descriptions.
  • 46.8% accuracy on the 1,156-category Objaverse-LVIS benchmark significantly surpasses previous methods, while ModelNet40 performance exceeds previous zero-shot approaches by at least 20%.
  • OpenShape embeddings retrieve shapes combining semantic and geometric elements from two inputs and integrate with CLIP-based models for captioning and image generation.

2 Related Work

Related work uses pretrained image-language models and multimodal contrastive learning to improve 3D tasks, but existing approaches differ in whether they provide semantic alignment during point-cloud pretraining.

  • CLIP-based methods support zero-shot text-to-3D generation and contribute priors to scene-level 3D vision tasks.
  • Self-supervised point-cloud methods improve shape representations through reconstruction, masked auto-encoding, estimation, and contrastive pretext tasks.
  • Multimodal 3D methods align 3D-native shape encoders with CLIP language or image embeddings using contrastive learning, with some adding masked auto-encoding.

3 Method

OpenShape learns a scalable joint representation for language, images, and 3D shapes by combining diverse training data, text enrichment, multimodal alignment, and scalable 3D backbones. Its pipeline also includes contrastive hard-negative handling and supports integration with CLIP-based models.

  • Multi-Modal Representation Alignment: OpenShape aligns language, images, and point clouds through multimodal contrastive learning with a trained 3D-native encoder and frozen CLIP encoders.The method uses sampled point-cloud, text, and image triplets, while freezing CLIP encoders to preserve their feature priors and avoid model collapse.
  • Multi-Modal Representation Alignment: OpenShape integrates its aligned embeddings with CLIP-based models for cross-modality tasks, including point-cloud captioning and point-cloud-conditioned image generation.The alignment enables interactions across 3D shapes, text, and images without requiring each downstream model to learn a separate embedding space.
  • Ensembling 3D Datasets: 876k training shapes come from four public 3D datasets, expanding category diversity while retaining higher-quality datasets alongside noisier Objaverse data.ShapeNetCore, 3D-FUTURE, and ABO provide human-verified shapes; Objaverse contributes broader categories but requires further processing because of uneven quality and unbalanced distributions.
  • Ensembling 3D Datasets: Training samples 10,000 surface points per shape, interpolates texture colors, renders 12 color images, and incorporates thumbnails when available.These point-cloud and image candidates are used to construct multimodal training triplets from the ensembled datasets.
  • Text Filtering and Enrichment: OpenShape filters noisy user text with GPT-4, adds captions from BLIP and Azure, and retrieves additional descriptions from LAION-5B images.GPT-4 removes inaccurate or uninformative texts, while captioning and retrieval broaden descriptive and fine-grained semantic coverage.
  • Scaling Up 3D Point Cloud Backbones: Model scaling improves all evaluated 3D backbones, while PointBERT and SparseConv offer stronger scalability than dense-point DGCNN and PointNet.DGCNN and PointNet become more time-consuming and memory-intensive when scaled because they operate on dense points or model dense-point relationships.

4 Experiments

OpenShape is evaluated through zero-shot classification, few-shot linear probing, ablations, retrieval, and cross-modal generation. Results show strong open-world recognition, benefits from data and model scaling and text enrichment, and support multimodal applications through CLIP alignment.

  • Zero-Shot Shape Classification: 85.3% zero-shot accuracy on ModelNet40 surpasses previous methods by at least 20% and reaches 96.5% top-3 and 98.0% top-5 accuracy.The reported zero-shot performance matches that of a fully supervised 3D learning method on ModelNet40.
  • Zero-Shot Shape Classification: OpenShape outperforms competing methods on zero-shot benchmarks including Objaverse-LVIS, ModelNet40, and ScanObjectNN.The evaluation compares against PointCLIP, PointCLIPv2, ReCon, CG3D, CLIP2Point, and ULIP.
  • Few-Shot Linear Probing: On Objaverse-LVIS, zero-shot OpenShape exceeds other methods’ few-shot linear probes, while margins are smaller on ModelNet40 and ScanObjectNN.Linear probing freezes representation vectors and trains a classifier using few-shot labels; performance is averaged over 10 random seeds.
  • Ablation Study: Using 1% of the ensembled dataset, about 8.8k shapes, achieves similar or better zero-shot performance than training without Objaverse shapes on about 77.1k shapes.The ablation indicates that training-data diversity can matter more than dataset scale, while model scaling is important on the large ensembled dataset.
  • Ablation Study: Text filtering and enrichment improve performance, with image captioning and text retrieval together producing the best enrichment results.PointBERT improves by more than 10 points from text enrichment; hard-negative mining further improves performance under unbalanced data.

5 Discussion and Conclusion

OpenShape learns scalable, generalizable multimodal representations for 3D shapes, supporting open-world recognition and cross-modality applications while leaving data scale, part-level information, and sim-to-real transfer as future directions.

  • OpenShape learns scalable and generalizable joint representations spanning language, images, and 3D shapes.
  • Its representations capture semantic and visual concepts for open-world 3D shape recognition.
  • Alignment with CLIP embeddings supports integration with off-the-shelf CLIP-based models for cross-modality applications.
  • Future work includes more 3D data, part-level supervision, and reducing the sim-to-real domain gap.The current model uses 876k 3D shapes and is mainly trained on synthetic data.

6 Appendix

The appendix provides additional examples of image-input and point cloud-input 3D shape retrieval using OpenShape embeddings and Objaverse shapes.

  • The appendix showcases additional multimodal 3D shape retrieval examples.
  • Image-input retrieval presents an input image alongside two 3D shapes retrieved with OpenShape embeddings from Objaverse.
  • Point cloud-input retrieval presents an input point cloud alongside two 3D shapes retrieved with OpenShape embeddings from Objaverse.

6.2 More Examples of Shape-Conditioned Multimodal Generation

The appendix showcases point cloud captioning and point cloud-conditioned image generation through additional examples pairing point clouds with generated outputs.

  • The appendix provides more examples of both point cloud captioning and point cloud-conditioned image generation.
  • Point cloud captioning: Point cloud captioning examples place input point clouds on the left and generated captions on the right.
  • Point cloud-conditioned image generation: Point cloud-conditioned image generation examples show input point clouds alongside generated images.

6.3 Details on Raw Text Generation and Filtering

OpenShape constructs raw shape descriptions from selected dataset metadata, filters uninformative texts with GPT-4, and removes approximately 30% of raw texts.

  • Raw text generation: Metadata from four datasets is selectively combined into informative raw descriptions for each shape.The selected attributes are intended to preserve text quality and relevance.
  • Raw text generation: Objaverse descriptions use each shape’s associated name.
  • Raw text generation: ShapeNetCore descriptions include names, category names, and concatenated subcategory names.The dataset provides 55 categories and 336 subcategories.
  • Raw text generation: 3DFuture descriptions include category or concatenated category, style, theme, and material attributes.
  • Raw text generation: ABO descriptions use item_name or product_type fields.
  • Raw text generation: The metadata process produces one or more raw texts for each shape.
  • Text filtering: GPT-4 filters raw texts in independent batches of 256 entries to identify uninformative descriptions.
  • Text filtering: The filtering prompt marks texts dominated by irrelevant or uninformative content as unsuitable descriptions.

6.4 Details on the Backbone Scaling Experiment

The experiment compares scalable 3D backbones under controlled point-cloud settings and finds that scaling behavior depends on architecture and model size. PointBERT provides a clear overfitting example at larger scale.

  • Experimental setup: Figure 4 evaluates backbone performance and scalability as model sizes increase, using 10,000-point inputs, batch size 200, and one A100 GPU.Models follow their original scaling instructions when available; otherwise, width or depth is expanded.
  • PointBERT: 72.1M-parameter PointBERT dramatically overfits and performs worse on all benchmarks than the 32.3M version.The 72.1M model extends beyond the 32.3M version reported in Figure 4.
  • SparseConv: SparseConv scaling adapts a 5.3M MinkowskiFCNN-derived model and larger MinkowskiResNet-based models with varying numbers of residual blocks.The specific scaling parameters are listed in Table 5.
  • PointNeXt: PointNeXt uses the official S, B, L, and XL configurations from its original paper.
  • Dense backbones: DGCNN and PointNet scale layer widths until GPU memory limits, using reduced resolutions of 1k and 4k points respectively.Their dense-point operation makes the default 10k-point resolution impractical.

6.5 Details on Training and Evaluation

OpenShape trains its 3D encoder and projection heads against frozen CLIP text and image encoders, then evaluates released and retrained baselines under specified benchmark protocols. Fine-tuning the CLIP text encoder did not improve results and may hinder generalization and CLIP-model integration.

  • Training: OpenShape freezes CLIP text and image encoders while training the 3D encoder and two projection heads with cross-modal contrastive loss.Training uses the ensembled dataset, batch size 200, one A100 GPU, cached embeddings, and about 300 A100 hours for convergence.
  • Encoder fine-tuning: Fine-tuning the CLIP text encoder for one epoch showed no noticeable benchmark improvement and could undermine CLIP generalization and integration.OpenShape therefore keeps the text encoder frozen.
  • Evaluation: Baselines use publicly released checkpoints, while ULIP is additionally retrained on the ensembled shapes with its official code and backbones.Retrained ULIP uses original raw dataset texts, with prompt engineering, rather than OpenShape’s filtered and enriched texts.

6.6 Details on Shape-Conditioned Multimodal Generation

OpenShape embeddings are adapted to the embedding spaces expected by existing CLIP-based multimodal systems. The resulting integrations support point-cloud captioning and point-cloud-conditioned image generation.

  • Point Cloud Captioning: For point-cloud captioning, a CLIP ViT-B/32 OpenShape-PointBERT variant directly supplies unnormalized point-cloud encodings to CLIPCap.This matches CLIPCap’s use of CLIP image embeddings for its 10-token prefix.
  • Point Cloud Conditioned Image Generation: For point-cloud-conditioned image generation, OpenShape replaces Stable Diffusion v2.1 unCLIP’s CLIP image-condition encoder with an OpenShape encoder.A CLIP ViT-L/14-compatible OpenShape-PointBERT variant supports conditioning on point clouds and optional text prompts.
  • Point Cloud Conditioned Image Generation: The image-generation setup uses 768 as the observed mean L2-norm of ViT-L/14 image embeddings and runs diffusion for 50 steps.The guidance scale can be tuned freely.
Loading 2305.10764v2…