PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning

Márcus Lobo, Vitor Matias, Jeová Farias, Moacir Ponti

arXiv:2608.29285v1cs.CV

TL;DR

Prior 3D multimodal methods lacked nested representations, while fixed-dimensional embeddings present a bottleneck for deployment under varied computational budgets. 3D-MRL aligns nested 3D embeddings with frozen CLIP image and text features, and achieves competitive recognition performance while improving the accuracy-efficiency trade-off.

  • Problem

    Recent 3D multimodal methods had not incorporated or evaluated nested representations, while fixed-dimensional embeddings are a bottleneck for deployment under varied computational budgets.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    3D-MRL aligns nested 3D embeddings with frozen CLIP image and text features across multiple dimensionalities, allowing one encoder to produce representations at different prefix lengths.

    Source: §1 · p. 2

  • Results

    3D-MRL achieves competitive zero-shot and few-shot recognition performance and improves the accuracy-efficiency trade-off, with graceful degradation across dimensionality budgets.

    Source: §1 · p. 2, §5 · p. 14

  • Takeaways & Limitations

    The learned nested structure organizes information across embedding dimensions, with retrieval progressing from broad semantic similarity to finer geometric similarity.

    Source: §5 · p. 14

    The approach assumes semantic abstraction emerges progressively along the dimensions of the learned 3D representation.

    Source: §3 · p. 4

Abstract

from arXiv · show

Vision-Language Models align point clouds with image and text embeddings, enabling zero-shot recognition, retrieval, and open-vocabulary understanding of 3D shapes. Existing multimodal 3D pre-training methods produce fixed-dimensional embeddings, requiring separate models for different computational budgets. We propose 3D Matryoshka Representation Learning (3D-MRL), a multimodal 3D pre-training framework based on Matryoshka Representation Learning. 3D-MRL learns nested 3D representations by aligning point clouds with frozen CLIP image and text embeddings while applying contrastive supervision across multiple embedding dimensions. The Matryoshka objective is applied only to the 3D encoder, allowing a single model to produce representations at different dimensionalities without retraining. Experiments on the Objaverse-LVIS, ModelNet40, and ScanNet datasets show that 3D-MRL achieves competitive performance on zero-shot and few-shot 3D recognition tasks. In addition, the learned representations support retrieval across different embedding dimensions within a single model. On Objaverse-LVIS, 3D-MRL improves Top-1 accuracy from 46.8% to 50.9%. Retrieval experiments further show that different embedding dimensions yield varying levels of semantic and geometric specificity.

1 Introduction

3D-MRL applies Matryoshka Representation Learning to multimodal 3D pre-training, aligning nested point-cloud embeddings with frozen CLIP image and text features. A single encoder can be truncated to fit different embedding budgets, addressing the fixed-dimensional embeddings used by earlier methods.

Introduction

  • Earlier 3D multimodal methods such as ULIP [37] and OpenShape [22] produce fixed 512-to-1280-dimensional embeddings that must be stored and compared in full, creating a deployment bottleneck.The paper identifies edge devices, large-scale retrieval pipelines, and strict latency budgets as settings where this one-size-fits-all approach is limiting.Source: §1 · pp. 1–2
  • Applying MRL to multimodal 3D must preserve geometric structure while aligning with frozen image and text representations, and truncation can remove local geometric cues.The formulation addresses these challenges with a logarithmic nesting schedule and parameter-free geometric truncation.Source: §1 · p. 2
  • 3D-MRL aligns nested point-cloud embeddings with frozen CLIP image and text features using contrastive supervision across multiple prefix lengths.At inference, the embedding can be truncated to the available budget, avoiding retraining or separate encoders for different operating points.Source: §1 · p. 2
  • 3D-MRL brings Matryoshka Representation Learning to multimodal 3D pre-training, producing useful embeddings at any prefix length from a single encoder.The paper describes this as the first framework to bring MRL to 3D multimodal pre-training.Source: §1 · p. 2
  • 3D-MRL achieves competitive zero-shot and few-shot 3D recognition performance on Objaverse-LVIS, ModelNet, and ScanNet.Its nested representations also support an accuracy-efficiency trade-off that degrades gracefully across a wide range of dimensionality budgets.Source: §1 · p. 2

2 Related Work

Prior 3D multimodal methods typically learn fixed-dimensional embeddings, while Matryoshka learning provides nested representations that can be used at multiple dimensionalities. 3D-MRL addresses the unexplored application of this principle to 3D multimodal learning by building nested structure into contrastive pre-training.

3D Multimodal Learning

  • ULIP [37] and OpenShape [22] align point-cloud features with pretrained image and text embeddings, while later methods improve supervision, relational structure, grounding, and retrieval.Examples include depth-aligned visual supervision in OpenDlign [23], relational distillation in MRD [32], and tri-modal grounding in TriCLIP-3D [18].Source: §2 · p. 3

Matryoshka Representation Learning

  • Matryoshka Representation Learning [16] trains a vector so each prefix remains independently useful, providing an accuracy-efficiency trade-off without retraining or model switching.Extensions include nesting across transformer layers [21], multi-vector retrieval [36], visual tokens [2], and 3D Gaussian primitives [12].Source: §2 · p. 3

3D Multimodal Learning

  • Recent 3D-CLIP methods [17] [18] [22] [37] had not incorporated or evaluated nested or multi-granularity representations.The related-work passage identifies this as an unexplored area of 3D multimodal representation learning.Source: §2 · p. 3
  • 3D-MRL trains a nested hierarchy of 3D representations, unlike post-hoc MRL, by coupling nesting with cross-modal contrastive pre-training from the start.This makes dimensionality reduction a property of the learned space.Source: §2 · pp. 3–4

3 3D Matryoshka Representation Learning

3D-MRL trains a 3D encoder to align nested point-cloud representations with frozen CLIP image and text embeddings at multiple dimensions. Its objective encourages compact prefixes to retain broad semantics while higher-dimensional prefixes add finer geometric and structural information.

Multi-Scale Contrastive Alignment

  • 3D-MRL applies multi-scale contrastive supervision only to the trainable 3D encoder, using frozen CLIP image and text encoders as fixed multimodal targets.The image and text encoders remain frozen throughout training.Source: §3 · p. 4

Nested Representation Structure

  • Eight log-spaced dimensions, {10,20,40,...,1280}, provide denser low-dimensional supervision and induce accurate representation spaces at intermediate dimensions.Training optimizes the selected scales while the nested structure supports smooth interpolation between them.Source: §3 · pp. 4–5

Multi-Scale Contrastive Alignment

  • At each scale, 3D-to-image and 3D-to-text InfoNCE alignment uses scale-specific linear heads, while prefix truncation shares the representation instead of requiring independent full-space projections.The linear transformations are used during pre-training, and the formulation applies them within each nested subspace.Source: §3 · pp. 5–6

Properties of the Learned Representation

  • The representation preserves lower-dimensional semantics in every larger prefix, while added dimensions allow progressively finer semantic, geometric, and structural distinctions.The design hypothesizes that broad category-level semantics emerge in compact prefixes and additional dimensions refine the representation.Source: §3 · pp. 4, 6

Application in Downstream Tasks

  • Zero-shot classification aggregates similarity across scales; few-shot evaluation trains linear classifiers on frozen nested embeddings at each dimension to measure the accuracy–compactness trade-off without retraining.The few-shot protocol keeps the 3D encoder fixed while sweeping the embedding dimension.Source: §3 · p. 6

4 Experiments

Experiments evaluate 3D-MRL on zero-shot, few-shot, and real-world recognition, matched-budget efficiency, and cross-modal retrieval. Results show strong benchmark performance and coarse-to-fine retrieval across embedding dimensions.

4.1 Experimental Setup

  • Pre-training used 875,665 Ensembled triplets from ShapeNetCore, 3D-FUTURE, ABO, and Objaverse; evaluation covered ModelNet40, ScanObjectNN, and Objaverse-LVIS.Point clouds sample 10,000 mesh-surface points, images use 12 preset views, and the tasks include zero-shot classification, few-shot linear probing, and real-world recognition.Source: §4.1 · p. 7

4.2 Zero-Shot 3D Shape Classification

  • On Objaverse-LVIS, 3D-MRL exceeds ULIP and OpenShape by +24.1% and +4.1% Top-1 accuracy; with PointBERT, it reaches 85.8% on ModelNet40, +1.4% over OpenShape.On ScanObjectNN, 3D-MRL with SparseConv achieves competitive performance with OpenShape under the same setting.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 Matched-Budget Compression and Efficiency

  • At 640D and 1280D, direct prefixes score 42.5% vs. 40.2% and 50.9% vs. 46.6%; 320D cuts storage 4× and CPU retrieval latency 4.1×.Post-hoc PCA yields 27.3/43.3/50.0% Top-1 accuracy for 3D-MRL at 40/80/160D, versus 14.7/29.5/40.8% for OpenShape.Source: §4.3 · pp. 8–9, Table 2 · p. 9, Table 3 · p. 9

4.4 Few-Shot Linear Probing 3D Classification

  • On few-shot ModelNet40, 3D-MRL surpasses OpenShape by 2.2%, 0.6%, and 1.4% in the 5-way 20-shot, 10-way 10-shot, and 10-way 20-shot settings.The linear classifier is trained without fine-tuning the full model; improvements are more evident in the 10-way settings.Source: Table 4 · p. 9, §4.4 · p. 9

4.5 Real-World Recognition

  • On ScanNet, 3D-MRL at 640 dimensions achieves 47.0% mean Top-1 accuracy, exceeding OpenShape by +1.4% and CLIP2 by +8.5%.Performance improves from 10 to 640 dimensions, with 640D best; 320D retains a competitive 38.4% mean average Top-1 accuracy.Source: Table 5 · p. 10, §4.5 · p. 10

4.6 Cross-Modal Granularity Applications

  • Across point-cloud, image, and text queries, increasing dimensions refine broad semantic retrieval into geometric or instance-specific matches; detailed text prompts yield more aligned results than OpenShape.The retrieval demonstrations use Objaverse-LVIS; text prompts include progressively specific descriptions, such as “ergonomic office chair.”Source: Figure 3 · p. 11, Figure 4 · p. 11, §4.6 · pp. 11–12

4.7 Ablation Study

  • On ModelNet40, the log-based schedule achieves 71.8% Top-1 accuracy, beating full by +1.2% and linear by +2.0%; on Objaverse-LVIS it reaches 12.4% Top-1 and 27.2% Top-5.The ablations also compare parameter-free geometric truncation with independent scale-specific projection heads; the schedule results support exponentially spaced granularities as an optimization regularizer.Source: §4.7 · pp. 13–14, Table 7 · p. 14

5 Conclusion

3D-MRL brings Matryoshka Representation Learning to multimodal 3D pre-training, enabling one model to produce nested representations at multiple dimensionality budgets without retraining. Experiments show competitive performance and retrieval that shifts from broad semantic similarity toward finer geometric and semantic specificity.

5 Conclusion

  • Across Objaverse-LVIS, ModelNet40, ScanObjectNN, and ScanNet, 3D-MRL achieves competitive performance while improving the accuracy-efficiency trade-off of multimodal 3D representations.The conclusion presents these results as evidence for using nested representations in multimodal 3D understanding and deployment under diverse computational constraints.Source: §5 · p. 14
  • 3D-MRL learns nested 3D representations aligned with frozen image and text embeddings, letting a single model operate across multiple dimensionality budgets without retraining.The paper presents this as the first framework to bring Matryoshka Representation Learning to multimodal 3D pre-training.Source: §5 · p. 14
  • The nested structure organizes information hierarchically across embedding dimensions, with retrieval progressing from broad semantic similarity to finer geometric and semantic specificity.The conclusion describes this retrieval behavior as a potential foundation for scalable multimodal 3D retrieval under diverse computational constraints.Source: §5 · p. 14

Related papers

Loading 2608.29285v1…