Source-linked AI summary
3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning
Márcus Lobo, Vitor Matias, Jeová Farias, Moacir Ponti
TL;DR
Existing multimodal 3D pre-training methods use fixed-dimensional embeddings, motivating a flexible representation that can meet different computational budgets. 3D-MRL addresses this gap by aligning nested 3D representations with frozen CLIP image and text embeddings across dimensions, achieving competitive recognition performance and supporting graceful accuracy-efficiency trade-offs from a single model.
Problem
Existing multimodal 3D methods produce fixed-dimensional embeddings, while nested representations remain unexplored in 3D multimodal representation learning.
Method
3D-MRL aligns nested 3D embeddings with frozen CLIP image and text features using contrastive supervision across multiple dimensionalities.
Results
3D-MRL achieves competitive zero-shot and few-shot recognition performance across Objaverse-LVIS, ModelNet, and ScanNet, while degrading gracefully across dimensionality budgets.
Takeaways & Limitations
A single 3D-MRL model supports multiple dimensionality budgets and retrieval behavior spanning broad semantic to finer geometric information.
Takeaways & Limitations
The framework is based on the hypothesis that semantic abstraction emerges progressively along representation dimensions.
Abstract
from arXiv · showhide
Vision-Language Models align point clouds with image and text embeddings, enabling zero-shot recognition, retrieval, and open-vocabulary understanding of 3D shapes. Existing multimodal 3D pre-training methods produce fixed-dimensional embeddings, requiring separate models for different computational budgets. We propose 3D Matryoshka Representation Learning (3D-MRL), a multimodal 3D pre-training framework based on Matryoshka Representation Learning. 3D-MRL learns nested 3D representations by aligning point clouds with frozen CLIP image and text embeddings while applying contrastive supervision across multiple embedding dimensions. The Matryoshka objective is applied only to the 3D encoder, allowing a single model to produce representations at different dimensionalities without retraining. Experiments on the Objaverse-LVIS, ModelNet40, and ScanNet datasets show that 3D-MRL achieves competitive performance on zero-shot and few-shot 3D recognition tasks. In addition, the learned representations support retrieval across different embedding dimensions within a single model. On Objaverse-LVIS, 3D-MRL improves Top-1 accuracy from 46.8% to 50.9%. Retrieval experiments further show that different embedding dimensions yield varying levels of semantic and geometric specificity.
1 Introduction
Existing multimodal 3D methods produce fixed-dimensional embeddings, limiting deployment flexibility. 3D-MRL learns nested representations so one encoder supports multiple dimensionality budgets, with competitive recognition performance and graceful accuracy-efficiency trade-offs.
- Fixed 512–1280-dimensional embeddings create a bottleneck for edge deployment, large-scale retrieval, and strict latency budgets.
- Matryoshka Representation Learning enables useful lower-dimensional prefixes and progressively higher capacity without retraining.
- 3D-MRL aligns nested 3D embeddings with frozen CLIP image and text features across multiple prefix lengths.
- A single 3D-MRL encoder produces high-quality embeddings at any prefix length.
- 3D-MRL achieves competitive performance on zero-shot and few-shot 3D recognition benchmarks including Objaverse-LVIS, ModelNet, and ScanNet.
- 3D-MRL degrades gracefully across a wide range of dimensionality budgets, enabling an accuracy-efficiency trade-off from one model.
2 Related Work
Prior work established multimodal 3D alignment and nested representations separately, but recent 3D multimodal methods did not incorporate nested or multi-granularity representations. 3D-MRL fills this gap by integrating Matryoshka structure into cross-modal 3D pre-training.
- 3D multimodal methods such as ULIP and OpenShape align point-cloud features with pretrained image and text embeddings.
- Matryoshka Representation Learning makes every selected-dimensional prefix independently useful, providing an accuracy-efficiency knob without retraining or model switching.
- Recent 3D-CLIP methods did not incorporate or evaluate nested or multi-granularity representations.
- 3D-MRL differs from post-hoc MRL by coupling nested structure with cross-modal contrastive pre-training from the start.
3 3D Matryoshka Representation Learning
3D-MRL trains a single 3D encoder to produce nested, multimodally aligned representations at multiple dimensionalities. Prefix-level supervision supports an accuracy–compactness trade-off without retraining while progressively refining semantic and geometric information.
- Framework: 3D-MRL aligns nested 3D point-cloud embeddings with frozen CLIP image and text embeddings across multiple dimensionalities.The image and text encoders remain frozen, while the 3D encoder is trained for multimodal alignment.
- Nested Representation Structure: A single 3D encoder produces valid representations at different prefix lengths, enabling variable dimensionality budgets without retraining or architecture changes.The representation uses prefixes of the 3D embedding, with smaller representations nested inside larger ones.
- Properties of the Learned Representation: Lower-dimensional prefixes preserve broad semantics, while larger prefixes add capacity for finer semantic, geometric, and structural distinctions.The hierarchy preserves lower-dimensional information as dimensionality increases rather than discarding it.
- Nested Representation Structure: The logarithmic dimension schedule spans 10 to 1280 dimensions, concentrating supervision at low dimensions while increasing representational capacity at larger prefixes.The chosen scales are D = {10,20,40,...,1280}, and the nesting structure can interpolate between designated scales.
- Multi-Scale Contrastive Alignment: 3D-MRL applies contrastive supervision independently at each embedding scale and aggregates the resulting multi-scale losses during training.Scale-specific classifiers support alignment at selected dimensionalities, with the encoder and classifiers jointly optimized.
- Application in Downstream Tasks: For downstream evaluation, zero-shot classification aggregates similarity across scales, while few-shot classifiers sweep dimensionalities using frozen nested embeddings.This characterizes the accuracy–compactness trade-off from one pretrained model.
4 Experiments
Across zero-shot, few-shot, real-world recognition, retrieval, and efficiency evaluations, 3D-MRL provides competitive accuracy while supporting nested representations across dimensionality budgets. Its embeddings also show coarse-to-fine retrieval behavior, and efficient formulations reduce overhead with only minor accuracy differences.
- Zero-Shot Recognition: 3D-MRL surpasses ULIP and OpenShape by +24.1% and +4.1% in Top-1 accuracy on Objaverse-LVIS.The comparison uses 3D-MRL trained on the Ensembled dataset and highlights its largest gains on the long-tailed benchmark.
- Zero-Shot Recognition: 85.8% top-1 accuracy on ModelNet40 surpasses OpenShape by +1.4% for PointBERT.On ScanObjectNN, performance remains comparatively fair, while SparseConv achieves competitive performance with OpenShape under the same setting.
- Compression and Efficiency: 50.9% vs. 46.6% at 1280D and 42.5% vs. 40.2% at 640D show that direct 3D-MRL prefixes outperform OpenShape at larger matched budgets on Objaverse-LVIS.Using 320D instead of 1280D reduces storage by 4× and CPU retrieval latency by 4.1× without retraining.
- Few-Shot Recognition: 3D-MRL surpasses OpenShape by 2.2%, 0.6%, and 1.4% in the 5-way 20-shot, 10-way 10-shot, and 10-way 20-shot settings.The improvements are reported for few-shot linear probing and become more evident in the more challenging 10-way settings.
- Real-World Recognition: 47.0% mean Top-1 accuracy at 640 dimensions outperforms OpenShape by +1.4% on zero-shot ScanNet recognition.Recognition improves progressively from 10 to 640 dimensions, while 640 dimensions outperform 1280 in this evaluation.
- Cross-Modal Retrieval: Low-dimensional prefixes retrieve broad semantic categories, while higher-dimensional embeddings increasingly refine geometric and semantic similarity.The coarse-to-fine behavior appears in point-cloud and text-input retrieval, including progressively more specific matches across dimensions.
- Ablation Study: The efficient configuration maintains competitive representational capacity while avoiding the overhead of K distinct parameter matrices.The non-efficient variant gains only 0.5 percentage points on Objaverse-LVIS Top-3 and 0.3 points on ModelNet40 Top-1.
5 Conclusion
3D-MRL introduces nested multimodal 3D representations aligned with frozen image and text embeddings, enabling one model to operate across dimensionality budgets without retraining. Experiments show competitive performance and progressively finer retrieval specificity.
- 3D-MRL aligns nested 3D representations with frozen image and text embeddings, allowing one model to operate across multiple dimensionality budgets without retraining.
- Experiments across Objaverse-LVIS, ModelNet40, ScanObjectNN, and ScanNet show competitive performance with improved accuracy-efficiency trade-offs.
- Nested representations organize information hierarchically across embedding dimensions, shifting retrieval from broad semantic similarity toward finer geometric and semantic specificity.
- 3D-MRL supports scalable multimodal 3D understanding, retrieval, and deployment under diverse computational constraints.