MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction

Zilin Xiao; Qi Ma; Mengting Gu; Chun-cheng Jason Chen; Xintao Chen; Vicente Ordonez; Vijai Mohan

Source-linked AI summary

MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction

Zilin Xiao, Qi Ma, Mengting Gu, Chun-cheng Jason Chen, Xintao Chen, Vicente Ordonez, Vijai Mohan

arXiv:2509.18095v2Sep 22, 2025cs.IRcs.CLcs.CV

TL;DR

Existing multimodal retrieval methods either compress inputs into single vectors that lose fine-grained information or use many vectors that are costly to store and compare. MetaEmbed uses learnable Meta Tokens and Matryoshka Multi-Vector Retrieval to provide compact, coarse-to-fine embeddings whose test-time budget is adjustable. It achieves state-of-the-art retrieval across MMEB and ViDoRe while scaling to 32B-parameter models, with deployment memory remaining a practical constraint.

Problem
Existing multimodal retrieval methods either lose fine-grained information through single-vector representations or incur prohibitive storage and computation costs with many vectors.
Method
MetaEmbed appends learnable Meta Tokens and trains their representations with Matryoshka Multi-Vector Retrieval for selectable coarse-to-fine late interaction.
Results
MetaEmbed achieves state-of-the-art retrieval across MMEB and ViDoRe, with test-time scalability remaining effective for models up to 32B parameters.
Takeaways & Limitations
MetaEmbed enables retrieval systems to balance accuracy, index size, and latency by selecting the retrieval budget at test time.
Takeaways & Limitations
Index memory grows with the retrieval budget, requiring balanced budgets or CPU offloading for large deployments.

Abstract

from arXiv · show

Universal multimodal embedding models have achieved great success in capturing semantic relevance between queries and candidates. However, current methods either condense queries and candidates into a single vector, potentially limiting the expressiveness for fine-grained information, or produce too many vectors that are prohibitive for multi-vector retrieval. In this work, we introduce MetaEmbed, a new framework for multimodal retrieval that rethinks how multimodal embeddings are constructed and interacted with at scale. During training, a fixed number of learnable Meta Tokens are appended to the input sequence. At test-time, their last-layer contextualized representations serve as compact yet expressive multi-vector embeddings. Through the proposed Matryoshka Multi-Vector Retrieval training, MetaEmbed learns to organize information by granularity across multiple vectors. As a result, we enable test-time scaling in multimodal retrieval where users can balance retrieval quality against efficiency demands by selecting the number of tokens used for indexing and retrieval interactions. Extensive evaluations on the Massive Multimodal Embedding Benchmark (MMEB) and the Visual Document Retrieval Benchmark (ViDoRe) confirm that MetaEmbed achieves state-of-the-art retrieval performance while scaling robustly to models with 32B parameters. Code is available at https://github.com/facebookresearch/MetaEmbed.

1 Introduction

Multimodal retrieval must balance the fine-grained expressiveness of multi-vector representations against their substantial storage and computation costs. MetaEmbed addresses this trade-off with learnable Meta Tokens and coarse-to-fine training, achieving strong retrieval performance while allowing test-time efficiency control.

Motivation: Single-vector methods can lose fine-grained cross-modal details, whereas naive multi-vector methods require storing and comparing hundreds or thousands of token embeddings.These costs make multimodal-to-multimodal retrieval computationally prohibitive.
MetaEmbed: MetaEmbed appends learnable Meta Tokens whose contextualized representations form compact multi-vector embeddings for late interaction.Its Matryoshka Multi-Vector Retrieval module organizes representations into nested coarse-to-fine groups that can be selectively used at test time.
Flexible retrieval: Users can trade retrieval quality against index storage and latency by increasing or decreasing the number of Meta Embeddings used during indexing and interaction.The nested groups support selecting retrieval granularity at test time without retraining.
Results: MetaEmbed achieves state-of-the-art retrieval performance across MMEB and ViDoRe, with test-time scalability remaining effective for models up to 32B parameters.The evaluations cover images, text, and visual documents, while larger models show minimal diminishing returns in test-time scalability.

2 Related Work

Prior work develops multimodal embeddings, multi-vector late interaction, and Matryoshka representations, but existing approaches leave efficiency and multimodal-to-multimodal scalability challenges. MetaEmbed combines these directions into a flexible retrieval framework.

Multimodal Embedding: Multimodal embedding models project heterogeneous inputs into a shared representation space and use contrastive training to align modalities.Representative systems include CLIP, MetaCLIP, BLIP, and SigLIP.
Multi-Vector Retrieval: ColBERT-style multi-vector retrieval preserves multiple embeddings for late interaction, but naive text-image systems remain computationally prohibitive when images supply hundreds of query-side tokens.Existing efficiency methods use approximation or compression, yet multimodal-to-multimodal retrieval remains unsupported by these approaches.
Matryoshka Representation Learning: Matryoshka Representation Learning organizes features at multiple granularities in a nested structure so retrieval can select prefixes according to computational budget.MetaEmbed extends this idea to multi-vector multimodal retrieval through nested groups of representations.

3 Methodology

MetaEmbed uses learnable Meta Tokens to produce compact contextualized multi-vector representations, then organizes them into nested prefixes for flexible late-interaction retrieval. Users can select retrieval budgets at test time to trade accuracy against computational cost without retraining.

Late Interaction: Late interaction selects each query vector’s maximum similarity with candidate vectors and sums these alignments into a retrieval score.This preserves multiple vector-level interactions rather than condensing each input into one vector.
MetaEmbed Recipe: MetaEmbed appends learnable Meta Tokens to queries and candidates, using their final hidden states as contextualized multi-vector Meta Embeddings.The representations are produced by the underlying VLM in separate query and candidate forward passes.
Motivation: The design reduces the number of vectors required for retrieval while maintaining retrieval quality, addressing the cost of storing and comparing many multimodal vectors.Using all vectors would make index size scale as O(N × Rc × D).
Matryoshka Multi-Vector Retrieval: MetaEmbed organizes vectors into prefix-nested groups, so early vectors provide coarse summaries while later vectors refine the representation.The training objective optimizes contrastive losses across groups in parallel, encouraging each prefix to remain discriminative.
Test-time Scaling: Figure 2 depicts a nested index, latency growth with index size, and MetaEmbed-7B performance across retrieval budgets.Latency is measured with 100,000 candidates per query on an A100 GPU.
Test-time Scaling: At test time, users select query and candidate prefix sizes according to latency constraints, trading faster coarse scoring for higher-precision larger-prefix retrieval.Different group sizes can be selected without retraining because the groups are optimized jointly.

4 Experiments

MetaEmbed is evaluated across multimodal benchmarks, model architectures, sizes, retrieval budgets, and ablations. Results show state-of-the-art performance, improved scaling with larger models and budgets, and a key dependence on Matryoshka Multi-Vector Retrieval training.

4.1 Settings: MetaEmbed uses VLMs spanning unified and cross-attention architectures, with experiments covering Qwen2.5-VL, PaliGemma, and Llama-3.2-Vision at multiple model sizes.MMEB evaluation covers 36 tasks across classification, VQA, retrieval, and visual grounding; ViDoRe v2 covers seven visual-document retrieval tracks.
4.2 Main Results: At 7B and 32B, MetaEmbed establishes state-of-the-art MMEB performance, reaching 76.6 and 78.7 overall while widening its advantage over baselines with scale.At 7B it exceeds MoCa-7B and mmE5 by over 5–7 points; the 32B model improves over stronger baselines and smaller variants.
4.2 Main Results: Backbone choice substantially affects performance: MetaEmbed-11B with Llama-3.2-Vision scores 42.1 on VQA and 65.1 overall, whereas Qwen2.5-VL initialization yields 74.2 VQA at 7B and 78.7 overall at 32B.The Llama-based model remains strong in grounding and classification but has a sharply lower VQA score.
4.2 Main Results: MetaEmbed performs strongly on ViDoRe v2 across all seven tracks, with especially large gains in multilingual and biomedical domains despite no explicit multilingual training data.The reported behavior suggests the method retains and uses cross-lingual capabilities from its backbone.
4.3 Ablation Studies: Increasing the retrieval budget from (1,1) to (16,64) steadily improves MMEB performance, with the largest reported gain reaching +6.6 points for MetaEmbed-32B.Higher-capacity models obtain larger gains, and the (16,64) setting is used for the main benchmark tables.
4.3 Ablation Studies: Matryoshka Multi-Vector Retrieval preserves flexible low-budget scaling: removing MMR causes a 9.0-point drop at (1,1), while MMR remains slightly better even at (16,64).The MMR model consistently outperforms the non-MMR model as retrieval budget increases, with negligible degradation on MMEB.

5 Discussion

MetaEmbed is efficient in practice because query encoding dominates latency, scoring remains lightweight under realistic budgets, and memory growth can be controlled through retrieval-budget and system-level choices.

Retrieval pipeline: The retrieval pipeline comprises query encoding, late-interaction scoring, and ranking, with ranking remaining lightweight after candidate scores are computed.MetaEmbed uses late interaction between multiple embeddings rather than single-vector dot products during scoring.
Efficiency findings: Scoring latency remains nearly flat across moderate retrieval budgets, although scoring FLOPs grow substantially and become compute-bounded at (16, 64).GPU throughput accommodates the additional FLOPs without becoming a bottleneck until the extreme budget case.
Efficiency findings: 42.72 TFLOPs and 788ms are required to encode a 1024-token image query, making query encoding far more costly than scoring.The passage indicates that efficiency improvements should primarily target encoding when only a small number of candidates is scored.
Efficiency findings: Index memory grows proportionally with retrieval budget, but balanced budgets or CPU offloading can mitigate the deployment burden.This is the principal efficiency trade-off identified for flexible multi-vector retrieval.

6 Conclusion

MetaEmbed uses learnable Meta Tokens with Matryoshka Multi-Vector Retrieval to organize multimodal information from coarse to fine granularity. This enables flexible late interaction that balances retrieval accuracy, index size, and latency, supporting test-time scalability.

Conclusion: MetaEmbed uses learnable Meta Tokens and Matryoshka Multi-Vector Retrieval to organize information across coarse-to-fine granularities.The framework uses a small set of tokens to construct multimodal embeddings for flexible interaction.
Conclusion: Flexible late interaction lets users balance retrieval accuracy, index size, and latency by scaling the number of vectors used at test time.The paper presents this controllability as a way to bridge fine-grained expressiveness with large-scale deployment.

A Implementation Details

The implementation uses standardized training infrastructure and distributed strategies across MetaEmbed variants, with checkpointing and FlashAttention reducing memory and attention costs.

Variant details: Table 4 lists the training details of the MetaEmbed variants.The table is the implementation-detail reference for the model configurations.
Training infrastructure: MetaEmbed variants use PyTorch 2.6.0+cu124, FlashAttention 2.0, and gradient checkpointing during training.Gradient checkpointing is used to reduce memory usage.
Distributed training: The 3B configuration uses Distributed Data Parallel, whereas larger configurations use Fully Sharded Data Parallel v2.The distributed strategy therefore changes with model size.

B Detailed MMEB Ablation Results

MetaEmbed consistently outperforms the strongest single-vector and naive multi-vector baselines under identical settings. Matryoshka Multi-Vector Retrieval adds flexible scaling without statistically significant retrieval-quality loss.

Single-vector baselines: The single-last baseline uses the final hidden state as one retrieval vector, while single-mean averages all final-layer hidden states.These are the single-vector baselines used for comparison.
Multi-vector baseline: The split-(16, 64) baseline partitions query and candidate hidden states into 16 and 64 mean-pooled vectors without adding parameters.It serves as a fixed-length naive multi-vector retrieval baseline.
Baseline comparison: MetaEmbed consistently outperforms both the top single-vector method and the naive multi-vector baseline under identical training settings.Table 5 compares these methods and identifies single-last as the best single-vector reference.
MMR ablation: MMR introduces flexibility without a statistically significant loss in retrieval quality.The comparison indicates that disabling MMR does not provide a statistically significant quality advantage.

C Baseline Method Introduction

The baselines span dual-encoders, instruction-guided retrievers, universal multimodal embedding models, and modality-aware or synthetic-data training approaches.

C Baseline Method Introduction: CLIP is a dual-encoder trained contrastively on 400M image–text pairs for aligned representations and zero-shot classification and retrieval.It provides a foundational cross-modal baseline for the comparisons.
C Baseline Method Introduction: Other competitive baselines use instruction-guided training, modality-aware hard negatives, synthetic multimodal data, or large-scale fine-tuning.These include MagicLens, MM-Embed, GME, MMRet, mmE5, and B3.
C Baseline Method Introduction: UniIR handles eight instruction-guided retrieval formats across text, image, and mixed-modality queries and candidates.It is jointly trained on ten heterogeneous datasets and reports robust in-distribution performance and zero-shot generalization.
C Baseline Method Introduction: VLM2Vec converts a pretrained vision-language model into a universal embedding model through instruction-tuned contrastive learning on MMEB.MMEB covers 36 tasks spanning classification, VQA, retrieval, and grounding.
C Baseline Method Introduction: MoCa uses modality-aware continual pre-training followed by heterogeneous contrastive fine-tuning across text, image, and mixed-modality pairs.Both MoCa-3B and MoCa-7B show competitive overall performance among baselines on MMEB.