MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs

• Universal multimodal retrieval accommodates diverse user-instructed tasks with text, image, or interleaved text-image queries and documents.Source: §1 · pp. 1–2
• MLLM retrievers outperform CLIP retrievers on challenging interleaved text-image queries but underperform on cross-modal retrieval because of modality bias.Source: §1 · p. 2
• Modality-aware hard negative mining and continuous fine-tuning produce MM-Embed, a universal retriever with state-of-the-art multimodal performance and competitive text retrieval.Source: §1 · p. 2
• Zero-shot MLLM rerankers further improve accuracy on especially challenging interleaved text-image queries.On CIRCO, reranking raises mAP@5 by over 7 points against the existing state-of-the-art composed-image retriever and the universal retrievers.Source: §1 · p. 2

• Instruction-aware dense retrieval uses task-specific instructions to learn diverse retrieval tasks beyond text retrieval.M-BEIR established instruction-aware fine-tuning as important for universal multimodal retrieval.Source: §2 · p. 2
• Vision-language retrieval research progressed from single-modal to cross-modal and complex multimodal tasks, often assuming homogeneous query and document modalities.Source: §2 · p. 2
• M-BEIR introduced heterogeneous queries and candidate pools with diverse retrieval tasks, but its study was limited to CLIP-based retrievers and omitted important text-to-text tasks.Source: §2 · pp. 2–3
• This work extends prior text-reranking research by prompting MLLMs as zero-shot rerankers for multimodal retrieval tasks.Source: §2 · p. 3

• Given a multimodal query and task instruction, the retriever ranks candidates from a pool of multimodal documents according to a specified ranking metric.Source: §3 · p. 3
• Queries and candidates may each be text, image, or interleaved text-image, while the same query can express different search intents through task-specific instructions.For example, an image query can request either an image caption or a similar image.Source: §3 · p. 3
• The paper considers text and image modalities only; audio and video are left for future work.Source: §3 · p. 4

4 METHOD

• Instruction-guided MLLM fine-tuning projects multimodal queries and documents into a shared space for k-nearest neighbor retrieval.The query representation includes the task instruction to capture retrieval-task intent.Source: §4 · p. 4, §4.1 · p. 4

4.1 FINE-TUNING MULTIMODAL LLMS FOR UNIVERSAL MULTIMODAL RETRIEVAL

• Modality-aware negative mining addresses MLLM modality bias by selecting incorrect-modality and same-modality candidates with unsatisfactory information.Incorrect-modality negatives rank above the labeled positive, while same-modality negatives rank below position k′; the experiment sets k′ = 45.Source: §4.1 · p. 4, §4.1.1 · p. 4

4.1.1 MODALITY-AWARE HARD NEGATIVE MINING

• The contrastive-learning candidate pool uses in-batch samples as random negatives before mining hard negatives from the retriever’s top-50 candidates.Hard-negative training uses twice as many candidate documents as random-negative mining under the same batch size, so batch sizes are adjusted for fair comparison.Source: §4.1.1 · pp. 4–5

4.1.2 CONTINUOUS FINE-TUNING

• Continuous fine-tuning combines universal multimodal and text-to-text retrieval data while retaining previously mined incorrect-modality negatives.The procedure rem ineates same-modality hard negatives with M hard and retains first-type negatives when M hard finds none.Source: §4.1.2 · p. 5

4.2 ZERO-SHOT MULTIMODAL LLM RERANKER

• A zero-shot LLaVa-Next reranker evaluates the top 10 retrieved candidates through True/False prompts and uses the True-token probability as the relevance score.The preliminary study reports improvements mainly for interleaved text-image queries, including composed image retrieval and visual question answering.Source: §4.2 · p. 5

Datasets and models

• M-BEIR contains 10 datasets with 16 multimodal retrieval tasks, while MTEB evaluation covers 15 diverse text retrieval datasets.M-BEIR training uses 1.1M queries and evaluation uses 190K test queries; MTEB reports average nDCG@10 across its datasets.Source: §5.1 · pp. 5–6
• LLaVa-Next MLLM retrievers use a Mistral 7B language model with a CLIP image encoder and vision-language projector, alongside CLIP-Large baselines.The experiments include LLaVa-Next variants and CLIPSF, a CLIP-based model that separately encodes image and text inputs.Source: §5.1 · p. 6

Main results

• MLLM retrievers outperform CLIPSF on multimodal-query tasks but lag on single-modal and especially cross-modal retrieval, whereas NVEmb leads text-to-text retrieval on WebQA.The results also show domain-dependent behavior for BLIPFF, which is stronger on fashion datasets than on VisualNews.Source: §5.2 · p. 7
• Hard-negative fine-tuning significantly improves MLLM retrieval accuracy, particularly for single-modal queries, while modality-aware mining mitigates bias toward retrieving relevant text instead of images.The authors attribute the gains partly to accelerating contrastive learning in MLLM retrievers and identify modality bias in top-1 errors.Source: §5.2 · pp. 7–8

Ablations and reranking

• Continuous fine-tuning improves text retrieval while retaining strong multimodal capability, and zero-shot reranking helps visual question answering but not FashionIQ or CIRR.Joint fine-tuning lowers multimodal accuracy by over 2 points, whereas staged fine-tuning incurs a 0.8-point multimodal drop; reranking benefits OVEN and InfoSeek.Source: §5.2 · p. 8, §5.3.2 · p. 9
• Zero-shot MLLM rerankers improve difficult interleaved text-image retrieval but degrade performance on most single-modal tasks, including after scaling from 7B to 34B.The authors hypothesize that cross-encoders better handle challenging multimodal queries, while lacking task- and domain-specific knowledge limits zero-shot performance.Source: §5.3.3 · p. 10

• Modality-aware hard negative mining improves MLLM-based retriever accuracy by 5 points on M-BEIR.Source: §6 · p. 10
• MM-Embed achieves state-of-the-art universal multimodal retrieval accuracy while ranking in the top five on MTEB retrieval tasks.Source: §6 · p. 10
• Zero-shot MLLM rerankers further boost retrieval accuracy on challenging multimodal-query tasks.Source: §6 · p. 10
• Future work includes distilling MM-Embed into smaller multimodal retrievers to improve retrieval efficiency.Source: §6 · p. 10
• Another proposed direction is distilling an MLLM reranker into a retriever for multimodal-query tasks.Source: §6 · p. 10

• Training and inference use Tevatron, while CLIP-based retrievers follow Wei et al.’s settings.Source: §A.1 · p. 15
• MLLM-based retrievers are fine-tuned with DeepSpeed Zero 2 and gradient checkpointing.Source: §A.1 · p. 15
• During M-BEIR fine-tuning, query and document maximum length is set to 128.Source: §A.1 · p. 15
• Continuous fine-tuning uses both M-BEIR and text-to-text retrieval training data.Source: §A.1 · p. 15

• Storage is measured on the 5.6M documents in the M-BEIR dataset.Source: §A.2 · p. 15
• Retrieval latency includes query encoding and vector search.Source: §A.2 · p. 15
• Query latency is measured by sampling 100 queries from each of the 16 M-BEIR test-query pools with batch size 1.Source: §A.2 · p. 15

Baseline reproduction

• The reproduced M rand(CLIPSF) implementation is intended to be equivalent to Wei et al.’s fine-tuned CLIPSF checkpoint.Source: §A.3 · p. 15

Benchmark statistics and comparisons

• M-BEIR includes statistics on datasets, document modalities, and comparisons with existing multimodal retrieval models.Source: Table 10 · p. 16, Table 11 · p. 16, Table 12 · p. 16
• Local M-BEIR evaluation restricts retrieval to each dataset’s sub-million-candidate, single-modality corpus.Source: Table 13 · p. 16

Modality analysis

• Most MLLM retrievers show low Task 1 modality accuracy because of modality bias, while modality-aware hard negatives resolve this issue for MM-Embed.Source: Table 14 · p. 17

Evaluation configurations

• The appendix reports detailed MTEB retrieval results and prompts for NVEmb, LLaVa-P, and reranking experiments.Source: Table 15 · p. 17, Table 16 · p. 17, Table 17 · p. 18

Negative samples

• Modality-aware negative samples can have incorrect modalities despite similar semantic meaning, whereas poor information-need matches are less accurate than correct answers.Source: Figure 2 · p. 19

Reranking analysis

• For composed image retrieval, retrieved and reranked top-1 candidates sometimes differ from the sole labeled positive while appearing correct.Source: Figure 3 · p. 20