PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Then Embed: Generative Context Improves Multimodal Embedding

Xuanming Cui, Jianpeng Cheng, Hong-you Chen, Satya Narayan Shukla, Abhijeet Awasthi, Xichen Pan, Chaitanya Ahuja, Shlok Kumar Mishra, Yonghuan Yang, Jun Xiao, Qi Guo, Ser-Nam Lim, Aashu Singh, Xiangjun Fan

arXiv:2510.05014v4cs.AIcs.LG

TL;DR

Universal multimodal embeddings need to represent increasingly complex instructions, while existing MLLM-based methods underuse their generative reasoning capacity. TTE inserts a reasoner before an embedder, and achieves state-of-the-art MMEB-V2 results while distilled and unified variants improve open-source performance and efficiency.

  • Problem

    Existing MLLM-based universal multimodal embeddings treat MLLMs mainly as encoders, limiting their handling of complex instructions that require compositional reasoning.

  • Method

    TTE uses a reasoner to generate embedding-centric reasoning traces, then conditions an embedder on the original input and those traces; smaller reasoners can be distilled and integrated with embedding heads.

  • Results

    TTEt-7B achieves the highest MMEB-V2 leaderboard score at 71.5%, while TTEs is best among open-source models and improves the 2B baseline by 5.1%.

  • Takeaways & Limitations

    Explicit reasoning benefits multimodal representation learning, and distilled or unified designs can retain strong retrieval performance with improved model efficiency.

  • Takeaways & Limitations

    Chain-of-thought reasoning has higher inference cost, motivating approaches that lower cost while preserving performance gains.

Abstract

from arXiv · show

There is a growing interest in Universal Multimodal Embeddings (UME), where models are required to generate task-specific representations. While recent studies show that Multimodal Large Language Models (MLLMs) perform well on such tasks, they treat MLLMs solely as encoders, overlooking their generative capacity. However, such an encoding paradigm becomes less effective as instructions become more complex and require compositional reasoning. Inspired by the proven effectiveness of chain-of-thought reasoning, we propose a general Think-Then-Embed (TTE) framework for UME, composed of a reasoner and an embedder. The reasoner MLLM first generates reasoning traces that explain complex queries, followed by an embedder that produces representations conditioned on both the original query and the intermediate reasoning. This explicit reasoning step enables more nuanced understanding of complex multimodal instructions. Our contributions are threefold. First, by leveraging a powerful MLLM reasoner, we achieve state-of-the-art performance on the MMEB-V2 benchmark, surpassing proprietary models trained on massive in-house datasets. Second, to reduce the dependency on large MLLM reasoners, we finetune a smaller MLLM reasoner using high-quality embedding-centric reasoning traces, achieving the best performance among open-source models with a 7% absolute gain over recently proposed models. Third, we investigate strategies for integrating the reasoner and embedder into a unified model for improved efficiency without sacrificing performance.

1 Introduction

Universal multimodal embeddings need to handle instructions whose complexity demands reasoning, but existing MLLM-based approaches mainly use models as encoders. Think-Then-Embed adds explicit reasoning before embedding and studies teacher, distilled, and unified designs.

  • 1 Introduction: Think-Then-Embed addresses complex multimodal instructions by adding an explicit thinking stage before embedding, rather than restricting MLLMs to encoder-only models.The motivation includes VQA, classification, visual grounding, and composed retrieval, where queries can require substantial reasoning.
  • 1 Introduction: TTE conditions task-specific representations on both the original multimodal input and generated reasoning traces.Figure 1 presents the representation as conditioned on the original input and the thinking result.
  • 1 Introduction: A powerful reasoner with a smaller embedder achieves state-of-the-art MMEB-V2 performance, surpassing close-sourced models trained with additional data.The contribution passage identifies Qwen2.5 VL 72B as an example reasoner and Qwen2 7B as the smaller embedder.
  • 1 Introduction: Distilling reasoning traces into a smaller MLLM reasoner achieves the best performance among open-source models across MMEB-V2.The distilled reasoner uses traces from the 72B reasoner and performs effectively across the full benchmark.
  • 1 Introduction: For unified models, two-stage autoregressive-then-contrastive training consistently outperforms joint training while almost halving parameters without degrading end-to-end TTE retrieval.The two-stage design uses a dedicated embedding head on top of the reasoner.

2 Related Work

Universal multimodal embeddings extend multimodal representation learning from content similarity to query- and instruction-dependent tasks. Related MLLM and query-rewriting approaches motivate TTE’s focus on reasoning-conditioned embeddings rather than external text retrieval.

  • 2 Related Work: Foundational multimodal models such as CLIP, BLIP, and SigLIP align separate image and text encoders using contrastive objectives.These models established the multimodal representation paradigm preceding universal multimodal embeddings.
  • 2 Related Work: Universal multimodal embeddings make representations depend on both the query and task instruction, supporting retrieval, VQA, grounding, and classification.MMEB and MMEB-V2 provide benchmarks spanning cross-modal retrieval and non-conventional retrieval tasks.
  • 2 Related Work: Recent MLLM-based embedding work improves performance through continued contrastive training, hard-negative mining, additional data, and related training strategies.These approaches build embedding models directly on powerful multimodal language models.
  • 2 Related Work: Query rewriting improves text retrieval by enriching or reformulating queries, but prior methods rely on external retrievers rather than learned embeddings.The cited approaches include prompt-based, iterative or reinforcement-learning-guided, and multi-reformulation methods.
  • 2 Related Work: Unlike text-to-video query rewriting for encoder models, TTE uses reasoning to improve general instruction-following multimodal embeddings for both queries and targets.This distinguishes TTE’s scope from approaches that enrich only textual queries to bridge information asymmetry.

3 Preliminary: MLLM-based Universal Multimodal Embedding

UME represents multimodal query–target pairs using a shared MLLM, with embeddings conditioned on visual input, text, and task instructions. Training uses pooled hidden states and a uni-directional InfoNCE objective based on cosine similarity.

  • Each UME query and target combines optional visual input, textual input, and a predefined task instruction before separate encoding by the same MLLM.
  • The MLLM embedder and pooling operation convert each multimodal input into its final representation, using the last token’s hidden state for pooling.
  • Training applies a uni-directional q →t InfoNCE loss, with cosine similarity scaled by temperature τ.

4 Think-then-Embed

Think-Then-Embed adds an explicit reasoning stage before multimodal embedding: a reasoner generates task-aligned ECR traces, and an embedder conditions representations on both inputs and traces. The framework also explores smaller reasoners, offline teacher inference, and unified reasoner–embedder designs to manage inference cost.

  • 4 Think-then-Embed: TTE first generates Embedding-Centric Reasoning traces and then produces task-specific representations conditioned on the original multimodal input and those traces.Formally, ψ = gω(V, [Ins], T), and h = Pooling(fθ(V, [Ins], T, ψ)).
  • 4.1 Embedding-Centric Reasoning: ECR adapts intermediate reasoning to task type, using step-by-step reasoning for VQA and detailed visual descriptions or summaries for visual document embedding.
  • 4.1 Embedding-Centric Reasoning: Manual task prompts format ECR as intermediate CoT followed by final reasoning, enabling task-aligned traces intended to produce more semantically aligned embeddings.The paper leaves retrieval-signal optimization of the reasoner for future work.
  • 4.2 TTE with Teacher Reasoner: Because CoT increases inference cost, TTE uses a powerful teacher reasoner with a lightweight embedder and can generate ECR offline once for indexed data.The reasoner is not used during online retrieval of existing data points.
  • 4.3 TTE with Finetuned Reasoner: Zero-shot ECR from the backbone improves classification, retrieval, VQA, and grounding, while the unified model seeks one-pass reasoning and embedding through joint training or an embedding head.
  • 4.3 TTE with Finetuned Reasoner: The embedding-head study compares learnable attention queries, an NV-Embed-style pooler, a Qformer-style head, and self-attention blocks initialized from the backbone MLLM.Green components are trainable; Q, K, and V denote attention query, key, and value, while MHSA denotes multihead self-attention.

5 Experiments

Experiments use MMEB-V1 and MMEB-V2, covering multimodal embedding tasks across classification, VQA, retrieval, grounding, video, and visual-document settings. The study compares multiple TTE variants and established dual-encoder and MLLM-based baselines under shared training procedures.

  • Datasets: MMEB-V1 contains 20 in-distribution and 16 out-of-distribution tasks spanning classification, VQA, retrieval, and grounding, while MMEB-V2 adds video and visual-document retrieval.
  • Models: The evaluated TTE variants are teacher-reasoner, SFT-ed student-reasoner, and unified reasoner–embedder models built on Qwen2-VL 2B and 7B backbones.
  • Models: Baselines include CLIP, SigLIP, UniIR, MagicLens, and VLM2Vec-V1 alongside the TTE variants.
  • Training Details: Training uses global batch size 8192, learning rates of 2e−4 for the backbone and 5e−4 for the embedding head, temperature 0.02, and 1 or 2.3 epochs for MMEB-V1 or MMEB-V2.
  • Training Details: The ECR reasoner is fully fine-tuned with the visual encoder frozen, using learning rate 2e−5, global batch size 128, and one training epoch.

6 Results

TTE achieves strong performance across MMEB-V1 and MMEB-V2, with the largest gains appearing on reasoning-intensive tasks and difficult video retrieval.

  • MMEB-V1: 12.7% improvement over VLM2Vec-V1 is achieved by TTEt on the 7B embedding model, exceeding TTEs and TTEu.TTE achieves the best performance in both the 2B and 7B categories, with TTEt outperforming the next-best approach by 6%.
  • MMEB-V2: 71.5% overall on MMEB-V2 makes TTEt-7B the highest-scoring model, surpassing systems trained with massive external data.TTEs is the best open-source model across both 2B and 7B variants.
  • Task-level analysis: 5% improvement on video retrieval exceeds the 2% gain on image retrieval for TTEt-2B, while video QA also improves significantly over VLM2Vec-V2.VQA and classification-based tasks show notable gains against the VLM2Vec-V2 baseline.

7 Ablations

Ablations show that reasoning traces improve embeddings, especially when reasoners are fine-tuned and include intermediate CoT, while the embedder remains more effective than direct ECR retrieval.

  • Zero-shot reasoning: Zero-shot ECR improves performance across all tested tasks, including classification, retrieval, VQA, and grounding.The experiment uses a Qwen2VL 2B reasoner and embedder on MMEB-V1 subsets.
  • Reasoner fine-tuning: Over 6% absolute gain follows fine-tuning the TTE reasoner on MMEB-V1, confirming that ECR quality materially affects embedding performance.The ablation compares zero-shot and fine-tuned reasoner-generated ECR.
  • Intermediate CoT: Consistent gains from intermediate CoT appear in classification, retrieval, and grounding, with larger improvements for 7B than 2B models.VQA receives only marginal gains because it primarily requires the final answer.
  • Role of ECR: Direct text-to-text retrieval from ECR performs significantly worse than using ECR-conditioned TTE embeddings, showing that ECR supplies signals rather than replacing multimodal encoding.The embedder can extract useful information from imperfect ECRs without blindly relying on them, although different reasoners can still change performance.
  • Embedding geometry: TTEt produces greater overlap between query and target embeddings than baseline VLM2Vec in t-SNE visualizations across selected task types.The visualization compares three subtasks for each task type on MMEB-V1.
  • Embedding heads: Attention and NV-Embed-style poolers underperform TTEs, while a self-initialized MHSA head outperforms a QFormer-style head.Increasing attention-pooler queries from 1 to 16 yields little improvement; earlier-layer representations with expressive heads are favored.

8 Conclusion

Think-Then-Embed uses explicit reasoning to improve universal multimodal embeddings, distills reasoning into compact models, and supports efficient unified inference.

  • Conclusion: TTE significantly outperforms baselines and recent methods on MMEB-V1 and MMEB-V2 without requiring additional data.The framework is presented as effective and robust across both benchmarks.
  • Conclusion: A large reasoner can improve a smaller embedder, while distillation enables a compact reasoner and embedder to share the same backbone.The paper presents reasoning traces as beneficial for multimodal representation learning and efficiency.
  • Conclusion: A pluggable embedding head enables single-forward-pass embeddings and halves model parameters in the unified design.This integration targets lower inference and parameter costs.

A.1 Results on I2T/T2I retrieval.

TTEs outperform VLM2Vec and B3 on MSCOCO retrieval, with strong gains in both image-to-text and text-to-image settings. Noisy ECR construction further improves contrastive training across model sizes.

  • Results on I2T/T2I retrieval.: 82.0 Precision@1 for 7B TTEt on MSCOCO T2I surpasses B3's 62.8 by almost 20% absolute.Both TTEs and TTEt surpass VLM2Vec and B3 by a large margin on the reported retrieval results.
  • Results on I2T/T2I retrieval.: The evaluation covers zero-shot image-to-text and text-to-image retrieval on Flickr30k, with results reported for MSCOCO retrieval.The supplied passages identify the evaluation scope but do not provide the full table values.
  • Results on I2T/T2I retrieval.: 1–2% performance gains across TTEs and TTEt follow from noisy ECR construction for contrastive training across both model sizes.The noisy construction uses rephrasing and 50% non-ground-truth ECR.

B Joint Contrastive-Autoregressive Training

The unified model jointly optimizes contrastive embedding and autoregressive generation objectives. Its performance degrades when supervised fine-tuning is emphasized, while increasing the contrastive-loss weight partially restores performance.

  • Joint Contrastive-Autoregressive Training: Joint SFT and contrastive training combines autoregressive generation with embedding learning through a weighted loss.The joint objective is L_joint = λL_InfoNCE + L_SFT, where λ controls the contrastive-loss weight.
  • Joint Contrastive-Autoregressive Training: The unified model uses the second-last token as the embedding token because it is the last token generated during inference.A simple MLP block is applied to this token to obtain the embedding.
  • Joint Contrastive-Autoregressive Training: Training uses LoRA with rank 16 and alpha 64, a 2e−4 learning rate, global batch size 8192, and one epoch.
  • Joint Contrastive-Autoregressive Training: Adding the SFT objective degrades performance against the contrastive baseline, while increasing λ partially recovers it.The authors conjecture that conflicting gradients or suboptimal last-layer hidden states may explain the degradation.

D Full results

The paper reports full MMEB-V2 results in Table 13 and separate Visrag results in Table 13's following table. The supplied passages identify the comparison sources but do not provide result values.

  • Full results: Full MMEB-V2 results are reported in Table 13, using the official VLM2Vec 2B result and a reproduced VLM2Vec 7B baseline.The supplied text specifies the baseline provenance but not the numerical outcomes.
  • Full results: Visrag results on MMEB-V2 are reported separately in Table 13's following table.The supplied passages identify the table scope without reporting cell values or comparisons.
Loading 2510.05014v4…