Source-linked AI summary
Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings
Peixi Wu, Ke Mei, Feipeng Ma, Bosong Chai, Zhibin Lan, Chenxi Zhao, Shannan Yan, Jie Chen, Zhangchi Hu, Yansong Peng, Bo Lin, Junjie Zhou, Dacheng Yin, Tianyi Wang, Fengyun Rao, Jing Lyu, Hebei Li, Xiaoyan Sun
TL;DR
General retrieval exposes redundancy and semantic ambiguity in CoT-based multimodal generative embeddings. RIME jointly learns retrieval-friendly rewriting and embeddings, adds CMA for mutual retrieval and Refine-RL for anchor-guided optimization, and reports stronger benchmark performance with shorter thinking. The framework nonetheless retains considerable inference latency.
Problem
CoT-based generative embeddings can produce redundant reasoning and semantically ambiguous summaries for broader retrieval scenarios, while joint generation and embedding optimization can impair embedding discriminability.
Method
RIME jointly optimizes retrieval-friendly rewriting and multimodal embeddings, using CMA to align embedding spaces and Refine-RL to use discriminative embeddings as semantic anchors.
Results
RIME substantially outperforms prior generative embedding models across MMEB-V2, MRMR, and UVRB while reducing the thinking process by approximately 50%.
Takeaways & Limitations
Retrieval-friendly rewriting provides a more compact yet semantically faithful representation, while CMA supports flexible deployment through mutual retrieval between embedding modes.
Takeaways & Limitations
Although rewriting reduces redundant reasoning and unnecessary summaries, generative embeddings still introduce considerable inference latency.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have emerged as a promising foundation for universal multimodal embeddings. Recent studies have shown that reasoning-driven generative multimodal embeddings can outperform discriminative embeddings on several embedding tasks. However, Chain-of-Thought (CoT) reasoning tends to generate redundant thinking steps and introduce semantic ambiguity in the summarized answers in broader retrieval scenarios. To address this limitation, we propose Rewrite-driven Multimodal Embedding (RIME), a unified framework that jointly optimizes generation and embedding through a retrieval-friendly rewrite. Meanwhile, we present the Cross-Mode Alignment (CMA) to bridge the generative and discriminative embedding spaces, enabling flexible mutual retrieval to trade off efficiency and accuracy. Based on this, we also introduce Refine Reinforcement Learning (Refine-RL) that treats discriminative embeddings as stable semantic anchors to guide the rewrite optimization. Extensive experiments on MMEB-V2, MRMR and UVRB demonstrate that RIME substantially outperforms prior generative embedding models while significantly reducing the length of thinking.
1 Introduction
MLLM-based generative embeddings extend multimodal representation learning with reasoning, but CoT can add redundancy and ambiguity in general retrieval. RIME replaces that pattern with retrieval-friendly rewriting, aligns generative and discriminative spaces, and reports stronger benchmark performance with shorter thinking.
- CoT-based generative embeddings introduce intermediate reasoning to enhance semantic understanding, but general retrieval can make those steps redundant and targets semantically ambiguous.These limitations affect both decoupled and jointly trained CoT-based architectures.
- RIME replaces CoT trajectories and summarized answers with structured rewriting of images, videos, documents, and text for retrieval.The modality-agnostic framework supports multi-granularity, complex logical, and mixed-modality retrieval.
- Cross-Mode Alignment uses mutual contrastive learning to align generative and discriminative embeddings for positive queries while separating negative samples.This alignment enables mutual retrieval between the two embedding spaces.
- Approximately 50% less thinking and significantly better MMEB-V2 performance are reported versus prior methods.The paper also reports broad improvements on MMEB-V2, UVRB, and MRMR.
- Refine Reinforcement Learning uses discriminative embeddings as anchors to guide rewriting toward greater semantic richness and stronger discriminative capability.The framework also jointly optimizes generation and embedding through Rewrite-Driven Joint SFT.
2 Related Works
Multimodal embedding research has progressed from fixed dual encoders to MLLM-based unified spaces, while generative methods use reasoning to improve semantic modeling. Related query-rewrite work motivates semantic expansion, but CoT-based embedding methods retain computational overhead and redundant thinking.
- Multimodal Representation Learning: Traditional dual encoders such as CLIP and BLIP use fixed structures that struggle with videos, multi-page documents, and mixed-modal content.These architectures are trained with contrastive learning for shared multimodal representations.
- Multimodal Representation Learning: MLLM-based models use instruction tuning and contrastive learning to project heterogeneous modalities into a shared semantic space.Examples include VLM2Vec, GME, MM-Embed, and LLaVE.
- Multimodal Generative Embedding: Generative embedding methods add intermediate reasoning or summarization to exploit generative models' semantic understanding for embedding learning.CoT is a widely used paradigm for enhancing complex query understanding.
- Multimodal Generative Embedding: CoT-based methods can conflict with contrastive embedding objectives, increase computational overhead, and produce redundant thinking.Decoupled TTE designs and reinforcement-learning approaches address the tension in different ways.
- Retrieval-oriented Query Rewrite: Query rewriting methods bridge query-document semantic gaps through expansion, rationale generation, pseudo-documents, decomposition, and constraint inference.MLLMs extend rewriting to semantic-level operations and diverse query generation.
- Retrieval-oriented Query Rewrite: The multimodal rewrite template presents input text, images, videos, or combinations alongside discriminative and generative embedding tags.The figure and prompt demonstration support SFT cold-start data construction.
3 Method
RIME replaces CoT-based embedding generation with retrieval-friendly rewriting, then aligns generative and discriminative spaces and refines rewriting with discriminative-embedding rewards. Its joint training addresses semantic mismatch and inference latency while preserving flexible retrieval pathways.
- Motivation: CoT can mismatch retrieval needs because targets often require no summarization, causing granularity mismatches and semantic distortions.CoT also introduces long reasoning processes that increase inference cost.
- Rewrite-Driven Joint SFT: Rewrite-Driven Joint SFT jointly trains structured rewriting and contrastive learning for queries and targets in an end-to-end pipeline.The framework rewrites images, videos, documents, and text rather than generating reasoning trajectories followed by summarized answers.
- Rewrite-Driven Joint SFT: The joint loss combines autoregressive rewrite loss with cross-mode InfoNCE, balancing rewrite quality against retrieval performance through λ.LCM_InfoNCE is the cross-mode contrastive loss, while LRewrite is the autoregressive language-modeling loss.
- Cross-Mode Alignment: Cross-Mode Alignment maps generative and discriminative embeddings into a unified semantic space and supports mutual retrieval through inter-mode and intra-mode alignment losses.Deployment can choose discriminative embeddings for low-latency retrieval or generative rewriting when deeper semantic reasoning is required.
- Refine Reinforcement Learning: Refine Reinforcement Learning uses discriminative embeddings as stable anchors and rewards rewrites that improve positive-versus-negative similarity gaps while preserving format adherence.GRPO optimizes the rewriting policy using grouped rewrite outputs, with clipping and KL regularization controlled by the stated objective.
4 Experiments
RIME is evaluated across multimodal, expert-reasoning, and video retrieval benchmarks, with ablations testing its components, rewriting strategy, and cross-mode inference. It achieves strong benchmark performance while shortening generated thinking and supporting flexible retrieval modes.
- MMEB-V2: RIME-7B achieves 68.6 on MMEB-V2, surpassing UME-R1-7B by 4.1 points and VLM2Vec-7B by 16.3 points.
- MRMR: RIME-7B achieves 50.2 on MRMR, surpassing UME-R1-7B by 2.2 points, Ops-MM-Embed-7B by 2.1 points, and GME-7B by 14.0 points.
- UVRB: RIME-7B achieves 55.6 on UVRB, surpassing Unite-7B by 1.8 points and GME-7B by 2.6 points across 16 subtasks.
- Rewrite versus CoT: Rewrite outperforms CoT by 4.4 points on MMEB-V2, while removing the summarized answer improves CoT to 60.5 and Rewrite surpasses its answer-conditioned variant by 2.3 points.
- Efficiency: Rewrite uses 212 average tokens, and Refine-RL reduces average tokens by approximately 52% relative to UME-R1.
- Embedding modes: Gen.-Gen. averages 58.1 across benchmarks, while Disc.-Gen. reaches 57.0 with zero query tokens and Gen.-Disc. reaches 57.1 with 232 query tokens.
5 Conclusion
RIME replaces traditional CoT with retrieval-friendly rewriting and achieves strong generalization while reducing inference overhead. However, rewriting still introduces considerable inference latency, motivating future latent-thinking approaches.
- RIME substitutes chain-of-thought reasoning with retrieval-friendly rewriting for efficient and generalizable generative multimodal embeddings.
- RIME achieves state-of-the-art performance across MMEB-V2, MRMR, and UVRB while significantly reducing inference overhead.
- Rewriting still introduces considerable inference latency, limiting practical large-scale retrieval applications.
A.1 Training Data Composition
RIME’s training data spans image, video, and visual-document retrieval scenarios, with separate supervised fine-tuning and reinforcement-learning datasets. Modality-aware prompts generate structured rewrites tailored to each input type and task.
- Data composition: Rewrite SFT uses approximately 1.5 million samples spanning images, videos, and visual documents.
- Data composition: Image data covers matching, visual question answering, classification, and composed image retrieval scenarios.
- Data composition: Video data includes video question answering, video retrieval, and caption retrieval subsets.
- Data composition: Refine-RL uses approximately 15K non-overlapping samples uniformly sampled across image, video, and visual-document modalities.
- Rewrite annotation: Modality-aware prompts produce semantic rewrites for text and descriptive reconstructions for images and videos, including temporal structure and key events.
- Rewrite annotation: Qwen2-VL-72B-Instruct generates rewrite annotations that are post-processed into a standardized embedding-compatible format.
B Additional Experimental Results
Additional experiments report benchmark and hyperparameter analyses, including MR2-Bench results and the effect of the Rewrite SFT loss weight. The supplied passages identify these evaluations but provide no corresponding numerical outcomes.
- MR2-Bench: MR2-Bench reports nDCG@10 across 12 datasets spanning biology, cooking, gardening, physics, chemistry, earth science, economics, mathematics, nature, spatial, puzzle, and analogy tasks.
- Hyperparameter analysis: The hyperparameter study evaluates the effect of λ in the joint Rewrite SFT loss using average MMEB-V2 scores.
B.1 Ablation on Hyperparameter 𝜆
The ablation shows that λ balances rewrite generation quality against retrieval performance, with equal weighting producing the best MMEB-V2 score.
- 64.1 is the optimal MMEB-V2 overall score at λ = 1.0, where rewrite and contrastive objectives receive equal weighting.Lower weighting reaches 59.7 at λ = 0.1, while higher weighting reaches 62.3 at λ = 1.5.
- Increasing λ from 0.1 to 0.5 improves the overall score from 59.7 to 61.1.
- Overemphasizing rewrite generation at λ = 1.5 slightly reduces performance to 62.3, weakening discriminative embedding capability.
B.2 Results on MR2-Bench
RIME-7B performs strongly on MR2-Bench, surpassing UME-R1 overall and improving especially on several reasoning-intensive subtasks, although Puzzle remains difficult.
- 28.6 is RIME-7B’s comprehensive MR2-Bench score, exceeding UME-R1’s 27.5 and approaching Seed-1.6-embed’s 30.7.
- RIME-7B improves over UME-R1 by 8.5 points on Economics, 2.7 on Spatial reasoning, and 2.9 on Analogy.The corresponding scores are 64.2 versus 55.7, 40.4 versus 37.7, and 15.1 versus 12.2.
- Puzzle remains challenging for all models, with scores near zero on this abstract visual reasoning task.
- The results indicate that rewrite-driven embeddings activate internal knowledge for reasoning-intensive retrieval without explicit chain-of-thought reasoning.
B.3 Detailed Results on MMEB-V2/UVRB
RIME-7B achieves strong benchmark-wide performance, including the best reported overall UVRB result and detailed evaluations across MMEB-v2 modalities.
- 55.6 is RIME-7B’s average UVRB score across 16 video-retrieval datasets, the benchmark’s best overall performance.
- MMEB-v2 evaluation covers all 78 subtasks across image, video, and visual-document modalities.
- The paper provides detailed per-task MMEB-v2 and UVRB results to support fair comparison and reproducible research.
C Examples of Data Construction
The data-construction examples illustrate how modality-aware prompts transform text, images, videos, and multimodal inputs into retrieval-friendly rewrites.
- Data construction: The figures document data-construction examples across different modalities and provide concrete details of the rewriting process.
- Text rewriting: Text rewriting analyzes type, resolves ambiguity, extracts logical relationships, and produces concise semantic transformations.
- Prompt design: Modality-aware rewrite prompts are organized by input modality and task type, including text, image, and image-text VQA or description inputs.
- Image rewriting: Image rewriting reconstructs subjects, backgrounds, colors, and object relationships while maintaining objectivity.
- VQA rewriting: VQA examples assess image information, analyze visual content, answer the question, and encode the resulting rewrite with the target.
- Caption retrieval: Caption-retrieval examples rewrite a cat-and-carrots caption through sentence transformations before matching it to an image target.