Source-linked AI summary
Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings
Peixi Wu, Ke Mei, Feipeng Ma, Bosong Chai, Zhibin Lan, Chenxi Zhao, Shannan Yan, Jie Chen, Zhangchi Hu, Yansong Peng, Bo Lin, Junjie Zhou, Dacheng Yin, Tianyi Wang, Fengyun Rao, Jing Lyu, Hebei Li, Xiaoyan Sun
TL;DR
CoT-based multimodal embeddings can be redundant, ambiguous, and costly for general retrieval. RIME replaces CoT with retrieval-friendly rewriting and aligns generative and discriminative embeddings, outperforming prior generative models across benchmarks while reducing thinking length by about 50%.
Problem
CoT-based multimodal embeddings generate redundant reasoning and semantically ambiguous summaries that can increase inference cost and mismatch general retrieval needs.
Method
RIME uses retrieval-friendly rewriting with cross-mode alignment and reinforcement learning anchored by discriminative embeddings to jointly optimize multimodal generation and embedding.
Results
RIME achieves state-of-the-art performance across MMEB-V2, MRMR, and UVRB while reducing the thinking process by approximately 50% versus prior methods.
Takeaways & Limitations
Retrieval-friendly rewriting provides a more compact, semantically faithful representation while supporting flexible generative and discriminative retrieval modes.
Takeaways & Limitations
The framework adopts shared-parameter reasoning rather than an external reasoning model, leaving external-model alternatives outside its baseline evaluation.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have emerged as a promising foundation for universal multimodal embeddings. Recent studies have shown that reasoning-driven generative multimodal embeddings can outperform discriminative embeddings on several embedding tasks. However, Chain-of-Thought (CoT) reasoning tends to generate redundant thinking steps and introduce semantic ambiguity in the summarized answers in broader retrieval scenarios. To address this limitation, we propose Rewrite-driven Multimodal Embedding (RIME), a unified framework that jointly optimizes generation and embedding through a retrieval-friendly rewrite. Meanwhile, we present the Cross-Mode Alignment (CMA) to bridge the generative and discriminative embedding spaces, enabling flexible mutual retrieval to trade off efficiency and accuracy. Based on this, we also introduce Refine Reinforcement Learning (Refine-RL) that treats discriminative embeddings as stable semantic anchors to guide the rewrite optimization. Extensive experiments on MMEB-V2, MRMR and UVRB demonstrate that RIME substantially outperforms prior generative embedding models while significantly reducing the length of thinking. Code is available at https://github.com/PeppaWu/RIME.
1 Introduction
Generative multimodal embeddings use reasoning to improve semantic representations, but redundant or lengthy Chain-of-Thought increases cost and can create retrieval ambiguity. The proposed rewrite-driven paradigm replaces CoT with retrieval-friendly rewriting, while cross-mode alignment and refine reinforcement learning improve efficiency and embedding quality.
- Motivation: Generative embeddings add intermediate reasoning to multimodal representation learning, extending beyond discriminative models that directly extract token embeddings [7] [15] [21].MLLM-based embedding models emerged as general solutions because traditional dual encoders struggle with complex multimodal inputs and structured visual data such as documents and videos [37] [25].
- Problem: Long CoT reasoning increases inference cost, while TTE requires an external reasoner and other jointly trained approaches retain costly lengthy reasoning [6] [7] [15] [21] [30] [53].These limitations hinder low-latency, high-throughput deployment and motivate a retrieval-oriented alternative.
- Contribution: Rewrite-Driven Joint SFT replaces traditional CoT reasoning with modality-agnostic, task-general rewriting for images, videos, documents, and text.The framework structurally rewrites visual inputs for recaptioning and text for further explanation without producing a summarized final answer.
- Contribution: Cross-mode alignment uses mutual contrastive learning to align discriminative and generative embeddings for identical queries while separating negative samples.This enables cross-retrieval between embedding spaces, reducing rewriting-related inference overhead and improving flexibility.
- Results: Approximately 50% less thinking than prior methods [21] [30] accompanies significantly stronger performance than existing works on MMEB-V2 [17].The introduction reports both reduced reasoning length and improved benchmark performance.
- Contribution: Refine reinforcement learning uses discriminative embeddings as semantic anchors to guide rewriting toward better generative embeddings.The method is presented as a mechanism for refining the generative embedding through anchor-guided rewriting.
2 Related Works
Related work progresses from contrastive dual encoders toward MLLM-based unified embeddings and generative embedding methods, while query rewriting provides a route to bridge semantic gaps between queries and documents.
- Contrastive dual encoders such as CLIP [37], BLIP [25], and ALIGN [14] embed heterogeneous modalities into unified spaces for multimodal retrieval.These methods combine vision encoders with language models to strengthen semantic alignment.
- Fixed-structure encoders lack sufficient modeling capability for complex inputs, including videos, multi-page documents, and mixed-modal content.
- MLLMs [2] [3] [22] [23] [31] [39] enable unified embedding learning by projecting heterogeneous modalities into shared spaces through instruction tuning and contrastive learning.Representative models include VLM2Vec [17] [35], GME [50], MM-Embed [27], and LLaVE [20].
- Generative embedding methods use generation and intermediate reasoning steps, including Chain-of-Thought (CoT) [12] [18] [41] [42], to strengthen representation quality and complex query understanding.
- Joint autoregressive generation and embedding alignment can bias models toward language modeling and impair embedding discriminability [8] [26] [45], motivating decoupled designs such as Think-then-Embed (TTE) [6] [7].Other works [21] explore reinforcement learning and reward mechanisms to alleviate this conflict.
- Query rewriting bridges the semantic gap between users and documents, evolving from lexical feedback methods to LLM-based expansion, rationale generation, and pseudo-document construction [34].MLLMs further support semantic-level rewriting, including query decomposition and implicit constraint inference.
3 Method
RIME replaces CoT-based embedding generation with retrieval-oriented rewriting, jointly optimizing rewrite generation and contrastive learning. It further aligns generative and discriminative spaces and uses reinforcement learning to improve rewrite informativeness while preserving format and retrieval separation.
- Preliminaries: The generative embedding baseline shares parameters between the reasoning model and embedding model rather than using an external reasoning model, avoiding parameter redundancy and optimization issues.Generative embeddings use the hidden state of a designated token after producing reasoning and a summary.
- Rewrite-Driven Joint SFT: RIME replaces CoT reasoning and answer summarization with structured rewriting of multimodal queries and targets, reducing semantic mismatch with retrieval needs.CoT can introduce redundancy, inappropriate summarization granularity, and semantic distortion because retrieval queries lack unique answers and targets often need no summarization.
- Rewrite-Driven Joint SFT: The rewrite-driven framework jointly trains autoregressive rewriting and multimodal contrastive learning end to end, with λ balancing rewrite quality against retrieval performance.Unlike traditional query rewriting, RIME rewrites both queries and targets to align them in a shared rewritten semantic space.
- Cross-Mode Alignment: Cross-Mode Alignment maps generative and discriminative embeddings into one semantic space, enabling low-latency discriminative retrieval or deeper generative rewriting at inference.Its cross-mode and intra-mode losses support mutual retrieval and prevent generative embeddings from drifting away from stable discriminative semantics.
- Refine Reinforcement Learning: Refine-RL uses discriminative embeddings as stable anchors and rewards rewrites that increase positive–negative similarity gaps, follow the required format, and improve over the discriminative baseline.The rewrite policy is optimized with Group Relative Policy Optimization using grouped sampled rewrites and group-normalized rewards.
4 Experiments
RIME achieves strong multimodal retrieval performance across MMEB-V2, MRMR, and UVRB, while its rewrite-based approach improves over CoT reasoning and supports flexible embedding-mode retrieval through CMA. Ablations show gains from Rewrite-Driven Joint SFT, CMA, and Refine-RL, with rewriting reducing unnecessary summarized-answer effects.
- MMEB-V2: 68.6 overall on MMEB-V2 makes RIME-7B best among open-source models, surpassing UME-R1-7B by 4.1 points and VLM2Vec-7B by 16.3 points.RIME-7B scores 73.4 on image tasks, 49.4 on video tasks, and 75.6 on visual-document tasks; it matches Think-Then-Embed without an additional Reasoner module and exceeds CAFe by 8.0 points.
- MRMR: RIME-7B achieves the best MRMR comprehensive score of 50.2, exceeding UME-R1-7B by 2.2 points, Ops-MM-Embed-7B by 2.1 points, and GME-7B by 14.0 points.MRMR evaluates 11 expert-level reasoning-intensive subtasks across Knowledge, Theorem, and Contradiction categories using nDCG@10 as the main metric.
- Ablation studies: Progressively adding Rewrite-Driven Joint SFT, CMA, and Refine-RL yields measurable improvements, while CMA raises pure discriminative retrieval from 55.8 to 56.3 and enables competitive hybrid modes.Gen.-Gen. averages 58.1 across benchmarks; Disc.-Gen. reaches 57.0 with zero query tokens, and Gen.-Disc. reaches 57.1 with 232 query tokens.
- Rewrite versus CoT: Rewrite outperforms CoT by 4.4 points among non-RL variants, while removing forced summarized answers improves CoT to 60.5 and lets Rewrite exceed its answer-forced variant by 2.3 points.These results support retrieval-friendly rewriting over chain-of-thought reasoning and show that forced summarized answers can introduce semantic ambiguity.
- UVRB: RIME-7B achieves the best overall UVRB performance, significantly outperforming prior state-of-the-art models under comparable training-data settings across 16 video-retrieval subtasks.UVRB primarily uses Recall@1 and spans diverse task types, application domains, and textual, composed, and visual retrieval settings.
5 Conclusion · Appendix · A Data Construction Details
RIME replaces chain-of-thought reasoning with retrieval-friendly rewriting for efficient, generalizable generative multimodal embeddings, achieving state-of-the-art performance across three benchmarks. The appendix emphasizes that high-quality rewrite annotations for queries and targets are critical to retrieval performance.
- 5 Conclusion: RIME directly replaces traditional chain-of-thought reasoning with a retrieval-friendly rewriting paradigm.The framework is designed for generative multimodal embeddings.
- 5 Conclusion: RIME produces efficient yet highly generalizable generative multimodal embeddings.
- 5 Conclusion: RIME achieves state-of-the-art performance across the MMEB-V2, MRMR, and UVRB benchmarks.
- 5 Conclusion: RIME simultaneously reduces inference overhead significantly while maintaining broad benchmark performance.
- 5 Conclusion: The conclusion acknowledges that the proposed framework still exhibits certain limitations.The supplied passage does not specify those limitations further.
- A Data Construction Details: The data construction pipeline is critical for training RIME because rewrite annotations for both queries and targets directly determine retrieval performance.The appendix presents a comprehensive overview of this pipeline.
A.1 Training Data Composition · A.2 Modality-Aware Rewrite Annotation
RIME trains rewriting on approximately 1.5 million multimodal samples and separately applies Refine-RL to 15K non-overlapping samples. Modality-aware prompts produce retrieval-friendly rewrites tailored to text, images, videos, and visual documents, with teacher-generated annotations standardized for embedding.
- A.1 Training Data Composition: Approximately 1.5 million VLM2VEC-v2 [35] samples support Rewrite SFT across images, videos, and visual documents.The image datasets cover matching, visual question answering, classification, and composed image retrieval; video data covers video QA, retrieval, and caption retrieval; document data covers document-as-image retrieval involving charts, tables, and papers.
- A.1 Training Data Composition: Image SFT data spans MSCOCO, ImageNet-1K, ChartQA, A-OKVQA, DocVQA, SUN397, Visual7W, N24News, VOC2007, HatefulMemes, VisualNews, InfographicsVQA, CIRR, VisDial, WebQA, NIGHTS, and OK-VQA.Together, these datasets cover image-text matching, visual question answering, image classification, and composed image retrieval scenarios.
- A.1 Training Data Composition: Video training uses LLaVA-Hound [48] subsets for video question answering, video-text retrieval, and caption retrieval.These subsets provide coverage of video understanding scenarios.
- A.1 Training Data Composition: Visual-document training uses ViDoRe [9] and VisRAG [46] for document-as-image retrieval involving complex charts, tables, and papers.This extends the training composition beyond ordinary image and video understanding to visually rendered documents.
- A.1 Training Data Composition: A separate Refine-RL dataset contains approximately 15K uniformly sampled image, video, and visual-document examples with no overlap with SFT data.The held-out dataset provides the GRPO exploration space for learning rewrites that maximize process reward and supports evaluation of generalization.
- A.2 Modality-Aware Rewrite Annotation: Modality-aware prompts guide retrieval-friendly descriptions by input modality and task type [34].For text, rewriting identifies text type, resolves ambiguity, extracts logical relationships, and summarizes concisely; for images, it reconstructs subjects, actions, and scenes.
- A.2 Modality-Aware Rewrite Annotation: Video rewrites capture scene transitions, temporal progression, key frames, event nodes, and chronological order, while video VQA rewrites handle temporal dynamics, multiple events, and dialogue or text across frames.The video strategy extends the image VQA approach to explicitly represent temporal structure and multi-event content.
- A.2 Modality-Aware Rewrite Annotation: Qwen2-VL-72B-Instruct [39] generates rewrite annotations asynchronously, which are post-processed into standardized embedding-ready outputs.Failed generations are filtered, content is wrapped in <think>...</think> tags, and a standard answer format ends with “All can be embedded into <gen_emb>.”
B Additional Experimental Results
This section provides additional experiments that deepen analysis of RIME across multiple evaluation dimensions, including MR2-Bench performance and the effect of the Rewrite SFT loss hyperparameter.
- MR2-Bench Results: Table 8 evaluates RIME on MR2-Bench using nDCG@10 across 12 subject-specific subtasks and reports their average score.The subtasks span Biology, Cooking, Gardening, Physics, Chemistry, Earth Science, Economics, Mathematics, Nature, Spatial, Puzzle, and Analogy.
- Hyperparameter Ablation: Table 9 studies the effect of λ in the joint loss for Rewrite SFT using average scores on MMEB-V2.The experiment is an ablation of the Rewrite SFT loss hyperparameter.
B.1 Ablation on Hyperparameter 𝜆
The λ ablation on MMEB-V2 shows that increasing emphasis on rewrite generation improves overall retrieval performance, reaching an optimal score of 64.1.
- B.1 Ablation on Hyperparameter λ: Overall performance rises from 59.7 at λ=0.1 to 61.1 at λ=0.5, with an optimal score of 64.1 as λ increases.The joint-loss weight λ controls the trade-off between rewrite generation quality and retrieval performance; insufficient rewrite emphasis yields suboptimal semantic understanding.
B.2 Results on MR2-Bench
On MR2-Bench, RIME-7B achieves a comprehensive score of 28.6, outperforming UME-R1 and approaching Seed-1.6-embed. Its strongest gains appear on reasoning-intensive sub-tasks, while Puzzle remains challenging for all models.
- B.2 Results on MR2-Bench: 28.6: RIME-7B achieves a comprehensive MR2-Bench score of 28.6, outperforming UME-R1 (27.5) and approaching Seed-1.6-embed (30.7).MR2-Bench evaluates multimodal reasoning-intensive retrieval across 12 sub-tasks in Multimodal Knowledge Retrieval, Visual Illustration, and Visual Relation.
- B.2 Results on MR2-Bench: 64.2 vs. 55.7: RIME improves over UME-R1 by 8.5 points on Economics, with additional gains of 2.7 points on Spatial reasoning (40.4 vs. 37.7) and 2.9 points on Analogy (15.1 vs. 12.2).These are notable improvements on reasoning-intensive sub-tasks.
- B.2 Results on MR2-Bench: Near-zero scores: Puzzle remains challenging for all models, with scores near zero.Puzzle is one of MR2-Bench’s Visual Relation sub-tasks.
B.3 Detailed Results on MMEB-V2/UVRB
This section reports comprehensive per-task evaluations on MMEB-v2 [17] [35], covering 78 subtasks across image, video, and visual document modalities. The detailed results are provided to support reproducibility, fine-grained analysis, and future comparisons.
C Examples of Data Construction
This section illustrates how modality-aware rewriting converts text, images, and visual question-answering inputs into retrieval-friendly representations. The examples use structured analysis, multiple rewrites, and summarized embeddings across varied input modalities and tasks.
- Examples of Data Construction: Modality-aware rewriting transforms multimodal inputs into retrieval-friendly representations, with examples spanning text, images, videos, and combinations thereof.The section presents illustrative construction examples across different input modalities and references Figures 6–10.
- Text-to-Rewrite: Text rewriting analyzes the input type and produces up to three meaning-preserving transformations before summarizing the input and rewrites for embedding.The text-analysis prompt distinguishes single words, conventional sentences or paragraphs, and disciplinary professional text, while the example rewrites a caption in three ways.
- Image-to-Rewrite: Image-to-rewrite processing first classifies images by information density, then describes visual content and answers questions or summarizes the scene for embedding.The VQA prompt distinguishes high-information from low-information images and structures image assessment, content analysis, and question answering.
- Task Coverage: The examples cover diverse retrieval and understanding tasks, including caption retrieval, computer counting, infographic question answering, image classification, and Llavahound caption retrieval.Illustrative cases include a cat-caption image query, an image containing three computers, a recovery-ratio question, toyshop classification, and Llavahound_Caption_Retrieval.