PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings

Peixi Wu, Ke Mei, Feipeng Ma, Bosong Chai, Zhibin Lan, Chenxi Zhao, Shannan Yan, Jie Chen, Zhangchi Hu, Yansong Peng, Bo Lin, Junjie Zhou, Dacheng Yin, Tianyi Wang, Fengyun Rao, Jing Lyu, Hebei Li, Xiaoyan Sun

arXiv:2604.22280v3cs.CV

TL;DR

Answer-oriented chain-of-thought reasoning has limited suitability for general retrieval, where queries often lack unique answers or summaries. RIME jointly trains retrieval-friendly rewriting and multimodal embeddings, using cross-mode alignment and discriminative-anchored reinforcement learning. It outperforms prior generative embedding models across MMEB-V2 [17], UVRB [13], and MRMR [49], while reducing thinking by approximately 50% versus prior methods [21] [30].

  • Problem

    Answer-oriented chain-of-thought reasoning has limited suitability for general retrieval because queries often lack unique answers and retrieval targets need no summarization.

    Source: §3.2 · p. 4

  • Method

    RIME jointly trains rewriting and multimodal embeddings, aligns generative and discriminative spaces, and refines rewrites using discriminative embeddings as reinforcement-learning anchors.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    RIME outperforms prior generative embedding models across MMEB-V2 [17], UVRB [13], and MRMR [49], while reducing thinking by approximately 50% versus prior methods [21] [30].

    Source: §1 · p. 2

  • Takeaways & Limitations

    Hybrid retrieval offers a deployment option pairing offline generative corpus encoding with discriminative queries requiring no generated query tokens.

    Source: §4.4.3 · p. 8

    The study adopts a baseline that shares parameters between reasoning and embedding models rather than using an external reasoner.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have emerged as a promising foundation for universal multimodal embeddings. Recent studies have shown that reasoning-driven generative multimodal embeddings can outperform discriminative embeddings on several embedding tasks. However, Chain-of-Thought (CoT) reasoning tends to generate redundant thinking steps and introduce semantic ambiguity in the summarized answers in broader retrieval scenarios. To address this limitation, we propose Rewrite-driven Multimodal Embedding (RIME), a unified framework that jointly optimizes generation and embedding through a retrieval-friendly rewrite. Meanwhile, we present the Cross-Mode Alignment (CMA) to bridge the generative and discriminative embedding spaces, enabling flexible mutual retrieval to trade off efficiency and accuracy. Based on this, we also introduce Refine Reinforcement Learning (Refine-RL) that treats discriminative embeddings as stable semantic anchors to guide the rewrite optimization. Extensive experiments on MMEB-V2, MRMR and UVRB demonstrate that RIME substantially outperforms prior generative embedding models while significantly reducing the length of thinking. Code is available at https://github.com/PeppaWu/RIME.

1 Introduction

RIME replaces chain-of-thought reasoning with retrieval-friendly rewrites, aligns discriminative and generative embeddings, and refines rewriting through reinforcement learning. Experiments report stronger benchmark performance with approximately 50% less thinking than prior methods [21] [30].

1 Introduction

  • CoT-based embeddings face redundant reasoning, target ambiguity, and costly inference that limit their suitability for general multimodal retrieval.Studies [5] [7] [45] report that joint contrastive and autoregressive training can weaken discrimination. Think-then-Embed [6] [7] [15] adds external-reasoner parameter redundancy, while coupled reinforcement-learning approaches [21] [30] [53] retain costly long CoT.Source: §1 · p. 2
  • Rewrite-Driven Joint SFT replaces CoT with retrieval-friendly rewrites, enabling one model to produce discriminative and rewrite-based generative embeddings without summarized answers.It recaptions images, videos, and documents and expands text explanations, supporting multi-granularity, complex logical, and mixed-modality retrieval.Source: §1 · p. 2
  • Cross-mode alignment uses mutual contrastive learning to align discriminative and generative embeddings for identical queries while separating negative samples, enabling cross-retrieval.Cross-retrieval between the two embedding spaces provides flexibility intended to mitigate rewriting-related inference overhead.Source: §1 · p. 2
  • Refine-RL rewards rewrites when generative embeddings’ similarity gap exceeds discriminative embeddings’, using the latter as stable semantic anchors.Building on the aligned spaces, this process reward guides rewriting toward richer semantics and stronger discriminative capability.Source: §1 · p. 2
  • Approximately 50% less thinking than prior methods [21] [30] accompanies RIME’s superior performance over existing generative embeddings on MMEB-V2 [17], UVRB [13], and MRMR [49].The thinking reduction is reported alongside significantly stronger performance on MMEB-V2; the contribution summary additionally reports improvements on UVRB and MRMR.Source: §1 · p. 2

2 Related Works

Prior multimodal embedding approaches face limitations in complex-input modeling, compatibility between generation and alignment, and reasoning overhead. RIME instead uses rewriting to interpret and expand information across modalities.

2.1 Multimodal Representation Learning

  • Early contrastive dual encoders [36] [37] [47] align modalities, but fixed-structure models struggle to represent videos, multi-page documents, and mixed-modal content.CLIP [37] learns shared image-text representations from large-scale web data for cross-modal retrieval. BLIP [25] and ALIGN [14] combine vision encoders with language models to strengthen semantic alignment.Source: §2.1 · pp. 2–3
  • MLLM-based embeddings use instruction tuning and contrastive learning to align heterogeneous inputs, with instruction-following capabilities supporting complex multimodal queries.Examples include VLM2Vec [17] [35], GME [50], MM-Embed [27], and LLaVE [20], which use unified LLM backbones. Subsequent studies [11] [15] [19] [33] [43] improve scalability and performance.Source: §2.1 · p. 3

2.2 Multimodal Generative Embedding

  • Generative embedding models [15] [21] [29] [32] typically generate stepwise rationales before producing embeddings, explicitly modeling input semantics to strengthen representations.Chain-of-Thought (CoT) [12] [18] [41] [42] is widely used to improve complex-query understanding through intermediate reasoning.Source: §2.2 · p. 3
  • Joint generation and alignment can favor language modeling over embedding discriminability [8] [26] [45], while CoT methods add computational overhead and redundant thinking.Think-then-Embed (TTE) [6] [7] separates reasoning generation from embedding encoding. Other work [21] uses reinforcement learning and rewards to balance the objectives within one framework.Source: §2.2 · p. 3

2.3 Retrieval-oriented Query Rewrite

  • RIME uses rewriting for interpretation and expansion across modalities, unlike CoT-dependent generative embeddings [7] [15] [21], whose stepwise derivation and summarization introduce task biases.These biases limit general retrieval.Source: §2.3 · p. 3
  • Prior query rewriting evolved toward LLM-driven expansion and MLLM semantic reasoning, but complex-retrieval approaches often depend on external LLMs [6] [49] [51] or remain modality-specific [28] [38].Earlier techniques include pseudo-relevance feedback and keyword reweighting; newer approaches generate rationales, pseudo-documents, decompositions, inferred constraints, and diverse queries. Modality-specific methods focus on images while overlooking modalities such as text.Source: §2.3 · p. 3

3 Method

RIME replaces CoT-centered embedding generation with jointly trained retrieval-oriented rewrites and contrastive learning. Cross-mode alignment supports mutual retrieval between generative and discriminative embeddings, while Refine-RL uses discriminative embeddings as stable anchors for rewrite optimization.

3.1 Preliminary

  • RIME adopts a generative embedding baseline that shares reasoning and embedding model parameters, avoiding external reasoning models’ parameter redundancy and optimization issues.Discriminative models extract normalized last-input-token representations. Generative models first produce reasoning and a summary, then extract a designated token’s hidden state.Source: §3.1 · pp. 3–4

3.2 Rewrite-Driven Joint SFT

  • CoT can distort retrieval semantics through mismatched summarization granularity and introduce redundant or erroneous reasoning because its deterministic-answer objective differs from retrieval needs.Retrieval queries need not have unique answers, and targets often require no summarization. Reasoning-intensive retrieval can still rely on CoT, but these scenarios are highly task-specific.Source: §3.2 · p. 4
  • Rewrite-driven joint SFT reconstructs images, videos, and text through structured descriptions, rewriting both queries and targets while jointly optimizing generation and contrastive learning.Training combines autoregressive rewrite loss with cross-mode InfoNCE loss. A hyperparameter balances rewrite quality against retrieval performance.Source: §3.2 · pp. 4–5

3.3 Cross-Mode Alignment

  • CMA aligns generative and discriminative embeddings for mutual retrieval, enabling low-latency direct encoding or generative rewriting when deeper semantic reasoning is needed.This addresses the query latency introduced by autoregressive rewriting. Cross-mode InfoNCE aligns the spaces, while intra-mode alignment limits generative semantic drift.Source: §3.3 · p. 5

3.4 Refine Reinforcement Learning

  • Refine-RL rewards rewrites whose positive–negative similarity gaps exceed discriminative baselines, alongside rewards for format adherence and larger generative similarity gaps.GRPO samples K rewrites per query and optimizes using group-normalized rewards. Its objective includes clipping and KL regularization against a fixed reference policy.Source: §3.4 · p. 5

4 Experiments

RIME-7B achieves leading overall results on MMEB-V2, MRMR, and UVRB. Ablations examine the contributions of training components, rewriting versus chain-of-thought reasoning, and cross-mode embedding alignment.

4.1 Implementation

  • Approximately 1.5 million samples support Rewrite SFT, followed by approximately 15K uniformly sampled three-modality examples for Refine-RL.Experiments use Qwen2-VL-2B-Instruct and Qwen2-VL-7B-Instruct on 32 A800 GPUs; each training stage runs one epoch. Unless specified otherwise, ablations use the 2B backbone with generative embeddings.Source: §4.1 · p. 5, §4.4 · p. 7

4.3.1 MMEB-V2 Benchmark.

  • 68.6 on MMEB-V2 places RIME-7B ahead of UME-R1-7B by 4.1 points and VLM2Vec-7B by 16.3 points.Image, video, and visual-document scores reach 73.4, 49.4, and 75.6, exceeding UME-R1 by 2.1, 1.9, and 8.5 points, respectively. RIME-7B matches Think-Then-Embed’s 68.6 without an additional Reasoner module and exceeds CAFe by 8.0 points.Source: Table 1 · p. 6, §4.3.1 · p. 6

4.2 Datasets and Metrics.

  • MMEB-V2, MRMR, and UVRB evaluate general multimodal retrieval, expert reasoning-intensive retrieval, and cross-domain video retrieval, respectively.MMEB-V2 uses Hit@1 across 36 image and 18 video sub-datasets, and NDCG@5 for visual documents. MRMR uses nDCG@10 except Negation’s Hit@1; UVRB primarily uses Recall@1 across 16 subtasks, with generative embeddings evaluated by default.Source: §4.2 · p. 6, Table 2 · p. 7

4.3.2 MRMR Benchmark.

  • 50.2 gives RIME-7B the best MRMR comprehensive score, exceeding UME-R1-7B, Ops-MM-Embed-7B, and GME-7B by 2.2, 2.1, and 14.0 points, respectively.Across 11 expert-level subtasks, RIME-7B outperforms all baselines in theorem reasoning and shows knowledge-task strengths in Medicine, Science, and Humanities. The authors attribute these gains to activating internal knowledge rather than deriving answers through CoT reasoning.Source: §4.3.2 · pp. 6–7, Table 2 · p. 7

4.4.1 Ablation on Core Components.

  • 1.9 points come from rewriting alone; progressively adding CMA and Refine-RL produces the strongest core-component configuration, scoring 64.1 on MMEB-V2.Joint generation-and-embedding training improves over purely discriminative training, while CMA aligns embedding spaces for mutual retrieval and adds a modest performance gain. Refine-RL further improves rewriting coherence and effectiveness.Source: Table 3 · p. 7, §4.4.1 · p. 7

4.4.2 Ablation on Rewrite vs. Chain-of-Thought.

  • 4.4 points separate Rewrite from CoT on MMEB-V2, making Rewrite the strongest non-RL variant while using only 212 average tokens.Removing forced summarized answers raises CoT to 60.5 (+1.4 points), while Rewrite exceeds Rewrite (w/ ans.) by 2.3 points, supporting the finding that summarized answers introduce retrieval noise. After Refine-RL, average token usage falls approximately 52% relative to UME-R1.Source: Table 4 · p. 7, §4.4.2 · p. 8

4.3.3 UVRB Benchmark.

  • 55.6 gives RIME-7B the best UVRB comprehensive score, surpassing Unite-7B by 1.8 points and GME-7B by 2.6 under comparable training-data settings.Coarse-grained retrieval reaches 56.2, exceeding Unite-7B by 2.1 points, while temporal-sensitive retrieval reaches a baseline-leading 42.7. Long-context performance exceeds all baselines, including GVE despite its additional training data.Source: §4.3.3 · p. 7, Table 5 · p. 8

4.4.3 Ablation on Two Embedding Modes.

  • 58.1 is the best cross-benchmark average for Gen.-Gen.; Disc.-Gen. achieves 57.0 with zero generated query tokens.With CMA, pure discriminative performance rises from 55.8 to 56.3; Gen.-Disc. scores 57.1 with 232 query tokens. Disc.-Gen. combines offline generative corpus encoding with discriminative queries, avoiding query-token generation.Source: Table 6 · p. 8, §4.4.3 · p. 8

5 Conclusion · Appendix · A Data Construction Details

RIME replaces chain-of-thought with retrieval-friendly rewriting, achieving state-of-the-art performance on MMEB-V2, MRMR, and UVRB while reducing inference overhead. Considerable remaining latency motivates future work on implicit latent reasoning.

5 Conclusion

  • RIME achieves state-of-the-art performance across MMEB-V2, MRMR, and UVRB.The conclusion reports this comparison without numerical benchmark results.Source: §5 · p. 8
  • RIME substitutes traditional chain-of-thought reasoning with retrieval-friendly rewriting to produce efficient, generalizable generative multimodal embeddings.Rewriting partially alleviates redundant reasoning and unnecessary summarized answers.Source: §5 · p. 8
  • RIME significantly reduces inference overhead alongside its benchmark improvements.The conclusion does not quantify this reduction.Source: §5 · p. 8
  • Considerable inference latency remains despite rewriting, limiting the framework’s practicality for large-scale retrieval.The authors describe this latency as an inherent limitation of generative embedding models.Source: §5 · p. 8
  • Future work aims to convert explicit reasoning steps into an implicit latent form to alleviate latency and support practical retrieval deployment.Latent thinking is proposed as a promising direction, not demonstrated as a solution here.Source: §5 · p. 8

A Data Construction Details

  • High-quality rewrite annotations for both queries and targets directly determine retrieval performance, making data construction critical for training RIME.The appendix introduces an overview of the training-data construction pipeline.Source: A Data Construction Details · p. 11

A.1 Training Data Composition · A.2 Modality-Aware Rewrite Annotation

Rewrite SFT uses approximately 1.5 million multimodal samples, while Refine-RL uses approximately 15K separate, nonoverlapping samples. Modality- and task-aware prompts guide retrieval-friendly rewrites, with teacher-generated annotations standardized through post-processing.

A.1 Training Data Composition

  • Approximately 1.5 million samples from VLM2VEC-v2 [35] train Rewrite SFT across images, videos, and visual documents.Image datasets cover image-text matching, visual question answering, image classification, and composed image retrieval.Source: §A.1 · p. 11
  • LLaVA-Hound [48] supplies video QA, video retrieval, and caption retrieval, while ViDoRe [9] and VisRAG [46] support document-as-image retrieval.Visual documents include complex charts, tables, and paper figures requiring fine-grained text recognition and layout understanding.Source: §A.1 · p. 11
  • Approximately 15K Refine-RL samples are uniformly drawn from images, videos, and visual documents, with no overlap with SFT training data.This separate dataset supports generalization evaluation and provides the exploration space for GRPO optimization of rewrites against the process reward.Source: §A.1 · p. 11

A.2 Modality-Aware Rewrite Annotation

  • Modality- and task-specific prompts [34] guide retrieval-friendly rewrites, with Qwen2-VL-72B-Instruct [39] generating annotations through asynchronous batch processing.Text rewriting identifies text type, resolves ambiguity, extracts logical relationships, and summarizes concisely. Post-processing filters failed generations and standardizes annotations as <think>{rewrite}</think>All can be embedded into <gen_emb>.Source: §A.2 · p. 11
  • Image rewrites objectively describe subjects, backgrounds, colors, and object relationships, including how accompanying text complements visual content.Image VQA uses structured reasoning to assess information density, perform OCR when necessary, and provide direct answers with confidence levels.Source: §A.2 · p. 11
  • Video rewrites describe subjects, backgrounds, actions, scene transitions, and temporal progression, prioritizing key frames and important events in chronological order.Video VQA extends the image approach to temporal dynamics, multiple events, and dialogue or text recognition across frames.Source: §A.2 · p. 11

B Additional Experimental Results

The additional experiments cover MR2-Bench evaluation and an MMEB-V2 ablation of the Rewrite SFT joint-loss hyperparameter λ.

MR2-Bench Evaluation

  • MR2-Bench evaluation reports nDCG@10 for all subtasks, with “All” denoting the average across 12 datasets.The supplied caption contains no numerical scores or model comparisons.Source: Table 8 · p. 12
  • MR2-Bench covers biology, cooking, gardening, physics, chemistry, earth science, economics, mathematics, nature, spatial, puzzle, and analogy subtasks.This scope spans scientific, everyday, and reasoning-oriented tasks.Source: Table 8 · p. 12

Rewrite SFT Ablation

  • The Rewrite SFT ablation examines the joint-loss hyperparameter λ using average scores on MMEB-V2.The supplied caption does not identify a preferred λ or report its effect on performance.Source: Table 9 · p. 12

B.1 Ablation on Hyperparameter 𝜆

On MMEB-V2, λ balances rewrite generation and retrieval performance, with equal weighting of rewrite and contrastive objectives achieving the best overall score.

B.1 Ablation on Hyperparameter 𝜆

  • 64.1 is the best MMEB-V2 overall score, achieved at λ = 1.0 with equal weighting of rewrite and contrastive objectives.The joint loss is LJoint = λ·LRewrite + LCM_InfoNCE, where λ controls the trade-off between rewrite generation quality and retrieval performance.Source: §B.1 · p. 12
  • 59.7 at λ = 0.1 improves to 61.1 at λ = 0.5, indicating that insufficient emphasis on rewrite generation limits semantic understanding.Both values are MMEB-V2 overall scores.Source: §B.1 · p. 12
  • 62.3 is the MMEB-V2 overall score at λ = 1.5, below the optimum, suggesting that overemphasizing generation may weaken embeddings’ discriminative capability.Source: §B.1 · p. 12

B.2 Results on MR2-Bench

RIME-7B outperforms UME-R1 overall on MR2-Bench [51], with gains in Economics, Spatial reasoning, and Analogy. Puzzle scores remain near zero across models.

B.2 Results on MR2-Bench

  • 12 subtasks across Multimodal Knowledge Retrieval, Visual Illustration, and Visual Relation make up MR2-Bench [51], which evaluates multimodal reasoning-intensive retrieval.Knowledge Retrieval covers Biology, Cooking, Gardening, Physics, Chemistry, and Earth Science; Visual Illustration covers Economics, Mathematics, and Nature. Visual Relation covers Spatial, Puzzle, and Analogy.Source: §B.2 · p. 12
  • 28.6 is RIME-7B’s comprehensive score, exceeding UME-R1 (27.5) and approaching closed-source Seed-1.6-embed (30.7).The authors attribute these results to rewrite-driven activation of internal knowledge without explicit chain-of-thought reasoning [21] [30] [53].Source: §B.2 · p. 12
  • 8.5-, 2.7-, and 2.9-point improvements over UME-R1 occur in Economics, Spatial reasoning, and Analogy, respectively.RIME versus UME-R1 scores are Economics: 64.2 versus 55.7; Spatial reasoning: 40.4 versus 37.7; and Analogy: 15.1 versus 12.2.Source: §B.2 · p. 12
  • Near-zero Puzzle scores across all models indicate persistent difficulty with abstract visual reasoning requiring complex logical deduction.Source: §B.2 · p. 12

B.3 Detailed Results on MMEB-V2/UVRB

RIME-7B achieves the best overall performance on UVRB [13]. Detailed per-task results on MMEB-v2 [17] [35] and UVRB support fair comparison and reproducible research.

B.3 Detailed Results on MMEB-V2/UVRB

  • 55.6 is RIME-7B’s average score on UVRB [13], achieving the best overall performance.Detailed per-task results are provided in Tables 12 and 13.Source: §B.3 · p. 12
  • 78 sub-tasks on MMEB-v2 [17] [35] cover image, video, and visual document tasks, with detailed results in Tables 10 and 11.These fine-grained results support further analysis and direct comparison in subsequent studies.Source: §B.3 · p. 12
  • 16 sub-tasks in UVRB [13] systematically evaluate video retrieval across three dimensions: Tasks, Domains, and Sub-domains.Source: §B.3 · p. 12

C Examples of Data Construction

The examples illustrate how modality-aware rewriting transforms multimodal inputs into retrieval-friendly representations. Text analysis and visual question answering use structured rewrites followed by a shared embedding summary.

C Examples of Data Construction

  • Modality-aware data construction transforms multimodal inputs into retrieval-friendly representations through a shared rewriting-and-embedding template.The template places the rewriting process inside <think> tags, then uses <gen_emb> to summarize both that process and the original inputs.Source: C Examples of Data Construction · pp. 12, 18
  • Text rewriting analyzes ambiguities in words or phrases, provides up to three meaning-preserving sentence rewrites, and extracts specialized terminology from professional text.Sentence rewrites include brief explanations; professional-text analysis also identifies the likely discipline and deeper implications.Source: C Examples of Data Construction · p. 15
  • Visual question answering assesses image complexity, adapts content analysis accordingly, and produces direct answers with confidence-labeled alternatives when uncertain.Low-information images receive concise visual descriptions; high-information images require key elements, relationships or spatial structure, and accurate text recognition.Source: C Examples of Data Construction · p. 16
  • Worked examples connect caption paraphrases with image descriptions and visual computer counting with semantic analysis of the target answer “Three.”The caption example rewrites a cat-and-carrots meme while the image description records its scene and text overlay. The counting example identifies three computers with high confidence, while the target rewrite discusses cardinal, ordinal, and collective interpretations.Source: C Examples of Data Construction · pp. 18–19

Related papers

Loading 2604.22280v3…