PaperlayerView exact revision on arXiv

Source-linked AI summary

PLUME: Latent Reasoning Based Universal Multimodal Embedding

Chenwei He, Xiangzhao Hao, Tianyu Yang, Yuxiang Ma, Yuheng Jia, Lingxiang Wu, Chaoyang Zhao, Haiyun Guo, Jinqiao Wang

arXiv:2604.02073v3cs.CV

TL;DR

Universal multimodal embedding needs intermediate computation to handle complex query intent without sacrificing retrieval efficiency. PLUME replaces explicit chain-of-thought with adaptive latent rollouts and outperforms UME-R1 while using fewer than ten latent steps and delivering over 30× faster inference.

  • Problem

    Universal multimodal embedding typically forms embeddings in a single pass, leaving limited intermediate computation for complex query intent while preserving retrieval efficiency.

  • Method

    PLUME replaces explicit rationales with compact latent rollouts guided by semantic anchors and progressively transfers verbalized reasoning into hidden-state computation.

  • Results

    Over 30× faster inference accompanies PLUME’s stronger performance than UME-R1 on the 78-task MMEB-v2 benchmark, using fewer than ten latent steps instead of hundreds of tokens.

  • Takeaways & Limitations

    PLUME is particularly effective for retrieval tasks with dense, structurally complex evidence, including video and visual document retrieval.

  • Takeaways & Limitations

    Formal interpretability guarantees for continuous latent trajectories remain an open problem.

Abstract

from arXiv · show

Universal multimodal embedding (UME) maps heterogeneous inputs into a shared retrieval space with a single model. Recent approaches improve UME by generating explicit chain-of-thought (CoT) rationales before extracting embeddings, enabling multimodal large language models to better infer complex query intent. However, explicit CoT incurs substantial inference overhead and can compress rich multimodal evidence into a narrow textual bottleneck. We propose PLUME, a latent reasoning framework that advances UME by replacing verbalized CoT with a short autoregressive rollout of continuous latent states. To support diverse multimodal queries, PLUME further introduces a semantic-anchor-guided transition adapter that steers latent rollout along different reasoning trajectories under the same fixed computation budget. To stabilize training, PLUME adopts a progressive explicit-to-latent curriculum that uses verbalized reasoning only as a temporary training scaffold and gradually transfers this behavior into hidden-state computation, eliminating explicit CoT at inference. On the 78-task MMEB-v2 benchmark, PLUME outperforms strong explicit-CoT UME baselines while reducing reasoning from hundreds of generated tokens to fewer than 10 latent steps, delivering over 30x faster inference. PLUME is especially well suited to retrieval settings where relevant evidence is dense, structurally complex, and difficult to organize through verbalized intermediate rationales, such as video and visual document retrieval. These results show that structured latent computation can preserve the benefits of intermediate reasoning without the overhead of explicit rationale generation, providing a stronger and more efficient paradigm for practical retrieval systems.

1. Introduction

PLUME replaces costly verbalized chain-of-thought with compact latent reasoning for universal multimodal embedding, addressing complex query interpretation while preserving retrieval efficiency. It outperforms explicit-CoT baselines on MMEB-v2 with fewer than ten latent steps and over 30x faster inference.

  • 1. Introduction: Existing single-pass UME methods collapse complex query interpretation, evidence integration, and representation formation into one forward pass, limiting deliberate computation for complex intent.Such queries may require compositional spatial understanding, knowledge-intensive visual inference, or aggregation of temporally and structurally dispersed evidence.
  • 1. Introduction: The paper argues that retrieval needs intermediate computation rather than necessarily verbalized intermediate text, motivating latent rollouts that preserve sequential reasoning structure without long-form generation.This perspective addresses the latency and token-cost drawbacks of explicit CoT while retaining room for deliberate computation.
  • 1. Introduction: PLUME internalizes multi-step reasoning as a short continuous latent process, replacing explicit rationale generation while preserving intermediate computation for heterogeneous multimodal inputs.The framework targets text, images, videos, and visual documents within a shared embedding space.
  • 1. Introduction: PLUME uses a progressive explicit-to-latent curriculum that shifts from verbalized intermediate reasoning to hidden-state computation until explicit CoT is unnecessary at inference.This training strategy uses explicit rationales as a temporary scaffold rather than an inference-time requirement.
  • 1. Introduction: PLUME outperforms strong explicit-CoT UME baselines on MMEB-v2 while reducing reasoning to fewer than ten latent steps and delivering over 30x faster inference.It is especially effective for video and visual document retrieval, where evidence is dense, structurally complex, and difficult to organize verbally.

2. Related Work

Universal multimodal embedding has progressed from contrastive dual encoders toward unified, broader-coverage retrieval systems, but most methods still use single-pass embedding extraction. Recent work adds explicit or latent reasoning to improve complex multimodal retrieval while addressing inference efficiency.

  • Universal multimodal embedding: UME methods evolved from contrastive image-text encoders such as CLIP [34], ALIGN [17], SigLIP, and BLIP-2 [27] toward unified multi-task frameworks including UniIR and MagicLens [52].These early dual-encoder approaches are less effective on complex multimodal compositions.
  • Universal multimodal embedding: VLM2Vec-V2 [31], GME [54], UniME [10], LamRA [30], LLaVE [23], MoCa [3], and DUME improve retrieval quality and modality coverage, alongside work on multi-vector representations [6], data synthesis [56], visual document retrieval [49], and reinforcement-learning alignment [48].Most methods derive embeddings from a single forward pass without modeling intermediate reasoning, limiting their ability to organize complex multimodal intent.
  • Explicit reasoning for embeddings: CoT prompting [41], multimodal CoT, preference-optimized reasoning, and reasoning-specialized models such as DeepSeek-R1 [11] motivate explicit reasoning for embedding extraction.Embedding-focused methods include Think-then-Embed (TTE) [5], UME-R1 [24], TRACE, and Embed-RL [18].
  • Latent reasoning: Latent-reasoning research moves computation beyond explicit CoT through pause tokens [8], internal thoughts, continuous hidden states [12] [36], and compressed thought traces [33].In retrieval, LaSER internalizes explicit reasoning into latent space for dense text retrieval.

3. Method

PLUME forms universal multimodal embeddings through a short autoregressive rollout of continuous latent states, replacing explicit chain-of-thought decoding while preserving sequential reasoning structure. A semantic-anchor-guided routed adapter and progressive explicit-to-latent curriculum make this computation input-adaptive and eliminate explicit CoT at inference.

  • Training and embedding extraction: The backbone is fully fine-tuned, while only a lightweight routed adapter and anchor-conditioned router are added; normalized backbone hidden states directly provide retrieval embeddings without separate heads.PLUME’s retrieval objective remains standard UME training with bidirectional InfoNCE [32] and an auxiliary causal language-modeling loss for the decoded query and positive-target suffixes.
  • Latent reasoning rollout: PLUME replaces explicit CoT decoding with a short autoregressive rollout of continuous hidden states, then extracts the retrieval embedding from the hidden state at <gen>.The rollout reuses the multimodal prefix’s KV cache, attends causally to preceding latent states, and can use as few as K latent steps instead of hundreds of explicit reasoning tokens.
  • Latent reasoning rollout: The rollout preserves explicit generation’s sequential dependency structure because latent vectors occupy the same causal positions with identical attention-mask, positional-encoding, and KV-cache mechanics.Each step refines the previous latent state before feeding it through the full transformer backbone, while retaining access to the full multimodal prefix and prior latent states.
  • Semantic-anchor-guided transition adapter: A semantic-anchor-guided routed adapter uses shared and specialized experts to adapt each latent transition, enabling different multimodal inputs to follow distinct reasoning paths under one rollout budget.The fixed anchor stabilizes routing, while step embeddings distinguish early and late transitions; the adapter is lightweight and leaves the backbone unchanged, following mixture-of-experts designs [7].
  • Progressive explicit-to-latent curriculum: PLUME uses a progressive explicit-to-latent curriculum that allocates more training to the fully latent stage, transferring verbalized reasoning into hidden-space computation without explicit CoT at inference.Earlier curriculum stages stabilize the transfer, while the final stage emphasizes compact latent rollout.

4. Experiments

PLUME improves universal multimodal retrieval on MMEB-v2 through short latent rollouts, outperforming explicit-CoT and single-pass baselines while substantially reducing inference cost. Ablations show that progressive transfer, iterative latent computation, and routed transition design are important for accuracy and specialization.

  • Overall benchmark results: PLUME scores 61.6 overall, surpassing UME-R1 [24] by 1.5 points and VLM2Vec-V2 [31] by 3.6 points, with a +9.2 gain on Video over VLM2Vec-V2 [31].Under the same backbone and training data, PLUME uses 8 latent steps instead of hundreds of generated reasoning tokens.
  • Modality-group results: PLUME achieves 44.1 on Video and 67.5 on VisDoc versus UME-R1’s 42.2 and 63.9, while scoring 66.3 on Image versus 66.6.It also sets the best scores on Image Grounding (79.7) and VisDoc OOD (57.4).
  • Inference efficiency: PLUME reduces reasoning from 403 generated tokens to 8 latent steps and latency from 9023 ms to 298 ms, achieving a 30.3× speedup over UME-R1 [24].Compared with VLM2Vec-V2 [31], PLUME adds less than 150 ms while improving overall accuracy by 2.1 points.
  • Component ablation: Removing the progressive curriculum causes the largest ablation drop (−6.8 overall and −7.6 on Video), showing that abrupt explicit-to-latent transfer destabilizes training.Removing latent transition reduces accuracy by 2.8 overall, while replacing the MoE adapter with a shared MLP costs 2.4 points, especially on VisDoc (−3.1).
  • Latent-step analysis: At K = 6, PLUME surpasses UME-R1’s accuracy while running over 30x faster; increasing to K = 8 yields +1.2 accuracy, while latency grows from 232 ms to 300 ms.K = 8 provides the best absolute accuracy, whereas K = 6 offers a favorable accuracy–speed balance.
  • Routing analysis: The routed adapter develops task specialization: Expert 2 peaks at 75.2% on video classification, while Expert 0 peaks at 62.9% on image grounding.Formal interpretability guarantees for continuous latent trajectories remain an open problem.

5. Conclusion

PLUME replaces explicit chain-of-thought generation with a short hidden-space rollout for universal multimodal embedding, combining latent reasoning, anchor-guided routed adaptation, and progressive explicit-to-latent training. On MMEB-v2, it surpasses UME-R1 trained on the same data while reducing reasoning overhead.

  • PLUME replaces explicit chain-of-thought generation with a short hidden-space rollout for universal multimodal embedding.
  • Its design combines latent multi-step reasoning, anchor-guided routed adaptation, and a progressive explicit-to-latent curriculum.These components transfer reasoning into compact embeddings.
  • On the 78-task MMEB-v2 benchmark, PLUME surpasses UME-R1 trained on the same data while reducing reasoning from hundreds of tokens to fewer than the supplied passage specifies.

A. Curriculum Ablation Details

PLUME’s curriculum ablation shows that performance depends non-monotonically on curriculum granularity: four stages best balance retrieval accuracy, optimization stability, and curriculum efficiency. Abrupt explicit-to-latent transitions are weaker, while overly fragmented schedules may underutilize each stage.

  • Curriculum Ablation Details: Two stages yield the weakest result because abrupt transitions impose a large distribution shift, whereas excessively fragmented curricula can make stages too short to exploit fully.The observed effect is non-monotonic, so greater curriculum granularity is not uniformly better.
  • Curriculum Ablation Details: Increasing from two to four stages substantially improves performance across all three domains, with the largest gap occurring in visual document retrieval.A more progressive transition smooths replacement of explicit reasoning with latent rollout, reducing optimization difficulty and stabilizing embedding formation.
  • Curriculum Ablation Details: Four curriculum stages provide the best overall trade-off between retrieval accuracy and curriculum efficiency, becoming PLUME’s default setting.The curriculum varies only the number of transition stages while holding the backbone, data, epochs, and other optimization settings fixed.

B. Training Time and Computational Cost Analysis

PLUME’s training cost increases with latent rollout length because variants share the same backbone and training recipe, making latent-step count the main computational difference. Training requires approximately 2562, 2838, and 3119 H20 GPU hours for 4, 6, and 8 latent steps, respectively.

  • Training PLUME with 4 latent steps requires approximately 2562 H20 GPU hours.
  • Increasing the rollout from 4 to 6 latent steps raises training cost to about 2838 H20 GPU hours.All variants use the same backbone and training recipe.
  • Using 8 latent steps further increases training cost to about 3119 H20 GPU hours.The number of latent reasoning steps is the main computational-cost difference across variants.

C. Failure Case Analysis

PLUME remains vulnerable when retrieval requires fine-grained textual preservation, dense document understanding, or externally grounded factual knowledge. ChartQA and InfographicsVQA failures indicate that it often locates the correct semantic neighborhood but loses precise numerical or answer granularity, partly reflecting limitations of embedding-based QA evaluation.

  • C. Failure Case Analysis: PLUME struggles with fine-grained textual preservation, dense document understanding, and externally grounded factual knowledge despite performing strongly on compact multi-step retrieval tasks.These cases characterize the remaining limitations of latent reasoning rather than a general failure of retrieval-oriented representation formation.
  • C. Failure Case Analysis: In ChartQA, PLUME ranks nearby distractors above correct answers, including 4 above 5 and several values above the correct average 21.5.The failures involve count-over-threshold queries and multi-step numerical aggregation, suggesting insufficient preservation of small numerical distinctions rather than loss of task intent.
  • C. Failure Case Analysis: In InfographicsVQA, PLUME identifies the relevant semantic neighborhood but ranks broader or compositionally related answers above the exact target, such as twitter, facebook, pinterest above facebook.Infographic inputs require aggregating fine-grained textual cues across multiple regions and maintaining precise answer granularity under complex layouts.
  • C. Failure Case Analysis: MMEB’s visual question-answering formulation evaluates retrieval from a fixed candidate pool, which is not fully aligned with realistic question-answering settings.The query combines an instruction, image, and question, so some observed QA weaknesses may reflect the benchmark formulation as well as model limitations.

D. Additional Routing Visualization

Figure 9 shows that the semantic-anchor-guided routed adapter allocates experts in structured, non-uniform patterns across modalities. This differentiated allocation supports adaptive latent transition pathways rather than a single fixed computation pattern.

  • D. Additional Routing Visualization: Expert 2 activates strongly across nearly all modalities, especially text and video, suggesting broad utility across diverse reasoning situations.Expert 1 instead shows clearer specialization, indicating that experts develop differentiated modality preferences rather than uniform usage.
  • D. Additional Routing Visualization: The routed adapter avoids uniform expert usage while preserving partial overlap, supporting differentiated latent computation for heterogeneous multimodal retrieval inputs.These structured allocation patterns are consistent with using adaptive latent transition pathways instead of a single fixed computation pattern.
  • D. Additional Routing Visualization: Figure 9 visualizes average expert activation across six input types, revealing modality-dependent routing patterns.The modalities are text (T), image (I), video (V), document (D), text-image (TI), and text-video (TV).

E. Full MMEB-V2 Results

The full MMEB-V2 evaluation reports per-task scores for all compared methods, complementing averaged category-level metrics. PLUME’s gains are broadly distributed across task families rather than concentrated in a small subset of benchmarks.

  • E. Full MMEB-V2 Results: Per-task scores provide a more complete view than the averaged category-level metrics reported in the main paper.
  • E. Full MMEB-V2 Results: PLUME’s gains span task families broadly, rather than arising from only a small subset of MMEB-V2 benchmarks.Table 7, 8 provides the detailed performance of all compared methods supporting this full-benchmark comparison.
  • E. Full MMEB-V2 Results: The section reports detailed per-task performance for all compared methods on MMEB-V2.

F. Additional Baseline Comparisons

Qualitative retrieval cases show that explicit CoT can be derailed by textual, semantic, visual, or action-description errors, whereas PLUME avoids committing to a fixed verbal path and recovers the correct target after latent rollout.

  • Qualitative comparisons: Across representative cases, PLUME retrieves the correct target after latent rollout when explicit CoT is misled by spurious text, coarse compression, surface priors, or incorrect action descriptions.These failure modes are illustrated by the ARM​​YJUNK cue, overly coarse intent summaries, the cardboard egg-carton prior, and an incorrect video action description.
  • Latent rollout analysis: PLUME’s rollout trajectories diagnostically show changing cosine similarity between intermediate latent states and final retrieved candidates, while the correct target remains recoverable and is ultimately selected.The trajectories are diagnostic only and do not imply that every step corresponds to an interpretable verbal reasoning trace.
  • Qualitative comparisons: PLUME’s latent rollout avoids locking retrieval into an early verbal mistake because it does not externalize intermediate reasoning as a discrete textual rationale.The examples support this contrast even though latent-state similarities may fluctuate during rollout rather than monotonically approach the correct target.
Loading 2604.02073v3…