PaperlayerView exact revision on arXiv

Source-linked AI summary

UEmbed: Unified Sparse and Dense Multimodal Embeddings

Tingyu Song, Mingxin Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Zhijie Nie, Yilun Zhao, Shu Wu

arXiv:2608.02583v1cs.CVcs.AIcs.CLcs.IR

TL;DR

Learned sparse retrieval has remained tied to bidirectional encoders, limiting its extension to decoder-only multimodal models. UEmbed unifies dense and sparse representations in one causal forward pass, with UEmbed-9B reaching 71.8 dense and 71.0 sparse on MMEB-v2.

  • Problem

    Learned sparse retrieval remains largely tied to bidirectional encoders, while decoder-only multimodal extensions face challenges from causal attention and auxiliary cross-modal modules.

  • Method

    UEmbed appends learnable special tokens that summarize inputs under causal attention and predict sparse weights over disjoint vocabulary subsets, alongside dense representations.

  • Results

    71.8 dense and 71.0 sparse on MMEB-v2, with sparse performance remaining competitive across multimodal and BEIR retrieval evaluations.

  • Takeaways & Limitations

    UEmbed combines effectiveness with compatibility with high-throughput serving, inverted-index search, hybrid scoring, and lower tool-call costs in agentic search.

  • Takeaways & Limitations

    Training data is predominantly English and Chinese, resulting in limited cross-lingual generalization of sparse-head activations.

Abstract

from arXiv · show

Sparse retrieval underpins modern search systems, from web search to retrieval-augmented generation. Existing work has introduced Learned Sparse Retrieval (LSR) to push beyond exact lexical matching toward richer semantics. Yet LSR has so far remained tied to encoder-style bidirectional architectures, and its extension to multimodal settings still relies heavily on auxiliary cross-modal modules. To address these limitations, we introduce UEmbed (Unified Embedding), a decoder-only multimodal embedding model that produces both sparse lexical and dense representations in one causal forward pass. UEmbed appends N learnable special tokens to the input and partitions the vocabulary into N disjoint subsets. Each token's causal hidden state predicts sparse weights over its assigned subset, and the N subsets are concatenated into the full sparse vector. Trained on public data, we release UEmbed at 2B, 4B, and 9B scales. UEmbed-9B reaches 71.8 (dense) and 71.0 (sparse) on MMEB-v2, outperforming multimodal embedding models trained on publicly available data (e.g., RzenEmbed). On BEIR, UEmbed also remains competitive with strong dense and sparse baselines. Furthermore, we demonstrate the practical utility of UEmbed across three dimensions: effectiveness, efficiency, and agentic applications. Overall, UEmbed offers a new paradigm: it unifies dense and sparse embeddings in one model, while further extending sparse retrieval to unify text and multimodal inputs.

1 Introduction

UEmbed addresses limitations of learned sparse retrieval by producing sparse and dense representations from a single decoder-only multimodal backbone. Its unified design supports text and multimodal retrieval while delivering competitive benchmark performance and practical benefits in effectiveness, efficiency, and agentic search.

  • Limitations: Existing learned sparse retrieval methods largely rely on bidirectional encoders, while decoder-only models face causal-masking and serving-compatibility constraints.Extending LSR to decoder-only models has required bidirectional conversion or dedicated curricula, adding training cost and disrupting optimized causal-model serving frameworks.
  • Method: UEmbed appends N learnable special tokens and assigns each token a disjoint vocabulary subset to overcome the causal model’s single-token information bottleneck.The subsets are derived through k-means clustering, and the token-specific predictions are combined into the sparse representation.
  • Evaluation: 71.8 (dense) and 71.0 (sparse) are achieved by UEmbed-9B on MMEB-v2, with the two modes remaining within one point at every tested scale.On BEIR across 9 datasets using nDCG@10, UEmbed remains competitive with strong dense and sparse baselines including SPLADE-v3.
  • Practical utility: Hybrid scoring improves text and visual-document retrieval, while UEmbed’s sparse mode improves efficiency and effectiveness and performs strongly in agentic search.These findings motivate UEmbed’s practical utility across effectiveness, efficiency, and agentic applications.
  • Unified embedding: UEmbed produces both sparse and dense retrieval representations from a single decoder-only backbone.The model is designed to unify these representations within one model rather than relying on separate retrieval architectures.
  • Contributions: UEmbed unifies modalities in one model, is extendable to diverse modality settings, and establishes a new state of the art in sparse multimodal retrieval on MMEB-v2.The paper describes UEmbed as the first sparse model evaluated on MMEB-v2.

2 Related Works

Prior work has advanced learned sparse retrieval through neural term reweighting and semantic vocabulary expansion, while multimodal embedding models built on MLLM backbones have achieved competitive benchmark results. However, multimodal sparse retrieval remains comparatively underexplored.

  • Learned Sparse Retrieval: Learned sparse retrieval neuralizes fixed lexical statistics by reweighting observed terms and adding semantically related terms absent from the input.This extends beyond fixed lexical methods such as BM25.
  • Learned Sparse Retrieval: SPLADE-based methods use contextual token weighting and vocabulary expansion to compete with strong dense baselines.The passage attributes this progress to SPLADE-based research.
  • Multimodal Embeddings: Universal multimodal embedding models built on MLLM backbones achieve highly competitive results on benchmarks such as MMEB-v2.The passage situates these models within recent progress in multimodal large language models.
  • Multimodal Sparse Retrieval: Multimodal sparse retrieval remains much less explored than universal multimodal embedding modeling.The passage identifies this as a gap in existing research.

3 Method

UEmbed adapts learned sparse retrieval to decoder-only models by appending special tokens and partitioning the vocabulary into disjoint subsets. It jointly produces sparse and dense representations using a combined retrieval objective and publicly available multimodal, text, and long-document data.

  • Partitioned Sparse Heads: UEmbed appends N learnable special tokens so each can attend to the full preceding input, overcoming decoder-only models’ inability to max-pool future-inaccessible hidden states.The special tokens summarize the full input while addressing the limitations of unidirectional attention.
  • Compressing Vocabulary: Vocabulary compression merges tokens sharing canonical forms, reducing vocabulary size from 248,320 to 184,016 while retaining their maximum scoring weight.Compression applies accent stripping, lowercasing, and whitespace collapsing before partitioning.
  • Partitioned Sparse Heads: N=16 special tokens correspond to approximately equal vocabulary subsets formed through k-means clustering, creating distinct semantic subspaces.Each special token is assigned one of the N disjoint subsets V1, . . . , VN.
  • Partitioned Sparse Heads: Each subset-specific sparse head projects its special token’s final hidden state into weights for assigned terms, and concatenating subset vectors yields the sparse representation.For term t ∈Vk, the responsible special token ⟨sk⟩ supplies the hidden state used by the corresponding linear projection.
  • Dense Representation: The EOS hidden state provides the dense embedding, while omitting special tokens when dense-only retrieval is needed incurs no extra forward computation.The dense embedding is d ∈R^D and is taken from the EOS token preceding the special tokens.
  • Unified Objective: UEmbed trains both retrieval modes with combined InfoNCE losses and FLOPS regularizers, whose coefficients balance four losses and encourage sparsity.InfoNCE uses cosine similarity and inner product for dense and sparse retrieval, respectively; FLOPS regularizes squared mean term weights.

4 Experiments

UEmbed is evaluated at 2B, 4B, and 9B scales using single checkpoints that support dense and sparse retrieval, across multimodal MMEB-v2 and text BEIR benchmarks. It achieves competitive multimodal performance, near-parity between sparse and dense retrieval, and strong BEIR results while unifying retrieval modes and multimodal inputs.

  • Experimental Setup: UEmbed uses 2B, 4B, and 9B Qwen3.5 backbones, with one checkpoint per scale supporting both dense and sparse retrieval.The experiments instantiate UEmbed at three parameter scales and train a single checkpoint for each scale.
  • MMEB-v2: 71.8: UEmbed-9B achieves this aggregate dense score on MMEB-v2, leading models trained on publicly available data but trailing Qwen3-VL-Embedding-8B.Qwen3-VL-Embedding-8B benefits from multi-stage training with vast, proprietary datasets, making direct comparison challenging.
  • MMEB-v2: 1.0 point: the maximum overall gap between UEmbed’s sparse and dense MMEB-v2 performance across scales, including 71.8 versus 71.0 for UEmbed-9B.The results show sparse embeddings remain close to dense counterparts across all three model scales.
  • BEIR: 56.3: UEmbed-9B achieves the highest average dense nDCG@10 on the evaluated BEIR datasets, followed by UEmbed-4B at 56.0.UEmbed-9B also reaches 89.9 on Quora and 40.5 on NFC.
  • BEIR: 55.2: UEmbed-9B matches Echo-Mistral-SPLADE’s average sparse nDCG@10 while supporting dense retrieval and multimodal inputs within one backbone.The result positions UEmbed as a unified model that sacrifices little text performance while gaining robust multimodal and dense capabilities.

5 Analysis

UEmbed’s analysis attributes its gains to the unified causal formulation and semantic vocabulary partitioning, while showing robust multimodal behavior and practical benefits from combining sparse and dense retrieval. The model also supports efficient autoregressive deployment and agentic-search evaluation.

  • Component Analysis: UEmbed-2B surpasses the bidirectional SPLADE baseline by +3.2 points for dense (61.3 →64.5) and +2.1 for sparse (61.3…).The baseline uses the same Qwen3.5 backbone, training data, and FLOPS regularization, but bidirectional attention with a standard SPLADE max-pooling head.
  • Component Analysis: On IMG-QA, UEmbed improves over the baseline by +8.2 dense and +6.0 sparse, indicating benefits from the unified causal formulation.The analysis attributes the gains to better use of the autoregressive backbone’s QA capabilities, beyond serving convenience.
  • Component Analysis: Semantic clustering gives the best sparse performance among the tested vocabulary partitions, while random partitioning performs worst.The compared strategies are random uniform assignment, max-distance pairing of k-means clusters, and assigning individual k-means clusters to distinct subsets.
  • Cross-modal Robustness: UEmbed identifies semantically relevant concepts across modalities, including inferring Singapore from a skyline and recognizing a rocket as SpaceX.The case studies also report limitations involving anomalous-token generation, though the supplied passage is truncated before further details.
  • Practical Advantages: Hybrid scoring improves Text (+0.3) and VisDoc (+0.5) without an additional encoding pass, while autoregressive serving and inverted-index compatibility support deployment.The sparse and dense representations are produced in one forward pass, allowing their similarities to be linearly interpolated.

6 Conclusion

UEmbed is a decoder-only multimodal embedding model that unifies dense and sparse retrieval within one causal forward pass. Evaluations show state-of-the-art sparse multimodal retrieval while remaining highly competitive in dense retrieval.

  • Conclusion: UEmbed natively unifies dense and sparse retrieval within a single causal forward pass.The model is decoder-only and multimodal.
  • Conclusion: UEmbed eliminates reliance on bidirectional encoders and auxiliary cross-modal modules by overcoming causal-attention bottlenecks.This is presented as a systematic consequence of the model design.
  • Conclusion: UEmbed establishes a new state of the art in sparse multimodal retrieval while remaining highly competitive in dense retrieval.These conclusions are based on comprehensive evaluations.

Limitations

UEmbed’s main limitations are limited cross-lingual generalization from English- and Chinese-skewed training data and a larger sparse–dense performance gap in video. Video’s information density and temporal dynamics may also create capacity bottlenecks for flat sparse representations.

  • Language and Cultural Bias: English- and Chinese-skewed training data limits the sparse head’s cross-lingual generalization.Broader sparse-language coverage would require highly language-diverse, large-scale training data.
  • Modality-Specific Performance Gaps: Sparse and dense representations maintain close parity on text and static visual documents but show a more pronounced gap in video.The paper attributes this gap to video frames’ high information density and temporal dynamics.
  • Modality-Specific Performance Gaps: Naively pooling spatiotemporal video data into a flat sparse vector may encounter capacity bottlenecks.This concern follows from video’s high information density and temporal dynamics.

Ethical Consideration · A Implementation Details · A.1 Training Data

UEmbed is trained exclusively on publicly available data, with released weights supporting reproducibility, and its training mixture spans text, image, video, and visually rich document retrieval. The paper also discusses bias risks and interpretability benefits, while detailing modality-specific datasets and sequence or frame budgets.

  • Ethical Consideration: The authors release weights and train exclusively on publicly available datasets to facilitate reproducibility.They caution that models may inherit biases from training data and the backbone LLM.
  • Ethical Consideration: Practitioners should evaluate fairness across demographic groups before deploying the embedding model in high-stakes retrieval applications.Sparse representations may aid bias auditing by surfacing activated vocabulary terms.
  • A.1 Training Data: The training mixture covers text, image, video, and visually rich document retrieval, with source ratios reported in Figure 4.Per-dataset retrieval instructions are listed in Tables 7 and 8.
  • A.1 Training Data: The text subset combines Echo-Embedding and M3-Embedding pairs across NLI, DuReader, ELI5, FEVER, HotpotQA, MIRACL, MrTyDi, and other retrieval datasets.Standard text tasks use 1,500-token maximum sequences, while MLDR uses 1,800 tokens.
  • A.1 Training Data: Video retrieval uses VideoCaption300k and VideoQA240k, sampling 8 frames per video with up to 200 tokens per frame.These datasets come from LLaVA-Hound.
  • A.1 Training Data: Visually rich document retrieval incorporates ColPali train (118k), VisRAG-Synthetic (239k), and VisRAG-IID (123k).The sources are drawn from ViDoRe and VisRAG.
  • A.1 Training Data: Image-text retrieval includes MMEB training datasets covering CIRR, NIGHTS, MSCOCO, and VisualNews.The supplied passage lists these datasets as part of the multimodal training data.
  • A Implementation Details: Training dynamics are compared using loss, max negative gap, and mean negative gap.Figure 5 presents these three quantities during training.

A.2 Training Details … B.2 Evaluation Settings

UEmbed training combines parameter-efficient adaptation, frozen visual features, mixed-precision distributed optimization, and balanced dense–sparse objectives. Evaluation compares single-checkpoint dense and sparse embeddings using benchmark-specific metrics and defined score functions.

  • A.2 Training Details: N=16 special tokens are added to a pretrained multimodal backbone and tokenizer for UEmbed training.The embedding matrix is expanded to accommodate the special tokens.
  • A.2 Training Details: LoRA adapts attention and MLP projections while the visual encoder remains frozen.The adapted projections are q_proj, k_proj, v_proj, up_proj, down_proj, and gate_proj.
  • A.2 Training Details: bf16 mixed precision, DeepSpeed ZeRO, and gradient checkpointing are enabled during training.These implementation choices are stated as part of the UEmbed training setup.
  • A.2 Training Details: A cosine schedule uses a 3×10−5 peak learning rate and 0.1 warmup ratio, with τ=0.03 for dense retrieval and τs=32 for sparse retrieval.The FLOPS regularizer coefficients are αq = αd = 1×10−4, and the sparse loss factor is λ=1.0.
  • A.3 Training Analysis: Text training data stabilizes and accelerates multimodal sparse training, whose monitored metrics include loss, max negative gap, and mean negative gap.Sparse-score inner products can be unstable because unnormalized similarities have a large dynamic range.
  • B.1 Evaluated Models: UEmbed is compared with baselines on MMEB-v2 for multimodal models and nine BEIR datasets for text-only models.Table 9 notes that UEmbed produces both dense and sparse embeddings from a single checkpoint.
  • B.2 Evaluation Settings: MMEB-v2 uses its dataset-specific metrics, while BEIR uses nDCG@10; dense scores are EOS-token cosine similarities and sparse scores are partitioned-vocabulary inner products.The dense representation uses the last content token preceding the special tokens.

B.3 Details of Ablation Study · B.4 Extra Experiment Results · C Case Study

The appendix details UEmbed’s hybrid scoring and sparse-retrieval efficiency, then reports seed-42 ablations showing consistent effects for joint training, special-token count, and sparse-training temperature.

  • B.3 Details of Ablation Study: Hybrid scoring linearly combines dense cosine similarity sdense(q, d) with sparse inner product ssparse(q, d).The dense and sparse components operate on different scales.
  • B.3 Details of Ablation Study: α = 1.0, while β is tuned per modality on a held-out split.The reported β values are 5e−8 for Text, 5e−4 for Image, 1e−4 for Video, and 7e−4 for VisDoc.
  • B.3 Details of Ablation Study: Plain text uses β = 5e−8 because raw sparse inner products grow rapidly with activated-token count.Multimodal inputs use larger β values because they produce much sparser activation patterns.
  • B.3 Details of Ablation Study: Sparse representations integrate natively with inverted indices for large-scale lexical retrieval.BrowseComp-Plus offline retrieval compares a Faiss-based dense index with a Lucene-backed sparse inverted index.
  • B.3 Details of Ablation Study: Capping Kprune exposes a latency–accuracy trade-off in sparse retrieval.Figure 6 compares search efficiency and effectiveness between dense retrieval and sparse inverted-index retrieval.
  • B.4 Extra Experiment Results: Seed-42 MMEB-v2 trends broadly match the main-paper settings.The extra experiments report that joint dense-sparse training preserves dense performance while yielding a competitive sparse branch, too many special tokens hurt sparse retrieval, and higher sparse-training temperature helps.

C.1 Qualitative Analysis · C.2 Case Study

UEmbed’s qualitative gap with Qwen3-VL-Embedding is modality-dependent, especially for fine-grained image recognition and question-grounded VQA. Case studies show that its sparse activations are highly relevant to query intent on QA, Retrieval, and Grounding tasks across multiple modalities.

  • C.1 Qualitative Analysis: UEmbed’s errors relative to Qwen3-VL-Embedding are modality-dependent.The analysis recomputed per-query Hit@1 on MMEB-v2 and inspected queries where Qwen3-VL-Embedding succeeds but UEmbed fails.
  • C.1 Qualitative Analysis: Image gaps mainly involve fine-grained recognition and question-grounded VQA, where UEmbed can confuse visually similar categories or follow misleading lexical cues.The affected datasets include ImageNet-A, SUN397, N24News, GQA, and ScienceQA.
  • C.1 Qualitative Analysis: The video analysis identifies action understanding as an area where Qwen3-VL-Embedding is stronger.The supplied passage reports this comparison but ends before specifying further video-case details.
  • C.2 Case Study: Case studies visualize UEmbed’s top-10 activated sparse tokens, distinguishing query, corpus, and shared co-activated tokens by color.Blue denotes query activations, green corpus activations, and red shared co-activated tokens.
  • C.2 Case Study: Sparse representations perform well on QA, Retrieval, and Grounding tasks because their activated tokens are highly relevant to query intent.The case studies span different tasks and modalities.
  • C.2 Case Study: The case-study figures cover classification on N24News, question answering on OK-VQA, text-to-image retrieval on MSCOCO, and grounding on MSCOCO.Each figure visualizes the top-10 activated sparse tokens.
Loading 2608.02583v1…