PaperlayerView exact revision on arXiv

Source-linked AI summary

ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compression

Zijing Cai, Yuzhe Wang, Jingxian Zhu, Fengbin Zhu, Richang Hong

arXiv:2609.37225v1cs.CVcs.AI

TL;DR

Compact multimodal embeddings must balance fine-grained visual evidence against storage and interaction costs. ResComEmb uses trainable coarse-to-fine compression and bidirectional late interaction; it reports leading results across MMEB and ViDoRe V1/V2, with higher ViDoRe retrieval performance than ColQwen2.5 at 37.5% of its visual-token budget.

  • Problem

    Single-vector embeddings can limit fine-grained expression, while multi-vector representations preserve local evidence but increase storage and pairwise-interaction costs.

    Source: §1 · p. 1

  • Method

    ResComEmb encodes global, intermediate, and fine-grained views, applies trainable Residual Homogeneity Compression and nested supervision, then scores queries and documents with length-adaptive bidirectional late interaction.

    Source: §1 · p. 2

  • Results

    Across MMEB and ViDoRe V1/V2, ResComEmb achieves the best averages; on ViDoRe V1, it scores 90.4 NDCG@5, 1.0 point above ColQwen2.5.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ResComEmb reports higher retrieval effectiveness on both ViDoRe benchmarks using 37.5% of full-token ColQwen2.5’s visual-token budget.

    Source: §1 · p. 2

    The benchmark composition may reflect domain, language, synthetic-data, or annotation biases, so results do not establish fairness or suitability for high-stakes deployment.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have shown strong potential for universal multimodal representation learning. However, existing methods either compress each input into a single vector, limiting fine-grained expressiveness, or retain long sequences of visual-token vectors, incurring substantial storage and interaction costs. To resolve this trade-off, we propose ResComEmb, a trainable framework for effective and efficient universal multi-vector multimodal embedding. ResComEmb first encodes each input at native dynamic resolution into ordered global, intermediate, and fine-grained views. After MLLM contextualization and embedding projection, a trainable Residual Homogeneity Compression (RHC) module reduces within-granularity redundancy and cross-granularity repetition under explicit visual token budgets. Then, ResComEmb introduces a length-adaptive Bidirectional Late-Interaction Matching mechanism for robust query-document scoring, which averages the strongest token-level matches in each direction and combines the two scores using a weight based on how many valid tokens each side has. Extensive experiments on MMEB, ViDoRe V1, and ViDoRe V2 show that ResComEmb produces higher-quality universal multimodal embeddings than VLM2Vec-V2, and outperforms ColQwen2.5 in visual document retrieval using only 37.5% of its full visual token budget, demonstrating a favorable effectiveness-efficiency trade-off.

1 INTRODUCTION

The introduction frames compact multimodal embeddings as a trade-off between fine-grained visual evidence and storage or interaction costs. ResComEmb addresses this with trainable coarse-to-fine compression and reports strong results across general multimodal and visual document retrieval benchmarks.

1 INTRODUCTION

  • Single-vector embeddings can limit fine-grained expression, while multi-vector representations preserve local evidence at higher storage and pairwise-interaction costs.Existing pruning or merging can sacrifice holistic context, while fixed query-slot capacity can restrict locality and input-specific detail under tight budgets.Source: §1 · pp. 1–2
  • MURE combines multi-resolution sampling with non-trainable post-hoc clustering, motivating joint optimization of compression and representations.ResComEmb aims to merge semantically similar tokens within granularities and prioritize finer details absent from coarser representations.Source: §1 · p. 2
  • ResComEmb encodes global, intermediate, and fine-grained views, then trainably compresses contextualized projected representations and applies nested supervision and length-adaptive bidirectional matching.RHC reduces within-granularity redundancy and cross-granularity repetition while preserving finer-grained evidence under explicit token budgets.Source: §1 · p. 2
  • 67.4 Precision@1 on MMEB exceeds VLM2Vec-V2 by 2.5 points, while ViDoRe V1/V2 scores of 90.4/61.6 NDCG@5 surpass ColQwen2.5 by 1.0 point each at 37.5% of its visual-token budget.The evaluation covers MMEB and ViDoRe V1/V2.Source: §1 · p. 2

2 RELATED WORK

Related work traces multimodal embedding from modality-specific contrastive encoders to unified MLLM representations and multi-vector approaches. It also distinguishes visual compression before MLLM contextualization from compression of contextualized embeddings.

2.1 MULTIMODAL EMBEDDING

  • Contrastive encoders such as CLIP and SigLIP use modality-specific towers, while MLLMs process interleaved image–text sequences in a unified backbone for instruction-conditioned embeddings.Globally pooled retrieval embeddings do not preserve explicit local visual–text correspondences; multi-vector methods retain independently accessible features.Source: §2.1 · p. 3
  • Single-vector methods reduce storage and comparison costs but may suppress localized or compositional evidence; multi-vector methods retain regions, objects, and text as separate features.RESCOMEMB builds on hierarchical multi-resolution encoding with trainable Residual Homogeneity Compression.Source: §2.1 · p. 3

2.2 VISUAL REPRESENTATION COMPRESSION

  • Upstream compression reduces visual tokens processed within the MLLM, whereas downstream compression reduces contextualized embeddings for storage and comparison.MURE retains input-aligned multi-resolution evidence but uses non-trainable clustering and does not explicitly separate within-granularity redundancy from cross-granularity repetition; RESCOMEMB introduces a downstream-trainable compressor for these forms under explicit budgets.Source: §2.2 · p. 3

3 METHOD

ResComEmb formulates budgeted multimodal embedding and combines multi-granularity visual encoding, residual compression, nested representations, and bidirectional late-interaction matching.

3.1 PROBLEM FORMULATION

  • Text and visual inputs map to token-level embeddings under a visual-token budget, while all text tokens are retained and only visual tokens are compressed.Source: §3.1 · p. 3

3.2 MULTI-GRANULARITY VISUAL ENCODING

  • Each input yields global, aspect-ratio-aware intermediate, and fine-grained views, which a shared MLLM contextualizes with the text before embedding projection.The three view granularities use 1, 2, and 4 crops, respectively.Source: §3.2 · p. 4

3.3 RESIDUAL HOMOGENEITY COMPRESSION

  • At each stage, an assessor scores token importance and within-stage similarity, while preceding compressed stages serve as anchors for estimating novelty.Source: §3.3 · pp. 4–5
  • Importance and novelty determine preservation priority; similar source–destination pairs are merged until the stage budget is met, and aggregated outputs remain differentiable.Priority also modulates value aggregation to enable gradients to train the assessor.Source: §3.3 · p. 5
  • Surviving visual outputs are assembled coarse-to-fine into prefixes, with each level retaining the same text prefix and adding progressively finer compressed stages.Source: §3.3 · p. 5

3.4 BIDIRECTIONAL LATE-INTERACTION MATCHING

  • Bidirectional matching combines query-to-document support and document-to-query coverage, averaging each direction’s strongest token matches with a count-adaptive weight.Each direction uses up to K valid tokens; when the document is longer, weighting shifts toward query-to-document support while retaining reverse coverage.Source: §3.4 · pp. 5–6

3.5 TRAINING

  • Each active nested level uses its matching score in a contrastive loss over in-batch documents and any available explicit negatives, with level-specific loss weights.Source: §3.5 · p. 6

4 EXPERIMENTS

RESCOMEMB is evaluated on general multimodal embedding and visual document retrieval benchmarks, where it reports strong performance with compact visual-token budgets. Ablations examine nested supervision, RHC scoring signals, prefix lengths, and token budgets.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers MMEB Precision@1 and ViDoRe V1/V2 NDCG@5, comparing single- and multi-vector embedding models.The comparisons include general multimodal embedding tasks and visual document retrieval.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • 67.4 is RESCOMEMB’s highest MMEB average; its ViDoRe V1 and V2 averages are 90.4 and 61.6, respectively.On MMEB it exceeds VLM2Vec-V2 by 2.5 points; on ViDoRe V1 and V2 it improves over ColMate by 0.5 points and ColQwen2.5 by 1.0 point, respectively.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7
  • 384 visual tokens—37.5% of ColQwen2.5’s budget—improve average NDCG@5 by 1.0 point on both ViDoRe benchmarks.All RESCOMEMB entries use per-stage RHC budgets of 128/128/128.Source: §4.2 · pp. 6–7

4.3 ABLATION STUDIES AND IN-DEPTH ANALYSIS

  • Removing MRL lowers ViDoRe V1 by 1.0–2.8 points, ViDoRe V2 by 5.7–7.6 points, and MMEB by 2.8–3.0 points across 128-, 256-, and 384-token prefixes.Without MRL, every MMEB task category also declines: classification by 3.6–3.9 points, VQA by 0.5–1.2, retrieval by 4.1–5.0, and grounding by 1.5–2.5.Source: §4.3 · p. 8, Table 4 · p. 9
  • Removing the importance score lowers ViDoRe V1/V2 by 0.3/2.1 points; removing novelty lowers them by 0.7/3.5 points.The larger drops without novelty are consistent with its role in distinguishing residual fine-grained evidence from content represented by coarser views.Source: Figure 2 · p. 8, §4.3 · p. 9
  • The 128-token prefix retains 98.7% of 384-token performance on each ViDoRe benchmark and 99.0% on MMEB; scaling to 384 adds 1.2, 0.8, and 0.7 points, respectively.These are prefixes of the same MRL-trained model, supporting inference at different representation lengths without retraining.Source: Figure 2 · p. 8, §4.3 · p. 9
  • Scaling separately trained budgets from 48 to 192 vectors improves ViDoRe V1/V2 by 3.3/5.1 points and MMEB by 1.0 point; 384 vectors offer the best effectiveness–efficiency balance.Increasing from 192 to 384 adds 0.9 points on V1, 1.0 on V2, and 0.1 on MMEB; doubling to 768 adds little on ViDoRe and lowers MMEB by 0.7 points.Source: Figure 2 · p. 8, §4.3 · p. 9

5 CONCLUSION

RESCOMEMB combines residual compression, nested supervision, and bidirectional late interaction to build compact universal multimodal embeddings. The paper reports strong benchmark results with a reduced visual-token budget, while noting that benchmark biases preclude conclusions about fairness or high-stakes suitability.

5 CONCLUSION

  • RESCOMEMB’s coarse-to-fine design reduces within-granularity redundancy and cross-granularity repetition while preserving complementary visual information.The authors suggest this structured redundancy perspective motivates task-adaptive redundancy modeling and trainable compression in other representation-generation settings.Source: §5 · pp. 9–10

AI USE STATEMENT

  • Generative AI assisted with translation, literature search, manuscript organization, editing, formatting, and drafting gradient analysis; the authors checked the analysis and take responsibility for the work.They state that AI did not generate synthetic datasets or experimental results.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The evaluation uses public benchmarks whose domain, language, synthetic-data, or annotation biases mean results do not establish fairness or high-stakes suitability.The authors report no additional privacy, security, legal, or conflict-of-interest concerns introduced by the evaluated method.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The experiments use publicly released data resources, with repositories listed for training pairs, hard negatives, evaluation sets, and the model checkpoint.The paper points to Section 4 for the evaluation protocol and Appendix A for benchmark composition and training configuration.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXPERIMENTAL DETAILS

The experiments evaluate ResComEmb across three multimodal benchmarks with distinct task and domain coverage. The gradient-flow analysis explains how retrieval supervision reaches RHC’s learned signals through differentiable value aggregation despite discrete matching.

A EXPERIMENTAL DETAILS

  • MMEB covers 36 tasks across classification, visual question answering, retrieval, and visual grounding; ViDoRe V1 covers 10 page-retrieval subsets, while V2 covers 7 challenging out-of-domain subsets.MMEB reports macro-averaged Precision@1; both ViDoRe benchmarks report macro-averaged NDCG@5.Source: A EXPERIMENTAL DETAILS · p. 15
  • Models use AdamW for 30,000 steps with global batch size 64, and all active nested prefixes receive supervision unless an ablation explicitly changes the objective.Training uses eight NVIDIA A100 GPUs, LoRA rank and scaling 32, and a learning rate decaying from 10^-4 to zero.Source: A EXPERIMENTAL DETAILS · p. 15

B GRADIENT FLOW THROUGH RESIDUAL HOMOGENEITY COMPRESSION

  • RHC holds discrete merge assignments fixed during backpropagation, while gradients pass through merged values and Equation 9 connects the assessor to retrieval loss without a straight-through estimator or auxiliary compression loss.Assignments are recomputed after each parameter update.Source: B GRADIENT FLOW THROUGH RESIDUAL… · pp. 15–16
  • MinMax importance is relative within each stage, and the modulation remains bounded by [1, 1 + ρ] while preserving token direction.Importance and novelty are equally weighted in the modulation; group averaging preserves the mean of the original members.Source: B GRADIENT FLOW THROUGH RESIDUAL… · p. 15
  • Under strict selection margins and other stated regularity assumptions, the fixed partition yields a locally differentiable value path; switching ties are measure-zero boundaries where the hard map is not differentiable.Gradients reach both importance and gate heads when their local derivatives are nonzero, and the matching signal routes through current MaxSim and TopK matches.Source: B GRADIENT FLOW THROUGH RESIDUAL… · pp. 15–16

C DETAILED RHC SCORING-SIGNAL RESULTS

Controlled RHC ablations remove either importance or novelty while retaining the other scoring and retrieval components. Their ViDoRe V2 losses are concentrated in particular subsets, consistent with the complementary role of cross-stage coverage.

C DETAILED RHC SCORING-SIGNAL RESULTS

  • The controlled variants remove either the importance score or novelty score while retaining similarity scoring, merging, nested supervision, and late-interaction scoring.Both variants use the same initialization, stage budgets, and optimization as the complete model.Source: C DETAILED RHC SCORING-SIGNAL RESULTS · p. 16
  • Removing novelty costs 4.8 points on Eco Mul and 8.3 on ESG Syn-Mul, while removing importance has its largest effect, 4.3 points, on Bio.Table 6 reports RHC scoring-signal ablations on ViDoRe V2 subsets using NDCG@5.Source: Table 6 · p. 17, C DETAILED RHC SCORING-SIGNAL RESULTS · p. 17
  • Without novelty, later stages rank candidates only by salience, leaving cross-stage redundancy unpenalized; novelty favors candidates weakly covered by earlier stages.The authors suggest that concentrated losses on multilingual and synthetic subsets indicate greater importance of complementary coverage under those distribution shifts.Source: C DETAILED RHC SCORING-SIGNAL RESULTS · p. 17

D DETAILED LATE-INTERACTION RESULTS

Late-interaction ablations show that scoring operations contribute differently across benchmark families. The method also adapts directional weights to valid query and document token counts.

D DETAILED LATE-INTERACTION RESULTS

  • Directional weights depend on valid query and document token counts: they are equal when counts match, while longer documents increase query-to-document weight without eliminating the reverse direction.The query-to-document weight is capped by ρmax.Source: D DETAILED LATE-INTERACTION RESULTS · p. 17
  • Table 7 shows that, on MMEB, reverting to vanilla MaxSim reduces the average by 22.9 points, while removing mean aggregation alone causes a 9.3-point drop.Source: D DETAILED LATE-INTERACTION RESULTS · p. 17
  • On ViDoRe V2, removing TopK lowers performance by 7.1 points, while fixed 0.5/0.5 directional weights lower it by 7.6 points.Removing bidirectionality lowers the MMEB average by 2.0 points while retaining query-to-document TopK-mean MaxSim.Source: D DETAILED LATE-INTERACTION RESULTS · p. 17

E PRELIMINARY MULTI-GRANULARITY STUDY

The multi-granularity study compares fixed visual scales with a per-query oracle across document retrieval and MMEB. Oracle gains over the strongest fixed granularity are largest on ViDoRe V2.

E PRELIMINARY MULTI-GRANULARITY STUDY

  • The oracle selects the best retrieval outcome per query using ground-truth relevance, estimating adaptive multi-resolution potential rather than providing a deployable routing method.The study separately evaluates global, aspect-ratio-aware intermediate, and fine-grained models.Source: E PRELIMINARY MULTI-GRANULARITY STUDY · p. 17
  • The per-query oracle improves over the strongest fixed granularity by 4.7 points on V1, 10.8 on V2, and 10.2 on MMEB Precision@1.Figure 3 reports NDCG@5 for ViDoRe V1/V2 and Precision@1 for MMEB.Source: Figure 3 · p. 18, E PRELIMINARY MULTI-GRANULARITY STUDY · p. 18
  • The largest oracle gain is on ViDoRe V2, where the study associates stronger complementary-scale value with domain shift and motivates preserving multiple scales before compression.The reported oracle gains extend beyond document retrieval to MMEB.Source: E PRELIMINARY MULTI-GRANULARITY STUDY · p. 18

F ADDITIONAL MULTI-GRANULARITY ABLATIONS

At a fixed 384-visual-token budget, the complete global, intermediate, and fine-grained hierarchy performs strongest across benchmark families, with all three granularities contributing complementary evidence.

F ADDITIONAL MULTI-GRANULARITY ABLATIONS

G ADDITIONAL TOKEN-BUDGET ABLATIONS

Most gains appear by 384 vectors: the 192-vector model nearly matches the main configuration, while 768 vectors add little to ViDoRe and lower the MMEB average. The selected 128/128/128 stage budget balances capacity and storage.

G ADDITIONAL TOKEN-BUDGET ABLATIONS

Related papers

Loading 2609.37225v1…