Source-linked AI summary
e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings
Haonan Chen, Sicheng Gao, Radu Timofte, Tetsuya Sakai, Zhicheng Dou
TL;DR
Omni-modal embeddings need reliable comparison across heterogeneous modalities, but VLM-inherited implicit alignment leaves similarity scales, negative hardness, and shared-space geometry mismatched. e5-omni adds three explicit alignment modules without changing the backbone, and experiments report consistent gains on MMEB-V2 and AudioCaps, with transfer to other VLMs. The recipe is aimed at similarity geometry and retrieval optimization rather than higher-level reasoning.
Problem
VLM-based omni-modal embeddings may retain modality-dependent similarity scales, imbalanced negative hardness, and mismatched cross-modal geometry.
Method
e5-omni adds modality-aware temperature calibration, a debiased controllable negative curriculum, and batch whitening with covariance alignment while keeping the VLM backbone unchanged.
Results
e5-omni delivers consistent gains over strong baselines on MMEB-V2 and AudioCaps, and transfers well to other VLM backbones.
Takeaways & Limitations
Explicit alignment provides a lightweight recipe for more robust omni-modal embeddings across diverse retrieval benchmarks and VLM backbones.
Takeaways & Limitations
The recipe primarily targets retrieval similarity geometry and optimization, not higher-level reasoning or compositional understanding.
Abstract
from arXiv · showhide
Modern information systems often involve different types of items, e.g., a text query, an image, a video clip, or an audio segment. This motivates omni-modal embedding models that map heterogeneous modalities into a shared space for direct comparison. However, most recent omni-modal embeddings still rely heavily on implicit alignment inherited from pretrained vision-language model (VLM) backbones. In practice, this causes three common issues: (i) similarity logits have modality-dependent sharpness, so scores are not on a consistent scale; (ii) in-batch negatives become less effective over time because mixed-modality batches create an imbalanced hardness distribution; as a result, many negatives quickly become trivial and contribute little gradient; and (iii) embeddings across modalities show mismatched first- and second-order statistics, which makes rankings less stable. To tackle these problems, we propose e5-omni, a lightweight explicit alignment recipe that adapts off-the-shelf VLMs into robust omni-modal embedding models. e5-omni combines three simple components: (1) modality-aware temperature calibration to align similarity scales, (2) a controllable negative curriculum with debiasing to focus on confusing negatives while reducing the impact of false negatives, and (3) batch whitening with covariance regularization to better match cross-modal geometry in the shared embedding space. Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines, and the same recipe also transfers well to other VLM backbones. We release our model checkpoint at https://huggingface.co/Haon-Chen/e5-omni-7B.
1 Introduction
e5-omni addresses the challenges of embedding heterogeneous modalities in a shared space by adding explicit alignment to VLM-based models. Its three lightweight components calibrate similarity scales, improve negative sampling, and harmonize cross-modal geometry, yielding consistent benchmark gains.
- Omni-modal retrieval must compare heterogeneous text, image, audio, and video items in a shared embedding space.
- Implicit alignment inherited from VLM pretraining leaves modality-specific similarity scales and local geometry insufficiently calibrated.Contrastive fine-tuning may not fully correct these mismatches.
- Its three modules calibrate logits, select harder negatives with debiasing, and align batch covariances across modalities.Together, they target contrastive scale, negative hardness, and second-order shared-space geometry.
- e5-omni adapts VLMs without changing their backbone architecture through explicit alignment.
2 Method: e5-omni
e5-omni preserves a VLM backbone while adding explicit alignment components for heterogeneous omni-modal embeddings. The recipe calibrates logits, focuses training on informative negatives, and regularizes shared-space geometry.
- Overview: e5-omni maps heterogeneous omni-modal inputs into a shared embedding space while preserving the backbone and adding a lightweight training recipe.Matched pairs are encouraged to be close and mismatched pairs separated.
- Modality-aware Temperature Calibration: A single global temperature can make modality-specific logits overly sharp or flat, producing imbalanced gradients and unstable optimization.The method addresses this with learnable modality-aware scaling and symmetric pairwise temperatures.
- Controllable Negative Curriculum: The negative curriculum progressively retains fewer, harder negatives, while debiased contrastive learning reduces the impact of potential false negatives.The schedule starts with more negatives, then increases hardness after a warmup period.
- Training Objective: The final objective combines the debiased contrastive loss with a CORAL-style covariance regularizer.The covariance term is controlled by λcoral.
- Batch Whitening and Covariance Alignment: Batch whitening and covariance alignment regularize mismatched second-order statistics that can distort similarity geometry and destabilize ranking.A shared whitening transform is applied to query and target embeddings, followed by covariance regularization.
3 Experiments
e5-omni is evaluated on multimodal retrieval benchmarks, through ablations, and with distribution-level diagnostics of cross-modal embedding alignment. It consistently outperforms strong baselines while reducing measured embedding-space mismatch.
- Evaluation Setup: MMEB-V2 spans 9 meta-tasks and 78 tasks, while AudioCaps contains about 4.4K text–audio pairs.MMEB-V2 uses Hit@1 for image/video tasks and NDCG@5 for visual-document tasks; AudioCaps uses Recall@1.
- Overall Results: e5-omni outperforms strong bi-modal and omni-modal baselines on both MMEB-V2 and AudioCaps.The result supports the effectiveness of the proposed framework across the two evaluated benchmarks.
- Overall Results: The method improves image/video and visual-document retrieval, while also achieving higher AudioCaps Recall@1 than omni-modal baselines.The reported gains extend across heterogeneous visual inputs and audio retrieval.
- Ablation Study: Removing any alignment component degrades performance on MMEB-V2 and AudioCaps, indicating that the techniques are complementary.The ablation removes one design choice at a time while keeping the other settings fixed.
- Embedding-space Diagnostics: The full model yields smaller centroid and covariance gaps than e5-omni w/o. alignment in VOC2007 diagnostics.PCA overlap measures first- and second-order mismatch, while covariance heatmaps further indicate reduced second-order discrepancy.
- Embedding-space Diagnostics: The covariance-difference heatmap shows fewer high-magnitude entries for e5-omni, providing distribution-level evidence of a more consistent shared embedding space.The heatmap visualizes entrywise query–target covariance differences after projection to 32D.
3.5 Hyperparameter Analysis
The hyperparameter analysis examines training settings and tests whether explicit alignment transfers across VLM backbones. Results identify a trade-off for covariance regularization and show consistent gains across backbones.
- Sensitivity Analysis: Hyperparameters are tuned on 1K-sample validation splits, with MMEB-V2 test performance reported in Fig. 5.The figure reports overall scores using the same metric as Table 1.
- Backbone Generalization: Applying e5-omni to multiple VLMs with different sizes and architectures consistently improves MMEB-V2 results over corresponding w/o. alignment baselines.The backbone comparison omits AudioCaps because few public VLMs support audio inputs out of the box.
4 Related Work
Prior work expanded embedding models from text–image retrieval toward unified multimodal spaces, but commonly relies on implicit alignment from VLM pretraining. e5-omni instead adds explicit calibration and alignment mechanisms.
- Bi-modal Embedding Models: Early bi-modal embedding models align text and images in a shared space using dual-encoder contrastive learning.CLIP is cited as an example trained on paired image–caption data at scale.
- Omni-modal Embedding Models: Recent omni-modal models extend coverage to video and audio through unified embedding architectures built on VLM backbones.Examples include Omni-Embed-Nemotron and LCO-Embed.
- Research Gap: These omni-modal models largely rely on implicit VLM-pretraining alignment and typically lack explicit calibration and alignment mechanisms.The limitation becomes more critical as the number of supported modalities grows.
- Research Gap: e5-omni introduces a lightweight plug-and-play explicit-alignment recipe to improve robustness under mixed-modality training.The comparison is made against omni-modal models that broaden modality coverage but rely primarily on implicit alignment.
5 Conclusion
e5-omni keeps VLM backbones unchanged while adding three explicit alignment components for unified omni-modal embeddings. Experiments on MMEB-V2 and AudioCaps show consistent gains over strong baselines.
- Conclusion: e5-omni turns off-the-shelf VLM backbones into unified omni-modal embedding models without modifying the backbone.The recipe adds temperature calibration, a debiased negative curriculum, and batch whitening with covariance alignment.
- Conclusion: Experiments on MMEB-V2 and AudioCaps show consistent gains over strong baselines, supporting explicit alignment for omni-modal embeddings.The conclusion identifies explicit alignment as the central contribution of the recipe.
Limitations
The paper identifies scope, batch-statistics, and evaluation-coverage limitations that constrain how broadly e5-omni’s retrieval results should be interpreted.
- Scope of alignment: The recipe targets retrieval similarity geometry and optimization, not higher-level reasoning or compositional understanding.Therefore, gains may be smaller on tasks requiring multi-step inference beyond embedding similarity.
- Dependence on batch statistics: Batch whitening and covariance alignment depend on mini-batch estimates whose quality varies with batch size, modality composition, and distributed training.Estimates may remain noisy for small or highly imbalanced omni-modal batches despite jitter and shrinkage stabilization.
- Training and evaluation coverage: Evaluation covers MMEB-V2 and AudioCaps, leaving broader audio/video domains, long-horizon retrieval, and real-world multimodal corpora for future assessment.MMEB-V2 covers text/image/video retrieval, while AudioCaps covers text–audio retrieval.
A Detailed Results on MMEB-V2
The MMEB-V2 evaluation reports overall and modality-group results across image, video, and visual-document retrieval tasks, with detailed task-level breakdowns.
- Overall results: The results section compares e5-omni with strong baselines on overall MMEB-V2 performance.Detailed results are organized into summary, image, video, and visual-document evaluations.
- Benchmark coverage: MMEB-V2 contains 78 tasks spanning Image, Video, and Visual Document modality groups.The benchmark includes 36 image tasks, 18 video tasks, and 24 visual-document tasks.
- Metrics and breakdowns: Image and video task results use Hit@1, while visual-document tasks use NDCG@5.The visual-document breakdown covers document-level retrieval benchmarks such as ViDoRe.
B Implementation Details
The implementation details specify how whitening, covariance regularization, debiased negative training, and hyperparameter selection are operationalized for reproducibility.
- Whitening and covariance regularization: Whitening uses concatenated query and target embeddings to derive one transform applied to both sets before covariance regularization.A CORAL-style Frobenius penalty aligns second-order statistics, while group-wise computation and jitter δ = 10^-4 improve numerical stability.
- Debiased negative training: Debiased contrastive learning is combined with curriculum masking to reduce false-negative bias during negative selection.The curriculum avoids overly aggressive hard negatives early and gradually increases negative pressure later.
- Hyperparameter selection protocol: Hyperparameter sweeps use 1K-sample validation splits drawn from the corresponding training sets.Reported results use MMEB-V2 test sets for consistency with prior work.
C Additional Hyperparameter Studies
Additional studies examine negative-curriculum settings on the 7B backbone, including initialization and warmup choices, with results reported on MMEB-V2.
- Study setup: The curriculum sweeps use the 7B backbone with a fixed training budget and 1K-sample validation sets.Results are reported on the MMEB-V2 test set, matching the main experimental protocol.
- Curriculum start: The initial quantile threshold ρinit controls how many negatives are retained by the controllable negative-selection curriculum.The appendix varies ρinit as part of the curriculum sensitivity analysis.
- Warmup cutoff: Warmup improves curriculum stability and final accuracy relative to starting hardness increases immediately.Starting at t0 = 0 underperforms, while the best observed setting is t0 = 4000 steps, used by default.
- Detailed benchmark tables: Detailed MMEB-V2 tables separately report image, video, and visual-document tasks with their corresponding metrics.Image and video tables use Hit@1, while the visual-document table uses NDCG@5.
- Sensitivity analysis: Figure 6 presents sensitivity to negative-curriculum settings on MMEB-V2 using the 7B backbone.The figure summarizes the additional curriculum sweep results.