e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings

• Omni-modal retrieval must compare heterogeneous text, image, audio, and video items in a shared embedding space.Source: §1 · p. 1
• Implicit alignment inherited from VLM pretraining leaves modality-specific similarity scales and local geometry insufficiently calibrated.Contrastive fine-tuning may not fully correct these mismatches.Source: §1 · pp. 1–2
• Its three modules calibrate logits, select harder negatives with debiasing, and align batch covariances across modalities.Together, they target contrastive scale, negative hardness, and second-order shared-space geometry.Source: §1 · p. 2
• e5-omni adapts VLMs without changing their backbone architecture through explicit alignment.Source: §1 · p. 2

Overview

• e5-omni maps heterogeneous omni-modal inputs into a shared embedding space while preserving the backbone and adding a lightweight training recipe.Matched pairs are encouraged to be close and mismatched pairs separated.Source: §2 · p. 2, §2.1 · p. 3

Modality-aware Temperature Calibration

• A single global temperature can make modality-specific logits overly sharp or flat, producing imbalanced gradients and unstable optimization.The method addresses this with learnable modality-aware scaling and symmetric pairwise temperatures.Source: §2.2 · p. 3

Controllable Negative Curriculum

• The negative curriculum progressively retains fewer, harder negatives, while debiased contrastive learning reduces the impact of potential false negatives.The schedule starts with more negatives, then increases hardness after a warmup period.Source: §2.3 · p. 4

Training Objective

• The final objective combines the debiased contrastive loss with a CORAL-style covariance regularizer.The covariance term is controlled by λcoral.Source: §2.3 · p. 4, §2.4 · p. 4

Batch Whitening and Covariance Alignment

• Batch whitening and covariance alignment regularize mismatched second-order statistics that can distort similarity geometry and destabilize ranking.A shared whitening transform is applied to query and target embeddings, followed by covariance regularization.Source: §2.4 · p. 4

Evaluation Setup

• MMEB-V2 spans 9 meta-tasks and 78 tasks, while AudioCaps contains about 4.4K text–audio pairs.MMEB-V2 uses Hit@1 for image/video tasks and NDCG@5 for visual-document tasks; AudioCaps uses Recall@1.Source: §3.1.3 · p. 6

Overall Results

• e5-omni outperforms strong bi-modal and omni-modal baselines on both MMEB-V2 and AudioCaps.The result supports the effectiveness of the proposed framework across the two evaluated benchmarks.Source: §3.2 · p. 6
• The method improves image/video and visual-document retrieval, while also achieving higher AudioCaps Recall@1 than omni-modal baselines.The reported gains extend across heterogeneous visual inputs and audio retrieval.Source: §3.2 · p. 6

Ablation Study

• Removing any alignment component degrades performance on MMEB-V2 and AudioCaps, indicating that the techniques are complementary.The ablation removes one design choice at a time while keeping the other settings fixed.Source: §3.3 · p. 6, Table 3 · p. 6

Embedding-space Diagnostics

• The full model yields smaller centroid and covariance gaps than e5-omni w/o. alignment in VOC2007 diagnostics.PCA overlap measures first- and second-order mismatch, while covariance heatmaps further indicate reduced second-order discrepancy.Source: §3.4 · p. 7, Figure 3 · p. 7, Figure 4 · p. 7
• The covariance-difference heatmap shows fewer high-magnitude entries for e5-omni, providing distribution-level evidence of a more consistent shared embedding space.The heatmap visualizes entrywise query–target covariance differences after projection to 32D.Source: §3.4 · p. 7, Figure 4 · p. 7

Sensitivity Analysis

• Hyperparameters are tuned on 1K-sample validation splits, with MMEB-V2 test performance reported in Fig. 5.The figure reports overall scores using the same metric as Table 1.Source: §3.5 · p. 7, Figure 5 · p. 8

Backbone Generalization

• Applying e5-omni to multiple VLMs with different sizes and architectures consistently improves MMEB-V2 results over corresponding w/o. alignment baselines.The backbone comparison omits AudioCaps because few public VLMs support audio inputs out of the box.Source: §3.6 · p. 8

Bi-modal Embedding Models

• Early bi-modal embedding models align text and images in a shared space using dual-encoder contrastive learning.CLIP is cited as an example trained on paired image–caption data at scale.Source: §4 · p. 8

Omni-modal Embedding Models

• Recent omni-modal models extend coverage to video and audio through unified embedding architectures built on VLM backbones.Examples include Omni-Embed-Nemotron and LCO-Embed.Source: §4 · p. 8

Research Gap

• These omni-modal models largely rely on implicit VLM-pretraining alignment and typically lack explicit calibration and alignment mechanisms.The limitation becomes more critical as the number of supported modalities grows.Source: §4 · p. 8
• e5-omni introduces a lightweight plug-and-play explicit-alignment recipe to improve robustness under mixed-modality training.The comparison is made against omni-modal models that broaden modality coverage but rely primarily on implicit alignment.Source: §4 · p. 8, Table 4 · p. 8

Conclusion

• e5-omni turns off-the-shelf VLM backbones into unified omni-modal embedding models without modifying the backbone.The recipe adds temperature calibration, a debiased negative curriculum, and batch whitening with covariance alignment.Source: §5 · p. 8
• Experiments on MMEB-V2 and AudioCaps show consistent gains over strong baselines, supporting explicit alignment for omni-modal embeddings.The conclusion identifies explicit alignment as the central contribution of the recipe.Source: §5 · p. 8

Scope of alignment

• The recipe targets retrieval similarity geometry and optimization, not higher-level reasoning or compositional understanding.Therefore, gains may be smaller on tasks requiring multi-step inference beyond embedding similarity.Source: Limitations · p. 9

Dependence on batch statistics

• Batch whitening and covariance alignment depend on mini-batch estimates whose quality varies with batch size, modality composition, and distributed training.Estimates may remain noisy for small or highly imbalanced omni-modal batches despite jitter and shrinkage stabilization.Source: Limitations · p. 9

Training and evaluation coverage

• Evaluation covers MMEB-V2 and AudioCaps, leaving broader audio/video domains, long-horizon retrieval, and real-world multimodal corpora for future assessment.MMEB-V2 covers text/image/video retrieval, while AudioCaps covers text–audio retrieval.Source: Limitations · p. 9

Overall results

• The results section compares e5-omni with strong baselines on overall MMEB-V2 performance.Detailed results are organized into summary, image, video, and visual-document evaluations.Source: A Detailed Results on MMEB-V2 · p. 12

Benchmark coverage

• MMEB-V2 contains 78 tasks spanning Image, Video, and Visual Document modality groups.The benchmark includes 36 image tasks, 18 video tasks, and 24 visual-document tasks.Source: A Detailed Results on MMEB-V2 · p. 12

Metrics and breakdowns

• Image and video task results use Hit@1, while visual-document tasks use NDCG@5.The visual-document breakdown covers document-level retrieval benchmarks such as ViDoRe.Source: A Detailed Results on MMEB-V2 · p. 12

Whitening and covariance regularization

• Whitening uses concatenated query and target embeddings to derive one transform applied to both sets before covariance regularization.A CORAL-style Frobenius penalty aligns second-order statistics, while group-wise computation and jitter δ = 10^-4 improve numerical stability.Source: B Implementation Details · p. 12

Debiased negative training

• Debiased contrastive learning is combined with curriculum masking to reduce false-negative bias during negative selection.The curriculum avoids overly aggressive hard negatives early and gradually increases negative pressure later.Source: B Implementation Details · p. 12

Hyperparameter selection protocol

• Hyperparameter sweeps use 1K-sample validation splits drawn from the corresponding training sets.Reported results use MMEB-V2 test sets for consistency with prior work.Source: B Implementation Details · p. 12

Study setup

• The curriculum sweeps use the 7B backbone with a fixed training budget and 1K-sample validation sets.Results are reported on the MMEB-V2 test set, matching the main experimental protocol.Source: C Additional Hyperparameter Studies · p. 12

Curriculum start

• The initial quantile threshold ρinit controls how many negatives are retained by the controllable negative-selection curriculum.The appendix varies ρinit as part of the curriculum sensitivity analysis.Source: C Additional Hyperparameter Studies · p. 12

Warmup cutoff

• Warmup improves curriculum stability and final accuracy relative to starting hardness increases immediately.Starting at t0 = 0 underperforms, while the best observed setting is t0 = 4000 steps, used by default.Source: C Additional Hyperparameter Studies · p. 12

Detailed benchmark tables

• Detailed MMEB-V2 tables separately report image, video, and visual-document tasks with their corresponding metrics.Image and video tables use Hit@1, while the visual-document table uses NDCG@5.Source: Table 5 · p. 13, Table 6 · p. 14, Table 7 · p. 14

Sensitivity analysis

• Figure 6 presents sensitivity to negative-curriculum settings on MMEB-V2 using the 7B backbone.The figure summarizes the additional curriculum sweep results.Source: Figure 6 · p. 13