PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillation

Mohammed Irfan Kurpath, Jaseel Muhammad Kaithakkodan, Sahal Shaji Mullappilly, Ivan Laptev, Hisham Cholakkal

arXiv:2610.02148v1cs.CV

TL;DR

Text embedding models can lose retrieval quality when extended to new modalities, while existing omni-modal embedders use multi-billion-parameter models. Omni-Embed-Mini aligns media to dense-caption embeddings from a frozen text backbone; its 2.3B variant edges gemini-embedding-2 on the overall-modality average, while caption quality remains an important limitation.

  • Problem

    Extending text embedders to new modalities can degrade text retrieval, while existing omni-modal embedders compensate with multi-billion-parameter models.

    Source: §1 · p. 2

  • Method

    Omni-Embed-Mini aligns media embeddings to dense-caption targets produced by the same frozen backbone used for text.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    The 2.3B variant edges gemini-embedding-2 on overall-modality average (51.39 against 49.51), while the 0.9B model is ∼2.7× to 9.5× smaller than the compared open omni-modal embedders.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Keeping text weights bit-identical means training cannot regress the text path while the 0.9B model adds five modalities.

    Source: §1 · p. 2

    Caption quality matters: replacing dense cascaded captions with original dataset captions costs 7.01 points on the mean of speech, audio, image, and video, and 4.8 nDCG@10 on visual documents.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Extending a text embedding model to new modalities typically degrades text retrieval quality, and existing omni-modal embedders compensate with multi-billion parameters. We present Omni-Embed-Mini, a 0.9B-parameter model that maps text, speech, audio, images, video, and visually-rich documents into a single shared cosine space without updating any text-side parameter. Our key insight is that the teacher signal requires no separate embedding model: each media sample is paired with a dense cascaded caption, and the teacher target is simply the frozen backbone's own embedding of that caption. Because teacher and student share the same backbone weights, they inhabit byte-identical geometry, and lightweight projectors plus phased LoRA adapters on the modality encoders suffice for alignment. Training combines a Matryoshka SigLIP contrastive loss with an online hybrid hard-negative miner whose negatives sharpen as the encoder improves. The recipe carries over to a 2.3B variant by swapping in a native vision-language backbone. Omni-Embed-Mini-0.9B keeps its text weights bit-identical to the backbone, so training cannot regress text retrieval (49.57 nDCG@10 on MTEB-v2 BEIR-8), while extending it to five additional modalities, and is ~2.7x to 9.5x smaller than every open omni embedder we compare against. The 2.3B variant is competitive with the closed gemini-embedding-2, edging ahead of it on the overall-modality average. Models, code, data and evaluation harness are on our project page: https://omniembed.cvmbzuai.com

1 Introduction

Omni-Embed-Mini addresses text forgetting and model size in multimodal retrieval by aligning media to a frozen text backbone. Its 0.9B model adds five modalities while leaving the text path untrained, and the 2.3B variant edges gemini-embedding-2 on the overall-modality average.

1 Introduction

  • The paper frames multimodal expansion as a trade-off between text-side catastrophic forgetting and the larger models used to absorb additional modalities.It names this coupled problem the multimodal expansion dilemma and notes that text remains critical for real-world search.Source: §1 · p. 2
  • Omni-Embed-Mini uses each media sample’s dense cascaded caption as a teacher target, embedded by the same frozen backbone that processes the student.A modest projector and phased LoRA adapters on modality encoders close the modality gap without changing the shared backbone weights.Source: §1 · p. 2
  • ∼2.7× to 9.5× smaller than the open omni-modal embedders listed, the 0.9B model keeps its text path untrained while covering five additional modalities, though it trails larger baselines on media retrieval.The paper reports 49.57 nDCG@10 on MTEB-v2 BEIR-8 for the text path and says the text weights are bit-identical to the stock 0.6B backbone.Source: §1 · p. 2
  • 51.39 against 49.51 on the overall-modality average puts the 2.3B variant ahead of gemini-embedding-2, leading on image, video, and visual documents while trailing on text, speech, and audio.The comparison is reported for the closed gemini-embedding-2.Source: §1 · p. 2
  • The recipe combines Matryoshka SigLIP, online hybrid hard-negative mining, and phased LoRA, and carries over to 2.3B by swapping in a native vision-language backbone.The 0.9B text branch remains bit-identical to the stock 0.6B backbone.Source: §1 · p. 2

2 Related Work

Related work spans modality-specific encoders, anchored alignment, and jointly trained omni-modal embedders. Omni-Embed-Mini distinguishes itself with dense-caption distillation to an unchanged backbone, while reusing established training components.

2 Related Work

  • Per-modality reference systems include text encoders such as Qwen3-Embedding and audio encoders such as Whisper and Dasheng, which lack an aligned text tower.The vision-language systems listed adapt pretrained models for retrieval, usually with contrastive training, but do not handle audio.Source: §2 · pp. 2–3
  • Table 1 reports 68.32 M trained parameters for the 0.9B model, against 935.35 M at inference; its text backbone is never updated.The trained parameters comprise 65.37 M for projectors and pooler and 2.95 M of r=16 LoRA adapters.Source: Table 1 · p. 3
  • e5-omni-7B leads Omni-Embed-Mini-2.3B on overall average (52.93 against 51.39), while Omni-Embed-Mini-2.3B edges e5-omni-3B (51.39 against 49.28).Omni-Embed-Mini leads on video (55.18 against 32.92 and 44.73) and uses half to a quarter of e5-omni’s parameters; its 0.9B variant holds a strong text score.Source: §2 · p. 3
  • Omni-modal baselines target shared spaces across text, image, and audio, with most also supporting video; Gemini Embedding 2 is closed and its parameter count is undisclosed.Among compared open omni embedders, Omni-Embed-Mini alone leaves its backbone entirely frozen, adding modalities through external encoders and projectors.Source: §2 · p. 3
  • Unlike ImageBind and LanguageBind, Omni-Embed-Mini uses dense captions as fixed per-sample distillation targets, not just contrastive text or naturally co-occurring pairs.The same frozen backbone serves as teacher and text encoder; interleaved multimodal queries are native to the architecture, but their contribution is not experimentally isolated.Source: §2 · pp. 3–4
  • The training recipe draws on refreshed hard-negative pools, Matryoshka representation learning, SigLIP’s pair-wise sigmoid loss, and LoRA adapters.Its hybrid miner generalizes hard-negative refresh to multimodal Matryoshka training, while the SigLIP loss scores pairs independently of batch-level softmax normalization.Source: §2 · p. 4
  • The caption-only corpus figure covers 227k effective rows across five non-text modalities, and visual-document captions average ∼6× longer than others.The captions enumerate layout, headings, tables, and charts.Source: Figure 2 · p. 4, Figure 3 · p. 4

3 Method

Omni-Embed-Mini keeps its backbone fixed and aligns modality inputs to caption embeddings produced by that same backbone. It combines phased modality adaptation, Matryoshka contrastive learning, and online hard-negative mining.

3.1 Asymmetric modality alignment

  • The backbone stays frozen; projectors train first, then encoder LoRA begins after 20% of steps to avoid adapting to unaligned projections.The text path is unmodified, and the text and vision paths of the 2.3B variant remain bit-identical to their stock backbone weights.Source: §3.1 · pp. 4–5

3.2 Architecture

  • Media inputs use modality-specific encoders and projectors before EOS pooling, while text passes through the unmodified backbone.Whisper and Dasheng outputs are temporally interleaved; image, document, and video use a shared vision path, with video spatially pooled.Source: Figure 4 · p. 5, §3.2 · p. 5

3.3 Cascaded-caption self-distillation

  • Each media sample’s dense caption supplies a cached teacher embedding from the frozen backbone, placing teacher and student in identical geometry without an auxiliary projection head.The student embedding is the EOS-pooled output for the sequence containing spliced-in media.Source: §3.3 · pp. 5–6

3.4 Hybrid Matryoshka contrastive learning

  • SigLIP contrastive losses are computed across truncated Matryoshka dimensions and combined with cosine self-distillation between student and teacher representations.The dimensional weighting favors aggressive truncations.Source: §3.4 · p. 6

3.5 Online hybrid hard-negative mining

  • Online text and modality-specific media miners refresh hard negatives from the current model, making negatives harder as the encoder improves.The text-side index stays stable because the text branch receives no gradient; the media-side index is refreshed.Source: §3.5 · p. 6

4 Training Data: Dense Caption Grounding

The training-data contribution is a dense, grounded caption signal for frozen-backbone distillation. The authors treat caption density as central to the recipe and use modality-specific prompts to prepare captions.

4 Training Data: Dense Caption Grounding

  • Dense captions are treated as a central ingredient because they expose more sample facets than short, crowdsourced captions.The paper identifies objects, attributes, layout, temporal events, acoustic environment, and OCR text as captioned facets.Source: §4 · p. 6
  • Filtering the caption corpus yields 242,080 single-turn samples, then 227,704 effective training rows after duration filtering.The source corpus contains 590,858 total rows; omni-split and chat samples are excluded because their narrow answers are unsuitable distillation targets.Source: §4 · p. 6
  • Captions come from one low-temperature Qwen3-Omni call per row, with source captions used as grounding references except for visual documents.Visual-document prompts use an expert document analyst because no reference caption exists.Source: §4 · p. 6
  • Machine-generated caption noise may propagate to learned geometry; prompts and pooled-embedding targets limit propagation but do not verify factuality.The paper says captions are not filtered for factuality.Source: §4 · pp. 6–7

5 Experiments

Across the reported omni-modal comparison, the 2.3B variant edges gemini-embedding-2 on the overall-modality average, while the 0.9B model is the smallest open omni embedder. The evaluation uses English benchmark subsets and reports a video-evaluation constraint from the benchmark’s pre-extracted frames.

5.1 Benchmarks

  • The benchmarks cover English text retrieval, audio tasks, image and video tasks, and visually-rich documents, using nDCG@10, mixed metrics, or hit@1 as specified.The evaluation uses BEIR-8 for MTEB-v2 and supported English tasks for MMEB, MAEB, and ViDoRe; identifier-level audits found no shared items with training sources.Source: §5.1 · p. 7

5.2 Setup

  • MMEB-V2 video evaluation embeds eight benchmark-pre-extracted frames as images and mean-pools them, a uniform comparison protocol that caps achievable video performance.Table 2 compares dense and source-shipped captions on a 0.9B mining-ablation checkpoint, not the final checkpoint.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 Main results

  • 51.39 to 49.51 on the overall-modality average: the 2.3B variant leads closed gemini-embedding-2 and leads the omni block on video.It is competitive on image and visual documents; speech and audio remain the weakest modalities.Source: §5.3 · p. 7, Figure 5 · p. 8
  • The 0.9B model is the smallest open omni embedder, about 2.7× smaller than the next entry and 9.5× smaller than the 7B comparator.Its text score is 49.57 on Text, second among open omni embedders only to omni-embed-nemotron-3B at 50.48.Source: §5.3 · p. 7
  • The 0.9B variant scores 26.29 on Image and 18.48 on Video, while the 2.3B variant scores 64.80 on Image and 55.18 on Video.The authors say this is consistent with vision-encoder capacity, rather than the alignment recipe, limiting the 0.9B model.Source: §5.3 · p. 7

6 Ablations and Analysis

The ablations examine backbone freezing, caption quality, mining, and Matryoshka behavior. Combined mining performs best overall, while the other analyses show how freezing preserves text retrieval and how captions and embedding dimension affect media retrieval and index size.

6.1 Backbone invariance

  • At 0.9B, backbone LoRA improves every media modality but costs text; at 2.3B, it degrades every modality, especially speech and audio.At 0.9B, media gains are +1.10 to +5.66 and text drops by 3.13 points; the frozen model scores 49.57 nDCG@10 on MTEB-v2 BEIR-8 with bit-identical text weights.Source: Table 3 · p. 8, §6.1 · pp. 8–9

6.3 Hybrid mining

  • Combined mining improves the mean over five media modalities by +2.6 versus no mining and gives the best overall score.Text-only mining slightly hurts several modalities, while media-only gives small improvements on most; visual-document retrieval varies by at most 0.6 nDCG@10 across mining configurations.Source: Figure 6 · p. 9, §6.3 · p. 9

6.2 Dense vs. original captions

  • Dense captions outperform original captions most on video (+14.06), while speech has the smallest gap (+2.68).On ViDoRe-V3, replacing dense captions with original captions costs 4.8 nDCG@10, from 46.98 to 42.21.Source: §6.2 · p. 9, §6.3 · p. 9

6.3 Hybrid mining

  • Across three seeds, the five trained modalities have reported mean ± standard deviation scores, while text is identical at 49.57.The trained-modality scores are image 27.48 ± 0.43, general audio 33.96 ± 0.59, video 19.82 ± 0.95, speech 43.10 ± 1.03, and visual documents 45.48 ± 1.76.Source: §6.3 · p. 9

6.4 Matryoshka behaviour

  • At 128 dimensions, truncating the 0.9B embedding cuts a 10M-item float32 omni index from ~41 GB to ~5 GB, at modest quality cost.Averaged-probe AVG falls only slightly from 1024 to 128 for both variants, while omni-specific baselines fall more steeply.Source: §6.4 · p. 9

7 Conclusion

Omni-Embed-Mini aligns six modalities in a shared space through self-distillation against a frozen text backbone. The model preserves text retrieval while remaining competitive with larger open omni embedders, though cross-modal evaluation remains an open gap.

7 Conclusion

  • A shared frozen text backbone supports self-distillation that binds text, speech, audio, images, video, and visually-rich documents into one retrieval space.The 0.9B model remains competitive with open omni embedders ∼2.7× to 9.5× larger.Source: §7 · p. 9
  • The 2.3B variant carries the approach to a larger backbone, and future text-embedder improvements can in principle be adopted by rerunning projector and LoRA training.The paper illustrates the backbone swap from the 0.6B text encoder to Qwen3-VL-Embedding-2B.Source: §7 · p. 10
  • Existing benchmarks each probe one modality slice, and none evaluates arbitrary query-to-corpus retrieval across all six modalities simultaneously.The conclusion also identifies learned audio-token mixing, specialist encoder routing, and native video pooling as open threads.Source: §7 · p. 10

Limitations

The frozen text backbone and caption-based supervision constrain what the shared space can represent, while audio design, model scale, and benchmark coverage bound the reported scope.

Limitations

  • Replacing dense captions with original captions costs 7.01 points on mean speech, audio, image, and video retrieval, and 4.8 nDCG@10 on visual documents.Captions are not checked for factuality, and the authors did not run a controlled caption-perturbation study.Source: Limitations · p. 10
  • The frozen text backbone may underrepresent difficult-to-verbalise phenomena, including complex acoustic textures, abstract visual patterns, and fine-grained temporal cues.Teacher captions may also pass hallucinations, omissions, and biases to the student; their quality is not verified against the source media.Source: Limitations · p. 10
  • Fixed temporal interleaving of speech and environmental-audio tokens may be suboptimal when inputs are dominated by only one audio type.The compact 0.9B and 2.3B scales may also limit complex reasoning-heavy multimodal retrieval.Source: Limitations · p. 10
  • Available benchmarks lack unified any-to-any retrieval across all six modalities, so results may not generalise across query-corpus combinations, languages, domains, data qualities, or workflows.The authors identify benchmark coverage as a boundary on generalisation.Source: Limitations · p. 10

Ethical Considerations

The paper frames Omni-Embed-Mini as a research and educational tool, and advises validation and human supervision before sensitive use. Its appendix documents the benchmark protocols and evaluated text subset.

Ethical Considerations

  • Omni-Embed-Mini is intended for research and education, not autonomous high-stakes decisions without appropriate validation and human supervision.The authors identify risks including inaccurate retrieval, bias amplification, and privacy leakage, and recommend safety, fairness, privacy, and robustness evaluation.Source: Ethical Considerations · p. 10

Appendix

  • The appendix expands the paper with benchmark protocols, caption-data generation, dataset licenses and overlap audits, mining, safeguards, ablations, loss equations, hyperparameters, and further limitations.It also includes per-task benchmark results and qualitative caption examples.Source: Appendix · p. 14

A Detailed benchmark protocols

  • The protocol describes evaluated task subsets and metrics alongside public-model results; published parameter counts are used, with undisclosed closed-model sizes marked “?”.Closed-source models with undisclosed sizes are not included in size-matched comparisons.Source: A Detailed benchmark protocols · p. 14

A.1 MTEB-v2 (8-task English BEIR subset)

  • The MTEB-v2 evaluation uses an eight-task English BEIR subset spanning short and long queries and corpora, with nDCG@10 averaged across tasks.The subset comprises ArguAna, three CQADupstack tasks, FiQA-2018, NFCorpus, SCIDOCS, and SciFact.Source: §A.1 · p. 14

A.2 MAEB (22-task English subset)

This appendix details modality-specific benchmarks, caption generation and safeguards, corpus construction, and overlap checks. It also reports per-modality rankings and describes the mining subsystem.

A.2 MAEB (22-task English subset)

  • 49.57 MTEB nDCG@10 is reported for Omni-Embed-Mini-0.9B, whose text weights remain bit-identical while it adds five modalities; it is 2.7× to 9.5× smaller than each open omni baseline.The 2.3B variant inherits the Qwen3-VL-Embedding-2B image, video, and visual-document strengths.Source: Figure 7 · p. 15

A.2 MAEB (22-task English subset); A.3 MMEB-V2 (16-task English subset)

  • The MAEB subset contains 22 English tasks split into 10 audio and 12 speech tasks; MMEB-V2 uses 10 image and six video tasks, excluding NExTQA and EgoSchema video QA.MAEB averages scored tasks within each modality; MMEB-V2 reports hit@1 averages for image, video, and combined tasks.Source: §A.2 · p. 16, §A.3 · pp. 16–17

A.4 ViDoRe-V3 (7 English domains)

  • ViDoRe-V3 evaluation covers seven English domains and reports mean nDCG@10, rather than the leaderboard’s nDCG@5 headline metric.The French-only Finance task is excluded.Source: §A.4 · p. 17

B Cascaded-caption data generation; B.1 Modality system prompts

  • Each caption row comes from a multimodal Qwen3-Omni call using raw media, a modality-specific prompt, and the source caption as a grounding prior; prompts request dense descriptions tailored to each modality.The speech prompt uses a transcription marker that is post-processed into a span around the source transcript.Source: B Cascaded-caption data generation · pp. 17–18, §B.1 · pp. 18–19

B.4 Qualitative dense-caption examples; C Per-source datasets, licenses, citations

  • The caption slice contains 242,080 rows from 16 datasets across five modalities; generated captions are released under CC BY-SA 4.0, while source media retain their dataset licenses.The appendix also gives sample records and one qualitative example per modality.Source: §B.4 · pp. 20–21, C Per-source datasets, licenses,… · p. 21

C.1 Train/test overlap audit

  • An identifier-level audit found zero shared items across the three audited training sources and benchmark tests; the reported cross-modal results therefore remain unchanged.For MSR-VTT, the training and test identifiers are disjoint complementary 9k/1k halves.Source: §C.1 · p. 23

D Mining subsystem

  • The miner alternates text and media cycles to refresh hard-negative caches; a shared live model avoids the roughly 16 GB extra HBM cost of a separate copy.Text cycles run roughly ten times per epoch; media cycles run roughly five times per epoch.Source: D Mining subsystem · p. 23

E Caption grounding and noise safeguards

  • Caption safeguards include source-caption grounding, pooled-vector targets, a 0.92 cosine cutoff for likely false negatives, and a frozen text path; they contain noise but do not verify captions.Visual documents receive no reference-caption grounding, and their strong retrieval does not measure robustness to hallucinated or systematically biased captions.Source: E Caption grounding and noise safeguards · p. 23

F Additional ablations

The appendix adds analyses of truncation, cross-modal alignment, media-to-media comparisons, and the single-stage schedule, alongside the loss specification and training settings.

F Additional ablations

  • Across modality probes, Omni-Embed-Mini’s truncation curves remain near-monotone and close to full-dimension performance at 128 dimensions, unlike steeper, non-monotone omni-specific baselines.Vision-language embedders with native Matryoshka training degrade comparably gently.Source: Figure 8 · p. 24
  • A SciFact text-versus-speech NDCG@10 probe tracks whether the speech path reaches the text geometry, with convergence of the two curves serving as the headline indicator.The probe is monitored every 10% of an epoch.Source: F Additional ablations · p. 24
  • The shared text axis allows direct comparison of media pairings without a media-to-media training objective, yielding emergent alignment through the common reference frame.The authors liken this to ImageBind’s emergent alignment, induced here by a text anchor rather than pairwise modality contrast.Source: F Additional ablations · pp. 24–25
  • Training uses one dataset loader and one loss across two optimizer phases: projectors first, then projectors plus encoder LoRA.Dense cascaded captions carry both alignment and task signal in each training row.Source: F Additional ablations · p. 25

G Loss equations

  • The loss uses d-truncated, L2-normalised student, teacher-positive, and mined-negative embeddings, with a per-dimension SigLIP term and cosine distillation.The learned temperature is clamped to [0.01, 0.3], and the total loss adds distillation with λ = 1.Source: G Loss equations · p. 25

H Hyperparameters and training schedule

  • The reported 0.9B training setup uses eight GPUs, BF16 mixed precision, AdamW, and a cosine schedule; batch size is 32 for image, document, audio, and speech, and 8 for video.Each variant’s reported numbers come from a single checkpoint evaluated across all six modalities.Source: H Hyperparameters and training schedule · p. 25

I Additional Limitations

The reported limitations concern differing vision strategies across model variants, truncation of long visual-document captions, and video embeddings that omit temporal encoding.

I Additional Limitations

  • Cross-variant vision ablations are left to future work because the 0.9B and 2.3B variants use different vision strategies.The 0.9B variant uses an extracted ViT, while the 2.3B variant uses its backbone’s native vision module.Source: I Additional Limitations · p. 25
  • The 2,048-token context truncates long visual-document captions, losing upper-end detail from the contrastive signal.Truncation is consistent between teacher and student paths, so cosine alignment is unaffected.Source: I Additional Limitations · p. 25
  • Per-frame mean pooling keeps video results comparable on MMEB-V2 but caps performance below what a native temporal encoder could reach.The benchmark’s eight frames per video are embedded as images and mean-pooled for all omni-style models.Source: I Additional Limitations · p. 25

Related papers

Loading 2610.02148v1…