PaperlayerView exact revision on arXiv

Source-linked AI summary

jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers

Florian Hönicke, Michael Günther, Andreas Koukounas, Mohammad Kalim Akram, Saba Sturua, Han Xiao

arXiv:2605.08384v4cs.CL

TL;DR

Multimodal search needs embeddings for text, images, audio, and video while preserving stable text embedding geometry. GELATO connects frozen modality encoders to frozen text models through compact projectors, producing competitive cross-modal models, with the small variant best-performing below 2B parameters among open-weight models supporting all four modalities.

  • Problem

    Search increasingly requires querying images, audio, and video alongside text while relying on stable embedding geometry for retrieval and related pipelines.

  • Method

    GELATO extends frozen text embedding models to vision and audio by connecting frozen modality encoders to the same text-sequence backbone through compact projectors.

  • Results

    jina-embeddings-v5-omni-small is the best-performing open-weight embedding model below 2B parameters supporting text, audio, images, and video, with particularly strong visual document retrieval.

  • Takeaways & Limitations

    GELATO provides a competitive model suite for broad cross-modality applications and suggests projector-only extension can generalize across text backbones.

  • Takeaways & Limitations

    Overall video performance remains weak, and the choice of non-text encoders is inadequately explored.

Abstract

from arXiv · show

In this work, we introduce GELATO (Geometry-preserving Embeddings via Locked Aligned TOwers), a novel approach to multimodal embedding models. We build on the VLM-style architecture, in which non-text encoders are adapted to produce input for a language model, which in turn generates embeddings for all varieties of input. We present the result: the jina-embeddings-v5-omni suite, a pair of models that encode text, image, audio, and video input into a single semantic embedding space. GELATO extends the two Jina Embeddings v5 Text models to support additional modality by adding encoders for images and audio. The backbone text embedding models and the added non-text modality encoders remain frozen. We only trained the connecting components, representing 0.35% of the total weights of the joint model. Training is therefore much more efficient than full-parameter retraining. Additionally, the language model remains effectively unaltered, producing exactly the same embeddings for text inputs as the Jina Embeddings v5 Text models. Our evaluations show that GELATO produces results that are competitive with the state-of-the-art, yielding nearly equal performance to larger multimodal embedding models.

1 Introduction

The work presents jina-embeddings-v5-omni, which extends text embedding models to image, video, and audio while preserving unchanged text inputs. GELATO uses modular encoder alignment and is evaluated for competitive performance, design rules, and training efficiency.

  • Motivation: Multimodal search increasingly requires querying images, audio, and video alongside text in retrieval and classification pipelines.Examples include screenshots, page scans, infographics, speech, music, natural sounds, and video.
  • Model suite: jina-embeddings-v5-omni extends a text embedding backbone to image, video, and audio while leaving text inputs entirely unchanged.The suite includes nano and small models based on text-only backbones.
  • Model suite: 0.24B parameters belong to the jina-embeddings-v5-text-nano base model, while jina-embeddings-v5-text-small has 0.67B parameters.The corresponding omni models are jina-embeddings-v5-omni-nano and jina-embeddings-v5-omni-small.
  • Method: GELATO aligns independently pretrained, language-aligned non-text encoders with text embedding models through small trainable projectors rather than joint retraining.The integrated encoders include vision encoders and the Qwen2.5-Omni audio encoder.
  • Contributions and evaluation: The released collection comprises two base models and eight task-specific variants for retrieval, classification, clustering, and text-matching across Small and Nano scales.The work evaluates the suite and comparable models across standard benchmarks, and analyzes projector training, encoder choice, Matryoshka truncation, and training efficiency.

2 Related Work

Prior multimodal embedding work spans contrastive image–text, VLM-derived, omni-modal, and frozen-tower approaches. GELATO is positioned against methods that commonly adapt language models, media encoders, or both.

  • Contrastive multimodal embeddings: CLIP established separately encoded image–text towers, while SigLIP, SigLIP2, EVA-CLIP, and ImageBind refined or extended contrastive multimodal alignment.Jina CLIP v1/v2 preserves text-embedding performance while supporting other media, but contrastive multimodal embedders suffer from a modal gap.
  • VLM-style embedding models: VLM-style architectures connect non-text encoders to the language model through projectors or connector modules.LLaVA, BLIP-2, Qwen2-VL, and Qwen3-VL exemplify this architecture; derived embedding models include E5-V, GME, and Qwen3-VL-Embedding.
  • VLM-style embedding models: VLM-derived embedding models demonstrate strong multimodal retrieval performance but adapt the language model, non-text media encoders, or both.This adaptation distinguishes them from approaches that preserve both components unchanged.
  • Omni-modal systems: Omni-style systems jointly train or align multiple modalities, supporting video and audio in addition to images.Examples include E5-Omni, WAVE, and LCO-Embedding-Omni.
  • Frozen-tower methods: Frozen-tower methods such as LiT and Nomic Embed Vision freeze the text encoder while adapting other media towers.MARVEL adds a CLIP visual encoder and linear projection to frozen text retriever T5-ANCE, then fine-tunes the text retriever end-to-end for multimodal retrieval.

3 Architecture

GELATO extends the frozen Jina Embeddings v5 Text backbone to image, audio, and video inputs using Qwen-derived encoders and trainable projection components. Inputs are serialized into one multimodal token sequence, with modality-specific projector and delimiter weights selected by task.

  • Encoder integration: The models add scale-matched Qwen3.5 vision and Qwen2.5-Omni audio encoders to the same frozen Jina text-sequence backbone.The small and nano variants use Qwen3.5-2B and Qwen3.5-0.8B vision encoders, respectively.
  • Projection layers: New projectors map mismatched image and audio encoder outputs into the Jina text hidden space.The audio projector maps native 1280-dimensional states to 1024-dimensional Small or 768-dimensional Nano inputs.
  • Vision projector: Vision features pass through LayerNorm, a 2×2 spatial merge, fc_vision_1, GELU, and fc_vision_2 before entering the text sequence.The spatial merge concatenates four neighboring patch embeddings and reduces spatial token count by 4×.
  • Input serialization: Each input becomes one sequence combining text tokens with modality placeholders, while videos concatenate frame segments and place extracted audio first.Mixed-modality segments are concatenated in document order.
  • Trainable components: Training updates fc_vision_2, fc_audio, and selected modality-delimiter embeddings, while projected encoder features overwrite modality placeholders.Small learns vision and audio delimiters; Nano learns only audio delimiters.
  • Task conditioning: Dynamic task selection activates the corresponding LoRA adapter, projectors, and learned special-token embeddings for retrieval, classification, clustering, and text matching.Task-specific projector and delimiter weights follow the same task-specific variation as the text adapters.

4 Training

Projector training uses bidirectional in-batch InfoNCE with Matryoshka representation learning, summing the loss across prefix dimensions. Optimization uses AdamW with specified regularization, warmup, mixed precision, and distributed training, while projectors are trained separately by modality and task.

  • Training objective: Projector training uses bidirectional in-batch InfoNCE with Matryoshka representation learning.The loss is defined over paired examples and their left and right embeddings.
  • Training objective: τ = 0.02 is used as the temperature for the contrastive objective.The temperature applies to the left and right embeddings and their prefixes.
  • Training objective: The training loss sums the InfoNCE term over Matryoshka prefix dimensions.The represented prefix dimensions are selected from K_Nano = {32, 64, 128, 256, 512, 768}.
  • Optimization and execution: AdamW uses β1 = 0.9, β2 = 0.999, weight decay 0.01, gradient clipping at ∥∇∥2 ≤ 1, and learning rate 2·10^-4 with 500 warmup steps.Training uses bf16 mixed precision, distributed data parallelism across 4 NVIDIA H100 GPUs, and a global batch size of 256 paired examples.
  • Optimization and execution: Projector training runs separately for each modality and task.The vision projector includes fc_vision_2 and, where applicable, vision modality-delimiter embeddings.

5 Evaluation

Evaluation shows that jina-embeddings-v5-omni-small is strongest on text and competitive overall, including visual document retrieval, while video and some generic modality tasks remain weaker. Cross-modal analyses find broadly shared embedding geometry, but retrieval still trails co-trained LCO models, especially for audio.

  • Evaluation suites: The evaluation covers text, image, video, and audio benchmarks, including MMTEB, MIEB, MMEB-Video, and MAEB.Image tasks include classification, clustering, visual STS, retrieval, document retrieval, compositional reasoning, and vision-centric tasks; video covers classification, VQA, retrieval, and moment retrieval.
  • Evaluation suites: Text scores use published MMTEB results from the inherited Jina Embeddings v5 Text encoders, which are shared bit-for-bit.The text encoders are inherited unchanged by jina-embeddings-v5-omni.
  • Overall benchmark results: 54.04 is jina-embeddings-v5-omni-small’s four-modality average, ranking above LCO-Embedding-Omni-3B at 53.83 and below LCO-Embedding-Omni-7B at 54.43 among comparable omni-style models.The small model has the strongest text-only performance and the best overall score among models below 5B parameters, while video lags baseline models.
  • Document retrieval: 79.25 is achieved by both jina-embeddings-v5-omni-small and jina-embeddings-v5-omni-nano on visual document retrieval, surpassing LCO-Embedding-Omni-3B at 78.24.The small model uses 0.92B active text+image-path parameters, while nano matches the score with 0.31B active parameters.
  • Cross-modal retrieval: 68.0% / 57.0% are jina-embeddings-v5-omni-small’s MS-COCO image→text and text→image Recall@1 scores, below LCO-Omni-3B at 71.6% / 58.0% and LCO-Omni-7B at 74.0% / 63.6%.The small frozen-tower model ranks third, while jina-embeddings-v5-omni-nano reaches 36.6% / 27.7%.
  • Cross-modal retrieval: 16.3% / 15.2% are jina-embeddings-v5-omni-small’s audio→text and text→audio Recall@1 scores, trailing LCO-Omni-7B at 27.5% / 29.8%.The audio gap is larger than the image gap, suggesting a weaker audio bridge into the text-aligned subspace; nevertheless, the frozen-tower models produce broadly interleaved multimodal geometry rather than disjoint modality clusters.

6 Ablation Studies

Ablations find that training only the modality projectors is sufficient for vision and audio under the tested budget, while encoder unfreezing helps only as an audio continuation. Matryoshka truncation largely preserves image, text, and audio retrieval information but degrades video more strongly at small dimensions.

  • Ablation setup: The released recipe trains only fc_vision_2 and fc_audio, keeping the encoders and remaining projector layers frozen while widening the trainable set is evaluated as an alternative.The ablation setup starts from jina-embeddingsv5-omni-small-retrieval, uses global batch 128, and runs for 5 000 optimizer steps.
  • Audio ablations: Audio’s fc_audio-only recipe reaches 0.398, while unfreezing the encoder from step 0 reaches 0.367; a two-stage continuation reaches 0.419, gaining 0.022 over I.The gain accumulates gradually during the continuation stage at a 20×-reduced learning rate.
  • Matryoshka truncation: Image and text embeddings lose roughly 0.18–0.21 nDCG@10 at 32 dimensions, while audio preserves most of its score at 256 dimensions and video degrades more at small dimensions.Curves remain near-flat from full dimension to roughly 128, with degradation becoming visible below 64.

7 Conclusion

GELATO extends frozen text embedding models to multiple modalities through compact projector layers while preserving text-only performance. The resulting jina-embeddings-v5-omni models show strong multimodal retrieval performance, and the ablations motivate projector-only alignment as an initialization for richer training.

  • Contributions: GELATO connects frozen modality-specific encoders to frozen text embedding models through compact, easily trained projectors.A single projector layer per modality can extend text-only models to photos, documents, video, speech, music, and sounds while preserving text-only performance.
  • Results: jina-embeddings-v5-omni-small is the best-performing open-weight embedding model below 2B parameters supporting text, audio, images, and video.It is particularly strong on visual document retrieval against comparable modality-specific and VLM-derived embedders.
  • Extensibility: Extending two text embedding models with different backbone architectures suggests that GELATO is an extensible strategy.The jina-embeddings-v5-omni-small and jina-embeddings-v5-omni-nano models extend completely different text embedding models.
  • Limitations and future work: Ablations suggest projector-only alignment can provide a compatibility-preserving initialization for richer multimodal training.Future work includes exploring non-text encoder choices and jointly training projectors for multiple modalities under different conditions.
  • Limitations and future work: jina-embeddings-v5-omni is comparatively closer to the baselines on moment retrieval.The conclusion identifies this comparison as an area where the model is relatively less differentiated.
Loading 2605.08384v4…