Source-linked AI summary
LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning
Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, Jinsong Su
TL;DR
LMM-based embedding models trained with standard InfoNCE struggle to distinguish positive from hard negative pairs because their similarity distributions overlap. LLaVE addresses this with hardness-weighted contrastive learning and cross-device negative gathering, achieving stronger MMEB performance and zero-shot transfer to video retrieval. Its main scope boundary is that it is trained only on image-text data, leaving video generalization room for improvement.
Problem
Standard InfoNCE-trained LMM embedding models exhibit overlapping positive and negative similarity distributions, making hard negative pairs difficult to distinguish.
Method
LLaVE uses hardness-weighted contrastive learning with adaptive negative weights and cross-device negative gathering.
Results
LLaVE achieves consistent improvements across MMEB metrics, with LLaVE-7B reaching a 70.3 overall score and outperforming MMRet by 6.2 points.
Takeaways & Limitations
The framework improves positive-negative separation, scales with model size, and transfers zero-shot to text-video retrieval despite image-text training.
Takeaways & Limitations
LLaVE is trained only on image-text data, and its zero-shot generalization to video embedding tasks still has significant room for improvement.
Abstract
from arXiv · showhide
Universal multimodal embedding models play a critical role in tasks such as interleaved image-text retrieval, multimodal RAG, and multimodal clustering. However, our empirical results indicate that existing LMM-based embedding models trained with the standard InfoNCE loss exhibit a high degree of overlap in similarity distribution between positive and negative pairs, making it challenging to distinguish hard negative pairs effectively. To deal with this issue, we propose a simple yet effective framework that dynamically improves the embedding model's representation learning for negative pairs based on their discriminative difficulty. Within this framework, we train a series of models, named LLaVE, and evaluate them on the MMEB benchmark, which covers 4 meta-tasks and 36 datasets. Experimental results show that LLaVE establishes stronger baselines that achieve state-of-the-art (SOTA) performance while demonstrating strong scalability and efficiency. Specifically, LLaVE-2B surpasses the previous SOTA 7B models, while LLaVE-7B achieves a further performance improvement of 6.2 points. Although LLaVE is trained on image-text data, it can generalize to text-video retrieval tasks in a zero-shot manner and achieve strong performance, demonstrating its remarkable potential for transfer to other embedding tasks.
1 Introduction
Existing dual-encoder vision-language models struggle with complex multimodal tasks, while LMM-based embeddings support richer inputs but still confuse positive and hard negative pairs under standard InfoNCE. LLaVE addresses this by weighting negatives according to difficulty and improves discriminative representation learning.
- Dual-encoder vision-language models encode images and text separately, causing poor performance on interleaved image-text retrieval.
- LMM-based embedding models naturally support interleaved text-image inputs and offer stronger multimodal semantic understanding than traditional vision-language models.
- Standard InfoNCE produces substantial overlap between positive and negative similarity distributions, especially for hard negatives.
- LLaVE assigns larger adaptive weights to harder negative pairs, making them more influential during training.The framework also decouples hardness estimation from the policy model and gathers negatives across devices.
- The framework increases the similarity gap between positive and negative pairs, indicating more discriminative multimodal representations.
- LLaVE models are trained from open-source LMMs at multiple scales and evaluated on MMEB, with LLaVE-0.5B comparable to VLM2Vec (Phi-3.5-V-4B).
2 Preliminary Study
The preliminary study examines multimodal contrastive learning and finds that standard InfoNCE leaves positive and negative similarity distributions substantially overlapping, especially for hard negatives. This motivates explicitly strengthening learning on negative pairs according to their difficulty.
- Universal retrieval requires query-target similarity to exceed similarity with other negative candidates across image, text, and interleaved image-text inputs.
- Contrastive learning pulls positive query-target representations together while pushing negative representations apart.
- Standard InfoNCE obtains separate query and target representations from an LMM and optimizes their cosine similarities within a minibatch.
- The preliminary model defines the five highest-similarity nonpositive pairs as hard negatives and the five lowest-similarity pairs as easy negatives.
- InfoNCE models show similarity differences of no more than 0.09 between positive and negative pairs, with differences of 0.02 on CIRR and Wiki-SS-NQ.
- Smaller positive-negative similarity differences coincide empirically with lower precision, motivating stronger learning for negatives of varying difficulty.
3 Our Framework
The framework combines hardness-weighted contrastive learning with cross-device negative-sample gathering. It uses reward-model feedback to emphasize difficult negatives and expands the negative set across devices without significantly increasing memory consumption.
- 3.1 Hardness-Weighted Contrastive Learning: The framework relates preference learning to contrastive learning through pairwise target preferences and their one-to-N InfoNCE-compatible extension.The embedding model is treated as both the reward model and policy model to directly optimize the embedding representation.
- 3.1 Hardness-Weighted Contrastive Learning: Hardness-weighted contrastive learning assigns larger weights and heavier penalties to more difficult negative pairs.The revised objective uses learning-difficulty weights so challenging negatives contribute more strongly to training.
- 3.1 Hardness-Weighted Contrastive Learning: The reward model estimates negative-pair difficulty, while stop-gradient updates keep it aligned with the policy model after each step.The reward model does not backpropagate and can alternatively use a structure other than the policy model.
- 3.1 Hardness-Weighted Contrastive Learning: Negative-pair gradients are proportional to reward-weighted terms, giving harder negatives a more significant role in gradient updates.This gradient relationship explains how estimated difficulty changes the learning contribution of negative pairs.
- 3.2 Cross-Device Negative Sample Gathering: Cross-device gathering increases each device’s negative-pair count by the device-number factor K while avoiding a substantial increase in memory consumption.Each device uses targets gathered from other devices for similarity calculation and loss computation.
4 Experiments
Experiments evaluate LLaVE across MMEB’s four meta-tasks, ablations, zero-shot video retrieval, and qualitative hard-sample retrieval. LLaVE-7B reaches 70.3 overall, while ablations show benefits from balanced data, diverse negatives, and hardness weighting.
- Experimental Setup: MMEB evaluates models on 20 in-distribution and 16 out-of-distribution datasets spanning classification, VQA, multimodal retrieval, and visual grounding.The study reports average Precision@1 over corresponding datasets.
- Main Results: LLaVE-7B achieves a 70.3 overall score, outperforming MMRet by 6.2 points and VLM2Vec by 4.5 points.It also reaches 91.9 in grounding, 65.4 in VQA, and 65.7 in classification.
- Main Results: LLaVE performance scales consistently with model size, while LLaVE-2B reaches 65.2 and surpasses pretrained MMRet-7B.LLaVE-0.5B is already comparable to VLM2Vec (Phi-3.5-V-4B).
- Ablation Study: Freezing the image encoder improves out-of-distribution performance but slightly reduces in-distribution performance, whereas freezing the projector harms performance.The authors attribute projector degradation to the need for re-adaptation when converting an LMM into an embedding model.
- Ablation Study: Increasing the sampling limit to 100K improves in-distribution performance but reduces generalization, favoring balanced data across meta-tasks.With limited data, the model’s generalization ability is constrained.
- Ablation Study: Cross-device negative gathering improves IND by 8.1, OOD by 0.1, and overall performance by 4.5.The result highlights the importance of diverse negative samples.
- Ablation Study: Hardness-weighted contrastive learning adds 1.4 points on OOD datasets and increases the similarity gap between positive and negative pairs.Focal-InfoNCE slightly improves OOD performance but reduces in-distribution performance.
- Zero-shot Video Retrieval: Without text-video contrastive training, LLaVE-7B still improves MSR-VTT R@1, R@5, and R@10 by 2.1, 2.5, and 1.4 points, respectively.The video evaluation uses sampled frames and expanded input length to enable video embedding.
5 Related Work
Related work progresses from dual-encoder multimodal representations to LMM-based embeddings and hardness-aware contrastive learning. The paper positions its method as dynamically weighting negative pairs with a decouplable reward model.
- Multimodal Embeddings: Dual-encoder vision-language models provide unified representations but struggle with interleaved image-text retrieval and instruction-following multimodal retrieval.Their separate image and text encoders limit performance on more complex multimodal tasks.
- LMM-based Multimodal Embeddings: E5-V and VLM2Vec transform LMMs into multimodal embedding models through contrastive learning, leveraging multimodal understanding and interleaved input support.Several concurrent studies further explore LMMs for multimodal embeddings.
- Contrastive Learning: Contrastive learning distinguishes positive from negative samples, and prior work shows that more negative samples can improve representation learning.The paper focuses specifically on estimating negative-pair hardness and weighting negatives accordingly.
- Contrastive Learning: The proposed hardness-weighted approach uses a reward model to estimate negative-pair difficulty and applies weighting only to negative pairs.The reward model can be decoupled from the policy model.
6 Conclusion
The paper identifies hard-negative discrimination as a challenge for InfoNCE-trained LMM embeddings and proposes a framework combining hardness weighting with cross-device negatives. Experiments support improved positive–negative discrimination and broader transfer potential.
- Conclusion: InfoNCE-trained LMM embedding models struggle to distinguish hard negative pairs.The conclusion frames this finding as the paper’s preliminary study.
- Conclusion: The framework combines hardness-weighted contrastive learning with cross-device negative sample gathering to improve learning across negative-pair difficulty levels.The stated goal is to enhance the model’s capacity to distinguish positive and negative pairs.
- Conclusion: Experimental results and in-depth analyses validate the effectiveness of the proposed framework.The conclusion states that the framework significantly improves positive–negative discrimination.
- Future Work: The authors plan to construct a universal multimodal benchmark for video-text retrieval and release the models and code.The planned benchmark targets more universal multimodal embedding models.
Limitations
LLaVE is trained only on datasets combining text and image modalities, limiting its direct training coverage of video. Although it generalizes to video-containing embedding tasks zero-shot, substantial improvement remains possible.
- LLaVE is trained only on embedding datasets containing arbitrary combinations of text and image modalities.
- LLaVE generalizes to embedding tasks involving video in a zero-shot manner.
- Significant room remains to improve LLaVE’s zero-shot generalization to video embedding tasks.
A.1 Training Details
The training details include memory- and acceleration-oriented techniques, while resource constraints affected the hardware used for LLaVE-7B training and leave room for faster training with A100 GPUs.
- LLaVE training uses Gradient Checkpointing and Flash Attention to save memory and accelerate training.
- Resource constraints led to training LLaVE-7B on 910B GPUs.
- Training time is expected to decrease substantially on A100 GPUs.
A.2 The Impact of Different Batch Sizes
With one training epoch, increasing batch size improves overall performance, but larger batches do not provide further gains without increasing the training schedule. Similarity distributions remain relatively stable across batch sizes.
- Increasing batch size improves overall performance when training for one epoch.
- Larger batch sizes do not yield further performance improvements when the total number of training epochs remains unchanged.
- The similarity distribution remains relatively stable across different batch sizes.
A.3 The Impact of Training Epochs
Training duration and hardness-weighting affect LLaVE performance: longer training eventually harms out-of-domain performance, while appropriate hard-negative weighting improves both in-domain and out-of-domain results. The method is also evaluated across additional LMM backbones on MMEB.
- A.3 The Impact of Training Epochs: After three epochs, overfitting degrades out-of-domain performance and limits further overall improvement.The authors therefore train for one epoch to balance efficiency and generalization.
- A.4 Hyperparameter Analysis: Increasing α improves both IND and OOD performance until performance stops increasing.
- A.4 Hyperparameter Analysis: Hardness-weighted contrastive learning consistently outperforms training without hardness weighting, where α=0.
- Generalization Across LMMs: The proposed methods are extended to Qwen2-VL models at 2B and 7B scales and evaluated on MMEB.