PaperlayerView exact revision on arXiv

Source-linked AI summary

What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies

Duo Li, Haojun Pan, Zuhao Yang, Xintong Wang, Chengfu Huo, Shijian Lu, Liang Ding

arXiv:2610.15035v1cs.CV

TL;DR

The paper examines whether contrastive strategies produce meaningful gains under controlled multimodal embedding comparisons, where prior evaluations used different architectures and settings. It evaluates representative strategies on MMEB and analyzes top-1 retrieval behavior; most strategies yield little improvement, while jointly combining complementary strategies achieves the strongest performance.

  • Problem

    Different architectures and experimental settings make it difficult to isolate the effectiveness of contrastive strategies, leaving their comparative gains and failure explanations unclear.

    Source: §1 · p. 2

  • Method

    The paper compares representative strategies in a unified MMEB setting and uses an MLLM-as-a-Judge framework to categorize top-1 outcomes.

    Source: §1 · p. 2

  • Results

    B3 raises the overall MMEB score from 65.1 to 69.0, while most individual strategies provide little or inconsistent improvement over the aligned baseline.

    Source: §1 · p. 2, §3.3 · p. 5

  • Takeaways & Limitations

    Hard-negative mining, false-negative mitigation, and hard-batch construction address complementary aspects of supervision and achieve the strongest performance when jointly applied.

    Source: §5 · p. 10

Abstract

from arXiv · show

Contrastive learning has become the standard paradigm for multimodal representation learning, aligning semantically matched pairs while separating mismatched ones in a shared embedding space. Although various contrastive training strategies have been proposed across negative selection, batch construction, and gradient modulation, their actual effectiveness under controlled, reproducible conditions remains poorly understood. This motivates a fundamental question: Do sophisticated contrastive strategies consistently bring meaningful downstream gains in multimodal embedding learning? To address this, we systematically evaluate representative strategies under a unified benchmarking framework on MMEB. Surprisingly, direct evaluations reveal that most existing strategies offer marginal gains over a standard aligned baseline, with several even degrading overall performance. To uncover the underlying failure modes, we introduce an MLLM-as-a-Judge diagnostic framework that dissects top-1 retrieval behaviors into labeled positives, false negatives (unlabeled valid matches), and hard/easy negative errors. Our systematic analysis yields three key insights: (1) Gradient distortion: Upweighting gradients from high-similarity negatives consistently hurts downstream generalization; (2) Distribution illusion: Wider separation between positive and negative similarity distributions does not necessarily translate to better retrieval performance; and (3) Synergistic training: While false-negative mitigation or hard-batch construction yields limited gains in isolation, integrating hard-negative mining, false-negative mitigation, and hard-batch construction achieves optimal performance, delivering an average improvement of 6.0% on MMEB with Qwen2-VL-2B. Overall, our findings demystify popular contrastive strategies and provide actionable recipes for training universal multimodal embedding models.

1 INTRODUCTION

The paper asks whether contrastive training strategies deliver meaningful gains and what explains their success or failure under controlled comparisons. It finds that most provide little improvement, while combining complementary strategies achieves the strongest performance.

1 INTRODUCTION

  • Contrastive learning aligns semantically matched multimodal inputs and separates mismatched ones in a shared representation space.The resulting embeddings support downstream tasks including classification, retrieval, and visual question answering.Source: §1 · p. 1
  • The study addresses uncertainty about strategy effectiveness because prior evaluations use different architectures and experimental settings.It asks both how strategies compare and what explains their success or failure.Source: §1 · p. 2
  • Most strategies yield no meaningful downstream gains over a well-aligned baseline, and many degrade performance in controlled MMEB comparisons.The comparison uses Qwen2-VL-2B/7B and Qwen3.5-2B.Source: §1 · p. 2
  • The MLLM-as-a-Judge framework separates top-1 predictions into labeled positives, valid but unlabeled matches, and hard- or easy-negative errors.This provides a fine-grained diagnosis beyond aggregate benchmark scores.Source: §1 · p. 2
  • Jointly combining hard-negative mining, false-negative mitigation, and hard-batch construction achieves the strongest performance.The strategies address complementary aspects of contrastive supervision.Source: §1 · p. 2

2 RELATED WORK

Multimodal embedding research has moved from CLIP-style dual encoders toward MLLM-based models that handle more diverse inputs and tasks. Contrastive strategies span several optimization dimensions, but prior variation in architectures and experimental settings leaves their comparative effectiveness unclear.

2.1 MULTIMODAL EMBEDDING LEARNING

  • CLIP-style dual encoders align independently encoded images and text, but global alignment and independent encoding limit fine-grained grounding and instruction-conditioned representations.The cited work also identifies limitations in compositional relationships, long descriptions, and high-resolution visual details.Source: §2.1 · pp. 2–3
  • MLLM-based embedding models use instruction-following and multimodal understanding capabilities to accommodate more diverse inputs and tasks.Examples include VLM2Vec, which established MMEB across 36 datasets spanning classification, VQA, retrieval, and visual grounding.Source: §2.1 · p. 3

2.2 CONTRASTIVE TRAINING STRATEGIES FOR MULTIMODAL EMBEDDINGS

  • Contrastive strategies target gradient modulation, false-negative mitigation, batch construction, and hard-negative mining.UniME-V2, EGA, and HOBIT respectively combine several of these approaches or specialize in mining, gradient amplification, or batch selection.Source: §2.2 · p. 3
  • Differences in model architectures and experimental settings leave the strategies’ relative effectiveness and reasons for success or failure unclear.The paper therefore benchmarks representative approaches under a unified setting and diagnoses behavior beyond aggregate scores.Source: §2.2 · p. 3

3 EXPERIMENTS

In controlled MMEB experiments, B3 raises the overall score from 65.1 to 69.0, while most individual strategies produce no consistent gains. The study aligns training choices, evaluates eight strategies, and characterizes their optimization dimensions and dependencies.

3.1 CONTRASTIVE TRAINING STRATEGIES

  • The study characterizes eight strategies by dependencies and optimization dimensions, including external scoring, preprocessing, and static or adaptive signals.The dimensions are gradient modulation, false-negative mitigation, batch construction, and hard-negative mining.Source: §3.1 · pp. 3–4

3.2 BASIC SETUP

  • The aligned VLM2Vec baseline achieves an MMEB score of 65.1, comparable to LLaVE-2B’s reported 65.2 and above UniME-V2-2B’s reported 63.6.The selected configuration uses a learning rate of 1 × 10−4, up to 2,880 visual tokens, LoRA rank 16, and last-token pooling.Source: §3.2 · p. 4, Table 2 · p. 4

3.3 MMEB RESULTS

  • The overall score rises from 65.1 to 69.0 with B3, which jointly optimizes false-negative mitigation, hard-negative mining, and batch construction.B3 gains consistently across all four meta-task categories; adding EGA reduces the score from 69.0 to 68.6.Source: Table 3 · p. 5, §3.3 · p. 5
  • Gradient-modulation and false-negative-mitigation methods do not surpass the 65.1 baseline, while batch-construction results are inconsistent.LLaVE scores 63.9, EGA 64.8, GloFND 64.3, FN Risk 65.0, FALCON 64.8, and HOBIT 65.3.Source: §3.3 · p. 5, Table 3 · p. 5

4 UNDERSTANDING THE EFFECTS OF CONTRASTIVE TRAINING STRATEGIES

Across four analyses, the paper uses finer-grained top-1 judgments to explain why popular contrastive strategies often fail to improve retrieval. Its central result is that false-negative mitigation, hard-negative mining, and hard-batch construction work best together, while similarity-based gradient weighting and distributional separation alone are unreliable.

4.1 MLLM-AS-A-JUDGE ANALYSIS

  • The MLLM-as-a-Judge framework classifies top-1 predictions as labeled positives, false negatives, hard negatives, or easy negatives, separating semantic validity from error types.The evaluation samples 200 cases from each of MMEB’s 36 datasets; Semantic Valid sums labeled positives and false negatives.Source: §4.1 · pp. 5–6, Figure 1 · p. 6

4.4 RELIABLE CONTRASTIVE TRAINING REQUIRES JOINT DESIGN

  • 78.81% Semantic Valid, up from 75.36%, and an overall MMEB score of 69.0 show B3’s strongest performance when it combines false-negative treatment, hard-negative mining, and hard-batch construction.B3’s HN and EN errors fall from 17.19% and 6.85% to 15.58% and 4.94%; removing any component reduces performance.Source: Table 4 · p. 7, §4.4 · p. 9

4.2 UPWEIGHTING HIGH-SIMILARITY NEGATIVES DEGRADES PERFORMANCE

  • LLaVE/EGA reduce Semantic Valid by 1.15/0.37 and raise Semantic Error by 1.21/0.24, consistent with high-similarity gradient upweighting degrading performance.The degradation persists when EGA is added to B3; similarity reflects hardness, not whether a negative is semantically invalid.Source: §4.2 · pp. 6–7

4.3 SIMILARITY SEPARATION DOES NOT RELIABLY PREDICT PERFORMANCE

  • LLaVE and EGA shift positive-negative overlap in opposite directions yet both degrade MMEB, while B3 and LLaVE have similar overlaps but B3 scores 5.1 MMEB points higher.Aggregate distribution separation can conceal query-level ranking errors and is not a reliable standalone performance indicator.Source: §4.3 · pp. 8–9, Figure 2 · p. 8

4.4 RELIABLE CONTRASTIVE TRAINING REQUIRES JOINT DESIGN

  • False-negative treatment and batch-construction strategies yield limited, inconsistent gains in isolation, with small FN improvements offset by LP or HN errors.GloFND and FN Risk lower Semantic Valid relative to baseline, while FALCON and HOBIT both reach 75.21%, near the 75.36% baseline.Source: §4.4 · p. 9

5 CONCLUSION

The conclusion reports that most evaluated contrastive strategies provide little benefit over a well-aligned baseline, with many degrading performance across the tested models. It highlights a finer-grained diagnostic framework and the stronger results from coordinating complementary training components.

5 CONCLUSION

  • Across Qwen2-VL-2B/7B and Qwen3.5-2B, most strategies provide little improvement over a well-aligned baseline, and many degrade performance.The authors report that previously reported gains do not persist under their controlled comparisons.Source: §5 · p. 9
  • The MLLM-as-a-Judge framework distinguishes labeled positives, false negatives, and hard- and easy-negative errors beyond aggregate benchmark scores.The conclusion identifies high-similarity gradient upweighting and aggregate similarity separation as unreliable, while jointly applying three complementary interventions yields the strongest performance.Source: §5 · pp. 9–10
  • The authors present their findings as practical guidance for training universal multimodal embedding models.They conclude that false-negative mitigation, hard-negative mining, and hard-batch construction address complementary aspects of contrastive supervision.Source: §5 · p. 10

A.1 CONTRASTIVE TRAINING STRATEGIES

The appendix describes the evaluated contrastive strategies and the model backbones used to compare them. It includes approaches for gradient modulation, false-negative mitigation, negative mining, and batch construction, alongside Qwen models at different scales and generations.

A.1 CONTRASTIVE TRAINING STRATEGIES

  • VLM2Vec supplies the shared contrastive-learning baseline, encoding queries and candidates with one backbone and using other mini-batch candidates as InfoNCE negatives.All evaluated training strategies are implemented on this common baseline.Source: §A.1 · p. 13
  • LLaVE weights negatives by query-negative similarity, while EGA redistributes negative gradients according to each negative’s similarity margin from the labeled positive.Both strategies assign more weight to higher-similarity negatives.Source: §A.1 · p. 13
  • GloFND excludes negatives above dynamically estimated per-instance thresholds, whereas FN Risk down-weights high-risk negatives using similarity and neighborhood information.FN Risk uses neighborhood overlap and mutual-neighbor relationships; GloFND estimates thresholds from sampled mini-batches.Source: §A.1 · p. 13
  • FALCON schedules mini-batches by predicted hardness quantiles, while HOBIT greedily selects examples to maximize collective in-batch negative hardness.UniME-V2 uses an MLLM judge to filter false negatives, select diverse hard negatives, and provide soft semantic targets; B3 builds batches from communities of hard-negative relationships.Source: §A.1 · p. 13

A.2 MODEL BACKBONES

  • The study uses Qwen2-VL-2B for primary controlled comparisons, Qwen2-VL-7B to examine scale, and Qwen3.5-2B to test generalization across model generations and architectures.Table B.1 extends the comparison to Qwen3.5-2B and Qwen2-VL-7B, reporting mean Precision@1 across the 36 MMEB datasets.Source: §A.2 · p. 14, Table B.1 · p. 14

A.3 EVALUATION BENCHMARK

MMEB evaluation averages Precision@1 across 36 datasets and four meta-task categories, with consistent prompt formatting across models and strategies. Cross-backbone tests find B3 strongest overall, with gains over VLM2Vec on both additional backbones.

A.3 EVALUATION BENCHMARK

  • MMEB covers 36 datasets across image classification, visual question answering, image-level retrieval, and visual grounding, all formulated as retrieval tasks.The evaluation reports Precision@1 averaged across datasets, with meta-task and in- and out-of-distribution scores.Source: §A.3 · p. 14
  • The same prompt formatting is used across models, strategies, checkpoints, and ablations, controlling for prompt-format differences.Queries use task-specific instructions; textual targets use modality-level templates, while images and cropped regions retain their original instructions.Source: §A.3 · p. 14
  • Table B.3 reports mean Precision@1 for B3 component removals, with an overall average across all 36 MMEB datasets.The ablations remove false-negative mitigation, hard-negative mining, or batch construction while keeping other training and evaluation settings unchanged.Source: Table B.3 · p. 15

B.1 CROSS-MODEL COMPARISON

  • B3 improves over VLM2Vec by 1.5 points on Qwen3.5-2B and 3.3 points on Qwen2-VL-7B, with gains across most meta-tasks.Other methods provide limited or inconsistent improvements, and adding EGA to B3 reduces the overall score by 0.2 and 0.6 points, respectively.Source: §B.1 · p. 15

B.2 ROBUSTNESS OF MLLM-AS-JUDGE

The MLLM-as-Judge analysis is tested across evaluation-set sizes and judge models. The sampled subset closely matches full-set outcomes, and changing the judge produces similarly small differences.

B.2 ROBUSTNESS OF MLLM-AS-JUDGE

  • The judge checks whether the dataset-designated positive is valid, then classifies the top-1 prediction as a designated positive, false negative, hard negative, easy negative, or uncertain case.Task-specific criteria guide judgments across five MMEB task families, and constrained JSON supports automatic parsing and aggregation.Source: Figure C.1 · p. 16
  • Reducing evaluation from 36,000 cases to 7,200 changes each outcome by at most 0.21 percentage points.The authors conclude that the 7,200-case evaluation is representative of the full evaluation set.Source: §B.2 · p. 16
  • Replacing Qwen3.8-Flash with GPT-5.4 on the 7,200-case subset changes positive and negative outcome categories by at most 0.40 percentage points.The semantic-error patterns are consistent across the two judge models.Source: §B.2 · p. 16

B.3 ABLATION OF THE JOINT DESIGN IN B3

Controlled B3 ablations show that false-negative mitigation, hard-negative mining, and hard-batch construction contribute complementary benefits. The full configuration achieves the strongest overall MMEB performance.

B.3 ABLATION OF THE JOINT DESIGN IN B3

  • Replacing hard-batch construction with random batching reduces overall MMEB score from 69.0 to 67.9; removing false-negative filtering or hard-negative mining lowers it to 68.4 or 68.8.Some ablations improve individual meta-tasks marginally, but none matches full B3 overall.Source: §B.3 · p. 16
  • The controlled ablation removes false-negative mitigation, hard-negative mining, or hard-batch construction individually while keeping other training and evaluation settings unchanged.The comparison isolates the contribution of each optimization dimension within B3.Source: §B.3 · p. 16
  • False-negative mitigation, hard-negative mining, and hard-batch construction provide complementary benefits and achieve the strongest performance when optimized jointly.Source: §B.3 · p. 16

C MLLM-AS-JUDGE EVALUATION PROTOCOL

The protocol uses an MLLM judge to validate the designated positive before classifying the model’s top-1 prediction. Task-specific criteria and a constrained output schema support consistent, machine-readable judgments.

C MLLM-AS-JUDGE EVALUATION PROTOCOL

  • For each case, the judge receives the query, designated positive, top-1 prediction, task type, and explicit image-role mappings.The same prompt and output schema are used for all methods, with Qwen3.8-Flash as the default judge.Source: C MLLM-AS-JUDGE EVALUATION PROTOCOL · p. 16
  • The judge first verifies whether the designated positive is valid, then labels the top-1 prediction as a designated positive, false negative, hard negative, easy negative, or uncertain case.Source: C MLLM-AS-JUDGE EVALUATION PROTOCOL · p. 16
  • Task-specific criteria cover five MMEB task families, and judgments use a constrained JSON schema for automatic parsing.Source: C MLLM-AS-JUDGE EVALUATION PROTOCOL · p. 16

Related papers

Loading 2610.15035v1…