PaperlayerView exact revision on arXiv

Source-linked AI summary

MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings

Haonan Chen, Hong Liu, Yuping Luo, Liang Wang, Nan Yang, Furu Wei, Zhicheng Dou

arXiv:2506.23115v1cs.CVcs.AIcs.CL

TL;DR

Existing VLM-based multimodal embeddings use causal attention and rely heavily on labeled image-text pairs, limiting embedding quality, scalability, and data diversity. MoCa combines modality-aware continual pre-training with heterogeneous contrastive fine-tuning to build bidirectional multimodal embeddings. It consistently improves MMEB and ViDoRe-v2 performance, achieves state-of-the-art results on MMEB, and scales with model and training-data size.

  • Problem

    Causal attention may be suboptimal for embeddings, while contrastive learning depends on high-quality paired data and limited training diversity.

  • Method

    MoCa jointly denoises interleaved text and image inputs during modality-aware continual pre-training, then applies heterogeneous contrastive fine-tuning with diverse multimodal data.

  • Results

    MoCa consistently improves performance across MMEB and ViDoRe-v2, achieves state-of-the-art results on MMEB, and scales with model size and training data.

  • Takeaways & Limitations

    Continual pre-training with bidirectional objectives and heterogeneous multimodal data is effective for improving multimodal embedding robustness, generalization, and scalability.

  • Takeaways & Limitations

    Future work should extend evaluation to additional modalities, more advanced objectives, and complex real-world applications such as multi-hop retrieval.

Abstract

from arXiv · show

Multimodal embedding models, built upon causal Vision Language Models (VLMs), have shown promise in various tasks. However, current approaches face three key limitations: the use of causal attention in VLM backbones is suboptimal for embedding tasks; scalability issues due to reliance on high-quality labeled paired data for contrastive learning; and limited diversity in training objectives and data. To address these issues, we propose MoCa, a two-stage framework for transforming pre-trained VLMs into effective bidirectional multimodal embedding models. The first stage, Modality-aware Continual Pre-training, introduces a joint reconstruction objective that simultaneously denoises interleaved text and image inputs, enhancing bidirectional context-aware reasoning. The second stage, Heterogeneous Contrastive Fine-tuning, leverages diverse, semantically rich multimodal data beyond simple image-caption pairs to enhance generalization and alignment. Our method addresses the stated limitations by introducing bidirectional attention through continual pre-training, scaling effectively with massive unlabeled datasets via joint reconstruction objectives, and utilizing diverse multimodal data for enhanced representation robustness. Experiments demonstrate that MoCa consistently improves performance across MMEB and ViDoRe-v2 benchmarks, achieving new state-of-the-art results, and exhibits strong scalability with both model size and training data on MMEB.

1 Introduction

MoCa addresses limitations in VLM-based multimodal embeddings by replacing causal processing with bidirectional objectives and broadening scalable training beyond labeled image-caption pairs. Its two-stage framework consistently improves benchmark performance and achieves state-of-the-art results on MMEB.

  • 1 Introduction: Causal attention may be suboptimal for embedding models, while bidirectional attention has shown advantages for text and multimodal embeddings.Existing multimodal embedding approaches commonly inherit causal attention from their VLM backbones.
  • 1 Introduction: Contrastive learning is difficult to scale because it depends on diverse, high-quality labeled image-text pairs, motivating unlabeled-data objectives and broader training data.CPT with unlabeled interleaved data addresses scalability, while heterogeneous data expands beyond mainly image-caption pairs.
  • 1 Introduction: MoCa combines modality-aware continual pre-training with heterogeneous contrastive fine-tuning to transform pre-trained VLMs into bidirectional multimodal embedding models.The first stage jointly denoises interleaved text and image inputs, while the second uses heterogeneous multimodal data and cross-modal interactions.
  • 1 Introduction: The framework uses joint reconstruction to enhance bidirectional cross-modal reasoning and heterogeneous multimodal pairs to improve generalization and alignment across domains.These stages directly target the limitations of causal attention, pair-data dependence, and limited training diversity.
  • 1 Introduction: MoCa consistently improves MMEB and ViDoRe-v2 performance, with strong scalability to model and training-data size on MMEB.After 30B tokens, the 3B model matches or surpasses competitive 7B contrastive baselines; the 7B model achieves new state-of-the-art MMEB results.

2 Method: MoCa

MoCa transforms causal VLMs into bidirectional multimodal embedding models through modality-aware continual pre-training and heterogeneous contrastive fine-tuning. The two stages jointly reconstruct masked text and images, then align diverse multimodal data with hard negatives and task-aware batching.

  • 2.1 Preliminaries: MoCa removes causal attention masks and uses bidirectional VLM representations for multimodal embeddings.Prior embedding models inherit causal attention, while MoCa uses bidirectional hidden states to improve representation quality.
  • 2.2 Modality-aware Continual Pre-training: Modality-aware continual pre-training jointly masks text tokens and image patches, reconstructing both from interleaved multimodal context.MLM predicts masked text tokens, while MAE uses a lightweight decoder and MSE to reconstruct masked image patches; their losses are weighted together.
  • 2.2 Modality-aware Continual Pre-training: The MLM objective uses bidirectional context so masked tokens attend to visible text and image patches, strengthening local, global, and cross-modal representations.The model predicts each masked token from the surrounding multimodal context using cross-entropy.
  • 2.3 Heterogeneous Contrastive Fine-tuning: Heterogeneous contrastive fine-tuning aligns queries with positive documents while suppressing hard-negative similarities across unimodal and interleaved multimodal data.The similarity function uses temperature-scaled cosine similarity, and training instances may contain text, image, or text-image documents.
  • 2.3 Heterogeneous Contrastive Fine-tuning: Task-aware batching keeps instances from the same task together, producing harder in-batch negatives than mixing visually dissimilar tasks.This strategy addresses the weakness of trivial negatives caused by large differences between task types.

3 Experiments

MoCa is evaluated on broad multimodal and document-retrieval benchmarks using heterogeneous continual-pretraining and contrastive-training corpora.

  • Continual-pretraining data: The continual-pretraining corpus combines text-only, common image-text, and document-level multimodal data, totaling approximately 30B tokens.Each dataset contributes 500K randomly sampled instances.
  • Implementation: Experiments use Qwen-2.5-VL as the backbone with 2048-token inputs and model-specific text and image masking ratios.Training uses 32 NVIDIA H100 GPUs, with MLM probabilities of 0.4 and 0.6 and image-masking ratios of 0.5 and 0.6 for MoCa-3B and MoCa-7B, respectively.
  • Contrastive fine-tuning data: Contrastive fine-tuning uses long-form multimodal, common multimodal, and text-only retrieval pairs, totaling approximately 2M training pairs.Each dataset contributes 50K randomly sampled instances.
  • Implementation: Contrastive training initializes from the modality-aware continual-pretraining checkpoint and uses task-aware batching with two hard negatives per positive pair.The contrastive batch size is 2048 and the temperature parameter is fixed at 0.03.
  • Benchmarks: MMEB covers 36 tasks spanning classification, visual question answering, retrieval, and visual grounding, while ViDoRe-v2 evaluates visually rich document retrieval.MMEB reports Precision@1; ViDoRe-v2 reports NDCG@5 across diverse document types, languages, and settings.

3.2 Overall Results

MoCa consistently outperforms strong baselines on MMEB and ViDoRe-v2, with bidirectional attention, continual pre-training, and heterogeneous contrastive learning jointly supporting its strongest results.

  • Overall comparison: MoCa consistently outperforms all strong baselines on MMEB and ViDoRe-v2, establishing the framework’s effectiveness across multimodal and document-level retrieval.The comparisons are reported in Table 1 for MMEB and Table 2 for ViDoRe-v2.
  • Component combination: The “bidirectional + CPT + CL” configuration outperforms both “causal + CL” and “bidirectional + CL” without CPT.This comparison supports the combined contribution of bidirectional attention, continual pre-training, and heterogeneous contrastive learning.
  • Model scaling: After continual pre-training on 30B tokens, MoCa-3B surpasses 7B baselines trained only with contrastive learning.The comparison is reported as part of the MMEB results.
  • Model scaling: Scaling MoCa to 7B parameters yields substantial improvements across all MMEB task categories and new state-of-the-art MMEB results.Across the full MMEB benchmark, larger models show consistent improvements.
  • Cross-benchmark behavior: Although MoCa-7B performs better overall on MMEB, MoCa-3B achieves slightly higher results on ViDoRe-v2.The passage attributes this pattern to ViDoRe-v2’s smaller sample sizes and the larger model’s greater susceptibility to overfitting in low-resource settings.

3.3 Ablation Study

Ablations show that both modality-aware reconstruction objectives and heterogeneous contrastive data are necessary for robust performance on MMEB and ViDoRe-v2.

  • Overall ablations: Removing any key component from either training stage consistently reduces performance on both MMEB and ViDoRe-v2.Table 3 evaluates components from modality-aware continual pre-training and heterogeneous contrastive fine-tuning.
  • Modality-aware Continual Pre-training: Removing either MLM or MAE lowers performance, while removing both produces the largest decline.The results indicate that text and image reconstruction jointly support robust bidirectional embeddings.
  • Heterogeneous Contrastive Fine-tuning: Removing text-only contrastive pairs causes a noticeable drop, indicating their importance for maintaining strong language representations.This ablation concerns the heterogeneous contrastive fine-tuning stage.
  • Heterogeneous Contrastive Fine-tuning: Excluding long-form document-retrieval pairs hurts performance, especially on ViDoRe, supporting their value for reasoning over extended contexts.The excluded data come from VisRAG and the ColPali training set.

3.4 Data Scaling Effect for Continual Pre-training

Increasing continual-pretraining duration steadily improves MMEB performance, with a 3B model reaching 7B-baseline performance after approximately 20B tokens.

  • Scaling with CPT: MMEB performance improves consistently as continual-pretraining steps increase for both 3B and 7B models.The analysis uses checkpoints from a single continual-pretraining run and evaluates them through contrastive learning.
  • Scaling with CPT: After approximately 2,200 steps, corresponding to 20B tokens, the 3B model matches the 7B baseline trained without continual pre-training.The result shows the effect of continual-pretraining scale on downstream MMEB performance.
  • Scope: The study is constrained by computational resources, leaving further scaling of continual pre-training as future work.The authors suggest that more data and training steps may continue improving performance while requiring cost trade-offs.

3.5 Hyperparameter Analysis of Continual Pre-training

The analysis evaluates how CPT settings affect MoCa (3B) on MMEB, finding that moderate masking, balanced loss weighting, and a stable learning rate support performance.

  • Experimental Setup: The experiments evaluate MoCa (3B) under different CPT settings on MMEB using a fixed data amount and validation-selected hyperparameters.Results are reported on the MMEB test set for consistency with earlier experiments.
  • Mask Ratio: Moderate masking rates of MLM: 40% and MAE: 50% provide a balance for cross-modal reasoning without excessive information removal.Higher masking can reduce the signal-to-noise ratio and degrade learning.
  • Loss Weight: An MAE loss weight of 0.5 provides the best trade-off between visual reconstruction and language modeling.Underweighting MAE can weaken visual-semantic integration, while overweighting it can distort the language objective.
  • Learning Rate: A learning rate of 2 × 10^-6 supports stable optimization while adapting to denoising objectives without catastrophic forgetting.Lower rates may underfit, whereas larger rates can disrupt pretrained knowledge and destabilize training.

4 Related Work

Prior multimodal embedding methods use dual-encoder or VLM-based architectures, while related continual pre-training work motivates MoCa’s bidirectional and multimodal adaptation.

  • Multimodal Embedding: Multimodal embedding models encode modalities separately or adapt VLMs, commonly using contrastive learning to align cross-modal representations.Examples include ALIGN [15], BLIP [23], CLIP [31], VLM2Vec [16], and related VLM-based methods.
  • Multimodal Embedding: Most existing approaches still rely on causal models, motivating exploration of bidirectional VLMs for stronger multimodal embeddings.Related fine-tuning methods also improve alignment through hard negatives and task-aware batch sampling.
  • Continual Pre-training for Multimodal Models: Continual pre-training has been used to adapt pretrained models with additional data or objectives, including multimodal MLM-based representation learning.Existing multimodal CPT work often uses relatively small transformer architectures.

5 Conclusion

MoCa combines modality-aware continual pre-training with heterogeneous contrastive fine-tuning to improve bidirectional multimodal embeddings. The authors report state-of-the-art performance and scalability, while identifying broader modalities and applications as future directions.

  • Conclusion: MoCa combines modality-aware continual pre-training and heterogeneous contrastive fine-tuning to improve cross-modal interactions, robustness, and generalization.Its continual pre-training uses bidirectional attention and joint reconstruction, while its fine-tuning uses diverse multimodal data.
  • Conclusion: MoCa achieves state-of-the-art performance and scales with both model size and training-data size on MMEB.The conclusion presents continual pre-training as effective for advancing multimodal embedding research.
  • Future Work: Future work includes adding video, speech, and structured data, improving denoising or encoder-decoder designs, and evaluating more complex retrieval scenarios.The proposed directions target generality, efficiency, representation quality, and robustness in real-world applications.

A Detailed Results on MMEB

The detailed MMEB results cover 36 tasks across classification, VQA, retrieval, and visual grounding, comparing baseline models with MoCa at 3B and 7B scales.

  • A Detailed Results on MMEB: Table 4 reports detailed performance across 36 MMEB tasks spanning classification, VQA, retrieval, and visual grounding.The table presents results for baseline models and MoCa at both 3B and 7B scales.
Loading 2506.23115v1…