PaperlayerView exact revision on arXiv

Source-linked AI summary

EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE

Junyi Chen, Longteng Guo, Jia Sun, Shuai Shao, Zehuan Yuan, Liang Lin, Dongyu Zhang

arXiv:2308.11971v2cs.CVcs.CLcs.LGcs.MM

TL;DR

Vision-language pre-training must reconcile shared multimodal encoding with modality differences, while multiple objectives can increase training costs and limit scalability. EVE uses a unified Transformer with Modality-Aware MoE and one masked-signal objective. The paper reports 3.5x faster pre-training than with ITC and ITM and strong performance across vision-language downstream tasks.

  • Problem

    Vision-language pre-training must balance joint encoding with modality differences, while multiple objectives can raise training time and compute, require manual weighting, and hinder scalability.

    Source: Introduction · pp. 1–2

  • Method

    EVE combines shared Transformer attention with Modality-Aware MoE and uses one masked-signal objective to reconstruct masked image and text signals from visible signals.

    Source: Introduction · p. 2

  • Results

    EVE reports 3.5x faster pre-training than with ITC and ITM and outperforms existing methods across various vision-language downstream tasks.

    Source: Introduction · p. 2, Conclusion · p. 8

  • Takeaways & Limitations

    EVE is reported as easy to scale to larger models or more pre-training data while achieving gains across vision-language downstream tasks.

    Source: Conclusion · p. 8

Abstract

from arXiv · show

Building scalable vision-language models to learn from diverse, multimodal data remains an open challenge. In this paper, we introduce an Efficient Vision-languagE foundation model, namely EVE, which is one unified multimodal Transformer pre-trained solely by one unified pre-training task. Specifically, EVE encodes both vision and language within a shared Transformer network integrated with modality-aware sparse Mixture-of-Experts (MoE) modules, which capture modality-specific information by selectively switching to different experts. To unify pre-training tasks of vision and language, EVE performs masked signal modeling on image-text pairs to reconstruct masked signals, i.e., image pixels and text tokens, given visible signals. This simple yet effective pre-training objective accelerates training by 3.5x compared to the model pre-trained with Image-Text Contrastive and Image-Text Matching losses. Owing to the combination of the unified architecture and pre-training task, EVE is easy to scale up, enabling better downstream performance with fewer resources and faster training speed. Despite its simplicity, EVE achieves state-of-the-art performance on various vision-language downstream tasks, including visual question answering, visual reasoning, and image-text retrieval.

Introduction

Vision-language models must balance joint encoding with the differences between modalities, while proliferating pre-training objectives can increase cost and hinder scalability. EVE addresses these challenges with a unified modality-aware Transformer and masked signal modeling, reporting faster pre-training and strong downstream performance.

Introduction

  • EVE reports faster pre-training and state-of-the-art results on image-text retrieval and vision-language understanding, including VQA and NLVR2.Figure 1 compares VQA test-dev performance under different training hours; the paper says all models’ training hours were reproduced on A100 GPUs.Source: Figure 1 · p. 1, Introduction · p. 2
  • Earlier unified Transformers simplify multimodal encoding but may overlook modality gaps, while adding many pre-training tasks can increase time, compute, and manual loss-weight tuning.The paper characterizes images as continuous and low-level, and text as discrete and high-level.Source: Introduction · pp. 1–2
  • EVE uses shared self-attention with Modality-Aware MoE to jointly encode modalities while capturing modality-specific information through expert routing.The shared Transformer is described as concise and flexible, while modality routing selects more appropriate experts.Source: Introduction · p. 2
  • 3.5x faster pre-training than incorporating ITC and ITM is reported for masked signal modeling on raw image-text signals.The method combines masked pixel and language modeling without additional visual tokenizers or momentum models.Source: Introduction · p. 2

Related Work

Prior vision-language work trades off architectural simplicity, modality-specific representation, and pre-training complexity. EVE combines a unified architecture with Modality-Aware MoE and reconstructs masked raw image and text signals in a single objective.

Related Work

  • Unified architectures offer flexible multimodal or unimodal encoding, but standard shared Transformers can ignore modality gaps; dual encoders perform well at higher time and architecture complexity.The paper notes that MoME uses shared attention but may neglect modality-specific information through shared deep-layer FFNs.Source: Related Work · p. 2
  • Prior masked-prediction methods use visual tokens, momentum features, or additional ITC and ITM tasks, while one unified image-language masking approach is not used in downstream tasks.The cited approaches include VL-BEiT, DAVINCI, MAMO, and VLMAE.Source: Related Work · pp. 2–3
  • EVE unifies masked pixel and language modeling by reconstructing masked raw signals from visible signals, simplifying and accelerating training while supporting scalability.The overview describes random masking of image and text, with masked image and complete text for image modeling, and the reverse for language modeling.Source: Related Work · p. 3, Figure 2 · p. 3
  • Modality-Aware MoE is proposed as a simple alternative to prior MoE approaches that increase complexity or limit performance through manual routing or ignoring modality information.The method simplifies the auxiliary loss and captures modality-specific information through expert switching.Source: Related Work · p. 3

Methods

EVE uses a shared multimodal Transformer with Modality-Aware MoE and one masked-signal modeling objective. Its components encode and fuse modalities while reconstructing masked image or text signals from visible inputs.

Backbone Network

  • The backbone concatenates image and text embeddings for shared-attention and Modality-Aware MoE encoding, then serves as either a fusion or unimodal encoder after fine-tuning.Image patches and text tokens are projected into a shared embedding space before concatenation.Source: Backbone Network · p. 3

Modality-Aware Mixture-of-Experts

  • Modality-Aware MoE adds modality routing to general MoE, selectively switching experts to capture modality-specific information while fusing modalities.A general MoE routes each token to k selected experts, whose outputs form a weighted average.Source: Modality-Aware Mixture-of-Experts · pp. 3–4

Modality Routing

  • The router uses a best-effort strategy to preserve all tokens and adds modality-specific embeddings to address imbalanced routing across modalities.The method identifies differing modality token counts as a cause of routing imbalance; it uses separate embeddings for images and text.Source: Modality Routing · p. 4

Auxiliary Loss

  • A single load-balancing loss encourages uniform token routing (α=0.001); efficiency uses soft top-k=2 routing in deep layers and modality-selected hard routing in shallow layers.The hard router has one expert per modality.Source: Auxiliary Loss · p. 4

Pre-training Task: Masked Signal Modeling

  • Four times more computational resources are required to achieve similar speed when pre-training includes ITC and ITM than when it omits them.The comparison reports that additional pre-training tasks can also decrease speed and affect scalability.Source: Pre-training Task: Masked Signal… · p. 4
  • EVE’s single masked-signal objective combines masked image and language modeling on image-text pairs, reconstructing masked signals from visible ones and achieving competitive performance.It uses raw image-text signals without additional techniques; image modeling pairs masked images with complete text, while language modeling pairs complete images with masked text.Source: Pre-training Task: Masked Signal… · pp. 4–5

Masked Language Modeling (MLM)

  • Masked language modeling randomly masks text tokens and predicts them from image information and corrupted text, using cross-entropy on masked tokens and a two-layer MLP decoder.Source: Masked Language Modeling (MLM) · p. 5

Masked Image Modeling (MIM)

  • Masked image modeling reconstructs raw pixels on masked patches using a lightweight decoder, with only observed image patches and all text tokens entering the encoder.The image reconstruction loss is mean squared error computed on masked patches.Source: Masked Image Modeling (MIM) · p. 5

Experiments

EVE is evaluated on VQA, NLVR2, and image-text retrieval after pre-training on public image-text datasets. Its downstream results and ablations examine the unified masked-signal objective and modality-aware MoE design.

Pre-training Datasets

  • Pre-training uses four public datasets totaling about 4M images and 10M image-text pairs, with EVE-Large also trained on 21M pairs, and models use 12 or 24 Transformer blocks.Downstream test-set images are excluded from pre-training; EVE-Base trains for 480k steps and EVE-Large for 280k steps.Source: Pre-training Datasets · p. 5, Implementation Details · p. 5

Vision-Language Downstream Tasks

Results on Downstream Tasks

  • EVE-Base outperforms prior methods on vision-language understanding and COCO retrieval, while EVE-Large16M surpasses SimVLM-Huge; EVE-Base improves over VLMO by 1.77% on VQA test-dev and 0.70% on NLVR2 test-P.For COCO retrieval R@1, EVE-Base improves over VLMO by 6.42% on text retrieval and 8.39% on image retrieval.Source: Results on Downstream Tasks · p. 6, Table 2 · p. 6, Table 4 · p. 7

MIM Target

  • Pixel reconstruction outperforms image-token reconstruction across all evaluated tasks, while more complex MIM targets do not provide the expected benefit.The comparison uses tokenizers from BEiT v2 and DALL-E.Source: MIM Target · p. 6, Table 3 · p. 6
  • Higher vision masking improves performance, and EVE performs better at a higher text masking ratio than prior work.The authors attribute the text result to vision helping predict masked text tokens through deeper vision-language integration.Source: MIM Target · p. 6, Figure 4 · p. 6
  • With top-k = 2, increasing the number of experts improves VQA and retrieval, whereas top-k = 1 performance deteriorates as expert count grows, especially on retrieval.Performance also deteriorates when the number of selected experts decreases from two to one.Source: MIM Target · p. 7, Figure 5 · p. 7
  • MLM with a high masking ratio is sufficient for learning vision-language interaction, MIM further improves results, and jointly masking both modalities reduces performance.MIM alone yields minimal fusion; under equal pre-training GPU hours, MIM and MLM alone outperform adding further pre-training tasks on retrieval and understanding.Source: MIM Target · p. 7, Table 5 · p. 7, Table 6 · p. 7
  • Deep-layer shared FFN performs better than modality-specific MoE, while soft routing improves performance over a deeper architecture and modality routing helps distinguish input modalities.The authors report that some last-layer experts specialize in image or text tokens, and MoE can mitigate the modality gap that shared FFN does not address.Source: MIM Target · pp. 7–8, Figure 6 · p. 8, Table 7 · p. 8

Visualization

Grad-CAM heatmaps visualize EVE self-attention during masked signal modeling and VQA, with the results provided in the Appendix.

  • Grad-CAM heatmaps visualize EVE self-attention during masked signal modeling and VQA, with results provided in the Appendix.Source: Visualization · p. 8

Conclusion

EVE combines masked signal modeling with Modality-Aware MoE in a unified multimodal model. The paper reports 3.5x faster pre-training than with ITC and ITM and better performance across vision-language downstream tasks.

  • EVE uses masked signal modeling with Modality-Aware MoE to encode different modalities in a unified manner.Source: Conclusion · p. 8
  • 3.5x faster pre-training than pre-training with ITC and ITM, with EVE outperforming existing methods across vision-language downstream tasks.Source: Conclusion · p. 8

Pre-training and Inference Speed

EVE surpasses other methods in pre-training speed, particularly VL-BEiT and VLMO. Its inference time does not increase significantly after adding MoE, and is slightly slower than VLMO but faster than ALBEF.

  • EVE surpasses other methods in pre-training speed, particularly VL-BEiT and VLMO.At inference on the VQA test set, EVE is slightly slower than VLMO but much faster than ALBEF; adding MoE does not significantly increase its inference time.Source: Pre-training and Inference Speed · p. 11

More Ablation Studies

The ablations examine decoder depth, soft-router placement, and shallow-layer FFN design. They report little decoder-depth sensitivity except for overly deep decoders, and favor top-layer soft routing and modality-aware shallow experts, with soft routing adding computational overhead.

MIM Decoder Depth

  • Different masked-image-modeling decoder depths show no noticeable downstream-performance difference, though an overly deep decoder causes a slight drop.The comparison is described as Table 11, but no table passage is supplied.Source: More Ablation Studies · p. 11

Position of Soft Router

  • Soft-router performance is highest in top layers, followed by bottom layers and then middle layers; using it across layers performs slightly worse than continuous use.The authors attribute the placement pattern to more pronounced modality information in low-level features and greater fusion needs in high-level features.Source: More Ablation Studies · p. 11, Table 12 · p. 11

Shallow FFN

  • Shallow modality-specific MoE outperforms modality-shared FFN, while a soft router combines their advantages at increased computational overhead.Table 13 covers FFN designs in the bottom 10 layers.Source: More Ablation Studies · p. 11, Table 13 · p. 11

Model Architecture Implementation Details

EVE-Base and EVE-Large use different Transformer sizes and pre-training budgets, with soft routing in selected top layers and hard routing elsewhere. Pre-training masks image patches and text at different rates.

Model architecture

  • EVE-Base uses 12 Transformer blocks, 12 attention heads, and 768 hidden size; EVE-Large uses 24 blocks, 16 heads, and 1024 hidden size.Both use 32 experts; soft routing covers the top 2 EVE-Base layers and top 3 EVE-Large layers, with hard routing elsewhere.Source: Model Architecture Implementation… · p. 11

Pre-training setup

  • EVE-Base is pre-trained for 480k steps on 16 A100 GPUs, while EVE-Large is trained for 280k steps on 32 A100 GPUs, both at batch size 2048.Both use AdamW with β1 = 0.9, β2 = 0.98, weight decay 0.05, 10k-step linear warmup, and cosine decay; peak learning rates are 5e-4 and 2e-4, respectively.Source: Model Architecture Implementation… · p. 11
  • Pre-training uses 224 × 224 images with 16 × 16 patches, masks 75% of image patches and 50% of text, and initializes EVE with BEiTv2.Random resized cropping, horizontal flipping, and mixed precision are also used.Source: Model Architecture Implementation… · pp. 11–12

Downstream Tasks Fine-tuning Details

Fine-tuning details specify task-specific datasets, input handling, optimization, and retrieval reranking. VQA and NLVR2 use classification heads, while retrieval selects candidates with ITC scores and reranks them with ITM scores.

Downstream Tasks Fine-tuning Details

  • Fine-tuning uses AdamW with weight decay 0.01, cosine learning-rate scheduling, and linear warmup over the first 10% of steps.Input resolution is 480 × 480 for VQA and 384 × 384 for other tasks.Source: Downstream Tasks Fine-tuning Details · p. 12

Image-Text Understanding Task

  • VQA fine-tuning uses VQA2.0 train and validation data, excludes Visual Genome question-answer augmentation, and classifies among 3,129 answers.The image-question representation is the T cls token; models train for 10 epochs with batch size 128 and peak learning rate 3e-5.Source: Image-Text Understanding Task · p. 12, Downstream Tasks Fine-tuning Details · p. 12
  • NLVR2 converts each triplet into two image-text pairs, fuses their representations with bi-attention, and predicts the answer with a two-layer classifier.Models train for 10 epochs with batch size 128; peak learning rates are 4e-5 for EVE-Base and 3e-5 for EVE-Large.Source: Image-Text Understanding Task · p. 12

Image-Text Retrieval Task

  • MSCOCO and Flickr30K use Karpathy splits; retrieval selects the top 128 candidates by ITC score and reranks them by ITM score.MSCOCO models train for 10 epochs at batch size 256, while Flickr30K uses 10 epochs at batch size 128.Source: Image-Text Retrieval Task · p. 12

Visualization

Grad-CAM visualizations show EVE attending to cross-modal information related to masked signals and to image regions relevant to VQA questions. The examples illustrate image-to-text and text-to-image attention during masked signal modeling.

Visualization

  • EVE attends to information in the other modality that is semantically related to the masked portion, illustrating complementary information learned through masked signal modeling.The authors state this occurs without necessarily using complex pre-training tasks.Source: Visualization · p. 12

Visualization on Masked Language Modeling

Visualization on Masked Image Modeling

Visualization on VQA

Loading 2308.11971v2…