Scaling Vision-Language Models with Sparse Mixture of Experts

1. Introduction

• Unlike LIMOE’s contrastive-pretraining and vision-only downstream focus, this study investigates generative vision-language modeling and broader multimodal evaluation.The authors identify overfitting with increased MoE capacity under a simple contrastive objective and limited coverage from vision-only benchmarking as concerns.Source: §1 · p. 1
• VL-MoE scales both text-based and vision-based feed-forward networks as modality-specific experts within a unified vision-language framework.The text and vision representations are aligned using three masked-data modeling objectives.Source: §1 · p. 1
• Image-only inputs route through V-MoE or V-FFN, text-only inputs through T-MoE or T-FFN, and image-text pairs through V-MoE, T-MoE, and VL-FFN.Early layers scale V-FFN and T-FFN into modality-specific expert modules.Source: Figure 1 · p. 2
• A 2B-parameter VL-MoEBASE/32E applies 180M parameters per token and achieves competitive performance with dense models using similar or more pretraining image-text pairs and 3–4× more parameters per token.The experiments cover vision-language classification and retrieval, plus vision-only and language-only tasks.Source: §1 · pp. 1–2
• The study compares increasing dense-model size, increasing expert counts, and scaling only the text or vision feed-forward network to examine performance and complexity trade-offs.The authors also present ablations of model behavior, interpretability, and design choices.Source: §1 · p. 2

Vision-language modeling

• Separate encoders can perform well on image-text retrieval but struggle with complex vision-language tasks, while cross-modal fusion improves representations at an efficiency cost.MOME Transformers unify dual-encoder and fusion-encoder designs using modality experts in a shared Transformer block.Source: §2 · p. 2

Pretraining objectives

• Vision-language pretraining uses discriminative objectives such as contrastive learning and image-text matching, or generative objectives such as masked language and region modeling.BEIT-3 combines masked image, masked language, and masked multimodal modeling.Source: §2 · p. 2

Sparse Mixture of Experts

• Sparse MoE uses input-dependent gating to activate only k of E experts, increasing parameter count without a proportional increase in computation.The paper adopts sparse MoE in a MOME backbone with generative masked-data modeling objectives.Source: §2 · p. 3

Related multimodal MoE

• Prior multimodal MoE work such as LIMOE [47] uses CLIP-style contrastive pretraining and vision or retrieval downstream tasks; VL-MoE studies generative pretraining and a broader task set.The authors characterize their work as the first MoE scaling study of generative modeling in vision-language pretraining.Source: §2 · p. 3

3.1. Vision-Language Masked Data Modeling

• Three masked-data objectives train language representations, vision representations, and aligned vision-language representations using text, image, and image-text data.The multimodal objective predicts masked text tokens and image patches from joint representations.Source: §3.1 · p. 3

3.2. VL-MoE Architecture

• Text and image inputs are embedded separately, then concatenated to form image-text representations for the shared multimodal Transformer.Text representations sum word and position embeddings; image patches are projected into embeddings and paired with a prepended [I CLS] token.Source: §3.2 · pp. 3–4
• Sparse routing activates only k ≪ E experts per input, while Batch Priority Routing prioritizes higher-weight tokens when fixed expert capacity is exceeded.The paper identifies Batch Priority Routing as crucial for stable VL-MoE training.Source: §3.2 · p. 4
• VL-MoE replaces selected text and vision feed-forward layers with sparse experts, while routing inputs to modality-specific experts and later-layer VL-FFN experts.The design uses a shared self-attention module and modality-specific feed-forward expert pools.Source: §3.2 · p. 4

4.1. Pretraining Setup

• Pretraining combines ImageNet-22K images, Wikipedia and BookCorpus text, and four image-text datasets totaling 4 million images and 10 million pairs.The base model uses 12 Transformer layers; the default BASE/32E setup activates one of 32 experts per token and replaces every second text or vision FFN sublayer.Source: §4.1 · p. 5
• Across scales, VL-MoE’s pretraining cost-performance trade-off in FLOPs dominates dense models, with maximum wall-clock overhead reduced to 13%.The reported scaling trend spans SMALL/8E to BASE/32E.Source: §4.1 · p. 5

4.2. Vision-and-Language Downstream Tasks

• VL-MoE outperforms previous base-size models on VQA, NLVR2, COCO, and Flickr30K; it exceeds VLMOLARGE and ALBEF on retrieval and is competitive on VQA and NLVR2.VL-MoEBASE/32E has 2 billion total parameters and applies 180 million parameters per token.Source: Table 1 · p. 6, §4.2 · pp. 6–7

4.3. Vision/Language-Only Downstream Tasks

• VL-MoE performs competitively with prior supervised and self-supervised models on ImageNet-1K and outperforms other strong vision-language models on MNLI-m.The comparison includes BEIT-3BASE and SIMVLM.Source: §4.3 · p. 7, Table 2 · p. 7

Ablation studies

• The ablations evaluate models on NLVR2, Flickr30k, ImageNet-1k, and MNLI-m.These cover visual reasoning, image-text retrieval, image classification, and natural-language inference, respectively.Source: §5 · p. 7

Scaling Strategy

• 70.6% averaged performance is reported for scaling both vision and language modalities, the most balanced-performing strategy.Table 3 covers VL-MoESMALL/E32 scaling strategies with 32 experts.Source: §5 · p. 7, Table 3 · p. 8

Number of Experts

• Larger expert pools consistently yield downstream performance improvements.The study investigates expert number in Figure 5.Source: §5 · p. 8, Figure 5 · p. 8

Effects of the Auxiliary Losses

• The averaged loading-balance-and-importance loss (“vloss”) gives the most stable training, while loading-balance-only training is unstable and underperforms.Z-loss hurts vision-and-language pretraining; BPR also helps stabilize training.Source: §5 · p. 8

Token Routing Examples in VL-MoE

• Vision tokens specialize in semantic expert groups, while language tokens show syntax specialization, with some experts processing padding and others nouns or adjectives.Language-token experts exclude prepositions, determiners, or verbs in the described routing examples.Source: §5 · p. 8

Conclusion

• MoE divides a large vision-language model into specialized sub-models and achieves state-of-the-art performance on several benchmarks while reducing computational costs.The paper characterizes MoE as promising for improving VLM efficiency and effectiveness.Source: §6 · p. 8
• The authors report that larger expert pools yield consistent performance improvements and that MoE can improve interpretability by clarifying how inputs are processed.Source: §6 · p. 8
• The conclusion identifies visual question answering, visual reasoning, and image-text retrieval as directions for further study.The authors hope the findings will inspire further investigation of MoEs in these tasks.Source: §6 · p. 8

“Dropped” Tokens

• Dropped tokens are more balanced in MLM and MIM than in VLM, and dropped image tokens are more severe than dropped text tokens.Figure 6 analyzes VL-MoEBASE/32E across the three pretraining tasks.Source: §A.1 · p. 13, Figure 6 · p. 14

Pretrain Losses for Different Scaling Strategies

• Scaling each modality supports improved optimization of both its modality-specific pretraining loss and the VLM loss.The analysis reports this pattern across MLM, MIM, and VLM pretraining losses for VL-MoESMALL/32E.Source: §A.1 · p. 13

Comparison with LIMOE

• With contrastive loss on 4M image-text pairs, both VL-MoE and LIMOE quickly overfit, while LIMOE shows more instability; LIMOE's MDM setup fails to converge.The MDM failure is attributed as likely related to modality imbalance from its three losses.Source: §A.1 · p. 13

Visual Question Answering (VQA)

• VQA base-size models are fine-tuned for 10 epochs with batch size 128, peak learning rate 3e-5, and 480 × 480 images.Source: §A.2 · p. 14

Natural Language for Visual Reasoning (NLVR2)

• NLVR2 base-size models use 10 epochs, batch size 128, peak learning rate 5e-5, and 384×384 images; ablations use 224 × 224 images and average three runs.Ablation learning rates are selected from {5e-5, 1e-4}.Source: §A.2 · p. 14

COCO and Flickr30K

• COCO fine-tuning uses 20 epochs, batch size 2048, peak learning rate 2e-5, and 384 × 384 images; Flickr30K uses 40 epochs and initializes from the COCO fine-tuned model.Flickr30K uses batch size 2048, peak learning rate 1e-5, and 384 × 384 images; ablations use 10 epochs, batch size 1024, 5e-5, and 224 × 224 images.Source: §A.2 · p. 15

ImageNet-1k and MNLI

• ImageNet-1k fine-tunes VL-MoE with V-MoE and V-FFN only for 15 epochs, while MNLI fine-tunes T-MoE and T-FFN only for 10 epochs.Both use peak learning rate 3e-5; batch sizes are 2048 for ImageNet-1k and 32 for MNLI.Source: §A.2 · p. 15

• For a token x, g(x)=softmax(Wx) gives gating weights across the E experts, with W as the routing parameters.Source: §A.3 · p. 15

Importance loss

• The importance loss Ωimp encourages gating weights to be evenly distributed across experts.It is defined via the squared coefficient of variation of expert importance.Source: §A.3 · p. 15

Load loss

• The load loss Ωload targets an even number of assignments by balancing experts’ probabilities of being selected in the top-k.Selection probability is estimated under noisy routing.Source: §A.3 · p. 15

Z-loss

• The z-loss Ωzloss controls the maximum magnitude of router activations A={W xi}.The activations are formed from routing parameters W applied to tokens xi.Source: §A.3 · p. 16

v-loss

• The v-loss used in Section 5 is 0.5 times the importance loss plus 0.5 times the load loss.Source: §A.3 · p. 16