Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts

1 Introduction

• LIMoE is a large-scale multimodal mixture-of-experts model that processes images and text in one architecture and aligns their representations via contrastive learning [7].The authors explore MoEs for multimodal modeling because sparse models may handle distinct tasks better.Source: §1 · pp. 1–2
• Entropy-based regularizers address MoE failure modes that arise when multiple modalities share one architecture, stabilizing training and improving performance.The paper reports that prior approaches to MoE regularization fall short for multimodal learning.Source: §1 · pp. 1–2
• 84.1% zero-shot ImageNet-2012 accuracy is achieved by LIMoE-H/14, a 5.6B-parameter model applying 675M parameters per token, competitive with two-tower models using modality-specific pre-training and feature extractors and 3–4x more parameters per token.The paper also reports LIMoE’s accuracy as comparable to state-of-the-art contrastive models with per-modality backbones and pre-training.Source: §1 · pp. 1–2
• Across architecture scales, LIMoE improves zero-shot ImageNet accuracy by 7% to 13% over equivalent dense models.The authors also report that a range of LIMoE models significantly outperforms compute-matched dense baselines.Source: §1 · pp. 1–2
• The paper presents ablations and analysis to examine LIMoE’s behavior and design decisions.Source: §1 · p. 2

2.1 Multimodal contrastive learning

• For image-caption pairs, LIMoE uses a learned-temperature contrastive objective that brings paired image and text representations closer than unpaired inputs [7] [11].This one-tower setup contrasts with the typical two-tower approach, which uses independent encoders for each modality [7] [10].Source: §2.1 · p. 2, §2 · p. 2

2.2 The LIMoE Architecture

• A shared Transformer embeds each modality separately, processes all tokens without explicit modality conditioning, average-pools representations, and applies modality-specific projections for contrastive learning.LIMoE adds sparse MLP expert layers: a lightweight router selects K of E experts for each token, whose outputs are combined using gating weights.Source: §2.2 · pp. 2–3

2.2.1 Challenges for multimodal MoEs

• With 3–17× more image tokens than text tokens, globally balanced routing can still overload one expert with minority-modality tokens and destabilize training.In the standard B/16 setup, the importance loss could be within 0.5% of its minimum while all text tokens were dropped.Source: §2.2.1 · p. 3

2.2.2 Auxiliary losses

• Classic auxiliary losses [1] were insufficient for stable, performant multimodal MoEs; per-modality local and global entropy losses address unstable preferences and modality-level expert collapse.The local loss stabilizes token-level preferences, while the global loss prevents all tokens of one modality from collapsing onto one expert, yielding stable allocations and high routing success rates.Source: §2.2.2 · p. 3, Figure 3 · p. 4

2.2.3 Priority routing

• Batch Priority Routing (BPR) [1] prioritizes tokens by routing weight, stabilizes training, and avoids systematically favoring either modality when tokens are dropped.The method requires care because batch order can group tokens by modality, making rank-based dropping potentially biased.Source: §2.2.3 · p. 5

3 Experiments

• Evaluation focused on zero-shot ImageNet classification and MS-COCO retrieval, alongside linear adaptation and 10-shot ImageNet accuracy; controlled runs used 781k steps and batch size 16k.Source: §3 · p. 5, §3.1 · p. 5

3.2 Scaling up LIMoE

• The H/14 scaling run used 32 layers, 12 nonuniform expert layers with 32 experts each and K = 1, 25% JFT-4B images per batch, and 5.6B total versus 675M active parameters.All routers together account for less than 0.5M parameters.Source: §3.2 · pp. 5–6

3.1 Controlled study across scales

• LIMoE’s compute-performance frontier dominated dense models from S/32 to L/16, averaging absolute gains of 10.1% on zero-shot and 12.2% on 10-shot ImageNet, with strongest COCO retrieval gains at small scales.Source: §3.1 · p. 5, Figure 4 · p. 6

3.2 Scaling up LIMoE

• 84.1% zero-shot ImageNet classification accuracy made LIMoE-H/14 comparable with the best two-tower models and the first performant one-tower model at this scale.It was trained from scratch without pretrained components and uses a shared backbone.Source: Table 1 · p. 6, §3.2 · p. 6

4 Ablations

• LIMoE greatly outperforms both dense baselines on ImageNet 0- and 10-shot; retrieval confidence intervals overlap with the two-tower model, which is twice as large and expensive yet still falls behind LIMoE.The comparison uses B/16 models trained for 100,000 steps at batch size 8096, averaging three trials.Source: §4 · p. 6, Table 2 · p. 7

4.1 Routing and auxiliary losses

• Both entropy losses are important for text, whereas the global loss is not impactful for images and the local loss is harmful.The final loss combination was chosen using validation accuracy together with qualitative observations of training stability and routing success rate.Source: §4.1 · pp. 6–7, Table 3 · p. 7
• Image threshold τ_image did not affect image expert usage and its global loss is usually inactive, while text threshold τ_text typically yielded approximately S = e^τ_text text experts.Performance was robust to τ_text unless it was too low; a low threshold can limit text experts for later pruning.Source: §4.1 · p. 7
• The mutual-information loss produced 1.5% worse zero-shot and 0.1% worse 10-shot performance than Table 2 in a comparable setup.It combines local and global entropy losses for both modalities without threshold parameters.Source: §4.1 · p. 7
• Entropy routing outperformed classical routing across image:text balances, including 1:1, and longer image sequences improved performance with entropy routing.The comparison varied image patch size to change image-token counts without changing data information content; the performance gap over the classical setup grew across the tested balances.Source: Figure 5 · p. 7, §4.1 · pp. 7–8
• Batch priority routing improved training stability and performance over first-in-first-out or random token ordering, whose models performed poorly or remained much worse.First-in-first-out models had low router success rates whether images or text were routed first; BPR also ameliorated token dropping.Source: Figure 6 · p. 8, §4.1 · p. 8

4.2 Other ablations

• Per-modality router variants performed comparably to the modality-agnostic router, while separate expert pools were more stable without auxiliary regularization but harder to scale; heuristics enabled modality-specific pruning for unimodal passes.Pruning avoids expert collapse under unimodal batches.Source: §4.2 · p. 8
• Increasing K raised zero-shot accuracy from 55.5% at K = 1 to 61.0% at K = 5, and increasing the expert pool at fixed K improved performance.LIMoE also retained performance improvements over a comparable dense model when trained on publicly available data.Source: §4.2 · p. 8

Multimodal experts arise

• Without explicit specialization constraints, LIMoE develops both modality-specific experts and experts that process images and text.The analysis uses B/32 and B/16 models with 8 experts and H/14 with 32 experts.Source: §5 · p. 8

Qualitative analysis

• Image experts specialize in semantically meaningful features, ranging from low-level colours and lines to complex features such as faces and text.The paper illustrates these patterns with example data and Figure 2.Source: §5 · p. 9

BPR ranking

• Although the local loss encourages high max-routing weights for text and BPR ranks accordingly, later layers often prioritize important image patches over text.Source: §5 · p. 9

• Unimodal task-specific networks increasingly converged on Transformer architectures [23] [26] in NLP and computer vision [13] [28] [29], while multimodal models aim to process multiple data types with one network.Source: §6 · p. 9
• Unlike modality-fusion approaches [30] [31] [32] [33], LIMoE resembles unimodal feature extractors; other models either co-train tasks without aligning or fusing representations [34] [36] or combine unimodal and fused functionality.Source: §6 · p. 9
• Sparse MoEs were studied separately in computer vision [1] and NLP [14] [3] [8], typically for transfer learning, but had not been explored for multimodal learning.Source: §6 · p. 9
• No prior research, to the authors’ knowledge, used a single dense or sparse model to process both images and text for contrastive learning.Source: §6 · p. 9
• Multimodal contrastive learning progressed from medical image-report pairs [11] to noisy web data [7] [10], enabling image classification and cross-modal retrieval without downstream finetuning.Follow-up work scaled models and used pretrained models [16], generative multitask training, or other vision tasks.Source: §6 · p. 9

• +10.2% average zero-shot accuracy over FLOP-matched dense baselines, while LIMoE-H/14 reaches 84.1%, competitive with current SOTA approaches.The H/14 result is reported for a large model.Source: §7 · p. 9
• LIMoE is presented as the first multimodal sparse mixture-of-experts model, with entropy-based auxiliary losses that stabilize training and support high performance.Source: §7 · p. 9
• LIMoE does not explicitly address potential harms of large-scale models, contrastive models [7], or web-scale multimodal data; pruning may also disproportionately reduce performance for some subgroups.The authors suggest expert specialization could instead improve performance on underrepresented groups.Source: §7 · p. 9
• Although training large models is costly, prior work finds most environmental impact occurs during inference and MoEs are more efficient there, making LIMoE a candidate for efficient, large-scale multimodal models.Source: §7 · p. 9
• Routing interference remains poorly understood; MoE findings do not transfer perfectly between NLP and vision, LIMoE shows image-text differences, and extensions to more modalities merit study.Source: §7 · pp. 9–10
• The authors identify sparse models as a natural way to scale while handling diverse tasks and data, and call for larger models and further research.Source: §7 · p. 10

A Training details

• Training uses Adafactor, 1 × 10−3 learning rate, 1 × 10−5 decoupled weight decay, cosine decay with linear warmup; models use 32 experts every two layers and 512-dimensional outputs except H/14.Architecture parameters otherwise follow ViT [13]; outputs are cast to bfloat16, text sequences use 16 tokens, images are renormalized to [-1, 1], and contrastive temperature is initialized at 10. Warmup is 40k steps for longer scaling-study models and 10k for ablations; training used Cloud TPU-v2, v3, and v4 pods.Source: A Training details · p. 14

A.1 Scaling study

• At 224px, scaling-study models trained for 781,250 steps at batch size 16,384, matching CLIP's number of examples while using a smaller batch and lower final resolution.CLIP [7] used batch size 32,768, increased resolution in its final epoch, and a larger final contrastive feature dimension; the passage says these changes improve performance.Source: §A.1 · p. 14

A.2 Ablations

• Ablation models were B/16 runs of 100,000 steps at batch size 8,192; the text entropy threshold encouraged use of at least nine experts, while the image loss was inactive.The text threshold was τT = log(9); the image threshold was τT = log(20), at which the loss was not applied.Source: §A.2 · p. 14

A.3 LIMoE-H/14

• H/14 used batch size 21,502 at resolution 288, MoE layers in 12 specified blocks, and a 3:1 default-data-to-JFT-4B mixture with class names concatenated as text.Text length was 16; entropy thresholds were τtext = log(4) for text and τtext = log(25) for images. JFT-4B was deduplicated as in previous work [16].Source: §A.3 · p. 14
• LIMoE-H/14's final soup combined eight same-run checkpoints selected greedily by ImageNet validation zero-shot accuracy across cooldowns that varied in duration and JFT-data inclusion.The method used a reverse-square-root schedule with linear cooldown, allowed checkpoints to be reused, and evaluated with the smaller CLIP prompt subset [7]. Training ran 700k steps before cooldowns, including a 125k cooldown and several 40k cooldowns; Figure 7 shows the checkpoints.Source: §A.3 · p. 14, Figure 7 · p. 15

B.1 Definitions of all the auxiliary losses

• Importance loss [1], inspired by [14], balances gate-weight profiles, while load loss [1], also inspired by [14], balances expert assignments using noisy Top-K selection probabilities.The load loss uses noisy routing and estimates an expert's probability of remaining in the Top-K when only that expert's noise is resampled.Source: §B.1 · p. 16
• Z-loss controls maximum router-activation magnitude, while the mutual-information loss captures dependence between experts and modalities rather than token-conditioned expert–token dependence.The sum of local and global entropy losses corresponds to negative mutual information between experts and tokens conditioned on modality.Source: §B.1 · pp. 16–17
• The combined auxiliary loss averages the selected losses using one regularization parameter, set to 0.04 in all experiments and reported as robust after preliminary tuning.The same parameter weights the average over whichever auxiliary losses are combined.Source: §B.1 · p. 17

B.2 In-depth analysis of global entropy threshold

• The threshold τ can be interpreted as a soft minimum: the modality must use at least S = e^τ experts to satisfy the loss.The authors describe ST = e^τ as the more intuitive soft-minimum threshold.Source: §B.2 · p. 17
• The performance analysis examines the effect of the entropy threshold, with plotted results expressed as the soft minimum ST = e^τ.The passage introduces Figure 8 but does not provide additional numerical results beyond the reported stability and invariance patterns.Source: §B.2 · p. 17, Figure 8 · p. 17
• A sufficiently high text threshold stabilizes performance; beyond that point, performance is fairly consistent, and the image threshold makes no systematic difference.Figure 8 plots the soft minimum ST = e^τ.Source: §B.2 · p. 17, Figure 8 · p. 17
• Image routing entropy is always high, while text routing entropy closely tracks its target threshold; image entropy tends to decrease as the text threshold rises.The global entropy loss is therefore generally not applied to images when their entropy exceeds the threshold. Figure 9 distinguishes pre-dispatch router entropy from post-dispatch entropy after top-K selection and capacity limits; the two are similar for text but not fully equivalent for images, where some tokens are dropped.Source: §B.2 · p. 17, Figure 9 · p. 18
• Raising the text threshold increases the number of text experts and decreases the number of image experts, behaving as a soft minimum on text expert use.Almost all experts process images, some experts process both modalities, and the figure estimates modality-specific experts by counting those needed to reach a routing success rate.Source: §B.2 · p. 17, Figure 10 · p. 18
• The analysis leaves open why increasing text expert use does not affect performance and whether text-image routing differences reflect training imbalance or fundamental modality differences.The authors note that the text entropy loss behaves as expected, increasing the number of text experts as its threshold rises.Source: §B.2 · p. 17

C.1 Scaling comparison, and architecture definitions.

• Table 4 presents the Figure 4 results alongside LIMoE-H/14 results and parameters.The supplied passages identify the results table but do not provide its cell values.Source: Table 4 · p. 19, §C.1 · p. 19

C.2 All tabular results

• The supplementary sweeps vary selected and total experts, group size, and routing-dispatch ordering, including a fixed k = 1 expert-count study.The selected-expert study adjusts local entropy loss and BPR; the group-size study frames a tradeoff between compute efficiency and stability, and Table 7 covers router variations.Source: §C.2 · p. 20, Table 7 · p. 20
• Figure 5 varies image sequence length with text sequence length fixed at 16 to study modality balancing.The supplied Table 5 caption also labels the table as covering all trained models.Source: §C.2 · p. 20, Table 5 · p. 21

Appendix B.2: Varying global entropy thresholds τtext and τimage independently

• Appendix B.2 is titled as varying global entropy thresholds τtext and τimage independently; its supplied passage identifies Table 8 as training on publicly available LAION400M data.Source: Appendix B.2 · p. 21

D.1 Unimodal evaluation with multimodal experts

• Expert capacity is set at CR × N/E tokens for N tokens and E experts, with slack CR ≥ 1 intended to let experts handle similar token loads.Source: §D.1 · p. 22
• Unimodal zero-shot evaluation can drop tokens because capacity assumes all experts are used although modality-specific experts may be inactive; LIMoE uses CR = 16.For zero-shot classification and retrieval, text and image tokens are processed separately. Alternatives include processing both modalities simultaneously or pruning modality-specific experts to evaluate at normal capacity.Source: §D.1 · p. 22

D.2 Understanding the compute costs of LIMoE

• Compute comparisons assume a full batch of images and texts, so they may not reflect workloads with many texts but few images.Zero-shot ImageNet evaluation requires 6,000 text and 50,000 image forward passes with six prompts, or 80,000 text passes with 80 prompts.Source: §D.2 · p. 22
• LIMoE is Pareto-optimal by total FLOPs and remains so by step time for zero- and 10-shot classification, while COCO retrieval gains only justify its costs.The train- and evaluation-cost patterns are similar; step-time profiling used a TPUv3 v3-32.Source: §D.2 · p. 22, Figure 11 · p. 23

E.1 Increasing the number of selected experts

• With K > 1, modified local entropy objectives and BPR scoring by summed top-K probabilities yield consistent gains, especially with merged entropy; unchanged BPR can leave models unstable.Target entropy encourages local entropy of log K, while merged entropy encourages routing weight to stay within the top K experts without requiring a uniform split.Source: §E.1 · p. 24, Table 6 · p. 24

E.2 Increasing the total number of experts

• Larger expert pools yield consistent performance improvements for LIMoE.The study varies the total number of experts, for which prior MoE research had no consensus on an optimal count.Source: Figure 12 · p. 24, §E.2 · p. 24

E.3 Router design choice

• Joint, per-modality, and disjoint router designs perform comparably; disjoint routing is more stable and needs no entropy regularization, but may not generalize to many modalities or tasks.Disjoint routing enforces modality-specific balance by assigning image and text tokens to separate expert pools.Source: §E.3 · p. 25, Table 7 · p. 25

E.4 Pruning Multimodal Experts

• Text routing is concentrated enough that 28 experts can be pruned at τtext = 4, whereas image tokens are distributed across almost all experts.Experts are ranked for pruning by the fraction of a modality’s tokens they handled during training; text evaluation can be reduced to four experts without significant performance harm, while less image pruning is better.Source: §E.4 · pp. 25–26, Figure 13 · p. 26
• When many experts are pruned, router-pred performs significantly better than router-drop on ImageNet 10-shot evaluation, while both approaches perform similarly when few are pruned.Router-pred preserves the original probability scaling by predicting weights for unavailable experts and then excluding them from use.Source: §E.4 · pp. 25–26, Figure 14 · p. 26

E.5 Grouped routing

• With four routing groups, step time falls 30% while capturing 75% of potential efficiency gains; more groups destabilize training and harm performance.Grouped routing reduces routing cost, but smaller groups make batch-wise auxiliary-loss statistics less reliable and increase the chance of nearly homogeneous groups.Source: §E.5 · pp. 26–27, Figure 16 · p. 27

E.6 Experiments on public data

• ImageNet zero-shot improved +5.0% absolute (+8.9% relative) and 10-shot improved +6.6% absolute (+13.8% relative) over a dense baseline.The B/16 models were trained on LAION-400M for 5 epochs at batch size 16,384, across three trials.Source: §E.6 · p. 27

F.1 Routing Distributions

• Training-data routing works well, with most experts handling both modalities and little dropping; zero-shot COCO and ImageNet shifts bring unbalanced expert loads and more dropped tokens.The analyses use B/32 and B/16 models with 8 experts; capacity is enforced locally per device.Source: §F.1 · p. 28, Figure 17 · p. 28, Figure 19 · p. 30

F.2 Routing Examples

• ImageNet routing examples reveal semantic patterns in many experts, especially for image tokens, while text tokens tend to use fewer experts.The examples show only patches with the highest routing confidence; the authors expect other training-data concepts to be served almost exclusively by individual experts.Source: §F.2 · p. 34

F.3 Routing for Individual Inputs

• For individual ImageNet inputs, image and text tokens are routed to experts separately and do not interact in the forward pass except through shared expert capacity.Source: §F.3 · p. 36

F.4 Routing Trajectories

• Image-token routes through the first two or three MoE layers are hard to predict, whereas text-token trajectories are more predictable in both B/32 and B/16.Source: §F.4 · p. 38, Figure 30 · p. 38, Figure 31 · p. 39

F.5 BPR rankings

• Local entropy loss and BPR favor high-pmax text tokens, but priorities can mix modalities, with some image patches processed before other text tokens.This pattern is reported for training data and zero-shot COCO and ImageNet evaluations.Source: §F.5 · p. 40

G LIMoE-H/14 Analysis

• Image experts naturally specialize in semantic concepts including body parts, textures, fauna, food, and doors.The analysis points to qualitative specialization in Figure 2.Source: G LIMoE-H/14 Analysis · p. 46
• With text:image tokens near 1:27, the entropy loss was set to require approximately four text experts, consistent with the observed token distribution across experts.Figure 39 shows routed token counts by modality for each LIMoE-H/14 MoE layer and expert; tokens above expert capacity may be dropped.Source: G LIMoE-H/14 Analysis · p. 46, Figure 39 · p. 47
• Image-token routing becomes increasingly confident across MoE layers, while most text-token maximum routing probabilities are close to one-hot under the local entropy loss.Figure 38 reports average and one-standard-deviation per-token maximum routing probabilities for training data.Source: Figure 38 · p. 46

G.1 Preliminary analysis of text routings

• Text-routing analysis groups tokens as verbs, nouns, adjectives, prepositions, or determiners using NLTK, but SentencePiece subwords prevent perfect parsing by expert.A word may be split into smaller units that are not necessarily handled by the same expert.Source: §G.1 · p. 46
• Because image tokens predominate, only 3–4 experts handle text in this setting, and preliminary routing breakdowns show signs of semantic specialization.Some experts process a mixture of token types, while others often process mostly padding tokens.Source: §G.1 · p. 46, Figure 40 · p. 47
• In Layer 14, expert 1 handles nouns and adjectives but no prepositions, determiners, or verbs; another expert 1 example handles few nouns or adjectives and instead processes padding.These examples illustrate specialization alongside padding-token routing.Source: §G.1 · p. 46, Figure 40 · p. 47