Source-linked AI summary
Scaling Laws for Native Multimodal Models
Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, Alaaeldin El-Nouby
TL;DR
The paper asks whether late fusion is inherently superior for multimodal modeling and studies native models trained from scratch. It finds no inherent late-fusion advantage, reports benefits from early fusion and sparse modality specialization, and notes that downstream-performance links remain unclear.
Problem
Whether late-fusion architectures are inherently optimal remains open, while separate pretrained components bring distinct scaling properties and may impede cross-modality learning.
Source: §1 · p. 1Method
Results
Early and late fusion perform similarly overall, with a slight early-fusion advantage at low compute; sparse models improve performance at the same inference cost.
Source: §1.1 · p. 2, §8 · p. 8Takeaways & Limitations
Early fusion offers an efficient alternative to late fusion, while sparse architectures support modality-specific specialization and improve performance within the studied setting.
Source: §1 · p. 2, §8 · p. 8The relationship between validation-loss scaling and downstream performance remains unclear, and extrapolation to extreme model sizes is open.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Building general-purpose models that can effectively perceive the world through multimodal signals has been a long-standing goal. Current approaches involve integrating separately pre-trained components, such as connecting vision encoders to LLMs and continuing multimodal training. While such approaches exhibit remarkable sample efficiency, it remains an open question whether such late-fusion architectures are inherently superior. In this work, we revisit the architectural design of native multimodal models (NMMs)-those trained from the ground up on all modalities-and conduct an extensive scaling laws study, spanning 457 trained models with different architectures and training mixtures. Our investigation reveals no inherent advantage to late-fusion architectures over early-fusion ones, which do not rely on image encoders or tokenizers. On the contrary, early-fusion exhibits stronger performance at lower parameter counts, is more efficient to train, and is easier to deploy. Motivated by the strong performance of the early-fusion architectures, we show that incorporating Mixture of Experts (MoEs) allows models to learn modality-specific weights, significantly benefiting performance.
1. Introduction
The paper tests whether late fusion is inherently better than early fusion and finds no such advantage; early fusion performs well at lower compute, while sparse early-fusion models benefit from modality specialization.
1. Introduction
- The study addresses whether late fusion is inherently optimal and whether separately pretrained components complicate multimodal scaling.The authors raise the possibility that unimodal pretraining can introduce biases that impede leveraging cross-modality dependencies.Source: §1 · p. 1
1.1. Summary of our findings
- Early and late fusion achieve similar validation loss at equal compute, while early fusion uses fewer parameters and sparse early fusion lowers loss at the cost of more training tokens.At lower compute, early fusion has a slight advantage; compute-optimal models perform similarly as compute grows.Source: Figure 1 · p. 1, §1.1 · p. 2
1. Introduction
- NMMs follow scaling laws similar to text-only LLMs, and the observed trends are similar across the training mixtures studied.The authors note that broader exploration is needed to validate the mixture finding.Source: §1 · p. 2
- Sparse early-fusion MoEs improve performance and convergence, with experts specializing by modality, especially in early and late layers.For MoEs, scaling training tokens matters more than scaling active parameters.Source: §1 · p. 2
1.1. Summary of our findings
- Modality-agnostic routing consistently outperforms routing based on predefined modality rules in sparse NMMs.Source: §1.1 · p. 2
2. Preliminaries
The preliminaries define native multimodal models and fusion and routing variants, describe a power-law framework for predicting loss and compute-optimal configurations, and outline the model and evaluation setup.
2.1. Definitions
- NMMs are trained from scratch on all modalities; early fusion processes raw text and image inputs in one transformer, while late fusion delays interaction until after separate unimodal processing.Modality-agnostic routing uses a jointly trained router; modality-aware routing follows predefined rules such as token modality.Source: §2.1 · p. 2
2.2. Scaling Laws
- The scaling analysis predicts final loss and compute-optimal model size or token count using a power-law relationship between loss, parameters, and training tokens.The framework assumes compute is proportional to parameters times tokens; FLOPs are estimated as C = 6ND, with a modified estimate for late fusion.Source: §2.2 · pp. 2–3
2.3. Experimental setup
- Models use an autoregressive transformer and are evaluated on held-out interleaved, image-caption, and text-only data.Early fusion linearly projects image patches; late fusion follows CLIP, and training uses a mixture of public and private multimodal datasets.Source: §2.3 · p. 3
3. Scaling native multimodal models
This section studies scaling across fusion architectures, training mixtures, and initialization strategies. Early fusion matches late fusion at scale while differing in compute-optimal allocations and training efficiency.
3.1. Scaling laws of NMMs
- NMM loss follows L ∝ C^-0.049, close to text-only LLM scaling; fitted exponents indicate that parameters and training tokens should scale roughly proportionally.The NMM estimates are α = 0.301 and β = 0.335; the compute-optimal exponents are a = 0.526 and b = 0.473.Source: §3.1 · pp. 3–4
- Early and late fusion have nearly identical loss-scaling exponents (−0.0492 vs −0.0494), with early fusion ahead at smaller scales and converging at larger sizes.The early-fusion advantage is also described as slight overall and strongest at smaller model sizes.Source: Figure 3 · p. 4, §3.1 · p. 4
- At fixed compute, early fusion’s compute-optimal models use more training tokens and fewer parameters, and train faster with less memory.The training-efficiency advantage becomes more pronounced as compute increases.Source: §3.1 · p. 5
3.2. Scaling laws for different data mixtures
- Across four mixtures, scaling trends stayed similar but coefficients shifted: more image-caption data lowered a and raised b, while more interleaved and text data reversed this.At fixed model size, increasing text-only and interleaved data also favored early fusion.Source: §3.2 · p. 5
3.3. Native multimodal pre-training vs.
- Fewer than 100B multimodal tokens let native models match initialized models on image captioning; interleaved and text data may require up to 1T tokens.The authors suggest native training could be more efficient for achieving the same multimodal benchmark performance.Source: §3.3 · p. 6
4. Towards multimodal specialization
The section investigates sparse early-fusion MoEs as a way to specialize parameters within a unified multimodal model. MoEs outperform dense models at matched active parameters and exhibit both modality specialization and sharing.
4. Towards multimodal specialization
- The sparse setup uses dropless top-1 routing over eight experts and a 0.01 auxiliary load-balancing loss.Each token selects its top-k experts; the reported configuration uses k = 1 and E = 8.Source: §4 · p. 6
4.2. Scaling laws for sparse early-fusion models
- MoE loss scales with compute at exponent −0.048, close to dense NMMs’ −0.049, but its multiplicative constant is lower (27.086 vs 29.574).Sparse models take longer to reach saturation and their scaling analysis favors increasing training tokens faster than active parameters.Source: §4.2 · p. 6, Figure 9 · p. 6
4.1. Sparse vs dense NMMs when scaling FLOPs
- At matched active parameters, MoEs significantly outperform dense models, especially at small sizes, with the gap narrowing as dense models grow.The comparison varies both active parameters and training-token amounts.Source: §4.1 · p. 6, Figure 10 · p. 7
4.3. Modality-aware vs. Modality-agnostic routing
- Modality-agnostic MoEs outperform modality-aware image/text experts on image-caption and interleaved data.The comparison uses experts dedicated to image or text tokens as the modality-aware alternative.Source: §4.3 · p. 7, Figure 11 · p. 7
4.4. Emergence of expert specialization and sharing
- Modality-agnostic MoEs show greater modality specialization in early and final layers, with more cross-modality sharing in middle layers.The reported visualizations are from a 1B-active-parameter early-fusion MoE trained for 300B tokens.Source: §4.4 · p. 7
5. Evaluation on downstream tasks with SFT
The downstream SFT evaluation confirms the ranking of the model configurations, while noting that their overall scores remain below current state of the art.
5. Evaluation on downstream tasks with SFT
- SFT results confirm that early fusion outperforms late fusion and MoEs outperform dense models, although overall scores remain below current state of the art.The models were native, at 1.5B scale, pretrained on 300B tokens, and fine-tuned on a small dataset.Source: §5 · p. 7
6. Related work
Related work situates the study among late-fusion systems, less-explored early fusion, and limited multimodal scaling-law and MoE research.
6. Related work
- Late fusion processes modalities separately before combining them and is well established [41] [81]; early fusion combines them earlier and has few public models [18].Source: §6 · p. 8
- Native multimodal models are trained from scratch on all modalities simultaneously [67], while some scratch-trained systems use pretrained image tokenizers [20] [70].Source: §6 · p. 8
- Few studies examine multimodal scaling; this work studies raw-input early fusion on interleaved data, unlike prior work on tokenized multimodal generation [5].Source: §6 · p. 8
- Prior multimodal MoE work covered contrastive models [50], late-fusion LLMs [38] [40], and modality-specific experts [7] [12] [60]; this study focuses on early-fusion MoEs.Source: §6 · p. 8
7. Limitations
Scaling-law coefficients appear broadly consistent across mixtures, but the study identifies limits in validating that pattern, linking validation loss to downstream performance, and extrapolating to extreme model sizes.
7. Limitations
- The observed consistency of scaling-law coefficients across training mixtures needs broader exploration for validation.The study also says prediction accuracy improves at higher FLOPs, while extrapolation to extreme model sizes remains open.Source: §7 · p. 8
- How predictably validation loss scales with compute relates to downstream performance remains unclear.The authors say this relationship warrants further investigation.Source: §7 · p. 8
- Extrapolating scaling-law predictions to extreme model sizes remains an open question.Prediction accuracy improves with higher FLOPs.Source: §7 · p. 8
8. Conclusion
The paper concludes that native multimodal models scale similarly to LLMs and that early fusion has no inherent disadvantage relative to late fusion. Sparse architectures encourage modality-specific specialization and improve performance while maintaining inference cost.
8. Conclusion
- Native multimodal models follow scaling laws similar to text-only LLMs, while early fusion is more efficient to train and outperforms late fusion at lower compute budgets.The conclusion reports no inherent advantage for late fusion.Source: §8 · p. 8
- Sparse architectures encourage modality-specific specialization, leading to performance improvements while maintaining the same inference cost.Source: §8 · p. 8
- The paper concludes that late fusion has no inherent advantage over early fusion, although both have similar scaling properties.Source: §8 · p. 8
A. Experimental setup
The experiments compare early- and late-fusion models across scaling and training configurations, using a multimodal data mixture and shared training conventions. Across the tested models, early fusion consistently outperforms late fusion as FLOPs are scaled.
A. Experimental setup
- The default training mixture uses 45% image captions, 45% interleaved documents, and 10% text-only data.Models use 24 layers, a 1k context length, and a batch of 2k samples accounting for 2M tokens; late-fusion vision encoders use a smaller learning rate when trained from scratch and are frozen when initialized from pre-trained models.Source: Table 5 · p. 13, A. Experimental setup · p. 13, Table 7 · p. 13
B. Late vs early fusion
- With mixture changes, early-fusion models improve as the proportion of interleaved documents increases.The compared early- and late-fusion models have 1.63B and 1.75B parameters, respectively.Source: Figure 15 · p. 14
B.1. Scaling FLOPs
- Early fusion consistently outperforms late fusion across the trained models as FLOPs scale, with the gap narrowing mainly as model size increases.The comparison scales model size and training tokens; increasing training tokens is not the main reason the gap decreases.Source: §B.1 · p. 14, Figure 14 · p. 14
B.2. Changing the training data mixture
The additional experiments test how data mixture, resolution, model matching, context length, and initialization affect early-versus-late fusion comparisons. The scaling-law analysis estimates compute-optimal configurations and loss trends from the training runs.
B.2. Changing the training data mixture
- Increasing text-only and interleaved data favors early fusion when model size is fixed, while increasing either data type can hurt performance on image captions.More interleaved data also harms image-caption performance, whereas more text-only data slightly improves interleaved performance; text-only and interleaved data are correlated across setups.Source: §B.2 · p. 14
B.3. Scaling image resolution is in favor of early-fusion
- Early fusion outperforms late fusion across image resolutions, with the gap widening at higher resolutions, particularly on multimodal data.Higher resolution increases visual-token count; loss on text and interleaved data also rises as resolution increases.Source: §B.3 · pp. 14–15, Figure 17 · p. 15
B.4. Early-fusion is consistently better when matching the late-fusion model size
- Early fusion outperforms late fusion when matched for total parameters and then FLOPs; when text-model size is matched, it performs better at higher interleaved-data ratios.Source: §B.4 · p. 15, Figure 18 · p. 16
B.5. Different late-fusion configuration
- With the late-fusion vision encoder fixed at 300M parameters, the performance gap narrows as the text model scales, though late fusion lags at smaller sizes.The authors interpret this as support for allocating more parameters to shared components.Source: §B.5 · p. 15, Figure 19 · p. 16
B.6. Different context lengths
- Initial experiments found that varying context length did not significantly affect the early-versus-late fusion comparison.The paper uses a 1k context length and ignores context-length effects in its compute estimate.Source: §B.6 · p. 15, Figure 20 · p. 16
B.7. Initializing from LLM and CLIP
- With DCLM-1B and CLIP-ViT-L initialization, early fusion can match late fusion on text and interleaved data when trained longer, but closing the image-caption gap is harder.Including pretraining cost, early fusion requires significantly longer training to compensate for the vision encoder’s pretraining cost.Source: §B.7 · p. 15, Figure 21 · p. 16
C. Scaling laws
- Scaling-law coefficients are derived by fitting loss across runs and regressing compute-optimal model-size relationships in log space.For loss versus compute, the method constructs minimum-loss points across FLOP budgets and fits a log-space linear relationship; regression estimates are close to the closed-form coefficients for early fusion.Source: C. Scaling laws · p. 17, Table 9 · p. 17, Figure 22 · p. 17
C.4. Scaling laws for different target data type
Across target data types, training mixtures, and model configurations, the scaling laws show broadly similar trends with some variation in coefficients. They predict observed losses reasonably well, including for an 8B model, while sparse-model coefficients are conditioned on the MoE sparsity configuration.
C.4. Scaling laws for different target data type
- On image-caption data, early and late fusion scale similarly but early fusion has a lower multiplicative constant; MoEs improve overall performance on text and interleaved data while learning slightly more slowly.The model learns image-caption data faster than interleaved data despite equal 45% data shares; text-only learning is slower, likely because it receives 10% of the mixture.Source: §C.4 · pp. 17–18
C.5. Scaling laws for different training mixtures
- Increasing image-caption data from 30% to 40% raises the absolute scaling exponent from 0.056 to 0.061, while mixture changes produce similar overall scaling trends.Text-only scaling coefficients do not change significantly with its mixture proportion.Source: §C.5 · p. 18
C.7. Scaling laws for sparse NMMs.
- Sparse NMM scaling laws use active rather than total parameters and condition coefficients on the MoE sparsity configuration; model-size exponents exceed training-token exponents.The scaling-law terms for active parameters decompose into two components.Source: §C.7 · pp. 18–19
C.6. Scaling laws evaluation
- Scaling laws predict observed losses with high accuracy, particularly at lower losses and larger FLOPs, and estimate 8B-model performance with reasonable accuracy.Bootstrapping gives more precise estimates for β than α, attributed to having fewer model sizes than token counts.Source: §C.6 · p. 18, Figure 24 · p. 19
D. Discussion and Limitations
The discussion identifies limits in mixture coverage, downstream-task evidence, and extrapolation to extremely large models. The MoE experiments report modality specialization and scaling behavior, but use a narrow set of configurations.
D. Discussion and Limitations
- Scaling-law coefficients across mixtures need broader validation; downstream performance and extrapolation to extremely large models remain open questions.High-resolution early fusion greatly increases vision-token counts, so alternative approaches may be necessary.Source: D. Discussion and Limitations · p. 19
- MoE scaling findings are limited to top-1 routing with 8 experts; other routing, load-balancing, or expert implementations may yield different findings.The paper reports that this single configuration worked reasonably well in its setup.Source: D. Discussion and Limitations · p. 19
E.1. MoEs configuration
- The authors vary the number of experts and top-k, with a sample of SFT results reported in Table 13.The supplied caption identifies Table 13 as SFT results across MoE configurations.Source: §E.1 · p. 19, Table 13 · p. 20
E.2. MoEs specialization
- MoE experts show modality-specific specialization, especially in early layers; specialization declines through mid-layers and rises again in the final layers.Some experts are shared between text and image modalities, unlike hard-routed or predefined modality-specific experts.Source: Figure 25 · p. 19, §E.2 · p. 19
- MoEs have scaling exponents close to dense models, but lower overall loss and take longer to saturate.Figure 27 compares late-fusion dense, early-fusion dense, and early-fusion MoEs across image-caption, interleaved, and text-only validation data.Source: Figure 26 · p. 20, Figure 27 · p. 21