Source-linked AI summary
Scaling Native Multimodal Pre-Training From Scratch
Haoyuan Wu, Aoqi Wu, Hai Wang, Jiajia Wu, Jinxiang Ou, Bei Yu
TL;DR
Native multimodal pre-training lacks established compute-optimal scaling guidance, limiting predictable allocation of model size, tokens, and data mixtures. This paper derives separate scaling laws and a joint efficiency frontier, finding distinct language and multimodal allocation behaviors plus positive cross-modal transfer.
Problem
It remains unclear whether native multimodal pre-training follows compute-optimal scaling laws and whether language and multimodal resource allocations align under fixed compute.
Method
The study decouples language and multimodal objectives and estimates their compute-optimal frontiers using IsoFLOP profiles and training-curve envelopes.
Results
Language allocation is largely composition-invariant, whereas multimodal allocation depends strongly on data composition, with text-heavy mixtures favoring larger models; native multimodal training also improves text spatial reasoning.
Takeaways & Limitations
The resulting language-multimodal Pareto frontier provides quantitative guidance for scaling model size, text tokens, multimodal tokens, and data mixtures.
Takeaways & Limitations
Because solid multimodal validation metrics are lacking, the study uses smoothed training loss as a proxy for test loss.
Abstract
from arXiv · showhide
Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain systematically uncharacterized. To address this gap, we investigate the optimal model size and token count for training a transformer-based vision-language model under a fixed computational budget. We demonstrate that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct scaling behaviors. The language allocation law is largely invariant to the composition of the data, indicating stable language learning regardless of the multimodal data ratio. Conversely, the multimodal allocation law is highly sensitive to this composition. Specifically, text-heavy mixtures become compute-efficient only at larger model scales, shifting the optimal resource allocation toward greater model capacity. Additionally, by modeling the influence of data composition on compute laws and allocation exponents, we derive an efficiency frontier specifying precise configurations of model size, token count, and data mixture. Downstream evaluations further reveal that native multimodal pre-training induces positive cross-modal transfer, thereby enhancing pure-text spatial reasoning and enabling robust multimodal in-context learning. In summary, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.
1 Introduction
The introduction motivates native multimodal pre-training as a way to overcome text-only limitations and late-fusion asymmetry, then establishes compute-optimal scaling laws and downstream benefits for this paradigm.
- Motivation: Text-only pre-training cannot ground multimodal concepts in the physical world, motivating multimodal training on diverse data.Multimodal pre-training incorporates diverse multimodal data to mitigate this limitation.
- Motivation: Native multimodal pre-training trains models from scratch, enabling deep modality integration and shared representational capacity while posing fixed-compute allocation challenges.It is proposed to resolve the fundamental asymmetry introduced by late-fusion architectures.
- Scaling laws: Compute-optimal scaling laws derived with IsoFLOP profiles and training-curve envelopes show predictable optimal loss and power-law scaling of model size and token count.The close agreement between estimators indicates that the results reflect data-distribution properties rather than a specific functional form.
- Scaling laws: Language allocation remains largely composition-invariant, whereas multimodal allocation is composition-sensitive, with text-heavy mixtures becoming efficient only at larger model scales.The resulting language-multimodal Pareto frontier specifies deployable configurations of model size, text tokens, and multimodal tokens for each compute budget.
- Downstream implications: Native multimodal pre-training improves pure-text spatial reasoning and enables robust multimodal in-context learning.Spatial understanding acquired through multimodal training generalizes to unimodal text tasks.
2 Preliminaries
This section formulates compute-optimal native multimodal pre-training as allocating a fixed budget between model size, training tokens, and multimodal data ratio. It defines loss-based evaluation and separate language/multimodal scaling analyses, with both objectives following power-law allocation relationships.
- Problem formulation: Under fixed compute C, the allocation problem optimizes model size N, training tokens D, and data composition through the multimodal ratio r.N is the number of activated non-embedding parameters, and the allocation determines how resources are divided between model capacity and tokens.
- Problem formulation: The computational constraint is C = 6ND, and final pre-training loss L(N, D) is minimized subject to this fixed budget.This formalization makes compute strictly determined by model size and token count.
- Evaluation assumption: Smoothed training loss serves as the test-loss proxy because multimodal validation metrics are limited and each token is seen roughly once during pre-training.The proxy standardizes evaluation across language and multimodal objectives.
- Objective decomposition: Because language and multimodal objectives share parameters, the study decouples them and analyzes their compute allocations independently.This tests whether the objectives have analogous scaling laws or conflicting optima under shared capacity.
- Scaling laws: Both objectives exhibit compute-optimal allocations that follow power-law relationships, governed by scaling exponents a and b.The exponents characterize how the optimal allocation changes with compute.
- Estimation methodology: The IsoFLOP profile is the primary allocation estimator, while the training-curve envelope independently cross-validates it; joint Pareto analyses use only IsoFLOP estimates.IsoFLOP minima identify Nopt(C), after which Dopt(C) = C/(6Nopt) follows algebraically.
3 Scaling Native Multimodal Pre-Training
Native multimodal scaling has predictable compute-optimal allocations, but language and multimodal objectives respond differently to data composition. Under unified compute, increasing the multimodal ratio shifts resources from parameter growth toward larger token budgets.
- Language objective: Language compute-optimal allocations follow power-law frontiers whose slopes vary minimally across mixture ratios r.IsoFLOP minima and training-curve envelopes agree closely.
- Multimodal objective: Multimodal compute-optimal scaling is composition-variant: increasing r from 0.1 to 0.3 substantially decreases the optimal model-size exponent.The decline is consistent across IsoFLOP profiles and training-curve envelopes.
- Joint optimization: Under unified compute, r = 0.1 yields optimal parameter scaling Nopt ∝C0.69total, whereas r = 0.3 reduces it to C0.66total.The higher multimodal ratio imposes larger data requirements across the system.
- Joint optimization: At r = 0.3, system-wide token allocation scales more aggressively, with allocation proportionality ∝C0.34total.The unified model therefore constrains parameter expansion in favor of substantially larger token budgets.
4 Downstream Implications
Native multimodal pre-training preserves aggregate text performance while improving text-only spatial reasoning through cross-modal transfer. Multimodal in-context learning emerges and strengthens with model scale and training data, with gains concentrated in spatial reasoning.
- Text capabilities: Aggregate accuracy across 16 text benchmarks remains unaffected by multimodal data, with average scores deviating by less than one percentage point at every scale.The comparison fixes 250B text tokens and varies multimodal data from 0B to 75B across models from 71M to 3B.
- Cross-modal transfer: Multimodal models consistently outperform text-only baselines on text-only abstract spatial reasoning tasks, with the gap widening at larger scales.Figure 9 compares r = 0.3 multimodal models with r = 0 text-only baselines on SpatialEval’s MazeNav and SpatialMap subtasks.
- Multimodal in-context learning: Multimodal in-context learning gains increase from near zero for A71M to +1.80 points for A874M and +2.43 points at A3B.At the largest scale, accuracy consistently improves with more shots, indicating that in-context learning emerges naturally from native multimodal pre-training.
- Multimodal in-context learning: Few-shot benefits emerge later during training, with the onset of shot-dependent improvements depending heavily on model capacity.Early 0-, 1-, and 3-shot curves are virtually indistinguishable, while the expected ordering appears during later training stages.
- Task-specific effects: Few-shot gains are most pronounced and consistent on spatial reasoning, while OCR- and recognition-oriented benchmarks plateau or degrade.Structured-diagram and aggregate suites also show notable gains, whereas OCR and recognition tasks do not benefit consistently from templates.
- Data scaling: A3B few-shot gains trend upward with total training tokens for all three r values, with 3-shot gains generally exceeding 1-shot gains.The aligned trends indicate that multimodal in-context learning strengthens as the model sees more training data rather than being tied to a particular r.
5 Related Works
Prior work largely uses late-fusion vision-language models that combine separately pretrained unimodal components, while scaling-law research provides compute-guided principles for resource allocation. These foundations motivate studying native multimodal scaling under fixed computational budgets.
- Multimodal Pre-Training Paradigms: Late-fusion vision-language models typically combine a pretrained language model and vision encoder through a lightweight projection layer for multimodal adaptation.This modular design efficiently reuses robust unimodal checkpoints but introduces a fundamental asymmetry.
- Compute-Optimal Scaling Laws: Scaling laws relate model performance to scale, training data, and compute, guiding resource allocation under fixed training budgets.Kaplan et al. established power-law relationships between loss and scale, while Hoffmann et al. refined compute-optimal analysis using IsoFLOP profiles and training-curve envelopes.
6 Conclusion
The paper establishes compute-optimal scaling laws for native multimodal pre-training, showing that language and multimodal objectives scale predictably but follow fundamentally different allocation laws. Language scaling remains largely insensitive to data composition.
- 6 Conclusion: The study establishes compute-optimal scaling laws that systematically characterize native multimodal pre-training.The compute-optimal frontier was estimated using complementary methodologies.
- 6 Conclusion: Language and multimodal objectives both scale predictably, but follow fundamentally different allocation laws.The joint training objective was decoupled into language and multimodal components to analyze their scaling behavior.
- 6 Conclusion: Language scaling remains largely insensitive to data composition.
A Implementation Details
Native multimodal pre-training uses Muon with specified regularization and optimization settings, a large global batch, and a 4096-token sequence limit. Loss is computed only on text tokens, with vision tokens masked out.
- A Implementation Details: Muon optimization uses 0.1 weight decay and a 1.0 gradient clipping threshold.Training also follows a warmup-stable learning-rate schedule.
- A Implementation Details: Loss is computed only on text tokens, while vision tokens are masked out from loss computation.This objective design follows Kimi et al. (2026).
B Experiment Results
The experiment results comprehensively evaluate A3B training dynamics and final model-family performance across text and multimodal benchmarks. Analyses vary pre-training token budgets, multimodal data ratios, model sizes, and multimodal in-context settings.
- Training dynamics: A3B training dynamics are evaluated across successive pre-training token budgets and multimodal data ratios r ∈{0, 0.1, 0.2, 0.3}.Text results appear in Tables 2–5, while multimodal results appear in Tables 6–8; the text component uses 250B tokens.
- Final text performance: Final text performance is reported across model sizes ranging from 71M to 3B activated parameters for four multimodal data ratios.All models use 250B text tokens, with multimodal training conditions of 0B, 25B, 50B, or 75B tokens.
- Multimodal evaluation: Multimodal benchmark performance is evaluated in 0-shot, 1-shot, and 3-shot settings across model sizes from 71M to 3B activated parameters.These evaluations use models trained on 250B text tokens and 75B multimodal tokens.