PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws for Multimodal Data Mixtures

Aditi Khandelwal, Ayush K. Tarun, Yixuan Xu, Imanol Schlag, Golnoosh Farnadi, Siva Reddy, Luke Zettlemoyer, Antoine Bosselut

arXiv:2610.14423v1cs.CLcs.LG

TL;DR

The paper addresses the limited public understanding of how text, vision, and audio mixtures affect multimodal pretraining. It fits a unified scaling law to MoE experiments and finds modality-specific compute optima, asymmetric cross-modal interference, and diminishing returns from repeated non-text data.

  • Problem

    Public evidence lacks a unified account of text, vision, and audio mixtures that considers data imbalance, repeated data, and cross-modal effects.

    Source: §1 · pp. 1–2

  • Method

    The authors develop a repetition-aware multimodal scaling law for MoE pretraining and fit it using controlled runs across text, vision, and audio mixtures.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Audio favors extra model capacity, text and vision favor scaling parameters and tokens together, cross-modal interference is asymmetric, and repeated non-text data has sharply diminishing returns after roughly two passes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework can help estimate the cost of adding a modality and guide choices between repeating scarce data and scaling model capacity.

    Source: §8 · p. 10

    The fitted law is context-specific rather than universal, and validation beyond ≫10B active parameters or >1T tokens remains future work.

    Source: §9 · p. 10

Abstract

from arXiv · show

Frontier AI systems are increasingly natively multimodal, jointly pretrained on multiple modalities, such as text, vision, and audio. However, the scientific understanding of the multimodal data mixtures used to pretrain these systems --- how much of each modality to mix, and at what potential interference cost to the others --- is largely absent or gatekept. In this work, we derive data mixing scaling laws for three modalities: text, vision, and audio. Multimodal Large Language Models (MLLMs) increasingly adopt Mixture-of-Experts (MoE) architectures, as MoEs scale efficiently and partially limit cross-modal interference. We therefore use MoEs to conduct 268 experiments ranging from 457M to 8.3B parameters, on up to 150 billion tokens. Our work leads to multiple actionable insights into multimodal data mixing for training MLLMs. First, compute optima are modality-specific: audio modality is model-heavy, favoring scaling parameters over tokens, while text and vision require balanced scaling of model size and data. Second, repeating vision or audio data beyond 2x yields negligible benefits. Finally, cross-modal interactions are highly asymmetric: vision strongly interferes with audio, whereas audio's interference with learning text and vision is negligible. Overall, we provide a principled foundation for understanding the multimodal data mixtures needed to train frontier MLLMs.

1 Introduction

The paper addresses the lack of a unified, repetition-aware account of text, vision, and audio mixtures, then fits a multimodal scaling law and tests its predictions. It finds modality-specific compute optima, asymmetric interference, and diminishing returns from repeated non-text data.

1 Introduction

  • A unified framework is needed because prior work covers parts of multimodal scaling, but not joint text, vision, and audio mixtures that account for repetition and data imbalance.The paper introduces a scaling law for target-modality validation loss in the presence of other modalities and fits it on MoE runs from 457M to 8.3B parameters, with up to 150B tokens per run.Source: §1 · p. 2
  • Audio favors spending extra compute on model capacity, while text and vision favor growing parameters and tokens together.The paper also finds that vision strongly disrupts audio, whereas audio's interference with text and vision is negligible; repeated vision and audio data show sharply diminishing returns after roughly two passes.Source: §1 · p. 2
  • Vision is the most disruptive source modality, especially for audio, while audio's interference with text and vision is negligible.The introduction characterizes audio as effectively inert in those mixtures, but costly to mix when audio is an evaluation target.Source: §1 · p. 2
  • Across three held-out runs, 8 of 9 modality loss predictions match observed validation loss within 2.6%, and 6 of 9 within 1.1%, with correlation > 0.9.The fitted law is tested on larger compute budgets, out-of-distribution mixture compositions, and continual pretraining.Source: §1 · p. 2
  • At every model scale, repeating 5B unique non-text tokens 5× underperforms training on roughly 15B unique tokens.The comparison averages validation loss across modality-specific evaluation sets for text+vision and text+audio training at 100B total tokens.Source: §1 · p. 2, Figure 2 · p. 3

2 Method

The method extends the standard scaling law by replacing raw model capacity and token count with target-modality effective quantities. These account for capacity sharing, directional cross-modal transfer, and diminishing returns from repeating scarce data.

2.3 Unified Multimodal Scaling Law

  • The unified law models each target modality's validation loss using effective capacity and effective data that account for mixture composition.It reduces to the standard unimodal law in the unimodal, non-repeated limit and generalizes prior bimodal interaction laws [1] to directional, unbalanced, repetition-aware mixtures.Source: §2 · p. 3, §2.3 · p. 5

2.1 Mixture-dependent Capacity Sharing N (m)

  • The capacity-sharing coefficient c_m captures whether non-target mixture mass reduces or increases the effective capacity available to modality m.When c_m < 0, non-target data dilutes capacity; the first-order penalty diminishes as model capacity increases, and the scalar formulation avoids overfitting on limited-scale data.Source: §2.1 · pp. 3–4

2.2 Effective Target-Modality Data D(m)

  • The repetition function ϕ_m(r_m) treats the first pass as fully valuable, while repeated passes help with diminishing marginal returns.Its saturation level R*_m represents the maximum additional fresh-data-equivalent epochs available through repetition.Source: §2.2 · p. 4
  • Directional transfer coefficients v_k→m encode whether source modality k helps or interferes with target modality m, while the exponential form keeps effective data positive.Unlike additive effective data, this log-scale formulation remains positive even under severe interference or highly skewed mixtures.Source: §2.2 · pp. 4–5

2 Method

  • The effective-data term combines repetition correction with transfer from co-trained modalities, complementing the law's effective-capacity adjustment.The formulation starts from the standard Chinchilla form and modifies the capacity and data terms to capture multimodal effects.Source: §2 · p. 3, §2.2 · p. 5

3 Experimental Setup

The experiments fit the scaling framework using controlled MoE pretraining runs across six parameter scales and curated text, vision, and audio mixtures. The grid isolates unimodal scaling, repetition, pairwise transfer, and three-way interference.

3 Experimental Setup

  • The controlled suite spans six MoE parameter scales from 457M to 8.3B total parameters and uses curated multimodal mixtures.All runs are autoregressive and evaluated on held-out modality-specific validation sets; their losses jointly fit the scaling law.Source: §3 · pp. 5–6
  • Models vary hidden dimension from 384 to 2048 while holding core architecture choices fixed, including 32 layers and 16 routable experts.Two experts are selected per token; the setup uses one shared expert and untied embeddings.Source: §3 · pp. 5–6
  • The study uses high-quality, permissively licensed modality-specific datasets and tunes learning rate and batch size before running the experiment grid.The chosen settings are a 1× Step Law learning rate and a 2× Step Law batch size; three-seed checks on the smallest and largest models found < 0.3% validation-loss variance.Source: §3 · p. 6
  • The experiment grid covers unimodal baselines, text–vision and text–audio mixtures, repetition ablations, vision–audio mixtures, and three-way mixtures.Three-way runs use 150B total tokens, with 100B text tokens and varied vision/audio allocations.Source: §3 · p. 6

4 Unimodal Scaling Laws

This section asks how compute should be divided between model size and tokens for each modality. Text and vision favor balanced scaling, while audio favors a more model-heavy allocation.

4 Unimodal Scaling Laws

  • Vision has the largest fitted exponents in this setup (αvis=0.50, βvis=0.51), exceeding text’s corresponding 0.34 values and audio’s model exponent of 0.32.The authors report that increasing either model size or training tokens reduces vision loss faster than for the other modalities.Source: §4 · p. 7, Table 1 · p. 7
  • Text’s α=β=0.34 yields a balanced split close to Chinchilla, so standard text-only recipes remain reasonable for the text portion of a multimodal mixture.The section distinguishes this text-specific conclusion from the more model-heavy audio allocation.Source: §4 · p. 7
  • Nopt ∝ C^0.62 and Dopt ∝ C^0.38 for audio, versus balanced a=b=0.50 splits for text and vision.The fitted compute-optimal exponents show audio favors model size, while text and vision divide compute evenly between model size and tokens.Source: Table 1 · p. 7, §4 · p. 7

5 Modality Data Mixing: Transfer, Interference, and Repetition

The section examines directional transfer, repetition, and capacity sharing in multimodal mixtures. It finds asymmetric interference, rapidly diminishing returns from repeated non-text data, and the greatest capacity dilution for audio.

5 Modality Data Mixing: Transfer, Interference, and Repetition

  • Audio transfer to text and vision is insignificant, while vision most strongly disrupts audio (vvision→audio=−0.795); text and vision also interfere mutually.The fitted transfer coefficients for text→vision and vision→text are −0.583 and −0.558, respectively.Source: §5 · p. 7, Figure 3 · p. 8
  • Task-specific image fits show positive text and audio transfer to image understanding, while image generation drives the negative joint text→vision transfer.The separately refitted text→vision coefficients are +0.400 for understanding and −0.716 for generation.Source: §5 · p. 7
  • By epoch three, another pass is worth 23% of unique vision data and 43% for audio; by epoch five, those values fall to 3% and 14%.The fitted repetition limits are about 2.0 fresh-data-equivalent epochs for vision and 2.76 for audio.Source: §5 · pp. 7–8, Figure 3 · p. 8
  • At a 10% target-token share, effective parameters retain ≈0.66N for text, ≈0.47N for vision, and ≈0.19N for audio.All three capacity-sharing coefficients are negative; audio’s effective capacity is eroded most by non-target modality data.Source: §5 · p. 8

6 Validation at Scale and Practical Implications

The fitted law is tested on held-out runs, including continual pretraining, and applied to an example mixture optimization. Predictions closely match observed validation losses across the tested settings.

6 Validation at Scale and Practical Implications

  • Validation spans a larger, text-skewed MoE run and two continual-pretraining runs, a regime absent from the fitted data.The first run uses a 12.6B-A3.8B MoE trained for 200B tokens; the two CPT runs use the 8.3B-A2.6B scale and start from a text-only checkpoint.Source: §6 · pp. 8–9
  • Routing analysis finds no modality-wise expert segregation, suggesting capacity dilution arises because modalities share experts.This provides a mechanistic perspective on the fitted capacity-dilution effect.Source: §6 · p. 8
  • Eight of nine modality-level predictions are within 2.6% of observed validation loss, and six are within 1.1%, with Pearson r=0.999.The authors report that these results suggest extrapolation beyond the training distribution across the three held-out settings tested.Source: §6 · p. 9
  • For a 120B-A29B model trained on 15T tokens, the example optimization selects 9.7T text tokens, 1T unique vision tokens repeated 2.6 times, and 500B audio tokens repeated 5.3 times.The example illustrates context-specific optimal training mixtures, not universal coefficients.Source: §6 · p. 9

7 Related Works

Related work covers scaling laws across modalities and sparse architectures, along with studies of multimodal interference and data mixtures. The section identifies non-text repetition limits and unified multimodal scaling as areas with less systematic coverage.

7 Related Works

  • Scaling laws extend from language [25] to autoregressive image, video, and image-text modeling [17], sparse MoE [63] [59], and individual vision and audio modalities [53] [36].The section also notes broader scaling formulations across tasks and modalities [8].Source: §7 · p. 9
  • Mixed-modal scaling studies include bi-modal formulations [1], native early-fusion models [60] [53], modality-specific tokenization [57], and MiCo across 10 modalities [69].MM1 [37] reports sensitivity to data mixtures and tokenization but lacks a scaling formulation.Source: §7 · p. 9
  • Text repetition has remained beneficial up to 4× [40], while systematic studies of non-text repetition limits are lacking.The section also notes that MoE approaches route tokens to modality-aware experts to reduce negative transfer and compute [32] [29].Source: §7 · p. 9

8 Conclusion

The framework treats multimodal scaling as a modality-specific resource-allocation problem, with distinct compute optima, asymmetric interference, and repetition limits. Its findings also suggest a practical role in planning multimodal training and reducing costly trial-and-error.

8 Conclusion

  • Text and vision favor balanced growth in parameters and tokens, while audio benefits more from added capacity; audio is about 4× more sensitive than text and 2× more than vision to capacity sharing.The capacity-sharing sensitivity is consistent with audio’s model-heavy compute-optimal frontier.Source: §8 · pp. 9–10
  • Audio’s interference with text and vision is negligible, whereas text and vision interfere with one another, with vision especially disruptive to audio.Vision is the most expensive modality to mix when audio is an evaluation target.Source: §8 · p. 9
  • Vision and audio show sharply diminishing returns after roughly two passes over the same data; beyond that, compute is better spent on unique tokens or a larger model.This repetition tolerance differs from text, whose observed tolerance does not directly carry over to continuous modalities.Source: §8 · p. 9
  • The framework can help estimate the cost of adding a modality, decide whether scarce data should be repeated or replaced with model scaling, and identify likely sources of representation degradation.The authors present it as a planning tool for reducing reliance on expensive trial-and-error searches.Source: §8 · p. 10

9 Limitations and Future Work

The framework is context-specific rather than a universal prescription, and the experiments document modality-specific effects of learning-rate and batch-size choices, repetition, and mixture composition.

9 Limitations and Future Work

  • Validation at substantially larger scales, extension to modalities such as long-form video, and a mechanistic explanation of audio’s limited interference remain future work.The paper reports extrapolation to a larger from-scratch model and a continual-pretraining setup, but not validation beyond the stated scale boundary.Source: §9 · p. 10
  • The scaling law is best treated as a context-specific fitting framework: its coefficients may shift with tokenizers, loss objective, MoE hyperparameters, and dataset quality.The authors expect qualitative ordering between models or mixtures to be more robust to these variations.Source: §9 · p. 10

A Impact Statement

  • The impact statement identifies potential compute and environmental savings from avoiding redundant ablations, while acknowledging possible misuse through higher-quality deepfakes and voice clones.The authors characterize the contribution as foundational rather than directly capability-unlocking.Source: A Impact Statement · p. 17

B Relation to Additive Bimodal Scaling Laws

  • The formulation generalizes prior bimodal interaction laws to directional transfer, arbitrary mixture weights, repeated data, and positive effective data under negative transfer.It omits a constant asymptotic synergy term for target-modality loss, but permits a mutual-information offset for joint likelihood evaluation.Source: B Relation to Additive Bimodal Scaling… · p. 17

C Learning Rate and Batch Size

  • Across both model sizes and all three modalities, 1× learning rate with 2× batch size yields the lowest or near-lowest loss; smaller batches degrade performance.The Step Law learning rate remains appropriate, while the preferred batch size is twice its predicted value.Source: C Learning Rate and Batch Size · p. 18, Figure 5 · p. 19

D Training Configurations

  • The experiments use high-quality permissive data aligned with real-world MLLM mixtures, across text, vision, and audio; the full sweep comprised 268 runs on 64 GH200 GPUs.The audio mixture combines audio-only examples with paired speech–text data; vision includes standard and reversed caption-image pairs.Source: E Datasets · pp. 18–19, D Training Configurations · p. 18

F.1 Effect of repeating vision data

  • Repeating vision 5B tokens five times performs worse than 15B unique tokens above half a billion parameters; repeating audio 5B tokens five times yields loss roughly equivalent to 10B unique tokens.Text-only performance remains stable across vision-data compositions, while text evaluation stays flat as vision and audio proportions change.Source: §F.1 · p. 21, §F.2 · p. 21, §F.3 · p. 21

F.3 Interference between modalities

  • With text fixed at 100B tokens, vision and audio evaluation losses track the absolute data volume of their own modality, while text evaluation remains flat across mixture ratios.The authors suggest MoE routing may mitigate gradient conflicts, but describe the observed curves as predictable and linear.Source: §F.3 · pp. 21–22

G Goodness-of-Fit Results and Scaling Law Variants

The law is stable across interpolation and random-subset refits, and its M6 predictions improve as more model scales enter the fit. Ablations and variant comparisons show that repetition saturation and capacity dilution matter, while a more complex directional-capacity form offers no consistent advantage.

Cross-validation (Interpolation) across model scales

Random-subset resampling of the configuration grid

Extrapolation to the largest scale

Component ablations

  • 49.4% and 38.4% are the overall RMSElog increases when repetition saturation and capacity dilution are removed, respectively.Removing cross-modal transfer changes worst-case error from 2.16% to 2.61%.Source: G Goodness-of-Fit Results and Scaling… · p. 24
  • 8.27% versus 10.55%, 9.96%, and 9.30% is the R1 maximum relative error for the full law versus its component-ablated variants.The full law performs much better when fitting uses only M1 and M2; the variants perform comparably in R2–R4.Source: G Goodness-of-Fit Results and Scaling… · p. 24

Directional-capacity variant

  • The six-coefficient directional-capacity variant is comparable on R1–R3 but slightly worse on R4: 2.54% versus 2.46% maximum relative error on M6.Its vision errors are strictly worse than the default, so the paper retains the three-coefficient form.Source: G Goodness-of-Fit Results and Scaling… · pp. 24–25

H Scaling Law Validation Results

The fitted law predicts held-out runs closely overall, with one larger error in non-text-heavy continual pretraining that the authors hypothesize longer training could reduce. The full law has the lowest maximum relative error in its component ablation, while the directional-capacity variant has worse vision errors than the default.

Out-of-domain datapoints

  • For out-of-domain datapoints, all obtained loss values are within less than 2.6% error of predictions, except the test loss in non-text-heavy CPT, which has 6.73% error.The authors hypothesize that longer training would stabilize text loss and bring it closer to their law’s prediction.Source: H Scaling Law Validation Results · p. 25

Component ablations

  • The full law achieves the minimum maximum relative error among the component ablations.The table also reports relative RMSElog increases when each component is removed.Source: Table 7 · p. 26

Directional-capacity variant

  • The directional-capacity variant has strictly worse vision errors than the non-directional version across the reported extrapolation comparison.The table compares errors on M6 across training-prefix regimes.Source: Table 8 · p. 26

I Routing Analysis

Routing analyses find that modalities use broadly the same experts, consistent with capacity dilution arising from sharing rather than partitioning the expert pool. The fitted law's held-out predictions also remain close on continual-pretraining runs, including the specified text-heavy and non-text-heavy mixtures.

Routing analysis

  • Negative capacity-dilution coefficients indicate that non-target data shrinks a target modality's effective parameter budget; routing evidence points to modalities sharing experts rather than partitioning them.The text describes per-modality validation assignments compared using normalized Jensen–Shannon divergence.Source: I Routing Analysis · pp. 26–27, Table 11 · p. 27
  • At M6, text–vision and text–audio routing divergences are 0.003 and 0.003–0.004, and divergences stay below 0.03 across scales.Routing is diffuse, close to uniform, and expert loads are balanced across modalities; the authors hypothesize load-balancing loss may partly explain this pattern.Source: I Routing Analysis · pp. 26–27, Table 11 · p. 27

Continual-pretraining validation

  • 8 of 9 held-out modality losses are predicted within 2.51% without retuning, including runs using 10B text, 20B vision, and 20B audio or 40B text, 5B vision, and 5B audio.The two continual-pretraining runs use the same text-only checkpoint.Source: Table 10 · p. 27, Figure 9 · p. 27

Related papers

Loading 2610.14423v1…