PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Junlin Han, Shengbang Tong, David Fan, Minghao Chen, Philip Torr, Filippos Kokkinos, Mike Lewis

arXiv:2608.05000v2cs.CVcs.LGcs.MM

TL;DR

Unified multimodal pretraining remains governed largely by heuristics, with its modality interactions underexplored. This paper uses controlled real-world and synthetic experiments to study knowledge flow, synergy, and training dynamics, finding asymmetric, concept-dependent transfer, benefits from early joint training, and efficient recipes for strong generative performance.

  • Problem

    The fundamental mechanisms governing modality interactions in unified multimodal pretraining remain underexplored, leaving the expanding design space guided largely by heuristics.

  • Method

    The paper systematically probes multimodal pretraining through controlled real-world and synthetic experiments spanning knowledge flow, modality synergy, architecture, tokenizer designs, and training timing.

  • Results

    The study finds asymmetric, concept-dependent multimodal transfer, synergy shaped by task complexity and architectural sharing, and early joint training that prevents vision laziness.

  • Takeaways & Limitations

    Unified multimodal models can co-evolve modalities efficiently, with strong generative fidelity achieved using only a small fraction of the training budget.

  • Takeaways & Limitations

    The synergy analysis isolates one modality’s task complexity while holding the other modality completely fixed, limiting the scope of that evidence.

Abstract

from arXiv · show

Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.

1 Introduction

The paper addresses the underexplored mechanisms governing unified multimodal pretraining, a field still navigated largely by heuristics and often built by retrofitting vision onto pretrained language models. It develops four insights covering knowledge flow, modality synergy, early unification, and scalable pretraining recipes.

  • Motivation: Unified multimodal pretraining remains underexplored, leaving the field to navigate its increasingly complex design space largely by heuristics.Existing approaches often retrofit pretrained LLMs or MLLMs, treating vision as a post-hoc alignment module.
  • Knowledge Flow: Knowledge Flow examines transfer among language, visual understanding, and visual generation, revealing starkly asymmetric knowledge transfer.The study also uses a cleaner controlled setting to separate visual understanding–generation interplay from real-world distributional confounders.
  • Modality Synergy: Modality Synergy identifies data and task complexity as conditions determining whether modalities compete or synergize, and locates relevant effects within Transformer architecture.Simplified language or vision tasks can promote cross-modal synergy, while shared attention and normalization with modality-specific feed-forward layers are identified as promising designs.
  • Early Unification: Early Unification shows that vision must be introduced early and trained jointly, because modalities need to co-evolve rather than follow isolated curriculum steps.The study uncovers a “vision laziness” phenomenon associated with delayed visual integration.
  • Recipes: Recipes combine optimized asymmetric data mixing, parameter-sharing architectures, and early unified training, then validate scalability with 13.5B MoE models trained on 2T tokens.The scaled evaluations use controlled, single-variable comparisons and provide a practical baseline for multimodal foundation models.

2 Experimental Setup

The experiments use a unified Transfusion-based decoder-only Transformer that jointly models text and vision, with multiple visual tokenization configurations and standardized training data, optimization, and evaluation protocols. Language, visual understanding, and visual generation are assessed through complementary benchmark suites and generation metrics.

  • Pretraining setup: The default model follows Llama-3 architecture and uses Transfusion to unify discrete text next-token prediction with continuous visual flow matching.The architecture includes SwiGLU, RoPE, pre-RMSNorm, grouped-query attention, QK-norm, and FlashAttention.
  • Visual tokenization: Four visual tokenization configurations are evaluated, including diffusion-based rectified-flow models and an autoregressive UniTok configuration with residual-quantized visual codes.Diffusion models use a 25-step Euler sampler with classifier-free guidance scale 5.0, while UniTok predicts codebook factors sequentially.
  • Optimization: Training uses AdamW, cosine decay with 8000-step linear warm-up, bf16 precision, 4096-token context, and a 3.0× weight on diffusion flow-matching loss.The optimizer uses β1 = 0.9, β2 = 0.95, weight decay 0.1, and gradient clipping 1.0.
  • Pretraining data: Pretraining combines DCLM language data with roughly 350M Shutterstock-Image image-text pairs used for both visual understanding and text-to-image generation.Visual-understanding scores are obtained after one supervised-fine-tuning epoch on Cambrian-7M.
  • Evaluation: Evaluation covers few-shot language accuracy and perplexity, 16 visual-understanding benchmarks, and visual generation through GenEval, DPG-Bench, CLIP similarity, and held-out diffusion loss.Generation alignment is reported across short, medium, and long prompt-length buckets.

3 Demystifying Modality Knowledge Flow

Controlled mixture and concept-removal experiments show that modality knowledge flow is asymmetric and depends strongly on the concept: language broadly boosts visual abilities, understanding supports structural generation, and generation supplies latent priors for low-level understanding without broadly transferring zero-shot.

  • Experimental Framework: The study combines real-data mixture experiments with synthetic concept-removal benchmarks to convert correlational modality-transfer findings into causal statements.The synthetic benchmark permits controlled concept insertion or removal and per-concept evaluation, while the reported trends also hold across alternative vision encoders and decoders.
  • 3.1 Real-Data Knowledge Flow: Language acts as a universal booster for visual understanding and generation, while visual understanding strongly supports generation and visual generation has largely neutral effects on other abilities.Increasing language data monotonically improves visual understanding and strengthens text-to-image alignment and compositional quality; visual-generation data causes only minor fluctuations elsewhere.
  • 3.2 Concept-Dependent Transfer: Knowledge transfer is concept-dependent: zero-shot transfer fails for low-level concepts, whereas visual understanding transfers structural concepts to generation.Color and shape removal collapses performance in the corresponding task, while retained spatial relations, size, and count improve zero-shot generation.
  • 3.2 Concept-Dependent Transfer: Generation provides minimal zero-shot transfer to understanding for structural concepts, with count as the only reported exception showing slight positive transfer.Removing relation and size from understanding leaves VQA near control levels, while count performs slightly above control.
  • 3.2 Concept-Dependent Transfer: Generation creates strong latent priors for low-level concept understanding, accelerating fine-tuning across tested concepts with ∆mean acc ranging from +0.133 to +0.273.Prior generative exposure substantially speeds recovery of color and shape understanding, whereas understanding exposure provides virtually no advantage for color generation and only marginal improvement for shapes.

4 Synergy vs. Competition in Unified Pretraining

Unified pretraining is governed by a tradeoff between cross-modal synergy and capacity competition, determined by task complexity and parameter sharing. Simple tasks can boost other modalities, while complex tasks compete for capacity; sharing attention and normalization with modality-specific feed-forward networks promotes synergy across vision tokenization designs.

  • 4.1 Task Complexity: Task complexity governs modality interaction: simple tasks act as cross-modal boosters, whereas complex tasks induce capacity competition that outweighs synergy.These effects arise because modalities share a finite parameter space while learning tasks with different semantic richness and difficulty.
  • 4.1 Task Complexity: −0.211 ∆ppl for backgrounds and −0.183 ∆ppl for noise improve language perplexity over a pure language baseline.The gains occur when language is paired with extremely simple visual targets.
  • 4.1 Task Complexity: +0.052 ∆PPL for video and +0.075 ∆PPL for SSTK natural images degrade text perplexity versus the unimodal baseline.As visual data becomes semantically richer and more complex, capacity competition outweighs the underlying synergistic effect.
  • 4.2 Parameter Sharing: Completely shared dense weights increase language ppl by 0.272 and visual diffusion loss by 0.0537 relative to unimodal baselines, indicating severe competition.Forcing distinct modalities into the same parameter subspace is structurally suboptimal.
  • 4.2 Parameter Sharing: Splitting feed-forward networks improves language ppl by 0.211 and lowers vision loss by 0.0168 beyond unimodal baselines, while shared attention and normalization preserve synergy.Decoupling attention reduces the language ppl improvement to 0.045 and the vision loss reduction to 0.0064.
  • 4.3 Vision Tokenization: Cross-modal synergy occurs across all four vision encoder designs, and Raw Pixels paired with backgrounds achieves −0.266 ∆PPL versus −0.211 for the default RAE setup.The results indicate that pre-aligned or shared vision representations are not required for modality transfer or synergy.

5 The Necessity of Early Unification

Early, simultaneous multimodal training outperforms delayed or sequential integration: postponing vision yields marginal language gains but systematically degrades visual understanding and generation. This delay induces vision laziness, as the visual pathway becomes less active, less integrated, and increasingly bypassed by language priors.

  • Training Sequence: Simultaneous joint training decisively outperforms every isolated modality-training sequence on visual tasks.A vision-first progression does not outperform the unified baseline across all dimensions.
  • Training Sequence: A 12.5% replay buffer slightly improves sequential training but remains far below the joint baseline.Replay mitigates catastrophic forgetting without restoring the cross-modal synergy achieved by joint training.
  • Vision Laziness: Longer language warm-ups induce vision laziness: the visual pathway becomes increasingly underdeveloped and disconnected from the language manifold.Four independent mechanistic measurements show declining visual activity and integration as language pretraining lengthens.
  • Vision Laziness: Late-aligned models increasingly rely on language capabilities to shortcut visual problems, preventing vision from functioning as an equally important modality.This under-optimization is associated with simplified generated shapes and failures in detailed attribute binding.

6 Designing Unified Pretraining Recipes

The paper derives an asymmetric multimodal pretraining recipe in which language and visual understanding support visual generation, with a 70/25/5 Language/Understanding/Generation split emerging as optimal. Large-scale 13.5B MoE evaluations validate this mixture, modality-decoupled architecture, and early unification.

  • Data mixture: The optimal data mixture is 70/25/5 for Language, Visual Understanding, and Visual Generation, respectively.The grid search finds that language needs a dominant share, understanding benefits from around 25%, and generation approaches peak performance with only 5%.
  • Architecture: A 13.5B-parameter MoE model with 1.5B active parameters per token, 256 experts, and top-16 routing implements shared attention and normalization with separated feed-forward experts.Two active experts are fixed as modality-specific, while the remaining 14 are dynamically routed.
  • Data mixture: The asymmetric L70/U25/G5 mixture improves language accuracy from 52.86% to 54.31% and visual understanding average from 41.42% to 43.08% versus the balanced recipe.Text-to-image alignment also improves despite five times fewer generative tokens, although diffusion loss slightly increases from 0.261 to 0.272.
  • Recipe implications: Pretraining everything jointly from scratch enables strong visual generation with few generation tokens because language and visual understanding priors bootstrap generative modeling.Increasing the generation ratio later or adding dedicated visual-generation midtraining remains beneficial for better generative modeling.
  • Architecture: The MoE balanced recipe outperforms the dense baseline, raising language accuracy from 52.03% to 52.86% and visual understanding average from 40.49% to 41.42%.Visual generation diffusion loss also decreases from 0.266 to 0.261 under the same data mix and early unification.
  • Early unification: Early unified pretraining outperforms late fusion, increasing language accuracy from 51.78% to 52.86% and visual understanding average from 40.66% to 41.42%.It also lowers visual generation diffusion loss from 0.269 to 0.261; the late-fusion baseline introduces vision tokens only after 60% of training.

7 Related Work

Multimodal research is shifting from late-fusion systems toward unified architectures and shared vision representation spaces that support both visual comprehension and generation. This work builds on that shift by examining knowledge flow, cross-modal synergy, parameter sharing, and vision representation designs.

  • Unified multimodal models increasingly target simultaneous visual comprehension and generation, alongside vision representation spaces that bridge perception and generation.
  • Late-fusion systems connect pretrained LLMs and independently pretrained visual encoders through adapters and post-hoc fine-tuning, but constrain visual signals to a pre-existing language space.
  • Unified multimodal models often surpass specialized text-to-image systems on complex generation tasks, suggesting benefits from language and visual-understanding capabilities.
  • A central design challenge is maximizing cross-modal synergy while minimizing modality competition, including determining the optimal degree of parameter sharing.
  • The study compares three prevalent vision representation designs and evaluates their effects on knowledge flow within unified multimodal models.

8 Discussion and Conclusion

The study systematically clarifies modality knowledge flow, architectural synergy, and training dynamics in unified multimodal pretraining through controlled experiments. Its findings support efficient early-fusion training and motivate extending these principles to richer omnimodal systems.

  • Discussion and Conclusion: Controlled experiments across synthetic and real-world environments clarify modality knowledge flow, architectural synergy, and training dynamics.The study moves beyond prevailing heuristics to investigate the underlying mechanisms of unified multimodal pretraining.
  • Discussion and Conclusion: Multimodal transfer is inherently asymmetric and concept-dependent, while early joint training is necessary for unified multimodal pretraining.These findings explain key training dynamics underlying modality interaction.
  • Discussion and Conclusion: Early-fusion architectures enable diverse modalities to co-evolve in an efficient, scalable manner rather than treating generative modeling as an isolated objective.The study presents this co-evolution as a defining takeaway for next-generation unified foundation models.
  • Discussion and Conclusion: Balancing synergy and competition will become increasingly important for omnimodal foundation models supporting arbitrary input-output formats.The paper suggests extending early joint training and parameter decoupling beyond vision to continuous video, audio, 3D, actions, and other modalities.

A Limitations and Future Work

The study’s principles remain empirically unvalidated for more complex modalities and extreme scales. Future work should examine dynamic modalities such as video and audio, and whether modality interactions change beyond 1T-parameter models.

  • Generalization: The principles of asymmetric knowledge flow, early unification, and architectural decoupling require validation on more complex modalities and extreme scales.The authors hypothesize these principles will largely hold, but note that the relevant frontiers remain empirically untested.
  • Dynamic modalities: The empirical study is largely limited to text and static images, leaving video and audio as open domains for testing modality interactions.Video adds temporal structure, increasing data complexity and computational demands.
  • Dynamic modalities: Future research should determine whether static-image findings transfer to video generation and how temporal dimensions affect knowledge flow and modality synergy.The study presents static-image results as a potentially transferable baseline, not as a validated conclusion for dynamic modalities.
  • Extreme scaling: The proposed recipes are validated at 13.5B parameters and 2T tokens, but modality-interaction physics may evolve at frontier scales exceeding 1T parameters.Greater capacity could shift the synergy–competition threshold or enable new cross-modal interactions.

B Modality Transfer on Real-world Data Additional Results

Across Raw Pixels, CLIP + VAE, and AR (UniTok) visual designs, modality-transfer experiments show consistent knowledge-flow patterns: language boosts vision, visual understanding benefits generation, and visual generation has a neutral effect on other capabilities. These results indicate that modality knowledge flow is governed more by learning objectives than by tokenizer design.

  • Cross-design consistency: Across three alternative visual tokenizer designs, modality-transfer experiments exhibit highly consistent knowledge-flow behaviors.The designs are Raw Pixels, CLIP + VAE, and AR (UniTok), with supplementary evaluations focused primarily on conditional visual generation.
  • Knowledge flow: Increasing the language data ratio from 0% to 80% monotonically and significantly improves all visual understanding metrics and generative quality.This language-as-universal-booster effect holds across the alternative tokenization setups.
  • Knowledge flow: Scaling visual understanding tokens consistently lowers diffusion loss and improves text-to-image alignment scores across all three settings.The result confirms transfer of visual-understanding knowledge into generative processes.
  • Knowledge flow: Increasing the visual generation compute budget has a largely neutral, highly stable effect on language and visual understanding capabilities.Both benchmark types show only minor, non-directional fluctuations across generation ratios, without systematic degradation.
  • Implication: The findings show that modality knowledge flows are tied more closely to learning objectives than to specific visual tokenization designs.The same transfer patterns recur under Raw Pixels, CLIP + VAE, and AR (UniTok).

C CLEVR Generation and Evaluation Details · C.1 Generation

The CLEVR-based synthetic environment expands visual concepts, generates approximately one million non-overlapping scenes, and pairs each image with dense and sparse captions for both understanding and generation. Evaluation uses noise-free scene-graph-derived VQA covering color, shape, relation, size, and count through balanced attribute, comparison, counting, and spatial-reasoning queries.

  • C.1 Generation: CLEVR scenes expand the color and shape vocabularies beyond standard assets, adding custom hues and authored geometric primitives.Expanded colors include red, blue, green, gray, brown, purple, cyan, and yellow; added shapes include cross, ellipse, pentagon, rectangle, and triangle.
  • C.1 Generation: Programmatically sampled scene graphs define each object’s color, shape, size, and 3D position, while placement constraints prevent intersections and overlap.Each scene contains 1–7 objects, with roughly one million scenes generated for training and a held-out evaluation split.
  • C.1 Generation: Each rendered image receives dense and sparse captions generated with Qwen3-VL-8B-Instruct, and both caption types support image-to-text understanding and text-to-image generation.Generation uses temperature 0.2, top-p 0.8, and a maximum of 256 tokens.
  • C.1 Generation: Replicated modality-transfer experiments with SigLIP-2 plus SD3 VAE and with discrete visual tokenization show language as a universal booster and understanding as a strong generation prior.Visual generation has a stable, neutral, or slightly fluctuating effect on other capabilities across both continuous diffusion and autoregressive settings.
  • C.1 Generation: Knowledge-flow analysis evaluates five conceptual axes: color, shape, spatial relation, size, and object count.Relations include cardinal and diagonal directions computed analytically from 3D scene-graph coordinates; size covers absolute scales and relative comparisons.
  • C.1 Generation: VQA pairs are derived directly from scene-graph metadata, providing noise-free open-ended short-answer questions across attribute identification, counting, size comparison, and spatial reasoning.Referent uniqueness filtering removes ambiguous attribute questions, while spatial queries include single-hop and multi-hop directional reasoning with diagonal outputs.
  • C.1 Generation: Answers are formatted as single words or short phrases, and VQA generation balances the five target concepts to avoid training bias.Each rendered-scene instance contains one image, two captions, and one to three VQA questions.

C.2 Evaluation

The evaluation measures concept transfer in both Understanding→Generation and Generation→Understanding directions using controlled leave-one-concept-out studies. Each held-out concept is evaluated with 100 samples, using VLM-judged generation accuracy or VQA accuracy.

  • Evaluation design: Concept transfer is evaluated symmetrically by ablating one concept at a time from either the generation or understanding stream.The study uses a controlled leave-one-concept-out design across two directions.
  • Evaluation design: Each held-out concept is evaluated using 100 evaluation samples.This sample count applies to every held-out concept.
  • Understanding→Generation: Understanding→Generation removes the concept from generation training data while preserving understanding data, then tests whether prompted images contain the held-out concept.Per-axis accuracy is measured on generated images restricted to the ablated axis.
  • Understanding→Generation: Generated images are scored by Qwen3-VL-8B-Instruct, which answers binary questions about whether the requested held-out concept appears correctly.The resulting metric is per-axis accuracy on the ablation set.
  • Generation→Understanding: Generation→Understanding removes the concept from captions and VQA pairs while preserving generation data, then evaluates answers on held-out VQA examples.The 100 ablation questions come from held-out scenes whose ground-truth answer is the removed concept.
  • Generation→Understanding: Generation→Understanding reports average VQA accuracy over 100 questions for each held-out concept.The questions cover color, shape, spatial relation, counting, and size concepts.

D CLEVR Additional Results

This section supplements the CLEVR analyses with detailed training trajectories and qualitative examples. The results examine concept recovery and illustrate where structural knowledge transfer succeeds while low-level semantic transfer fails.

  • Overview: Supplementary CLEVR results provide quantitative and qualitative evidence supporting the analyses and findings presented in §3.2.The section adds additional results on the synthetic dataset.
  • Concept Prior Study: Training trajectories over the first 1000 optimization steps examine recovery of held-out color and shape concepts under understanding priors.Figure 17 plots diffusion-loss curves for recovering low-level generative concepts, while Figure 18 provides the symmetric text cross-entropy trajectories.
  • Zero-Shot Transfer: Qualitative generation examples show successful structural transfer for spatial relations, size, and object count, but failed low-level semantic transfer for color and shape.These examples complement the quantitative zero-shot transfer rates in Figure 5 and §3.2.2.
  • Concept Prior Study: Figure 17 compares diffusion-loss trajectories with visual understanding priors against a no-prior control for recovering red, yellow, cylinder, and sphere concepts.The plotted trajectories reportedly show no observable benefit from prior exposure via visual understanding for these concepts.

E Additional Text-to-Image Generation Results

Early unification produces text-to-image samples that adhere more closely to multi-attribute prompts and preserve sharper structures than late alignment under matched compute. The observed degradation in late-fusion outputs is consistent with vision laziness and reduced optimization of visual pathways.

  • Qualitative comparison: Under matched compute, the 13.5B MoE Balanced Recipe early-unification model is qualitatively compared with the Late Fusion baseline across diverse prompts.The comparison is presented in Figure 20.
  • Qualitative analysis: Early fusion preserves clearer boundaries in multi-attribute prompts, while late fusion blends distinct attributes into generic outputs.For the noodle prompt, early fusion separates red, white, and green segments rather than producing a blended pasta dish.
  • Qualitative analysis: Early fusion better represents geometric and symbolic structures, whereas late fusion often produces unstructured or abstract shapes.The Bitcoin cloud prompt yields a recognizable outlined shape with early fusion but standard unstructured clouds with late fusion.
  • Qualitative analysis: Early fusion renders contextual details such as reflections and hanging objects more consistently, while late fusion may omit or ambiguously depict them.The passage notes that late-fusion outputs can contain highly ambiguous shapes for these requested elements.
  • Vision laziness: These late-fusion weaknesses align with vision laziness: extended pure-language pretraining may leave visual pathways under-optimized, reducing structural and textural definition.The qualitative evidence is not definitive proof of optimization dynamics but agrees with quantitative measurements of reduced visual pathway activity.
Loading 2608.05000v2…