PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Language Modeling: An Exploration of Multimodal Pretraining

Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan Ghazvininejad, Mike Lewis, Nicolas Ballas, Amir Bar, Michael Rabbat, Jakob Verbeek, Luke Zettlemoyer, Koustuv Sinha, Yann LeCun, Saining Xie

arXiv:2603.03276v1cs.CV

TL;DR

Native multimodal pretraining lacks clear evidence about which design choices govern unified models. The paper addresses this with controlled, from-scratch Transfusion experiments and finds that RAE, complementary data, emergent world modeling, and MoE scaling support unified multimodal learning.

  • Problem

    The design space for native multimodal models remains opaque because many existing approaches initialize from pretrained language models rather than isolating multimodal pretraining from scratch.

  • Method

    The paper trains a single model from scratch with Transfusion, combining next-token language prediction and visual diffusion on text, video, image–text, and action-conditioned video.

  • Results

    Unified pretraining yields a single RAE representation for visual understanding and generation, complementary multimodal data, emergent world modeling, and MoE scaling that accommodates differing modality data requirements.

  • Takeaways & Limitations

    MoE provides structural flexibility for modalities with different scaling behaviors to coexist in a unified multimodal model.

  • Takeaways & Limitations

    The conclusions come from from-scratch pretraining and do not establish how the findings transfer to finetuning pretrained dense language models.

Abstract

from arXiv · show

The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.

1 Introduction

The paper studies unified multimodal pretraining from scratch to clarify design choices without inherited language-model capabilities. It finds complementary data, emergent world modeling, modality-specialized architecture, and a vision–language scaling asymmetry addressed by MoE.

  • Motivation: Text-only pretraining captures symbols but misses high-fidelity physics, geometry, and causality in the physical world.
  • Approach: The study uses controlled, from-scratch Transfusion pretraining with next-token prediction for language and diffusion for vision across diverse data and evaluations.
  • Visual Representation: RAE is identified as an optimal unified visual representation for both visual understanding and generation.
  • Data: Visual and language data show minimal interference and positive synergy in selected downstream capabilities.
  • World Modeling: Physical prediction capabilities emerge primarily from general multimodal pretraining, including video, rather than domain-specific data.
  • Architecture and Scaling: MoE studies reveal modality separation and unification, while IsoFLOP experiments find that vision is significantly more data-hungry than language and that MoE bridges this gap.

2 Experiment Setup

The experiments train a unified decoder-only Transformer from scratch on mixed text, image, video, paired image–text, and action-conditioned data. Text uses autoregressive prediction, while visual states use flow matching with modality-aware masking and visual representations.

  • Study Protocol: The study controls compute budgets and training hyperparameters within each ablation family and evaluates models at the end of pretraining unless specified otherwise.
  • Unified Objective: A single autoregressive model predicts text with next-token prediction and visual states with flow matching.
  • Visual Training: Flow matching interpolates noisy and clean visual latents, then trains the model to predict a velocity field using squared error.
  • Data Mixture: Each batch mixes text-only, video-only, paired image–text, and action-conditioned examples under a weighted combination of language-modeling and flow-matching objectives.
  • Masking: Visual training uses block-wise causal masking, with bidirectional attention within each frame and attention to previous tokens.
  • Model and Tokenization: The model uses a decoder-only Transformer with linear projections, modality-specific FFNs, standard BPE text tokens, and frozen visual encoders that produce latent tokens.
  • Training Data: Training data spans large-scale web text, raw video, paired image–text data, and action-conditioned navigation trajectories formatted as I + T →I.
  • Evaluation: Evaluation covers text perplexity, diffusion loss, image-generation scores on DPGBench and GenEval, and VQA accuracy across 16 Cambrian benchmarks.

3 Visual Representations for Unified Multimodal Pretraining

The experiments evaluate unified visual representations for both understanding and generation. Semantic RAE-based encoders, especially SigLIP 2, provide a single representation that performs strongly across both capabilities while preserving text performance.

  • Unified representation: RAE-based representations bridge visual understanding and generation without requiring dual encoders.This simplifies the architecture while retaining competitive performance across both visual objectives.
  • Unified representation: A single RAE-based encoder suffices for both visual understanding and generation.The results motivate adopting SigLIP 2 as the default vision encoder.
  • Language performance: All visual representations achieve similar or slightly better text perplexity than the text-only baseline.The marginal differences indicate that visual representation choice does not significantly affect language capabilities.
  • Representation comparison: Semantic encoders consistently outperform VAE-based encoders on visual understanding and generation.SigLIP 2 outperforms FLUX.1 on VQA, DPGBench, and GenEval.
  • Representation comparison: Raw pixels underperform semantic encoders in generation quality but remain comparatively close in VQA accuracy.The authors identify direct learning from pixels as a promising direction with more compute and scale.

4 Understanding the Impact of Data

The data-composition study finds that multimodal training can preserve language performance while contributing distinct visual capabilities. The experiments compare text, video, image-text, and action-conditioned data to identify complementary effects and trade-offs.

  • Data mixtures: The unified framework models combinations of text, video, image-text pairs, and action-conditioned video.The study compares text-only training with multimodal mixtures and analyzes the role of text distribution in image-text data.
  • Language performance: Text + Video achieves the best perplexity among multimodal mixtures on DCLM and Notes.On DCLM, it even outperforms the text-only baseline, indicating compatibility between raw video and language modeling.
  • Language performance: Text + MetaCLIP has the worst perplexity among the evaluated data mixtures.The full mixture only slightly degrades text performance relative to the text-only baseline.
  • Language performance: Multimodal pretraining introduces a minor trade-off in text generalization on the more out-of-distribution Notes corpus.The relative trends across mixtures remain consistent despite the degradation.

COCO FID

The data experiments show that image-text sources contribute differently across language, generation, and VQA, while multimodal co-training improves downstream visual capabilities. Diverse data can outperform simply increasing task-specific training.

  • Image-text data: I/T data is essential for acquiring visual understanding and generation capabilities.Text + Video contains no I/T data, so image generation is omitted.
  • Image-text distribution: Caption distributional distance from DCLM correlates with degradation in language perplexity.Synthetic recaptions and in-house data are farther from the pretraining corpus than standard MetaCLIP captions.
  • Image-text data: Combining MetaCLIP for image-to-text with Shutterstock for text-to-image captures complementary strengths across metrics.Recaptioned data improves VQA, whereas high-aesthetic Shutterstock data improves generation quality.
  • Cross-modal synergy: Adding text tokens to a fixed vision budget consistently improves diffusion loss and GenEval score.The multimodal models surpass the multimodal-only baseline across the tested token budgets.
  • Downstream synergy: All mixed variants outperform the 100B VQA-only baseline despite using 5× less in-domain data.The comparison uses 20B VQA tokens supplemented with 80B heterogeneous data.
  • Downstream synergy: Multimodal pretraining improves VQA over text-only pretraining after finetuning across visual representations.Semantic encoders such as SigLIP 2 consistently outperform VAE-based encoders in this comparison.

5 Towards World Modeling in Unified Multimodal Models

Unified multimodal pretraining supports world modeling without architectural changes by representing navigation actions as text. World-modeling performance emerges largely from general training, requires little in-domain data, and enables natural-language control.

  • World-model formulation: Navigation actions are represented as standard text tokens, framing world modeling as I + T → I prediction.The model uses no action-specific adapters or architectural modifications.
  • General pretraining: Adding multimodal data outperforms scaling domain-specific NWM data from 50B to 100B tokens.Pure video provides the biggest gain, while MetaCLIP and text also improve results.
  • Data efficiency: Competitive world-modeling performance is reached with as little as 1% in-domain NWM data.Higher domain-data ratios provide negligible additional gains under a fixed 200B-token budget.
  • General pretraining: Navigation and VQA are primarily learned from general pretraining and require minimal in-domain data.This conclusion is supported by the rapid saturation of world-modeling performance with domain-data volume.
  • Natural-language control: The model generates visually consistent trajectories for WASD controls and out-of-distribution natural-language commands.Language-driven navigation emerges zero-shot from semantic transfer during multimodal pretraining.

6 Unified Multimodal Architecture Design

The architecture study finds that learned capacity separation, especially MoE, improves unified multimodal modeling while producing modality-specific and later-integrated expert behavior. A unified semantic visual representation also supports both understanding and generation.

  • Expert granularity: Increasing granularity from G=1 to G=16 substantially improves language and vision modeling, with vision saturating at G=4 and language at G=16.Gains plateau beyond G=32, so G=16 is used subsequently.
  • Sparsity: At fixed compute, increasing total experts from 32 to over 1000 decreases text perplexity and improves visual generation.Both modalities benefit from increased sparsity.
  • Shared experts: Per-modality shared experts outperform global shared experts across all metrics, indicating distinct capacity needs across modalities.The result motivates dedicated computation for different modalities.
  • Emergent specialization: Expert routing allocates more experts to text, processes modalities separately in early layers, and increasingly fuses them in later layers.Vision experts show minimal timestep specialization, while understanding and generation activate highly correlated experts with r ≥0.90.
  • Capacity separation: Replacing shared FFNs with modality-specific FFNs improves perplexity from 15.93 to 15.13 and DPG score from 0.45 to 0.47 without additional inference cost.Only one modality-specific FFN is active per token.
  • Visual representation: SigLIP 2 improves DPG score from 0.47 to 0.57 and reduces perplexity from 15.13 to 15.06, outperforming separate-encoder alternatives.The dual-encoder baseline underperforms the single SigLIP 2 encoder on both metrics.
  • Mixture-of-Experts: MoE reaches the lowest perplexity of 12.49 and highest DPG score of 0.63, outperforming dense and MoT parameter-separation strategies.The comparison uses SigLIP 2 with modality-specific FFNs.
  • Scope: The conclusions are established in a from-scratch pretraining setting, while extending routing networks to multimodal finetuning remains a prospective direction.The authors note that MoT has also been effective when finetuning pretrained dense language models.

7 Scaling Laws for Unified Multimodal Models

The scaling analysis shows that vision requires more data than language under compute-optimal training, creating a widening resource mismatch. MoE reduces this asymmetry and lets one sparse model closely track unimodal baselines across modalities.

  • Method: The analysis derives joint vision-language scaling laws using IsoFLOP sweeps over model sizes and token counts at fixed compute budgets.Validation loss is measured separately on held-out DCLM and CC12M data.
  • Dense scaling: Dense scaling exponents are a≈0.47, b≈0.53 for language and a≈0.37, b≈0.63 for vision, indicating that vision is more data-hungry.The differing exponents imply no single compute-optimal trend governs unified models.
  • Scaling asymmetry: The required vision-to-language data ratio grows as O(N^0.57), increasing 14× at 100B parameters and 51× at 1T from a 1B baseline.This widening gap forces a compromise between under-training vision and over-training language.
  • MoE scaling: MoE halves the parameter scaling exponent gap from 0.10 for dense models to 0.05, with language b≈0.59 and vision b≈0.64.The sparse regime uses 16× more total experts than activated experts.
  • Balancing data demands: MoE increases the language data exponent from 0.53 to 0.59, moving it closer to vision’s data-intensive regime.The authors identify sparsity as an architectural lever for balancing divergent modality data demands.
  • Compute efficiency: At 10^21 FLOPs, unified MoE reaches DCLM PPL 12.3 versus 12.0 for MoE Text-Only and FID 39.2 versus 39.8 for MoE T2I-Only.Across all four reported metrics, the unified model closely tracks unimodal MoE baselines.

8 Related Work

Related work situates the paper among unified multimodal pretraining, visual representation learning, world modeling, and scaling-law studies. The paper extends these lines by combining Transfusion-style unified training with systematic representation and scaling analyses.

  • Multimodal pretraining: Unified multimodal pretraining trains models from scratch on mixed-modal data, with recent methods combining language objectives and diffusion-based image generation.Earlier unified approaches quantized images into discrete tokens for next-token prediction.
  • Unified architectures: Unified models typically input and output vision and language, while architectural work studies how multimodal computation is organized.The paper positions its MoE design study within this broader unified-model literature.
  • Visual representations: Semantic visual latents can support generation as well as understanding, whereas VAE latents degrade visual understanding in the paper’s experiments.The paper reports that RAE-based semantic latents excel at both tasks.
  • World models: World-model research has progressed from in-domain policy-training models to large-scale video generators and multimodal planning systems.Recent approaches still require in-domain trajectories for planning.
  • Scaling laws: Prior scaling studies mainly address unimodal language, while this work derives scaling laws jointly for vision and language and examines architectural effects.The analysis reveals a modality asymmetry that MoE architectures help harmonize.

9 Discussion

The discussion argues that unified multimodal pretraining can reconcile visual understanding, generation, language, and world modeling through suitable representations, data, and architectures. It identifies RAE and MoE as particularly important design choices while noting remaining limitations in reconstruction, scope, and efficiency.

  • Visual representations: RAE provides a single high-dimensional semantic representation that excels at both visual understanding and generation.The same experts are often activated for both tasks, and RAE representations continue improving as capacity scales.
  • Visual representations: Semantic vision encoders still lag behind VAEs in fine-grained reconstruction.The authors identify generation-aware semantic representations as a direction for closing this gap.
  • Modality interaction: Vision-language competition is largely attributable to caption distribution shifts and dense capacity allocation rather than vision itself.Pure video is complementary to language, while modality-specific FFNs and MoE improve capacity allocation; slight degradation remains for out-of-distribution text generalization.
  • Scaling with MoE: Both modalities consistently improve as total MoE experts grow at fixed active compute.The result confirms that sparse MoE scaling extends to unified multimodal training.
  • Scaling with MoE: Vision is significantly more data-hungry than language, creating a scaling asymmetry that dense models cannot optimize simultaneously.In sparse MoE models, language scaling shifts toward a more data-hungry regime, aligning with vision scaling.
  • Scaling with MoE: MoE provides structural flexibility for modalities with fundamentally different scaling behaviors to coexist.The discussion presents sparsity as more than an efficiency mechanism for unified multimodal models.

D.3 Text Performance on Data Composition Studies

Adding text tokens to a fixed multimodal budget consistently improves performance, while adding visual tokens to a fixed text budget has little effect on text perplexity. The study evaluates these interactions across systematically varied text and image-token mixtures.

  • Adding text tokens to a fixed multimodal budget consistently improves performance and surpasses the vision-only baseline.The comparison uses multimodal models and unimodal baselines across varying budgets.
  • Visual tokens minimally affect text perplexity at a fixed text budget, with small DCLM improvements and slight Notes degradation.
  • The composition study spans 25 text–image token configurations, using 0, 25, 50, 75, or 100 billion tokens for each modality.The 0-0 configuration is excluded, and language is evaluated with held-out text benchmarks.
  • IsoFLOP MoE sweeps keep active computation approximately constant while scaling total capacity through the expert pool.The sweep increases experts from 32 to 1008 while active parameters remain approximately 1.5B.

E.8 Analysis on MoE models

The MoE analysis measures how experts route text and image tokens, classifies modality preference, and compares visual expert usage across generation and understanding. Routing reveals shared vision specialization across both visual objectives.

  • Expert routing is analyzed on a 1T-token model trained with 50% text, 25% video, 22.5% image-text, and 2.5% action data.
  • Selection rates normalize expert routing by modality token counts and the number of active experts per token, k = 16.
  • The specialization score ranges from −1 to +1, with positive values indicating text preference and negative values indicating image preference.A score of +1 or −1 denotes exclusive selection by text or image tokens, respectively.
  • Experts are classified as Text when S_i > 0.5, Vision when S_i < −0.5, and Multimodal when −0.5 ≤ S_i ≤ 0.5.The thresholds correspond to one modality being selected more than three times as often, or to balanced routing.
  • Expert usage for image generation and understanding shows strong positive correlations across all layers.Experts frequently selected for generation are also frequently selected for understanding, indicating shared vision experts.

F.1 Implementation Details

The implementation evaluates navigation planning with sampled future trajectories, varied action-conditioned data, and both keyboard and natural-language controls. It also documents the data sources and distribution-similarity procedure used in the experiments.

  • The CEM planner uses an 8-step horizon, samples N=120 action sequences, predicts 8 future frames, and averages the top K=5 trajectories.Trajectories are scored by LPIPS, and evaluation reports ATE and RPE over three repetitions.
  • Navigation training varies action-conditioned data from 0.1% to 25% of the total budget within a mixture containing text, video, and multimodal data.The base mixture uses 25% text, 25% video, and 50% multimodal data.
  • WASD controls map to robot-centric translation and yaw tuples, with forward and backward motion encoded as (+0.5, 0, 0) and (−0.5, 0, 0).Turning actions add a small forward component: left is (+0.2, 0, +0.5) and right is (+0.2, 0, −0.5).
  • Free-form natural-language commands provide zero-shot action conditioning beyond a predefined keyboard action set.Examples include “go on the road” and “take big steps forward.”
  • The navigation examples include WASD and natural-language rollouts, plus counterfactual trajectories from the same context under different commands.
  • The action-video annotations use 1–4 context frames, a text action, and 1–4 resulting frames to supplement navigation-world-model trajectories.About 12M video-text action annotations are collected through model-based captioning and self-reflection verification.
Loading 2603.03276v1…