PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws for Generative Mixed-Modal Language Models

Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, Luke Zettlemoyer

arXiv:2301.03728v1cs.CLcs.AIcs.LG

TL;DR

Mixed-modal generative language models can combine many tokenized modalities, but their scaling behavior and modality interactions remain an open question. This paper conducts over 250 experiments and develops scaling laws that combine individual modality contributions with interaction terms. The study also reports training phenomena and uses the laws to guide mixed-modal model design.

  • Problem

    The study asks whether scale alone can overcome the competition that arises when multiple modalities are trained in one model.

  • Method

    The paper studies mixed-modal generative language models over tokenized modalities and develops scaling laws with additive terms for modality contributions and interactions.

  • Results

    The study identifies interaction-driven competition and synergy, predicts a regime eliminating Speech–Text competition, and reports modality-prioritized training dynamics.

  • Takeaways & Limitations

    The scaling laws provide insights and guidelines for designing and training unified models that handle multiple modalities simultaneously.

Abstract

from arXiv · show

Generative language models define distributions over sequences of tokens that can represent essentially any combination of data modalities (e.g., any permutation of image tokens from VQ-VAEs, speech tokens from HuBERT, BPE tokens for language or code, and so on). To better understand the scaling properties of such mixed-modal models, we conducted over 250 experiments using seven different modalities and model sizes ranging from 8 million to 30 billion, trained on 5-100 billion tokens. We report new mixed-modal scaling laws that unify the contributions of individual modalities and the interactions between them. Specifically, we explicitly model the optimal synergy and competition due to data and model size as an additive term to previous uni-modal scaling laws. We also find four empirical phenomena observed during the training, such as emergent coordinate-ascent style training that naturally alternates between modalities, guidelines for selecting critical hyper-parameters, and connections between mixed-modal competition and training stability. Finally, we test our scaling law by training a 30B speech-text model, which significantly outperforms the corresponding unimodal models. Overall, our research provides valuable insights into the design and training of mixed-modal generative models, an important new class of unified models that have unique distributional properties.

1 INTRODUCTION

This study examines how mixed-modal generative language models scale when multiple tokenized modalities are trained together. It develops scaling laws for modality contributions, competition, synergy, training phenomena, and hyperparameter selection.

  • 1 INTRODUCTION: Mixed-modal models represent arbitrary modality subsets and orders by training one discrete language model over modality-specific token sequences.Examples include image tokens from VQ-VAEs and speech tokens from HuBERT.
  • 1 INTRODUCTION: The central question is whether increasing scale can overcome competition introduced by adding modalities to one model.
  • 1 INTRODUCTION: Over 250 experiments across seven modalities and 8 million–30 billion parameter models identify scaling laws with additive interaction terms for modality competition or synergy.
  • 1 INTRODUCTION: The scaling laws predicted a 30B-model, 45B-token regime where competition between Speech and Text was completely reduced.
  • 1 INTRODUCTION: Training exhibits modality-prioritized optimization at different stages, while the scaling-law interaction terms guide explanations and hyperparameter selection.

2 RELATED WORK

Prior work established neural scaling laws for model size, data size, and compute, but had not fully characterized interactions among generative modalities. This paper positions mixed-modal competition and synergy as a distinct scaling-law problem.

  • 2 RELATED WORK: Neural scaling laws relate model size, dataset size, compute budget, and neural-network performance through power-law behavior.
  • 2 RELATED WORK: Subsequent work extended scaling laws to compute-optimal training, mixture-of-experts models, lower precision, and neural machine translation.
  • 2 RELATED WORK: Earlier studies examined scaling in image generation and acoustic models, while multimodal work had not specifically focused on generative models.
  • 2 RELATED WORK: The paper identifies interactions, competition, and interference among multiple modalities and provides a recipe for optimal mixed-modal training.
  • 2 RELATED WORK: Prior multilingual findings similarly associate interference with multimodal training and improved synergy with larger models.

3 DEFINITIONS

The paper defines modalities empirically through cross-dataset perplexity and models unimodal loss as the sum of irreducible, approximation, and optimization errors. This provides a measurable basis for comparing scaling across data types.

  • 3 DEFINITIONS: A modality is empirically distinguished when a secondary dataset’s perplexity exceeds σ times the primary dataset’s mean perplexity.The paper uses σ = 3 and applies the definition across source, domain, sensor, and language differences.
  • 3 DEFINITIONS: Source modality denotes the token type contained in a sample, including Text, Speech, or Image.
  • 3 DEFINITIONS: The selected scaling-law parameterization expresses loss using model parameters N and dataset tokens |D| through three additive components.
  • 3 DEFINITIONS: These components are minimal achievable loss, functional approximation error, and optimization or convergence error.
  • 3 DEFINITIONS: The exponents α and β have upper bounds of 1, clarifying how transformer and gradient-descent scaling compares with modality-specific optimal scaling.

4 EMPIRICAL SETTING

The empirical study trains decoder-only models on unified discrete tokens from text, images, speech, code, and molecules, using causal masking and infrastructure for large-scale experiments. The setup spans diverse datasets, tokenizers, model sizes, and controlled training procedures.

  • 4.1 DATASETS: Text, image, image-text, speech, speech-text, code, and molecule datasets provide the study’s modality and coupling data.The datasets use OPT text, Make-A-Scene image tokens, aligned image captions, web and public speech, InCoder code, and SMILES molecules.
  • 4.2 TOKENIZATION: A unified tokenizer converts all modalities into discrete tokens that one model processes jointly with a single loss.
  • 4.2 TOKENIZATION: VQGAN converts images into discrete tokens using spatial compression and an 8192-entry codebook.
  • 4.2 TOKENIZATION: HuBERT converts speech into discrete units through self-supervised representation learning and K-means clustering with a 2000-token codebook.
  • 4.2 TOKENIZATION: The study adds a BPE vocabulary over sampled multimodal data, with image and speech tokens treated as single tokens and additional digit splitting.
  • 4.3 MODEL AND TRAINING: Decoder-only GPT-3- and OPT-style models are trained up to 6.7B parameters for scaling laws and 30B parameters for generalization tests.
  • 4.3 MODEL AND TRAINING: Causal masking supports bidirectional context and sequence infilling, including generation with modality orderings not seen strictly left-to-right during training.

5 SCALING LAWS

The study fits scaling laws for individual modalities and extends them to mixed-modal models by adding interaction terms that capture competition or synergy. These laws are used to analyze modality interactions, predict competition barriers, and validate a speech-text model at scale.

  • UNI-MODAL SCALING LAWS: Seven modalities are trained across model sizes from 8 million to 6.7 billion and dataset sizes of 5B, 10B, and 100B tokens.The experiments establish uni-modal scaling behavior before extending the analysis to mixed-modal settings.
  • UNI-MODAL SCALING LAWS: Scaling dynamics differ substantially across modalities, model scales, and dataset sizes.The fitted uni-modal parameters also vary significantly, with Code and Molecules using scale more efficiently than Image.
  • BI-MODAL SCALING LAWS: The mixed-modal law averages independent modality losses and adds interaction terms for mutual information, functional competition, and optimization competition.Its additive structure also permits extension to n-modal scaling laws.
  • BREAKING THE COMPETITION BARRIER: The scaling laws predict regimes where modality competition transitions into synergy and define a competition barrier separating the two regimes.The barrier is used to identify compute-optimal model size and token count for overcoming competition.
  • BREAKING THE COMPETITION BARRIER: A 30B-parameter, 50B-token Speech|Text model crosses the predicted competition barrier, supporting the scaling-law prediction.The selected training regime was close to the predicted optimum of N = 28.35B and D = 45.12B.

6 EMERGENT PHENOMENA

Mixed-modal training exhibits modality-specific pauses and optimization rates that vary with model scale and interaction parameters, alongside links between these parameters, batch size, and stability.

  • Intermittent Coordinate Ascent Like Training: Different modalities are optimized at different paces, with some pausing for substantial training intervals.Average perplexity remains smooth and generally decreasing, while individual modality perplexities can flatten.
  • Scale and Flatness: Flatness is measured as the percentage of training-curve regions where submodality loss does not decrease, excluding warm-up.The analysis covers speech and image perplexity in bimodal couplings.
  • Scale and Flatness: Increasing model size reduces intermittent coordinate-ascent-like training, while α_i,j correlates with optimization flatness.The authors find no correlation between β_i,j and optimization flatness.
  • Optimal Batch Size: The optimal batch size for modality pairs is correlated with β_i,j.Experiments compare batch sizes from 1M to 8M tokens, with additional 0.5M experiments for text-containing modalities.
  • Training Stability: Training deterioration is associated with α_i,j and model size, measured through the count of gradient-norm spikes.The reported relationship links log(N)/α_i,j to gradient spikes, while β_i,j shows no correlation with spike count.

7 CONCLUSION

The paper develops scaling laws for mixed-modal generative models that represent both individual modalities and their interactions. It also connects these laws to observed training dynamics and presents them as a tool for understanding unified models.

  • Conclusion: The study develops scaling laws that represent individual modalities and interactions between them.The analysis targets the scaling properties and training dynamics of mixed-modal generative models.
  • Conclusion: The findings include empirical training phenomena that can be primarily explained through the interaction terms in the scaling laws.The authors present the resulting analysis as a tool for practitioners.
  • Conclusion: The work advances understanding of mixed-modal generative models and supports the development of unified models handling multiple modalities simultaneously.

A.1 MODEL ARCHITECTURE

The models use a pre-norm architecture with ReLU activations and regularization, while truncated normal initialization improves reported training stability and performance.

  • Architecture and Initialization: Models use pre-norm, ReLU activation, dropout of 0.1 except on embeddings, and weight decay of 0.1.
  • Architecture and Initialization: Truncated normal initialization with mean zero and standard deviation 0.006 substantially improves training stability and performance.The distribution is truncated within two standard deviations.
  • Architecture and Initialization: Table 2 reports layers, embedding size, attention heads, head dimension, batch size, learning rate, and context length.

A.2 CAUSAL MASKED VS. CAUSAL OBJECTIVE

The paper compares causal and causal-masked objectives on Speech and Text scaling laws while keeping training procedures constant. Their fitted scaling laws are close to identical within approximation error.

  • Objective Comparison: The objective comparison keeps datasets, training procedures, and other conditions constant while fitting scaling laws for Speech and Text.
  • Objective Comparison: The causal and causal-masked objectives produce scaling laws that appear close to identical within approximation error.Both objectives optimize the joint probability of tokens.
  • Objective Comparison: Table 3 reports uni-modal scaling-law parameters for causal and causal-masked objectives.

A.3.1 QUALITY OF IMAGE TOKENIZATION

The image tokenization study compares VQGAN variants that trade compression, token count, vocabulary size, and reconstruction quality. Lower compression and larger vocabularies generally improve visual detail, while VQGANMAS variants better preserve text and numbers.

  • Tokenization design: VQGAN tokenizers reduce image context size by a factor of 3∗X^2, where X is the spatial reduction rate and 3 is the number of image channels.The tokenizer encodes images into discrete grids and maps encoded vectors to a learned codebook.
  • Tokenization design: 256-token VQGAN(f16) representations use higher spatial reduction, whereas VQGAN(f8) and VQGANMAS representations use 1024 tokens.
  • Reconstruction quality: Lower spatial reduction and larger visual vocabularies produce lower reconstruction FIDs and more realistic image reconstructions.The qualitative examples show improved recovery of details such as door handles, clouds, and mirrors when compression is decreased.
  • Reconstruction quality: VQGANMAS models reconstruct textbook pages and scientific-paper tables more successfully than the other tested models, preserving readable text and numbers.VQGANMAS256 produces sharper edges, whereas VQGANMAS512 produces smoother reconstructions.
  • Resolution and token budget: Improved image representation requires longer token sequences when spatial reduction is decreased or image resolution is increased.For 512×512 inputs, VQGANMAS models produce 4096 tokens, while VQGAN(f16) produces 1024 tokens.

A.3.2 DETAILS OF SPEECH TOKENIZATION

The speech tokenizer uses a pretrained HuBERT model to convert speech into discrete units. The supplied details describe its architecture, large-scale pretraining, compression, and evaluation through reconstructed-audio speech recognition.

  • HuBERT model: The speech tokenizer uses the BASE HuBERT model, comprising a convolutional encoder and a 12-layer Transformer with 768-dimensional embeddings.Its feed-forward layers have dimension 3072 and its self-attention uses 12 heads.
  • HuBERT model: HuBERT was pretrained for three iterations on 32 GPUs, using 221K hours of unlabeled multilingual speech data.Each iteration involved 400K updates.
  • Compression and evaluation: HuBERT encodes 16 kHz speech at 50 Hz with a 2000-entry codebook, reducing the bitrate from 64 kbps to 548 bps.The resulting effective compression rate is roughly 117 while retaining speech information for evaluation.
  • Compression and evaluation: Speech-tokenizer quality is evaluated by comparing ASR word error rates for original audio and audio reconstructed from HuBERT representations.The comparison covers LJSpeech and LibriSpeech datasets and multiple HuBERT configurations.
Loading 2301.03728v1…