PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?

Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q. Tran, Dani Yogatama, Donald Metzler

arXiv:2207.10551v1cs.LGcs.CL

TL;DR

This paper addresses limited understanding of how architecture affects scaling and transfer. It compares ten architectures across scales and finds that scaling behavior varies by architecture and compute region, while upstream performance may not predict downstream results.

  • Problem

    How different inductive biases and architectures scale, and how this affects upstream and downstream performance, remains insufficiently understood.

    Source: §1 · p. 1

  • Method

    The authors pretrain and fine-tune more than 100 models across ten diverse architectures and multiple compute regions and scales.

    Source: §1 · p. 1

  • Results

    Scaling behavior differs across architectures and compute regions, and models can perform well on upstream perplexity yet fail to transfer downstream.

    Source: §1 · p. 2, §3.8 · p. 7

  • Takeaways & Limitations

    Architecture evaluation should consider downstream performance and how models perform across compute regions, not upstream perplexity alone.

    Source: §3.8 · pp. 7–8

    The largest Switch Transformer was scaled suboptimally and may not represent the family’s full potential.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

There have been a lot of interest in the scaling properties of Transformer models. However, not much has been done on the front of investigating the effect of scaling properties of different inductive biases and model architectures. Do model architectures scale differently? If so, how does inductive bias affect scaling behaviour? How does this influence upstream (pretraining) and downstream (transfer)? This paper conducts a systematic study of scaling behaviour of ten diverse model architectures such as Transformers, Switch Transformers, Universal Transformers, Dynamic convolutions, Performers, and recently proposed MLP-Mixers. Via extensive experiments, we show that (1) architecture is an indeed an important consideration when performing scaling and (2) the best performing model can fluctuate at different scales. We believe that the findings outlined in this work has significant implications to how model architectures are currently evaluated in the community.

1 Introduction

The paper investigates how architecture and inductive bias affect language-model scaling, using experiments across diverse architectures and scales. It finds that scaling behavior differs by architecture and compute region, and upstream performance may not predict downstream transfer.

1 Introduction

  • The study pretrains and fine-tunes more than 100 models across ten architectures, spanning multiple compute regions and sizes from 15M to 40B parameters.The architectures include Transformer variants, efficient models, and non-Transformer approaches such as convolutions and MLP-Mixers.Source: §1 · p. 1
  • Architecture choices can alter scaling across performance, FLOPs, speed, and parameter count, complicating comparisons across compute regions.The introduction contrasts parameter sharing in Universal Transformers and ALBERT with the high parameter-to-FLOPs ratio of Switch Transformers.Source: §1 · pp. 1–2
  • Among the ten architectures, the vanilla Transformer has the best scaling behavior, although it does not have the greatest absolute performance at each compute region.The paper derives architecture-specific scaling laws and reports that their scaling coefficients differ greatly.Source: §1 · p. 2
  • Models that perform well in one compute region may not perform best in another, and some models struggle to scale despite decent low-compute performance.The authors also report that some models do not scale or show a negative trend, with Performer struggling when scaled up.Source: §1 · p. 2
  • Upstream pretraining perplexity may not correlate well with downstream transfer, making architecture and inductive bias relevant to downstream performance.Source: §1 · p. 2

2 Related Work

Prior scaling studies found benefits from increasing model scale, but largely used the same Transformer implementation within each study. This paper extends architecture comparisons across the model-scale axis.

2 Related Work

  • Prior work reported power-law loss scaling with model size, dataset size, and training compute, while model width and depth had minimal effects across a broad scale range.Related studies also reported scaling benefits for autoregressive generation and transfer learning.Source: §2 · pp. 2–3
  • Transfer studies found downstream performance increasing with scale from 60M to 11B parameters, while multilingual encoder studies reported improved cross-lingual ability as models scaled.Source: §2 · p. 3
  • Earlier scaling studies generally retained the same Transformer implementation, motivating systematic comparison of how architectural inductive biases affect scaling.The paper extends architecture-performance analysis to the model-scale axis.Source: §2 · p. 3

3 Methods

The experiments use sequence-to-sequence models to compare Transformer and non-Transformer architectures, including variants that change attention, parameter sharing, token mixing, or convolutional processing.

3 Methods

  • The experimental setup uses a sequence-to-sequence framework following T5, chosen because encoder-decoder models can express both encoding and decoding tasks.Source: §3.1 · p. 3

3.1 Models

  • The Transformer variants include Evolved Transformers with convolutions and attention, Universal Transformers with shared parameters and recurrent-like computation, and sparsely activated Switch Transformers.The basic Transformer follows the T5-style setup; Evolved Transformers are scaled like vanilla Transformers, while Universal Transformer scaling varies computation separately from parameter count.Source: §3.1 · p. 3
  • Efficient Transformer variants include ALBERT, which shares layer parameters and factorizes embeddings and output softmax, Performer, which uses kernel attention, and Funnel Transformer, which downsamples the encoder sequence.The Performer experiments use the ReLU-kernel variant; the Funnel implementation uses a vanilla Transformer decoder.Source: §3.1 · p. 4
  • The study tests Mixture of Softmaxes at the softmax layer and Gated Linear Units in place of position-wise feed-forward networks.These candidates were selected as general improvements not necessarily tied to Transformers.Source: §3.1 · p. 4
  • Non-Transformer models include lightweight depthwise convolutions, dynamic convolutions with time-dependent kernels, and MLP-Mixers that mix input tokens in the encoder.Token-mixers are used only in the input encoder because Mixers had not been used in autoregressive decoding.Source: §3.1 · p. 4

3.2 Experiment Setup

Models share a common pretraining and fine-tuning setup, with several sizes tested per architecture. Scaling constraints differ by architecture, and the largest Switch Transformer is a stated exception.

3.2 Experiment Setup

  • All models are pretrained for 219 steps on English C4 and fine-tuned for 100K steps on GLUE, SuperGLUE, and SQuAD, with upstream perplexity and downstream transfer evaluated.Training uses a shared tokenizer and Mesh TensorFlow implementation, with 16 TPU-v3 chips.Source: §3.2 · p. 4
  • The study tests tiny through XL sizes where scaling is straightforward, but Universal Transformers and ALBERT have limitations in matching T5 sizes without substantial FLOPs costs.For convolutional models, dmodel denotes the hidden channel depth.Source: §3.2 · p. 4
  • The largest Switch Transformer was scaled suboptimally, so its final data point may not represent the family’s full potential.Source: §3.2 · p. 4

3.3 Main Results

The reported results compare ten model architectures across pretraining and finetuning, including compute and transfer-related measures.

3.3 Main Results

  • The comparison reports trainable parameters, single-pass FLOPs, speed, validation perplexity, and aggregate results on 17 downstream tasks.The downstream aggregates cover GLUE, SuperGLUE, and SQuAD; the authors also report verifying that T5 base achieves results similar to Raffel et al. (2019).Source: §3.3 · p. 5

3.4 Do all models scale the same way?

Architectures show distinct scaling curves, and upstream pretraining performance does not reliably track downstream transfer. Some models that fare well upstream struggle on downstream tasks.

3.4 Do all models scale the same way?

  • Most models, including LConv and Evolved Transformer, match or outperform vanilla Transformer upstream but fail to scale with higher compute.Performer’s pretraining perplexity decreases 2.7% from base to large, compared with 8.4% for vanilla Transformer.Source: §3.4 · p. 5
  • Most architectures have distinct upstream and downstream scaling curves, and their upstream and downstream scaling trends often differ.The authors report that Funnel Transformer and LConvs hold up upstream but suffer substantially downstream, with a greater disparity for Performer downstream than upstream.Source: §3.4 · p. 5
  • SuperGLUE generally requires encoder pseudo-cross-attention, which convolutional models lack, and some models struggle downstream despite strong upstream performance.The authors connect this architectural mismatch with difficulty learning downstream tasks.Source: §3.4 · p. 5

3.5 Are the best models at each scale different?

The best-performing architecture can change across scales and compute regions, so no single model leads throughout the examined range.

3.5 Are the best models at each scale different?

  • The best model varies by scale and compute region: Evolved Transformer leads vanilla Transformer in tiny-to-small downstream settings, but loses that advantage at larger scales.MoS-Transformer also outperforms vanilla Transformer in some regions but not others.Source: §3.5 · p. 5

3.6 Scaling Law for Each Model

The fitted scaling coefficient differs across architectures: vanilla Transformer has the highest values, while Switch Transformer leads on the upstream-to-downstream scaling metric.

3.6 Scaling Law for Each Model

  • α is the slope of a fitted line across model-scaling scenarios and summarizes relative performance across scales.For α_F,U, the fit relates FLOPs to upstream performance.Source: §3.6 · p. 5
  • Vanilla Transformer has the highest α values overall; Evolved, GLU, MoS, and Funnel Transformers have similar scaling properties.GLU-Transformer scales similarly but slightly worse than vanilla Transformer, despite better absolute performance in some compute regions.Source: §3.6 · p. 5
  • LConv, Universal Transformer, MLP-Mixer, and Performer are difficult to scale, especially downstream; this ALBERT version trends negatively as it scales up.The authors note that parameter sharing across its encoder and decoder may partially explain ALBERT’s scaling difficulty.Source: §3.6 · p. 5
  • Switch Transformer performs best on α_U,D, the metric for how downstream performance scales with upstream performance.The passage defines a higher α_U,D as better transfer to downstream tasks as a model scales.Source: §3.6 · p. 7
  • Architectures that change less from the main Transformer tend to retain similar scaling behavior, while changing inductive bias alters scaling properties.The passage names GLU-Transformer and MoS-Transformer as examples of models making fewer architectural changes.Source: §3.6 · p. 7

3.7 Do Scaling Protocols influence model architectures in the same way?

Scaling depth and width produce different downstream patterns across architectures, although upstream scaling trends remain similar. Depth has a more substantial downstream impact than width.

3.7 Do Scaling Protocols influence model architectures in the same way?

  • Depth scaling affects downstream performance similarly across most architectures, with LConv as the exception.Upstream negative-log-perplexity scaling trends remain quite similar despite differences in absolute performance.Source: §3.7 · p. 7
  • Evolved Transformers scale slightly better than the other architectures when scaling FFN width downstream.Source: §3.7 · p. 7
  • Depth scaling has a more substantial impact on downstream scaling than width scaling.Source: §3.7 · p. 7

3.8 Epilogue and Conclusion

Across ten architectures, scaling behavior differs, and the model that performs best can change with compute region or model size. The authors emphasize evaluating upstream and downstream behavior and scaling choices when developing architectures.

3.8 Epilogue and Conclusion

  • Across ten architectures, scaling behavior differs, and models that perform well in one compute region or size may not do so in another.Source: §3.8 · p. 7
  • Architectures that perform well on upstream perplexity may fail to transfer to downstream tasks.Source: §3.8 · p. 7, Figure 3 · p. 8
  • Evaluation at one compute region may miss architectural differences; the authors also suggest considering whether inductive biases affect emergent capabilities.Source: §3.8 · p. 8
  • Different architectures may respond differently to scaling protocols, making model comparison and benchmarking challenging.Source: §3.8 · pp. 8–9
  • The authors advise caution with expensive runs on architectures that substantially modify attention, citing Mixers and Performers as generally high-risk options in their experiments.Source: §3.8 · p. 9
  • For practitioners who do not need billion-parameter scaling, inductive biases tailored to small or low compute may be sufficient.Source: §3.8 · p. 9

4 Appendix

The appendix describes model-size configurations and scaling choices, including separate considerations for Switch Transformers and Universal Transformers.

4 Appendix

  • For most models, the appendix follows uniform scaling across standard T5 sizes and lists configurations by layer count, MLP width, hidden size, key-value size, and head count.Source: §4.1 · p. 11, Table 3 · p. 11
  • Switch Transformer scaling is documented separately, with its configuration table identifying the number of experts as a parameter.Source: §4.1 · p. 11, Table 4 · p. 11
  • Universal Transformer scaling explores increasing MLP width to 32K after increasing recurrent operations did not improve performance despite added FLOPs.Source: §4.1 · p. 11, Table 5 · p. 11

Related papers

Loading 2207.10551v1…