PaperlayerView exact revision on arXiv

Source-linked AI summary

What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?

Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, Colin Raffel

arXiv:2204.05832v1cs.CLcs.LGstat.ML

TL;DR

The paper asks which Transformer architectures and pretraining objectives yield the strongest zero-shot generalization, and evaluates them systematically with and without multitask finetuning. It finds that causal decoder-only models with full language modeling are best after pretraining alone, while encoder-decoder models with masked language modeling are best after multitask finetuning, motivating cross-architecture adaptation.

  • Problem

    Prior work provided limited systematic evidence about which architecture and pretraining objective best support zero-shot generalization, especially at large scale and with multitask finetuning.

  • Method

    The study compares six architecture–objective combinations across causal and non-causal decoder-only and encoder-decoder models, evaluating zero-shot performance before and after multitask finetuning and testing adaptation between designs.

  • Results

    Causal decoder-only models pretrained with full language modeling perform best after unsupervised pretraining, whereas encoder-decoder models pretrained with masked language modeling perform best after multitask finetuning.

  • Takeaways & Limitations

    Architecture and objective choices should depend on whether the model will undergo multitask finetuning, while adaptation offers a route toward models supporting different uses.

  • Takeaways & Limitations

    The study omits encoder-only architectures because their fixed output length limits applicability to zero-shot settings.

Abstract

from arXiv · show

Large pretrained Transformer language models have been shown to exhibit zero-shot generalization, i.e. they can perform a wide variety of tasks that they were not explicitly trained on. However, the architectures and pretraining objectives used across state-of-the-art models differ significantly, and there has been limited systematic comparison of these factors. In this work, we present a large-scale evaluation of modeling choices and their impact on zero-shot generalization. In particular, we focus on text-to-text models and experiment with three model architectures (causal/non-causal decoder-only and encoder-decoder), trained with two different pretraining objectives (autoregressive and masked language modeling), and evaluated with and without multitask prompted finetuning. We train models with over 5 billion parameters for more than 170 billion tokens, thereby increasing the likelihood that our conclusions will transfer to even larger scales. Our experiments show that causal decoder-only models trained on an autoregressive language modeling objective exhibit the strongest zero-shot generalization after purely unsupervised pretraining. However, models with non-causal visibility on their input trained with a masked language modeling objective followed by multitask finetuning perform the best among our experiments. We therefore consider the adaptation of pretrained models across architectures and objectives. We find that pretrained non-causal decoder models can be adapted into performant generative causal decoder models, using autoregressive language modeling as a downstream task. Furthermore, we find that pretrained causal decoder models can be efficiently adapted into non-causal decoder models, ultimately achieving competitive performance after multitask finetuning. Code and checkpoints are available at https://github.com/bigscience-workshop/architecture-objective.

1 Introduction

The paper systematically examines how Transformer architecture, pretraining objective, and multitask finetuning affect zero-shot generalization. It finds that the best combination depends on whether multitask finetuning follows pretraining, and explores adaptation across model designs.

  • Motivation: Zero-shot generalization enables language models to perform varied tasks without additional task-specific data or training.The paper motivates systematic study because existing architecture and objective comparisons largely focused on transfer learning and smaller models.
  • Study design: The study compares six architecture–objective pairs across decoder-only and encoder-decoder models, with and without multitask finetuning.Experiments use models with 5 billion parameters, encoder-decoder models with 11 billion parameters, 168 billion pretraining tokens, and 30 evaluation datasets.
  • Main findings: Causal decoder-only models trained with full language modeling perform best immediately after unsupervised pretraining, whereas masked-language-modeling models perform better after multitask finetuning.After finetuning, decoder-only models perform worse and models pretrained with masked language modeling perform significantly better.
  • Adaptation: The paper studies adaptation as a way to obtain models suited to both generative use cases and multitask finetuning.This addresses the opposing architecture and objective preferences observed across evaluation settings.
  • Implications: The results clarify the interplay among architecture, objective, multitask finetuning, and zero-shot generalization while identifying paths toward more efficient model development.The proposed paths target either purely generative prompted use cases or multitask finetuning.

2 Background

This section introduces Transformer architecture variants, pretraining objectives, model adaptation, multitask finetuning, and zero-shot prompting considered in the study.

  • Architectures: Transformer variants differ primarily in attention masking, with causal decoders restricted to prior tokens and non-causal conditioning representations allowed bidirectional attention.Encoder-decoder models place conditioning information in an encoder, while non-causal decoders modify the conditioning region's attention mask.
  • Architectures: Encoder-only models are omitted because they produce only as many tokens as their input, limiting applicability in zero-shot settings.
  • Pretraining objectives: Full language modeling predicts every next token, prefix language modeling predicts tokens outside a non-causal prefix, and masked language modeling reconstructs masked tokens or spans.Masked language modeling uses sentinel tokens for masked spans in text-to-text models.
  • Model adaptation: Adaptation extends pretraining with a different architecture or objective, including conversion between causal and non-causal decoders by switching the attention mask.The study examines both MLM-to-FLM language modeling adaptation and the reverse FLM-to-MLM non-causal adaptation.
  • Multitask finetuning: Multitask finetuning trains models on prompted tasks in natural-language formats to improve zero-shot performance beyond purely unsupervised pretraining.The study uses openly available datasets and prompts developed for T0.
  • Zero-shot evaluation: Zero-shot prompting formulates tasks in natural language without supervised examples, but performance can be sensitive to prompt wording.

3 Methods

The study systematically compares architecture–objective combinations, multitask finetuning, and evaluation settings under approximately matched compute budgets. Models are pretrained on C4 at large scale, then optionally finetuned on the T0 task mixture and evaluated on two zero-shot benchmarks.

  • Study design: The study pretrains all architecture–objective combinations on 168 billion C4 tokens and evaluates zero-shot performance with and without multitask finetuning.The study also examines adaptation across architectures and objectives.
  • Architectures and objectives: The comparison covers causal decoder, non-causal decoder, and encoder-decoder architectures using FLM, PLM, or MLM where compatible.Causal decoders use FLM or MLM; non-causal decoders and encoder-decoders use PLM or MLM.
  • Compute matching: The experiments approximately match pretraining compute across architectures, with encoder-decoder models scaled to 11B parameters versus 4.8B for decoder-only models.The encoder-decoder architecture has roughly twice as many layers and approximately 10% higher runtime cost due to cross-attention.
  • Multitask finetuning: Multitask finetuning uses the T0 training mixture for 13 billion tokens, following T0 configurations without an intermediate language-modeling adaptation stage.The authors report no systematic improvement from adding language-modeling adaptation first.
  • Evaluation: Zero-shot performance is measured on T0-Eval and EAI-Eval using benchmark-specific prompt and aggregation procedures.T0-Eval averages median per-task accuracy across 11 datasets, while EAI-Eval averages accuracy across 31 datasets.

4 Experiments

Without finetuning, causal decoder models with full language modeling perform best, while multitask finetuning reverses the preferred architecture–objective pattern. Encoder-decoder MLM models lead after finetuning, and benchmark outcomes partly reflect task composition and prompt design.

  • After unsupervised pretraining only: Causal decoder models systematically outperform other architectures after language-modeling pretraining alone, while encoder-decoder models lag far behind.The non-causal decoder remains within a percent of the causal decoder, whereas T0-Eval performance is close to random baseline.
  • After multitask finetuning: Encoder-decoder models with MLM achieve the best zero-shot performance after multitask finetuning.They lead by a large margin on T0-Eval and also achieve the best performance on EAI-Eval; non-causal MLM is a close second on EAI-Eval.
  • After multitask finetuning: Multitask finetuning reverses the unsupervised-pretraining pattern: MLM models improve relative performance, while decoder-only models perform worse.The paper reports this as a broad impact of multitask finetuning on architecture and objective choice.
  • Influence of tasks and prompts: EAI prompts outperform average T0 prompts for every model, especially causal decoder language models without multitask finetuning.EAI prompts were hand-tuned for GPT-3, whereas T0 prompts emphasized diversity and naturalness rather than model performance.
  • Influence of tasks and prompts: EAI-Eval’s decoder-only advantage reflects specific tasks rather than differences in prompts when EAI prompts are used for both task subsets.The comparison separates tasks shared with T0-Eval from non-T0-Eval tasks.

5 Can models be adapted from one architecture/objective to another?

The study examines adapting decoder-only models between causal/full-language-modeling and non-causal/masked-language-modeling configurations. Adaptation accelerates convergence and can improve zero-shot performance while producing models suited to both generation and multitask finetuning.

  • Adaptation methods: Adaptation switches a decoder-only model between causal FLM and non-causal MLM by changing the attention mask.The model parameters and overall architecture can remain unchanged.
  • Language modeling adaptation: 105B additional tokens match the loss reached after 168B tokens of FLM pretraining, giving LM adaptation a 1.6× speed-up.This starts from a non-causal decoder pretrained with MLM and continues training it as a causal decoder with FLM.
  • Non-causal masked language modeling adaptation: MLM convergence accelerates 3.3× versus non-causal training from scratch and up to 9.1× versus causal training from scratch with MLM.This reverse adaptation starts from a causal FLM-pretrained decoder and continues training it as a non-causal MLM decoder.
  • Non-causal masked language modeling adaptation: The reverse adaptation produces both a zero-shot model and an excellent generative model for 1.3× the cost of training a single model.The reported cost is lower than the previously considered language modeling adaptation.
  • Zero-shot evaluation: After multitask finetuning, the MLM-adapted model performs best by a significant margin and outperforms every considered model on EAI-Eval.The comparison controls for total training tokens across the evaluated variants.
  • Practical recommendation: For the smallest total compute budget, the study recommends causal FLM pretraining followed by non-causal MLM adaptation before multitask finetuning.This recipe is intended to obtain both generative and multitask models.

6 Conclusion

The paper systematically compares architectures and objectives for zero-shot generalization with and without multitask finetuning. It finds opposite best choices across these settings and proposes adaptation as an efficient compromise.

  • The study compares FLM and MLM across causal/non-causal decoder-only and encoder-decoder architectures, with and without multitask finetuning.
  • Causal decoder-only FLM performs best immediately after unsupervised pretraining, whereas encoder-decoder MLM performs best after multitask finetuning.
  • Causal FLM pretraining followed by non-causal MLM adaptation accelerates convergence and yields both an excellent generative model and a strong zero-shot model.

B.1 Societal impact

The paper discusses societal risks associated with large language models and states that its study focuses on benchmarking modeling aspects. It did not examine how architecture and objective choices affect toxicity or bias.

  • The study is oriented toward benchmarking modeling aspects and does not introduce a novel societal challenge beyond those already identified.
  • The released models are intended for research, and the authors did not evaluate their potential biases or guarantee that they will not produce harmful content.
  • The study did not explore whether specific architectures and objectives affect a model's toxicity and biases.The authors identify this as a limitation and note that the released models enable follow-up research.

C Implementation: prefix language modeling for encoder-decoder on TPU

To train encoder-decoder models with prefix language modeling under fixed sequence-size constraints, the implementation concatenates two independent examples into one sequence.

  • Two 626-token examples are concatenated into a 1,252-token encoder-decoder sequence while masking keeps them independent.The batch size is reduced from 2,048 to 1,024 sequences to keep the token count constant.

D Evaluation: benchmarks composition and baselines

Evaluation uses EAI-Eval and T0-Eval, with task and prompt assignments specified separately. Random baselines are provided for all considered tasks.

  • EAI-Eval and T0-Eval contain the tasks used for zero-shot evaluation, while T0 also includes tasks used for multitask finetuning.T0 tasks use multiple prompts; EAI-Eval tasks use a single prompt.
  • EAI-Eval and T0-Eval share tasks but have no shared prompts, separating evaluation prompt formats across the benchmarks.
  • Random baselines are reported for all tasks considered across the EAI harness and T0 Eval.The baselines were obtained from the papers introducing these tasks.

E.1 Preliminary results and evolution throughout pre-training

Preliminary results motivate excluding masked-language-model evaluations after pretraining alone and omitting systematic language-model adaptation before multitask finetuning. These choices follow early evidence that MLM remained near random and adaptation often reduced performance.

  • Design decisions: The study therefore omitted span-corruption evaluation after pretraining alone and did not systematically apply LM adaptation before multitask finetuning.
  • Preliminary results: After 84B pretraining tokens, masked language modeling achieved close to the random 33.3% baseline on EAI-Eval and underperformed full and prefix language modeling.
  • Preliminary results: LM adaptation before multitask finetuning did not consistently improve zero-shot generalization and worsened it in most early-model cases.The decision was based on models pretrained for 42B tokens.
  • Design decisions: LM adaptation alone was insufficient, while most performance gains came from multitask finetuning.

E.2 Complete Results

The complete results compare all experiments across architectures, objectives, training stages, and evaluation checkpoints. Multitask finetuning, dropout, and cross-architecture adaptation materially shape zero-shot performance.

  • Evaluation: The experiments report intermediary checkpoints and all multitask-finetuned checkpoints on T0-Eval.
  • Complete results: Table 8 organizes experiments by architecture, objective, token count, and training stages including pretraining, adaptation, and multitask finetuning.Architectures are causal decoder-only, non-causal decoder-only, and encoder-decoder.
  • Baselines: The T5-LM baseline, ED:MLM (1.28T) plus ED:PLM (131B), reports 39.0 on EAI-Eval and 43.2 on T0-Eval.
  • Baselines: The random baseline reports 32.9 on EAI-Eval and 41.7 on T0-Eval.
  • Pretraining scale: With only a quarter of the pretraining tokens, the best encoder-decoder MLM model already exceeded the final performance of the other configurations after multitask finetuning.The ordering did not change significantly throughout pretraining.
  • Multitask finetuning: Dropout significantly boosts zero-shot generalization during multitask finetuning, with proportionally similar effects across architectures and objectives.
  • Architecture adaptation: Converting between causal and non-causal decoder-only models requires only switching the attention mask, whereas encoder-decoder conversion requires pruning or adding parameters.
Loading 2204.05832v1…