PaperlayerView exact revision on arXiv

Source-linked AI summary

UL2: Unifying Language Learning Paradigms

Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, Donald Metzler

arXiv:2205.05131v3cs.CL

TL;DR

Existing pretrained models are commonly specialized for particular tasks, and the field lacks consensus on the right architecture and pretraining setup. UL2 combines diverse denoising paradigms with mode switching, then demonstrates broad improvements over T5- and GPT-like models, including state-of-the-art performance on 50 tasks at 20B parameters. The authors note scope boundaries in the 20B scaling run and leaderboard comparisons.

  • Problem

    Pretrained models are often geared toward particular problem classes, leaving open how to build one that works universally across many tasks.

  • Method

    UL2 combines multiple span-corruption denoisers in Mixture-of-Denoisers and associates downstream behavior with pretraining schemes through mode switching.

  • Results

    UL2 outperforms T5 and GPT-like models across diverse supervised and few-shot tasks, with state-of-the-art performance on 50 of 50–60 evaluated tasks and setups at approximately 20B parameters.

  • Takeaways & Limitations

    UL2 provides a single pretrained framework intended to support diverse task types and prompting regimes rather than requiring task-specific model choices.

  • Takeaways & Limitations

    The 20B run did not specifically control or mitigate common scaling instabilities, and leaderboard comparisons use published work as the SOTA reference.

Abstract

from arXiv · show

Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives -- two concepts that are commonly conflated. Next, we present a generalized & unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. We furthermore introduce a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes. We conduct extensive ablative experiments to compare multiple pre-training objectives and find that our method pushes the Pareto-frontier by outperforming T5 & GPT-like models across multiple diverse setups. By scaling our model up to 20B parameters, we achieve SOTA performance on 50 well-established supervised finetuning based NLP tasks. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization. On 0-shot MMLU, UL2 20B outperforms T0 and T5 models. UL2 20B also works well with chain-of-thought prompting and reasoning, making it an appealing choice for research into reasoning at a small to medium scale of 20B parameters. Finally, we apply FLAN instruction tuning to the UL2 20B model, achieving MMLU and Big-Bench scores competitive to FLAN-PaLM 62B. We release Flax-based T5X checkpoints for the UL2 20B & Flan-UL2 20B.

1 Introduction

UL2 addresses the task-dependent choice of pretrained models by combining diverse denoising objectives and mode switching in a framework designed for broad effectiveness. Across diverse supervised and prompt-based setups, it improves the balance between discriminative fine-tuning and open-ended generation while outperforming established baselines.

  • NLP model selection often depends on the downstream task, motivating a search for models that work universally well across tasks.
  • Mixture-of-Denoisers combines regular, sequential, and extreme denoising objectives, varying corruption patterns and context to support diverse tasks.R-denoising uses standard span corruption; S-denoising follows sequence order; X-denoising uses extreme span lengths and corruption rates.
  • Mode switching associates pretraining tasks with sentinel tokens, allowing downstream learning to dynamically select among R, S, and X denoisers.
  • UL2 separates backbone architecture from self-supervision and argues that denoiser choice has more impact, making the framework compatible with decoder-only and encoder-decoder models.The backbone remains an efficiency trade-off rather than the defining source of pretrained behavior.
  • UL2 outperforms T5 and GPT-like baselines across all 9 evaluated setups, with average gains of +43.6% over T5 and +76.1% over a language model.
  • At approximately 20B parameters, UL2 achieves state-of-the-art results on a vast majority of more than 50 diverse NLP tasks and strong zero- and few-shot performance.It outperforms GPT-3 175B on zero-shot SuperGLUE and triples an LM-adapted T5-XXL model on one-shot summarization.

2 Background: Pre-trained Language Models

Pretrained language modeling spans autoregressive, masked, permutation, and sequence-to-sequence paradigms, with architectures and objectives often paired in task-specific ways. Prior unified approaches combine objectives within shared Transformer models, but the broader landscape includes distinct efficiency and modeling trade-offs.

  • Pretrained representations are foundational to modern NLP, with GPT introducing causal language modeling and BERT demonstrating the value of bidirectional modeling.
  • BERT’s masked language modeling reconstructs inputs in place using bidirectional receptive fields, while XLNet’s permutation language modeling models dependencies among masked tokens.
  • T5-style encoder-decoder models improved classification and sequence-to-sequence performance but showed limited open-text generation and prompt-based inference.
  • Decoder-only models commonly use causal language modeling, whereas encoder-decoder models process inputs and targets with separate parameter sets and cross-attention.
  • Sparse mixture-of-expert models pursue state-of-the-art performance with a different FLOP-per-parameter ratio, making sparsity orthogonal to pretraining objectives.
  • UniLM combines unidirectional, bidirectional, and sequence-to-sequence language-modeling objectives in one Transformer using a cloze-style formulation.

3 Unifying Language Learning Paradigms (UL2)

UL2 unifies pre-training tasks by separating architecture from objective and expressing denoising schemes through a shared input-to-target formulation. Its Mixture-of-Denoisers combines complementary paradigms, with mode switching adapting pre-training behavior to downstream tasks.

  • Unified perspective: UL2 formulates diverse pre-training tasks as predicting targets from model-conditioned context, enabling approximate reductions between objectives.Span corruption approaches language modeling when the corrupted span covers the full sequence, and can mimic local LM objectives with large spans.
  • Unified perspective: The SpanCorrupt function parameterizes denoising inputs and targets by mean span length µ, corruption rate r, and number of corrupted spans n.Corruptions are applied to spans sampled around µ, while corrupted spans become targets for recovery.
  • Mixture-of-Denoisers: UL2 mixes R-, S-, and X-denoisers: standard span corruption, sequential prefix-language-model denoising, and aggressive recovery of long targets from limited context.R-denoising masks roughly 15% of tokens in short spans, while X-denoising masks approximately 50% of the input sequence.
  • Mixture-of-Denoisers: The final objective mixes seven denoisers with approximately equal participation, omitting causal LM because it is treated as a special case of Prefix-LM.An alternative mixture increases S-denoisers to 50%, with the remaining denoisers sharing the rest.
  • Mode switching: Mode switching adds paradigm tokens [R], [S], and [X] so the model can select a pre-training mode during fine-tuning and downstream few-shot learning.The model can switch among R, S, and X denoisers on demand after pre-training.
  • Model architecture: UL2 keeps architecture choice separate from the pre-training objective, offering decoder-only and encoder-decoder variants while treating their difference as an efficiency trade-off.The paper uses a standard vanilla T5 Transformer architecture for its encoder-decoder formulation.

4 Ablative Experiments

The ablations compare pre-training objectives and architectures across diverse supervised and prompt-based tasks, finding that UL2 consistently outperforms T5-like and GPT-like alternatives. They also show that objective mixtures, mode switching, and balanced denoiser mixtures affect performance.

  • Experimental setup: UL2 outperforms T5-like and GPT-like models on 9 out of 9 tasks.The experiments cover diverse supervised and prompt-based few-shot learning tasks.
  • Architecture comparisons: Encoder-decoder models generally trade higher parameter counts for similar speed, while UL2 decoder exceeds T5 encoder-decoder performance by +14.6%.Under parameter constraints, Prefix-LM decoder is identified as a suitable alternative; UL2 decoder still does not exceed UL2 encoder-decoder.
  • Objective comparisons: UniLM and SCLM outperform standard span corruption on encoder-decoder models, while UniLM gains +9.4% and SCLM +16.1% over causal LM on decoder models.Both objectives outperform T5 on 6 out of 9 tasks, and SCLM performs best on one-shot SGD and ToTTo generation.
  • Mode switching: Mode-switching prompts can create a 48% Rouge-1 performance gap on one-shot XSum, whereas SuperGLUE is less prompt-sensitive.For XSum, selecting the appropriate paradigm prompt is crucial for good performance.
  • Mixture-of-Denoisers ablations: Extreme denoising complements regular denoising but performs poorly alone, while approximately 20% S-denoising is preferred over 50%.Removing S-denoising improves one task but substantially degrades another, supporting its necessity in the mixture.

5 Scaling to 20B Parameters

UL2 is scaled to approximately 20B parameters using an encoder-decoder architecture for the large-scale run. Although UL2 is architecture agnostic, the authors softly recommend encoder-decoder models due to intrinsic sparsity.

  • The scaled run uses an encoder-decoder architecture despite UL2 being architecture agnostic.The authors selected encoder-decoder for this run based on insights from the ablation experiments.
  • The authors softly recommend defaulting to encoder-decoder architectures because of intrinsic sparsity.
  • Approximately 20B parameters are used for the scaled UL2 model.The paper describes 20B as a medium-scale proof-of-concept relative to larger language models.

5.1 Pretraining and Model Configuration

The 20B model is pretrained on C4 for 1 trillion tokens using 512 TPUv4 chips and a 512/512 input-target sequence length.

  • 1 trillion tokens from the C4 corpus are used for pretraining over 2 million steps.
  • 512 TPUv4 chips and a batch size of 1024 are used during pretraining.
  • The input and target sequence lengths are both set to 512, and dropout is set to 0.

5.2 Experiments at 20B scale

The 20B experiments evaluate UL2 across nearly 50+ NLP tasks spanning generation, understanding, reasoning, structured knowledge grounding, and information retrieval. Results compare UL2 with prior state-of-the-art systems using task-appropriate metrics, with many tasks reaching or exceeding previous results.

  • Experimental scope: Nearly 50+ NLP tasks cover generation, human-evaluated generation, understanding, classification, question answering, reasoning, knowledge grounding, and information retrieval.The task suite includes the GENIE, SuperGLUE, Scrolls, UnifiedSKG, and differentiable search index settings.
  • Evaluation protocol: UL2 is evaluated against the best previous SOTA result for each dataset using dominant prior-work metrics.Generation tasks generally use ROUGE-2, while BLEU scores use sacrebleu; external knowledge-base methods are excluded for commonsense comparisons.
  • Representative results: 90.7 Accuracy on QASC (w IR) Test compares with 89.6 from the previous result.
  • Representative results: UL2 reports strong gains across long-range reasoning, including 45.8 EM on QUALITY Test versus 26.0 previously.
  • Representative results: 88.7 EM on ContractNLI Test compares with 77.4 from the previous result.
  • Representative results: 73.8 HITS@10 on DSI-NQ Dev compares with 70.3 from the previous result.

5.2.4 Results on Supervised Finetuning

UL2 achieves state-of-the-art performance on around 50+ NLP tasks and setups, while remaining generally competitive where it does not reach SOTA. Human evaluation also finds solid generation quality on several GENIE metrics.

  • Around 50+ NLP tasks and setups achieve state-of-the-art performance with UL2.The authors note that margins vary substantially across benchmarks and that SOTA difficulty differs by task.
  • UL2 is generally competitive on tasks where it does not achieve SOTA.The authors caution readers to judge the value of individual SOTA results in light of differing benchmark difficulty and baselines.
  • UL2 20B performs well on human-evaluated GENIE tasks and outperforms SOTA on several metrics.The authors describe its generation quality as reasonably solid.

5.2.5 Tradeoffs between Finetuning and Prompt-based Zero-shot Learning (SuperGLUE)

UL2 remains competitive with T5-11B on SuperGLUE while extending strong performance to zero- and few-shot settings, including reasoning tasks. Its results show gains from chain-of-thought and self-consistency prompting at 20B parameters.

  • 5.2.5 Tradeoffs between Finetuning and Prompt-based Zero-shot Learning (SuperGLUE): UL2 20B remains competitive on SuperGLUE and outperforms T5-11B, although it does not achieve state of the art.It still trails ST-MoE-32B.
  • 5.2.6 Generative Few-shot: XSUM Summarization: Zero-shot SuperGLUE results compare UL2 20B with GPT-3, GLaM, PaLM, and compute-matched models.The table reports average scores across these comparisons.
  • 5.2.6 Generative Few-shot: XSUM Summarization: One-shot XSUM performance for UL2 20B is about 3x that of the LM-adapted T5-XXL model.UL2 20B also outperforms LaMDA 137B and compares favorably with compute-matched PaLM 8B, while larger PaLM models perform best.
  • 5.2.7 UL2 for chain-of-thought prompting: Chain-of-thought prompting is motivated by its ability to elicit multi-step reasoning without additional fine-tuning, but it can hurt smaller models.Prior successful applications used much larger models such as LaMDA 137B and PaLM 540B.
  • 5.2.7 UL2 for chain-of-thought prompting: UL2 20B successfully uses chain-of-thought prompting without fine-tuning for multi-step arithmetic and commonsense reasoning tasks.The experiments compare chain-of-thought with standard prompting across five arithmetic and five commonsense benchmarks.
  • 5.2.7 UL2 for chain-of-thought prompting: 22.5% average improvement over standard prompting results from adding self-consistency to chain-of-thought on five arithmetic benchmarks.An external calculator further improves arithmetic performance by a large margin.
  • 5.2.7 UL2 for chain-of-thought prompting: 14.4% average improvement over standard prompting occurs when chain-of-thought with self-consistency is applied to five commonsense benchmarks.The combined approach outperforms standard prompting on four of five benchmarks.
  • 5.2.7 UL2 for chain-of-thought prompting: UL2 20B is a smaller publicly available pretrained model that can leverage chain-of-thought reasoning, although the mechanism behind this ability remains a hypothesis.The authors suggest Mixture-of-Denoisers may contribute and leave further investigation for future work.

5.3 Instruction Tuned UL2 20B with FLAN

FLAN instruction tuning adapts UL2 20B for MMLU and Big-Bench evaluation, where it outperforms Flan-T5-XXL and approaches Flan-PaLM 62B. Chain-of-thought gains are present but direct prompting remains stronger overall.

  • 5.3 Instruction Tuned UL2 20B with FLAN: FLAN instruction tuning adapts UL2 20B with additional training and longer context lengths.Because the FLAN mixture lacks mode-switching prompts, UL2 is trained for another 100K steps without mode tokens.
  • 5.3.1 Few-shot MMLU and Big-Bench Results after Flan training of UL2: FLAN-UL2 20B outperforms Flan-T5-XXL by +1.8% on the MMLU test set and +4.7% on MMLU dev.Its Big-Bench hard score remains competitive and marginally exceeds Flan-T5-XXL.
  • 5.3.1 Few-shot MMLU and Big-Bench Results after Flan training of UL2: FLAN-UL2 20B’s best dev scores nearly reach Flan-PaLM 62B on both MMLU and Big-Bench hard.Table 15 reports the MMLU and BBH comparisons.
  • 5.3.2 Comparisons on using Chain-of-thought vs Direct Prompting: Flan-UL2 outperforms Flan-T5-XXL across all four direct and chain-of-thought evaluation setups.The comparisons use MMLU and Big-Bench hard with both prompting styles.
  • 5.3.2 Comparisons on using Chain-of-thought vs Direct Prompting: 7.4% relative improvement appears on MMLU-CoT for Flan-UL2 over Flan-T5-XXL.The authors report larger gains on chain-of-thought tasks.
  • 5.3.2 Comparisons on using Chain-of-thought vs Direct Prompting: Direct prompting still outperforms chain-of-thought variants for these Flan models, as also observed for PaLM 62B and Flan-PaLM 62B.This comparison is based on the reported dev scores.
  • 5.3.2 Comparisons on using Chain-of-thought vs Direct Prompting: 49.1 versus 49.9 is Flan-UL2’s average across setups compared with Flan-PaLM 62B, while Flan-PaLM 540B remains substantially better.The average covers direct and chain-of-thought MMLU and Big-Bench setups.

6 Conclusion

UL2 combines Mixture-of-Denoisers pretraining with mode switching to train broadly effective language models. Across supervised and few-shot evaluations, it outperforms the cited T5 and GPT-like baselines and achieves state-of-the-art results on 50 tasks.

  • 6 Conclusion: UL2’s two key ideas are Mixture-of-Denoisers pretraining and mode switching for associating downstream behavior with upstream pretraining.MoD mixes multiple pretraining tasks framed as span corruption.
  • 6 Conclusion: +76.1% normalized overall gain and 9 out of 9 tasks show UL2 outperforming T5 in supervised and few-shot evaluations.The conclusion also reports consistent gains over GPT-like models across a wide range of tasks.
  • 6 Conclusion: 50 tasks receive state-of-the-art performance after scaling UL2 to 20B parameters across 50 to 60 NLP tasks and setups.Pretrained UL2 and Flan-UL2 20B checkpoints are released.

8 Author Contributions

The author contributions span project conception, model implementation and training, benchmark evaluation, reasoning experiments, infrastructure, and advising. Multiple contributors led specialized evaluations and supported open-source release.

  • 8 Author Contributions: Yi Tay proposed and led the project, drove implementation and ablations, pretrained UL2 20B, and ran most large-model evaluations.His work covered finetuning and in-context learning experiments.
  • 8 Author Contributions: Mostafa Dehghani co-led the effort, ran initial experiments and SuperGLUE ablations, and helped with UL2 open sourcing.He also contributed to early brainstorming.
  • 8 Author Contributions: Vinh Q. Tran contributed to early project discussions and implemented and trained UL2 across several task and baseline runs.Examples include SamSum, GENIE human evaluations, and CommonsenseQA.
  • 8 Author Contributions: Xavier Garcia optimized the seqio UL2 pipeline and ran machine-translation experiments, while Siamak supported experiments, infrastructure, and algorithm improvements.Neil and Donald served as technical advisors and sponsors; Huaixiu Steven Zheng helped develop and write the paper.
  • 8 Author Contributions: Jason Wei and Xuezhi Wang ran chain-of-thought and self-consistency experiments on reasoning benchmarks, respectively.These contributions supported the reasoning evaluation section.
  • 8 Author Contributions: Hyung Won ran MMLU experiments and wrote that section, while Dara Bahri supported UnifiedSKG SuperGLUE experiments.Tal Schuster evaluated UL2 on Scrolls, VitaminC, and Programming Puzzles.

9 Appendix

The appendix describes UL2’s implementation as a mixture of denoising objectives, with optional PrefixLM and task-specific prefixes, and documents released checkpoints and fine-tuning procedures.

  • Model Release: Released 20B checkpoints use T5X with JAX/Flax, and three fine-tuning checkpoints are provided because results came from continuous fine-tuning.The listed checkpoints are 1.87M, 2.05M, and 2.65M steps.
  • Model Release: The released model uses [NLG], [NLU], and [S2S] mode tags for X-, R-, and S-denoising, respectively.These tags are prepended to example inputs for the particular checkpoint.
  • Implementation: UL2 implementation mixes multiple SeqIO pre-training tasks, optionally adding PrefixLM as the S-denoising task.The generalized implementation applies span corruption with different noise settings and samples among processed datasets.
  • Implementation: The objective accepts configurable noise densities, mean span lengths, task sampling rates, dataset sharding, and optional prefixes.PrefixLM must be the last task prefix when included.
  • Implementation: The preprocessing pipeline can merge examples, reserve space for packing, shard or copy datasets, apply denoising, and sample processed tasks.It validates task-prefix counts and checks that target lengths fit the configured sequence length.
  • Fine-tuning: Most supervised tasks use single-task fine-tuning, while GLUE, SuperGLUE, and selected reasoning tasks use benchmark-specific mixtures.Rainbow co-training includes SIQA, PIQA, AbductiveNLI, Winogrande XL, and CosmosQA.
Loading 2205.05131v3…