Source-linked AI summary
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu
TL;DR
Transfer-learning research for NLP spans diverse objectives, architectures, datasets, and methods, making rigorous comparison difficult. The paper uses a unified text-to-text framework and empirical survey, then combines the findings with larger models and data; denoising objectives outperform language modeling and deshuffling, while translation remains below state of the art.
Problem
The diversity of transfer-learning techniques, datasets, benchmarks, and fine-tuning methods makes it difficult to compare algorithms and understand their contributions systematically.
Method
The paper empirically surveys transfer-learning techniques through a unified text-to-text framework, combining systematic comparisons with scaled models and the C4 dataset.
Results
Across the studied tasks, denoising objectives outperform language modeling and deshuffling, while scaling the combined approach achieves state-of-the-art results on many tasks.
Takeaways & Limitations
Among the denoising variants studied, objective choice should mainly reflect computational cost because their performance differences were not remarkable.
Takeaways & Limitations
The study does not achieve state-of-the-art performance on the translation tasks examined, suggesting English-only pre-training and scale are insufficient against more sophisticated methods.
Abstract
from arXiv · showhide
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.
1. Introduction
The paper frames NLP transfer learning as a diverse, difficult-to-compare landscape and introduces a unified text-to-text framework for systematically studying it. Combining this framework with large-scale models and C4 yields state-of-the-art results on many benchmarks while releasing resources for future work.
- Motivation: Transfer learning research spans varied pre-training objectives, unlabeled data sets, benchmarks, and fine-tuning methods, making contributions difficult to compare.The paper motivates a more rigorous understanding of this rapidly developing landscape.
- Unified framework: Every text-processing task is represented as text input mapped to generated target text, including translation, question answering, and classification.This formulation supports a common model, objective, training procedure, and decoding process across tasks.
- Unified framework: The unified framework enables systematic comparisons of transfer-learning objectives, unlabeled data sets, and related factors across diverse English NLP problems.The study includes tasks such as question answering, document summarization, and sentiment classification.
- Study scope: The paper studies existing techniques rather than proposing new methods, combining survey, exploration, and empirical comparison.Its experiments scale models up to 11 billion parameters to explore current approaches' limits.
- Contributions: Scaling the study's insights produces state-of-the-art results on many considered tasks, alongside releases of C4, data sets, pre-trained models, and code.C4 is described as hundreds of gigabytes of clean English text scraped from the web.
2. Setup
The paper uses Transformer-based text-to-text modeling and constructs C4 as a large cleaned source of unlabeled English text for pre-training. Downstream tasks are converted into generation problems, with STS-B handled by discretizing similarity scores and WNLI excluded from validation reporting because of overlap with WSC.
- Model and framework: T5 is a Transformer-based text-to-text model that provides a common framework for the paper's NLP tasks.The implementation uses an encoder-decoder Transformer, while all studied models are based on the Transformer architecture.
- Model and framework: The Transformer processes token embeddings through encoder and decoder blocks using self-attention, including causal self-attention in the decoder.Relative position embeddings provide an explicit position signal because self-attention is order-independent.
- Experimental setup: The study examines model scalability using model and data parallelism across slices of Cloud TPU Pods.Training large models is presented as computationally demanding and potentially unsuitable for a single machine.
- Unlabeled data: C4 contains about 750 GB of filtered Common Crawl text and is intended to provide reasonably clean, natural English for pre-training.The filtering removes pages or lines containing code, boilerplate, citation markers, offensive-language terms, and placeholder text.
- Downstream tasks: Downstream tasks are generally cast as text generation, but STS-B scores are rounded to the nearest 0.2 and represented as literal strings.STS-B is the stated exception because it is a regression task with similarity scores from 1 to 5.
- Downstream tasks: The paper excludes WNLI from training and validation-set reporting because its data overlap with WSC risks leaking validation examples.WNLI is also omitted from validation-set average GLUE scores, while test-set reporting in Section 3.7 follows a different policy.
3. Experiments
The experiments systematically compare transfer-learning objectives, architectures, pre-training data, and scaling choices in a unified NLP framework. Denoising and encoder-decoder designs generally perform best, while large-scale training with C4 achieves state-of-the-art results across many tasks.
- Experimental overview: The study empirically surveys pre-training objectives, architectures, unlabeled data sets, and transfer approaches before combining its findings with larger-scale training.The final experiments train models up to 11 billion parameters and introduce the Colossal Clean Crawled Corpus (C4).
- Baseline performance: Pre-training substantially improves performance across almost all benchmarks, with marginal gains only on the high-resource English-to-French translation task.The authors compare the baseline with training from scratch and use English-to-French translation to probe the high-resource regime.
- Architectures: The encoder-decoder architecture with a denoising objective performs best across tasks, while halving encoder and decoder layers significantly hurts performance.Sharing parameters across the encoder and decoder performs nearly as well despite reducing the total parameter count.
- High-level objectives: The BERT-style pre-training objective performs best overall, although prefix language modeling reaches similar performance on translation tasks.Deshuffling performs considerably worse than both alternatives.
- Denoising variants: Variants of the denoising objective perform similarly overall, but span length and corruption format affect particular benchmarks and training speed.Average span length 3 slightly outperforms i.i.d. noise on most non-translation benchmarks, while span corruption shortens training sequences.
- Objective discussion: Denoising objectives outperform language modeling and deshuffling, so objective selection among denoising variants should mainly reflect computational cost.The authors suggest exploring fundamentally different ways to leverage unlabeled data rather than additional closely related objective variants.
- Pre-training data: Removing heuristic filtering from C4 uniformly degrades performance, while a more domain-constrained corpus can outperform diverse C4 on some tasks.The comparison shows that pre-training data quality and domain composition both influence transfer performance.
4. Reflection
The paper finds that a unified text-to-text framework supports diverse NLP tasks and achieves strong performance when combined with suitable architectures, data, objectives, and scale. It also identifies efficiency, data-selection, language coverage, and task-similarity questions that remain open.
- Text-to-text: The text-to-text framework applies one model, loss function, and decoding procedure to summarization, classification, and regression tasks, achieving comparable or state-of-the-art performance.The framework handles abstractive summarization, natural language inference, and STS-B within the same setup.
- Architectures: The original encoder-decoder Transformer performed best in the text-to-text framework, while parameter sharing halved total parameter count without a substantial performance drop.Although encoder-decoder models use twice as many parameters as encoder-only or decoder-only architectures, their computational cost is similar.
- Unsupervised objectives: Most denoising objectives performed similarly, motivating short target sequences for more computationally efficient unsupervised pre-training.The objectives train models to reconstruct randomly corrupted text.
- Data sets: C4 provides heuristically cleaned Common Crawl text; in-domain data can improve some downstream tasks, but narrow or repeatedly reused data sets can reduce performance.Constraining data to a single domain typically produces a smaller corpus, and small corpora may be repeated excessively during pre-training.
- Scaling: Larger models generally perform better, but smaller models remain important where inference or computational resources are constrained.The paper specifically highlights client-side inference, federated learning, and low-resource applications as settings favoring cheaper models.
- Outlook: The study leaves open more efficient knowledge extraction, principled task-similarity measures, and language-agnostic models, while English-only pre-training did not reach state-of-the-art translation results.The authors note that their denoising approach may not efficiently teach general-purpose knowledge and that task relatedness could guide unlabeled-data choices.
Appendix A. Contributions
The contributions appendix assigns responsibility for the project’s scope, ideas, implementation, experiments, engineering, and data-set construction. It also records a specific evaluation preprocessing decision for WNLI.
- Contributions: Colin designed the project scope, wrote the paper, ran Sections 3.1–3.6 experiments, and contributed substantially to the codebase.The passage also attributes a large portion of the codebase to Colin.
- Contributions: Noam contributed the text-to-text framework, unsupervised objectives, data-set mixing strategies, Transformer implementations, and Section 3.7 experiments.The passage attributes both conceptual and implementation work to Noam.
- Contributions: Adam oversaw engineering and created the C4 data set.The passage also states that Adam implemented part of the project’s data-processing infrastructure, though the excerpt ends mid-sentence.
- WNLI evaluation: The study does not train on WNLI data and evaluates WNLI by converting its test set into a referent-noun-prediction format using a model trained on WSC and DPR.The WNLI preprocessing is inspired by an earlier procedure.
Appendix C. Example Predictions on CNN/Daily Mail
The appendix presents randomly selected CNN/Daily Mail validation examples comparing ground-truth summaries with T5-11B predictions. The examples cover wildlife, indoor biodiversity, an execution delay, and a football transfer story.
- Examples: The appendix uses randomly selected T5-11B validation decodes rather than cherry-picked examples.Each example is shown alongside its ground-truth summary.
- Example 1: The wildlife example’s prediction preserves the leopard–porcupine encounter and its location while paraphrasing the event.The prediction describes the leopard attempting to attack the porcupine before being deterred.
- Example 2: The indoor-biome prediction retains the North Carolina bacterial and archaeal finding and adds quantitative details about land area, arthropods, and fungi.It reports more than 8,000 bacterial and archaeal taxa and additional household biodiversity findings.
- Example 3: The execution example preserves the delayed execution, cloudy pentobarbital dose, and opposition context.The prediction identifies Kelly Gissendaner and the second delay.
- Example 4: The Dani Alves example preserves his non-selection for Brazil and charity single while summarizing the Barcelona-related details.The prediction also mentions his filmed food-service appearance.
Appendix D. Preprocessed Examples
The appendix introduces examples of the preprocessing applied to every data set used in the study. Its purpose is to document how the diverse task inputs were prepared before training and evaluation.
- Preprocessing: The appendix provides preprocessing examples for each data set considered in the study.It presents these examples as documentation of the data-preparation stage.
- Scope: The examples cover the study’s full collection of data sets rather than a single benchmark.The passage explicitly refers to each data set considered.
- Purpose: The appendix focuses on how data sets were preprocessed before the paper’s experiments.The passage frames the section as an account of preprocessing procedures.
D.1 CoLA
The CoLA example presents the sentence “John made Bill master of himself” and assigns it the target label “acceptable.”
- The input sentence is “John made Bill master of himself.”
- The processed target for the CoLA example is “acceptable.”
D.2 RTE Original input:
The RTE example pairs a claim about Italians settled in Slovenia with the statement that Slovenia has 3,000 inhabitants, producing the target “not_entailment.”
- The first sentence states that a smaller proportion of Yugoslavia’s Italians were settled in Slovenia, including approximately 3,000 declared ethnic Italians.
- The second sentence states that Slovenia has 3,000 inhabitants.
- The processed RTE target is “not_entailment.”
D.3 MNLI
The MNLI example contrasts the hypothesis that the St. Louis Cardinals have always won with a premise describing them as mostly a losing team, yielding “contradiction.”
- The hypothesis claims that the St. Louis Cardinals have always won.
- The premise describes the St. Louis Cardinals as mostly a losing team.
- The processed MNLI target is “contradiction.”
D.4 MRPC
This section presents processed examples from multiple language understanding tasks, pairing task-specific inputs with labeled targets or answers. The examples cover paraphrase equivalence, question answering, duplicate detection, sentiment, textual entailment, causal effects, reading comprehension, word sense, and coreference.
- D.4 MRPC: The MRPC example presents two statements about seeing existing evidence in a new light through September 11 and labels them “equivalent.”
- D.4 QNLI: The QNLI example asks where Jebe died, gives “on the road back to Samarkand,” and labels the answer “entailment.”
- D.4 QQP: The QQP example compares questions about desirable attributes in ancient Rome and joining an IT company as a fresher, labeling them “not_duplicate.”
- D.4 SST-2: The SST-2 example labels a positive sentence praising Fincher’s use of technical knowledge for psychological insight as “positive.”
- D.4 STS-B: The STS-B example pairs two statements that representatives could not be reached or located for comment.
- D.4 Additional tasks: Additional examples assign contradiction to “Valence was helping,” present two effects of political violence, answer Joey’s breakfast question with pie, and provide word-sense and coreference inputs.
D.14 CNN/Daily Mail
This example presents Marouane Fellaini and Adnan Januzaj as Manchester United and Belgium teammates who are also best friends. The text describes their night out dancing with friends before Manchester United’s match against Newcastle.
- Fellaini and Januzaj are described as both teammates and best friends.
- The Manchester United and Belgium duo posted restaurant pictures on Monday night before Wednesday’s game against Newcastle.
- The Belgian duo took to the dance floor with friends ahead of the Newcastle game.
D.15 SQuAD
This question-answering example asks what increased oxygen concentrations in a patient’s lungs displace. The context explains that hyperbaric oxygen helps displace carbon monoxide from hemoglobin’s heme group.
- Hyperbaric medicine uses special oxygen chambers to increase the partial pressure of O 2 around the patient.
- Increased oxygen concentrations in the lungs displace carbon monoxide from hemoglobin’s heme group.
- Hyperbaric oxygen chambers are sometimes used to treat carbon monoxide poisoning, gas gangrene, and decompression sickness.
D.16 WMT English to German
This translation example converts an English sentence about Luigi’s wishes into German. The target preserves the meaning that he did not want the brothers to end up in court.
- The English sentence says Luigi never wanted the brothers to end up in court.
- The translated sentence retains the reporting phrase “she wrote.”
- The German target translates the sentence as Luigi saying he never wanted the brothers to land before the court.
D.17 WMT English to French
This translation example describes an infrared Spitzer telescope image as a family portrait of generations of stars. The French target preserves the distinction between older blue stars and harder-to-identify pink newborn stars.
- The Spitzer image is described as a family portrait of countless generations of stars.
- The oldest stars appear as blue dots, while the pink newborn stars are more difficult to identify.
- The French translation places the pink newborn stars in the universe’s star delivery room.
D.18 WMT English to Romanian
The section illustrates English-to-Romanian translation by pairing an original sentence with its Romanian target and showing their processed forms.
- The original input is an English Taco Bell sentence about adding 2,000 US locations by 2022.
- The processed input and target preserve the translation example in task-formatted text, with each sequence split across passages.
- The original target is the corresponding Romanian translation, split across two passages.
Appendix E. Scores on Every Task for All Experiments
Appendix E records scores for every task and experiment, with condensed-result table references and baseline rows identified for comparison.
- The appendix table lists scores achieved on every task in the experiments described in Sections 3.2 to 3.6.
- The table covers GLUE, SuperGLUE, WMT, CNN/Daily Mail, and SQuAD, with task-specific metrics including accuracy, F1, BLEU, and exact match.
- Each row identifies the main-text table containing condensed results, and a star marks the baseline model described in Section 3.1.