Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Motivation

• Unsupervised pre-training on abundant unlabeled text has driven state-of-the-art NLP results and benefits from neural networks’ scalability.The Common Crawl produces about 20TB of web text monthly, motivating larger-data and larger-model approaches.Source: §1 · p. 2
• The field’s diverse objectives, datasets, benchmarks, and fine-tuning methods make rigorous comparison and understanding difficult.The paper addresses this need through a unified approach for systematically studying transfer-learning methods.Source: §1 · p. 2

Unified framework

• Every text-processing task is cast as text input mapped to target text, enabling the same model, objective, training procedure, and decoding process across tasks.The framework covers tasks including translation, question answering, classification, summarization, and sentiment classification.Source: §1 · pp. 2–3, Figure 1 · p. 3

Contributions

• The work surveys and empirically compares existing transfer-learning techniques, then scales the resulting insights to models with up to 11 billion parameters.Its stated goal is a comprehensive perspective on the field rather than proposing new methods.Source: §1 · p. 3
• The study introduces the Colossal Clean Crawled Corpus, consisting of hundreds of gigabytes of clean English web text, and releases its code, datasets, and pretrained models.These resources support transfer learning in data-scarce settings.Source: §1 · p. 3

2. Setup

• T5 casts diverse NLP problems into a unified text-to-text format, using task prefixes and maximum-likelihood training for both pre-training and fine-tuning.Classification predicts label words, while translation and summarization remain generative; STS-B is converted into 21-class classification.Source: §2 · p. 4, §2.4 · pp. 8–9

2.1 Model

• The models use encoder-decoder Transformers with self-attention, residual connections, dropout, relative position embeddings, and shared input-output embeddings.The implementation removes Layer Norm bias, places normalization outside the residual path, and uses shared, logarithmically spaced relative-position buckets through offset 128.Source: §2.1 · pp. 4–5

2.2 The Colossal Clean Crawled Corpus

• C4 is a roughly 750 GB collection assembled from April 2019 Common Crawl text using filters for natural-language quality, unwanted content, and duplication, yielding reasonably clean, natural English text.It is orders of magnitude larger than most pre-training data sets and is released as part of TensorFlow Datasets.Source: §2.2 · pp. 6–7

2.3 Downstream Tasks

• Evaluation spans GLUE and SuperGLUE classification, CNN/Daily Mail summarization, SQuAD question answering, and English-to-German, French, and Romanian translation.GLUE and SuperGLUE are fine-tuned as concatenated combined tasks, with DPR added to SuperGLUE; pre-training uses English data only.Source: §2.3 · pp. 7–8

2.4 Input and Output Format

• Special conversions handle regression and pronoun-resolution tasks: STS-B becomes 21-class classification, while Winograd examples predict referent nouns under leakage-avoidance rules.WNLI is excluded from training and validation reporting, and omitted from average GLUE scores except for test-set results in Section 3.7.Source: §2.4 · pp. 9–10

3. Experiments

• The experiments vary one factor at a time from a fixed baseline to compare objectives, architectures, data, transfer approaches, and scaling, while acknowledging that interactions among factors remain unexplored.This coordinate-ascent design keeps most factors fixed but may miss second-order effects because exhaustive combinations would be prohibitively expensive.Source: §3 · pp. 10–11

3.1 Baseline

• The baseline is a roughly 220-million-parameter standard encoder-decoder Transformer with 12-block encoder and decoder stacks, 12 attention heads, d_model = 768, and dropout 0.1.Its encoder and decoder are each similar in size and configuration to a BERTBASE stack, making the model about twice as large in parameters because it has two layer stacks.Source: §3.1 · p. 11, §3.1.1 · p. 11

3.1.2 Training

• All tasks use the text-to-text format with maximum-likelihood training, while models are pre-trained on C4 and then fine-tuned separately on downstream tasks.Pre-training uses 524,288 steps, 512-token sequences, and batches of roughly 65,536 tokens, totaling approximately 34B tokens; fine-tuning uses 262,144 steps and selects the best validation checkpoint.Source: §3.1.2 · pp. 11–12

3.1.3 Vocabulary

• The shared 32,000-wordpiece vocabulary mixes English C4 data with German, French, and Romanian data, but restricts the model to a predetermined set of languages.The vocabulary is trained using 10 parts English data and 1 part each of German, French, and Romanian data, and is shared across model inputs and outputs.Source: §3.1.3 · p. 12

3.1.4 Unsupervised Objective

• The baseline denoising objective randomly drops 15% of input tokens, replaces consecutive dropped spans with unique sentinel tokens, and predicts the missing spans.Span corruption and predicting only dropped-out tokens reduce pre-training computation; Figure 2 illustrates the input-output transformation.Source: §3.1.4 · pp. 12–13, Figure 2 · p. 13

3.1.5 Baseline Performance

• Pre-training significantly improves performance across almost all benchmarks, except high-resource WMT English-to-French translation, where gains are marginal.The baseline is comparable to similarly sized models, achieving SQuAD exact match 80.88 and MNLI-matched accuracy 84.24; these values are not directly comparable to BERTBASE because the models and pre-training budgets differ.Source: §3.1.5 · pp. 14–15, Table 1 · p. 14
• The baseline’s run-to-run standard deviation is below 1% of the baseline score for most tasks, but variance is higher for low-resource CoLA, CB, and COPA.For CB, the average F1 is 91.22 with a standard deviation of 3.237, partly because its validation set contains only 56 examples.Source: §3.1.5 · p. 15

3.2 Architectures

• The architecture comparison centers on attention masks across encoder-decoder, language-model, and prefix-LM structures.Encoder-decoder models use fully visible encoder attention and causal decoder attention; language models use causal masking throughout, while prefix LMs use fully visible prefix attention followed by causal target attention.Source: §3.2 · p. 15, §3.2.1 · pp. 15–16, Figure 3 · p. 16

3.2.4 Results

• The denoising encoder-decoder performs best across all tasks, while shared encoder-decoder parameters nearly match it and halving encoder-decoder depth substantially hurts performance.The best model uses 2P parameters but the same computational cost as P-parameter decoder-only models; the shared variant outperforms the decoder-only prefix LM, indicating a benefit from explicit encoder-decoder attention.Source: §3.2.4 · pp. 19–20, Table 2 · p. 20

3.3.1 Disparate High-Level Approaches

• The BERT-style denoising objective performs best among prefix language modeling, masked language modeling, and deshuffling, while prefix language modeling is similar on translation tasks and deshuffling performs considerably worse.All objectives corrupt tokenized unlabeled text into an input-target pair and train by maximum likelihood; the denoising objective also outperforms language modeling downstream.Source: §3.3 · pp. 20–21, §3.3.1 · p. 21, Table 4 · p. 22

3.3.2 Simplifying the BERT Objective

• Simplified BERT-style objectives perform similarly overall, but dropping corrupted tokens slightly improves GLUE through CoLA while performing worse than sentinel replacement on SuperGLUE.Replacing corrupted spans or dropping them shortens targets and can speed training, motivating the baseline that predicts only corrupted tokens marked by sentinel tokens.Source: §3.3.2 · pp. 22–23, Table 5 · p. 22

3.3.3 Varying the Corruption Rate

• A 50% corruption rate significantly degrades GLUE and SQuAD, whereas corruption rates of 10%, 15%, and 25% have limited effects, so the experiments retain 15%.Higher corruption rates also produce longer targets that can slow training.Source: §3.3.3 · p. 23, Table 6 · p. 23

3.3.4 Corrupting Spans

• Span corruption with average span length 3 slightly but significantly outperforms i.i.d. corruption on most non-translation benchmarks and speeds training by producing shorter sequences.Across average span lengths 2, 3, 5, and 10, differences are limited; length 10 slightly underperforms some alternatives.Source: §3.3.4 · p. 24, Table 7 · p. 24

3.3.5 Discussion

• Across objective experiments, denoising matters more for performance than the specific denoising variant, so objective choice should primarily reflect computational cost and sequence length.The exploration suggests further variations on these objectives may yield limited gains, whereas different ways of leveraging unlabeled data may be more promising.Source: §3.3.5 · p. 24, Figure 5 · p. 25

3.4.1 Unlabeled Data Sets

• Removing C4’s heuristic filtering uniformly degrades performance, although more constrained corpora sometimes outperform diverse C4; the study releases its dataset variants for comparison.The evaluated sources include C4, unfiltered C4, RealNews-like, WebText-like, Wikipedia, and Wikipedia plus the Toronto Books Corpus.Source: §3.4.1 · pp. 25–26, §3.4 · p. 25, Table 8 · p. 27
• In-domain unlabeled data improves downstream performance, but its benefits are task-specific and motivate more diverse pre-training data for arbitrary domains.SuperGLUE rose from 71.36 with C4 to 73.24, driven largely by MultiRC Exact Match increasing from 25.78 to 50.93 with Wikipedia + TBC; ReCoRD Exact Match increased from 68.16 to 73.72 with RealNews-like data.Source: §3.4.1 · p. 27

3.4.2 Pre-training Data set Size

• Smaller pre-training data sets degrade downstream performance as repeated examples produce lower training loss and possible memorization, although 64 repetitions have limited effects.The experiments truncate C4 to 2^29, 2^27, 2^25, and 2^23 tokens, corresponding to 64–4,096 repetitions over 2^35 ≈34B training tokens; the authors therefore recommend large data sets when possible.Source: §3.4.2 · p. 28, Figure 6 · p. 29

3.5.1 Fine-tuning Methods

• Adapter layers provide parameter-efficient fine-tuning when their dimensionality matches task size, whereas gradual unfreezing causes minor degradation despite speeding fine-tuning.Lower-resource tasks such as SQuAD work well with smaller adapter dimensionality, while higher-resource tasks require larger dimensionality; GLUE and SuperGLUE require large dimensionality when treated as concatenated tasks.Source: §3.5.1 · pp. 29–30

3.5.2 Multi-task Learning

• Multi-task training generally underperforms separate pre-training followed by fine-tuning, with equal mixing especially damaging and temperature-scaled mixing at T = 2 often strongest among the alternatives.Examples-proportional mixing typically has a task-dependent sweet spot for K, while WMT English to French benefits from higher mixing proportions across the tested range.Source: §3.5.2 · pp. 31–32, Table 11 · p. 33

3.5.3 Combining Multi-Task Learning with Fine-Tuning

• Fine-tuning after multi-task pre-training achieves performance comparable to the baseline, leave-one-out training is only slightly worse, and supervised-only pre-training is significantly worse except for translation.The results suggest that multi-task pre-training can support adaptation to unseen tasks without dramatic task interference, while unsupervised pre-training is more important for non-translation tasks.Source: §3.5.3 · pp. 33–34, Table 12 · p. 34

3.6 Scaling

• Increasing training time or model size consistently improves performance, with larger models generally adding more than extra time or batch size; ensembling is also effective but task-dependent.Four-model ensembles significantly outperform other scaling approaches on CNN/DM, WMT English to German, and WMT English to Romanian, whereas neither ensemble improves substantially over baseline on SuperGLUE.Source: §3.6 · pp. 34–35, Table 13 · p. 35
• Scaling choices involve deployment trade-offs: larger models increase fine-tuning and inference costs, whereas longer pre-training of a small model can amortize across downstream tasks.An ensemble of N models has a cost similar to a model with N× higher computational cost, so the eventual use case should guide the scaling strategy.Source: §3.6 · pp. 35–36

3.7 Putting It All Together

• The study then combines these experimental insights to push performance on popular NLP benchmarks with larger models and large amounts of data.The authors begin from the baseline training approach and introduce additional changes to explore the limits of transfer learning for NLP.Source: §3.7 · p. 36
• The models were pre-trained on C4 for 1 million steps with batches of 211 sequences of length 512, totaling about 1 trillion pre-training tokens, and ranged from about 60 million to about 11 billion parameters.Source: §3.7 · pp. 36–37
• T5 achieved a state-of-the-art average GLUE score of 90.3 and outperformed the previous SQuAD state of the art by over one Exact Match point.The GLUE result was especially strong on MNLI, RTE, and WNLI; SQuAD comparisons were affected by ensembles and external data in competing systems.Source: §3.7 · pp. 38, 40
• T5 achieved state-of-the-art performance on 18 of the 24 tasks considered, and its largest 11-billion-parameter model performed best among the model-size variants across all tasks.Source: §3.7 · p. 38
• SuperGLUE improved from 84.6 to 88.9, nearly matching human performance of 89.8, while exceeding human performance on MultiRC and ReCoRD but trailing humans on COPA and WSC.The results indicate strong reading-comprehension performance alongside remaining difficulties on some low-resource linguistic tasks.Source: §3.7 · p. 40
• T5 did not achieve state-of-the-art performance on WMT translation, where backtranslation, cross-lingual training, and larger or different training data complicated comparisons.On CNN/Daily Mail, T5 reached state of the art, with the largest improvement occurring on ROUGE-2-F despite known limitations of ROUGE and extractive baselines.Source: §3.7 · p. 40
• T5-Base substantially outperformed a baseline trained on the same 1 trillion tokens across all downstream tasks, showing that non-scaling design changes contributed beyond additional pre-training.Additional pre-training also improved performance over the original baseline trained on about 34B tokens.Source: §3.7 · pp. 40–41, Table 15 · p. 41

4.1 Takeaways

• The unified text-to-text framework applied one loss and decoding procedure across generative, classification, and regression tasks, matching task-specific architectures and reaching state-of-the-art performance with scale.The approach covered abstractive summarization, natural language inference, and STS-B.Source: §4.1 · p. 41
• The original encoder-decoder architecture worked best, while sharing encoder-decoder parameters halved the parameter count without a substantial performance drop.Although encoder-decoder models use twice as many parameters as encoder-only or decoder-only models, their computational cost is similar.Source: §4.1 · p. 41
• Denoising objectives performed similarly, favoring short targets for efficiency; large, diverse C4 data mitigated repetition risks, while in-domain data sometimes improved downstream tasks.Full fine-tuning outperformed parameter-efficient updates, and no multi-task mixing strategy surpassed unsupervised pre-training followed by supervised fine-tuning, though task-mixture pre-training was comparable.Source: §4.1 · p. 42
• Scaling data, model size, and ensembles each improved performance, with larger models often outperforming smaller models trained longer and independently trained ensembles outperforming shared-base ensembles.Fine-tune-only ensembling still substantially outperformed a single model.Source: §4.1 · p. 42
• Models with up to 11 billion parameters achieved state-of-the-art results across many benchmarks using C4 span denoising, multi-task pre-training, and task-specific fine-tuning.The models were trained on over 1 trillion tokens, and the authors released code, C4, and pre-trained weights for replication and extension.Source: §4.1 · p. 42

4.2 Outlook

• The outlook prioritizes cheaper models for low-resource settings, more efficient knowledge extraction, principled task-similarity measures, and language-agnostic models after English-only pre-training underperformed on translation.The authors also note that in-domain data can help downstream performance and that language-agnostic modeling would avoid advance vocabulary-language choices.Source: §4.2 · p. 43

Appendix A. Contributions

• The authors divided responsibility across project scope, research ideas, experiments, engineering, data, infrastructure, documentation, and open-sourcing, with all authors helping set the research direction.Colin led the scope, paper, major experiments, and code; Noam contributed the text-to-text framework and objectives; Adam oversaw engineering and created C4; other authors handled experiments, data, infrastructure, documentation, and fine-tuning.Source: Appendix A · p. 45

Appendix B. Converting WNLI to Our Text-to-Text Format

• WNLI data are excluded from training, but its test set is evaluated by converting examples into the referent noun prediction format using a model trained on WSC and DPR.The preprocessor is inspired by He et al. (2019).Source: Appendix B · p. 45
• The conversion identifies premise pronouns whose surrounding words form the longest case- and punctuation-insensitive substring of the hypothesis, selecting among multiple candidates by maximum length.The selected pronoun is marked with asterisks in the premise.Source: Appendix B · p. 45
• The candidate noun is formed by removing the matched hypothesis substring and optionally making the result non-possessive, then comparing it with the model prediction to derive the True-or-False label.For the example, the candidate becomes “the demonstrators.”Source: Appendix B · p. 45

Appendix C. Example Predictions on CNN/Daily Mail

• Randomly selected, non-cherry-picked T5-11B validation decodes are presented alongside ground-truth summaries to illustrate fluent generation.The appendix frames these examples as qualitative evidence rather than a selectively curated showcase.Source: Appendix C · p. 45
• The wildlife prediction preserves the leopard–porcupine confrontation, location, and photographer while rephrasing the story fluently.It adds the concise framing that the leopard tried to make lunch of the porcupine, consistent with the reported fight.Source: Appendix C · p. 46
• The remaining examples retain key facts about indoor biodiversity, a delayed execution, and Dani Alves’s Brazil-selection and Barcelona news.The indoor-biodiversity prediction includes more than 8,000 bacterial and archaeal taxa, while the execution prediction retains the cloudy pentobarbital dose and delay.Source: Appendix C · p. 46
• The Dani Alves prediction preserves the friendlies, charity single, hot-dog-bar footage, and contract uncertainty, while adding a generic Barcelona-news prompt.The displayed continuation includes “click here for all the latest barcelona news.”Source: Appendix C · pp. 46–47

Appendix D. Preprocessed Examples

• The section provides preprocessing examples for every data set considered.Source: Appendix D · p. 47

D.1 CoLA

• The original CoLA sentence is “John made Bill master of himself.”Source: §D.1 · p. 47
• The sentence is processed as “cola sentence: John made Bill master of himself.”Source: §D.1 · p. 47
• The processed target is “acceptable.”Source: §D.1 · p. 47

Original input:

• The original input states that some 3,000 Slovenian inhabitants declared themselves ethnic Italians, while separately stating that Slovenia has 3,000 inhabitants.The first statement concerns ethnic identity and settlement; the second gives a total population figure.Source: §D.2 · p. 47
• The example is converted into a single text-to-text input prefixed with `rte`, preserving both sentences as `sentence1` and `sentence2`.The processed input concatenates the first statement and the separate Slovenia population statement.Source: §D.2 · p. 47
• The processed target label is `not_entailment`.This is the assigned RTE target for the example.Source: §D.2 · p. 47

D.3 MNLI

• The hypothesis states that the St. Louis Cardinals have always won.Source: §D.3 · p. 47
• The premise says the speaker is from Saint Louis and that the Cardinals were mostly a losing team.Source: §D.3 · p. 47
• The processed MNLI input concatenates the hypothesis and premise, producing the target label contradiction.Source: §D.3 · pp. 47–48

D.4 MRPC

• MRPC represents two sentences as a processed input and assigns the target “equivalent.”The example preserves both source sentences while formatting them under the mrpc task prefix.Source: §D.4 · p. 48

D.5 QNLI; D.6 QQP Original input:

• QNLI pairs a question with a sentence and labels the relationship “entailment,” while QQP formats two questions with the target “not_duplicate.”The examples show task prefixes and concatenated question–sentence or question-pair inputs.Source: §D.5 · p. 48, §D.6 · pp. 48–49

D.7 SST2; D.8 STSB

• SST2 converts a review sentence into a processed input with the target “positive,” while STSB presents two related sentences as a similarity example.The STSB passage supplies the paired sentences and their processed representation but no explicit target value.Source: §D.7 · p. 49, §D.8 · p. 49

D.9 CB Original input:

• CB frames a hypothesis and premise as a processed input whose target is “contradiction.”The example contrasts the hypothesis “Valence was helping” with a premise questioning whether he was helping.Source: §D.9 · p. 49

D.10 COPA

• COPA formats a premise, two choices, and an effect question into one processed input for causal-choice reasoning.The example asks which consequence follows from political violence breaking out in the nation.Source: §D.10 · p. 50

D.11 MultiRC

• MultiRC combines a question, candidate answer, and supporting paragraph in a processed reading-comprehension input.The example explains Joey’s surprise by noting that only pie was available instead of traditional breakfast foods.Source: §D.11 · p. 50

D.12 WiC Original input:

• WiC presents two contexts containing “deliberation” and marks the word pair as positive in the processed input.The contexts use deliberation for an act and deliberations for a jury, with the word explicitly identified.Source: §D.12 · p. 51

D.13 WSC and DPR

• WSC formats a passage with the pronoun “*it*” marked for resolution, while the section also covers DPR.The supplied WSC example concerns whether “it” refers to the stable or the swinging window, but no DPR example is shown.Source: §D.13 · p. 51

• Fellaini and Januzaj, Manchester United and Belgium teammates, are portrayed as close friends who shared restaurant and dance-floor photos with companions.The article also mentions their friendship around matches, including celebrations and interactions with manager Louis van Gaal.Source: §D.14 · p. 52
• The reference summary states that the Belgian duo took to the dance floor with friends on Monday night before facing Newcastle in the Premier League.Source: §D.14 · p. 53
• The target adds that Manchester United sought only their second league away win in seven games while sitting two points ahead of Liverpool in fourth place.Source: §D.14 · p. 53

D.15 SQuAD

• The processed input combines the question and hyperbaric-medicine context into a single text-to-text prompt.The original question asks what increased oxygen concentrations in the lungs displace.Source: §D.15 · p. 53
• Increased oxygen concentration helps displace carbon monoxide from hemoglobin’s heme group.The passage describes this as an effect of hyperbaric oxygen treatment.Source: §D.15 · p. 53
• The target answer is carbon monoxide.The processed target preserves the same answer.Source: §D.15 · p. 54

• The original English input is transformed into the instruction-formatted text “translate English to German” while preserving the sentence content.The processed input splits the sentence across two passages but retains the original wording.Source: §D.16 · p. 54
• The German target is shown both as the original target and as a processed target with the same translation.The translation is “Luigi sagte oft zu mir, dass er nie wollte, dass die Brüder vor Gericht landen”, schrieb sie.Source: §D.16 · p. 54

D.17 WMT English to French

• The English source describes a Spitzer infrared image showing old blue stars and harder-to-identify pink newborn stars.The passage calls the image a family portrait of countless generations of stars and places the newborns in a star delivery room.Source: §D.17 · p. 54
• The processed input reformulates the source as an instruction to translate English to French.The instruction precedes the full English description of the Spitzer image.Source: §D.17 · p. 54
• The French target translates the image description, retaining the family-portrait framing, blue oldest stars, pink newborns, and cosmic delivery-room metaphor.The target identifies the image as an infrared photograph taken by the Spitzer telescope.Source: §D.17 · p. 54

• The original English statement says Taco Bell plans to add 2,000 US locations by 2022.Source: §D.18 · p. 54
• The processed input reformulates the example as an instruction to translate English to Romanian.Source: §D.18 · p. 54
• The Romanian target renders the plan as opening 2000 restaurants in the United States by 2022.Source: §D.18 · p. 54

Appendix E

• Table 16 reports scores on every task considered for all experiments in the paper.The appendix covers experiments described in Sections 3.2 to 3.6.Source: Appendix E · p. 56, Table 16 · p. 57
• The table’s first column identifies where condensed results for each experiment were presented, while ⋆ denotes the baseline model from Section 3.1.Source: Table 16 · p. 57