Transcending Scaling Laws with 0.1% Extra Compute

1 Introduction

• UL2R continues an existing causal language model with UL2’s mixture-of-denoiser objective, requiring roughly 0.1% to 1% of the original training FLOPs and no new data sources.The approach is designed as an efficient restoration of pretrained models.Source: §1 · p. 2
• UL2 combines prefix language modeling with long-short span corruption and enables inference-time mode switching for language understanding and infilling.These objectives are intended to complement the original causal language-modeling objective.Source: §1 · p. 2
• At 540B scale, UL2R reaches final PaLM performance with approximately half the computation, saving up to 4.4 million TPUv4 hours.The reported computation savings rate is approximately 2x.Source: §1 · p. 2
• U-PaLM performs competitively across English NLP, multilingual, reasoning, and challenging BIG-Bench tasks, including tasks where PaLM struggles.On selected BIG-Bench tasks, UL2R also elicits emergent behavior at smaller scales such as 62B or 8B.Source: §1 · p. 2
• Only a small amount of UL2R, such as 0.1% of tokens or FLOPs, gives U-PaLM bidirectional multi-span infilling prompting.This adds a secondary prompting mode alongside the model’s existing capabilities.Source: §1 · p. 2

Large language models

• Large language models above 100B parameters have predominantly used decoder-only causal objectives, whereas bidirectional models have been more common at smaller scales.GPT-3, Gopher, and PaLM are cited as causal examples.Source: §2 · p. 3

Scaling laws

• Scaling-law research links model scale and compute to quality, while downstream performance is treated as more relevant to usability than upstream cross entropy.Prior studies also found that inductive bias can affect scaling behavior.Source: §2 · p. 3

Emergent Abilities

• Emergent abilities are behaviors absent in smaller models that appear above a critical scale, but UL2R seeks to unlock them without increasing parameter count.The motivation is to reduce reliance on expensive model scaling for new capabilities.Source: §2 · p. 3

Continued Training of Language Models

• Most prior continued-training methods add datasets, instructions, or human annotations, whereas UL2R reuses pretraining data without requiring new data sources.This makes UL2R complementary to continued-training methods based on additional data.Source: §2 · p. 3
• UL2R extends self-supervised adaptation by uptraining a heavily pretrained language model with diverse denoising tasks and reporting rapid improvement across setups.This follows earlier work adapting models between causal and masked language-modeling objectives.Source: §2 · p. 4

Unified language learner (UL2)

• UL2 combines prefix language modeling and span corruption through a mixture-of-denoisers objective with prompts that switch modes during downstream tasks.The cited UL2 model bridges generative causal and bidirectional language modeling.Source: §2 · p. 4

3 U-PaLM

• U-PaLM is initialized from PaLM, retains its architecture, and continues training with UL2R.The section presents U-PaLM as PaLM adapted through UL2R rather than fully pretrained anew.Source: §3 · p. 4

3.1 Training Data

• U-PaLM uses the same data mixture as PaLM and adds no labeled or unlabeled data sources.The training-data choice is intended to keep the comparison consistent.Source: §3.1 · p. 4

3.2 Prefix Language Model Architecture

• The PrefixLM architecture applies bidirectional attention to input tokens while retaining a decoder-only model structure.Its combined sequence uses a prefix and target portion, with later packing and padding optimizations improving example-level sample efficiency.Source: §3.2 · pp. 4–5

3.3 Loss Objectives

• The UL2R mixture contains regular span denoising, extreme span denoising, and sequential denoising, the latter corresponding to the PrefixLM objective.Regular denoising uses shorter spans and lower corruption, whereas extreme denoising uses longer spans or higher corruption.Source: §3.3 · p. 5
• The main 540B mixture uses 50% PrefixLM, 25% long span corruption, and 25% regular span corruption with UL2 mode prompts.This simplified three-task mixture was selected as efficient for continued training.Source: §3.3 · p. 5

3.4 Training

• The 540B model trains for 20k steps on approximately 1.3 billion additional tokens, representing 0.16% extra computation.Training uses a cosine learning-rate decay from 10^-4 to 10^-6 and finishes in about five days on 512 TPUv4 chips.Source: §3.4 · p. 5

4 Experiments

• The section presents experimental results for U-PaLM.Source: §4 · p. 5
• U-PaLM is evaluated through experiments reported in this section.Source: §4 · p. 5
• The section serves as the experimental-results portion of the paper.Source: §4 · p. 5

Improved Scaling Properties

• U-PaLM substantially outperforms original PaLM models at both 8B and 540B scales on the aggregated few-shot evaluation.The evaluation averages 26 zero-shot and one-shot NLP tasks, including NLU and NLG benchmarks.Source: §4.1 · pp. 5–6

8B versus 540B

• At 8B, UL2R’s gains narrow as PaLM approaches a performance plateau, whereas at 540B gains continue growing through 780B tokens.The authors attribute the continued 540B gains to remaining headroom beyond 780B tokens.Source: §4.1 · p. 6

Savings Rate

• ∼2x savings at 540B let UL2R reach final PaLM 540B performance with substantially less computation, equivalent to about 4.4 million TPUv4 hours.The savings rate is measured against continuing standard causal-language-model training and varies with training stage.Source: §4.1 · pp. 6–7, Figure 2 · p. 6
• Savings depend on training stage: they are higher at middle stages than near the start or convergence for the 8B model.For 540B, the savings rate may continue increasing beyond 2.35x because PaLM was not trained to convergence.Source: §4.1 · pp. 6–7

Breakdown on individual tasks

• U-PaLM 540B outperforms PaLM 540B on 21 of 26 zero-shot and one-shot tasks and achieves state-of-the-art performance on those tasks.Figure 3 reports the individual task scores for models trained for 780B tokens.Source: §4.1 · p. 7, Figure 3 · p. 7

4.2 BigBench Emergent Suite

• BBES targets tasks where PaLM remains nearly flat at 8B and 62B before improving at 540B, plus tasks that remain difficult for PaLM 540B.The suite uses default BIG-Bench prompting rather than chain-of-thought prompting, so it is not entirely comparable to BBH.Source: §4.2 · p. 8

4.2.1 BIG-Bench results

• U-PaLM outperforms PaLM on 19 of 21 BBES tasks at 540B, with a +5.4% relative quality gain on the aggregated average.Gains include navigate from 55.3% to 67.0% and snarks from 69.1% to 86.1%.Source: Table 1 · p. 7, §4.2.1 · p. 8
• UL2R shifts emergence earlier: several tasks begin improving at 62B rather than 540B, and U-PaLM 8B sometimes exceeds PaLM 62B.Examples include crass_ai, vitaminc, identify_odd_metaphors, snarks, and understanding_fables.Source: §4.2.1 · p. 8

4.2.2 Analyzing individual task performance on BIG-Bench

• U-PaLM 8B outperforms PaLM 540B on spatial or visual reasoning tasks such as navigate and geometric_shapes.The authors associate this advantage with bidirectional attention and additional PrefixLM training, which provide a suitable inductive bias.Source: §4.2.2 · p. 8
• Knowledge-task gains are generally milder without new data, while multi-step and analogical reasoning tasks show modest 540B improvements with scaling patterns similar to PaLM.U-PaLM underperforms PaLM on the two identified tasks logical_sequence and english_proverbs.Source: §4.2.2 · pp. 9–10
• U-PaLM shows earlier emergence on context-reasoning tasks such as vitaminc_fact_verification and understanding_fables, although final 540B performance is similar.The authors postulate that PrefixLM helps by enabling bidirectional reasoning over context.Source: §4.2.2 · pp. 9–10

4.3.1 Commonsense Reasoning

• U-PaLM 540B outperforms PaLM 540B on all four tasks, with an average (+1.4%) relative improvement, and attains the best performance across all models.Source: §4.3.1 · p. 10

4.3.2 Question Answering and Reading Comprehension

• U-PaLM 540B improves few-shot closed-book question answering and reading comprehension by 2.3% on average, reaching 11.4% on natural questions.Source: §4.3.2 · p. 11

4.3.3 Reasoning and Chain-of-thought Experiments

• U-PaLM 540B outperforms PaLM 540B and Minerva 540B on chain-of-thought reasoning benchmarks, with especially strong gains on GSM8K and BBH.Source: §4.3.3 · p. 11

4.3.4 Multilingual Few-shot Reasoning and Question Answering Tasks

• Table 5 reports that U-PaLM outperforms PaLM by a considerable margin: 3.2% on TydiQA and 8.7% on MGSM.Source: §4.3.4 · p. 12

4.3.5 Massively Multi-Task Language Understanding

• U-PaLM outperforms PaLM by a 2.0% relative gain on the 5-shot MMLU test set.Source: §4.3.5 · p. 12

• U-PaLM strongly improves fine-tuning quality, especially at 8B scale, while gains at 62B remain modest.The experiments cover SuperGLUE and TydiQA at 8B and 62B scales.Source: §4.4 · p. 12, Table 7 · p. 12

5.1 Infilling Ability

• UL2R enables U-PaLM to infill one or multiple blanks, adding a second prompting approach alongside traditional continuation.Source: §5.1 · p. 13, §5 · p. 13
• U-PaLM correctly solves an infilling example that both PaLM and traditional U-PaLM prompting fail to answer.Source: §5.1 · p. 13, Figure 5 · p. 13

5.2 Leveraging Specific Pretraining Modes

• UL2 mode tokens expose distinct pretraining objectives, allowing inference-time control without changing the learned model or inference algorithm.Source: §5.2 · p. 14
• In a cross-lingual question-answering example, only U-PaLM queried in [S2S] mode returns the requested Vietnamese answer.Source: §5.2 · p. 14, Figure 6 · p. 14

5.3 Improved Diversity for Open-ended Generation

• Combining mode tokens and infilling in different ways increases output diversity without alternative decoding techniques such as sampling.Source: §5.3 · p. 15
• With greedy decoding, U-PaLM’s prompting variants produce pleasantly varying haikus for the same structured open-ended prompt.Source: §5.3 · p. 15, Figure 7 · p. 15

• With approximately 0.1% additional compute, U-PaLM improves PaLM’s scaling and outperforms it across English, reasoning, multilingual, MMLU, and BIG-Bench tasks.At 540B scale, UL2R enables savings of 4.4 million TPUv4 hours.Source: §6 · p. 15
• Whether UL2R works well with multi-task fine-tuning methods such as FLAN remains an open question.Source: §6 · p. 15

7.1 Details of Scaling Curves for Few-shot Experiments

• The 26-task aggregate averages 21 zero-shot rank-classification and 5 one-shot generative tasks, using accuracy or exact match except MultiRC’s F1a.These scores correspond to the 540B scaling plot.Source: §7.1 · p. 20
• Table 8 reports PaLM-versus-U-PaLM results at different FLOPs and token counts for the 540B model.Source: Table 8 · p. 20

7.2 Details of Vocab and Sentinel Tokens

• For infilling, U-PaLM uses sentinel tokens by repurposing PaLM’s last 100 subwords rather than adding new vocabulary embeddings.The setup also uses end-of-sequence symbols during training.Source: §7.2 · p. 20

7.3.1 What about training from scratch?

• UL2R can be viewed as a schedule that shifts from causal language modeling to the UL2 mixture, a curriculum toward infilling and bidirectional receptive fields, or a practical reuse strategy for existing models.The taxonomy of pretraining tasks remains an open question, while the approach is presented as useful for improving existing causal language models with little compute.Source: §7.3.1 · p. 21

7.3.2 What about supervised finetuning on many new tasks like FLAN or T0?

• The appendix states that UL2R and supervised finetuning methods such as FLAN are complementary.Source: §7.3.2 · p. 21