Source-linked AI summary
Inverse Scaling: When Bigger Isn't Better
Ian R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, Andrew Gritsevskiy, Daniel Wurgaft, Derik Kauffman, Gabriel Recchia, Jiacheng Liu, Joe Cavanagh, Max Weiss, Sicong Huang, The Floating Droid, Tom Tseng, Tomasz Korbak, Xudong Shen, Yuhui Zhang, Zhengping Zhou, Najoung Kim, Samuel R. Bowman, Ethan Perez
TL;DR
The paper asks whether increasing language-model scale reliably improves task performance, given that next-token training may encourage undesirable behaviors. It uses a public contest and cross-model evaluation to collect and analyze inverse-scaling tasks. The authors find 11 cases, identify four potential causes, and show that scaling trends can reverse, limiting scale alone as a path to progress.
Problem
Scaling laws predict improved language-model loss with scale, but task performance may worsen when the training objective encourages undesirable generalization.
Method
The authors ran a public contest and evaluated zero-shot and few-shot submissions across model series from OpenAI, Anthropic, and DeepMind over more than 5 orders of magnitude in training FLOPs.
Results
The study discovered 11 inverse-scaling cases and identified four potential causes: strong priors, unwanted imitation, distractor tasks, and spurious few-shot correlations.
Takeaways & Limitations
U-shaped and inverted-U trends show that smaller-model scaling directions may not predict larger-model behavior, so increased scale alone may not produce progress on every task.
Takeaways & Limitations
Many inverse-scaling tasks did not sufficiently demonstrate real-world implications of failure, and no submission met the grand or second prize criteria.
Abstract
from arXiv · showhide
Work on scaling laws has found that large language models (LMs) show predictable improvements to overall loss with increased scale (model size, training data, and compute). Here, we present evidence for the claim that LMs may show inverse scaling, or worse task performance with increased scale, e.g., due to flaws in the training objective and data. We present empirical evidence of inverse scaling on 11 datasets collected by running a public contest, the Inverse Scaling Prize, with a substantial prize pool. Through analysis of the datasets, along with other examples found in the literature, we identify four potential causes of inverse scaling: (i) preference to repeat memorized sequences over following in-context instructions, (ii) imitation of undesirable patterns in the training data, (iii) tasks containing an easy distractor task which LMs could focus on, rather than the harder real task, and (iv) correct but misleading few-shot demonstrations of the task. We release the winning datasets at https://inversescaling.com/data to allow for further investigation of inverse scaling. Our tasks have helped drive the discovery of U-shaped and inverted-U scaling trends, where an initial trend reverses, suggesting that scaling trends are less reliable at predicting the behavior of larger-scale models than previously understood. Overall, our results suggest that there are tasks for which increased model scale alone may not lead to progress, and that more careful thought needs to go into the data and objectives for training language models.
1 Introduction
This paper challenges the reliability of standard scaling expectations by documenting tasks where larger language models perform worse, and by identifying four potential causes of inverse scaling. A public contest produced 11 robust cases across model families, while the resulting trends also revealed reversals such as U-shaped and inverted-U scaling.
- Motivation: Inverse scaling occurs when task performance worsens as language-model loss on the original training objective improves.The paper argues that next-token prediction may actively incentivize behaviors that harm performance on tasks not explicitly trained for.
- Empirical approach: The public Inverse Scaling Prize evaluated zero-shot and few-shot submissions across OpenAI, Anthropic, and DeepMind model series spanning over 5 orders of magnitude in training FLOPs.The contest attracted 99 submissions and awarded prizes to 11 submissions that robustly demonstrated inverse scaling, including on held-out model series.
- Potential causes: Strong prior causes models to repeat memorized sequences instead of following in-context instructions, as in Resisting Correction, Memo Trap, Redefine, and Prompt Injection.These tasks test whether models can override learned sequence preferences when prompts explicitly require alternative behavior.
- Potential causes: Unwanted imitation reflects imitation of undesirable training-data patterns, while distractor tasks cause models to perform an easier task instead of the intended harder task.The distractor examples include Pattern Match Suppression, NeQA, Sig Figs, and Into the Unknown; Modus Tollens is identified as a task fitting unwanted imitation.
- Potential causes: Spurious few-shot examples are correctly labeled demonstrations that contain misleading correlations, causing models to infer the wrong task.Hindsight Neglect and Repetitive Algebra are prize-winning examples of this category.
- Implications: The study discovered 11 inverse-scaling cases and showed that scaling trends can reverse, producing U-shaped or inverted-U behavior.Inverse scaling is not the only possible reversal: some tasks improve at larger scales after initially declining, while Prompt Injection worsens after initially improving.
2 The Inverse Scaling Prize
The Inverse Scaling Prize collected text-completion datasets and evaluated them across pretrained and fine-tuned language models, including held-out model series. Submissions used zero-shot or few-shot prompts and classification-loss or sequence-probability metrics.
- Contest design: The contest collected participant-submitted input-output datasets formatted as text-completion tasks for evaluating inverse scaling.Participants were asked to provide examples and an evaluation metric.
- Model evaluation: Evaluations included pretrained autoregressive models and private Anthropic and DeepMind model series to test robustness beyond public model families.The private models were used to check whether findings generalized beyond quirks of specific model series.
- Model evaluation: The study included pretrained models, FeedME models, and Anthropic Context Distilled models with different fine-tuning procedures.These procedures included human-rated LM-generated samples and matching the outputs of a helpful, harmless, and honest chatbot.
- Model evaluation: Additional results covered GPT-4, GPT-4 RLHF, and PaLM after the contest period.PaLM results were taken from prior work, while GPT-4 results were received through private correspondence.
- Submission format and metrics: Submissions required at least 300 examples per task, with around 1000 recommended for clearer scaling trends.The thresholds were based on observations from standard scaling on LAMBADA using GPT-3.
- Submission format and metrics: Winning submissions used classification loss or loss on a sequence at the end of a prompt as evaluation metrics.Classification loss normalizes probabilities over the provided options, while sequence probability evaluates the correct completion.
3 Inverse Scaling Prize Tasks
The Inverse Scaling Prize identified robust cases where larger language models performed worse on tasks involving instruction conflicts, distractors, or misleading demonstrations. Scaling behavior varied across model series, including inverse, U-shaped, and inverted-U trends.
- Prize Overview: 11 third-prize datasets were released after two contest rounds, although no submission met the grand- or second-prize criteria.The contest received 50 submissions in Round 1 and 49 in Round 2.
- Strong Prior: Larger models increasingly failed Resisting Correction, Redefine, and Memo Trap by following memorized or conventional patterns instead of overriding them with in-context instructions.Resisting Correction involved repeating atypical text verbatim; Redefine required using altered symbol meanings; Memo Trap required varying a familiar phrase.
- Strong Prior: Prompt Injection showed inverted-U scaling: loss initially decreased, then worsened after around 10^21 FLOPs for all model series.The figure negates loss so scaling direction is comparable to accuracy, and the later reversal is opposite to the preferred U-shaped pattern.
- Distractor Task: Distractor tasks produced inverse scaling when larger models handled an easier subtask while missing the intended task, as in negated questions and pattern or redundancy judgments.In NeQA, larger models handled the question format but often failed to apply negation; related tasks also fell below random accuracy at larger scales.
4 Related Work
Related work documents inverse scaling across evaluation settings, including memorization, undesirable imitation, prompt sensitivity, and symbolic generalization. These findings connect prior observations to the paper’s proposed causes.
- Language Model Evaluation Suites: GLUE, SuperGLUE, MMLU, BIG-Bench, and HELM provide broad LM evaluations, but MMLU emphasizes tasks expected to improve with scale.BIG-Bench includes some inverse- and U-shaped-scaling tasks without actively soliciting them or analyzing their causes.
- Unwanted Imitation: Larger LMs perform worse on TruthfulQA, illustrating unwanted imitation of misinformation present in training data.The cited work treats repeating common misconceptions as an inverse-scaling failure.
- Other Inverse-Scaling Findings: Inverse scaling also appears in code robustness, prompt-format sensitivity, lexical generalization, and inference with intervening distractors.These studies report worsening relative correctness, greater prompt-format variance, reduced compositional generalization, or poorer inference with scale.
- Unwanted Imitation: Larger GPT-3 models produce more toxic text, another example of unwanted imitation from internet training data.The largest models in the cited evaluation had higher toxicity scores.
- Other Inverse-Scaling Findings: Symbolic generalization can worsen with scale because larger models may rely more heavily on pretraining priors on out-of-distribution examples.The paper presents this as a possible explanation rather than an established causal account.
5 Discussion
Discussion shows that scaling trends can reverse in either direction and vary across tasks, model families, training progress, and few-shot conditions. The authors emphasize that future behavior and reversal points remain difficult to predict.
- Open Questions: The future direction of Unwanted Imitation and Strong Prior trends is difficult to predict, because pretraining information may become more strongly represented as optimization continues.The paper also identifies prompt-injection susceptibility as a class that may continue showing inverse scaling, while noting possible mitigation through context-part separation.
- Trend Reversals: U-shaped trends can reverse inverse scaling, but Prompt Injection exhibits inverted-U scaling, reversing from improvement to degradation at larger scales.Resisting Correction, Memo Trap, and NeQA are among tasks with U-shaped behavior, although larger PaLM models can remain below smaller ones.
- Few-Shot Effects: Additional few-shot examples improved trends for Pattern Match Suppression, Prompt Injection, Repetitive Algebra, and Modus Tollens, often converting inverse scaling into U-shaped or regular scaling.Few-shot examples did not improve all trends; Hindsight Neglect, Redefine, and Memo Trap retained inverse or inverted-U behavior in specified model families.
- Few-Shot Effects: Few-shot examples were actively harmful for some settings, including Sig Figs across models and NeQA on larger Anthropic LMs.Resisting Correction also worsened at the two largest numbers of demonstrations on Anthropic LM, Chinchilla, and Gopher.
- Scaling Through Training: Training-time scaling varied widely: Hindsight Neglect, NeQA, and Pattern Match Suppression showed inverse scaling, while Redefine, Memo Trap, and Repetitive Algebra more often showed regular scaling at smaller sizes.Into the Unknown shifted from mostly flat or regular behavior to inverse scaling at the largest scales.
- Scaling Through Training: Many tasks showed inverted-U behavior through training at large scales, including Redefine, Memo Trap, Repetitive Algebra, and Prompt Injection.Resisting Correction and Modus Tollens also showed complex reversals, including multiple flips and, for the largest model, U-shaped followed by inverted-U behavior.
6 Conclusion
The conclusion presents inverse scaling as a documented phenomenon supported by 11 contest-winning tasks and organizes these failures into four potential causes. It positions the released tasks as a starting point for further study of scaling reversals and emergent behavior.
- Findings: The paper reports 11 prize-winning tasks demonstrating inverse scaling in language models.These cases were collected through the public Inverse Scaling Prize.
- Proposed Causes: Four potential causes are strong prior, unwanted imitation, distractor task, and spurious few-shot.They respectively involve relying on memorized information, imitating undesirable training patterns, performing an easier task, and following misleading few-shot correlations.
- Future Investigation: The released Inverse Scaling Prize tasks and accompanying takeaways are proposed as a useful starting point for investigating inverse scaling, emergent behaviors, and trend reversals.The conclusion frames these topics as requiring further discovery.
A Task Authors
This appendix credits the authors of selected contest tasks and summarizes the evaluated model series. The overview records few-shot coverage and Anthropic training-token checkpoints.
- Task Authors: Resisting Correction and Prompt Injection were submitted by Joe Cavanagh, Andrew Gritsevskiy, and Derik Kauffman, with Aaron Kirtland also credited for Prompt Injection.Cavendish Labs is listed for the first three contributors, and Brown University for Kirtland.
- Task Authors: Hindsight Neglect was submitted by The Floating Droid, listed as anonymous.The appendix provides no additional institutional attribution for this task.
- Model Series Overview: The model-series overview marks few-shot evaluation by submission round and evaluates Anthropic LM sizes through training checkpoints from 0.03B to 400B tokens.The table also notes that OpenAI model names follow the text-*-001 API naming pattern.
C FLOP Computation
Training FLOP estimates combine reported values with 6ND-based estimates, while Figure 8 relates parameter count to pretraining FLOPs and training-corpus size. Several model-series estimates rely on stated token-count assumptions or external sources.
- FLOP estimation: Training FLOPs are estimated with the 6ND approximation when unavailable, where N is model parameters and D is training tokens.Fine-tuning FLOPs are excluded for Anthropic Context Distilled, GPT-3 FeedME, and Anthropic RLHF because they are difficult to estimate.
- Source-specific estimates: Anthropic FLOP counts use 6ND estimates that closely match values from Askell et al. (2021).The paper also uses externally sourced FLOP counts for Gopher, Chinchilla, GPT-3, and GPT-3 FeedME.
- FLOP estimation: Figure 8 plots parameter count against pretraining FLOPs, with more training tokens shifting model series to the right.Anthropic checkpoints span 0.03B to 274.88B training tokens, with a final checkpoint at 400B tokens.
- Source-specific estimates: OPT FLOPs use 6ND with 180B pretraining tokens, while GPT-2 FLOPs use 6ND with an estimated 28B-token WebText corpus.The GPT-2 token estimate assumes WebText has GPT-3’s tokens-per-byte ratio.
D Inverse Scaling Prize Details
The appendix documents how the Inverse Scaling Prize was reviewed and how prize decisions were informed. Table 4 presents the submission-review criteria.
- Prize documentation: The appendix provides additional details about the contest rubric, unused metrics, and submission and review processes.These details supplement the main description of the Inverse Scaling Prize.
- Prize documentation: Table 4 presents the criteria used to review submissions and inform prize decisions.The table is labeled “Inverse scaling evaluation criteria.”
D.2 Other Metric Details
The appendix describes alternative log-odds metrics and their risks, including spurious inverse scaling caused by comparing unrelated tasks. These metrics were not used by prize winners.
- Log-odds metrics: The signed logodds metric measures how changing a prompt alters the relative probabilities of two possible responses.The example concerns undesirable prompt effects such as bias against specific demographics.
- Metric limitations: Comparing unrelated tasks can create spurious inverse scaling when both scale normally but one improves faster than the other.The resulting performance difference can grow with scale even without inverse scaling in either task.
- Metric limitations: Winners did not use logodds metrics because participants were required to justify that effects were not caused by unrelated prompt differences.The requested justification included control experiments, such as tests for prompt-length effects.
- Log-odds metrics: Absolute logodds is the absolute value of signed logodds and is used when a prompt difference is expected but its direction is unknown.It carries the same potential for spurious inverse scaling, so strong justification was required.
- Other evaluation options: Round 2 allowed submissions demonstrating standard scaling on an incorrect answer, but no prize-winning task used this option.Eligible formats included classification with one incorrect answer or sequence probability with one inappropriate completion.
D.3 Submission details
Submissions documented the task, its expected inverse-scaling rationale, data generation, annotation requirements, and empirical scaling behavior. Optional components tested persistence under few-shot conditioning, instruction tuning, or fine-tuning.
- Submission requirements: Participants described the task behavior, intended good behavior, and why poor performance matters for safety or understanding LM failures.They also explained why they expected inverse scaling.
- Submission requirements: Submissions reported how data were generated, including the resources, templates, or programs used.They also specified the expertise required for human annotators to verify labels.
- Submission requirements: Participants plotted GPT-3 task performance across model sizes using a provided Google Colab notebook.This supplied a standardized way to display scaling behavior.
- Optional analyses: Optional analyses tested whether inverse scaling persisted after few-shot examples, instruction-following training, or fine-tuning.For few-shot tests, submitters could report how many examples were needed to eliminate inverse scaling.
D.4 Contestant Support
Contestants received evaluation resources, iterative feedback, collaboration support, and assistance with dataset creation. During Round 2, organizers also published speculative sources of inverse scaling to inspire submissions.
- Contestant Support: Contestants could evaluate tasks on GPT-3, GPT-2, and OPT model series using provided Google Colab notebooks.OPT access extended up to 13B with Colab Pro+, while GPT-3 queries required OpenAI API credits.
- Contestant Support: Round 1 participants received reviewer feedback and private-model evaluation results before revising submissions for Round 2.Revised submissions could supersede corresponding Round 1 entries.
- Contestant Support: A Slack workspace enabled contestants to ask organizers questions, find collaborators, and discuss relevant literature and ideas.Organizers also used it for competition announcements and literature sharing.
- Contestant Support: Surge AI supported dataset creation by offering bespoke assistance and $500 annotation credits to the first 10 participants who requested them.The support specifically targeted contestants generating example datasets.
- Contestant Support: Organizers published a list of speculative potential sources of inverse scaling during Round 2 to provide inspiration for new submissions.The list combined original ideas with public suggestions from Slack and Twitter.
D.5 Submission Assessment
The competition used expert review and crowd-worker validation to assess submissions, ensuring tasks were judged relevant and had answers humans considered correct.
- Submission Assessment: A reviewer panel with relevant machine learning and NLP experience judged the competition submissions.The panel assessed tasks using six rubric dimensions, including inverse scaling strength and dangerous behavior.
- Submission Assessment: Surge AI crowd workers validated all tasks to ensure they were solvable in principle and had answers humans would judge as correct.This validation complemented the expert review process.
E.1 Modus Tollens Corrected Plot
The corrected Modus Tollens analysis removes a small set of grammatically flawed examples without materially changing the results. For Prompt Injection, average probability is plotted because the sequence-probability metric lacks an accuracy score.
- E.1 Modus Tollens Corrected Plot: Less than 10% of Modus Tollens examples contained grammatical errors discovered after paper preparation.Removing them had very little effect on the results, so the analysis remains unchanged.
- E.1 Modus Tollens Corrected Plot: Prompt Injection uses sequence probability rather than accuracy, so Figure 10 plots average probability to make the results easier to interpret.The figure presents scaling behavior for probability on the Prompt Injection task.
- E.1 Modus Tollens Corrected Plot: Figure 9 shows scaling behavior for a Modus Tollens version with grammatically erroneous examples removed.The corrected plot corresponds to the post-preparation error discovery and unchanged analysis.