PaperlayerView exact revision on arXiv

Source-linked AI summary

Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve

R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, Thomas L. Griffiths

arXiv:2309.13638v1cs.CLcs.AI

TL;DR

LLMs are flexible but difficult to understand holistically, motivating an analysis grounded in the problem they were trained to solve. Using a teleological approach, the paper tests probability-based predictions across tasks and finds robust sensitivity to task, output, and input probabilities. It concludes that LLMs should be evaluated as systems shaped by their own training pressures, not simply as humans.

  • Problem

    LLMs can be applied to many tasks, but human-centric evaluations provide an incomplete basis for understanding their properties and limitations.

  • Method

    The paper uses a teleological approach, deriving hypotheses from next-word prediction over Internet text and testing them with adversarial low-probability tasks.

  • Results

    Across seven common–rare task pairs, models performed substantially better on common variants, while output probability had a larger effect than input probability in the investigated tasks.

  • Takeaways & Limitations

    LLMs should be understood and evaluated as statistical next-word prediction systems shaped by their training pressures, especially when used in low-probability situations.

  • Takeaways & Limitations

    The teleological perspective can be incomplete or misleading, so its predictions must be tested empirically rather than treated as definitive.

Abstract

from arXiv · show

The widespread adoption of large language models (LLMs) makes it important to recognize their strengths and limitations. We argue that in order to develop a holistic understanding of these systems we need to consider the problem that they were trained to solve: next-word prediction over Internet text. By recognizing the pressures that this task exerts we can make predictions about the strategies that LLMs will adopt, allowing us to reason about when they will succeed or fail. This approach - which we call the teleological approach - leads us to identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input. We predict that LLMs will achieve higher accuracy when these probabilities are high than when they are low - even in deterministic settings where probability should not matter. To test our predictions, we evaluate two LLMs (GPT-3.5 and GPT-4) on eleven tasks, and we find robust evidence that LLMs are influenced by probability in the ways that we have hypothesized. In many cases, the experiments reveal surprising failure modes. For instance, GPT-4's accuracy at decoding a simple cipher is 51% when the output is a high-probability word sequence but only 13% when it is low-probability. These results show that AI practitioners should be careful about using LLMs in low-probability situations. More broadly, we conclude that we should not evaluate LLMs as if they are humans but should instead treat them as a distinct type of system - one that has been shaped by its own particular set of pressures.

1 Introduction

The paper argues that LLMs should be understood through their training objective—statistical next-word prediction over Internet text—rather than evaluated only with human-centric tests. It develops probability-based predictions, tests them experimentally, and finds robust effects on LLM accuracy.

  • Motivation: Human-centric evaluations may miss properties unique to LLMs because they emphasize abilities shared with humans.The paper therefore advocates analyzing LLMs on their own terms.
  • Motivation: LLMs are trained for autoregressive next-word prediction over Internet text, so this objective may influence performance even on apparently different tasks.The paper frames an LLM solving mathematics as a statistical next-word prediction system being used for that purpose.
  • Predictions: The paper predicts that deterministic tasks will be easier when their task, target output, or input has higher probability.These predictions concern task probability, output probability, and input probability.
  • Evidence: Across seven pairs, models performed substantially better on common task variants than rare variants.The experiments deliberately used low-probability situations, such as decoding simple ciphers, to expose sources of difficulty.
  • Evidence: The findings provide robust evidence that task or example probability substantially changes LLM accuracy in the predicted directions.The paper also uses additional experiments and a literature review to connect these effects with broader LLM behaviors.

2 A teleological approach to understanding LLMs

The teleological approach explains LLM behavior by first characterizing the problem they solve and then deriving hypotheses about their capacities and biases. For LLMs, that problem is next-word prediction over Internet text using neural-network mechanisms, creating sensitivity to word-sequence statistics and probability.

  • Teleological approach: The teleological approach characterizes a system’s goal and uses that characterization to generate hypotheses about its capacities and biases.The approach is presented as a lens rather than a complete explanation of every LLM property.
  • What problem do LLMs solve?: For standard LLMs, the training task is next-word prediction, the distribution is Internet text, and the architecture is a Transformer neural network.Together, these factors define the problem the model must solve.
  • Scope: Instruction tuning is an additional training phase that this paper does not analyze, leaving it for future work.The paper’s teleological analysis focuses only on next-word prediction.
  • Hypothesized embers of autoregression: Because LLMs are statistical systems trained on Internet text, they are predicted to be sensitive to word-sequence statistics.The paper derives predictions about task, answer, and input probability from this training setup.
  • Hypothesized embers of autoregression: LLMs are predicted to perform worse on rare tasks and to remain sensitive to answer and input probability even on deterministic tasks.Input probability is expected to matter less than output probability.
  • Experimental framing: The paper studies these predictions mainly with basic prompting, while noting that prompting technique can substantially affect LLM accuracy.This sensitivity is treated as consistent with the models’ autoregressive roots.

3 Motivating our predictions

The authors derive predictions about LLM difficulty by analyzing autoregression and test whether task, output, and input probabilities affect deterministic cipher decoding. Results support stronger effects for task and output probability than for input probability.

  • Task probability: Task probability is the expected frequency with which a task appears in sampled text, and more training examples should improve performance.The authors therefore predict better accuracy on frequent than rare tasks.
  • Task probability: Both GPT-3.5 and GPT-4 performed much better on frequent rot-13 decoding than equally complex, rare rot-2 decoding.This supports sensitivity to task probability.
  • Output probability: LLMs are predicted to favor high-probability outputs even in deterministic tasks, where probability should be irrelevant.The formal framing treats generation as maximizing P(output|input), which can favor high-probability outputs through P(output).
  • Output probability: Both models became more accurate as the log probability of the correct rot-13 output increased.The result supports influence from the unconditional probability of candidate outputs.
  • Input probability: GPT-4 was more accurate for high-probability inputs, whereas GPT-3.5 showed no significant input-probability effect.Input probability is expected to matter mainly when task performance depends on experience with the specific input.
  • Input versus output probability: Output probability is expected to influence more situations than input probability because uncertainty commonly requires choosing among possible outputs.The authors characterize input probability as occasional and output probability as broadly influential.

4 Overview of experiments

The experiments target low-probability settings and use mostly deterministic tasks to expose sources of LLM errors. GPT-3.5 and GPT-4 are evaluated with controlled datasets, while the authors caution that results may vary across model versions and interfaces.

  • Tasks: The task suite was chosen to create non-negligible error rates by pushing models into low-probability situations.Most tasks also have simple deterministic algorithms invariant to task variations.
  • Controls: Both models scored close to 100% on spelling 1,000 single-token words, supporting the use of character-level manipulation tasks.The reported scores were 99.8% for GPT-3.5 and 99.9% for GPT-4.
  • Sentences: The experiments include high-, medium-, and low-probability sentence sets for testing performance on specific examples.An example ranges from natural wording to a scrambled low-probability sentence.
  • Models: GPT-3.5 and GPT-4 are large-scale Transformer language models evaluated through the OpenAI API at temperature 0.0.The study used time-stamped versions gpt-3.5-turbo-0613 and gpt-4-0613.
  • Tasks: The common-versus-rare task comparisons cover shift ciphers, Pig Latin, acronyms, linear functions, and sorting.Examples contrast rot-13 with rot-2 and alphabetical sorting with reverse alphabetical sorting.
  • Scope: Results may not transfer identically to other language models, newer versions, or models accessed through a chat interface rather than the API.The chat interface was not systematically investigated because the datasets made that evaluation prohibitive.

5 Sensitivity to task probability

Across seven task pairs, GPT-3.5 and GPT-4 performed substantially better on common variants than equally simple rare variants, supporting sensitivity to task probability rather than inherent complexity.

  • Shift ciphers: GPT-4 scored 0.50 or above on shift levels 1, 3, and 13, but below 0.03 on every other shift.The three highest-performing shifts were also the three most frequent in the C4 corpus.
  • Pig Latin: GPT-4’s Pig Latin encoding accuracy was 0.39 versus 0.13 for an equally complex but unattested system.Among four minimally differing variants found in natural corpora, accuracy ranged from 0.23 to 0.42 and increased with corpus frequency.
  • Acronyms: GPT-4 achieved 0.76 accuracy joining first letters of words versus 0.03 joining second letters, despite matched outputs and input vocabularies.The difference was statistically significant for both models.
  • Linear functions: GPT-4 scored 0.33 on the common Celsius-to-Fahrenheit function versus 0.00 on a similarly structured rare function.The common function is frequently illustrated in natural text, whereas the rare function has no special significance.
  • Ordering: GPT-4 scored 0.80 for alphabetical word sorting versus 0.32 for reverse alphabetical sorting, but numeric ascending and descending scores were 0.82 and 0.80.The larger word-ordering gap coincided with a much larger frequency difference than for numeric orderings.
  • Overall pattern: Across all seven task pairs, both models performed substantially better on the common task variant than the rare one.The comparisons covered shift ciphers, Pig Latin, acronyms, linear functions, and ordering tasks.

6 Sensitivity to output probability

Across deterministic tasks, LLM accuracy was consistently higher when the correct output was more probable. The pattern appeared across ciphers, reversal, Pig Latin, article swapping, acronyms, and counting.

  • Shift ciphers: 0.51 versus 0.13: GPT-4’s rot-13 decoding accuracy fell when the correct output sentence changed from high- to low-probability.Errors often replaced low-probability answers with incorrect outputs having higher probability.
  • Pig Latin: 0.83 versus 0.28: GPT-4 translated Pig Latin more accurately when the English output sentence was high-probability than low-probability.
  • Article swapping: 0.83 versus 0.02: GPT-4 swapped articles more accurately for high-probability outputs than for random word sequences.The authors could not confidently separate input-probability from output-probability effects, although expanded experiments implicated output probability most strongly.
  • Acronyms: 0.76 versus 0.67: GPT-4 formed acronyms more accurately when the resulting word was common than when it was rare.GPT-3.5 likewise scored 0.39 in the highest-probability condition versus 0.18 in the lowest-probability condition.
  • Counting: GPT-4 counted more accurately when the correct count was a commonly used number than when it was a rarer number.Accuracy tracked the log frequency of the target number more smoothly than its magnitude.

7 Sensitivity to input probability

The experiments test whether input probability affects LLM accuracy across several tasks and find that its influence is generally smaller and less consistent than output probability, except in birthday retrieval.

  • 7.1 Shift ciphers, reversal, Pig Latin, and acronyms: 21% versus 11%: GPT-4’s rot-13 encoding accuracy was higher for high-probability input sentences than low-probability ones.For reversal, Pig Latin, and acronyms, GPT-4 showed no statistically significant input-probability effect.
  • 7.1 Shift ciphers, reversal, Pig Latin, and acronyms: Input probability significantly affected GPT-4 shift-cipher performance and GPT-3.5 acronym formation, while edit distance detected effects for both models on shift ciphers.The accuracy effects were p < 0.05 for GPT-4 shift-cipher processing and p < 10^-5 for GPT-3.5 acronym formation; edit-distance regressions found p < 0.01 for both models.
  • 7.2 Counting: Counting results were inconsistent: input probability sometimes helped and sometimes hurt, with very small effects except for GPT-3.5 character counting.The expected-direction effects occurred for GPT-3.5 character counting and GPT-4 word counting; opposite-direction effects occurred for GPT-3.5 word counting and GPT-4 character counting.
  • 7.3 Asymmetry between input probability and output probability: Output probability influenced accuracy more pervasively than input probability across the experiments.All output-probability experiments produced significant effects, often with large effect sizes, whereas only a few input-probability conditions did.
  • 7.3 Asymmetry between input probability and output probability: 0.51 versus 0.21: GPT-4 decoded high-probability rot-13 outputs more accurately than it encoded high-probability inputs.GPT-3.5 showed the same asymmetry, scoring 0.21 for decoding and 0.02 for encoding.
  • 7.4 Birthdays: 0.99 versus 0.23: GPT-4’s birthday accuracy was much higher for people frequently mentioned online than for people rarely mentioned.This was the largest input-probability effect observed in these experiments.

8 Other phenomena

The paper extends its analysis beyond probability sensitivity to embodiment and wording. Models struggle with keyboard-based symbol manipulation and perform differently when equivalent tasks are framed in familiar linguistic contexts or formats.

  • 8.1 Keyboard cipher: 0%: GPT-3.5 and GPT-4 both failed the keyboard cipher under the basic prompt.The task required shifting each letter to the right on a QWERTY keyboard, despite being straightforward for a human with a keyboard.
  • 8.1 Keyboard cipher: Providing a keyboard layout reduced GPT-4’s edit distance but still left about 8 mistakes per sentence.Providing explicit letter correspondences improved GPT-4 further, but its accuracy remained far below 100%.
  • 8.1 Keyboard cipher: Text-only models can acquire some keyboard-structure information, but the paper argues that such embodied information is difficult rather than impossible to learn from text.Even without a keyboard, models performed better than random guessing based only on prompt examples.
  • 8.2 Wording: GPT-4 performed significantly better on f(x) = (9/5)x + 32 when the prompt identified it as Celsius-Fahrenheit conversion.Both models improved in the primed condition relative to the basic and priming-control conditions, with p < 0.01 for each model.
  • 8.2.1 Linear function: The wording effect may reflect either easier access to a systematic ability or reliance on memorized Celsius-Fahrenheit pairs.The experiments used rare input values to reduce the chance that performance reflected direct retrieval of memorized equivalences.
  • 8.2.2 Multiplication: 0.46, 0.39, 0.35, and 0.17: GPT-4’s multiplication accuracy varied across digits, lowercase words, all caps, and alternating capitals.Many format comparisons were statistically significant, showing sensitivity to how numerical language is written.

9 Discussion

The discussion presents the teleological approach as a way to identify model-specific failure modes and unifying principles, while cautioning that human comparisons require fair, task-matched evaluation.

  • Teleological approach: Teleological analysis can reveal non-obvious deviations that desideratum-driven testing may miss.The authors argue that desideratum testing remains necessary but should be informed by this analysis.
  • Teleological approach: The approach offers unifying principles that support more general predictions than lists of model scores.It uses knowledge of model architecture and training objectives despite limited access to internal workings.
  • Teleological approach: The teleological approach starts from the pressures shaping a model to predict challenging scenarios without accessing the model itself.Here, that analysis led the authors to test low-probability spaces.
  • Human comparison: The proposed two-step perspective aims to identify both human-like and qualitatively non-human-like model properties before comparing models with humans.Direct human-designed tests can overestimate similarity and omit model-specific errors.
  • Findings and implications: LLMs achieve higher accuracy on common tasks and high-probability answers, even when tasks are equally complex or deterministic for humans.The discussion contrasts task probability and answer probability as distinct influences on performance.
  • Findings and implications: Advanced prompting can improve overall shift-cipher accuracy without eliminating probability-related difficulty patterns.The cited experiments also report formatting-related sensitivity in multiplication, while human comparisons remain incomplete without evaluating humans on the same tasks.

10 Related work

Related work situates the paper’s teleological perspective within cognitive science and research on training objectives, language processing, data effects, and autoregressive limitations.

  • Cognitive-science foundations: The teleological perspective draws on cognitive science’s emphasis on understanding systems through the problems they evolved or were trained to solve.Different systems with the same goal may converge on similar strategies despite different implementations.
  • Language processing: Because LLMs manipulate word sequences rather than directly manipulating ideas or structured world models, they can struggle with entity tracking and related tasks.The paper frames this as an indirect consequence of language-centered training and processing.
  • Autoregressive interaction: Autoregressive generation can produce errors when models cannot plan far ahead or revise text after producing it.The related work identifies one-word-at-a-time generation as a source of this limitation.
  • Training data: Frequent repetition in a training dataset can cause models to memorize sentences that are not especially notable outside that dataset.The Boat Race articles illustrate how corpus frequency determines sentence probability from the model’s perspective.
  • Training task: Next-word prediction gives LLMs broad task exposure because many tasks can be expressed as predicting subsequent words.The paper gives summarization as an example of a task represented through continuation prediction.
  • Training distribution: Internet text exposes models to many input types, reducing some brittleness on out-of-distribution examples.This benefit concerns the breadth of the training distribution rather than the next-word objective itself.
  • Model architecture: Neural-network architectures support similarity-based generalization, multiple constraint satisfaction, graceful degradation, and content-addressable memory.The cited work also emphasizes efficient learning from naturalistic data.

11 Conclusion

The conclusion contrasts broad claims about language models with their simpler identity as statistical next-word predictors. This framing highlights systematic difficulty on infrequent tasks and low-probability text.

  • Conclusion: Recent work characterizes language models as multitask learners, few-shot learners, general-purpose interfaces, reasoning systems, and knowledge graphs.These titles reflect the advanced properties the conclusion says should be recognized and explored.
  • Conclusion: Language models are also statistical next-word prediction systems whose behavior reflects that training objective.The conclusion presents this simpler characterization alongside broader descriptions of model capabilities.
  • Conclusion: LLMs have greater difficulty with infrequent tasks than frequent tasks, even when the tasks seem equally complex to humans.The conclusion also points to greater difficulty producing low-probability text.

Limitations

The authors qualify the teleological approach as incomplete and potentially misleading, and they limit the experiments through proprietary models and relatively simple tasks.

  • General limitations: The teleological perspective should be one of multiple approaches because it can be incomplete.The authors caution against treating it as the field’s only way to study LLMs.
  • General limitations: Teleologically derived predictions may be wrong when the analysis of a model’s goal is incorrect, so they require empirical testing.The paper explicitly states that such predictions should not be assumed correct.
  • Experimental scope: The tested models are proprietary, leaving precise architecture and training-data details unavailable.The authors nevertheless argue that available information was sufficient for their teleological analyses.
  • Experimental scope: The investigated tasks are relatively simple and have limited practical utility, leaving effects in more practical settings for future work.The authors selected simple tasks to reason clearly about the general principles driving performance.

A Dataset details

The datasets span cipher, acronym, article-swapping, and other tasks, with controlled probability, tokenization, formatting, and sentence-construction choices. Dataset statistics and probability bins support comparisons across input and output conditions.

  • Cipher datasets: High-probability cipher sentences were grammatical, nonviolent, nonsexual, and free of proper names except common names.
  • Probability manipulation: Medium-probability sentences replaced tokens with same-length alternatives between the top 90% and 95% probability-mass boundaries.
  • Probability manipulation: Low-probability sentences were created by shuffling internal words or replacing one word with a grammatical near-neighbor at Levenshtein distance at most 2.
  • Prompt construction: Prompts included examples clarifying capitalization, punctuation, apostrophes, consonant- versus vowel-initial words, and non-letter handling.
  • Cipher datasets: 100 sentences per condition were used for shift, reversal, and keyboard cipher datasets.
  • Acronym datasets: Acronym comparisons held output words, input vocabulary, example split points, and word probability bins under explicit controls.

C Evaluation details

Evaluation used exact or task-specific correctness criteria across GPT-3.5 and GPT-4, with regressions accounting for imperfect matching among probability-related predictors. Results generally showed significant effects of output probability, while input-probability effects varied by task and metric.

  • Evaluation procedure: Responses were manually normalized or inspected when task rules allowed extra wording, capitalization, spaces, punctuation, or alternative valid decryption outputs.
  • Statistical analysis: The analyses used logistic or linear regressions and z-scored continuous or integer predictors to address imperfect condition matching and scale differences.
  • Evaluation procedure: GPT-3.5 and GPT-4 were evaluated with correctness as a binary response variable unless otherwise stated.
  • Probability effects: Input probability significantly affected encoding accuracy for GPT-4 but not GPT-3.5, with p < 0.05 and p = 0.61 respectively.
  • Probability effects: Using Levenshtein edit distance instead of correctness, input probability significantly affected both GPT-3.5 and GPT-4, with p < 0.01 for each.
  • Probability effects: Output probability significantly predicted accuracy for both models, with p < 10^-4 for GPT-3.5 and p < 10^-7 for GPT-4.

D.5.2 Expanded experiment

The expanded experiment disentangled probability effects in article swapping, acronym, counting, and sorting tasks. Output probability was consistently significant, whereas input probability and task or direction effects depended on the task and model.

  • Article swapping: The expanded article-swapping experiment introduced alternative framings to disentangle input-probability and output-probability effects.
  • Acronyms: Task significantly affected acronym accuracy for GPT-3.5 and GPT-4, with p < 10^-4 and p < 10^-15 respectively.
  • Acronyms: Input log probability significantly affected acronym accuracy for GPT-3.5 but not GPT-4, with p < 10^-5 and p = 0.092 respectively.
  • Acronyms: Output probability significantly affected both acronym models, with p < 10^-15 for GPT-3.5 and p < 10^-9 for GPT-4.
  • Counting: Output log probability significantly affected counting accuracy in all four model-by-unit conditions, with p < 10^-15 throughout.
  • Counting: Input probability significantly affected low- versus high-probability counting for both models and both unit types, but some effects reversed the predicted direction.
  • Sorting: Sorting direction significantly affected word sorting for both models but not number sorting, with p < 10^-8 and p < 10^-6 for words and p = 0.70 and p = 0.96 for numbers.

D.9.1 Comparing task variants

Additional comparisons tested alternative linear-function tasks, prompt types, number formats, and birthday-name frequency. Task and prompt choices significantly affected accuracy, while name and answer probabilities also showed strong effects in birthday naming.

  • Linear functions: Task significantly affected linear-function accuracy for GPT-3.5 and GPT-4, with p < 0.001 and p < 0.01 respectively.
  • Linear functions: Task remained significant when outputs rather than inputs were controlled, with p < 10^-7 for GPT-3.5 and p < 0.001 for GPT-4.
  • Prompt comparisons: Prompt type significantly affected accuracy for both LLMs, with p < 10^-12 for GPT-3.5 and p < 10^-18 for GPT-4.
  • Prompt comparisons: Prompt effects persisted for unusual values, with p < 10^-18 for both models and all pairwise comparisons significant at p < 0.01.
  • Prompt formatting: Number formatting significantly affected accuracy for GPT-3.5 and GPT-4, with p < 10^-15 and p < 10^-8 respectively.
  • Birthdays: Input probability of the queried name significantly affected birthday-naming accuracy for both models, with p < 10^-15.

E Estimating task probability

The paper estimates task probabilities from corpus mentions and manually filters retrieved examples, while acknowledging assumptions about representativeness and training-data distributions. Corpus frequencies indicate that alphabetical order is far more common than reverse alphabetical order, whereas ascending and descending order differ less.

  • Estimating task probability: Corpus analyses of C4 estimate task-variant probabilities by counting phrases commonly used to invoke each task.Because task instances are rarely labeled, phrase counts serve as a proxy for task-instance counts.
  • Shift ciphers: The shift-cipher search used case-insensitive templates for terms including Caesar cipher, rotN, rot-N, and alphabetic-letter descriptions.The authors manually checked retrieved examples and counted each document only once.
  • Shift ciphers: Manual checking excluded non-cipher uses such as the ROT2 gene and numeric rot-5 encoding, while documents containing several search phrases were counted once.The search did not require encoded examples because checking for them would have made the labor-intensive procedure longer.
  • Assumptions: The shift-cipher estimates assume C4 resembles the models’ training distributions, retrieved texts represent all discussions, and mention frequency tracks illustration probability.These assumptions connect corpus observations to estimated task probabilities.
  • Pig Latin: The Pig Latin analysis similarly searched for phrase occurrences, manually categorized specified variants, and assumed variant mention frequency tracks illustration probability.Its assumptions also include similarity between C4 and the Pile and the models’ training distributions.
  • Corpus frequencies: 95,492 alphabetical-order occurrences versus 629 reverse-order occurrences imply alphabetical order is approximately 150 times as common, while ascending and descending order differ less.These analyses provide evidence about relative frequency in Internet text.

F Spelling

The spelling evaluation tests whether models can access token spelling when outputs are forced into individual letters. GPT-4 performed nearly perfectly, while GPT-3.5’s strict errors largely reflected formatting instructions rather than spelling.

  • Dataset: The evaluation tested 1,000 lowercase Roman-alphabet words that were single tokens both with and without a preceding space.The words were drawn from the first 1 billion words of the Pile dataset.
  • Prompt design: The prompt required listing each word’s letters in order with spaces, forcing the model to break the token into individual letters.The example asks the model to spell “additional” using space-separated letters.
  • Results: 99.9% was GPT-4’s score, with one spelling error: “b e c k e n d” instead of “b a c k e n d.”The reported error was a letter substitution in the target word.
  • Results: 98.1% was GPT-3.5’s strict score, and 17 of its 19 errors used hyphens instead of the required spaces.The authors regard these formatting errors as instruction-following failures rather than failures at spelling.
Loading 2309.13638v1…