Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve

1 Introduction

• LLM evaluations based on human abilities can reveal shared properties while missing weaknesses that arise from models’ distinct training pressures.The paper proposes approaching LLMs on their own terms by considering next-word prediction over Internet text.Source: §1 · p. 3, Figure 1.1 · p. 3
• The teleological approach explains LLM behavior by analyzing the problems, goals, and environments that shaped a system, analogous to explaining species through the problems they evolved to solve.The paper applies this perspective to LLMs’ next-word prediction objective.Source: §1 · pp. 3–4
• The authors predict that deterministic tasks will be easier when the correct output is high-probability than when it is low-probability.They frame LLMs as statistical next-word prediction systems even when those systems are used for tasks such as mathematics.Source: §1 · p. 4
• The experiments deliberately use low-probability situations, such as simple cipher decoding, to expose sources of LLM error.This adversarial design creates settings with enough errors to test which hypothesized factors affect accuracy.Source: §1 · p. 4

Counting

• The introduction connects these predictions to failures on counting, article swapping, shift ciphers, and linear functions, where common or high-probability variants outperform rare or low-probability ones.Figure 1.2 includes the Celsius-to-Fahrenheit function as a common Internet-text variant and contrasts it with a rare linear function.Source: §1 · p. 4, Figure 1.2

2 A teleological approach to understanding LLMs

• The teleological approach first characterizes the problem a system solves, then uses that characterization to generate hypotheses about its capacities and biases.The authors present it as a useful lens rather than a complete explanation of every LLM property.Source: §2 · pp. 5–6

2.1 What problem do LLMs solve?

• Standard LLMs are trained to perform next-word prediction over Internet text using neural-network mechanisms, specifically Transformer architectures.The paper defines the training problem through task, distribution, and architecture.Source: §2.1 · p. 6
• The mismatch between next-word prediction and downstream tasks such as translation or code generation can influence model behavior, just as mismatches between evolved traits and current environments affect biological systems.The paper identifies possible mismatches in task, training distribution, and architecture.Source: §2.1 · pp. 6–7

2.2 Hypothesized embers of autoregression

• The paper predicts that LLMs will perform worse on rare tasks, remain sensitive to output probability on deterministic tasks, and show weaker sensitivity to input probability.These predictions target probability effects even when the evaluated task itself is not inherently probabilistic.Source: §2.2 · p. 7

2.3 What we are not arguing

• The claims concern relative difficulty rather than inability: a task or example can be harder for LLMs without being impossible for them.Accordingly, the experiments compare conditions using the same basic prompting approach rather than claiming optimal or universal failure.Source: §2.3 · pp. 7–8
• Prompting effects themselves are consistent with sensitivity to word sequences, while the paper argues that autoregressive origins have been underused in constructing balanced evaluations.The authors distinguish their claim from the stronger claim that researchers have completely ignored autoregressive limitations.Source: §2.3 · p. 8

3.2 Sensitivity to task probability

• Both GPT-3.5 and GPT-4 decoded frequently used rot-13 more accurately than equally simple, rarely used rot-2.Source: §3.2 · p. 8

3.3 Sensitivity to output probability

• Output probability can influence predictions because maximizing P(output|input) is equivalent to considering P(input|output)P(output).Source: §3.3 · p. 10
• In deterministic settings, imperfect likelihood estimates leave multiple candidate outputs, allowing output probability to affect predictions in practice.Source: §3.3 · p. 10
• GPT-3.5 and GPT-4 became more accurate at rot-13 decoding as the correct output’s log probability increased.Source: §3.3 · pp. 8, 10, Figure 3.1 · p. 9

3.4 Sensitivity to input probability

• GPT-4 performed better with high-probability inputs, whereas GPT-3.5 showed no significant input-probability effect.Source: §3.4 · p. 10

3.5 Asymmetry between input probability and output probability

• Output probability is expected to matter more broadly than input probability because uncertainty commonly requires choosing among candidate outputs.Source: §3.5 · pp. 10–11

4.1 Tasks

• The study evaluates tasks designed to push models into low-probability situations while remaining solvable by simple, deterministic algorithms.Source: §4.1 · p. 11
• A spelling test yielded 99.8% accuracy for GPT-3.5 and 99.9% for GPT-4, supporting character-level evaluation despite subword tokenization.Source: §4.1 · p. 11

4.2 Sentences

• The sentence sets contain 100 high-probability news sentences, medium-probability word substitutions, and low-probability word shuffles.Source: §4.2 · p. 11

4.1 Tasks

• The task suite includes character manipulations, counting, linear functions, acronyms, multiplication, sorting, keyboard ciphers, Pig Latin, and public-figure birthdays.Source: §4.1 · p. 11, Table 2 · p. 12, Acronyms · p. 12

4.3 Models

• The experiments used GPT-3.5 and GPT-4, both large-scale Transformer language models accessed through the OpenAI API at temperature 0.0.Source: §4.3 · p. 12
• Results may not transfer identically to other models, model versions, or chat-interface access, which was not systematically investigated.Source: §4.3 · pp. 12–13

5 Sensitivity to task probability

• The experiments used adversarially selected, low-probability situations and mostly deterministic tasks so that models would make measurable errors.For shift ciphers, prompts specified the shift and included an example, while answer sentences were high-probability sentences from GlobalVoices.Source: §5 · p. 13, Shift ciphers · p. 15

5.1 Shift ciphers

• GPT-4 scored 0.50 or above on the three most common shift levels but below 0.03 on every other shift.GPT-3.5 scored 0.21 on rot-13 and 0.00 on all other shifts; GPT-4’s rot-1 accuracy was 0.82, rot-3 was 0.76, and rot-2 was 0.02.Source: §5.1 · p. 13, Figure 5.2 · p. 14, Shift ciphers · p. 15
• The shift-cipher results cannot be explained by task complexity: GPT-4 scored 0.82 on rot-1 and 0.76 on rot-3 but only 0.02 on rot-2.The authors note that rot-13’s greater frequency in C4 does not explain why GPT-4 outperformed it on rot-1 and rot-3, possibly because document-type frequencies differ across training distributions.Source: Shift ciphers · p. 15

5.2 Pig Latin

• GPT-4’s Pig Latin encoding accuracy was 0.39 versus 0.13 for an unattested variant, while natural-variant accuracy ranged from 0.23 to 0.42 with corpus frequency.For both models, variant effects were significant for encoding but not decoding, possibly because removing unfamiliar endings is easier than producing them.Source: §5.2 · pp. 15–17

5.3 Acronyms

• GPT-4 achieved 0.76 on first-letter acronyms versus 0.03 on second-letter acronyms, despite matched outputs and shared input vocabulary.The difference was statistically significant for both GPT-3.5 and GPT-4.Source: §5.3 · pp. 17–18

5.4 Linear functions

• GPT-4 scored 0.33 on the commonly illustrated Celsius-to-Fahrenheit function versus 0.00 on a similarly structured rare function.When outputs were controlled across tasks, GPT-3.5 and GPT-4 still scored 0.60 and 0.44 on the common task versus 0.01 and 0.00 on the rare task.Source: §5.4 · pp. 18–19

5.5 Sorting

• GPT-4 scored 0.80 for alphabetical word sorting versus 0.32 for reverse alphabetical sorting, but showed similar number-sorting accuracy for ascending and descending order.The word-ordering frequency ratio was approximately 150, whereas descending and ascending number order differed by only about 1.5 in corpus mentions.Source: §5.5 · pp. 19–20, Table 3 · p. 20

5.6 Summary of task probability effects

• Across all seven task pairs, models performed substantially better on the common task variant than the rare one.This cross-task pattern supports the broader claim that task frequency in pretraining data influences model performance.Source: §5.6 · p. 20, §5.5 · p. 20

6 Sensitivity to output probability

• Output probability significantly affected accuracy across rot-13 decoding, reversal, Pig Latin, article swapping, acronyms, and counting tasks.The strongest reported contrasts included GPT-4 accuracy of 0.51 versus 0.13 for high- versus low-probability rot-13 outputs, 0.97 versus 0.53 for reversal, and 0.83 versus 0.28 for Pig Latin.Source: §6.1 · p. 22, §6.2 · p. 24, §6.3 · p. 24

6.1 Shift ciphers

• GPT-4’s rot-13 accuracy was 0.51 for high-probability outputs versus 0.13 for low-probability outputs, with output log probability significant for both models (p < 10^-4).When correct answers were low-probability, models often produced incorrect but more probable outputs.Source: §6.1 · pp. 20, 22
• Both models produced regularized, higher-probability sentences more often than the correct low-probability sentences in targeted rot-13 examples.The tendency extended beyond typo correction: models sometimes altered answers that were grammatical but merely unlikely, while familiar quotations also appeared in rarer shifts.Source: §6.1 · pp. 22–24

6.2 Reversal

• GPT-4 achieved 0.97 accuracy on reversing high-probability sentences versus 0.53 on low-probability sentences, and output probability significantly affected both models (GPT-3.5 p < 10^-8; GPT-4 p < 10^-11).Source: §6.2 · p. 24

6.3 Pig Latin

• GPT-4 achieved 0.83 accuracy for high-probability Pig Latin outputs versus 0.28 for low-probability outputs, with highly significant effects for GPT-3.5 (p < 10^-11) and GPT-4 (p < 10^-14).Source: §6.3 · pp. 24–25

6.4 Article swapping

• GPT-4’s article-swapping accuracy was 0.83 for high-probability outputs versus 0.02 for random word sequences, although input and output probability were highly correlated.Expanded experiments suggested output probability was the most important factor, but the authors could not confidently exclude effects from input probability.Source: §6.4 · p. 25, Figure 6.3 · p. 25

6.5 Acronyms

• For acronyms, GPT-3.5 accuracy was 0.39 for the highest-probability outputs versus 0.18 for the lowest, while GPT-4 achieved 0.76 versus 0.67.Input probability was held roughly constant, and output-probability effects were highly significant for GPT-3.5 (p < 10^-15) and GPT-4 (p < 10^-9).Source: §6.5 · p. 26, Figure 6.4 · p. 26

6.6 Counting

• Counting accuracy tracked number frequency more strongly than magnitude, and output log probability significantly predicted performance across conditions (p < 10^-15).Accuracy was higher for high-probability numbers than neighboring low-probability numbers even when average magnitude was matched; magnitude was nonsignificant for word counting (GPT-3.5 p = 0.56; GPT-4 p = 0.07).Source: §6.6 · p. 26, Counting · p. 28, Table 4 · p. 28

6.7 Summary of output probability effects

• GPT models’ performance is heavily influenced by target-output probability, despite the tasks being non-probabilistic.These results support the hypothesis that LLMs are sensitive to the probability of the sequences they must produce.Source: §6.7 · p. 28

7.1 Shift ciphers, reversal, Pig Latin, and acronyms

• GPT-4 encoding rot-13 reached 21% on high-probability inputs but 11% on low-probability inputs, while other tested tasks showed no significant GPT-4 input-probability effect.The broader analysis identifies input probability as a moderate influence in rot-13 encoding and a large influence in birthdays.Source: §7.1 · pp. 29–30

7.2 Counting

• Counting produced inconsistent input-probability effects, with expected and reversed directions across models and character-versus-word conditions; effect sizes were generally small.GPT-3.5 benefited from high-probability characters, whereas GPT-4 benefited from high-probability words, with opposite patterns in the other two cases.Source: §7.2 · p. 30

7.3 Asymmetry between input probability and output probability

• Output probability influenced performance more pervasively than input probability: all output-probability experiments showed significant effects, whereas only a few input-probability conditions did.For high-probability rot-13, decoding exceeded encoding for both GPT-4 and GPT-3.5, supporting greater sensitivity to output probability.Source: §7.3 · p. 31

7.4 Birthdays

• GPT-4 accuracy on public-figure birthdays was 0.99 for high-probability people versus 0.23 for low-probability people.GPT-3.5 ranged from 0.99 to 0.09 across the same conditions.Source: §7.4 · pp. 31–32
• The birthday result suggests that input probability matters when relevant information cannot be derived from the input and must have been encountered during training.The authors conjecture that input sensitivity can reflect memorized input-output mappings rather than systematic function use, including for rot-13 encoding.Source: §7.4 · p. 32

7.5 Summary of input probability effects

• Across the investigated tasks, input probability had a smaller impact than output probability but meaningfully affected performance under the right conditions.The paper identifies birthdays as a large effect and rot-13 encoding as a moderate statistically significant effect, while noting complementary arithmetic findings from prior work.Source: §7.5 · p. 33

8 Other phenomena

• The paper interprets these findings as evidence that LLMs are shaped by textual experience rather than operating as general-purpose human-like reasoning systems.The keyboard experiment links failures to missing embodiment, while the wording experiments link performance changes to prompt form and statistical cues.Source: §8.1 · p. 33, Keyboard cipher · p. 34, §8.2 · p. 35

8.1.1 Keyboard cipher

• GPT-4 scored 0% on a keyboard cipher, and supplying the keyboard layout reduced but did not eliminate its errors.With the layout provided, GPT-4’s median edit distance remained 8.0 mistakes per sentence.Source: §8.1.1 · p. 33, Keyboard cipher · p. 34

Keyboard cipher

• Providing explicit letter correspondences further improved GPT-4’s keyboard-cipher performance, but remaining errors indicate difficulty systematically manipulating symbols.GPT-3.5 instead performed worse with correspondences provided, possibly because the longer prompt was harder to process.Source: Keyboard cipher · pp. 34–35

8.2.1 Linear function

• GPT-4 performed significantly better on f(x) = (9/5)x + 32 when the prompt identified it as Celsius-Fahrenheit conversion than when it presented the abstract function alone.Both models also performed better in a generic priming-control condition than in the basic condition, despite no apparent additional task information.Source: §8.2.1 · pp. 35–37

8.2.2 Multiplication

• GPT-4 multiplication accuracy varied from 0.46 for digit notation to 0.17 for alternating-capital words despite identical numerical values.The formatting differences were often statistically significant, showing that linguistic form changes performance on the same computation.Source: §8.2.2 · p. 37

9.1 How to understand a new type of intelligence

• Teleological analysis reveals model-specific properties that generic evaluations may miss, because it uses knowledge of training goals, environments, and architectures to generate testable behavioral hypotheses.Unlike desideratum-focused evaluation, it can expose failures such as sensitivity to sentence probability that standard tests may overlook.Source: §9.1 · pp. 38–39

9.2 Adversarial testing of AI systems

• Counting accuracy was driven more by number commonness than size: models performed better when counting to frequent 100 than to rarer smaller numbers such as 59.The teleological adversarial strategy identifies difficulty by analyzing the pressures that shaped a model rather than inspecting its parameters or input-output behavior.Source: §9.2 · pp. 39–40
• The teleological approach is most useful when researchers seek holistic understanding and know at least something about a system’s training task, distribution, and architecture.White-box, black-box, and complexity-guided methods remain useful for automated error discovery or when training information is unavailable.Source: §9.2 · p. 40

9.3 Comparing models to humans

• Human-likeness should be assessed in two steps: characterize models in their own terms, then compare that characterization with human behavior rather than beginning with human-designed tests.Direct comparisons risk confirmation bias, overestimating similarity, and overlooking model-specific failure modes.Source: §9.3 · pp. 40–41
• LLMs appear more sensitive than humans to task and example probability, although humans may also regularize implausible outputs and show formatting-related differences in mathematical tasks.The paper cautions that strong claims about qualitative human-model differences require evaluating humans and LLMs on the same stimuli.Source: §9.3 · pp. 41–42

9.4 Evaluating models fairly: Prompts, scaling, and calibration

• Advanced prompting improves decoding accuracy but can hurt encoding, while models retain shift-level and probability-related difficulty patterns.Chain-of-thought and step-by-step prompting increased overall performance without erasing autoregressive effects.Source: §9.4 · pp. 42–43, Figure 9.1 · p. 42
• GPT-4 substantially outperformed GPT-3.5 across almost all tasks yet retained qualitative sensitivity to task and example probability.Scaling therefore improves overall performance but may not fully overcome influences arising from model nature.Source: §9.4 · p. 43
• GPT-4 rated all fifteen incorrectly answered difficulty queries as 1, suggesting that its self-reported confidence did not identify these failures.The queries covered counting, low-probability article swapping, and high-probability rot-10 decoding.Source: §9.4 · p. 43

10.1 Training task

• The paper relates its perspective to prior work emphasizing goals, resources, training objectives, data, and architectural limitations as determinants of model behavior.It frames teleological analysis as complementary to broader work on how systems learn and how their implementations constrain performance.Source: §10.1 · pp. 44–45, §10.3 · pp. 47–48

10.1.1 Difficulty in tasks that depend on meaning

• Next-word prediction makes LLMs primarily language models rather than direct models of thought, contributing to difficulties with entity tracking, negation, and everyday causal affordances.The paper remains agnostic about whether text-only systems capture meaning but argues that meaning-sensitive tasks do not come naturally to them.Source: §10.1.1 · p. 45

10.1.2 Inability to modify text that has already been produced

• Autoregressive generation prevents models from revising already-produced text, so early planning errors can propagate into later outputs.This phenomenon is described as self-delusion: prior output becomes input for subsequent predictions.Source: §10.1.2 · pp. 45–46, Figure 10.1 · p. 46

10.2 Training distribution

• Training distributions shape LLM behavior through societal biases, spurious correlations, and memorization of repeated dataset-specific material.Models trained on Wikitext-103 tended to generate Boat Race sentences because related articles and sentences were unusually frequent in that dataset.Source: §10.2.1 · p. 47, §10.2.2 · p. 47

10.3.1 Sensitivity to tokenization and other aspects of input formatting

• Architecture and input representation constrain behavior: subword tokenization can create brittleness to typos and make character-level manipulations less accessible.Input formatting has also been shown to affect the processing costs and robustness of language models.Source: §10.3.1 · p. 48

10.3.2 Limited compositionality and systematicity

• Neural networks can represent compositional and systematic behavior, but doing so is not necessarily straightforward and remains difficult in some settings.This distinction concerns expected ease of behavior rather than fundamental representational impossibility.Source: §10.3.2 · pp. 48–49

10.4 Beneficial embers of autoregression

• Next-word prediction also has benefits: its generality gives models experience with many tasks, while Internet text exposes them to diverse inputs and neural architectures support several useful forms of generalization.These benefits help explain why the training setup became prevalent despite its limitations.Source: §10.4 · p. 49

11 Conclusion

• LLMs are statistical next-word prediction systems despite their advanced capabilities and broad uses.The conclusion contrasts this simple training description with claims that language models are multitask learners, few-shot learners, interfaces, reasoners, or knowledge graphs.Source: §11 · p. 49
• LLMs have greater difficulty with infrequent tasks than frequent tasks, even when the tasks seem equally complex to humans.This pattern is attributed to training focused on word-sequence statistics.Source: §11 · p. 49
• LLMs also have greater difficulty producing low-probability answers than high-probability answers, even when the task is deterministic.The conclusion presents this as another consequence of autoregressive training.Source: §11 · p. 49
• Understanding LLMs requires considering both their sparks of artificial general intelligence and their embers of autoregression.The conclusion treats advanced capabilities and training-shaped limitations as jointly relevant to model understanding.Source: §11 · p. 49

• The teleological perspective should be one of several approaches because it can be incomplete and its analyses of model goals may be incorrect.The authors therefore do not claim it should be the field’s only way to study LLMs.Source: Limitations · p. 49
• Teleologically derived predictions should be tested empirically rather than assumed to be correct.Source: Limitations · p. 49
• The experiments use proprietary models whose precise architectures and training data are not publicly available.Source: Limitations · pp. 49–50
• The investigated tasks are relatively simple and lack substantial practical utility, leaving their effects in more practical settings for future work.The authors chose simple tasks to reason clearly about the general principles driving LLM performance.Source: Limitations · p. 50

A.1 Sample sizes

• Datasets generally used 100 examples per condition, while acronym experiments generated 1000 examples per condition and counting varied example counts by task.The dataset-size overview is reported in Table 7, while task-specific counts include 100 examples per condition for ciphers, Pig Latin, article swapping, sorting, multiplication, linear functions, and birthdays.Source: §A.1 · p. 62, §A.3 · p. 63, §A.4 · p. 63

A.2 Shift cipher sentences

• Shift-cipher sentences were drawn from recent Global Voices articles, filtered for ordinary grammatical content, and transformed into high-, medium-, low-, and minimally altered probability conditions.High-probability sentences were selected using GPT-2 token perplexity; medium-probability sentences used constrained token replacements, low-probability sentences shuffled internal words, and adversarial sentences changed one word within Levenshtein distance 2.Source: §A.2 · pp. 62–63

A.3 Reversal sentences and keyboard cipher sentences

• Reversal and keyboard-cipher experiments reused the shift-cipher sentences, with 100 sentences per condition, while Pig Latin excluded spelling–pronunciation ambiguities and lowercased all inputs and outputs.Pig Latin prompts also clarified punctuation, apostrophe handling, and consonant- versus vowel-initial words through an in-context example.Source: §A.3 · p. 63, §A.4 · pp. 63, 65

A.5 Article swapping sentences

• Article-swapping sentences contained internal articles without sentence-initial or sentence-final articles, yielding 100 examples per probability level.These constraints prevented trivial cases and avoided boundary problems during swapping.Source: §A.5 · p. 65

A.6 Acronyms

• Acronym datasets controlled output words, input probability, acronym position, and token split points, generating 1000 examples per condition from probability-binned vocabularies.The vocabulary contained 8203 lowercase and 2244 all-caps words after exclusions, and output and input log probabilities were estimated from GPT-2 sequences.Source: §A.6 · p. 65, Table 9 · p. 66

A.7 Counting

• Counting, sorting, multiplication, linear-function, and birthday datasets matched candidate structure while varying lexical or numerical probability and controlling relevant confounds.Counting used matched token patterns and repeated-character examples; linear-function out-of-distribution constraints excluded 99.8% or 99.7% of corpus conversion occurrences; birthday names were frequency-binned and birth-date matched.Source: §A.7 · p. 66, §A.8 · p. 66, §A.9 · p. 66

B.3 Comments on step-by-step and chain-of-thought prompts

• For shift-cipher prompting, the researchers selected the more successful middle-of-prompt step-by-step instruction and a letter-based chain-of-thought variant after informal comparisons.They expected sentence-level shifting to perform best, but a letter-based variant instead performed best and was used for later prompts.Source: §B.3 · p. 68

Transformation and symbolic tasks

• The dataset tests shift-cipher, word-reversal, Pig-Latin, article-swapping, acronym, counting, and alphabetical-sorting operations.Examples include decoding ROT-13, reversing word sequences, converting Pig Latin, rearranging articles, extracting initial letters, counting items, and sorting words.Source: §B.7 · p. 69, §B.8 · p. 69, §B.9 · p. 69

Article-swapping tasks

• Article-swapping tasks cross decoding and encoding directions with swaps applied either to the preceding or following word.Each prompt supplies an example transformation and a separate Input 2 sentence requiring the corresponding operation.Source: §B.12 · p. 70, §B.13 · p. 70, §B.14 · p. 70

Numerical tasks

• The numerical tasks include three-digit multiplication presented as numerals, lowercase words, all caps, or alternating capitals.The dataset also includes applying the transformation 9/5 × input + 32, with one version requiring nearest-integer rounding.Source: §B.24 · p. 71, §B.25 · p. 72, §B.26 · p. 72

Keyboard cipher tasks

• Keyboard-cipher encoding tasks vary whether the QWERTY layout, complete letter correspondences, or neither is provided.The cipher replaces each letter with the letter to its right, wrapping from the row’s end to its beginning; punctuation remains unchanged in the example.Source: §B.30 · pp. 72–73, §B.31 · p. 73, §B.32 · pp. 73–74

Factual recall

• The dataset includes factual recall through a question asking for Michelle Pfeiffer’s birth date in “Month Day, Year” format.The prompt specifies the required response format with an example date.Source: §B.33 · p. 74

C.1 Shift ciphers and reversal / C.2 Pig Latin / C.3 Acronyms / C.4 Sorting / C.5 Counting, linear functions, and three-digit multiplication / C.6 Birthdays

• Correctness was determined with task-specific matching rules: outputs generally had to exactly match the ground truth, while some tasks allowed manually checked equivalent formats or answers.Source: §C.1 · p. 75, §C.4 · p. 75, §C.5 · p. 75

D.1 General notes

• The general statistical procedure used logistic or linear regression, z-scored continuous predictors, VIF checks, and separate binary correctness outcomes for the two models.Source: §D.1 · pp. 75–76

D.2.1 Comparing task variants

• GPT-4 and GPT-3.5 showed significant shift effects in most cipher comparisons, while GPT-3.5 encoding lacked significance because accuracy was near the floor.For GPT-4 decoding, p < 10−7; for GPT-3.5 decoding, p < 0.01; GPT-3.5 encoding had p = 0.29 with accuracies of 0.00 and 0.02.Source: §D.2.1 · p. 76

D.2.2 Input probability

• Using edit distance instead of correctness made input probability significant for both GPT-3.5 and GPT-4.Both effects had p < 0.01, with VIF below 2.0 for input log probability.Source: §D.2.2 · p. 76

D.2.2 Input probability / D.2.3 Output probability

• Output probability significantly affected both models, whereas input probability affected GPT-4 but not GPT-3.5 under correctness scoring.Output probability effects were p < 10−4 for GPT-3.5 and p < 10−7 for GPT-4; input probability was p < 0.05 for GPT-4 and p = 0.61 for GPT-3.5.Source: §D.2.3 · p. 76, §D.2.2 · p. 76

D.2.4 Step-by-step and chain-of-thought prompting

• Output probability remained significant across basic, step-by-step, and chain-of-thought prompting, while input-probability and shift effects varied by prompt style and metric.Output effects had p < 10−7 for basic and step-by-step prompting and p < 10−8 for chain-of-thought prompting.Source: §D.2.4 · pp. 76–77

D.4.1 Comparing task variants

• Task variant significantly affected GPT-3.5 and GPT-4 encoding, and GPT-4 decoding, but not GPT-3.5 decoding.The reported p-values were below 0.01, 0.001, and 0.05 respectively, while GPT-3.5 decoding had p = 0.08.Source: §D.4.1 · p. 77
• Pig Latin variant probability significantly affected encoding for both models but not decoding.Encoding effects were p < 10−6 for GPT-3.5 and p < 10−8 for GPT-4; decoding effects were p = 0.27 and p = 0.11.Source: §D.4.1 · pp. 77–78

D.4.2 Input probability

• Input probability did not significantly affect Pig Latin performance for GPT-3.5 or GPT-4.The reported p-values were 0.70 and 0.88, with modest VIFs of 1.88 and 1.98.Source: §D.4.2 · p. 78

D.4.3 Output probability

• Output probability significantly affected Pig Latin performance for both GPT-3.5 and GPT-4, with acceptable collinearity.The effects were p < 10−11 and p < 10−14, with VIFs of 2.14 and 2.69.Source: §D.4.3 · p. 78

D.5.1 Main experiment

• In the main experiment, input and output log probabilities were too collinear to include together, so output log probability represented overall example probability and remained significant for both models.After removing input log probability, effects were p < 10−11 for GPT-3.5 and p < 10−15 for GPT-4, with VIFs of 1.69 and 2.22.Source: §D.5.1 · p. 78

D.5.2 Expanded experiment

• Grammatical outputs produced the highest accuracies across article-swapping variants, indicating output probability mainly drove the probability effect.The swap direction and input grammaticality did not explain the largest accuracy differences.Source: §D.5.2 · p. 78, Table 10 · p. 79

D.6.2 Input probability

• Input log probability significantly affected GPT-3.5 (p < 10^-5) but not GPT-4 (p = 0.092).Source: §D.6.2 · p. 80

D.6.3 Output probability

• Output probability significantly affected both GPT-3.5 and GPT-4, with p < 10^-15 and p < 10^-9, respectively.Source: §D.6.3 · p. 80

D.7 Counting

• Output probability significantly predicted counting success in all four common-input conditions, while magnitude mattered only for character-based counting.Output log probability had p < 10^-15 in all four cases; magnitude was nonsignificant for word-based conditions.Source: §D.7 · p. 80
• Input probability significantly affected controlled counting comparisons, but two small effects ran opposite to the expected direction.The reversed effects occurred for GPT-3.5 word counting and GPT-4 character counting.Source: §D.7 · p. 80

D.8 Sorting

• Ordering direction affected word sorting for both models but not number sorting, with p < 10^-8 and p < 10^-6 for words and p = 0.70 and p = 0.96 for numbers.Source: §D.8 · p. 80

D.9.1 Comparing task variants

• Task variants significantly affected both models even when input or output was controlled, with p < 0.001 for GPT-3.5 and p < 0.01 for GPT-4 in the initial comparison.The effect persisted when output rather than input was controlled, with p < 10^-7 for GPT-3.5 and p < 0.001 for GPT-4.Source: §D.9.1 · p. 81

D.9.2 Comparing prompts

• Prompt type significantly affected both models, and all three pairwise comparisons were significant; unusual-number prompts showed the same pattern.The initial prompt effect had p < 10^-12 for GPT-3.5 and p < 10^-18 for GPT-4; unusual-number conditions had p < 10^-18 for both.Source: §D.9.2 · p. 81

E Estimating task probability

• Task-probability estimates use corpus analyses of phrases invoking each task, but imperfect assumptions and manual review limit their precision.The authors describe the estimates as useful for obtaining a rough sense of task-variant frequency.Source: E Estimating task probability · p. 82

E.1 Shift ciphers

• Shift-cipher counts search several naming templates, manually exclude false positives, and categorize the remaining examples by shift level.Documents containing multiple relevant phrases were counted once, and encoded-text examples were not required.Source: §E.1 · p. 82
• Encoded-text searches for shift ciphers produced too many false positives to check reliably, including ordinary strings that resemble shifted words.The authors therefore relied on phrase-based searches rather than directly searching encoded text.Source: §E.1 · p. 83
• Shift-cipher estimates assume C4 resembles model training data, searched texts represent all mentions, and mention probability tracks illustration probability.Source: §E.1 · p. 83

E.2 Pig Latin

• Pig Latin counts favored the -way variant over -ay, -yay, and -hay in both C4 and the Pile, although the sample was small.The authors report similar relative frequencies across the two corpora and broadly similar results to Vaux and Nevins (2003).Source: §E.2 · p. 83

E.3 Sorting

• Sorting phrases occurred 95,492 times for alphabetical order versus 629 for reverse alphabetical order, whereas ascending and descending counts were 21,562 and 31,378.The authors estimate alphabetical order is about 150 times as common as reverse alphabetical order, while descending order is about 1.5 times as common as ascending order.Source: §E.3 · pp. 83–84

F Spelling

• The task sampled 1,000 lowercase Roman-alphabet words that were single tokens with and without a preceding space.Spaces between output letters forced models to break each token into individual letters.Source: F Spelling · p. 84
• GPT-4 achieved 99.9% spelling accuracy, with one error changing the target word backend to beckend.Source: F Spelling · p. 84
• GPT-3.5 achieved 99.8% after hyphen-formatting errors were counted as correct, compared with 98.1% under strict scoring.Seventeen of its 19 strict errors used hyphens instead of spaces, which the authors classify as instruction-following failures.Source: F Spelling · p. 84