PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Reflection in Pre-Training

Essential AI, :, Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Anthony Polloreno, Ashish Tanwer, Burhan Drak Sibai, Divya S Mansingka, Divya Shivaprasad, Ishaan Shah, Karl Stratos, Khoi Nguyen, Michael Callahan, Michael Pust, Mrinal Iyer, Philip Monk, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, Tim Romanski

arXiv:2504.04022v1cs.CLcs.AI

TL;DR

The paper asks whether reflection develops during pre-training, rather than only through post-training reinforcement learning. It measures reflection by testing models on tasks with deliberately erroneous reasoning. Reflection appears across pre-training checkpoints and improves with additional pre-training, while how it develops into autonomous reasoning remains an open question.

  • Problem

    The paper examines whether reflection emerges during pre-training, an issue contrasted with the commonly held view that it emerges during post-training through reinforcement learning.

    Source: §1 · p. 2, §6 · p. 12

  • Method

    The authors create six datasets across four domains and use adversarial reasoning to measure situational-reflection and self-reflection.

    Source: §1 · p. 3, §3.1 · p. 4

  • Results

    231 of 240 dataset-checkpoint pairs show at least one instance of situational-reflection, 154 of 240 show self-reflection, and average Pearson correlation between accuracy and log(pre-training compute) is 0.76 across tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Reflection is observed even in models trained with very few training flops, and these abilities strengthen as more compute is spent on pre-training.

    Source: §1 · p. 3, §6 · p. 12

    How early self-reflection develops into sophisticated autonomous reasoning abilities with post-training remains an open question.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

A language model's ability to reflect on its own reasoning provides a key advantage for solving complex problems. While most recent research has focused on how this ability develops during reinforcement learning, we show that it actually begins to emerge much earlier - during the model's pre-training. To study this, we introduce deliberate errors into chains-of-thought and test whether the model can still arrive at the correct answer by recognizing and correcting these mistakes. By tracking performance across different stages of pre-training, we observe that this self-correcting ability appears early and improves steadily over time. For instance, an OLMo2-7B model pre-trained on 4 trillion tokens displays self-correction on our six self-reflection tasks.

1 Introduction

The paper challenges the view that reflection develops only through reinforcement learning, proposing a framework to measure it across pre-training. Across six datasets, reflection appears in many pre-training checkpoints and improves with more compute.

1 Introduction

  • The framework distinguishes situational-reflection on reasoning from another source from self-reflection on the model’s own reasoning.It calibrates task solving in the presence of confounding reasoning that leads to an incorrect answer.Source: §1 · p. 2
  • A partially pre-trained OLMo-2-32B corrects its mistaken program-input prediction after its incorrect reasoning is followed by “Wait,”.The figure illustrates sample_485 from CruxEval.Source: Figure 2 · p. 2, §1 · p. 2
  • 231 of 240 dataset-checkpoint pairs show at least one instance of situational-reflection, 154 of 240 show self-reflection, and average Pearson correlation between accuracy and log(pre-training compute) is 0.76 across tasks.The evaluation spans six diverse datasets.Source: §1 · p. 3
  • The authors introduce six datasets across Code, Knowledge Acquisition, Logical Reasoning, and Mathematics to study reflection.Programmatically introduced arithmetic perturbations and logical inconsistencies control the reflection required while retaining the established chain-of-thought format [Wei et al., 2022].Source: §1 · pp. 2–3
  • With more pre-training, models correct more adversarial examples and increasingly recover from incorrect reasoning, with fewer test-time tokens needed for the same number of tasks.Explicit reflection also becomes more frequent and increasingly contributes to recovery from confounding chains-of-thought.Source: §1 · p. 3

2 Related Work

The paper situates its study among work on reasoning evaluation, adversarial inputs, and train-time versus test-time compute. It uses adversarial reasoning datasets to examine reflection during pre-training, rather than primarily evaluating post-trained models or demonstrating vulnerabilities.

Evaluating Reasoning in LLMs

  • Prior reasoning evaluations use outcomes, generation traces, interactive engagement, or internal-network analyses, while this work diagnoses reasoning across training phases using adversarial datasets.The cited approaches include Chain of Thought [Wei et al., 2022], Tree of Thought [Yao et al., 2023], and Self-Consistency [Wang et al., 2023b].Source: §2 · p. 3

Adversarial Attacks on LLMs

Train Time vs Test Time Tradeoff

  • The paper complements train-time versus test-time compute research by showing that some reasoning capabilities can gradually emerge during pre-training without fine-tuning, reinforcement learning, or specialized test-time techniques.The authors suggest these findings may affect how compute is allocated across training and inference.Source: §2 · pp. 3–4

3 Approach

The approach operationalizes reflection in two settings, elicits it with adversarial chains-of-thought, and measures explicit and implicit forms. Its datasets pair tasks with reasoning that contains errors, which models must navigate to reach correct answers.

3.1 Defining Reflection

  • Reflection is defined as examining information, evaluating its underlying reasoning, and adjusting subsequent behavior; the paper distinguishes situational-reflection from self-reflection.Situational-reflection concerns information created by another source, while self-reflection concerns the model’s own outputs.Source: §3.1 · p. 4

3.2 Eliciting Reflection with Adversarial Reflection Datasets

  • Adversarial datasets elicit reflection by presenting chains-of-thought with errors that models must identify and correct to solve the task.Situational chains-of-thought are created by corrupting correct ones with human-like logical or arithmetic errors; self-reflection uses the model’s own mistakes.Source: §3.2 · p. 4
  • The task design appends a trigger such as “Wait” to adversarial reasoning, and the algorithm provides separate dataset-generation variants for situational-reflection and self-reflection.Situational data are generated by introducing small errors into correct reasoning; self-reflection data retain chains-of-thought from model runs that produced incorrect answers.Source: §3.2 · p. 4, §3.3 · p. 5

3.3 Measuring Reflection

  • Explicit reflection is detected with a prompt-based LLM classifier, while correct answers in adversarial contexts count as implicit reflection even without explicit error-identifying tokens.By construction, correct outputs not identified by the explicit classifier are categorized as implicit reflection.Source: §3.3 · pp. 4–5

4 Experimental Setup

The setup tracks reflection across partially pretrained checkpoints and evaluates adversarial tasks for both external-chain and self-generated-error reflection, including explicit reflective behavior.

4 Experimental Setup

  • Experiments span partially pretrained models varying parameter count and training tokens, with situational and self-reflection tasks plus explicit-reflection assessment.Source: §4 · p. 5

OLMo-2

  • OLMo-2 provides 7B, 13B, and 32B variants, and the study evaluates 40 checkpoints, using the closest available checkpoints when equally spaced ones are unavailable.Source: OLMo-2 · p. 5

Qwen2.5

  • Qwen2.5 evaluation covers 0.5B, 3B, 7B, 14B, 32B, and 72B parameter variants.Source: Qwen2.5 · p. 6

4.2 Datasets

  • Six adversarial datasets adapt reasoning benchmarks; situational chains are deliberately corrupted, self-reflection sets use prior failures on checkpoint-shared questions, and automated plus manual checks target dataset quality.Source: §4.2 · p. 6

4.3 Evaluation

  • Evaluation reports correctness, explicit-reflection rate regardless of correctness, explicit-reflection accuracy, and implicit-reflection accuracy; a high-precision prompt classifier is validated against human labels.Source: §4.3 · p. 7

4.4 Infrastructure

  • OLMo-2 and Qwen inference use vLLM, DeepSeek-V3 uses SGLang, and jobs run on AMD MI300x accelerators scheduled with Kubernetes.Source: §4.4 · p. 7

5 Results

Across OLMo-2 checkpoints, reflection and recovery from confounders increase with pre-training, with explicit reflection increasingly contributing to recovery. The results also show reflection without triggers, a trade between training and test-time compute, and a similar performance pattern in Qwen2.5.

5 Results

  • Across OLMo-2 checkpoints, recovery from non-zero adversarial examples and explicit reflection increase with pre-training across tasks.Table 6 summarizes these results.Source: §5 · pp. 7–8, Table 6 · p. 8

5.1 Explicit Situational-Reflection is Prominent in all Models

  • 231 of 240 dataset-checkpoint pairs show situational reflection; with more compute, explicit reflection increasingly contributes to recovery, while TriviaQA improvements lean on implicit reflection.On GSM8K-Platinum, explicit reflection accounts for a majority of solved instances as pre-training increases.Source: §5.1 · p. 8, Figure 3 · p. 9

5.2 Models Reflect Even Without Triggers; ‘Wait,’ Enhances Explicitness and Accuracy

  • Without a trigger, models increasingly recover from situational confounders; ‘Wait,’ performance combines implicit-only behavior with explicit reflection and improves when explicit reflection occurs.With explicit reflection, ‘Wait,’ performance is comparable to a prompt that explicitly identifies the mistake; without it, performance aligns with the no-trigger condition.Source: §5.2 · p. 9, Figure 4 · p. 10

5.3 Explicit Self-Reflection is Harder but Advances With More Compute

  • About 64.2% of self-reflection task attempts show some self-correction, while explicit self-reflection emerges earlier than successful correction on cruxeval-i.These self-reflection datasets include only questions that every checkpoint in the model family answered incorrectly; how these abilities develop into post-training autonomous reasoning remains open.Source: §5 · p. 7, §5.3 · pp. 9–10, Figure 5 · p. 10

5.4 One Can Trade Training for Test-time Compute

  • For OLMo-2-32B, more train-time compute lowers test-time compute needed to reach the same GSM8K-Platinum accuracy under sequential ‘Wait,’ scaling.Figure 6 excludes an outlier checkpoint that distorted the plot.Source: §5.4 · p. 11, Figure 6 · p. 11, §5.5 · p. 11

5.5 Do we see similar results in stronger model families?

  • Qwen2.5 reproduces rising adversarial-task performance with increased pre-training compute, varied here through parameter count.Source: §5.5 · p. 11, Figure 7 · p. 12

6 Conclusion

The conclusion examines how reflective reasoning develops during pre-training, using a framework that measures reflection across pre-training checkpoints. It reports reflection at low compute and stronger abilities with more pre-training.

6 Conclusion

  • At 198 billion tokens, OLMo-2-7B demonstrates reflection across mathematics, code, verbal, and logical reasoning.The conclusion also reports that these abilities strengthen as more compute is spent on pre-training.Source: §6 · p. 12
  • The paper asks how reflective reasoning advances during pre-training, in contrast to the view that it emerges during post-training through reinforcement learning.It proposes a comprehensive framework to measure reflection throughout pre-training.Source: §6 · p. 12

B OLMo-2 and Qwen Pre-trained Checkpoints

  • The checkpoint inventory lists OLMo-2 7B, 13B, and 32B checkpoints and Qwen2.5 models from 0.5B to 72B.The OLMo-2 entries span multiple pre-training token counts; the Qwen2.5 entries are main models.Source: B OLMo-2 and Qwen Pre-trained… · p. 19

C.1 Plots for GSM8K, GSM8K-Platinum, cruxeval-i

Figure 8 reports accuracy and explicit-reflection accuracy for the OLMo-2 family on three reasoning tasks.

C.1 Plots for GSM8K, GSM8K-Platinum, cruxeval-i

  • Figure 8 covers the OLMo-2 family of models.The caption identifies accuracy and explicit-reflection accuracy as the plotted measures.Source: Figure 8 · p. 20
  • The plotted tasks are GSM8K, GSM8K-Platinum, and cruxeval-i.The caption names all three task datasets.Source: Figure 8 · p. 20
  • The figure presents both accuracy and explicit-reflection accuracy.The caption does not state the plotted values or which measure is higher.Source: Figure 8 · p. 20

C.2 Plots for cruxeval-o, TriviaQA, BBH

The plots cover OLMo-2 family reflection measures across adversarial evaluations and several task groups.

C.2 Plots for cruxeval-o, TriviaQA, BBH

  • Figure 9 plots explicit-reflection accuracy in the situational setting across adversarial evaluations.The listed task group is cruxeval-o, TriviaQA, and BBH.Source: Figure 9 · p. 21

D.1 Plots for GSM8K, GSM8K-Platinum, cruxeval-i

  • Figure 10 shows self-reflection accuracy and explicit-reflection rates for the OLMo-2 family on GSM8K, GSM8K-Platinum, and cruxeval-i.Source: Figure 10 · p. 22

D.2 Plots for cruxeval-o, TriviaQA, BBH

  • Figure 11 shows self-reflection accuracy and explicit-reflection rates for the OLMo-2 family on cruxeval-o, BBH, and TriviaQA.Source: Figure 11 · p. 23

E Do Models Understand Correct Answers?

On GSM8K-Platinum, models tend to assign lower perplexity to incorrect answers following misleading reasoning, although this gap narrows during pre-training. The analysis tests whether models favor correctness despite that misleading context.

E Do Models Understand Correct Answers?

  • The analysis compares perplexity for the same question and adversarial chain-of-thought followed by either a correct or incorrect answer.Perplexity is computed on the final answer and adversarial chain-of-thought, conditioned on the question, across OLMo-2 7B, 13B, and 32B pre-training checkpoints.Source: E Do Models Understand Correct Answers? · pp. 23–24
  • Across GSM8K-Platinum, models assign lower perplexity to incorrect answers after misleading reasoning, but the gap narrows during pre-training.Figure 12 depicts the correct-minus-incorrect perplexity difference; positive values indicate greater likelihood assigned to incorrect answers under misleading reasoning.Source: E Do Models Understand Correct Answers? · p. 24, Figure 12 · p. 24
  • The observed preference for incorrect answers under misleading reasoning may reflect a local consistency bias, in which models prioritize coherence with preceding context over correctness.Source: E Do Models Understand Correct Answers? · p. 24
  • Certain OLMo-2-32B checkpoints show near-zero perplexity differences, while differences remain non-negative and reflective trigger cues such as “Wait,” continue to help shift responses toward correct conclusions.Source: E Do Models Understand Correct Answers? · p. 24

F Adversarial Dataset Creation

The section builds adversarial reasoning contexts for mathematical reflection tasks, testing both responses to altered reasoning from another source and correction of a model’s own errors. Its perturbations target different positions in a reasoning chain while preserving surface fluency.

F.1 Math Reasoning

  • GSM8K validation uses 1,319 questions and GSM8K-Platinum validation uses 1,209, pairing a widely used noisy benchmark with a disambiguated version.The two benchmarks assess whether progress reflects reasoning improvements rather than overfitting to label noise.Source: §F.1 · p. 24

Adversarial Settings

  • Situational-reflection CoTs are created by converting original reasoning into Python, introducing targeted code errors, and converting the modified program back into natural language.DeepSeek-V3 generates the Python program [DeepSeek-AI, 2025b]; examples whose final answer remains unchanged are discarded, and “Wait,” is prepended to the adversarial context.Source: Adversarial Settings · pp. 24–25
  • Self-reflection uses each checkpoint’s own incorrect answers as adversarial contexts and evaluates checkpoints on questions all checkpoints in their parameter family answered incorrectly.The setup prepends “Wait,” to the question and adversarial context.Source: Adversarial Settings · p. 25

Attack Types in Situational-Reflection

  • Seven structured perturbations target the beginning, middle, or end of reasoning through variable changes, arithmetic changes, extraneous steps, or omitted steps.The modifications are designed to preserve fluent wording while disrupting reasoning logic.Source: Attack Types in Situational-Reflection · p. 25
  • End-position attacks also alter final arithmetic, append an unnecessary calculation, or remove the final step, leaving an unresolved answer.These perturbations preserve surface-level fluency while challenging verification of intermediate steps.Source: Attack Types in Situational-Reflection · p. 25

Dataset Augmentation Summary

The appendix describes adversarial dataset construction across math, code, knowledge, and broader reasoning tasks, along with human annotation used to calibrate the reflection classifier. It also reports how perturbations were filtered for the math datasets.

Dataset Augmentation Summary

  • 9,070 adversarial instances remain for GSM8K after 163 of 9,233 perturbed instances are filtered because their final answer is unchanged.The perturbations comprise seven modifications applied to each of 1,319 questions.Source: Dataset Augmentation Summary · p. 25

GSM8K-Platinum

  • 8,223 adversarial instances remain for GSM8K-Platinum after 240 of 8,463 perturbed instances are filtered because their final answer is unchanged.The perturbations comprise seven modifications applied to each of 1,209 questions.Source: GSM8K-Platinum · p. 25

F.2 Code Reasoning and Understanding

  • CruxEval [Gu et al., 2024] supplies 800 code functions for input-prediction and output-prediction tasks.The two task variants are cruxeval-i and cruxeval-o.Source: §F.2 · p. 26

Adversarial Settings

  • For CruxEval self-reflection, each checkpoint’s incorrect generated CoT becomes the adversarial context, and evaluation uses questions all checkpoints in its parameter family answered incorrectly.The setup prepends “Wait,” to the question and context.Source: Adversarial Settings · p. 26
  • For CruxEval situational reflection, adversarial answers are checked against each task’s assertion, and generated adversarial CoTs are filtered and verified.The pipeline generates explanations and correct CoTs before constructing adversarial answers and CoTs; it retains 605 cruxeval-i and 704 cruxeval-o instances.Source: Adversarial Settings · p. 26

F.3 Knowledge Acquisition and Reading Comprehension

  • TriviaQA [Joshi et al., 2017] uses 17,944 validation questions with answer-bearing reference documents and confounding passages that are semantically similar but omit the correct answer.The situational setup limits total context length to 500 words, yielding 2,961 instances.Source: §F.3 · p. 26, Adversarial Settings · p. 27

F.4 Verbal, Logical, and Mathematical Reasoning

  • BBH [bench authors, 2023] covers 27 reasoning subtasks, while situational reflection uses incorrect or deliberately generated incorrect answers and CoTs as adversarial contexts.The setup appends “Wait,” after the question and adversarial CoT; self-reflection uses the checkpoint’s own incorrect answers.Source: §F.4 · p. 27, Adversarial settings · p. 27

G Reflection Classifier Annotations

  • Twenty-four annotators independently labeled 360 questions across GSM8K, cruxeval-o, and TriviaQA, and a structured review aligned labels with the operational definition of explicit reflection.Each question received annotations from two annotators; the reviewed labels were used to evaluate and calibrate the classifier.Source: G Reflection Classifier Annotations · pp. 27–28

H Situational Reflection Dataset Specific Metrics

Table 8 is identified as explaining the metrics and filtering strategies used across the datasets.

H Situational Reflection Dataset Specific Metrics

I Examples of Explicit Reflection

This section presents examples of explicit reflection in adversarial chains of thought across tasks and model checkpoints. The examples show models revisiting initial answers or reasoning in arithmetic, factual, object-counting, and spatial-reasoning prompts, with revisions that may be correct, mistaken, or unresolved.

I Examples of Explicit Reflection

  • In one CRUXEval example, the model revises its reasoning about a missing delimiter and gives an input that produces the requested output.It first proposes an input without the delimiter, then considers the case where the delimiter is present and returns an assertion using “bpxa24fc5.”Source: I Examples of Explicit Reflection · p. 29
  • In another CRUXEval example, the model corrects its interpretation of the function and changes its answer to 4.It recognizes that the function sums list lengths, rather than taking the maximum of the two lists.Source: I Examples of Explicit Reflection · p. 31
  • In a GSM8K example, the model catches that its initial radiator-cost total omitted the mechanic and revises the total to $230.It adds the $150 mechanic charge to the $80 radiator cost.Source: I Examples of Explicit Reflection · p. 32
  • In a corn-chip example, the model rejects its initial total of 210 and revises the answer to 225 after accounting for the extra 15 chips.The response explicitly adds 15 to 210.Source: I Examples of Explicit Reflection · p. 33
  • The truck revision was mistaken: 0.0375 trucks was rounded down to 0.037, though the passage says a fractional truck requires rounding up to a whole truck; 14.67 liters was correctly rounded to 15.Source: I Examples of Explicit Reflection · pp. 34–35
  • The Toby example revised a 30-minute reading-time answer to 20 minutes after accounting for the 10-minute trip to the airport.Source: I Examples of Explicit Reflection · p. 34
  • On the Victor Krum wand question, the response reconsidered its initial “J K Rowling” answer by wondering whether Ollivander made the wand, without resolving the answer.Source: I Examples of Explicit Reflection · p. 35
  • On the Sherlock Holmes housekeeper question, the model answered “Sir Arthur Conan Doyle” and then said it thought the housekeeper was Mrs Hudson.Source: I Examples of Explicit Reflection · p. 36
  • For the Pi prompt, the response questioned the meaning of “to three decimal places,” called the question ambiguous, and gave 3.142 and 3.141 under different interpretations.Source: I Examples of Explicit Reflection · p. 36
  • The model corrected its object count after noticing the couch was omitted, and corrected a spatial answer from the pink textbook to the teal cup.In another counting example, it said it had forgotten the onion even though its preceding count already included it.Source: I Examples of Explicit Reflection · pp. 36–38
Loading 2504.04022v1…