PaperlayerView exact revision on arXiv

Source-linked AI summary

Shape of Thought: When Distribution Matters More than Correctness in Reasoning Tasks

Abhranil Chandra, Ayush Agrawal, Arian Hosseini, Sebastian Fischmeister, Rishabh Agarwal, Navin Goyal, Aaron Courville

arXiv:2512.22255v2cs.AIcs.LG

TL;DR

The paper asks whether reasoning-data quality depends only on final-answer correctness or also on how well traces match a model’s distribution. It studies supervised fine-tuning on synthetic and human traces, finding that even incorrect-answer synthetic traces can improve reasoning and often outperform fully correct human-written traces.

  • Problem

    Reasoning-data construction commonly treats final-answer correctness as the primary quality signal, leaving unclear whether closer-to-model but imperfect traces can directly help reasoning.

    Source: §1 · pp. 1–2

  • Method

    The authors compare supervised fine-tuning on human-written, synthetic-correct, and synthetic-incorrect CoTs, and test paraphrasing and progressively flawed traces across models and reasoning tasks.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Across MATH, GSM8K, Countdown, and MBPP, synthetic incorrect-answer CoTs can improve reasoning and often surpass fully correct human-written traces; on MATH, G-2B and L-8B reach max accuracies of 23% versus H baselines of 17% and 19%, respectively.

    Source: §5.1 · p. 6, §6 · p. 11

  • Takeaways & Limitations

    The findings indicate that CoT distribution relative to the student model matters alongside final-answer correctness, and that an incorrect answer does not necessarily mean every reasoning step is unusable.

    Source: §1 · p. 2, §6 · p. 11

    The evaluation primarily uses final-answer accuracy, leaving a need for methods that verify step-level correctness in natural-language reasoning.

    Source: §6 · p. 11

Abstract

from arXiv · show

We present the surprising finding that a language model's reasoning capabilities can be improved by training on synthetic datasets of chain-of-thought (CoT) traces from more capable models, even when all of those traces lead to an incorrect final answer. Our experiments show this approach can yield better performance on reasoning tasks than training on human-annotated datasets. We hypothesize that two key factors explain this phenomenon: first, the distribution of synthetic data is inherently closer to the language model's own distribution, making it more amenable to learning. Second, these `incorrect' traces are often only partially flawed and contain valid reasoning steps from which the model can learn. To further test the first hypothesis, we use a language model to paraphrase human-annotated traces -- shifting their distribution closer to the model's own distribution -- and show that this improves performance. For the second hypothesis, we introduce increasingly flawed CoT traces and study to what extent models are tolerant to these flaws. We demonstrate our findings across various reasoning domains like math, algorithmic reasoning and code generation using MATH, GSM8K, Countdown and MBPP datasets on various language models ranging from 1.5B to 9B across Qwen, Llama, and Gemma models. Our study shows that curating datasets that are closer to the model's distribution is a critical aspect to consider. We also show that a correct final answer is not always a reliable indicator of a faithful reasoning process.

1 INTRODUCTION

The introduction questions whether final-answer correctness is the best guide to reasoning-data quality, given that model-generated traces may better match the student model’s distribution. It studies whether incorrect-answer traces can still support learning and why.

1 INTRODUCTION

  • The introduction challenges the assumption that correctness determines data quality, noting that human-written traces may be distributionally distant and incorrect synthetic traces are often discarded.Prior work had not thoroughly tested whether incorrect traces directly improve math reasoning.Source: §1 · pp. 1–2
  • Synthetic traces may be easier to learn from because they are closer to the student model’s distribution, and many wrong-answer traces retain valid reasoning steps.These include correct intermediate steps, valid decompositions, and useful patterns.Source: §1 · pp. 1–2
  • The study asks whether model-generated traces with incorrect final answers can directly improve reasoning and whether distributional closeness should outweigh perfect correctness in data selection.The comparison is between fully correct human-written traces and closer-to-model, potentially imperfect synthetic traces.Source: §1 · p. 2
  • Across MATH, GSM8K, Countdown, and MBPP, synthetic incorrect-answer CoTs can improve reasoning and sometimes outperform fully correct human-written traces.The study tests these findings across Gemma, Llama, and Qwen models.Source: §1 · p. 2
  • Paraphrasing human solutions to better match the model distribution improves reasoning scores, while progressive CoT degradation probes error tolerance and final-answer checking’s limits.The paper also qualitatively examines generated traces and argues final-answer checking is not a holistic measure of CoT quality.Source: §1 · pp. 2–3

2 BACKGROUND AND PRELIMINARIES

The background frames supervised fine-tuning and noisy-data learning as context for studying how language models might extract reasoning patterns from imperfect demonstrations.

2.1 LARGE LANGUAGE MODELS AND SUPERVISED FINE-TUNING

  • An autoregressive language model predicts each next token conditioned on the preceding sequence, with a softmax over output logits and temperature controlling generation randomness.The model assigns probability to a text sequence by factorizing it into conditional token probabilities.Source: §2.1 · p. 3
  • Supervised fine-tuning adapts a pretrained model using labeled input-output pairs by minimizing cross-entropy loss, while reasoning distillation trains on step-by-step solutions from the same or another model.The paper extends this line of work by studying distillation from incorrect as well as correct CoTs.Source: §2.1 · p. 3

2.2 LEARNING FROM NOISY DATA IN LARGE NEURAL NETWORKS

  • Neural networks can learn consistent, low-frequency patterns before fitting noise, but localized mistakes are less damaging than logical errors that corrupt later steps.Prior work cited in the passage finds LLMs robust to many localized miscalculations but sensitive to small amounts of logical error.Source: §2.2 · p. 3

2.3 PARALLELS TO LEARNING FROM NOISY TEACHERS IN HUMANS

  • The paper draws an analogy to children regularizing inconsistent language input, and hypothesizes that LLM priors may similarly help extract reliable reasoning from imperfect demonstrations.The cognitive-science discussion presents this as a hypothesis about robustness to noisy learning data.Source: §2.3 · p. 4

3 RELATED WORK

The related work situates the study within reasoning-data curation and the use of negative traces, emphasizing its focus on direct supervised learning from imperfect CoTs.

Data-Centric Approaches for Enhancing Reasoning

  • Distillation research and large reasoning datasets emphasize high-quality traces and sample-efficient question design, while the paper studies unfiltered synthetic data across a broader set of settings.Prior findings largely focus on domains and datasets with available verifiers.Source: §3 · p. 4

Using Negative Data for Learning

  • Prior work uses incorrect CoTs mainly for preference learning or verifier training; this study instead tests direct SFT on incorrect-answer traces and highlights distributional differences from human traces.The passage also relates the study to work on imperfect reward signals and synthetic-CoT structure.Source: §3 · p. 4

4 EXPERIMENTAL SETUP

The experiments compare human-written, synthetic-correct, and synthetic-incorrect reasoning traces across several models, reasoning tasks, and evaluation settings.

4 EXPERIMENTAL SETUP

  • The experimental design targets the contribution of generated-CoT distribution and intermediate reasoning steps to base-model reasoning performance.This setup motivates comparisons among the three data categories across the selected tasks and models.Source: §4 · p. 4
  • The study uses Gemma-2-2B, Llama-3.1-8B, and Qwen2.5-1.5B base models, with Gemma-2-9B added to examine scaling.The larger Gemma model is compared with Gemma-2-2B.Source: §4 · p. 5
  • Evaluation covers MATH and GSM8K mathematics, Countdown arithmetic, and MBPP code generation, using the tasks’ stated test sets or subsets.Countdown is used to examine learning from incorrect synthetic solutions in a relatively harder task setting.Source: §4 · p. 5
  • Synthetic CoTs are sampled from the same or more capable models, classified as Gold or Wrong by final-answer verification, and selected alongside human traces for training.The generation pipeline samples 64 solutions at temperature 0.8 and constructs task-specific datasets; Countdown’s Wrong traces must follow task rules.Source: §4 · p. 5
  • The study fine-tunes on human-written, synthetic-correct, and synthetic-incorrect datasets, then evaluates with greedy decoding using final-answer accuracy or code Pass@1.The authors note that manually checking CoT correctness for all generated outputs is impractical.Source: §4 · p. 5

5 RESULTS AND DISCUSSION

Across reasoning and code-generation tasks, synthetic traces from a stronger model outperform human-written traces, including traces with incorrect final answers. The experiments also find benefits from distribution-matched human traces, tolerance to some flawed steps, and reusable reasoning within incorrect traces.

5.1 RESULTS ON ALL TASKS

  • On MATH, GSM8K, and MBPP, both correct and incorrect synthetic CoTs from G-27B-IT outperform human-written CoTs.For G-2B and L-8B, MATH accuracy reaches 23%, versus human-written baselines of 17% and 19%, respectively.Source: §5.1 · p. 6, Table 2 · p. 7, Table 3 · p. 7
  • On Countdown, correct synthetic CoTs yield larger gains than incorrect ones, yet models still learn from incorrect traces, including with near-zero baselines.The reported gains reach up to +20% absolute for G CoTs and up to +10% absolute for W CoTs over the four-shot baseline.Source: §5.1 · pp. 6–7, Figure 4 · p. 7
  • Qwen-2.5-1.5B shows limited fine-tuning gains on MATH and GSM8K but improves on Countdown; both synthetic trace types also improve MBPP results over human traces.The passage attributes limited math-task gains to Qwen’s already strong baselines and describes Countdown as a harder, out-of-distribution task.Source: §5.1 · p. 8

5.2 SCALING EXPERIMENT

  • At 9B scale, both correct and incorrect synthetic CoTs provide consistent gains over human-written traces on MATH.Scaling from 2B to 9B also improves baseline accuracy.Source: §5.2 · p. 8, Table 5 · p. 9

5.3 BRIDGING THE GAP: PARAPHRASING HUMAN COTS FOR BETTER MODEL ALIGNMENT

  • Paraphrasing human solutions lowers their perplexity under the base model from 3.996 to 3.873 and raises G-2B MATH accuracy from 17% to 20%.For G-9B, paraphrased traces outperform all other training datasets, including G traces; MBPP results also improve across iterations.Source: §5.3 · p. 9, Table 5 · p. 9

5.4 HOW TOLERANT ARE LMS TO ERRORS?

  • On Gemma-2B, finetuning with up to 25% completely flawed CoTs yields performance comparable to human-written CoTs (+0.01 accuracy difference), with a 0.05 absolute drop versus G CoTs.Accuracy declines steadily beyond 25% flawed CoTs, with a maximum 0.09-point drop when all CoTs are flawed. Model-generated CoTs have lower starting training losses (0.55–0.59) than human-written CoTs (0.86).Source: §5.4 · p. 9, Table 6 · p. 10
  • G-27B-IT-W performance is comparable to mixing 25% flawed CoTs with correct G traces, indicating that incorrect-answer traces retain useful training signals.The experiments describe W traces as containing signals that should not be discarded.Source: §5.4 · pp. 9–10

5.5 BEYOND FINAL ANSWER CORRECTNESS: UNDERSTANDING THE STRUCTURE OF COTS

  • Both correct-answer and incorrect-answer CoT datasets mix correct and incorrect reasoning steps, so an incorrect final answer does not make a trace random noise.The paper proposes that correct, reusable steps can provide learning signals for general reasoning patterns despite localized errors.Source: §5.5 · p. 10

Synthetic CoT Generated by G-27B-IT leading to correct final answer (G)

The example shows that a correct final answer can conceal double-counting, while sampled correct-answer and wrong-answer traces contain mixtures of usable and flawed steps.

Synthetic CoT Generated by G-27B-IT leading to correct final answer (G)

  • W CoTs had minor problems in 6/10 G-27B-IT samples and 7/10 Q-1.5B samples, while G CoTs were fundamentally flawed in 4/10 Gemma and 2/10 Qwen samples.The authors interpret both datasets as mixtures of correct and incorrect reasoning steps.Source: Synthetic CoT Generated by G-27B-IT… · p. 11
  • In the counting example, the model double-counts cases in its G CoT but reaches the correct total of 9 by chance.The worked solution counts three choices in each of three overlapping cases.Source: Synthetic CoT Generated by G-27B-IT… · pp. 10–11
  • The authors argue that useful reasoning steps in both G and W traces can support SFT despite localized errors.They relate increasing flaws to a lower proportion of correct steps and the observed performance decline.Source: Synthetic CoT Generated by G-27B-IT… · p. 11

6 CONCLUSIONS, LIMITATIONS & FUTURE WORK

Across four benchmarks and three model families, synthetic CoTs—even those ending in incorrect answers—can improve reasoning and often surpass fully correct human-written traces. The paper attributes this pattern to distributional similarity and reusable correct steps, while noting limits in evaluation and scope.

CONCLUSIONS

  • Across MATH, GSM8K, MBPP, and Countdown, model-generated CoTs that all end in incorrect answers can improve reasoning and often surpass fully correct human-written traces.The study covers Gemma, Llama, and Qwen models from 1.5B to 9B parameters.Source: §6 · p. 11
  • Paraphrasing human CoTs to better match the model distribution improves performance, while controlled error introduction produces steady degradation.The paper identifies closer distributional alignment and reusable partially correct steps as explanations for the findings.Source: §6 · p. 11

LIMITATIONS & FUTURE WORK

  • The study covers SFT, relies mainly on final-answer accuracy and indirect distribution proxies, and cautions that scaling beyond 9B may differ.The authors call for step-level correctness verification and study of reinforcement learning and principled distribution alignment.Source: §6 · p. 11

A.1 HYPERPARAMETERS AND COMPUTE

The appendix reports the compute setup and shared training settings used across the experiments.

A.1 HYPERPARAMETERS AND COMPUTE

  • Experiments ran on 4 A100 80GB GPUs and 8 A100 40GB GPUs.These are the reported compute resources for all experiments.Source: §A.1 · p. 14
  • G-2B and G-9B used batch size 64 and learning rate 2e-5; L-8B and Q-1.5B used batch size 256 and learning rate 1e-6.The batch size and learning rate differ by model group.Source: §A.1 · p. 14
  • All experiments used linear learning-rate decay with 0.1 warmup, weight decay 0.05, adamw_torch, and 10 training epochs.Source: §A.1 · p. 14

A.2 ACCURACY PLOTS AND OTHER RESULTS

Figures 7–16 provide accuracy and training-loss plots for all iterations across base models and test benchmarks.

A.2 ACCURACY PLOTS AND OTHER RESULTS

  • Figures 7–16 show accuracy plots for all iterations across the base models and test benchmarks.Source: §A.2 · p. 14
  • Figures 7–16 also show training-loss plots for all iterations across the base models and test benchmarks.Source: §A.2 · p. 14
  • The plots cover all base models and all test benchmarks.Source: §A.2 · p. 14

A.3 ERROR TOLERANCE SETUP

The authors generated completely flawed CoTs by prompting G-27B-IT for incorrect solutions and retaining traces with incorrect final answers.

A.3 ERROR TOLERANCE SETUP

  • They prompted G-27B-IT to generate completely flawed CoTs.Source: §A.3 · p. 14
  • They generated one sample per problem at a temperature of 0.8.Source: §A.3 · p. 14
  • When a requested incorrect solution produced a correct final answer, they sampled more solutions and retained ones with incorrect final answers.Source: §A.3 · p. 14

EXAMPLE OF A FLAWED COT GENERATED BY G-27B-IT

The example’s lemonade solution uses an incorrect ratio procedure and arrives at 9.26 lemons for four gallons.

EXAMPLE OF A FLAWED COT GENERATED BY G-27B-IT

A.4 PARAPHRASED COTS

This appendix examines paraphrased and synthetic chain-of-thought traces through examples of correct and flawed reasoning, alongside prompting and paraphrasing procedures and robustness checks. It illustrates that paraphrased human traces can improve performance and resemble their originals, while final-answer correctness may mask flawed reasoning and traces with wrong answers may still contain valid steps.

A.4 PARAPHRASED COTS

A.5 LOSS CURVES FOR GSM8K

  • Paraphrased human CoTs improved performance over the originals for G-2B and L-8B, while accuracy generally declined as CoTs were made more erroneous.G outperformed W across Llama-3.1-8B, Qwen-2.5-1.5B, and Gemma-2-2B; W nevertheless improved the base models, and Qwen training was unstable in one comparison.Source: Figure 11 · p. 18, Figure 12 · p. 18, Figure 13 · p. 18

Example I: G-27B-IT-G

Example II: G-27B-IT-G

  • For the nested-sum example, G’s mis-scaled reasoning reaches 4002 by coincidence, whereas W’s faulty factor and subtraction produce 4003.The supplied excerpts identify the G calculation as invalid despite its correct value, and the W final value as incorrect.Source: Example II: G-27B-IT-G · p. 22, Example II: G-27B-IT-W · p. 22

Example I: Q-14B-IT-G

Example II: Q-14B-IT-G

Countdown G-27B-IT-G — Target 57 from [50, 42, 63, 86]

GSM8K G-27B-IT-G; GSM8K G-27B-IT-W

  • The G example correctly calculates 21 kilos remaining, while W retains correct division steps but miscounts untouched bags and gives 12 kilos.The W trace identifies the correct total as 21 kilos after noting that each untouched bag still contains 6 kilos.Source: GSM8K G-27B-IT-G · p. 25, GSM8K G-27B-IT-W · p. 26

MATH PROBLEM SOLVING (ZERO-SHOT) PROMPT; MATH PROBLEM SOLVING (FOUR-SHOT) PROMPT; GSM8K PROBLEM SOLVING (ZERO-SHOT) PROMPT; GSM8K PROBLEM SOLVING (FOUR-SHOT) PROMPT; COUNTDOWN PROBLEM SOLVING (ZERO-SHOT) PROMPT; COUNTDOWN PROBLEM SOLVING (FOUR-SHOT) PROMPT

Prompt for Math Paraphrasing

  • The math paraphrasing prompt asks a model to restate a correct hinted solution in its own style while preserving its logical steps and final answer.It says not to generate a new solution, but to adhere to the hint.Source: Prompt for Math Paraphrasing · p. 31

PROMPT FOR GENERATING COMPLETELY FLAWED REASONING

  • The flawed-reasoning prompt requests problem-based solutions built from invented formulas and incorrect reasoning that end in an incorrect answer.It instructs the model not to disclose that the solution is intentionally incorrect.Source: PROMPT FOR GENERATING COMPLETELY… · p. 31

A.8 STATISTICAL SIGNIFICANCE AND HYPERPARAMETER ABLATIONS

  • Synthetic G and W CoTs outperform human H CoTs on MATH500 across the tested hyperparameter settings, and the reported trends hold across five random seeds.The study reports mean ± standard deviation and 95% confidence intervals over five seeds; the hyperparameter ablation varies batch size and learning rate.Source: §A.8 · p. 31, Table 11 · p. 31, Table 12 · p. 32

MBPP CODE GENERATION PROMPT

  • The MBPP code-generation prompt requests clean, efficient Python using only the standard library, satisfying the supplied assertion, and returning code only.Source: MBPP CODE GENERATION PROMPT · p. 32

Prompt for Code Paraphrasing

  • The code paraphrasing prompt asks for a stylistic rewrite of a correct hinted solution while strictly preserving its logic, functionality, and function signature.It says not to generate a new solution.Source: Prompt for Code Paraphrasing · p. 32

Related papers

Loading 2512.22255v2…