PaperlayerView exact revision on arXiv

Source-linked AI summary

Correct Answers, Invalid Traces: What Verifiable Grade-School Math Reveals About Chain-of-Thought Traces

Ratish Puduppully, Pranabendu Misra, Paarth Iyer, Durgesh Kalwar, Vardhan Palod, Subbarao Kambhampati

arXiv:2609.38107v1cs.CLcs.AI

TL;DR

The paper asks whether correct answers make natural-language thinking traces trustworthy records of model reasoning. Using mechanically checkable iGSM traces, it finds answer correctness and trace validity can diverge sharply, cautioning against treating visible traces as direct measurements of reasoning.

  • Problem

    Because natural-language thinking traces are rarely mechanically verifiable, it is unclear when they can be read as records of how models reached answers.

    Source: §1 · p. 2

  • Method

    The paper studies iGSM, where known problem quantities and dependencies allow programmatic checks of generated traces as derivations of final answers.

    Source: §3 · p. 4

  • Results

    Invalid traces among correct answers rise from below 1% in distribution to 31.6% on the hardest problems, and over half pass syntax and arithmetic checks.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Generated traces are observations of model behavior, not direct measurements of the reasoning process inside the model.

    Source: §5 · p. 9

    The experiments do not measure causal faithfulness or the accuracy of a safety monitor.

    Source: §5 · p. 9

Abstract

from arXiv · show

Chain-of-thought traces are widely read as records of how models reach their answers, informing debugging, agent auditing, and claims about reasoning. Testing this interpretation is difficult because natural-language thinking traces are rarely mechanically verifiable. We revisit it in iGSM, a synthetic grade-school mathematics benchmark designed to study thinking traces and used to support claims of learned reasoning and planning. Crucially, iGSM exposes the exact quantities and dependencies that a correct solution should use, allowing generated traces to be checked programmatically step by step and enabling us to test whether correct answers are reliably accompanied by valid traces. We first evaluate models trained exclusively on valid, minimal traces. Answer correctness and trace validity nearly coincide in distribution but decouple out of distribution: on the hardest instances, 31.6% of correct answers have invalid traces, over half of which pass all syntactic and arithmetic checks but fail semantic dependency checks. We then intervene on trace supervision. Non-minimal training traces induce non-minimal outputs, while re-asking the same problem with a different query reveals computations inherited from the original query, weakening minimality as evidence of selective planning. Shuffling tokens in 10% of training trace sentences preserves near-clean accuracy even out of distribution despite no trace passing verification. Swapped training traces likewise retain high in-distribution accuracy. We discuss the implications of these findings for chain-of-thought monitoring and interpretation in the context of AI safety.

1 INTRODUCTION

The introduction asks whether visible CoT traces can be trusted as records of model reasoning, then uses iGSM’s known quantities and dependencies to separate answer correctness from derivation validity. It reports that this separation widens under distribution shift and that trace features can reflect supervision rather than question-specific computation.

1 INTRODUCTION

  • CoT moved from elicited step-by-step responses into training, where STaR fine-tunes on generated traces that lead to correct answers.Modern outcome-based RL recipes reward answer correctness and output format without directly considering reasoning-trace content, unlike process supervision of intermediate steps.Source: §1 · pp. 1–2
  • Trace-based monitoring and interpretation rely on the assumption that emitted traces faithfully reflect, or correlate with, how models reached their answers.Thinking traces are also examined to interpret model and multi-agent behavior.Source: §1 · p. 2
  • iGSM exposes known quantities and dependencies for programmatic trace checks, testing whether a correct answer’s emitted trace is a valid derivation.Prior work has treated iGSM generalization, minimal solutions, and hidden-state probes as evidence of learned reasoning and planning.Source: §1 · p. 2
  • 31.6% of correct answers on the hardest problems have invalid traces; over half of those invalid traces pass syntactic and arithmetic checks.The models were trained only on standard valid, minimal traces.Source: §1 · pp. 2–3
  • As reasoning depth moves out of distribution, answer correctness and trace validity diverge despite valid-trace training and no incentive to conceal behavior.The gap grows as problem difficulty exceeds the training distribution.Source: §1 · p. 3
  • Non-minimal training traces change output minimality, while re-asking with a different quantity reveals computations inherited from the original query.The findings make minimality alone insufficient evidence that the model selected exactly the computations required by the current question.Source: §1 · p. 3
  • Swapped traces retain 82% in-distribution accuracy; 10% shuffled prefix-sentence tokens nearly preserve clean accuracy out of distribution despite no valid generated traces.More shuffling progressively harms generalization.Source: §1 · p. 3
  • An invalid emitted trace does not show that no useful reasoning occurred internally, and these experiments do not measure whether traces faithfully record the computation producing answers.The conclusions are deliberately narrower than claiming that chain-of-thought does not matter.Source: §1 · p. 3

2 RELATED WORK

Prior iGSM work treated out-of-distribution generalization, minimal solutions, and hidden-state probes as evidence of learned reasoning and planning. Related findings question whether traces reliably reflect reasoning, while this paper tests trace validity on iGSM and connects the issue to monitoring.

2 RELATED WORK

  • Prior work interpreted iGSM models’ out-of-distribution generalization, minimal solutions, and hidden-state probes as evidence of learned reasoning and planning.Later work studied error correction and extended the graph-based construction to larger problems.Source: §2 · p. 3
  • Earlier studies found plausible CoT explanations can omit causally influential factors, and correct answers can accompany semantically invalid traces in maze planning and question answering.Unlike benchmarks built for those studies, iGSM was an existing benchmark used in prior work with largely positive conclusions about trace semantics.Source: §2 · p. 3
  • Figure 1 contrasts a problem’s structure graph with its dependency graph, marking an unneeded stated parameter, abstract parameters, and RNG-supplied constants.The graph captions identify the visual encodings.Source: Figure 1 · p. 4
  • Figure 2’s running example includes an unused Estuary’s Walrus statement and two unstated hierarchy-derived totals in its reference trace.The displayed reference trace gives Salt Marsh’s Organs as 14.Source: Figure 2 · p. 4, §2 · p. 4
  • A monitor can identify problems in a trace only if the trace tells it enough about how the model reached its answer.The paper’s checker plays this monitoring role, and its results show traces can be invalid even when answers are correct.Source: §2 · p. 4

3 BACKGROUND

iGSM is a synthetic, controllable grade-school math benchmark whose problems retain the quantities and dependencies needed for reference solutions and automatic trace checking. Its structure and difficulty controls make it possible to specify which quantities a solution must compute and evaluate traces against those requirements.

3 BACKGROUND

  • iGSM retains each problem’s complete symbolic computation, enabling valid reference traces and automatic checking, while allowing controlled distributions and problem lengths.Its synthetic generator is estimated to yield more than 90 trillion distinct solution templates even when numerical values and several surface variations are ignored.Source: §3 · p. 4
  • An iGSM problem lists quantity statements in random order followed by a query, while its trace defines one quantity per sentence using previously defined quantities.All arithmetic is modulo 23; the example’s Walrus statement is unused, and two computed totals are not stated in the problem.Source: §3 · p. 4
  • The structure graph links adjacent entity-category layers, with edges representing instance parameters stated as constants or functions of other parameters.The problem text therefore fixes the structure graph’s edges.Source: §3 · p. 5
  • Abstract parameters are unstated hierarchy-derived totals; computing them across layers can require both sums and products.The example’s Manatee’s Organs sums organ types, while Salt Marsh’s Organs multiplies manatees per marsh by organs per manatee.Source: §3 · p. 5
  • The dependency graph is a directed acyclic graph recording which parameters are needed to compute each quantity, with RNG supplying constants.A valid reference trace computes the query’s dependency subgraph in topological order, excluding stated parameters outside that subgraph.Source: §3 · p. 5
  • iGSM difficulty is controlled by operation count and instance-parameter count, with training and out-of-distribution test ranges specified for iGSM-med and iGSM-hard.The hard setting trains through op = 21 and tests up to op = 32; the medium setting trains through op = 15 and tests at op ∈ {20, 21, 22, 23}.Source: §3 · p. 5
  • Trace checking verifies prior definitions, prescribed dependency inputs, arithmetic modulo 23, and whether the trace defines the queried parameter; answer correctness is checked separately.Using a quantity with the right value but the wrong prescribed dependency fails the dependency check, while the checker does not assess which operation combines two quantities.Source: §3 · p. 5

4 EXPERIMENTS AND RESULTS

Experiments on iGSM separate answer accuracy from trace validity and test how trace supervision shapes model outputs. Correct answers increasingly accompany invalid traces out of distribution, while trace minimality changes with training supervision and query phrasing.

Swapped traces

  • Swapped-trace models answer 82% of in-distribution problems correctly despite emitting no valid traces, but reach only 16–20% accuracy beyond the training distribution.Their accuracy is dominated by short problems and falls as operation count increases.Source: Table 1 · p. 6, §4 · p. 6

Shuffled prefixes

  • At p=10 shuffled-prefix training matches clean-model accuracy throughout, although no generated trace passes verification; greater corruption progressively harms generalization.At p=30, accuracy is 75% at op=15 and 44% at op=20, versus 25% and 18% for no-trace training.Source: Table 2 · p. 6, §4 · p. 6

Correct answers with invalid traces

  • 31.6% of correct answers at op=23 have invalid traces, as Figure 3 shows accuracy and trace-validity measures diverging beyond the training range.At op=15 and op=20, invalid traces accompany 0.5% and 7.5% of correct answers, respectively.Source: Figure 3 · p. 7, §4 · p. 7
  • 95.0%, 78.1%, and 51.2% of invalid traces accompanying correct answers at op=15, 20, and 23 are semantic failures that pass syntactic and arithmetic checks.These failures emerge only when steps are checked against the problem's dependency graph.Source: §4 · p. 7, Table 3 · p. 7

Non-minimal training traces

  • 2.5 unnecessary parameters per trace at op=15 and 40% minimal traces result from 100% non-minimal training, versus 99.9% minimal traces under standard training.With 10% minimal supervision, about three quarters of traces are minimal.Source: Table 4 · p. 8, §4 · p. 8

Is minimality a planning ability?

  • At op=20, re-asking raises padded traces from 0.4% to 3.7% in the standard setting and from 1.3% to 9.6% in the widened world.The extra computations tend to lie on the original chain to the sink, showing that minimality depends on how the question is posed.Source: Table 5 · p. 8, §4 · pp. 8–9

5 DISCUSSION

The experiments distinguish trace usefulness for producing answers from correctness as a derivation: shuffled or swapped traces can preserve accuracy, while clean-trace validity degrades with reasoning-depth shift.

5 DISCUSSION

  • 31.6% of correct answers at op = 23 have invalid traces, up from 0.5% at op = 15; half of the op = 23 invalid traces pass syntactic and arithmetic checks.The remaining failures point to lost problem-specific semantics despite plausible derivation form.Source: §5 · p. 9
  • With 30% of sentences shuffled, the model scores 75% at op = 15 versus 25% with no trace, while swapped traces support 82% in-distribution accuracy despite belonging to other problems.The shuffled and swapped interventions distinguish useful but invalid computation from ritual-like traces.Source: §5 · p. 9
  • The experiments measure neither causal faithfulness nor safety-monitor accuracy, and include no adversary, monitor in the loop, or incentive to conceal information.The authors caution against assuming reliability on familiar tasks transfers to harder or unfamiliar ones.Source: §5 · p. 9
  • Trace semantics matter for generalization, but useful intermediate computation need not match a correct, externally interpretable derivation token for token.The paper argues that trace validity, minimality, coherence, and problem-specific meaning can be shaped by supervision and problem construction, so reasoning mechanisms require counterfactual tests.Source: §5 · p. 9

6 CONCLUSION

In mechanically verifiable iGSM, correct answers can still carry invalid traces despite valid, minimal-trace training; interventions and model scale qualify what these results establish.

6 CONCLUSION

  • 31.6% of correct answers on the hardest problems have invalid traces, and over half of these pass all syntactic and arithmetic checks.The conclusion summarizes the increase from below 1% in distribution to 31.6% on the hardest problems.Source: §6 · p. 10
  • Minimality is shaped by supervision and query placement, while swapped or shuffled training traces can retain substantial answer accuracy despite producing no valid traces.The conclusion says non-minimal traces induce non-minimal outputs and re-asking with a different query reveals computations inherited from the original query.Source: §6 · p. 10
  • The models are small and synthetic, so the authors say direct extrapolation to frontier systems is unwarranted and reliability under distribution shift must be tested directly.They argue iGSM's exact verifiability makes the observed failures harder, not easier, to dismiss.Source: §6 · p. 10
  • The appendices provide training and evaluation details, additional runs and per-operation results, plus fifteen annotated example traces.Source: §6 · p. 10

AI USE STATEMENT

The authors report using generative AI tools for implementation and manuscript support, while excluding them from research design and result interpretation.

AI USE STATEMENT

  • Generative AI tools were used to implement methods, assist drafting and editing, search related work, and format references.Source: AI USE STATEMENT · p. 10
  • The authors say these tools did not propose research questions or hypotheses, design experiments, generate datasets, or interpret results.Source: AI USE STATEMENT · p. 10
  • The authors reviewed AI-assisted work and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A FURTHER RELATED WORK

Prior work finds that answer correctness and trace validity can separate, while filler, corrupted, or unrelated traces sometimes preserve performance. This paper situates its results alongside studies of longer problems, verifiable steps, benchmarks, and safety monitoring.

Traces, chain of thought and filler tokens

  • Filler tokens, relevant but incorrect demonstrations, and truncated or corrupted traces can preserve some benefits, while swapped traces look solution-like but belong to another problem.The cited studies report task-dependent effects, and filler training does not appear to recover standard serial chain-of-thought behavior.Source: A FURTHER RELATED WORK · p. 15

Verifying intermediate steps

Generalizing to longer and harder problems

  • Ye et al. (2025a) count only fully valid solutions; here, swapped-trace accuracy drops sharply against clean-trace models as operation counts grow.The paper contrasts this with maze settings where swapped-trace models generalize well, sometimes best, to out-of-distribution maze types.Source: A FURTHER RELATED WORK · p. 15

Grade-school math

  • GSM-Symbolic and GSM-Infinite score final answers only; GSM-Symbolic traces are not reliably program-verifiable, while GSM-Infinite's available graphs are not used to check traces.Source: A FURTHER RELATED WORK · pp. 15–16

Chain of thought in AI safety

  • Safety research uses traces to assess faithfulness, detect reward hacking, and interpret multi-agent behavior, but this study has no adversary or monitor in the loop.The authors note that invalid traces can still accompany correct answers in their setting.Source: A FURTHER RELATED WORK · p. 16

B EXPERIMENTS AND RESULTS

Clean-model answer accuracy is noisy across runs, while the share of correct answers with valid traces at op = 23 is stable. The evaluation uses a fixed iGSM setup and treats trace validity as passing every evaluator check.

Setup

  • The models are 124M-parameter, 12-layer GPT-2-style transformers trained for 100k steps on problems with op ≤15 and ip ≤20.Evaluation uses 4,096 fixed problems per level, with op ≤15 and op = 15 in distribution and op ∈ {20, . . . , 23} out of distribution.Source: B EXPERIMENTS AND RESULTS · p. 16
  • Because 14–16% of problems have answer 0, an always-zero model scores about 15%, so accuracies near that level are not statistically significant.Answers are integers modulo 23, making random-guess accuracy approximately 4.3%.Source: B EXPERIMENTS AND RESULTS · p. 16

Results

  • At op = 23, the share of correct answers with valid traces is 68% for run A, 68% for run B, and 71% for run C, despite answer-accuracy variation.Two recipe runs differ by ten points in answer accuracy at op = 23; doubling training to 200k steps changes answer accuracy from 58.0% to 58.5% and the share with both a correct answer and valid trace from 39.7% to 41.3%.Source: B EXPERIMENTS AND RESULTS · p. 16

C ADDITIONAL TABLES

The additional tables extend clean-model trace-failure counts across operation levels and document the scopes of several in-distribution comparisons. They also report substantial semantic-check failures among invalid traces accompanying correct answers.

Additional tables

  • At op = 15, 20, and 23, invalid traces accompanying correct answers that pass shallow but fail semantic checks account for 95.0%, 78.1%, and 51.2%, respectively.The corresponding shares at op = 21 and 22 are 63.2% and 55.1%.Source: C ADDITIONAL TABLES · pp. 16–17
  • Tables 6 and 7 cover in-distribution accuracy comparisons for clean, swapped, operation-matched, shuffled-prefix, and no-trace models.Table 6 reports operation-count accuracy for clean and swapped variants; Table 7 compares clean, shuffled-prefix, swapped, and no-trace models.Source: Table 6 · p. 17, Table 7 · p. 17
  • Table 8 organizes clean-model cases by trace validity and answer correctness, and reports that their correlation drops at larger operation counts.Table 10 compares clean-model accuracy with Ye et al.'s reported values under the same recipe; runs A and B differ by ten points at op = 23.Source: Table 8 · p. 17, Table 10 · p. 18
  • Table 9 extends the failure breakdown across five operation levels; its rows distinguish invalid traces among correct answers from failure-category shares among invalid traces.Its last column points to an annotated example for each row.Source: Table 9 · p. 18

D EXAMPLE TRACES

The examples illustrate how answer correctness and trace validity can diverge, and how training traces or re-asking a question can shape generated traces, including retaining computations from the original query.

D EXAMPLE TRACES

  • Examples 1–8 show correct clean-model answers with invalid traces, including semantic dependency errors and syntactic failures; some errors leave the answer unchanged.In one case, an incorrect dependency reaches intermediate values but later cancels, preserving the answer.Source: D EXAMPLE TRACES · pp. 18, 21, 22
  • Example 9 shows a valid trace with a wrong answer: it reads the right parameters but pairs them in the wrong products.The checker reports an answer of 22 instead of the true answer, 20.Source: D EXAMPLE TRACES · p. 23
  • Examples 10–11 show swapped-model traces naming parameters absent from the problems, despite producing correct answers.The checker rejects the traces because their named parameters do not occur in the respective problems.Source: D EXAMPLE TRACES · p. 23
  • Examples 12–13 show shuffled-prefix traces can yield either a correct answer after an unparsable prefix or a wrong answer after using an incorrect value.In the wrong-answer example, the later steps are arithmetically correct, but the first well-formed step uses the wrong input and the error reaches the answer.Source: D EXAMPLE TRACES · p. 24
  • Example 14 shows a correct answer from the non-minimal model with a valid trace containing unnecessary parameters.The supplied passage identifies this as a model trained on 90% non-minimal traces.Source: D EXAMPLE TRACES · p. 25
  • Both examples show valid traces padded with computations inherited from the original query after the question is re-asked.The examples describe correct answers; in the second, four steps define parameters unnecessary for the new question and belong to the original question’s chain.Source: D EXAMPLE TRACES · pp. 25–26
  • Example 15’s re-asked-question trace is valid but includes the original question’s chain.The passage characterizes the answer as correct and the trace as padded with that original chain.Source: D EXAMPLE TRACES · p. 25
  • In the Niagara Falls Aviary example, four unnecessary steps follow the original “Mahi Mahi’s Bone” question rather than the re-asked “Niagara Falls Aviary’s Animal” query.The trace is marked valid and gives the answer 8.Source: D EXAMPLE TRACES · p. 26
Loading 2609.38107v1…