PaperlayerView exact revision on arXiv

Source-linked AI summary

Beneath the Surface of Chains-of-Thought: A Mechanistic Interpretation of Reasoning Operations in LLMs

Seogyeong Jeong, Jaehui Hwang, Dongyoon Han, Geonmo Gu, Alice Oh, Taekyung Kim

arXiv:2609.04753v1cs.CL

TL;DR

The paper asks how explicitly distinguished reasoning operations are organized in LLM hidden representations. It analyzes operation-labeled reasoning traces across models and layers, finding separable geometric structures, distributed span-level signals, and dependence on preceding context. These results support a correspondence between linguistic reasoning operations and internal geometric organization.

  • Problem

    It remains unclear how distinct reasoning operations in chain-of-thought traces are organized in representation space and whether their structure generalizes beyond lexical and problem-specific content.

  • Method

    The study applies a Polya-based operation taxonomy to generated reasoning traces and analyzes hidden representations across mathematical reasoning datasets, model families, and layers.

  • Results

    Reasoning operations form separable hidden-representation structures, peak in separability at middle layers, distribute across spans, contextualize identical tokens, and depend on preceding reasoning context.

  • Takeaways & Limitations

    The findings support a representational correspondence between explicitly expressed reasoning operations and internal geometric organization in language models.

  • Takeaways & Limitations

    Operation annotations are approximate labels of textually expressed reasoning functions, based on GPT-5 generation and human validation over a limited subset of 84 spans.

Abstract

from arXiv · show

Reasoning in large language models unfolds through diverse functional operations, such as problem formulation, goal decomposition, and deduction. Although these operations are explicitly distinguished in text, little is known about how they are geometrically organized in representation spaces. To this end, we investigate whether distinct reasoning operations exhibit corresponding geometric structure in hidden representations. We find that operations are separable in held-out representations, with separability peaking in middle layers, and verify that this structure is not explained by lexical or positional confounds. Across layers, token-wise operation-alignment becomes more distributed over spans, while identical surface tokens are represented differently depending on the operation of its surrounding chunk. Attention-masking interventions further show that operation-aligned representations at chunk onset depend on preceding reasoning context. Consequently, our work demonstrates that language models maintain representational correspondence between linguistic reasoning expressions and their internal geometric structures. Code and project materials are available at https://github.com/naver-ai/beneath-cot.

1 Introduction

The paper studies whether reasoning operations in chain-of-thought traces have shared geometric structure in hidden representations beyond lexical content. It analyzes these representations across models and finds separable, context-dependent operation signals that peak in middle layers.

  • Research questions: The study asks whether hidden representations encode the reasoning operation being performed beyond the current token’s lexical identity.It also examines whether the same operation shares representational structure across problems and reasoning contexts.
  • Research questions: It investigates whether distinct operations are organized geometrically and whether that organization generalizes across tokens, problems, and model layers.These questions concern operation-level structure in representation space.
  • Approach: The analysis applies a Polya-based taxonomy to generated traces and examines hidden representations from mathematical reasoning datasets across Qwen and Gemma models.The datasets include DAPO-MATH-17K and TheoremQA.
  • Findings: Reasoning operations are separable in held-out hidden representations, with separability peaking in middle layers across models and operation types.The introduction identifies this as a central finding alongside context-dependent operation signals and weakening under factual errors.
  • Findings: Operation signals distribute across spans, identical surface tokens receive operation-specific representations, and preceding context contributes to subsequent operation representations.These findings connect token-level behavior, span-level structure, and contextual dependence.
  • Implication: The findings suggest a representational correspondence between explicitly expressed reasoning operations and the model’s internal geometric organization.The paper presents this correspondence as a lens for examining LLM cognition.

2 Preliminary

The paper represents a reasoning trace as functional operation chunks rather than a homogeneous token sequence. It defines a Polya-based taxonomy and analyzes hidden representations of these operations in reasoning models.

  • Representation setup: For each token position, the analysis extracts a layer-specific hidden representation to study how geometry reflects functional reasoning roles.The representations are vectors in the model’s hidden space.
  • Operation taxonomy: A reasoning trace is modeled as contiguous operation chunks, each serving a specific functional role in problem solving.The taxonomy emphasizes functional role rather than surface wording alone.
  • Operation taxonomy: The taxonomy is based on Polya’s four-stage problem-solving structure and supports a broader set of operation types and subtypes.The main analysis focuses on eight recurring operation types across generated traces.
  • Operation labels: The eight frequent operation types are used as span labels for representation analysis.Table 1 provides an overview of the main operations used in the analyses.
  • Operation examples: The operations include extraction, direct mapping, decomposition, recall, deduction, algebraic manipulation, and arithmetic computation across understanding, planning, and execution stages.These labels capture distinct functional steps in solution construction.

3 Geometric Structure of Reasoning Operations in Reasoning LLMs

Reasoning operations form separable structures in hidden representations across models, with strongest expression in middle layers and robustness beyond lexical and positional cues. Their signals become distributed across spans, contextualize identical tokens, and depend on preceding reasoning context, while remaining detectable but weakened under factual errors.

  • Operation annotation: 96.4% of annotated spans reached human-majority agreement, supporting aggregate analyses despite nonnegligible annotation uncertainty.Human-majority agreement was defined by at least two annotators agreeing on a canonical operation label.
  • Held-out separability: Reasoning operations are separable in held-out hidden representations across Qwen2.5-7B, Qwen3-8B, and Gemma4-31B, with strongest separability in middle layers.The analysis uses middle-token representations and reports consistently high peak-layer one-vs-rest AUROC across operations and models.
  • Robustness to confounds: Hidden-state probes capture operation-relevant information beyond lexical content, while lexically similar, position-controlled, and adversarial evaluations preserve strong operation prediction.Across models, hidden-state probes outperform text-only baselines; competing-operation vocabulary and matched-position tests remain strongly predictive.
  • Robustness and generalization: Operation-level separability persists without supervised LDA and transfers across models and datasets, including Llama-3-8B, GPQA-Diamond, and MATH-500.With 128 principal components, macro AUROC/AUPRC remains 0.938/0.716 for Qwen3-8B; Qwen3-8B probes transfer without task-specific retraining to GPQA-Diamond and MATH-500.
  • Within-span geometry: Operation-alignment becomes distributed across tokens within spans in middle layers, whereas early-layer signals are more cue-local and late-layer variance rises slightly.Shared surface tokens also separate by surrounding operation in middle-to-late layers, showing that operation alignment is not determined by lexical identity alone.
  • Context dependence: Masking the preceding 30 tokens reduces target operation-alignment at reasoning-chunk onset, indicating causal dependence on immediately preceding context.The intervention tests the same fixed generated sequence while masking attention from the first target-chunk token to preceding context.
  • Factual errors: Under factual errors, operation geometry remains detectable but aggregate separability is modestly attenuated, with the clearest reductions for Deduction and Arithmetic Computation.Mean-pooled factual-error spans achieve macro AUROC/AUPRC of 0.955/0.877 versus 0.971/0.901 for operation-matched non-error spans.

4 Related Work

This work connects research on structured reasoning traces with research on geometric structure in LLM representations by analyzing recurring local reasoning operations in hidden space.

  • Prior studies characterize reasoning traces through influence, cognitive behavior, hierarchical episodes, and discourse structure.
  • Prior representation studies find structured geometry for semantic and behavioral concepts, while reasoning models’ hidden states encode correctness and other reasoning signals.
  • Unlike trajectory-level behaviors spanning multiple steps, this work analyzes local operation spans that can recur at different positions within a reasoning trajectory.

5 Conclusion

The study finds that textual reasoning operations correspond to distinct, context-dependent geometric structures in LLM hidden representations. Its interventions indicate that these representations incorporate preceding reasoning context.

  • Reasoning operations form separable clusters in hidden space, and identical tokens receive operation-specific representations from their surrounding reasoning context.
  • Causal interventions show that operation representations are not locally self-contained but emerge through information propagation from preceding context.
  • The findings support viewing reasoning as a structured, layered, and context-dependent process with correspondence between linguistic expressions and internal geometry.

6 Limitations

The study’s conclusions are constrained by approximate operation annotations, limited task and model coverage, and a primarily diagnostic analysis.

  • Operation labels come from GPT-5 and were human-validated on only 84 spans, with validation focused on labels rather than span boundaries.The authors therefore treat annotations as approximate labels of expressed reasoning functions, not ground-truth latent cognitive states.
  • Experiments cover mathematical and theorem-driven tasks and a small set of reasoning-oriented LLMs, limiting evidence about other tasks and architectures.The authors note that observed geometry may differ in commonsense reasoning, planning, code generation, interactive tasks, or differently trained models.
  • The analysis is primarily diagnostic and does not evaluate whether learned operation vectors improve model behavior.Suggested future applications include failure detection, verification, decoding-time control, and activation-based steering.

7 Ethical Considerations

The supplementary analyses visualize operation-alignment across generated tokens and test whether span-distribution patterns generalize across model families. They show increasingly coherent middle-layer signals and context-dependent token scores.

  • Token-level heatmaps visualize whether operation signals are localized to cue tokens or sustained across broader generated segments.
  • Operation-specific scores are often localized in early layers, become more coherent across contiguous tokens in middle layers, and weaken as operation-specific signals in later layers.
  • Identical surface tokens can receive different operation-score intensities depending on their surrounding reasoning context.The comma token is cited as an example of differing scores across repeated occurrences in one trace.
  • Qwen2.5-7B and Gemma4-31B show the same qualitative intra-span variance pattern as Qwen3-8B: high early, lower middle, and slightly higher later.This cross-model recurrence supports the interpretation that middle-layer operation signals become more distributed across reasoning chunks.

A.3 Exploratory Temporal Ordering of Early- and Late-Layer Signals

The exploratory analysis finds operation-dependent differences in when sustained late- and early-layer alignment signals appear, with the clearest late-before-early tendency for Arithmetic Computation and Final Answer. Because signal shapes differ across regimes, this temporal characterization is detector-dependent.

  • Mean onset difference is negative for all eight operations, but the ordering’s strength and consistency vary substantially.
  • 77.0% of Arithmetic Computation and 87.5% of Final Answer examples show late-layer signals detected before early-layer signals.
  • Extraction, Decomposition, and Algebraic Manipulation have negative median onset differences, but negative examples are closer to one half.
  • Direct Mapping, Recall, and Deduction have median onset differences of zero, and fewer than half of examples show negative differences.Significant Wilcoxon results do not imply that late-layer onset occurs first in a majority of examples.
  • Early-layer signals are spike-like with higher token-level variance, whereas later-layer signals are more plateau-like across contiguous tokens.The same three-token averaging and thresholding procedure therefore may have different detection sensitivity across regimes.
  • The analysis is supplementary and detector-dependent, so it does not establish or rule out autoregressive cue propagation.

B.1 Additional Context-Intervention Results

Additional context interventions preserve the same ordering: masking immediately preceding tokens reduces operation-alignment scores most, followed by preceding chunks and random blocks. This supports a stronger role for local preceding context, with Recall as an exception.

  • Score decreases follow the ordering random < preceding-chunk < preceding-token.
  • The ordering suggests that target operation representations depend most strongly on immediately preceding local context, while prior chunks contribute more diffuse information.Random preceding blocks have weaker effects, indicating that reductions are not simply caused by removing arbitrary past tokens.
  • Recall is an exception: random-control masking produces a relatively large decrease, suggesting dependence on broader accumulated context.
  • The intervention pattern is reported as an additional result for probes applied to Llama-3-8B and for Qwen3-8B probes transferred to GPQA-Diamond and MATH-500 without task-specific retraining.

C.3 Operation Geometry under Factual Errors

Operation identity remains strongly detectable in reasoning spans containing factual errors, but aggregate separability is significantly lower than in operation-matched non-error spans. The effect varies by operation and representation choice.

  • Mean-pooled representations reach macro AUROC/AUPRC of 0.955/0.877 in factual-error spans, while middle-token representations reach 0.920/0.759.
  • Aggregate operation separability is significantly lower in factual-error spans than in operation-matched non-error spans under both representation choices.
  • Deduction and Arithmetic Computation show significant reductions in both AUROC and AUPRC for both representation choices.
  • Recall shows no significant reduction, while Algebraic Manipulation and Final Answer show representation- or metric-specific reductions.

D.1 Robustness to Projection Choice: PCA-Only Evaluation

Operation labels remain substantially separable without supervised LDA, showing that the central held-out separability result does not depend on the projection choice. LDA instead sharpens the structure in a compact supervised coordinate system.

  • Mean-pooled PCA-only macro AUROC ranges from 0.846 to 0.908 with 8 components and from 0.872 to 0.938 with 128 components.
  • The cited Table C caption describes test-only transfer of Qwen3-8B probes to GPQA-Diamond and MATH-500 without task-specific retraining.
  • LDA provides a compact supervised coordinate system that sharpens the structure, but held-out separability does not depend on LDA.

D.2 Robustness to Span-Representation Choice

Reasoning-operation separability is robust to span-representation choices and remains distinct from lexical and positional explanations. Alternative pooling and control analyses preserve the main pattern, while aggregation affects when signals become detectable.

  • Representation choice: Across alternative span representations, operation labels remain more separable than random baselines across models and layers.The evaluated representations include middle-token, mean-pooled, first-token, and last-token spans.
  • Layer-wise robustness: Separability generally increases from early to middle layers and remains above baseline across operation types and token positions.Mean pooling often produces stronger separability in earlier layers by aggregating within-span information.
  • Positional controls: The hidden-state probe outperforms the position-only predictor for every operation in all three models.Position contributes to some labels but does not fully explain the operation-level hidden-state structure.
  • Lexical controls: Lexical content is informative, but hidden-state probes outperform text-only baselines in aggregate; Decomposition is the main operation-level exception.For Decomposition, text-only AUPRC slightly exceeds hidden-state AUPRC.
  • Lexical controls: Lexically matched, competing-vocabulary, and numerically dense controls preserve operation separability, although lexical variation is not completely removed.Arithmetic Computation remains distinguishable in the numerically dense subset.
  • Statistical robustness: Across three models, all 48 AUROC confidence intervals remain above the 0.5 chance level, and all 96 permutation tests are significant at p < 0.001.These results support stability under held-out resampling and random-label controls.

E.3 Human Validation of Operation-Span Annotations

The paper validates operation-span annotations with multi-annotator judgments and documents the representation-analysis pipeline used to evaluate them. Human agreement is high for majority labels, while the annotations remain labels of expressed reasoning functions rather than latent cognitive states.

  • Annotation procedure: Each validation span was independently labeled by three of seven annotators using eight canonical operation labels plus NOT DETERMINED.The reference label was assigned by majority vote when at least two annotators selected the same canonical label.
  • Human validation: 81/84 validation spans received a human majority label, with Fleiss’ κ = 0.666.50/84 cases were unanimous, corresponding to 59.5% of the validation set.
  • Human validation: GPT-5 matched human majority labels exactly on 64/84 spans, with Cohen’s κ = 0.715.Soft agreement was 67.5/84 (80.4%), reported as a supplementary measure of annotation ambiguity.
  • Interpretation: Operation labels describe textually expressed reasoning functions rather than direct labels of latent cognitive states.The eight recurring labels include Extraction, Direct mapping, Decomposition, Recall, Deduction, Algebraic manipulation, Arithmetic computation, and Final answer.
  • Representation analysis: The analysis represents spans with first-token, middle-token, last-token, or mean pooling before normalization, PCA, and supervised LDA.The main analysis uses middle-token representations, while other pooling variants are reported separately.

F.2 Reproducibility Details for Trace Generation

Trace generation uses model-specific chat templates or a fallback prompt, stochastic decoding, and zero-shot correctness judging. Because generation is unseeded, repeated runs are not expected to produce bitwise-identical trace corpora.

  • Prompting and generation: Chat-template models receive a single user message, while unavailable templates use a fallback prompt; no system prompt or few-shot examples are added.The fallback instruction is “Solve step by step.”
  • Prompting and generation: Generation uses do_sample=True, temperature = 0.7, top_p = 0.9, and max_new_tokens = 4096.Other decoding parameters remain at library defaults, and generation stops at the default end-of-sequence token or maximum length.
  • Reproducibility: Trace generation is stochastic and uses no fixed random seed, so generated corpora are not expected to be bitwise reproducible.Question-level splitting, span sampling, and random-baseline construction use a fixed seed of 42 afterward.
Loading 2609.04753v1…