PaperlayerView exact revision on arXiv

Source-linked AI summary

Verifying Chain-of-Thought Reasoning via Its Computational Graph

Zheng Zhao, Yeskendir Koishekenov, Xianjun Yang, Naila Murray, Nicola Cancedda

arXiv:2510.09312v2cs.CLcs.AIcs.LG

TL;DR

Existing black-box and gray-box verification methods offer limited insight into why reasoning computations fail. CRV analyzes attribution graphs as computational traces and classifies their structural features; the paper reports domain-specific error signatures and successful targeted interventions on faulty reasoning.

  • Problem

    Black-box and gray-box verifiers provide limited insight into why a reasoning computation fails.

    Source: §1 · p. 1

  • Method

    CRV constructs attribution graphs as traces of reasoning computations and trains a diagnostic classifier to predict step correctness from graph structure.

    Source: §1 · pp. 1–2, §2.1 · p. 2

  • Results

    Error signatures are domain-specific, and targeted interventions on transcoder features successfully correct faulty reasoning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings establish CRV as a proof of concept for mechanistic analysis of reasoning failures through computational structure.

    Source: §6 · p. 10

    CRV focuses on standard instruction-tuned models generating autoregressive CoT and is too computationally intensive to serve as a practical drop-in verifier.

    Source: §1 · p. 2

Abstract

from arXiv · show

Current Chain-of-Thought (CoT) verification methods predict reasoning correctness based on outputs (black-box) or activations (gray-box), but offer limited insight into why a computation fails. We introduce a white-box method: Circuit-based Reasoning Verification (CRV). We hypothesize that attribution graphs of correct CoT steps, viewed as execution traces of the model's latent reasoning circuits, possess distinct structural fingerprints from those of incorrect steps. By training a classifier on structural features of these graphs, we show that these traces contain a powerful signal of reasoning errors. Our white-box approach yields novel scientific insights unattainable by other methods. (1) We demonstrate that structural signatures of error are highly predictive, establishing the viability of verifying reasoning directly via its computational graph. (2) We find these signatures to be highly domain-specific, revealing that failures in different reasoning tasks manifest as distinct computational patterns. (3) We provide evidence that these signatures are not merely correlational; by using our analysis to guide targeted interventions on individual transcoder features, we successfully correct the model's faulty reasoning. Our work shows that, by scrutinizing a model's computational process, we can move from simple error detection to a deeper, causal understanding of LLM reasoning.

1 Introduction

Existing black-box and gray-box verifiers can detect signals associated with reasoning errors but offer limited explanation of the computations behind them. The paper introduces CRV to study reasoning through attribution-graph structure, reporting domain-specific error signatures and successful corrections through targeted feature interventions.

1 Introduction

  • Black-box and gray-box methods can detect error-correlated signals but cannot explain why the underlying computation leads to an error.The paper identifies this as a limitation of methods analyzing generated text, logits, raw activations, or hidden-state trajectories.Source: §1 · p. 1
  • CRV treats attribution graphs as execution traces of latent circuits and trains a classifier on their structural properties to detect reasoning failures.The approach replaces MLPs with trained transcoders to make the model interpretable before constructing and analyzing the graphs.Source: §1 · pp. 1–2
  • CRV is intended as a scientific instrument, not a practical drop-in verifier, and focuses on standard instruction-tuned models generating autoregressive CoT.The paper says advanced reasoning mechanisms such as search and backtracking can obscure the computations of a single step, and that CRV is computationally intensive.Source: §1 · p. 2
  • Error signatures are domain-specific, with failures in different reasoning tasks manifesting as distinct computational patterns.Source: §1 · p. 2
  • Targeted interventions on individual transcoder features successfully correct faulty reasoning, supporting a causal role for the error signatures.Source: §1 · p. 2
  • The authors release datasets with step-level correctness labels for synthetic and real-world tasks, along with trained transcoders.Source: §1 · p. 2

2 Problem Formulation and Preliminaries

The paper frames step verification as predicting correctness from structural fingerprints extracted from attribution graphs. It motivates the graphs as traces of interpretable computations while noting that the proposed signatures are indirect evidence about latent circuits.

2.1 Problem Statement

  • For each reasoning step, CRV constructs an attribution graph, extracts a fixed-size structural fingerprint, and uses it to predict whether the step is correct.The graph represents causal influence among interpretable features and tokens.Source: §2.1 · p. 2

2.2 Preliminaries: Circuits in Transformers

  • In mechanistic interpretability, a circuit is a neural-network subgraph implementing a human-understandable algorithm.The paper hypothesizes that latent circuits cast detectable structural fingerprints onto its attribution graphs, but says it does not observe those circuits directly.Source: §2.2 · p. 2
  • The paper’s circuit interpretation is a hypothesis about graph signatures, not direct observation of the latent circuits themselves.Source: §2.2 · p. 2

2.3 Preliminaries: Transcoders for Interpretable Features

  • A sparse autoencoder represents an activation with a higher-dimensional, mostly zero feature vector whose elements are learned features.The supplied passage describes SAE reconstruction of its input before contrasting it with the transcoder variant used in this work.Source: §2.3 · p. 2
  • The pipeline replaces MLP modules with per-layer transcoders, builds a step-level attribution graph, extracts structural features, and classifies step correctness.Source: Figure 1 · p. 3
  • Unlike an SAE trained to reconstruct its input, a transcoder approximates a target component’s input-output function and serves as its functional substitute.Replacing an MLP with a transcoder represents intermediate computations as sparse combinations of learned features.Source: §2.3 · p. 3

3 Methodology

CRV is a white-box method for detecting flawed reasoning by analyzing structural fingerprints of computational graphs. It uses step-labeled data, an interpretable transcoder-replacement model, step-level attribution graphs, graph features, and a diagnostic classifier.

3 Methodology

  • CRV detects flawed reasoning by analyzing the structural fingerprint of a reasoning step’s computation.The paper positions this as a developer-facing alternative to judging step correctness alone, aimed at debugging a model with full internal access.Source: §3 · p. 3

3.1 Dataset Curation and Step-Level Annotation

  • The benchmark pairs the model’s CoT text and step labels with computational traces, covering controlled synthetic tasks and GSM8K.Existing text-only datasets are incompatible with this method because it requires the target model’s internal state during generation.Source: §3.1 · p. 3
  • Boolean-expression and multi-step arithmetic datasets provide automatically verifiable step correctness and controllable task complexity.Source: §3.1 · p. 3
  • For GSM8K, a stronger LLM judges step correctness using the full problem context, and manual review validates a substantial subset of labels.Source: §3.1 · p. 4

3.2 Circuit-based Reasoning Verification (CRV)

  • CRV’s four-stage pipeline replaces MLPs with interpretable transcoders, constructs attribution graphs, extracts structural features, and classifies correctness.Source: §3.2 · p. 4, §3.2.1 · p. 4

3.2.2 Step 2: Constructing Step-Level Attribution Graphs

  • For each reasoning step, a greedy path-finding algorithm traces high-attribution connections backward from final logits to form a sparse, weighted, directed graph.Its nodes are input tokens, active transcoder features, and output logits; edge weights quantify influence.Source: §3.2.2 · p. 4

3.2.3 Step 3: Extracting Interpretable Graph Features

  • The fixed-size graph fingerprint is computed on a subgraph pruned to influential components, including global, node-influence, activation, and topological features.The feature groups cover graph size and output uncertainty, feature activation and influence statistics, layer distribution, density, centrality, and connectivity.Source: §3.2.3 · p. 4

3.2.4 Step 4: Diagnostic Classifier

  • A Gradient Boosting Classifier predicts step correctness from the extracted feature vectors and provides feature-importance measures.The authors chose it for heterogeneous tabular features and benchmarked alternative classifiers in an appendix.Source: §3.2.4 · p. 5

4 Experiments

Experiments test CRV’s verification performance, structural error signatures, and intervention potential using Llama 3.1 8B Instruct on synthetic Boolean and arithmetic tasks and GSM8K. CRV outperforms the evaluated baselines, while its error signatures vary by task and can guide targeted corrections.

4.1 Experimental Setup

  • The evaluation uses Llama 3.1 8B Instruct with trained transcoders, three datasets, and black-box and gray-box baselines, measuring AUROC, FPR@95, and AUPR.The baselines include output-based scores, hidden-state probes, and trajectory-based methods; incorrect steps are treated as the positive class.Source: §4.1 · p. 5

4.2 Verification Performance and Robustness (RQ1)

  • CRV achieves 92.47 AUROC on Synthetic Arithmetic versus the strongest baseline’s 76.45, and reduces FPR@95 to 37.09% from 63.33%.The authors report that CRV consistently outperforms black-box and gray-box baselines across every dataset and metric.Source: §4.2 · pp. 5–6, Table 1 · p. 6
  • Combined training yields 90.51 AUROC on Arithmetic and 70.62 on GSM8K, while Arithmetic-only transfer to GSM8K reaches 57.04 AUROC, below the Energy baseline.The authors interpret limited zero-shot transfer as evidence that error fingerprints are task-specific, while combined training can learn diverse signatures.Source: §4.2 · p. 6, Table 2 · p. 6
  • On the synthetic arithmetic task, CRV maintains a consistent advantage over key baselines across difficulty levels (number of operators), while most methods show stable AUROC and FPR@95.AUPR generally improves for all methods as difficulty rises because harder problems increase the proportion of incorrect examples; CRV’s advantage persists despite these shifts.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 Mechanistic Analysis of Error Computations (RQ2)

  • Removing Node Influence & Activation features causes the greatest ablation degradation, including an increase of over 12 points in FPR@95; the full feature set performs best.Global Graph Statistics contribute substantially, while Topological & Path-Based features are least critical on Synthetic Arithmetic.Source: §4.3 · p. 7, Table 3 · p. 7
  • On GSM8K, distributions of five selected graph features differ significantly between correct and incorrect steps, with medium-to-large effect sizes.The authors also report that incorrect steps form a dense subset within the broader distribution of correct steps in a PCA projection of the full feature vectors.Source: §4.3 · p. 7, Figure 4 · p. 8, Figure 3 · p. 8

4.4 From Diagnosis to Causal Intervention (RQ3)

  • In a case study, CRV flagged an arithmetic error associated with high activation of a multiplication feature; clamping that feature to zero led to the correct next step and final answer.The authors also report correcting an error by amplifying an under-active feature, while describing the interventions as proof-of-concept rather than a general-purpose solution.Source: §4.4 · p. 8, Table 4 · p. 9

5 Related Work

Prior CoT verification work examines text or hidden states, while mechanistic interpretability studies latent circuits and computational structure. CRV applies attribution-graph structure to automated verification of reasoning failures.

5 Related Work

  • CoT prompting research spans few-shot and zero-shot elicitation, synthetic exemplars, and test-time compute scaling.The section also surveys work on CoT verification, faithfulness, improvement, and process reward models.Source: §5 · p. 9
  • Prior verification methods assess reasoning through text or hidden states, whereas this work operationalizes attribution-graph structure for automated verification.The cited review describes black-box and gray-box verification and notes the absence of prior structural computational-graph verification.Source: §5 · p. 9
  • Mechanistic interpretability treats specialized subgraphs as circuits and has begun examining latent algorithms underlying CoT reasoning.Prior work used transcoder-based attribution graphs for qualitative faithfulness analysis; CRV extracts quantitative structural features to diagnose computational failures.Source: §5 · p. 9

6 Conclusion

The conclusion presents CRV as a white-box approach that links attribution-graph structure with reasoning errors, domain-specific signatures, and successful targeted interventions. An appendix provides further dataset-construction, labeling, and statistics details.

6 Conclusion

  • CRV finds distinct structural fingerprints for correct and incorrect reasoning, with domain-specific error signatures and targeted transcoder-feature interventions that corrected faulty reasoning.The authors frame these findings as a proof-of-concept for mechanistic analysis of how and why LLMs fail to reason correctly.Source: §6 · p. 10

A Additional Details on Datasets

A.1 Synthetic Dataset Construction

The appendix describes procedurally generated Boolean and Arithmetic CoT datasets and a step-labeling protocol combining LLM judgments with programmatic verification for synthetic tasks. Human review reports moderate kappa but high raw agreement, with disagreements often following an earlier error.

A.1 Synthetic Dataset Construction

  • The Boolean and Arithmetic synthetic datasets use ground-truth expressions and Llama 3.1 8B Instruct-generated CoT, with 10,000 expressions per difficulty except Boolean n = 3.Difficulty is controlled by operator count n ∈ {3, 5, 7, 10}.Source: §A.1 · p. 14

A.2 Labeling Protocol

  • Synthetic-task steps are retained only when an LLM judge and programmatic state verifier agree on their correctness label.The state verifier compares the ground-truth value of the original expression with that of the model-reduced expression; equivalent values can occasionally let an incorrect reduction pass.Source: §A.2 · pp. 14–15

A.3 Human Validation of LLM-as-a-Judge Labels

  • Filtering ambiguous incorrect cases improves label confidence but makes the class distribution more imbalanced.The validation procedure upsampled the positive class; the authors report both kappa and raw agreement because extreme imbalance can depress kappa.Source: §A.3 · pp. 15–16
  • Human annotators had κ = 0.42 and 87.3% simple agreement, while consensus human labels versus LLM-judge labels had κ = 0.26 and 84.1% simple agreement.Disagreements mostly involved steps following an initial reasoning error, motivating truncation of the dataset after the first error.Source: §A.3 · p. 16

A.4 Handling Error Propagation

Steps after an initial reasoning error can be locally correct yet ambiguous in their contribution to the final answer. The dataset therefore retains each chain only through its first incorrect step.

A.4 Handling Error Propagation

  • The dataset retains steps only through the first incorrect step, discarding later steps on a corrupted computational path.Later calculations may be technically correct given a previously incorrect expression, making their local versus global correctness ambiguous.Source: §A.4 · p. 16

A.5 Dataset Statistics and Model Performance

The section describes the datasets and model evaluations used in the study, including step-level annotation for GSM8K and train/test splitting for synthetic tasks.

A.5 Dataset Statistics and Model Performance

  • GSM8K uses the official test split and LLM-as-a-Judge annotations, while synthetic data is randomly divided 80/20 for classifier training and testing.The judge pipeline is used because natural-language reasoning steps are not always programmatically verifiable.Source: §A.5 · p. 17
  • The human validation study reports moderate-to-high inter-annotator agreement, with lower Kappa scores attributed to extreme class imbalance.The supplied passage does not provide the agreement values.Source: Table 10 · p. 18
  • A flawed trace illustrates how an unrecoverable Step 2 error makes the correctness of later steps ambiguous.Source: Table 11 · p. 18
  • Table 12 summarizes dataset sizes and label distributions after annotation and filtering, while Table 13 reports Llama 3.1 8B Instruct task accuracy.Table 13 also breaks down synthetic-task accuracy by operator-count difficulty.Source: Table 12 · p. 19, Table 13 · p. 19

B Transcoder Training Details

This section details the transcoder and attribution-graph setup, feature and baseline choices, evaluation metrics, and classifier comparisons. It also reports that instruction-tuning data did not consistently improve verification, while attribution after a reasoning step generally performed better.

B Transcoder Training Details

  • TopK transcoders reconstruct each MLP’s output from its residual-stream input using 131,072 latent features and retaining the 128 largest activations.They were trained on 10B RedPajama-V2 tokens; decoder weights were normalized, and inactive features received revival loss.Source: B Transcoder Training Details · p. 20, Figure 5 · p. 20

B.1 Impact of Training Transcoders on Instruction-Tuning Data

  • Further transcoder training on instruction-tuning data yielded no consistent or meaningful verification improvement, so the main experiments use base transcoders.Table 14 compares the base and instruction-tuned variants.Source: §B.1 · p. 21, Table 14 · p. 21

B.2 Attribution Graph Computation

  • Computing attribution after the current step showed a clear, consistent advantage across nearly all metrics and domains over computing it before the step.The “After” setting uses the current step’s final token; the paper attributes its advantage to the executed computation being fully represented there.Source: §B.2 · p. 21, Table 15 · p. 21

C.1 Attribution Graph Features

  • The graph feature set spans global counts and density, node influence and activation statistics, layer-wise feature distributions, centrality, connectivity, and path lengths.Input-to-logit path length is used to measure information propagation from prompt tokens to the final decision.Source: §C.1 · p. 22

C.2 Additional Details on Baselines

  • Baseline implementations include final-token-logit black-box methods, hidden-state gray-box methods, and supervised probes selected or tuned using validation data.The LR probe searches across the model’s 32 layers; MLP probes also vary learning rate, weight decay, hidden size, and source layer.Source: §C.2 · pp. 22–23

C.3 Additional Details on Evaluation Metrics

  • Verification is evaluated with AUROC, FPR@95, and AUPR, treating incorrect reasoning steps as the positive class and higher scores as more error-like.FPR@95 uses a threshold at which TPR is at least and closest to 0.95.Source: §C.3 · pp. 23–24

C.4 Additional Classifier and Their Results

  • Logistic regression achieved the best AUROC on two datasets and strongest overall GSM8K results, while tree ensembles often led on other metrics; no classifier dominated across domains and metrics.Gradient Boosting was selected for the main experiments as a robust performer; the classifier comparison appears in Table 18.Source: Table 18 · p. 24, §C.4 · p. 24

C.5 Additional Results for RQs

The additional results show separable graph-feature patterns for correct and incorrect steps and give a case where feature-guided intervention corrected an arithmetic parsing error. The section also notes the authors’ limited use of language models as writing assistants.

C.5 Additional Results for RQs

  • Five selected graph features differed significantly between correct and incorrect steps on arithmetic and Boolean tasks, with p < 0.001 and medium-to-large effect sizes.Source: Figure 6 · p. 25, Figure 7 · p. 25
  • Amplifying a multiplication-related transcoder feature corrected a unary-minus parsing error, changing the model’s result from -42 to -41.The classifier flagged the step, and feature-importance analysis suggested the missing multiplication signal.Source: §C.5 · p. 25, Table 19 · p. 26
  • A t-SNE projection of full feature vectors shows separation between correct and incorrect steps, which the passage says validates CRV’s distinct structural error fingerprint.Source: Figure 8 · p. 25

D Usage of Language Language Models

E Limitations

The limitations concern CRV’s computational cost, its aggregative feature analysis, the scope of its empirical generalization, and dependence on imperfect interpretability tools.

E Limitations

  • CRV’s transcoder training and per-step graph construction make it orders of magnitude more resource-intensive than black- or gray-box verification, limiting its current role to deep analysis rather than real-time production verification.Source: E Limitations · p. 27
  • CRV’s features are primarily aggregative and do not yet reason symbolically about whether semantically specific features activate appropriately for inputs.Source: E Limitations · p. 27
  • The empirical results cover only Llama 3.1 at 8B scale, leaving transfer to other architectures or larger scales unresolved; identified error signatures are also domain-specific.Source: E Limitations · p. 27
  • The analysis depends on a non-exhaustive sparse feature basis and attribution graphs that incompletely approximate the model’s true information flow.Source: E Limitations · p. 27

Related papers

Loading 2510.09312v2…