PaperlayerView exact revision on arXiv

Source-linked AI summary

Verbalising Model's Computation for Chain-of-Thought Faithfulness Detection

Yujing Wang, Yulan He, Wenya Wang, Hanqi Yan

arXiv:2610.23275v1cs.CL

TL;DR

Existing CoT-faithfulness methods use model outputs or static activations and offer limited explanation of how unfaithfulness develops during generation. VERITAS verbalises internal computation along generated CoT traces, and its classifier improves on LLM-as-Judge and activation probing while supporting mechanistic analysis of reasoning divergence.

  • Problem

    Existing CoT-faithfulness methods assess outputs or static activations but provide limited explanation of how unfaithfulness develops during generation.

    Source: §1 · p. 1

  • Method

    VERITAS uses the J-lens to verbalise reasoning-relevant internal information as it emerges across layers and shapes each generated CoT token.

    Source: §1 · p. 1

  • Results

    28.4% gain over LLM-as-Judge and 16.0% over activation probing are reported for a classifier trained on VERITAS trace features.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VERITAS offers a human-readable, mechanistic account of reasoning evolution and is reported to generalise across datasets and model architectures.

    Source: §1 · p. 2, §6 · p. 10

    VERITAS traces should be treated as diagnostic tools rather than ground-truth explanations of model behaviour.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Current Chain-of-Thought (CoT) verification methods detect reasoning faithfulness from model outputs or static activations, but are limited in explaining how unfaithfulness develops during generation. We introduce VERITAS to verbalise the model's computation: how different reasoning anchors emerge, transfer across layers, and surface in the output. VERITAS adopts the layer-wise Jacobian projection to produce a verbalised account of the model's internal reasoning evolution at each generated CoT token. By training a classifier on the extracted features along the traces, we show that these structured features provide insightful perspectives on how and where reasoning diverges toward unfaithfulness. Experiments on four datasets from FaithCoT-Bench and BonaFide show that VERITAS achieves a significant performance gain over LLM-as-Judge, with a 28.4% gain in balanced accuracy, and an average improvement of 16.0% over activation probing. Further analysis shows VERITAS can be generalised to different datasets and model architectures. Our work shows that verbalising model computation provides a richer and more interpretable lens on CoT faithfulness, moving beyond detection toward a mechanistic understanding of how reasoning fails.

1 INTRODUCTION

Existing CoT-faithfulness methods assess visible outputs or static activations but do not explain how unfaithfulness develops during generation. VERITAS traces reasoning-relevant information across layers and generated tokens, using structured features to detect and explain divergence.

1 INTRODUCTION

  • Existing methods assess visible CoT plausibility or probe static activations, but do not explain how reasoning-relevant information emerges, transfers across layers, and appears in generated text.CoTs can also be affected by misleading cues, concealing reward hacking.Source: §1 · p. 1
  • VERITAS provides a continuous account of reasoning evolution by tracing how information arises, propagates through layers, and shapes each generated CoT token.The approach hypothesizes that faithful and unfaithful CoTs follow distinct internal trajectories that can reveal where and how reasoning diverges.Source: §1 · pp. 1–2
  • Map, Profile, and Dynamics jointly describe where anchor evidence appears, which anchor tokens are expressed, and how anchor states evolve across layers and CoT phases.VERITAS uses the J-lens to map developmental hidden states into an interpretable representation space and aligns trace features with faithfulness anchors for classification.Source: §1 · pp. 1–2
  • 28.4% gain over LLM-as-Judge and 16.0% over activation probing are achieved by a classifier trained on VERITAS trace features.The framework also provides mechanistic explanations of reasoning failures and generalises across datasets and model architectures.Source: §1 · p. 2

2 VERBALISE THE COT INTERNAL COMPUTATION

VERITAS reads CoT development through layer-wise internal computation and its expression in visible text. The Jacobian lens maps layer states into vocabulary-aligned representations, revealing phase-dependent differences between faithful and unfaithful trajectories.

2.1 UNDERSTANDING MODEL COMPUTATION VIA JACOBIAN LENS

  • The Jacobian lens linearizes how an intermediate-layer state influences the final-layer residual, then maps it through the model’s unembedding to a vocabulary distribution.Its layer-specific token directions form J-space, a shared representation for comparing layer states.Source: §2.1 · p. 2

2.2 READOUTS ALONG DEPTH AND REASONING PROGRESS

  • VERITAS compares reasoning-relevant information propagated through internal layers with its progressive expression in visible CoT across depth and reasoning phases.The method samples states at sentence and clause boundaries into a fixed layer-by-phase grid, then follows their J-space alignment across normalized CoT phases.Source: §2.2 · pp. 2–3
  • Faithful internal and visible-CoT trajectories show more similar phase-wise trends, particularly near the end of the CoT.The visible-CoT trajectory is represented by a lexical proxy projected alongside internal states onto the faithful-minus-unfaithful J-space direction.Source: §2.2 · p. 3, Figure 1 · p. 3
  • Faithful and unfaithful J-space traces follow similar depth paths but show phase-dependent offsets, clearest in middle layers during the final CoT phase.The separation suggests faithfulness is reflected in semantic movement through the network rather than one isolated layer.Source: §2.2 · pp. 3–4, Figure 2 · p. 3

3 TRACING COT DYNAMICS FOR FAITHFUL DETECTION

VERITAS traces reasoning-relevant information across model layers and CoT progress, verbalizing internal states with aligned anchors. It uses these trajectories to detect faithfulness and locate reasoning divergence.

3.1 OVERVIEW

  • VERITAS maps internal states to J-lens-aligned anchor features and trains a detector to predict CoT faithfulness.The feature function Φ produces a representation from the internal states, and the classifier predicts whether the CoT is faithful or unfaithful.Source: §3.1 · p. 4

3.2 CONSTRUCTING THE EXPLAINABLE ANCHOR

  • For each layer–phase cell, VERITAS averages within-question faithful–unfaithful state differences, then selects the highest-scoring positive token directions as anchors.Pairing CoTs for the same question is intended to keep task content out of the contrast.Source: §3.2 · pp. 4–5
  • Faithful-directed anchors retain Context, with Checking increasing later and in deeper layers, while unfaithful-directed anchors lose Context early and are dominated by Explanation and Reporting.Figure 4 groups anchors into those four functional stages and shows their composition across normalized CoT phases and model depth.Source: §3.2 · p. 5, Figure 4 · p. 5

3.3 FAITHFUL DETECTION WITH TRANSITIONAL FEATURES

  • Map preserves anchor-evidence strength by layer and phase, while Profile summarizes each selected anchor’s average alignment across the trajectory.Together, the features represent where evidence appears and which anchors are expressed.Source: §3.3 · pp. 5–6
  • Dynamics measures changes in anchor alignment between neighboring cells and captures directional consistency across model depth and CoT progress.The feature set concatenates Map, Profile, and Dynamics for an L2-regularized logistic-regression detector; the linear head is used so performance primarily reflects the extracted features.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments evaluate faithful-CoT detection across BonaFide and FaithCoT-Bench datasets, using Llama and Qwen backbones, and analyze feature behavior, sensitivity, and transfer.

4.2 UNDERSTANDING THE TRANSITION VIA STRUCTURED FEATURES

  • Each structured feature view contributes individually, and combining Map, Profile, and Dynamics yields the best detection results across settings.Map captures where anchor evidence appears, Profile preserves alignment by anchor, and Dynamics tracks changes across CoT phases and model depth.Source: §4.2 · p. 6

4.1 FAITHFUL DETECTION RESULTS

  • Averaged over five datasets, VERITASc reaches 0.749 with Qwen3 and 0.737 with Llama3.1, more than 9 points above the strongest baselines (Ans Tracing, 0.655; HSl, 0.644).VERITASc incorporates Profile in addition to Dynamics and Map.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 UNDERSTANDING THE TRANSITION VIA STRUCTURED FEATURES

  • Map shows higher overall anchor alignment for faithful CoTs, peaking near layers 22–23 versus unfaithful CoTs near layer 17.Faithful peaks shift slightly deeper in later CoT phases; Map preserves layer–phase structure that a pooled score would discard.Source: §4.2 · p. 7, Figure 5 · p. 7
  • Profile shows stronger faithful-CoT alignment on all 10 most discriminative anchors, while retaining differences among individual anchors.Unfaithful alignment remains moderately high for “text” but drops near zero for “assess,” “value,” and “automatically.”Source: §4.2 · p. 7, Figure 6 · p. 8
  • After the first 30% of the CoT, faithful alignments plateau while unfaithful alignments decline sharply near the end, in both motor and workspace layers.Dynamics measures whether consecutive alignment changes across CoT progress and model depth keep a consistent direction.Source: §4.2 · p. 7, Figure 6 · p. 8

4.3 SENSITIVITY ANALYSIS

  • Feature performance varies with CoT length, while coarser layer stages hurt Map and Profile most; Dynamics stays nearly flat across stage merging.The three views favor different lengths, and combining them gives the most stable performance, retaining 0.693 at K = 7 versus 0.695 with all layers.Source: §4.3 · p. 8, Figure 7 · p. 8

4.4 CROSS-SOURCE GENERALIZATION

  • Across datasets, anchor transfer retains 97–99% of in-domain balanced accuracy; classifier transfer retains 88–90% in both metrics, while cross-model transfer varies by feature and stage granularity.Cross-model Dynamics retains 91–96% of in-domain performance at every K; coarser stages better absorb depth mismatch between models.Source: §4.4 · pp. 8–9, Figure 8 · p. 9

5 RELATED WORK

Prior CoT-faithfulness work uses observable behavior or internal representations, but existing methods do not verbalize how unfaithfulness develops across layers and reasoning progress.

5 RELATED WORK

  • External methods use outputs and interventions to assess faithfulness, while internal methods inspect causal dependence or hidden representations; neither explains how reasoning-relevant information transits during generation.VERITAS addresses this gap with interpretable, token-named J-space anchors.Source: §5 · p. 9

6 CONCLUSION

VERITAS uses J-lens and three structured features to track internal computation across model depth and CoT progress; experiments report baseline improvements, transfer, and human-readable explanations.

6 CONCLUSION

  • Experiments on two benchmarks and two backbones show that VERITAS outperforms all baselines, transfers across datasets and models, and explains unfaithfulness in human-readable terms.The framework summarizes where anchor evidence appears, which anchors carry it, and how it evolves.Source: §6 · p. 10

ETHICS STATEMENT

The ethics statement describes the benchmarks as public research datasets and frames VERITAS traces as diagnostic tools rather than ground-truth explanations.

ETHICS STATEMENT

  • VERITAS traces should be treated as diagnostic tools, not ground-truth explanations of model behaviour.The work reports no direct harmful applications and states that improved faithfulness detection supports safer and more transparent LLM deployment.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The statement documents the method, hyperparameters, datasets, and model configurations needed to reproduce the experiments, and says the source code will be released as supplementary material.

REPRODUCIBILITY STATEMENT

  • VERITAS’s J-lens projection, faithful-anchor construction, and structured-feature extraction are described in Section 3, with implementation details in Appendix B.The statement also says the classifier and feature-extraction hyperparameters are fully specified in Appendix B.6.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Dataset statistics and preprocessing are documented in Appendix A, and the models are available through HuggingFace with versions and hardware configurations reported there.Experiments use the publicly available FaithCoT-Bench and BonaFide benchmarks.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The authors say they will release feature-extraction, classifier-training, and evaluation code as anonymous supplementary materials.The statement also identifies ablations and cross-dataset generalization results as reproducible using that code and the documented splits.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The authors used generative AI for language edits and figure-presentation code, reviewed the assisted work, and state that it was not used for research design or analysis.

AI USE STATEMENT

  • Generative AI was used to edit the Introduction and Related Work for readability and code to improve figure presentation.The authors state that they reviewed all AI-assisted work.Source: AI USE STATEMENT · p. 10
  • The authors say they revised AI-drafted content, validated AI-provided information, and double-checked AI-edited language.They accept responsibility for the final content, including text produced with AI assistance.Source: AI USE STATEMENT · p. 10
  • The authors state that generative AI was not used to develop theory, formulate claims or hypotheses, design experiments, implement methods, generate or clean data, or conduct qualitative analysis.Source: AI USE STATEMENT · p. 10

A BACKBONES AND DATASETS

This section specifies the datasets, backbones, feature and classifier procedures, and evaluation protocol. It reports complementary feature contributions and cross-source transfer results.

A BACKBONES AND DATASETS

  • FaithCoT-Bench covers four reasoning subsets, and BonaFide contributes 1,016 CoTs across 541 questions, with 87.5% labeled unfaithful in its diversionary setting.The evaluation uses Llama-3.1-8B-Instruct and Qwen3-4B-Thinking-2507 as replay backbones; FaithCoT-Bench labels were expert-annotated.Source: A BACKBONES AND DATASETS · p. 13, Table 2 · p. 13

B.1 J-LENS ESTIMATION; B.2 ANCHOR AND FEATURE CONFIGURATION

  • The J-lens is estimated from residual-state Jacobians averaged over positions and 1,000 WikiText-2 sequences, independently of labeled CoT data.CoTs are divided into five normalized phases, and the five highest-scoring positively aligned content-word anchors per layer–phase cell form a frozen dictionary.Source: §B.1 · p. 13, §B.2 · p. 13

B.3 LLM JUDGE; B.4 RAW HIDDEN-STATE PROBES; B.5 CLASSIFIER CONFIGURATIONS; B.6 EVALUATION PROTOCOL

  • The LLM judge sees the question, visible CoT, and answer but not reference labels or correct answers; Raw-HS probes use one fixed layer without layer search.VERITAS classifiers use fold-based imputation and standardization, L2-regularized logistic regression, and balanced class weights; the evaluation groups all CoTs for each question within one fold.Source: §B.3 · pp. 13–14, §B.4 · p. 14, §B.5 · p. 14

B.7 FUNCTIONAL CATEGORIES OF ANCHOR TOKENS

  • The four functional categories—Context, Explanation, Checking, and Reporting—are used only to interpret anchor tokens, not to alter discovery or classifier training.Displayed category values are normalized shares of discriminative anchor evidence, not raw token frequencies; tokens shared across faithful and unfaithful pools are removed.Source: §B.7 · p. 14, Table 3 · p. 15

C.1 FEATURE ABLATION

  • Combining Map, Profile, and Dynamics achieves the best balanced accuracy in every reported model and benchmark setting.The three features capture where anchor responses occur, which anchors are expressed, and how their representations evolve.Source: §C.1 · p. 15, Table 4 · p. 15

C.2 FEATURES’ CROSS-SOURCE EFFECT

  • Dynamics retains 94% of AUROC under zero-shot cross-source transfer, while its balanced-accuracy retention is 89%, indicating a threshold shift despite transferred ranking.Profile retains 86% in balanced accuracy and Map retains 92%; the passage attributes Profile’s lower retention to source-dependent discriminative anchors and Map’s transfer to its layer-and-phase computation structure.Source: §C.2 · p. 15

C.3 CROSS-SOURCE GENERALIZATION

  • Anchor transfer preserves 98–99% of in-domain AUROC, while zero-shot Ourp and Ourc retain 87% and 86%, respectively, versus 85% for HSl.The evaluation covers all twelve ordered source–target pairs among the four FaithCoT-Bench subsets; anchor transfer refits the classifier on the target, whereas zero-shot transfer freezes both anchors and classifier.Source: §C.3 · p. 15

C.4 SENSITIVITY ANALYSIS

The sensitivity analysis varies phase-bin count and anchors per layer–phase cell on FaithCoT-Bench. The reported results support using five phases and eight anchors as defaults, with a modest variant-specific trade-off for anchor count.

CoT phases

  • The phase sweep compares P ∈ {3, 5, 7} while holding other settings fixed and averages balanced accuracy over four FaithCoT-Bench sources.Source: §C.4 · p. 16
  • 0.703 and 0.671 are the respective best balanced accuracies for the two variants at P = 5; three phases provide insufficient resolution, while seven yield sparser cells.Figure 11 marks P = 5 as the default setting.Source: §C.4 · p. 16, Figure 11 · p. 17

Number of Anchors

  • 0.703 balanced accuracy is Ourp’s best result at κ = 8, while Ourc improves slightly at κ = 16; κ = 8 is the default to limit dictionary size while preserving the strongest Profile-based result.The anchor-count sweep varies κ ∈ {4, 8, 16}, with P = 5 fixed and results macro-averaged across four sources; Figure 12 marks κ = 8 as default.Source: §C.4 · p. 16, Figure 12 · p. 17
Loading 2610.23275v1…