Detecting Hidden Chain-of-Thought in Large Language Models with Linguistic, Behavioral, and Mechanistic Indicators

Motivation and approach

• Explicit CoT can improve complex reasoning performance while its visible steps may not faithfully represent the computations producing answers.This motivates studying latent reasoning without relying on self-reported traces.Source: §1 · p. 1
• HCDS compares neutral-prompt behavior with explicit CoT and explicit no-CoT behavior using linguistic, behavioral, and mechanistic features.The six features are entropy, entropy slope, latency, paraphrase consistency, perturbation sensitivity, and attribution-based suppression cost.Source: §1 · p. 1

Empirical findings

• Qwen3-4B Thinking may ignore no-CoT instructions, so the primary comparison force-closes its reasoning block while adjusting features for generation length.In an un-intervened diagnostic, Thinking produced a mean of 575 output tokens under a no-reasoning instruction; force-closure reduced the primary comparison pole to a six-token median.Source: §1 · p. 2
• On GSM8K, neutral behavior is significantly closer to explicit CoT than explicit no-CoT for both Qwen3-4B variants.The evaluation covers Qwen3-4B Instruct and Thinking, with StrategyQA reported separately because its answer format weakens the contrast.Source: §1 · p. 2

Contribution

• The framework combines behavioral alignment with mechanistic interventions to test whether identified internal computations influence outputs.Its interpretation remains comparative and does not directly establish an unexposed reasoning trace.Source: §1 · pp. 1–2

Faithfulness and latent reasoning

• Prior work shows that visible CoT explanations can be unfaithful and need not reveal the computations responsible for an answer.The paper therefore asks whether latent reasoning occurs, rather than whether an explicit rationale is faithful.Source: §2 · p. 2
• Other studies examine implicit reasoning from shortcut learning or steerable internal features, whereas this paper detects inference-time use under neutral prompting.The distinction is between reasoning emergence or steering and behavioral detection during inference.Source: §2 · p. 2

Diagnostic features

• Unlike PPCV, which maximizes paraphrastic agreement to improve answers, HCDS uses paraphrase consistency diagnostically and allows it to vary.The paper reports per-feature decomposition separately.Source: §2 · p. 2
• Related agent experiments show that longer interaction can plateau in performance, supporting the paper’s caution against treating response length alone as reasoning.The paper distinguishes length from deeper inference.Source: §2 · p. 2

Mechanistic probing

• The paper adapts intervention logic related to thought anchors, using anchor perturbation as one HCDS feature rather than as the end goal.Its target is CoT-like behavior under neutral prompting.Source: §2 · p. 2

HCDS construction

• HCDS evaluates explicit CoT, explicit no-CoT, and neutral prompts, treating the explicit conditions as comparison poles.For Thinking, neutral and no-CoT share a force-closure intervention.Source: §3 · p. 2, §3.3 · p. 4
• The six-dimensional feature vector contains latency, mean entropy, entropy slope, paraphrase consistency, perturbation sensitivity, and mechanistic intervention sensitivity.The final feature measures interventions on candidate reasoning-related internal states.Source: §3.1 · p. 3

Length adjustment

• Primary HCDS first residualises raw features on log generation length, then z-scores residuals before computing Euclidean distances.Output length is used only for adjustment, diagnostics, and matching.Source: §3.1 · p. 3, §3.2 · p. 3

Robustness

• All GSM8K cells remain significant across alternative missing-feature rescaling policies, with a largest policy difference of 0.40 HCDS.The pairwise policy omits the correction, while the complete policy uses one intersection across all three poles.Source: §3.1 · p. 3

Interpretation

• Positive HCDS means neutral behavior is closer to explicit CoT than explicit no-CoT, while the score does not establish causality.The adjusted score represents alignment not linearly explained by log generation length, but residualisation may remove reasoning-related variation.Source: §3.1 · p. 3, §3.2 · p. 3

Clean no-CoT pole

• Force-closure is load-bearing for Thinking because its chat template opens a reasoning block automatically and asymmetric intervention creates formatting and verbosity differences.With symmetric force-closure, GSM8K exceeds arithmetic, reversing the problematic unadjusted ordering.Source: §3.3 · pp. 3–4

Mechanistic feature

• Mechanistic sensitivity compares accuracy changes after ablating attribution-selected reasoning anchors with position-matched control interventions.A larger decrease after candidate suppression is evidence that the internal state causally contributes to the output.Source: §3.4 · p. 4

Models and datasets

• The study evaluates Qwen3-4B Instruct and Thinking on GSM8K multi-step arithmetic problems and StrategyQA implicit multi-hop commonsense questions.Both datasets use explicit CoT, explicit no-CoT, and neutral prompt conditions.Source: §4 · p. 4

Inference conditions

• Each model–dataset cell uses shared deterministic decoding settings across prompt conditions, including a common token cap within the cell.Caps differ across models and datasets.Source: §4 · p. 4

Measurements

• The experiment records accuracy, generation time, output length, latency, entropy, paraphrase consistency, perturbation sensitivity, and mechanistic sensitivity.Accuracy and output length are auxiliary diagnostics; the remaining six measurements form HCDS.Source: §4 · p. 5

Behavioral features

• Paraphrase consistency measures the fraction of meaning-preserving paraphrases that retain the original answer, while perturbation sensitivity uses a fixed prepended clause.Paraphrases preserve quantities, structure, named entities, and the gold answer.Source: §4 · p. 5

Statistical analysis

• Because decoding is deterministic, uncertainty reflects variation across questions rather than across generations.Question-level scores are analyzed with bootstrap confidence intervals and two-sided significance tests.Source: §4 · p. 5

Robustness analyses

• Robustness analyses include leave-one-feature-out, single-feature, length-matched, and open-weight mechanistic anchor-versus-control analyses.Additional implementation and decoding details are provided in the appendices.Source: §4 · p. 5

5.1 HCDS detects CoT-like behavior on GSM8K

• HCDS remains positive and highly significant for both Qwen3-4B variants on GSM8K after length adjustment.The result holds across two independent inference stacks.Source: §5.1 · p. 5
• Within 0.08, length-adjusted GSM8K scores agree across inference backends, compared with a 0.65 maximum difference before adjustment.Source: Table 2 · p. 5, §5.1 · p. 5

5.2 Length adjustment reduces calibration-control scores

• After length adjustment, control scores approach zero and are almost uniformly non-significant on single-step arithmetic and factual lookup.The unadjusted score instead produces large positive values in five of eight control cells, so it is treated only as a supplementary diagnostic.Source: §5.2 · p. 6, Figure 1 · p. 6

5.4 The result is robust to feature choice

• All 24 leave-one-feature-out variants remain positive and significant, with the weakest cell at +1.193 (p = 1.6 × 10−3).Removing both entropy features causes the largest reduction, but significance remains across both models and backends.Source: §5.4 · p. 6, Figure 2 · p. 7
• Entropy is the strongest and most consistent contributor (d = 0.56–1.13), while paraphrase consistency is the strongest non-entropy feature.Latency contributes little after residualisation, and perturbation sensitivity is positive but non-significant in either model on this sample.Source: §5.4 · pp. 6–7

5.5 Cross-task results

• StrategyQA does not provide an independent replication: Instruct is non-significant on both backends, while Thinking remains positive and is driven mainly by entropy slope.Its binary answer format compresses accuracy-based features near the chance floor.Source: §5.5 · p. 7

Calibration controls

• Length adjustment reduces positive scores on single-step arithmetic and factual lookup, where multi-step reasoning is unnecessary.These tasks serve as negative controls for reasoning-like behavior.Source: §6 · p. 8

Model comparison

• Thinking scores higher than Instruct on both backends when a valid no-CoT pole is created by force-closure.Prompt compliance makes the comparison easier to measure rather than implying lower latent reasoning.Source: §6 · p. 8

Calibration controls

• Five of eight control cells are significantly positive unadjusted, compared with one marginally positive length-adjusted cell.This pattern motivates treating the unadjusted score as vulnerable to verbosity-related artifacts.Source: Table 4 · p. 8

Robustness

• Backend replication checks runtime-specific artifacts but cannot eliminate a length dependence shared by both inference stacks.The marginal Table 4 exception neither replicates nor survives multiple-comparisons correction.Source: §6 · p. 8

Intervention dependence

• Thinking’s no-CoT pole is engineered through force-closure because the checkpoint lacks a promptable non-reasoning mode.Thus, Thinking results characterize behavior under intervention; the prefill is applied identically to both relevant conditions.Source: §7 · p. 8

Generalisation

• The main qualitative comparison uses one Qwen3-4B model family, limiting separation of Thinking behavior from its specific post-training recipe.An unadjusted Gemma-3-4B-it pilot is directionally consistent but cannot provide a matched, length-adjusted dual-variant replication.Source: §7 · p. 8

Mechanistic measurement

• The mechanistic feature is constrained by a single pinned layer/site, position-only controls, attribution saturation, answer-step eligibility, and sparse coverage.It is defined in only 41% of PyTorch/CUDA cells and 40% of MLX cells, with most defined values exactly zero.Source: §7 · pp. 8–9

Statistical reporting

• Statistical uncertainty reflects between-question variability rather than sampling uncertainty about underlying model behavior.The bootstrap also conditions on fitted residualisation and standardisation rather than refitting preprocessing.Source: §7 · p. 9

8 Conclusion

• HCDS combines linguistic, behavioral, and mechanistic indicators to compare neutral behavior with explicit CoT and no-CoT poles.Positive alignment is interpreted comparatively, not as direct observation of an internal reasoning trace.Source: §8 · p. 9, Figure C.1 · p. 13
• The neutral prompt omits reasoning and answer-format directives, whereas explicit prompts request reasoning or answer-only output formats.The evaluation conditions therefore differ in both reasoning-display instructions and, for explicit prompts, answer-format instruction.Source: A Prompt Templates · p. 11
• Paraphrases are generated by Instruct without exposing the gold answer, but reasoning leakage remains possible because rewrites can simplify relations.This makes paraphrase consistency a noisy behavioral feature rather than a leakage-free measure.Source: B Paraphrase Generation and Quality… · pp. 11–12
• Duplicate paraphrases occur for 138 of 200 questions (69%), reducing the effective K to one and making the feature ternary in practice.The observed values are exactly {0, 0.5, 1}.Source: B Paraphrase Generation and Quality… · p. 12
• Removing eight byte-identical rewording cases raises HCDS in all four cells, so retaining the full n = 50 avoids post-hoc exclusion.The reported increases range from +0.009 to +0.128 across backend-model cells.Source: B Paraphrase Generation and Quality… · p. 12
• The primary grid covers four task families, two models, and three prompts at n = 50, with a separate n = 500 output-length diagnostic.Features are z-scored within backend, model, and dataset before HCDS contrasts pole distances.Source: Table 5 · p. 12, Figure C.1 · p. 13
• The natural un-prefilled Thinking neutral-pole design inverts the unadjusted score, with trivial arithmetic outranking GSM8K.The paper therefore uses the symmetric force-closure design in the primary analysis.Source: D Output-Length Distributions · pp. 13–14, Table 6 · p. 14
• Natural no-CoT prompting suppresses Instruct outputs but not Thinking’s long-form generation, motivating the symmetric force-closure design for the primary comparison.The separate diagnostic and primary intervention answer different measurement needs.Source: Figure 3 · p. 13, D Output-Length Distributions · p. 14

• 0.50 chance accuracy on StrategyQA compresses the usable range of accuracy-based features, unlike GSM8K where chance is effectively zero.Source: Table 7 · p. 15
• Thinking remains positive on StrategyQA but is carried almost entirely by entropy slope, with every other feature contributing |∆| < 0.21.Source: F StrategyQA as a Secondary Benchmark · p. 15
• After length adjustment, Instruct StrategyQA HCDS does not survive on either backend.Source: Table 8 · p. 15, F StrategyQA as a Secondary Benchmark · p. 15
• Thinking survives in 52.4% of CUDA and 57.1% of MLX feature subsets, versus 84.1%–90.5% for GSM8K cells.Source: F StrategyQA as a Secondary Benchmark · p. 15
• On GSM8K, HCDS is positive in all six output-length tiers and significant in five; Instruct short is the exception at +0.465, p = 0.23.Source: G Length-Matched HCDS · p. 16, Figure 4 · p. 16, Table 9 · p. 16

Anchor selection

• Gradient×activation attribution toward the answer span selects candidate reasoning anchors from segmented generated-token steps.Attributions are aggregated over each step’s generated token positions.Source: H Mechanistic Anchor Methodology —… · pp. 16–17

Mechanistic feature

• ∆Amech = Ic − (I1 + I2)/2 compares matched-control correctness with the average correctness after two anchor ablations.CUDA uses separate anchor and control redecodings; MLX uses a joint comparison with coarser support {−1, 0, +1}.Source: H Mechanistic Anchor Methodology —… · p. 17

Intervention

• Residual_zero ablation multiplies the residual-stream output by 0 at selected step-token positions before greedy decoding continues.Other token positions and layers remain unchanged.Source: H Mechanistic Anchor Methodology —… · p. 17

Control selection

• Controls match anchors by trace position with step length as a weak tie-break, but not by step function.The absence of function matching is an explicit limitation.Source: H Mechanistic Anchor Methodology —… · p. 17

Weighting robustness

• 95.2%–98.4% of GSM8K feature subsets are positive and 84.1%–90.5% significantly positive across models and backends.Negative subsets are degenerate latency/mechanistic combinations without behavioral or linguistic features.Source: Table 11 · p. 20, J Sensitivity to Feature-Subset… · p. 20

Backend robustness

• The MLX 8-bit feature-ablation pattern matches PyTorch/CUDA bf16: removing latency raises HCDS, whereas removing both entropy features produces the largest reduction.Error bars represent 95% bootstrap confidence intervals.Source: Figure J.1 · p. 20

Continuous weighting

• Instruct remains positive in > 98% of sampled continuous weightings and significantly positive in > 90%, while Thinking remains positive in > 94% and significantly positive in > 84%.Weights are sampled from a uniform Dirichlet distribution over the feature simplex rather than optimized.Source: J Sensitivity to Feature-Subset… · pp. 20–21

Primary family

• Six of seven uncorrected significant cells survive Holm correction, including all four headline GSM8K cells.The largest adjusted p-value among the headline GSM8K cells is 2.5×10^-3.Source: K Multiple-Comparisons Correction · p. 21, Table 12 · p. 21
• Under Holm correction, no length-adjusted negative-control cell is significantly positive.This controls the reported false-positive rate but does not establish equivalence to zero.Source: K Multiple-Comparisons Correction · p. 21

Robustness families

• All 24 leave-one-feature-out ablations survive Holm at α = 0.05, with largest adjusted p = 4.0 × 10^-3.Source: K Multiple-Comparisons Correction · p. 21
• Holm correction reduces the GSM8K significant-subset fraction from 84.1%–90.5% raw to 65.1%–84.1%, while Benjamini–Hochberg leaves it at 84.1%–90.5%.Correction changes no point estimate, so the sign-based weighting claim is unaffected.Source: K Multiple-Comparisons Correction · p. 21

Scope

• Multiplicity corrections address reported test families, but deterministic one-trial-per-question testing still has question identity—not model stochasticity—as its unit of variation.No p-value adjustment changes what that test is a test of.Source: K Multiple-Comparisons Correction · p. 21

Motivation

• Six features derived from the same generated response may be correlated, so Euclidean distance can double-count shared information.Source: L Correlation-Aware (Mahalanobis) HCDS · p. 22

Metric

• Mahalanobis distance replaces Euclidean distance using a shrinkage covariance matrix Sλ = (1 − λ)S + λI.Because features are z-scored, S is their correlation matrix; λ = 1 recovers the published Euclidean score.Source: L Correlation-Aware (Mahalanobis) HCDS · p. 22

GSM8K results

• GSM8K remains significant in all four Mahalanobis cells, with Ledoit-Wolf changes of at most 0.09.Fully unregularised whitening reduces published values by at most 0.13.Source: L Correlation-Aware (Mahalanobis) HCDS · p. 22, Table 13 · p. 22

Controls

• Under Mahalanobis distance, no negative-control cell is significantly positive.The marginal CUDA Instruct arithmetic control crossing the threshold is treated only as a consistency check.Source: L Correlation-Aware (Mahalanobis) HCDS · p. 23

Metric comparison

• Length residualisation leaves mean feature correlations of 0.105 for Thinking and 0.158 for Instruct on CUDA GSM8K.The largest surviving correlation is r = 0.54 between perturbation and mechanistic sensitivity for CUDA Instruct GSM8K.Source: L Correlation-Aware (Mahalanobis) HCDS · p. 23

Reporting choice

• The Euclidean score remains primary because it is simpler, requires no covariance estimate from 50 questions, and is closely reproduced by the correlation-aware variant.Source: L Correlation-Aware (Mahalanobis) HCDS · p. 23