PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion

Anum Afzal, Florian Matthes, Gal Chechik, Yftah Ziser

arXiv:2505.24362v2cs.CL

TL;DR

The paper asks whether zero-shot CoT success can be predicted before completion and develops a probe using LLM representations to answer this question. The probe works before token generation and outperforms a text-only baseline, while early stopping helps over no CoT but remains below full reasoning.

  • Problem

    The paper investigates whether LLMs know whether zero-shot CoT will succeed before completing reasoning and whether this knowledge can enable more efficient reasoning.

  • Method

    The authors construct labeled math datasets, extract LLM hidden representations before and during CoT generation, and train a probing classifier to predict final correctness.

  • Results

    The representation-based classifier predicts CoT success before any token is generated, reaches 60%–76.4% accuracy across datasets and models, and outperforms BERT.

  • Takeaways & Limitations

    Early representations can guide CoT efficiency decisions, although zero-shot early stopping still trails full CoT and may require targeted training methods.

  • Takeaways & Limitations

    The study is limited to three math datasets, one human-evaluated LLM, a zero-shot setting, deterministic generation, and white-box model access.

Abstract

from arXiv · show

We investigate whether the success of a zero-shot Chain-of-Thought (CoT) process can be predicted before completion. We discover that a probing classifier, based on LLM representations, performs well \emph{even before a single token is generated}, suggesting that crucial information about the reasoning process is already present in the initial steps representations. In contrast, a strong BERT-based baseline, which relies solely on the generated tokens, performs worse, likely because it depends on shallow linguistic cues rather than deeper reasoning dynamics. Surprisingly, using later reasoning steps does not always improve classification. When additional context is unhelpful, earlier representations resemble later ones more, suggesting LLMs encode key information early. This implies reasoning can often stop early without loss. To test this, we conduct early stopping experiments, showing that truncating CoT reasoning still improves performance over not using CoT at all, though a gap remains compared to full reasoning. However, approaches like supervised learning or reinforcement learning designed to shorten CoT chains could leverage our classifier's guidance to identify when early stopping is effective. Our findings provide insights that may support such methods, helping to optimize CoT's efficiency while preserving its benefits.

1 Introduction

The paper asks whether LLMs can predict CoT success before completing generation and whether this knowledge can support more efficient reasoning. It introduces representation-based prediction and finds useful signals before token generation, with early stopping remaining promising but incomplete.

  • LLMs’ internal representations predict CoT success before generating a single token, reaching 60%–76.4% accuracy across datasets and models.The classifier also outperforms BERT, indicating that representations contain information about intermediate calculations beyond textual cues.
  • In two of six cases, later reasoning steps do not significantly improve prediction accuracy, and earlier representations are more similar to final-step representations.This pattern suggests that some relevant reasoning information is encoded early.
  • The study frames CoT success prediction as a way to determine whether reasoning will reach a correct conclusion before full generation.The motivation is to understand when LLMs know their answer and whether additional reasoning is useful.
  • Zero-shot early stopping remains below full CoT performance but slightly outperforms solving without CoT, motivating targeted methods for shortening reasoning chains.The authors suggest supervised or reinforcement learning as possible ways to identify effective stopping points.

2 Related Work

Related work examines CoT mechanisms, reliability, and internal representations, while this paper focuses on predicting CoT success from what the model internally represents.

  • Prior studies analyze how CoT affects mathematical reasoning, including improvements, degradation, sensitivity to perturbations, and errors in intermediate steps.The literature also investigates whether apparently meaningful CoT reflects genuine internal computation.
  • This paper’s novelty is to study what the model internally knows about whether an ongoing CoT process will succeed.
  • Representation-based research has used LLM internal states to predict future tokens and inspect properties such as explanation honesty and encoded task information.These studies motivate examining internal representations as predictive signals rather than relying only on generated text.

3 Methodology

The methodology builds a deterministic CoT success dataset, extracts hidden states before and during generation, and trains a compact classifier to predict eventual correctness.

  • The dataset labels deterministic CoT generations as correct or incorrect and extracts initial hidden states under white-box access to the LLM.Hidden states are represented across layers, samples, and hidden dimensions, using the last prompt token because prompt lengths vary.
  • A compact three-hidden-layer feedforward network with ReLU activations and a sigmoid output predicts CoT success from the extracted representations.Adam or SGD is selected per dataset based on empirical performance.
  • Prediction over time uses prefixes containing 10% through 90% of generated tokens concatenated with the initial prompt.This percentage-based design avoids defining atomic reasoning steps with varying lengths and styles.
  • Table 1 compares problem-solving outcomes with and without CoT using a balanced dataset where CoT helps on 50% of problems.

4 Experimental Setup

Experiments compare text-only and representation-based prediction across three math datasets and two LLMs, using balanced labels and manual checks for part of the evaluation.

  • The study evaluates BERT’s token-based baseline against LLM hidden representations to test whether textual cues alone predict CoT success.BERT provides a black-box comparison without access to internal LLM states.
  • Experiments use Olympiad, cn-k12, and AQuA with Llama-3.1-8B and Mistral-7B, producing six dataset–model variants.The datasets vary in mathematical difficulty and problem format.
  • Without CoT, Llama-3.1-8B achieves 9.2% on AQuA, 17.69% on Olympiad, and 25.55% on cn-k12, versus the enforced 50% CoT accuracy.These values are reported in the comparison summarized by Table 1.
  • Human and GPT-4o mini labels agree on 90.9%, 94.8%, and 93.4% for AQuA, Cn-K12, and Olympiad, respectively, but the disagreement requires caution.For Mistral-7B, all splits are annotated with GPT-4o mini.

5 Results and Discussion

Internal LLM representations predict CoT success before and during generation, with performance varying by layer, dataset, and reasoning stage. Early stopping remains brittle, although incomplete CoT can outperform no CoT in some datasets.

  • 5.1 Prediction before Generation: 76.4% accuracy on Llama-3.1-8B Olympiad versus BERT’s 69.1% demonstrates that internal representations outperform token-only prediction; AQuA reaches 60.0%.Across Llama-3.1-8B datasets, BERT ranges from 53.5% to 69.1%, while the internal-representation probe outperforms it; random chance is 50%.
  • 5.1 Prediction before Generation: Layer 14 and layer 16 are consistently effective across all three datasets, while middle layers 11–14 and 16–17, and sometimes final layers, encode success or failure.Figure 2 reports accuracy for all 33 Llama 3.1 8B Instruct layers.
  • 5.2 Prediction over Time: Mistral-7B AQuA negative examples contain linguistic cues that may help BERT classify them, illustrating a dataset-specific alternative to internal-representation signals.The reported cues include generations that question whether the provided question is incorrect.
  • 5.2 Prediction over Time: Later reasoning steps do not significantly improve prediction in two of six cases, specifically when earlier representations already resemble later reasoning states.The analysis links this pattern to higher similarity between early and final representations in cases where added CoT context is unhelpful.
  • 5.2 Prediction over Time: SVCCA compares representations at 10%–90% of reasoning with the 100% completion state; AQuA shows lower early-to-late similarity than Olympiad and Cn-K12.Higher SVCCA scores indicate greater similarity in encoded information, using Layer 14 for the three Llama-3.1-8B datasets.
  • 5.2 Prediction over Time: 57% consistency in Olympiad highlights that zero-shot early stopping is suboptimal because the model may not converge to a stable answer before nearly full generation.Nevertheless, halting midway in AQuA and Cn-K12 slightly outperforms the no-CoT setting, despite remaining below full CoT completion.

6 Conclusion

The study shows that CoT success can be predicted from LLM internal representations before any token is generated, while additional reasoning steps do not always improve prediction. Early representations may therefore support more efficient CoT, although zero-shot early termination does not fully preserve completed reasoning performance.

  • Before generating a single token, LLM internal representations can predict whether the CoT reasoning process will succeed.
  • In some cases, exposing the classifier to intermediate reasoning steps does not improve prediction accuracy.
  • Early steps resemble final steps more closely when additional reasoning context is unhelpful, indicating that useful answer information may be encoded early.
  • Initial zero-shot experiments suggest early termination may be possible, but zero-shot prompting alone does not fully unlock this potential.

Limitation

The study’s generalizability is constrained by its narrow evaluation setting and methodological assumptions. It covers one LLM, three math datasets, zero-shot prompting, deterministic generation, and white-box model access.

  • The evaluation uses human annotation for only one LLM and three math datasets in a zero-shot setting, limiting generalizability.
  • Temperature-zero generation reduces stochastic noise but may not represent variability in real-world stochastic LLM use.
  • The method assumes white-box access to internal model representations, which is generally unavailable for proprietary models.

Ethics Statement

The study reports no fine-tuning and uses open-source datasets, while manual evaluation targets math problems with one logically correct answer. These choices are presented as reducing potential sources of bias in the analysis.

  • The research evaluates LLM internal representations without performing fine-tuning that could introduce additional model bias.
  • The study uses open-source datasets and therefore reports no additional bias beyond what those datasets may already contain.
  • Manual evaluators corrected math problems with one logically correct answer, reducing the possibility of biased annotations.

A Technical Details

The classification models are tuned separately for each dataset and layer. The best hyperparameters are reported in Table 7 and reused in the prediction-over-time experiments.

  • Hyperparameters are tuned separately for every dataset and model layer during classifier training.
  • The best hyperparameters are documented in Table 7 for the classification models.
  • The selected hyperparameters are used in the prediction-over-time experiments.

B Additional Results

Additional analyses compare internal-representation probing with token-based baselines, examine representation similarity across reasoning stages and layers, and illustrate early stopping in CoT.

  • Complementary Analysis for Probing: SVCCA compares representations from reasoning steps at 10% through 90% with the completed 100% CoT representation.The analysis evaluates similarity across all possible step combinations.
  • Early Stopping in CoT Reasoning: Table 8 illustrates artificially halting generation after 30% of the original answer while retaining the correct answer in the example.This example demonstrates the paper’s early-stopping setup rather than establishing its aggregate performance.
  • Classification Performance Breakdown: 317 versus 250 true negatives on Cn-k12, 177 versus 131 on AQuA, and 388 versus 315 on Olympiad show our model outperforms BERT in detecting true negatives.The comparison uses a 1,000-example test set and complements the model’s performance on true positives.
  • Classification Performance Breakdown: The analysis frames internal representations as capturing predictive signals beyond generated-token cues, supporting a more nuanced account of factors associated with CoT success.This interpretation follows the reported comparison between the internal-representation model and BERT.
  • LLM Layer Analysis: Layer analysis compares hidden representations with and without CoT prompts to identify layers whose representations differ most across the two settings.The comparison is averaged over 1,000 samples in the reported five-layer analysis.

C Dataset Examples

The paper provides dataset and evaluation examples covering prediction at different generation steps, dataset construction, and a confusion-matrix comparison at T = 0.

  • Success Prediction over Time: Table 10 illustrates samples from the Success Prediction over Time method at two different generation steps.
  • Classification Comparison: Table 9 compares BERT and the proposed model at T = 0 using confusion matrices on balanced test sets and their best-performing layers.
  • Dataset Construction: Table 10 uses a Cn-k12 example to illustrate how the Prediction over Time dataset is constructed.
Loading 2505.24362v2…