PaperlayerView exact revision on arXiv

Source-linked AI summary

TERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoning

Alliot Nagle, Jakhongir Saydaliev, Dhia Garbaya, Michael Gastpar, Ashok Vardhan Makkuva, Hyeji Kim

arXiv:2603.12529v2cs.LGcs.AIcs.CL

TL;DR

LRMs often overthink after generating their eventual answer, but optimal CoT lengths are task- and model-dependent. Terminator identifies the first logical arrival of the final answer, uses those positions to train a token-level early-exit predictor, and reduces CoT lengths by 14%-55% across four datasets while cutting inference latency by more than 2x. The method outperforms current state-of-the-art methods on the reported accuracy-efficiency comparisons.

  • Problem

    LRMs can spend substantial compute double-checking or exploring after their final answer has appeared, while determining optimal CoT lengths remains task- and model-dependent.

  • Method

    Terminator identifies the first logical arrival of an LRM’s final answer, constructs optimal-length CoT supervision through answer extraction and verification, and trains a token-level early-exit predictor.

  • Results

    14%-55% reductions in CoT lengths are achieved across MATH-500, AIME 2025, HumanEval, and GPQA, with more than 2x lower inference latency than the original LRM.

  • Takeaways & Limitations

    Terminator provides an inference-time early-exit strategy with favorable accuracy-efficiency trade-offs across four evaluated LRMs and practical datasets.

  • Takeaways & Limitations

    Event-locked signals require multiple CoTs and full answer-position knowledge, so translating them directly into online inference remains challenging.

Abstract

from arXiv · show

Large Reasoning Models (LRMs) achieve impressive performance on complex reasoning tasks via Chain-of-Thought (CoT) reasoning, which enables them to generate intermediate thinking tokens before arriving at the final answer. However, LRMs often suffer from significant overthinking, spending excessive compute time even after the answer is generated early on. Prior work has identified the existence of an optimal reasoning length such that truncating reasoning at this point significantly shortens CoT outputs with virtually no change in performance. However, determining optimal CoT lengths for practical datasets is highly non-trivial as they are fully task and model-dependent. In this paper, we precisely address this and design Terminator, an early-exit strategy for LRMs at inference to mitigate overthinking. The central idea underpinning Terminator is that the first arrival of an LRM's final answer is often predictable, and we leverage these first answer positions to create a novel dataset of optimal reasoning lengths to train Terminator. Powered by this approach, Terminator achieves significant reductions in CoT lengths of 14%-55% on average across four challenging practical datasets: MATH-500, AIME 2025, HumanEval, and GPQA, while outperforming current state-of-the-art methods and reducing inference latency by more than 2x compared to the original LRM.

1 Introduction

Large Reasoning Models improve difficult-task performance through extended Chain-of-Thought reasoning but often overthink after reaching their final answer. TERMINATOR targets this waste by learning when the answer first logically arrives and exiting early.

  • Motivation: LRMs generate thousands of additional thinking tokens and may continue checking or exploring after producing the final answer.This overthinking wastes inference compute, while prior work indicates CoTs can often be shortened substantially with little accuracy loss.
  • Motivation: Prior work suggests an optimal reasoning length because performance rises, peaks, and then declines as reasoning length increases.Existing approaches either retrain LRMs or terminate inference without determining and using the optimal length for practical tasks.
  • Evaluation: TERMINATOR is evaluated against prior methods using accuracy and compression rate, where lower compression rates indicate greater token savings.The Pareto frontier connects non-dominated accuracy–compression solutions.
  • TERMINATOR: TERMINATOR defines hindsight-optimal reasoning length as the fewest tokens needed before the LRM reaches the same final answer as in the unshortened CoT.It marks the first logical arrival of the final answer as the exit position.
  • TERMINATOR: The first final-answer arrival is associated with shifts in token-level confidence and token-usage distributions that can train a binary probe for early exiting.These signals support a training dataset of optimal-length CoTs and online inference-time termination.

2 Preliminaries

The paper formalizes LRM reasoning as a generated Chain-of-Thought followed by a solution, defines hindsight-optimality over the completed trajectory, and introduces Token-Confidence for analyzing uncertainty during generation.

  • 2.1 Notation: An LRM autoregressively generates a reasoning sequence r during thinking and a solution sequence s containing a final answer.The final answer from the full CoT is distinguished from any answer produced after an early exit.
  • 2.3 Token-Confidence: Figure 2 uses event-locked averaging to align CoTs at their first final-answer position and examine confidence and log-probability trajectories.The averaged signal shows agreement around the answer position that is not readily visible in a single sample.
  • 2.1 Notation: Accuracy measures the proportion of problems for which the correct answer is produced.The notation distinguishes the LRM’s final answer from the ground-truth answer.
  • 2.2 Hindsight-optimality: HORL is the earliest position in a completed CoT where the LRM has logically reached its final answer.It is a retrospective property of the realized CoT and final answer.
  • 2.3 Token-Confidence: Token-Confidence measures the uncertainty of a chosen token during CoT generation.The paper uses this metric to analyze confidence trajectories across reasoning positions.
  • 2.3 Token-Confidence: Figure 3 compares normalized frequencies of thinking tokens before and after the final answer, with dot size representing relative CoT length.The Token-Confidence formulation averages negative log-probabilities over the Top-K predictions with K=20.

3 Motivation

LRMs can continue reasoning after first reaching their eventual answer, but detecting that arrival online is difficult. TERMINATOR addresses this by learning token-level signals from hindsight-optimal answer positions.

  • Motivation: Overthinking occurs when LRMs explore alternatives after already generating the final answer, increasing inference-time cost.Prior work found that CoT length can sometimes be reduced substantially without changing performance.
  • Motivation: Hindsight-optimal reasoning identifies the earliest position where the completed CoT logically reaches the LRM’s final answer.This position is determined retrospectively from the realized CoT and answer.
  • Answer-arrival signals: Token-confidence and thinking-token frequencies shift around the first answer occurrence across math, science, and coding data.The observed changes include higher frequencies of “hmm” and “okay” before the answer and “another” after it.
  • Online inference challenge: Event-locked averages and token-frequency analyses are difficult to use online because they require answer positions or multiple completed CoTs.These findings indicate an underlying trend but do not directly provide a single-CoT online stopping rule.
  • Motivation: TERMINATOR predicts whether the final answer has been generated using a probe classifier over final-layer hidden states and token-level training labels.Its training data uses hindsight-optimal CoTs rather than coarse heuristic chunks.

4 TERMINATOR: Methodology

TERMINATOR converts hindsight-optimal answer positions into scalable token-level supervision and uses a lightweight probe to predict when reasoning should stop. Its curation pipeline extracts, identifies, verifies, and indexes the earliest answer occurrence.

  • Answer-position dataset: A scalable pipeline constructs training labels by detecting the earliest logical arrival of the LRM’s final answer in completed CoTs.The pipeline is needed because reliable answer-position detection is a semantic search problem rather than simple pattern matching.
  • Answer-position dataset: The extract-identify-verify pipeline extracts the final answer, identifies a preceding span containing it, verifies that span, and retrieves its exact token position.Failed verification triggers identification retries with feedback; unresolved examples are excluded within the retry limit.
  • Probe classifier: TERMINATOR trains a small classifier on final-layer hidden states to make a binary exit prediction at every CoT position.The model reuses the LRM’s final transformer block and adds a prediction head.
  • Probe classifier: Class-weighted binary cross-entropy handles the imbalance between exit and non-exit labels.The labels and predicted probabilities are defined for each CoT position relative to the extracted answer-arrival index.
  • Design rationale: TERMINATOR is trained on hindsight-optimal CoTs and avoids data-calibrated thresholding while combining math, coding, and STEM sources.This design targets early exit as soon as the final answer is generated.

5 Experiments

Experiments evaluate TERMINATOR across multiple LRM families and four tasks against prompt- and probe-based baselines. It occupies a favorable accuracy-efficiency frontier, halves latency, and shows strong in-distribution but mixed OOD behavior.

  • Experimental setup: The evaluation covers Qwen3 and Ministral model families, with AIME 2025, MATH-500, HumanEval, and GPQA as test datasets.Training mixes AIME, MATH, OpenCoder-SFT, and OpenScience data.
  • Inference procedure: Inference uses a sliding window of 10 predictions and injects </think> when more than 50% of labels are 1.The main results use a 0.7 threshold for predicting label 1.
  • Main results: TERMINATOR consistently occupies a favorable accuracy-efficiency Pareto frontier across four evaluated LRMs.Methods with more aggressive token reduction can incur significant accuracy degradation.
  • Latency analysis: TERMINATOR halves average latency over the vanilla LRM, with 10.8% overhead for Qwen3-8B and 7.5% for Qwen3-14B.The latency comparison uses the same MATH-500 subset, batch size, hardware, and disabled prefix caching.
  • Early termination: TERMINATOR remains close to hindsight-optimal stopping across datasets in truncation evaluations.The hindsight-optimal marker is not achievable during ordinary inference.
  • OOD evaluation: OOD compression is best in-distribution, while OOD training can slightly improve accuracy but often delays exiting and reduces token savings.OpenCoder-SFT training improved GPQA accuracy but worsened compression to 96% in the cited example.
  • Signal recovery: TERMINATOR’s predicted exit positions recover the reported event-locked confidence and thinking-token phenomena, with prediction errors concentrated near zero and a median difference of 7.The recovery is evaluated by comparing ground-truth and predicted answer positions.

6 Related Work

Related work improves efficiency through prompt compression, latent reasoning, or early CoT termination. Early-exit methods differ in whether they are training-free or learned and generally use consistency-based stopping signals.

  • Prompt Compression: Prompt-compression methods shorten the input context using soft or hard prompts before LLM processing.Soft-prompt approaches use embeddings but are less suitable for black-box models and harder to analyze theoretically.
  • Efficient Reasoning: Latent-reasoning methods replace discrete reasoning tokens with latent hidden states to reduce model passes before the final answer.Examples include Coconut, CCoT, Soft Thinking, and LightThinker.
  • Early-Exit Reasoning: Early-exit reasoning methods terminate CoTs by injecting </think> when consistency-based signals cross a threshold.Existing approaches include training-free methods such as EAT, DEER, ES-CoT, and Dynasor, alongside learned methods.

7 Conclusion

TERMINATOR provides an early-exit method for LRM reasoning, supported by an optimal-length CoT dataset and analyses of Token-Confidence and “thinking token” usage. The authors identify more efficient training-data curation as future work.

  • TERMINATOR is an early-exit method for LRM reasoning trained using an optimal-length dataset of CoTs.
  • The paper analyzes Token-Confidence during reasoning and shifts in “thinking token” usage to provide insights into LRM behavior.
  • The training-data curation pipeline works well, but future work could make training more efficient because tens of thousands of CoTs are used.
  • The averaged Token-Confidence result is likened to event-related potential research, while the authors explicitly avoid claiming exact alignment with ERP findings.

A.1 Limitations

The approach is constrained by assumptions about answer commitment and answer presence, and its answer-position pipeline faces ambiguity in numerical, mathematical, and code outputs. Confidence-based early termination also raises risks of premature conclusions.

  • The method assumes that exiting at the earliest final-answer occurrence is sufficient for the LRM to use that answer in its final solution.
  • The LRM may not commit to the exited answer, causing an incorrect exited response even when full reasoning eventually produces the correct answer.
  • The approach depends on the final answer appearing within the CoT, which may not hold for open-ended generation tasks.
  • Confidence-based early termination can produce over-confidence and premature conclusions, especially where incorrect outputs carry significant consequences.
  • Pattern matching cannot reliably identify answer positions when numerical values recur as intermediate calculations or coincidental tokens.
  • Mathematical expressions may be semantically equivalent but syntactically different, while Python functions may be generated non-contiguously and absent verbatim.

B.4 TERMINATOR Training Details

TERMINATOR uses a compact transformer-based classifier with majority voting and thresholding at inference. Training analyses favor the copied-initialization transformer design, while threshold changes trade compression against modest accuracy changes.

  • Architecture comparisons: The proposed architecture uses one copied-initialization transformer layer followed by a one-layer MLP, alongside random-initialization, deeper-MLP, and MLP-only variants.
  • Architecture comparisons: MLP-only architectures incur a substantial 10–17% drop in Macro-F1, whereas adding a second MLP layer improves performance by only „0.9%.
  • Inference controls: A window of 10 recent predictions with strict majority voting serves as a consistency check against a single spurious early-exit prediction.
  • Inference controls: Once exit probability exceeds the threshold, it rarely drops below, so window size has minimal impact on the majority vote.
  • Inference controls: Lowering the predictive threshold substantially improves compression while only modestly affecting accuracy across the evaluated datasets.
  • Context length: TERMINATOR is trained and evaluated up to 32,768-token contexts, with Mistral AIME experiments extending to 65,536 tokens.

B.6 Implementation of the Baseline Methods

The baseline implementations include interval-based probing and linear probes with calibrated stopping controls, while supplementary figures examine TERMINATOR’s predicted exits against ground-truth answer positions and token-use biases.

  • Baseline methods: Dynasor probes every 64 tokens and exits after consistent answers appear across eight probing intervals in its mild configuration.
  • Baseline methods: Thought Calibration trains linear probes on segmented reasoning trajectories, with Supervised and Consistent variants predicting stopping-related conditions.
  • Baseline methods: Thought Calibration controls stopping through tolerance, a calibrated score threshold, and a window averaging consecutive reasoning steps.
  • Baseline methods: The reported Supervised probe outperformed the Consistent probe across test datasets, while both are compared in the broader results table.
  • Supplementary analyses: TERMINATOR’s predicted exit positions recover ground-truth event-locked average spiking, with their differences concentrated near zero on a log-scaled histogram.
  • Supplementary analyses: TERMINATOR’s predicted exits recover the same biases in “thinking token” occurrence rates as ground-truth answer positions.

C.1 Early-Exit Signal Analysis

TERMINATOR is motivated by answer-linked signals in reasoning, including token-confidence spikes and shifts in “thinking token” usage around the first answer. Its adaptive-threshold and domain-selection analyses show limited gains over fixed or unified configurations, while introducing practical overhead and assumptions.

  • Early-exit signals: Token-confidence spikes at the first occurrence of the final answer, then sharply decreases and later recovers before plateauing across data sources.This pattern supports using confidence as an early-exit signal, although its exact contours vary by source.
  • Early-exit signals: 63.9% of CoTs use “hmm” more often before the answer, while 91.5% use “okay” more often before it; “another” is more frequent afterward.Usage rates also differ substantially across datasets, such as “alternatively” appearing above-diagonal in 80.4% of MATH points versus 19.2% for OpenCoder-SFT.
  • Early-exit signals: Shorter CoTs correlate with higher occurrence rates for “hmm,” “okay,” and “another” after normalizing before- and after-answer frequencies.The side with the higher rate tends to decay most as CoT length increases.
  • Adaptation choices: Cross-domain training achieves 1% higher accuracy than single-domain training but a 1.5% worse compression rate.Both approaches yield good performance without a clear overall winner.
  • Adaptation choices: Domain-adaptive thresholding generally lies above fixed thresholding on the Pareto frontier, but its performance improvement is marginal.The fixed configuration uses threshold 0.7 across MATH-500, AIME 25, HumanEval, and GPQA, while adaptive results assume domain knowledge in practice.

C.3 TERMINATOR Prediction Analysis

TERMINATOR’s prediction behavior is examined through confidence streams, answer-position distributions, generalization, cost-benefit, and hard-problem case studies. The analyses indicate clearer exit transitions on some datasets and weaker detection on harder questions, while curation costs are substantial but potentially recoverable.

  • Prediction behavior: MATH-500 and HumanEval show sharp predicted-confidence transitions at the exit threshold, with good separation in event-locked and individual examples.The cited analysis covers predicted probability streams across AIME25, MATH-500, HumanEval, and GPQA.
  • Generalization: OOD compression is best when training and evaluation are in-distribution, but accuracy does not always follow the same pattern.For example, OpenScience training yields the lowest GPQA accuracy despite being in-distribution, while OpenCoder-SFT improves accuracy but worsens compression to 96%.
  • Prediction behavior: TERMINATOR may not detect the appropriate answer position cleanly on harder questions, whereas easy problems show a clearer transition to overthinking.The case study reports this behavior on Qwen3-14B CoTs.
  • Cost-benefit: The curation pipeline requires approximately 1,416 GPU-hours for 110,799 CoTs, while the break-even prompt count depends on token savings and prompt length.The cost-benefit analysis estimates GPU-seconds saved per prompt and prompts needed to offset one-time curation compute.
  • Cost-benefit: The estimated break-even prompt count is small relative to Qwen3-8B’s approximately 9.9 million monthly Hugging Face downloads.The paper notes that one prompt per download would compensate for curation compute many times over.

C.5 Disentangling Hindsight-Optimal Labels and Token-Level Exit Prediction

The ablations separate hindsight-optimal labels from prediction granularity, showing that both matter: the labels identify meaningful exit positions, while token-level prediction provides the resolution needed to exploit them.

  • Design combinations: TERMINATOR combines hindsight-optimal labels, marking the earliest final-answer occurrence, with token-level prediction at every generated token.This contrasts with consistency-based labels and chunk-level checkpoints used in prior early-exit approaches.
  • Prediction granularity: Chunk-level prediction substantially underperforms token-level TERMINATOR, with 56.7% accuracy and 24.1% compression versus 73.7% accuracy and 67.8% compression.The chunk-level variant exits less aggressively but still suffers large accuracy drops, especially on AIME25 and GPQA.
  • Label supervision: Consistency-based token-level labeling is computationally infeasible because it would require forced-answer generation after every token.At 8,000 tokens and one second per forced answer, labeling one CoT would require roughly 2.22 hours.
  • Additional analyses: Token-confidence and thinking-token analyses are reproduced across AIME, MATH, OpenCoder-SFT, and OpenScience, with additional results linking token usage to CoT length.These supplementary figures extend the signal analysis across datasets and examine predicted termination behavior.
  • Overall ablation finding: Hindsight-optimal labels set the achievable performance ceiling, while token-level prediction provides the granularity needed to reach it.The analysis attributes meaningful early-exit positions to the labels and fine-grained exploitation to token-level prediction.
Loading 2603.12529v2…