PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable Answers, Unfinished Reasoning: Why Self-Consensus Is Not a Safe Early-Exit Signal

Yunxiang Mo, Donghao Zhao, Hejia Geng

arXiv:2609.09989v1cs.CL

TL;DR

The paper asks whether self-consensus can safely and economically stop reasoning in a single partial trajectory, and whether one rule can be reused across settings. It replays 3,520 consensus rules on frozen trajectories under fixed gates and compares them with a boundary-confidence control. No consensus rule clears the gates because agreement often reflects a repeated placeholder rather than termination, while the control clears them.

  • Problem

    The paper examines whether agreement among repeated probes of one partial reasoning trajectory is both safe and token-saving, and whether one rule can be selected once and reused across models and benchmarks.

  • Method

    The study replays a preregistered sweep of 3,520 consensus rules on frozen trajectories from two models and three benchmarks, using fixed token accounting and comparison gates.

  • Results

    No consensus rule clears any of the three fixed acceptance gates, whereas a boundary-confidence control swept through the same pipeline clears all three.

  • Takeaways & Limitations

    Used alone, self-consensus is not a safe early-exit signal because agreement measures persistence of the current answer rather than reasoning termination.

  • Takeaways & Limitations

    The negative result covers one boxed-answer probe suffix and one preregistered consensus schema; other probe wordings and signals outside that schema were not searched.

Abstract

from arXiv · show

A natural way to cut reasoning-model inference cost is to repeatedly probe a single partial trajectory for its current answer and stop once probes agree -- self-consensus. We ask whether any such rule is both safe and token-saving, and whether one can be selected once and reused. A preregistered sweep of 3,520 consensus rules, replayed on frozen trajectories from two models and three benchmarks, clears none of three acceptance gates fixed in advance; the frontier reproduces on a held-out split and on two unseen models -- while a boundary-confidence control (DEER) swept through the same pipeline clears all three. The reason lies in the signal: agreement establishes that the current answer persists under a fixed probing procedure, not that the reasoning has terminated -- a consensus-termination gap. Stopping on it commits non-terminal answers. At a rule still saving 32% of the tokens, one stop in nine fires on an answer the trajectory itself later abandons, and most of those stops cut off a correction it would otherwise have made. Widening the agreement window does not remove them: the share levels off near 7%, and by then the saving has fallen to 8%. Probe re-wording and a hand-labelled error taxonomy show the agreed answer is often a placeholder the model had not settled on. Used on its own as the stop signal, agreement fails not because it is insufficiently strict, but because it repeatedly measures the wrong object.

1 Introduction

The paper tests whether repeated agreement from probes of one partial reasoning trajectory can provide a safe, reusable early-exit rule. Across direct measurements and an exhaustive preregistered sweep, agreement instead tracks answer persistence rather than reasoning termination.

  • Motivation: Self-consensus repeatedly probes one partial trajectory for its current answer and stops when recent probe answers agree.The paper distinguishes this from self-consistency, which samples multiple trajectories independently.
  • Core diagnosis: Agreement establishes that the current answer persists under a fixed probing procedure, not that reasoning has terminated.The paper names this mismatch the consensus–termination gap.
  • Evidence: 54% of early re-probes return a different answer under changed wording, versus 16% near the trajectory’s end.This supports the claim that early extracted answers are sensitive to the elicitation procedure.
  • Evidence: At 32% token saving, one stop in nine commits an answer the trajectory later abandons, with 155 of 216 such stops pre-empting later corrections.Only 16 of those stops bank a correct answer from a trajectory ending wrong.
  • Robustness: Near 7% non-terminal stops, widening the agreement window reduces saving to 8% without closing the gap.At matched savings, the cheapest consensus rules cost 6.0 pp at 28% saving and 8.3 pp at 33%, versus 0.5 and 2.0 pp for DEER.
  • Evaluation: A preregistered sweep of 3,520 consensus rules clears no acceptance gate, while a non-consensus signal clears the same gates.The sweep uses frozen trajectories from two models and three benchmarks to test safety and token savings under commitment.

2 Related Work

Related work places consensus early exit among probe-based stopping methods, cost-aware exit policies, reasoning-level signals, independent-sampling methods, and external budget controls. The paper’s distinction is that its negative result concerns agreement used alone, whereas some related methods gate or supplement agreement.

  • Probe-based / consensus early exit: Probe-based methods query a partial chain of thought for its current answer and halt when recent probe answers agree.Dynasor and CertaIndex formalize this with a certainty index over recent probe answers.
  • Cost-aware comparisons of exit policies: Cost-aware comparisons evaluate learned prefix-feature stoppers against confidence, entropy, and running-answer exits across 18 task–model settings.The related comparison identifies regimes where different stopping signals perform best and settings with no certifiable aggressive policy.
  • Gating agreement behind a reasoning-level signal: PUMA retains answer agreement but consults it only once a detector provides a reasoning-level signal.This contrasts with the paper’s evaluation of agreement as a standalone stop signal.
  • Scope distinction: The paper’s negative result applies to agreement used alone; related systems may retain agreement as one component gated by another signal.The paper therefore measures whether the consensus signal itself can satisfy fixed safety and saving gates.
  • Self-consistency and independent sampling: Independent-path self-consistency uses agreement across diverse sampled reasoning paths, unlike probing one single chain.In that setting, the vote can outweigh an error on any one path; this structural distinction does not by itself establish termination tracking.
  • Alternative termination signals: Alternative termination signals include verifier or process-reward scores, learned halting signals, confidence dynamics, and boundary confidence.These approaches differ from consensus in what they read from partial reasoning or model confidence.
  • Test-time budget control: Budget forcing controls test-time compute externally rather than detecting when reasoning has terminated.Its use of continued reasoning provides evidence relevant to the paper’s mechanism.

3 Experimental Setup

The experiments replay stopping rules offline on frozen reasoning trajectories, using explicit token accounting and a balanced multi-environment evaluation. This isolates stopping-policy effects from changes to the reasoning being observed.

  • Frozen trajectories: Each problem contributes one frozen main trajectory, and offline probes cannot re-enter decoding or alter the underlying reasoning.Rules are therefore scored against the same fixed trajectories.
  • Probe banks: The dense probe bank re-probes every 64 tokens with simple@32, while an adaptive bank also probes at entropy drops and conclusion, reflection, or answer markers.The adaptive bank adds event-triggered opportunities beyond regular probing.
  • Token accounting: Total decode cost is T = s + p, net saving is (B − T)/B, and gross saving is (B − s)/B.The gap between gross and net saving is the charged probe cost; probe prefill is not charged.
  • Token accounting: Accuracy drop compares the committed answer at stopping position s with the frozen final answer, not with an agreement rate.This directly measures the consequence of stopping early relative to the trajectory’s eventual result.
  • Models, benchmarks, and splits: Development covers 18 model×benchmark×seed environments from two models, three benchmarks, and seeds 42/43/44.Problems use 60/20/20 train/dev/test splits by problem id, while confirmation uses later seeds and two unseen models on test only.
  • Models, benchmarks, and splits: Headline metrics are macro-averages over model×benchmark×seed environments, weighting each environment equally.This prevents the largest benchmark from dominating policy selection.

4 The Consensus–Termination Gap

Consensus can appear stable before reasoning terminates: early probes often elicit placeholders that later change, and stopping on them frequently prevents correction.

  • 4.1 The First Actionable Consensus: 40.2 pp: first actionable consensus reaches 50.5% accuracy versus 90.7% at completion.It fires on 1,477 of 1,500 trajectories.
  • 4.1 The First Actionable Consensus: 736 of 1,500 trajectories later abandon an answer they had previously agreed on, and 84.2% of those still finish correct.End-of-trajectory agreement is therefore distinct from mid-trajectory agreement.
  • 4.1 The First Actionable Consensus: 27.5%: early consensus accuracy across development environments versus 85.2% for trajectories run to completion.Later consensus is more accurate, but waiting trades away saving.
  • 4.2 Probe Wording Versus Position: 54%: differently worded probes disagree in the first tenth of a trajectory, compared with 16% near the end.Across paired positions, the two wordings are nearly equally accurate, indicating sensitivity in which answer is elicited rather than one suffix being defective.
  • 4.3 Error Taxonomy: 56.7% of stopped-but-wrong cases concern answers the model had not converged on, while 18.7% are probe-format artifacts and 24.6% are settled wrong values.A consensus rule cannot distinguish a forced placeholder from a settled belief.
  • 4.4 The Cost of a Non-Terminal Stop: 10.6% of stops at W=12 commit answers later abandoned, while the rule still saves 32% of tokens.Of 216 non-terminal stops, 155 pre-empt a correction and 16 bank a correct answer from a trajectory ending wrong.
  • 4.4 The Cost of a Non-Terminal Stop: 60% to near 7%: the non-terminal share falls as W widens, while net saving drops from 92% to 8%.The residual share remains about 7% beyond W=24, so stricter agreement does not remove the underlying risk.

5 Searching the Consensus Rule Space

The preregistered search tests whether windowed consensus can satisfy fixed safety-and-saving gates and whether its failure is specific to the consensus statistic. None of 3,520 consensus rules passes, whereas DEER clears the same gates and the contrast persists out of distribution.

  • 5.1 Preregistered Rule Space and Gates: 3,520 candidate consensus rules span window size, share threshold, probing schedule, maturity, validity, and certainty settings.The search includes large windows and strict unanimity, not just a single threshold choice.
  • 5.1 Preregistered Rule Space and Gates: Three preregistered gates require bounded macro accuracy loss, sufficient net token saving, and positive saving on a minimum fraction of environments.The gates were fixed in advance and not relaxed post hoc.
  • 5.2 Gate Outcomes Across the Rule Space: 0 of 3,520 rules clears any gate; rules keeping accuracy loss at or below 1.0 pp save at most 0.2%.The first rule saving 10% already costs 2.66 pp, while 30% saving costs 11.8 pp.
  • 5.4 Held-Out and Out-of-Distribution Confirmation: The frontier reproduces at r=0.98 on held-out test data and r=0.97 and r=0.94 on two unseen models, while the conservative consensus gate remains empty.DEER clears the gates on dev, test, and both unseen models.
  • 5.6 Locating the Failure: 13.7% of the 28.1% saving gap comes from the statistic, 8.9% from read positions, and 5.5% from denser-probing cost.Only changing the statistic lowers the accuracy price at every saving floor.

6 Conclusion

Self-consensus is not a safe standalone early-exit signal because agreement measures answer persistence under a probe, not reasoning termination. Across the preregistered search, stricter agreement buys lower risk only by surrendering token savings, while boundary confidence clears the same gates.

  • 6 Conclusion: Agreement often repeats a placeholder from unfinished reasoning, causing stops to pre-empt corrections the trajectory would otherwise make.The resulting consensus–termination gap leaves the safe-and-saving corner empty across 3,520 rules.

Limitations

The negative result is bounded by the searched consensus schema, competition-mathematics domain, evaluation design, and probe-cost accounting. The mechanism is supported by multiple measurements but not by a direct manipulation of probe dependence.

  • Mechanism evidence: The mechanism evidence combines a structural argument, wording experiment, hand-labelled error taxonomy, and window sweep rather than directly manipulating probe independence.A stronger test would vary dependence in probe streams and measure the resulting accuracy tax.
  • Scope: The negative result covers 18 development environments and confirmation on a held-out split and two unseen models, but only the searched consensus schema.Other probe wordings and signals are outside the exhaustive search, and the argument against broader consensus variants is not an exhaustive proof.
  • Comparison scope: The DEER comparison establishes that one non-consensus signal clears the gates, not that DEER is the best possible early-exit method or a leaderboard entry.The comparison is limited to the trial-answer-submit variant under the stated evaluation pipeline.
  • Accounting: The savings axis depends on dense re-probing every 64 tokens with 32-token probes, although the accuracy-tax result does not depend on the schedule.Gross-saving gates are also reported, with the outcome unchanged; sparser or KV-cache-reusing schedules could alter net savings.
  • Domain: The evaluation uses competition mathematics with checkable final answers, leaving transfer to open-ended reasoning, code, and agentic tasks out of scope.The reported gap and accuracy tax therefore have no demonstrated scope beyond these benchmark types.
  • Evaluation design: The 18-environment macro-average weights each model×benchmark×seed equally, while small AIME24 and AMC23 held-out sets make individual cells noisy.The development splits contain 6 and 8 problems per seed for AIME24 and AMC23, respectively.

C Reproducibility

Reproducibility is supported by released trajectories, probe banks, sweep archives, and aggregation scripts, alongside safeguards for grading and baseline discrepancies. The reported accuracy values depend on explicit evaluator and baseline conventions that are reconciled without changing the main gate outcome.

  • Released artifacts: Frozen trajectories, offline probe banks, the sweep archive, and aggregation scripts support reproducing the reported tables and figures.The archive contains 126,720 train+dev metric rows plus the DEER threshold sweep, and Table 10 and Figure 4 were reproduced from it.
  • Grading dependencies: The grading stack must be pinned because incompatible antlr4-python3-runtime versions can break latex2sympy2 deserialization.The released pyproject.toml pins antlr4-python3-runtime==4.7.2, and the grader self-check verifies evaluator behavior.
  • Baseline reconciliation: 82.94% is the live regrade of full generation over 18 environments, compared with 82.55% reported in Table 2 under the consensus-sweep baseline.The 0.39 pp discrepancy is exactly reconciled through grading and scoping choices, including one MATH500 summand-order case.
  • Baseline conventions: The replay scores unfinished baseline trajectories as wrong, whereas the self-check grades the stored answer either way; this changes exactly one development trajectory.The DEER bank applies the budget gate and places its baseline at 82.88%.
  • Gate robustness: The baseline discrepancy cannot move a gate result because grading affects accuracy only, while the cheapest 10% net-saving rule costs 2.66 pp against a 1.0 pp cap.The best rule within the cap saves 0.21%, so the discrepancy is much smaller than the relevant margins.
  • Error coding: The error taxonomy distinguishes wrong numeric collapse, wrong closed-form collapse, sign or symbol errors, reasoning gaps, and format artifacts.Reasoning gaps include probes reading intermediate quantities, placeholders, or guesses before a final candidate forms.

D Frontier and Reproduction Tables

Across the swept consensus frontier, reducing accuracy loss requires sacrificing token savings, while probe costs further widen the gap. DEER reaches safe-and-saving points where consensus does not, and the consensus failure persists under alternative weighting and density settings.

  • Accounting and mechanism: Probe tax creates a 4–7 pp gap between gross and net savings, but the accuracy floor from ongoing correction—not probe tax—keeps consensus outside the safe-and-saving region.Sparser probes, shorter probes, and KV-cache reuse can reduce probe tax, but not the correction-related accuracy floor.
  • Ablations: Changing probe positions does not lower the accuracy price in any savings band at or above 10%, whereas changing the statistic lowers it in all nine bands.The ablation attributes the gap primarily to the statistic under gross accounting, with position and statistic splitting higher saving floors under net accounting.
  • Robustness: The consensus family has many more firing opportunities than DEER at dense intervals, yet matching DEER’s decision density still produces worse saving and accuracy.At interval 512, consensus has a median of 7 opportunities, but its cheapest rule loses 1.57 pp for 1.65% saving versus DEER’s 0.33 pp for 28.2%.

E Supporting Figures: Sweep Surface, Selection, and Frontiers

Supporting figures show that the consensus safe-and-saving corner remains empty across selection, scale, architecture, and benchmark views. DEER stays within the conservative gate across splits and models, while consensus rules fail selection or generalization requirements.

  • Generalization: On held-out models, no consensus rule clears the conservative gate on all four models, while the dev-selected DEER points stay inside it everywhere.The two lower panels test entirely unseen models.
  • Frontiers: DEER’s trial-answer-submit frontier clears all three gates around τ ∈[0.97, 0.995], unlike the consensus frontier.The table uses macro averaging over 18 environments and the same token accounting.
  • Selection: None of the 478 consensus rules that pass the conservative gate on train passes it on dev, whereas all three DEER points remain inside the gate on every split.The dev-selected DEER thresholds clear all three preregistered gates in the selection pipeline.

F Case Studies

Trajectory case studies show why repeated probe answers can be stable without representing settled reasoning. Placeholders and probe-format artifacts can persist across long spans, while some consensus stops also simply lock in an answer the trajectory would never correct.

  • Persistent placeholder: A placeholder answer repeats for 27 probes before the trajectory reaches the correct value, so even W=24 fires on an unsettled answer.The trajectory eventually holds −24/25 to the end, despite the earlier stable 0.
  • Probe-format artifact: A probe-format artifact returns D for more than two thirds of a trajectory despite no answer option being D; the correct value 1/8 appears only at the final probe.The repeated letter reflects plausible suffix completion rather than a reported belief.
  • Wrong either way: In the minority wrong-either-way case, a stop commits 52 while the trajectory ends at 154, and both answers are wrong.Continued reasoning was not going to reach the correct answer, so the stop changes one wrong answer into another.

G Self-Consensus: Full Analysis

End-of-trajectory agreement can correlate with correctness, but online early agreement is unreliable because it fires before the answer has settled. The full analysis shows that continued reasoning often recovers from intermediate consensus answers.

  • Setup: The analysis uses frozen trajectories and separate probe forward passes, so the controller observes the full answer sequence without halting or altering the main trajectory.The dense probe bank reads current answers every 64 generated tokens on the main trajectories.
  • End versus online agreement: End-state unanimous agreement is 97.8% correct across 186 trajectories, but this reliable signal is unavailable when an online rule must stop at the first mid-trajectory agreement.A final five-probe unanimous window is 90.4% correct across 1,205 trajectories.
  • Naive stop: A naive three-probe stop reaches only 50.5% accuracy versus 90.7% at completion, losing 40.2 pp while saving 3301 tokens on average.It fires on 1,477 of 1,500 trajectories and stops 731 on outright wrong answers.
  • Recovery: Of 736 trajectories whose three-probe consensus differs from the eventual answer, 620, or 84.2%, ultimately finish correct.This recovery pattern explains why early consensus is especially damaging.
  • Timing: Consensus reliability improves with later first-consensus position, making the earliest consensus—the point an online rule uses—the least reliable.The result is normalized by each trajectory’s own span.

H Using the Negative Result

Agreement can remain useful as a precondition, but it cannot safely carry an online stop by itself. The practical direction is to measure reasoning convergence directly and require rules to transfer across splits.

  • Using the Negative Result: Agreement can serve as a necessary precondition, while a separate signal decides whether the trajectory is settled enough to stop.Early agreement reflects persistence of an elicited answer, not necessarily reasoning termination.
  • Using the Negative Result: 54% of differently worded probes return different answers in the first tenth of a trajectory, showing that early readings depend substantially on the query.This supports treating agreement as a property of the probing procedure rather than a direct belief or convergence signal.
  • Using the Negative Result: Demanding more agreement buys safety by giving back token savings, so stricter agreement windows do not solve the underlying problem.The useful direction is a better measurement of whether reasoning has converged.
  • Using the Negative Result: Two unevaluated alternatives are cross-query agreement and semantic similarity between successive reasoning prefixes.Both shift attention from the boxed answer span toward evidence about convergence in the reasoning process.
  • Using the Negative Result: 478 rules clear the conservative gate in-sample and 364 are admissible on test, but none is selectable from dev or clears all three splits jointly.A rule can look safe and saving in one environment or split without being reusable.
Loading 2609.09989v1…