Learning to Stop without Learning to Stop: Self-Supervised Confidence Training Improves Reasoning Efficiency

1 INTRODUCTION

• ConfSFT trains models to predict confidence at intermediate points in their own reasoning, using self-derived targets and no loss for length, efficiency, or stopping.Inference uses standard generation without confidence elicitation or a stopping mechanism.Source: §1 · p. 2
• Confidence is associated with more reliable and stable intermediate answers, while the expected benefit of further reasoning decreases as confidence rises.This motivates confidence as a signal about both the current answer and the value of continuing.Source: §1 · p. 2
• Up to 25% fewer generated tokens at matched accuracy, with gains transferring from mathematical training problems to scientific and coding tasks.The gains are comparable to methods that explicitly optimize for shorter reasoning.Source: §1 · p. 2
• Confidence prediction improves during training as generated tokens decrease and accuracy remains stable, while high-level reasoning composition is largely preserved.Alternative targets, including randomly shuffled confidence labels, produce weaker or no efficiency gains.Source: §1 · p. 2

2 RELATED WORK

• Training-time efficiency methods explicitly favor shorter reasoning through length-penalized reinforcement learning or concise-trajectory fine-tuning.Other approaches train models to adapt or terminate reasoning through truncated rollouts, selective reasoning, or confidence-guided post-training.Source: §2 · p. 2
• Inference-time methods use confidence, uncertainty, answer stability, or probes to decide when reasoning should stop.ConfSFT does not use a stopping criterion during inference.Source: §2 · p. 2
• Confidence and metacognitive signals have been used for self-verification, calibration, confidence-dynamics shaping, and reinforcement-learning feedback.Examples include verbalized confidence labels and confidence-related alignment objectives.Source: §2 · p. 2
• ConfSFT derives intermediate confidence targets from the model’s token probabilities and trains without gold correctness labels or external feedback.It studies whether self-supervised confidence learning alone can induce more efficient reasoning.Source: §2 · p. 3

3 CONFIDENCE AND THE VALUE OF FURTHER REASONING

• Confidence is the length-normalized likelihood of a greedily generated trial answer at an intermediate reasoning state, computed from the model’s token probabilities.The score does not require knowing whether that trial answer is correct, and the paper does not interpret it as a calibrated probability of correctness.Source: §3 · p. 3
• Higher confidence is associated with greater trial-answer correctness and a higher likelihood that the trial answer matches the eventual final answer.The evaluation groups intermediate states from Nemotron AIME 2024 traces into equal-sized confidence quantiles, with roughly 1,700 states per bin.Source: §3 · p. 3
• After first reaching ct ≥0.95, Nemotron generates a median of 1,766 additional tokens, with many trajectories continuing for more than 10K tokens.The confidence analysis uses intermediate reasoning states immediately preceding each Wait in 480 traces, with 16 samples per problem.Source: §3 · p. 3
• The expected accuracy gain from continued reasoning decreases sharply with confidence, with little average improvement once confidence is high.Further reasoning is most beneficial at low-confidence states.Source: §3 · p. 3

• The method uses Wait markers to select intermediate reasoning prefixes, retaining at most M decision points per rollout when there are many.Selected points are spread approximately uniformly across the trajectory.Source: §4 · p. 4
• At each selected state, the model greedily generates a trial answer whose token probabilities define a length-normalized confidence score, then quantizes that score into a textual percentage target.The confidence score is the geometric mean of the generated answer’s token probabilities and does not require the gold answer.Source: §4 · p. 4
• Training concatenates the problem, reasoning prefix, priming prefix, and confidence label, but masks all tokens except the confidence target from the loss.At inference, the fine-tuned policy uses standard generation without confidence elicitation or an early-exit mechanism.Source: §4 · pp. 4–5, Figure 3 · p. 5
• ConfSFT repeatedly samples reasoning rollouts from the current policy, constructs confidence labels, and fine-tunes the policy across training rounds.Later rounds generally yield more efficient policies.Source: §4 · pp. 4–5

5.1 SETUP

• The setup evaluates Gemma, Qwen, Nemotron, and GPT-OSS on AIME2025, GSM8K, GPQA-Diamond, HumanEval, and LiveCodeBench, using accuracy and average generated tokens.Training uses AIME2000–2023 problems, with AIME2024 held out for validation; baselines include DEER, A&Z, and On-Policy SFT.Source: §5.1 · pp. 5–6

5.2 MAIN RESULTS

• ConfSFT reduces generated tokens by 11.1% on Nemotron, 10.3% on Gemma, 19.2% on Qwen, and 11.1% on GPT-OSS while maintaining base-model accuracy.Although trained only on AIME problems, the efficiency gains transfer to scientific reasoning and coding benchmarks.Source: §5.2 · p. 6, Table 1 · p. 6
• On Qwen, ConfSFT reduces tokens by 19.2% at 69.5 average accuracy, compared with 17.2% at 69.3 average accuracy for On-Policy SFT.Across evaluated model families, ConfSFT achieves comparable or larger efficiency gains despite no supervision favoring shorter reasoning.Source: §5.2 · p. 6
• DEER can reduce tokens more, but its task-dependent results include coding accuracy drops on Nemotron from 44.2% to 17.4% on LiveCodeBench and from 89.7% to 47.1% on HumanEval.ConfSFT reduces tokens without inference-time stopping and preserves accuracy across domains.Source: §5.2 · p. 7

5.3 CONFIDENCE LEARNING AND EFFICIENCY DYNAMICS

• During Nemotron training, confidence mean absolute error and generated tokens decrease while accuracy remains approximately unchanged; confidence predictions also span a broader range after training.Gemma shows similar confidence-learning dynamics.Source: §5.3 · p. 7, Figure 4 · p. 7

5.4 REASONING COMPOSITION UNDER EFFICIENCY TRAINING

• ConfSFT largely preserves base-policy episode shares, whereas L1-Max, ThinkPrune, and On-Policy SFT substantially redistribute computation across episodes such as Implement, Explore, and Verify.The analysis compares signed token-share changes and total variation distance between before- and after-training episode distributions.Source: Figure 5 · p. 7, §5.4 · p. 8

6.1 WHAT SUPERVISION SIGNAL MATTERS?

• The ablations replace the target with trajectory position, binary correctness, or shuffled confidence while keeping the training procedure fixed.Binary correctness uses the gold answer; shuffled confidence preserves the label distribution but breaks the state-label correspondence.Source: §6.1 · p. 8
• Shuffling confidence labels largely removes or reverses efficiency gains, while position and binary-correctness targets yield weaker, less consistent improvements.The comparisons suggest that preserving the correspondence between each reasoning state and its graded confidence target matters.Source: §6.1 · p. 8, Table 2 · p. 8

6.2 CHOICE OF DECISION POINTS

• Paragraph boundaries provide many more candidate decision points than Wait, and results with both markers support treating them as sampling mechanisms rather than semantic components.The paragraph-boundary variant uses fewer training problems per round and a smaller learning rate.Source: §6.2 · p. 9
• On Gemma, paragraph-boundary ConfSFT reduces generated tokens by 8.4% at maintained average accuracy, versus 10.3% with Wait.The variants use different training configurations, so this is not a controlled comparison of markers.Source: §6.2 · p. 9, Table 3 · p. 9

7 CONCLUSION

• Confidence training on only 600 AIME problems reduces generated tokens at matched accuracy across model families and mathematical, scientific, and coding benchmarks.The reported accuracy–efficiency tradeoffs are comparable to methods that explicitly optimize for shorter reasoning.Source: §7 · p. 9

A.1 REASONING PROMPT (ROLLOUTS AND EVALUATION)

• Rollouts and mathematical evaluations use the same boxed-answer prompt, while science and coding benchmarks use task-specific instructions.No confidence instruction is added at training or test time.Source: §A.1 · p. 13

A.2 DECISION-POINT MARKERS

• Decision points are sampled at Wait markers before the end-of-thinking marker, capped at 32 approximately uniformly spaced points; the variant substitutes paragraph boundaries.Special tokens are retained where needed to identify the end-of-thinking marker.Source: §A.2 · p. 13

A.3 TRIAL-ANSWER PROBE

• At each decision point, the method appends an answer-elicitation prompt to the reasoning prefix and greedily generates a trial answer of at most 16 tokens.Its confidence score is the geometric mean of the trial answer's token probabilities, and constructing the target does not use the gold answer.Source: §A.3 · pp. 13–14

A.4 PRIMING PREFIX AND LABEL

• Confidence scores are rounded up to one of 50 percentage labels in 2% increments, following the paper's priming prefix.The prefix asks for confidence in the answer so far.Source: §A.4 · p. 14

A.5 TRAINING SEQUENCE AND LOSS MASK

• Training examples concatenate the prompt, reasoning prefix, priming prefix, and confidence label, but only label tokens receive loss.Maximum training context is 16,384 tokens for Nemotron, Gemma, and Qwen, and 8,192 for GPT-OSS-20B.Source: §A.5 · p. 14

B.2 PER-MODEL SETTINGS / B.3 GPT-OSS-20B

• Nemotron, Gemma, and Qwen use full-parameter fine-tuning, whereas GPT-OSS-20B uses a LoRA adapter because a full update does not fit the training setup.The GPT-OSS adapter targets attention projections and leaves the base weights frozen.Source: §B.1 · p. 14, §B.2 · p. 15, §B.3 · p. 15

B.1 SHARED RECIPE

• The shared recipe divides 695 training problems into eight groups, generates eight rollouts per problem each round, and selects checkpoints using held-out AIME 2024 problems.The 30 AIME 2024 problems are reserved for validation and checkpoint selection.Source: §B.1 · p. 14

B.4 ABLATION AND DECISION-POINT VARIANTS

• Ablations keep training construction fixed and replace only the confidence target; the reported variants use distinct round and marker configurations.The Gemma ablation uses paragraph boundaries, four problems per round, a 1×10^-6 learning rate, and the round-five checkpoint.Source: §B.4 · p. 15

C EVALUATION PROTOCOL

• Evaluation samples 16 independent completions per problem, reports pass@1 and pass@8 accuracy, and measures mean generated tokens and token reduction relative to each base model.Pass@8 uses an unbiased estimator.Source: C EVALUATION PROTOCOL · pp. 15–16

C.1 MODELS AND DECODING

• Decoding follows each model family's recommended sampler, with token limits of 8,192 for GPT-OSS-20B and 16,384 for other models, raised to 32,768 for LiveCodeBench.The GPT-OSS limit matches the context used during confidence training.Source: §C.1 · p. 16

C.2 BENCHMARKS AND SAMPLE COUNTS

• The evaluation covers AIME 2025, GSM8K, GPQA-Diamond, LiveCodeBench, and HumanEval, with 182 LiveCodeBench problems from contests dated January–April 2025.The listed benchmark sample counts include 30 AIME 2025, 1,319 GSM8K, 198 GPQA-Diamond, and 164 HumanEval problems.Source: §C.2 · p. 16

C.3 EVALUATION PROMPTS

• Prompts vary by benchmark: GPQA-Diamond requests a boxed answer letter, while coding tasks request code in a Markdown fence and LiveCodeBench retains its official wording.AIME 2025 and GSM8K use the boxed-answer instruction.Source: §C.3 · pp. 16–17

C.4 ANSWER EXTRACTION AND GRADING

• Fixed extraction rules recover answers or code already present in completions, using benchmark-specific grading rather than marking every format deviation incorrect.For LiveCodeBench, extracted code is tested with the official tests and is not otherwise repaired.Source: §C.4 · pp. 17–18

C.5 MORE EVALUATION RESULTS

• Token reductions are consistent across model families, while accuracy remains close to the corresponding base models under paired question-level 95% confidence intervals.Each method is compared with its base model on the same questions.Source: §C.5 · p. 18
• The paired intervals express accuracy differences in percentage points and token differences in average generated tokens.The intervals are question-level and paired against the corresponding base model.Source: §C.5 · p. 18
• Across all four model families, ConfSFT substantially reduces generated tokens while keeping pass@8 close to the corresponding base model.Pass@8 is computed with the unbiased estimator described in the evaluation protocol.Source: §C.5 · p. 18

D ADDITIONAL ANALYSES ON GEMMA-4-E2B

• The Gemma-4-E2B analysis repeats the four-panel analysis of Figure 2 and confidence-learning analyses from Section 5.2.Figure 7 is described as the counterpart of Figure 4.Source: D ADDITIONAL ANALYSES ON GEMMA-4-E2B · p. 18, Figure 6 · p. 19, Figure 7 · p. 19
• C-MAE and generated tokens fall over training, while accuracy remains within a few points of Gemma-4-E2B’s base model.The analysis tracks these measures across training rounds.Source: D ADDITIONAL ANALYSES ON GEMMA-4-E2B · p. 19

E SUPERVISION-SIGNAL ABLATIONS OVER TRAINING ROUNDS

• ConfSFT is the only supervision variant whose token reduction grows steadily while accuracy remains near the base model on both models.Figure 8 shows accuracy in the left panels and token reduction relative to the pre-SFT baseline in the right panels.Source: E SUPERVISION-SIGNAL ABLATIONS OVER… · p. 19, Figure 8 · p. 20

F.1 DEER

• DEER exits at the first decision point with confidence ≥0.95, returning that point’s trial answer; coding confidence uses at most 50 trial-answer tokens.If no decision point crosses the threshold, DEER returns the original completion.Source: §F.1 · p. 19
• DEER’s accuracy-efficiency behavior is highly threshold-sensitive and varies across tasks, whereas ConfSFT achieves more consistent reductions without inference-time stopping.Raising DEER’s threshold to 0.98 reduces premature exits and improves accuracy, but yields smaller token savings.Source: §F.1 · p. 19

F.2 A&Z

• Despite using roughly 15× more training problems and additional checkpoint selection, A&Z yields smaller token reductions than ConfSFT on both Nemotron and Gemma.The reported comparison selects the checkpoint with greatest token reduction and accuracy close to the base model.Source: §F.2 · p. 20
• Nemotron’s response length and reward collapse early and take several hundred steps to recover, while Gemma shortens only modestly and performance is sensitive to the length-penalty coefficient.Small coefficients yield little efficiency gain; larger ones substantially degrade performance.Source: §F.2 · pp. 20–21

F.3 ON-POLICY SFT

• On-Policy-SFT uses AIME 2000–2023 for training and selects checkpoints by validation accuracy-efficiency tradeoff.The learning-rate search spans 1 × 10^-6 to 5 × 10^-6, with checkpoints saved every 10 batches up to 2 epochs.Source: §F.3 · p. 22

G SCHOENFELD’S ANATOMY

• The episode analysis uses an eight-category sentence taxonomy and a classifier conditioned on the previous and current sentences, reaching κ = 0.754 on held-out sentences.The categories characterize reasoning structures and dynamics.Source: G SCHOENFELD’S ANATOMY · p. 22
• ConfSFT changes episode shares only slightly, while L1-Max, ThinkPrune, and On-Policy SFT substantially redistribute computation across episodes.The preserved shares include Implement, Explore, and Verify, so shorter ConfSFT traces allocate computation across episodes in nearly the same way as the base policy.Source: G SCHOENFELD’S ANATOMY · pp. 22–23