Free Lunch for LLMs Confidence: The RL Critic Already Knows

1 Introduction

• Dedicated confidence training can improve confidence estimation but adds a training stage requiring additional data and compute.This motivates asking what prediction problem confidence training actually solves.Source: §1 · p. 1
• An external estimator trained on question–response–correctness traces estimates policy confidence as accurately as the policy without access to its internal representations.The self- and external estimators have comparable capacity and use the same behavioral traces.Source: §1 · p. 1
• Under correctness-based outcomes, confidence estimation and critic training predict the likelihood that a policy response will be correct from its trajectory.For continuous correctness or quality rewards, the connection holds when confidence is interpreted as expected outcome under the chosen reward.Source: §1 · p. 2
• For value-based RL, the existing critic can be reused without confidence-specific training or data, leaving the policy unchanged and providing estimates at intermediate reasoning steps.The critic learns from policy trajectories throughout RL, rather than only from final responses.Source: §1 · p. 2
• CaC is the best confidence discriminator on every in-distribution split and the best-calibrated on 14 of 16 across a dozen domains.The evaluation spans two LLMs and math, code, agentic tool calling, and general reasoning; fine-tuning the policy for confidence costs 3.9 accuracy points on average.Source: §1 · p. 2

Method

• External models with comparable capacity are trained on the target policy’s question–response–correctness traces, without access to its internal representations or token-level probabilities.The self-estimator and external estimators use the same behavioral dataset and LoRA-based supervised fine-tuning procedure.Source: §2 · pp. 2–3

Experimental setup

• The experiments use Llama-3-8B-Instruct responses, capped at 2,000 training examples per benchmark, across math, code generation, and science reasoning.The five benchmarks are GSM8K, NuminaMath, APPS-intro, TACO, and ARC-Challenge.Source: §2 · pp. 2–3

Main results

• Qwen2.5-7B-Instruct stays within 0.05 AUROC of the self-estimator on four of five benchmarks, and an external estimator ties or leads in AUROC on four.On every benchmark, at least one external estimator matches or improves on the self-estimator’s ECE; the same is true for Brier score on three of five benchmarks.Source: §2 · p. 3

Conclusion

• The results indicate that strong confidence signals can be learned from observable input–output behavior without privileged access to a policy’s internal representations.The next section connects this prediction problem to the value function learned under correctness-based reward.Source: §2 · p. 3

Equivalence

• With binary terminal correctness reward, the critic’s expected return is the probability of a correct final response, matching confidence estimation at every trajectory step.A critic trained on bootstrapped GAE targets has the same fixed point as the Monte Carlo value objective.Source: §3 · p. 4
• A calibrator and critic share the same optimum under squared loss for any model class, and under log loss when the model class can represent that optimum.The shared solution is perfectly calibrated and consistent along the trajectory.Source: §3 · p. 4

Practical advantages

• The critic can estimate confidence from the prompt alone through intermediate prefixes to the final response, while calibrators trained on response-level triples are supervised only at the final state.Critic supervision comes from the policy’s rollouts throughout RL, avoiding separate data collection and covering every training trajectory.Source: §3 · p. 4

Scope of the equivalence

• The equivalence applies to any fixed policy, permits stochastic graders, and is independent of estimator architecture; strictly proper losses recover the same correctness probability when representable.Bootstrapped GAE targets preserve the policy value as their unique population fixed point for every λ in [0, 1].Source: §3 · p. 4
• Applying the equivalence requires an undiscounted terminal correctness return, matching the policy and decoding settings, and recognizing that transfer beyond the RL prompt distribution is empirical.Adding KL penalties or other shaping terms to the reward changes the value target.Source: §3 · p. 4

Critic-as-Confidence (CaC)

• For CaC, the critic is used unchanged as the confidence estimator; clipping its output to [0, 1] guards against regression values outside that range.Reuse requires no confidence-specific data, training, or policy modification, and adds one critic forward pass at inference.Source: §3 · pp. 4–5

4.1 Setup

• Experiments use Qwen3-8B and Nemotron-Cascade-8B trained with PPO across math, code, tool calling, and general reasoning, evaluated on held-out in-distribution and external out-of-distribution benchmarks.Confidence is assessed with AUROC, 15-bin ECE, and Brier score; CaC reads the critic at the last response token and clips it to [0, 1].Source: §4.1 · p. 5

4.2 Results

• CaC has the highest AUROC on all eight in-distribution splits, the best or tied-best ECE on all 16 splits, and preserves task accuracy without updating the policy.R-Tuning lowers accuracy by 3.9 points on average; on out-of-distribution benchmarks CaC is best on seven of eight splits, while its ECE rises by 0.08 on average versus 0.12 for R-Tuning and Probe.Source: §4.2 · pp. 5–6, Table 3 · p. 6

4.3 Inference-Time Scaling with CaC

• CaC Selection improves single-sample accuracy by 4.1 points for Qwen3-8B and 5.6 for Nemotron-Cascade-8B on average, closing about half the oracle gap.Majority voting closes 26% and 35% of the gap, respectively; CaC scores samples independently and needs no task-specific rule for grouping equivalent outputs.Source: §4.3 · p. 6, Table 2 · p. 6

4.4 Extension to Critic-Free RL

• For a GRPO-trained Qwen3-8B policy, a post-hoc value model fit on existing rollouts is the best estimator on every metric and split.It reaches AUROC 0.90 ID and 0.81 OOD, versus 0.87 and 0.79 for the strongest baseline; this extension requires an auxiliary training stage.Source: §4.4 · p. 7, Table 5 · p. 7

4.5 CaC: Confidence along the Trajectory

• The critic estimates confidence at the prompt and intermediate response positions; most remaining discrimination gain arrives in the first quarter of generation.For the final general-reasoning checkpoint, prompt-only AUROC is 0.83 ID and 0.73 OOD, while ECE improves along the trajectory from 0.09 to 0.05 ID and 0.18 to 0.12 OOD.Source: §4.5 · pp. 7–8, Table 6 · p. 8

5 Related Work

• Training-free estimators use token probabilities or entropy, sampling-based consistency measures, or prompted confidence; sampling costs more inference and elicited confidence can be poorly calibrated across tasks.Training-based estimators instead learn correctness from graded policy responses and are described as the strongest family.Source: §5 · p. 8
• R-Tuning teaches “I am sure/unsure,” SaySelf produces fine-grained confidence with calibration-aware RL, APRICOT predicts confidence from text, and probes use hidden activations.The passage also notes that fine-tuning on as few as a thousand graded examples can outperform training-free baselines and generalize across domains (Kapoor et al., 2024).Source: §5 · p. 8
• The open question is whether confidence supervision needs internal representations: with the same behavioral data, self- and external estimators perform nearly identically, reducing prediction to f(x, y) → P(correct | x, y).The paper establishes a theoretical link to RL critics and evaluates it across a dozen domains.Source: §5 · p. 8

6 Conclusion

• Under correctness-based outcomes, confidence estimation and value-function learning are the same prediction problem, allowing the RL critic to be reused without extra training.The conclusion also says this approach extends to intermediate steps, test-time selection, and critic-free RL.Source: §6 · p. 8
• Across multiple domains, CaC is the best estimator in a dozen domains and extends to intermediate steps, test-time selection, and critic-free RL.The conclusion does not give specific metric values for these comparisons.Source: §6 · p. 8

Limitations

• CaC requires a critic, which GRPO and other critic-free RL methods do not train; fitting one on existing rollouts adds a training stage, so the exact free lunch applies only to value-based RL.Using the critic also adds one value-model forward pass per response.Source: Limitations · p. 9

A.1 Setting

• The formal setting uses terminal correctness c as reward with γ = 1; at each prefix, the value function is the probability of a correct final answer.The prompt, response prefix, and grader label define the prediction setting; a deterministic grader is a special case.Source: §A.1 · p. 12

A.3 Bootstrapped Critics

• For every λ ∈ [0, 1], bootstrapped updates have the unique fixed point Vπ = q and exact repeated updates from any bounded value function reach it.The proof’s backward argument establishes the correct values from terminal prefixes toward earlier prefixes.Source: §A.3 · p. 14

A.4 Properties of the Shared Target

• The shared target q(st) = Vπ(st) is calibrated and, when both labels occur, achieves the highest possible population AUROC among scores based on the prefix.Any strictly increasing transformation of q preserves its AUROC ranking.Source: §A.4 · p. 14

A.5 Assumptions and Extensions

• The identity concerns the final policy under its decoding settings, and the critic may lag its last updates; transfer beyond the RL prompt distribution is empirical.If reward includes shaping, the value includes expected shaping; with γ < 1 it discounts continuation length, so it is no longer correctness probability.Source: §A.5 · p. 14

A.2 Main Theorem

• At any prefix st, Vπ(st) equals q(st), the probability that the final response is correct, under the theorem’s correctness-reward setting.The equality holds almost surely, including for later prefixes along the response.Source: §A.2 · pp. 12–13
• With squared loss, the confidence and critic objectives have the same minimizers, including when evaluated on the same response sample.The theorem’s equality follows because the reward equals correctness for each response.Source: §A.2 · p. 13
• Any strictly proper confidence loss is uniquely minimized by predicting q(st) = Vπ(st); if q belongs to Gt, the calibrator and squared-loss critic can both attain that optimum.This is a target equivalence, not a claim that different estimator architectures are identical.Source: §A.2 · p. 13

• Value functions underpin methods such as GAE in PPO and AlphaGo’s value network, which predicts the probability of winning.Classification-trained value functions also connect value learning to the calibration objective (Farebrother et al., 2024).Source: B Related Work on Training Value… · p. 15
• Although critic-free methods reduce memory use and avoid difficult critic training, well-trained critics support finer credit assignment, lower variance, and greater sample efficiency in long-CoT reasoning and agentic settings.Value models are also reused for search and for obtaining correctness probabilities in process reward models.Source: B Related Work on Training Value… · p. 15
• This work presents the critic as a calibrated, general-purpose confidence estimator for LLMs.The PPO critic is standard in RLHF.Source: B Related Work on Training Value… · p. 15

• The actor and critic are initialized from Qwen3-8B or Nemotron-Cascade-8B, with a randomly initialized scalar value head for the critic.Thinking mode stays enabled, and prompts request a final answer in \boxed{}.Source: C Experimental Details · p. 15
• PPO with GAE uses a binary correctness reward at the final response token, and γ = λ = 1 makes the critic’s return target the correctness label at every response position.KL regularization affects only the actor loss; the critic uses unclipped MSE and trains for 300 steps without repeating prompts.Source: C Experimental Details · p. 15, Table 7 · p. 15
• Confidence is the critic’s scalar output at the last nonpadding token, evaluated using four sampled responses per prompt and rule-based grading.Truncated responses count as incorrect; AUROC uses raw outputs, while ECE uses 15 equal-width bins and ECE and Brier score use outputs clipped to [0, 1].Source: C Experimental Details · p. 15

• Tables 8–10 report AUROC, ECE, and Brier score for every math, code, and agentic tool-calling benchmark and both LLMs.Source: D Full Results · p. 15
• Math results cover six NuminaMath sources used for RL as in-distribution data and AIME 2024 and 2025 as out-of-distribution data.Table 8 marks the best result per column within each model and metric.Source: Table 8 · p. 16
• Code results cover three sources used for RL in-distribution and LiveCodeBench by difficulty out-of-distribution, while tool-calling results use held-out ToolRL and API-Bank levels 1 to 3.Tables 9 and 10 mark the best result per column within each model and metric.Source: Table 9 · p. 17, Table 10 · p. 18