PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models

Yu-Chung Hsiao

arXiv:2609.10996v1cs.CL

TL;DR

The paper asks whether logprobs remain the best soft signal for LLM-as-a-Judge as proprietary models and API capabilities change. It evaluates a verbalized-confidence recipe with an overconfidence advisory and self-debate across three benchmarks and up to 18 models, finding a compatibility shift toward verbalized confidence on post-2025 flagships. The authors consequently recommend reporting soft-score measures alongside hard predictions.

  • Problem

    The paper examines whether the longstanding preference for logprob-based soft scoring still holds on contemporary proprietary LLM judges.

  • Method

    The study evaluates a logprob-free verbalized-confidence protocol that adds an overconfidence advisory and self-debate to a rubric baseline across three benchmarks and multiple model generations.

  • Results

    Across the benchmarks, verbalized confidence improves calibration, score spread, and subjectivity robustness; post-2025 flagships retain prediction accuracy while pre-2025 models lose it.

  • Takeaways & Limitations

    Soft-score measures should be reported alongside hard predictions, with verbalized confidence treated as a portable interface for contemporary proprietary judges.

  • Takeaways & Limitations

    The study focuses on single-call, pointwise judging and excludes reference-based correctness tasks, pairwise or listwise judging, and multi-call or multi-agent debate.

Abstract

from arXiv · show

Verbalized confidence, long dismissed as overconfident, coarse, and prone to round-number clustering, is now the more robust soft-scoring mechanism for LLM-as-a-Judge on top-tier proprietary models. Across SummEval, AggreFact, and HelpSteer2, spanning up to 18 LLMs, we show that the standard advice to prefer log-probabilities no longer holds on post-2025 models, where verbalized confidence is the better signal. We call this a compatibility shift. On top of a standard verbalized-confidence baseline, we introduce two new ingredients: an overconfidence advisory and self-debate. Together they improve calibration, score-distribution spread, and robustness to task subjectivity. We further observe a generation effect: post-2025 models accommodate these two additions with little balanced-accuracy cost, whereas pre-2025 models pay a measurable penalty. Compared with logprob-based G-Eval, verbalized confidence is the more subjectivity-robust soft signal on GPT-family top-tier releases. The shift is invisible under accuracy-only reporting. Rather than defaulting to hard predictions, we recommend broader use of soft scoring in LLM-as-a-Judge. More broadly, verbalized confidence has moved from a weaker substitute for logprobs to a practical soft-scoring mechanism for contemporary LLM judges.

Introduction

This paper revisits whether logprobs or verbalized confidence provide the better soft signal for LLM-as-a-Judge, motivated by changing model capabilities and logprob availability. Across model generations and benchmarks, it identifies a compatibility shift favoring verbalized confidence on contemporary proprietary models.

  • Introduction: Across an 18-model sweep, verbalized confidence overtakes logprob-based G-Eval on GPT-family flagships and reverses the pre-2025 preference.The comparison is restricted to GPT because it is the family with usable logprob access across releases.
  • Introduction: Post-2025 flagships preserve balanced accuracy under the richer prompts, whereas pre-2025 models incur a significant balanced-accuracy loss.This generation effect explains why the same prompt recipe benefits newer models more reliably.
  • Introduction: The shift is visible in calibration and score-distribution spread but can be missed by accuracy-only reporting.The paper therefore treats soft-score quality as necessary for comparing judging protocols.
  • Introduction: An overconfidence advisory and self-debate improve calibration, score spread, and subjectivity robustness over the rubric-anchored verbalized baseline.The recipe argues both True and False before concluding.
  • Introduction: The study focuses on single-call, pointwise binary judging for factual faithfulness and subjective quality evaluation.It excludes reference-based correctness tasks, pairwise or listwise judging, and multi-call or multi-agent debate.

Method

The method compares logprob-based and verbalized-confidence protocols under a single-call binary-judging setup. Its full verbalized recipe adds an overconfidence advisory and self-debate to a rubric baseline, then evaluates both prediction and soft-score quality.

  • Method: The study restricts judging to one LLM call per example and pointwise binary classification.This excludes multi-agent setups while keeping evaluation costs low at larger scale.
  • Method: The full verbalized protocol augments Rubric with an overconfidence advisory and self-debate, while G-Eval supplies the logprob comparison anchor.The self-debate asks the judge to argue True, argue False, and then conclude.
  • Method: Verbalized protocols output a binary prediction with a 0–100 confidence score, whereas G-Eval requests a Likert value and derives its soft score from token logprobs.Rubric is the baseline; Ours-FreeReason replaces self-debate with unconstrained reasoning.
  • Method: The evaluation reports Prediction BA, Oracle BA, OP Gap, Debate Stress, calibration error, and Spread together because no single metric reveals the compatibility shift.Spread measures closeness of the confidence histogram to a uniform reference rather than generic dispersion.

Experimental Setup

The experiments cover factual faithfulness and subjective quality across SummEval, AggreFact, and HelpSteer2, using cohorts that span model families and training generations. The setup combines benchmark-specific subjectivity measures with prediction, calibration, spread, and stress metrics.

  • Experimental Setup: The setup reports balanced accuracy, oracle accuracy, the Oracle–Prediction Gap, Debate Stress, calibration error, and Spread together.These metrics jointly assess hard predictions, latent confidence quality, calibration, distributional breadth, and prompt stress.
  • Experimental Setup: SummEval evaluates consistency, coherence, and relevance on 100 documents paired with 17 generated summaries and expert 5-point ratings.Its inter-annotator agreement values provide a task-subjectivity covariate.
  • Experimental Setup: HelpSteer2 independently replicates the subjectivity analysis using roughly 21k prompt-response pairs rated by about 1,000 crowd annotators, with 800 sampled items.Its crowd annotation and task diversity differ from SummEval’s expert panel.
  • Experimental Setup: AggreFact probes binary factual faithfulness across nine tasks, using 300 examples per task for 2,700 total examples.The suite spans hallucination detection, claim verification, grounding, reasoning verification, and factchecking regimes.
  • Experimental Setup: The sweep spans up to 18 models across GPT, Claude, Gemini, and open-weight checkpoints, with pre-2025 and post-2025 cohorts.The GPT comparison uses gpt-4o, gpt-4.1, gpt-5.2, and gpt-5.4 because they return logprobs.

Soft-Scoring Quality

Across AggreFact, SummEval, and HelpSteer2, the proposed verbalized-confidence recipe improves calibration, score spread, and robustness to task subjectivity. Its subjectivity advantage is strongest for probability scores rather than binary predictions.

  • Overall result: Across all three datasets, the recipe improves soft-scoring quality in calibration, score-distribution spread, and robustness to task subjectivity.AggreFact supplies the calibration and spread result, while SummEval and HelpSteer2 replicate the subjectivity-robustness result.
  • Calibration Error and Score Spread Improve: AECE falls from 8.9% to 3.8% and Spread rises from 75.8% to 89.6% for Sonnet 4.5 on AggreFact.The prompts reduce overconfidence in low-probability regions and bring calibration closer to the diagonal while using more of the probability scale.
  • Ours: Soft Signal More Robust to Subjectivity: ∆slope = −0.143 for τ(prob), with one-sided p < 10−4, showing lower coupling between probability scores and task subjectivity than the Rubric baseline.The corresponding τ(pred) slope difference is not significant, with two-sided p = 0.46.
  • Ours: Soft Signal More Robust to Subjectivity: The subjectivity-robustness conclusion replicates on HelpSteer2 using per-example rater disagreement rather than pooled cross-dataset agreement.The authors treat the datasets as independent replications because Krippendorff’s α is interpretable only within a single study design.

Compatibility Shift and Generation Effect

On GPT-family flagships, verbalized confidence becomes more subjectivity-robust than logprob-based G-Eval in newer generations. The generation effect extends to the recipe’s accuracy cost: post-2025 models largely retain or improve prediction accuracy, whereas older models lose it.

  • Compatibility Shift: Verbalized confidence gains a subjectivity-robustness advantage over G-Eval in the GPT-5 era, while its judging capacity improves across GPT generations and G-Eval’s does not.The comparison uses GPT flagships that return logprobs: gpt-4o, gpt-4.1, gpt-5.2, and gpt-5.4.
  • Generation Effect: Post-2025 flagships avoid the significant Prediction BA drops seen in older models, with newer models shrinking the Oracle-Prediction Gap and Debate Stress toward zero.Later Claude models even gain prediction BA from debate, whereas older GPT and Gemini models show substantial positive stress.
  • Generation Effect: The recipe favors Ours in 67 of 90 model-task comparisons, with none favoring Rubric at 95% confidence, while significant prediction-accuracy drops remain concentrated in pre-2025 models.The calibration advantage appears across all 9 tasks; the generation-dependent accuracy pattern has Fisher one-sided p = 0.013.
  • Which Component Drives Which Effect: The full Adv+SD recipe changes Prediction BA by −1.8 percentage points for pre-2025 models and +0.7 percentage points for post-2025 models.Table 3 isolates the overconfidence advisory and self-debate across pre-2025 and post-2025 cohorts.

Discussion

The prompt and model effects are complementary: verbalized confidence improves soft-signal quality, while post-2025 flagships preserve prediction quality. This compatibility shift is missed by accuracy-only reporting and motivates reporting soft scores alongside accuracy.

  • Discussion: Verbalized confidence improves subjectivity robustness on SummEval and HelpSteer2, while reducing calibration error and broadening score distributions on AggreFact.The recipe improves τ(prob) robustness to task subjectivity and produces broader, better-calibrated scores across the evaluated benchmarks.
  • Discussion: Post-2025 flagships preserve prediction accuracy while older models often trade prediction accuracy for the same calibration gains.The generation effect separates prompt-level soft-signal improvements from model-level ability to convert them into stable predictions.
  • Discussion: Verbalized confidence overtakes logprob-based G-Eval for subjectivity robustness on GPT-5-era models and improves more steadily across GPT generations.The advantage changes sign across generations, with verbalized protocols improving steadily while G-Eval fluctuates.
  • Discussion: Benchmark practice should report soft-score measures alongside prediction accuracy and adopt logprob-free verbalized confidence as a portable interface.This recommendation follows both the generation-dependent reporting gap and declining logprob availability across production APIs.

Related Work

Prior work developed logprob-based soft scoring because verbalized confidence was viewed as coarse and overconfident. This paper instead connects calibration and self-debate to a generation-dependent compatibility shift in contemporary proprietary models.

  • Related Work: G-Eval uses token logprobs to recover finer-grained soft scores because verbalized Likert judgments can cluster on a few integer values.Prior evidence on verbalized confidence was mixed, with usefulness established in some QA-correctness settings but not uniformly across judging tasks.
  • Related Work: Figure 5 tracks Oracle-Prediction Gap and Debate Stress across model generations, with both measures shrinking toward zero on post-2025 flagships.Lower Oracle-Prediction Gap indicates better faithfulness capture by binary predictions; Debate Stress uses zero for full robustness and negative values for gains.
  • Related Work: On contemporary closed-source flagships, verbalized confidence is presented as practically useful rather than merely a weaker substitute for logprobs.The paper positions this usefulness within its broader compatibility-shift result.
  • Related Work: The paper’s self-debate is a single-call prompt constraint, unlike multi-call debate or aggregation systems.Calibration is assessed with adaptive ECE together with a Bhattacharyya spread measure because ECE alone can miss distributional collapse.

Conclusion

The paper argues that logprob-based soft scoring is no longer uniformly preferable: verbalized confidence improves soft-signal quality, while post-2025 flagships retain prediction accuracy. It therefore treats prompt-model compatibility as a moving target.

  • Conclusion: The pre-2025 preference for logprob-based soft scoring no longer holds uniformly across LLM-as-a-Judge.The conclusion attributes the shift to the combination of improved verbalized-confidence soft scores and generation-dependent prediction costs.
  • Conclusion: Across three benchmarks, the logprob-free recipe improves calibration, score spread, and robustness to task subjectivity.Post-2025 flagships keep prediction accuracy, whereas pre-2025 models lose it under the same recipe.
  • Conclusion: The recommended practice is to report hard predictions and soft-score measures together while treating prompt-model compatibility as a moving target.This preserves information about both prediction performance and the quality of the underlying confidence signal.

Supplementary Material

The supplementary material documents logprob-access constraints, subjectivity-robustness inference, benchmark protocols, component comparisons, and replication analyses. It also reports that the advisory drives calibration and spread gains, while self-debate adds spread on many models.

  • A. Logprob Access Is No Longer a Stable Primitive: Recent APIs increasingly restrict logprob access, making logprob-free confidence extraction more portable for cross-model evaluation.OpenAI narrowed top-token availability, Google withdrew support from Gemini 3.0 Pro onward, and Anthropic has never returned logprobs.
  • B. Bootstrap Inference for Subjectivity-Sensitivity Slopes: Subjectivity-sensitivity slopes are estimated by linear regression with a paired model-cluster bootstrap using 100,000 resamples.The bootstrap resamples models while preserving each model’s task or bin points, testing whether Ours has a flatter slope than Rubric.
  • C–E. Replication and Evaluation Protocols: The supplementary analyses cover SummEval, HelpSteer2, and AggreFact using dimension-level and per-example disagreement measures.HelpSteer2 uses 800 sampled items judged by nine flagship models across four vendors and two eras.
  • Supplementary Tables: The supplementary material includes vendor-level logprob availability and SummEval dimension-level subjectivity-robustness tables.Table 4 records API availability by vendor, while Table 5 reports Kendall τb, regression slopes, R2, and bootstrap slope-difference tests.
  • D. Component Ablation: The overconfidence advisory reduces AECE and broadens Spread on all 10 models, while self-debate adds Spread on 6/10 models.Removing the advisory reverses both calibration and spread changes on all 10 models; free-form reasoning is roughly neutral or slightly negative on Spread.

E. Full Per-Task Breakdown (10 Models × 9 Tasks)

Across 90 model-and-task comparisons, the compatibility shift remains visible at the individual-cell level. Calibration improves in both eras, while prediction-accuracy costs are concentrated among pre-2025 models.

  • The analysis uses ∆Brier and ∆BA for each of 90 combinations, with separate bootstrapping and Brier chosen because per-task AECE binning is noise-dominated.Each task contains n = 300 examples, making fine-grained AECE estimates unstable.
  • The compatibility shift holds across 90 model-and-task cells, with significant era differences for both ∆Brier and ∆BA.Fisher one-sided tests give p = 0.027 for ∆Brier and p = 0.013 for ∆BA.
  • All 9 tasks include at least one post-2025 flagship with a 95%-CI-below-zero ∆Brier, compared with 6 tasks for pre-2025 models.The benefit therefore spans tasks rather than concentrating on a single benchmark.

F. Data Processing

The evaluation processes fixed, source-grounded benchmark samples across SummEval, AggreFact, and HelpSteer2, using single-call judgments and multiple verbalized-confidence protocols. Stability checks and full-generation analyses show improved calibration and spread, with accuracy costs concentrated in older models.

  • Sampling and task selection: The study uses deterministic, uniformly applied samples: 2,700 examples across 9 AggreFact tasks and 800 HelpSteer2 items.Examples are selected by md5-hash ordering, so the exact samples can be regenerated without a random seed.
  • Dataset processing: SummEval uses consistency, coherence, and relevance, excluding fluency because the dataset preprocessing conflicts with its rubric.The authors avoid detokenization or truecasing to prevent a judge-by-restorer confound.
  • Sampling stability: Every significant Ours−Rubric result replicates on the disjoint tail-300 draw, preserving the pre-versus-post ∆BA pattern.The second draw has zero overlap on 8 of 9 tasks and 42/300 overlap on AggreFact-CNN.
  • Protocol components: The advisory reduces AECE on all 10 models and broadens Spread on all 10, while self-debate further broadens Spread on 6 of 10 models.Spread gains from the advisory range from +6.1 to +30.2 percentage points.
  • Generation trends: Across GPT, Claude, and Gemini generations, calibration improves under Ours and Ours-FreeReason, while balanced-accuracy costs concentrate on pre-2025 models.Oracle-Prediction Gap and Debate Stress approach zero for post-2025 models.
Loading 2609.10996v1…