Source-linked AI summary
Decomposing LLM-Judge Uncertainty to Target Expert Labels
Ryan Lail
TL;DR
Uncertainty in an LLM judge’s predictions mixes genuine disagreement in the expert pool with the judge’s ignorance about that pool. The paper uses a Bayesian model and existing labels to separate the components, then finds on ChaosNLI that epistemic ranking removes more error than total uncertainty at the same label budget, while fewest-labels-first ties it.
Problem
An LLM judge’s uncertainty mixes irreducible expert-pool disagreement with uncertainty about the pool distribution, raising the question of whether these components can be disentangled from judge outputs and collected labels.
Source: §1 · p. 1Method
Results
On ChaosNLI, epistemic ranking removed 83% more error than entropy at the same label budget, but did not outperform fewest-labels-first.
Source: §5.3 · p. 4, §6 · p. 4Takeaways & Limitations
The results support estimating judge ignorance separately from genuine annotator disagreement when directing expert labels, within the tested settings.
Source: §6 · p. 4The full method is demonstrated only on synthetic dice; on ChaosNLI, nearly flat fitted trust means the real-data test mainly exercises the label-count side.
Source: §6 · p. 4
Abstract
from arXiv · showhide
An LLM judge evaluates outputs at scale. Experts should label only where it is least sure. Its natural escalation signal conflates two uncertainties: aleatoric, real disagreement in the expert pool, which labels cannot reduce, and epistemic, the judge's ignorance, which labels do reduce. A small Bayesian model separates them: a regression on labels already collected learns how far to trust a black-box judge's prediction. Both components follow as simple formulas, with no sampling or further judge calls. The components isolate on a real LLM judge against exactly known truth, and stated confidence is no guide to its actual error. On real human disagreement (ChaosNLI) the epistemic ranking removes 83% more error than total uncertainty for the same expert labels, though simply escalating the least-labelled items does as well there. We demonstrate we can estimate where a judge is ignorant rather than where experts genuinely disagree, and propose using this to direct expert labelling. Code and data are available at https://github.com/composo-ai/judge-uncertainty-decomposition.
1 Introduction
Because LLM judges are stochastic and expert pools can genuinely disagree, uncertainty in a judge’s prediction mixes irreducible disagreement with the judge’s ignorance. The paper asks whether these components can be separated using the judge’s outputs and labels already collected.
1 Introduction
- The paper separates aleatoric uncertainty, which further labels cannot reduce, from epistemic uncertainty about the expert pool’s label distribution.The pool may read an item differently, so the judge should predict a distribution rather than one verdict.Source: §1 · p. 1
- A regression on collected labels learns how much to trust a black-box judge’s prediction, yielding both components without further judge calls.The natural escalation score, predictive entropy, is the sum of the components and therefore conflates ignorance with disagreement.Source: §1 · p. 1
- On ChaosNLI, epistemic ranking recovers more label-budget value than total uncertainty, but a count-based rule does as well.A controlled dice test against known truth finds that the components isolate, while stated confidence does not guide actual error.Source: §1 · p. 1
2 Related work
Related work addresses distributional labels, judge-side uncertainty, and uncertainty decomposition, but the paper distinguishes its target from prior grader calibration and escalation approaches.
2 Related work
- Distributional judging supplies a label-distribution prediction, while the paper models uncertainty about where that distribution lies.Calibration to majority labels is ill-posed under genuine disagreement [Baan et al., 2022; Plank, 2022], and LeWiDi-2025 standardises evaluation against the pool distribution [Leonardelli et al., 2025].Source: §2 · p. 2
- Prior escalation systems use undivided judge-side uncertainty to target a single verdict [Jung et al., 2025; Sheng et al., 2025; Badshah et al., 2026].Bachar et al. [2026] carry the distinction as self-reported flags; the paper instead estimates the components from labels already collected.Source: §2 · p. 2
- The decomposition matches the classical one [Kotelevskii et al., 2025], and its epistemic score is BALD’s [Houlsby et al., 2011; Bickford Smith et al., 2023; Nguyen et al., 2019].Unlike Dubois et al. [2026], who separate grader severity offsets from residual disagreement, this work separates pool disagreement from ignorance of the pool distribution.Source: §2 · p. 2
3 Model
The model distinguishes variation within the expert pool from uncertainty about the pool’s label distribution. A Dirichlet belief centered on the judge’s prediction is updated with collected labels, and its uncertainty decomposition supplies decision scores.
3 Model
- The pool’s label-distribution entropy is aleatoric uncertainty, an item-level property unreduced by labels.The model represents the pool at the marginal label-distribution level.Source: §3 · p. 2
- Epistemic uncertainty predicts the divergence between the estimated and actual pool distributions, rather than treating genuine pool disagreement as error.The paper uses Manhattan distance for categorical soft-label divergence, ranging from 0 for an exact match to 2 for probability on disjoint classes.Source: §3 · p. 2
- A Dirichlet belief centered on the judge’s predicted distribution uses trust as pseudo-label weight, then updates in closed form with collected labels.Trust is fitted log-linearly on item features; the judge prediction itself is not fitted.Source: §3 · p. 2
- Total predictive entropy is the sum of expected pool disagreement and epistemic information gain, with the latter corresponding to BALD’s acquisition score.Entropy ignores the evidence behind the predictive distribution, so its ranking can diverge from epistemic ranking when total evidence varies.Source: §3 · p. 2
4 Experimental setup
Two experiments use one real LLM judge: a dice setting with exactly known truth and a ChaosNLI setting with human disagreement. They evaluate component tracking and compare escalation rules at fixed label budgets.
4 Experimental setup
- The dice experiment tests whether estimated components track known disagreement and error across 500 three-sided dice with four description tiers.The judge predicts each die’s 100-roll distribution from its description, with 0–10 revealed rolls as labels and tier as its familiarity feature.Source: §4 · pp. 2–3
- The setup includes Table 2’s comparison of seven rules at B = 10% and Table 1’s component-isolation correlations on 500 dice.Table 2 reports Manhattan units of error removed; Table 1 reports Spearman rank correlation with true quantities.Source: Table 2 · p. 3, Table 1 · p. 3
- The ChaosNLI experiment uses 3,113 premise–hypothesis pairs, each labelled by 100 annotators, with disjoint halves for fitting and evaluation.The remaining half is not touched during training; the disjoint 50-label halves set a 0.158 Manhattan noise ceiling.Source: §4 · p. 3
- At a fixed budget, each escalation rule selects its top-ranked B% of items for expert labels, allowing comparison by error removed.Rules include epistemic scores, entropy, verbalised confidence, fewest-labels-first, random, and oracle.Source: §4 · p. 3
5 Results
Across simulations and experiments, the decomposition distinguishes judge ignorance from genuine annotator disagreement, while the value of epistemic escalation depends on the setting. On ChaosNLI it outperforms entropy ranking, but not a fewest-labels-first rule.
5.1 Simulations: the estimator on synthetic pools with known truth
- In calibrated simulations with varying label counts, epistemic ranking stayed at oracle level while total uncertainty forfeited a third of achievable error reduction.When evidence was equal, the gap was zero; posterior spread captured nearly all the gain.Source: §5.1 · p. 3
5.2 Experiment 1: a real LLM judge on dice with exactly known truth
- Across 500 dice and 200 draws (2.5–97.5 bands), aleatoric estimate held at 0.85–0.87 versus true 0.892, while epistemic uncertainty fell from 0.190 to 0.052 and total uncertainty reached the aleatoric floor.Source: Figure 1 · p. 4
- On dice, stated confidence was non-monotone in judge error, correlated positively with error (+0.22), and low-confidence escalation did no better than random.Table 3 presents the confidence pathology by tier.Source: §5.2 · p. 3, Table 3 · p. 4
- Dice were unsuitable for measuring escalation gains: the judge predicted uniform distributions when ignorant, making total uncertainty resemble ignorance rankings amid single-draw noise.The decision-level comparison was therefore deferred to ChaosNLI, where the two rankings come apart.Source: §5.2 · p. 3
5.3 Experiment 2: escalation on real human disagreement (ChaosNLI)
- On ChaosNLI, predicted annotator distributions missed at triple the noise ceiling despite stated confidence averaging above 90; neither confidence nor prediction entropy tracked error.Confidence was measured on a 0–100 scale.Source: §5.3 · p. 4
- ChaosNLI trust was near-uniform at about five labels per prediction, while label counts ranged from 0–10, leaving total evidence heterogeneous.The short crowdsourced NLI pairs gave familiarity features little to exploit.Source: §5.3 · p. 4
- On ChaosNLI, epistemic ranking removed 15.0 error units versus entropy’s 8.2—83% more per expert label—but fewest-labels-first did as well.The gap widened at larger budgets and was absent at smaller ones.Source: §5.3 · p. 4
- Separating the estimator’s value from the count heuristic requires real items with varied judge familiarity and dense labels, which no public dataset offers.The authors characterize this as a dataset gap rather than a missing experiment.Source: §5.3 · p. 4
6 Limitations and conclusion
The authors recommend escalating items where the judge is ignorant rather than where annotators disagree, using labels already collected. The full method is demonstrated on synthetic dice; on ChaosNLI, near-flat trust leaves the count-based part tested on real data.
6 Limitations and conclusion
- The full method is demonstrated only on synthetic dice with exactly known truth; on ChaosNLI, near-flat fitted trust means only the label-count side was really tested on real data.There the method removed nearly twice as much error as entropy ranking at the same budget, but tied the fewest-labels-first baseline.Source: §6 · p. 4
- Each experiment used one judge model and prompt design; simulated effect sizes depend on the generator, intervals resample a fixed item set, and the method lacks defense against deceptive familiarity signals.A worthless signal reduces the method to label-count ranking, which remained about 32 percentage points better than entropy ranking in simulations.Source: §6 · p. 4
- The authors propose escalating on judge ignorance rather than annotator disagreement, and identify data with genuinely varying judge familiarity as the decisive next test.They describe the work as a first step toward decomposing epistemic and aleatoric uncertainty in LLM judges.Source: §6 · p. 4
A Supporting simulations in full
The simulations test label-value ranking in calibrated and misspecified settings, including varying item trust and corrupted judge confidence. They examine when entropy-ranking fails and how well estimated label value tracks the oracle.
Calibrated simulation
- At B = 10%, estimated label value stayed at oracle level as trust spread grew; its advantage over entropy-ranking declined from 81pp to 60pp as the range narrowed from ×256 to ×4.Estimated-trust ranking matched the oracle condition at every tested trust width.Source: Figure 2 · p. 6
Simulation setup
- The calibrated simulation used 2,000 model-generated items; in the misspecified simulation, regret comparisons were meaningful only between rules because the estimator’s model was deliberately wrong.Regret is a rule’s top-B% selection shortfall below the best possible, measured in percentage points; advantage is the difference between two rules’ regrets.Source: A Supporting simulations in full · pp. 5–6
Calibrated simulation
- 80.9pp advantage and 1.0 oracle-tracking Spearman met the calibrated simulation’s pass criteria.The criteria were an advantage of at least 15pp at the criterion cell and oracle-tracking Spearman of at least 0.9.Source: A Supporting simulations in full · p. 6
- 34.8pp was lost to entropy-ranking when items had identical prior trust but different revealed-label counts; the true null advantage was 0.0pp when both were identical.The null result is the condition predicted by the model.Source: A Supporting simulations in full · p. 6
- With a worthless familiarity signal, regression beat entropy-ranking by 32pp; the advantage rose to 81pp with better signals and remained 50pp at trust–disagreement correlation 0.8.Posterior-spread ranking captured approximately 98% of the gain, staying within 1.2–1.8pp of label-value ranking.Source: A Supporting simulations in full · p. 6
Misspecified simulation
- At γ = 1, judge-only features yielded 3.6pp advantage; at γ = 1.5, regression exploited inverted confidence, flipping its fitted coefficient from −0.63 to +1.09.A middling judge-independent familiarity signal recovered 23.5pp at γ = 1.Source: A Supporting simulations in full · p. 6, Figure 3 · p. 7
B Model details: the estimator, fitting likelihood, closed forms, and the real-valued formulation
The model combines an LLM judge’s predicted distribution with collected labels, using fitted trust to weight the judge’s contribution. Its uncertainty decomposition and resulting scores distinguish disagreement in the annotator pool from uncertainty that labels can reduce.
Estimator
- The judge prediction forms a Dirichlet prior weighted by fitted trust λx; observed class counts update it to Dir(αx), so trust and labels add as evidence.The prior is centered on the judge’s predicted distribution, and trust is expressed in pseudo-label units.Source: B Model details: the estimator,… · p. 7
Real-valued formulation
- For real-valued scores, each rating adds Gaussian within-annotator noise to a latent annotator score; pool spread includes genuine disagreement that further labels do not reduce.The model uses the sum of within-annotator noise and pool disagreement because individual labels carry both.Source: B Model details: the estimator,… · pp. 7–8
Closed forms and variance decomposition
- The real-valued variance split treats within-annotator noise and pool disagreement as aleatoric, while uncertainty about the pool mean is epistemic and shrinks with labels.With one label per item–annotator pair, the two aleatoric sources are not separately identifiable; the model uses their sum.Source: B Model details: the estimator,… · p. 8
Fitting likelihood
- Maximum likelihood fits trust coefficients with a log-link and Dirichlet–multinomial response; only the trust scale is learned.Features include stated confidence, label count, and embedding distance or density to labelled items.Source: B Model details: the estimator,… · pp. 7–8
Derived scores
- Under squared error, posterior spread is the expected error from withholding a label, while one-label value is the expected error reduction from purchasing one.Withholding and purchasing optimize different scores, though they rank items nearly identically in this model class; both differ from entropy.Source: B Model details: the estimator,… · p. 8
When entropy-ranking is safe
- Entropy-ranking and label-value ranking nearly agree when total evidence is equal; varying evidence can make entropy anti-correlated with label value, even to full inversion.In partially labelled deployments, label counts make total evidence heterogeneous.Source: B Model details: the estimator,… · p. 8
C Experiment 1 auxiliary results
Dice results show that low-confidence escalation is indistinguishable from random, and single-draw noise dominates the reported regret difference. The auxiliary results also compare self-updating with conjugate updating and describe the component-isolation and confidence tables.
Escalation numbers on dice
- Entropy tracked judge error at 0.57 versus 0.66 for the epistemic estimate; low-confidence escalation was indistinguishable from random, and regret differences were dominated by single-draw noise (+6pp expected, per-draw −25 to +28).Low stated confidence yielded +1.8pp at B = 10%, with a per-draw 95% interval of −14 to +18.Source: C Experiment 1 auxiliary results · p. 8
Self-updating versus conjugate updating
- Conjugate updating of the judge’s prior had error 0.262 with the familiarity channel and 0.307 without, compared with 0.258 for the judge’s self-updated prediction (n = 380).The passage attributes the trust level to the familiarity channel and says the Bayesian machinery, rather than overhead, is where the correction lives.Source: C Experiment 1 auxiliary results · p. 8
Isolation of the estimated components
- Table 4 assesses component isolation through correlations between estimated components and their ground truths; isolation calls for strong diagonal and near-zero off-diagonal correlations.Its truth–truth column indicates that the two ground truths are uncorrelated or anti-correlated.Source: Table 4 · p. 9
Stated confidence and error
- Table 5 reports that stated confidence is non-monotone in judge error, with results grouped by description tier and also showing prediction spread across three prompt paraphrases.Each tier contains 125 dice; variant spread is the standard deviation of predicted face probabilities, and 0.0002 corresponds to paraphrases agreeing to the fourth decimal place.Source: Table 5 · p. 9
D Experiment 2 auxiliary results
In the auxiliary comparisons, I-ranking beats entropy at B = 20%, while smaller budgets are inconclusive and it does not beat the fewest-labels-first rule. Descriptive results also show near-uniform fitted trust and weak item-level error associations for stated confidence and prediction entropy.
Descriptives
- Mean Manhattan error was 0.458 against the 0.158 ceiling; stated confidence averaged 90.3 and correlated −0.13 with error, versus 0.11 for prediction entropy.Fitted trust was near-uniform (sd(log ˆλ) = 0.22 versus the planned condition’s ≥0.5), with each prediction worth about five pseudo-labels.Source: D Experiment 2 auxiliary results · p. 9
- The planned condition tested only the trust side, so its prediction of no separation was a mis-specification: the characterisation concerns α0, not λ alone.The passage frames this as a distinction between total evidence and trust alone.Source: D Experiment 2 auxiliary results · p. 9
Comparisons beyond Table 2
- I-ranking beats entropy by +12.8 [+4.3, +20.4] at B = 20%, while smaller budgets are inconclusive.The reported value clusters are 12.1–15.0 for count-aware rules and 6.9–8.3 for count-blind rules; I minus fewest-labels-first is +0.7 [−6.3, +7.1].Source: D Experiment 2 auxiliary results · p. 9
- The planned ranking by ∆ was inconclusive against entropy (+4.5 [−2.2, +9.4]), while the counts-only claim was refuted (+6.9, grazing zero at −0.4).Seven rules are compared, so the passage cautions that no interval stands alone.Source: D Experiment 2 auxiliary results · p. 9
ChaosNLI escalation table
- At B = 10%, Table 6 covers 311 items and reports Manhattan error removed and advantage over entropy, with means over 200 label re-draws and per-draw 95% intervals.Only the ∆ row was the planned headline comparison; the remaining advantages are exploratory.Source: Table 6 · p. 9