When Accuracy Gaps Fail to Certify: Auditing Cross-Domain Recalibration of LLM Judges

1 Introduction

• Calibration is distribution-local: a judge may preserve answer ordering while its reported probabilities become unusable after a task change.This matters when probabilities control acceptance, review, or deferral rather than serving only as ranking statistics.Source: §1 · p. 1
• A source–target accuracy gap is marginal, can conceal different conditional reliability, and consumes target labels to estimate.The paper says any gap-based rule should be compared with fitting a target-domain calibrator using the same label budget.Source: §1 · p. 1
• The paper separates the gap from recalibrated mean-score shift, shows the gap is insufficient, and derives an importance-weighting identity and a simultaneous rejection certificate.Non-rejection by the certificate is not approval.Source: §1 · p. 1
• The paper finds that the leak-free gap association does not survive the cross-generator standard, while probe-estimated raw target error does.Importance-weighting controls locate natural-domain failure in the observed text representation; the rejection certificate is valid but too weak at available audit sizes, while 16 target labels are more useful for direct temperature scaling.Source: §1 · pp. 1–2

2.1 LLM judges and their uncertainty

• Confidence quality is distinct from ranking and agreement, and prior confidence studies do not test transfer of a one-dimensional map across tasks under disjoint target evaluation.This paper holds the judge and response set fixed and reports ranking, proper scores, calibration metrics, and downstream risk separately.Source: §2.1 · p. 2

2.2 Post-hoc calibration and distribution shift

• Importance weighting can identify target risk under covariate shift when conditional labels are invariant and target support is covered; estimated pipeline failure alone does not establish conditional shift.The paper evaluates shift assumptions as distinct hypotheses.Source: §2.2 · p. 2

2.3 Measuring calibration

• The study uses population L1 calibration error for theory, Brier score and log loss as primary empirical outcomes, and equal-mass ECE as a secondary diagnostic.Debiased and smoothed ECE variants were not computed.Source: §2.3 · p. 2

2.4 Boundary with concurrent work

• Unlike concurrent work on conformal prediction-set coverage under traffic-mixture shift, this paper studies scalar post-hoc maps, proper-loss transfer, and a lower certificate for calibration error.It does not claim label-free target calibration or reuse the companion paper’s experiments as evidence.Source: §2.4 · p. 2

3 Problem Setup

• The judge input is X, its scalar correctness probability is S, reference correctness is Y, and a post-hoc map g produces Q = g(S).The protocol elicits S directly, without an intermediate binary verdict.Source: §3 · p. 2
• Temperature and constrained Platt scaling are fit by source log loss, never by target-test ECE.An unconstrained Platt variant was not run.Source: §3 · p. 3
• A strictly increasing recalibration map preserves AUROC, so preserved ranking is compatible with either good or poor calibration.Ranking alone cannot validate probability transfer.Source: §3 · p. 3

4.1 A population lower bound

• The accuracy gap warns of target calibration error only to the extent that it is not absorbed by a corresponding shift in the calibrator’s mean output.The gap is not itself a measure of conditional reliability drift.Source: §4.1 · p. 3

4.2 The accuracy gap is not sufficient

• Identical score distributions and source–target accuracy gaps can yield target calibration errors of 0 and 1/2 for the same source-calibrated map.Without restrictions on target conditional labels, some observationally indistinguishable targets force every source-plus-unlabeled procedure to incur at least 1/4 calibration error on one target; this is why a gap-based gate is not label-free.Source: §4.2 · p. 3

4.3 What exact importance weighting guarantees

• Under covariate shift, exact importance weighting recovers target proper loss when target inputs are supported by the source and conditional labels are invariant.Minimizing exactly weighted source Brier or log loss is equivalent to minimizing the corresponding target risk over the same function class; estimated pipeline failure is compatible with multiple explanations.Source: §4.3 · p. 3

4.4 A finite-sample certificate of transfer failure

• The simultaneous finite-sample certificate rejects a calibrator only when its lower bound exceeds the tolerance; non-rejection does not certify good calibration.The calibrator family must be fixed independently of audit samples, or audit-based selection must use a separate split or a predeclared family covered by the union bound.Source: §4.4 · pp. 3–4

5.1 Tasks, responses, and labels

• The study varies answer format and evaluation mechanism across eight English domains, with 40 items per domain and frozen items, outputs, and prompts.HumanEval uses unit-test execution; two proxy-label domains support robustness analyses rather than deterministic-contract calibration claims.Source: §5.1 · p. 4, Table 1 · p. 5

5.2 Judges and confidence extraction

• Eight open judges spanning 1.1B–7B parameters form the primary set; confidence is elicited as a verbalized probability and parsed under a fixed protocol.The design also scores four modern 7–9B judges and one frontier API judge; malformed responses receive a neutral score of 0.5 and are retained.Source: §5.2 · p. 4

5.3 Splits and transfer units

• Within each domain, seeded question groups split into 24-item fit and 16-item audit parts; target audit labels form the probe, while 24 fit items remain an untouched test.The four source/target fit, audit, and test roles are disjoint within each transfer; clustered inference accounts for item reuse across transfers.Source: §5.3 · p. 4
• The protocol covers 1,176 ordered transfers: 448 from eight primary judges across eight domains, 224 from modern judges, 56 from a frontier judge, and 448 in a second-generator replication.The replication uses all eight primary judges on Llama-3.2-1B responses; bootstrap resamples are not counted as new configurations.Source: §5.3 · pp. 4–5

5.4 Calibration methods

• Comparisons include source-fitted calibrators, importance-weighted Platt scaling, label-shift correction, and target-only temperature and Platt baselines using the same 16 probe labels.The density ratio is estimated from TF-IDF question text with logistic regression; weights are clipped at their 0.95 quantile and mean-normalized, without cross-fitting at this sample size.Source: §5.4 · p. 5

5.5 Synthetic identification controls

• Four synthetic controls separate mild covariate shift, conditional shift, label shift, and a designed case where importance weighting must matter.The positive control uses a piecewise conditional relationship outside the Platt class and opposing source and target tilts; each control has 30 replications per judge.Source: §5.5 · p. 5

5.6 Metrics and inference

• Brier score and clipped log loss are primary metrics; paired method contrasts use a two-way judge-and-domain cluster bootstrap and transfer-level inference.ECE and AUROC are secondary, with ECE interpreted only in aggregate at the 24-item test size; the gap model also adjusts for probe raw Brier, score-marginal distance, and effective sample size.Source: §5.6 · pp. 5–6

6.1 Calibration drifts independently of ranking

• Raw target Brier ranges from 0.125 to 0.875 (mean 0.414), while source-to-target AUROC changes by a mean magnitude of 0.117, separating probability drift from ranking drift.The AUROC change has a 90th percentile of 0.254.Source: §6.1 · p. 6

6.2 Does the accuracy gap predict recalibration benefit?

• The leak-free gap association is inconclusive across generators: r = 0.25 (95% CI [−0.09, 0.55]) falls to r = 0.09 on the second generator, while target temperature scaling reaches −0.155 with 0.05 harm rate there.Probe-split raw target error, unlike the gap, replicates as a predictor.Source: §6.2 · pp. 6–7

6.3 Importance weighting: assumption or implementation?

• Estimated importance weighting recovers the designed control’s oracle improvement (0.256 vs. 0.257), but natural-text IW matches Platt (0.362 vs. 0.362) under measured support failure.The authors locate the natural-transfer failure in the observed text representation and make no claim about other representations.Source: §6.3 · p. 7

6.4 Can the certificate detect harmful transfer?

• At m = 1024, certificate power is only 0.128 despite simultaneous coverage of 1.0; m = 16 target temperature scaling reaches −0.171 with a 0.09 harm rate.The certificate rejects nothing in the 2,688 real audit cells at benchmark budgets.Source: §6.4 · p. 8, Table 5 · p. 8

6.5 Proxy-label sensitivity and downstream decisions

• Proxy-label relabeling keeps the gap association in the r = 0.30–0.38 band, while pooled Platt raises decision cost from 0.507 to 0.728 and lowers coverage from 0.74 to 0.37.Strictly increasing recalibration leaves risk–coverage behavior unchanged exactly.Source: §6.5 · p. 8

Discussion and conclusion

• The accuracy gap is neither a direct measure of P(Y | S) shift nor sufficient for scalar recalibration transfer, and the analysis does not establish a generator-invariant association.Corollary 3 retains a narrower role for the gap as one term in a calibration-error lower bound.Source: §7 · p. 8
• Theorem 7 turns the population lower bound into a valid rejection rule, but the rule is too conservative at available label budgets.At m = 16, target temperature scaling uses labels more effectively and with less harm.Source: §7 · pp. 8–9
• Importance-weighting failure should be traced to overlap, representation, estimation, or conditional shift rather than attributed to conditional shift by default.Reliable scalar confidence transfer requires target evidence or a defended transport assumption; a marginal accuracy gap supplies only a warning.Source: §7 · p. 9

• The theory covers binary correctness and scalar confidence; ordinal scores and pairwise preferences require separate calibration targets.Source: Limitations · p. 9
• English-only experiments use two small generators and 40 items per domain, so noisy per-transfer estimates support only aggregate, cluster-adjusted claims.Source: Limitations · p. 9
• The certificate is conservative and one-sided, and its practical value at the larger budgets where it may bind is untested.No calibration metric alone establishes downstream utility; operational thresholds require a task-specific loss and separate validation.Source: Limitations · p. 9

• Automated judges can amplify verifier, language, and domain biases; released artifacts exclude private prompts and personal data, with label sources separately identified.The paper states that dataset terms, model licenses, and API policies will be followed, and that no human annotation is used.Source: Ethics Statement · p. 9

A Deferred Proofs

• A Jensen-based mean-residual bound feeds a simultaneous finite-sample certificate derived using Hoeffding’s inequality and a union bound.Source: A Deferred Proofs · pp. 10–11
• Identical zero accuracy gaps can accompany target calibration errors of 0 and 1/2, while indistinguishable unlabeled target data can leave worst-case calibration error at least 1/4.Source: A Deferred Proofs · pp. 10–11
• Under conditional invariance and absolute continuity, importance weighting yields pointwise target-risk equality, so minimizers over a common class coincide when they exist.Source: A Deferred Proofs · p. 11

C Judge Prompt and Parsing Contract

• The shared judge prompt requests one correctness score from 0 to 1; parsing clips numeric outputs and assigns 0.5 when no numeric literal or an inference exception occurs.Source: C Judge Prompt and Parsing Contract · p. 11

D Full Calibration Protocol

• Calibrators are fit by source-domain maximum likelihood with specified optimizer constraints, while raw scores are clamped before logit transforms.Source: D Full Calibration Protocol · p. 11

B Model and Inference Manifest

• The reproducibility manifest records judge and generator configurations, inference settings, and the missing primary-judge revision hashes from the original scoring.Source: B Model and Inference Manifest · p. 11, Table 6 · p. 12

E Reliability and Shift Diagnostics

• Pooled reliability diagrams can hide opposite domain-level calibration errors, while the accuracy-gap plot shows only limited predictive value for held-out target ΔBrier.Source: E Reliability and Shift Diagnostics · p. 12

F Proxy-Label Audit

• Hard relabeling agrees with committed labels on 32.5% of SAMSum items and 80% of TruthfulQA items, while the probe-gap association remains r = 0.30–0.38 across threshold variants.Source: F Proxy-Label Audit · p. 12

G Complete Second-Generator Results

• Table 7 provides the full Llama-3.2-1B replication method suite, parallel to Table 2, for auditing the selected replication results.Source: G Complete Second-Generator Results · p. 12

H Parse-Failure Sensitivity

• Figure 2 presents per-judge reliability diagrams pooled across domains, and Figure 3 plots accuracy gap against held-out target ΔBrier for 448 primary transfers.Pooled curves can hide opposite domain-level calibration errors.Source: Figure 2 · p. 13, Figure 3 · p. 13
• Excluding parse failures moves means by at most 0.008 Brier, leaves target-temperature harm rates unchanged, and barely changes the gap and raw-error regression coefficients.The target-temperature harm rates are 0.094 on the primary generator and 0.047 on the second generator under every treatment.Source: H Parse-Failure Sensitivity · pp. 12–13

I Prompt and Order Robustness

• Across prompt variants, target temperature was best or tied-best; mean |∆| ≤0.03 Brier for temperature, Platt, and target temperature, with sign agreement 0.67–0.94.The prompt and presentation-order controls covered all eight primary judges on the primary generator.Source: I Prompt and Order Robustness · p. 13
• Prompt paraphrases had 1.8%–10.7% failure rates, while prompt B drove Gemma-2 to 59%; the gap association stayed weak at r = 0.25–0.32.The paper reports that prompt choice neither rescues nor destroys the gap association.Source: I Prompt and Order Robustness · p. 13, Table 9 · p. 14

J Split-Seed Sensitivity

• Across ten seeds, primary-generator target temperature stayed best or tied-best (harm rate ≤0.16; mean ∆Brier [−0.186, −0.152]), while second-generator gap correlations and adjusted slopes spanned zero.The primary-generator gap correlation ranged from 0.21 to 0.45; the second-generator correlation ranged from −0.08 to 0.20 and adjusted slope from −0.25 to 0.12.Source: J Split-Seed Sensitivity · p. 13
• On the second generator, source-fitted transfer was milder, EM label-shift correction was the only harmful method, and equal-budget target temperature remained best; Table 8 compares parse-failure handling variants.Table 8 reports headline quantities under committed imputation, exclusion of flagged pairs, and second-generator digitconstrained decoding.Source: Table 7 · p. 14, Table 8 · p. 14

K Analyses Run and Not Run

• Prespecified checks covered judge/domain holdouts, label thresholds, certificate budgets, model and generator replications, parsing, prompt/order, and split seeds; stochastic decoding and several alternatives were not run.Unrun alternatives included other gap definitions, smoothed ECE variants, unconstrained Platt slope, weight-clipping sweeps, and Holm-adjusted contrasts.Source: K Analyses Run and Not Run · p. 14

L Supplementary Figures

• Only probe-split raw Brier’s adjusted coefficient excludes zero; transfer-gap and distribution-shift predictors remain uncertain.The adjusted regression is shown as a coefficient forest plot with cluster-bootstrap confidence intervals.Source: L Supplementary Figures · p. 14, Figure 4 · p. 15
• Target-aware temperature methods have the strongest gains and lowest harm rates, while certificate power improves modestly and false acceptance remains high at m = 1024.Figure 5 reports target Brier change and harm rate; Figure 6 reports certificate behavior by audit size.Source: L Supplementary Figures · p. 14, Figure 5 · p. 15, Figure 6 · p. 15