PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems

Eddie Landesberg, Manjari Narayan

arXiv:2512.11150v3stat.MEstat.APstat.ML

TL;DR

Uncalibrated LLM judges can misrank systems, while oracle outcomes are expensive to measure. CJE calibrates judge scores against a small oracle slice and audits calibration transport across target policies; on Arena prompts it reports high pairwise ranking accuracy at lower cost.

  • Problem

    Oracle labels for long-run outcomes are expensive, so teams use cheap judge scores that can diverge from actual value and invert rankings.

    Source: §1 · p. 1

  • Method

    CJE calibrates judge scores against oracle labels and audits whether the calibration remains valid for each target policy, recalibrating or refusing level claims when the audit fails.

    Source: §1 · p. 2

  • Results

    At a 5% oracle fraction, CJE achieves 99% pairwise ranking accuracy; accuracy averages 94% across configurations at 14× lower cost.

    Source: §1 · p. 3

  • Takeaways & Limitations

    For open-ended generation under nontrivial policy shifts, default to Direct, use bias-corrected bootstrap inference, and validate transportability with a mean residual test.

    Source: §7 · p. 23

    When logger coverage β is small, the CLE precision floor remains prohibitive regardless of weight stabilization, even with perfect propensities.

    Source: §2 · p. 5

Abstract

from arXiv · show

Measuring long-run LLM outcomes (user satisfaction, expert judgment, downstream KPIs) is expensive. Teams default to cheap LLM judges, but uncalibrated proxies can invert rankings entirely. Causal Judge Evaluation (CJE) makes it affordable to aim at the right target: calibrate cheap scores against a small oracle slice, then evaluate at scale with valid uncertainty. We treat surrogate validity as auditable: for each policy or deployment context, a small oracle audit tests whether the learned calibration remains mean-unbiased, turning an uncheckable identification condition into a falsifiable diagnostic. On 4,961 Chatbot Arena prompts comparing five policies with a 16x oracle/judge cost ratio, at a 5% oracle fraction CJE achieves 99% pairwise ranking accuracy at 14x lower cost; across all configurations (5-50% oracle, varying n), accuracy averages 94%. An adversarial policy fails the transport audit and is correctly flagged; in such cases CJE refuses level claims rather than reporting biased estimates. Key findings: naive confidence intervals on raw judge scores achieve 0% coverage (CJE: ~95%); importance-weighted estimators fail despite >90% effective sample size; and the Coverage-Limited Efficiency (CLE) bound and its TTC diagnostic explain why.

1 INTRODUCTION

Cheap LLM judges can systematically misrank configurations because their scores may diverge from oracle outcomes. CJE calibrates judge scores against a small oracle slice and audits whether that calibration remains valid for each target policy.

1 INTRODUCTION

  • CJE calibrates judge scores to oracle labels, then audits transport for each target policy and recalibrates or refuses level claims when the audit fails.The protocol makes calibration validity auditable rather than assumed.Source: §1 · p. 1
  • Uncalibrated judge scores can invert rankings because higher scores may predict lower oracle rewards, while judges may reward verbosity independent of quality.Teams may use these proxies because oracle labels for long-term outcomes are expensive.Source: §1 · pp. 1–2
  • Naive confidence intervals on uncalibrated scores cover the true value in 0/50 seeds, while bias-corrected bootstrap inference restores coverage to approximately 95%.At a 5% oracle fraction, calibration uncertainty contributes approximately 90% of total variance.Source: §1 · p. 2
  • CJE combines mean-preserving calibration, stabilized weights, and calibration-aware bootstrap inference; projection restrictions reduce variance and attain the surrogate information bound with cross-fitting.The calibration can use a two-stage option for covariate-dependent bias, such as response length.Source: §1 · pp. 2–3
  • Despite stabilized ESS above 80%, calibrated IPS remains near-random at 47% pairwise accuracy because low target-typical coverage imposes a precision floor.TTC is 19–49% for meaningfully different policies; OPE may remain viable for short outputs, near-clone shifts, or tool-choice bandits.Source: §1 · pp. 2–3
  • At a 5% oracle fraction, CJE achieves 99% pairwise ranking accuracy; accuracy averages 94% across configurations at 14× lower cost.The benchmark compares five policies.Source: §1 · p. 3

2 BACKGROUND AND SETUP

CJE treats a judge score as a surrogate for an expensive outcome, with mean-unbiased evaluation depending on whether the calibration residual transports to each target policy. It combines oracle-audited calibration with direct, logged-data, or hybrid evaluation, while distinguishing calibration transport from logged-data coverage.

Evaluation setup

  • The three evaluation modes are logs-only IPS/SNIPS, Direct estimation from fresh target-policy responses, and hybrid DR; each uses calibration learned from an oracle slice under the logger.Ranking requires calibration to preserve policy ordering, whereas reporting absolute levels requires mean-unbiasedness.Source: §2 · p. 4

CJE as surrogate index for LLM evaluation

  • The paper distinguishes calibration transport from action-space coverage: it audits transport with target-context oracle labels, while CLE/TTC diagnose whether logged actions cover the target.This oracle budget enables amortization when transport holds; the pipeline overview depicts a 5–25% oracle slice used to calibrate bulk evaluation.Source: §2 · p. 4, Figure 2 · p. 7

Coverage-Limited Efficiency

  • Low logger mass in target-relevant regions creates a CLE precision floor that weight stabilization cannot remove, even with perfect propensities.The bound combines a coverage penalty, within-region shape mismatch, and outcome noise; the paper notes that coverage can collapse exponentially with sequence length.Source: §2 · p. 5

Diagnostics for CLE

  • TTC estimates logger coverage of target-typical responses, while Bhattacharyya affinity measures judge-score distribution alignment; the diagnostics assess different forms of overlap.The paper recommends against logs-only IPS when TTC is below 0.7; its stated threshold is calibrated for MDE ≈0.02 at n≈5k.Source: §2 · p. 5

Surrogate interpretation

  • CJE’s surrogate estimate is mean-unbiased for a target policy if and only if the calibration residual has mean zero there, a condition tested on a small oracle audit slice.When the audit fails, the paper recommends re-anchoring calibration or refusing level estimates; the mean test does not guarantee subgroup or tail calibration.Source: §2 · p. 6

Transportability

  • CJE’s mean-residual transport condition is weaker than Prentice-style surrogacy and suffices for unbiased mean values, but not conditional calibration within subgroups or distribution tails.Decile-binned residual checks can detect compensating errors that cancel in the overall mean.Source: §2 · p. 6

3 METHODS

CJE combines mean-preserving reward calibration, optional stabilized importance weights, estimators for fresh or logged responses, and uncertainty-aware inference. Its calibration and inference address judge bias and calibration uncertainty, while logged-data methods remain constrained by overlap.

3 METHODS

  • The CJE pipeline combines reward calibration, weight stabilization, and uncertainty-aware inference; for nontrivial policy shifts, calibration and inference are essential while low TTC limits logged-data methods.The paper describes its model restrictions as monotonicity and mean-preservation for rewards, mean-one constraints for weights, nuisance orthogonality, and convex stacking.Source: §3 · p. 7

3.1 REWARD CALIBRATION (TWO-STAGE ISOTONIC)

  • Two-stage calibration fits a smooth index using judge scores and covariates, then applies isotonic regression to preserve mean-honesty while accommodating flexible patterns.Including response length as a first-stage covariate improves ranking accuracy in the experiments by addressing judges’ preference for longer responses independent of quality.Source: §3.1 · p. 8, Figure 3 · p. 8

3.2 WEIGHT STABILIZATION (UNIT-MEAN RATIOS VIA OOF STACKING OF S-MONOTONE CANDIDATES)

  • Weight stabilization stacks increasing and decreasing judge-score-based candidates while restoring mean-one weights, but low TTC limits its utility for open-ended generation under nontrivial shifts.The paper prefers Direct in low-coverage settings and identifies TTC ≥0.70 as a setting where stabilization remains useful.Source: §3.2 · pp. 8–9

3.3 ESTIMATORS: DIRECT, CALIBRATED IPS, AND CALIBRATED DR

  • Direct uses calibrated rewards on fresh target-policy responses without importance weights; DR adds a weighted residual correction that can add noise when TTC < 0.7.In the experiments, DR effectively reduces to its outcome model, and Direct achieves the same accuracy without requiring log probabilities.Source: §3.3 · p. 9

3.4 INFLUENCE-FUNCTION STACKING (VARIANCE-OPTIMAL CONVEX ENSEMBLING)

  • Influence-function stacking forms centered influence-function columns for candidate estimators and selects simplex weights to minimize their estimated covariance.Source: §3.4 · p. 9

3.5 CALIBRATION-AWARE INFERENCE

  • Bias-corrected augmented inference achieves ∼95% coverage for Direct estimates by accounting for calibration uncertainty; naive confidence intervals on uncalibrated scores achieve 0% coverage.At a 5% oracle fraction, calibration uncertainty accounts for over 50% of total variance; bootstrap refits the calibrator in each replicate.Source: §3.5 · p. 10

4 THEORY: EIF, MODEL RESTRICTION, AND EFFICIENCY

CJE’s projections are information-optimal: justified restrictions can tighten efficiency bounds without changing the estimand, and the proposed estimators attain those bounds under standard conditions. The theory connects these restrictions to calibration, weight stabilization, and influence-function stacking, while specifying assumptions and diagnostics for valid use.

4 THEORY: EIF, MODEL RESTRICTION, AND EFFICIENCY

  • Var(ϕsur) ≤ Var(ϕuncon): using the surrogate cannot increase variance, with strict reduction unless the weights are σ(S)-measurable and R⋆ is degenerate.The paper notes that coarser judges are strictly worse when the judge carries information about the oracle.Source: §4 · p. 11
  • Restricting the model’s tangent space weakly reduces the canonical-gradient norm, and stronger restrictions can further tighten the efficiency bound.The paper presents CJE’s projections as attaining these bounds under standard conditions.Source: §4 · p. 11
  • CJE treats reward calibration, weight stabilization, and influence-function stacking as model restrictions that operationalize the efficiency principle.Examples include conditioning on S, mean-one monotone weights, and convex combinations of candidate influence-function columns.Source: §4 · p. 11
  • When fresh responses under π′ are feasible, Direct is the recommended default: it needs no overlap assumption and attains the missing-data efficiency bound.With approximately uniform labeling, the one-step estimator attains the semiparametric efficiency bound.Source: §4 · pp. 11–13
  • For OPE, calibrated DR attains the surrogate efficiency bound under mean sufficiency and product-rate conditions; weight stabilization and IF stacking add variance control.The budgeted information bound trades some asymptotic efficiency for improved finite-sample behavior; jackknife variance consistency requires regularity conditions, complicated by isotonic regression’s non-smoothness.Source: §4 · p. 12
  • Mean sufficiency supports the EIF derivations, but valid estimation fundamentally requires policy-wise mean transport, which audits can test without ruling out subgroup or tail miscalibration.The theory also extends two-stage calibration by replacing S with a consistently estimated index Z(S, X).Source: §4 · pp. 12–13

5 EXPERIMENTS

Across 4,961 Chatbot Arena prompts, the experiments compare calibrated judge-based evaluation with oracle labeling and logged-data estimators. CJE achieves high ranking accuracy at lower cost, while transport and coverage diagnostics identify settings where estimates should not be trusted.

5.1 EXPERIMENTAL DESIGN

  • The benchmark retains 4,961 of 5,000 first-turn English Arena prompts and compares five policies across 13 estimators, five oracle fractions, five sample sizes, and 50 seeds.Oracle labels were approximately 16× more expensive than judge scores.Source: §5.1 · pp. 13–14

5.2 RESULTS

  • 94.4% pairwise accuracy for direct+cov versus 90.9% for naive-direct; bias-corrected bootstrap restores coverage from 0% to ∼95%.Direct achieves ∼95% coverage across oracle fractions with calibration-aware bootstrap inference.Source: §5.2 · pp. 14–15
  • 47% pairwise accuracy versus 50% chance persisted after stabilized ESS rose above 80%, as low TTC imposed a CLE precision floor.For non-clone policies, TTC ranges from 0.19 to 0.49 and CLE factors from 24–61×.Source: §5.2 · p. 15
  • 94.4% direct+cov slightly exceeds calibrated-dr+cov’s 94.1%; under low overlap, DR’s IPS term adds noise rather than information.The authors recommend Direct when teacher-forcing log probabilities are unreliable.Source: §5.2 · p. 15

5.3 COMPARISON TO MISCLASSIFICATION CORRECTION

  • Against Rogan–Gladen at τ=0.6, CJE achieves 93% lower RMSE (0.004 vs. 0.061) and 9× narrower confidence intervals (0.015 vs. 0.136).Both methods achieve near-nominal coverage in this Arena preference comparison.Source: Table 5 · p. 18, §5.3 · p. 19
  • Under class-balance shift, Rogan–Gladen bias stays below 0.007 while CJE bias reaches −0.14 or +0.14; with no shift, CJE RMSE is 0.009 vs. 0.032.This comparison identifies a robustness advantage for confusion-matrix methods when calibration and test distributions differ.Source: §5.3 · p. 19
  • 99% pairwise ranking accuracy at n=5,000 with 5% oracle labels costs 14× less than pure-oracle evaluation across five policies.Accuracy averages 94% across configurations; the 99% figure is the reported peak.Source: Table 6 · p. 20, §5.3 · p. 20

5.4 PRACTICAL TAKEAWAYS

  • For deployment, use Direct + two-stage calibration, audit levels per policy, bootstrap with θaug, and spread a fixed oracle-label budget across more prompts.The authors recommend refusing level claims when a transport audit fails; for a fixed label budget, larger n with a lower oracle fraction performs better.Source: Figure 7 · p. 19, §5.4 · p. 20

6 LIMITATIONS

CJE’s guarantees are bounded by overlap, surrogate validity, and calibration support, while data quality and deployment conditions can also limit inference. The paper identifies additional risks from temporal drift, selection, subgroup variation, and judge informativeness.

Overlap (positivity)

  • Poor overlap inflates uncertainty for importance-weighted estimators; Direct requires no overlap, while doubly robust estimation benefits from overlap.Weight stabilization reduces weight dispersion but does not remove the coverage limitation.Source: §6 · p. 21

Judge assumptions (surrogate validity)

  • A monotone single-index calibration is a stability choice, and policy-specific mapping shifts, score extrapolation, or temporal drift can undermine calibration.The paper proposes mean transport, coverage, and periodic residual diagnostics for these distinct failure modes.Source: §6 · p. 21

Scope of the mean transport test

  • The policy-wise mean transport test is necessary but not sufficient for full transportability, since it does not rule out subgroup or tail miscalibration.Subgroup tests, reliability diagrams, or richer moment conditions can probe those deviations.Source: §6 · p. 21

Calibration coverage (identification)

  • When a target policy produces judge scores outside the labeled range, isotonic calibration flattens at the boundary and levels are not point-identified.The paper recommends flagging limited support and reporting rankings or partial-identification bounds until targeted labels cover the region.Source: §6 · p. 21

Approximate sufficiency (bias modulus)

  • If the conditional outcome is not captured by the surrogate, calibration error can induce bias; the bound is informative only when its residual factor is controlled.The paper calls for diagnostics and REFUSE-LEVEL when bias is unbounded.Source: §6 · p. 21

Temporal dependence and logger drift

  • Non-stationarity, including launches or safety updates, can bias estimates or widen intervals, motivating dependence-robust errors and shorter analysis windows.The paper also recommends monitoring judge drift with rank- or residual-change detection.Source: §6 · p. 21

Data and oracle assumptions

  • Inference assumes an i.i.d. oracle slice with non-overlapping folds, and the oracle’s alignment with stakeholder values remains outside the study’s scope.The experiments also filter prompts for teacher-forcing conformance or missing target log probabilities, and report propensity error even for clone policies.Source: §6 · pp. 21–22

Selection, subgroups, and judge informativeness

  • Scanning many target policies risks winner’s curse, while calibration and effective-sample-size gains may differ across subgroups and judge coarseness widens intervals.The paper recommends multiplicity controls, subgroup diagnostics, and richer judge rubrics where feasible.Source: §6 · p. 22

7 CONCLUSION

CJE addresses failures of uncalibrated judge evaluation by calibrating cheap proxies against oracle outcomes and encoding justified assumptions as model restrictions. On Arena-derived logs, it reports improved ordering and near-nominal coverage, while limited calibration support leads to conservative rank-only conclusions.

Framework and methods

  • CJE combines mean-preserving reward calibration, weight stabilization, and calibrated doubly robust inference with cross-fitting and bias-corrected bootstrap intervals.The framework encodes justified assumptions as restrictions that preserve the estimand while weakly reducing variance.Source: §7 · pp. 22–23

Uncertainty quantification

  • Bootstrap inference with bias-corrected estimation achieves near-nominal confidence-interval coverage.The conclusion contrasts this with naive intervals on uncalibrated scores, which have near-zero coverage.Source: §7 · p. 23

Empirical results

  • On Arena-derived logs, weight stabilization yields large ESS gains, stacked DR achieves near-nominal coverage and near-√n scaling, and stacking improves ordering.When calibration support is limited, CJE flags the issue and reports robust rankings with conservative uncertainty.Source: §7 · p. 23

Theory

  • The Efficiency via Model Restriction result says justified restrictions lower the efficiency bound, attainable with projection-designed estimators and cross-fitting.The paper also states that richer judges strictly help and exact IsoMeanOneS projection weakly reduces weight dispersion.Source: §7 · p. 23

Takeaways

  • For open-ended generation under nontrivial shifts, the paper recommends Direct over logs-only OPE, auditing transport, and using bootstrap inference.It advises gating logged-data use on TTC ≥0.70; in the benchmark, only near-clone policies pass.Source: §7 · p. 23

Future work

  • Future work includes selection-aware inference, robust or differentially private isotonic calibration, active oracle budgeting, sequential evaluations, and subgroup-aware constraints.Source: §7 · p. 23

8 RELATED WORK

This section positions CJE’s reusable calibration and target-context audit alongside work on LLM judges, calibration, and off-policy evaluation. It also establishes variance and efficiency properties for surrogate-based estimators, calibrated importance sampling, doubly robust estimation, and influence-function stacking, including conditions for variance reduction, calibration-aware uncertainty, and effective-sample-size preservation.

8 RELATED WORK

  • CJE attempts cross-environment calibration reuse but audits transport with target-context oracle labels, unlike efficiency-only surrogates and single-environment surrogate indices.When the mean transport test fails, CJE flags the policy or context for recalibration or oracle-only evaluation; each target-context oracle slice also supports explicit pass/fail decisions, including refusing level claims.Source: §8 · pp. 23–24
  • CJE’s transportability approach parallels negative controls and proximal causal inference in making assumptions checkable, but is not a negative-control estimator.It tests the policy-wise moment restriction Eπ′[Y − f(S, X)] = 0 and provides decisions to reuse calibration, recalibrate, or refuse level claims.Source: §8 · p. 24
  • CJE calibrates a fixed LLM judge for unbiased policy-value estimates and calibration-aware confidence intervals, complementing work that ranks judges by correlation with human labels.It also complements CalibraEval’s label-free correction of position and token bias with oracle-grounded calibration and transportability guarantees.Source: §8 · p. 24
  • 13× more sample-efficient: continuous calibration versus binary misclassification correction, even when mean J = 0.41.The passage attributes the gap to standard error scaling as 1/J and information loss from binary thresholding; both approaches rely on calibration-to-test stability.Source: §8 · p. 24
  • CJE extends EIF-based judge debiasing to continuous scores, policy and deployment shift, and off-policy evaluation with weight stabilization.Its influence-function stacking combines estimators by minimizing the asymptotic variance of their combined influence function, rather than combining predictions to minimize MSE.Source: §8 · pp. 24–25

C.2 ISOTONIC REGRESSION: MEAN PRESERVATION AND MAJORIZATION

  • Isotonic fitting is mean-preserving and reduces dispersion: with sample mean one, it weakly lowers variance and weakly raises effective sample size.The result follows from the fitted vector being a mean-preserving adjacent pooling of the sorted observations.Source: §C.2 · p. 34

C.3 PROOF OF THEOREM 1 (SURROGATE EIF & VARIANCE DROP)

  • Under mean-sufficiency, the surrogate efficient influence function has variance no greater than the unconstrained one, with strict reduction except in the stated degenerate case.The exception requires Wπ′ to be measurable with respect to S and R⋆ to be degenerate.Source: §C.3 · p. 35

C.4 PROOF OF THEOREM 2 (EFFICIENCY VIA MODEL RESTRICTION)

  • For nested model restrictions, a smaller tangent space yields no greater canonical-gradient variance, and one-step corrections or cross-fitted TMLE can attain the restricted efficiency bound.Stronger restrictions correspond to smaller tangent spaces.Source: §C.4 · p. 36

C.5 PROOF OF COROLLARY 1

  • Using a finer surrogate S1 gives variance no greater than using a coarser S2 when σ(S2) ⊆ σ(S1), with strictness failing only under the stated measurability and degeneracy conditions.Strictness fails only if Wπ′ is σ(S2)-measurable and R⋆ is degenerate.Source: §C.5 · p. 36

C.7 PROOF OF THEOREM 3 (CALIBRATED DR √n LIMITS)

  • Cross-fitted calibrated DR is asymptotically normal with variance Var(ϕsur) under a one-of-two product-rate condition and cross-fitting.The product-rate condition is ∥Ŵπ′ − m⋆∥2 · ∥q̂ − q⋆R∥2 = op(n−1/2).Source: §C.7 · p. 37

C.8 PROOF OF THEOREM 4 (BUDGETED BOUND)

  • The budgeted bound is monotone in ρ, converges to the surrogate influence-function bound as ρ grows, and is attained by calibrated DR under the stated weight-stabilization condition.The bound projects m⋆ onto Wρ; attainment uses the same one-of-two rate argument.Source: §C.8 · p. 37

C.9 PROOF OF THEOREM 5 (INFLUENCE-FUNCTION STACKING) AND COROLLARY 2; C.10 PROOF OF PROPOSITION 3 (CALIBRATION-AWARE JACKKNIFE)

  • Variance-optimal influence-function stacking has asymptotic variance no greater than the best candidate, while the delete-one-oracle-fold jackknife consistently estimates calibration variance.When rank(Φ) = r, the stack needs at most r+1 base estimators.Source: §C.9 · p. 37, §C.10 · p. 37

C.11 AUXILIARY LEMMAS USED IN THE MAIN PROOFS

The auxiliary lemmas establish properties of cross-fitted calibration and weighting procedures, including mean preservation, remainder control, and stability of projections.

C.11 AUXILIARY LEMMAS USED IN THE MAIN PROOFS

  • Out-of-fold reward predictions preserve the sample mean up to op(1) under L2-consistency of the calibration learner.Source: §C.11 · p. 37
  • Lemma 4 identifies bVDR as calibrated DR with cross-fitted (q-hat, g-hatπ′) and calibrated W-hatπ′, and its heading states a second-order remainder for this estimator.The reference implementation uses a simpler absolute-cap formula α = min{1, ρ/Var(W stack)} and omits final isotonic re-projection, normalizing to mean one; this relaxation empirically yields similar ESS improvements but lacks Lemma 5’s deterministic guarantee.Source: §C.11 · p. 37
  • The weight blend's variance is bounded by the cap, and the subsequent mean-one isotonic projection cannot increase empirical variance.Source: §C.11 · p. 37
  • Metric projections onto closed convex sets are firmly non-expansive, so compositions of the projection modules are non-expansive.For serial or cluster dependence, the influence-function central limit arguments can be replaced by block or stationary bootstrap arguments.Source: §C.11 · p. 38

C.12 COVERAGE-LIMITED EFFICIENCY (CLE) BOUND

The appendix develops the CLE bound and related diagnostics for when logged policy evaluation is limited by coverage. It also describes direct-mode inference, transport audits, and the scope of the bounds.

C.12 COVERAGE-LIMITED EFFICIENCY (CLE) BOUND

  • The CLE precision floor grows with the coverage penalty α/√β, within-region shape mismatch, and outcome noise σT/√n; high global ESS alone is insufficient.The bound focuses on target-relevant regions, where effective sample size matters more than its global value.Source: §C.12 · p. 38

C.13 TTC COLLAPSE WITH SEQUENCE LENGTH (STRUCTURAL COVERAGE SPARSITY)

  • At L=400, TTC falls below 0.01 for η≥0.3, while IPS RMSE grows 20–50× and Direct RMSE remains ∼0.01.Figure 9 and the accompanying simulation show that coverage collapse persists with exact importance weights; the appendix attributes it to trajectory-space sparsity.Source: Figure 9 · p. 39, §C.13 · p. 39

C.14 EIF DERIVATION FOR DIRECT MODE

  • Under MAR, averaging the EIF with estimated outcome and labeling models yields a one-step estimator; the cross-fitted augmented estimator is asymptotically equivalent to TMLE.Source: §C.14 · p. 40

C.15 ASSUMPTIONS AND WHAT CJE RELAXES

  • CJE tests mean transport with an oracle audit of about 50–200 labels per target policy: a pass permits reuse, while failure prompts recalibration or refusal of level claims.The bridge assumption aligns the operational oracle with the idealized oracle; oracle selection remains a governance and construct-validity question.Source: §C.15 · p. 40

C.16 ADDITIONAL COROLLARIES

  • A garbled surrogate signal cannot improve the variance bound over a finer signal, and a rank-r influence-function matrix needs at most r+1 base estimators in its variance-optimal convex combination.Source: §C.16 · p. 41

C.17 WHAT THE BOUNDS DO NOT INCLUDE

  • The stated information bounds omit finite-sample dispersion control from the sample cap and oracle first-stage uncertainty, which calibration-aware inference adds separately.Source: §C.17 · p. 41

C.18 BUDGETED BOUND AND IF STACKING (OPE-SPECIFIC RESULTS)

For off-policy evaluation, this section gives a budgeted information bound and a variance-minimizing influence-function stacking result. It also specifies diagnostics and reporting procedures for evaluation and uncertainty.

C.18 BUDGETED BOUND AND IF STACKING (OPE-SPECIFIC RESULTS)

  • The optimal asymptotic variance under the weight cap is nonincreasing in ρ and approaches the surrogate bound as ρ grows; influence-function stacking attains variance no greater than any single candidate estimator.The cap trades some asymptotic efficiency for better finite-sample behavior.Source: §C.18 · p. 41

D DIAGNOSTICS, GATES, AND REPORTING (DETAILS)

D.1 WEIGHT BEHAVIOR & OVERLAP

  • Weight reporting includes the ESS fraction, ESS uplift, and max-weight share, alongside the empirical 99.5th percentile of weights.Source: §D.1 · p. 42

D.2 JUDGE CALIBRATION, COVERAGE, AND DRIFT

  • Reliability diagrams compare calibrated predictions with oracle bin means; out-of-range mass plus flat boundary slopes triggers limited calibration support and the refuse-level gate.Rank drift can also be monitored using Kendall’s τ on a fixed anchor set.Source: §D.2 · p. 42

D.3 DR ORTHOGONALITY AND DECOMPOSITION

  • The DR orthogonality diagnostic reports the mean residual contribution and its confidence interval; a near-zero mean indicates successful orthogonality.Source: §D.3 · p. 42

D.4 UNCERTAINTY: IF VARIANCE AND CALIBRATION UNCERTAINTY ADDITION

  • Bootstrap inference with the augmented Direct estimator achieves approximately 95% coverage by correcting both variance and bias.Table 8 reports coverage, centering, and calibration across oracle fractions.Source: §D.4 · p. 43, Table 8 · p. 43

D.5 MULTIPLICITY FOR MANY-POLICY COMPARISONS

  • For many-policy comparisons, report a pairwise win matrix with false-discovery-rate marks and Kendall’s τ over policy means.Source: §D.5 · p. 43

D.6 POLICY-WISE MEAN TRANSPORT DIAGNOSTIC

The appendix makes surrogate validity testable policy by policy, then specifies gates and implementation practices for calibrated evaluation. Its diagnostics distinguish transport failure and inadequate coverage from problems that weight stabilization alone can solve.

D.6 POLICY-WISE MEAN TRANSPORT DIAGNOSTIC

  • Rejecting a policy’s mean-transport test indicates its surrogate-only value estimate is systematically biased and does not transport.The test compares the policy’s mean residual with zero; a failed policy can be recalibrated with policy-specific oracle data or evaluated using oracle-only data.Source: §D.6 · p. 44

D.7 GATES: THRESHOLDS AND ACTIONS

  • The gate defaults are calibrated for n ≈5,000 and MDE ≈0.02; for example, ESS/n ≥0.30 bounds variance inflation to ≤3.3×.The appendix also recommends deriving tighter thresholds for high-stakes applications from the CLE bound.Source: §D.7 · p. 44
  • For level claims, transport is fundamental; when identification fails, refuse level estimates and report rank-only conclusions with conservative relative confidence intervals.Structural assumptions such as mean sufficiency can imply transport, but an oracle audit can test transport directly.Source: §D.7 · p. 44

D.8 PLANNER: MDE AND LABEL/LOG BUDGETS

  • The planner tabulates calibration-aware standard errors across prompt counts and label fractions, with iso-cost lines for the label budget.Source: §D.8 · p. 44

D.9 WEIGHT STABILIZATION VISUALIZATION

  • ESS improvements range from 4.6× for clone to >3000× for parallel universe prompt, yet high ESS alone does not guarantee ranking accuracy.Figure 10 reports that the premium policy’s raw weights span 130 orders of magnitude before stabilization; stabilization recovers ESS from <1% to >80%, while coverage remains binding.Source: Figure 10 · p. 45

E IMPLEMENTATION, ENGINEERING, AND REPRODUCIBILITY

E.3 FOLDS AND CROSS–FITTING

  • Use five stable hash-based folds so reward calibration, weight stabilization, and doubly robust nuisances share out-of-fold boundaries.Oracle folds intersect the shared fold map with the oracle-label indicator.Source: §E.3 · p. 47

E.4 NUMERICS AND STABILITY •

  • The reference implementation’s multiplicative mean-one rescaling may distort isotonicity at extremes; the exact IsoMeanOneS projection preserves both properties.The deterministic majorization guarantee applies only to the exact constrained projection, though similar ESS improvements were observed with rescaling.Source: §E.4 · p. 48

E.9 EXACT PROMPTS USED IN EXPERIMENTS

The experiment appendix records the policy, judge, and oracle prompts used in evaluation. It specifies the scoring instructions and the model and sampling settings for both evaluators.

E.9 EXACT PROMPTS USED IN EXPERIMENTS

  • All policies used temperature 0.7 without top-p or top-k truncation, so teacher-forcing log probabilities represent the exact sampling distribution.Base, clone, and premium shared the helpful-assistant prompt; the other policies used distinct prompt instructions.Source: §E.9 · p. 49
  • The judge was gpt-4.1-nano-2025-04-14 and the oracle was gpt-5-2025-08-07, using identical evaluation prompts but different models and temperatures.The judge used temperature 0.0 and the oracle 1.0; temperature 0 was unavailable for the oracle at experiment time.Source: §E.9 · p. 49
  • Evaluators rated response quality from 0 to 100 on relevance, helpfulness, clarity, and completeness, including when responses were incomplete or truncated.Source: §E.9 · p. 49

F EFFICIENT BUDGET ALLOCATION

The paper derives a variance-minimizing allocation of cheap surrogate scores and expensive oracle labels under a fixed budget. In the Arena example, a 5% oracle allocation trades some precision for lower cost, while the variance-optimal fraction is higher.

F EFFICIENT BUDGET ALLOCATION

  • The Square Root Allocation Law chooses surrogate and oracle sample sizes to minimize total variance subject to a fixed financial or latency budget.The derivation assumes additive variance components, including calibration variance of order 1/m.Source: F EFFICIENT BUDGET ALLOCATION · p. 50
  • At the optimum, the calibration-uncertainty share equals the budget share spent on oracle labels, providing a diagnostic for reallocating budget.A higher calibration uncertainty share than oracle budget share signals under-labeling; a lower share signals over-labeling.Source: F EFFICIENT BUDGET ALLOCATION · p. 50
  • At 5% oracle, the Arena allocation achieves 94% ranking accuracy at 14× lower cost than pure oracle labeling.The variance-optimal oracle fraction is approximately 17%, so the experiment’s allocation was below the variance-optimal share.Source: F EFFICIENT BUDGET ALLOCATION · p. 51
  • When calibration is reused across multiple policies, oracle costs remain fixed while surrogate costs scale with policy count, favoring larger oracle slices as policies grow.Source: F EFFICIENT BUDGET ALLOCATION · p. 51

G CODE GENERATION EXPERIMENT (MBPP)

On MBPP, CJE reduces RMSE for policies that pass transport validation, with benefits increasing as the oracle fraction grows. Two policies fail transport checks and are excluded from the reported RMSE comparison.

G CODE GENERATION EXPERIMENT (MBPP)

  • The benchmark uses 257 MBPP problems and 974 samples across five epochs, with actual unit-test outcomes as oracle labels.The judge is GPT-4o-mini predicting pass probability; monotone-only calibration is used because MBPP has no strong covariate effects.Source: G CODE GENERATION EXPERIMENT (MBPP) · p. 51
  • The mini policy’s mean residual is −0.065 (p=0.006) and the unhelpful policy’s is −0.280 (p<0.001), so both fail transport validation.The mini failure is attributed to a judge–policy confound; the unhelpful failure to sparse calibration density for low-support scores.Source: G CODE GENERATION EXPERIMENT (MBPP) · p. 51
  • The two transport-failing policies are excluded from RMSE computation because including them would bias the results.Source: G CODE GENERATION EXPERIMENT (MBPP) · p. 51
  • 18% RMSE improvement over naive estimation at 5% oracle grows to 32–84% at 10% or more for the two transport-valid policies.The comparison uses n=974 and reports RMSE for enhanced and terse; Table 12 describes diminishing returns, with 10% oracle capturing most of the benefit.Source: G CODE GENERATION EXPERIMENT (MBPP) · p. 51, Table 12 · p. 52
  • The authors report that binary correctness requires more labels to stabilize calibration than Arena’s ordinal, high-variance preference judgments.They describe this as a likely explanation for the different oracle-fraction tradeoffs across the tasks.Source: G CODE GENERATION EXPERIMENT (MBPP) · pp. 51–52

Related papers

Loading 2512.11150v3…