Source-linked AI summary
Optimal Design for Active Preference Learning with Biased LLM Judges
Zhongman Du, Huiming Zhang, Haodong Zhu, Baochang Zhang
TL;DR
Active acquisition with biased LLM judges can shift comparisons and expose residual judge errors, while joint estimation can leave less information about human preferences than raw reward information suggests. NAOD selects policy-relevant target information after nuisance adjustment. On Chatbot Arena, it reduced mean proxy policy regret by 29.11% relative to the matched Target-info design and achieved the best reported mean human-prediction metrics.
Problem
Active acquisition can shift the comparison distribution and expose residual LLM-judge bias, while jointly estimating reward and judge deviations can make target information difficult to distinguish.
Source: §1 · pp. 1–2Method
NAOD selects comparisons using nuisance-adjusted target information weighted by downstream policy relevance.
Source: §1 · p. 2Results
29.11% lower mean proxy policy regret than the matched Target-info design, with the lowest mean human cross-entropy and highest mean human choice accuracy on Chatbot Arena.
Source: §6.2 · p. 9Takeaways & Limitations
NAOD gains tend to be larger under stronger target–nuisance coupling, but representation error can change the preferred acquisition design.
Source: §7 · p. 9The evaluation uses archived Chatbot Arena feedback that may reflect population- and model-specific biases and may not represent other target populations or deployment settings.
Source: ETHICS STATEMENT · p. 10
Abstract
from arXiv · showhide
Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference learning reduces this cost by selecting informative comparisons, and LLM judges can provide additional scalable feedback. However, the preferences of the judges may deviate from those of the target human population. Even after calibration on trusted reference data, active acquisition can shift the comparison distribution and expose residual judge bias. We therefore incorporate judge deviations into the acquisition design rather than relying on a separate calibration stage. Under joint estimation, comparisons that appear highly informative about the reward may also reflect judge bias and therefore provide less information about human preferences. To address this issue, we propose Nuisance-Adjusted Optimal Design (NAOD), a comparison-selection strategy that prioritizes policy-relevant target information after nuisance adjustment and uses the Frank-Wolfe algorithm for optimization. Theoretically, we establish a sharp conditional local asymptotic minimax lower bound on policy risk and construct an estimator that attains it. We further characterize the finite-sample cost of learning the nuisance representation and show that representation error can reverse an oracle design advantage. Finally, we validate these predictions experimentally and evaluate NAOD on Chatbot Arena data across 17 judges, 15 budget configurations, and 15 random cluster-level splits. NAOD reduces the mean regret of proxy policy by 29.1% relative to a matched target-information design, outperforms existing methods, and improves human-preference prediction on held-out data.
1 INTRODUCTION
Active acquisition with LLM judges can shift comparisons so residual judge bias affects the learned human reward, while joint estimation can make reward and judge deviations difficult to distinguish. NAOD selects policy-relevant target information after nuisance adjustment; its Chatbot Arena evaluation reports lower proxy policy regret and improved held-out human-preference prediction.
1 INTRODUCTION
- Active acquisition changes the comparison distribution, potentially unbalancing judge residuals that cancel under the reference calibration distribution.Pair selection therefore affects both the information collected and which judge errors can influence the learned reward.Source: §1 · p. 1
- Joint estimation can confuse human-reward changes with judge-deviation changes, so apparent reward information may overstate precision about the human target.The paper establishes a sharp conditional local asymptotic minimax lower bound on policy risk and constructs an estimator that attains it.Source: §1 · p. 2
- NAOD selects comparisons using target information remaining after judge-deviation adjustment, weighted by how reward errors affect downstream policy decisions.The introduction notes that reward-model accuracy alone need not reflect downstream policy performance.Source: §1 · p. 2
- 29.1% lower mean proxy policy regret than the matched Target-info design was reported across 17 judges, 15 budget configurations, and 15 random cluster-level splits.The experiments also improved held-out human-preference prediction.Source: §1 · p. 2
- The paper frames biased-judge active learning as a joint target–nuisance design problem and develops practical comparison selection for finite candidate pools.Its contributions also include synthetic validation of theoretical risk predictions and Chatbot Arena evaluation with held-out human evaluation.Source: §1 · p. 2
2 RELATED WORK
Prior work studies active preference acquisition, information-based design, policy-aware acquisition, and nuisance-aware design, alongside LLM-judge bias and calibration. This paper combines policy-aware preference acquisition with nuisance-aware design for LLM-judge feedback.
2 RELATED WORK
- Prior acquisition research spans adaptive pairwise comparisons, preference-based reinforcement learning, query-efficient preference learning, and reliable LLM ranking selection.Source: §2 · p. 3
- Information-based methods use Fisher information or D-optimal design, while other methods incorporate downstream policy relevance or reward uncertainty into acquisition.Source: §2 · p. 3
- LLM-judge mitigation work includes debiasing, evaluator fine-tuning, prompt optimization, pairwise calibration, and human-data-based correction, whereas this paper focuses on how discrepancies change acquired comparisons.Source: §2 · p. 3
3 PROBLEM SETTING
The problem setting models human preferences over response pairs and evaluates learned rewards through their induced policies. Because judge calibration is defined on a reference distribution, acquisition can expose judge discrepancies that are balanced there.
3 PROBLEM SETTING
- The Bradley–Terry setup represents each ordered response comparison by a feature difference and models human preference probability from the reward difference.Source: §3 · p. 3
- Performance is measured by the loss incurred when deploying the policy induced by an estimated reward instead of the target-reward policy.The loss is expressed as a KL divergence between the induced policies, weighted by the regularization parameter and averaged over evaluation prompts.Source: §3 · p. 3
- Judge discrepancy is defined as the difference between the judge's preference probability and the target human probability, while calibration constrains its conditional mean under a reference distribution.Source: §3 · p. 4
- Acquisition can reweight judge discrepancies across reward-feature directions, producing a nonzero residual-feature moment even when that moment is zero under the reference distribution.Treating judge feedback as target feedback can then shift the population score away from zero at the target reward.Source: §3 · p. 4
4 NUISANCE-ADJUSTED OPTIMAL DESIGN
NAOD jointly models human reward and judge deviations, then optimizes policy-relevant target information after nuisance adjustment. Its local policy-risk objective has a minimax interpretation, and the finite-pool method selects distinct comparisons using a convex relaxation.
4.1 JOINT TARGET-NUISANCE MODEL
- Human and judge preferences are modeled jointly, with a fixed judge-deviation representation and nuisance parameter; the theory treats that representation as fixed and conditions on historical data used to construct it.Current judge labels are allocated across finite comparison types, alongside an independent trusted human sample.Source: §4.1 · p. 4
4.2 EFFECTIVE TARGET INFORMATION
- The effective information for the human reward is the target-information block adjusted by the Schur complement for nuisance estimation.The target–nuisance cross-information also governs sensitivity of exposed target score and information loss from nuisance estimation.Source: §4.2 · p. 5
4.3 POLICY-WEIGHTED DESIGN AND MINIMAX RISK
- NAOD minimizes a weighted design criterion combining policy curvature with inverse effective target information, rather than optimizing raw reward information.Source: §4.3 · p. 5
- Theorem 4.1 gives a conditional local policy-risk lower bound, attained uniformly on fixed local balls by the projected one-step estimator under the stated additional conditions.Its leading risk is the NAOD objective divided by sample size.Source: §4.3 · p. 6
- A matched Target-info design substitutes raw target information for effective information and can incur strictly larger leading policy risk.Source: §4.3 · p. 6
4.4 FINITE-POOL COMPARISON SELECTION
- For finite pools, the method relaxes selection of B distinct comparisons, accounts for group constraints and a feasible seed, then uses Frank–Wolfe optimization followed by rounding and exchanges.The statistical guarantee evaluates information in the realized selected set, retaining rounding effects.Source: §4.4 · pp. 6–7
5 LEARNING THE JUDGE-DEVIATION REPRESENTATION
Learning the judge-deviation representation adds a design-dependent cost to policy risk, which can overturn oracle design rankings. The section characterizes when that cost matters and how it depends on representation error.
5.1 FINITE-SAMPLE REPRESENTATION COST
- The learned representation uses m historical units, while the nuisance parameter is reestimated from current data.The setup evaluates policy risk at the human target.Source: §5.1 · p. 7
- Theorem 5.1 separates oracle sampling risk from representation-learning cost under its stated regularity and convergence conditions.The theorem considers n/m → λP and a limiting distribution for the scaled representation error.Source: §5.1 · p. 7
- Representation-learning cost adds to oracle sampling risk and depends on representation-error direction, including both covariance and bias contributions.The directional dependence is expressed through R0.Source: §5.1 · p. 7
5.2 DESIGN RANKING REVERSALS
- When ΦN < ΦT, the leading-order ranking reverses if λP(CP,N − CP,T) > ΦT − ΦN.This is the condition under which NAOD’s representation-cost disadvantage outweighs its lower oracle sampling risk.Source: §5.2 · p. 7
- When n/m → 0, representation cost vanishes and oracle ordering returns; when n/m → λP > 0, it can contribute at leading order.NAOD optimizes the fitted design criterion rather than the full two-stage risk.Source: §5.2 · p. 8
6 EXPERIMENTS
Synthetic experiments support the predicted effective-information, representation-cost, and design-reversal behavior. On Chatbot Arena, NAOD improves proxy policy regret and human-preference metrics, with gains varying by budget and judge-level target–nuisance coupling.
6.1 SYNTHETIC EXPERIMENTS
- With limited historical data, Target-info has lower scaled risk; by m/n = 2, NAOD has lower risk, confirming a design-ranking reversal.The difference is indistinguishable from zero around m/n = 1.Source: Figure 2 · p. 8, §6.1 · p. 8
- Across n ∈ {240, 960, 3840, 15360}, empirical scaled risks closely track theoretical information constants, supporting the effective-information characterization of leading policy risk.The comparison uses a scalar three-type Bernoulli construction and 6,000 independent replications per setting.Source: Figure 2 · p. 8, §6.1 · p. 8
- In the two-type construction, empirical scaled risks closely follow Φ0 + (n/m)CP; the largest finite-sample deviation occurs at the smallest representation-learning sample size.Here Φ0 = 50/49 and CP = 25/98.Source: Figure 2 · p. 8, §6.1 · p. 8
6.2 CHATBOT ARENA EVALUATION
- 29.11% lower mean proxy policy regret than matched Target-info; NAOD also achieves the lowest mean human cross-entropy and highest mean human choice accuracy.The overall results average 17 judges, 15 budget regimes, and 15 random cluster-level splits.Source: Table 1 · p. 9, §6.2 · p. 9
- NAOD reduces mean proxy policy regret versus Target-info in all 15 budget settings, but its advantage narrows as judge budget B increases at fixed human budget H.Settings with the same H/B ratio can still show substantially different gains.Source: §6.2 · p. 9
- NAOD’s acquisition gain tends to be larger for judges where Target-info induces stronger target–nuisance coupling, consistent with the theoretical prediction.The analysis reports this pattern across judges.Source: §6.2 · p. 9
7 CONCLUSION
The conclusion presents NAOD as a policy-relevant design that adjusts for judge deviations, while the technical foundations specify the conditional model and how nuisance and policy-null directions affect target information.
7 CONCLUSION
- NAOD prioritizes policy-relevant target information after nuisance adjustment, but representation error can change the preferred acquisition design.The conclusion motivates jointly learning judge-deviation representations and selecting comparisons.Source: §7 · p. 9
A.1 NOTATION, REGULARITY, AND POLICY LOSS
- The fixed-representation theory conditions on historical information and assumes bounded features, a positive-definite information limit, and a smooth policy value.For the Gibbs policy, the directed policy loss is a Bregman divergence, which supplies NAOD’s policy weighting.Source: §A.1 · p. 14
A.2 SCORES, INFORMATION, AND LOCAL ASYMPTOTIC NORMALITY
- Under the correctly specified Bernoulli model, the score covariance matches information curvature, and the conditional likelihood has a local asymptotic normal expansion.The appendix treats soft feedback separately because that equality is specific to the Bernoulli experiment.Source: §A.2 · p. 15
A.3 EFFICIENT TARGET INFORMATION AND POLICY-NULL DIRECTIONS
- The Schur complement removes target-score variation reproducible by nuisance scores, and judge feedback adds no effective target information when its target-score directions lie entirely in the nuisance-score span.Exact policy-null target coordinates can also be removed without changing policy loss.Source: §A.3 · pp. 15–16
A.4 SOFT FEEDBACK AND SANDWICH POLICY RISK
For bounded soft feedback, the paper distinguishes logistic curvature from score covariance and characterizes NAOD’s criterion as a worst-case leading sandwich policy-risk objective. It also relates acquisition-dependent residual exposure to target–nuisance cross-information.
A.4 SOFT FEEDBACK AND SANDWICH POLICY RISK
- Soft labels share logistic curvature with Bernoulli labels, but their score covariances may differ, so policy risk uses a sandwich characterization.For archived Arena soft targets, mean misspecification can still displace the joint population root from the human target.Source: §A.4 · pp. 16, 18
- NAOD minimizes the worst-case leading sandwich policy-risk coefficient over bounded, mean-correct soft feedback.Conditionally Bernoulli feedback with the same means attains the worst case; under correct Bernoulli feedback, the relevant inequalities become equalities.Source: §A.4 · pp. 17–18
B.1 ACQUISITION-DEPENDENT RESIDUAL EXPOSURE
- Acquisition reweights judge discrepancies across reward-feature directions, so policy impact depends on how the resulting residual aligns with reward-score directions.The target–nuisance cross-information also controls local sensitivity of the exposed target score.Source: §B.1 · p. 18
B.2 CONDITIONAL LOCAL MINIMAX RISK AND ATTAINMENT
The theory establishes a conditional local minimax bound attained by a projected one-step estimator, distinguishes NAOD from Target-info through a scalar construction, and gives finite-pool optimization a fitted-criterion certificate.
B.2 CONDITIONAL LOCAL MINIMAX RISK AND ATTAINMENT
- The projected one-step estimator attains the conditional local minimax lower-bound constant under the stated conditions.The analysis conditions on historical information, and its attainment assumptions include positive trusted curvature and a positive nuisance Gram matrix.Source: §B.2 · pp. 18–20
B.3 SEPARATION AND TARGET–NUISANCE COUPLING
- In the scalar two-type construction, Target-info allocates all judge labels to one type, while NAOD selects a mixture that increases effective target information.Both designs use the same correctly specified joint model and final estimator; Target-info favors the largest raw target block even when target and nuisance variation cannot be separated.Source: §B.3 · pp. 20–21
- The coupling measure identifies target-score directions nearly reproducible by nuisance variation, but policy-risk impact also depends on their weight under G0.The Arena analysis uses the measure as a confounding diagnostic, not as a finite-sample prediction of NAOD’s realized gain.Source: §B.3 · p. 21
C.1 FINITE-POOL OPTIMIZATION AND CERTIFICATES
- Finite-pool NAOD uses a convex relaxation, Frank–Wolfe optimization, feasible rounding, and improving exchanges to select a subset.The procedure freezes the selected set before observing its judge labels.Source: §C.1 · p. 22
- The final Frank–Wolfe gap and rounding-and-exchange change certify optimization under the frozen fitted criterion, not statistical uncertainty or population-information accuracy.The gap bounds suboptimality for the relaxed fitted objective, not whether fitted information equals population information.Source: §C.1 · pp. 21–22
C.2 STATISTICAL GUARANTEES FOR DISTINCT COMPARISONS
For distinct selected comparisons, the statistical guarantee is conditional on the realized set and its true information. The accompanying analysis separates plug-in error from numerical and rounding error and describes representation-learning costs.
C.2 STATISTICAL GUARANTEES FOR DISTINCT COMPARISONS
- The distinct-comparison guarantee retains the realized subset’s statistical effect and does not require its empirical proportions to approach a fixed-support allocation.It is conditional on the selected set and its true information; fitted positive definiteness does not ensure true positive definiteness or an information floor.Source: §C.2 · pp. 22–23
C.3 PLUG-IN ERROR, NUMERICAL ACCURACY, AND COMPUTATION
- Plug-in error and numerical or rounding error enter separately, and a small Frank–Wolfe certificate does not establish population accuracy or correct joint-model specification.The certificate measures optimization of the fitted objective.Source: §C.3 · p. 23
D LEARNING THE JUDGE-DEVIATION REPRESENTATION
- The representation-learning analysis covers representation-error effects, cancellation of modeled nuisance directions, finite-sample representation cost, and possible design-ranking reversal.Source: D LEARNING THE JUDGE-DEVIATION… · p. 24
D.1 REPRESENTATION ERROR AND ORACLE RECOVERY
The local population-root shift from judge-logit representation error is governed at first order by the target-influence operator. Errors in the learned nuisance span are absorbed, while small projected errors can satisfy an oracle-recovery condition.
D.1 REPRESENTATION ERROR AND ORACLE RECOVERY
- A sufficiently small deterministic judge-logit error shifts the fitted population root locally, with its target-coordinate effect given by Bm e and a quadratic remainder.The expansion assumes the trusted model remains correctly specified.Source: §D.1 · p. 24
- Errors in the learned nuisance span have no first-order target effect and, under the stated interior condition, are absorbed exactly by shifting the nuisance coefficient.Estimating that coefficient still reduces effective target information.Source: §D.1 · p. 24
- If ∥em∥ = o(n^-1/4) and √n∥Bmem∥ → 0, the target-root displacement is o(n^-1/2) and the estimator recovers oracle leading policy risk.For random representation errors, expected-risk recovery also requires moment control.Source: §D.1 · pp. 24–25
D.2 FINITE-SAMPLE REPRESENTATION COST AND DESIGN REVERSALS
Learning the judge-deviation representation adds a design-dependent term to policy risk, which can alter the oracle ranking of acquisition designs. The appendix gives higher-order error conditions and controlled experiments on these costs and reversals.
D.2 FINITE-SAMPLE REPRESENTATION COST AND DESIGN REVERSALS
- When n/m → λP > 0, design-dependent representation cost can contribute at the same leading order as oracle sampling risk; the oracle ordering is recovered when n/m → 0.This follows by comparing NAOD and Target-info under the same representation-learning protocol, budgets, and final estimator.Source: §D.2 · p. 26
D.3 HIGHER-ORDER REPRESENTATION EFFECTS
- First-order cancellation alone may not ensure oracle recovery at the n^-1/4 error scale; a second-order condition and corresponding moment control are also needed.Errors exactly in the learned nuisance span are absorbed by recentering and avoid this generic remainder condition.Source: §D.3 · p. 26
E SYNTHETIC EXPERIMENTS; E.1 EXPERIMENTAL PROTOCOL
- Controlled studies hold the outcome model and final estimator fixed across acquisition rules, isolating differences in information collected; nested-study uncertainty is computed across historical realizations.The appendix provides the synthetic constructions and diagnostics underlying the main-text results.Source: §E.1 · p. 27, E SYNTHETIC EXPERIMENTS · p. 27
E.2 CONTROLLED MECHANISM STUDIES
- With equal likelihood and estimator across methods, the Target-info gap isolates acquisition geometry, and empirical risks track the information constants across budgets.The study identifies target-score variation as valuable only when distinguishable from nuisance variation.Source: §E.2 · p. 28, Table 3 · p. 28
- With learned representation, expected scaled risk includes the predicted term Φ0 + (n/m)CP, and empirical risks approach these levels as both samples grow at fixed m/n.The largest finite-sample discrepancy occurs in the setting with nonzero clipping.Source: §E.2 · pp. 28–29, Table 4 · p. 28
- At m/n = 1/4 the NAOD-minus-Target-info scaled-risk difference is +1.588, while at m/n = 2 it is −0.446, reversing which design has lower risk.The sign change reflects representation sensitivity dominating with scarce historical data and oracle acquisition advantage dominating once representation error is reduced.Source: §E.2 · p. 29, Table 5 · p. 29
E.3 ADDITIONAL ABLATIONS AND DIAGNOSTICS
The diagnostics show that policy risk depends on policy-weighted target information after nuisance adjustment, not on raw error magnitude or nuisance rank alone. The Arena protocol freezes representations and compares acquisition methods under shared budgets and estimators.
E.3 ADDITIONAL ABLATIONS AND DIAGNOSTICS
- Equal-magnitude aligned and near-null perturbations produce sharply different policy costs; the relevant quantity is the policy-weighted target projection through the target-influence operator and G0.Figure 3(a) compares local predictions with empirical scaled policy risks.Source: §E.3 · p. 29, Figure 3 · p. 30
- NAOD remains below the Unweighted design across budgets under the same policy loss, isolating the value of policy weighting after nuisance adjustment.Effective information determines identifiable target directions, while G0 weights their downstream policy importance.Source: §E.3 · p. 30, Figure 3 · p. 30
- Underfitting creates omitted-discrepancy bias, overfitting adds variance cost, and saturation removes judge-derived target information when nuisance variation spans every judge target-score direction.The effects of nuisance dimension depend on representation adequacy and target–nuisance geometry.Source: §E.3 · p. 30, Table 6 · p. 30
- The realized-subset criterion tracks empirical policy loss in the finite-pool experiment, which selects distinct comparisons under a shared charged seed.The table reports risks in 10^-3 units across 24 pool-level means.Source: §E.3 · p. 30, Table 7 · p. 31
F.1 DATA, REPRESENTATIONS, AND EXPERIMENTAL PROTOCOL
- Arena representations are learned upstream and frozen before acquisition; connected text clusters define split ownership and acquisition capacity.At most one eligible row per connected cluster can be acquired.Source: §F.1 · pp. 31–32, Table 8 · p. 31
F.2 METRICS, BASELINES, AND STATISTICAL ANALYSIS
- The primary endpoint is proxy policy regret, with held-out human cross-entropy and tie-aware choice accuracy also reported; paired comparisons are aggregated across clusters, judges, and budgets.The 15 repeated cluster-level splits are the inference unit, and positive paired gains favor NAOD.Source: §F.2 · pp. 32–33
F.3 COMPLETE RESULTS AND DIAGNOSTICS
Across Arena budgets, NAOD favors Target-info, and its judge-level policy gains are associated with stronger target–nuisance coupling. NAOD also reduces selected-set coupling, while the reported representation and optimization diagnostics show positive nuisance-fit gains and small optimization gaps.
F.3 COMPLETE RESULTS AND DIAGNOSTICS
- 15/15 budget cells favored NAOD over Target-info, with 12 paired intervals entirely above zero.Table 11 reports paired proxy-regret gains across the budget grid.Source: Table 11 · p. 34, §F.3 · p. 34
- At H/B = 1/8, relative reductions were 50.1%, 36.7%, and 15.3%, while selection overlap rose from 0.446 to 0.505 to 0.568.Absolute information scale and finite-pool geometry matter in addition to the human-to-judge budget ratio.Source: §F.3 · p. 34
- 15/17 judges had positive mean policy gain; the largest gains occurred for Meta-Llama-3-8B and two dolphin judges with coupling above 0.95, while Meta-Llama-3-70B-Instruct had essentially no gain despite strong residual predictability.The judge-level coupling–gain Spearman correlation is 0.576.Source: §F.3 · pp. 34–35, Figure 5 · p. 34
- Mean squared coupling fell from 0.700 under Target-info to 0.290 under NAOD, with mean selected-set overlap of 0.521.The comparison indicates that NAOD changes which comparisons are acquired rather than applying a different estimator to essentially the same set.Source: §F.3 · p. 35
- All 255 split-by-judge nuisance fits had positive out-of-fold soft-label cross-entropy gain, averaging 0.0370.The learned nuisance representation captured systematic judge residual structure across the evaluation.Source: §F.3 · p. 35
- No listed acquisition or design failures appeared in 22,950 arrays; maximum Frank–Wolfe gap was 8.62 × 10−8 and maximum integer certificate was 6.01 × 10−7.The audit concludes that unresolved selection optimization does not explain the observed acquisition differences.Source: §F.3 · p. 35