PaperlayerView exact revision on arXiv

Source-linked AI summary

Whether or Where? Tail Calibration Signals Finite-Budget Reward Hacking in Best-of-N Selection

Cheng Gui, Yihua Shao, Lei Li, Jingcai Guo, Maolin Wang

arXiv:2610.34375v1cs.LGcs.AIcs.CL

TL;DR

Best-of-N can raise proxy scores while true correctness falls, because large selection budgets probe reward-model tails that bulk metrics may miss. The paper develops a rank-based tail audit and cross-fitted budget cap, finding localized failures in code generation and held-out cap gains competitive with rank-Soft-BoN. It presents tail calibration as a complement to rank accuracy, not a replacement.

  • Problem

    Best-of-N can keep raising proxy scores while correctness falls, and bulk reward-model metrics may miss defects in the extreme tail that selection probes.

    Source: §1 · p. 1

  • Method

    The paper uses a monotone-invariant tail-excess statistic and cross-fitted reward-response curve to estimate and validate a task- and model-specific budget cap.

    Source: §1 · p. 2

  • Results

    Negative tail excess concentrates in code generation, and cross-fitted budget caps yield held-out gains competitive with rank-Soft-BoN.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Tail calibration complements rank accuracy with a monotone-invariant audit connecting reward-model evaluation to search budgets.

    Source: §1 · p. 2, §5 · p. 9

    Policy-stratified results show risk depends on the policy–reward-model pairing and do not establish a causal shortcut.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Best-of-N (BoN) inference returns the candidate with the highest proxy reward, so a larger $N$ tests the proxy's extreme tail more than its average ranking quality. We show that the exact BoN value is a Beta-weighted integral of conditional true reward over within-prompt score quantiles. It follows that BoN depends on scores only through their ranks. It also follows that BoN can decline only if the calibration curve in the tail turns downward, and the budget where the decline appears depends on where that turn lies. Under a bounded additive model, the error tail sets the limit: Gaussian errors keep improving, Laplace errors reach a plateau, and heavy-tailed errors return to the prior mean. When error variance is largest at low true reward, even Gaussian noise can push BoN below random selection. Synthetic phase maps agree with this taxonomy. On real candidate pools spanning several verifiable domains, base policies, and learned reward models, tail failures are rare and all occur in code generation. A held-out tail calibration statistic reliably predicts where a cross-fitted budget cap helps. It does about as well as rank accuracy, and the cap is competitive with a Soft-BoN baseline. Tail calibration therefore does not replace ranking metrics. It is an audit that is unchanged by any monotone rescaling of the reward scores and shows which budgets are at risk.

1 INTRODUCTION

The paper frames Best-of-N reward hacking as a tail-calibration problem: larger budgets can raise proxy scores while true correctness falls. It proposes a rank-based tail audit and cross-fitted budget cap, validated on synthetic and public candidate pools.

Motivation

  • BoN can keep increasing proxy scores while verifiable correctness turns over and falls, creating reward hacking through selection budget alone.The practical questions are whether a reward model is exposed and, if so, at what N.Source: §1 · p. 1
  • BoN concentrates selection on vanishingly small top quantiles, so defects there can escape bulk metrics while determining deployment outcomes.At N = 64, half the selection mass lies above the 99th percentile.Source: §1 · p. 1

External validation

  • Tail failures are sharply localized rather than ambient, and a top-percentile reading yields budget caps that still pay off on held-out data.Validation spans synthetic settings, a controlled input-perturbation study, and public correctness data across domains and learned reward models.Source: §1 · p. 2

Contribution

  • The paper formalizes risk as a tail class determining whether a finite optimal budget exists and a correlation scale determining where it appears.BoN depends on the upper end of the reward-response curve: whether it peaks depends on a downward turn, and the budget scale on how soon it occurs.Source: §1 · p. 2
  • The proposed audit uses a monotone-invariant tail-excess statistic and cross-fitted reward-response curve to estimate a task- and model-specific budget cap.One data split estimates the statistic and cap; the other checks the cap's value.Source: §1 · p. 2

2 RELATED WORK

The paper situates its contribution among work on reward-model overoptimization, tail-aware selection, and reward-model evaluation. It distinguishes its audit by measuring the joint relation between true utility and score rank within a fixed candidate pool.

Reward-model overoptimization

  • Prior work studies proxy overoptimization, BoN distribution shift and control, and reward-model ensembles; this paper narrows focus to rank–utility calibration within a fixed candidate pool.The cited lines of work address proxy optimization [Gao et al. (2023)], BoN bounds and controls [Beirami et al. (2024); Huang et al. (2025)], and ensembles [Coste et al. (2023a)].Source: §2 · p. 2

Inference-time reward hacking and tail-aware selection

  • The paper builds on the probability-integral-transform and Beta order-statistic scaffold, but contributes a score-transform-invariant joint-tail audit and held-out budget rule.Best-of-Tails and ITP use score-tail shape or pessimistic selection, whereas this audit requires labeled outcomes.Source: §2 · pp. 2–3

Reward-model evaluation

  • RewardBench, RM-Bench, and RMB assess reward-model capabilities, while PPE supplies public fixed candidate pools reused for external validation.The paper's masked and shuffled controls perturb outcome-level reward-model inputs; it does not evaluate intermediate-step PRMs.Source: §2 · p. 3

3 PROBLEM SETUP AND TAIL REPRESENTATION

The section represents Best-of-N through a reward-response curve over within-prompt score quantiles, then relates that curve's shape and tail to selection value. It distinguishes tail regimes, heteroskedastic departures, tail location, and budget guidance.

3 PROBLEM SETUP AND TAIL REPRESENTATION

  • The setup separates whether BoN value can turn down as N grows, where it turns, and how fast.Responses have true utilities and proxy scores, with binary utilities in verifiable experiments and real-valued utilities in the synthetic study.Source: §3 · p. 3, §3.1 · p. 3

3.1 SCORE SPACE, QUANTILE SPACE, AND TRUE UTILITY

  • BoN value is determined by the reward-response curve Q over score quantiles, so a candidate pool gives values across budgets and is invariant to strictly increasing score recalibration.The selected quantile has a Beta order-statistic weighting; empirically Q is estimated from within-prompt ranks and averaged over prompts.Source: §3.1 · p. 3

3.2 CALIBRATION SHAPE CONSTRAINS BON

  • BoN cannot have more local extrema than Q; a single-peaked Q yields a single-peaked value curve, with a finite peak when Q decreases in the upper tail.For single-peaked Q, stopping at the first budget with nonpositive marginal value is globally optimal; the finite-pool shape constraint also holds.Source: §3.2 · p. 4

3.3 ADDITIVE NOISE AND THE PRIOR TILT

  • Under independent additive errors and bounded rewards, Gaussian errors drive Q toward the maximum reward, Laplace errors to a plateau, and heavy-tailed errors back to the prior mean.Log-concave additive errors make G, Q, and BoN value nondecreasing for every prior; heavy-tailed errors can permit a finite optimal budget after a finite budget beats random.Source: §3.3 · p. 4

3.4 HETEROSKEDASTIC NOISE: SCALE BEATS SHIFT

  • With heteroskedastic Gaussian noise, BoN's limiting value lies below random selection when the reward level of maximum noise scale is below the prior mean.The scale enters the log-likelihood at order s^2 while reward enters at order s; multiplicative and conditional-variance tests probe such departures.Source: §3.4 · p. 5

3.5 TAIL LOCATION SETS THE RELEVANT BUDGET

  • A budget N reads Q mainly on the top 1/N band, so a defect confined to the top δ becomes visible only around N ∼ 1/δ.A heavy-tailed contaminating component can preserve the observed curve yet force an eventual return to the prior; a finite pool cannot certify monotonicity beyond its size.Source: §3.5 · p. 5

3.6 PRESCRIPTIONS

  • The prescription is to identify the tail regime from low-reward score log-survival, stop at the first nonpositive estimated marginal value for single-peaked Q, and never extrapolate beyond pool size.In plateau and heavy-tailed regimes, cap N near empirical saturation or the peak; pursue Gaussian √log N gains only when samples are cheap.Source: §3.6 · p. 5, §3.3 · p. 5

4 EXPERIMENTS

Experiments test tail-regime predictions in synthetic pools, controlled MATH-500 and GSM8K candidate pools, and public PPE pools, then evaluate cross-fitted budget caps.

4.1 EXPERIMENTAL SETUP

  • The experiments used 20,000 synthetic candidate sets per cell, controlled pools of 256 generated answers per problem, and PPE pools with 32 responses per prompt.Source: §4.1 · p. 6

4.2 SYNTHETIC TESTS OF TAIL REGIME AND BUDGET SCALE

  • Gaussian cells were monotone over N ≤ 512 (and at every budget); Laplace never produced downturns, while t2.5, t3, t4, and t6 had supported downturns at all six correlations; t30 was monotone.Across 54 primary cells, 24 were downturns, 20 monotone, and 10 plateau/boundary; increasing correlation shifted and attenuated heavy-tailed downturns without changing their regime.Source: §4.2 · p. 6
  • In the scale stress test, four decreasing-scale cells had supported downturns; the largest drop, 1.939, put BoN below random selection despite rank accuracy of 0.828 in a failing cell.Source: §4.2 · p. 6

4.3 THE CONTROLLED MATH FAILURE AND FORMAT CONTROLS

  • Masked-question and shuffled-question controls had unsupported drops, so the answer-only failure cannot be attributed to missing question semantics alone; chat role/template change remains a confound.Source: §4.3 · p. 7
  • Tulu on MATH-500 answer-only peaked at N = 4 and dropped 0.0597 in correctness by N = 256, while QA had no supported downturn and top-1% excess changed from +.054 to −.031.The reported simultaneous 95% interval for the drop was [0.0316, 0.0878]; Table 2 reports the input-format controls.Source: §4.3 · p. 6, Table 2 · p. 8

4.5 HELD-OUT BUDGET CAPS AND DIAGNOSTIC COMPARISONS

  • Nine of 130 learned-cell folds (0.069 prevalence) had positive held-out cap gains; tail risk predicted them with ROC AUC 0.983 [0.954, 0.998] and AP 0.825 [0.449, 0.973].Source: Figure 3 · p. 8, §4.5 · p. 9

4.4 EXTERNAL VALIDATION: FAILURES CONCENTRATE IN CODE

  • All six negative top-12.5% tail-excess cells were on MBPP+; Eurus-RM-7B was the sole strict interior downturn, while three other negative-tail models were already harmful for N > 1.Those three were Starling-7B (drop 0.165), InternLM2-7B (0.171), and Nemotron-340B (0.076).Source: Figure 3 · p. 8, §4.4 · p. 8

4.5 HELD-OUT BUDGET CAPS AND DIAGNOSTIC COMPARISONS

  • Against BoN-32, cap and rank-Soft-BoN gained 0.0085 [0.0019, 0.0168] and 0.0068 [−0.0003, 0.0156] across 65 cells; on six negative-tail cells, gains were 0.0924 and 0.0916.The hard rule was competitive, not materially superior.Source: §4.5 · p. 9

5 CONCLUSION

Tail calibration connects best-of-N budgets to true utility: the curve’s shape determines whether selection can decline and where that decline appears. The paper concludes that held-out, task-specific budget choices are supported when labeled candidate pools and independent validation are available.

5 CONCLUSION

  • Tail calibration complements rank accuracy by linking reward-model evaluation to search budgets and supporting task–policy–reward-model-specific choices over the tested range.Cross-fitted budget caps yield held-out gains competitive with rank-Soft-BoN.Source: §5 · p. 9

AI USE STATEMENT

  • The authors report using generative AI for literature synthesis, drafting and editing, code organization, figure preparation, and LaTeX formatting, with numerical claims checked against analysis artifacts.They state that authors remain responsible for the final content and artifacts.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The ethics statement says the work uses no personal data and does not target deployed systems; it frames tail diagnostics as pre-deployment audits requiring independent ground-truth verification.The stated concern is that the diagnostic could help optimize against a weak reward model.Source: ETHICS STATEMENT · p. 10

A PROOFS FOR SECTION 3

  • The finite-pool proof uses order-statistic and total-positivity arguments, alongside a posterior-tilting account of what extreme scores reveal about reward.The latter describes the posterior at a large score as a tilt of the prior, with rate set by the error hazard slope or class-conditional log-likelihood growth.Source: A PROOFS FOR SECTION 3 · p. 12

Appendix A.9 combines the two to separate whether a failure occurs from where it becomes visible.

  • The appendix conventions define sign blocks for signed measures, specify total positivity through nonnegative minors, and record the Beta order-statistic facts used in the analysis.The accompanying bounds give the mode, mean, and standard deviation of Beta(N + 1, 2).Source: Appendix A.9 · p. 12

A.1 PROOF OF PROPOSITION 3.1

  • A BoN winner reads the calibration curve at a Beta(N, 1) quantile, so increasing N moves the reading point toward 1 and score rankings alone determine selection.Strictly increasing score transformations leave the ranking, and therefore the result, unchanged.Source: §A.1 · p. 12
  • The marginal value of another sample depends on calibration increments near the current record, with the kernel u^N(1 − u) peaking at u = N/(N + 1).For differentiable Q, the budget increment averages Q′ over a window of width about 1/N near 1 − 1/N.Source: §A.1 · p. 13
  • For bounded-variation calibration curves, the BoN value tends to the curve’s top-end value as N grows; the marginal-sample kernel concentrates near u = 1.The integral representation also makes the remaining gap to that limit explicit.Source: §A.1 · p. 13

A.2 FINITE POOLS: PROOF OF PROPOSITION A.1

For finite candidate pools, every turn in the BoN value curve is witnessed by a turn in the ordered reward profile. The proof connects this result to a total-positive transform of rank-profile increments.

A.2 FINITE POOLS: PROOF OF PROPOSITION A.1

  • The finite-pool setup orders responses by score, averages tied blocks, and selects the highest-ranked response among N draws without replacement.The construction also applies to prompt averages when pools share a common size.Source: §A.2 · pp. 13–14
  • The finite-pool kernel gives the probability that the current best draw has rank at most j while the next draw ranks above j, paralleling the continuous crossing kernel.The passage states that this finite-pool expression converges to the continuous kernel when j/K approaches u with N fixed.Source: §A.2 · p. 14
  • The proof establishes total positivity by reducing the kernel to the Pascal matrix, whose minors count non-intersecting lattice-path families.The variation-diminishing property then supplies the sign-change result.Source: §A.2 · p. 14
  • Every turn in the finite-pool BoN curve is witnessed by a turn in the rank profile, because the curve is a total-positive transform of the profile’s increments.Noise may add sign changes to the profile, but cannot hide the witness of a turn.Source: §A.2 · pp. 13–14

A.3 PROOF OF THEOREM 3.2

This section connects BoN’s increments, stopping behavior, and limiting rewards to calibration-curve sign changes and error-tail behavior. It also characterizes how quickly best-of-N reward approaches its limit under different error tails.

A.3 PROOF OF THEOREM 3.2

  • BoN’s increment sign changes are bounded by calibration-curve sign changes, and its eventual increment sign matches the top calibration block.For piecewise-monotone profiles, calibration turning points correspond to local extrema of BoN value.Source: §A.3 · pp. 14–15
  • For a single-peaked calibration curve, once another sample stops helping, it never helps again; heterogeneous prompt peaks can make the aggregate curve multi-peaked.The first nonpositive increment gives a global maximizing budget for a single-peaked profile; differing per-prompt peak quantiles can create multiple turns in the average.Source: §A.3 · pp. 15–16

A.4 PROOF OF COROLLARY A.3, AND THE SPEED OF RETURN

  • If the bounded calibration curve returns to the prompt mean at its top, any BoN value above that mean guarantees a maximizer at finite N.As N grows, BoN value converges to the prompt mean; its excess is governed by the calibration profile near the top quantiles.Source: §A.4 · p. 16

A.5 ADDITIVE NOISE: THE TILT REPRESENTATION AND PROPOSITION A.5

  • In additive-noise models, log-concave error densities are exactly the condition ensuring calibration is nondecreasing for every prior; extreme-score posteriors are tilted by the error’s local hazard slope.For Gaussian error, G′(s) = Var(R | S = s)/σ2; non-log-concave errors permit a binary-reward prior with a decreasing calibration curve.Source: §A.5 · pp. 17–18

A.6 PROOF OF THEOREM 3.3

  • As extreme scores grow, the posterior reward tends to the best reward when the hazard slope diverges, a tilted-prior mean when it converges positively, and the prior mean when it vanishes.The limiting tilt is proportional to e^(λ⋆r) for finite positive λ⋆; the appendix illustrates hazard-slope limits across several error-tail families.Source: §A.6 · pp. 18–19

A.7 RATES OF APPROACH: PROPOSITION A.7

  • Heavy-tailed errors return toward the prior at a polynomial rate, exponential tails yield a plateau, and Gaussian tails approach the top reward more slowly than any power of N.The appendix summarizes the BoN implications as wasted samples beyond a geometric horizon at a plateau, super-polynomial cost under Gaussian tails, and harm beyond the peak under heavy tails.Source: §A.7 · p. 19
  • For Student-t3 noise, VN − µ is asymptotic to (ν + 1)Γ(1 + 1/ν) Var(R) N^-1/ν L(1/N).For R ∈ {0, 1} with µ = 1/2, the ratios of VN − µ to the asymptotic expression are 0.81, 0.95, 0.99, and 0.998 at N = 10^2, 10^3, 10^4, and 10^5, respectively; the rate follows because the winner's score grows as N^1/ν and its local tilt is (ν + 1)/sN.Source: §A.7 · p. 20
  • With exponential error tails, G(s) equals m(λ⋆) above t0 + rmax, so VN approaches this tilted value rather than necessarily reaching rmax.The passage also bounds the budget horizon by solving (rmax − rmin)FS(t0 + rmax)^N ≤ δ.Source: §A.7 · p. 20
  • For binary rewards with Gaussian error, −log(1 − VN) = (1 + o(1))√(2 log N)/σ, a slow rate because the winning score grows only as σ√(2 log N).At the winning score, the tilt is aN/σ², so raising the top score by one unit requires multiplying N by roughly e^(aN/σ²).Source: §A.7 · p. 20
  • For continuous rewards with density proportional to (rmax − r)^β near rmax, Gaussian error gives rmax − VN ∼ (β + 1)σ/√(2 log N).The posterior reward gap at score s is asymptotic to (β + 1)σ²/s.Source: §A.7 · p. 21

A.8 PROOF OF PROPOSITION 3.4 AND OF PROPOSITION A.8

Extreme-score behavior, not bulk ranking, determines the posterior reward and the eventual BoN limit. In heteroskedastic Gaussian settings, the largest noise scale can outweigh reward and make BoN converge to the reward level with that scale.

A.8 PROOF OF PROPOSITION 3.4 AND OF PROPOSITION A.8

  • For binary rewards, the tail likelihood ratio sets the limiting calibration value: 1, μ, or 0 when Λ(s) tends to infinity, 1, or 0, respectively.Any other finite positive limit gives a plateau in (0, 1).Source: §A.8 · p. 21
  • With heteroskedastic Gaussian noise, the posterior concentrates on the reward level with maximum noise scale, so BoN converges to that level.If the unique maximum-scale reward is the minimum reward, BoN converges to the worst reward level.Source: §A.8 · pp. 21–22
  • With homoskedastic noise, the Gaussian score’s reward-dependent term favors the highest reward; with heteroskedastic noise, the scale term decides.The Gaussian log-likelihood ratio for unequal class variances has a leading quadratic term whose sign depends on which class has the larger variance.Source: §A.8 · pp. 21–22
  • When incorrect scores have the heavier tail, or the wider scale at equal tail shape, BoN converges to the worst outcome; location decides only when shape and scale agree.The extreme score is explained first by the heaviest tail, then by the largest scale, and only then by the largest reward.Source: §A.8 · p. 22

A.9 PROOFS FOR SECTION 3.5

A heavy-tailed component can control extreme scores even when it is rare, making BoN return to the prior mean after an arbitrarily long delay. Tail features also determine when declines appear, while a finite observed budget range cannot rule out a later return.

A.9 PROOFS FOR SECTION 3.5

  • Any heavy-tailed contamination component, however small, eventually makes the calibration curve return to the prior mean; a smaller contamination rate only delays the turn.At small budgets, rare mis-scoring is invisible regardless of the contaminant’s form, while its tail determines the eventual behavior.Source: §A.9 · pp. 22–23
  • No BoN curve observed through budget K rules out a later return; diagnostics therefore target λ⋆ and Λ directly.The two curves can remain within δ through K yet eventually differ by rmax − μ; the guaranteed invisibility window scales linearly with 1/π.Source: §A.9 · pp. 22–23
  • A tail feature of height c on the top-t region is nearly invisible when Nt ≪ 1 and fully expressed when Nt ≫ 1, with the switch near N ≈ 1/t.The localization heuristic places a feature at quantile u⋆ near budget Nloc = u⋆/(1 − u⋆).Source: §A.9 · p. 22
  • For Student-t errors with ν = 1, the optimal budgets are 5, 11, and 22–24 at σ = 1, 3, and 3, respectively.The tail index places the peak, while noise scale mainly sets its height; as ν → ∞, the peak recedes toward the monotone Gaussian case.Source: §A.9 · p. 23

B DETAILED EXPERIMENTAL PROTOCOL

The protocol combines synthetic tail-regime tests, controlled fixed-candidate studies, and cross-fitted PPE audits to examine how prior, error structure, and budget affect BoN. Its results show that correlation can shift when heavy-tail failures appear, while prior tails and heteroskedastic noise change which regimes are possible.

B DETAILED EXPERIMENTAL PROTOCOL

  • The study tests 162 synthetic cells across priors, correlations, and error families, plus 12 heteroskedastic cells, using 20,000 candidate sets per cell.Student-t errors are variance standardized; the heteroskedastic cells use conditionally Gaussian errors with scale exp(βR/3).Source: B DETAILED EXPERIMENTAL PROTOCOL · pp. 23–24
  • Controlled pools use 256 Qwen2.5-7B-Instruct answers per MATH-500 and GSM8K problem, scored by four reward models under QA and answer-only formats.MATH controls mask or shuffle questions while preserving the user–assistant template; answer-only also changes role and information.Source: B DETAILED EXPERIMENTAL PROTOCOL · pp. 23–24
  • PPE evaluation covers 32 responses per prompt across five tasks, four base policies, and 13 learned reward models, with 31 nonnumeric Nemotron rows excluded.The protocol records source hashes, exclusions, and policy strata.Source: B DETAILED EXPERIMENTAL PROTOCOL · pp. 24–25
  • Exact finite-pool rank profiles estimate BoN expectations; PPE uses top-12.5% excess to trigger a calibration-selected cap, evaluated on the opposite policy-stratified fold.The cap is selected from budgets {1, 2, 4, 8, 12, 16, 24, 32}; otherwise the policy retains N = 32, with tests BH-corrected across 70 cells.Source: B DETAILED EXPERIMENTAL PROTOCOL · p. 24

B.1 CORRELATION AND FINITE-BUDGET LOCALIZATION

  • For Uniform t2.5 errors, the detected peak shifts from N = 12 at ρ = .5 to N = 32 at ρ = .95, while the drop contracts from 0.495 [0.471, 0.520] to 0.296 [0.282, 0.310].For t6 at ρ = .95, the peak moves to N = 128 and the drop falls to 0.0177 [0.0118, 0.0235]; smaller noise scale delays the heavy-tail region without changing the eventual return.Source: §B.1 · p. 25

B.2 HETEROSKEDASTIC SCALE RESULTS

  • With Gaussian error variance largest at low reward, four of six β = −1 cells have supported downturns, with drops from 1.939 to 0.600 at ρ = .5–.8.The largest drop leaves BoN below random selection; at ρ = .8, rank accuracy remains 0.828.Source: §B.2 · p. 25

B.3 PRIOR AND MULTIPLICATIVE STRESS TESTS

  • Binary-prior tests yield 28 downturn, 19 plateau, and 7 monotone cells, while the unbounded Gaussian prior yields 17/6/31 across those outcomes.The unbounded prior lies outside Theorem 3.3 because its tail competes with the error tail; beyond bounded rewards, phase depends jointly on prior, error, and budget.Source: B DETAILED EXPERIMENTAL PROTOCOL · p. 23, §B.3 · p. 25
  • The multiplicative control is monotone for Uniform and Gaussian priors and plateaus for binary rewards; scale growth with |R| alone is insufficient to predict the upper tail.The example is at the boundary of the uniqueness condition, and the first-order shift favors the positive reward edge.Source: §B.3 · p. 26

C.1 RELATION TO BULK RANKING METRICS

Bulk ranking metrics and tail-risk measures are closely associated on the evaluated PPE cells, but neither determines the other in general.

C.1 RELATION TO BULK RANKING METRICS

  • Tail-risk AUC/AP is 0.972/0.749, while rank-accuracy AUC/AP is 0.972/0.747; paired intervals include zero, and bad-tail PPE cells also have low rank accuracy.The paper's non-identifiability result permits curves alike below the extreme tail to have different tail regimes, even though bulk ranking quality predicts the problem in these observed cells.Source: §C.1 · p. 26

D COMPLETE CONTROLLED RESULTS

The controlled study compares QA, answer-only, and structure-aware question controls to localize where Tulu's calibration curve turns. The answer-only condition places the estimated turn much farther into the score distribution than the other conditions.

D COMPLETE CONTROLLED RESULTS

  • The masked-question and shuffled-question controls retain the user–assistant template, distinguishing missing question content from deliberately mismatched semantics.These controls address the confound that answer-only changes both information and role/template.Source: D COMPLETE CONTROLLED RESULTS · p. 26

D.1 BUDGET LOCALIZATION

  • Tulu’s exact tied-rank N* is 68 for QA, 3 for answer-only, 82 when masked, and 18 when shuffled, locating the answer-only turn near the 75th percentile rather than the extreme top.Top-1% excesses in the same order are +.0538, −.0310, +.0495, and +.0231; answer-only visual estimates remain 3–4 across tested grids and bandwidths.Source: §D.1 · p. 26

E PPE REWARD MODELS AND ADDITIONAL RESULTS

The PPE results identify policy–reward-model interactions, descriptive features of code-generation errors, and checks on the audit pipeline. The supplied passages also specify the scope of the controlled-condition table but do not provide its cell values.

E PPE REWARD MODELS AND ADDITIONAL RESULTS

  • Eurus-RM’s MBPP+ tail excess is +.130 for Meta-Llama-3-8B responses but −.127 for Claude-3-Haiku, with peak-to-32 drops .011 and .178, respectively.The other five negative cells also reverse sign across policies, so the aggregate tail depends on policy mix and does not support a task-independent safe budget.Source: E PPE REWARD MODELS AND ADDITIONAL… · p. 26
  • Top-tail MBPP+ errors are about 13% longer and add three lines; across six reward models they more often parse, define the target, include a docstring, and place prose outside the fence.Repeated examples or assertions are 13.3 points less common; these shared-pool correlates are descriptive, not causal.Source: E PPE REWARD MODELS AND ADDITIONAL… · p. 27
  • NaiveVerbosityModel has negative tail excess on all five PPE tasks and supported BH-corrected cap gains on MATH, MMLU-Pro, and MBPP+.It is retained as an explicit stress baseline, not as evidence of learned-reward-model prevalence.Source: E PPE REWARD MODELS AND ADDITIONAL… · p. 27
  • Among learned cells, any tied score occurs in a median 73.2% of prompts, but top-four-boundary ties occur in 7.0%; all six negative cells stay negative across deterministic and 200 randomized tie orders.Table 4 is scoped to original controlled conditions and defines drop relative to the selected interior peak and N = 256, but its cell values are not supplied here.Source: Table 4 · p. 27, E PPE REWARD MODELS AND ADDITIONAL… · p. 28

F DESCRIPTIVE EXTREME-ANSWER FEATURES

The section describes which answer forms occupy extreme reward-score tails in MATH and MBPP+, while treating MATH feature patterns as associations rather than causal reward-model shortcuts.

F DESCRIPTIVE EXTREME-ANSWER FEATURES

  • In failing answer-only Tulu MATH top-1%, extreme answers had an equation-marker contrast of −0.88 [−1.56, −0.23] and fewer boxed or extractable final answers.Table 7 reports the top-1% comparison; the analysis also examines top 5% and 10% score ranks.Source: F DESCRIPTIVE EXTREME-ANSWER FEATURES · p. 28, Table 7 · p. 28
  • Masking reverses correctness and final-answer-marker contrasts, and the failing pattern is not a simple verbosity effect.The reported feature associations describe which response forms occupy the extreme tail; they do not identify a causal reward-model shortcut.Source: F DESCRIPTIVE EXTREME-ANSWER FEATURES · p. 28
  • For MBPP+, responses are extracted from the longest Python fence or full unfenced text and parsed with Python’s AST; Table 8 contrasts incorrect top-tail responses across six negative-RM cells.Its confidence intervals do not treat the six repeated views of the shared response pool as independent.Source: F DESCRIPTIVE EXTREME-ANSWER FEATURES · p. 28, Table 8 · p. 28
Loading 2610.34375v1…