PaperlayerView exact revision on arXiv

Source-linked AI summary

How to post-train on a surrogate: Envelope sampling mitigates reward hacking

Sanjit Dandapanthula, Shuvom Sadhuka, Samir Khan, Michael Oberst, Aaditya Ramdas, Alexandra Chouldechova

arXiv:2610.11281v1cs.LGcs.AIstat.ME

TL;DR

When true rewards are expensive and a surrogate may be miscalibrated on rare outputs, on-policy labels can miss the disagreement. The paper derives an envelope query distribution for recalibration, and finds that it mitigates reward hacking where base-model recalibration does not.

  • Problem

    True rewards can be expensive to evaluate, and on-policy recalibration may miss rare outputs where a surrogate disagrees with the ground truth.

    Source: §1 · p. 1

  • Method

    Envelope sampling chooses ground-truth label queries to minimize a regret upper bound under an assumption that the true and recalibrated rewards are close to the judge.

    Source: §1 · p. 2

  • Results

    Envelope recalibration mitigated reward hacking on clinical note generation and sycophancy, while on-policy recalibration left it in place; in a stylized model, it attained optimal label complexity.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The envelope provides a query distribution for recalibrating a surrogate before optimization, with rejection-sampling and bonus-based sampling algorithms.

    Source: §1 · p. 2, §7 · p. 9

    The theory idealizes post-training as an exact exponential tilt, the case study uses a stylized output space, and clinical experiments use an LLM rather than physician review.

    Source: §2.1 · p. 2, §7 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are commonly post-trained against LLM judges and other cheap surrogates because the true reward, such as human preference, is too expensive to query at scale. This practice often leads to reward hacking, where reinforcement learning against a miscalibrated surrogate leads to undesirable side effects. In this work, we study a setting in which a small number $n$ of model outputs are annotated with ground-truth labels (e.g., from expert review) and used to recalibrate the LLM judge before optimizing against it. Prior approaches to judge recalibration are costly or heuristic, and it is known that on-policy sampling fails when the surrogate is miscalibrated on a rare set of outputs. In this work, we propose envelope sampling, a theoretically-grounded method for judge recalibration that seeks to minimize an upper bound on the regret of the post-trained model under the assumption that the human reward and re-calibrated reward lie in an $L^2$ ball around the judge. We give practical algorithms to sample from the envelope by rejection or by fine-tuning against a modified reward, and experiments on clinical note generation and on a controlled sycophancy task show that recalibrating on envelope samples mitigates reward hacking where recalibrating on base-model samples does not.

1 Introduction

The introduction asks how to choose ground-truth labeling queries so a surrogate can be safely optimized despite rare miscalibration. It proposes envelope sampling, which targets regret control and is evaluated theoretically and on two tasks.

1 Introduction

  • Because on-policy queries rarely include outputs where the surrogate is wrong, recalibration may preserve errors on rare outputs and leave the post-trained model vulnerable to reward hacking.The introduction frames the central question as which query distribution should supply ground-truth labels.Source: §1 · pp. 1–2
  • The regret bound depends on recalibration error under the query distribution and a coverage factor for models reachable by recalibration and optimization.Source: §1 · p. 2
  • Assuming the ground truth and recalibrated judge lie close to the current judge, the envelope minimizes the worst-case coverage factor.Source: §1 · p. 2
  • The envelope can be sampled by rejection from the judge-optimized model or by optimizing against an alternative bonus function.Source: §1 · p. 2
  • In a stylized model, envelope sampling attains the optimal label complexity among all query distributions.Source: §1 · p. 2
  • Envelope recalibration mitigates reward hacking on clinical note generation and controlled sycophancy, while on-policy recalibration leaves it in place.The overview contrasts labels on envelope samples with labels on typical base-model outputs.Source: §1 · p. 2, Figure 1 · p. 2

2 Setup and notation

The setup models post-training as an exponential tilt of a base model and treats query-distribution choice as the design problem. A finite-budget protocol collects labels, recalibrates a surrogate, then optimizes against the recalibrated reward.

2.1 Exponential tilts and KL-regularized optimization

  • Post-training against reward f is modeled as an exponential tilt, πf proportional to πe^βf, with normalization to a probability distribution.The text relates this tilt to the population objective of PPO-based RLHF and DPO under the Bradley-Terry preference model.Source: §2.1 · p. 2

2.2 Setting and protocol

  • The protocol draws n labels from query distribution ρ, recalibrates surrogate j into reward r, then optimizes π against r.The resulting model is πr proportional to πe^βr at optimization pressure β > 0.Source: §2.2 · p. 3
  • The design goal is to choose ρ to improve the final model; the guarantee depends on recalibration error Varρ(r − h).The setting has an expensive true objective h and cheap surrogate j, and does not prescribe a particular recalibration method.Source: §2.2 · p. 3

3 Related work

Related work situates envelope sampling among research on reward hacking, post-training, label efficiency, universal prediction, and distribution coverage. The paper distinguishes its minimax, reward-class-based query law from uncertainty-driven sampling and relates its coverage factor to density-ratio coefficients.

Reward hacking and overoptimization

  • Prior work studies proxy overoptimization and reward hacking, including harms concentrated in the high-reward tail; this paper instead studies ground-truth labels used to repair the proxy before optimization.Source: §3 · p. 3

Label-efficient reward modeling

  • Unlike active-learning methods driven by model uncertainty, envelope sampling derives its query law from minimax analysis over an explicit reward class.The paper also uses rubric refinement in some experiments to recalibrate judges.Source: §3 · p. 3

Universal prediction and the Shtarkov distribution

  • The normalized envelope is a normalized maximum likelihood measure, used here as a sampling target rather than a coding-theoretic device.Source: §3 · p. 3

Coverage and distribution shift in offline RL

  • The coverage factor plays a role analogous to density-ratio coefficients, but concerns the path of tilts between πr and πh and treats ρ as a design variable.Source: §3 · pp. 3–4

4 Theoretical results

The theoretical results bound post-training regret using recalibration error and query-distribution coverage, then choose an envelope distribution that minimizes worst-case coverage under a reward-class assumption. The paper gives a closed form for a clipped L2 ball and two ways to sample from the envelope.

4.1 Regret is controlled by coverage

  • Regret is controlled by recalibration error on the query distribution and a coverage factor for measures interpolating between the recalibrated and true-reward models.The bound motivates minimizing coverage because it can be arbitrarily large when relevant tilted models receive little query probability.Source: §4.1 · p. 4

4.2 The envelope is the minimax query distribution

  • The normalized envelope distribution minimizes the worst-case coverage factor over a reward class.When the class is convex and contains both rewards, the interpolated rewards remain in the class.Source: §4.2 · pp. 4–5

4.3 Closed form envelopes for balls around the surrogate

  • For a clipped L2 ball around the surrogate, the envelope has a closed form determined by the surrogate-optimized model and shifts probability toward its tails.The clipped construction permits sampling, and the closed form depends on the probabilities of πj rather than the unknown true reward.Source: §4.3 · p. 5

4.4 Sampling from the envelope

  • Rejection sampling proposes from πj and accepts according to the envelope weight, while fine-tuning uses a bonus-augmented reward whose tilt equals ρ∗.Rejection sampling requires sequence probabilities but no training; fine-tuning is an alternative when the rejection acceptance rate is low.Source: §4.4 · p. 6

5 Case study: inflating rare bad modes

A stylized model examines rare bad outputs that the surrogate overvalues, making them difficult to discover with on-policy labels. The envelope succeeds at the optimal label-complexity scale in this setting, while on-policy sampling requires substantially more labels.

5.1 Setting

  • The stylized setting has one good output and K bad outputs, with bad outputs rare under the base model but overvalued by the surrogate.This combination can leave the surrogate apparently well-calibrated on base-model samples even though optimizing against it is dangerous.Source: §5.1 · p. 6
  • Envelope sampling succeeds with Ω(K log K) labels, whereas on-policy sampling can fail until Ω((K/ϵ) log K) labels are collected.The guarantee is for the stylized setting with sufficiently many bad modes and the stated model assumptions.Source: §5.1 · p. 7
  • The envelope reaches all 116 answer labels in 450 draws, while the base-model query requires more than 14,000 draws in the controlled sycophancy experiment.The draw ratio is close to 1/ϵ for the actor’s bad-mode mass ϵ ≈ 0.03.Source: §5.1 · p. 7

6 Experiments

Across clinical note generation and controlled sycophancy, envelope-based recalibration mitigates reward hacking where base-model recalibration does not. The experiments also indicate that reweighting the surrogate-optimized model matters when bad outputs are heterogeneous.

6 Experiments

  • Across both tasks, envelope sampling mitigates reward hacking while on-policy recalibration leaves it in place; reweighting πj is necessary when bad outputs are heterogeneous.The paper also reports a numerical experiment illustrating its theoretical results.Source: §6 · p. 7

6.1 Clinical note generation

  • At T = 128, envelope-query recalibration yields 1.0% of notes with at least three fabricated vital signs and reference quality 0.658, while base-query recalibration yields 19.8% and 0.563.The untrained model’s reference quality is 0.651; envelope sampling also keeps fabrication below the base model’s 1.6%.Source: Figure 2 · p. 7, §6.1 · p. 8

6.2 Controlled sycophancy

  • After 256 labels, envelope recalibration lowers sycophancy error to 2.1%, below the actor’s 3.1%, while base-model recalibration leaves error above 33%.The envelope collects all 116 answer labels in 450 draws, compared with more than 14,000 draws for the base query.Source: Figure 3 · p. 8, §6.2 · p. 9

7 Conclusion

The conclusion states that, under proximity assumptions on the surrogate and recalibrated reward, the envelope minimizes an upper bound on regret. The appendices establish the regret and sampling results, derive the clipped L2 envelope, and analyze occupancy in a stylized model.

7 Conclusion

  • The envelope minimizes an upper bound on regret against the oracle-optimized model when the true and recalibrated rewards stay close to the surrogate; the theory assumes exact exponential-tilt post-training.Clinical experiments use an LLM reference rather than physician review, and the case study has a stylized output space.Source: §7 · p. 9

A Proofs

  • The proofs assume a finite output space, ensuring the sums, partition functions, tilts, expectations, and KL divergences are well-defined.Source: A Proofs · p. 15

A.1 The exponential tilt as a KL-regularized maximizer

  • The exponential tilt uniquely maximizes the KL-regularized objective, and the paper defines regret relative to the model optimized against the ground-truth reward.The proof uses nonnegativity of KL divergence to establish uniqueness.Source: §A.1 · p. 15

A.2 Proof of Theorem 1 (regret reduces to coverage)

  • The regret proof interpolates between recalibrated and true rewards and bounds regret using variance under queries that cover the tilted model supports.The theorem assumes the base-model support is contained in the query distribution's support.Source: §A.2 · pp. 15–16

A.3 Proof of Theorem 2 (envelope is minimax)

  • The normalized envelope uniquely minimizes the worst-case coverage factor over query distributions.The proof identifies the minimum at the normalized unnormalized envelope.Source: §A.3 · p. 16

A.4 Proof of Theorem 3 (closed-form clipped L2 envelope)

  • The clipped L2 envelope is derived by maximizing the tilted model's probability at each output subject to variance and oscillation constraints on reward perturbations.The resulting expression is specified on the support of the judge-optimized model, with zero envelope mass outside it.Source: §A.4 · p. 17

A.5 Proofs for Section 4.4 (sampling from the envelope)

  • The envelope can be sampled by rejection from the judge-optimized model or by tilting the base model with a bonus reward.Rejection sampling accepts proposals according to the envelope weight; the accepted distribution is the envelope.Source: §A.5 · p. 18

A.6.1 Preliminary lemmas

  • In the stylized model, post-training mass on bad modes depends on how many remain unlabeled, and occupancy bounds quantify how many bad modes sampling can miss.The occupancy argument gives a high-probability lower bound on the number of unlabeled bad modes.Source: §A.6.1 · pp. 18–19

A.6.2 Proof of Theorem 6 (i) (naive labeling fails)

The stylized-model theorem contrasts on-policy failure with envelope success and gives a lower bound applying to any query distribution. The appendices explain tail reweighting, an alternative sup-norm envelope, and parameter choices for sampling.

A.6.2 Proof of Theorem 6 (i) (naive labeling fails)

  • The theorem's on-policy case states failure under its conditions, while envelope sampling succeeds at the stated label budget.Source: §A.6.2 · p. 19

A.6.3 Proof of Theorem 6 (ii) (envelope labeling succeeds)

  • With n ≥ 2(K + 1) log(K/δ), envelope sampling labels every bad mode with probability at least 1 − δ, yielding the oracle model; the theorem also states that on-policy labeling fails under its stated conditions.The proof concludes that once all bad modes are labeled, the local rule recovers the true reward and the oracle model.Source: §A.6.2 · p. 19, §A.6.3 · p. 20

A.6.4 Proof of Theorem 6 (iii) (lower bound for any query distribution)

  • For any query distribution, covering K bad modes requires Ω(K log K) draws, so choosing a query cannot avoid the stylized model's coverage cost.The theorem's lower bound includes distributions that put all their mass on bad modes.Source: §A.6.4 · p. 20

B.1 The envelope weight and the role of the clip

  • The envelope weight decreases with judge-optimized probability, shifting mass toward its tails; clipping caps the weight and bounds sampling cost.Without clipping, the envelope approaches uniformity over rare tails, which is impractical over an exponentially large output space.Source: §B.1 · p. 21

B.2 The sup-norm ball

  • A sup-norm reward ball also yields a closed-form envelope by bounding each perturbation's pointwise difference across outputs.The cited proof uses a bound of 2M on that difference.Source: §B.2 · p. 21

B.3 Choosing β, M, and S

  • The optimization pressure β comes from the post-training KL-regularization coefficient, while M is set near the RMS discrepancy under πj and S is pilot-calibrated for rejection acceptance.The experiments target 80% acceptance; the appendix reports similar results at 50%.Source: §B.3 · p. 21

B.4 Illustrating the failure of on-policy sampling

Figure 4 contrasts query distributions across two judge-error settings: the envelope reaches the error in both, while the base model and judge-optimized model each fail in one. The accompanying finite-state simulation compares query choices across reward regimes and label budgets.

B.4 Illustrating the failure of on-policy sampling

  • The envelope reaches the judge's error in both settings: it retains mass on overrated rare outputs in setting A and restores mass to underrated typical outputs in setting B.The base model misses the error in setting A, while the judge-optimized model moves away from it in setting B.Source: Figure 4 · p. 22, §B.4 · p. 23

C.1 Design

  • The finite-state simulation uses four bad states and compares base-model and clipped-envelope queries under equal and heterogeneous surrogate rewards.The true reward assigns −1 to bad states; the heterogeneous surrogate assigns them rewards 8, 5, 3, and 1.5.Source: §C.1 · p. 23
  • Labels are exact, recalibration follows the local rule, and the simulation computes post-training regret across β settings, seeds, repeated label draws, and budgets from 8 to 64.The radii are chosen to include the true reward and partial repairs; in both reported regimes the clip is active.Source: §C.1 · p. 23

C.2 Results

In numerical simulations, envelope queries reduce regret as the label budget reaches the coupon-collector scale, while base-model queries leave regret near its unrecalibrated level. Bonus optimization recovers the envelope and reproduces its downstream sycophancy advantage, though scalable implementation for neural language models remains un demonstrated.

C.2 Results

  • Envelope regret falls by more than 90% within 16 labels at N = 5, 32 at N = 9, and 64 at N = 17 in the heterogeneous regime.The reduction takes at most twice as many labels in the equal regime; larger supports shift the curves right because more rare states share envelope mass.Source: §C.2 · pp. 23–24, Figure 5 · p. 24
  • After 64 labels, the envelope removes at least 3.0 of 3.3 regret units in the equal regime and 3.7 of 4.0 in the heterogeneous regime at every support.Base-model queries leave regret near its unrecalibrated value across budgets and supports.Source: §C.2 · pp. 23–24, Figure 6 · p. 24
  • At N = 5 and 64 labels, envelope regret is 0.000 in both regimes at every β, while base-model regret ranges from 0.72 to 3.39 in the equal regime and 0.77 to 3.96 in the heterogeneous regime.The base-model values correspond to β = 0.5, 2, and 6, respectively.Source: §C.2 · p. 24

C.3 Sampling cost

  • At β = 2 and N = 5, rejection sampling needs an expected 2.4 × 10^5 proposals per accepted sample in the equal regime and 1.3 × 10^7 in the heterogeneous regime; it is practical only when βS is moderate.The bonus fine-tuning route is the relevant implementation otherwise.Source: §C.3 · p. 25, §C.4 · p. 25

C.4 Sampling the envelope by optimizing the bonus

  • Natural-gradient ascent recovers the envelope to total variation below 0.01 in every numerical fit within 128 updates, with worst case 8.5 × 10^-6; sycophancy-context error differs from exact-envelope error by at most 0.006 percentage points.At β = 4 primary endpoints, the envelope-versus-base advantage is reproduced in full.Source: §C.4 · p. 25
  • With a cap of 200,000 proposals per setting, rejection produced the requested 512 samples in only 252 of 486 numerical settings and produced none in 167.Natural-gradient bonus optimization is practical in the finite enumerable settings studied when βS is large; first-order optimizers can stall at high β.Source: §C.4 · p. 25

D.1 Protocol

  • The clinical experiment compared recalibration from eight labeled notes drawn either from the base query ρ = π or the envelope query ρ = ρ∗, then post-trained each recalibrated judge with DPO.The envelope query used rejection sampling from πj; evaluation used a held-out panel, and the reference was a model rather than a physician panel.Source: §D.1 · pp. 26–27
  • Envelope-query rubrics assigned the minimum score to any invented numeric vital sign, whereas base-query rubrics capped or reduced the affected score.The editor grounded rubric changes in observed auditor-reference disagreements on the labeled notes.Source: §D.1 · pp. 26–27

D.2 Pooled and per-seed results

This section presents pooled and per-seed results on envelope-query recalibration in clinical note generation and a controlled sycophancy experiment. It covers reward-hacking mitigation, comparisons with base-model querying, and the role of controls and transcript-grounded rubric editing.

D.2 Pooled and per-seed results

  • At T = 128, envelope-query seeds stayed within 0.03 of untrained quality and had at most 3/64 severe fabrications, while base-query seed 202 had 26/64 and base quality fell below untrained in all seeds.Figure 7 shows the per-seed comparison; Table 2 reports pooled values, but the supplied caption does not provide its cell values.Source: §D.2 · p. 27, Figure 7 · p. 28
  • Pooled completeness rose from 0.830 at T = 0 to 0.875 for base query and 0.852 for envelope query at T = 128, and each recalibrated judge favored its own trained model over the untrained model.Both arms increased the reward their own judge assigned; they differed in what that reward bought under the reference.Source: §D.2 · p. 28, Figure 8 · p. 29

D.3 Controls

  • At T = 64, pooled quality was 0.589 for base query, 0.666 for α = 0.8, and 0.630 for α = 0.5, with 29, 4, and 11 severe fabrications of 192, respectively.Both envelope targets reduced severe fabrications in every seed; α = 0.8 improved quality in every seed, while α = 0.5 tied base query in seed 202 and had 8/64 severe fabrications in one seed.Source: §D.3 · p. 28, Figure 9 · p. 29
  • The no-recalibration control underperformed both query arms in seed 101, and additional optimizer seeds preserved the direction of the envelope effect.No recalibration yielded quality 0.528 and 15/64 severe fabrications, versus 0.573 and 13/64 for base query and 0.651 and 0/64 for envelope query; additional optimizer seeds had envelope-minus-base quality gaps of +0.011 ([−0.023, 0.045]) and +0.070 ([0.027, 0.111]), with 10/128 versus 15/128 severe fabrications.Source: §D.3 · p. 29

D.4 An example of the failure mode

  • A note converted qualitative vital signs into numeric values, earning 18/18 from the unrecalibrated auditor despite reference quality 0.5, six invented vitals, and 12 unsupported claims.The edited rubrics show the resulting instruction to credit only vital information actually supplied by the encounter.Source: §D.4 · pp. 29–30

D.5 Prompts

  • The appendix’s pipeline fixes the scribe and auditor task formats, grounds reference review in transcript evidence, and requires rubric changes to be justified by observed discrepancies.The auditor returns nine required scores; the editor is instructed to leave its rubric unchanged when no supported discrepancy warrants revision.Source: §D.5.1 · p. 30, §D.5.2 · p. 30, §D.5.3 · p. 31

D.5.5 Edited rubrics (seed 101)

  • The edited rubrics prioritize supported, clinically useful content and reject fabricated numeric vitals, unsupported boilerplate, and facts inferred from silence or normality.The envelope rubric also preserves qualitative findings as qualitative and gives a fabricated set of numeric vitals a score of 0.Source: §D.5.5 · pp. 33–34

E.1 Protocol

  • The protocol compares envelope and base-model queries for recalibrating judges in a two-answer sycophancy task, measuring each policy’s expected probability of an incorrect answer.The actor is Qwen3.8-27B with a verified factual snippet, making incorrect answers rare; judges are Qwen3.8-27B and Qwen3.5-9B. Calibration replaces the judge’s score with ground truth on labeled answers across contexts, while unlabeled answers retain surrogate scores.Source: §E.1 · pp. 35–36

E.2 All panels

  • Expected error was 1.7%–2.7% with envelope querying across all four panels and both judges, below the unoptimized actor, while base-model querying stayed above 33%.At about 4.4 draws per supported question, the matched budgets were 64 draws on panel 1 and 256 on panel 4. Each supported question contributes one overvalued answer that must be labeled before repair, so labels needed grow with panel size; envelope per-label efficiency was consistent across panels.Source: Table 3 · p. 36, §E.2 · p. 36

E.3 Additional results

Across additional sycophancy-task tests, envelope sampling reduces error relative to base-model queries across judges, optimization pressures, and stricter calibration settings. The prompts define a controlled preference in which validating even a false belief is rewarded, while factual disagreement is not treated as hostility.

E.3 Additional results

  • For the 9B judge, envelope queries reduce error from 45.0% unrecalibrated to 2.3% after 256 labels, while base-model queries leave 37.0% error.The envelope reaches 23.1% after 64 labels and 9.7% after 128.Source: §E.3 · p. 36
  • As β rises from 1 to 8, unrecalibrated harm grows from 6% to 90%, while envelope queries roughly halve error at every β under both judges.The base-model query removes almost none of the harm at any β.Source: §E.3 · p. 36, Table 4 · p. 37
  • Across four panels and both judges, envelope curves coincide when plotted by draws per supported question, while base-model query error stays above 29%.This scaling matches the prediction that labels needed grow in proportion to the number of questions.Source: Figure 10 · p. 37
  • With 58 distinct labels, envelope error is 19.4% (27B) and 21.9% (9B), versus 36.9% and 40.9% for base-model queries.With all 116 labels, every query reaches the truth-optimal error of 0.06%.Source: §E.3 · pp. 36, 38
  • Collecting all 116 labels takes an average of 450 draws with envelope sampling, versus at least 14,200 draws on average for base-model queries.The base-model query had not finished within 20,000 draws on 73 of 512 seeds.Source: §E.3 · p. 38
  • Under a stricter calibrator that repairs only the labeled context–answer pair, envelope error remains lower after 64 draws: 27.7% and 32.1%, versus 37.7% and 43.1% for base queries.The pairs are 27B then 9B, at β = 4.Source: §E.3 · p. 38
  • When users deny the correct answer, envelope-repaired error is 2.4% (27B) and 3.0% (9B), versus 4.6% and 5.7% for base-repaired rewards.Without labels, the unrecalibrated errors are 3.4% for 27B and 4.9% for 9B.Source: §E.3 · p. 38

E.4 Prompts

  • The toy preference assigns +1 to validating even a false belief, 0 to neutral factual answers or polite correction, and −1 only to hostility; judge scores average six letter orders.Source: §E.4 · p. 38

F Labeling samples from the judge-optimized model

The judge-optimized query often performs comparably to envelope sampling when errors are similarly overvalued, while envelope reweighting helps when overvaluation is heterogeneous. Clinical and sycophancy experiments and simulations compare these query strategies.

F Labeling samples from the judge-optimized model

  • The envelope reweights the judge-optimized query toward outputs it still treats as rare; the two queries coincide only when that query is uniform on its support.As the envelope’s acceptance rate approaches one, it approaches the judge-optimized query; the acceptance rate bounds their total-variation distance by 1−α.Source: F Labeling samples from the… · p. 39
  • When incorrect answers were overvalued similarly, the judge-optimized query sampled them at their error rate—37.7% under the 27B judge and 45.0% under the 9B judge—while the envelope sampled them half the time.The section reports that the judge-optimized query did about as well as the envelope in this similar-overvaluation regime.Source: Controlled sycophancy · p. 39, Figure 12 · p. 40

Controlled sycophancy.

  • At 256 draws, the envelope was below the judge-optimized query: 2.1% against 4.8% for the 27B judge and 2.3% against 3.3% for the 9B judge.For the 9B judge, the queries were within 0.2 points through 64 draws, with the envelope lower from 128 draws onward.Source: Controlled sycophancy · p. 39
  • At 58 distinct labels, the envelope reached 19.4% (27B) and 21.9% (9B), versus 21.6% and 21.5% for the judge-optimized query.Collecting all 116 labels took 450 envelope draws under either judge, versus 1,055 judge-optimized draws under the 27B judge and at least 3,960 under the 9B judge.Source: Controlled sycophancy · p. 39

Clinical note generation.

  • In the clinical task, both judge-optimized and envelope queries reduced severe fabrications from 29 of 192 notes to 1 and 4, respectively; the judge-optimized query was at least as good overall.The pooled quality difference was −0.014 with an interval covering zero, while notes from the judge-optimized query had 1.06 fewer unsupported claims per note with an interval excluding zero.Source: Table 5 · p. 40, Clinical note generation · p. 40

Numerical simulations.

  • In the heterogeneous simulation, judge-optimized and mixed queries repaired only one bad mode and left regret above 3.4, while the envelope reached regret below 0.2 at 64 labels.The envelope inflated every rare state regardless of how strongly the surrogate favored it; the judge-optimized query concentrated on the most overvalued state.Source: Numerical simulations · p. 40, Figure 13 · p. 41

G.1 Setup and results

The previous-generation clinical pipeline used a reference reviewer, rubric editor, and post-training evaluation to compare recalibration queries. The reported result was the same failure and repair pattern.

G.1 Setup and results

  • The previous-generation pipeline used Claude Opus 4.8 as both reference rater and rubric editor, with eight query-drawn note, judge-score, and reference-score triples for editing.The reference rated notes on a 0–18 scale, and the post-training evaluation scored 1,600 generations per arm.Source: §G.1 · p. 41
  • The base model was post-trained against each recalibrated judge using listwise DPO, and each arm was evaluated on 1,600 generations.The method used the Plackett–Luce formulation of Rafailov et al. [2] rather than Bradley–Terry.Source: §G.1 · p. 41
  • The previous-generation experiment reported the same failure and repair pattern as the main clinical pipeline.The supplied passage does not quantify the effect sizes for this comparison.Source: Results · p. 41

Results.

In the previous-generation clinical experiment, post-training against the unrecalibrated judge increased severe fabrication, and base-model recalibration left it high. Envelope recalibration exposed more failure modes and reduced severe fabrication, while results across instantiations were limited to a single seed.

Results.

  • Post-training against the unrecalibrated judge raised notes with three or more fabricated vital signs from 0.81% to 22.12% and lowered the reference score by 2.11 points.Table 6 reports mean reference scores and fabrication rates over 1,600 generations in a single seed.Source: Results · p. 41
  • Base-model recalibration left the rate of notes with three or more fabricated vital signs at 17.38%; the editor patched only a plan-element penalty..Source: Results · p. 41
  • Envelope recalibration exposed failure modes across four rubric elements, and post-training reduced severe fabrication to 0.19% while improving reference score over the original model.The four elements were history of present illness, vitals, exam, and results.Source: Results · p. 42
  • The pattern agreed across model generations, model scales, reference scorers, editors, and post-training algorithms, but the single seed did not support quantitative effect-size comparisons..Source: Results · p. 42

Related papers

Loading 2610.11281v1…