PaperlayerView exact revision on arXiv

Source-linked AI summary

Better Policies, Worse Verifier Audits

Daniel Cieślak, Jacek Łubiński

arXiv:2610.29029v1cs.LG

TL;DR

A component verifier audited against the full outcome on fresh policy samples can become misleading as the policy masters that component. The paper characterizes how the changing failure mixture affects audit identification and verifier choice, then tests refresh outcomes and a support-preserving repair.

  • Problem

    When a component verifier is audited against the full outcome on fresh policy samples, policy improvement can change the failure mixture and make the audit systematically wrong.

    Source: §1 · pp. 1–2

  • Method

    The paper derives mixture-based identification and inversion results, tests them across oracle, language, and ECG settings, and evaluates component-labeled refresh with and without restored failure support.

    Source: §1 · p. 2, §4.1 · p. 5, §5 · p. 6

  • Results

    40/40 paired evaluations showed outcome refresh lowered source-generator off-policy F-AUC; restoring base-policy candidates preserved F-AUC at 1.000 in every source setting.

    Source: §5 · pp. 7–8

  • Takeaways & Limitations

    Auditing the component on its own failures and retaining those failures in refresh data avoids the damage observed when support is depleted.

    Source: §1 · p. 2, §5 · p. 8

    The GoEmotions stress test is post-hoc and uses a small bag-of-words verifier, so it does not measure the refresh effect with an LLM-scale semantic verifier.

    Source: §7 · p. 9

Abstract

from arXiv · show

Verifier stacks increasingly sit inside post-training and test-time compute loops. We study a specific control loop: a component verifier is evaluated on fresh policy outputs against the full success label and refreshed when that aggregate score falls. For a conjunctive target $y = F \wedge M$, its on-policy AUC is $(1-q)A_{\mathrm{own}} + qA_{\mathrm{cross}}$, where $q = \Pr[F=1 \mid y=0]$. As the policy masters $F$, $q$ can rise toward one, removing weight from $F$-failures and rewarding information about $M$. We call this on-policy audit collapse. We derive a sharp identified set, a complete-erasure threshold, and the exact point at which the audit can prefer a worse component verifier; its population optimum is the full-outcome predictor, not the component oracle. An exact oracle exhibits the collapse, a natural multi-label text control shows inversion can select a head with lower canonical $F$-AUC at the observed failure mixture, and a prediction-registered ECG study shows that acting on the audit can reduce off-policy $F$-AUC, transfer the damage to an unseen generator, and increase downstream constraint violations. Preserving own-failure support prevents the damage. Matching label-complexity bounds and a controlled-mixture audit give the corresponding repair: audit the component on its own failures, and keep those failures in the refresh distribution.

1 INTRODUCTION

The paper studies a verifier-refresh loop that evaluates component verifiers against the full outcome on fresh policy samples. As a policy masters one conjunct, the resulting failure-mixture shift can make the audit misleading or actively favor a verifier that is worse on the component itself.

1 INTRODUCTION

  • For y = F ∧ M, on-policy AUC is (1 − q)Aown + qAcross, where q = Pr[F = 1 | y = 0].As F saturates while M remains imperfect, q can rise, shifting audit weight away from F-failures and toward information about M.Source: §1 · p. 1
  • At a calculable threshold, the aggregate audit can prefer a verifier with worse own-failure separability because it carries more information about M.On-policy refresh can further entangle the component with the full outcome or lose the F-failures needed to identify its boundary.Source: §1 · p. 2
  • A component audit against the full outcome can become systematically wrong as the policy improves, because optimization changes the data used to decide whether the verifier should be replaced or refreshed.The paper calls this a control-plane failure and limits its claim to the interface where a component head is audited against the full outcome and the audit informs comparison, stopping, or refresh.Source: §1 · pp. 1–2
  • Across the tested settings, an exact-oracle audit collapses, text stress tests show inversion, and ECG refresh damages off-policy component performance while restoring failure support prevents the damage.The experiments include an exact component oracle, language-domain controls, and a prediction-registered ECG study.Source: §1 · p. 2

2 MECHANISM-FIRST EVALUATION

The evaluation is organized to test the proposed failure chain across complementary settings. It distinguishes own-failure separability from canonical component quality so that audit behavior is not mistaken for actual component degradation.

2 MECHANISM-FIRST EVALUATION

  • Engine tests the inversion threshold with an exact component oracle, GSM8K is a negative control, GoEmotions tests semantic criteria, and ECG is the operational stress test.The ECG evaluation uses a frozen benchmark, three generator families, held-out subjects, two refresh model classes, and downstream gating.Source: §2 · p. 3
  • Own-failure separability Aown compares target successes with F-failures, whereas canonical component quality AF compares all F=1 examples with F=0 examples.The estimands need not agree; the paper reports AF separately when claiming actual component degradation.Source: §2 · p. 3
  • The paper treats best-of-K supply, selection efficiency, and Answer@K as separate measures rather than substitutes for component AUC.The supplied passages locate their definitions and audits in Appendix AC.Source: §2 · p. 3

3 THE PARADOX: POLICY IMPROVES, AUDIT COLLAPSES

In engine and GSM8K, policy success improves while the component head’s aggregate full-outcome AUC falls toward chance. An exact-oracle control shows that this symptom can arise without component failure.

3 THE PARADOX: POLICY IMPROVES, AUDIT COLLAPSES

  • 3.6× higher engine target success coincided with feasibility-head aggregate AUC falling from 0.80 to 0.51; 2.1× higher GSM8K target success coincided with format-head AUC falling from 0.92 to 0.51.Native selector efficiency fell to 0.05 and 0.09, respectively.Source: §3 · p. 3
  • The exact-oracle control shows aggregate collapse cannot by itself identify component degradation: own-property AUC remained 0.940–0.991 on matched engine pools, and the GSM8K format head retained 0.995 accuracy.The passage also reports mass-head AUCs of 0.96–0.98 for two unused engine critics.Source: §3 · p. 3

4 ON-POLICY AUDIT COLLAPSE

The aggregate outcome audit mixes a component’s own-failure separability with its ordering of failures that already satisfy that component. As the policy changes this mixture, the audit can lose identification and even prefer a worse component verifier.

4 ON-POLICY AUDIT COLLAPSE

  • For each pool, aggregate AUC is Ar = (1 − qr)Aown,r + qrAcross,r, so increasing qr reduces the weight placed on the component’s own-failure task.Across compares target successes with failures that already satisfy F.Source: §4 · pp. 3–4
  • The sharp identified set for component own-failure separability is constrained by the observed aggregate AUC and failure-bin weights, but may remain broad.With multiple failure bins, enough deliberately distinct mixture rows can identify the binwise AUCs when the mixture matrix has full column rank.Source: §4 · p. 4
  • When q ≥ max(A, 1 − A), the identified interval can span all of [0, 1]; every matched engine row is in this complete-erasure regime.Those rows have q ≥ 0.916 and aggregate AUC 0.49–0.58 despite direct bin audits measuring 0.94–0.99.Source: §4 · p. 4
  • On ECG, observing all negative scores or estimating q from 50 labels only modestly reduces erased pools, leaving 34–48% erased across the reported settings.The passage reports 37% to 34% for Llama and 48% to 48% for Phi when all negative scores are observed.Source: §4 · p. 4
  • Even an exact F-oracle has AUC(F; y) = 1 − q/2, falling from 0.833 to 0.515 in engine and 0.922 to 0.519 in GSM8K.This is a composition-only path with Aown = 1 and Across = 1/2.Source: §4 · p. 4

4.1 AUDIT INVERSION: CROSS-COMPONENT SIGNAL CAN WIN

  • The aggregate audit prefers a head with worse own-failure separability when its cross-component advantage outweighs that deficit; its population maximizer is the full-outcome predictor.An exact component oracle is weakly dominated at every q, strictly unless the outcome predictor is constant on F-satisfying candidates.Source: §4.1 · pp. 4–5

4 ON-POLICY AUDIT COLLAPSE

  • Figure 1 shows aggregate outcome AUC changing with q despite strong own-bin ordering, while the Across = 1/2 reconstruction tracks measured AUC.The complete-erasure region is shown separately in Appendix P.Source: Figure 1 · p. 5

4.1 AUDIT INVERSION: CROSS-COMPONENT SIGNAL CAN WIN

  • At observed engine q = 0.970, the audit ranks the oracle below the imperfect head (0.515 vs 0.532), despite the oracle’s canonical F-AUC of 1.The predicted crossing is q∗ = 0.64; GSM8K has no crossing because the format head’s Across is below 1/2.Source: §4.1 · p. 6

5 THE REFRESH TRAP: ACTING ON AUDIT COLLAPSE

The ECG study tests whether acting on a collapsed audit damages a component verifier and downstream decisions. Outcome-based refresh damages off-policy component performance, while restoring own-failure support preserves it.

5 THE REFRESH TRAP: ACTING ON AUDIT COLLAPSE

  • At natural q = .522 on GoEmotions, the aggregate audit preferred an entangled score (.681) over the specialist (.442), although its canonical F-AUC was lower (.570 versus .679).The deliberately chosen post-hoc positive control had predicted crossing q∗= .126.Source: §5 · p. 6
  • Before the ECG refresh study, the deployability head’s own-failure bin was empty in every evaluable Llama pool and in 64–72% of Phi pools, versus 13% and 36% at base.The protocol used held-out subjects, two refresh classes, and evaluated source-generator F-AUC, unseen-Gemma transfer, and a best-of-16 gate.Source: §5 · p. 6
  • Across all four family×class settings, the pure component head had F-AUC 1.000 yet the lowest on-policy aggregate AUC among heads trained with mixed component and outcome labels.The result supports the prediction that adding outcome information improves the audit score while moving the verifier away from a pure component representation.Source: §5 · p. 7
  • Outcome refresh lowered source-generator F-AUC by 0.09–0.12 for MLPs and 0.43–0.48 for GBMs, and unseen-Gemma F-AUC by 0.29–0.51.These declines had the same sign in all 40 paired evaluations; the count reflects sign consistency across overlapping splits, not independent replications.Source: §5 · p. 7
  • Using outcome-refreshed verifiers as best-of-16 feasibility gates raised violations by +0.011 to +0.055 above the unavoidable floor and lowered Answer@16 by 0.02–0.06.The sign was consistent across all 40 paired evaluations.Source: §5 · p. 8
  • Re-injecting base-policy candidates while retaining component labels preserved source F-AUC at 1.000, unseen-generator AUC at 0.998–1.000, and gate metrics at deployed-specialist levels.Correct component labels alone did not suffice: RF lost 0.235 source-generator F-AUC in the Llama-GBM setting.Source: §5 · p. 8
  • With a 0.05 aggregate-AUC trigger, refresh changed mean F-AUC/violation/Answer@16 from 1.000/.148/.530 to .719/.179/.491; a reference-distribution F-specific trigger fired in none.The aggregate trigger activated in all four primary settings despite deployed specialists retaining F-AUC 1.000.Source: §5 · p. 8
  • Five of six registered predictions held; the evidence linked entanglement, depleted own-failure support, refresh-related forgetting, and increased downstream violations, while refresh did not reliably repair the monitored AUC.The reported rescue preserves or deliberately reconstructs the component’s own-failure distribution.Source: §5 · pp. 6, 8

6 REPAIR: RESTORE FAILURE SUPPORT

The repair is to measure component performance on its own failures and preserve those failures in the refresh data. Passive audits become costly as own-failure support vanishes; controlled mixtures can restore identification.

6 REPAIR: RESTORE FAILURE SUPPORT

  • If own-failure support is absent, construct controlled negative mixtures by reweighting or resampling fixed bin-conditional candidates, then invert the two-mixture system.Changing either bin-conditional score distribution makes naive two-row inversion biased; the operation requires designed mixtures, not arbitrary policy checkpoints.Source: §6 · p. 8
  • The audit card recommends direct own-failure auditing when supported, controlled mixtures when support is missing, and off-policy quality reporting rather than aggregate outcome AUC alone for refresh decisions.It also calls for auditing the downstream interface that consumes the score.Source: §6 · p. 8
  • Passive-audit label budgets scale with m(1−q): matching upper and lower bounds show estimation becomes expensive as own-failure prevalence falls and impossible when its bin is empty.The designed-audit error bound instead depends on the mixture-weight gap |q2−q1|, not the deployed policy’s q.Source: §6 · p. 9
  • On ECG, m(1−q) · RMSE stayed at 0.20–0.24 for q ≤0.95, while two-mixture inversion reached RMSE 0.01–0.02 at |q2−q1| = 0.6.With 100 labels, subject-bootstrap coverage was 96–98%.Source: §6 · p. 9
  • Re-injecting base-policy failures preserved ECG source F-AUC at 1.000 in every setting and recovered Aown with RMSE ≤0.02 even where post-DPO own-failure bins were empty.The same re-injection distinguished the protected refresh from component-labeled refresh without restored support.Source: §6 · p. 9

7 SCOPE: WHAT THE RESULTS DO AND DO NOT ESTABLISH

The evidence spans several controlled tests, but the ECG refresh study is limited to a CPU-only signal-processing benchmark and does not validate clinical or LLM-scale settings.

7 SCOPE: WHAT THE RESULTS DO AND DO NOT ESTABLISH

  • The ECG refresh study is CPU-only and uses one benchmark with a deterministic resource-constraint component, two head classes, and ten subject splits.The authors did not refresh an LLM-scale verifier or study online RL with a refreshed verifier.Source: §7 · p. 9
  • The ECG benchmark is not a clinical validation, and its results do not establish refresh effects for an LLM-scale semantic verifier.Contemporary rubric-RL findings establish interface relevance in the paper, not validation of its inversion or refresh claims.Source: §7 · p. 9
  • The GoEmotions test is a post-hoc natural multi-label stress test using a small bag-of-words verifier; AG News is a constructed control with 24 deterministic robustness splits.The authors do not claim the refresh effect has been measured with an LLM-scale semantic verifier.Source: §7 · p. 9
  • The audit requires trusted component labels because proxy bins change the estimand; the engine oracle is simplified, and the strongest training result is offline DPO with three seeds.Engine and GSM8K policies use Qwen2.5, while ECG training is informative in one family.Source: §7 · p. 9

8 RELATED WORK

The paper situates its audit interface among work on rubric aggregation, policy-dependent criteria, and verifier-loop failures, while drawing on classical statistical tools. Its claimed novelty is their combination under policy-induced semantic failure mixtures.

8 RELATED WORK

  • Rubric RL aggregates separately scored prompt-specific criteria into scalar on-policy rewards, while policy-aware rubric work finds criterion informativeness can change as criteria saturate or become dead.Fresh policy interactions are also used to refine reward or preference models.Source: §8 · p. 9
  • Prior work supplies the statistical ingredients: case-mix effects, partial identification, finite-mixture identification, within/cross AUC comparisons, and classical optimal ranking.The paper explicitly does not claim these ingredients as new.Source: §8 · p. 9
  • The paper’s claimed novelty is combining those ingredients under policy-induced semantic failure mixtures to show loss of component identification, verifier-ranking inversion, and audit-driven maintenance failure.Adjacent work studies behavior loss under on-policy optimization and verifier regression after task shift; this paper focuses on the audit interface.Source: §8 · pp. 9–10

AI USE STATEMENT

Generative AI supported manuscript development, but the authors retain responsibility for the final work.

AI USE STATEMENT

  • Generative AI helped improve manuscript clarity and presentation.It also assisted with software implementation and debugging and provided critical feedback on claims and arguments.Source: AI USE STATEMENT · p. 10
  • The authors treated AI-generated suggestions as editorial and technical assistance, not as evidence or independent validation.Source: AI USE STATEMENT · p. 10
  • The authors take responsibility for the final claims, proofs, code, citations, analyses, and submission.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The engineering and ECG evaluations are bounded benchmarks, not safety or clinical validations.

ETHICS STATEMENT

  • Engine evaluations use a simplified geometry, mass, and printability model rather than safety certification or fabrication validation.Source: ETHICS STATEMENT · p. 10
  • The ECG benchmark uses public, de-identified PhysioNet recordings under their published terms.It is a signal-processing deployment benchmark, not diagnostic or clinical validation.Source: ETHICS STATEMENT · p. 10
  • ECG outputs should not be used for patient care; the authors also note propulsion design’s dual-use implications.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The supplement supports exact paper rebuilding and independent reruns of public-data semantic controls, while documenting protocols and frozen summaries for larger experiments that cannot be regenerated end to end from the lightweight artifact.

REPRODUCIBILITY STATEMENT

  • The supplement enables an exact paper rebuild and reruns of the public-data semantic controls, but not end-to-end regeneration of every large system experiment from raw candidates.Original large engine, ECG, and GSM8K candidate pools and model checkpoints are not bundled; their protocols and frozen summaries are documented.Source: REPRODUCIBILITY STATEMENT · p. 10

APPENDIX ROADMAP

  • The appendix separates evidence supporting the core propositions and refresh-trap conclusion from optional downstream-selection and DPO boundary analyses.Scripts and frozen summaries are indexed in the supplement README.Source: APPENDIX ROADMAP · p. 13

A THE ENGINE ORACLE

  • The engine solver evaluates geometry, integrated mass, printability, and envelope fit, returning feasibility in 7.9 ms.Validation checks compare thermochemistry with NASA CEA and test flow against the Sod problem and oblique-shock relation.Source: A THE ENGINE ORACLE · p. 13

B THE IDENTITY FOR THE MASS HEAD

  • As q approaches one, the mass head’s aggregate AUC against the conjunctive target approaches its own accuracy: 0.937 on the base pool and 0.98 on trained pools.Its own sub-problem AUC is 0.96–0.99 across arms.Source: B THE IDENTITY FOR THE MASS HEAD · p. 13

C GSM8K WITH A CONJUNCTIVE TARGET

  • At K=64, GSM8K answer rate under the two-stage gate is 0.762 for base, versus 0.645 ± 0.012 and 0.657 ± 0.011 for the reported trained arms.On trained-policy samples, on-target and off-target pairs separate at 0.70 versus 0.59 under each tested rule.Source: C GSM8K WITH A CONJUNCTIVE TARGET · p. 13

D ENGINE ARMS NOT IN THE MAIN TEXT

  • Engine training-side variants report feasible rates of 0.39–0.87 for DPO and IPO, while latent search reaches supply 0.937 ± 0.017 at K=64 versus 0.336 ± 0.003 for direct sampling.The latent-search pools have Across = 0.79; the single-seed 7B K=64 reversal is explicitly not claimed.Source: D ENGINE ARMS NOT IN THE MAIN TEXT · pp. 13–14

E THE DESIGNS IN PARAMETER SPACE

  • Across every policy, on-target designs average 2.85–2.97 active zones, achieve a printability score of 1.0, and place mass at 0.99 of the budget on average.The benchmark—oracle, critic, splits, pools, and every result file—is released with the paper.Source: E THE DESIGNS IN PARAMETER SPACE · p. 14

F THE ESCAPE

  • The critic-trained decoder produces a block with radius controls clipped from far outside the box, despite a pre-clipping critic feasibility score of 0.9999.That design passes the mass-and-printability criterion.Source: Figure 4 · p. 15

H 7B AND ONLINE GRPO EXPERIMENTS: REGISTRATION AND SUPPORT

Registered follow-ups report both held predictions and misses, alongside post-hoc audits and matched-pair training tests. The results distinguish selection performance from training-pair construction and record uncertainty boundaries for the added experiments.

H 7B AND ONLINE GRPO EXPERIMENTS: REGISTRATION AND SUPPORT

  • The preregistered 7B DPO predictions held for per-draw and composed answer on each seed, while feasibility was lower on every gated seed.One Aown threshold missed on gated seed 1; the report retains the run and does not change its threshold.Source: H 7B AND ONLINE GRPO EXPERIMENTS:… · p. 16
  • The registered GRPO supply-advantage prediction failed, while feasibility above 0.8 and coordinate spread below 0.17 held for both rewards.Gate per-draw exceeded sum on every seed, but the registered threshold of 0.20 failed; endpoint outcomes do not establish the predicted trajectory.Source: H 7B AND ONLINE GRPO EXPERIMENTS:… · p. 16

K COMPUTE

  • The reported compute record cannot identify which added 7B jobs used A100-80GB versus H100 accelerators.Original DPO took approximately 2.5 hours per 1.5B seed and 5 hours per 7B seed.Source: K COMPUTE · p. 18

L ADDITIONAL GEOMETRY OBSERVATIONS

  • On-target designs after training retain distinct geometries, but coordinate spread falls to 0.07 after sum-ranked round two versus 0.13 after gate-ranked round two.The sum-ranked designs use two of four lattice families, compared with three under gate ranking.Source: L ADDITIONAL GEOMETRY OBSERVATIONS · p. 19

M POST-HOC SELECTION AND UNCERTAINTY AUDIT

  • In matched 1.5B sum-DPO pools, raw sum misses an available solution on 0.446–0.482 of requirements versus 0.035–0.048 under two-stage selection.The analysis also audits top-1 failures and nested K answer curves.Source: M POST-HOC SELECTION AND UNCERTAINTY… · p. 20

I REGISTERED PREDICTIONS AND OUTCOMES

  • Registered predictions include substantial misses: tuned-sum and log-probability training fell 0.187 and 0.082 below the gate, respectively, on composed answer.The tuned-sum q prediction also missed, at 0.57–0.63, while Aown exceeded 0.92 throughout those arms.Source: O MATCHED TRAIN-PAIR FOLLOW-UP: TUNED… · p. 22

O MATCHED TRAIN-PAIR FOLLOW-UP: TUNED SUM, LOG-PROBABILITY AND MASS-ONLY DPO

  • Contrast-matched 1.5B training reached composed answer 0.770 ± 0.023, and a later 7B follow-up reached 0.565 ± 0.065 versus 0.562 ± 0.026 for the matched hard gate.The 7B follow-up is not preregistered; the paired contrast-matched-minus-gate difference was 0.003 [−0.015, 0.020].Source: O MATCHED TRAIN-PAIR FOLLOW-UP: TUNED… · pp. 22–23, Table 15 · p. 22
  • The critic-only maximin rule selects weights in the 0.05–0.10 region, and its chosen point is stable across all reported test-pool bootstrap resamples.The criterion selects w = 0.074989 at 1.5B and w = 0.056234 at 7B.Source: O MATCHED TRAIN-PAIR FOLLOW-UP: TUNED… · p. 23

P PROOF OF THE SHARP AUC IDENTIFIED SET

The aggregate AUC is a mixture of separability across negative failure bins, so observing it and the mixture weights generally identifies a set—not each bin’s AUC. The set is sharp, and when failures satisfying F dominate sufficiently, the statistic can contain no information about A_own.

P PROOF OF THE SHARP AUC IDENTIFIED SET

  • A = πᵀa, so the observed aggregate AUC and negative-bin mixture weights constrain the bin-specific AUC vector to a linear slice of [0, 1]^J.Here each a_j is positive-versus-bin AUC, with ties receiving half credit.Source: P PROOF OF THE SHARP AUC IDENTIFIED SET · p. 23
  • Every vector in that slice is attainable, making the identified set sharp; for two bins, A = (1 − q)A_own + qA_cross.If q = 1, A_own drops out completely.Source: P PROOF OF THE SHARP AUC IDENTIFIED SET · pp. 23–24
  • K controlled negative mixtures identify all J bin-specific AUCs for every possible value exactly when the mixture matrix has rank J.Ordinary checkpoints cannot automatically be treated as controlled mixtures because their conditional score distributions may drift.Source: P PROOF OF THE SHARP AUC IDENTIFIED SET · p. 24
  • When q < 1, the sharp interval for A_own is all of [0, 1] exactly when 1 − q ≤ A ≤ q.All four matched engine rows satisfy this complete-erasure condition; it concerns the aggregate statistic, not direct own-bin estimates with adequate support.Source: P PROOF OF THE SHARP AUC IDENTIFIED SET · p. 24, Figure 7 · p. 24

Q WHY VALIDATION-TUNED SELECTION WEIGHTS NEED NOT BE PAIR-OPTIMAL

Validation top-1 performance need not produce training pairs with balanced component contrasts, and the observed best scalar’s ordering overlaps poorly with the hard gate’s complete pairs. Post-hoc controls also show that fixing format classification does not restore GSM8K answers, while rescaling logits changes raw-sum answers without changing head AUCs.

Q WHY VALIDATION-TUNED SELECTION WEIGHTS NEED NOT BE PAIR-OPTIMAL

  • Away from pairwise crossing weights, within-pool rankings are fixed, so identical chosen–rejected multisets give exactly the same DPO loss.The score itself is no longer an input to DPO once pairs are selected.Source: Q WHY VALIDATION-TUNED SELECTION… · p. 24
  • The reference-gradient contrast bound is a sufficient-condition bridge, not an empirically established mechanism, because per-pair gradient artifacts were not retained.Its assumption is a local feature model for the preference-log-ratio gradient.Source: Q WHY VALIDATION-TUNED SELECTION… · p. 25
  • At w = 0.5, top-1 success is 1, but top-versus-bottom pairs have average oracle contrasts (∆F, ∆M) = (1, 0).At w = 2, top-1 success falls to 1/2 while average contrasts become (1/2, 1/2); this is an existence example, not evidence that the mismatch explains the engine experiment.Source: Q WHY VALIDATION-TUNED SELECTION… · p. 25
  • The best-performing scalar’s chosen candidates overlap the gate by 65%, but only 15% of complete chosen–rejected pairs coincide.All round-1 rules use 4,000 pairs from the same training pool.Source: Q WHY VALIDATION-TUNED SELECTION… · p. 25
  • Across five matched scalar rules, bottleneck-contrast correlation with composed @16 is 0.90, versus −0.60 for exact pair overlap; this is a descriptive check with n = 5.The source cautions that this is not a significance result or fitted predictive model.Source: Q WHY VALIDATION-TUNED SELECTION… · pp. 25–26

R PERFECT-CONJUNCT AND FINITE-POOL CONTROLS

  • With exact format gating, GSM8K answers after sum-DPO are 0.679 at k = 16 and 0.639 at k = 64, versus base 0.737 and 0.758.The control removes format-classification error without repairing the learned correctness ranker; it does not establish impossibility for other correctness models or selection methods.Source: R PERFECT-CONJUNCT AND FINITE-POOL… · p. 26, Figure 9 · p. 27
  • Scaling feasibility logits by T ∈ {0.01, 0.1, 1, 10, 100} preserves every head AUC but changes mean raw-sum answers to 0.544, 0.312, 0.185, 0.162, 0.160.This is a fixed test-set sensitivity analysis, not validation-based hyperparameter tuning.Source: R PERFECT-CONJUNCT AND FINITE-POOL… · p. 26

S TRANSFER FORECAST OF ON-POLICY AUC

The transfer forecasts use source-pool bin terms and target-pool q to predict on-policy AUC without re-auditing target ordering, then test that relation across policy arms and checkpoints. Performance varies by setting: source terms add value in a pre-specified later-arm test, while a label-free diagnostic does not transfer across domains and a feasibility substructure control shows that a shifted failure mix alone need not collapse AUC.

S TRANSFER FORECAST OF ON-POLICY AUC

  • On the 11 non-saturated stored arms, zero-parameter MAE is 0.0429, versus 0.2863 for chance and 0.1596 for persistence; transporting the cross term gives 0.0404.Across all 57 stored comparisons, source-own forecast MAE is 0.0280, slightly above the 0.0261 perfect-head and q-only linear baselines.Source: S TRANSFER FORECAST OF ON-POLICY AUC · pp. 27–28, Table 17 · p. 29
  • On twelve arms trained after source terms were frozen, the registered forecast error was 0.020 against a 0.030 target; source-own terms beat 1 − q/2 by 0.0098 and the historical linear baseline by 0.0054.The outcome rates had been read before the control, so this is a pre-specified transfer test, not a fully prospective forecast.Source: S TRANSFER FORECAST OF ON-POLICY AUC · p. 29, Table 18 · p. 29
  • Within runs, the zero-parameter forecast has arm-level error 0.049 versus 0.096 for persistence and 0.138 for chance over 18 transitions, beating persistence on 15.The mass-only null run has q in [0.28, 0.41], and the forecast gains only 0.010 there.Source: S TRANSFER FORECAST OF ON-POLICY AUC · pp. 29–30, Table 19 · p. 30, Figure 11 · p. 30
  • Across five OOD generator families, forecast MAE is 0.0282 versus 0.0752 for persistence and 0.3640 for chance; the capped decoder is the exception, with own-bin AUC falling from 0.965 to 0.821.Transporting A_own assumes its ordering remains stable across policies, and the forecast still uses target-side oracle-label statistic q.Source: S TRANSFER FORECAST OF ON-POLICY AUC · pp. 28, 30, 31

T A LABEL-FREE VERSION OF THE DIAGNOSTIC DOES NOT TRANSFER

  • Adding GSM8K makes both label-free statistics worse than predicting the mean: saturated fraction has r = 0.02 and SD ratio r = −0.14.This is a negative result for three specific statistics and two targets, not proof that no label-free diagnostic exists.Source: T A LABEL-FREE VERSION OF THE… · p. 31

U FEASIBILITY’S OWN SUB-CONSTRAINTS: A CASE WHERE THE MIX MOVES AND THE AUC DOES NOT

  • Printability reaches 0.93–1.00 across matched trained arms while envelope fit ends at 0.811–0.880, shifting the remaining-failure share that prints but does not fit to 0.21–0.93.For 4.9% of test requirements, even the policy’s shortest setting cannot fit the envelope.Source: U FEASIBILITY’S OWN SUB-CONSTRAINTS: A… · p. 31
  • Despite q reaching 0.98, no collapse follows in the feasibility substructure control because the head retains its ordering in the inherited envelope-failure bin.Envelope-only support is 33–201 requirements on the strongest arms, so those A_env estimates have correspondingly wide uncertainty.Source: U FEASIBILITY’S OWN SUB-CONSTRAINTS: A… · pp. 31–32

V THE SAME CRITIC AS A TRAINING TARGET AND AS A SEARCH OBJECTIVE

Across the engine studies, critic-guided search preserves coverage better than critic-based decoder training, ranking rules shape which requirement conjunct policies learn, and adding an unscored readiness conjunct improves extended-target outcomes at a cost to original-target performance.

V THE SAME CRITIC AS A TRAINING TARGET AND AS A SEARCH OBJECTIVE

  • Latent search reaches 0.342 on target and 0.937 supply, while critic-trained decoders lose coverage; the search comparison is not compute-matched.The authors do not claim that no decoder training against a critic can work.Source: V THE SAME CRITIC AS A TRAINING TARGET… · p. 32

W WHAT THE GENERATOR LEARNS TO CONDITION ON THE REQUIREMENT

  • At w = 0.1, the contrast-matched scalar score installs unit budget tracking (0.98 ± 0.02) and envelope conditioning (0.69 ± 0.06), the only rule that installs both.The authors conclude that the earlier plateau came from the wrong weights, not from scalar scores.Source: W WHAT THE GENERATOR LEARNS TO… · p. 34

X A PHYSICS CONJUNCT OUTSIDE EVERY SCORE

  • Among feasible bodies, 29–64% survive; the 7B gated policy has 0.31 survival, versus 0.54 for the sum policy, alongside a hotter median face (604°C versus 409°C).The readiness screen evaluates the liner, cooling channels, yield margin, and fatigue life; critic-trained decoders have no cooling zone and zero survival.Source: X A PHYSICS CONJUNCT OUTSIDE EVERY SCORE · pp. 34–35, Table 22 · p. 35
  • Adding readiness to the gate raises survival among feasible bodies from 0.40 to 0.55 and extended composed answer from 0.30 to 0.37, while original-target per-draw falls from 0.311 to 0.257.The readiness-aware gate retains mass tracking, but its policy still does not condition channel area on thrust.Source: X A PHYSICS CONJUNCT OUTSIDE EVERY SCORE · p. 36

Y SELECTION AMBIGUITY AND THE AUDIT CARD: PROOFS AND VALIDATION

This section combines formal results and validation analyses to examine how aggregate audits can favor worse component verifiers and how monotone-invariant audits leave composite selection and training preferences ambiguous. Natural-language controls and frozen ECG analyses assess the role of own-failure support and the actual selection or training interface, while qualifying transfer claims.

Y SELECTION AMBIGUITY AND THE AUDIT CARD: PROOFS AND VALIDATION

  • Any Pareto candidate can be selected by some strictly increasing score transform, but positive linear rescalings select only upper-hull Pareto points.Thus a statistic invariant to monotone reparameterization cannot identify the selected candidate; measured ECG pools also exhibit non-singleton Pareto sets and ranges of audit-equivalent Answer@16 values, though the supplied passages give no table cell values.Source: Y SELECTION AMBIGUITY AND THE AUDIT… · p. 36, Table 23 · p. 37
  • Deployability-head identified intervals averaged [0.472, 1] for Llama and [0.452, 1] for Phi, with 37% and 48% of pools individually erased.For the utility head, q was 0.013–0.018 and intervals were [0.894, 0.900] and [0.872, 0.877]; controlled-mixture RMSE rose from 0.010–0.021 at (q1, q2) = (0.2, 0.8) to 0.075–0.090 at (0.45, 0.55).Source: Y SELECTION AMBIGUITY AND THE AUDIT… · pp. 36–37

Z PREREGISTERED EMBEDDED-ECG STRESS TEST

  • The preregistered ECG stress test rejects the gate-free contrast criterion as a transferable recipe; Phi’s pair-weight runs instead favored a requirement-blind program.That program accounted for 97.1–100% of parsed outputs in all three Phi pair-weight seeds, versus 0–8% in Phi gate and inference arms; the study does not establish that gate-referenced contrast matching works beyond near-identical-pair settings.Source: Z PREREGISTERED EMBEDDED-ECG STRESS TEST · p. 39

AA.1 PROOF OF PROPOSITION 3

  • Passive own-failure auditing requires Θ(log(1/δ)/(ε2(1 −q))) labels up to universal constants, while two-mixture estimation recovers Aown by solving the aggregate-AUC equations.The upper bound requires m(1 −q) to scale with log(1/δ)/ε2; the lower bounds show the dependence on both own-failure frequency and target accuracy.Source: AA.1 PROOF OF PROPOSITION 3 · pp. 39–40

AA.2 PROOF OF PROPOSITION 6; AA.4 SELECTION AMBIGUITY VERSUS POOL SIZE

AA.5 AUDIT DECOMPOSITION ON POST-DPO POOLS

  • Re-injecting base-policy own failures into post-DPO mixtures estimated deployability Aown at 0.998–1.003 against a direct value of 1.000, checking the procedure rather than detecting degradation.The deployability head was near-perfect; the design mixes trained-pool inherited negatives with base-policy own failures at q1 = 0.2 and q2 = 0.8.Source: AA.5 AUDIT DECOMPOSITION ON POST-DPO… · pp. 40–41

AA.3 PROOF OF PROPOSITION 7(IV); AA.8 PREDICTION-REGISTERED CONVEX DPO RERANKER

AB.1 IDENTIFIED SET WHEN EVERY NEGATIVE’S SCORE IS OBSERVED; AB.3 RULE-LEVEL PAIR GEOMETRY FOR ALL TRAINED SCALAR RULES; AUC-CHANGE DECOMPOSITION AND ENGINE SELECTION CONTROLS

AUC-CHANGE DECOMPOSITION AND ENGINE SELECTION CONTROLS

AD.1 PROOF OF PROPOSITION 2

  • For y = F ∧ M, the full-outcome predictor’s audit advantage over the component oracle is q(Across − 1/2), and refresh data without own failures cannot identify how a head ranks them.On own failures, the full-outcome predictor has Aown = 1; on F = 1 it ranks by Pr[M = 1 | x].Source: AD.1 PROOF OF PROPOSITION 2 · p. 45

AD.4 REGISTERED PREDICTIONS AND OUTCOMES

AD.5 CLOSED-LOOP TRIGGER ANALYSIS

  • The frozen ECG threshold sweep shows that aggregate-audit triggers can deploy outcome-refresh heads, lowering canonical F-AUC while increasing constraint violations and reducing Answer@16.The no-refresh/component-specific control had F-AUC 1.000, violation .148, and Answer@16 .530; the threshold sweep was illustrative, not preregistered.Source: AD.5 CLOSED-LOOP TRIGGER ANALYSIS · pp. 46–47, Figure 12 · p. 47, Table 37 · p. 47

AD.6 NATURAL MULTI-LABEL SEMANTIC STRESS TEST: GOEMOTIONS

  • At q = .522, the aggregate audit preferred the worse annoyance/gratitude verifier (.681 vs .442) despite its lower canonical F-AUC (.570 vs .679).The second criterion was selected post hoc, so the intervals quantify stability on this fixed test split rather than a confirmatory discovery. Targeted replay of 80 own failures raised canonical F-AUC to .606 ± .008, versus .550 ± .004 for 80 random examples; all 303 own failures yielded .679.Source: Figure 13 · p. 48, AD.6 NATURAL MULTI-LABEL SEMANTIC… · p. 48

SELECTION INTERFACE: AUDIT EQUIVALENCE DOES NOT IMPLY EQUAL

  • Strictly increasing score transformations leave component AUCs unchanged yet can select any Pareto candidate; at K = 16, Answer@16 ranges over [0.33, 0.54] for Llama and [0.46, 0.60] for Phi.Positive rescalings select only the upper-convex-hull subset of the Pareto set. On matched 1.5B DPO pools, rescaling the feasibility logit leaves its AUC unchanged but moves raw-sum best-of-16 from 0.544 to 0.160.Source: SELECTION INTERFACE: AUDIT EQUIVALENCE… · pp. 49, 51

AD.7 ADDITIONAL CONSTRUCTED SEMANTIC-LANGUAGE CONTROL: AG NEWS

  • In AG News, the audit preferred the leaked verifier at q = .9 in every split, although its canonical F-AUC was 0.607 ± 0.025 versus 0.949 ± 0.011 for the specialist.For the specialist, audit AUC fell from 0.951 at q = 0 to 0.625 at q = .98; re-injecting 80 baseline World failures raised held-out Aown to 0.941 ± 0.011, versus 0.904 ± 0.014 with eight failures.Source: AD.7 ADDITIONAL CONSTRUCTED… · p. 49, Figure 15 · p. 50

TRAINING INTERFACE: A GOOD SELECTOR NEED NOT BUILD GOOD DPO DATA

  • Matched DPO reached 0.311 target/draw and 0.758 Answer@16 with gate pairs, versus 0.134 and 0.596 with unit-sum pairs; the preregistered ECG pair-weighting criterion was gate-equivalent in no family.Proposition 7 bounds objective and gradient differences by pair overlap, but low overlap does not certify endpoints; in the external stress test, gate and inference policies shared 79.6–99.5% of pairs and differed by at most 0.007.Source: TRAINING INTERFACE: A GOOD SELECTOR… · pp. 51–52, Table 39 · p. 52, Table 40 · p. 52

Related papers

Loading 2610.29029v1…