PaperlayerView exact revision on arXiv

Source-linked AI summary

Elucidating the Design Space of Regression-based Diffusion Reinforcement Learning

Toyota Li, David Zhao, Alan Zhao

arXiv:2609.33444v1cs.LGcs.CV

TL;DR

Regression-based diffusion RL methods lacked a principled account connecting their reward-weighting choices and the design decisions that affect performance. This paper derives them as divergence-constrained reward maximization and examines the resulting design space. DiffusionRFT combines sparsemax projection with a variance-minimizing baseline and achieves the strongest reported configuration.

  • Problem

    DiffusionNFT, FlowAWR, and RAM use reward-weighted regression, but their relationships and performance-driving design choices lacked a principled account.

    Source: §1 · p. 1

  • Method

    The paper derives the methods as reward maximization on a frozen rollout buffer under a divergence penalty, with the convex generator determining the tilt.

    Source: §1 · p. 1

  • Results

    DiffusionRFT, combining sparsemax projection and a variance-minimizing baseline, is the safest and strongest tested configuration, reaching 0.983 evaluation reward.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    The empirical design-space results favor sparsemax and a variance-minimizing baseline, while the tested frozen-anchor approach has no safe operating point.

    Source: §1 · p. 2

    The group estimator substitutes the endpoint normalizer for the posterior mean, which differs unless the posterior equals the prior.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

A nascent family of methods that forgoes the policy gradient and reweights a supervised regression instead has garnered momentum in reinforcement learning for diffusion and flow models. DiffusionNFT, FlowAWR, and RAM are representative regimes with contrasting motivations. It is yet opaque what, if anything, they share. We substantiate that each is the solution of one divergence-constrained reward-maximization problem, and they are differentiated only by the convex generator that defines the constraint. Under the unified modeling framework, we unravel the relaxations that prior art made during building the advantage-embedded regression target: approximating the KKT condition and posterior normalizer for the linear and exponential tilt shapes DiffusionNFT and FlowAWR respectively, while preserving the exact sparsemax projection onto the probability simplex for linear tilt leads to another superior model type in this work. Beyond the theoretical underpinnings, we further empirically investigate the design space and shed light on the training recipe for regression-style diffusion RL. Retaining the merits discovered during our exploration gives rise to DiffusionRFT, our paradigm that converges faster, trains more stably, and attains the top performance.

1 INTRODUCTION

DiffusionNFT, FlowAWR, and RAM preserve supervised regression while incorporating reward, but their relationships and performance-driving design choices were unclear. The paper derives them within a divergence-constrained framework and finds that exact sparsemax projection, combined with a variance-minimizing baseline, yields the strongest configuration.

1 INTRODUCTION

  • Policy-gradient and direct reward-backpropagation methods require long denoising trajectories, exhibit high variance, and depart from the MSE loss used in pre-training.Post-training with external reward is important for human-preference behaviors that pre-training dictates only weakly, including prompt adherence and text rendering.Source: §1 · p. 1
  • The paper unifies these methods as reward maximization on a frozen rollout buffer under a divergence penalty, with the convex generator distinguishing their tilts.Reverse KL yields FlowAWR’s exponential tilt; Pearson χ2 yields the linear tilt associated with DiffusionNFT.Source: §1 · p. 1
  • The exact Pearson χ2 optimum is a sparsemax projection, while DiffusionNFT’s dense linear form drops the non-negativity constraint and collapses when its exactness condition fails.Retaining the projection repairs collapse and achieves higher rewards.Source: §1 · p. 2
  • DiffusionRFT combines the exact projection with the baseline, producing the safest and strongest tested configuration at 0.983 evaluation reward.The paper characterizes this design as faster-converging and more stable than the alternatives explored.Source: §1 · p. 2
  • Empirical results favor sparsemax, and a variance-minimizing baseline stabilizes and improves all three tilts.A frozen anchor has no safe operating point in the tested settings; decoupling renoising timesteps steers loss toward noise levels the sampler never visits.Source: §1 · p. 2

2 REWARD MAXIMIZATION UNDER A DIVERGENCE CONSTRAINT

The section frames policy improvement on a frozen rollout buffer as reward maximization regularized by divergence from the behavior policy. For diffusion models, an endpoint reweighting becomes a velocity-field correction, connecting that objective to supervised regression.

2.1 POLICY IMPROVEMENT ON A FROZEN ROLLOUT BUFFER

  • On a frozen rollout buffer, policy improvement raises expected reward while limiting divergence from the behavior policy so the batch remains informative.Within an update, the sampling distribution is fixed, making the update an offline optimization problem.Source: §2.1 · p. 2
  • Expressing divergence through the density ratio confines the regularizer’s effect to the convex generator; reverse KL gives a closed-form, reward-monotone exponential tilt.The optimum redistributes mass toward high-reward actions rather than assigning mass to unvisited actions.Source: §2.1 · p. 2
  • For diffusion models, the tilted endpoint law specifies the distribution an update should target, without depending on how the batch was explored.A velocity field, rather than an endpoint density, parameterizes the flow model.Source: §2.1 · p. 2

2.2 FROM A TILTED ENDPOINT LAW TO A VELOCITY FIELD

  • Flow matching reduces generative modeling to supervised regression under plain MSE, motivating forward-process RL methods that inject reward while preserving this property.The target velocity is the conditional expectation of the flow-matching velocity given the noised sample and prompt.Source: §2.2 · p. 3
  • Reweighting endpoints becomes an additive correction to the sampling field, and the exact target’s score gradient cancels the endpoint partition function.The remaining correction is a posterior expectation, not a per-sample quantity, so converting it into a trainable target is still necessary.Source: §2.2 · p. 3

3 FROM A HEURISTIC LINEAR TILT TO THE EXACT SIMPLEX PROJECTION

The section recovers DiffusionNFT and FlowAWR as regression updates arising from divergence-constrained reward maximization, then identifies the exact constrained linear-tilt solution as sparsemax. The methods differ by the divergence generator and by approximations used to construct their regression targets.

3.1 A REGRESSION FORM FOR THE DIFFUSIONNFT UPDATE

  • A single DiffusionNFT update is a least-square regression onto an advantage-weighted target, and its linear reward weighting is distinct from FlowAWR’s exponential tilt.DiffusionNFT parameterizes implicit positive and negative branches through their velocity difference; its weight is linear in reward and βC acts as temperature.Source: §3 · p. 3, §3.1 · pp. 3–4
  • For a general endpoint reweighting, the velocity correction uses the relative tilt κ = w/z, comparing a sample’s weight with the posterior mean among endpoints compatible with the same noised state.Because the residual has zero conditional mean, subtracting a constant baseline leaves the target unchanged; b = 1 is a centering convention, not a derivation.Source: §3.1 · p. 4
  • DiffusionNFT’s group-standardized reward and fixed constant produce its heuristic dense linear weighting; the paper asks which divergence yields that tilt and whether it is the constrained optimum.Its linear form is not the reverse-KL solution, which instead yields an exponential tilt.Source: §3.1 · p. 4

3.2 RETURNING TO THE KL CONSTRAINT: THE EXPONENTIAL TILT

  • The reverse-KL solution is an exponential tilt, and its group estimator recovers the FlowAWR objective.Replacing the posterior normalizer with an endpoint group estimate yields a rescaling centered at one, with variance bounded by E[A^2]; it preserves the optimal field’s direction and changes only step length.Source: §3.2 · pp. 4–5
  • The group estimator uses independent endpoint samples to estimate the endpoint normalizer Z, not the posterior mean z conditioned on a noised state.The two coincide only when posterior and prior agree; at the latent size used here, posterior reweighting concentrates on the generating endpoint and would annihilate the update.Source: §3.2 · p. 5
  • The Pearson χ2 constraint requires normalized, nonnegative relative densities, and its KKT solution truncates the linear tilt at a reward threshold fixed by normalization.Samples below the threshold receive zero mass; the resulting sparsemax projection is the exact constrained solution, unlike the dense interior branch.Source: §3.2 · pp. 5–6
  • The dense linear form is exact only when its non-negativity constraint is inactive, a condition that fails on 86.8% of the dense run’s steps.DiffusionNFT’s heuristic meets its stated bound only with designated hyperparameters, whereas sparsemax is feasible at every temperature and group size.Source: Figure 1 · p. 6, §3.2 · p. 6
  • The three algorithms differ by their weight choice: reverse KL gives softmax, Pearson χ2 gives sparsemax, and DiffusionNFT adopts the latter’s dense interior branch heuristically.The paper identifies these as entropic and Euclidean projections of the same linear objective.Source: §3.2 · p. 7

4 DESIGN SPACE NAVIGATION

This section varies regression parameterization, reward shaping, anchor choice, and renoising-timestep law, then identifies a stable combination and practical training choices. Its experiments favor the projected linear tilt with a variance-minimizing baseline, while showing that frozen anchors lock or collapse and trajectory-coupled renoising is better conditioned.

4 DESIGN SPACE NAVIGATION

  • Equation 22 varies the regression variable and noise weighting, reward tilt and baseline, rolling versus frozen anchor, and the renoising-timestep law.The experiments use GenEval and Stable Diffusion 3.5-Medium at 512 × 512 resolution.Source: §4 · p. 7, §4.1 · p. 7

4.2 ABLATION STUDIES

  • The x-prediction loss geometry yields a +0.34 gain in terminal training reward, so the authors retain it rather than treating it as another tuning knob.At a matched horizon, v-prediction has the largest advantage and movement budget but the least reward; its regression loss is an order of magnitude higher.Source: §4.2 · p. 8
  • 0.983 evaluation reward: DiffusionRFT combines projection and the variance-minimizing baseline, which stabilize all three tilts and together form the safest, strongest configuration.The two strategies are complementary: projection keeps the linear tilt in its optimum region, while the baseline advances the envelope of any tilt.Source: §4.2 · p. 8
  • Replacing the rolling anchor with a frozen pretrained anchor makes three variants stagnate below 0.25 training reward; larger updates escape only temporarily before collapse.Small updates lock, while large updates diverge; with a rolling anchor, re-centering keeps displacement of order one at every epoch.Source: §4.2 · pp. 8–9
  • Trajectory-coupled renoising reaches evaluation rewards of 0.972 at K = 1 and 0.975 at K = 8, versus 0.964 and 0.953 under uniform renoising; increasing K is neutral, while the timestep law matters.The uniform-law K = 8 run loses a further 0.086 before the budget ends; trajectory-coupled K = 1 is retained as cheaper and better conditioned.Source: §4.2 · p. 9

5 CONCLUSION

The paper unifies regression-based diffusion reinforcement learning and reports a training recipe built around an exact projected linear tilt and complementary stabilization choices. It situates this work amid diffusion models’ limited direct modeling of sample-level preferences and the still-undisclosed basis of DiffusionNFT’s reward-integrated regression target.

5 CONCLUSION

  • Regression-based diffusion reinforcement learning has a unified formulation whose divergence generator selects the tilt: exponential for FlowAWR and linear for DiffusionNFT, with the exact sparsemax solution motivating DiffusionRFT.The conclusion favors the projected linear tilt, a variance-minimizing baseline, x-prediction with loss calibration, an online cumulative anchor, and backward-aligned renoising.Source: §5 · p. 9

A.1 DIFFUSION GENERATIVE MODELING

  • Diffusion’s supervised MSE training objective does not straightforwardly model sample-level human preferences such as visual aesthetics and physical plausibility.Source: §A.1 · pp. 14–15

A.2 DIFFUSION REINFORCEMENT LEARNING

  • The paper frames diffusion RL against the difficulty of tractable likelihoods and notes that DiffusionNFT’s underlying insight for its reward-integrated regression target was not disclosed.It links DiffusionNFT to constrained reward maximization and explores the design space of advantage-weighted regression.Source: §A.2 · p. 15

B DERIVATION OF THE REWEIGHTED VELOCITY FIELD

The derivation carries an endpoint reward tilt through the forward noising kernel, differentiates the ratio of noised marginals, and converts the resulting posterior score into a sample-computable residual. It also identifies baseline choices that preserve a zero-mean reweighting or minimize its second moment.

B DERIVATION OF THE REWEIGHTED VELOCITY FIELD

  • The endpoint tilt becomes a tilt on noised marginals weighted by the posterior mean of the endpoint weight, z, rather than by the weight itself.The noising kernel enters at this step, and z cannot be evaluated per sample.Source: B DERIVATION OF THE REWEIGHTED… · p. 15
  • Differentiating the noised-marginal ratio and using the conditional score trades the posterior score for the computable residual ut − vold.Here ut is the conditional target and vold is the reference field; the residual has zero conditional mean under the reference process.Source: B DERIVATION OF THE REWEIGHTED… · pp. 15–16
  • The numerator’s differentiation under the integral is legitimate when the tilt is pold-integrable and the Gaussian kernel dominates, conditions stated to hold here because w ≥ 0 has unit mean.Source: B DERIVATION OF THE REWEIGHTED… · p. 15
  • The baseline b = 1 makes the reweighting zero-mean, while b = b* instead minimizes the second moment.Source: B DERIVATION OF THE REWEIGHTED… · p. 16

C THE LINEAR WEIGHT IMPLIED BY THE DIFFUSIONNFT BRANCHES

The appendix derives DiffusionNFT’s linear reward weight from its binary optimality branch. At velocity level, replacing a posterior reward normalizer with its endpoint counterpart yields the implemented group-centered advantage form, with a rescaling of step length but unchanged direction.

C THE LINEAR WEIGHT IMPLIED BY THE DIFFUSIONNFT BRANCHES

  • DiffusionNFT’s binary optimality variable makes its positive branch the sampling law conditioned on o = 1, which corresponds to a tilt weighted by reward up to normalization.Source: C THE LINEAR WEIGHT IMPLIED BY THE… · p. 16
  • At velocity level, the exact correction uses the posterior reward mean α(xt, c) as its normalizer, and the free baseline follows from the residual’s zero-mean property.Source: C THE LINEAR WEIGHT IMPLIED BY THE… · p. 17
  • Replacing α with the endpoint mean yields a linear reward weight without a posterior quantity; the resulting factor rt rescales step length but leaves direction unchanged.Setting the baseline to the endpoint mean and scaling gives the group-centered advantage equation at temperature βCσ.Source: C THE LINEAR WEIGHT IMPLIED BY THE… · p. 17

D NORMALIZATION AT THE ENDPOINT AND INTERMEDIATE LEVEL

The three tilt families share a regression step whose target uses endpoint normalization in place of a posterior normalizer, rescaling the ideal displacement. The resulting factor preserves direction, has a reward-group variance bound, and becomes less dispersed with increasing noise, while the implemented group estimate carries an additional finite-group bias.

D NORMALIZATION AT THE ENDPOINT AND INTERMEDIATE LEVEL

  • For exponential, projected-linear, and dense-linear tilts, the shared step uses the endpoint normalizer in place of the posterior normalizer.Their respective weights and normalizers are exp(R/γ) with Z(c), [R−τ]+/γ with Z=1, and R with Z=R̄(c).Source: D NORMALIZATION AT THE ENDPOINT AND… · p. 17
  • Replacing z(x_t,c,t) with Z(c) scales the conditional displacement by r_t, preserving its direction; changing the baseline cannot alter this discrepancy.The baseline controls variance, not this bias.Source: D NORMALIZATION AT THE ENDPOINT AND… · pp. 17–18
  • The self-normalized group estimate Ẑ introduces an O(1/G) bias that is non-negligible at the default G=24, so the bound for Z does not fully describe diagnostics for Ẑ.The conditional expectation does not pass through the random denominator when Ẑ is used.Source: D NORMALIZATION AT THE ENDPOINT AND… · p. 18
  • Var(r_t) is bounded by E[A²], with the bound tightest at t→0 and the substitution asymptotically exact at the noisy end.Gaussian smoothing makes the density ratio less dispersed as noise increases; a trajectory-coupled buffer concentrates timesteps where the discrepancy is smaller.Source: D NORMALIZATION AT THE ENDPOINT AND… · p. 19
  • The factor r_t can be read as the tilted-to-sampling density ratio at noise level t, avoiding a likelihood-ratio estimate but differing from plain AWR by this factor.An alternative logarithmic route removes the endpoint normalizer through within-group centering but requires estimating a diffusion-model log-ratio with an ELBO surrogate.Source: D NORMALIZATION AT THE ENDPOINT AND… · p. 19

E WHY THE PER-SAMPLE TARGET MAY REPLACE THE EXACT ONE

A tractable per-sample target replaces an exact posterior expectation in the regression objective. Its conditional expectation recovers the exact field, so the regression minimizer is unchanged; stochastic-target variance affects gradient estimation, not that optimum.

E WHY THE PER-SAMPLE TARGET MAY REPLACE THE EXACT ONE

  • The exact field requires averaging over all endpoints consistent with a given x_t, whereas the implemented target uses one endpoint draw.The sampled target is computable from the pair (x_0,x_t).Source: E WHY THE PER-SAMPLE TARGET MAY… · p. 20
  • Conditioning the per-sample target on x_t recovers v*, so the tractable and exact regression objectives differ only by a θ-independent constant and have the same minimizer.Their gradients coincide.Source: E WHY THE PER-SAMPLE TARGET MAY… · p. 20
  • The stochastic target’s conditional variance sets gradient-estimator variance but does not bias the optimum, and the argument applies to exponential and linear tilts.The regression uses reweighted rollouts without likelihood ratios, stochastic sampling, or differentiation through reward.Source: E WHY THE PER-SAMPLE TARGET MAY… · p. 20

F DERIVATION OF THE χ2 OPTIMUM AND ITS GROUP ESTIMATOR

The Pearson χ²-constrained reward optimum is a nonnegative truncated linear tilt, with a dense linear form only under a stated reward condition. On a finite group, the exact solution is a sparsemax simplex projection that can be computed by sorting.

F DERIVATION OF THE χ2 OPTIMUM AND ITS GROUP ESTIMATOR

  • The χ²-constrained optimum is ρ⋆=(R−τ)+/γ, with τ chosen to make the weights normalize; the KKT conditions set weights to zero below the threshold.The solution is unique, and the dense linear form is exact only when ess inf R > R̄−γ.Source: F DERIVATION OF THE χ2 OPTIMUM AND ITS… · pp. 20–21
  • The dense linear tilt is exact only when the truncation is inactive, and the group-size guarantee is G≤1+(βC)²; exceeding it removes the guarantee but does not itself prove infeasibility.The bound is tight in the stated one-hot-group case; the default βC=5 and G=24 lies within it.Source: F DERIVATION OF THE χ2 OPTIMUM AND ITS… · p. 21
  • The projected tilt has endpoint normalizer Z=1 exactly, so the group estimator needs no estimated endpoint partition function, although its posterior normalizer remains uncomputed.The shared endpoint-normalization substitution makes the implemented advantage ρ_i−b.Source: F DERIVATION OF THE χ2 OPTIMUM AND ITS… · pp. 21–22
  • For a finite group, the exact weights are the Euclidean projection onto the probability simplex, equivalently the sparsemax operator.Sorting rewards yields the solution in O(G log G).Source: F DERIVATION OF THE χ2 OPTIMUM AND ITS… · pp. 21–22

G IMPLEMENTATION AND EXPERIMENTAL DETAILS

Under the Pickscore reward, DiffusionRFT converges faster and reaches a higher performance ceiling than DiffusionNFT. The appendix also specifies the inherited training setup, evaluation protocol, and diagnostic quantities used to compare runs.

G IMPLEMENTATION AND EXPERIMENTAL DETAILS

  • The setup is inherited from DiffusionNFT and may place the authors’ methods at a disadvantage.Source: G IMPLEMENTATION AND EXPERIMENTAL… · p. 22
  • Runs use SD3.5-Medium with rank-32 LoRA adapters, GenEval train and evaluation splits, and 10-step CFG-free rollouts.The backbone is frozen; the training pool has 50,000 records and evaluation uses 2,212 records across 553 prompts.Source: G IMPLEMENTATION AND EXPERIMENTAL… · p. 22
  • Evaluation uses a 40-step solver every 10 epochs and runs continue until plateau or collapse, up to 1,000 epochs.The higher evaluation solver budget is intended to prevent apparent gains from adapting only to the coarse rollout sampler.Source: G IMPLEMENTATION AND EXPERIMENTAL… · p. 22
  • The tracked diagnostics include effective step size, policy–EMA gap, implicit KL, and the fraction of prompt groups with zero reward variance.The implicit KL is proportional to per-step KL under Gaussian transitions, while zero-variance groups indicate reward saturation and loss of usable learning signal.Source: G IMPLEMENTATION AND EXPERIMENTAL… · pp. 22–23
  • On Pickscore, DiffusionRFT converges faster and reaches a higher performance ceiling than DiffusionNFT.Figure 6 reports training reward (a) and validation reward (b), with consistent superiority for DiffusionRFT.Source: Figure 6 · p. 23, G IMPLEMENTATION AND EXPERIMENTAL… · p. 23

H.1 THE PREDICTION SPACE AND THE IRREDUCIBLE RESIDUAL

Regression loss has an irreducible floor that varies with prediction space, so raw loss values need not measure fit quality comparably. The section connects this floor to a prediction-space weighting and contrasts that analysis with the dense linear tilt’s constraint violations and collapse.

H.1 THE PREDICTION SPACE AND THE IRREDUCIBLE RESIDUAL

  • The irreducible residual is conditional variance in the reward-weighted target, which no parameterization or amount of training removes.It sets the loss floor because the network can match only the target’s conditional mean given the noised state.Source: §H.1 · p. 23
  • In v-space, the prediction-space factor is t^-2, averaging 2.08 over the nine retained timesteps; x- and v-objectives share a minimizer but weight noise levels differently.The approximation to the floor is relevant when A^2 and squared residual norm q are close to uncorrelated; the measured correlation median is 0.013.Source: §H.1 · pp. 23–24
  • The v-space run ends with 8.9× the residual and reward 0.442 against 0.693 for x-space, despite similar movement and more signal.The movement is 0.190 against 0.177 in squared distance to the reference field, while E|A| is 0.538 against 0.420.Source: §H.1 · p. 24
  • The 8.9× residual ratio does not identify excess error because the runs have different floors, which could account for the entire ratio.The estimated floor ratio is bounded by [1.45, 8.47], and the measured ratio is not a pure measure of worse fit.Source: §H.1 · p. 24

H.2 WHY THE DENSE LINEAR TILT COLLAPSES.

  • The centered dense linear tilt violates its positivity condition on 86.8% of training steps and ends 0.185 below its peak, while sparsemax ends 0.001 below its peak.The authors posit that collapse reflects a violated density constraint; retaining the projection restores stability in the tested comparison.Source: §H.2 · p. 25

H.3 HOW THE OPTIMAL BASELINE COMES INTO PLAY.

The section derives an additional baseline shift that minimizes the regression target’s second moment, then measures its size and tests what accounts for the dense linear tilt’s stabilization. A constant shift recovers most of the control variate’s reward gap in the reported comparison.

H.3 HOW THE OPTIMAL BASELINE COMES INTO PLAY.

  • The additional shift b* minimizes the second moment of the regression target, and its variance reduction is governed by covariance between advantage and squared residual.The reduction vanishes when those quantities are uncorrelated; the comparison is incremental to the centering baseline b = 1.Source: §H.3 · p. 25
  • In practice, |b*|/E|A| is about 1%, implying a relative second-moment reduction of at most 10^-4 under the stated near-uncorrelation condition.The matched-budget reward contribution is +0.011 ± 0.002, too small for the second-moment reduction alone to explain stabilization.Source: §H.3 · p. 25
  • At matched 390 epochs, the constant baseline shift reaches terminal reward 0.941 versus 0.718 without a control variate and 0.964 with the optimal one.The constant shift recovers 90% of the reward gap despite having no per-batch adaptivity or variation.Source: §H.3 · pp. 25–26
  • The projection removes the sensitivity seen near the stability boundary without requiring a control variate; the projected tilt holds its peak within 0.001.The compared configurations remain infeasible under the dense-tilt condition, and a 0.36% advantage-scale perturbation changes terminal reward from 0.718 to 0.941.Source: §H.3 · p. 26

H.4 WHAT CONFOUNDS ANCHOR-LOCK WITH RAM’S ESCAPE.

The anchor-lock and RAM escape depend on both target displacement and anchor quality; increasing the multiplier can raise reward but is followed by collapse. For decoupled buffers, the key difference is noise-level support: uniform renoising evaluates levels the sampling trajectory does not visit.

H.4 WHAT CONFOUNDS ANCHOR-LOCK WITH RAM’S ESCAPE.

  • RAM’s 100× advantage multiplier and CFG-guided anchor confound target displacement with anchor quality; unit scaling is roughly 40× too weak, while CFG raises base-model reward from 0.164 to 0.410.RAM guides both sampling and regularization, whereas the compared grid is CFG-free.Source: §H.4 · p. 26
  • RAM’s official run reaches 1.000 training reward after multiplier increases, but both tested high-multiplier implementations later collapse to zero.As reward saturates, γ contracts while the target displacement demand grows with a fixed anchor, making the multiplier too large late in training.Source: §H.4 · p. 26
  • Frozen-anchor runs retain stronger reward signal, yet remain near the reference policy: implicit KL is 1.9–4.5 × 10^-4 versus 1.5 × 10^-1 for rolling-anchor runs.Frozen speed stays within 2.6–3.7×10^-6, while rolling speed exceeds that ceiling at epoch 8; the frozen policy remains in a neighborhood rather than slowly drifting out.Source: §H.4 · p. 26

H.5 WHY THE DECOUPLED BUFFER IS WORSE.

  • Trajectory-coupled sampling uses t ≥ 0.43, while uniform renoising puts 43% of loss evaluations below that support, including when stratified.The shifted grid’s nine loss timesteps never fall below t = 0.43, so decoupling changes support, not merely noise-axis weighting.Source: Figure 7 · p. 27, §H.5 · p. 27

H.4 WHAT CONFOUNDS ANCHOR-LOCK WITH RAM’S ESCAPE.

  • The official RAM run’s output-delta norm grows 7.2× (0.031 → 0.224) as reward falls from 0.98 to 0.92, consistent with runaway target displacement rather than loss of signal.A rolling anchor follows the policy, preventing the target from separating indefinitely from the sampling distribution.Source: §H.4 · p. 27

Related papers

Loading 2609.33444v1…