PaperlayerView exact revision on arXiv

Source-linked AI summary

Timestep Weighting: A Hidden Key to Effective ELBO-Based Flow-Matching RL

Qinwei Ma, Jingzhe Shi, Simin Fan, Ling Li, Alex Lamb

arXiv:2609.32665v2cs.LGcs.AIcs.CV

TL;DR

Timestep weighting in ELBO-based flow-matching RL is often inherited from pretraining configurations, although reward-driven updates depend on the policy’s current behavior. The paper studies weighting preferences and cross-noise gradient coordination, then tests strategies that improve performance beyond conventional target choices.

  • Problem

    Timestep weighting remains under-researched and is often inherited from pretraining configurations, despite being an explicit choice for distributing effort across noise levels during reward-driven learning.

    Source: §1 · p. 1

  • Method

    The paper studies controlled image and robotics experiments, analyzes gradient coordination across noise levels, and evaluates static weighting, budgeted profile selection, and dynamic schedules.

    Source: §1 · p. 2

  • Results

    Useful weighting varies with reward, policy behavior, and learning stage; high-to-low-noise annealing beats the best tested fixed-α weightings on three image rewards.

    Source: §1 · p. 2, §6.1 · p. 9

  • Takeaways & Limitations

    Timestep weighting is an important ELBO-based flow-matching RL design choice, motivating methods that select and adapt allocations throughout learning.

    Source: §1 · p. 2, §6.1 · p. 9

    The study mainly ablates restricted reweighting families, and its explanation is primarily local and gradient-based rather than an analysis of final performance by noise bin.

    Source: §6.2 · pp. 9–10

Abstract

from arXiv · show

ELBO-based reinforcement learning offers a sampler-agnostic approach to fine-tuning flow matching models with reward feedback. Timestep weighting in ELBO-based RL has large impact on performance, and it also provides a unified view (as we show in this work) to understand prediction losses heuristically chosen in prior work, yet it remains under-researched and is often chosen to inherit pretrain configs. We investigate impacts and dynamics of timestep weighting in ELBO-based RL. We show that effective weighting depends on both the reward landscape and stage of learning. (1) Through experiments on controlled CIFAR image generation, complemented by robotics, we investigate how weighting impacts reward-driven updates across noise levels. (2) Through gradient analysis, we reveal distinct patterns of cross-noise coordination across tasks and their evolution during training. These findings motivate the hypothesis that useful weighting depends on the gap between the policy's current behavior and the behavior favored by the reward. (3) Guided by this analysis, we study simple static weighting, budgeted profile selection, and dynamic schedules that improve performance beyond conventional target choices. Our results establish timestep weighting as an important design choice for flow-matching RL and motivate further research into methods that choose and adapt it throughout learning.

1 INTRODUCTION

The paper treats timestep weighting as an explicit design choice because reward-driven updates depend on both the reward and the policy’s current behavior. It analyzes these dependencies and tests static, selection-based, and dynamic strategies.

1 INTRODUCTION

  • Prediction targets such as noise, velocity, and the clean sample implicitly weight errors differently across noise levels.The paper frames this implicit weighting as a design choice beyond inheriting pretraining configurations.Source: §1 · p. 1
  • For CIFAR-10 image rewards, global rewards intuitively favor more high-noise emphasis, while the deeper analysis relates preferences to behavioral gaps.Figure 1 summarizes validation peaks across six image rewards under different schedules.Source: Figure 1 · p. 2, §1 · p. 2
  • Reward-driven updates depend on both the reward landscape and the policy’s current behavior, with useful allocations changing across tasks and learning stages.The analysis links this pattern to the gap between current and reward-preferred behavior.Source: §1 · p. 2
  • High-to-low-noise annealing outperforms the best tested fixed-α weightings on three image rewards, demonstrating value in stage-dependent allocation.The paper also studies static softmax weights and successive halving to select fixed profiles.Source: §1 · p. 2

2 PRELIMINARIES: TIMESTEP-WEIGHTED POLICY OPTIMIZATION

The preliminaries express prediction-target losses as timestep-weighted velocity losses, separate profile shape from overall scale, and describe an ELBO-surrogate policy-optimization procedure.

2.1 TIMESTEP WEIGHTING AND PREDICTION TARGETS

  • Noise-to-data interpolation defines the path, and noise, velocity, and clean-sample targets correspond to timestep factors t^2, 1, and (1 − t)^2.These identities apply to predictions induced by one velocity output, not separately parameterized networks.Source: §2.1 · pp. 2–3

2.2 SEPARATING PROFILE SHAPE FROM OVERALL SCALE

  • The common-coordinate timestep-weighting profile can be applied explicitly or through sampling, while calibration matches initial whole-loss RMS to separate profile shape from overall scale.The bank stays fixed as model losses and gradients evolve; proportional effective profiles have identical calibrated losses on the same bank.Source: §2.1 · p. 3, §2.2 · p. 3

2.3 ELBO-SURROGATE POLICY OPTIMIZATION

  • The ELBO-surrogate procedure retains advantage estimation and PPO clipping, samples weighted times, recomputes paired losses, and updates the policy using the clipped objective.The CIFAR implementation uses GRPO advantages and applies timestep weighting through qk sampling.Source: §2.3 · pp. 3–4

3 REWARD- AND TASK-DEPENDENT WEIGHTING PREFERENCES

Experiments vary timestep weighting in CIFAR-10 post-training and from-scratch robot control. The preferred allocation differs across image rewards, robot tasks, and native objective families.

3.1 EXPERIMENTAL SETUP AND WEIGHTING PROFILES

  • CIFAR-10 post-training uses a shared pretrained unconditional U-Net, while robot policies learn from scratch through online interaction.The evaluated implementations are FPO on MuJoCo Playground and FPO++ on Isaac Lab.Source: §3.1 · p. 4
  • The scalar α varies emphasis along a high-to-low-noise axis, with reference profiles tied to the native objectives and setting.For CIFAR and FPO++, α = 0 and 1 correspond to clean-sample and velocity-derived profiles; native FPO uses a noise-prediction reference at αϵ = 1.Source: §3.1 · p. 4

3.2 DIFFERENT REWARDS AND TASKS FAVOR DIFFERENT PROFILES

  • Automobile-related and semantic CIFAR rewards generally favor stronger high-noise emphasis, whereas Edge and Digit7 favor profiles closer to clean-target weighting.Auto-A35 has a broad high-noise plateau, while Edge peaks near the clean-target reference.Source: §3.2 · p. 4
  • Acrobot, Walker, and Cartpole favor higher-noise emphasis than their native references, while Spot favors its native allocation or a modest shift toward lower noise.Figure 2 compares mean per-run peak returns for fixed α against each method’s native target.Source: §3.2 · p. 4, Figure 2 · p. 5

4 BEHAVIORAL GAPS AND EVOLVING CROSS-NOISE COORDINATION

The analysis relates useful timestep weighting to the corrections required by the current policy and how those corrections coordinate across noise levels. These relationships differ between behavioral refinement and acquisition, and evolve during training.

4.1 BEHAVIORAL GAPS SHAPE CROSS-NOISE LEARNING ACROSS TASKS AND REGIMES

  • Reward-relative fields compare the current policy with its reward-favored distribution, and weighting emphasizes different smoothed views of the resulting correction.The relevant distinction is whether the policy must acquire or refine behavior favored by the reward.Source: §4.1 · p. 5
  • Cross-batch gradient cosines measure reproducibility within noise bins and agreement between bins, distinguishing stable directional signal from raw gradient magnitude.Independent on-policy batches help separate this structure from agreement induced by shared examples.Source: §4.1 · p. 5
  • Successful CIFAR post-training shows selective cross-noise agreement, whereas acquiring new behavior can benefit from broader coordination across noise levels.The better Go2 profile preserves positive coordination where its poorer counterpart has opposing noise groups; Digit7 is intermediate.Source: §4.1 · p. 5, Figure 3 · p. 6
  • The behavioral-gap hypothesis holds that useful weighting depends on how much reward-favored behavior the policy must acquire, rather than on the reward alone.The analysis distinguishes refinement of existing behavior from acquisition of missing behavior.Source: §4.1 · pp. 5–6

4.2 STAGE-DEPENDENT COORDINATION AND LEARNING ALLOCATION

  • During post-training, learning shifts toward middle and low noise; from-scratch learning begins with broader cross-bin agreement that later weakens or reorganizes.Even a fixed high-noise-emphasizing profile can support the post-training shift, as early high-noise gains saturate; the changing coordination motivates different schedules across regimes.Source: §4.2 · p. 6

5 STRATEGIES FOR CHOOSING AND ADAPTING TIMESTEP WEIGHTS

The paper develops static, budgeted-selection, and stage-adaptive strategies for choosing timestep weights. Their results show that useful allocations vary by reward and learning stage, and that adaptive schedules can outperform fixed profiles.

5.1 STATIC SOFTMAX COMPLEMENTS TARGET AND FIXED-PROFILE CHOICE

  • Static softmax improves on the best tested fixed exponent for CLIP, Mixed, and Digit7, while other rewards show plateaus or little change.Greater concentration is not uniformly better.Source: §5.1 · p. 7, Figure 5 · p. 8

5.2 SUCCESSIVE HALVING SCREENS FAVORABLE REGIONS EFFICIENTLY

  • Successive halving selects profiles that perform as well as or better than native-target references across image and robotic rewards at 2–3 full-run budgets.Task-dependent preferences motivate screening profiles through observed performance; the text also notes that preferences appear relatively stable across training stages.Source: §5.2 · p. 7, Figure 6 · p. 8

5.3 STAGE-AWARE WEIGHTING IMPROVES THE LEARNING TRAJECTORY

  • On Mixed, CLIP, and Digit7, dynamic schedules exceed their corresponding best tested fixed-α peaks; on Acrobot and Walker, they exceed fixed endpoints and best tested fixed-sweep references.CIFAR schedules shift emphasis from larger to smaller noise, while robotics schedules shift in the opposite direction.Source: §5.3 · p. 8, Figure 7 · p. 9

6 CONCLUSIONS AND DISCUSSIONS

The paper concludes that timestep weighting is consequential in ELBO-based RL, with preferences varying across rewards, tasks, and learning stages. It presents simple reweighting strategies while noting that the analysis and tested weighting families are limited in scope.

6.1 CONCLUSIONS

  • Timestep-weighting preferences differ across rewards and tasks, and gradient alignment varies with behavioral gap and training stage.From-scratch tasks or rewards far from the pretrained model show a co-learning phase with higher cross-bin alignment; the authors propose static softmax, successive halving, and α scheduling as reweighting strategies.Source: §6.1 · p. 9

6.2 LIMITATIONS AND FUTURE WORK

  • The study mainly ablates restricted reweighting families, and its explanation is primarily gradient-based and local.The authors note that some robotics gains over fixed-α configurations are small and suggest further work on timestep scheduling.Source: §6.2 · pp. 9–10

A RELATED WORK

  • The related-work discussion distinguishes this study’s reward- and stage-dependent allocation from pretraining methods targeting a fixed data distribution.It also places the results within ELBO/matching-loss approaches and notes that the discussed surrogate constructions are prominent, not exhaustive.Source: A RELATED WORK · p. 15

B.1 PREDICTION-TARGET CONVERSION

  • Converting noise- and clean-sample prediction errors to velocity coordinates yields target-dependent factors, while architectural parameterization and preconditioning may add factors.The robotics native-ϵ FPO baseline retains an intrinsic t^2 factor that a constant shift in α cannot absorb into the velocity-based power family.Source: §B.1 · p. 16

B.2 NATIVE OBJECTIVES AND WEIGHT PLACEMENT

The section aligns weighting conventions across objectives and target coordinates, then connects noise-level weighting to reward-conditioned policy changes. It also states the conditions and implementation distinctions that limit these interpretations.

B.2 NATIVE OBJECTIVES AND WEIGHT PLACEMENT

  • FPO++ applies weighting after per-draw exponentiation and trust-region processing, while moving weights across exponentiation changes the objective.The calibrated CIFAR objective likewise aggregates before exponentiation and PPO clipping, while retaining its own sampling and scale conventions.Source: §B.2 · pp. 16–17

B.3 REFERENCE-FRAME INVARIANCE OF WHOLE-LOSS CALIBRATION

  • Whole-loss calibration matches initial aggregate objective RMS across effective shapes, enabling shape and scale to vary separately without equating loss functions or later training scales.It uses a common frozen bank, x0 anchor, and homogeneous normalizer; the construction requires a positive scale functional and nonzero bank vector.Source: §B.3 · pp. 17–18

B.4 KL TILT AND THE CONDITIONAL REWARD FIELD

  • The KL-regularized target is the reward-tilted distribution, and its conditional reward field depends on both the reward and reference distribution.The target is valid when β > 0 and 0 < ZR < ∞; behavior already represented by the generator presents a different adaptation problem from rarely generated structure.Source: §B.4 · p. 18

B.5 GAUSSIAN AUGMENTATION AND THE NOISE-LEVEL ELBO VIEW

  • Weighting allocates learning across differently smoothed views of reward-induced policy change, while the KL-tilted endpoint target stays fixed for fixed reward, reference, and temperature.The noise-level ELBO interpretation requires conditions on the weight profile; full-endpoint limits are not automatic for weights singular at t = 0 or 1.Source: §B.5 · pp. 19–20

B.6 SAMPLING AND SCHEDULE EQUIVALENCE

  • A fixed-integrand weighted expectation can be implemented by sampling times from the weighted distribution, but the associated Monte Carlo estimators need not have the same variance.Nonlinear exponentiation or clipping changes expected objectives and gradient distributions in an implementation-dependent way.Source: §B.6 · p. 20

B.7 SHARED MINIMIZERS AND FINITE SHARED-PARAMETER UPDATES

  • With a common minimizer and positive weights, all profiles preserve jointly realizable population minimizers; finite capacity, optimization, and moving on-policy losses shape practical deviations.In the local additive model, positive inner products support simultaneous loss reduction, but reward effects also depend on the reward gradient.Source: §B.7 · p. 21

C.1 REWARD DEFINITIONS

  • The CIFAR reward suite includes semantic and structural scores, while its differentiable proxies can differ from the black-box rewards used for training and evaluation.For example, the measured multiplicative rewards remain the products even when separate differentiable proxies use attribute coefficients.Source: §C.1 · p. 22

C.2 JOINT SHAPE–SCALE GRID

Across calibrated CIFAR scales, most rewards retain their preferred weighting shape, while Mixed and Digit7 show scale-dependent changes. Additional analyses compare sampling with explicit weighting, local reward responses, and cross-noise gradient coordination.

C.2 JOINT SHAPE–SCALE GRID

  • Four rewards preserve their preferred shape across calibrated scales, while Mixed’s preferred allocation changes with scale and Digit7 switches its leading shape centrally.The grid evaluates six rewards, seven profiles, five scales, and three seeds per cell; shape-first tuning may need a local recheck for Mixed and Digit7.Source: §C.2 · pp. 22–23, Figure 8 · p. 23

C.3 SAMPLING VERSUS EXPLICIT WEIGHTING

  • Edge improves consistently in the historical matched sampling comparison, while Auto-A35 has a positive mean with substantial profile/seed heterogeneity; sampling is not uniformly superior.These are observed validation-peak outcomes, not measurements of estimator variance.Source: §C.3 · p. 23, Table 2 · p. 24

D.1 LOCAL REWARD RESPONSE

  • Initial local reward responses vary by reward and do not alone determine the early advantage of high-noise weighting over x0; the observed association depends on the outcome window.The explicit-cohort association is not statistically significant and changes with the outcome summary or omission of a reward.Source: Figure 10 · p. 25, §D.1 · p. 25

D.2 INTERPRETING THE MAIN HEATMAPS

  • At Go2 iteration 300, the coordination contrast appears in both seeds, while most off-diagonal G1 late-checkpoint cosines in each seed have absolute value below .1.The heatmaps compare recorded cross-batch parameter-gradient cosines at individual checkpoints, not peak checkpoints or phase averages.Source: §D.2 · pp. 25–26

D.3 CIFAR WITHIN- AND CROSS-BATCH MEASUREMENTS

The measurements distinguish reproducible cross-batch directions from within-batch agreement and show that coordination patterns vary across rewards, learning stages, and tasks. Static and online weighting outcomes also vary, with full profile replacement performing poorly in the tested setup.

D.3 CIFAR WITHIN- AND CROSS-BATCH MEASUREMENTS

  • Batch-fluctuation energy exceeded estimated stable-mean energy in 75 of 80 states, with positive mean off-diagonal fluctuation correlation in all 80.Fluctuation correlations were spatially structured, stronger in middle/low-noise neighborhoods than in high-to-low coupling.Source: §D.3 · p. 27
  • A paired control’s mean same-bin cosine exceeded its independent-batch counterpart in 75 of 80 states, with median difference .1860.It reused generated endpoints, initial noise, raw rewards, and group advantages while redrawing timestep and corruption noise.Source: §D.3 · p. 27
  • Digit7’s α = 0 reward rose as S fell from .2018 to .0728 to −.1437; high-performing SimCLR −2 retained weak stable-mean estimates.The negative final estimate denotes unresolved stable signal with four batches, not a universal measure of learning quality.Source: §D.3 · p. 27

D.4 STAGE DEPENDENCE AND COMPLEMENTARY GEOMETRIES

  • Deteriorating Auto-A35 and SimCLR α = 0 branches developed stronger cross-bin agreement, while CLIP α = 0 deterioration accompanied falling agreement.At α = −2, CLIP retained a localized middle-to-low-noise positive block; these patterns do not quantify realized update contributions.Source: §D.3 · pp. 27–28, §D.4 · p. 29

D.5 ROBOTICS: STAGE-DEPENDENT COOPERATION AND CONFLICT

  • All 783 robotics checkpoints had higher mean off-diagonal within-batch than cross-batch cosine, with a median gap of .1588.Across 44 task–profile groups, 31 had lower late than early means; these changes are not a monotonic criterion for policy quality.Source: §D.5 · p. 31

E.1 OUTPUT-GRADIENT ESTIMATOR AND STATIC DEPLOYMENT

  • Digit7’s selected static mean was .710581, versus .641535 for its best fixed-sweep reference and .674514 for its matched x0 control.Static softmax improves selected reward peaks, but excessive concentration can reverse the gain.Source: §E.1 · p. 32

E.2 ONLINE REFRESH RULES AND MAIN OUTCOMES

  • EMA online weighting improved SimCLR over its x0 control, while CLIP, Auto-A35, and Mixed had lower online mean peaks than their controls.Digit7’s online mean exceeded x0 with mixed paired differences; all 33 full-replacement runs stopped by the degradation criterion.Source: §E.2 · pp. 32–33

E.3 TRAINING PATHS AND STAGE-DEPENDENT GAINS

Prescribed schedules outperform tested fixed profiles on the reported image rewards and robot-control tasks, while fixed-profile rankings vary with task and training budget. Longer-budget comparisons also show different stage trends for Acrobot and Walker.

E.3 TRAINING PATHS AND STAGE-DEPENDENT GAINS

  • Mixed, CLIP, and Digit7 schedules beat their named fixed-profile comparators across all three seeds, while Digit7 −2→0 remained below fixed 0 on mean.CLIP static softmax had a selected mean peak close to its larger-span schedule, without a general ranking between them.Source: §E.3 · pp. 33–34

E.4 ROBOT-CONTROL SCHEDULES IN NATIVE COORDINATES

  • Acrobot’s scheduled mean peak was 208.097670 versus best fixed .5 at 193.492714; Walker’s was 780.010782 versus fixed .5 at 760.164659.The comparisons are descriptive and use pooled ordinary fixed-reference runs, including reruns.Source: §E.4 · p. 34

E.5 FIXED PROFILES ACROSS LONGER TRAINING BUDGETS

  • Across four matched budgets, Acrobot’s low-exponent advantage narrowed then widened, while Walker retained αϵ = 1 > .5 > .25 among complete arms.Walker αϵ = 0 had two divergent runs and was not ranked as a complete three-seed arm.Source: §E.5 · p. 34

E.6 FOUR-CANDIDATE SELECTION AND TRAINING COST

Four-candidate selection reduces CIFAR training cost while exceeding native-target references on four evaluated rewards, though CLIP remains below its best fixed-sweep reference. Robotics replay shows both gains and misselection, with outcomes sensitive to the first-cut budget.

E.6 FOUR-CANDIDATE SELECTION AND TRAINING COST

  • Four-candidate selection used half the exhaustive training budget and exceeded native-target references on four CIFAR rewards, while CLIP remained below its best fixed-sweep reference.The cost comparison excludes evaluation runtime; the best-fixed values come from a separate cohort.Source: §E.6 · pp. 34–35
  • Across matched three-seed fixed-profile cohorts, Acrobot’s leading exponent alternated between .25 and .5 across budgets, while Walker retained 1 > .5 > .25.The plotted lines connect budgets, not changing-weight schedules.Source: Figure 16 · p. 35

E.7 RETROSPECTIVE ROBOTICS SUCCESSIVE-HALVING REPLAY

  • In robotics replay, successive halving improved Acrobot and Walker over native references, but a 20% first cut missed Spot’s full-budget winner.At a 40% first cut, the replay selected Spot’s native α = 1 with probability one.Source: §E.7 · p. 35
  • Replay search cost is measured relative to exhaustive profile search, and its reported selected score is not an independent confirmation on fresh training seeds.The replay uses draws from an empirical seed bank; evaluation runtime and fresh post-selection retraining are excluded from cost accounting.Source: §E.7 · p. 36
  • At the 20% first cut, replay selected Acrobot αϵ = 0, .25, .5 with probabilities 11/27, 15/27, 1/27; Walker selected .5, 1, 1.25, 1.5 with probabilities 17/27, 3/27, 6/27, 1/27.Spot selected α = .5 with probability one, below its full-budget native winner; Table 16 separates selected expectation from the fixed oracle.Source: §E.7 · p. 36

F V-GRPO: HELD-OUT EVALUATION AND TRAINING DYNAMICS

Held-out V-GRPO results show that preferred timestep exponents vary across rewards and evaluation horizons. The experiments are exploratory, with substantial seed-to-seed variability and high computational cost limiting method exploration.

F V-GRPO: HELD-OUT EVALUATION AND TRAINING DYNAMICS

  • Held-out weighting preferences varied by reward and horizon across OCR, TextPecker, GenEval, and PickScore, rather than identifying one preferred exponent.The displayed comparisons use one training seed per arm and are descriptive.Source: Figure 17 · p. 37, F V-GRPO: HELD-OUT EVALUATION AND… · p. 37
  • OCR favored moderate exponents early and midway, but α = −1 caught up late and achieved the highest observed held-out peak; TextPecker favored .5–1.Matched short training runs instead favored velocity over clean-target weighting in training reward.Source: F V-GRPO: HELD-OUT EVALUATION AND… · p. 37
  • GenEval’s held-out terminal scores favored the clean-target profile, while PickScore’s velocity and α = −.5 peaks nearly tied and −.5 and −1 exceeded velocity by update 300.At update 200, velocity had the highest of the four PickScore scores.Source: F V-GRPO: HELD-OUT EVALUATION AND… · p. 37
  • V-GRPO exploration was limited by substantial seed-to-seed variability, relatively saturated selected rewards, and high computational cost.The authors identify more challenging rewards and better weighting-search methods as future-work needs.Source: F V-GRPO: HELD-OUT EVALUATION AND… · p. 37
  • V-GRPO self-normalization can induce a different effective timestep allocation because it rescales residual contributions using time-dependent native-branch loss statistics.The paper treats its interaction with target parameterization as part of the timestep-weighting design dimension.Source: F V-GRPO: HELD-OUT EVALUATION AND… · p. 37

G.1 CIFAR MODEL, OPTIMIZATION, AND EVALUATION

The supplementary results describe the experimental settings and additional evidence on reward-dependent profiles, gradient coordination, and weighting methods. Across the supplied tasks, results vary by reward, policy, and learning stage.

G.1 CIFAR MODEL, OPTIMIZATION, AND EVALUATION

  • The shared CIFAR generator was a pretrained unconditional CIFAR-10 flow-matching checkpoint; controlled image runs used Euler sampling over 20 flow steps and three seeds per replicated main-image cell.Validation used fixed latent banks, with observed validation peaks serving as the reported selection metric.Source: §G.1 · p. 39

G.2 ROBOT-CONTROL ENVIRONMENTS AND TRAINING

  • Robotics experiments used online environment transitions with policies trained from scratch; FPO and FPO++ covered distinct simulation environments and training budgets.The supplied settings include 512 parallel Cartpole environments and 2,048 for locomotion tasks.Source: §G.2 · p. 40

G.3 WEIGHTING-METHOD CONFIGURATIONS AND COMPARISON REFERENCES

  • Static-softmax concentrations were selected from mean per-seed validation peaks, while the schedules were designed from observed learning curves as path interventions rather than learned online controllers.Online concentration was fixed at 300; Digit7 had a distinct initialization from the five-reward comparison.Source: §G.3 · pp. 40–41

G.4 SUCCESSIVE-HALVING CONFIGURATIONS

  • CIFAR successive halving began with four exponents, retained two at update 200 and one at update 400, then continued the winner to update 800.The replay enumerated ordered seed draws and evaluated alternative first-cut budgets.Source: §G.4 · p. 41

G.5 V-GRPO EVALUATION SETTINGS

  • V-GRPO OCR and TextPecker evaluations used held-out checkpoints through update 25, while GenEval used its update-50 terminal score and PickScore used updates 100, 200, and 300.Each displayed arm had one training seed.Source: §G.5 · p. 41

H.1 CIFAR FIXED PROFILES AND SHAPE–SCALE GRID

  • Across five calibrated whole-loss scales, most CIFAR rewards retained similar profile-shape rankings, while Mixed and Digit7 changed leading profiles.The figure reports within-scale ranks from mean per-run validation peaks, not reward units.Source: Figure 19 · p. 42

H.2 ROBOT-CONTROL FIXED-PROFILE RESULTS

  • Among the reported robot-control fixed-profile results, Fish had a broad high-return region, while G1 and Go2 favored their respective native FPO++ profiles.Ball’s zero-return seed for αϵ ≤ 0 is included in its mean and uncertainty.Source: §H.2 · p. 43

H.4 GRADIENT MEASUREMENTS, PANEL IDENTITIES, AND PHASE SUMMARIES

  • At Go2 iteration 300, both seeds showed broad positive coordination for α = 1, while α = 0 showed local positive blocks and negative clean–noisy coupling; the supplied learning-evolution panels also report weak late G1 off-diagonal structure.The Go2 trajectory peaks were 41.59/39.77 for seed 9200 and 41.59/39.00 for seed 9201 (α = 1/0); the corresponding current returns were distinct from those peaks.Source: §H.4 · p. 45

H.7 ROBOT-CONTROL SCHEDULES: ARMS AND COHORT RESULTS

Robot-control weighting schedules and replayed profile selection show that outcomes depend on cohort, task, and search budget. The gradient atlas likewise records task- and phase-specific cross-noise coordination patterns.

H.7 ROBOT-CONTROL SCHEDULES: ARMS AND COHORT RESULTS

  • Separate fixed-profile controls reached Acrobot 214.492 and Walker 815.786; a later Acrobot cohort mean of 190.525 was below its corresponding fixed-.5 reference of 202.226.The pooled gain does not establish improvement in every cohort.Source: §H.7 · p. 49
  • Reported observed-prefix peaks for three nonfinite Walker runs were 800.6 for ordinary fixed .5, 511.1 for ordinary fixed 1, and 720.5 for separate-family fixed .5.Including these peaks yields mixed-precision means of 763.841 (n = 11), 699.838 (n = 15), and 799.905 (n = 6), respectively.Source: §H.7 · p. 49

H.8 FOUR-CANDIDATE SEARCH AND ROBOTICS REPLAY RESULTS

  • The robotics replay table separates selected expected empirical mean-peak scores from fixed-oracle scores and reports selection probabilities and search cost relative to exhaustive search.Its sensitivity table reports exact ranges across bootstrap draws, not confidence intervals over independent training replications.Source: Table 16 · p. 50, Table 17 · p. 50

I FULL ROBOTICS HEATMAP ATLAS

  • Across Acrobot, Ball in Cup, Cheetah, and Fish, the phase atlas reports different coordination trajectories, including early-to-late declines for Fish and αϵ = 0 on Cheetah.Ball in Cup’s αϵ = 2 late panel contains only seed 902 after seed 901 failed at block 114.Source: Figure 20 · p. 51, Figure 21 · p. 52, Figure 22 · p. 53
  • Swimmer’s αϵ = .5 late panel retained more agreement than its early panel, while Walker’s αϵ = 0 late panel gained agreement and native-1 panels included opposing noise blocks.Cartpole remained broadly high in agreement across profiles, in contrast to G1’s weak off-diagonal agreement.Source: Figure 24 · p. 55, Figure 25 · p. 56, Figure 26 · p. 57
  • Go2’s native α = 1 lost much early agreement by late training despite a higher peak return than α = 0, whereas Spot’s α = 1 late phase retained more agreement than α = 0.The atlas therefore records different within-task trajectories for the two tasks.Source: Figure 30 · p. 61, Figure 31 · p. 62

Related papers

Loading 2609.32665v2…