PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty-Gated Exploration Noise Suppresses Task Collapse in Online RL Fine-Tuning of a Flow-Matching Vision-Language-Action Policy

Mehmet Turan Yardımcı, Yunus Emre Çoğurcu

arXiv:2609.28838v1cs.ROcs.LG

TL;DR

Online fine-tuning can erode individual tasks even when aggregate performance looks healthy, making task preservation relevant to continued robot learning. The paper compares three exploration-noise policies under a matched LIBERO-10 regime, finding that the uncertainty-gated controller avoids pooled collapse in its three seeds while the fixed and learned arms do not. The findings support task preservation in the evaluated regime, without an aggregate gain over behavior cloning.

  • Problem

    Online RL fine-tuning can destroy competence on individual tasks while aggregate performance still appears healthy, complicating robots’ continued learning after deployment.

    Source: §1 · p. 1

  • Method

    The study compares fixed, learned, and uncertainty-gated exploration noise in a matched online RL fine-tuning setup, with the controller using task-agnostic novelty and competence signals.

    Source: §1 · p. 2

  • Results

    No controller seed showed pooled collapse, compared with two of three fixed-noise seeds and all three learned-noise seeds; no arm improved over behavior cloning in this budget.

    Source: §1 · p. 2, §9 · p. 9

  • Takeaways & Limitations

    The matched results support task preservation by adaptive exploration in the evaluated regime, without an aggregate gain over behavior cloning.

    Source: §9 · p. 9

    Task preservation is shown for one VLA policy on seven jointly trained LIBERO tasks, and generalization to other policies and training regimes is untested.

    Source: §8 · p. 9

Abstract

from arXiv · show

Online reinforcement learning fine-tuning of pretrained flow-matching vision-language-action (VLA) policies promises robots that keep learning after deployment, but continued updates often destroy competence on individual tasks while the aggregate still looks healthy. We study this failure mode, which we call task collapse, under a matched small-compute budget on LIBERO-10 with a 450M-parameter SmolVLA policy trained by PPO with stochastic (SDE) sampling. Three exploration-noise policies differ in one live variable: a fixed noise scale, a ReinFlow-style learned noise network, and an uncertainty-gated controller that redistributes exploration across task streams from task-agnostic novelty and competence signals, without task labels or episode boundaries. Under the pooled definition, fixed noise collapses tasks in two of three seeds and learned noise in every seed measured to iteration 200, while the controller collapses none in any of its three seeds. Measured parameter displacement shows the controller's action expert keeps changing, while its mean applied noise is close to the fixed scale in the available logs. The matched comparison supports the controller's effect on task preservation; the separate contributions of its adaptation across states and over time are not disentangled. A lower fixed scale slows the decline but does not stop it. No arm improves on the behavior-cloning baseline in this budget. Two properties of that regime are measured beside this result, not offered as its cause: following the reference recipe, training runs in bfloat16 with no fp32 master copy, under which 96.02% of the action expert's elements stay bit-identical across three consecutive iterations, and an fp32 master copy at the reference learning rate collapses both arms in a single-seed observation. We release tools measuring per-task collapse under four definitions, rescoring noise and instrument tares.

1 INTRODUCTION

The paper studies whether adaptive exploration can prevent individual tasks from collapsing during online fine-tuning, a failure that aggregate success can conceal. It compares three noise policies and reports task preservation by the uncertainty-gated controller in the matched regime.

Motivation

  • Individual tasks can fall while aggregate performance still looks healthy, and a task lost completely may no longer receive reward that could support recovery.The paper frames preventing collapse as a precondition for lifelong learning.Source: §1 · p. 1

Approach

  • The controller regulates a flow policy’s injected noise using task-agnostic novelty and value statistics, not a calibrated confidence estimate.The signals read robot observations and the policy’s own value estimates.Source: §1 · p. 2

Research gap

  • Prior online fine-tuning reports surveyed by the authors provide aggregate success rates but not per-task training dynamics or their dependence on exploration-noise policy.The paper positions its comparison as addressing that reporting gap.Source: §1 · p. 2

Study design

  • The matched comparison holds the policy, benchmark, PPO recipe, data budget, and numerical regime fixed while changing only the exploration-noise policy.The arms are fixed noise, a ReinFlow-style learned noise network, and the uncertainty-gated controller.Source: §1 · p. 2

Main result

  • No controller seed showed pooled collapse, versus two of three fixed-noise seeds and all three learned-noise seeds; no arm improved over behavior cloning in this budget.The three-arm matched-budget comparison used three seeds per arm, with all nine runs reaching iteration 200.Source: §1 · p. 2

Update absorption

  • Under the reference recipe, 96.02% of action-expert elements remained bit-identical across three consecutive iterations, bounding what any arm could show in this regime.The recipe uses in-place bfloat16 parameters and no fp32 master copy.Source: §1 · p. 2

2 RELATED WORK

Related work covers online policy-gradient fine-tuning of flow and diffusion action heads, task interference and forgetting, adaptive exploration, and numerical precision. The paper distinguishes its per-task training analysis from prior aggregate-success reporting.

Online RL fine-tuning of flow and diffusion VLAs

  • Prior online flow and diffusion VLA methods inject stochasticity into denoising for policy-gradient fine-tuning, while reporting aggregate success rather than per-task training dynamics.The paper adds analysis of per-task dynamics and their dependence on exploration-noise policy.Source: §2 · p. 2

Collapse and forgetting under RL fine-tuning

  • Prior work studies task interference and forgetting, while gradient surgery acts on gradients and the controller here acts on exploration, making them orthogonal approaches.Some VLA methods retain imitation objectives during online updates as a stability measure.Source: §2 · p. 3

Exploration noise, entropy, and adaptive noise

  • Adaptive stochasticity includes entropy-regularized objectives and diffusion-policy methods, while runtime failure predictors use policy internals for alarms rather than update-time exploration.The paper uses related signal families to regulate exploration during updates.Source: §2 · p. 3

Numerical precision

  • Prior precision work reports that small bfloat16 updates can disappear, but the cited language-model RL measurement uses fp32 master weights, unlike the in-place VLA recipe studied here.The paper reports a robot-policy measurement under that in-place recipe.Source: §2 · p. 3

3 METHOD

The method applies PPO to a stochastic flow-matching policy and compares fixed, learned, and controller-selected SDE noise. The controller updates per-stream noise from task-agnostic features and competence signals.

3.1 FLOW-MATCHING POLICY AND SDE SAMPLING

  • The policy integrates a learned velocity field from Gaussian noise over 10 denoising steps, and Gaussian noise at each step gives tractable transition log-densities in a two-layer MDP.The first five executed action steps form the action whose log-density enters the policy-gradient ratio.Source: §3.1 · p. 3

3.2 PPO ON THE TWO-LAYER MDP

  • PPO uses the reference operators, while a single value head computes advantages and the controller’s five-head ensemble enters neither the advantage nor the PPO loss.The recipe was adopted rather than selected for this study.Source: §3.2 · p. 3

3.3 EXPLORATION-NOISE ARMS

  • The arms vary the SDE noise scale: fixed noise stays at 0.25, learned noise is predicted per denoising step, and the controller selects a per-stream value from four scales.The learned noise network is trained jointly with the policy; the controller recomputes its scale at every environment step.Source: §3.3 · p. 4

3.4 THE CONTROLLER

  • The controller combines novelty and competence signals into a multiplier for base scale σ0 = 0.3, then applies the computed scale at the following environment step.Its standardization statistics are pooled across slots, and the novelty predictor is trained once per outer iteration.Source: §3.4 · p. 4

4 EXPERIMENTAL SETUP

The experiments use a seven-task LIBERO-10 pool with a fixed training budget and compare task collapse using pre-registered thresholds. Evaluation reports both sampling uncertainty and rescoring variability.

Policy, benchmark, and budget

  • The study fine-tunes a 450M SmolVLA on LIBERO-10 tasks 1, 2, 3, 5, 6, 7, and 9 for 200 outer iterations, 2044 samples per iteration, and four optimizer steps per iteration.The pool is selected from the long-horizon suite.Source: §4 · p. 4

Evaluation rulers

  • Evaluation gives both Wilson intervals and run-to-run standard deviations because they describe different uncertainty sources and neither contains the other.The pooled Wilson interval assumes episodes are independent Bernoulli trials of one rate, so it does not bound variation between tasks or across the training window.Source: §4 · p. 5

Collapse metrics

  • A task is collapsed below half its same-task tare, with pooled collapse over the last five scans designated primary among four definitions.The other definitions use the last scan, two consecutive scans, or any measured scan; survival analysis uses first single-scan crossing.Source: §4 · p. 5

Controller objective

  • The controller is designed to spread exploration across tasks rather than concentrate it where reward is already plentiful, so success rate is not its control signal.The signal rationale is described in Appendix B.Source: §4 · p. 5

5 RESULTS

In the matched comparison, pooled collapse affected fixed- and learned-noise seeds but none of the controller seeds. Across four collapse definitions, transient threshold crossings can differ from persistent task loss, and controller-seed variability is small relative to the fixed arm’s.

5.1 LEARNING CURVES

  • Controller traces stayed near their tare throughout the run, while fixed seeds diverged and two ended far below tare and all learned-noise seeds fell early.The learned-noise seeds each reached their first collapse within 40 iterations; controller seeds stayed within the larger measured rescoring floor of one another.Source: §5.1 · p. 5, Figure 1 · p. 5

5.2 PLATEAU POOLS

  • Table 1 reports each run’s pooled success and four collapsed-task counts against its own instrument tare; campaign rows pool iterations 120–200.The four counts distinguish pooled, single-scan, sustained, and ever-collapsed tasks; every row is a single seed, not an arm-level verdict.Source: §5.2 · p. 6, Table 1 · p. 6

5.3 TASK COLLAPSE AND SURVIVAL

  • Under pooled collapse, two of three fixed seeds and all three learned-noise seeds had at least one collapsed task, versus none of three controller seeds.The pooled definition can differ from a single-scan result: fixed seed 1337 had 4 of 7 pooled collapsed tasks but 7 of 7 on its last scan.Source: §5.3 · pp. 6–7
  • Sustained collapse affected all fixed and learned seeds (3/3 each), versus 0/3 controller seeds; only one controller seed crossed the point-wise threshold, at iteration 160.The other two controller seeds were censored at iteration 200 without a point-wise event.Source: §5.3 · pp. 6–7

5.4 SEED VARIANCE

  • Controller pooled values were 0.585, 0.589, and 0.564 (SD 0.013), versus a fixed-seed spread of 0.194; controller spread was within the 0.0375 floor but just outside 0.0147.The authors claim only the contrast with the fixed arm that survives both rescoring floors.Source: §5.4 · p. 7

6 ANALYSIS: IS THE PROTECTION JUST LESS EXPLORATION?

The controller’s logged mean noise is close to the fixed scale, while its allocation differs across task groups. Lower fixed noise helped in some runs but did not replicate consistently; the reference bfloat16 recipe also leaves most action-expert elements bit-identical across three iterations.

Mean noise

  • Controller mean applied noise was 0.2399, 0.2444, and 0.2353 across logged runs, versus fixed noise at 0.2500; learned noise averaged 0.2340, logged from iteration 54.Controller logs cover 144 local iterations, 200 campaign iterations for seed 1337, and 155 for seed 2337; learned-noise logging begins at iteration 54.Source: §6 · p. 7

Noise allocation across tasks

  • Lower-baseline tasks received mean scales of 0.288 and 0.274, versus 0.180 and 0.185 for higher-baseline tasks in the logged controller runs.Task labels were used only for post-hoc grouping, not as controller inputs; seed 3337’s log was unavailable.Source: §6 · p. 7

Lower fixed noise

  • At seed 1337, fixed σ=0.18 pooled at 0.496 with one collapsed task, and σ=0.15 at 0.559 with zero; the second σ=0.15 seed pooled at 0.463 with two.The σ=0.18 ablation had one seed and σ=0.15 had two, so the first seed’s preservation did not replicate.Source: §6 · pp. 7–8

Precision

  • 96.02% of action-expert elements stayed bit-identical across three iterations, although 112 of 145 expert tensors changed in at least one element.Across 40-iteration windows, action-expert displacement was 0.0008–0.0010 under bfloat16 versus 0.0032–0.0035 with an fp32 master copy; these are separate measurements.Source: §6 · p. 8

7 DISCUSSION

The controller’s advantage is protection against task loss rather than uniform superiority, while aggregate success does not improve in this budget. The evidence is limited to the evaluated settings and does not establish gradient competition or lifelong-learning performance.

Reading the results

  • The controller’s advantage is protection where baselines break; gradient competition was not measured directly, and the matched intervention evaluates the controller as a whole.Fixed-noise outcomes vary across seeds, while controller endpoints fall within the larger measured rescoring SD of one another.Source: §7 · p. 8

Why no gain here

  • No aggregate-success gain is explained by the design: several possible factors are noted, but their contributions cannot be separated, and an appropriate fp32 learning rate was not established.The fp32-master pilot collapsed at the reference rate, motivating retuning.Source: §7 · p. 8

Breadth, and what it scopes

  • The eight-task benchmark re-run does not reproduce LIBERO’s simultaneous language-conditioned mixture, so a benefit from reallocating noise across task streams need not follow there.A stricter reading retains no positive separation; stage-level behavior for the Kitchen mixed drop was not analyzed.Source: §7 · p. 8

Why simultaneous rather than sequential

  • The study measures simultaneous multi-task training, not sequential lifelong learning; its narrower claim treats exploration, update size, and plasticity as related parts of that program.The fleet-level design is described in Appendix I.Source: §7 · p. 8

8 LIMITATIONS

The paper’s conclusions are bounded by its simultaneous-training scope, evaluation choices, precision regime, and the specific controller and learned-noise variants tested. Several broader settings and alternative configurations remain untested.

Necessary, not sufficient

  • Resistance to per-task collapse is presented as necessary, not sufficient, for continued learning without losing existing skills; sequential lifelong learning is not measured.The paper measures collapse under simultaneous multi-task training.Source: §8 · p. 9

No gain over behavior cloning

  • No arm exceeds its behavior-cloning tare; the result covers one policy, suite, and budget, and aggregate success was not the objective or tuned target.The paper notes sparse rewards, 800 optimizer steps, and bfloat16 in-place updates alongside the small room above the tare.Source: §8 · p. 9

Precision and learning rate

  • With in-place bfloat16 training and no fp32 master copy, 96.02% of action-expert elements remain bit-identical over three iterations; the appropriate fp32 learning rate is unknown.The fp32 pilot holds the reference learning rate fixed.Source: §8 · p. 9

Breadth

  • Task preservation is shown for one VLA policy on seven jointly trained LIBERO tasks; generalization to other policies and training regimes is untested.Across Appendix F.1’s eight complete tasks, the controller has no positive separation under the stricter reading and two negative ones.Source: §8 · p. 9

The learned-noise family is represented by one member

  • The learned-noise comparison uses one member of the Flow-Noise family; the Flow-Noise variant of Chen et al. (2025a) was not run.The tested R arm is not claimed to be the family’s best case.Source: §8 · p. 9

Evaluation budget and instrument

  • A canonical scan uses 30 episodes per task, below the stated de facto LIBERO standard of 50; the pooled definition averages five scans, or 150 episodes.Comparisons are read within one instrument because the same weights score 15 episodes apart on two GPUs.Source: §8 · p. 9

Scope of the evaluated controller

  • The reported controller effect applies to the evaluated two-signal, four-cell rule; additional code drivers were inactive and are untested, not shown ineffective.One multiplier-table cell departs from the design record and is identified in Appendix B.Source: §8 · p. 9

Exposure

  • All LIBERO campaign arms were measured through iteration 200 and scanned every 20 iterations, so seed-level counts compare three complete fixed seeds with three complete controller seeds.Run outcomes use each run’s last scan; the local-instrument row and Appendix F.1 re-run are outside that scope.Source: §8 · p. 9

9 CONCLUSION

The study examines task-level outcomes during small-budget online fine-tuning of a pretrained flow-matching VLA. In the evaluated LIBERO regime, the uncertainty-gated controller suppresses pooled task collapse, but aggregate success does not rise above behavior cloning.

9 CONCLUSION

  • Task collapse dominates the observed outcomes: fixed noise loses tasks at a seed-dependent rate, while the evaluated learned-noise arm loses tasks early and systematically.The study asks how individual-task outcomes depend on exploration-noise policy during online fine-tuning.Source: §9 · p. 9
  • No controller seed exhibits pooled collapse, compared with two of three fixed-noise seeds and all three learned-noise seeds; no controller seed exhibits sustained collapse.These findings are reported for the matched LIBERO training regime.Source: §9 · p. 9
  • The results support task preservation during continued updates in the evaluated regime, without an aggregate gain over behavior cloning.The comparison uses a small matched budget and pretrained flow-matching VLA.Source: §9 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies measurement protocols and artifacts used to check reported results. It also distinguishes released checkpoints from training-resume state and flags readings from an earlier run outside the released arms.

REPRODUCIBILITY STATEMENT

  • Appendix G lists run manifests, evaluation protocol, instrument-specific scoring rulers, four collapse definitions and thresholds, the seven-task pool, and the pinned reference configuration.The protocol specifies 210 fixed-identity episodes per canonical scan, SDE sampling with σeval = 0.25, and horizon 480.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The supplementary archive includes evaluation trees, measurement and gate scripts, controller and training code, campaign configuration, and measurement files; model weights are released separately.The archive names measurement files and describes their provenance.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Released checkpoints omit optimizer and controller state, so training does not resume from them; two readings use earlier seed-1337 controller checkpoints from outside the released arms.Those checkpoints are available in the anonymous repository, with their run configuration and logs identified in the archive.Source: REPRODUCIBILITY STATEMENT · p. 10

STATEMENT ON THE USE OF AI

The paper situates its controller among work on VLA confidence signals, exploration noise, and task interference, then describes a task-agnostic rule driven by novelty and competence. It defines collapse using four thresholds applied to per-task outcomes during simultaneous training.

STATEMENT ON THE USE OF AI

  • The authors disclose AI assistance with implementation, literature work, drafting, figures, and methodological feedback, while retaining final research and interpretation decisions.They report checking AI-assisted code and analyses with tests and planted-violation checks, and checking claims against measurement files, code, and sources.Source: STATEMENT ON THE USE OF AI · p. 10

A EXTENDED RELATED WORK

  • Unlike monitors that leave actions unchanged or selectors that choose candidates, this controller changes injected noise scale and reports per-task collapse under continued updates.Related work also distinguishes its pretrained VLA setting and RND-plus-critic signals from state-based exploration regulators and task-level success-rate schedules.Source: A EXTENDED RELATED WORK · pp. 14–15

B CONTROLLER: RATIONALE, CONSTRAINTS AND WHAT RAN

  • The controller does not observe contact phases, although the design rationale seeks caution during contact-rich phases and more freedom during transport.The authors identify this as design intent, not a measured property.Source: B CONTROLLER: RATIONALE, CONSTRAINTS… · p. 16
  • The controller combines RND novelty with PPO value and five-head critic spread in a rule table, using the signals to distinguish unfamiliar difficulty from familiar difficulty.Its design excludes task labels and episode boundaries, uses pooled running statistics and a one-step lag, and the reported runs use the four-cell rule rather than inactive drivers or the sixteen-cell variant.Source: B CONTROLLER: RATIONALE, CONSTRAINTS… · pp. 16, 18

C COLLAPSE DEFINITIONS

  • All four collapse definitions use half of each task’s behavior-cloning tare as a threshold, but differ in pooling, scan timing, persistence, or cumulative crossing.Pooled is primary; point-wise supports survival analysis, sustained requires two consecutive scans, and ever counts any crossing, including one followed by recovery.Source: C COLLAPSE DEFINITIONS · pp. 18–19
  • The authors distinguish simultaneous-task interference from continual-learning backward transfer because all seven tasks remain in training and a declining task can recover.They report three seeds per arm and note that task-seed pairs share training trajectories, so they are not independent training runs.Source: C COLLAPSE DEFINITIONS · p. 19

D.1 NOISE ALLOCATION ACROSS TASKS: HOW IT WAS READ

The allocation analysis finds substantial seed-to-seed variation in fixed-noise runs and reports controller noise and parameter movement from the available logs. It also documents limits on attribution and contrasts the bfloat16 reference regime with a single-seed fp32 pilot.

D.1 NOISE ALLOCATION ACROSS TASKS: HOW IT WAS READ

  • The logs cannot recover step-level noise allocation, and the matched arms do not isolate redistribution across task streams from adaptation over time or their interaction.The seed-2337 allocation figures cover only iterations 0–154, while the seed-3337 log is absent; no interpolation is performed.Source: §D.1 · pp. 19–21
  • The controller’s logged mean and unweighted mean of seven per-task values differ by at most 0.0033 on an iteration and 0.0002 in the run mean.The per-task scale is indexed by rollout stream, not task identity; task labels are used only to group output after the fact.Source: §D.1 · pp. 20–21
  • 0.19% of its weight norm is the action expert’s movement over 200 iterations in controller seed 2337, while fixed and learned arms move 0.0008 to 0.0010 per 40-iteration window.The controller’s per-window action-expert movement is 0.0008 to 0.0009; movement is diffusive, and these values are of the same order in runs that lose or keep tasks.Source: Figure 3 · p. 21, §D.1 · p. 21
  • 0.194 is the spread of fixed-noise pooled plateaus across three seeds at σ = 0.25, exceeding the 0.0375 run-to-run rescoring SD.The plateaus are 0.3162, 0.4724, and 0.5105, with first crossings at iterations 20, 120, and 80, respectively.Source: §D.1 · p. 21
  • 0.013 is the standard deviation of controller pooled readings 0.585, 0.589, and 0.564, within the 0.0375 rescoring SD.One seed was continued from iteration 180 to 200, and the effect of that continuation on its reading is unmeasured.Source: §D.1 · p. 22

E PRECISION: CONFIGURATION EVIDENCE AND THE FP32 PILOT

  • In the single-seed pilot, over ten canonical scans, the controller seed pooled at 0.603 [0.582, 0.624] in bfloat16 versus 0.388 [0.367, 0.409] with an fp32 master copy; intervals did not overlap.At equal exposure to iteration 200, pooled collapsed tasks were 3/7 for the fp32 controller arm versus 7/7 for the fixed arm. Each pilot arm used one seed.Source: E PRECISION: CONFIGURATION EVIDENCE… · p. 22

E.1 PARAMETER MOVEMENT, READ FROM THE WEIGHTS

The weight measurements show comparable action-expert movement across bfloat16 exploration arms, while most action-expert elements remain unchanged over a three-iteration check. The appendix also contrasts evaluation breadth and cost with ReinFlow’s benchmarks, with incomplete cost metadata limiting direct comparison.

E.1 PARAMETER MOVEMENT, READ FROM THE WEIGHTS

  • Across every measured bfloat16 window, action-expert displacement is 0.0008–0.0010, and fixed-noise movement is the same at σ = 0.25 as at 0.15.The controller’s task preservation coincides with continued parameter change, but comparable displacement does not establish similar update dynamics or explain preservation.Source: §E.1 · p. 23
  • Across measured 40-iteration windows, 11.9–13.3% of trainable tensors were bit-identical under bfloat16, versus 1.4% in the fp32-master pilot.Within bfloat16, value heads moved most, LoRA adapters next, and the action expert least, by roughly a factor of five; the precision regimes are not a controlled comparison.Source: §E.1 · p. 23, Table 5 · p. 25
  • 96.02% of action-expert elements were bit-identical across three iterations, while 112 of 145 tensors changed in at least one element.This is an earlier controller run, and the measurement does not determine whether unchanged elements received sub-resolution updates or none.Source: Table 4 · p. 24
  • The movement table reports relative weight displacement and bit-identical tensor fractions over 40-iteration windows; 39 of 63 cells are measured, and absent windows are not zero.The table merges six duplicate-directory rows into 33 distinct windows.Source: §E.1 · p. 22, Table 5 · p. 25

F BEYOND THE VLA: REINFLOW’S BENCHMARKS

  • ReinFlow reports relatively small networks and leaves scaling to large flow-based VLA models as future work; this paper instead evaluates a VLA and small flow policies.The comparison presents different evaluation profiles, not a difference in rigor.Source: F BEYOND THE VLA: REINFLOW’S BENCHMARKS · p. 26
  • The evaluation profiles trade breadth for depth: ReinFlow spans ten tasks across three benchmark families, while this work covers one campaign family and seven tasks, plus nine of ReinFlow’s tasks in a re-run.The re-run completed eight tasks in all three arms; reported per-iteration costs lack units and hardware, so cost comparison is only order-of-magnitude.Source: F BEYOND THE VLA: REINFLOW’S BENCHMARKS · p. 26, Table 6 · p. 26

F.1 THE REFERENCE MECHANISM RE-RUN ON ITS OWN BENCHMARKS

The re-run evaluates the three exploration arms on completed reference benchmark families using a two-ruler screening rule. The controller has downward separations on two families under that rule, while no family is reported in its favor.

F.1 THE REFERENCE MECHANISM RE-RUN ON ITS OWN BENCHMARKS

  • Nine families were attempted, but only eight completed in all three arms; Square had no completed runs, and Transport was never attempted.The baseline columns remain empty because per-task DPPO and FQL values were unavailable in the figures or pinned repository.Source: §F.1 · pp. 26, 28
  • The screening rule requires the arm’s mean to lie outside the other arms’ seed range and its difference from fixed noise to exceed both measured spread rulers.The rulers are the spread of final returns across seeds and the largest within-evaluation standard deviation; this is not a hypothesis test, and neither passing nor failing establishes significance or equivalence.Source: §F.1 · p. 27
  • On eight completed families, the controller separates downward on Hopper and Kitchen mixed under the governing stricter rule; no family is reported in its favor.The reported margins are 25.31 on Hopper and 2.24 on Kitchen mixed; the stage-level explanation offered for Kitchen mixed is a hypothesis, not a measured cause.Source: §F.1 · p. 27
  • No learned-against-fixed comparison clears both thresholds on any of the eight complete families, while Kitchen complete separates under the owner’s rule but not the stricter governing reading.The stricter reading was defined after the owner’s verdicts were seen and differs on Kitchen complete alone.Source: §F.1 · p. 27

G MEASUREMENT PROTOCOL, INSTRUMENTS AND REPRODUCIBILITY

The measurement protocol fixes the instrument, evaluation cadence, and a seven-task LIBERO-10 pool, while the discussion bounds conclusions to the measured model, regime, and budget. These choices support interpreting task-collapse results without treating them as evidence of absolute gains or generalization.

G MEASUREMENT PROTOCOL, INSTRUMENTS AND REPRODUCIBILITY

  • The behavior-cloning tare differs across instruments for the same weights: 136/210 (0.648) on the campaign instrument and 121/210 (0.576) locally.Accordingly, arms are compared only against the tare measured on their own instrument.Source: G MEASUREMENT PROTOCOL, INSTRUMENTS… · p. 31
  • No arm exceeds its own behavior-cloning tare, and the measured condition is simultaneous multi-task task-collapse resistance, not sufficient evidence of lifelong learning.The paper leaves sufficiency for sequential lifelong learning to future work.Source: G MEASUREMENT PROTOCOL, INSTRUMENTS… · p. 32
  • Training follows the reference bfloat16 in-place recipe without an fp32 master copy, and 96.02% of action-expert elements stayed bit-identical across three iterations.The authors measure this as a possible swallowing of updates, not as the cause of the outcome.Source: Table 7 · p. 31, G MEASUREMENT PROTOCOL, INSTRUMENTS… · p. 32
  • The fp32-master pilot at the reference learning rate is an observation rather than a controlled single-variable comparison, and an optimized fp32 recipe was not evaluated.The authors state that the reference rate may not be appropriate under fp32; retuning it is outside this work.Source: G MEASUREMENT PROTOCOL, INSTRUMENTS… · pp. 32–33
  • The findings cover one 450M policy, one seven-task LIBERO-10 pool, three seeds per arm, and training to iteration 200; they do not generalize to another model or suite.One controller seed was resumed from iteration 180 to reach 200.Source: G MEASUREMENT PROTOCOL, INSTRUMENTS… · p. 33
  • The learned-noise arm represents one member of that family; the Flow-Noise variant with per-step noise and exact log-likelihoods was not run.The paper cautions against reading its learned-noise arm as the family’s best case.Source: G MEASUREMENT PROTOCOL, INSTRUMENTS… · p. 33
  • The evaluated controller uses two signal families and a four-cell rule; additional code drivers were inactive, so reported effects apply to that rule, not the fuller design.The running rule raises σ to 0.36 in the familiar-and-low-competence cell, above its 0.30 base.Source: G MEASUREMENT PROTOCOL, INSTRUMENTS… · p. 33
  • Canonical scans use 30 episodes per task, and the seven-task pool excludes three LIBERO-10 tasks whose behavior-cloning success was near zero on the available instrument and budget.The excluded tasks were not treated as impossible; their tares at the 480-step horizon were not measured.Source: G MEASUREMENT PROTOCOL, INSTRUMENTS… · pp. 33–34

H.1 WHAT THIS PAPER MEASURES THAT THE LEARNED-NOISE REFERENCE DOES NOT

The paper extends learned-noise online RL evaluation to a language-conditioned VLA and measures per-task collapse, which aggregate success rates can hide. In this regime, the uncertainty-gated controller preserves pooled task performance across its three seeds, while no arm improves on behavior cloning; the discussion frames update gating and fleet learning as future directions.

H.1 WHAT THIS PAPER MEASURES THAT THE LEARNED-NOISE REFERENCE DOES NOT

  • Unlike the learned-noise reference, this study uses a 450M-parameter VLA, language instructions shared across tasks, and per-task collapse measurements.The reference reports aggregate episode reward or success, not per-task training dynamics.Source: §H.1 · p. 36
  • The controller preserved every task above the pooled collapse threshold in all three seeds; fixed noise collapsed tasks in two of three seeds and learned noise in every measured seed.No arm improved on the behavior-cloning baseline in this budget.Source: §H.1 · p. 37, I EXTENDED DISCUSSION AND OUTLOOK · p. 38
  • In the reference bfloat16 recipe without an fp32 master copy, 96.02% of action-expert elements remain bit-identical across three consecutive iterations.The paper reports this update-absorption measurement alongside task-preservation results, not as their cause.Source: §H.1 · p. 37
  • A lower fixed noise scale slows decline but does not stop it; the σ = 0.15 arm collapsed two tasks in one of its two reported seeds.The controller’s mean applied noise was only 0.006–0.015 below the fixed arm across available logs, so noise magnitude alone does not match the reported ordering.Source: §H.1 · p. 37
  • Rescoring variability is reported separately from single-scan Wilson intervals because the two evaluation rulers capture different uncertainty.A single 210-episode scan has a Wilson 95% interval half-width of about 0.067; campaign rescoring has a run-to-run standard deviation of 0.0375.Source: §H.1 · p. 37

I EXTENDED DISCUSSION AND OUTLOOK

  • A task below the collapse line may later recover, so zero measured success is not evidence that competence is irreversibly lost.The discussion connects task loss to missing future reward that could otherwise teach the task back.Source: I EXTENDED DISCUSSION AND OUTLOOK · p. 38
  • The paper presents its results as a first step toward policies governed by their own uncertainty, not as evidence that the fleet-learning design works.The fleet proposal selects novel or failed-on experience for local training; no model weights move between robots.Source: I EXTENDED DISCUSSION AND OUTLOOK · p. 38
  • The proposed next mechanisms are gating update size with the same signals and later managing plasticity as task capacity fills.The full four-signal design was measured only through its logs, and novelty detection at scale remains to be tested.Source: I EXTENDED DISCUSSION AND OUTLOOK · p. 38

Related papers

Loading 2609.28838v1…