PaperlayerView exact revision on arXiv

Source-linked AI summary

Noise Space Learning for VLA Adaptation

Junjie Lu, Chuheng Zhang, Yucen Wang, Yuhua Jiang, Xinyao Qin, Min Xu, Li Zhao

arXiv:2610.13943v1cs.RO

TL;DR

Adapting pretrained VLAs to specific tasks must balance limited robot interaction, computation, and safety while preserving useful priors. The paper introduces NSL, which learns context-conditioned noise distributions for a frozen VLA, and evaluates PPO, SAC, and AWR variants. Its results show that optimizer suitability varies with resource budgets and deployment setting.

  • Problem

    VLA adaptation must acquire task-specific behavior efficiently while preserving pretrained priors, yet fixed-backbone approaches face trade-offs in execution effectiveness, inference cost, and behavioral change.

    Source: §1 · p. 1

  • Method

    NSL adapts flow-matching VLAs by learning a lightweight, context-conditioned noise distribution while keeping the pretrained VLA fixed, with PPO, SAC, and AWR variants.

    Source: §1 · pp. 1–2

  • Results

    NSL is competitive with generator and residual adaptation at low gradient-update cost; AWR suits limited collection and updates, SAC benefits from larger simulation budgets, and PPO benefits from more rollouts.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Choosing among NSL optimizers should reflect interaction and computation budgets, with AWR practical for online robot learning using small collection batches and human corrections useful on complex tasks.

    Source: §5 · p. 10

    NSL's adaptation capacity is constrained by the fixed generator and the expressiveness of the noise actor; generalization, skill retention, and execution reliability in complex contact interactions remain for future study.

    Source: §5 · p. 10

Abstract

from arXiv · show

Pretrained vision-language-action (VLA) models encode broad manipulation priors, yet adapting them to specific tasks requires balancing limited robot interaction, computational resources, and safety constraints. Noise space learning (NSL) adapts flow-matching-based VLAs by learning a context-conditioned distribution over noise while keeping the pretrained VLA fixed. Building on prior noise-steering methods, we formulate NSL as an interface for VLA adaptation that accommodates different learning algorithms and instantiate it with PPO, SAC, and AWR. Theoretically, we provide an information-theoretic analysis of how NSL enables efficient adaptation. Empirically, we study these NSL algorithms on LIBERO, RoboTwin, and four real-robot manipulation tasks under interaction, computation, and execution constraints. Across these settings, we develop a suite of effective NSL variants and derive practical recommendations for choosing among them under different resource and deployment regimes.

1 Introduction

The paper asks how to adapt VLAs efficiently while preserving pretrained priors, and proposes learning a context-conditioned noise distribution with the VLA backbone fixed. It evaluates PPO, SAC, and AWR across simulation and real-robot settings, finding that the suitable variant depends on resource and deployment constraints.

1 Introduction

  • Fixed-backbone adaptation preserves pretrained priors and can improve efficiency, but action-head updates may underfit, candidate search raises inference cost, and conservative residual corrections can limit behavioral change.Low-data supervised fine-tuning can also fixate on training objects or targets despite a different instruction.Source: §1 · p. 1
  • NSL trains a lightweight, context-conditioned noise actor to steer action generation while keeping the pretrained VLA and its noise-to-action mapping fixed.The paper instantiates this interface with PPO, SAC, and AWR and studies noise limits, task balancing, and success/failure balancing.Source: §1 · pp. 1–2
  • The evaluation spans LIBERO, RoboTwin, and four real-robot tasks under interaction, computation, and execution constraints; noise distributions far from standard Gaussian are associated with unsafe motions and emergency stops on robots.The real-robot tasks are cup stacking, toy fishing, sequential insertion, and bead placement.Source: §1 · p. 2
  • NSL-SAC performs well in simulation with ample interaction but costs more computation, whereas NSL-AWR better suits interaction-limited real-robot adaptation with lower computational cost and safer execution.NSL-PPO is also competitive in simulation, benefiting from more rollouts while showing limited average gains from additional updates.Source: §1 · p. 2

2 Related Work

Prior VLA adaptation methods update the generator, correct its outputs, or change inference-time selection, while noise-space methods adjust the distribution that feeds a generative policy. This paper positions NSL as a unified noise-distribution interface for flow-matching VLAs.

2 Related Work

  • VLA adaptation includes full-model or parameter-efficient updates, reinforcement-learning updates, residual corrections, hybrid generator-and-action methods, and inference-time integration or candidate selection.These approaches trade off behavioral change, retention, adaptation cost, and inference efficiency.Source: §2 · p. 2
  • Existing noise-space methods use conditional noise actors, corrective-action inversion, context-independent noise search, constrained perturbations, or jointly learned noise priors and generators.DSRL [33], UniSteer [18], FlowDAgger, Flow Reversal Steering, Golden Ticket [21], LP-DS [30], and LeaP [6] illustrate these approaches.Source: §2 · p. 2
  • For a deterministic generator, NSL adapts the input noise distribution, generator adaptation changes the noise-to-action map, and residual adaptation modifies generated actions before execution.The figure marks trainable modules in orange and fixed modules in blue.Source: Figure 1 · p. 3

3 Methods

The methods define NSL as learning the initial noise distribution while holding the pretrained action generator fixed, then compare adaptation approaches through information costs and instantiate NSL with three optimizers.

3.1 Noise Space Learning

  • NSL samples context-conditioned noise from a learned actor and passes it through the fixed pretrained generator to produce action chunks, optimizing the actor for reward.The context typically includes image observation, proprioception state, and task instruction; the standard noise prior is typically Gaussian.Source: §3.1 · p. 3

3.2 Information-Theoretic View of Different Adaptation Approaches

  • The information-theoretic analysis treats noise as a source distribution and the pretrained generator as a channel, with NSL changing the source, generator adaptation changing the channel, and residual adaptation modifying outputs.Under the stated conditions, NSL matches the behavioral information lower bound while retaining the pretrained generator.Source: §3.2 · pp. 3–4
  • For a fixed deterministic generator and an absolutely continuous target action distribution with finite KL divergence, an optimal noise distribution attains minimum information cost; NSL's cost is the KL divergence from its noise actor to the prior.The comparison defines a common context distribution, utility, target threshold, and reference law across adaptation approaches.Source: §3.2 · p. 4

3.3 Optimization Methods for Noise Space Learning

  • PPO uses recent rollouts and estimated advantages, SAC uses noise-space critics, and success-filtered AWR fits replay noises with estimated-advantage weights.The methods update the noise actor and auxiliary value models without backpropagating through the frozen VLA; AWR's success indicator and weight cap are specified in the method.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate three NSL algorithms on LIBERO-10 and compare them with generator- and residual-adaptation baselines, then test NSL-AWR on four real-robot tasks. Results show improved success with the VLA frozen, while the strongest algorithm depends on available interaction and update budgets.

4.1 Experimental Setup

  • LIBERO-10 experiments use SmolVLA and π0.5 checkpoints trained with one demonstration per task or all 379 filtered demonstrations, with one shared task-conditioned noise actor per run.Each epoch evaluates all 500 task–reset pairs once; baselines start from the same checkpoint within each backbone–data setting.Source: §4.1 · p. 5

4.2 Results of Noise Space Learning

  • 95.8% (SmolVLA-Full) and 99.8% (π0.5-Full) peak success; in One, NSL raises SmolVLA from 24.4% to 77.0% and π0.5 from 40.4% to 71.2%.All three NSL methods improve over their base-policy references; NSL-SAC has the highest peak scores, tying NSL-PPO on SmolVLA-Full.Source: Figure 2 · p. 6, §4.2 · p. 6

4.3 Comparison with Generator and Residual Adaptation

  • 83.5% (NSL-SAC) and 81.6% (NSL-PPO) average success rates exceed RLT (81.0%) and Action-AWR (79.4%); NSL-SAC takes 9.93 seconds per collection batch.NSL-AWR averages 79.1% success and has the lowest update time, 1.09 seconds per batch.Source: Table 1 · p. 7, §4.3 · p. 7

4.4 Analysis of Update Budget

  • AWR performs best at 1× updates, SAC benefits most from added updates, and PPO’s mean learning curves change little across 1×, 2×, and 4× budgets.With larger update budgets, SAC reaches higher final success; the comparison holds the rollout protocol fixed.Source: §4.4 · p. 7, Figure 3 · p. 8

4.6 Analysis of Training Components

  • SAC benefits from task and success/failure balancing and is particularly sensitive to noise limits; PPO benefits mainly from task balancing, while AWR remains comparable without the components.Without a noise limit, SAC’s actor means drift farther from initialization and critic-target upper tails exceed the return bound; both are lower with bounded noise.Source: Table 2 · p. 8, §4.6 · pp. 8–9

4.5 Analysis of Rollout Count

  • More rollouts improve all three methods, with SAC benefiting most; SAC reaches the highest final success rate as collection increases.Gains from four to eight rollouts are smaller; AWR is competitive with limited collection, while PPO improves more sustainably with additional rollouts.Source: §4.5 · pp. 7–8, Figure 4 · p. 9

4.7 Real-Robot Experiments

  • 71.3% mean success for autonomous NSL-AWR, versus 58.8% for Action-AWR and 70.0% for RLT, from 37.5% for the initial policy.Performance varies across tasks, with RLT doing better on sequential insertion.Source: Table 3 · p. 10, §4.7 · p. 10
  • 83.8% mean success with human-guided NSL-AWR exceeds DAgger (66.3%) and UniSteer (73.8%); guided NSL-AWR takes 2.04 minutes per task.The largest reported gains are on sequential insertion (70% to 90%) and bead placement (35% to 55%).Source: §4.7 · p. 10, Table 3 · p. 10

5 Discussion and Conclusion

NSL uses a frozen action generator and a learned noise distribution; the paper reports competitive adaptation at low gradient-update cost and optimizer trade-offs across resource budgets. Its analysis distinguishes behavioral change from intervention-mechanism change and states conditions and limits for comparing intervention costs.

5 Discussion and Conclusion

  • NSL-AWR suits small collection batches, while NSL-SAC gains from larger simulation budgets and NSL-PPO benefits from more rollouts; human corrections improve complex real-robot tasks.The paper recommends choosing optimizers according to interaction and computation budgets.Source: §5 · p. 10
  • Adaptation remains limited by the fixed generator and noise actor’s expressiveness; future work includes richer noise parameterizations, selective generator updates, generalization, skill retention, and contact reliability.Source: §5 · p. 10

A.1 Definitions and Scope

  • For stochastic inference, the action generator is modeled as a fixed conditional kernel, and NSL changes the noise supplied to it rather than controlling additional decoder randomness.Controlling all decoder randomness would define a different adaptation interface.Source: §A.1 · p. 14

A.2 Noise and Behavioral Information

  • The KL decomposition gives a common behavioral-information lower bound; equality holds when adapted and reference noise posteriors agree almost everywhere, without requiring a differentiable or invertible generator.Source: §A.2 · p. 14

A.3 Comparing Intervention Locations

  • Intervention cost also includes conditional redistribution of internal variables given executed actions, so the shared behavioral lower bound does not establish a strict cost ranking.Hybrid interventions require a full chain-rule treatment; EXPO-FT cannot be substituted for a single-factor correction law without redefining internal variables.Source: §A.3 · p. 14
  • Finite-cost comparisons involving changed deterministic generators require a shared Gaussian relaxation, whose resulting costs depend on its scale.The unchanged deterministic NSL map is an exception to that relaxation requirement.Source: §A.3 · p. 15

A.4 Proof of the Minimum-Information Lift

The appendix establishes a minimum-information noise reweighting for deterministic generators and extends the comparison to utility-constrained adaptation. It also qualifies the scope of the results and explains why behavior anchoring is not a physical safety guarantee.

A.4 Proof of the Minimum-Information Lift

  • For deterministic generators, noise reweighting can attain the target action distribution at minimum information cost while preserving the reference noise posterior within each action fiber.This construction does not transfer to arbitrary stochastic channels; full support, reachability, coverage, and finite-budget learnability are distinct.Source: §A.4 · p. 15

A.5 Utility-Based Information Costs

  • With unrestricted noise distributions in the deterministic system, NSL matches the common behavioral information requirement under utility constraints.The comparison does not establish strict separation from another interface without a positive gap in that interface’s own infimum.Source: §A.5 · p. 16

A.6 Regularized Noise Actor Improvement and AWR

  • AWR’s regularized target weights reference noise by exponentiated value, but filtering or clipping changes the unclipped optimum and no successful replay yields no actor-learning signal.Weighted likelihood is a forward-KL projection under a reweighted context marginal, not generally the original replay context distribution; finite empirical weighted log density is a Monte Carlo approximation.Source: §A.6 · pp. 16–17

A.7 Why Behavior Anchoring Is Not a Safety Bound

  • Behavior anchoring does not bound physical safety: noise-moment inequalities do not guarantee collision, force, or joint-limit constraints without additional system assumptions.Physical safety remains an experimental and engineering requirement.Source: §A.7 · p. 17

B.1 Implementation Details

The implementation section specifies the noise actors, optimization schedules, and data handling for LIBERO, then describes the real-robot tasks, online collection protocols, and evaluation procedure.

B.1 Implementation Details

  • LIBERO actors use pooled VLA features and proprioception, four 512-unit hidden layers, and separate 32-dimensional mean and log-standard-deviation heads; update schedules scale with collected transitions and multiplier.The noise vector is repeated across the 50-step horizon and bounded to (−1, 1) in the main SAC setting after initial Gaussian exploration.Source: §B.1 · p. 17
  • PPO and AWR use categorical value networks, while SAC uses twin critics; LIBERO evaluation uses deterministic actor means for NSL and native Gaussian sampling for the base policy.Stochastic training-rollout success is tracked separately.Source: §B.1 · pp. 17–18

B.2 Real-Robot Protocol

  • Real-robot policies predict 50-step action chunks and execute ten steps before replanning across four manipulation tasks.Each task uses a separately fine-tuned π0.5 base policy shared by compared methods.Source: §B.2 · pp. 18–19
  • Online adaptation alternates trajectory collection and updates; human-guided NSL-AWR converts human action chunks into noise with fixed-point inversion and combines them with robot samples.Figure 6 reports mean trajectories per round, separating model and human counts; evaluations use 40 attempts per task and equally weight task success rates.Source: §B.2 · pp. 19–20, Figure 6 · p. 20

B.3 NSL-AWR Noise Evolution on Fixed Observations

On fixed LIBERO observations, NSL-AWR develops task-related noise-mean structure while continuing to refine context-dependent outputs. The diagnostics also describe SAC noise-limit behavior and real-robot training, followed by the RoboTwin experimental setup.

B.3 NSL-AWR Noise Evolution on Fixed Observations

  • At epoch 60, task centroids explain 52.8–79.9% of total actor-mean variation in the original 32-dimensional space.Task-related structure appears after the first epoch, alongside differences between observations within each task.Source: §B.3 · p. 21
  • SmolVLA ends with average conditional standard deviation near 0.80, compared with 0.95 for π0.5; One has larger final mean displacement than Full for either backbone.The backbones use separate PCA bases and fixed axis limits in Figure 7; Figure 8 tracks mean shift, epoch-to-epoch drift, standard deviation, and between-task variance.Source: Figure 7 · p. 21, §B.3 · p. 22, Figure 8 · p. 22
  • For π0.5-One, adjacent-epoch mean displacement is 0.205 over epochs 51–60, versus 0.171 over epochs 1–5.Task-related structure coexists with continued refinement of observation-dependent outputs.Source: §B.3 · p. 22

B.4 Noise-Space SAC Diagnostics in Simulation

  • The simulation noise-limit diagnostics compare mean Gaussian-mean displacement before tanh with the mean minibatch-maximum Bellman target, against a return upper bound of 1.The figure caption describes the diagnostic quantities but does not report their observed values.Source: Figure 9 · p. 23, §B.4 · p. 23

B.5 Noise-Space SAC Diagnostics on the Real Robot

  • In bead placement, some deterministic noise outputs approach the amplitude limit, and the final minibatch maximum absolute output is 4.96.The mean absolute output rises from 0.60 to 1.04; Figure 10 also tracks twin-critic disagreement and actor-proposal minus replay Q value.Source: §B.5 · p. 23, Figure 10 · p. 23
  • Over the last 200 updates, twin-Q disagreement is 0.916 for actor proposals versus 0.151 for replay noise, while logged actor-proposal Q exceeds replay by 4.43.The experiment collected 32 trajectories and 2,789 replay transitions; unintended table contact occurred.Source: §B.5 · pp. 23–24

C.1 Experimental Setup

  • RoboTwin uses eight tasks with Easy and Hard settings, shared fixed training and held-out manifests, and independent task-specific noise actors conditioned on pooled VLA features and proprioception.The main runs use 30 epochs, eight rollouts, and update multipliers of 4×, 2×, and 4× for AWR, PPO, and SAC; the SFT reference averages 31.50% on Easy and 32.75% on Hard.Source: §C.1 · p. 24

C.2 Main Results

All three NSL methods improve over SFT on RoboTwin Easy and Hard, with NSL-SAC achieving the highest mean success rate. Its epoch-30 gains over SFT are substantial in both settings.

C.2 Main Results

  • 84.25% Easy and 77.25% Hard: NSL-SAC has the highest mean success rate at epoch 30, improving over SFT by 52.75 and 44.50 percentage points, respectively.All three methods improve both settings; NSL-AWR and NSL-PPO also provide substantial gains, with their relative ranking depending on setting.Source: Table 7 · p. 24, §C.2 · p. 24

C.3 Per-Task Results

At epoch 30, every NSL method exceeds SFT on all eight RoboTwin tasks in both difficulty settings. The strongest method varies by task, and place_a2b_left remains the weakest across the six main runs.

C.3 Per-Task Results

  • The best method varies by task: NSL-AWR leads on Easy dual-bottle picking and microphone handover, NSL-PPO leads on Hard dual-bottle picking, and place_a2b_left is weakest across all six main runs.Source: §C.3 · p. 25
  • At epoch 30, every method exceeds SFT on all eight tasks in both settings, including NSL-SAC’s Easy move_can_pot gain from 4% to 86% and Hard place_phone_stand gain from 2% to 62%.Table 8 reports per-task success rates for Easy and Hard.Source: Table 8 · p. 25, §C.3 · p. 25

C.4 Analysis of Update Budget

The preferred RoboTwin Easy update budget depends on the optimizer: SAC benefits up to a point, PPO deteriorates with larger budgets, and AWR is comparatively stable. A modest update budget is sufficient for strong AWR performance.

C.4 Analysis of Update Budget

  • NSL-SAC improves by 17.00 percentage points when U rises from one to four, but increasing U to eight reduces its epoch-20 success rate.Table 9 reports eight-task mean success rate at epoch 20 with R = 8.Source: §C.4 · p. 25, Table 9 · p. 25
  • NSL-PPO performs best at 2×; its 8× run declines from 60.75% at epoch 4 to 50.50% at epoch 20.Source: §C.4 · p. 25
  • NSL-AWR is comparatively stable across tested update budgets, performing best at 2× and maintaining similar success rates at 4× and 8×.The authors conclude that a modest update budget is sufficient for strong NSL-AWR performance.Source: §C.4 · p. 25

C.5 Analysis of Rollout Count

On RoboTwin Easy, more rollouts improve epoch-20 success rates for NSL-AWR, NSL-PPO, and NSL-SAC, while SAC’s advantage is reduced under limited collection.

C.5 Analysis of Rollout Count

  • With one rollout per scenario, AWR and SAC finish at similar performance and both exceed PPO, reducing SAC’s advantage under limited collection.Source: §C.5 · p. 26
  • SAC gains 9.75 percentage points from four to eight rollouts, a pattern consistent with value-guided optimization benefiting from richer experience.Source: §C.5 · p. 26
  • From one to eight rollouts, epoch-20 SR rises 7.75 percentage points for NSL-AWR, 16.75 for NSL-PPO, and 20.50 for NSL-SAC.Table 10 reports eight-task mean SR (%) at epoch 20.Source: Table 10 · p. 26, §C.5 · p. 26
Loading 2610.13943v1…