PaperlayerView exact revision on arXiv

Source-linked AI summary

The Low-Rank Structure of VLA Reinforcement Learning

Minjae Oh, Yoonah Park, Jongwon Lim, Yohan Jo

arXiv:2609.34599v1cs.LG

TL;DR

How reinforcement learning encodes learning signals in VLA parameters remains poorly understood, particularly for flow-based action experts. Across VLA families and manipulation benchmarks, the paper analyzes parameter updates and their task-related structure, finding low-rank changes concentrated in Timestep Modules and informative shift-vector directions.

  • Problem

    How RL reshapes VLA parameters remains poorly understood, particularly given the distinct flow-based action expert architectures used in modern VLAs.

    Source: §1 · p. 1

  • Method

    The paper studies RL-induced parameter updates across flow-based VLA families and manipulation benchmarks, using module-replacement, probing, and steering experiments to characterize Timestep Module signals.

    Source: §1 · pp. 1–2, §7 · p. 9

  • Results

    RL induces low-rank updates concentrated in Timestep Modules, and shift-update directions predict task success with ROC-AUC up to 99.6%.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The results suggest that much of RL post-training in VLAs can be explained by timestep-specific vector-level modulations of the action expert’s hidden states.

    Source: §7 · p. 9

Abstract

from arXiv · show

Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, yet how RL reshapes these policies remains poorly understood. We find that RL across widely used flow-based VLA models, including $\pi_{0.5}$ and GR00T~N1.5/N1.6, on LIBERO, ManiSkill, MetaWorld, and CALVIN induces substantially lower-rank parameter updates that are highly concentrated in the action expert's Timestep Modules, a small and previously overlooked component. Through systematic module-replacement experiments, we further show that these modules capture a disproportionate share of the performance gains from RL. We then characterize what is encoded in these Timestep Modules. First, we show that RL specializes them to the discrete denoising timesteps used during rollouts, and that this discrete-timestep training underlies the low-rank updates. Second, we find that among their outputs, the shift vector changes most distinctly under RL, and through probing, we show that shift update directions strongly predict task success (ROC-AUC up to $99.6\%$). Third, we find that the geometry of shift updates reflects task relationships, as their pairwise similarity correlates with cross-task transfer patterns. Building on these findings, we show that steering along shift update directions further improves RL-trained policies without additional RL training. Overall, we provide a systematic understanding of how RL reshapes VLA policies by studying how learned signals are encoded in parameter space, offering insights into more efficient and interpretable VLA post-training.

1 INTRODUCTION

The paper investigates where and how reinforcement learning reshapes VLA parameters, finding low-rank updates concentrated in the action expert’s Timestep Modules. It further traces these signals to discrete-timestep specialization and compact shift-vector updates associated with task success and transfer.

1 INTRODUCTION

  • Across flow-based VLA families and four manipulation benchmarks, RL induces low-rank parameter updates concentrated in the Timestep Modules.These modules capture a disproportionate share of RL performance gains, and targeting them with LoRA outperforms standard LoRA configurations.Source: §1 · pp. 1–2
  • RL specializes Timestep Modules to rollout denoising timesteps, and discrete-timestep training underlies their low-rank updates.Shift updates carry most task-related information among module outputs, which the paper characterizes as compact, timestep-specific vectors.Source: §1 · p. 2
  • Shift-update directions predict task success with ROC-AUC up to 99.6%, while their similarity correlates with cross-task transfer patterns at Spearman ρ = 0.794.The paper also reports that steering along task-specific shift update directions further improves RL-trained policies at test time.Source: §1 · p. 2
  • The authors characterize RL’s VLA learning signal as low-rank, structured updates concentrated in Timestep Modules and driven by discrete denoising-timestep training.They identify parameter-efficient post-training, adaptation, and task-behavior composition and transfer as directions for future work.Source: §1 · p. 2

2 BACKGROUND

Flow-based VLA action experts learn to transform noisy action chunks toward demonstrated actions, with behavior cloning trained over continuous timesteps. Timestep Modules turn denoising timesteps into hidden-state modulations, while on-policy PPO updates use discrete rollout timesteps.

2.1 FLOW-BASED VLA POLICIES

  • Flow-based action experts use a flow-matching objective to predict the transformation from noisy action chunks toward demonstrated chunks; behavior cloning samples denoising timesteps continuously, while inference uses discrete steps.The action expert generates action chunks for continuous robot control.Source: §2.1 · p. 2
  • Timestep Modules transform scalar denoising timesteps into vector embeddings that modulate hidden states, producing timestep-dependent scale, shift, and gate vectors.In π0.5, a Time MLP and AdaRMS generate the vectors, which depend only on the timestep; a fixed K-step schedule yields K distinct modulation sets per layer at inference.Source: §2.1 · pp. 2–3

2.2 REINFORCEMENT LEARNING FOR VLA

  • On-policy PPO updates the action expert using rollout trajectories at discrete denoising timesteps, unlike behavior cloning’s training across continuous timesteps in [0, 1].The PPO objective uses a policy ratio, estimated advantage, and clipping threshold.Source: §2.2 · p. 3

3 PARAMETER UPDATES DURING VLA REINFORCEMENT LEARNING

Across flow-based VLA policies, RL produces dense but low-rank updates concentrated in the Timestep Modules. Module-replacement experiments show that these modules retain more of the RL gain than the other parameters, despite their smaller size.

3.1 WHERE AND HOW DOES RL UPDATE VLA PARAMETERS

  • RL updates to Timestep Modules have effective rank 6–30 versus 250–650 for BC, despite their 70–95% update density.Attention and MLP modules have less than 5% update density; Figure 2 shows density for LIBERO-Spatial π0.5 and effective rank across π0.5 checkpoints.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 DO TIMESTEP MODULES CAPTURE THE RL GAIN?

  • Timestep Modules only nearly matches the full RL policy, while MLP+Attn only loses up to 39.7% on ManiSkill.In π0.5, Timestep Modules only reaches 69.6% on MetaWorld and falls behind the full policy by 1.4%, 1.6%, and 0.7% on LIBERO, ManiSkill, and CALVIN, respectively.Source: Table 1 · p. 5, §3.2 · p. 5
  • Timestep Modules comprise 14.83–27.58% of parameters, yet preserve more of the RL gain than MLP+Attn only.The replacement experiments start from RL checkpoints and revert either attention and FFN parameters or Timestep Module parameters to their base values.Source: §3.2 · p. 5
  • Top-four-direction reconstructions retain most Timestep Module performance, while removing those directions largely degrades it.Timestep-targeted LoRA also converges faster and reaches a higher success rate than LoRA on other modules.Source: §3.2 · p. 5

4 WHAT DO THE TIMESTEP MODULES ENCODE?

The paper examines how AdaRMS encodes RL learning signals in its timestep-conditioned outputs. RL specializes updates to rollout timesteps, while shift vectors change more distinctly than scale and gate vectors.

4.1 SPECIALIZATION TO DISCRETE TIMESTEPS

  • RL produces sharply localized AdaRMS responses around rollout timesteps, unlike standard BC’s smooth responses.Discrete-timestep BC shows similarly localized responses and likewise low-rank updates, indicating that discrete-timestep training underlies the low-rank structure.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 DISENTANGLING SCALE, SHIFT, AND GATE

  • RL-induced changes align more with base scale and gate outputs than with shift outputs, making shift the more distinct change.The relative alignment varies more across denoising timesteps than across Transformer sublayers.Source: §4.2 · pp. 6–7

5 UNDERSTANDING THE SHIFT VECTOR

RL-induced shift updates capture outcome-relevant directions and task relationships, and steering along them improves trained policies. The section probes their predictive value, cross-task geometry, and use at inference time.

5.1.1 DO SHIFT UPDATES ENCODE TASK OUTCOMES?

  • 98.6–99.6 ROC-AUC: shift-update projections predict episode outcomes across four benchmarks, well above random-label controls.Probes are predictive for both base and RL policies; a single sublayer–timestep direction reaches 96.9–99.0 ROC-AUC on LIBERO-Spatial, LIBERO-Object, and ManiSkill.Source: §5.1.1 · pp. 7–8

5.1.2 DOES SHIFT UPDATE GEOMETRY ENCODE TASK RELATIONSHIPS?

  • 0.504 mean task-wise correlation (range: 0.224–0.794): shift-update alignment is positively associated with cross-task transfer behavior.Figure 4 compares pairwise shift-vector similarity with cross-task transfer similarity; their structures closely resemble one another.Source: Figure 4 · p. 8, §5.1.2 · p. 8

5.2 EXPLOITING THE SHIFT UPDATES

  • 2.0–4.0% gains: adaptive shift steering improves final RL success across all five benchmarks and beats matched-norm random directions.On LIBERO-Spatial, success rises from 86.7% to 90.0%; on LIBERO-Goal, it rises from 90.7% to 94.7%.Source: Table 3 · p. 8, §5.2 · pp. 8–9

6 RELATED WORK

The related work describes flow-based VLA action experts and online RL post-training, while identifying a gap in understanding how RL changes those experts.

6.1 VISION-LANGUAGE-ACTION MODELS

  • Flow-based VLAs use dedicated action experts to generate action chunks, and prior work has studied their representations through probing and steering.The paper distinguishes its focus on how RL reshapes representations from prior VLA representation studies (H¨aon et al., 2025; Wang et al., 2026a; Gu et al., 2025; Buurmeijer et al., 2026).Source: §6.1 · p. 9

6.2 REINFORCEMENT LEARNING

  • Prior work post-trained VLAs through online RL in simulation, but what happens to action experts during RL had not been studied.The cited work includes online VLA RL (Tan et al., 2025) and recipes for optimizing flow-based policies (Chen et al., 2025).Source: §6.2 · p. 9

7 CONCLUSION AND DISCUSSION

The paper concludes that RL signals in flow-based VLAs are concentrated in low-rank Timestep Module updates, especially their shift vectors. It points to parameter-efficient post-training and task-behavior composition as future directions.

7 CONCLUSION AND DISCUSSION

  • Across π0.5 and GR00T on four benchmarks, RL learning signals appear in low-rank Timestep Module updates, with much of the signal concentrated in the shift vector.Probing and steering experiments support a vector-level account of RL post-training as timestep-specific modulation of action-expert hidden states.Source: §7 · p. 9
  • Future work could investigate discrete-timestep behavior in BC and whether small residual modules capture much of full-parameter RL’s benefit.The paper presents these as directions motivated by its findings, not established outcomes.Source: §7 · p. 9
  • The authors plan to release experiment code and relevant artifacts, including trained checkpoints, upon acceptance.Detailed hyperparameters and settings are provided in § B.Source: REPRODUCIBILITY STATEMENT · p. 10

A VLA ARCHITECTURES AND TIMESTEP MODULES

This appendix defines how Timestep Modules are counted and summarizes their implementations across π0.5, GR00T, and SmolVLA, followed by experiment and checkpoint details.

A VLA ARCHITECTURES AND TIMESTEP MODULES

  • Timestep Modules count parameters that map timestep embeddings to conditioning vectors, excluding non-timestep transformations and counting shared parameters once.For joint timestep-action inputs, the count includes only timestep-input weights and excludes the action-input block, joint bias, and downstream mixed-feature projections.Source: A VLA ARCHITECTURES AND TIMESTEP MODULES · p. 16
  • π0.5 uses a shared Time MLP and layer-specific AdaRMS projections that produce scale, shift, and gate vectors; Timestep Modules comprise 27.58% of its action expert.AdaRMS accounts for 98.23% of those modules.Source: A VLA ARCHITECTURES AND TIMESTEP MODULES · p. 16

A.2 GR00T: ADALAYERNORM AND ACTION–TIMESTEP ENCODING

  • GR00T N1.5/N1.6 combine adaptive LayerNorm with an action–timestep encoder; their Timestep Modules contain 158.52M and 234.07M parameters, respectively.The counted modules account for 14.83% and 16.50% of the respective action experts.Source: §A.2 · pp. 16–17

A.3 SMOLVLA: ACTION–TIMESTEP MLP

  • SmolVLA uses an action–timestep MLP whose timestep-input block contains 0.52M parameters, or 0.52% of the action expert.Unlike the other described designs, it does not generate layerwise timestep-dependent scale, shift, or gate vectors.Source: §A.3 · p. 17

B EXPERIMENT SETTINGS

  • Experiments used four NVIDIA H100 80GB GPUs paired with an Intel Xeon Platinum 8480+.The appendix identifies the hardware used across experiments.Source: B EXPERIMENT SETTINGS · p. 18

B.1 MODEL LIST

  • The checkpoint set includes public and in-house models; RL checkpoints use PPO except SmolVLA, which uses GRPO.π0.5 uses Flow-SDE-based RL checkpoints for MetaWorld, ManiSkill, and CALVIN.Source: §B.1 · p. 18, Table 5 · p. 18

B.2 IN-HOUSE TRAINING SETUP

The authors train in-house π0.5 and GR00T policies for specified LIBERO analyses, using PPO with Flow-SDE for RL. They also provide additional results across the extended-results sections.

B.2 IN-HOUSE TRAINING SETUP

  • In-house runs cover π0.5 RL on LIBERO-Spatial, Object, and Goal; GR00T N1.5/N1.6 RL on LIBERO; and π0.5 BC on Spatial and LIBERO.The GR00T runs cover a few steps; π0.5 BC supports task-controlled training and the discrete-timestep ablation.Source: §B.2 · p. 18
  • PPO is the base RL algorithm and Flow-SDE provides exploration, with 240 interaction steps for π0.5 LIBERO-Object and five denoising steps for LIBERO-Spatial.The authors otherwise mostly follow recommended hyperparameters.Source: §B.2 · p. 18, Table 6 · p. 19

C EXTENDED RESULTS

C.1 THRESHOLD ABLATIONS ON PARAMETER SHIFT ANALYSES

The supplementary analyses test threshold robustness and extend the update and replacement findings across VLA and image/video models. They also compare timestep-conditioned directions and the scale, shift, and gate outputs.

C.1 THRESHOLD ABLATIONS ON PARAMETER SHIFT ANALYSES

  • Even at 99% explained energy, AdaRMS updates use 8.2–9.3% of available rank under RL versus 35.3–46.0% under BC; attention and MLP ranks remain similarly high.The Timestep-Module BC–RL rank difference persists across tested energy thresholds.Source: Figure 5 · p. 20, §C.1 · p. 20

C.2 PARAMETER UPDATES ON FLOW-BASED VLA MODELS

C.3 PARAMETER UPDATES IN IMAGE AND VIDEO DIT MODELS

  • Attention and MLP account for approximately 90–99% of measured update energy in RL-trained image and video models, while AdaNorm ranks vary across models.AdaNorm updates are relatively high-rank in SeFi-Image and VideoRLVR but lower-rank in DanceGRPO and Pref-GRPO; Time MLP ranks also vary, including low-rank image-model updates.Source: Figure 8 · p. 22, §C.3 · p. 22

C.4 LOW-RANK PARAMETER REPLACEMENT

  • The top four singular directions alone retain nearly all π0.5 Timestep-Modules-only performance, while removing them drops performance below the base policy on five of six benchmarks.For GR00T, the top directions still carry most of the gain despite high-rank noise from low-precision training.Source: Table 9 · p. 23, §C.4 · p. 23, Table 10

C.5 TARGETED LORA

  • RLinf’s default LoRA targets omit the π0.5 Time MLP and AdaRMS projections.Source: Figure 9 · p. 23

C.6 TIME CONDITIONING DIRECTIONS IN ADARMS UPDATES

  • The layerwise analysis compares BC and RL AdaRMS input directions across MetaWorld and CALVIN, and examines LIBERO-Spatial RL and discrete-timestep BC settings.The analysis covers all 18 action-expert layers, separating attention and MLP AdaRMS sites and plotting the top three input singular directions for joint scale–shift–gate updates.Source: §C.6 · p. 24

C.7 COMPARING SCALE, SHIFT, AND GATE

C.8 PROBING RESULTS

The authors probe whether shift directions encode task outcomes using π0.5 base and RL policies. Single-position directions can be highly predictive, and probe predictions separate successful from failed episodes early in execution.

C.8 PROBING RESULTS

  • 96.9–99.0 ROC-AUCs are reached by single sublayer–timestep shift-direction probes across LIBERO-Spatial, LIBERO-Object, and ManiSkill.Probes use per-benchmark 70:30 train–test splits; the single position is selected by best training-split F1.Source: Table 12 · p. 34, §C.8 · p. 34
  • Across single positions, larger projections associate with success for some directions and the opposite for others, in similar proportions for base and RL policies.Source: Table 13 · p. 34, §C.8 · p. 34
  • Predicted success separates between successful and failed trajectories early in episode execution.Figure 22 plots probe trajectories over normalized episode progress with 95% confidence intervals.Source: Figure 22 · p. 37, §C.8 · p. 37

C.9 STEERING RESULTS

Steering the base policy along learned shift directions produces substantial gains and recovers a substantial portion of the RL improvement. The best steering strength varies across benchmarks, and excessive steering reduces performance.

C.9 STEERING RESULTS

  • Steering the base policy along learned shift directions produces substantial gains and recovers a substantial portion of the RL improvement.Steering is applied simultaneously across all sublayer–timestep pairs.Source: §C.9 · p. 38, Table 14 · p. 38
  • The optimal steering coefficient α varies across benchmarks, and performance declines when steering becomes too strong.Performance generally improves as α approaches its optimum.Source: Table 14 · p. 38, §C.9 · p. 38

Related papers

Loading 2609.34599v1…