PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Noise Steering: Dual-Latent Space Reinforcement Learning for Generative Robot Policy

Pengfei Zhang, Teng Sun, Xianchao Xiu

arXiv:2609.11270v1cs.RO

TL;DR

Pretrained generative robot policies often need downstream adaptation under distribution shift, while existing reinforcement-learning interfaces mainly steer initial noise and do not directly control intermediate action representations. DLSRL jointly learns initial-noise and action-representation latents, maps the latter to adapter features, and injects them into intermediate action-token hidden states without updating the base policy. Across robotic manipulation tasks and generative policy architectures, it generally accelerates online adaptation while maintaining competitive success rates.

  • Problem

    Generative robot policies require downstream adaptation under pretraining-distribution mismatch, but existing methods mainly steer noisy initial conditions instead of directly modulating intermediate action representations.

  • Method

    DLSRL jointly optimizes initial-noise and action-representation latents, maps the representation latent to adapter features, and residually injects them into intermediate action-token hidden states while keeping the base policy frozen.

  • Results

    DLSRL generally improves online adaptation speed across RoboMimic and LIBERO while maintaining competitive success rates; on Can, it reaches approximately 99% success versus about 90% for DSRL later in training.

  • Takeaways & Limitations

    Combining initial-noise steering with intermediate representation modulation makes more effective use of online interactions when adapting frozen generative robot policies.

Abstract

from arXiv · show

Pretrained generative robot policies learn expressive action priors from demonstrations. However, existing reinforcement learning methods only steer the noisy space but fail to modulate intermediate action representations during the generation process, resulting in performance degradation and inefficiency. To address this limitation, we propose a novel Dual-Latent Space Reinforcement Learning (DLSRL) framework, which complements initial-noise steering with representation-level control inside the frozen generator. Specifically, our actor network predicts two distinct latent variables: an initial-noise latent variable that steers behavior generation, and an action-representation latent variable for intermediate feature modulation. Moreover, this representation latent variable is mapped to adapter features and ingeniously injected into the hidden states of intermediate action tokens via residual connections. Our dual-control design enables direct adjustment of action representations without updating the base policy. Experiments across generative policy architectures and robotic manipulation tasks show that DLSRL effectively accelerates online robot policy adaptation and achieves competitive performance. Our code is available at \href{https://github.com/xianchaoxiu/DLSRL}{https://github.com/xianchaoxiu/DLSRL}.

I. INTRODUCTION

Pretrained generative robot policies need online adaptation when deployment environments differ from pretraining, but existing frozen-policy steering mainly controls initial noise rather than intermediate action representations. DLSRL addresses this gap with dual-latent control and reports faster adaptation while maintaining competitive final performance.

  • I. INTRODUCTION: Pretrained VLA policies require downstream adaptation when deployment distributions differ, especially for long-horizon tasks where small errors can cause failure.Offline adaptation data may not reveal mismatches that emerge during closed-loop execution, while full fine-tuning is computationally expensive.
  • I. INTRODUCTION: Existing approaches either fine-tune the generator or steer its initial noise, leaving later action representations only indirectly controlled.This motivates combining global behavioral-mode selection with direct intermediate representation modulation.
  • I. INTRODUCTION: DLSRL combines initial-noise steering with action-token hidden-state modulation inside a frozen generative policy.Its actor jointly produces initial-noise and action-representation latents, while adapter features are injected through residual connections without updating the base policy.
  • I. INTRODUCTION: DLSRL improves online adaptation speed across diffusion and flow-matching policies while maintaining competitive final performance.The experiments evaluate the framework across generative policy architectures and robotic manipulation tasks.

II. RELATED WORK

Generative robot policies can be adapted by modifying the generator or by applying external steering at different stages of action generation. Existing methods include denoising-trajectory guidance and initial-noise control, while intermediate-representation adapters have mainly been developed for image generation rather than online robot feedback.

  • II. RELATED WORK: Generative robot policies support adaptation through generator updates or external steering mechanisms applied during action generation.
  • II. RELATED WORK: DSRL learns an observation-conditioned initial-noise policy for a frozen diffusion policy, whereas DynaGuide modifies the denoising trajectory using an external dynamics model.

C. Intermediate-Representation Adapters

The framework targets online adaptation of frozen generative robot policies by learning lightweight external control while preserving the pretrained generator. It applies the same conditional generative-process formulation to diffusion and flow-matching policies and optimizes discounted task return from online interaction.

  • C. Intermediate-Representation Adapters: Intermediate-representation adapters map external conditions into features that control frozen diffusion models, but DLSRL learns action-representation latents from environmental feedback for robot adaptation.
  • C. Intermediate-Representation Adapters: The problem formulation represents observations with visual input, robot proprioception, and optionally language, then generates continuous action chunks over a prediction horizon.
  • C. Intermediate-Representation Adapters: The formulation covers both diffusion and flow-matching policies because each transforms initial noise into continuous action chunks.
  • C. Intermediate-Representation Adapters: DLSRL learns a lightweight external control module through online interaction while keeping the pretrained base policy frozen.This preserves previously learned actions and avoids end-to-end online fine-tuning of large generative models.

B. Dual-Latent Actor

DLSRL extends noise-space steering with an actor that jointly produces initial-noise and action-representation latents, enabling task-conditioned control beyond generation initialization.

  • B. Dual-Latent Actor: Unlike noise-space RL, DLSRL directly modifies intermediate action representations rather than only selecting the generator’s initial conditions.Noise-space steering primarily influences the behavioral mode produced by the frozen base policy.
  • B. Dual-Latent Actor: The dual-latent actor produces an initial-noise latent z_t and an action-representation latent u_t from the current observation.The initial-noise branch steers the starting point, while the representation branch supports intermediate control.
  • B. Dual-Latent Actor: The action-representation latent is mapped through a lightweight adapter feature mapper before influencing the frozen generator.The adapter is parameterized by ω and produces f_t = A_ω(u_t).
  • B. Dual-Latent Actor: Both latent variables are optimized with online task rewards while the base-policy parameters remain frozen.The dual-latent actor and adapter mapping are updated during training, with critics used in the optimization procedure.

C. Action-Token Hidden-State Modulation

DLSRL injects adapter features into action-token hidden states at selected frozen Transformer blocks, providing representation-level modulation without changing the base architecture.

  • C. Action-Token Hidden-State Modulation: The adapter feature is scaled by λ_inj and residually added to action-token hidden states at selected frozen Transformer blocks.The resulting modulated states are passed to the next Transformer block.
  • C. Action-Token Hidden-State Modulation: The hidden-state formulation applies across diffusion reverse-denoising steps and flow-matching integration steps.The index k denotes the corresponding generation update step for each policy family.
  • C. Action-Token Hidden-State Modulation: Adapter features and action-token hidden states share shape R^(N_a×d_h), eliminating the need for additional projection layers inside the base policy.N_a is the number of action tokens and d_h is the Transformer hidden dimension.
  • C. Action-Token Hidden-State Modulation: The shared adapter feature is injected only at action-token positions, leaving contextual tokens and frozen base-network parameters unchanged.Visual, language, proprioceptive, and other contextual tokens are not directly modified.

D. Dual-Latent Policy Optimization

Dual-latent policy optimization trains critics and the actor around a frozen generator, combining online value learning with latent-space value distillation.

  • D. Dual-Latent Policy Optimization: DLSRL uses an action-space critic and a latent-space critic to optimize dual-latent control without backpropagating through the frozen generation process.The action-space critic estimates returns for decoded action chunks, while the latent critic learns their joint latent-control effect.
  • D. Dual-Latent Policy Optimization: Across six LIBERO tasks, DLSRL generally improves success rates more rapidly than DSRL with the pretrained π0 policy.The figure compares online success rates against the frozen base policy reference.
  • D. Dual-Latent Policy Optimization: The actor objective propagates latent-value gradients through both latent branches and the adapter feature mapper while applying entropy regularization to the initial-noise branch.The actor, adapter mapper, and critics are updated alternately; the base generator remains frozen.

IV. NUMERICAL EXPERIMENTS

The experiments evaluate DLSRL across generative policy architectures and manipulation tasks under fixed environment-interaction budgets, with comparisons including frozen and adapted baselines.

  • Experiments span RoboMimic and LIBERO, including diffusion and flow-matching policies evaluated on multiple manipulation tasks.RoboMimic uses Lift, Can, and Square with a frozen Transformer-based Diffusion Policy; LIBERO evaluates six tasks with the pretrained π0 policy.
  • The numerical experiments include average episode length on six LIBERO simulation tasks, with the best results labeled in bold.
  • The Can-task ablation isolates action-token hidden-state modulation by removing learned initial-noise steering.
  • Success rate is evaluated over 100 episodes during training, with DSRL serving as the main comparison for initial-noise steering.

A. Can DLSRL Improve Online Adaptation Efficiency?

On RoboMimic tasks, DLSRL improves adaptation more rapidly than the compared methods, with the clearest gains on Can and Square while final performance is not uniformly higher.

  • Approximately 99% success on Can versus about 90% for DSRL demonstrates DLSRL's stronger later-stage performance under the same interaction budget.
  • DLSRL improves performance faster across Lift, Can, and Square, reaches near-saturation earlier on Lift, and gains more stably on Square.
  • DLSRL's main advantage is faster adaptation rather than uniformly higher final performance, especially on the more precision-demanding Can and Square tasks.

B. Does DLSRL Transfer to Flow-Matching VLA Policies?

Across six LIBERO tasks using a flow-matching-based VLA policy, DLSRL generally adapts faster, maintains higher performance, and reaches saturation with fewer interactions than DSRL.

  • DLSRL generally improves success rates more rapidly and maintains higher performance over most of the interaction budget across six LIBERO tasks.
  • DLSRL typically reaches saturated performance with fewer environment interactions than DSRL.
  • DLSRL achieves shorter average episode lengths on all six LIBERO tasks, indicating more efficient execution across manipulation scenarios.

C. How Does Representation Modulation Help?

Representation-level modulation independently improves online adaptation, while its effectiveness depends on choosing an injection strength that balances speed and stability.

  • DLSRL-Rep improves Can success from approximately 27% to 52% without learned initial-noise steering, showing an independent benefit from intermediate representation modulation.
  • λinj = 0.09 adapts fastest initially but fluctuates later, whereas λinj = 0.03 improves more gradually and stably.
  • λinj = 0.06 provides the best balance between adaptation speed and stability and eventually reaches 100% success.

V. CONCLUSION

DLSRL enables efficient online adaptation of frozen generative robot policies by combining initial-noise steering with lightweight modulation of intermediate action-token representations. Experiments on RoboMimic and LIBERO show generally faster online adaptation while maintaining competitive success rates.

  • DLSRL combines initial-noise steering with lightweight modulation of intermediate action-token representations inside a frozen generator.This provides an additional control interface without updating the base policy.
  • Experiments on RoboMimic and LIBERO show generally improved online adaptation speed with competitive success rates.
  • Future work will evaluate DLSRL in real-world environments and study injection layers, generation steps, and injection strengths.
Loading 2609.11270v1…