PaperlayerView exact revision on arXiv

Source-linked AI summary

RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulation

Entong Su, Tyler Westenbroek, Anusha Nagabandi, Abhishek Gupta

arXiv:2602.01789v3cs.RO

TL;DR

Pretrained generative imitation policies often fail to generalize across test-time scenarios. RFS jointly steers latent noise for global variation and uses residual corrections for local refinement, consistently outperforming conventional baselines across simulated and real-world dexterous manipulation.

  • Problem

    Pretrained imitation policies often fail to generalize to all test-time scenarios, motivating adaptation of their behavior.

    Source: §1 · p. 1

  • Method

    RFS jointly modulates a flow-matching policy’s latent noise for global behavioral variation and residual actions for local refinement.

    Source: §1 · p. 2, §4.1.2 · p. 4

  • Results

    RFS consistently outperforms conventional baselines across simulation and real-world dexterous manipulation; in simulation, its average success rate is 0.87.

    Source: §5.1.3 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    RFS supports data-efficient fine-tuning of pretrained policies in real-world dexterous manipulation.

    Source: §6 · p. 9

    Point-cloud observations provide limited semantic and contextual understanding, and performance can degrade in cluttered or semantically rich environments.

    Source: §6 · p. 9

Abstract

from arXiv · show

Imitation learning has emerged as an effective approach for bootstrapping sequential decision-making in robotics, achieving strong performance even in high-dimensional dexterous manipulation tasks. Recent behavior cloning methods further leverage expressive generative models, such as diffusion models and flow matching, to represent multimodal action distributions. However, policies pretrained in this manner often exhibit limited generalization and require additional fine-tuning to achieve robust performance at deployment time. Such adaptation must preserve the global exploration benefits of pretraining while enabling rapid correction of local execution errors. We propose Residual Flow Steering(RFS), a data-efficient reinforcement learning framework for adapting pretrained generative policies. RFS steers a pretrained flow-matching policy by jointly optimizing a residual action and a latent noise distribution, enabling complementary forms of exploration: local refinement through residual corrections and global exploration through latent-space modulation. This design allows efficient adaptation while retaining the expressive structure of the pretrained policy. We demonstrate the effectiveness of RFS on dexterous manipulation tasks, showing efficient fine-tuning in both simulation and real-world settings when adapting pretrained base policies. Project website:https://weirdlabuw.github.io/rfs.

1 INTRODUCTION

Generative imitation learning captures multimodal behavior and performs well on complex manipulation, but pretrained policies may not generalize to all test-time scenarios. RFS adapts them through joint input and output modulation, and the paper reports effective simulation pretraining and data-efficient real-world fine-tuning.

1 INTRODUCTION

  • Imitation learning with generative models captures multimodal human behavior and scales to complex manipulation, but often fails to generalize across test-time scenarios.This leaves room for test-time fine-tuning toward production-level performance.Source: §1 · p. 1
  • Prior imitation-plus-RL methods typically rely on closed-form likelihoods and reparameterization, making them incompatible with expressive diffusion and flow models.The adaptation challenge also includes balancing new-task adaptation against preservation of pretrained knowledge.Source: §1 · p. 1
  • RFS jointly modulates initial latent noise and applies an affine residual correction, combining global semantic adaptation with local dexterous refinement.It adapts pretrained generative policies by changing inputs or outputs rather than model parameters.Source: §1 · p. 2
  • RFS controllers achieve robust multiobject grasping in simulation, transfer effectively to the real world, and can be improved with limited human demonstrations.The paper also reports offline-RL real-world fine-tuning that outperforms bootstrapped imitation-learning baselines.Source: §1 · p. 2

2 RELATED WORK

Related work uses demonstrations to bootstrap RL, but fine-grained dexterous control and stable adaptation of generative policies remain challenging. Existing output and input modulation methods offer complementary capabilities, with limitations that motivate combining them.

2 RELATED WORK

  • Demonstrations accelerate RL, including in high-DoF dexterous manipulation, but remain insufficient for the fine-grained precision required in challenging tasks.Prior approaches use imitation losses, replay buffers, reset strategies, or demonstration data alongside online rollouts.Source: §2 · p. 2
  • Offline-to-online methods can improve sample efficiency, but action-space operation can struggle with distribution shift and fine-grained dexterous corrections.Applying RL to diffusion and flow policies is challenging because iterative refinement and high-dimensional action chunks can make fine-tuning unstable.Source: §2 · p. 2
  • Output residuals support local corrections but may not induce global behavioral changes, while latent-noise steering changes high-level behavior but limits off-manifold exploration and fine refinement.These complementary constraints frame the motivation for combining input and output modulation.Source: §2 · p. 2

3 BACKGROUND

The background presents flow matching as a way to generate actions by integrating a learned velocity field, including conditioning that field on state. It then frames RL adaptation through expected discounted return and introduces complementary output- and input-modulation paradigms.

3.1 FLOW MATCHING

  • Flow matching learns a time-dependent velocity field that transports a base distribution toward a target distribution, and Euler integration samples from the learned field.For a conditional policy, the field is conditioned on state and trained using expert state-action data and interpolated action paths.Source: §3.1 · pp. 2–3
  • Integrating the learned conditional velocity field produces policy actions while supporting expressive multimodal behavior and downstream adaptation.The paper uses πθ and vθ interchangeably for flow-based policies.Source: §3.1 · p. 3

3.2 REINFORCEMENT LEARNING

  • The RL objective is to learn a stochastic policy that maximizes expected discounted return in an MDP defined by states, actions, transitions, rewards, and a discount factor.The choice of RL algorithm is not central to the paper's focus on policy modulation.Source: §3.2 · p. 3
  • Residual RL is output modulation that adds a learned correction to a pretrained policy, whereas diffusion steering is input modulation that learns to adjust initial latent noise.The latter steers generative behavior through the denoising process.Source: §3.2 · p. 3

4 RESIDUAL FLOW STEERING: METHOD AND APPLICATIONS

RFS adapts a pretrained flow-matching policy by combining latent-noise steering for global behavioral variation with residual actions for local correction. The method is instantiated through simulation training and real-world offline fine-tuning for dexterous manipulation.

4 RESIDUAL FLOW STEERING: METHOD AND APPLICATIONS

  • RFS unifies latent-noise steering for global adaptation with residual actions for precise local corrections on a pretrained base policy.The framework is presented as supporting efficient learning in high-dexterity tasks and robust sim-to-real transfer.Source: §4 · p. 3

4.1.1 POLICY MODULATION ALGORITHMS FOR ADAPTING PRETRAINED GENERATIVE MODELS

  • Policy modulation adapts a pretrained generative policy through input or output transformations without changing its underlying parameters.The paper frames residual RL and diffusion steering as output and input modulation, respectively.Source: §4.1.1 · p. 4

4.1.2 RESIDUAL FLOW STEERING FOR JOINT LOCAL AND GLOBAL ADAPTATION

  • RFS jointly produces latent noise for global steering and residual actions for local refinement, including compensation for off-manifold requirements or base-policy imperfections.The latent variable shapes overall behavior, while the residual action adds a fine-grained correction.Source: §4.1.2 · p. 4, Figure 1 · p. 4

4.2 INSTANTIATING RFS FOR DEXTEROUS MANIPULATION

  • The dexterous-manipulation pipeline applies online simulation fine-tuning and offline real-world fine-tuning to state- and vision-based flow-matching policies.The overview includes distillation into visuomotor policies and offline adaptation using human-corrected real-world transitions.Source: §4.2 · p. 4, Figure 2 · p. 5

4.2.1 SIMULATION TRAINING: DATA GENERATION AND ONLINE RL

  • A flow-matching policy trained from a small set of simulated VR demonstrations initializes online PPO training for grasp success and stability, then generates demonstrations for visuomotor-policy distillation.The base policy captures coordinated hand-arm behavior; privileged low-level state is available when generating additional simulation demonstrations.Source: §4.2.1 · pp. 4–5

4.2.2 REAL-WORLD FINE-TUNING: OFFLINE RFS

  • For offline real-world adaptation, human corrections are converted into residual-action transitions, then used to train RFS with TD3+BC.The critic uses combined actions, while the actor maximizes the critic with a behavior-cloning regularizer on residual actions.Source: §4.2.2 · p. 5

5 EXPERIMENTS

Experiments compare RFS with reinforcement-learning baselines and component variants across simulated dexterous tasks and real-world adaptation. RFS leads all six simulation tasks, and its full form performs strongest on unseen real objects.

5.1.1 SETUP

  • Six simulated dexterous tasks use approximately 400 VR demonstrations each; comparisons include flow-RL, offline-to-online, demonstration-RL, residual-learning, and latent-only methods.Source: §5.1.1 · p. 6, §5.1.2 · p. 6

5.1.3 EMPIRICAL RESULTS IN SIMULATION

  • RFS yields at least 0.35 higher success across tasks than full-trajectory RL; IQL leads offline-to-online methods but struggles on packing and push-to-grasp.Source: §5.1.3 · pp. 6–7
  • RFS reaches ~0.80 success on packing and push-to-grasp and ~0.90 on other tasks, while residual methods degrade on sequential tasks and DSRL falls to 0.15 stacking and 0.26 pouring.Source: §5.1.3 · p. 7
  • 0.87 average success rate; RFS records the highest rates across all tasks: 0.89 Grasping, 0.94 Pick & Place, 0.78 Packing, 0.72 Push-to-Grasp, 0.95 Stacking, and 0.87 Pouring.Source: Table 1 · p. 7, §5.1.3 · p. 7

5.2 REAL-WORLD OFFLINE FINETUNING

  • The sim-to-real pipeline distills privileged-state simulation data into a point-cloud policy, then adapts it offline using 50 real corrections, compared with zero-shot, BC, co-training, and critic variants.Source: §5.2 · pp. 7–8, §5.2.1 · p. 8

5.3 EMPIRICAL RESULTS IN THE REAL WORLD

  • Co-training reaches 83.3% grasping and 60% pick-and-place on known objects, but unseen generalization is limited; real-only pretraining gains 70%→73% seen and 35%→50% unseen grasping.Source: §5.3 · pp. 8–9
  • Executed-action critic Q(ab + ar, o) outperforms residual-only and residual-plus-base critics; full RFS is strongest on unseen objects, combining global exploration with precise local refinement.Source: Figure 5 · p. 8, §5.3 · pp. 8–9
  • Zero-shot transfer shows offsets, insufficient finger closure, and mistimed or misplaced actions; illustrated RFS corrections jointly adjust timing, grasp pose, and target location.Source: §5.3 · p. 8, Figure 6 · p. 9

6 CONCLUSION

RFS combines latent-noise steering for global exploration with residual corrections for local refinement. The paper reports strong results in simulation and real-world dexterous manipulation, while identifying limits in semantic understanding and online real-world adaptation.

6 CONCLUSION

  • RFS unifies latent-noise steering for global exploration with residual corrections for local refinement.Source: §6 · p. 9
  • Across simulation and real-world dexterous manipulation tasks, RFS consistently outperforms conventional baselines, including residual-only methods and diffusion-based steering.The conclusion attributes this to effective simulation pretraining and data-efficient real-world fine-tuning.Source: §6 · p. 9
  • Point-cloud observations provide limited semantic and contextual understanding, and performance can degrade in cluttered or semantically rich environments requiring higher-level reasoning.Source: §6 · p. 9
  • Real-world adaptation is restricted to offline fine-tuning, preventing online updates for changing conditions and limiting deployment responsiveness and robustness.Source: §6 · p. 9

A.1 SIMULATION SETUP AND TRAINING

The appendix describes simulation design choices, PPO training, and policy distillation for sim-to-real transfer. It specifies the simulator and observations, disturbance and data-collection procedures, and point-cloud robustness measures.

A.1.1 REINFORCEMENT LEARNING DESIGN CHOICES

  • IsaacLab simulation uses robot, object, and target states with fingertip contacts, custom dexterous teleoperation, and joint disturbances every 2–5 steps; Vision Pro is used for data collection.Source: §A.1.1 · p. 15

A.2 PPO HYPERPARAMETERS

  • Online RFS fine-tuning uses PPO with the same hyperparameters across tasks, and its policy outputs latent-noise steering and a residual action.Table 3 lists the PPO hyperparameters used in simulation.Source: §A.2 · p. 15, Table 3 · p. 15

A.2.1 POLICY DISTILLATION

  • Policy distillation uses multi-view camera calibration to the robot base frame, camera-extrinsic and point-cloud noise, and 1,000 demonstrations each for grasping and pick-and-place.Source: §A.2.1 · p. 15

A.3.1 REAL ROBOT EVALUATION

Real-robot evaluation was conducted in a defined workspace, with separate trial counts for known and unknown objects.

A.3.1 REAL ROBOT EVALUATION

  • The real-robot evaluation workspace measured 30×50 cm.Source: §A.3.1 · p. 16
  • The workspace was located 0.35–0.65 m from the robot base and spanned −0.25 to 0.25 m horizontally.Source: §A.3.1 · p. 16
  • Evaluation used 20 trials per known object and 10 trials for each of seven unknown objects.Source: §A.3.1 · p. 16

A.4 TELEOPERATION DATA COLLECTION

The co-training setup used a Vision Pro application for real-world teleoperation data collection, operating at a controlled frequency of 10 Hz.

A.4 TELEOPERATION DATA COLLECTION

  • A Vision Pro application supported real-world teleoperation data collection in the co-training setup.Source: §A.4 · p. 16
  • The application was based on Park and Agrawal (2024).Source: §A.4 · p. 16
  • Teleoperation ran at a controlled frequency of 10 Hz.Source: §A.4 · p. 16

A.4.1 OFFLINE DATA COLLECTION

The appendix describes offline real-world data collection and reward construction for RFS, then specifies real-world fine-tuning and simulation baseline protocols. Human corrections are bounded residuals, while the real-world reward combines object-to-palm distance and lifting height.

A.4.1 OFFLINE DATA COLLECTION

  • SpaceMouse corrections are enabled within ∼10 cm of the table and applied as bounded residuals, limited to ≤1.5 cm translation and ≤0.05 rad finger motion.The CFM rollout is compared with operator input; corrections are conditioned on observation, with the SpaceMouse z-axis mapped to finger motion.Source: §A.4.1 · p. 16, Figure 8 · p. 17

A.4.2 REWARD FUNCTION DESIGN

  • The real-world reward combines object-center-to-palm distance from forward kinematics with object lifting height to encourage stable grasp execution.SAM2 tracks the object in image space and extracts its point cloud; Figure 7 shows reward progression over time.Source: §A.4.2 · p. 16, Figure 7 · p. 17

A.5 TD3+BC HYPERPARAMETERS

  • Real-world RFS fine-tuning uses offline TD3+BC on fixed human-corrected transitions, with point-cloud observations and a PointNet-adapted perception backbone.The hyperparameters for all real-world experiments are summarized in Table 4.Source: §A.5 · p. 16, Table 4 · p. 16

B SIMULATION BASELINES AND CODEBASES

  • Simulation baselines span generative-policy fine-tuning, offline-to-online RL, demonstration-based RL, and residual RL, using recommended protocols and matched simulator, observations, and task specifications.The comparison uses open-source implementations.Source: B SIMULATION BASELINES AND CODEBASES · p. 17
Loading 2602.01789v3…