Source-linked AI summary
SmoothRL: Online Reinforcement Learning During Asynchronous Execution
Guang Gao, Yuxuan Nong, Baifu Huang, Jianan Wang
TL;DR
Generalist robot policies must improve reliability through online RL while maintaining smooth control despite rising inference latency, but asynchronous execution complicates gradient-based optimization. SmoothRL models asynchronous execution during training and updates only executed actions; across three real-world tasks, success substantially improves over frozen policies after online fine-tuning.
Problem
Generalist robot policies require reliable online adaptation and smooth real-time execution, yet asynchronous deployment and gradient-based online RL remain underexplored together.
Method
SmoothRL fine-tunes pretrained policies with value gradients, partitioning chunks into committed, execution, and discarded regions and propagating gradients only through executed actions.
Results
After 250 episodes, dynamic tossing improves from 39% to 94%, pen capping from 8% to 83%, and box opening from 30% to 90%.
Takeaways & Limitations
SmoothRL improves fine-grained real-robot task success while preserving real-time responsiveness, and raw action space supports direct use of human interventions as training data.
Takeaways & Limitations
The framework requires each chunk-level inference to meet a fixed latency budget, and its RL module's expressive capacity is constrained by the base policy.
Abstract
from arXiv · showhide
Deploying robot policies in the physical world requires satisfying two fundamental desiderata: reliability and smooth real-time execution. However, deploying state-of-the-art generalist models presents challenges on both fronts. Achieving the precision and robustness required for real-world deployment necessitates sample-efficient online reinforcement learning (RL) to adapt pretrained models. Meanwhile, the increasing scale of robot foundation models has led to higher inference latency. To satisfy real-time constraints under high latency, modern systems adopt asynchronous inference with action chunking, overlapping policy computation with chunk execution to hide latency and enable smooth control. Despite their complementary roles, integrating asynchronous execution with gradient-based online RL remains underexplored. We present SmoothRL, an online RL framework that fine-tunes a pretrained policy within an asynchronous inference loop. SmoothRL follows a value-gradient paradigm, directly updating policy parameters using gradients of the action-value function with respect to policy actions. To enable correct optimization under asynchronous execution, SmoothRL explicitly models the asynchronous inference process during training. Specifically, each generated action chunk is partitioned by frame index into three regions: a committed region, consisting of actions committed by the previous inference cycle; an execution region, containing newly generated actions executed by the robot; and a discarded region, containing actions superseded by the next inference cycle. Gradients are propagated only through the execution region, ensuring policy optimization aligns with the trajectory distribution induced by asynchronous execution. We evaluate SmoothRL on real-world robotic tasks requiring high precision, as well as highly dynamic tasks that necessitate asynchronous execution.
1. Introduction
SmoothRL targets the joint need for reliable adaptation and smooth real-time robot control by combining online value-gradient RL with asynchronous inference. It models asynchronous execution explicitly, updating policy parameters only through actions that are actually executed.
- Motivation: Real-world deployment requires sample-efficient online RL for reliability and asynchronous inference to manage rising policy latency.These requirements address robustness on diverse tasks and smooth control under high-latency foundation models.
- Framework: SmoothRL combines online learning, asynchronous execution, and value-gradient optimization into one framework for pretrained robot policies.The value function directly supplies action gradients to the policy parameters, unlike approaches that use advantages only for weighting or conditioning.
- Framework: Asynchronous execution partitions each generated chunk into committed, execution, and discarded regions according to which actions affect the robot.The execution region contains newly generated actions actually executed before the next inference cycle supersedes the remainder.
- Training Design: Policy gradients are truncated to the execution region, while the critic retains committed actions to represent the action sequence generating the interval reward.This separates optimization through executed actions from critic conditioning needed for unbiased chunk-skip bootstrapping and concurrent decision modeling.
- Training Design: Raw-action training lets human demonstrations serve directly as behavioral-cloning targets and critic transitions without latent inversion or separate offline collection.SmoothRL is evaluated on high-precision manipulation and dynamic tasks requiring asynchronous execution.
2. Related Work
Prior work addresses asynchronous deployment, offline learning, or synchronous value-gradient updates, but does not jointly solve asynchronous online RL with direct action-value gradients. SmoothRL addresses the resulting mismatch between generated chunks and executed actions.
- Asynchronous Policy Deployment: Asynchronous action chunking is widely used because large robot policies require forward passes lasting tens to hundreds of milliseconds.Synchronous per-segment inference would constrain control frequency to an unacceptable rate and can produce discontinuities between successive chunks.
- Asynchronous Policy Deployment: Chunk stitching and trajectory blending improve continuity but differ in when the executed action is determined.That distinction matters for online RL because the value function must evaluate the exact action executed while retaining a differentiable path to policy parameters.
- RL Approaches: Offline RL can support asynchronous deployment, but its value-based signal influences policies through weighting or conditioning rather than direct value-gradient backpropagation.The policy remains fixed at deployment in the offline setting described.
- RL Approaches: Online actor-critic methods directly propagate value gradients but assume synchronous receding-horizon execution, so partial chunk execution is not modeled.Existing approaches include updates to attached modules or entire pretrained policies, with some operating outside closed-loop real-robot deployment.
- Open Problem: Asynchronous partial execution contaminates exact chunk transitions, motivating alternatives that keep value gradients away from raw-action policy parameters.These alternatives use latent-space learning, value-based action selection, or value functions to rank or verify sampled candidates.
3. Method
SmoothRL integrates online value-gradient reinforcement learning with asynchronous robot inference by modeling which action-chunk frames are committed, executed, or discarded. Training fixes a latency-budget execution window and propagates policy gradients only through actions that enter the environment.
- Problem: The paper asks how online RL fine-tuning can be integrated into an asynchronous deployment loop for smooth real-time robot control.The setting overlaps inference with execution and issues the latest available action chunk without waiting for inference to complete.
- Asynchronous inference loop: Each action chunk is partitioned into committed, execution, and discarded regions according to asynchronous inference timing.Committed actions come from the preceding chunk, execution actions are issued as generated, and discarded actions are superseded by the next chunk.
- Asynchronous inference loop: A latency budget n fixes the committed region to [0,n), the execution region to [n,2n), and the discarded region to [2n,H).This replaces independently fluctuating region lengths with an identical frame-indexed execution window for every chunk, while converting latency slack into waiting time.
- Gradient-based RL: Value-gradient RL backpropagates the action-value gradient through the deterministic policy to directly update policy parameters.The chunk-level formulation treats each action chunk as an atomic MDP action and conditions the critic on the action sequence associated with the accumulated reward.
- Chunk-level RL under asynchronous deployment: SmoothRL restricts policy gradients to the execution region because committed actions are overwritten and discarded actions never affect the environment.The critic still receives the full relevant action sequence, including committed actions, so its backup remains conditioned on the actions that generated the interval reward.
4. Experiments
SmoothRL is evaluated on three real-robot manipulation tasks spanning dynamic and high-precision settings. Across 250 online RL episodes, performance improves substantially over the frozen base policy, while intervention results show the value of residual corrections for dynamic tossing.
- Tasks: The evaluation spans dynamic tossing, bimanual pen capping, and high-precision box opening on a real robot.The tasks cover high-speed dynamic manipulation and precision-sensitive bimanual manipulation.
- Training setup: The asynchronous training setup uses a frozen base policy, attached actor and critic networks, and concurrent rollout and off-policy update processes.Algorithm 1 initializes the frozen policy, latency budget, discount, update parameters, and shared replay/update processes.
- Training setup: Residual intervention reaches approximately 80% success on farthest tossing configurations, compared with approximately 30% for direct chunk-level teleoperation.Residual intervention preserves the base policy’s velocity profile while the operator supplies corrections, reducing control burden and preserving motion continuity.
- Results: After 250 episodes, dynamic tossing improves from 39% to 94%, pen capping from 8% to 83%, and box opening from 30% to 90%.These success rates are measured against the frozen base policy under the evaluation protocol.
- Results: The frozen policy exhibits systematic pose- and distance-dependent errors, while online RL progressively corrects these offsets across tasks.Pen capping fails to adapt to cap poses, box opening has a leftward blade bias, and tossing miscalibrates release velocity and lateral direction.
- Results: At the final checkpoint, remaining errors are small spreads around the correct pose, including 1–2 mm left-right deviations during box opening.The comparison emphasizes that these tasks require closing the final few millimeters for success.
5. Conclusion, Limitations and Future Work
SmoothRL fine-tunes pretrained robot policies with online RL under asynchronous inference, improving fine-grained real-robot task success while preserving real-time responsiveness. The framework has two stated limitations: fixed latency-budget assumptions and residual-policy expressiveness bounded by the base policy.
- SmoothRL improves success on fine-grained real-robot tasks while preserving real-time responsiveness under asynchronous inference.
- The framework partitions asynchronous action chunks and truncates policy gradients to the executed region, restoring correspondence between optimized and environment-executed actions.
- State augmentation supports concurrent execution, while raw-action operation allows human interventions to enter training naturally as data.
- Limitations: The current instantiation requires each chunk-level inference to complete within a fixed latency budget; latency overruns can destabilize asynchronous execution.
- Limitations: The RL module produces bounded residual corrections, so systematic base-policy biases may exceed its expressive capacity to rectify.
- Future Work: Future work could propagate value gradients through entire generative policies and accommodate broader policy architectures.
6. Contributions
The supplied contribution passage identifies the contributors and project lead.
- Guang Gao and Yuxuan Nong are contributors, and Jianan Wang is the project lead.