PaperlayerView exact revision on arXiv

Source-linked AI summary

SmoothRL: Online Reinforcement Learning During Asynchronous Execution

Guang Gao, Yuxuan Nong, Baifu Huang, Jianan Wang

arXiv:2608.29768v1cs.RO

TL;DR

Integrating online RL with asynchronous robot inference remains underexplored, despite demands for reliable, smooth real-time deployment. SmoothRL models asynchronous execution during policy fine-tuning, and after 250 episodes all three evaluated tasks outperformed the frozen base policy.

  • Problem

    Integrating online RL fine-tuning into asynchronous deployment remains an open question for reliable, smooth real-time robot control.

    Source: §1 · p. 2, §3 · p. 6

  • Method

    SmoothRL fine-tunes pretrained policies with value gradients and models asynchronous execution by restricting policy updates to the actions actually executed.

    Source: §1 · p. 3, §5 · p. 20

  • Results

    After 250 episodes, success improved across all three tasks: dynamic tossing 39% to 94%, pen capping 8% to 83%, and box opening 30% to 90%.

    Source: §4.5 · p. 19

  • Takeaways & Limitations

    SmoothRL improves success on fine-grained real-robot tasks while preserving real-time responsiveness.

    Source: §5 · p. 20

    The framework requires chunk-level inference to meet a fixed latency budget; latency fluctuations that exceed it can destabilize asynchronous execution.

    Source: Limitations · p. 21

Abstract

from arXiv · show

Deploying robot policies in the physical world requires satisfying two fundamental desiderata: reliability and smooth real-time execution. However, deploying state-of-the-art generalist models presents challenges on both fronts. Achieving the precision and robustness required for real-world deployment necessitates sample-efficient online reinforcement learning (RL) to adapt pretrained models. Meanwhile, the increasing scale of robot foundation models has led to higher inference latency. To satisfy real-time constraints under high latency, modern systems adopt asynchronous inference with action chunking, overlapping policy computation with chunk execution to hide latency and enable smooth control. Despite their complementary roles, integrating asynchronous execution with gradient-based online RL remains underexplored. We present SmoothRL, an online RL framework that fine-tunes a pretrained policy within an asynchronous inference loop. SmoothRL follows a value-gradient paradigm, directly updating policy parameters using gradients of the action-value function with respect to policy actions. To enable correct optimization under asynchronous execution, SmoothRL explicitly models the asynchronous inference process during training. Specifically, each generated action chunk is partitioned by frame index into three regions: a committed region, consisting of actions committed by the previous inference cycle; an execution region, containing newly generated actions executed by the robot; and a discarded region, containing actions superseded by the next inference cycle. Gradients are propagated only through the execution region, ensuring policy optimization aligns with the trajectory distribution induced by asynchronous execution. We evaluate SmoothRL on real-world robotic tasks requiring high precision, as well as highly dynamic tasks that necessitate asynchronous execution.

1. Introduction

SmoothRL fine-tunes pretrained robot policies online under asynchronous inference, aligning value-gradient updates with the actions actually executed. It embeds the asynchronous loop in training and targets reliable, smooth real-time control.

1. Introduction

  • Real-world deployment requires sample-efficient online RL for reliability and asynchronous inference to maintain continuous motion despite high policy latency.Synchronous execution pauses at chunk boundaries, while asynchronous chunk switching can create discontinuities; prior chunk-stitching methods address continuity [1] [2] [3] [4].Source: §1 · p. 2
  • SmoothRL targets asynchronous online RL with value gradients, addressing a setting where superseded actions can contaminate policy updates unless the objective excludes them.Offline RL lacks online adaptation, synchronous online RL assumes synchronous execution, and asynchronous restricted online RL updates a latent steering space rather than policy parameters [29] [30] [7] [9] [11] [12] [13] [31].Source: §1 · p. 3
  • SmoothRL partitions each action chunk into committed, execution, and discarded regions, and propagates value gradients only through the execution region.With a fixed latency budget of n frames, the execution window is a function of frame index; the critic retains committed actions to condition reward bootstrapping and model concurrent decisions.Source: §1 · pp. 3–4
  • The asynchronous inference loop runs during training rollouts, so replay records actions executed under the same timing and schedule used at deployment.The in-flight chunk is part of the state for the next decision because inference and execution overlap [35].Source: §1 · pp. 3–4
  • Raw-action human demonstrations can directly supervise the policy over the execution region and provide critic transitions, without latent inversion or separate offline collection.SmoothRL is evaluated on real-world manipulation tasks requiring high precision and dynamic tasks that require asynchronous execution.Source: §1 · p. 4

2. Related Work

The related work distinguishes asynchronous deployment mechanisms from RL methods by whether learning is online or offline and whether value gradients reach policy parameters. It identifies partial execution under asynchronous inference as a constraint on online value-gradient learning.

2.1. Asynchronous Policy Deployment

  • Chunk stitching [1] [2] [3] [4] and trajectory blending [6] [29] smooth asynchronous action chunks differently, but online RL requires the value function to evaluate the exact executed action while preserving a differentiable path to policy parameters.Stitching fixes actions before emission; blending determines them after overlapping chunks are available. This distinction motivates modeling asynchronous inference during online RL.Source: §2.1 · pp. 4–5

2.2. Offline RL

  • Offline methods χ0 [29] and RECAP [30] support asynchronous deployment without direct value-gradient updates to policy parameters, while CO-RFT [10] performs offline chunk-level Q-learning with synchronous deployment.For offline RL, asynchronous execution is an inference-scheduling choice rather than a change to the learning signal; in closed-loop robot learning, generated-versus-executed action mismatch affects that signal.Source: §2.2 · p. 5

2.3. Synchronous Online RL

  • Synchronous online actor-critic methods propagate ∇aQ into policy parameters but assume receding-horizon execution, so partial chunk execution is absent from their formulations.Chunk-level methods [9] [34] [52] share SmoothRL’s optimization unit, whereas per-step methods [11] [12] [13] [27] [28] cannot express gradients restricted to a chunk sub-segment. SmoothRL’s chunk-role partitioning and gradient truncation apply to residual-policy adaptation and end-to-end fine-tuning.Source: §2.3 · pp. 5–6

2.4. Asynchronous Restricted Online RL

  • GR-RL [31] learns latent noise for a frozen diffusion policy [32], bounding partial-execution effects to its pretrained action distribution but limiting improvement and preventing direct regression from human interventions.Other asynchronous online methods avoid applying ∇aQ to raw-action policy parameters; SmoothRL instead embeds asynchronous execution as an optimization constraint and learns directly in raw action space.Source: §2.4 · p. 6

3. Method

SmoothRL integrates online, value-gradient RL with asynchronous robot execution by modeling chunk timing and restricting policy gradients to actions that enter the environment. Its method also instantiates this objective with smoothness constraints and human intervention support.

3.1. The Asynchronous Inference Loop

  • The asynchronous loop issues the latest available action without waiting for inference, while a fixed latency budget n makes chunk boundaries and the execution window predictable.For H ≥ 2n, the committed, execution, and discarded regions are [0,n), [n,2n), and [2n,H); inference slack becomes waiting time.Source: §3.1 · pp. 6–8

3.2. Gradient-based RL in Continuous Action Space

  • SmoothRL treats each action chunk as an atomic action and backpropagates the action-value gradient through the deterministic policy to update its parameters.The critic uses a chunk-skip Bellman backup conditioned on the action sequence that generated the reward.Source: §3.2 · p. 8

3.3. Chunk-Level RL under Asynchronous Deployment

  • SmoothRL optimizes over the executable [0,2n) span, treats the committed region as fixed, and propagates policy gradients only through the executed [n,2n) region while the critic evaluates the full chunk.The chunk-skip backup uses the same 2n-frame span, with reward accumulated over that interval and bootstrap state s_t+2n.Source: §3.3 · p. 9
  • The critic augments state with the committed actions already in flight, while the actor searches executable chunks under explicit smoothness constraints and training rollouts follow asynchronous execution.The fixed latency budget sets the remaining time until the in-flight actions finish; the smoothness constraint is needed because maximizing value alone does not guarantee continuity.Source: §3.3 · pp. 9–10

3.4. Instantiation

  • The instantiation freezes the base policy and uses a lightweight attached actor-critic: the actor adds bounded residual corrections, and its loss combines value maximization, behavioral cloning, and smoothness penalties.The critic receives the committed and execution regions; smoothness penalties bound per-frame velocity, acceleration, and jerk.Source: §3.4 · pp. 11, 13, 14
  • Human intervention can replace the actor chunk or modify it residually; the issued action is recorded as the committed-region input and supplies the behavioral-cloning target for intervened chunks.Residual intervention suits dynamic tasks by preserving chunk temporal structure, while absolute intervention suits tasks requiring greater flexibility and precision.Source: §3.4 · pp. 12–13

4. Experiments

The experiments evaluate SmoothRL on three real-robot tasks spanning dynamic and high-precision manipulation, comparing task success before and after online reinforcement learning. They assess asynchronous execution, task-specific challenges, evaluation protocol, and how fine-tuning changes performance and residual errors.

4.1. Platform

  • On the Astribot S1, inference runs at 5 Hz with a 6-frame latency budget, so each 32-frame chunk has 6 execution frames and 20 discarded frames.The attached actor and critic modify only the 20 arm-action dimensions; torso and head actions remain unchanged.Source: §4.1 · pp. 15–16

4.2. Task Specification

  • The tasks test continuous momentum and release timing in tossing, roughly 5 mm relative-pose precision in pen capping, and millimeter-level blade insertion into box tape seams.Tossing also requires spatial accuracy; in box opening, the seam guides the blade after insertion, making insertion the primary precision challenge.Source: §4.2 · pp. 16–17

4.3. Training Setup

  • Residual intervention achieved approximately 80% success on far-bin tossing configurations, versus approximately 30% for direct VR teleoperation, by preserving the base policy’s velocity profile.Operators provide only corrective residuals rather than controlling release direction and magnitude while maintaining continuity; task rewards are sparse binary judgments at episode end.Source: §4.3 · pp. 17–18

4.4. Evaluation

  • Evaluation uses 10 box-opening, 18 tossing, and 12 pen-capping configurations once each per checkpoint, comparing the frozen policy with checkpoints after 150, 200, and 250 rollout episodes.All checkpoints use the same hardware, asynchronous inference loop, and latency budget; supervised pretraining used 1,500 demonstrations for box opening and pen capping and 500 for tossing.Source: §4.4 · pp. 18–19

4.5. Results

  • After 250 episodes, success rates rose from 39% to 94% for dynamic tossing, 8% to 83% for pen capping, and 30% to 90% for box opening.These comparisons are reported for the frozen base policy and final online-RL checkpoint, respectively.Source: §4.5 · p. 19, Figure 7 · p. 19, Table 1 · p. 20
  • The frozen policy’s failures were systematic: it under-adapted to cap poses, biased the blade left into cardboard, and mis-scaled tossing velocity by target distance.Tossing also showed left-right deviations, with overshooting nearby bins and undershooting distant ones.Source: §4.5 · p. 19
  • Pen capping rose from 8% to 67% success in the first 150 episodes, while box opening dipped to 20% before recovering through 40% to 90%.Tossing and pen capping improved monotonically, with their largest gains in the first 150 episodes; the box-opening drop likely reflects exploration while discovering effective corrections.Source: §4.5 · pp. 19–20
  • At 250 episodes, remaining failures clustered near the correct pose: box-opening blade deviations were 1–2 mm, and tossing and pen-capping errors were visibly smaller than the base policy’s.For tossing, objects landed closer to the bin and the fixed-bias pattern was reduced; for pen capping, axial misalignment was likewise smaller.Source: §4.5 · p. 20

5. Conclusion, Limitations and Future Work

SmoothRL fine-tunes pretrained robot policies with online RL during asynchronous inference, restricting value-gradient updates to executed actions to preserve correspondence with environment behavior. The conclusion also identifies latency and expressiveness limits and outlines directions for broader policy optimization, action blending, and evaluation.

5. Conclusion, Limitations and Future Work

  • SmoothRL fine-tunes pretrained robot policies with online RL during asynchronous inference, restricting policy updates to executed actions and supporting state augmentation and human intervention as training data.Chunk-role partitioning and gradient truncation restore correspondence between optimized and executed actions; raw-action optimization enables concurrent-execution state augmentation.Source: §5 · pp. 20–21

Limitations

  • Latency fluctuations that exceed the fixed inference budget can prevent scheduled chunk handover and destabilize the asynchronous execution loop.The framework requires each chunk-level inference to finish within a predefined latency budget.Source: Limitations · p. 21
  • The RL module’s expressiveness is bounded by VLA-derived features and fixed-radius residual corrections, which may not rectify systematic base-policy biases on target-domain data.The residual radius balances correction range against trust-region regularization.Source: Limitations · p. 21

Future Work

  • Future work could extend value-gradient truncation to broader offline-to-online algorithms and policy architectures, including generative policies such as QAM [55].For generative policies, preserving committed actions becomes a prefix-constrained generation problem.Source: Future Work · p. 22
  • Future work could incorporate action blending into the learning objective so the critic evaluates executed actions, while addressing its temporal-dependency and continuity costs.Blending may require an additional actor forward pass per contributing chunk; its weight decays toward chunk junctions, potentially trading off smoothness against learning-signal strength.Source: Future Work · p. 22
  • Broader task distributions and increasing shift from the pretrained policy could reveal when local corrections fail and distinguish learning-algorithm limits from policy-parameterization limits.Candidate responses include more expressive adapters, fully generative policy updates, or progressively relaxing the reference constraint.Source: Future Work · p. 22

Related papers

Loading 2608.29768v1…