PaperlayerView exact revision on arXiv

Source-linked AI summary

RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning

Kun Lei, Huanyu Li, Dongjie Yu, Zhenyu Wei, Lingxiao Guo, Zhennan Jiang, Ziyu Wang, Shiyu Liang, Huazhe Xu

arXiv:2510.14830v4cs.ROcs.AIcs.LG

TL;DR

Real-world robotic manipulation needs reliability, efficiency, and robustness comparable to skilled human operation, but safe improvement beyond demonstrations remains difficult. RL-100 combines imitation-trained diffusion policies with staged real-world reinforcement learning, unified denoising-time policy optimization, and one-step consistency distillation. Across eight tasks, it achieves 100% success in 1000/1000 trials, matches or surpasses expert completion efficiency, and maintains robustness during deployment and distribution shifts.

  • Problem

    Real-world manipulation requires human-level reliability, efficiency, and robustness while learning systems must combine scarce human demonstrations with safe, sample-efficient autonomous improvement.

  • Method

    RL-100 builds on imitation-trained diffusion visuomotor policies with staged offline and online RL, a unified clipped PPO objective during denoising, and consistency distillation for one-step control.

  • Results

    100% success was achieved in 1000/1000 trials across eight real-world tasks, while matching or surpassing expert teleoperators in time-to-completion.

  • Takeaways & Limitations

    The results indicate a practical path toward deployment-ready manipulation by starting from human priors and extending performance through human-grounded self-improvement.

  • Takeaways & Limitations

    Broader generalization, long-horizon composition, and orientation-sensitive control remain less explored in the reported experiments.

Abstract

from arXiv · show

Real-world robotic manipulation in homes and factories demands reliability, efficiency, and robustness that approach or surpass those of skilled human operators. We present RL-100, a real-world reinforcement learning framework built on diffusion visuomotor policies. RL-100 unifies imitation and reinforcement learning under a single clipped PPO surrogate objective applied within the denoising process, yielding conservative and stable improvements across offline and online stages. To meet deployment latency requirements, a lightweight consistency distillation method compresses multi-step diffusion into a one-step controller for high-frequency control. The framework is task-, embodiment-, and representation-agnostic, and supports both single-action and action-chunking control. We evaluate RL-100 on eight diverse real-robot tasks, from dynamic pushing and agile bowling to pouring, cloth folding, unscrewing, multi-stage juicing, and long-horizon box folding. RL-100 attains 100 percent success across evaluated trials, for a total of 1000 out of 1000 episodes, including up to 250 out of 250 consecutive trials on one task. It matches or surpasses expert teleoperators in time to completion. Without retraining, a single policy attains approximately 90 percent zero-shot success under environmental and dynamics shifts, adapts in a few-shot regime to significant task variations (86.7 percent), and remains robust to aggressive human perturbations (about 96 percent). Notably, our juicing robot served random customers continuously for about seven hours without failure when deployed zero-shot in a shopping mall. These results suggest a practical path to deployment-ready robot learning by starting from human priors, aligning training objectives with human-grounded metrics, and reliably extending performance beyond human demonstrations.

INTRODUCTION

RL-100 combines human demonstrations with staged real-world reinforcement learning to improve deployment-oriented reliability, efficiency, and robustness. It uses diffusion-based visuomotor control with unified policy-gradient refinement and latency-reducing distillation across diverse tasks and representations.

  • Motivation: RL-100 addresses the challenge of combining strong human priors with autonomous exploration for reliable, efficient, and robust robotic manipulation.The framework targets deployment metrics including success rate, time-to-completion, and robustness.
  • Framework: A three-stage pipeline uses imitation pretraining, iterative offline RL, and targeted online RL to improve deployment performance.Most learning budget goes to offline updates, while online RL targets residual rare failure modes.
  • Generality: RL-100 supports task-, embodiment-, and visual-representation-agnostic operation, including 3D point clouds, 2D RGB images, single-step actions, and action chunks.Observation encoders can be swapped without modifying the rest of the framework.
  • Framework: A unified clipped PPO-style policy-gradient objective refines the diffusion sampler across offline and online training phases.The objective is applied within denoising, while consistency distillation compresses multi-step diffusion into a one-step controller.
  • Deployment results: 100% success was achieved across eight real-robot tasks, with performance matching or exceeding human teleoperation efficiency on multiple tasks.The task suite includes dynamic, deformable, fluid, tool-use, and precision-assembly settings.

REAL-WORLD RESULTS

The evaluation spans eight real-world manipulation tasks, varied embodiments, randomized initial layouts, and multiple control modes. RL-100 shows higher success and shorter completion times than baselines and human operators, with robustness evaluated under zero-shot and few-shot variations.

  • Evaluation suite: Eight tasks cover dynamic rigid-body control, deformable objects, fluids, and precision assembly across single- and dual-arm embodiments.The suite includes UR5, Franka with LeapHand, and xArm–Franka systems, plus single-step and chunked action control.
  • Evaluation protocol: Randomized initial object placements are used to encourage policy generalization across trials.Task-specific initialization ranges are provided in the evaluation protocol.
  • Results: RL-100 achieves higher success rates and shorter time-to-completion than baselines and human operators across tasks.The largest gains occur where fast online corrections or stable dexterous grasping under pose variability are required.
  • Robustness: Robustness evaluation tests zero-shot and few-shot transfer under changed objects, visual distractors, and external perturbations.These protocols assess generalization across environmental and interaction variations.

Main Results

RL-100 improves imitation-based manipulation across diverse tasks, reaching perfect success in evaluated trials and strong zero-shot transfer under altered dynamics.

  • 45.3% and 67.8% mean success rates for DP-2D and DP3 rise to 91.8% after iterative offline RL.
  • +48 points on Box Folding and +44 points on Pouring are the largest gains over DP3 among the hardest tasks.
  • 450/450 DDIM episodes and 550/550 consistency-model episodes succeed across evaluated tasks, including 250/250 Soft-towel Folding trials.
  • 100% success is maintained on altered friction in Dynamic Push-T and modified surface properties in Agile Bowling without retraining.

Few-shot Adaptation

RL-100 adapts to task variations with limited additional training and remains effective under substantial physical disturbances, while improving deployment efficiency.

  • Few-shot adaptation: 1–3 hours of fine-tuning yields 100% success for altered towel material and modified container shape or size.
  • Physical disturbances: 100% success is maintained in Box Folding under combined grasp-stage, pushing, and repeated touch disturbances.
  • Physical disturbances: 96.0% average success is maintained across tested human-applied disturbances.
  • Execution efficiency: 41.4s for RL-100 (CM) versus 65.1s for DP-2D makes Box Folding 1.57× faster in wall-clock completion time.
  • Execution efficiency: 20 successful Dynamic Push-T episodes per unit time exceed the human expert’s 17 and beginner’s 13.
  • Execution efficiency: 322 steps for RL-100 DDIM versus 822 for DP-2D represents up to 2.55× fewer steps when failures and timeouts are included.

Training Efficiency

RL-100 reaches deployment-grade performance with stable online learning and limited human data relative to subsequent policy rollouts.

  • 100% rollout success is sustained after approximately 190 Agile Bowling on-policy episodes.
  • 115 human demonstrations per task are followed by approximately 566 offline-RL and 434 online-RL rollouts.
  • 804 human-collected episodes take 12.5 hours, about 13% of the total data-collection budget under 100 hours.

Comparison to Humans and Human–Robot Collaboration

RL-100 is evaluated against humans in competitive, service, and throughput settings, showing strong performance beyond laboratory demonstrations.

  • Competitive settings: 25 bowling successes for RL-100 compare with 14 for human players across 25 attempts per side.
  • Service deployment: Seven hours of continuous zero-shot shopping-mall juicing without failure demonstrates service deployment outside the lab.
  • Throughput comparison: 20 successful T-push trials match the expert comparison in the same wall-clock budget, versus 13 for a novice.

DISCUSSION

RL-100 combines human demonstrations with iterative self-improvement to achieve reliable, efficient, and robust manipulation across diverse real-world tasks. The authors identify broader evaluation and scaling directions for deployment in unstructured environments.

  • Results: 100% success across 1000/1000 trials demonstrates reliable performance across eight real-world manipulation tasks.The evaluated suite includes dynamic pushing, folding, bowling, pouring, unscrewing, juicing, and box folding.
  • Results: RL-100 matches or surpasses expert teleoperators in time-to-completion while remaining stable during extended deployment.A public mall demonstration ran for more than seven hours without failure.
  • Implications: The framework starts from a small set of human demonstrations and uses human-grounded objectives with iterative self-improvement.The authors connect this approach to reduced data-collection burden and autonomous on-robot rollouts.
  • Future directions: Future work includes scaling to larger multi-task, multi-robot models and studying transfer, scaling laws, and semantic generalization.The authors specifically identify larger diffusion or vision-language-action models as directions for extending the model-agnostic pipeline.
  • Limitations and future work: Evaluation should expand to cluttered, partially observable environments, while autonomous reset and recovery remain practical bottlenecks.Proposed settings include occlusions, changing illumination, non-stationary layouts, and dynamic multi-object scenes.

MATERIALS AND METHODS

RL-100 refines diffusion-based imitation policies with a unified, conservative RL objective that operates across denoising steps and offline-to-online training. Its design supports varied observations, embodiments, control regimes, and deployment requirements while reducing inference latency.

  • Motivation: Learning from scratch on hardware is costly and unsafe for contact-rich, long-horizon tasks with sparse success signals.Diffusion-based imitation learning supplies a low-variance prior grounded in human demonstrations and constrains exploration to physically plausible actions.
  • Motivation: Imitation learning alone does not directly optimize success rate, time-to-completion, or robustness, and may leave rare failures unresolved.
  • RL fine-tuning: RL-100 models diffusion denoising as a two-level MDP and shares the environment-level advantage across denoising steps for stable PPO refinement.This addresses weak credit assignment from applying RL only to the final sampled action.
  • Deployment: Consistency distillation compresses the K-step diffusion policy into a one-step controller, reducing deployment latency while maintaining or improving efficiency and robustness.Single-step control suits reactive tasks, whereas action chunking can reduce jitter in precision tasks.
  • Policy design: The policy accepts RGB images or 3D point clouds and predicts either single-step actions or action chunks through a shared backbone.Observation conditioning combines recent visual frames with proprioception, while output heads differ by control mode.
  • Representation stabilization: Reconstruction and variational information bottleneck regularization stabilize representations, with their weights reduced during RL fine-tuning.The imitation objective combines imitation, reconstruction, and KL terms before the RL-stage reduction.
  • RL fine-tuning: A unified clipped PPO surrogate is used across offline and online stages to support a smooth transition from refinement to exploration.The formulation is intended to prevent performance collapses associated with switching optimization regimes.
  • Training pipeline: The pipeline combines imitation pretraining, offline dataset-expansion rounds, OPE-gated PPO updates, and continuous online post-training.The behavior policy advances after accepted inner updates within each outer offline iteration.

Statistical Analysis

The experiments define success rate as the proportion of successful rollouts and wall-clock time as average completion time over successful trajectories.

  • Success rate is computed as successful rollouts divided by total trajectories.

Description of Tasks

The paper evaluates manipulation across dynamic, dexterous, deformable, fluid, and long-horizon tasks, each designed around demanding contact, coordination, adaptation, or recovery requirements.

  • Dynamic Push-T: Dynamic Push-T requires precise high-frequency pushing into a narrowly fitting slot under randomized block poses and variable edge friction.
  • Dynamic Push-T: Generalization tests vary block poses, surface friction, visual clutter, and external disturbances during Dynamic Push-T.
  • Agile Bowling: Agile Bowling demands trajectory-sensitive pushing on a smooth surface, with small motion differences substantially changing the curling stone’s path.
  • Agile Bowling: Agile Bowling generalization includes varied object positions, a coarser surface for zero-shot transfer, and reversed pin placement for finetuning.
  • Pouring: Pouring requires stable grasping, accurate alignment, and adaptive wrist control for randomized containers and heterogeneous contents, including liquids and altered spouts.
  • Other tasks: Unscrewing, towel folding, and box folding require contact adaptation, bimanual coordination, deformable-object handling, recovery, and resistance to long-horizon error compounding.

PRELIMINARIES

The preliminaries formulate manipulation as an MDP and interpret diffusion denoising as a sub-MDP whose transitions can be optimized with policy gradients, then distilled for fast inference.

  • MDP formulation: Robotic manipulation is modeled as an MDP with states, actions, transition dynamics, rewards, and a discount factor.
  • Offline-to-online RL: PPO unifies offline and online RL stages, using IQL advantages offline and GAE advantages online.
  • Diffusion sampling: DDIM samples through a possibly subsampled denoising schedule with K ≪ T and variance σ_t→m ≥ 0, interpolating deterministic and stochastic transitions.
  • Diffusion sampling: Positive denoising variance permits Gaussian log-likelihood optimization, whereas zero variance yields a deterministic mapping and invalidates Gaussian density calculations.
  • Consistency models: Consistency distillation trains a one-step model against a frozen multi-step teacher, enabling single-evaluation inference.
  • Action execution: The framework supports single actions and action chunks, treating each chunk as one decision with cumulative discounted reward and an equivalent inter-chunk discount.
  • RL formulation: Embedding diffusion denoising as a K-step sub-MDP exposes terminal task rewards to denoising sub-policies through policy-gradient updates.

SIMULATION RESULTS AND ANALYSIS

Simulation results show RL-100 performs strongly across locomotion and manipulation benchmarks, while ablations identify fast consistency inference, bounded noise, representation adaptation, and ε-prediction as important design choices.

  • Benchmark results: RL-100 reaches the highest or near-highest performance across all ten simulated tasks, including strong returns and success rates against diffusion and flow baselines.
  • Benchmark results: RL-100 reaches approximately 10,000 return on halfcheetah-medium-v2, compared with 4,500 for DPPO and 3,000 for DSRL.
  • Sample efficiency and stability: RL-100 converges within 1–2M steps on most tasks versus 3–5M for baselines and shows narrower confidence intervals across seeds.
  • Inference efficiency: RL-100 (CM) reaches 378 Hz inference, exceeding DSRL at 35 Hz and DPPO at 30 Hz, while retaining a compact model size.
  • Consistency distillation: A one-step consistency model matches multi-step DDIM performance and final success near 1.0 while providing K× speedup and enabling 100+ Hz deployment.
  • Ablations: Joint representation adaptation is most stable, moderate noise clipping gives the best trade-off, and ε-prediction achieves approximately 40% higher final success than x₀-prediction.

SUPPLEMENTARY EXPERIMENT DETAILS

The experiments combine calibrated 3D perception, asynchronous multi-robot control, and task-matched teleoperation to collect and evaluate diverse manipulation data. The dataset varies demonstration scale to examine both performance limits and sample efficiency.

  • Demonstration collection: Teleoperation interfaces match task complexity: Apple Vision Pro supports dexterous 3D motions, while a joystick records planar delta-action demonstrations.Approximately 100 demonstration trajectories are collected per task.
  • Perception pipeline: Calibrated depth observations are converted into 3D point clouds, transformed into the robot root frame, spatially filtered, and downsampled to 512 points.A buffered camera pipeline supports asynchronous observation acquisition during deployment.
  • Experimental platform: The platform uses UR5, xArm, and Franka Emika Panda manipulators with dexterous, gripper, or passive fixed end-effectors.All control modules operate asynchronously for low-latency, high-frequency execution.
  • Control execution: Robot-specific execution uses 30 Hz commands with interpolation ranging from 100 Hz to 1000 Hz, depending on the manipulator.The UR5 executes at 125 Hz, the xArm at 200 Hz, and the Franka controller at 1000 Hz.
  • Task coverage: The eight-task dataset spans dexterous three-dimensional manipulation, planar or constrained motion, folding, pouring, unscrewing, and juicing.The collection strategy aims to cover task spaces while keeping demonstration acquisition practical.
  • Dataset design: Demonstration counts vary across tasks, with larger datasets testing performance limits and smaller datasets evaluating sample efficiency.This variation is intentional rather than uniform across the task suite.

ADDITIONAL RELATED WORK

Related work has integrated reinforcement learning with expressive diffusion or flow-based policies while addressing offline-to-online distribution shift and computational cost. Hybrid offline-online strategies and policy-specific optimization provide the immediate context for RL-100.

  • Diffusion and flow policies: Diffusion-based policies model complex, multimodal action distributions and have advanced visuomotor control, trajectory planning, and real-world deployment.Diffusion Policy uses visual conditioning and receding-horizon control for visuomotor action generation.
  • RL for generative policies: Prior work fine-tunes diffusion models with reinforcement learning, including policy-gradient methods designed for diffusion-based policies.These methods seek online optimization while retaining diffusion policies’ expressive benefits.
  • Computational efficiency: Flow Q-Learning trains an independent single-step policy to match a flow model’s output, reducing denoising-related computation while preserving expressiveness.It reports state-of-the-art performance on offline RL benchmarks.
  • Offline-to-online transition: Offline-to-online reinforcement learning must manage distribution shift and catastrophic forgetting while enabling continued improvement.Conservative Q-Learning and Advantage-Weighted Regression are cited as foundational offline-RL approaches.
  • Hybrid learning: Hybrid strategies mix offline and online experiences to improve sample efficiency relative to purely offline or purely online learning.RLPD and other frameworks provide examples of combining offline pretraining with online fine-tuning.
Loading 2510.14830v4…