Source-linked AI summary
Benchmarking Reinforcement Learning Algorithms on Real-World Robots
A. Rupam Mahmood, Dmytro Korenkevych, Gautham Vasan, William Ma, James Bergstra
TL;DR
Real-robot reinforcement learning lacks reproducible benchmark tasks and supporting setup guidance. The paper introduces six tasks across three commercial robots and benchmarks four off-the-shelf algorithms, finding strong hyper-parameter sensitivity but some effective configurations across tasks. It also reports computational limitations that constrain faster action cycles.
Problem
Real-world reinforcement learning lacks benchmark tasks and supporting source code, while task-interface details can strongly affect learning performance and reproducibility.
Method
The paper introduces six tasks on three commercial robots and evaluates four off-the-shelf continuous-control algorithms across them.
Results
Performance varied widely across hyper-parameter configurations, and hyper-parameter choices could matter more than algorithm choice across tasks.
Takeaways & Limitations
Careful task-interface and computation design can make some existing learning implementations applicable to physical robots, while tuning remains important.
Takeaways & Limitations
Sequential learning updates and policy passes are expensive, so the experiments used moderately large action-cycle times rather than faster cycles for finer policies.
Abstract
from arXiv · showhide
Through many recent successes in simulation, model-free reinforcement learning has emerged as a promising approach to solving continuous control robotic tasks. The research community is now able to reproduce, analyze and build quickly on these results due to open source implementations of learning algorithms and simulated benchmark tasks. To carry forward these successes to real-world applications, it is crucial to withhold utilizing the unique advantages of simulations that do not transfer to the real world and experiment directly with physical robots. However, reinforcement learning research with physical robots faces substantial resistance due to the lack of benchmark tasks and supporting source code. In this work, we introduce several reinforcement learning tasks with multiple commercially available robots that present varying levels of learning difficulty, setup, and repeatability. On these tasks, we test the learning performance of off-the-shelf implementations of four reinforcement learning algorithms and analyze sensitivity to their hyper-parameters to determine their readiness for applications in various real-world tasks. Our results show that with a careful setup of the task interface and computations, some of these implementations can be readily applicable to physical robots. We find that state-of-the-art learning algorithms are highly sensitive to their hyper-parameters and their relative ordering does not transfer across tasks, indicating the necessity of re-tuning them for each task for best performance. On the other hand, the best hyper-parameter configuration from one task may often result in effective learning on held-out tasks even with different robots, providing a reasonable default. We make the benchmark tasks publicly available to enhance reproducibility in real-world reinforcement learning.
1 Introduction
Physical-robot reinforcement learning has lagged behind simulation because benchmark tasks, reproducible setups, and supporting code are limited. This work introduces shared real-robot benchmarks and studies how task design affects learning.
- Motivation: Physical-robot reinforcement learning has not fully embraced learning directly from first-hand interaction with the world.Prior successes often rely on additional state information, simulation preparation, collaboration, or demonstrations.
- Motivation: Learning performance can depend strongly on action space, action-cycle time, and system delays, making reproduction difficult when setup details are omitted.Without careful task setup, learning with physical robots can become insurmountably difficult.
- Contributions: The work introduces six reinforcement learning tasks based on three commercially available robots and benchmarks four continuous-control algorithms.Most tasks require no additional hardware beyond the basic robot setup.
- Contributions: The benchmark is intended to support shared evaluation and an empirical study of policy-learning algorithms across multiple physical robots.The authors identify benchmark tasks, learning-conducive setups, and multi-robot algorithm comparison as core contributions.
2 Robots
The benchmark spans three commercially available robot platforms: UR5 collaborative arms, Dynamixel MX-64AT actuators, and Create 2 mobile bases. Each platform supports two distinct learning tasks.
- Robot platforms: The study uses three commercially available robots as the basis for its learning tasks.The platforms are UR5 arms, Dynamixel MX-64AT actuators, and iRobot Create 2 mobile bases.
- UR5: UR5 provides six-joint sensing and low-level position or velocity control for UR-Reacher-2 and UR-Reacher-6.Its sensory packets include joint angles, velocities, target accelerations, and currents.
- Dynamixel MX-64AT: The Dynamixel MX-64AT supports position, velocity, or current control and sensing of position, velocity, temperature, current, and load.The study uses it for DXL-Reacher and DXL-Tracker through half-duplex serial communication.
- Create 2: Create 2 offers two actuated wheels, multiple infrared and bump sensors, and streaming sensor access for Create-Mover and Create-Docker.The interface sends target wheel speeds in millimeters per second.
3 Tasks
The benchmark tasks vary in robot, control dimensionality, sensing, objective, and physical difficulty. Their interfaces specify observations, actions, rewards, episode conditions, and real-time computation settings.
- UR5 tasks: UR-Reacher-2 uses two-joint angular-speed control to reach randomly generated targets, with four-second episodes supporting exploration.Targets are sampled within a 0.7m × 0.5m boundary.
- UR5 tasks: UR-Reacher-6 is substantially harder than UR-Reacher-2 because it controls all six joints in three-dimensional target space.Higher-dimensional action and observation spaces and physical reachability constraints enlarge the policy space.
- Dynamixel tasks: DXL-Reacher uses one-dimensional current control for reaching, while DXL-Tracker uses current control to track a moving target.Their action ranges are [−100, 100] mA and [−50, 50] mA, respectively.
- Create 2 tasks: Create-Mover trains forward motion in an enclosed arena using wheel-speed actions, wall sensing, recent-action observations, and distance-based rewards.Episodes last 90 seconds unless a bump sensor triggers termination.
- Create 2 tasks: Create-Docker rewards successful docking while penalizing bumps and encouraging forward motion toward a perpendicular charging station.The charging station is attached to the middle of a wider arena wall.
- Real-time interface: The implementation separates environment and agent computations into processes to reduce execution delays, with task-specific action-cycle times.Action cycles are 150ms for Create-Mover, 45ms for Create-Docker, and 40ms for the remaining tasks.
4 Reinforcement learning algorithms
The study evaluates four continuous-control algorithms using established implementations: TRPO, PPO, DDPG, and Soft-Q. They differ mainly in how they constrain policy updates, optimize deterministic policies, or represent exploration.
- Algorithms: The benchmark compares TRPO, PPO, DDPG, and Soft-Q using implementations from OpenAI Baselines, Rllab, and the original Soft-Q authors.TRPO and PPO are represented through OpenAI Baselines, DDPG through Rllab, and Soft-Q through its authors’ implementation.
- TRPO: TRPO constrains each policy update by limiting the expected KL divergence from the previous policy.It solves the constrained optimization problem using conjugate-gradient methods.
- PPO: PPO replaces TRPO’s KL-divergence constraint with clipping in the objective function to control policy changes.Optimization uses several stochastic-gradient-ascent epochs per update.
- Soft-Q: Soft-Q represents policies as energy-based probability distributions derived from soft action-value functions.The energy model can represent complex multimodal behaviors and provides a natural exploration mechanism.
- DDPG: DDPG learns a deterministic policy that maximizes an estimated action-value function, with exploration added as noise to the policy output.The described implementation uses an independent noise model for exploration.
5 Experiment Protocol
The experiments evaluate four reinforcement learning algorithms across six physical-robot tasks, while probing hyper-parameter sensitivity, initialization effects, cross-task consistency, and held-out-task effectiveness.
- Evaluation design: Four algorithms were run on six robotic tasks to assess sensitivity, consistency, and overall learning effectiveness.The algorithms were evaluated using off-the-shelf implementations.
- Hyper-parameter search: Thirty independently sampled configurations were tested for seven hyper-parameters per algorithm on UR-Reacher-2 and DXL-Reacher.Values were sampled uniformly in logarithmic scale from predetermined ranges.
- Initialization analysis: Four hyper-parameter configurations were rerun with five network initializations to estimate initialization effects.The initial random search used one network initialization per configuration.
- Held-out evaluation: Best configurations from UR-Reacher-2 were evaluated on four held-out tasks to measure cross-task effectiveness.Scripted agents were also evaluated for qualitative comparison with learned policies.
- Training budget: Runs lasted 40,000–300,000 steps, with wall times ranging from about 45 minutes to four hours when reported.All wall times included resets, and Create-Docker reset duration depended on performance.
6 Experimental results and discussion
The experiments show generally repeatable physical-robot learning, strong hyper-parameter effects, and useful but imperfect transfer of configurations across tasks. TRPO, PPO, and Soft-Q learned effectively under selected settings, while computation costs and hardware-specific failures constrained some experiments.
- Repeatability: Learning runs were generally repeatable, with small variation that did not diverge over time except on Create-Mover.The authors attribute this reliability to tight control over system delays.
- Hyper-parameter sensitivity: Hyper-parameter configurations produced widely varying policy quality, including both effective behavior and no learning on UR-Reacher-2 and DXL-Reacher.Figure 3 summarizes 30 configurations per algorithm using boxplots.
- Hyper-parameter sensitivity: TRPO had the smallest interquartile range on both tasks, indicating the least sensitivity to hyper-parameter variation among the compared algorithms.DDPG had the lowest median performance on both tasks.
- Sources of variation: Hyper-parameter choices could matter more than algorithm choice, while network initialization usually caused smaller performance variation.Initialization effects were smaller except in one DDPG case.
- Cross-task consistency: Performance correlations across DXL-Reacher and UR-Reacher-2 were positive for all algorithms, although configuration rankings were not consistent.The result supports using a good configuration from one task as a baseline for another while still optimizing for best performance.
- Algorithm performance: TRPO, PPO, and Soft-Q improved learning across evaluated tasks; Soft-Q learned fastest on UR5 and DXL tasks, while TRPO was never substantially worse than the best final performance.DDPG performed poorly on all UR5 and DXL tasks and was not run on the Create tasks.
- Learned versus scripted control: Scripted solutions outperformed reinforcement learning by a large margin where they were established or easy to script, whereas learned solutions were more competitive for Create-Docker.TRPO also docked successfully quite often on Create-Docker, though its movement was not smooth.
- Limitations: Sequential learning updates and policy forward passes required moderately large action cycles, limiting efficient learning of finer policies at faster cycle times.The authors identify efficient computation ordering and inexpensive or asynchronous updates as necessary for that setting.
7 Conclusions
The study benchmarks four policy-learning algorithms on multiple commercial physical robots and finds strong hyper-parameter sensitivity, while some configurations remain effective across tasks. Its benchmark tasks and source code support reproducible real-world reinforcement learning research.
- The study evaluates TRPO, PPO, DDPG, and Soft-Q across multiple commercially available physical robots.
- Algorithm performance is highly sensitive to hyper-parameter values, requiring retuning on new tasks for best performance.
- More than 450 independent experiments consumed over 950 hours of robot usage, with most experiments highly repeatable and many producing effective learning.
- The publicly available benchmark tasks and supporting source code enable reproducible real-world reinforcement learning research and easier adoption of physical robots.
A.1 Additional details of the robots
The robot-control setups use wired connections between commercially available robots and control computers, with interfaces tailored to each platform.
- All robot setups use wired connections to link the robots with control computers.
- The UR5 communicates over TCP/IP, while MX-64AT actuators use a USB2AX controller and a 12V, 5A power adapter.
- The Create 2 uses a serial connection with iRobot’s Open Interface and streams internal-controller data packets every 15ms.
A.2 Additional details of Create-Docker
Create-Docker is a charging-station docking task whose interface combines wheel-speed control, sensor-derived observations, shaped rewards, adaptive resets, and bounded network size. The appendix also documents the experimental and hyper-parameter-search setup used to compare algorithms.
- Task objective: Create-Docker succeeds when the robot docks with an active charging signal, after which its controller switches to Passive mode and ignores external wheel commands.
- Task interface: The task uses wheel-speed actions in [−150mm/s, 150mm/s]2 and a 20-dimensional observation vector built from charging, wall, bump, previous-action, and processed docking-infrared signals.
- Reward design: Its reward combines a large docking reward with bumping penalties and bonuses for forward motion and perpendicular alignment to the charging station.
- Reward design: The reward weights are a = 150, b = 10, c = 5, and d = 4, with successful docking continuing to provide reward for the rest of the episode.