PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Play Table Tennis From Scratch using Muscular Robots

Dieter Büchler, Simon Guist, Roberto Calandra, Vincent Berenz, Bernhard Schölkopf, Jan Peters

arXiv:2006.05935v1cs.ROcs.LG

TL;DR

Table tennis demands fast, precise robot control, while safe real-hardware RL must accommodate extended exploration at high speeds. The paper trains PAM-driven arms with model-free RL and hybrid sim-and-real training; the robot returns and smashes real balls at reported average speeds of 5 m/s and 12 m/s, respectively.

  • Problem

    Dynamic table tennis requires precise fast motions, but applying step-based RL on real systems makes safe exploration over millions of high-speed time steps a challenge.

    Source: §I · p. 2

  • Method

    The authors use soft PAM-driven arms for model-free RL on real hardware and introduce hybrid sim-and-real training with recorded ball trajectories.

    Source: §I · p. 2, C. HYSR: Hybrid Sim and Real Training · pp. 4–5

  • Results

    The robot returns balls at 5 m s−1 on average and smashes them at 12 m s−1 on average.

    Source: B. Learning to Smash · p. 7

  • Takeaways & Limitations

    The experiments show that PAM-driven robots can support safe real-hardware RL for dynamic table tennis without algorithmic safety constraints or real-ball contact during training.

    Source: §I · p. 2, §IV · p. 8

Abstract

from arXiv · show

Dynamic tasks like table tennis are relatively easy to learn for humans but pose significant challenges to robots. Such tasks require accurate control of fast movements and precise timing in the presence of imprecise state estimation of the flying ball and the robot. Reinforcement Learning (RL) has shown promise in learning of complex control tasks from data. However, applying step-based RL to dynamic tasks on real systems is safety-critical as RL requires exploring and failing safely for millions of time steps in high-speed regimes. In this paper, we demonstrate that safe learning of table tennis using model-free Reinforcement Learning can be achieved by using robot arms driven by pneumatic artificial muscles (PAMs). Softness and back-drivability properties of PAMs prevent the system from leaving the safe region of its state space. In this manner, RL empowers the robot to return and smash real balls with 5 m\s and 12m\s on average to a desired landing point. Our setup allows the agent to learn this safety-critical task (i) without safety constraints in the algorithm, (ii) while maximizing the speed of returned balls directly in the reward function (iii) using a stochastic policy that acts directly on the low-level controls of the real system and (iv) trains for thousands of trials (v) from scratch without any prior knowledge. Additionally, we present HYSR, a practical hybrid sim and real training that avoids playing real balls during training by randomly replaying recorded ball trajectories in simulation and applying actions to the real robot. This work is the first to (a) fail-safe learn of a safety-critical dynamic task using anthropomorphic robot arms, (b) learn a precision-demanding problem with a PAM-driven system despite the control challenges and (c) train robots to play table tennis without real balls. Videos and datasets are available at muscularTT.embodied.ml.

I. INTRODUCTION

Dynamic tasks demand precise, accelerated motions, but safety checks that work for slow robot skills can restrict the fast exploration table tennis requires. The paper uses PAM-driven arms to learn real table tennis from scratch, including smashing and training without real balls.

I. INTRODUCTION

  • Safety checks can restrict the explosive accelerations needed to reach distant hitting points in time, making fast dynamic tasks difficult to explore safely.The paper notes that tuning safety heuristics is harder at higher speeds and that conservative checks limit robot capabilities.Source: §I · p. 2
  • PAM-driven robots let model-free RL explore fast real-hardware motions without algorithmic safety constraints, while a reward term directly favors faster returned balls.The method uses stochastic Gaussian MLP actions on low-level desired pressures and learns from scratch.Source: §I · p. 2
  • Hybrid sim-and-real training replays recorded ball trajectories in simulation while applying actions to the real robot, avoiding real-ball contact during training.This is presented as a way to avoid the practical burden of collecting, removing, and resupplying balls.Source: §I · p. 2
  • The setup avoids solution priors such as safety constraints, regularizers, models, or demonstrations, leaving task-related reward terms and the hardware to shape the learned behavior.The authors argue such priors can steer optimization toward degenerated performance.Source: §I · p. 2
  • The authors identify learning table tennis with muscular robots, learning real-robot smashes, safe model-free RL for dynamic tasks, and training without real balls as novel contributions.These are presented as firsts to the authors’ knowledge.Source: §I · p. 2

II. TRAINING OF MUSCULAR ROBOT TABLE TENNIS

The paper combines muscular-robot actuation with reinforcement learning to return and smash real balls at desired landing points. It introduces dense task rewards and hybrid simulation-real training to make learning practical.

II. TRAINING OF MUSCULAR ROBOT TABLE TENNIS

  • The paper uses RL with muscular robots to learn returns and smashes to desired landing points.The approach is framed as leveraging powerful actuation while addressing PAM control difficulties.Source: §II · p. 2
  • Dense rewards provide feedback for the table-tennis experiments.The training section introduces rewards for returning and smashing.Source: §II · p. 2
  • HYSR combines simulation and real-robot training to enable practical learning of the task.The section introduces this procedure alongside the task setup and rewards.Source: §II · p. 2

A. Muscular Robot Table Tennis Task Setup

The task is to strike an incoming ball so it lands at a desired point, despite the rapid, precise motions and uncertain trajectories that characterize table tennis. PAM actuation offers fast, compliant motion, but its control challenges make learning precision demanding.

A. Muscular Robot Table Tennis Task Setup

  • The robot must hit an incoming ball so it lands at a desired point on the table.The ball trajectory is represented through ball states, with successful strokes evaluated at impact and landing.Source: A. Muscular Robot Table Tennis Task… · pp. 2–3
  • With fixed launcher settings, recorded ball trajectories still vary substantially, especially after the bounce, so fixed-delay timing is insufficient.The described deviations at hit range are around 40 cm along x and 50 cm along z; first-bounce variation is around 50 cm along y.Source: Figure 2 · p. 3
  • Dynamic tasks require quick reactions, precise motions, and high accelerations, unlike many manipulation tasks where slow motion is sufficient.Fast strikes can support smashes and help the racket reach distant hitting positions in time.Source: A. Muscular Robot Table Tennis Task… · p. 3
  • Anthropomorphic robots face safety risks during fast table-tennis motions, including broken cables or exceeding joint limits before the system can stop.High accelerations are harder to generate safely on these systems than on parallel or Cartesian robots.Source: A. Muscular Robot Table Tennis Task… · p. 3
  • PAMs provide high accelerations and passive compliance, while pressure limits can prevent joint-limit violations during fast motion.Despite these properties, PAM nonlinearities, temperature and wear effects, and hysteresis make precise control difficult; the paper uses RL to address those challenges.Source: A. Muscular Robot Table Tennis Task… · pp. 3–4

B. Dense Reward Functions For Returning and Smashing

The reward design separates reaching the ball from directing its post-impact flight, with a dense near-hit signal and task rewards for landing position and speed. The RL problem uses robot and ball observations with desired PAM pressure changes as actions.

B. Dense Reward Functions For Returning and Smashing

  • The RL problem is formulated as an episodic MDP whose objective is to maximize expected discounted reward over state-action trajectories.The state includes ball and robot information, and actions are changes in desired pressures for the PAMs.Source: B. Dense Reward Functions For… · p. 4
  • The reward separates ball contact from post-impact flight evaluation, since landing location changes only when the racket hits the ball.A conditional reward uses the table-tennis score after contact and a hitting reward otherwise.Source: B. Dense Reward Functions For… · p. 4
  • The table-tennis reward scores landing-point proximity, while the smash task also rewards post-impact ball speed.The landing-distance normalization and reward cap are chosen to keep typical values in range and limit penalties for errant high-speed hits; the speed and precision terms are combined multiplicatively.Source: B. Dense Reward Functions For… · p. 4
  • A dense hitting reward uses the minimum temporal distance between ball and racket trajectories to provide feedback on how close the racket came.The racket trajectory is obtained from joint positions using forward kinematics, with racket orientation ignored.Source: B. Dense Reward Functions For… · p. 4
  • The reward contains no safety penalties for joint ranges, acceleration, or action changes; instead, it favors faster hits and task-related outcomes.The authors leave the solution to emerge from the hardware and reward function.Source: B. Dense Reward Functions For… · p. 4

C. HYSR: Hybrid Sim and Real Training

HYSR combines real-robot actions with simulated balls to reduce the practical burden of long training and avoid relying on a full robot simulation. It replays recorded ball trajectories and uses simulation after impact to estimate outcomes for the reward.

C. HYSR: Hybrid Sim and Real Training

  • HYSR applies policy actions to the real robot while keeping the ball in simulation during training.This avoids the practical effort of collecting, launching, and removing real balls, while also avoiding reliance on a simulated PAM-driven robot.Source: C. HYSR: Hybrid Sim and Real Training · p. 4
  • Each episode replays one trajectory sampled from the prerecorded ball dataset.The dataset contains 100 recorded trajectories, which vary partly because of vision-system and launcher noise.Source: C. HYSR: Hybrid Sim and Real Training · p. 5
  • The simulation copies the real robot state each timestep and starts simulating the ball after racket contact, using a rebound model to estimate its outgoing velocity.The rebound model uses incoming ball velocity, racket speed at impact, and racket restitution, and assumes no spin.Source: C. HYSR: Hybrid Sim and Real Training · p. 5
  • HYSR uses simulation to estimate landing position and reward-related context, while avoiding real-ball collection and launching.The procedure could also serve as an entry point for domain randomization or curriculum learning.Source: C. HYSR: Hybrid Sim and Real Training · p. 5

III. EXPERIMENTS AND EVALUATIONS

The experiments use HYSR and reward functions to train a PAM-driven robot to return and smash table-tennis balls. They assess learning on a difficult dynamic task and illustrate system robustness through ball speed and noisy low-level actions.

III. EXPERIMENTS AND EVALUATIONS

  • The experiments test whether RL can learn to return and smash table-tennis balls with a complex real PAM-driven system.The authors highlight robustness by showing returned-ball speed and stochastic-policy actions on the real system’s low-level controls.Source: §III · p. 5
  • The experiments use HYSR training and the paper’s reward functions for both return and smash tasks.The paper directs readers to supplemental videos for the results.Source: §III · pp. 5–6
  • The authors frame the experiments as enabling RL to explore fast soft-robot motions without added safety precautions.Source: §III · p. 5

A. Learning to Return

The return task asks a PAM-driven robot to hit launcher-shot balls toward a desired landing point, despite difficult control and impact-precision demands. After stochastic-policy training, the robot returned most of the real test balls, and its policy also learned to prepare for the hit.

A. Learning to Return

  • The return task requires precise racket control at impact, while PAMs are difficult to model and control.The robot was trained to send launcher-shot balls toward a desired landing position on the opposite side of the table.Source: A. Learning to Return · p. 6
  • The robot trained for 1.5 million timesteps from a randomly initialized stochastic policy, using pressure-change actions and PPO.Each strike was one episode, with a dense return reward at episode end.Source: A. Learning to Return · p. 6
  • The final policy hit 96% and returned 77% of 107 real balls shot by the launcher.Landing points formed a circle around the desired point that overlapped, but was not fully contained within, the opponent’s side.Source: A. Learning to Return · p. 6
  • The learned policy prepared for the hit as well as intercepting the ball, although the stated goal was only to return it to a desired landing point.The paper describes this as learning sensor-to-low-level-control mappings directly; it notes that such an approach requires sufficiently robust hardware.Source: A. Learning to Return · p. 6

B. Learning to Smash

The smash task rewards both ball speed and proximity to a desired landing point, requiring fast but precise motions. Adding a speed bonus raised average returned-ball speed, but the faster hits were less precise and harder to learn.

B. Learning to Smash

  • The smash reward maximizes ball speed while minimizing distance to the desired landing point, favoring explosive motions rather than algorithmic safety constraints.The task is harder than returning because imprecise motions can send the ball off the table.Source: B. Learning to Smash · p. 6
  • The stochastic policy switches antagonistic PAM pressures across their range, with a minimum-to-maximum switch at 0.6 s before ball impact at 0.9 s.The figure marks preparation and hit as separate phases.Source: Figure 4 · p. 6
  • 5 m s−1 on average for return and 12 m s−1 for smash; the final smash policy produced significantly higher post-impact speeds.The speed bonus increased ball speed, and the smash average exceeds the cited 10 m s−1 threshold for smashes in human play.Source: Figure 5 · p. 7, B. Learning to Smash · p. 7
  • The faster hits came at the cost of lower precision, with a higher chance of failing to return the ball.The smash experiment also had higher reward standard deviation and was harder to learn than the return task.Source: B. Learning to Smash · pp. 6–7

C. Robustness

The PAM-driven robot completed extensive real-system training without hardware issues, while its pressure limits kept random-start exploration within joint limits. The agent used noisy low-level actions and learned to switch across the pressure range before impact.

C. Robustness

  • Both tasks trained for about 14 hours and ended at algorithmic convergence, not hardware failure, across 15,676 return and 15,161 smash strokes.The runs used 1.5 million time steps at 100 Hz; the return and smash tasks took 14 h 18 min and 14 h 10 min, respectively.Source: C. Robustness · p. 7
  • Gaussian-MLP actions varied substantially at every time step, and the agent switched first-DoF pressure commands from minimum to maximum and back before the hit.The preparation phase used the full pressure range to place the robot in a beneficial state for hitting; the authors say comparable signals could damage or wear a traditional motor-driven arm.Source: C. Robustness · p. 7
  • The policy began with random weights, yet motions stayed within joint limits because pressure ranges were set to stretch an antagonistic muscle close to its limit.This enabled training without human supervision; the authors contrast it with action filtering, whose tuning can slow motions or risk damage.Source: C. Robustness · p. 7

IV. CONCLUSION

The paper uses PAM-driven arms and model-free RL to learn table-tennis returns and smashes, including training without real balls. It reports a speed–precision tradeoff, identifies sample efficiency and cable-drive friction as limitations, and releases motion datasets.

IV. CONCLUSION

  • The learned return motion initializes, prepares for a hit, and then performs the hit in two distinct striking stages.Initialization takes 2 to 4 seconds.Source: Figure 6 · p. 8
  • Smash landings were faster but more dispersed and shifted from the desired point, whereas return landings’ fitted mean coincided with the target.At smash speeds, small racket-orientation errors produce larger landing deviations, making the strategy more aggressive but riskier.Source: Figure 7 · p. 8
  • The approach learned from scratch to return and smash real balls with PAM-driven robots, without algorithmic safety constraints or real balls during training.HYSR supports thousands of strokes without the robot touching real balls during training.Source: §IV · p. 8
  • The authors identify lower sample efficiency than prior table-tennis approaches and severe nonlinear friction and stiction from the cable drive as limitations.They state that improving sample efficiency and cable-drive effects could improve performance.Source: §IV · p. 8
  • The released datasets contain motions at different speeds and are proposed as benchmarks for dynamics models.The paper also makes experiment data and videos available.Source: §IV · pp. 8–9

APPENDIX A VISUALIZATION OF A RETURN AND SMASH

The visualizations show learned return and smash motions, including a preparation phase followed by a hit. Their joint-angle and pressure traces distinguish the two strategies and place impact at about 0.9 seconds.

APPENDIX A VISUALIZATION OF A RETURN AND SMASH

  • The agent learned distinct preparation and hit phases, using different DoF combinations for return and smash motions.The return uses DoFs 1 and 3; the smash uses DoFs 1 and 4, with DoF 1 gaining momentum from the beginning of the smash episode.Source: Appendix A · p. 10
  • Figure 8 plots joint angles and normalized pressures, marking preparation and hit phases by green and red backgrounds; impact occurs at about 0.9 seconds.Pressure is normalized over 0%–100%, corresponding to 0–3 bar.Source: Figure 8 · p. 11
Loading 2006.05935v1…