Source-linked AI summary
HITTER: A HumanoId Table TEnnis Robot via Hierarchical Planning and Learning
Zhi Su, Bike Zhang, Nima Rahmanian, Yuman Gao, Qiayuan Liao, Caitlin Regan, Koushil Sreenath, S. Shankar Sastry
TL;DR
Table tennis exposes the difficulty of rapid dynamic manipulation for humanoid robots because balls exceed 5 m/s and require sub-second reactions. The paper combines model-based trajectory and racket planning with an RL whole-body controller, achieving up to 106 consecutive shots against a human and autonomous humanoid rallies.
Problem
Humanoid robots remain limited in manipulation tasks involving fast-moving objects, where table tennis demands perception, prediction, planning, and striking within sub-second reaction times.
Method
A hierarchical framework combines a model-based planner for ball trajectory and racket targets with an RL whole-body controller trained using consecutive strikes and human motion references.
Results
106 consecutive shots were achieved against a human opponent, and two humanoids sustained continuous rallies autonomously.
Takeaways & Limitations
The results demonstrate real-world humanoid table tennis with agile, human-like striking under sub-second reaction times.
Takeaways & Limitations
The system assumes a fixed hitting plane and negligible spin, limiting reachability, striking strategies, and stroke diversity.
Abstract
from arXiv · showhide
Humanoid robots have recently achieved impressive progress in locomotion and whole-body control, yet they remain constrained in tasks that demand rapid interaction with dynamic environments through manipulation. Table tennis exemplifies such a challenge: with ball speeds exceeding 5 m/s, players must perceive, predict, and act within sub-second reaction times, requiring both agility and precision. To address this, we present a hierarchical framework for humanoid table tennis that integrates a model-based planner for ball trajectory prediction and racket target planning with a reinforcement learning-based whole-body controller. The planner determines striking position, velocity and timing, while the controller generates coordinated arm and leg motions that mimic human strikes and maintain stability and agility across consecutive rallies. Moreover, to encourage natural movements, human motion references are incorporated during training. We validate our system on a general-purpose humanoid robot, achieving up to 106 consecutive shots with a human opponent and sustained exchanges against another humanoid. These results demonstrate real-world humanoid table tennis with sub-second reactive control, marking a step toward agile and interactive humanoid behaviors.
I. INTRODUCTION
Humanoid table tennis requires sub-second perception, prediction, planning, and agile whole-body control. The paper addresses this with a hierarchical planner-controller framework validated in autonomous real-world rallies.
- Motivation: Ball speeds exceeding 5 m/s force table-tennis robots to perceive, predict, plan, and strike within sub-second reaction times.Success also requires coordinated arm swings, waist rotations, stepping, balance recovery, and readiness for subsequent rallies.
- Approach: The framework combines a model-based planner for striking position, velocity, and timing with an RL-based whole-body controller for human-like strikes.The planner supplies racket targets while the controller generates coordinated motions.
- Approach: Consecutive-strike training teaches agile motions and balance recovery, while two human reference motions encourage natural forehand and backhand strikes.The controller is trained to switch between swing types and track planner-specified targets.
- Validation: The system is demonstrated on a general-purpose humanoid robot without specialized hardware or teleoperation, including rallies against humans and humanoids.The experiments evaluate autonomous table-tennis play in real-world settings.
- Validation: 106 consecutive shots were achieved against a human opponent.This result is reported alongside humanoid-human and humanoid-humanoid rally experiments.
II. RELATED WORK
Prior robotic table-tennis research spans model-based control, learning-based methods, and specialized hardware, while humanoid whole-body control has progressed through imitation and policy decomposition. This work builds on those lines for rapid humanoid striking.
- Robotic table tennis: Earlier robotic table-tennis systems used biologically inspired motion generation, kinesthetic demonstrations, model predictive control, and model-based hitting-velocity control.These approaches addressed motion generation, motion mixtures, diverse hit styles, or accurate returns.
- Robotic table tennis: Learning-based table-tennis research includes fast techniques, hierarchical sim-to-real adaptation, and reinforcement learning that can learn rallies from scratch in under 200 trials.The cited work also includes progress beyond specialized robotic table-tennis devices.
- Humanoid control: Humanoid whole-body control has advanced from motion-specific imitation toward general motion trackers and, in some approaches, separate upper- and lower-body policies.These developments provide foundations for coordinating diverse humanoid movements.
III. SYSTEM OVERVIEW
The system combines motion capture, model-based ball prediction, and an RL whole-body controller. Planner and controller performance are evaluated separately for prediction accuracy and agile target reaching.
- System overview: Nine OptiTrack cameras track the ball at 360 Hz with millimeter-level accuracy, supplying position estimates to the planner.The robot uses these estimates to predict hitting position and time and compute desired racket velocity.
- System overview: The planner predicts racket striking position, velocity, and timing, then computes a base target from the robot base and predicted racket position.The WBC policy receives component observations and proprioceptive information to generate whole-body motions.
- Ball trajectory prediction: Ball velocity is estimated by fitting a second-order polynomial to the nearest 31 position measurements before trajectory prediction.The hybrid dynamics model combines continuous flight dynamics with discrete bounce-impact dynamics and neglects spin-induced effects.
- Ball trajectory prediction: The planner uses explicit step-by-step integration and a predefined virtual hit plane at x = −1.37 m to compute hitting time and position.The return objective uses simplified racket-ball interaction assumptions, including a racket plane perpendicular to velocity.
- Evaluation: 943 successful simulated trials yielded a 94.3% success rate, with targets reached in under 0.8 s on average for initial distances below 0.75 m.This is faster than the 0.86 s strike time.
C. Racket-ball Interaction
The paper simplifies racket-ball interaction by targeting a valid return rather than an exact landing position. Desired outgoing velocity and racket velocity are computed from the target landing point, impact position, flight time, and predicted incoming velocity, then passed to the whole-body controller.
- The interaction model assumes the racket plane is perpendicular to its velocity at impact and prioritizes a valid return over precise landing control.The racket-ball model uses a coefficient of restitution along the racket normal and neglects tangential friction.
- The desired outgoing ball velocity is computed from the target landing position, hitting position, and predefined flight time under gravity.The target landing position is set to the center of the opponent’s side of the table.
- The desired racket velocity is derived from the desired outgoing velocity, predicted incoming velocity, and their difference direction.The unit vector u points in the direction of vo − vi, and the resulting racket command is combined with predicted hitting position and time.
- The computed racket velocity, hitting position, and hitting time are passed to the learning-based whole-body controller to generate robot motion.
V. LEARNING-BASED WHOLE-BODY CONTROLLER
The whole-body controller receives planner-generated racket targets and learns to produce the corresponding humanoid motions. It is trained end-to-end with PPO in Isaac Lab and deployed to the real robot zero-shot.
- The model-based planner predicts the desired racket striking position, velocity, and timing for the whole-body controller.
- The Whole Body Controller policy is trained end-to-end with the model-free PPO reinforcement learning algorithm in Isaac Lab.
- The trained policy is deployed to the real humanoid robot in a zero-shot manner.
A. Human Motion References
The system uses two human swinging references to encourage continuous, human-like humanoid striking motions. Video demonstrations are reconstructed as SMPL motion clips and retargeted to the robot, while the resulting references support agile whole-body behavior.
- Continuous humanoid motion is shaped toward two swinging references: forehand and backhand.
- The references come from video-based human demonstrations rather than kinesthetic teach-ins.
- The whole-body policy produces agile reaching and balance-preserving strikes during real-world play.The robot can move from the right side of the table to the left while maintaining balance and striking the ball.
- Human video is reconstructed into an SMPL motion clip and retargeted to the humanoid robot.The resulting motion clip contains base pose and joint positions at 30 Hz.
B. Markov Decision Process (MDP) Setting
The MDP separates base and racket commands and trains the policy on consecutive, alternating swing targets. This setting supports rapid repositioning, human-like striking, and transitions between forehand and backhand motions.
- 1) Separate Commands for Base and Racket: Separate commands specify the desired base position and the racket’s position relative to the base and velocity.The base command is expressed in the world frame, while racket position is relative to the base and racket velocity is expressed in the world frame.
- MDP Setting: Each 10 s episode samples the next forehand or backhand swing and conditions target positions and velocity on that swing type.The striking plane is fixed 0.4 m in front of the robot, so only the racket target’s y and z coordinates are sampled.
- MDP Setting: The policy’s whole-body control produces human-like strikes with coordinated waist rotation during hits.
- MDP Setting: The total reward combines upper-body imitation, commanded-goal tracking, and regularization.The terms are weighted by wi, wg, and wr, respectively.
2) Reward Functions:
The reward design combines imitation, goal tracking, and regularization, with dense rewards throughout episodes and sparse tracking rewards concentrated around strikes. The experiments evaluate planner accuracy, whole-body agility, and integrated real-world play.
- Reward composition: The reward function combines upper-body imitation, commanded-goal tracking, and regularization terms.The corresponding weights are wi, wg, and wr.
- Reward timing: Dense imitation and regularization rewards apply throughout each episode, while sparse goal-tracking terms activate near task-specific events.Racket position, velocity, and orientation tracking activate around hitting time; base-position tracking activates before the strike.
- Training setup: The asymmetric actor-critic gives the critic additional training-only information, including robot body poses, to improve return estimation.This design is used because several goal-tracking terms are sparse and episodic return depends on remaining strikes.
- Deployment: During deployment, motion capture and predicted racket position determine the swing type and desired base position, while the swing-type variable is excluded from policy observations.The system heuristically selects forehand or backhand from the robot base and predicted racket position.
- Evaluation questions: The experiments ask whether the planner predicts accurately, whether the whole-body policy is agile, and whether the integrated system can return balls against humans or humanoids.These questions correspond to planner evaluation, whole-body control evaluation, and real-world integrated play.
A. Model-based Planner
The planner and controller are evaluated through prediction accuracy, simulated agility, and real-world rallies. Results show accurate near-contact prediction, rapid base repositioning, human-like motion, high return rates, and sustained exchanges.
- Planner accuracy: At 0.5 s before striking, position prediction error falls below 7.5 cm, while at 0.3 s, time error falls below 20 ms.Both errors reach minimal levels at 0.1 s before the strike.
- Whole-body agility: 94.3% of 1000 simulated roll-outs reached within 1 cm of the desired base position after discarding 57 unsuccessful cases.For initial distances below 0.75 m, nearly all trials converged within 0.8 s.
- Whole-body agility: In real-world tests, the controller used a single step for rapid lateral repositioning and produced human-like strikes with waist rotation.The paper contrasts this with slower multi-step lateral movements under velocity commands.
- Real-world returns: 96.2% of 26 thrown balls were hit and 92.3% were returned successfully.The robot made 24 successful returns, missed one return after a hit, and completely missed one ball.
- Extended rallies: The humanoid achieved rallies of up to 106 consecutive shots against humans and sustained fully autonomous rallies against another humanoid.It also returned human smashes with a 0.42 s reaction time.
VII. DISCUSSION
The system combines modular planning and whole-body control, while experiments expose strong return performance alongside important deployment and gameplay limitations.
- System Design: The hierarchical design separates long-horizon prediction and planning from short-horizon whole-body control, allowing the modules to be evaluated and improved independently.The model-based and learning-based combination also targets sample efficiency, robustness to perception errors, and real-time response.
- Limitations: The fixed hitting plane constrains striking strategies and reduces effectiveness against very short or deep balls.Human opponents must avoid overly short balls that the robot cannot reach.
- Experimental Results: 96.2% hit rate and 92.3% return rate were achieved across 26 incoming balls distributed over the virtual hit plane.The robot returned 24 balls, with one hit that produced no return and one complete miss.
- Limitations: External motion capture provides ball and base-pose information, restricting deployment to controlled environments.Vision-based sensing is identified as a route toward more natural and diverse settings.
- Limitations: Negligible-spin assumptions and flat-push returns limit performance relative to professional play involving heavy spin and diverse strokes.Future extensions would perceive spin and generate appropriate counter-strokes.
C. Future Work
The paper identifies autonomous serving, multi-agent learning, and adaptation to skilled opponents as directions for extending humanoid table tennis beyond the demonstrated baseline.
- Multi-agent training and serving: The humanoid robots sustained a game using the same policy without joint training of different policies.Explicit multi-agent training is proposed to further improve competitiveness.
- Multi-agent training and serving: The robots cannot yet serve, so human players must initiate rallies in both humanoid-human and humanoid-humanoid settings.Autonomous serving is identified as an important direction for future work.
- Learning to play against skilled opponents: Learning against skilled opponents requires watching them to learn their characteristics and adapting stroke making accordingly.The authors frame the current system as a baseline for humanoid table-tennis shot making.
- Conclusion: The conclusion positions the demonstrated system as a step toward agile, interactive, and human-level humanoid behaviors.The paper ultimately aims toward championship-caliber play against skilled opponents.