Source-linked AI summary
Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data
Zhikai Zhang, Haofei Lu, Yunrui Lian, Ziqing Chen, Yun Liu, Chenghuai Lin, Han Xue, Zicheng Zeng, Zekun Qi, Shaolin Zheng, Qing Luan, Jingbo Wang, Junliang Xing, He Wang, Li Yi
TL;DR
Humanoid robots lack complete, precise tennis motion references for reproducing athletic rallies. LATENT learns from imperfect primitive motion fragments, corrects and composes them through a latent action space, and deploys the resulting policy on a Unitree G1 that sustains multi-shot rallies with human players. The paper also identifies current boundaries: motion-capture dependence and a task formulation that differs from real two-player matches.
Problem
Complete and precise human-tennis motion data are difficult to collect, while direct human-humanoid tele-operation is infeasible for athletic tennis.
Method
LATENT learns tennis skills from imprecise and incomplete primitive motion fragments, using latent-action designs to correct and compose them for tennis returns.
Results
LATENT is deployed on a real-world humanoid robot and stably sustains multi-shot rallies with human players.
Takeaways & Limitations
Imperfect motion fragments can provide human primitive-skill priors for learning tennis returns while preserving natural motion styles.
Takeaways & Limitations
The method relies on motion capture, and its randomly initialized ball-return task still differs from a real two-player tennis match.
Abstract
from arXiv · showhide
Human athletes demonstrate versatile and highly-dynamic tennis skills to successfully conduct competitive rallies with a high-speed tennis ball. However, reproducing such behaviors on humanoid robots is difficult, partially due to the lack of perfect humanoid action data or human kinematic motion data in tennis scenarios as reference. In this work, we propose LATENT, a system that Learns Athletic humanoid TEnnis skills from imperfect human motioN daTa. The imperfect human motion data consist only of motion fragments that capture the primitive skills used when playing tennis rather than precise and complete human-tennis motion sequences from real-world tennis matches, thereby significantly reducing the difficulty of data collection. Our key insight is that, despite being imperfect, such quasi-realistic data still provide priors about human primitive skills in tennis scenarios. With further correction and composition, we learn a humanoid policy that can consistently strike incoming balls under a wide range of conditions and return them to target locations, while preserving natural motion styles. We also propose a series of designs for robust sim-to-real transfer and deploy our policy on the Unitree G1 humanoid robot. Our method achieves surprising results in the real world and can stably sustain multi-shot rallies with human players. Project page: https://zzk273.github.io/LATENT/
1 Introduction
LATENT addresses the difficulty of learning athletic humanoid tennis from incomplete or imprecise motion data by correcting and composing primitive human skills. The system combines latent-action designs with robust transfer methods and demonstrates real-world multi-shot rallies.
- Motivation: Tennis demands rapid, precise whole-body behavior, making direct human-humanoid tele-operation data collection infeasible for imitation learning.Players may sprint above 6 m/s, react to balls traveling at 15–30 m/s, and strike within a few milliseconds.
- Motivation: Complete human-tennis motion capture is difficult because tennis combines large-scale movement, sustained rallies, and precise wrist motions.Broadcast-video alternatives require complex pipelines for detection, camera estimation, pose estimation, and motion correction.
- Approach: LATENT learns from imperfect motion fragments representing primitive skills such as forehand strokes, backhand strokes, lateral shuffles, and crossover steps.These fragments are imprecise in wrist motion and incomplete regarding how primitives should be used for tennis returns, but provide natural motor-skill priors.
- Approach: The system constructs a latent action space and trains a high-level policy to correct and compose primitive actions for tennis returns.The latent space accommodates policy-predicted action corrections, while a latent action barrier constrains exploration toward the state-based action prior.
- Evaluation: Robust sim-to-real transfer uses dynamics randomization and observation noise for both the robot and tennis ball.The method is deployed on Unitree G1 hardware and sustains multi-shot rallies with human players after simulation and real-world validation.
2 Related Work
Prior humanoid sports research has explored simulated and real-world athletic skills, but existing systems often lack the dynamic motion, agile coordination, or rapid reactions required for athletic tennis. LATENT focuses on correcting and composing imperfect primitive motion data to preserve natural motion styles while learning tennis skills.
- Latent-action control: Latent-action methods encode reusable primitive skills from prerecorded human motion and train a high-level policy to sample within the latent space.This approach addresses low reinforcement-learning sampling efficiency and unnatural motions associated with learning directly on high-degree-of-freedom humanoids.
- LATENT: LATENT differs by learning athletic tennis from imprecise and incomplete primitive skills, then correcting and composing those priors while preserving natural motion styles.The approach targets tennis skills without requiring complete, high-quality human-tennis motion sequences.
- Real-world sports skills: Real-world humanoid sports systems have demonstrated table tennis, badminton, football, and boxing, but often lack highly dynamic motions, agile coordination, or rapid reactions.LATENT positions its contribution around athletic humanoid sports skills with these demands.
3 Learning Tennis Skills with LATENT
LATENT learns athletic humanoid tennis behavior from imperfect primitive-skill motion fragments through a correctable latent action space and high-level policy. Its design addresses wrist inaccuracies and constrains exploration so the robot can compose natural skills for tennis returns.
- Imperfect Human Motion Data: LATENT collects primitive tennis skills rather than complete match sequences, reducing motion-capture requirements for training data.The collected skills include forehand, backhand, lateral shuffle, and crossover steps; the capture area is 3m×5m, more than 17× smaller than a tennis court.
- Motion Tracker and Latent Space: A motion tracker imitates the collected fragments, after which online distillation builds a compact continuous latent representation of those primitive skills.The student uses an encoder-decoder with a conditional variational information bottleneck and reconstructs the tracker’s actions.
- Sim-to-Real Transfer: LATENT trains its policy in simulation with PPO and transfers it toward real-world deployment using dynamics randomization and observation noise.The system runs on Unitree G1 hardware with a tennis racket and uses high-level and low-level controllers at 50 Hz.
- Latent Action Barrier: The learnable conditional prior models state-dependent action distributions, while the Latent Action Barrier constrains residual exploration around the prior mean.This addresses task policies that might exploit the latent space by switching between primitives in low-quality ways; the barrier is based on the prior’s state-dependent distribution.
- High-Level Policy Learning: A high-level planner uses robot state, root pose, and ball state to correct and compose latent primitive actions for the tennis return task.The planner is trained to combine stored skills while retaining compatibility with potential action corrections.
- Action Correction: The latent action space is designed to tolerate wrist corrections because retargeted striking motions are inaccurate across human and humanoid embodiments.Training removes right-wrist control signals and applies random wrist perturbations so the remaining motion remains stable under unknown wrist disturbances.
4 Sim-to-Real Transfer
LATENT addresses the sim-to-real gap caused by highly dynamic humanoid motion, fast tennis balls, and precise ball striking. It combines dynamics randomization, observation-noise modeling, and motion-capture-based deployment procedures.
- Transfer challenges: Sim-to-real transfer is difficult because humanoid movement and tennis-ball interactions are highly dynamic and require precise state estimation.The paper identifies dynamics, ball speed, and interaction precision as key transfer challenges.
- Dynamics randomization: LATENT randomizes robot friction, armature, mass, and center of mass to reduce the simulation-to-reality dynamics gap.
- Dynamics randomization: Ball dynamics randomization covers mass, restitution, tangential damping, and air drag because contacts and aerodynamics strongly affect trajectories.The randomized air-drag model uses f_air = −k * m * v * ||v||.
- Observation noise: The simulation models observation errors with uniform noise, frame dropouts, and latency in policy observations.
- Observation noise: A four-frame sliding window averages ball velocity in simulation and the real world to reduce errors from noisy position differencing.
- Deployment: Real-world deployment uses an optical motion-capture system, reflective markers, and a wrist-mounted racket on the humanoid.
5 Experiments
Experiments evaluate LATENT in simulation through baseline comparisons and ablations, then assess real-world rally performance and sim-to-real components. The reported results show advantages in tennis return performance, action quality, and real-world transfer.
- Experiment setting: Experiments use the 29-DoF Unitree G1 and evaluate simulated tennis returns across stroke types and court regions.Results are averaged over 10,000 trials.
- Metrics: Success Rate measures returns landing within 2.5 meters of the target, while Distance Error measures average landing distance from the target.
- Metrics: Smoothness measures averaged joint acceleration, and Torque measures averaged joint torque as indicators of action naturalness, stability, feasibility, and hardware safety.
- Baseline comparison: LATENT outperforms PPO, MotionVAE, AMP, ASE, and PULSE in tennis return task performance and action quality.The paper attributes baseline difficulties to learning without motion priors, tracking errors, and difficulty balancing task completion with motion-prior adherence.
- Ablation study: Removing wrist correction significantly lowers success rate, while removing the latent action barrier produces jittery motions and degrades overall performance.
- Real-world evaluation: Real-world evaluation uses 20 consecutive human-robot rally matches with randomized ball initial positions and velocities.Successful returns are analyzed by landing region and forehand or backhand stroke.
- Real-world evaluation: Adequate randomization of ball dynamics and observations enables effective sim-to-real transfer without precise system identification.
6 Conclusion
LATENT learns humanoid tennis skills from imperfect human motion data and is deployed on a real-world humanoid robot. The method sustains multi-shot rallies with human players, while the current formulation remains narrower than a real two-player match.
- LATENT learns athletic humanoid tennis skills from imperfect human motion data.
- The method was successfully deployed on a real-world humanoid robot and sustained multi-shot rallies with human players.
- The framework may generalize to tasks such as soccer and parkour where complete, high-quality human motion data are unavailable.
- The current system relies on motion capture and returns randomly initialized incoming balls to target locations rather than modeling a full two-player match.The authors suggest active vision and multi-agent training as directions for improvement.
A Additional Experiments
This section introduces further analysis of the learned policy. The additional experiments examine the policy beyond its primary evaluation.
- The section provides further analysis of the learned policy.
- The analysis focuses on behavior of the learned policy beyond its initial presentation.
- The section frames the following material as additional policy analysis.
A.1 Robot Movement Coverage
The movement-coverage analysis aggregates pelvis positions across consecutive returns to examine spatial coverage. The robot explores a wide court region while preserving stable movement structure and dynamically repositioning during repeated rallies.
- Pelvis-position heatmaps aggregate trajectories over 8, 16, 80, and 400 consecutive ball returns.Positions below 0.01% density are filtered for visualization.
- The robot consistently explores a wide court region while maintaining a stable movement structure across episodes.
- The learned policy dynamically repositions the robot and maintains broad court coverage during repeated rallies.
A.2 Robot-Robot Self-Play
Simulation self-play places two instances of the policy on opposite sides of the court to exchange returns. The robots maintain stable rallies, reaching up to 25 consecutive rallies, while self-play reinforcement learning is proposed for future improvement.
- Two policy instances are deployed on opposite sides of the court to exchange tennis returns in simulation.
- Up to 25 consecutive rallies were achieved in robot-robot self-play, demonstrating stable rally behavior.
- Future work may use reinforcement learning with self-play to improve robot-human and robot-robot rally performance.Suggested approaches include Neural Fictitious Self-Play and other multi-agent reinforcement-learning methods.
B.1 Training in Simulation
Simulation examples show incoming-ball trajectories, robot posture at contact, and resulting return trajectories, illustrating representative hitting events.
- These examples illustrate representative hitting events in simulation.
- Red curves encode incoming ball trajectories, while green curves show the trajectories of returned balls.
- The robot posture is shown at the moment of ball contact in each simulated hitting event.
B.2 Close-up Views of the Robot During Rallies in the Real World
Real-world rally views show the robot adapting posture, swing timing, footwork, and whole-body coordination across diverse returns.
- The system uses hardware modifications on the Unitree G1 to improve robustness during motion capture and interaction.
- The robot exhibits diverse motion patterns when interacting with incoming balls in real-world rallies.
- It adapts body posture and swing timing to reach incoming balls and execute stable returns.
- Close-up examples include forehand and backhand strokes, dynamic footwork, and coordinated whole-body motion.