Source-linked AI summary
ATHLETE: Learning Reactive Humanoid Tennis via Trajectory-Guided Motion Matching
Zhihao Tian, Zifan Wang, Enzheng Zhu, Xun Yang, Jialong Qin, Mengbo Wang, Junwei Liang
TL;DR
Reactive humanoid tennis requires broad coverage of approach states, contact locations, and timing, but task-specific motion-capture demonstrations are costly. ATHLETE combines generic locomotion and video-derived strikes to train a reactive policy; on a real humanoid it achieves forehand and backhand returns and multi-shot rallies.
Problem
Reactive humanoid tennis must handle diverse approach states, contact locations, and arrival times, while collecting task-specific motion-capture demonstrations for those variations is costly.
Source: §I · p. 1Method
ATHLETE composes generic locomotion and video-derived strikes into task-conditioned references, then uses teacher–student learning to produce a reactive policy without reference-motion input at deployment.
Source: §I · p. 2Results
The robot hits 27/30 forehand balls (90.00%) and 26/30 backhand balls (86.67%), and ATHLETE supports multi-shot rallies with human players.
Source: §I · p. 2, C. Real World Experiments and Benchmark · p. 7Takeaways & Limitations
ATHLETE demonstrates that reusable locomotion data and sparse visual demonstrations can support dynamic whole-body tennis skills on a real humanoid robot.
Source: §I · p. 2The current system relies on external motion capture for ball-state estimation.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Humanoid tennis requires reactive whole-body coordination under rapidly changing ball dynamics, yet collecting task-specific motion-capture demonstrations that cover diverse approach states, contact locations, and timing conditions is costly. Inspired by how humans acquire new skills by combining reusable motor experience with task-specific demonstrations, we present ATHLETE, a framework for learning reactive humanoid tennis from generic locomotion data and ordinary tennis videos. ATHLETE combines LAFAN1 locomotion with video-derived striking motions and uses Trajectory-Guided Motion Matching (TGMM) to construct diverse task-conditioned whole-body references. These references are transformed into a reactive policy through physics-based teacher--student learning with online DAgger distillation, followed by PPO fine-tuning for the tennis task. The resulting policy responds directly to robot and ball observations without requiring reference-motion inputs at deployment. On a real humanoid robot, ATHLETE achieves forehand and backhand hit rates of $90.00\%$ and $86.67\%$, respectively, and enables multi-shot rallies with human players. These results demonstrate that ATHLETE can learn dynamic humanoid tennis skills from reusable locomotion data and video demonstrations. Result videos and code can be found at https://anonymous.4open.science/w/ATHLETE2027/.
I. INTRODUCTION
ATHLETE addresses the costly coverage demands of reactive humanoid tennis by combining reusable locomotion with video-derived strikes. It uses TGMM and teacher–student learning to produce a reactive policy and validates real-world tennis skills.
I. INTRODUCTION
- ATHLETE combines generic locomotion with video-derived strikes under approach, contact, and transition constraints to generate diverse whole-body tennis references without dedicated tennis motion-capture demonstrations.The approach treats locomotion and striking as complementary sources rather than requiring complete tennis demonstrations.Source: §I · pp. 1–2
- A deadline-adaptive teacher–student framework transfers reference-motion and contact-timing supervision into a deployable policy that uses no reference motions, online matching, or explicit planner at deployment.The controller responds to robot state, ball observations, and task target.Source: §I · p. 2
- ATHLETE demonstrates reactive forehand and backhand returns and multi-shot human–robot rallies using reusable locomotion data and sparse visual demonstrations.Source: §I · p. 2
A. Data Generation for Humanoid Robots
- Prior motion-matching methods retrieve and blend database motions, while Learned Motion Matching approximates retrieval with neural networks; ATHLETE instead composes optimized approach trajectories with strikes under contact and transition constraints.Perceptive Humanoid Parkour [7] also combines sparse atomic skills with locomotion to generate references with diverse approach states.Source: A. Data Generation for Humanoid Robots · p. 2
B. Humanoid Whole-Body Control
ATHLETE draws on prior humanoid-control and tennis methods while composing LAFAN1 locomotion with video-derived strikes. Contact matching selects a suitable strike clip for the target, and TGMM connects it to locomotion.
B. Humanoid Whole-Body Control
- LATENT [1], AdaPT [2], and TaskNPoint [24] address tennis through latent-action composition, hierarchical planning with speed-adaptive tracking, and interaction-window-focused learning, respectively.ATHLETE adopts motion-tracking and goal-conditioned striking rewards, adds deadline-conditioned phase adaptation, and distills the skills into one reactive policy.Source: B. Humanoid Whole-Body Control · p. 2
III. MODEL-BASED REFERENCE MOTION GENERATION
- The reference-motion pipeline combines LAFAN1 locomotion sequences with video-derived striking motions and uses simulated Mimic-policy rollouts to dynamically correct composed references.Source: §III · p. 2
A. Source Skills and Contact Matching
- The motion library includes 16 LAFAN1 locomotion sequences and 164 video-derived strike clips: 67 backhand and 97 forehand clips.The clips come from a 10-minute tennis practice video, are reconstructed with GVHMR, and retargeted with GMR.Source: A. Source Skills and Contact Matching · p. 2
- For a stationary target, candidate strikes are first restricted to forehand or backhand by lateral position, then ranked by contact cost, with greater weight assigned to height difference.The lowest-cost candidate is selected.Source: A. Source Skills and Contact Matching · p. 2
- TGMM optimizes planar trajectories for smoothness, converts them into whole-body motion, and connects the locomotion and striking phases.The figure traces the racket sweet spot through the strike.Source: Figure 3 · p. 3
B. Trajectory-Guided Motion Matching
TGMM plans a planar approach to strike entry and uses its optimized trajectory to retrieve matching locomotion frames. The resulting whole-body locomotion reference is connected to the selected striking motion.
B. Trajectory-Guided Motion Matching
- The approach planner uses distinct nearby and distant modes: nearby targets retain initial heading before alignment, while distant targets require turning, forward travel, and strike-entry heading recovery.Both modes optimize their phase durations.Source: B. Trajectory-Guided Motion Matching · p. 3
- The optimizer favors short, direct approaches with smooth controls while enforcing terminal position, heading, velocity, mode, and total-duration constraints.Path length is normalized by the straight-line distance to strike entry, lower-bounded at 1 m; the trajectory is solved with trapezoidal collocation and IPOPT.Source: B. Trajectory-Guided Motion Matching · p. 3
- Optimized planar positions, headings, and root velocities form a motion-matching query that retrieves eligible LAFAN1 frames using weighted feature distance.The feature set also includes foot and racket-hand positions and velocities, plus future planar positions and headings at 0.4, 0.8, and 1.2 seconds.Source: B. Trajectory-Guided Motion Matching · p. 3
- The best-matching frames are connected to convert the planned planar path into whole-body locomotion and link it to the contact-matched strike.Source: B. Trajectory-Guided Motion Matching · p. 3
C. Composition and Dynamic Correction
ATHLETE blends the selected strike motion into locomotion references, then uses physics simulation to correct residual tracking inconsistencies. The framework subsequently trains and deploys a reactive policy for real-world rallies.
C. Composition and Dynamic Correction
- At the locomotion-to-strike transition, ATHLETE selects a strike entry frame matching the current pose and interpolates toward it, using root alignment and offset-decay blending to reduce discontinuities.Joint offsets are blended over a duration, with root translations blended similarly and orientations interpolated using quaternions.Source: C. Composition and Dynamic Correction · p. 3
- Because blending can leave residual discontinuities and kinematic inconsistencies, a Mimic policy tracks the composed references in physics simulation and its executed trajectories become dynamics-corrected training references.Source: C. Composition and Dynamic Correction · pp. 3–4
- The framework proceeds from TGMM reference construction and dynamic correction through time-adaptive teacher training, online DAgger student distillation, and PPO fine-tuning before robot deployment.The deployed student enables real-world rallies with human players, using motion capture for ball-state estimation.Source: Figure 4 · p. 4
IV. THREE-STAGE TENNIS POLICY LEARNING
The three-stage policy-learning pipeline trains a goal-conditioned, time-adaptive teacher, distills it into a student without reference-motion inputs, and then fine-tunes the student for tennis. Stage I retimes the same reference motion to meet different strike deadlines.
IV. THREE-STAGE TENNIS POLICY LEARNING
- The three-stage pipeline trains a time-adaptive teacher, distills a reference-free student, and fine-tunes that student with tennis task rewards.The training sequence comprises a goal-conditioned teacher, student distillation, and Stage III task refinement.Source: §IV · p. 4
A. Stage I: Time-Adaptive Teacher Policy
- Fixed-rate playback reaches the striking state only at a predetermined time, limiting adaptation to balls with different speeds and flight times.Stage I instead trains a time-adaptive teacher to adapt the same striking reference to different contact times.Source: A. Stage I: Time-Adaptive Teacher Policy · p. 4
- Strike deadlines are sampled every 0.1 s from half the nominal strike time to the nominal time, and PPO trains the teacher to strike a stationary ball at that deadline.The reward combines motion tracking and goal-conditioned striking terms with phase timing; phase acceleration is constrained to keep reference joint velocities within prescribed limits.Source: A. Stage I: Time-Adaptive Teacher Policy · p. 4
- Stage I retimes the same reference motion to meet different strike deadlines, allowing Stage II to match it to faster incoming balls and expand covered ball conditions.Feasible deadlines earlier than nominal playback are used for matching.Source: A. Stage I: Time-Adaptive Teacher Policy · p. 4
B. Stage II: Ball Trajectory Matching and Student Policy Distillation
Stage II matches simulated ball trajectories to reference strike points, then distills the teacher into a student using online DAgger and latent supervision. The student’s history-based representation provides motion intent and timing without requiring references or explicit strike deadlines at execution.
B. Stage II: Ball Trajectory Matching and Student Policy Distillation
- Ball trajectories simulated with gravity, drag, and ground rebound are matched to reference strike points between the first and second bounces.A valid match must satisfy the contact-point tolerance and deadline interval; its matched time becomes the teacher’s deadline, using a library of 200 generated references.Source: B. Stage II: Ball Trajectory Matching… · pp. 4–5
- Online DAgger queries teacher action labels at every visited state while gradually increasing student control, providing supervision on states reached after student deviations.Stage II disables PPO updates and optimizes action and representation distillation objectives.Source: B. Stage II: Ball Trajectory Matching… · p. 5
- The student temporal encoder fuses robot and ball observations at five historical instants, yielding motion intent and timing without reference motions or explicit strike deadlines at execution.The privileged encoder cross-attends from robot-history queries to 12 future reference joint states; reference motions remain in the training supervision branch.Source: B. Stage II: Ball Trajectory Matching… · p. 5
- Reference reconstruction gives the privileged latent motion semantics, while latent alignment transfers that supervision to a student representation conditioned on observation histories.The Stage II objective combines action distillation, reference reconstruction, and a 0.05-weighted latent term.Source: B. Stage II: Ball Trajectory Matching… · p. 5
C. Stage III: Task Refinement via PPO
Stage III fine-tunes the distilled student with PPO and tennis task rewards while preserving learned representations. The deployed policy uses current observations and encoded histories, without a teacher, reference motions, or a phase controller.
C. Stage III: Task Refinement via PPO
- PPO fine-tuning updates the policy and value heads while freezing both latent encoders and the reference decoder.A retained 0.05 action-distillation coefficient regularizes refinement toward the teacher’s learned skills.Source: C. Stage III: Task Refinement via PPO · p. 5
- Task rewards score pre-contact swinging and post-contact ball speed, net clearance, and landing accuracy.The outgoing-speed, net-clearance, and landing terms use the post-strike ball state, with landing accuracy evaluated for valid net-crossing trajectories.Source: C. Stage III: Task Refinement via PPO · p. 5
- Deployment retains only the student encoder and action network, producing 29-dimensional joint actions at 50 Hz from current observations and encoded histories.The teacher, privileged reference encoder, decoder, reference motions, and phase controller are removed.Source: C. Stage III: Task Refinement via PPO · p. 6
- Ball mass, rebound and drag properties, racket mass, racket center-of-mass offsets, and racket restitution are randomized during fine-tuning.The randomization exposes the student to variations in flight, ground rebound, and racket-contact dynamics.Source: C. Stage III: Task Refinement via PPO · p. 6
A. Reference Motion Coverage
TGMM-generated references substantially broaden matching coverage, and deadline-adaptive teacher control supports physical contact across varied arrival times. The full method improves simulated hitting and returns over the distilled policy without PPO fine-tuning.
A. Reference Motion Coverage
- A candidate group counted as successful when at least one of its first K qualified trajectories matched a reference in strike location and timing.Qualified trajectories cleared the net, passed within 3 m horizontally of the initial robot position before the second bounce, and reached at least 0.5 m ball-center height.Source: A. Reference Motion Coverage · p. 6
- At 164 motions and K = 1, TGMM-generated references achieved 42.64% matching success versus 7.57% for the original library; 200 generated references reached 99.21% mean group matching success at K = 8.Source: A. Reference Motion Coverage · p. 6
B. Simulation Experiments
- 493 of 500 teacher trials made physical racket contact, and 362 contacts fell within ±50 ms of the target deadline.Mean absolute timing error across the 493 contacts was 71.23 ms; deadline adaptation lets one reference support more arrival times and motion–trajectory pairings for distillation.Source: Figure 6 · p. 6, B. Simulation Experiments · p. 6
- 94.97%/95.04% forehand-/backhand-side hit rates and 91.19%/92.20% return rates were achieved by the full method, versus 1.26%/5.67% hits and no net clears without PPO fine-tuning.The full method also reduced mean sweet-spot distances to 5.61/6.54 cm and mean landing errors to 1.150/1.314 m.Source: Table II · p. 7, B. Simulation Experiments · p. 7
C. Real World Experiments and Benchmark
In manual-throw tests, ATHLETE hit most incoming forehand- and backhand-side balls and cleared the net in some trials. The robot also sustained multi-shot rallies with human partners.
C. Real World Experiments and Benchmark
- The robot hit 27/30 forehand balls (90.00%) and 26/30 backhand balls (86.67%), clearing the net in 14/30 forehand trials (46.67%) and 16/30 backhand trials (53.33%).Source: C. Real World Experiments and Benchmark · p. 7
- Among video-derived-motion approaches, ATHLETE reports higher forehand and backhand hit and net-clearance rates than AdaPT under their respective evaluation protocols.The comparison retains the methods’ original experiment environments and protocols.Source: C. Real World Experiments and Benchmark · p. 7
- The longest observed human–robot rallies lasted 5 rounds with a novice and 8 rounds with a professional player.The trials involved human partners with different levels of tennis experience.Source: C. Real World Experiments and Benchmark · p. 7
D. Qualitative Comparison
ATHLETE reuses public locomotion and video-derived strikes, then distills composed references into a reactive policy for real-robot tennis. Its current system relies on external motion capture for ball-state estimation.
D. Qualitative Comparison
- ATHLETE avoids dedicated tennis motion-capture collection by reusing public locomotion and sparse video-derived strikes, and trains its policies on a single RTX 4090.LATENT reports using five hours of newly captured amateur-player motion primitives and eight training GPUs.Source: D. Qualitative Comparison · pp. 7–8
- The deployed ATHLETE policy supports forehand and backhand returns and continuous rallies without online motion planning, future-ball prediction, or reference tracking.All three real-robot evaluations discussed use external motion capture for state estimation.Source: D. Qualitative Comparison · p. 8
VI. CONCLUSION
- External motion capture remains necessary for ball-state estimation; future work will investigate onboard ball perception and preservation of motion-style diversity.The conclusion also reports real-robot forehand and backhand returns and continuous rallies with human players.Source: §VI · p. 8