Source-linked AI summary
SkillX: Unified Multi-Skill Policy Learning for Humanoid Soccer
Zhangchen Ye, Enxuan Ruan, Yifei Bao, Runhan Huang, Jiankun Yang, Jiakang Jin, Yixiao Huo, Pengyuan Wang, Yinan Han, Huaxing Huang, Wenhao Cui, Yiming Li, Xiaoyu Tian
TL;DR
Humanoid soccer demands coordinated whole-body control and long-horizon skill composition, while existing methods often rely on task-specific pipelines. SkillX uses a single command-conditioned policy with skill-specialized training signals and an object-aware temporal encoder. Experiments demonstrate multi-skill execution in simulation and on a real humanoid, including long-horizon composition and sim-to-real deployment.
Problem
Existing humanoid sports methods often rely on task-specific multi-stage pipelines, limiting joint learning and composition of object-interactive skills within one deployable policy.
Source: §1 · p. 2Method
SkillX uses one command-conditioned policy with skill-specific adversarial motion priors, skill-specific critics, and an object-aware temporal encoder.
Source: §1 · p. 2Results
Takeaways & Limitations
The results suggest that pairing a unified deployable policy with skill-specialized training signals is promising for multi-skill object-interactive control.
Source: §5 · p. 9SkillX relies on externally provided high-level skill commands and does not address strategic planning or autonomous skill sequencing; experiments are single-robot tasks.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Humanoid soccer is a challenging testbed for dynamic whole-body control, requiring robots to coordinate balance, locomotion, object interaction, and skill switching over long horizons. Existing humanoid sports methods often rely on task-specific multi-stage pipelines, making it difficult to jointly learn and compose multiple object-interactive skills within a single deployable policy. To address this, we present SkillX, a unified reinforcement learning framework that learns and composes multiple atomic soccer skills through a single command-conditioned policy. SkillX integrates three core designs: skill-specific adversarial motion priors, skill-specific critics, and an object-aware temporal encoder, enabling the robot to execute atomic skills and transition among them such as dribbling, trapping, and shooting. Experiments in simulation and on a real Noetix E1 humanoid demonstrate robust multi-skill execution, long-horizon skill composition, and successful sim-to-real deployment.
1 Introduction
Humanoid soccer requires coordinated whole-body control and long-horizon skill composition, while existing methods often depend on task-specific pipelines. SkillX combines skill-specialized training signals and an object-aware temporal encoder in one policy, with simulation and real-robot experiments demonstrating multi-skill execution.
1 Introduction
- Existing humanoid sports methods often use task-specific multi-stage pipelines, making it difficult to jointly learn and compose object-interactive skills in one deployable policy.The task also requires coordinating balance, locomotion, object interaction, and skill switching over long horizons.Source: §1 · p. 2
- SkillX uses skill-specific motion priors and critics plus an object-aware temporal encoder to address motion-style heterogeneity, skill-dependent objectives, and partial observability.The encoder estimates hidden robot and object states from observation histories; adaptive command sampling, a command-duration curriculum, and object domain randomization support command-driven skill composition.Source: §1 · p. 2
- 81.7% success on the hard trap-dribble-shoot task versus 27.4% for AMP; the real robot averaged 72.5% across four soccer tasks.The evaluation also reports robust sim-to-real deployment and command-driven skill transitions.Source: §1 · p. 2
- A single policy executes multiple atomic skills and composes them into long-horizon tasks, with simulation and real-world deployment validating the framework.The contributions include multi-skill training mechanisms and robust sim-to-real deployment.Source: §1 · p. 2
2 Related Work
Prior work addresses multi-task locomotion and athletic ball interaction through several learning strategies, each with noted tradeoffs. SkillX instead combines skill-specialized training signals with a unified policy for command-driven execution and transitions.
2.1 Multi-Task Reinforcement Learning in Locomotion
- Hierarchical, latent disentanglement, and mixture-of-experts approaches address heterogeneous locomotion skills, but face multi-stage training, latent entanglement, or expert imbalance and unstable specialization, respectively.SkillX decouples style regularization and value estimation across skills while retaining one unified deployment policy [28].Source: §2.1 · p. 3
2.2 Humanoid Athletic Ball Interaction
- Humanoid ball-interaction studies often target isolated skills or use multi-stage refinement and distillation, whereas SkillX studies composing ball-control skills under partial observations.Prior examples include table-tennis strikes [30], soccer dribbling [31], shooting [32] [33] [12], progressive refinement [16] [17], and policy distillation [13] [14].Source: §2.2 · p. 3
3 Method
SkillX trains a single command-conditioned actor for multi-skill soccer, using skill-specific training signals and an object-aware history encoder. Its command representation supports soft transitions between skills.
3 Method
- SkillX pairs one command-conditioned actor with skill-specific motion priors and critics, plus an object-aware temporal encoder for history features.The actor receives proprioception, object states, and skill commands and outputs joint-position targets.Source: §3 · p. 3, Figure 2 · p. 4
3.1 Problem Formulation
- The policy uses a history window of proprioceptive and object observations with skill commands, and its command vector interpolates during skill switches.The policy outputs target joint positions for low-level PD controllers.Source: §3.1 · p. 3
3.2 Skill-Specific Adversarial Motion Priors
- Each skill has a discriminator trained against same-command expert segments; style rewards blend during soft switches, while switch-spanning segments are excluded from discriminator updates.Source: §3.2 · pp. 4–5
3.3 Multi-Critic PPO with Skill-Specific Critics
- Skill-specific critics estimate values for each skill while a shared actor is retained, with command-conditioned values blended during soft transitions.The critics share a feature extractor and use separate skill-specific value heads.Source: §3.3 · p. 5
3.4 Object-Aware Temporal Encoder
- The temporal encoder applies positional encoding and causal self-attention to observation histories, then uses auxiliary temporal-consistency and hidden-state estimation objectives.The estimated root and object velocities and history latent are provided to the actor as compact features.Source: §3.4 · p. 5
3.5 Training Strategy
- Training strategies adapt command sampling to recent skill failures, shorten command durations over a curriculum, and randomize ball physical properties.The randomizations cover ball mass, friction, and restitution.Source: §3.5 · p. 6
4 Experiments
SkillX is evaluated on simulated skill-composition tasks, real-robot soccer tasks, onboard-vision deployment, and non-soccer whole-body skills. Results include higher simulation and hardware success than baselines, ablation findings on training components, and execution of diverse skills with one policy.
4.1 Implementation Details
- Policies train in Isaac Sim and Isaac Lab at 50 Hz control frequency against 200 Hz simulation, using MoCap soccer motions retargeted to the 25-DoF Noetix E1.Real-world experiments also use the 25-DoF Noetix E1.Source: §4.1 · p. 6
4.2 Simulation Results
- In simulation, evaluation covers atomic skills and Medium DRIBBLE→SHOOT and Hard TRAP→DRIBBLE×3→SHOOT tasks, with event-based command transitions and 1,000 trials per method.The listed baselines include AMP, Conditional AMP, MoE-based AMP, MoE-Encoder AMP, and Vanilla PPO; methods share task rewards, training budget, and evaluation protocol.Source: §4.2 · pp. 6–7
- 88.0% success on Medium exceeds the best baseline by 20.8 percentage points, while Hard reaches 81.7% versus 45.6% for MoE-Encoder AMP.Hard requires trapping, three-waypoint dribbling, and shooting.Source: §4.2 · p. 7
4.3 Ablation Study
- Ablations associate skill-specific priors with better success on both tasks, critics with a larger Hard-task benefit, and temporal architecture plus interaction-relevant supervision with the temporal encoder’s gain.Raw Stacked History reaches 49.0% on Medium and 20.0% on Hard; removing command-duration curriculum drops Hard success from 81.7% to 30.4%, versus 88.0% to 85.5% on Medium.Source: §4.3 · pp. 7–8, Table II · p. 7
4.4 Real-World Experiments
- SkillX succeeds on 80% of atomic dribbling and shooting trials and 70% and 60% on composite tasks, while AMP succeeds in only 1 of 10 atomic trials and none of the composite trials.The hardware comparison uses 10 trials per task under the MoCap setup.Source: Table III · p. 8, §4.4 · p. 8
4.5 Onboard Vision Deployment
- With a head-mounted camera, visual-inertial odometry, and YOLOv8 ball detection, the policy executes ball interactions despite noisier observations than MoCap.Training with lower-rate updates and observation delays accounts for perception noise; Fig. 5 shows the dribble-then-shoot deployment.Source: §4.5 · pp. 8–9, Figure 5 · p. 9
4.6 Skill Diversity Beyond Soccer Interaction
- With object observations and interaction rewards removed, one policy transitions among kicking, saluting, back kicking, single-leg standing, defending, and run-to-shoot motions.The evaluation retains proprioception, skill commands, and the same training framework.Source: §4.6 · p. 9, Figure 6 · p. 9
5 Conclusion
SkillX uses one command-conditioned policy for multi-skill humanoid soccer, with skill-specific training signals and an object-aware temporal encoder. Simulation and real-robot experiments demonstrate improved atomic execution, sim-to-real deployment, and long-horizon skill composition.
5 Conclusion
- SkillX combines a single command-conditioned policy with skill-specific motion priors and critics plus an object-aware temporal encoder for multi-skill humanoid soccer.The framework targets style interference, skill-dependent value estimation, and partial observability.Source: §5 · p. 9
6 Limitations
SkillX leaves strategic planning and autonomous skill sequencing to external commands and evaluates only single-robot tasks. Its deployment uses a shared policy interface across MoCap and onboard vision, while each sensing setup has distinct practical constraints.
6 Limitations
- SkillX does not address strategic planning or autonomous skill sequencing, and its experiments cover single-robot tasks rather than play with teammates or opponents.The policy handles low-level execution and transitions under externally provided high-level skill commands.Source: §6 · p. 9
A Deployment Details
- Both sensing backends convert measurements to a common robot-centric observation for the same exported policy, reducing deployment-specific changes.The policy runs onboard at 50 Hz; MoCap supports controlled evaluation, while onboard vision enables infrastructure-free execution.Source: A Deployment Details · p. 13
A.1 Motion-Capture Deployment
- MoCap provides global robot, ball, and goal measurements, but tracking jitter, occlusion, calibration error, and latency can affect relative observations.Filtering and invalidating stale measurements precede closed-loop rollouts in which the robot switches between distinct skills.Source: §A.1 · p. 13
A.2 Onboard Vision Deployment
- Onboard vision detects the ball with a head-mounted camera, and the policy approaches and interacts with it without external MoCap.Training simulates noisy, delayed, and intermittent observations; vision is less reliable than MoCap, but both backends use the same policy interface.Source: §A.2 · pp. 13–14
B.1 Evaluation Protocol
The evaluation tests isolated soccer primitives and command-driven skill compositions in simulation, using randomized initial conditions and sequential episodes. Five baselines share the task and training setup, subject to method-specific network structures and adversarial modules.
B.1 Evaluation Protocol
- The evaluation field is a 25.0 m by 15.0 m, 0.25×-scaled soccer field.The goal is 2.745 m wide and 1.22 m high; the evaluation ball has a 0.11 m radius and 0.45 kg mass.Source: §B.1 · p. 14
- Atomic evaluations isolate trapping, dribbling, or shooting under a fixed skill command, with success defined separately for each behavior.Policies are assessed by success rate and completion time over successful episodes, with domain randomization and observation corruption disabled.Source: §B.1 · p. 14
- Composite tasks use event-triggered commands: DRIBBLE→SHOOT for Medium and TRAP→DRIBBLE×3→SHOOT for Hard.The Hard task requires the ball to enter the goal within 40 seconds.Source: §B.1 · pp. 14–15
B.2 Baseline Settings
- The five baselines are Vanilla PPO, AMP, Conditional AMP, MoE-based AMP, and MoE-Encoder AMP, evaluated with shared robot, simulation, observation, reward, motion-data, and protocol settings.PPO hyperparameters are also shared except for method-specific network structures and adversarial modules.Source: §B.2 · p. 15
B.3 Details of Ablation Experiments
The ablations remove or alter SkillX’s skill-specific training signals and temporal encoder components to isolate their contributions. Baseline methods and SkillX share training hyperparameters unless otherwise specified.
B.3 Details of Ablation Experiments
- The framework ablations replace skill-specific discriminators or critics with shared ones, or train on the final command-duration range from the outset.These variants test skill-specific motion regularization, value estimation, and the command-duration curriculum.Source: §B.3 · p. 15
- Temporal-encoder ablations distinguish flattened history, an MLP in place of the Transformer, and removal of ball-velocity or all velocity-estimation objectives.The variants separate effects of explicit history representation, temporal architecture, and interaction-relevant supervision.Source: §B.3 · pp. 15–16
- Shared training hyperparameters are used for SkillX and all baselines unless a method is specified otherwise.Table IV summarizes the shared hyperparameters.Source: Table IV · p. 16
B.4 Additional Results
SkillX performs best across the three isolated atomic skills, and stricter trapping tests also favor it over the compared baselines. Ablations indicate that the framework’s components strengthen both primitives and multi-stage tasks.
B.4 Additional Results
- 95.0% shooting success and 100.0% success on both trapping and dribbling make SkillX the best overall across the three atomic skills.Vanilla PPO performs well on trapping but fails on shooting and dribbling; Conditional AMP and MoE-based AMP do not consistently learn all skills.Source: Table V · p. 16, §B.4 · p. 16
- Removing skill-specific AMP or critics degrades shooting, while raw stacked history lowers shooting and dribbling performance despite retaining five-frame inputs.The reported ablations also confirm that the components strengthen the atomic skills needed for successful composition.Source: §B.4 · pp. 16–17, Table VI · p. 17
- Under the stricter contact-based trapping criterion, SkillX reaches 63.0% success, versus 31.0% for AMP, 24.0% for Conditional AMP, and 50.0% for MoE-based AMP.Success requires stopping and keeping the ball near the robot within the stated post-contact interval.Source: §B.4 · p. 17
C Implementation Details
The actor uses a five-frame history of proprioception, ball and target positions, and a skill command, while separate critic and discriminator observations support training. Only actor observations are available during deployment.
C.1 Observation Details
- Actor observations are available in training and deployment, whereas critic and discriminator observations are training-only.The critic uses the uncorrupted actor sequence plus privileged information, discarded before deployment without extra actor inference overhead.Source: §C.1 · pp. 17–18
- A five-frame actor input combines proprioception, ball and target positions, and the skill command, yielding 440 scalar values in the three-skill setup.Each frame has 88 values; observation noise is added during training to all components except the skill command.Source: §C.1 · p. 17
- The critic adds ball velocity and binary foot-contact states to the actor sequence, producing 96 input dimensions in the three-skill setup.Source: §C.1 · p. 18
- The discriminator uses motion-centric observations of 136 dimensions per frame, including joint, body-position, and foot-contact information.For the 25-DoF Noetix E1, the robot has 26 bodies; the discriminator trains on sequential motion segments.Source: §C.1 · p. 18
- The discriminator’s sequential multi-step motion observations train adversarial motion priors but are not used by the deployed policy.Source: §C.1 · p. 18
C.2 Training Strategy
Training prioritizes underperforming skills, gradually shortens command durations to encourage transitions, randomizes ball properties, and blends commands locally near skill transitions.
C.2 Training Strategy
- After 5,000 warm-up iterations, skill sampling allocates more updates to higher-failure skills while retaining a uniform component.Failure rates use an EMA with a 0.99 smoothing coefficient; the mixture weight is 0.5.Source: §C.2 · p. 18
- Command durations shift from 5.0–10.0 seconds to 0.2–10.0 seconds, first supporting stable single-skill behavior and then more frequent transitions.The lower bound is linearly annealed from 5.0 to 0.2 seconds between iterations 9000 and 12000.Source: §C.2 · p. 18
- Ball friction, restitution, and mass are randomized within specified ranges to improve generalization to varying interaction dynamics.The randomized ranges are static friction 0.30–0.70, dynamic friction 0.10–0.55, restitution 0.55–0.95, and mass 0.40–0.48 kg.Source: §C.2 · pp. 18–19
- Command blending applies only during a five-frame, 0.1-second transition window, approximating local value changes without assuming globally linear returns.Source: §C.2 · p. 19