Source-linked AI summary
Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
Tuomas Haarnoja, Ben Moran, Guy Lever, Sandy H. Huang, Dhruva Tirumala, Jan Humplik, Markus Wulfmeier, Saran Tunyasuvunakool, Noah Y. Siegel, Roland Hafner, Michael Bloesch, Kristian Hartikainen, Arunkumar Byravan, Leonard Hasenclever, Yuval Tassa, Fereshteh Sadeghi, Nathan Batchelor, Federico Casarini, Stefano Saliceti, Charles Game, Neil Sreendra, Kushal Patel, Marlon Gwira, Andrea Huber, Nicole Hurley, Francesco Nori, Raia Hadsell, Nicolas Heess
TL;DR
The paper asks whether Deep RL can produce sophisticated, safe, and composable movement for a low-cost humanoid in dynamic soccer. It trains a two-stage policy in simulation and transfers it to real robots, achieving agile skills, context-dependent tactics, and strong improvements over a scripted controller. The approach still relies on domain-specific reward knowledge, hand-designed get-up poses, and manually selected distillation targets.
Problem
The paper addresses limited evidence that low-cost humanoids can perform and compose sophisticated full-body movements for long-horizon, dynamic multi-agent tasks.
Method
Deep RL trains separate soccer and get-up skills, then distills them into a single 1v1 agent refined through self-play and regularization.
Results
181% faster walking, 302% faster turning, 63% less get-up time, and 34% faster mean kicking than a scripted controller with a run-up.
Takeaways & Limitations
The learned agent combines agile movement, ball interaction, recovery, defending, and fluid transitions while adapting behavior to game context.
Takeaways & Limitations
The pipeline relies on domain-specific reward knowledge, hand-designed get-up poses, and manually chosen skill targets during distillation.
Abstract
from arXiv · showhide
We investigate whether Deep Reinforcement Learning (Deep RL) is able to synthesize sophisticated and safe movement skills for a low-cost, miniature humanoid robot that can be composed into complex behavioral strategies in dynamic environments. We used Deep RL to train a humanoid robot with 20 actuated joints to play a simplified one-versus-one (1v1) soccer game. The resulting agent exhibits robust and dynamic movement skills such as rapid fall recovery, walking, turning, kicking and more; and it transitions between them in a smooth, stable, and efficient manner. The agent's locomotion and tactical behavior adapts to specific game contexts in a way that would be impractical to manually design. The agent also developed a basic strategic understanding of the game, and learned, for instance, to anticipate ball movements and to block opponent shots. Our agent was trained in simulation and transferred to real robots zero-shot. We found that a combination of sufficiently high-frequency control, targeted dynamics randomization, and perturbations during training in simulation enabled good-quality transfer. Although the robots are inherently fragile, basic regularization of the behavior during training led the robots to learn safe and effective movements while still performing in a dynamic and agile way -- well beyond what is intuitively expected from the robot. Indeed, in experiments, they walked 181% faster, turned 302% faster, took 63% less time to get up, and kicked a ball 34% faster than a scripted baseline, while efficiently combining the skills to achieve the longer term objectives.
Introduction
The paper applies Deep RL to full-body control of a low-cost humanoid in simplified 1v1 soccer, targeting agile movement, context-dependent behavior, and sim-to-real deployment. A two-stage training approach combines pre-trained soccer and get-up skills into an agent that can discover and smoothly sequence behaviors.
- Deep RL builds on prior learning-based work for simulated and physical motor control, including quadrupedal locomotion, recovery, soccer, and manipulation.
- Deep RL trains low-cost humanoid robots for simplified 1v1 soccer using proprioceptive and motion-capture observations, with policies deployed directly on real robots.
- The agent learns agile movements and smoothly composes walking, side stepping, kicking, fall recovery, and ball interaction.
- Learning discovers game-context-specific skills and tactics, including kicking moving balls, defensive running, adaptive footwork, prediction, and long-horizon coordination.
- The training pipeline first learns separate soccer and get-up policies, then distills them into a full 1v1 agent trained through self-play against partially trained policy copies.
- Pre-training soccer and get-up skills is the minimal set needed for success; end-to-end training otherwise produced rolling or standing-only degenerate solutions.
Results
On physical OP3 robots, the learned agent combined agile locomotion, recovery, kicking, and opponent-aware tactics while adapting behavior to game context. It transferred from simulation with modest behavior-level performance gaps and exceeded scripted baselines on key locomotion measures.
- Emergent behaviors: The agent smoothly combined getting up, fall recovery, running, turning, ball control, shooting, moving-ball kicks, shot blocking, and defensive positioning during 1v1 play.During matches, agents transitioned fluidly between these behaviors.
- Baseline comparisons: 181% faster walking, 302% faster turning, and 63% less time to get up were achieved versus scripted baseline controllers.The learned policy also reached 2.8 m/s mean kicking speed after a run-up, 34% faster than the scripted controller.
- Baseline comparisons: The learned gait leaned forward, pushed off foot-plate edges, and landed on the heels rather than keeping the feet nearly parallel to the ground.This dynamic gait differed from the scripted controller’s weight-centered movement.
- Behavior embeddings: Joint-angle embeddings showed richer, non-periodic learned gaits than the scripted walking controller’s cyclic trajectory.The full 1v1 agent occupied a dense region containing varied cyclic gaits, while kicking and getting up showed less variation.
- Sim-to-real reliability: In the get-up-and-shoot set piece, the robot scored 29/50 trials (58%) in reality versus 35/50 (70%) in simulation, while getting up and kicking every real-world trial.The transfer reduced scoring consistency but retained majority success.
- Sim-to-real reliability: Real-robot execution differed from simulation by 13% faster walking, 11% slower turning, 28% more time to get up, and 5% slower kicking.The reported gaps were not extreme, and the optimized corner-foot pivot appeared in both settings.
- Opponent awareness: When the opponent possessed the ball, the agent moved into the shooting path; when the opponent was away, it approached the ball directly.The interception behavior emerged from task-reward optimization rather than manual specification.
- Opponent awareness: Opponent-aware positioning extended to routing and footwork: the agent walked around an obstructing opponent, scored in 9/10 trials, and used 30 short steps versus 20 longer strides when rushing.These behaviors adapted to whether the opponent obstructed or possessed the ball.
Discussion
This work extends learning-based control toward agile, whole-body behavior for low-cost humanoids in dynamic soccer settings. It also identifies practical limitations involving domain knowledge, sim-to-real transfer, sensing, hardware maintenance, and training stability.
- Comparison to Robot Learning Literature: Humanoid soccer remains less explored than quadrupedal locomotion, especially for low-cost platforms with less precise actuators and sensors.Prior soccer work focused on narrower skills, while quadrupeds provide a more stable learning platform.
- Limitations: The approach still relies on domain-specific reward design, hand-designed get-up poses, manually selected skill targets, and simulation-only transfer.The authors suggest automatic skill selection, real-robot fine-tuning, or mixing real data into training as possible improvements.
- Limitations: Motion-capture ball tracking is sensitive to marker angle and occlusion, while hardware degradation requires maintenance and limits reliable operation.The control stack also missed its nominal 25 ms action deadline in practice, and battery charge limited operation time to 5 to 10 minutes per charge.
- Limitations: Self-play sometimes produces unstable learning, and the weighted auxiliary-reward design required extensive hyperparameter tuning.The authors identify population-based, multi-objective, or constrained RL as potential alternatives.
Materials and Methods
The system trains a miniature humanoid in simulation and transfers it to a corresponding physical soccer environment. Control uses 20-dimensional joint-position actions, filtered before PID-based actuation, with proprioceptive and motion-capture game-state observations.
- Environment: The simulated training environment uses MuJoCo and a 5 m by 4 m soccer pitch with two 0.8 m-wide goals.Ramps return the ball to the pitch bounds in both simulation and reality.
- Control: The agent acts at 40 Hz with 20-dimensional joint-position set points passed through an exponential action filter.Filtered actions are sent to PID controllers, producing torques in simulation and voltages on the real robot.
- Observations: Observations combine joint positions, linear acceleration, angular velocity, gravity direction, and filter state with game-state information.Motion capture supplies agent, ball, opponent, and goal locations and velocities in the real environment.
- Hardware: The platform is a 51 cm, 3.5 kg Robotis OP3 humanoid actuated by 20 Dynamixel XM430-350-R servomotors.The robot uses proportional position control and magnetic rotary encoders for joint-position observations.
- Motion Capture: A 14-camera Optitrack PrimeX 22 motion-capture system tracks the robots and ball using reflective markers and stickers.Tracked positions are streamed wirelessly to the robots through VRPN and ROS.
- Hardware Modifications: Safety bumpers and redesigned forearms were added to reduce fall damage and prevent the limbs from snagging the robot’s cabling.These modifications supported evaluation of many prototype policies.
Policy Optimization
The soccer task is modeled as a POMDP and optimized with Distributional MPO. A deep stochastic policy generates continuous joint-position actions, while a critic estimates expected discounted return for policy improvement.
- Problem Formulation: The soccer environment is modeled as a Partially Observable Markov Decision Process with state, action, transition, reward, initial-state, and discount components.The agent observes features extracted from the underlying state rather than the full state directly.
- Policy and Reward: The policy outputs 20-dimensional continuous actions corresponding to desired robot-joint positions.The reward is a weighted sum of K reward components, with stage-specific components described for training.
- Objective: A policy and system dynamics induce a trajectory distribution, and training maximizes expected discounted cumulative reward.The return sums discounted rewards over the trajectory.
- Policy Parameterization: The policy is a deep feed-forward neural network producing the mean and diagonal covariance of a multivariate Gaussian.This parameterization defines a stochastic action policy.
- Optimization: Distributional MPO alternates critic-based policy evaluation with policy improvement.The critic estimates Q^πθ(s, a), the expected return after taking action a in state s and following the policy.
- Self-Play: Sampling opponents from previous agent snapshots makes the full-task objective non-stationary and partially observed.Despite this, the agent eventually learned a well-performing policy.
Training
Training uses a two-stage pipeline: separate soccer and get-up skills are learned first, then distilled into a single agent trained through self-play. This staged setup addresses sub-optimal behavior from direct full-task training.
- Stage 1: The first stage trains separate policies for scoring goals and getting up from the ground.These skills provide the initial behaviors used in subsequent training.
- Stage 2: The second stage distills the skills into one 1v1 agent and continues training through self-play against increasingly strong opponents.Distillation regularization stops after performance exceeds a preset threshold.
Stage 1: Skill Training
Stage 1 trains separate soccer and get-up skills, using goal-scoring and stable, collision-free recovery objectives.
- Soccer Skill: Soccer skill training maximizes goals while using movement, ball-interaction, transfer, and breakage-reduction rewards.Episodes randomize player and ball positions and terminate after falls, boundary violations, penalty-area entry, opponent scores, or 50 seconds.
- Get-Up Skill: The get-up skill follows interpolated target poses derived from three key poses for recovering from the front or back.The target poses bias the policy toward stable and collision-free trajectories.
- Get-Up Skill: The policy conditions on target joint angles and egocentric gravity direction, while the gravity target remains independent of heading.The actual joint positions and gravity direction are compared with their desired targets during training.
- Get-Up Skill: Conditioning the converged policy on the standing key pose produces the get-up skill used in the next training stage.This standing-conditioned version enables recovery from the ground into the upright pose.
Stage 2: Distillation and Self-Play
Stage 2 combines pretrained soccer and get-up skills through distillation, adaptive regularization, and self-play to train a full 1v1 agent.
- Full 1v1 Training: The second-stage agent learns walking, kicking, getting up, scoring, and defending against increasingly strong opponents.Episodes end when either player scores or after 50 seconds, while behavior is initially regularized toward the skill policies.
- Distillation: Policy distillation regularizes the agent toward pretrained skill-policy outputs so it can learn from those skills.The student policy is encouraged to match the relevant skill policy during training.
- Distillation: Because the skills apply in mutually exclusive states, upright states use the soccer skill while non-upright states favor the get-up skill.The relevant skill is selected state by state through critic-based regularization.
- Adaptive Regularization: Adaptive regularization increases cloning pressure below skill-return thresholds and falls to zero after the agent surpasses them.This lets reinforcement learning improve beyond the pretrained policies and learn effective transitions between them.
- Self-Play: Self-play samples opponents from a pool of partially trained agent copies to improve high-level gameplay and opponent awareness.The critic is conditioned on opponent identity to address value-function dependence on opponent strategy and ability.
Sim-to-Real Transfer
The transfer pipeline targets zero-shot deployment by identifying actuator dynamics, using high-frequency position feedback, and reducing simulation mismatch.
- Transfer Strategy: Zero-shot transfer combines system identification, domain randomization, training perturbations, and shaping rewards for robustness and robot safety.The pipeline explicitly targets the sim-to-real gap and behaviors less likely to damage the robot.
- System Identification: Actuator parameters are identified by fitting simulated joint-angle trajectories to responses from sinusoidal controls applied under a known load.The fitted position-controlled model includes damping, armature, friction, maximum torque, and proportional gain parameters.
- System Identification: Although the model does not exactly match servo operation, high-frequency position feedback sufficiently hid the mismatch during training.The position-control model uses fast stabilizing feedback, unlike the servos’ coil-voltage operating mode.
- System Identification: Direct current control failed zero-shot transfer because its sim-to-real gap was too large.The authors expect a more accurate model could reduce this gap further.
Domain Randomization and Perturbations
Targeted dynamics randomization, perturbations, action limits, and reward regularization improve transfer robustness while constraining unsafe robot behavior.
- Randomization: Domain randomization varies selected dynamics because excessive randomization could make the policy conservative and reduce performance.The approach addresses remaining simulation mismatch and robot-to-robot variation from wear and battery state.
- Transfer Outcome: Agents trained without randomization and perturbations fell every one or two steps on real robots and could not score.This demonstrates that both training interventions were needed for successful zero-shot deployment in the reported setting.
- Safety Regularization: Joint action ranges preserve sufficient motion while reducing self-collision risk.The limits are part of the safety-oriented training design.
- Safety Regularization: Reward shaping penalizes stresses associated with dynamic gaits and kicks that can break knee gears.The policies are regularized to reduce damage from foot-ground and foot-ball impacts.
- Ablations: Ablations separately test skill-policy regularization, self-play, and reward components.These experiments examine the contribution of the training design choices.
Importance of Regularization to Skill Policies
Regularization to separately pretrained get-up and soccer skills was essential for avoiding degenerate behaviors and enabling successful 1v1 learning. Ablations also show that self-play produced better fixed-opponent performance than direct training against those opponents, while sampling all policy snapshots destabilized learning.
- Skill regularization: Without skill regularization and reward shaping, agents learned to roll to the ball and knock it into the goal instead of getting up.The corresponding ablation did not learn to get up from the ground.
- Skill regularization: With reward shaping but no skill regularization, agents learned to get up and stand still, failing to walk around or score.This indicates that jointly learning recovery and soccer remained too difficult in this setting.
- Self-play: Self-play agents performed better against six fixed opponents than agents trained directly against those same opponents.The evaluation used 100 matches per opponent, with draws counted as half a win.
- Self-play: Training against all saved policy snapshots produced less stable learning and worse converged performance than sampling from the first quarter.The full method combined self-play with regularization to pretrained skills.
Supplementary Methods
The supplementary methods describe the robot safeguards, observations and rewards, the DMPO training procedure, and supporting simulation-to-real experiments. They also report skill learning behavior, learned gait mechanics, and real-world set-piece performance.
- Robot setup: The OP3 robots used protective bumpers, arms, motion-capture markers, and independently limited joint ranges to reduce damage and self-collisions.The joint limits remained wide enough to permit agile dynamic behaviors and did not completely eliminate collisions.
- Task specification: The training setup specified the agent’s observations and reward components for both skill learning and the full 1v1 task.Reward-component weights for each training stage are listed in the supplementary reward table.
- Skill training: The get-up skill was guided by three key poses for recovery from the front and back, based on a scripted get-up controller.These poses were used to guide training rather than directly serving as the complete learned behavior.
- DMPO: Distributional MPO trained the agents with an actor-critic policy-iteration procedure whose critic outputs a categorical Q-value distribution.Policy evaluation used five-step Q-learning, while policy improvement formed an improved action distribution under a KL constraint before distillation into a parametric policy.
- DMPO: The KL-divergence constraints limited policy changes during improvement, stabilizing learning and avoiding premature convergence.The supplementary implementation used separate online and target networks for the policy and Q-function.
- Learning behavior: At 40 Hz control, longer soccer-skill training improved anticipation and ball handling, including kicking a moving ball.The learned gait used arms for balance and contacted primarily the edges of the feet, unlike the conservative scripted gait.
- Sim-to-real evaluation: In the penalty-kick set piece, the robot scored in all 10 simulation trials but scored 6 of 10 real-world trials.The authors suggest visual variation and sensor noise could explain the discrepancy.