PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Vision-Driven Reactive Soccer Skills for Humanoid Robots

Yushi Wang, Changsheng Luo, Penghui Chen, Jianran Liu, Weijian Sun, Tong Guo, Kechang Yang, Biao Hu, Yangang Zhang, Mingguo Zhao

arXiv:2511.03996v2cs.RO

TL;DR

Humanoid soccer requires reactive coordination of locomotion and unreliable vision, while existing systems often separate perception from control or assume ideal sensing. This paper trains a unified vision-driven RL controller with modeled perceptual uncertainty; it reports reactive soccer skills across real-world scenarios, including RoboCup competition.

  • Problem

    Humanoid soccer requires long-horizon, real-time adaptation to noisy, delayed, and spatially constrained vision, while many approaches separate perception from control.

    Source: §1 · p. 2

  • Method

    A unified RL controller trains in simulation with virtual onboard-vision imperfections and an encoder-decoder architecture to couple visual perception and locomotion.

    Source: §1 · p. 3, §3 · p. 11

  • Results

    The controller performed reactive soccer skills using onboard vision and demonstrated effectiveness in real-world competitive settings, including RoboCup.

    Source: §1 · p. 3, §3 · p. 12

  • Takeaways & Limitations

    The results suggest that integrating perception and motion control can support vision-driven humanoid soccer skills transferred from simulation to real-world environments.

    Source: §3 · p. 13

    The controller observes the ball and goal but not other agents, limiting team behaviors such as collaborative passing, defensive interception, and dynamic role allocation.

    Source: §3 · p. 13

Abstract

from arXiv · show

Humanoid soccer poses a representative challenge for embodied intelligence, requiring robots to coordinate agile locomotion with unreliable visual perception in dynamic environments. However, existing systems typically rely on modular pipelines that separate perception from control or assume ideal sensing, making it difficult to achieve coherent and reactive behavior under real-world perceptual limitations. In this work, we present a unified reinforcement learning-based controller that enables humanoid robots to learn vision-driven reactive soccer skills by directly coupling visual perception with locomotion control. The robot is trained in simulation to acquire soccer behaviors, and adversarial motion priors guide policy learning toward natural motion patterns. To support robust performance under imperfect sensing, we introduce an encoder-decoder architecture together with a virtual perception system that models key characteristics of onboard vision, exposing the policy to perceptual noise and detection failures during training. This design encourages the policy to internalize perceptual uncertainty and continuously adapt its motion in a closed loop. The resulting controller produces coordinated soccer behaviors using only onboard vision, including ball searching, chasing, and multidirectional kicking. It reduces ball position estimation error by 46% and shortens time-to-kick by up to 64% compared with a rule-based baseline, achieving around 90% kicking success in frontfield positions. Experiments across diverse environments and dynamic scenarios, including real RoboCup competitions, further demonstrate the robust performance of the controller. These results highlight the practical effectiveness of integrating perceptual uncertainty directly into policy learning for achieving reliable vision-driven behaviors in humanoid robots operating under real-world conditions.

1 Introduction

Humanoid soccer requires long-horizon, reactive coordination despite noisy, delayed, and spatially constrained vision. The paper proposes a unified vision-driven RL controller that learns soccer behavior while incorporating perceptual uncertainty into control.

1 Introduction

  • Noisy, delayed, spatially constrained onboard vision makes real-time adaptation and tightly coordinated humanoid soccer behavior difficult.The robot must also coordinate locomotion, foot placement, orientation, and kick timing under physical and visual disturbances.Source: §1 · p. 2
  • Modular pipelines and skill-specific learning approaches separate perception, planning, and control, limiting integrated reactive behavior in dynamic soccer environments.The modular approach also depends heavily on accurate perception estimates and extensive manual tuning.Source: §1 · p. 2
  • Earlier end-to-end RL relied on motion-capture state, while a later vision-based approach required scene-specific modeling and showed sim-to-real degradation.These constraints leave robust vision-driven reactive control in real-world humanoid soccer an open challenge.Source: §1 · p. 2
  • The proposed unified RL framework trains in simulation with adversarial motion priors, virtual onboard-vision imperfections, and an encoder-decoder that couples perception with locomotion.The controller uses visual perception and locomotion in a closed loop to adapt to uncertainty; experiments report searching, chasing, and multidirectional kicking using onboard vision.Source: §1 · p. 3

2 Results

The results show that a simulation-trained controller performs soccer skills across varied real-world settings, including competitions, while coordinating visual perception and locomotion. Evaluations examine its deployment pipeline, kicking success, perception-action behavior, and learned gaits.

2.1 System overview

  • On Booster T1, a head-mounted camera supplies BEV ball detections directly to the policy, while field landmarks pass through odometry to infer goal location.The policy receives proprioception, ball detections, and goal information; the camera pipeline runs at 25 Hz and joint commands at 50 Hz.Source: §2.1 · p. 3, Figure 1 · p. 4

2.2 Soccer skills performance

  • Deployed zero-shot after simulation-only training, the controller tracked the ball and adapted locomotion across varied terrain and visual conditions, including occasional detection failures.It also sustained performance without field landmarks by using proprioceptive odometry, and searched actively when ball tracking was lost.Source: §2.2 · p. 4
  • The Tsinghua Hephaestus team won the 2025 RoboCup Adult-Size Humanoid League and World Humanoid Robot Games championships with 76 goals scored and 11 conceded.The controller was used under competition constraints that required autonomous operation with onboard computation and sensing.Source: §2.2 · p. 4

2.3 Kicking success rates

  • Hardware kicking success was ∼80 to 90% in frontfield positions and 60 to 70% in backfield positions over 10 tests per position, with no falls recorded.Hardware rates were comparable to simulation results.Source: §2.3 · p. 6

2.4 Perception-action coordination

  • Compared with raw visual perception, the policy reduced minimum ball-position RMSE in the last 1 s before kicking from 0.344 to 0.186 m, demonstrating perceptual denoising.The policy kept the ball in the camera’s field of view for the majority of the time and adaptively rotated the robot’s head and torso to reposition it when it neared the FOV edge.Source: §2.4 · p. 6

2.5 Gait behavior analysis

  • Five gait clusters corresponded to walking, left and right turns, and left- and right-foot kicks, while trajectories extended beyond reference motions to task-specific behavior.One example is a pivot hook kick when the goal was behind the robot, avoiding a full-body rotation.Source: §2.5 · p. 9, Figure 5 · p. 10
  • The learned policy took ∼1.5 s to kick across orientations, versus ∼2 s facing the goal and about 5 s when the rule-based strategy faced away.It combined the kick with the final approach step; measured turning speeds were 3 to 4 rad/s versus ∼2 rad/s for the rule-based strategy.Source: §2.5 · p. 9, Figure 6 · p. 11
  • Near the ball, the gait cycle shortened from ∼0.7 to 0.8 s at greater distances to ∼0.3 to 0.4 s, supporting faster foot-placement adjustments.When chasing a rolling ball, the policy used lateral interception steps or reoriented in as few as two steps.Source: §2.5 · p. 10

3 Discussion

The controller couples visual perception with agile locomotion and shows stable, responsive soccer performance across real-world conditions, including RoboCup competition. Its design combines perception-aware learning and adversarial motion guidance, while the current scope excludes team coordination and several broader soccer behaviors.

3 Discussion

  • The RL controller integrates visual perception with agile locomotion, using an encoder-decoder and virtual perception system to support consistent representations across simulation and hardware.The framework incorporates adversarial motion priors and is designed to adapt to perceptual uncertainty in a closed loop.Source: §3 · p. 11
  • The robot maintained stable, fluent performance across varied terrains and visual and interaction conditions, gained ball possession faster than rule-based strategies, and scored goals in RoboCup competition.The experiments also included rolling-ball responses and kicking, with Figure 7 comparing simulation bars from 8,192 trials per velocity against real-world markers from 10 trials per velocity.Source: §3 · pp. 11–12, Figure 7 · p. 12
  • An adversarial discriminator supplies implicit motion guidance, while the unified policy adjusts gait, head orientation, and kicking angle to real-time ball and goal positions.The approach is contrasted with modular pipelines that separate strategic planning from motor control and feature-based imitation that depends on strict temporal alignment.Source: §3 · pp. 12–13
  • The perception pipeline uses compact ball-detection-derived state and odometry memory, while virtual perception and the encoder-decoder support active perception and ball-position estimation during motion.The discussion connects these design choices to robustness under motion blur, partial observability, and temporary loss of visual landmarks.Source: §3 · p. 13
  • The controller is limited to individual skills and ball-and-goal observations, and does not yet reproduce behaviors such as dribbling, trapping, or strategic passing.The authors identify multiagent training and strategic priors or task-level cues as directions for addressing these limits.Source: §3 · p. 13

4 Materials and Methods

The method trains a vision-driven soccer policy in simulation and transfers it to hardware, combining partial-observation control, temporal state reconstruction, virtual perception, and adversarial motion guidance. Training randomizes field interactions and sensing conditions, while the evaluation reports independent simulation rollouts and real-world trials.

4.1 Training environment

  • Training in Isaac Gym used a RoboCup-sized field, randomized ball dynamics and mild terrain unevenness, and introduced ball and robot disturbances to represent match events and physical confrontations.Episodes ended on falls, out-of-bounds balls, goals, or a 60-second limit; after a goal or out-of-bounds event, only the ball was reset to support continuous kicking practice.Source: §4.1 · pp. 13–15

4.2 Training procedure

  • The policy was trained as a POMDP with asymmetric actor-critic PPO: the actor used sensor-available partial state, while the critic accessed full simulator state.A sequence of 50 preceding observations spanning one second was encoded into a 64-dimensional latent vector; a decoder reconstructed auxiliary information, including dynamic parameters and true ball position. Dense approach and goal-progress rewards supplemented sparse goal rewards, and separate critics estimated goal-related and auxiliary returns.Source: §4.2 · p. 15

4.3 Perception system

  • The perception system gives the policy ball position and a visibility flag, goal-center coordinates, and goal-normal direction in robot coordinates; YOLO detections and fused projections provide ball and landmark estimates.A virtual perception system models detection probability, distance-dependent noise, update frequency, latency, and head-pose-relative field of view, exposing the policy to realistic perceptual uncertainty during training.Source: §4.3 · p. 16
  • The virtual perception model was fitted using about one hour of real-robot observations across head orientations and relative positions, with Gaussian positional noise whose variance depended linearly on relative position.The authors note a potential distribution discrepancy because data collection used a different gait from the learned policy, while reporting that the simple model generalized across locomotion patterns.Source: §4.3 · p. 16

4.4 Learning from demonstration

  • AMP uses retargeted walking and arch-kicking demonstrations to train a discriminator whose style reward is combined with environmental rewards, allowing the policy to compose walking-to-kicking transitions without explicit sequencing.The demonstration set comprised 76.28 seconds of omnidirectional walking and 30 seconds of arch-based kicking; Wasserstein GAN training, reference-state initialization, and mirror-symmetry loss supported stable training and bilateral kicking.Source: §4.4 · pp. 16–17
  • The authors characterize the integration of AMP with perception, physical modeling, and domain randomization as supporting deployment under partial observability and real-world sensing and actuation imperfections.The stated aim is reliable deployment on real humanoid hardware.Source: §4.4 · p. 17

4.5 Statistical analysis

  • Simulation rollouts and real-world trials were treated as independent samples, with means and standard deviations computed where applicable and figure captions specifying sample sizes and analysis details.Error bars and shaded regions represent standard deviation unless otherwise specified.Source: §4.5 · p. 17

A Training details

Training used large-scale parallel simulation and corrupted only actor observations with simulated sensor noise to match hardware sensing.

A Training details

  • The policy trained for 20,000 epochs in 16,384 parallel environments using eight NVIDIA V100 GPUs, taking about one day.The training pipeline used NVIDIA Isaac Gym and a sim-to-real framework.Source: A Training details · p. 22
  • Table 1 summarizes the training hyperparameters.The accompanying training details also refer to domain-randomization ranges and actor, critic, discriminator, and reconstruction spaces in Tables 2 and 3.Source: Table 1 · p. 22, A Training details · p. 22

B Reward functions

The reward design combines task, style, and regularization signals, while separate critics estimate goal-related and auxiliary reward groups before PPO combines their advantages.

B Reward functions

  • The reward function combines task, style, and regularization rewards to guide policy learning.The component breakdown is listed in Table 4.Source: B Reward functions · p. 22, Table 4 · p. 24
  • A goal-related critic estimates scoring, ball-approach, and goal-progress rewards, while an auxiliary critic estimates the remaining rewards.Each critic learns a value estimate from its own reward subset.Source: B Reward functions · p. 22
  • PPO uses a weighted sum of the two critics’ advantages, with the goal-related advantage weighted 2 and the auxiliary advantage weighted 1.The authors report that this decomposition reduced negative interference between reward components and improved PPO-update stability.Source: B Reward functions · p. 22

C AMP training

AMP uses adversarial motion demonstrations to shape policy learning toward reference-like motions. The ablations report that WGAN reduces motion mode collapse, while removing AMP blurs behavior distinctions and can produce physically implausible actions.

C AMP training

  • The AMP discriminator scores expert and policy motion transitions, and the policy is trained to minimize their Wasserstein distance.The implementation adds a soft boundary constraint and uses a gradient penalty on random interpolations to enforce Lipschitz continuity.Source: C AMP training · p. 24
  • Without AMP, walking, turning, and kicking remain present but become more entangled, with blurred distinctions in the learned motion space.The UMAP embeddings without AMP appear compact and lack clear cluster separation.Source: C AMP training · pp. 24–25, Figure 9 · p. 25
  • Without sufficient motion constraints, the policy can learn aggressive, physically implausible motions despite high simulation task rewards.The authors note that reward shaping and regularization could address this but typically require substantial task-specific engineering and tuning.Source: C AMP training · p. 25
  • WGAN produces smoother, more humanlike motions than LSGAN, whose motions are less natural and lower in quality.The nearest-neighbor analysis also links LSGAN’s larger-distance samples to collapsed kicking behaviors.Source: C AMP training · p. 25
  • WGAN concentrates nearest-neighbor distances at small values and alleviates mode collapse, unlike LSGAN’s larger-distance peak associated mainly with collapsed kicks.The distance distribution compares policy-generated motions with the reference dataset in AMP observation space.Source: C AMP training · p. 25, Figure 10 · p. 26

D Symmetry loss

The symmetry loss encourages consistent policy responses to mirrored states, supporting bilateral kicking. In the ablation, removing it led to single-foot bias and slower execution when the ball was on the opposite side.

D Symmetry loss

  • The loss compares policy actions for an observation and its mirrored counterpart, constructed by swapping left-right joint states and inverting lateral ball and goal observations.The action is mirrored correspondingly to encourage consistent responses to symmetric states.Source: D Symmetry loss · p. 26
  • This regularization preserves bilateral motion capability and lets the policy choose either foot according to task context.The study was designed to evaluate the effect of the mirror symmetry loss.Source: D Symmetry loss · p. 26
  • With symmetry loss, the policy selects its kicking foot according to ball position; without it, the policy favors one foot and may rotate around the ball before kicking from the opposite side.The resulting repositioning increases execution time and reduces overall efficiency.Source: D Symmetry loss · p. 26

E Perception system and modeling

The perception system combines a fine-tuned onboard detector with a virtual model of visual noise, latency, update frequency, and missed detections. Real-world ball-error measurements followed the model’s distance-dependent trend, supporting its fidelity for policy training.

E Perception system and modeling

  • A YOLOv8s detector was fine-tuned on approximately 30,000 onboard-camera frames for ball, goalpost, penalty-point, and field-intersection detection.Remaining failures mainly occurred under severe motion blur or at extreme viewing distances, including missed detections and confusion between similar intersections.Source: E Perception system and modeling · p. 26, Figure 12 · p. 27
  • The virtual perception model samples distance-dependent positional noise, models latency at N(116 ms, (18 ms)^2) and frequency at N(25.36 Hz, (1.06 Hz)^2), and sets 90% ball detection within 7 m in the field of view.Beyond 7 m, the detection rate gradually decays.Source: E Perception system and modeling · pp. 26–27, Figure 13 · p. 28
  • Real-world ball-position errors followed a distance-dependent trend close to the virtual model, whose measured errors were slightly higher than those observed during policy execution.The authors attribute the lower real-world errors to improved motion stability and reduced camera jitter; the analysis covered approximately 7,000 perception frames.Source: E Perception system and modeling · p. 27, Figure 14 · p. 28
  • The authors conclude that the virtual perception system faithfully approximates real-world visual characteristics and supports sim-to-real transfer.The measured real-world error distribution was used to validate the model.Source: E Perception system and modeling · pp. 27–28

F Odometry implementation

The hybrid odometry system combines proprioceptive motion estimates with vision-based landmark localization, falling back to proprioception during degraded visual updates. During policy execution, visual corrections constrained position error while pure proprioceptive error accumulated over time.

F Odometry implementation

  • The proprioceptive module processes a 1 s window of proprioceptive observations and prior odometry output to predict incremental displacement.Its MLP is trained on Isaac Gym trajectories to minimize mean squared error against simulated motion increments.Source: F Odometry implementation · p. 29
  • A particle filter matches detected landmarks in the robot-centric BEV frame to a global field map and uses localization quality to correct proprioceptive drift.The correction is weighted by landmark count and residual error.Source: F Odometry implementation · p. 29
  • When visual localization is unavailable or fails its convergence checks, the system defaults to proprioceptive odometry for continuous localization.The fusion strategy is intended to compensate for drift while maintaining localization through temporary visual occlusions or degraded perception.Source: F Odometry implementation · p. 29
  • At least one landmark was detected in approximately 89% of frames, and more than five were detected in about 52% during real-world deployment.The RoboCup field contained 30 recognizable landmarks, and detected categories tracked their field proportions.Source: F Odometry implementation · p. 29, Figure 15 · p. 30
  • Vision-based corrections constrained localization error within approximately 1 m, whereas purely proprioceptive odometry accumulated error at about 0.04 m/s.The comparison used motion-capture ground truth during policy execution; Figure 15C reports the localization-error trajectories.Source: F Odometry implementation · p. 29, Figure 15 · p. 30

G Deployment in RoboCup matches

RoboCup deployment combined a learned ball-handling policy with hierarchical match logic for rules, roles, and coordination. The policy was activated in defined attacker contexts, while a walking controller handled navigation and positioning outside those windows.

G Deployment in RoboCup matches

  • A behavior tree handled rule compliance, role assignment, and multi-robot coordination, activating the learned policy according to game state and spatial relationships.When the policy was inactive, a velocity-tracking walking controller handled navigation and positioning.Source: G Deployment in RoboCup matches · p. 30
  • Only the primary attacker could enter the learned policy, when the ball was within a predefined distance and frontal space was sufficiently clear.The primary attacker was selected by spatial advantage relative to the ball; teammates used the walking controller for support behaviors.Source: G Deployment in RoboCup matches · p. 30
  • Ball-range exit or a maximum execution time ended the learned policy and returned control to the walking controller.This deployment operated amid match rules, discrete game states, and role-dependent behaviors.Source: G Deployment in RoboCup matches · p. 30
  • The hierarchy reserved tightly coupled perception-action control for critical ball-handling phases while other components handled broader tactical and rule-based behaviors.Source: G Deployment in RoboCup matches · p. 30

H Human motion retargeting

The human-motion retargeting pipeline adapted H2O to the robot's morphology and multiple human-motion representations. It produced kinematically consistent reference motions that preserved human movement structure while respecting robot constraints.

H Human motion retargeting

  • The retargeting pipeline adapted H2O to the robot's morphology and supported multiple human-motion representations.Source: H Human motion retargeting · p. 31
  • Shared keypoints linked BVH or SMPL human-body representations to the robot, with fitted scaling and offset parameters aligning limb lengths and body proportions.Source: H Human motion retargeting · p. 31
  • Per-frame optimization solved robot joint angles and base pose using weighted position, orientation, and smoothness terms, subject to joint limits and kinematic feasibility.The pipeline additionally optimized base pose and added link-orientation and smoothness terms; trajectories were smoothed for continuity and execution.Source: H Human motion retargeting · p. 31
  • The resulting reference motions preserved human movement structure while respecting robot constraints, yielding kinematically consistent trajectories for policy training.Source: H Human motion retargeting · p. 31
Loading 2511.03996v2…