Banana Kick: Response-Informed Skill Evolution for Humanoid Soccer

I. INTRODUCTION

• RISE addresses first-order learning starvation by adapting objectives to reachable responses and retaining updates only when they advance physical responses while preserving correct-foot reliability.The aim is to move beyond the prior’s familiar contact regime, not merely refine speed or accuracy.Source: §I · p. 2

A. Curriculum learning and reward design

• Curriculum and reward-design methods alter tasks or learning signals, whereas RISE selects reward changes using local response sensitivity on the current rollout distribution.Prior work includes intermediate-task curricula, automatic teachers, and reward shaping [4, 8, 11, 14].Source: A. Curriculum learning and reward design · p. 2

B. Policy adaptation and physical skill discovery

• Motion priors provide coordinated behavior, but adapting a competent kick into a distinct spin-generating contact regime differs from refining existing soccer behaviors.Residual policies, skill discovery, and exploration methods address related adaptation or diversity goals.Source: B. Policy adaptation and physical… · p. 2

C. Continuation and sensitivity-guided search

• RISE scores candidate rewards directly on cached responses, then verifies selected changes through policy training and measured response progress.This differs from continuation methods that warm-start parameterized solutions and from search methods using surrogate models.Source: C. Continuation and sensitivity-guided… · p. 2

III. METHODOLOGY

• The pipeline combines an imitation prior with reinforcement learning and RISE to change the foot–ball contact regime under calibrated contact and Magnus-force aerodynamics.The architecture separates skill initialization from skill evolution.Source: §III · p. 2

A. Motion capture, retargeting, and imitation

• Captured human motion is retargeted to a robot-feasible reference, and imitation trains a feedback policy to execute it under simulated dynamics.The ordinary-kick prior supplies balance and swing coordination for later contact adaptation.Source: A. Motion capture, retargeting, and… · p. 2

B. Reinforcement learning with calibrated physics

• The imitation policy initializes curved-kick reinforcement learning, which changes foot–ball interaction to generate spin while retaining learned whole-body coordination.The actor uses deployable proprioception, motion-reference cues, ball-relative state, and the previous action.Source: B. Reinforcement learning with… · p. 2
• Calibrated elastic contact and Magnus-force aerodynamics determine ball velocity and spin consequences, while the simulation transition model remains fixed during policy adaptation.PPO updates the policy from rollouts under that model; RISE selects response-reward parameters.Source: B. Reinforcement learning with… · p. 3
• RISE adapts contact and ball-motion response channels, including patch error, tangential impact speed, spin, and outgoing ball speed, while maintaining a minimum correct-foot-contact probability.The fixed reward terms and task targets remain unchanged.Source: B. Reinforcement learning with… · pp. 3–4

• RISE scores candidate objectives using response sensitivity on cached rollouts, with fixed contexts and normalization constants, before policy training.Source: C. Response sensitivity and local… · p. 4
• A channel is first-order starved when its reward derivative with respect to that response is zero almost surely under the current policy.Source: C. Response sensitivity and local… · p. 4
• If response-reward derivatives vanish almost surely and differentiation can pass through expectation, the response-mediated policy gradient is zero.Small response derivatives likewise imply small response-mediated gradients when the response Jacobian is bounded.Source: C. Response sensitivity and local… · p. 4
• For the bell kernel at a fixed response distinct from its target, sensitivity is maximized when its scale equals the response-target distance.Source: C. Response sensitivity and local… · p. 4
• Rescaling a saturated spin reward cannot restore first-order sensitivity at zero spin, whereas a coupled contact response can initiate spin.Source: C. Response sensitivity and local… · p. 4
• Bounded warm-started objective changes have a local continuation interpretation when the loss is twice continuously differentiable, the stationary point is regular, and the policy Hessian is nonsingular.Under those conditions, the implicit function theorem yields a locally smooth stationary branch as objective parameters vary.Source: C. Response sensitivity and local… · p. 4

• At each accepted iterate, candidate objectives must lie in the admissible domain and remain within the objective trust-region radius.The proposal also penalizes objective change, and experiments enumerate a discrete candidate set.Source: D. Objective continuation and policy… · p. 4
• Table II reports a five-seed cross-method comparison under a fixed evaluation score, with brackets denoting 95% matched-context bootstrap intervals.Source: Table II · p. 5
• Figure 3 compares simulated straight and banana-kick trajectories and physical versus simulated contact-side sweep and follow-through; Figure 11 reports quantitative flight trajectories.Source: Figure 3 · p. 5
• Figure 4 compares fixed-objective score and normalized response regret across methods, while Figure 5 shows paired score differences and patch-precision improvement versus spin change.Source: Figure 4 · p. 5, Figure 5 · p. 5
• For a selected objective, PPO warm-starts from the accepted policy while the objective parameters and response model remain fixed during the inner update.The changed reward then guides contact adaptation from the accepted checkpoint.Source: D. Objective continuation and policy… · p. 5
• A candidate is accepted when normalized physical-response displacement exceeds a threshold and required reliability is retained.Matched contexts estimate policy means for the targeted response coordinates and their desired directions.Source: D. Objective continuation and policy… · p. 5
• The continuation loop expands its radius after acceptance and contracts it after rejection; the final accepted policy is transferred to hardware without further learning.The loop can terminate when its training budget is exhausted, its radius falls below a minimum, or no admissible proposal provides sufficient excitation gain.Source: D. Objective continuation and policy… · p. 5

A. Experimental setup

• Experiments use a 23-joint Unitree G1, with IsaacSim/PhysX at 200 Hz and the policy at 50 Hz.The ordinary-kick prior is trained from retargeted motion-capture references.Source: A. Experimental setup · p. 5
• RISE is compared with a learning-progress curriculum, direct-final PPO, and two ablations under equal interaction budgets and five random seeds.Each terminal policy is evaluated using the actor mean on 4,096 matched ball-pose contexts with offsets within ±5 cm.Source: A. Experimental setup · p. 5
• Fig. 6 depicts straight- and curved-kick approach, contact, and follow-through, while Fig. 7 tracks checkpoint gains, response coordinates, paired-score increments, and patch-error–spin distributions.The overlaid paths illustrate post-contact differences.Source: Figure 6 · p. 6, Figure 7 · p. 6
• Every evaluated trajectory is rescored using one fixed physical-response objective because rewards change during adaptation.Evaluation reports score, score gap, joint coverage, lower-tail CVaR10, response regret, and the fraction of contexts where RISE outperforms each comparator.Source: A. Experimental setup · p. 6
• Joint coverage is the fraction of matched contexts with zero response regret, with thresholds covering patch error, tangential speed, spin, and contact alignment.Evaluation parameters and thresholds are listed in Table I.Source: A. Experimental setup · p. 6

B. Cross-method performance and component ablations

• 19.8% higher mean evaluation score than the learning-progress curriculum, with joint target coverage rising from 15.23 to 50.88% and response regret falling from 0.1427 to 0.0149.RISE attains a mean evaluation score of 1093.7.Source: B. Cross-method performance and… · p. 6
• Removing sensitivity ranking raises spin to 13.210 rad/s but degrades patch error, tangential speed, and joint target coverage to 0.073%.The curriculum reaches nearly identical mean spin to RISE.Source: B. Cross-method performance and… · p. 6
• Without acceptance feedback, the policy produces the highest ball speed but only 1.887 rad/s spin and 11.099 cm patch error.Direct-final PPO produces zero spin in terminal simulation evaluation.Source: B. Cross-method performance and… · p. 6
• RISE evolves toward a coordinated contact regime that jointly satisfies spin generation, contact placement, tangential impact, and reliability.Source: B. Cross-method performance and… · p. 6

C. Response evolution during adaptation

• Figs. 8 and 10 organize sensitivity-scale and response-space diagnostics, while Table III covers simulated terminal responses for five policies using spin, patch error, and speeds.Fig. 8 shows excitation, scale mismatch, and accepted-versus-rejected candidate diagnostics; Fig. 10 shows patch-error support against reward sensitivity and responses before and after correction.Source: Figure 8 · p. 7, Figure 10 · p. 7, Table III · p. 7
• At the final checkpoint, score reaches 1093.7, response regret falls to 0.0149, and joint target coverage reaches 50.88%.Earlier checkpoints raise score from near zero to 688.2 without joint coverage, then reach 792.8 while reducing regret from 0.367 to 0.105.Source: C. Response evolution during adaptation · pp. 6–7
• Widening the patch-kernel scale from 0.02 to 0.10 m follows initial patch error of 15.78 cm and raw excitation of 2.07 × 10−16; patch error then drops to 4.45 cm and spin rises to 12.671 rad/s.Spin was zero across evaluation contexts at initialization.Source: C. Response evolution during adaptation · p. 7
• The sequence supports adapting a coupled contact response first, moving beyond the prior’s contact regime before substantial spin emerges despite an initially flat spin channel.Source: C. Response evolution during adaptation · p. 7
• For bell-kernel scale edits, the excitation-maximizing candidate matches the recorded choice in four of five transitions; the remaining choice differs by only 2% in normalized excitation.The authors present Fig. 8 as a mechanism-consistency diagnostic rather than an independent benchmark.Source: C. Response evolution during adaptation · pp. 7–8

D. Physical deployment

• 30 motion-capture-recorded physical trials exhibit the same bending direction as simulated flights after the final policy is frozen and transferred without further learning.All reported physical statistics use the full 30-trial set.Source: D. Physical deployment · p. 8
• Simulated flights have a median top-view bow of 0.381 m and a median heading change of 33.1 degrees.Source: D. Physical deployment · p. 8
• For hardware transfer, the final RISE policy is frozen and deployed without further learning; Fig. 11 displays 20 simulated and 20 physical trajectories for clarity, while physical statistics use all 30 physical trials.Source: D. Physical deployment · p. 8

V. CONCLUSIONS

• RISE adapts the objective to reachable responses and retains updates only after verified response progress and kicking reliability, allowing the ordinary-kick prior to leave familiar contacts.The prior serves as a scaffold for qualitatively new contact-rich behavior rather than a terminal skill template.Source: §V · p. 8
• The response-space analysis explains how coupled contact responses can restore a viable learning direction when the target response itself is locally uninformative.This characterizes first-order learning starvation as a limitation of the objective’s information about the policy’s current responses.Source: §V · p. 8
• More than 11.55 rad/s mean ball spin, a 19.8% higher mean evaluation score than a learning-progress curriculum, and joint target attainment rising from 15.2% to 50.9% mark RISE’s results.Repeated physical trials reproduce the learned curved-kick behavior on real hardware without further fine-tuning.Source: §V · p. 8