Source-linked AI summary
Drone Soccer: Learning to Manipulate with Multicopter Downwash
Neelay Joglekar, Bavin Saravanan, Yutong Wang, Varun Kandiyappan, Junyi Geng, Sebastian Scherer
TL;DR
Aerial manipulation has generally relied on physical contact, and prior aerial work has ignored downwash or treated it as a disturbance. This paper develops a simplified downwash model and trains a closed-loop RL policy for drone soccer; hardware tests show goal scoring and passing, with performance bounded by ball speed.
Problem
Prior aerial manipulation work either ignores downwash or treats it as a disturbance, while modeling it for dynamic manipulation is difficult because accurate CFD is costly and simpler models assume stationary drones away from surfaces.
Source: §I · p. 1Method
The authors develop a simplified downwash dynamics model and use it to train a closed-loop RL policy for a drone soccer task in which the drone pushes a ball using downwash only.
Source: §II · p. 2, A. Our Contribution · p. 2Results
3/4 goals were achieved within the geofence, and the policy completed 10 passes in human-in-the-loop experiments, but struggled to redirect a 2.52 m/s kick.
Source: B. Hardware Experiments · p. 4Takeaways & Limitations
The authors conclude that their heuristic downwash model is sufficient to train an RL policy that transfers well to real-world drone soccer.
Source: §IV · p. 4The policy performs well when ball speed stays below approximately 1.5 m/s, and could not handle the final 2.52 m/s kick in the human-in-the-loop experiment.
Source: B. Hardware Experiments · p. 4
Abstract
from arXiv · showhide
Although multicopter drones are traditionally designed for "perception-only" tasks, like mapping and exploration, recent work has sought to develop Unmanned Aerial Manipulators (UAMs) to solve mobile manipulation tasks. Aerial manipulation performance can be impacted by "downwash," the airflow produced by propellers, but current state-of-the-art UAMs either ignore downwash or treat it as a disturbance. Instead, is it possible to actively use downwash as a tool during manipulation? We design a drone soccer task to explore the feasibility of downwash-based manipulation. Specifically, we develop a simplified downwash dynamics model which we use to train an RL policy to dribble a soccer ball. We further demonstrate that our policy transfers to real world deployment. This work provides key insights into novel manipulation capabilities for multicopters.
I. INTRODUCTION
Aerial manipulation has largely relied on direct contact, while the role of propeller downwash in manipulation remains underexplored. The paper asks whether drones can actively use downwash, despite the difficulty of modeling it during dynamic flight near surfaces.
I. INTRODUCTION
- Multicopter aerial manipulation has typically used direct-contact arms, cables, or other tools for tasks such as carrying loads and playing racket sports.The paper contrasts contact-based approaches with using propeller airflow as an interaction mode.Source: §I · p. 1
- Downwash-based manipulation is largely unexplored for multicopters: prior aerial manipulation work ignores downwash or treats it as a disturbance.Pesticide drones passively use downwash to penetrate leaf canopies, but that does not demonstrate active manipulation.Source: §I · p. 1
- Existing downwash models are inadequate for dynamic manipulation: accurate CFD is computationally intensive, while simpler models assume stationary drones without nearby surfaces.The paper motivates a model that can support MPC- or RL-based control under more complex conditions.Source: §I · p. 1
- Prior closed-loop control results with simplified models motivate testing whether a closed-loop policy can enable downwash-based manipulation.The cited background includes sim-to-real transfer using simplified particle-based fluid dynamics for a wheeled robot.Source: §I · pp. 1–2
A. Our Contribution
The paper presents drone soccer as a test of downwash-based manipulation and contributes a simplified model and a real-world-transferred closed-loop RL policy.
A. Our Contribution
- The authors develop the first UAM control policy that actively uses downwash to manipulate an object.The policy is evaluated in a drone soccer task.Source: A. Our Contribution · p. 2
- The authors design a simple downwash dynamics model that mimics fluid-ground interaction effects.Source: A. Our Contribution · p. 2
- A closed-loop RL policy trained with the downwash model transfers from simulation to real-world deployment.The paper presents this transfer as a demonstration of the approach's feasibility.Source: A. Our Contribution · p. 2
II. METHODS
The methods define drone soccer as a downwash-only ball-pushing task, develop a model to map drone actions to ball forces, and train an RL policy in simulation.
II. METHODS
- Drone soccer requires a drone to push a ball at high velocity toward a goal using downwash, without direct contact.Source: §II · p. 2
- The authors develop a downwash dynamics model to translate drone actions into forces on the ball.Source: §II · p. 2
- The model is used to train an RL drone soccer policy in simulation.Source: §II · p. 2
A. Downwash Model
The model maps motor thrust to an estimated downwash velocity field and then to force on the ball, using simplifying assumptions whose resulting inaccuracies are handled by a closed-loop controller.
A. Downwash Model
- Motor thrust actions are converted into an airflow velocity field, beginning with an induced-velocity estimate based on average propeller thrust and Momentum Theory.Induced velocity is the flow-aligned airspeed produced at the rotor plane.Source: A. Downwash Model · p. 2
- The simplified downwash model estimates downwash speed from induced velocity, using fitted parameters and neglecting radial flow components treated as negligible.The parameters were fitted from anemometer data.Source: A. Downwash Model · p. 2
- A heuristic spreading angle represents the airflow direction radiating outward near the ground, with the velocity field transformed from the drone’s local frame to the world frame.The field is defined in mixed polar coordinates relative to the drone; the world frame is level with the ground.Source: A. Downwash Model · p. 2
- At the ball’s position, a blunt-drag equation estimates force from the downwash velocity, but does not capture nonuniform, turbulent downwash effects.The authors report that this approximation is sufficient for drone soccer.Source: A. Downwash Model · p. 2
- The model relies on assumptions and heuristics, and a closed-loop controller handles the resulting dynamics inaccuracies.Source: A. Downwash Model · p. 3
B. RL Policy
The policy is framed as an MDP over drone, ball, and goal configurations, with mixed-frame waypoint actions. PPO training uses a curriculum that gradually increases episode difficulty.
B. RL Policy
- The task is represented as an MDP with state space, action space, stochastic transitions, and a reward function, defining a policy and its value function.The optimal policy is specified using the discount factor γ.Source: B. RL Policy · p. 3
- The state captures relevant drone, ball, and goal configurations, while actions specify mixed-frame waypoints supplied to a low-level controller at 10 Hz.The state uses relative xy positions and omits drone yaw because the downwash model does not depend on yaw.Source: B. RL Policy · p. 3
- The reward encourages pushing the ball toward the goal, adds a sparse goal reward and crash penalties, and penalizes large changes in waypoint position.The waypoint smoothness cost discourages large changes in absolute waypoint position.Source: B. RL Policy · p. 3
- PPO training gradually increases difficulty, moving from a drone initialized behind a stationary ball to randomized drone, ball, and goal positions and increasingly varied ball velocities.The curriculum first helps the policy learn to push the ball.Source: B. RL Policy · p. 3
A. Implementation & Environment Setup
The implementation uses a compact policy network and configures simulation and hardware for real-world deployment.
A. Implementation & Environment Setup
- A 2-layer MLP with hidden dimension 64 represents the policy, with waypoint displacements bounded to [−3, 3] meters and height to [0.5, 1.5] meters.The simulation sums rewards between policy inferences before passing them to PPO.Source: A. Implementation & Environment Setup · p. 3
- Deployment uses a Modal AI Starling 2 Max, a 148 g ball, motion capture, PX4 EKF drone-velocity estimates, and a separate Kalman filter for ball velocity.The ball diameter is 13.5 cm.Source: A. Implementation & Environment Setup · p. 3
- The MuJoCo environment matches measured and specified hardware parameters and PX4 controller gains, while an 8 m × 10 m workspace geofence limits commanded waypoints.The downwash parameter s0 was set to −18.07 to match collected ball-downwash data.Source: A. Implementation & Environment Setup · p. 3
B. Hardware Experiments
Hardware experiments test the policy across geofenced configurations and a human-in-the-loop rally. The policy succeeds in most scenarios, with ball speed limiting performance.
B. Hardware Experiments
- The four geofenced trials vary ball initialization: the first two start at zero velocity, and the next two start with nonzero velocity.Drone, ball, and goal positions are initialized within the geofence for these trials.Source: B. Hardware Experiments · p. 3
- 3 of 4 geofenced configurations reached the goal, while the HITL rally completed 10 consecutive passes before an excessive-speed failure on pass 11.Insufficient force sometimes slowed convergence or caused failure.Source: Figure 3 · p. 4, B. Hardware Experiments · p. 4
- In the HITL trial, the goal is outside the geofence and a human kicks the ball back into the drone's workspace for repeated passes.The goal is initialized at (0, 4), beyond the geofence ending at y = +2.Source: B. Hardware Experiments · p. 4
- The policy performs well when ball speed stays below ≈1.5 m/s, but could not handle the final 2.52 m/s kick in the HITL experiment.The authors identify the sim-to-real gap as a likely explanation for insufficient force to redirect the ball quickly.Source: B. Hardware Experiments · p. 4
IV. DISCUSSION & CONCLUSION
The authors conclude that a simple heuristic downwash model can train an RL policy that transfers to real-world drone soccer. They identify extensions to other manipulation tasks and multi-agent settings.
IV. DISCUSSION & CONCLUSION
- A simple heuristic downwash model is sufficient to train an RL policy that transfers well to real-world drone soccer, though performance could improve, possibly through real-world reinforcement learning.The work demonstrates that downwash can be used for manipulation.Source: §IV · p. 4
- The drone soccer formulation could inform farm-animal herding, multi-particle collection, collaborative multi-agent soccer, and adversarial swarm strategy.The paper notes that herding has different dynamics but the same problem formulation as drone soccer.Source: §IV · p. 4