PaperlayerView exact revision on arXiv

Source-linked AI summary

AgilePE: Autonomous UAV Pursuit-Evasion via Self-Play Reinforcement Learning

Wenhao Tang, Tianyang Chen, Zhejun Cui, Boyuan An, Jiayu Chen, Ruize Zhang, Huidong Liu, Tianyue Wu, Qingmin Liao, Fei Gao, Yu Wang, Chao Yu

arXiv:2608.14135v1cs.ROcs.LG

TL;DR

Autonomous UAV pursuit-evasion demands rapid decisions and agile maneuvering against changing opponents. AgilePE combines end-to-end CTBR control, competitive self-play, and hardware-aligned simulation, achieving emergent tactics and zero-shot deployment on physical quadrotors.

  • Problem

    Autonomous UAV pursuit-evasion requires real-time decisions and agile maneuvering under adversarial, highly non-stationary aerial conditions.

  • Method

    AgilePE combines end-to-end CTBR control, bilateral self-play training with PFSP, and hardware-aligned sim-to-real deployment.

  • Results

    PFSP achieved a 0.88 capture rate against scripted baselines, while learned policies displayed emergent flanking and distance-management maneuvers with zero-shot physical deployment.

  • Takeaways & Limitations

    The integrated framework supports autonomous 1v1 UAV pursuit-evasion with emergent tactics transferred from simulation to physical quadrotors without task-specific tuning.

  • Takeaways & Limitations

    Experiments were restricted to 1v1 engagements in obstacle-free environments, leaving multi-agent coordination and cluttered settings untested.

Abstract

from arXiv · show

Autonomous pursuit-evasion is a fundamental challenge for Unmanned Aerial Vehicles (UAVs), requiring rapid decision-making under tightly coupled dynamics and continuously changing opponent behaviors. Traditional rule-based or differential-game approaches often struggle with high-dimensional aerial interactions and agile maneuvering. We present AgilePE, a complete system for autonomous UAV pursuit-evasion via self-play reinforcement learning. AgilePE integrates agile low-level control, competitive policy optimization, and sim-to-real deployment in a unified framework. The policy directly maps onboard state observations to Collective Thrust and Body Rates (CTBR) commands, enabling end-to-end agile maneuvering without intermediate trajectory planners or waypoint controllers. For training, we use competitive self-play with Prioritized Fictitious Self-Play (PFSP) and a diversified opponent pool, enabling agents to improve against historical policies while stabilizing optimization and reducing policy oscillation. This process leads to the emergence of sophisticated pursuit and evasion strategies. For real-world deployment, we develop a hardware-aligned simulation pipeline that models actuator-response dynamics, communication latency, and domain randomization. The learned policies transfer zero-shot to real quadrotors without task-specific tuning. Real-world experiments reproduce pursuit-evasion tactics observed in simulation, including rapid dodging and flanking, and demonstrate interactive two-agent zero-shot deployment.

I. INTRODUCTION

AgilePE addresses autonomous UAV pursuit-evasion, a representative zero-sum adversarial task involving interception and avoidance under highly non-stationary conditions. It integrates end-to-end agile control, competitive self-play, and hardware-aligned sim-to-real deployment to overcome limitations of existing approaches.

  • Motivation: Pursuit-evasion models aerial confrontation as a zero-sum competition in which pursuers seek interception while evaders delay or avoid capture.The paradigm is relevant to counter-UAS operations against unauthorized intruders.
  • Limitations: Existing methods struggle with six-degree-of-freedom aerial dynamics, adversarial-motion model dependence, and failures caused by the sim-to-real gap.The gap arises from unmodeled aerodynamics, actuator latency, and sensor noise; opponent shifts can also cause catastrophic forgetting during co-evolution.
  • End-to-End Agile Control Architecture: AgilePE directly maps state observations to low-level CTBR commands, replacing cascaded planners and waypoint controllers with end-to-end agile control.Its multi-objective reward balances adversarial objectives with physical safety and actuation smoothness.
  • Bilateral Adversarial Training Pipeline: AgilePE uses bootstrapping, curriculum learning, and progressive self-play from naive SP through FSP to PFSP, enabling emergent baiting, dodging, and counterpursuit.The bilateral training pipeline co-evolves pursuer and evader policies under algorithmic evolutionary pressure.
  • Hardware-Aligned Sim-to-Real Deployment Pipeline: AgilePE aligns simulation with hardware through calibrated actuator dynamics, latency and noise modeling, domain randomization, and reward refinement for zero-shot physical deployment.Randomization covers mass, aerodynamics, and latency, while reward refinement smooths control commands.

II. RELATED WORK · A. Methodologies for UAV Pursuit-Evasion Games

UAV pursuit-evasion research spans mathematical differential-game frameworks, classical planning and control methods, and deep reinforcement learning. However, existing DRL work typically uses high-level commands, motivating low-level CTBR control for peak physical agility.

  • A. Methodologies for UAV Pursuit-Evasion Games: Pursuit-evasion games provide a rigorous mathematical framework for analyzing antagonistic aerial interactions, originating with Isaacs’ differential games.This establishes the theoretical foundation for UAV pursuit-evasion methodology.
  • A. Methodologies for UAV Pursuit-Evasion Games: Artificial Potential Fields and MPC offer computationally efficient or constraint-aware solutions for UAV pursuit-evasion.These approaches represent early practical methodologies within the broader pursuit-evasion framework.
  • A. Methodologies for UAV Pursuit-Evasion Games: APF and MPC remain limited by local minima, unpredictable maneuvers, and reliance on accurate forward-predictive models.These limitations constrain their robustness in dynamically changing aerial interactions.
  • A. Methodologies for UAV Pursuit-Evasion Games: Differential games address antagonistic interactions through a rigorous mathematical formulation.The passage situates differential games as a foundational methodology for pursuit-evasion analysis.
  • A. Methodologies for UAV Pursuit-Evasion Games: Deep reinforcement learning has emerged as a powerful paradigm for high-dimensional policy optimization in UAV pursuit-evasion.DRL expands the methodological toolkit beyond classical analytical and model-based approaches.
  • A. Methodologies for UAV Pursuit-Evasion Games: Existing aerial pursuit-evasion DRL studies predominantly operate at high abstraction levels, including velocity commands and waypoint planning.This abstraction separates policy decisions from direct actuator control.
  • A. Methodologies for UAV Pursuit-Evasion Games: Peak physical agility motivates extending DRL to low-level actuator commands, particularly Collective Thrust and Body Rates.CTBR is identified as the relevant low-level control modality for agile flight.
  • A. Methodologies for UAV Pursuit-Evasion Games: CTBR policies have been shown to significantly outperform velocity-level policies in agile flight benchmarks.This supports the methodological shift from velocity-level control toward direct low-level commands.

B. Adversarial Learning and Self-Play in UAV Tasks · C. Sim-to-Real Transfer for Agile Flight · III. AGILEPE SYSTEM DESIGN

AgilePE addresses brittle adversarial learning and sim-to-real challenges through integrated self-play, end-to-end CTBR control, hardware-aligned deployment, and three tightly coupled system subsystems. The architecture combines bilateral adversarial training with robust physical transfer and validates its components in simulation and physical experiments.

  • B. Adversarial Learning and Self-Play in UAV Tasks: Static or hand-scripted opponents often produce brittle UAV policies that generalize poorly to novel adversarial tactics.
  • B. Adversarial Learning and Self-Play in UAV Tasks: Self-play generates an automatic curriculum of progressively stronger opponents, but naive self-play can cause strategy cycling and unilateral collapse.
  • C. Sim-to-Real Transfer for Agile Flight: The sim-to-real gap arises from unmodeled aerodynamic effects, sensor noise, latency, and parameter mismatch.
  • C. Sim-to-Real Transfer for Agile Flight: Successful zero-shot transfer typically combines System Identification with Domain Randomization, including randomized physical parameters and, in advanced variants, simulator dynamics.
  • C. Sim-to-Real Transfer for Agile Flight: AgilePE integrates end-to-end CTBR control, bilateral adversarial training, and hardware-aligned deployment to address agility, tactical diversity, and sim-to-real gaps.
  • C. Sim-to-Real Transfer for Agile Flight: The proposed components are detailed and evaluated through large-scale simulation and physical experiments.
  • III. AGILEPE SYSTEM DESIGN: AgilePE comprises an End-to-End Control Subsystem defining the state-action interface, flight dynamics, and multi-objective reward architecture.
  • III. AGILEPE SYSTEM DESIGN: A Bilateral Training Subsystem governs adversarial policy evolution through hierarchical self-play, while a Real-World Deployment Subsystem supports transfer to physical platforms.

A. Task Description · B. End-to-End Control Subsystem

AgilePE formulates 1v1 aerial pursuit-evasion as a partially observable zero-sum game and controls both 6-DOF UAVs directly through CTBR commands. Its reward design combines FOV-aware tactical objectives with safety and command-smoothness regularization for physically viable control.

  • A. Task Description: The task is a partially observable 1v1 zero-sum game in a bounded three-dimensional workspace.The pursuer minimizes relative distance while keeping the evader in its forward-facing conical FOV; the evader seeks to break detection and maximize separation.
  • A. Task Description: The environment runs N = 2048 parallel arenas in a 6 m × 6 m × 3 m enclosed workspace with invisible boundary walls.Both agents are modeled as 6-DOF rigid bodies driven through a CTBR interface.
  • A. Task Description: Each agent observes its own state, temporally windowed opponent state, and relevant relative positions under conical-FOV and obstacle-occlusion masking.Visibility uses frustum checks and cylinder occlusion testing, with missing entries replaced by a mask value.
  • B. End-to-End Control Subsystem: The control subsystem models each UAV with state xt = [p, q, v, ω]T and CTBR actions u = [Td, ωx,d, ωy,d, ωz,d]T.This provides direct low-level control without an intermediate action representation in the supplied formulation.
  • B. End-to-End Control Subsystem: The reward balances tactical objectives with physical safety through FOV-distance, speed, boundary, proximity, and smoothness terms.The FOV-distance term masks inter-agent distance with a constant when the evader leaves the detection cone, avoiding prohibitively sparse win/loss signals.
  • B. End-to-End Control Subsystem: Opposing rewards form a zero-sum core that encourages persistent tracking under realistic sensing constraints.The pursuer and evader receive opposing rewards scaled by ˜d.
  • B. End-to-End Control Subsystem: Three safety regularizers penalize overspeed, boundary proximity, and mid-air collision risk, while smoothness penalizes high-frequency CTBR oscillations.The overspeed cost is exponential, the boundary cost is quadratic, and smoothness supports stable zero-shot deployment on physical hardware.

C. Bilateral Training Subsystem

The bilateral training subsystem builds on MAPPO and compares three progressively sophisticated adversarial self-play frameworks that co-evolve pursuer and evader policy populations. These frameworks differ in historical-opponent sampling, addressing the nonstationarity and forgetting caused by naive self-play.

  • Framework Design: The subsystem instantiates three adversarial training frameworks that maintain historical checkpoint populations for both pursuer and evader policies.All frameworks are built on MAPPO and differ in how opponents are sampled during co-evolution.
  • Naive Self-Play: Naive self-play pairs the latest pursuer and evader policies synchronously across all environments, providing an ablation baseline without historical overhead.Each agent optimizes against a single rapidly shifting opponent.
  • Naive Self-Play: Naive self-play suffers severe nonstationarity, causing policy oscillation and catastrophic forgetting.The instability arises because each agent continually adapts to one rapidly changing opponent.
  • Fictitious Self-Play: FSP stabilizes training by dividing environments into two halves and training each agent against a uniformly sampled mixture of historical opponent checkpoints.One half trains the pursuer against historical evader checkpoints, while the other trains the evader against historical pursuer checkpoints.

D. Real-World Deployment Subsystem

The real-world deployment subsystem uses a hardware-calibrated deterministic kinematic integrator for CTBR-controlled UAVs, replacing uncertain contact solvers during aggressive maneuvers. It models actuator latency and stochastic noise, then integrates the resulting state with RK4 over ten sub-steps per control cycle for sim-to-real deployment.

  • Kinematic Integration: A hardware-calibrated kinematic integration scheme directly integrates UAV pose and velocity from CTBR commands, replacing uncertain physics-engine contact solvers with deterministic, numerically stable state evolution.The approach differs from standard force-torque-based Isaac Sim simulations and is designed for aggressive maneuvers.
  • Kinematic Integration: RK4 integration uses 10 sub-steps per control cycle, with ∆t = 0.016 s and dt = 0.0016 s.The fourth-order Runge-Kutta scheme maintains numerical stability during high-speed pursuit.
  • Actuator Modeling: Actuator-response modeling combines first-order latency with stochastic noise, using nominal response times of τthrust = 45 ms and τω = 30 ms.Training injects multiplicative noise of ±10% on thrust and ±30% on body rates to improve robustness to hardware variance.
  • Deployment Pipeline: The end-to-end pipeline generates CTBR commands, emulates actuator latency and noise, integrates kinematic state with RK4, and updates Isaac Sim poses while bypassing its internal physics solver.The pipeline performs direct pose updates after ten integration sub-steps per control cycle.

IV. SIMULATION EXPERIMENTS AND RESULTS · A. Bilateral Adversarial Framework Evaluation

The AgilePE system is evaluated in high-fidelity OmniDrones simulation using a systematic training pipeline, with three hierarchical adversarial frameworks compared quantitatively and through cross-evaluation against scripted baselines. The evaluation contrasts naive SP, FSP, and PFSP, including their policy evolution and cross-play payoff matrices.

  • IV. SIMULATION EXPERIMENTS AND RESULTS: AgilePE is evaluated in OmniDrones, a high-fidelity simulation platform designed for UAV reinforcement learning.The section reports quantitative performance for three adversarial regimes and their cross-evaluation against scripted baselines.
  • IV. SIMULATION EXPERIMENTS AND RESULTS: All policies follow a systematic pipeline comprising warm-up, multi-dimensional curriculum learning, and hierarchical adversarial training.
  • IV. SIMULATION EXPERIMENTS AND RESULTS: The three adversarial regimes are evaluated quantitatively and cross-evaluated against scripted baselines.
  • A. Bilateral Adversarial Framework Evaluation: The bilateral training subsystem compares three hierarchical adversarial frameworks: naive SP, FSP, and PFSP.Each framework represents a progressively more sophisticated approach to maintaining tactical diversity and driving policy evolution.
  • A. Bilateral Adversarial Framework Evaluation: Naive SP training is examined through the evolution curve of its in-FOV rate.
  • A. Bilateral Adversarial Framework Evaluation: FSP and PFSP are compared using cross-evaluation payoff matrices that report each Predator i against Prey j as a mean in-FOV rate.The lower-left triangle of the PFSP matrix shows higher saturation, indicating more robust strategy dominance.

1) Naive self-play: · 2) Fictitious self-play (FSP):

Naive self-play initially produces diverse adversarial behaviors but converges to an evader strategy collapse, with in-FOV rates exceeding 0.9. Fictitious self-play addresses this through historical opponent pools, yielding evolutionary policy improvement and higher evader survival against scripted predators.

  • 1) Naive self-play:: Naive self-play was evaluated across multiple independent random-seed trials using in-FOV rate as the pursuit-success indicator.Figure 4 tracks in-FOV rate over training iterations.
  • 1) Naive self-play:: Early naive self-play shows significant in-FOV oscillations, indicating iterative development of pursuer and evader counter-strategies.The oscillations correspond to diversified adversarial behaviors during exploration.
  • 1) Naive self-play:: Late naive self-play converges across all seeds to an in-FOV rate exceeding 0.9, indicating unilateral evader strategy collapse.The process stagnates at a sub-optimal equilibrium rather than reaching a robust Nash Equilibrium.
  • 2) Fictitious self-play (FSP):: At iteration k, each active agent trains against a uniformly sampled opponent policy excluding the current version, then archives the optimized policy.Training rounds are warm-started from the previous iteration, progressively diversifying the training distribution.
  • 2) Fictitious self-play (FSP):: FSP cross-evaluation shows evolutionary progression: later pursuers such as Predator 6 track more historical evaders, while later evaders such as Prey 6 improve survival.The payoff matrix evaluates in-FOV rate across archived pursuer–evader policy combinations.
  • 2) Fictitious self-play (FSP):: 0.81 is the FSP-trained evader’s in-FOV rate against a scripted predator at iteration 4, rising from 0.28 at iteration 0.Naive SP remains at approximately 0.5 against scripted opponents.

3) Prioritized fictitious self-play (PFSP): · V. EMERGENT BEHAVIOR ANALYSIS

PFSP prioritizes historical opponents according to the current policy’s failure rate, while incremental self-play adds converged policies to the strategy pool. The learned agents exhibit tactical behaviors including flanking, latency exploitation, distance management, and precise tracking against scripted and heterogeneous opponents.

  • 3) Prioritized fictitious self-play (PFSP):: PFSP samples historical strategies in proportion to the current agent’s failure rate, creating an automated curriculum beyond uniform opponent selection.The framework extends FSP with weighted opponent sampling from strategy pool H.
  • 3) Prioritized fictitious self-play (PFSP):: Each PFSP training round inherits the previous round’s weights, and its converged policy is added to the strategy pool.This preserves the incremental training regime of FSP.
  • 3) Prioritized fictitious self-play (PFSP):: Over 0.82 in-FOV rate is achieved by the PFSP-trained Predator (Iter 6) against every historical Prey version in cross-evaluation.The result is reported for Figure 5b and supports PFSP’s consistent performance across the historical opponent pool.
  • 3) Prioritized fictitious self-play (PFSP):: Table II evaluates different policies against scripted baselines using In-FOV Rate, including naive self-play policy performance.The supplied table caption and accompanying subsection text identify the evaluation metric and naive SP policy comparison.
  • V. EMERGENT BEHAVIOR ANALYSIS: The emergent-behavior analysis assesses learned policies against scripted and heterogeneous opponents to examine tactical depth, robustness, and generalization.The analysis focuses on interpretable maneuvers produced by the adversarial framework.
  • V. EMERGENT BEHAVIOR ANALYSIS: Against a rigid direct-tracking pursuer, the SP-trained evader performs a dash-and-flank maneuver toward the pursuer’s rear hemisphere.The maneuver is identified in Figure 6a as tactical flanking against a scripted pursuer.
  • V. EMERGENT BEHAVIOR ANALYSIS: The evader exploits rotational latency by decelerating during reorientation, then circling outside the pursuer’s detection cone.This behavior targets structural vulnerabilities in rule-based adversaries.
  • V. EMERGENT BEHAVIOR ANALYSIS: Against a fast-spinning evader, the SP-trained pursuer maintains a strategic standoff distance and uses small-amplitude angular adjustments for boresight tracking.The conservative strategy avoids overshooting and tracking instability associated with aggressively closing range.

VI. REAL-WORLD VALIDATION

AgilePE transfers zero-shot to physical quadrotors through hardware-aligned simulation and onboard CTBR control. Real-world experiments reproduce emergent evasion tactics, including flanking and rapid lateral oscillation to break pursuit tracking.

  • Physical Deployment: Hardware-aligned simulation enables zero-shot deployment of AgilePE on real quadrotor hardware.The pipeline calibrates actuator latencies, applies domain randomization, and enforces control smoothness.
  • Onboard System: The policy runs onboard at 60Hz on a Jetson Orin NX, generating CTBR commands for PX4 control.Pose measurements arrive at 120Hz from motion capture, fuse with IMU data, and support bodyrate delays below 20ms.
  • Emergent Evasion Tactics: The evader performs a high-speed dash, reverses at the arena boundary, and flanks behind the pursuer to break line-of-sight lock.The maneuver exploits the pursuer’s turning latency and mirrors flanking behavior observed in simulation.
  • Emergent Evasion Tactics: Rapid lateral oscillations create high relative angular velocity, causing repeated loss of the pursuer’s visual lock.The evader alternates abrupt left-right accelerations to complicate tracking.

VII. CONCLUSION AND FUTURE WORK

AgilePE integrates end-to-end CTBR control, bilateral adversarial training, and hardware-aligned sim-to-real transfer for autonomous UAV pursuit-evasion. Future work targets multi-agent coordination, cluttered terrain, and vision-based perception beyond the current 1v1, obstacle-free, state-based setting.

  • Conclusion: AgilePE couples end-to-end CTBR control, bilateral adversarial training, and a hardware-aligned sim-to-real pipeline for autonomous UAV pursuit-evasion.The system uses SP, FSP, and PFSP for adversarial training.
  • Conclusion: PFSP sustains tactical diversity and drives robust policy equilibria, while learned policies exhibit emergent flanking and distance-management maneuvers without hand-crafted reward shaping.These findings summarize the reported experimental outcomes.
  • Limitations and Future Work: Experiments were restricted to 1v1 engagements in obstacle-free environments, motivating future scaling to multi-agent coordination and cluttered terrain.The paper identifies this scaling as an important next step.
  • Limitations and Future Work: The agents relied on state-based observations; integrating vision-based perception could enhance autonomy in unstructured environments where precise state estimation is unavailable.Vision-based perception is proposed to address environments lacking precise state estimates.
  • Future Outlook: The developed methodologies lay a foundation for more complex, multi-agent autonomous systems in dynamic and contested environments.This is the paper’s stated broader outlook.
Loading 2608.14135v1…