Source-linked AI summary
DRG-MAPPO: Hierarchical Dynamic Role-Graph Multi-Agent Reinforcement Learning for Cooperative Air Combat
Junlin Liu, Chengwei Li, Yang Gao, Hui Chang, Xinchen Zhang, Zhijun Zhao, Hao Zhao
TL;DR
Cooperative air-combat MARL lacks structured modeling of changing battlefield relations and explicit tactical roles, limiting sophisticated coordination. DRG-MAPPO combines graph attention, dynamic hierarchical role assignment, role-conditioned maneuver control, and a target-priority auxiliary task; it achieves an 87% win rate in high-fidelity BVR simulation. The paper concludes that the framework supports complex collaborative behaviors such as bait-and-flank and focus-fire.
Problem
Existing MARL methods lack structured relational modeling for time-varying battlefield interactions and explicit tactical roles for dynamic task allocation.
Method
DRG-MAPPO combines graph-based relational encoding with hierarchical dynamic role assignment and role-conditioned low-level maneuver control.
Results
87% win rate is achieved in high-fidelity BVR simulation, with advanced collaborative tactics including bait-and-flank and focus-fire emerging.
Takeaways & Limitations
The framework supports complex multi-agent decision-making with tactical coordination that is described as robust and interpretable.
Takeaways & Limitations
The evaluated formulation is a decentralized partially observable setting in which each agent receives local observations, while scalability to larger-scale swarm confrontations remains future work.
Abstract
from arXiv · showhide
Multi-Agent Reinforcement Learning (MARL) has emerged as a pivotal paradigm for complex decision-making in autonomous systems and air combat. While MARL has demonstrated significant potential in air combat, achieving sophisticated tactical coordination remains a non-trivial challenge. This difficulty is largely attributed to two primary limitations: (1) the absence of structured relational modeling hinders agents from capturing complex, time-varying interactions among battlefield entities; and (2) conventional flat architectures often lack the capability to explicitly model tactical roles, leading to ambiguous task allocation in highly dynamic environments. To address these challenges, we propose Hierarchical Dynamic Role-Graph Multi-Agent Proximal Policy Optimization (DRG-MAPPO), a novel MARL framework that integrates graph-based relational modeling with dynamic role assignment. Specifically, DRG-MAPPO constructs a graph-based representation of battlefield interactions and leverages graph attention mechanisms to extract critical relational features among allies, enemies, and threats. Subsequently, a high-level policy employs a dynamic role assignment mechanism to determine tactical responsibilities (e.g., ``leader'' and ``supporter''). Conditioned on these roles and encoded graph-relational features, a low-level policy executes discrete maneuver actions, facilitating the joint optimization of tactical strategy and collaborative execution. Furthermore, a target-priority auxiliary task is designed to foster the emergence of behaviors such as focus-fire. Experimental results demonstrate that DRG-MAPPO achieves a state-of-the-art win rate of 87%, suggesting that our framework effectively balances relational modeling, interpretability, and optimization stability for cooperative air combat.
1 Introduction
The paper identifies missing relational modeling and explicit tactical roles as barriers to coordinated air-combat decision-making. DRG-MAPPO addresses them with graph-based representations, hierarchical role assignment, auxiliary target prioritization, and temporal commitment, achieving an 87% win rate.
- Rule-based, optimization-based, and handcrafted-indicator systems encode domain knowledge but struggle when missile, radar, and teammate interactions jointly shape dynamic engagements.The paper motivates MARL as a response to these scaling challenges.
- Existing MARL frameworks struggle with time-varying battlefield relations and explicit tactical roles, causing limited situational awareness and role confusion.These limitations hinder asymmetric coordination such as leader-supporter behavior.
- DRG-MAPPO combines graph-attention relational modeling with dynamic role assignment in a hierarchical MARL framework.A high-level policy assigns roles, while a low-level policy executes role-conditioned maneuvers.
- The target-priority auxiliary task is designed to foster asymmetric cooperative behaviors, while temporal commitment reduces destabilizing role oscillation during training.The framework separates tactical role assignment from low-level maneuver control.
- 87% win rate demonstrates that DRG-MAPPO outperforms state-of-the-art MARL baselines in high-fidelity air-combat simulation.The evaluation also reports robust and interpretable tactical coordination.
2 Related Work
Related work spans MARL for autonomous air combat, hierarchical reinforcement learning for long-horizon decisions, and graph networks for structured inter-entity coordination. These strands motivate combining decentralized execution, decision decomposition, and relational perception.
- MARL algorithms including MAD-DPG, MATD3, and MAPPO establish CTDE foundations for multi-agent autonomous air-combat systems.Prior work also improves cooperative perception under partial observability through context-aware feature fusion.
- Hierarchical reinforcement learning addresses long-horizon air-combat tasks by separating abstract subtask generation from specific maneuver control.Recent hierarchical architectures build on this high-level/low-level decomposition.
- Graph neural networks, particularly graph attention networks, model complex inter-entity interactions and extract relational features for multi-agent decision-making.Prior applications use graph methods to capture topological dependencies in cooperative optimization settings.
3 Preliminaries
The paper formulates cooperative air combat as a Dec-POMDP and builds on MAPPO with centralized training and decentralized execution. Local actors use observations while a centralized critic evaluates global states and clipped objectives regulate policy updates.
- 3.1 Decentralized Partially Observable Markov Decision Process: Cooperative air combat is modeled as a Dec-POMDP with global states, local observations, joint actions, transitions, observations, rewards, and discounting.The goal is to learn a joint policy maximizing each agent’s expected discounted cumulative return.
- 3.2 Multi-Agent Proximal Policy Optimization (MAPPO): MAPPO uses a shared decentralized actor for local observations and a centralized critic that evaluates global states during training.Parameter sharing is used to improve sample efficiency within CTDE.
- 3.2 Multi-Agent Proximal Policy Optimization (MAPPO): The actor’s clipped surrogate objective restricts policy-update magnitude through the probability ratio between current and previous policies.The clipping parameter ϵ bounds policy updates.
- 3.2 Multi-Agent Proximal Policy Optimization (MAPPO): The critic minimizes mean squared error between estimated state values and discounted empirical returns, while GAE balances advantage-estimation variance and bias.The GAE formulation uses TD error, discount factor γ, and smoothing parameter λ.
4 Simulation Environment
The simulation represents battlefield entities as a directed graph, uses tactical discrete actions, and combines shared combat rewards with dense shaping terms. Cooperative behaviors such as focus-fire are intended to emerge from the graph, role-assignment, and target-priority mechanisms rather than explicit reward engineering.
- Observation Space: The battlefield is modeled at each step as a directed graph whose nodes represent the self agent, allies, enemies, and incoming missiles.The self node encodes intrinsic state, while other node types encode relative or threat-related features.
- Action Space: The low-level policy selects from 12 discrete maneuver and weapon commands, with an underlying autopilot translating commands into flight trajectories.The action space covers adjustment, pursuit, evasion, and missile engagement.
- Reward Function: The reward combines team-shared combat outcomes with dense tactical-advantage, threat-avoidance, and boundary penalties.Combat rewards include win/loss and destruction events, while shaping uses engagement geometry, missile threats, and operational-zone violations.
- Reward Function: Cooperative behaviors such as focus-fire are not explicitly rewarded but are intended to emerge from graph attention, dynamic role assignment, and target-priority supervision.This design aims to avoid complex reward shaping or reward over-engineering.
5 Methods
DRG-MAPPO combines graph-based relational encoding, hierarchical role assignment, and centralized training with decentralized execution for cooperative air combat. Its policies use graph-derived context, persistent tactical roles, auxiliary target prioritization, and joint PPO-based optimization.
- Framework overview: DRG-MAPPO integrates graph-based relational encoding, hierarchical role-conditioned policies, target-priority supervision, and CTDE into one framework.The graph encoder captures time-varying entity interactions; high-level roles condition low-level maneuvers, while centralized critics support training.
- Graph-Based Modeling: Graph attention operates over entity representations to capture pairwise relational importance and produce agent-, entity-, and team-level features.The attention matrix encodes how entities attend to one another, supporting multi-level relational reasoning.
- Hierarchical Policy: The high-level policy assigns discrete tactical roles, whose learned embeddings condition decentralized low-level action selection.Role embeddings provide richer downstream representations than one-hot encodings, while shared global context supports policy inputs.
- Hierarchical Policy: Roles are resampled only at fixed commitment intervals, enforcing behavioral consistency, enabling teammates to anticipate intentions, and stabilizing high-level training.The commitment horizon reduces the effective decision frequency of role assignment.
- Target-Priority Auxiliary Task: The auxiliary target-prioritization head predicts enemy targets from self, graph, and role features using heuristic labels and cross-entropy supervision.The heuristic incorporates distance, radar-lock status, and friendly missiles tracking each enemy.
- Joint Training Objective: The end-to-end objective combines PPO actor and role losses, centralized value estimation, auxiliary cross-entropy, and entropy bonuses, with role gradients restricted to resampling intervals.A distinct clipping coefficient accommodates the lower decision frequency of role assignments.
6 Experiments and Results
DRG-MAPPO is evaluated in a 2v2 BVR engagement through training curves, ablations, cross-play, and tactical-behavior visualization. It reaches 87% peak win rate, 69.9% average cross-play win rate, and exhibits bait-and-flank and focus-fire behaviors.
- 6.1 Simulation Setting: The 2v2 BVR environment uses a 200 km × 100 km theater, two aircraft per side, four missiles per aircraft, and an 800-second time limit.Episodes terminate after total team elimination, boundary breach, or the maximum time limit.
- 6.2 Main Results and Performance Analysis: 87% peak win rate after approximately 1250 episodes outperforms the baselines, with integrated relational modeling and hierarchical role assignment exceeding either component alone.All methods show a similar 0–800-episode cold-start bottleneck; DRG-MAPPO ultimately achieves the highest reported training win rate.
- 6.2 Main Results and Performance Analysis: Approximately 950 asymptotic average reward exceeds mid-tier methods plateauing between 500 and 650, while initial rewards are approximately −450.The curves surpass zero around episode 400 as basic engagement behaviors emerge.
- 6.3 Ablation Experiments: The ablation evaluation attributes performance and tactical efficiency to graph modeling, dynamic role assignment, target-priority auxiliary learning, and temporal commitment.The ablation win rates are presented in Fig. 4a.
- 6.4 Confrontation Experiments: 69.9% average win rate in round-robin cross-play consistently outperforms all baselines across 200-episode pairings.The authors attribute diagonal deviations from 0.5 to simulation stochasticity and interpret the result as generalization across adversarial behaviors.
- 6.5 Tactical Behavior Visualization: Visualization shows a Supporter drawing enemy fire for a Leader’s flank, followed by coordinated focus-fire and a Supporter transition from defense to offense.The tactical episode is shown in Fig. 5 while DRG-MAPPO controls the blue side against a rule-based red side.
7 Conclusion
The paper concludes that DRG-MAPPO addresses multi-UAV coordination by combining graph-based relational encoding with dynamic role assignment. Simulations report an 87% win rate and emergent bait-and-flank and focus-fire tactics, while future work targets larger-scale swarm confrontations.
- 7 Conclusion: DRG-MAPPO combines graph-based relational encoding and dynamic role assignment to separate high-level tactical intent from low-level maneuver execution.The framework is presented as a hierarchical MARL approach for coordination challenges in multi-UAV air combat.
- 7 Conclusion: 87% win rate and emergent bait-and-flank and focus-fire tactics are reported in high-fidelity BVR simulations.These results are presented as evidence of complex collaborative decision-making within the evaluated setting.
- 7 Conclusion: Future work will examine whether the hierarchical paradigm scales to larger-scale swarm confrontations.The conclusion identifies scalability as the next research direction.