PaperlayerView exact revision on arXiv

Source-linked AI summary

Threat-Aware Energy-Efficient Deployment for Dynamic UAV Networks: A Multi-Agent RL Approach

Faisal Al-Kamali, Hussein A. Ammar, Francois Chan, James H. Bayes, Yasser Gadallah, Mohamed H. Ahmed

arXiv:2609.30690v1cs.ITcs.AIcs.LGeess.SPmath.IT

TL;DR

Existing studies often omit threat avoidance, limiting their applicability in safety-critical scenarios. The paper combines threat-aware clustering, optimal matching, and MATD3; simulations report higher energy efficiency, faster convergence, zero safety violations, and performance above GPSO and near OPSO.

  • Problem

    Existing studies often do not incorporate threat avoidance as a fundamental design constraint, limiting their applicability in safety-critical scenarios involving threat zones.

    Source: B. Motivation and Novelty · p. 2

  • Method

    The framework uses TAKM to determine the minimum UAV count and safe initial positions, then applies matching and threat-aware MATD3 to optimize trajectories, power, and user associations.

    Source: §VII · p. 18

  • Results

    The framework achieved higher EE, faster convergence, and zero safety violations, surpassed GPSO, approached OPSO, and retained more than 92% of training performance on unseen user distributions.

    Source: §VII · p. 18

  • Takeaways & Limitations

    The authors conclude that integrating TAKM with MATD3 provides a practical and scalable solution for safe, energy-efficient UAV deployment in threat-constrained environments.

    Source: §VII · p. 18

Abstract

from arXiv · show

Ensuring operational safety in threat-prone environments remains a critical challenge for multi-UAV networks serving as aerial base stations. This paper proposes an efficient framework to maximize global energy efficiency (EE) while promoting safe operation through threat-aware clustering and reward-based safety enforcement. The proposed framework is executed in three steps. First, a threat-aware K-means (TAKM) algorithm determines the minimum required UAVs and computes safe initial placements. Second, an optimal matching stage assigns physical UAVs to these centroids to minimize energy expenditure. Third, a threat-aware multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm dynamically optimizes trajectories, power, and user associations. Simulation results show that the proposed framework achieves zero observed safety violations in the considered scenarios while achieving superior EE and faster convergence than other learning methods and non-clustering baselines. Compared to heuristic optimization, the proposed framework outperforms the greedy particle swarm optimization (GPSO) and achieves performance comparable to that of the optimized PSO (OPSO), while incurring significantly lower online deployment computational complexity. Furthermore, the proposed framework demonstrates effective generalization to unseen user distributions, large UAV fleets, and different threat geometries, while maintaining zero safety violations.

I. Introduction

UAV networks can extend connectivity, but single-UAV capacity and multi-UAV energy and deployment challenges motivate cooperative systems. The introduction identifies threat avoidance as a gap in existing work and presents a framework combining threat-aware clustering, matching, and multi-agent learning.

A. Background

  • Multi-UAV cooperation can serve users over larger areas and improve throughput, latency, and scalability, while limited battery capacity constrains sustained coverage.Single-UAV systems are limited in communication, caching, and computational capabilities.Source: A. Background · p. 1

B. Motivation and Novelty

  • Existing studies often omit threat avoidance, while conventional K-means can place UAVs inside threat zones and existing RL approaches may lack safety-aware initialization.The paper frames threat avoidance as a fundamental design constraint for safety-critical scenarios.Source: B. Motivation and Novelty · p. 2

C. Related Works

  • Prior UAV studies address trajectory, power, and resource optimization, but the proposed framework integrates threat-aware clustering, safe-centroid matching, and threat-aware MATD3.The related-work discussion describes prior MATD3 clustering work but identifies threat-prone environments as the distinct focus here.Source: C. Related Works · pp. 2–3
  • The framework supports non-uniform user distributions, optimizes the number of UAVs, and updates user associations dynamically, distinctions absent from the compared works.It also uses TAKM for safety during initialization and iterative updates.Source: C. Related Works · p. 3

D. Contribution

The paper’s contribution is a risk-aware UAV placement framework that combines safe clustering with learning-based control for dynamic user association. Its system model places aerial base stations among stationary, non-uniformly distributed users and requires UAVs to avoid threat zones.

D. Contribution

  • The framework combines safety- and QoS-constrained deployment with TAKM safe placement and MATD3 learning for trajectories and transmit power, while updating user associations deterministically.TAKM also determines the minimum feasible fleet size; the contribution describes safe intermediate and final placement points.Source: D. Contribution · p. 3

E. Organization and Notation

  • The paper’s organization assigns system modeling to Section II, problem formulation to Section III, deployment and matching to Section IV, and learning and evaluation to Sections V–VI.Section VII concludes and outlines future research directions.Source: E. Organization and Notation · p. 3

II. System Model

  • UAVs serve stationary, non-uniformly distributed ground users, including users inside threat zones, while the UAVs themselves must remain outside those zones.The model treats UAVs as aerial base stations over a fixed time horizon.Source: §II · p. 4
  • The system models probabilistic LoS/NLoS links and user rates, with UAV power consumption accounting for transmission and flight.Each UAV uses one power level shared equally among its associated users, and flight power depends on speed.Source: §II · pp. 4–5

III. Problem Formulation

The problem formulation maximizes global energy efficiency while jointly selecting the UAV count, associations, trajectories, and transmit powers under coverage, QoS, mobility, and safety requirements. The resulting mixed-integer nonconvex problem is decomposed into fleet sizing and safe initialization, UAV matching, and threat-aware learning-based control.

III. Problem Formulation

  • The objective maximizes global energy efficiency, defined as total user sum rate divided by total UAV energy consumption, while jointly optimizing UAV count, associations, trajectories, and transmit power.The formulation requires full coverage and minimum user rates, collision avoidance, threat-zone avoidance with a safety margin, and velocity and transmit-power limits.Source: §III · pp. 5–6
  • The formulation is NP-hard because it combines binary associations with continuous trajectories and power, a fractional nonconvex objective, and nonconvex safety and rate constraints.Source: §III · p. 6
  • The solution decomposes into determining fleet size and safe TAKM initialization, matching UAVs to clusters by minimizing flight distance, and MARL optimization of associations, trajectories, and power.Pre-placement narrows the learning search space and accelerates convergence.Source: §III · p. 6

A. Threat-Avoidance K-means Clustering Algorithm

TAKM modifies K-means to keep centroids in safe regions, while a constraint-aware search selects the minimum UAV count that satisfies user-rate and safety requirements.

A. Threat-Avoidance K-means Clustering Algorithm

  • TAKM initializes centroids at safe positions and projects unsafe cluster means to the nearest safe position, preserving feasibility while minimizing deviation from the cluster mean.The method states that it maintains CKM’s convergence properties while keeping intermediate and final solutions safe.Source: A. Threat-Avoidance K-means Clustering… · p. 6

B. Joint Determination of the Minimum Number of UAVs and Initial User Association

  • Unlike approaches that assume a fixed fleet size, the procedure starts at M = 1 and checks coverage, minimum rate, and threat avoidance as it increases M.The paper frames selecting the minimum number of UAVs as important for cost-effective deployment.Source: B. Joint Determination of the Minimum… · pp. 6–7
  • Algorithm 2 increments M until every user meets Rmin while maintaining safety, returning the minimum UAV count, safe centroids, and initial user associations.Candidate clusterings are checked using per-user transmit power pm = Pmax/Nm and are valid only if every user’s rate is at least Rmin.Source: B. Joint Determination of the Minimum… · p. 7

C. UAV Matching

The framework matches physical UAVs to safe centroids by minimizing total movement, then uses a MARL formulation to address joint trajectory, power, and association optimization.

C. UAV Matching

  • The Hungarian algorithm finds a one-to-one UAV-to-centroid assignment that minimizes global movement cost in polynomial time.The assignment minimizes cumulative Euclidean distance from UAV launch positions to designated centroids to support energy-efficient initial deployment.Source: C. UAV Matching · p. 7

V. Threat-Aware MARL for user association, trajectory design and power control

  • After the required UAV count is fixed, the learning-stage objective maximizes global energy efficiency by optimizing user associations, trajectories, and power allocation.The resulting problem is nonlinear, nonconvex, and mixes discrete association variables with continuous control variables.Source: §V · p. 7

A. Markov Decision Process Formulation

  • The formulation models sequential UAV decisions as a POMDP because movement affects user rates and transmit power in current and future states.The policy is selected to maximize cumulative reward, which combines energy efficiency, coverage, and a safety penalty.Source: A. Markov Decision Process Formulation · pp. 7–8
  • Each agent observes only its own horizontal coordinate, while the reward penalizes collisions, threat-zone entry, and departures from operational boundaries.The safety flag is binary, and the safety weight is set higher to strongly penalize violations.Source: A. Markov Decision Process Formulation · p. 8

B. Multi-Agent RL Method

The method combines continuous-action MATD3 with deterministic user association, centralized training, and decentralized UAV execution.

B. Multi-Agent RL Method

  • MATD3 is selected for continuous state-action control because its double critics and delayed policy updates mitigate Q-value overestimation.The paper contrasts this with tabular methods, which it says are unsuitable for the continuous variables in this task.Source: B. Multi-Agent RL Method · p. 9
  • User associations are recomputed deterministically from UAV positions, transmit powers, and channel conditions rather than emitted as agent actions.Agents learn association effects indirectly through the global reward; replay memory reuses joint transitions to improve sample efficiency.Source: B. Multi-Agent RL Method · p. 9, 1) Decentralized Execution · p. 10
  • CTDE pairs local-coordinate observations at execution with centralized critics that use global state, including threat boundaries and neighbor positions, during training.The binary safety penalty shapes decentralized policies, while local observations reduce real-time coordination overhead in the fixed scenario with stationary, known users and threats.Source: B. Multi-Agent RL Method · pp. 9–10
  • During interaction, each UAV actor outputs velocity, flight angle, and transmit power, and the environment updates associations before returning the shared reward and next state.The resulting joint experience is stored in replay memory for training.Source: 1) Decentralized Execution · p. 10

1) Decentralized Execution:

MATD3 uses centralized critics to evaluate joint UAV behavior and two critics to reduce overestimation bias.

1) Decentralized Execution:

  • A centralized critic evaluates the joint state-action pair during training, capturing dependencies among UAV agents such as collision avoidance.The method uses two critics to reduce overestimation bias.Source: 2) Centralized Training · p. 10
  • MATD3 uses two critics to reduce overestimation bias when estimating the target Q-value.The passage introduces the target Q-value computation after describing centralized evaluation.Source: 2) Centralized Training · p. 10
  • The centralized critic is used during training to account for interagent dependencies, including collision avoidance.This evaluation is based on the joint state-action pair.Source: 2) Centralized Training · p. 10

2) Centralized Training:

The centralized MATD3 training process uses twin critics, delayed actor updates, and target-network smoothing to stabilize learning. It also uses uniform replay sampling, relying on TAKM initialization to quickly populate the buffer with high-quality transitions.

2) Centralized Training:

  • Soft updates are applied to actor and critic target networks using τ as the soft-update factor.The passage specifies τ ≪ 1.Source: 2) Centralized Training · p. 10
  • Uniform mini-batch replay is used, with TAKM initialization quickly filling the buffer with high-quality transitions and reducing the need for prioritization.Prioritized experience replay is identified as a possible direction for more complex environments.Source: 2) Centralized Training · p. 10
  • Twin centralized critics form target Q-values using the lower critic output, mitigating the overestimation bias associated with MADDPG's single-critic design.MADDPG uses one centralized critic per agent, whereas MATD3 shares two centralized critics among agents.Source: 2) Centralized Training · p. 10
  • Delayed actor updates, double critics, and target-policy smoothing stabilize training while agents learn cooperative strategies that discourage safety violations through reward penalties.The actor updates less frequently than the critics, keeping target values reliable for multi-UAV optimization.Source: 2) Centralized Training · p. 10

C. Meta-heuristic Solution

The paper benchmarks its learning-based framework against PSO variants and compares offline training costs with online execution complexity. MATD3 has substantial one-time training cost, but its actor-only online execution is lightweight and can adapt to deployment conditions that fixed offline PSO plans cannot handle.

C. Meta-heuristic Solution

  • GPSO optimizes UAV trajectories and user associations at fixed maximum power, whereas OPSO jointly optimizes transmit power, trajectories, and user associations.PSO is included solely as a comparison baseline, not as part of the proposed framework.Source: C. Meta-heuristic Solution · pp. 10–11

D. Computational Complexity and Scalability

  • Once training is complete, MATD3 discards critic and target networks and uses actor forward passes, making online complexity independent of the number of GUs.The online computation also avoids iteratively calculating the user-association matrix during flight.Source: D. Computational Complexity and… · p. 11
  • ≈4.7 × 10^5 operations for total swarm online execution per time step versus approximately 1.6 × 10^10 operations for PSO to solve the same problem.MATD3 requires approximately 6.4 × 10^10 operations for one-time offline training; the online figure uses M = 4 UAVs, K = 400 users, and tmax = 100 time steps.Source: D. Computational Complexity and… · p. 11, Table III · p. 11
  • MADDPG requires approximately 5.8 × 10^10 offline operations versus 6.4 × 10^10 for MATD3, while both have identical online complexity.The offline difference is attributed to MADDPG using one critic rather than MATD3's twin-critic architecture.Source: D. Computational Complexity and… · p. 11
  • MATD3 can adjust actions to wind deviations and the actual active-user set, whereas offline PSO plans cannot revise fixed trajectories and powers after deployment.A fixed plan may direct UAVs into threat regions after deviations or waste energy flying toward inactive users.Source: D. Computational Complexity and… · p. 11
  • MATD3's centralized critic increases training complexity as UAV count grows and may slow convergence for very large swarms.During decentralized execution, each actor's execution time remains independent of total swarm size because it uses local observations.Source: D. Computational Complexity and… · pp. 11–12

VI. Simulation Results

In the simulated urban area, TAKM's threat-aware centroid updates preserve safe UAV placements while retaining clustering balance. CKM can place a centroid inside a threat zone, whereas TAKM moves it outside the hazardous region with only slight changes in user assignments.

VI. Simulation Results

  • The simulation places M UAVs in an urban 4000 × 4000 m² area with two threat zones of radii 500 m and 950 m, partly overlapping high-density user areas.All UAVs operate at fixed altitude and maintain a safety margin around each threat boundary.Source: §VI · p. 12

A. The Comparison of CKM and TAKM Clustering Algorithms

  • Algorithm 2 selects four UAVs as the minimum required to cover the considered area under a minimum data rate of Rmin = 0.128 Mbps.The experiments use M = 4 for the remaining evaluations.Source: A. The Comparison of CKM and TAKM… · p. 12
  • TAKM incorporates threat avoidance during centroid updates, maintaining safety while preserving CKM's convergence properties and computational efficiency.CKM minimizes within-cluster variance without accounting for constraints, which can lead to unsafe placements.Source: A. The Comparison of CKM and TAKM… · p. 12
  • TAKM places Cluster 3 510 m from Threat 1, outside its 500 m radius, while CKM places it 151.31 m away inside the threat zone.TAKM changes Cluster 2 from 200 to 198 GUs and Cluster 3 from 52 to 55, maintaining overall cluster balance.Source: A. The Comparison of CKM and TAKM… · p. 12, Figure 3 · p. 12, Figure 4 · p. 12
  • The matching of physical UAVs to TAKM centroids is assumed to use the Hungarian algorithm and is excluded from simulation because it incurs a fixed one-time energy cost.Experiments focus on clustering and learning stages.Source: A. The Comparison of CKM and TAKM… · p. 13

B. Performance of the Proposed MATD3 with TAKM Clustering and Pre-placement Scheme

The evaluation finds that threat-aware initialization supports high energy efficiency with fewer safety violations than non-TAKM alternatives. Additional tests examine altitude, network scale, user associations, and robustness.

B. Performance of the Proposed MATD3 with TAKM Clustering and Pre-placement Scheme

  • Removing the energy-efficiency reward term raises average transmit power from 73 W to 80 W, while removing the coverage term lowers served-user coverage to 98%.Across safety weights 3, 5, 10, and 15, agents maintain required safety distances with zero threat-zone entries; restricted observations also retain zero safety violations and 100% coverage.Source: B. Performance of the Proposed MATD3… · pp. 13, 18
  • The proposed scheme achieves the highest final EE among evaluated learning schemes, reaching 4.48 ± 0.25 × 10^6 b/J, with faster convergence and stable learning.It is competitive with MADDPG using TAKM initialization; MATD3 with CKM clustering reaches 4.45 × 10^6 b/J but has persistent safety violations.Source: B. Performance of the Proposed MATD3… · pp. 13–14, Figure 6 · p. 14
  • With TAKM initialization, MATD3 approaches nearly zero safety violations after roughly 300 episodes; pre-placement alone does not eliminate unsafe behavior by episode 400.CKM initialization retains about 70% safety violations after 400 episodes, while rate-violation percentages across schemes converge toward zero after approximately 150–200 episodes.Source: B. Performance of the Proposed MATD3… · pp. 13–14, Figure 7 · p. 15
  • At 300 m, UAVs serve all 400 users and achieve the highest average reward among the tested altitudes of 100–300 m.Other tested altitudes do not fully cover all users; 300 m is selected for the proposed scheme.Source: B. Performance of the Proposed MATD3… · pp. 14–15, Figure 9 · p. 15
  • With four UAVs, coverage falls to 95% at 600 users, while nine UAVs serve 1,500 users with 100% coverage and zero safety and rate violations.For 600 users, increasing the fleet to five UAVs restores full coverage and increases average reward.Source: B. Performance of the Proposed MATD3… · p. 16
  • MATD3 dynamically changes user groupings along learned trajectories, reducing cluster SDNU from 57.40 with TAKM to 48.66 with MATD3.At final positions, the UAVs serve 64, 183, 67, and 86 users, with average rates of 1.31, 0.45, 1.06, and 0.96 Mbps, respectively.Source: Figure 12 · p. 16, B. Performance of the Proposed MATD3… · pp. 16–17
  • MATD3 achieves higher EE than GPSO and competitive performance with OPSO, while online execution requires only one neural-network forward pass.The PSO comparison uses fixed benchmark parameters selected before the final performance comparison.Source: Figure 14 · p. 17, B. Performance of the Proposed MATD3… · p. 17
  • On unseen user distributions, testing retains over 92% of training performance; under a new threat layout, the framework achieves 4.13 × 10^6 b/J, 100% coverage, and zero safety violations.For the new threat layout, the threat zones are centered at (1000, 2200) m and (2800, 1500) m, with radii of 800 m and 600 m.Source: B. Performance of the Proposed MATD3… · pp. 17–18

VII. Conclusion

The paper combines threat-aware clustering and safe placement with MATD3 to optimize UAV trajectories, power allocation, and user associations. Simulations report higher EE, faster convergence, and zero safety violations, while real flight effects remain untested.

VII. Conclusion

  • The framework uses TAKM to enforce safe clustering and determine UAV count and initial positions, then MATD3 jointly optimizes trajectories, transmit power, and user associations.An optimal matching stage links the safe configuration to physical UAVs.Source: §VII · p. 18
  • Simulations report higher EE, faster convergence, and zero safety violations; the approach outperforms GPSO, approaches OPSO, and retains over 92% of training performance on unseen user distributions.The paper also reports zero violations under different threat configurations and lower online deployment complexity than heuristic optimization.Source: §VII · p. 18
  • The framework was validated in simulation without real flight effects such as wind, GPS drift, or latency; hardware validation under real-world disturbances remains future work.Source: §VII · pp. 18–19

Related papers

Loading 2609.30690v1…