Cooperative Multi-Agent Deep Reinforcement Learning for Reliable Surveillance via Autonomous Multi-UAV Control

Won Joon Yun; Soohyun Park; Joongheon Kim; MyungJae Shin; Soyi Jung; David A. Mohaisen; Jae-Hyun Kim

Source-linked AI summary

Cooperative Multi-Agent Deep Reinforcement Learning for Reliable Surveillance via Autonomous Multi-UAV Control

Won Joon Yun, Soohyun Park, Joongheon Kim, MyungJae Shin, Soyi Jung, David A. Mohaisen, Jae-Hyun Kim

arXiv:2201.05843v1Jan 15, 2022eess.SYcs.AIcs.LGcs.RO

TL;DR

UAV surveillance must remain reliable and energy-efficient despite uncertain user movements, neighboring-UAV behavior, malfunctions, and incomplete communication. The paper proposes a cooperative model-free MADRL scheme called CommNet, which learns autonomous trajectories and coverage control. Simulations report superiority over comparison schemes in surveillance and computational performance, including specific computational-cost trade-offs.

Problem
Reliable UAV surveillance requires joint energy and reliability management under changing users, UAV failures, and highly dynamic distributed networks where centralized and non-communicative approaches are limited.
Method
The paper proposes cooperative model-free MADRL based on CommNet, using communication among UAV agents to optimize trajectories, surveillance coverage, energy consumption, and cooperation.
Results
The proposed scheme outperforms comparison schemes in surveillance coverage, user support capability, and computational efficiency; its cost is 16.1% lower than Comp1 and 5.3% higher than Comp2 per time step.
Takeaways & Limitations
CommNet-based autonomous coordination provides a promising solution for reliable, robust, energy-efficient surveillance coverage control across distributed UAVs.

Abstract

from arXiv · show

CCTV-based surveillance using unmanned aerial vehicles (UAVs) is considered a key technology for security in smart city environments. This paper creates a case where the UAVs with CCTV-cameras fly over the city area for flexible and reliable surveillance services. UAVs should be deployed to cover a large area while minimize overlapping and shadow areas for a reliable surveillance system. However, the operation of UAVs is subject to high uncertainty, necessitating autonomous recovery systems. This work develops a multi-agent deep reinforcement learning-based management scheme for reliable industry surveillance in smart city applications. The core idea this paper employs is autonomously replenishing the UAV's deficient network requirements with communications. Via intensive simulations, our proposed algorithm outperforms the state-of-the-art algorithms in terms of surveillance coverage, user support capability, and computational costs.

I. INTRODUCTION

The paper addresses reliable surveillance with distributed UAVs under uncertain dynamics, energy constraints, malfunction risks, and incomplete inter-UAV information. It proposes a CommNet-based MADRL scheme for autonomous cooperation, trajectory and coverage optimization, and evaluates surveillance performance and computational efficiency.

Motivation: UAV surveillance must jointly optimize energy consumption and reliability while target movements, user deployments, and neighboring-UAV behavior remain uncertain.The paper highlights battery depletion, collisions, malfunctions, and changing user demands as operational concerns.
Motivation: Centralized optimization approaches cannot provide online computational solutions for highly dynamic, distributed UAV-enabled networks.The study motivates distributed machine-learning computation over multiple UAVs and users.
Research gap: Partially observable multi-agent settings make conventional machine-learning approaches unsuitable when UAVs have different information and cannot fully communicate.Prior approaches also focus on communication systems despite available visual information.
Research gap: The authors identify autonomous cooperative UAV surveillance under incomplete information exchange as an insufficiently studied MADRL problem.They position the proposed scheme as guidance for future dynamic UAV-based autonomous surveillance studies.
Proposed approach: The proposed MADRL algorithm enables distributed UAV cooperation, estimates environmental uncertainty, and optimizes energy consumption, surveillance reliability, and operation.The scheme is based on the communication neural network, or CommNet.
Evaluation: Experiments examine the relationship between captured surveillance-data resolution and the area covered by the corresponding UAV.The paper evaluates reliable and robust cooperative management of distributed UAVs.

II. DEEP REINFORCEMENT LEARNING

This section formalizes reinforcement learning through Markov decision processes and extends the formulation to multi-agent systems. It describes trajectories, expected return, policies, and joint multi-agent actions.

MDP formulation: An MDP is defined by states, actions, transition probabilities, rewards, and a finite time horizon.The tuple is (S, A, P, R, T).
MDP formulation: A policy πθ maps states to action probabilities and is optimized to maximize discounted cumulative rewards over the finite horizon.The parameter θ represents neural-network parameters such as weights and biases.
Trajectory and return: The probability of a stochastic trajectory is formed from the starting-state distribution, transition probabilities, and policy action probabilities.The trajectory contains a sequence of states and actions.
Trajectory and return: The expected return J(πθ) is the trajectory-reward expectation, and reinforcement learning seeks a policy that maximizes it.The objective is defined over trajectories sampled according to the policy.
Multi-agent extension: A multi-agent MDP generalizes the MDP by using Cartesian-product state spaces and joint actions across agents.This formulation represents actions executable collectively by the agents.

B. Deep reinforcement learning

The section reviews DQN and PPO before motivating MADRL for UAV coordination. It emphasizes stabilization techniques, policy-update control, and the non-stationarity created by other agents’ behavior.

Deep reinforcement learning: DQN approximates the Q-function with a neural network for model-free reinforcement learning in high-dimensional state spaces.Experience replay and a target network stabilize optimization.
Deep reinforcement learning: DQN stores transition experiences in a replay buffer and periodically resamples minibatches to update policy parameters.The target network parameters are updated from the Q-network at predefined intervals.
Deep reinforcement learning: PPO improves training stability by clipping the probability ratio so each policy update remains close to the previous policy.The allowed update distance is controlled by the hyperparameter ϵ < 1.
MADRL motivation: Single-agent methods do not directly account for other agents’ concurrent behavior in multi-agent environments.This changes the environment continuously and creates a non-stationary problem whose predecessor convergence guarantees generally do not hold.
System model: The proposed system models users, cooperative UAV agents, and non-agent UAVs, with a leader UAV handling communication among agents.The leader receives information for multi-agent cooperation.
System model: The surveillance model assumes camera sensors are unaffected by zooming or UAV movement, so higher surveillance resolution corresponds inversely to surveillance area.This assumption is justified through dynamic camera-pixel control.
System objective: The ACR objective is reliable, high-resolution monitoring despite changing users and UAVs being dropped, malfunctioning, or energy-exhausted.The approach also seeks to optimize energy use and surveillance coverage.

B. MADRL formulation

The MADRL formulation defines UAV state information, energy accounting, surveillance coverage, and discrete movement and resolution-control actions. These elements encode coordination under positional, energetic, and monitoring conditions.

B. MADRL formulation: The ACR formulation requires explicit state-space, action-space, and reward definitions for the MADRL problem.
1) State space:: Each UAV state combines location, energy, and surveillance information, including monitored users and assigned resolutions.The state is represented as sm = {pm, em, cm}.
1) State space:: Location information includes absolute positions and relative positions or distances between each agent, users, and non-agent UAVs.These variables describe the UAV’s spatial relationships with relevant entities.
1) State space:: Each UAV consumes basic operational energy for aviation and monitoring, with energy depending on surveillance coverage range.The formulation separately describes aviation-related and monitoring-related consumption.
1) State space:: Equation (4) models flight energy using minimum flight power, motor-speed effects, operating time, and lifting power; equation (5) models monitoring energy from coverage range and a coefficient.The prose identifies δ, ζ, v, t, P(h/v), RCm, and ρRC as the relevant quantities.
1) State space:: Surveillance resolution is represented by q(bm), while each UAV’s surveillance coverage is defined as a positional set over the surveillance field.An allocation variable determines whether a UAV surveils a user.
1) State space:: The surveillance indicator records whether users are monitored and which UAV monitors them at which resolution.This information forms the surveillance component of each UAV state.

2) Action space:

The ACR action space combines UAV movement with surveillance-resolution control, while its reward design balances energy use, surveillance coverage, cooperation, and total user utilization.

Action space: The ACR action space contains 10 discrete actions: eight movement actions and two surveillance-resolution controls.Movement changes position along x, y, or both axes; resolution actions adjust the surveillance level by ±1.
Reward design: UAV rewards account for energy consumption, battery discharge, and the number of users served.Energy and aviation status use separate scaling factors in the reward formulation.
Reward design: The surveillance reward is the ratio of users currently covered to users covered at maximum surveillance coverage.Increasing this reward corresponds to higher surveillance resolution and more monitored users.
Reward design: Cooperation rewards target reduced overlap among UAV coverage areas because overlap lowers energy and resource efficiency.The formulation introduces an overlapped-area threshold, ωth, as a reward constraint.
Reward design: The total-utilization reward represents the number of users receiving service relative to the total number of users.Its contribution is scaled by ρu before inclusion in each agent’s total reward.

C. Algorithm for learning cooperation

The proposed ACR uses CommNet-based multi-agent reinforcement learning so UAVs can exchange information and learn autonomous coordination policies under uncertainty.

Algorithm for learning cooperation: ACR agents exchange information about users and other UAVs to maintain surveillance under environmental uncertainty.CommNet is applied as the communication-based multi-agent deep reinforcement learning architecture.
Algorithm for learning cooperation: Each CommNet agent uses its observable state, while a shared deep neural network enables communication among multiple agents.The network calculates communication and hidden-state representations before producing action probabilities.
Algorithm for learning cooperation: A randomly selected leader UAV gathers embedded state messages from other agents, averages them, and forwards the result to the next network layer.Each non-leader sends its embedded state information to the leader.
Algorithm for learning cooperation: The softmax output represents the probabilities of discrete actions when a UAV agent observes its state.The method adopts actor and critic reinforcement learning networks based on CommNet.
Algorithm for learning cooperation: Training initializes actor, critic, and target networks, collects transitions in a replay buffer, and updates network parameters during learning.Actor and critic parameters are shared across UAV agents, giving them the same cooperation policy.

A. Performance metric

The evaluation studies convergence in a dynamic environment and compares CommNet-based ACR with communication and disconnection benchmarks.

A. Performance metric: The dynamic environment continuously changes the number of monitored users and transfers the latest UAV locations.Episodes begin with randomly distributed agent UAVs and randomly assigned areas of interest.
A. Performance metric: The common objective is to provide the best surveillance service for each assigned area.The proposed ACR is evaluated against state-of-the-art benchmark techniques.
A. Performance metric: Comp1 provides agents with others’ observations and uses a CommNet-based policy for action decisions.The comparison considers both performance and computational cost.
A. Performance metric: Comp2 removes UAV-to-UAV communication, creating a non-communicative neural-network structure with reduced cooperation.The benchmark is used to contrast communication-enabled and disconnected coordination.

B. Simulation setup

The simulation uses a 2,400 × 2,400 m² grid with 25 randomly distributed users and four cooperative UAV agents, evaluated over 40-step episodes.

B. Simulation setup: The experimental map is a 2-dimensional 2,400 × 2,400 m² grid containing N = 25 randomly distributed users.The setup models a multi-agent UAV system for numerical performance analysis.
B. Simulation setup: The multi-agent system contains M = 4 UAV agents and runs each episode for 40 time steps.Agent UAVs initially occupy the grid center, while non-agent UAVs begin 750 m from the center.

C. Evaluation results

The evaluation examines reward convergence, surveillance performance, and trained UAV behaviors across the proposed scheme, Comp1, and Comp2. Results indicate more stable training and strong surveillance performance for the proposed approach, with distinct cooperative behaviors under UAV malfunctions.

Reward convergence: The proposed scheme and Comp1 converge to total rewards around [540, 580], while Comp2 fluctuates within [300, 580].The proposed scheme and Comp1 show more stable reward and loss tendencies than Comp2.
Reward convergence: The proposed scheme’s training loss starts at 1.7 and converges at 1.2, while Comp2 shows the highest loss tendency.The proposed scheme begins with the highest loss but converges to the second-ranked level; Comp1 has the minimal loss tendency.
Surveillance reliability: The proposed scheme surveils 12–15 users on average, while Comp1 surveils 12–14 users across 25 test iterations.The proposed scheme’s b1 and b3 agents surveil more users than other agents or the three non-agent UAVs.
Surveillance reliability: Surveillance resolution ranks proposed scheme, Comp1, then Comp2, while support rate is highest for the proposed scheme at t = 1.At t = 40, Comp1 matches the proposed scheme and Comp2 is lowest in support rate.
Behavior pattern analysis of UAV agents: In the proposed scheme, the CommNet UAV occupies the position with most users while DNN UAVs cover remaining users and enhance resolution during non-agent UAV malfunctions.The proposed scheme combines one CommNet UAV with three DNN UAVs; non-agent UAVs malfunction during t ∈(5, 10], t ∈(25, 30], and t ∈(35, 40].

4) Computational cost and efficiency:

The proposed scheme balances computational cost with cooperative surveillance performance by combining CommNet-based and DNN-based policies across its application schemes. It uses fewer FLOPS than Comp1 while retaining high performance and outperforming Comp2.

22.7% larger computational cost characterizes the CommNet-based policy compared with the DNN-based policy.The proposed scheme uses CommNet-based policies for the leader and Comp1, and DNN-based policies for Comp2.
16.1% fewer FLOPS than Comp1 are required by the proposed scheme at each time step.The proposed scheme requires 5.3% more FLOPS than Comp2.
Stable convergence and high performance comparable to Comp1 are achieved despite Comp1 having higher computational cost.The comparison concerns the proposed scheme, Comp1, and Comp2.
The proposed scheme shows performance superiority to Comp2 despite requiring higher computational cost.The paper attributes cooperation in the proposed scheme and Comp1 to the cooperative reward, whereas Comp2 lacks communication between UAVs.
The proposed CommNet-based management scheme is presented as a computationally efficient approach for reliable, energy-efficient surveillance UAV deployment.Its intended outputs include optimal trajectories and surveillance coverage control that can cover as many users as possible.