PaperlayerView exact revision on arXiv

Source-linked AI summary

Cooperative Internet of UAVs: Distributed Trajectory Design by Multi-agent Deep Reinforcement Learning

Jingzhi Hu, Hongliang Zhang, Lingyang Song, Robert Schober, H. Vincent Poor

arXiv:2007.14297v1eess.SPeess.SY

TL;DR

Cooperative UAV sensing and transmission require challenging distributed trajectory design to minimize AoI. The paper introduces a distributed sense-and-send protocol, formulates the problem as an MDP, and solves it with CA2C deep reinforcement learning. Simulations report that CA2C outperforms four benchmark algorithms and that cooperation lowers AoI relative to non-cooperation.

  • Problem

    Distributed trajectory design is challenging because centralized design has high computational complexity and cooperative UAV trajectories influence one another.

  • Method

    The paper proposes a distributed sense-and-send protocol, formulates trajectory design as an AoI-minimizing MDP, and develops the CA2C algorithm based on DQN and DDPG.

  • Results

    CA2C outperforms four benchmark algorithms, while cooperative sensing and transmission reduce AoI compared with a non-cooperative approach.

  • Takeaways & Limitations

    Cooperative UAVs achieve lower AoI than non-cooperative UAVs, with the reduction increasing as the number of UAVs grows.

Abstract

from arXiv · show

Due to the advantages of flexible deployment and extensive coverage, unmanned aerial vehicles (UAVs) have great potential for sensing applications in the next generation of cellular networks, which will give rise to a cellular Internet of UAVs. In this paper, we consider a cellular Internet of UAVs, where the UAVs execute sensing tasks through cooperative sensing and transmission to minimize the age of information (AoI). However, the cooperative sensing and transmission is tightly coupled with the UAVs' trajectories, which makes the trajectory design challenging. To tackle this challenge, we propose a distributed sense-and-send protocol, where the UAVs determine the trajectories by selecting from a discrete set of tasks and a continuous set of locations for sensing and transmission. Based on this protocol, we formulate the trajectory design problem for AoI minimization and propose a compound-action actor-critic (CA2C) algorithm to solve it based on deep reinforcement learning. The CA2C algorithm can learn the optimal policies for actions involving both continuous and discrete variables and is suited for the trajectory design. {Our simulation results show that the CA2C algorithm outperforms four baseline algorithms}. Also, we show that by dividing the tasks, cooperative UAVs can achieve a lower AoI compared to non-cooperative UAVs.

I. INTRODUCTION

The paper addresses distributed trajectory design for cooperative UAV sensing and transmission while minimizing accumulated AoI. It proposes a distributed protocol and CA2C deep reinforcement learning algorithm, with simulations showing advantages over benchmark methods and non-cooperative UAVs.

  • I. INTRODUCTION: Distributed design is needed because centralized deployment has high computational complexity, while cooperative UAV trajectories influence one another and jointly affect sensing success and transmission rate.
  • I. INTRODUCTION: The paper proposes a distributed sense-and-send protocol coordinating UAVs for multiple sensing tasks in the cellular Internet of UAVs.
  • I. INTRODUCTION: The trajectory problem is formulated as an MDP minimizing accumulated AoI, with UAV policies learned from previous design experiences using deep reinforcement learning.
  • I. INTRODUCTION: The work targets continuous sensing and transmission for AoI minimization, a setting not addressed by prior cooperative or centralized cellular Internet of UAV studies.
  • I. INTRODUCTION: CA2C outperforms DDPG, DQN, and two conventional baseline algorithms in simulation for cooperative UAV trajectory design.
  • I. INTRODUCTION: Cooperative sensing and transmission reduce AoI compared with a non-cooperative approach, and the reduction increases with the number of UAVs.

II. SYSTEM MODEL

The system models a single-cell cellular Internet of UAVs in which multiple UAVs cooperatively sense and transmit task results to keep information fresh. Sensing success depends on target coverage and location, while UAVs cannot locally determine validity and rely on the BS for verification.

  • System setting: A single-cell system contains a BS, M UAVs, K uplink subcarriers, and N sensing tasks, with K>M and UAVs operating cooperatively within circular cell coverage.UAVs are assumed to share altitude and use collision-avoidance mechanisms outside the paper’s scope.
  • Cooperative execution: Each task targets a sensing location, and UAVs execute tasks through sensing followed by transmission of the sensing results to the BS.Cooperation dynamically divides tasks into sets, with each UAV executing one set; task assignment affects trajectory design.
  • UAV sensing: Sensing is not always successful because of range and precision limitations, so UAVs transmit results to the BS, which determines whether they are valid.A valid result is one produced by successful sensing; each sensing result contains Ds bits.
  • UAV sensing: Sensing succeeds with nonzero probability only when the target lies within the UAV’s sensing range and maximum sensing angle, and probability decreases with sensing distance under the adopted model.The sensing range is modeled as a ground circle of radius rs = h · tan φ, and λ depends on sensor quality.

B. UAV Transmission

The transmission and freshness model combines orthogonal uplink communication with an AoI metric that resets after successful task execution and otherwise grows over time. The sense-and-send protocol organizes UAV operation into recurring cycles with BS-mediated state exchange.

  • B. UAV Transmission: UAVs transmit sensing results over orthogonal uplink channels, with received SNR and data rate determined by transmit power, pathloss, noise, bandwidth, and allocated subcarriers.The channel model averages line-of-sight and non-line-of-sight pathlosses using the LoS probability.
  • B. UAV Transmission: The transmitted data amount is D(ki, xi) = ki · W · log2(1 + γ(xi)) · (tc −te), linking allocated subcarriers and location to transmission capacity.W is the bandwidth of one subcarrier; te is the information-exchange duration and tc is the cycle duration.
  • B. UAV Transmission: AoI for Task j is measured from its last successful execution, becomes zero after successful execution, and otherwise increases with time.Successful execution requires successful sensing and completed transmission of the sensing result.
  • B. UAV Transmission: The protocol uses four cycle types—decision, empty, sensing, and transmission—with common information exchange between UAVs and the BS at every cycle.The BS broadcasts all UAV states and each task’s AoI after receiving UAV reports.
  • B. UAV Transmission: After a decision cycle, a UAV moves toward its selected sensing location, senses after arrival, and transmits in subsequent cycles until the BS receives a valid result.The number of empty cycles depends on travel distance, while transmission duration depends on generated data and transmission rate.

1) Decision Cycle:

The protocol makes each UAV repeatedly select a task and sensing location, move there, sense, and transmit until the BS validates a result. Failed validation triggers another sensing cycle, while multiple UAVs can collectively complete a task when any valid result arrives.

  • 1) Decision Cycle:: During a decision cycle, a UAV selects its next task and sensing location, then remains at its current location until the decision process finishes.The distributed decision mechanism is introduced later in the paper.
  • 1) Decision Cycle:: During empty cycles, the UAV moves directly toward its selected sensing location at maximum speed until it arrives.The trajectory accounts for whether the UAV can reach the sensing location within the current cycle.
  • 1) Decision Cycle:: A sensing cycle collects Ds bits at the sensing location, after which the UAV transmits the result because it cannot determine validity locally.The sensing process is assumed to finish within one cycle, with the UAV hovering during sensing.
  • 1) Decision Cycle:: During transmission, UAVs keep their locations unchanged until sensing results are successfully transmitted to the BS.Otherwise, an invalid result could require flying back and sensing the target again.
  • 1) Decision Cycle:: If the BS declares a result invalid, the UAV immediately begins another sensing cycle and retransmits new data until a valid result is received.When multiple UAVs execute the same task, the task is completed once any valid result reaches the BS.
  • 1) Decision Cycle:: The paper formulates the cellular Internet of UAVs as a distributed AoI-minimization trajectory-design problem.The formulation treats the environment as an MDP and ultimately expresses trajectory design as distributed optimization.

A. Markov Decision Process Formulation

The trajectory-design problem is represented as a multi-agent MDP whose cycle-based states, compound actions, transitions, and rewards capture UAV motion, sensing, transmission, and task freshness. Actions combine discrete task selection with continuous sensing-location selection under distributed execution constraints.

  • A. Markov Decision Process Formulation: The MDP is appropriate because UAVs make sequential trajectory decisions whose effects propagate to future states and task AoI.State transitions depend on the previous state and the actions taken by the UAVs.
  • A. Markov Decision Process Formulation: The MDP uses the tuple (S, A, R, T), with the cellular Internet of UAVs and sensing tasks as the environment and the M UAVs as agents.A cycle is the basic time step, and each UAV selects an action from the available action set.
  • 3) State Transition Function:: The transition reward represents the AoI reduction caused by task executions during the cycle.The paper interprets this reduction as the sum of AoI reductions for Task j and illustrates it in Fig. 5.
  • 1) State:: Each state captures cycle-level system information, including UAV locations, pending sensing data, task AoI, selected tasks, sensing locations, and cycle types.UAVs obtain environmental state information through exchanges with the BS, and AoI is discretized in cycle durations.
  • 2) Action:: Each UAV action combines a selected task with a sensing location, while ongoing empty, sensing, and transmission cycles retain the current task and location.Sensing locations are continuous variables, and tasks already being executed by other UAVs are excluded from selection.
  • 3) State Transition Function:: The state transition function advances cycles, updates UAV locations and data, and changes task AoI according to execution outcomes.A successful task execution resets its AoI to zero; otherwise, AoI increases by one cycle, while sensing and transmission update data differently.
  • 3) State Transition Function:: Orthogonal subcarrier allocation gives UAV i a positive allocation under the assumption K>M, with equal allocation across UAVs.The transition model uses an indicator function to determine allocation based on the transmission-cycle condition.

4) Reward:

The paper formulates cooperative UAV trajectory design around minimizing accumulated AoI, using AoI reduction as an equivalent maximization objective and reinforcement learning because transitions are unknown and spaces are large and mixed.

  • AoI reduction from successful task executions is used as the reward, with larger task AoI reductions producing greater reward.
  • The UAVs optimize policies for AoI minimization while accounting for both current and future task AoI.
  • The reward is shared among cooperating UAVs to support distributed trajectory optimization under a common AoI objective.
  • Maximizing total AoI reduction is equivalent to minimizing accumulated AoI because maximum accumulated AoI is constant.
  • Unknown sensing and transmission models leave the state transition function unspecified, motivating reinforcement learning without prior transition knowledge.
  • Deep reinforcement learning is adopted because the large state and action spaces contain continuous and discrete variables, making tabular Q-learning inefficient.

V. COMPOUNDED-ACTION ACTOR-CRITIC ALGORITHM

The paper introduces CA2C for trajectory design in a cellular Internet of UAVs, targeting compounded actions that combine discrete task selection with continuous sensing-location selection.

  • CA2C is proposed to solve the cellular Internet of UAVs trajectory-design problem using deep reinforcement learning.
  • CA2C handles compounded actions containing both continuous and discrete variables.
  • Existing deep reinforcement learning algorithms cannot directly address the problem because each UAV action combines discrete task selection with continuous sensing-location selection.
  • The proposed method combines ideas from DDPG for continuous actions and DQN for discrete actions.

B. Action Selection Process

CA2C decomposes action selection into task choice and sensing-location prediction, then evaluates candidate tasks through a critic network to select the highest-valued action.

  • The action policy is decomposed into a discrete task-selection policy and a continuous sensing-location policy.
  • Neural networks approximate the optimal Q-function because the state and action spaces are too high-dimensional for exact estimation.
  • The action-selection process combines the current state and candidate task to generate a sensing location before evaluating the resulting state-task-location combination.
  • For each candidate task, the actor predicts a sensing location and the critic estimates its Q-function value; the UAV selects the task with the highest estimate.
  • Each UAV stores completed-task transitions in a replay buffer and trains its actor and critic networks from randomly sampled experience batches.
  • Target actor and critic networks provide next-state estimates, while soft updates and double Q-learning address training stability and Q-value overestimation.

VI. ALGORITHM ANALYSIS

The training procedure records transitions when selected tasks complete, then updates each UAV’s actor and critic networks using replayed experiences and target networks.

  • The critic update computes next-state task and sensing-location choices with target networks before minimizing its loss and softly updating the target critic.
  • The actor update follows Q-function gradients with respect to sensing locations and then softly updates the target actor parameters.
  • CA2C trains actor and critic networks for each UAV from replay-buffer experiences collected during episodes.
  • Algorithm 3 initializes network parameters and replay buffers, iterates over training episodes, selects exploratory or learned tasks, and executes the associated action.

A. Computational Complexity

The CA2C algorithm has action-selection complexity O(N^2 · M) and training complexity O(N_b · N^2 · M), comparable to DQN and DDPG in complexity while supporting mixed discrete-continuous actions.

  • A. Computational Complexity: CA2C requires O(N^2 · M) computational complexity for action selection.The dominant cost is evaluating Q-function values across the N tasks for each UAV.
  • A. Computational Complexity: CA2C requires O(N_b · N^2 · M) computational complexity for training.The leading cost comes from calculating and comparing task Q-values for each sampled experience.
  • A. Computational Complexity: CA2C has the same action-selection complexity as DDPG but higher training complexity, O(N_b · N^2 · M) versus O(N_b · N · M).CA2C is reported to outperform DDPG and DQN because it is tailored to continuous and discrete action spaces.

B. Convergence

The paper relates CA2C to Q-learning and uses a shrinking learning rate, but formal convergence guarantees are limited because simultaneous multi-agent learning is difficult to prove.

  • B. Convergence: CA2C is described as an extended Q-learning algorithm with a shrinking learning rate that supports convergence toward a local optimum and reduces oscillation.The inverse-time learning-rate schedule uses larger early updates and smaller later updates.
  • B. Convergence: Standard Q-learning convergence to the optimal Q-function is stated for a single-agent Markov decision process under the theorem's conditions.The supplied theorem statement concerns one agent adopting Q-learning in the environment.
  • B. Convergence: For multiple UAVs learning simultaneously, convergence is hard to prove and is guaranteed only in special cases such as iterated-dominance-solvable games and team games.The paper evaluates CA2C convergence through simulations rather than establishing a general multi-agent proof.

VII. SIMULATION RESULTS

Simulations compare CA2C with four baselines and examine learning, cooperation, network resources, altitude, and learning-rate effects. CA2C converges fastest among the reinforcement-learning methods and achieves lower AoI through coordinated task division and nearby sensing locations.

  • VII. SIMULATION RESULTS: CA2C has the highest training speed and converges to the lowest normalized accumulated AoI among the reinforcement-learning algorithms.It also achieves lower normalized accumulated AoI than the two non-learning algorithms after 6000 training episodes, while DDPG does not converge within 10,000 episodes.
  • VII. SIMULATION RESULTS: An initial learning rate of α0 = 1 causes high-value AoI oscillations, whereas α0 = 0.01 trains more slowly than α0 = 0.1.The paper attributes the first effect to unstable network updates and the second to smaller parameter changes.
  • VII. SIMULATION RESULTS: Cooperative UAVs achieve lower normalized accumulated AoI than non-cooperative UAVs, and the cooperation gain increases with the number of UAVs.Non-cooperative UAVs may pursue the same high-AoI task, whereas cooperation enables task division and parallel execution.
  • VII. SIMULATION RESULTS: CA2C divides tasks into non-overlapping sets and selects nearby sensing locations, shortening flight distances and reducing normalized accumulated AoI relative to DQN.DQN partitions tasks but fixes each sensing location directly above its target, causing longer inter-location flights.
  • VII. SIMULATION RESULTS: Increasing the number of subcarriers decreases normalized accumulated AoI, while each subcarrier setting has an optimal flight altitude.Low altitude increases flight distance through shorter sensing range, whereas high altitude lowers data rates and lengthens transmission.

VIII. CONCLUSION

The paper formulates cooperative UAV trajectory design for minimizing task AoI and proposes distributed protocol and CA2C reinforcement learning to solve it. Simulations show benefits over benchmark algorithms, cooperation, and appropriate altitude and subcarrier configurations.

  • The distributed sense-and-send protocol coordinates UAVs executing multiple sensing tasks through cooperative sensing and transmission.
  • The trajectory design problem is formulated as an MDP minimizing normalized accumulated task AoI.
  • CA2C combines DQN- and DDPG-based learning to handle actions containing both discrete and continuous variables.
  • CA2C outperforms four benchmark algorithms in terms of AoI minimization.
  • Cooperative sensing and transmission considerably reduce AoI relative to a non-cooperative approach, with larger reductions as the number of UAVs increases.
  • An optimal flying altitude exists for AoI minimization, and it increases with the number of available subcarriers.
Loading 2007.14297v1…