Source-linked AI summary
3D UAV Trajectory and Data Collection Optimisation via Deep Reinforcement Learning
Khoi Khac Nguyen, Trung Q. Duong, Tan Do-Duy, Holger Claussen, and Lajos Hanzo
TL;DR
UAV-assisted IoT resource allocation is difficult under limited power and flight time. The paper uses DRL to jointly optimize the UAV’s trajectory and data collection, balancing throughput with flight distance and time. The reported evaluation characterizes trajectory, reward, and total sum-rate, including DQL and dueling DQL in simple and complex settings.
Problem
Limited UAV power and flight time complicate resource allocation, while existing contributions often assume static environments and ignore stringent flight-time constraints.
Method
The paper proposes DRL techniques that jointly optimize throughput and UAV flight distance while satisfying quality-of-service, power, and communication constraints.
Results
The evaluation characterizes the proposed techniques through UAV trajectory, expected reward, total sum-rate, and convergence speed in simple and complex environmental settings.
Takeaways & Limitations
After training, the UAV can autonomously select actions for data collection while balancing throughput, trajectory, and time spent.
Takeaways & Limitations
The study assumes limited UAV power and associated communication constraints.
Abstract
from arXiv · showhide
Unmanned aerial vehicles (UAVs) are now beginning to be deployed for enhancing the network performance and coverage in wireless communication. However, due to the limitation of their on-board power and flight time, it is challenging to obtain an optimal resource allocation scheme for the UAV-assisted Internet of Things (IoT). In this paper, we design a new UAV-assisted IoT systems relying on the shortest flight path of the UAVs while maximising the amount of data collected from IoT devices. Then, a deep reinforcement learning-based technique is conceived for finding the optimal trajectory and throughput in a specific coverage area. After training, the UAV has the ability to autonomously collect all the data from user nodes at a significant total sum-rate improvement while minimising the associated resources used. Numerical results are provided to highlight how our techniques strike a balance between the throughput attained, trajectory, and the time spent. More explicitly, we characterise the attainable performance in terms of the UAV trajectory, the expected reward and the total sum-rate.
I. INTRODUCTION
UAV-assisted wireless networks face constrained power, flight time, and changing environments, while prior work has not jointly optimized 3D trajectory and data collection. DRL is motivated as a way to address high-dimensional decision-making and reduce delays associated with offline optimization.
- Motivation: Limited on-board power and flight time make resource allocation difficult, while changing environments and operational tasks hinder optimal UAV operation.The introduction also notes that existing technologies may not adapt optimally to surveying, collection, and obstacle avoidance.
- Research gap: Existing UAV-network contributions often assume static environments and ignore stringent flight-time constraints in real-life applications.
- Research gap: Model-free reinforcement learning struggles with large state and action spaces, motivating deep reinforcement learning for high-dimensional UAV-network problems.The supplied passages describe DRL as combining reinforcement learning with neural networks.
- Related work: Prior studies apply optimization and learning to resource allocation, path planning, navigation, and data collection, but generally address these tasks separately.
- Research gap: Earlier techniques mostly operate offline and may impose excessive delay, creating a need for faster decision-making in UAV-assisted networks.
- Research gap: The cited literature has not addressed joint trajectory and data-collection optimization while also considering interference, 3D trajectory, and dynamic environments.
B. Contributions and Organisation
The paper formulates single-UAV IoT data collection as maximizing throughput while minimizing travel distance, then uses DRL to learn autonomous trajectory decisions under communication and flight constraints.
- B. Contributions and Organisation: The proposed solution uses a joint reward based on shortest flight distance and uplink transmission rate to find an optimal UAV path.
- B. Contributions and Organisation: The objective is to maximize total network throughput while minimizing the UAV distance travelled.
- B. Contributions and Organisation: DRL techniques divide the area into a grid and enable the trained UAV to choose its next action autonomously at each position.The stated purpose is to support fast convergence and eliminate the need for human navigation.
- B. Contributions and Organisation: The evaluation considers scenarios with three or five clusters and assesses sum-rate, trajectory, and associated performance.
- B. Contributions and Organisation: Both DQL and dueling DQL are used, with dueling DQL intended to improve system performance and convergence speed.
II. SYSTEM MODEL AND PROBLEM FORMULATION
The system models one UAV visiting randomly distributed, mobile user clusters in 3D, with a trajectory constrained by coverage and return-to-dock requirements. The observation model uses line-of-sight free-space propagation and derives user throughput and aggregate sum-rate over time.
- II. SYSTEM MODEL AND PROBLEM FORMULATION: A single-antenna UAV visits M user clusters to cover their users, and its 3D position is represented by time-varying coordinates.
- II. SYSTEM MODEL AND PROBLEM FORMULATION: Each cluster contains K users randomly distributed within coverage radius C, with users moving according to a random-walk model capped at velocity v.
- II. SYSTEM MODEL AND PROBLEM FORMULATION: The trajectory objective is to cover all users and return the UAV to the dock.
- II. SYSTEM MODEL AND PROBLEM FORMULATION: The system model depicts the UAV-aided IoT communication arrangement used for the trajectory and data-collection problem.
- A. Observation model: The observation model defines the UAV-to-user distance at each time step and assumes line-of-sight communication links.
- A. Observation model: Channel power gain follows a free-space path-loss model, with β0 denoting the gain at the reference distance d = 1m.
- A. Observation model: Achievable user throughput is defined when the distance constraint is satisfied, using bandwidth B and noise power α2.
- A. Observation model: The total sum-rate aggregates throughput from each user over T time steps.
B. Game formulation
The game formulation models UAV trajectory and data collection as an MDP in which position-based actions determine movement, user coverage, rewards, and mission completion. Its reward design balances user sum-rate against movement and supports different operational priorities under flight, distance, and transmission constraints.
- The UAV is formulated as an agent seeking an optimal policy π∗ that maximises expected mission reward while flying from its initial position through user clusters to the destination.
- The state is the UAV position st = (xt, yt, Ht), and the grid-based action space contains left, right, forward, backward, upward, downward, and hover.
- Users begin data collection when within UAV coverage and are marked collected after reaching the required rate, preventing revisits during the mission.
- The reward depends on associated users’ total sum-rate and UAV movement, with β and ζ controlling the trade-off and Rplus rewarding mission completion.
- Adjusting β/ζ supports fast emergency deployment, maximising total sum-rate, or maximising UAV-user connections.
- Transmission distance, UAV position, flight time, and limited power constrain optimisation, motivating DRL techniques for the trajectory and data-collection problem.
III. PRELIMINARIES
The preliminaries define Q-learning as a value-based reinforcement-learning framework for selecting UAV actions under an MDP. The formulation uses discounted rewards, ε-greedy exploration, distance-constrained communication, and terminal mission conditions.
- The finite-game formulation assumes an optimal policy π∗ satisfying Bellman optimality, with the action-value function defined from state, action, reward, and successor-state transitions.
- Q-learning updates the action-value function from observed rewards and successor states, with α controlling the Q-value update and γ discounting future rewards.
- The UAV selects actions with an ε-greedy policy, communicates with users only when dm,k ≤ dcons, and marks served users as unavailable for later revisits.
- Each transition advances the UAV to st+1, updates the Q-value, and ends an episode when the destination is reached under the flight-duration constraint.
IV. AN EFFECTIVE DEEP REINFORCEMENT LEARNING APPROACH FOR UAV-ASSISTED IOT
The paper develops a deep Q-learning approach for the UAV trajectory and data-collection trade-off game, replacing large Q-tables with neural networks. Experience replay and target networks are used to stabilise learning, while two episode-reward definitions are compared.
- Deep Q-learning addresses excessive Q-table size in large state and action spaces by using deep neural networks to represent the state-action relationship.
- The algorithm stores transitions in a replay buffer, samples mini-batches for training, and updates network parameters by minimising a loss function.
- A separate target network Q′ estimates target values because using the same network can shift dramatically during training; its parameters are periodically updated from Q.
- The method compares immediate reward defined as the difference between consecutive rewards with total episode reward accumulated across steps.
V. DEEP REINFORCEMENT LEARNING APPROACH FOR UAV-ASSISTED IOT NETWORKS: A DUELING DEEP Q-LEARNING APPROACH
The dueling deep Q-learning approach decomposes action evaluation into state-value and advantage streams for UAV trajectory and data collection. It uses replay-based training and target-network updates to improve convergence and stability without extra supervision.
- The advantage function A(s, a) = Q(s, a) − V(s) describes the importance of each action relative to its state.
- The dueling network estimates a scalar state value V(s; θV) and an action-advantage vector A(s, a; θA), combining them to form Q-values.
- Algorithm 2 selects ε-greedy actions, executes them to obtain rewards and successor states, stores transitions, samples mini-batches, and periodically updates the target network.
- The mean-subtraction formulation addresses value-advantage identifiability, with the chosen action satisfying Q(s, a∗; θ, θA, θV) = V(s; θV).
- The paper proposes a dueling DQL algorithm for joint UAV trajectory and data collection optimisation in UAV-assisted IoT networks.
VI. SIMULATION RESULTS
The simulations evaluate joint UAV trajectory and data-collection optimisation in three- and five-cluster environments. The trained DQL trajectory visits users while respecting flight-duration and power constraints.
- Simulation setup: The study compares a simple three-cluster scenario with a more complex five-cluster scenario for UAV-assisted IoT optimisation.Training uses TensorFlow 1.13.1 and the Adam optimiser, with remaining parameters specified in Table II.
- Trajectory result: The trained DQL trajectory starts at (0, 0), visits about 40 users, and lands at the destination.The trajectory visual distinguishes cluster coverage, user nodes, and UAV states after actions.
- Operational constraints: In complex environments, visiting all users while satisfying flight-duration and power constraints is challenging.This constraint motivates evaluating the joint trajectory and data-collection problem across different scenarios.
A. Expected reward
The expected-reward experiments compare DQL and dueling DQL across cluster counts, trajectory designs, reward functions, and β/ζ settings. Dueling DQL and exponential trajectory design generally provide stronger or faster-converging performance.
- Five-cluster reward comparison: In the five-cluster scenario, episode reward performs better than immediate reward for both trajectory designs.This result is reported for the expected-reward comparison in Fig. (4).
- Five-cluster comparison: For β/ζ = 1 : 1, dueling DQL with exponential trajectory design reaches the best performance and converges faster than DQL.The corresponding DQL and dueling DQL rewards are similar and stable after about 400 episodes under the compared setting.
- Algorithm and trajectory comparison: The dueling DQL algorithm delivers better rewards than DQL across β/ζ pairs, while exponential trajectory design outperforms binary design.These comparisons are reported for both three-cluster and five-cluster settings.
- Reward-function comparison: The episode reward outperforms the immediate reward, especially when β/ζ ≥1 : 1, while exponential trajectory design remains superior to binary design.The same reward-function pattern is observed for dueling DQL in the five-cluster experiments.
B. Throughput comparison
Throughput experiments examine trajectory cost, average throughput, algorithm choice, reward design, and β/ζ trade-offs. The strongest reported setting is dueling DQL with exponential trajectory design and episode reward, although alternatives perform better in some scenarios.
- Three-cluster scenario: In a three-cluster binary-design setting, β/ζ = 1 : 1 yields higher throughput than other tested values, while increasing β degrades performance.Some configurations can also cause the UAV to become stuck, increasing flight time and distance travelled.
- Throughput reward: The best reported throughput uses dueling DQL with β/ζ = 1 : 1 and binary trajectory design, exceeding DQL with β/ζ = 1 : 2.This comparison reflects the joint trajectory-cost and average-throughput reward formulation.
- β/ζ trade-off: Increasing β degrades throughput but can let the UAV finish the mission faster, creating a throughput-versus-mission-time trade-off.The authors recommend choosing an approximate β/ζ value according to the specific objective.
- Reward and trajectory design: Episode reward produces higher throughput than immediate reward across different β/ζ values, while exponential trajectory design exceeds binary design.The expected throughput is stable across β/ζ values when episode reward is used.
- Overall comparison: Dueling DQL with exponential trajectory design and episode reward achieves the best overall reported performance, although another setting can be better in some scenarios.The paper therefore describes algorithm selection as a trade-off rather than a universally fixed configuration.
C. Parametric Study
The parametric study evaluates exploration settings, mini-batch sizes, and learning rates for DQL, identifying configurations that improve performance and convergence. It also frames the DRL approach around jointly optimising UAV trajectory and data collection under resource constraints.
- Mini-batch size: K = 32 delivers optimal DQL performance in the 5-cluster scenario and the fastest convergence among tested batch sizes.The study therefore sets the mini-batch size to K = 32 for DQL.
- Learning rate: The learning rate α = 0.01 causes fluctuating performance, whereas α = 0.0001 and α = 0.00001 slow convergence and may lead to local optima.The experiments consequently select α = 0.001 for the algorithms.
- Parametric evaluation: The study evaluates DQL in a 5-cluster scenario while varying batch size and learning rate to characterize performance and convergence.Figures 13 and 14 provide the corresponding parametric comparisons.
- Optimisation objective: The proposed DRL techniques jointly optimise UAV flight trajectory and data collection while balancing flight time, throughput, and quality-of-service constraints.Both DQL and dueling DQL are designed to maximise throughput while following the shortest path under limited UAV resources.