Source-linked AI summary
Multi-UAV Path Planning for Wireless Data Harvesting with Deep Reinforcement Learning
Harald Bayerlein, Mirco Theile, Marco Caccamo, David Gesbert
TL;DR
The paper tackles multi-UAV IoT data harvesting when mission parameters and urban conditions vary, making fixed, repeatedly retrained policies impractical. It formulates cooperative path planning as a Dec-POMDP and uses map-based MARL to learn generalized control policies. The resulting agents cooperate across complex environments, balancing data collection with flight-time and navigation constraints while enabling system-level trade-off analysis.
Problem
Multi-UAV IoT harvesting must handle changing UAV, sensor, timing, and urban-environment parameters, while existing DRL methods typically require retraining after scenario changes.
Method
The paper translates cooperative non-communicating UAV path planning into a Dec-POMDP and applies DDQN-based MARL with centered global-local map representations.
Results
The learned agents find efficient trajectories that balance data collection with safety and navigation constraints without prior knowledge of challenging urban wireless-channel characteristics.
Takeaways & Limitations
Generalized policies can be reused across broad scenario-parameter changes and support analysis of collection-performance trade-offs, including UAV count versus mission time.
Abstract
from arXiv · showhide
Harvesting data from distributed Internet of Things (IoT) devices with multiple autonomous unmanned aerial vehicles (UAVs) is a challenging problem requiring flexible path planning methods. We propose a multi-agent reinforcement learning (MARL) approach that, in contrast to previous work, can adapt to profound changes in the scenario parameters defining the data harvesting mission, such as the number of deployed UAVs, number, position and data amount of IoT devices, or the maximum flying time, without the need to perform expensive recomputations or relearn control policies. We formulate the path planning problem for a cooperative, non-communicating, and homogeneous team of UAVs tasked with maximizing collected data from distributed IoT sensor nodes subject to flying time and collision avoidance constraints. The path planning problem is translated into a decentralized partially observable Markov decision process (Dec-POMDP), which we solve through a deep reinforcement learning (DRL) approach, approximating the optimal UAV control policy without prior knowledge of the challenging wireless channel characteristics in dense urban environments. By exploiting a combination of centered global and local map representations of the environment that are fed into convolutional layers of the agents, we show that our proposed network architecture enables the agents to cooperate effectively by carefully dividing the data collection task among themselves, adapt to large complex environments and state spaces, and make movement decisions that balance data collection goals, flight-time efficiency, and navigation constraints. Finally, learning a control policy that generalizes over the scenario parameter space enables us to analyze the influence of individual parameters on collection performance and provide some intuition about system-level benefits.
I. INTRODUCTION
The paper addresses multi-UAV IoT data harvesting under changing mission parameters, urban navigation constraints, and wireless-channel uncertainty. It proposes generalized MARL policies and map-based inputs to avoid repeated retraining while supporting cooperative path planning.
- Motivation: The mission varies the number of UAVs, sensor devices, device locations, data amounts, and flying times in complex urban environments.UAVs must collect data while managing limited battery duration, obstacles, NFZs, and LoS/NLoS blocking.
- Method rationale: DRL is used because it can combine navigation constraints and data-collection objectives in the reward while offering computationally efficient inference.The paper motivates DRL for non-convex and often NP-hard UAV control and deployment problems.
- Research gap: Most prior RL approaches optimize one scenario at a time and require full or partial retraining when scenario parameters change.The paper targets direct policy generalization across a broad scenario-parameter space.
- Contributions: The proposed approach combines parameter generalization with centered dual global-local maps to improve adaptation to large maps and state spaces.The learned policy covers UAV counts, start positions, flying times, and sensor-device configurations without restarting training.
- System-level analysis: Generalized policies enable direct analysis of parameter effects and system-level trade-offs such as deploying more drones versus reducing mission time.The paper uses this capability to compare collection performance across scenario parameters.
C. Organization
The paper models UAVs in a discretized rectangular grid with fixed-altitude movement, operational states, battery levels, obstacles, NFZs, and start or landing zones. Its organization introduces these models before the communication, optimization, learning, and evaluation sections.
- Environment model: The environment is a rectangular grid containing start/landing positions, forbidden positions, tall obstacles, and smaller buildings that may be flown over.Tall buildings and NFZs block UAV movement, while buildings can also cause wireless shadowing.
- UAV state: Each UAV state includes three-dimensional position, binary operational status, and battery energy level.The model uses ground level or a constant flight altitude and permits different constant altitudes if altitude is observable.
- Model scope: The model restricts the study to 2D trajectory optimization because climbing consumes battery energy and high-rise buildings require long climbing phases.UAVs must ultimately land at ground level, further limiting the benefit of 3D control under mission-time constraints.
- Motion model: Movement occurs in discrete time slots with horizontal speed 𝑉 = 𝑐/𝛿𝑡 or zero, and landing is allowed only in a landing zone.The feasible action set depends on the UAV’s current position.
- Mission termination: The mission ends when all UAVs have safely landed and become inactive, under a constant-operating-energy assumption that treats battery content as remaining flying time.The constant-consumption approximation is justified because hovering dominates the modeled power demand.
B. Communication Channel Model
The communication model separates mission and communication time scales, tracks finite sensor data, and computes throughput from LoS/NLoS wireless links and scheduling decisions. It assumes TDMA and orthogonal inter-UAV channels while leaving multiple-access design outside the scope.
- Time scales: Each mission slot is divided into 𝜆 communication slots, allowing link performance and data transmission to be modeled at a finer time scale.The communication-slot duration is 𝛿𝑡/𝜆 and the total communication index reaches 𝑁 = 𝜆𝑇.
- Sensor data: IoT devices occupy fixed ground locations and have finite data volumes that decrease over communication time as data is uploaded.Initial device data is specified at the mission start.
- Channel model: The channel uses LoS/NLoS point-to-point links with log-distance path loss and shadow fading, while effective rates account for remaining device data.The model captures distance dependence and urban obstruction effects without requiring the planner to know the channel model.
- Scope assumptions: The model assumes TDMA within each UAV and orthogonal resource blocks across UAVs, so inter-UAV interference is not considered.Multiple-access protocol design is explicitly outside the paper’s scope, although the learned geographic task division may partly mitigate interference.
- Scheduling: Scheduling follows a max-rate rule that selects the sensor with the highest available SNR, subject to a TDMA constraint for each UAV.Throughput sums scheduled rates over communication slots and sensor nodes while depending on UAV operational status.
C. Optimization Problem
The optimization problem maximizes total throughput collected by all deployed UAVs over the mission. This objective is subject to mobility, safety, flying-time, landing, starting-state, and scheduling constraints.
- Objective: The central objective is maximizing throughput over the full mission and all deployed UAVs.The optimization jointly selects UAV actions while respecting the modeled communication schedule.
- Constraints: The optimization must satisfy collision avoidance, obstacle and NFZ avoidance, operation-time, starting-position, starting-status, and scheduling constraints.The constraints ensure UAVs remain safe and land in the designated landing zone before battery depletion.
III. MARKOV DECISION PROCESS (DEC-POMDP)
The multi-UAV harvesting problem is modeled as a cooperative Dec-POMDP with shared data-collection rewards and safety-aware individual penalties. A safety controller rejects actions that violate navigation rules, while flight-time compliance is handled through penalties rather than hard enforcement.
- Problem formulation: The path-planning problem is formulated as a Dec-POMDP with state, joint actions, transitions, rewards, observations, and a discount factor.The state includes environment, agent, and device information, while the agents operate through individual observations.
- State space: The state represents maps of landing zones, obstacles, and NFZs together with UAV positions, remaining flight times, statuses, and IoT-device data.
- Safety controller: The safety controller replaces rejected actions with hovering and excludes landing unless the UAV is in a landing zone.It enforces collision avoidance, NFZ avoidance, and obstacle avoidance.
- Constraint handling: The safety controller cannot enforce flying time and safe landing as hard constraints, so late landing receives a high penalty instead.Agents with negative remaining battery time are treated as crashed and non-operational in simulation.
- Reward design: The reward combines collective collected-data reward with individual penalties for rejected actions, late landing, and movement.The movement penalty incentivizes shorter flight times and efficient trajectories.
IV. MAP-PROCESSING AND OBSERVATION SPACE
The map-processing pipeline converts spatial state information into centered, dual-resolution representations. A detailed local map supports immediate actions, while a compressed global map preserves broader directional information and reduces computational cost.
- Map centering: Centering relocates the map around each UAV, making convolutional features relative to the agent rather than tied to absolute positions.Centering improves learning performance but increases map and observation sizes.
- Global-local mapping: The centered representation is split into an uncompressed cropped local map and a compressed global map.The local map is a central l×l crop, while the global map uses average pooling with cell size g.
- Global-local mapping: Distant features receive lower spatial detail for directional decisions, whereas close features retain detail for immediate actions such as collision avoidance.
- Computational cost: Global-local processing offsets the computational increase caused by centering and considerably reduces floating-point operations compared with centering alone.Table II reports inference cost in million floating point operations for different map-processing regimes.
- Mapping: Map layers encode spatial state variables such as device data, UAV flight times, and operational status, which can be stacked into tensors.A general mapping function assigns values to grid coordinates and uses zero elsewhere.
- Global-local mapping: Increasing l enlarges the local map, while increasing g enlarges pooling cells and decreases global-map size.
B. Observation Space
Each UAV receives an observation composed of local and global map views plus its remaining flying time. Compression and cropping create partial observability while substantially reducing neural-network size and training time.
- Observation contents: The observation space contains local and global representations of the environment, collectible data, other UAVs’ remaining flight times, and their operational statuses.
- Observation contents: Each UAV’s remaining flying time is included both in the local map representation and as a scalar input.This redundancy helps the agent interpret the remaining flying time.
- Partial observability: Using observations instead of the full state converts the path-planning problem into a partially observable MDP.Partial observability results from the restricted local map and compressed global map.
- Partial observability: Map compression greatly reduces neural-network size and leads to a significant reduction in training time despite partial observability.The cited passage states that partial observability does not make the problem infeasible, even for a memory-less agent.
A. Q-Learning
The approach uses deep Q-learning to approximate action values in large UAV planning state spaces. Experience replay, target networks, and double Q-learning stabilize training and reduce action-value overestimation.
- Q-learning: Q-learning iteratively learns a state-action value function that guides a policy toward maximizing discounted cumulative reward.
- Deep Q-networks: Deep Q-networks replace inefficient Q-tables in large UAV planning spaces while using neural-network generalization to improve data efficiency.The passage motivates DQNs by the size of the state and action spaces and notes training-stability challenges.
- Training stabilization: Experience replay stores state-transition experiences and samples minibatches uniformly to reduce correlations in training data.
- Training stabilization: A separate target network estimates next-state maximum Q-values, with soft updates controlling how quickly target parameters adapt.The update factor τ determines the adaptation pace.
- Double deep Q-learning: Double deep Q-learning avoids overestimation of action values under certain conditions in standard DQN and supplies the network’s training target.
C. Multi-agent Q-learning
The approach uses decentralized execution with centralized training for homogeneous, non-communicating UAVs, combining centered global-local maps with scalar inputs in a DDQN architecture. Missions randomize key scenario parameters so one learned policy can generalize across changing data-harvesting conditions.
- Multi-agent setting: The UAV team is homogeneous and non-communicating, with identical structures, domain knowledge, and action spaces.Agents still observe other UAVs’ positions, which may be conveyed through required command-and-control links.
- Training and execution: Training centrally pools experiences from independently acting agents, while each trained control system is deployed separately on its UAV.This is decentralized deployment or execution with centralized training.
- Network architecture: The DQN processes centered global and local maps of environment constraints, IoT devices, and other UAVs through convolutional layers before combining them with scalar inputs.Map channels include NFZs, obstacles, start or landing areas, IoT devices, and other UAVs’ flying times and operational status.
- Scenario randomization: A new mission samples the number of UAVs, IoT-node count and positions, data amounts, available flying time, and UAV start positions from specified ranges.The policy is intended to generalize over this scenario-parameter space rather than require a new policy for each mission.
- Evaluation: Evaluation uses successful landing, collection ratio, and their product, with Monte Carlo sampling used to assess performance across many scenario combinations.The collection ratio is total collected data divided by data available at mission start; collection ratio and landed multiplies this by successful landing.
- Constraints: The simulated UAVs fly at constant altitude and must avoid tall buildings, NFZs, and leaving the grid world while collecting data under finite mission time.The grid cell size is c=10m, altitude is h=10m, and each mission time slot contains λ=4 scheduled communication slots.
B. Training with Map-based vs. Scalar Inputs
Map-based DRL is compared with scalar-input networks under changing scenario conditions and across two urban maps. The map-based agent learns effectively, supports large environments without proportional network growth, and produces coordinated multi-UAV trajectories.
- Training comparison: Scalar-input agents cannot effectively adapt to changing scenario conditions, while absolute versus relative position encoding does not alter their results.Small scalar networks show only a slight edge over large scalar networks.
- Training comparison: The map-based agent reaches final performance metric levels after the first 20% of training steps.Further training optimizes flight-time efficiency and multi-UAV coordination; three million steps took around 40 hours on a 2017 Nvidia Titan Xp GPU.
- Manhattan32 scenario: The global-local map approach solves the larger 32×32 Manhattan32 scenario without increasing network size.Map scaling enables the larger scenario to use the same network size as the smaller related scenario.
- Manhattan32 scenario: With two UAVs in the Manhattan32 example, the collection ratio increases to 76.5%; with three UAVs, all data is collected with in-time landing while avoiding NFZs.The UAVs divide the harvesting task among themselves and use alternative trajectories to collect devices missed by another UAV.
- Urban50 scenario: In the larger Urban50 example, two UAVs divide devices between lower-left and lower-right regions while ignoring isolated devices.The map has an order of magnitude more cells than the scenarios in the cited prior work.
- Urban50 scenario: A network without map scaling would require 34,061,446 trainable parameters for the Urban50 scenario.The global-local map approach uses a higher global-map scaling or compression factor and yields a smaller network than the Manhattan32 network.
E. Influence of Scenario Parameters on Performance and System-level Benefits
The generalized policy supports analyzing how UAV number, IoT-device load, initial data, and flying time affect collection performance, including outside the training range. This exposes system-level trade-offs such as using more coordinating drones to reduce mission time while maintaining collection performance.
- System-level benefits: Two coordinating UAVs complete the Manhattan32 mission with b_0 = 75, compared with b_0 = 150 for one UAV, while both achieve around 88% average data collection ratio with in-time successful landing.The example illustrates a trade-off between extra equipment and substantially reduced mission time.
- UAV deployment: Increasing the number of deployed UAVs improves collection performance but also increases collision-avoidance requirements and early safety-controller activations.
- IoT-device parameters: More IoT devices create more complex trajectory requirements and reduce performance because devices are randomly distributed across unoccupied map space.
- Data volume and flying time: Higher initial data volume benefits collection roughly up to D_k,init ∈ [10, 12.5] data units, after which flying-time constraints force UAVs to abandon some data.
- Data volume and flying time: More available flying time improves collection performance, but its effect shrinks once most data is collected and UAVs prioritize shorter flights and safe landing.
- Generalization: The agents preserve parameter trends outside training and successfully extrapolate to out-of-distribution settings, while increasing agent count can reduce overall performance through lower all-agent landing probability.