PaperlayerView exact revision on arXiv

Source-linked AI summary

UAV Path Planning for Wireless Data Harvesting: A Deep Reinforcement Learning Approach

Harald Bayerlein, Mirco Theile, Marco Caccamo, David Gesbert

arXiv:2007.00544v2cs.LGcs.ITcs.ROeess.SPmath.ITstat.ML

TL;DR

Urban UAV data harvesting requires trajectory planning under limited flight time and navigation constraints, while changing mission parameters can necessitate expensive retraining. The paper uses a DDQN with combined experience replay and a centered multi-layer map to adapt its policy across scenarios. The agent effectively balances data collection, safety, and mission time, and centered maps improve learning performance.

  • Problem

    UAV data harvesting must account for limited flight time and navigation constraints, while changes in scenario parameters can require expensive retraining or relearning.

    Source: §I · pp. 1–2

  • Method

    A DDQN with combined experience replay uses centered multi-layer maps to learn a UAV data-harvesting policy across varying scenario parameters.

    Source: §I · p. 2, B. Double Deep Q-learning and Combined… · p. 3, C. Centered Global Map · p. 4

  • Results

    The agent adapts to variations in device count, position, collectible data, and flying time, while centered maps show a clear performance advantage over non-centered maps.

    Source: B. Centered vs. Non-Centered Map · p. 5, §V · p. 6

  • Takeaways & Limitations

    The UAV agent balances data collection, obstacle avoidance, and mission-time minimization without prior information about urban wireless-channel characteristics.

    Source: §V · p. 6

Abstract

from arXiv · show

Autonomous deployment of unmanned aerial vehicles (UAVs) supporting next-generation communication networks requires efficient trajectory planning methods. We propose a new end-to-end reinforcement learning (RL) approach to UAV-enabled data collection from Internet of Things (IoT) devices in an urban environment. An autonomous drone is tasked with gathering data from distributed sensor nodes subject to limited flying time and obstacle avoidance. While previous approaches, learning and non-learning based, must perform expensive recomputations or relearn a behavior when important scenario parameters such as the number of sensors, sensor positions, or maximum flying time, change, we train a double deep Q-network (DDQN) with combined experience replay to learn a UAV control policy that generalizes over changing scenario parameters. By exploiting a multi-layer map of the environment fed through convolutional network layers to the agent, we show that our proposed network architecture enables the agent to make movement decisions for a variety of scenario parameters that balance the data collection goal with flight time efficiency and safety constraints. Considerable advantages in learning efficiency from using a map centered on the UAV's position over a non-centered map are also illustrated.

I. INTRODUCTION

UAV data harvesting in urban environments must balance limited flight time, obstacle and no-fly-zone avoidance, and variable wireless conditions. The paper proposes a DDQN approach designed to adapt to changing mission parameters, with a centered map that improves learning efficiency.

I. INTRODUCTION

  • Urban data harvesting combines restricted mission duration, obstacle and no-fly-zone avoidance, and wireless-channel fluctuations caused by alternating LoS and NLoS links.Source: §I · p. 1
  • Existing non-RL approaches generally solve one set of scenario parameters at a time, while prior RL methods may require expensive retraining or relearning when parameters change.The cited approaches also vary in whether they account for obstacles and no-fly zones.Source: §I · pp. 1–2
  • The method plans UAV trajectories for IoT data harvesting while maximizing collected data under flight-time and navigation constraints, without prior wireless-channel information.Source: §I · p. 2
  • A centered multi-layer environment map considerably increases the RL agent’s learning efficiency.Source: §I · p. 2
  • The DDQN policy is designed to adapt to changes in sensor count, sensor positions, and other scenario parameters without expensive retraining.The paper identifies retraining after even small scenario changes as a challenge for deep RL.Source: §I · p. 2

II. SYSTEM MODEL AND PROBLEM FORMULATION

The system models a UAV collecting data from static IoT devices over a constrained grid and time horizon. Its objective is to maximize collected throughput while minimizing flight duration, subject to safety and landing constraints.

A. System Model

  • The mission takes place on an M × M grid with K static IoT devices at ground-level coordinates.Source: A. System Model · p. 2
  • The UAV flies at constant altitude and either hovers or moves at constant velocity V during each mission time slot.Source: A. System Model · p. 2
  • Each mission time slot is divided into δ communication slots so the UAV position and channel gain can be treated as constant within each communication slot.Source: A. System Model · p. 2
  • UAV-device links use LoS/NLoS point-to-point channels with log-distance path loss and shadow fading; the urban propagation parameters vary with the LoS/NLoS condition.The stated SNR is averaged over small-scale fading.Source: A. System Model · p. 2
  • In each communication slot, TDMA serves the device with remaining data that has the highest SNR.Source: A. System Model · p. 2
  • Throughput in a mission time slot is the sum of achieved rates across its communication slots and the K sensor devices.Source: A. System Model · p. 2
  • The trajectory objective maximizes mission throughput and minimizes flight duration, subject to flight-time, no-fly-zone, obstacle-avoidance, and safe-landing constraints.The optimization is translated into an MDP reward function and solved using deep reinforcement learning.Source: A. System Model · p. 2

B. Markov Decision Process

The MDP represents each mission through device data, UAV position, remaining flight time, the environment map, and device locations. Its action set and reward components encode movement choices, data collection, safety, and mission completion.

B. Markov Decision Process

A. Q-Learning

Q-learning uses agent–environment interaction to improve a policy through a state-action value function. The value represents expected discounted return from the current state and action to a terminal state.

A. Q-Learning

  • The agent observes a state, takes an action, receives a reward, and transitions to the next state in a repeated interaction cycle.The policy is the behavior rule the agent aims to learn to maximize received reward.Source: A. Q-Learning · p. 3
  • The Q-function is iteratively improved to guide and evaluate learning a policy.Source: A. Q-Learning · p. 3
  • The Q-value is the expected discounted cumulative return from the current state and action through a terminal state; γ balances immediate and future rewards.Source: A. Q-Learning · p. 3

B. Double Deep Q-learning and Combined Experience Replay

The method approximates the Q-function with a deep Q-network and trains it as a DDQN. Combined experience replay always includes the agent’s latest transition to reduce sensitivity to replay-buffer size.

B. Double Deep Q-learning and Combined Experience Replay

C. Centered Global Map

The global input map combines environmental information with device-data layers, and the map is centered on the UAV. This makes flattened features agent-relative rather than tied to absolute map positions.

C. Centered Global Map

  • The global map combines a static environmental map with two real-valued device-data layers: available data and data already collected.Source: C. Centered Global Map · p. 3
  • The figure compares non-centered and centered input maps, marking the UAV with a green star and the dashed-line intersection.Source: Figure 1 · p. 4
  • The paper applies centering to global maps for the first time in a UAV data-collection scenario and reports that it benefits generalization over varying scenario parameters.Source: C. Centered Global Map · pp. 3–4
  • The map-centering process expands maps to (2M −1) × (2M −1) and translates them so the expanded map is centered on the UAV, preserving the entire map in view.Source: C. Centered Global Map · p. 4
  • Centering changes flattened neurons from representing features at absolute map positions to representing features relative to the UAV, which aligns with actions based on relative feature positions.The paper links this representation to considerably increased learning efficiency.Source: C. Centered Global Map · p. 4

D. Neural Network Model

The network processes the centered map with convolutional layers, then combines its flattened representation with remaining flight time. Fully connected layers produce action Q-values used for evaluation or training policies.

D. Neural Network Model

  • Centered maps pass through convolutional layers with ReLU activation, are flattened, and are concatenated with the scalar input for remaining flight time.Source: D. Neural Network Model · p. 4
  • Fully connected layers with ReLU activation feed a final linear layer of size |A| whose outputs are Q-values for each action.Source: D. Neural Network Model · p. 4
  • Evaluation uses a deterministic greedy policy, while training uses a soft-max policy whose temperature β scales exploration versus exploitation.Source: D. Neural Network Model · p. 4

IV. SIMULATIONS

The simulations evaluate UAV data collection in a constrained grid environment using episode-level metrics and Monte Carlo sampling across scenario variations.

A. Simulation Setup

  • The UAV operates in a 16×16 grid of 10m cells at 10m altitude, with flight time decremented after every move or hover and restrictions on buildings, NFZs, and grid boundaries.Missions begin with T steps of remaining flight time.Source: A. Simulation Setup · p. 4
  • The agent has no prior knowledge of the shadowing maps or wireless-channel characteristics used in the urban micro-scenario.Propagation uses LoS/NLoS path-loss and shadow-fading parameters, with shadowing maps computed by ray tracing.Source: A. Simulation Setup · p. 4
  • Performance is measured by cumulative reward, in-time landing, collection ratio, and their product, collection ratio and landed.Training progress is evaluated every ten episodes in a randomly selected scenario, while trained-agent performance is assessed by Monte Carlo sampling because exhaustive evaluation is infeasible.Source: A. Simulation Setup · p. 4
  • Figure 3 compares centered and non-centered map training processes using three-run averages and 99% quantiles, with episodic metrics grouped into 5,000-step bins.Figure 2 depicts the DQN architecture with the device map encoded in separate layers and visualized in RGB channels.Source: Figure 3 · p. 5, Figure 2 · p. 5

B. Centered vs. Non-Centered Map

Centered map inputs improve learning performance and generalization relative to non-centered inputs. The same agent also changes its route as collectible data at devices changes.

B. Centered vs. Non-Centered Map

  • Centered map input gives a clear performance advantage throughout training and is reported to improve the DDQN agent’s learning performance and generalization.The comparison averages three training runs; the non-centered input is padded to match the centered input’s shape and adds UAV position as a 2D one-hot map layer.Source: B. Centered vs. Non-Centered Map · p. 5

C. Collectible Data and Device Accessibility

  • With two devices and randomized collectible data, the agent enters the difficult courtyard only when the orange device’s data makes it necessary.When the orange device’s data is lower, the UAV collects the unobstructed green node first and takes only the required detour to the orange node on the way back.Source: C. Collectible Data and Device… · p. 5

D. Manhattan Scenario

In the Manhattan scenario, one agent is evaluated across varying device counts, device locations, data amounts, and flight-time limits. The conclusion reports effective adaptation while balancing collection, obstacle avoidance, and mission time.

D. Manhattan Scenario

  • The Manhattan tests vary device count from 2 to 5, device data from 5.0 to 20.0 units, flight time from 35 to 70 steps, and starting position across eight possibilities.Device positions are randomized in unoccupied map space; the city includes regularly distributed blocks, streets, and an NFZ district.Source: D. Manhattan Scenario · p. 5
  • Both agents achieve similarly high successful-landing performance in Monte Carlo evaluation, though collection can fall below 100% when assigned flight time and device parameters constrain a mission.The evaluation averages performance over 1,000 randomly selected scenario iterations; the supplied table caption does not provide metric values.Source: D. Manhattan Scenario · pp. 5–6, Table II · p. 6
  • Example missions show adaptation to obstacles, distance, and time limits: the agent may ignore an isolated device when time is insufficient, or order collection efficiently when all data is reachable.Repeated runs of the same configuration produce trajectory variation that adapts to random communication-channel fading.Source: D. Manhattan Scenario · p. 6

V. CONCLUSION

  • The DDQN with combined experience replay adapts to changes in device count and position, collectible data, and maximum flying time without expensive retraining or recollecting training data.The conclusion also identifies scalability to larger maps as future work because trainable parameters in the flatten layer increase linearly with map area.Source: §V · p. 6
  • The method balances data collection, obstacle avoidance, and mission-time minimization without prior information about urban wireless-channel characteristics.The conclusion proposes future extensions including multi-task or transfer learning and altitude control.Source: §V · p. 6

Related papers

Loading 2007.00544v2…