PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectory Design for UAV-Based Internet-of-Things Data Collection: A Deep Reinforcement Learning Approach

Yang Wang, Zhen Gao, Jun Zhang, Xianbin Cao, Dezhi Zheng, Yue Gao, Derrick Wing Kwan Ng, Marco Di Renzo

arXiv:2107.11015v1cs.ITcs.LGeess.SP

TL;DR

The paper addresses UAV trajectory design for IoT data collection when urban 3D propagation and imperfect CSI make simplified optimization models inadequate. It reformulates the problem as an MDP and proposes TD3-TDCTM, whose simulations show superior performance to three conventional non-learning baselines.

  • Problem

    Existing UAV trajectory designs commonly assume simplified LoS models or perfect CSI, whereas practical urban environments contain building-dependent LoS conditions and imperfect channel information.

  • Method

    The paper uses TD3 to learn UAV trajectories for completion-time minimization, using service statuses, UAV position, and merged pheromone information within an MDP.

  • Results

    TD3-TDCTM consistently outperforms conventional baselines, saving 54.2 s, 84.9 s, and 300.3 s versus RRT, ACO, and Scan for 25 IoT nodes.

  • Takeaways & Limitations

    The learned trajectory policy can adapt UAV movement to urban blockages while reducing mission completion time and flight energy in the evaluated settings.

Abstract

from arXiv · show

In this paper, we investigate an unmanned aerial vehicle (UAV)-assisted Internet-of-Things (IoT) system in a sophisticated three-dimensional (3D) environment, where the UAV's trajectory is optimized to efficiently collect data from multiple IoT ground nodes. Unlike existing approaches focusing only on a simplified two-dimensional scenario and the availability of perfect channel state information (CSI), this paper considers a practical 3D urban environment with imperfect CSI, where the UAV's trajectory is designed to minimize data collection completion time subject to practical throughput and flight movement constraints. Specifically, inspired from the state-of-the-art deep reinforcement learning approaches, we leverage the twin-delayed deep deterministic policy gradient (TD3) to design the UAV's trajectory and present a TD3-based trajectory design for completion time minimization (TD3-TDCTM) algorithm. In particular, we set an additional information, i.e., the merged pheromone, to represent the state information of UAV and environment as a reference of reward which facilitates the algorithm design. By taking the service statuses of IoT nodes, the UAV's position, and the merged pheromone as input, the proposed algorithm can continuously and adaptively learn how to adjust the UAV's movement strategy. By interacting with the external environment in the corresponding Markov decision process, the proposed algorithm can achieve a near-optimal navigation strategy. Our simulation results show the superiority of the proposed TD3-TDCTM algorithm over three conventional non-learning based baseline methods.

I. INTRODUCTION

The paper targets UAV trajectory design for IoT data collection in practical urban environments, where conventional optimization relies on simplified channel models and perfect CSI. It reformulates completion-time minimization as an MDP and proposes TD3-TDCTM with information enhancement and reward-shaping techniques.

  • Motivation: Existing trajectory designs often use simplified LoS or statistical channel models and assume perfect CSI, limiting their representation of practical urban environments.Actual LoS links depend on physical building blockages, while offline optimization requires accurate channel information.
  • Problem Formulation: The paper formulates UAV-assisted IoT data collection as a completion-time minimization problem under throughput and flight-movement constraints.The formulation concerns multiple IoT ground nodes in an urban scenario.
  • Method: The original trajectory problem is reformulated as an equivalent MDP so deep reinforcement learning can obtain a near-optimal solution in the complex environment.The MDP formulation addresses the intractability of conventional convex optimization under practical building distributions and imperfect CSI.
  • Method: TD3-TDCTM uses service statuses, UAV position, and merged pheromone information to continuously and adaptively control UAV movement with limited information exchange.Merged pheromone represents UAV and environmental state information and is adopted as a reward reference.
  • Evaluation: Information enhancement, reward shaping, dimension-spread, and done-or-terminated techniques are introduced to facilitate convergence, and simulations evaluate their effectiveness.The paper also compares TD3-TDCTM with multiple baseline methods and studies hyperparameter settings.

II. SYSTEM MODEL

The system models a UAV collecting delay-tolerant data from distributed IoT nodes in a 3D region with building-aware ground-to-air channels. LoS availability is determined from simulated building geometry, with distinct fading models for LoS and NLoS links.

  • System Setup: A UAV with a single omni-directional antenna collects delay-tolerant data from K randomly distributed IoT nodes in a D × D m2 region.The UAV position is q(t) = [x_t, y_t, H], while node k is positioned at w_k = [x̄_k, ȳ_k, 0].
  • Channel Model: The ground-to-air channel uses large-scale and small-scale fading calculated from a simulated 3D map containing buildings as propagation scatterers.This contrasts with simplified LoS and probabilistic LoS channel models.
  • Channel Model: Building density and height are characterized using the land-area coverage ratio α, building density β, and Rayleigh-distributed building-height mean λ.These parameters describe the urban environment used in the channel model.
  • Channel Model: LoS availability is determined geometrically by checking whether buildings block the direct UAV-to-node communication link.The resulting large-scale fading depends on the UAV-node distance, carrier frequency, and LoS or NLoS propagation losses.
  • Channel Model: Small-scale fading is modeled as Rayleigh for NLoS links and Rician with a 15 dB factor for LoS links.Doppler effects from UAV mobility are assumed to be estimated and compensated at the receiver.

III. PROPOSED DRL-BASED TD3-TDCTM SCHEME

The proposed scheme reformulates UAV trajectory optimization as an MDP for deep reinforcement learning and applies TD3-TDCTM to minimize mission completion time. The design also includes convergence-facilitating techniques.

  • MDP Reformulation: The trajectory optimization problem is reformulated as an MDP, the standard representation used by the DRL framework.This reformulation supports applying DRL to the UAV trajectory design problem.
  • TD3-TDCTM: TD3-TDCTM is designed with three convergence-facilitating tricks for mission completion-time minimization.The supplied passage identifies the scheme and its convergence objective but does not enumerate the three tricks.

A. Problem Formulation

The paper formulates UAV data collection as a mission-completion-time minimization problem with discretized flight and hovering, throughput constraints, and location-dependent wireless channels.

  • A. Problem Formulation: Each time step combines fixed flight time with hovering time for data collection, and the mission ends when all IoT-node data has been collected.The hovering duration is determined by the maximum upload duration among served nodes.
  • A. Problem Formulation: The UAV moves using average speed and horizontal direction variables constrained by maximum speed and the operating region.The movement distance is defined as m_n = δ_ftυ_n.
  • A. Problem Formulation: IoT nodes awaken only when the UAV provides sufficient SNR, and at most K_up nodes are simultaneously served through OFDMA.When more nodes qualify than the access limit, nodes with the largest SNR are selected.
  • A. Problem Formulation: The objective is to minimize mission completion time while satisfying data-service, movement, and operational constraints.The formulation models completion through cumulative service indicators for each IoT node.
  • A. Problem Formulation: The UAV trajectory problem is a mixed-integer non-convex optimization whose channel conditions depend on instantaneous UAV, IoT-node, and building locations.These dependencies make traditional optimization methods difficult to apply in the considered environment.

B. Preliminaries

The preliminaries introduce reinforcement learning and TD3 as the basis for continuous UAV trajectory control, emphasizing mechanisms that reduce value overestimation and update instability.

  • B. Preliminaries: Reinforcement learning models an agent interacting with an environment through states, actions, transitions, and rewards to learn a reward-maximizing policy.The return aggregates rewards using a discount factor γ.
  • B. Preliminaries: TD3 addresses DDPG overestimation by learning twin Q-functions and using the smaller target value during critic updates.This clipped double Q-learning mechanism limits the effect of overestimated value estimates.
  • B. Preliminaries: TD3 delays policy and target-network updates relative to critic updates to reduce value-estimation variance and stabilize policy learning.The critic is allowed to become more stable before influencing actor updates.
  • B. Preliminaries: TD3 smooths target policies by adding clipped random noise to target actions, reducing sensitivity to narrow value-estimate peaks.The noise is clipped so the target remains close to the original action.

C. MDP Formulation

The MDP formulation represents UAV control through node-service information, UAV position, and merged pheromone state, with continuous movement actions and shaped rewards for faster completion.

  • C. MDP Formulation: The TD3-TDCTM training loop stores interaction transitions, samples replay batches, updates twin critics and the actor, and terminates when the task or maximum-step condition is reached.The algorithm adds completion rewards and records a termination flag when all nodes are served.
  • C. MDP Formulation: The state combines node coverage indicators, serving flags, UAV coordinates, and merged pheromone, giving a 2K + 3-dimensional representation.The pheromone merges environment and UAV information to enhance decision efficiency.
  • C. MDP Formulation: The action consists of the UAV’s next-step horizontal direction and flying speed, making trajectory optimization a continuous-control problem.The direction lies in (0, 2π] and speed lies in [0, υ_max].
  • C. MDP Formulation: The reward-shaping mechanism converts the original sparse reward into dense feedback using pheromone changes and a smoother tanh-like function.The shaped reward supplies intermediate exploration feedback.
  • C. MDP Formulation: A remaining-time reward at task completion encourages the UAV to finish data collection as soon as possible.The reward is defined as N_re = N_max − n at the completion step.

D. TD3-Based UAV Trajectory Design

TD3-TDCTM augments TD3 with merged pheromone information, state-dimension spreading, and separate done and terminated handling to improve UAV trajectory learning.

  • D. TD3-Based UAV Trajectory Design: The proposed TD3-TDCTM algorithm uses TD3 as the continuous-control foundation for UAV trajectory design and adds training-stabilization techniques.Its pseudocode is provided as Algorithm 1.
  • D. TD3-Based UAV Trajectory Design: Merged pheromone is added to the state as ant-colony-inspired information representing data collected from active IoT nodes and transferred to the UAV.This additional state information is intended to enhance learning efficiency.
  • D. TD3-Based UAV Trajectory Design: A dimension-spread network makes location and pheromone dimensions comparable to the larger set of IoT coverage dimensions.The technique addresses imbalance among state dimensions.
  • D. TD3-Based UAV Trajectory Design: The algorithm distinguishes artificial episode termination at the maximum step from genuine task completion when constructing critic targets.Only task completion sets the future Q value to zero; conflating the cases can cause critic oscillation and performance degradation.

E. Flow of the Proposed Algorithm

The TD3-TDCTM algorithm initializes actor–critic networks, explores through noisy actions, and updates networks from replayed experience to optimize UAV trajectory completion time.

  • E. Flow of the Proposed Algorithm: TD3-TDCTM initializes the critic and actor networks and their target networks before training.The target networks are initialized in the same manner as the original networks.
  • E. Flow of the Proposed Algorithm: During exploration, the actor selects actions from the current state and adds decaying Gaussian noise to balance exploration.The environment is initialized at each episode and returns an initial state and termination flag.
  • E. Flow of the Proposed Algorithm: The algorithm handles completed missions by terminating the episode without further reward signals.Completion ends the data-collection task and the corresponding episode.
  • E. Flow of the Proposed Algorithm: Boundary violations incur a pheromone penalty and cancel the corresponding UAV movement, while early completion receives a positive remaining-time reward.The penalty is assigned to the pheromone and the UAV stays put for the invalid movement.
  • E. Flow of the Proposed Algorithm: The actor and critic networks are updated by storing collected samples in a replay buffer and sampling mini-batches for training.This update process follows the classical TD3 structure.

IV. SIMULATION RESULTS

The simulation-results section evaluates the performance of the proposed TD3-TDCTM algorithm through numerical experiments.

  • IV. SIMULATION RESULTS: Numerical results are conducted to evaluate the performance of the proposed TD3-TDCTM algorithm.
  • IV. SIMULATION RESULTS: The evaluation is presented as a numerical-results study of the proposed algorithm.
  • IV. SIMULATION RESULTS: The section focuses on assessing TD3-TDCTM performance rather than introducing a new algorithmic component.

A. Simulation Settings

The simulations use a dense 3D urban environment with 40 IoT nodes, compare TD3-TDCTM against conventional baselines, and vary UAV altitude and trajectory conditions.

  • A. Simulation Settings: The simulation environment models a 1,000×1,000 m2 urban area with dense high-rise buildings and building heights clipped to 10–50 m.The building realization uses the stated statistical-model parameters.
  • A. Simulation Settings: The communication setup includes PTx = 10 dBm, PN = −75 dBm, ρth = 0 dB, and Dfile = 10 Mbits, with at most six served nodes per time step.
  • A. Simulation Settings: The experiments use 40 IoT nodes and visualize both 2D and 3D UAV trajectories produced by TD3-TDCTM.The trajectory figure marks served nodes and the UAV path.
  • A. Simulation Settings: The experiments examine average mission completion time across different UAV altitudes and use fixed altitude H as an optimization parameter.
  • A. Simulation Settings: The actor and critic networks use two-layer fully connected architectures with 400 neurons, implemented in PyTorch on three NVIDIA 2080TI GPUs.
  • A. Simulation Settings: The study compares TD3-TDCTM with three conventional baseline methods: scan, ACO-based, and RRT-based approaches.The scan strategy covers the target region, while ACO and RRT construct node-serving routes.

B. Result and Analysis

The proposed TD3-TDCTM algorithm learns UAV trajectories for efficient data collection in 3D urban environments, balancing LoS coverage, completion time, energy use, and imperfect CSI. Across the reported experiments, it converges reliably and outperforms conventional trajectory strategies.

  • Completion time and energy: For 25 IoT nodes, TD3-TDCTM saves 54.2 s, 84.9 s, and 300.3 s versus RRT, ACO, and Scan, respectively, in average mission completion time.The comparison is reported in Fig. 6(a).
  • Completion time and energy: For 25 IoT nodes, TD3-TDCTM reduces average flight energy consumption by 43.6%, 54.9%, and 81.2% versus RRT, ACO, and Scan, respectively.The comparison is reported in Fig. 6(b).
  • Ablation and design choices: Reward shaping enables convergence at 40 IoT nodes, while information enhancement reduces convergence time from about 5000 to about 2000 episodes.Without dimension-spread or done-or-terminated, Q-loss fluctuations are relatively large.
  • Convergence and robustness: After about 2000 training episodes, accumulated reward becomes smooth and stable across different IoT-node counts, indicating convergence and robustness.The training behavior is summarized in Fig. 8.
  • LoS coverage: The designed trajectories maintain an average LoS coverage ratio above 85%, enabling more IoT nodes to be covered simultaneously along the UAV route.This result is reported in Fig. 9.
  • Hyperparameter selection: With buffer size R = 1 × 10^5, increasing neuron numbers from 200 to 400 decreases average mission completion time from 71.49 to 70.52.The reported setting is identified as the optimal hyperparameter combination in Table IV.

V. CONCLUSION

The paper presents TD3-TDCTM for UAV trajectory design in practical 3D urban environments with imperfect CSI. The algorithm uses service status, UAV position, and merged pheromone information to adapt movement and minimize completion time under throughput and flight constraints, with numerical gains over optimization-based methods.

  • Conclusion: TD3-TDCTM designs efficient UAV trajectories for IoT data collection in 3D urban environments with imperfect CSI.The algorithm targets completion-time minimization under flight and throughput constraints.
  • Conclusion: The algorithm takes IoT-node service statuses, UAV position, and merged pheromone information as state inputs for adaptive movement decisions.The merged pheromone represents UAV and environmental state information for reward design.
  • Conclusion: Numerical results show significant performance gains over existing optimization-based methods.
Loading 2107.11015v1…