PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine Learning Empowered Trajectory and Passive Beamforming Design in UAV-RIS Wireless Networks

Xiao Liu, Yuanwei Liu, Yue Chen

arXiv:2010.02749v1eess.SP

TL;DR

The framework addresses UAV wireless-network weaknesses by leveraging RISs and NOMA while formulating energy minimization across network design choices. It proposes D-DQN, whose decaying learning rate balances faster training with convergence, and reports reduced UAV energy consumption, including lower consumption for RIS-NOMA than RIS-OMA.

  • Problem

    Limited UAV coverage areas motivate leveraging RISs in wireless networks, while reinforcement learning is considered against unsuitable conventional techniques.

  • Method

    The paper formulates energy minimization involving NOMA, power-allocation policy determination, and RIS integration, then combines DQN with a decaying learning rate in D-DQN.

  • Results

    The D-DQN algorithm balances accelerated training with convergence, while RIS assistance significantly reduces UAV energy consumption and RIS-NOMA consumes less energy than RIS-OMA.

  • Takeaways & Limitations

    RIS integration extends coverage and reduces energy consumption, while D-DQN supports convergence with a balance between training speed and local-optimal convergence.

Abstract

from arXiv · show

A novel framework is proposed for integrating reconfigurable intelligent surfaces (RIS) in unmanned aerial vehicle (UAV) enabled wireless networks, where an RIS is deployed for enhancing the service quality of the UAV. Non-orthogonal multiple access (NOMA) technique is invoked to further improve the spectrum efficiency of the network, while mobile users (MUs) are considered as roaming continuously. The energy consumption minimizing problem is formulated by jointly designing the movement of the UAV, phase shifts of the RIS, power allocation policy from the UAV to MUs, as well as determining the dynamic decoding order. A decaying deep Q-network (D-DQN) based algorithm is proposed for tackling this pertinent problem. In the proposed D-DQN based algorithm, the central controller is selected as an agent for periodically observing the state of UAV-enabled wireless network and for carrying out actions to adapt to the dynamic environment. In contrast to the conventional DQN algorithm, the decaying learning rate is leveraged in the proposed D-DQN based algorithm for attaining a tradeoff between accelerating training speed and converging to the local optimal. Numerical results demonstrate that: 1) In contrast to the conventional Q-learning algorithm, which cannot converge when being adopted for solving the formulated problem, the proposed D-DQN based algorithm is capable of converging with minor constraints; 2) The energy dissipation of the UAV can be significantly reduced by integrating RISs in UAV-enabled wireless networks; 3) By designing the dynamic decoding order and power allocation policy, the RIS-NOMA case consumes 11.7% less energy than the RIS-OMA case.

I. INTRODUCTION

The introduction motivates RIS-enhanced UAV networks as a way to address UAV coverage and energy limitations, and surveys machine-learning approaches for dynamic wireless-network design.

  • RISs reconfigure incident signals through reflecting elements, forming concatenated virtual LoS links that can extend coverage and reduce energy consumption.
  • UAV-enabled wireless networks provide flexible aerial base stations for temporary traffic hotspots and connectivity after disasters.
  • UAV networks remain constrained by limited coverage areas and meager energy supplies.
  • Prior work applies machine learning and optimization to UAV trajectories, resource allocation, beamforming, passive phase shifts, throughput, energy efficiency, and security.
  • Existing RIS studies address joint active and passive beamforming across MISO, OFDM, wireless-security, and millimeter-wave systems.

3) NOMA in UAV-enabled/RIS-enhanced wireless networks:

The paper frames RIS-NOMA UAV networks as dynamic energy-optimization systems in which mobility, connectivity, and decoding decisions must be coordinated over time.

  • NOMA improves spectrum efficiency and massive connectivity by exploiting users’ different channel conditions and power levels.
  • Prior RIS-NOMA work improves energy efficiency or sum rate through RIS deployment, passive beamforming, active beamforming, and convex-approximation methods.
  • UAV energy consumption is dominated by propulsion-related costs, while this paper models only energy for hovering and mobility.
  • Continuously roaming users require UAV repositioning, but RIS phase shifts can form virtual LoS links without always moving the UAV.
  • Dynamic UAV and user mobility makes decoding order depend on more than channel gains, requiring rate constraints and timeslot-by-timeslot re-determination.
  • The expanded joint search over trajectory, passive beamforming, decoding order, and power allocation motivates reinforcement learning with discrete state and action spaces.

C. Contributions

The paper proposes a RIS-UAV framework that jointly optimizes UAV movement and RIS control for energy minimization in both OMA and NOMA settings, using D-DQN for dynamic decisions.

  • The framework targets long-term network benefits by jointly controlling RISs and designing UAV trajectories to enhance connectivity and reduce UAV movement.
  • The energy-minimization formulation jointly decides UAV movement and passive RIS beamforming.
  • Both OMA and NOMA are investigated, with zero-forcing linear precoding used to eliminate multiuser interference.
  • Dynamic decoding order is considered in NOMA-RIS-UAV networks because both UAVs and mobile users move.
  • D-DQN combines DQN with a decaying learning rate to balance training speed, local convergence, and oscillation avoidance.

D. Organization and Notations

The paper presents its system model, channel and mobility modeling, energy-minimization formulation, D-DQN solution, simulations, and notation table. The modeled network uses a UAV, roaming users, and an RIS with passive reflecting elements.

  • The paper is organized around system modeling and energy-consumption minimization, D-DQN-based solution algorithms, simulation results, and conclusions.
  • System Model: The network model includes a multi-antenna UAV serving roaming single-antenna users, with an RIS deployed on a high-rise building and equipped with reflecting elements.
  • System Model: RIS phase shifts can form concatenated virtual LoS propagation between the UAV and users, reducing the UAV’s energy dissipation.
  • System Model: The received signal comprises direct UAV–MU and reflecting UAV–RIS–MU links, with RIS coefficients parameterized by phase shifts and amplitude reflection factors.
  • Channel Model: The channel and environment models account for LoS/NLoS states, path loss, UAV altitude, user distance, carrier frequency, and 3D urban-building geometry.
  • Channel Model: User mobility is modeled using location prediction from online social-network data, balancing prediction accuracy against processing and communication-resource requirements.

D. Energy Dissipation Model for the UAV

The UAV energy dissipation model separates communication-related and propulsion-related consumption, with propulsion usually dominating total energy use.

  • Energy components: UAV energy dissipation comprises communication-related and propulsion-related components.Communication energy supports radiation, signal processing, and hardware circuitry; propulsion energy supports hovering and mobility.
  • Energy components: More than 95% of total energy consumption is usually attributed to propulsion-related dissipation.
  • Propulsion model: For rotary-wing UAVs, propulsion energy consumption depends on velocity and acceleration.
  • Propulsion model: The propulsion model includes blade profile power, induced power, and parameters describing rotor geometry, air density, drag, and UAV weight.The cited parameters include s, ρ, δ, A, Ω, R, κ, W, v0, and fd.

E. Signal Model for OMA Scheme

The OMA and NOMA signal models use linear zero-forcing precoding, while NOMA additionally models clustered users, successive interference cancellation, power allocation, and dynamic decoding-order constraints.

  • Precoding: The UAV uses zero-forcing linear precoding based on the pseudo-inverse of the combined UAV-RIS-MU channel.The resulting precoding metrics are constructed from the combined channel for serving users or clusters.
  • OMA signal model: In OMA, each MU is represented by a data symbol, beamforming vector, allocated transmit power, and bandwidth-dependent instantaneous rate.
  • NOMA signal model: NOMA superimposes signals at different power levels, and users in each cluster apply SIC according to strong- and weak-user roles.The strong MU removes intra-cluster interference caused by the weak MU, whereas the weak MU directly decodes its received signal.
  • NOMA signal model: NOMA power allocation factors for the two users in a cluster sum to one.The factors are denoted α_l,a and α_l,b for the strong and weak users, respectively.
  • Interference management: ZF removes inter-cluster interference for strong users, while successful SIC removes intra-user interference; weak users still experience inter-cluster interference.
  • Dynamic decoding order: In MISO-NOMA with RISs, decoding order depends on rate constraints and RIS-modified channel responses and must be satisfied at every timeslot.Because the UAV and MUs roam continuously, the decoding order is re-determined at each timeslot.

G. Problem Formulation

The problem minimizes UAV energy consumption while maintaining wireless service quality through joint control of trajectory, RIS phase shifts, power allocation, and decoding order in a dynamic environment.

  • Objective: The optimization minimizes UAV energy consumption while guaranteeing wireless service quality for users.
  • Constraints: User data-demand constraints must be satisfied at every timeslot, alongside RIS passive-beamforming and UAV altitude constraints.
  • Constraints: The total transmit power cannot exceed the UAV’s maximum power constraint, and NOMA decoding-order constraints must ensure successful SIC.
  • Optimization challenge: The formulated problem is non-convex and dynamically changing because the UAV and MUs move continuously.
  • Optimization challenge: The expanded search space makes conventional gradient-based optimization unsuitable, motivating reinforcement learning for sequential decision-making.
  • Joint design: The decision variables jointly include RIS phase shifts, UAV movement, transmit power allocation, and dynamic decoding order.

III. PROPOSED SOLUTIONS

The proposed solution models joint RIS phase-shift control and UAV trajectory design as an MDP and applies a D-DQN algorithm to learn long-term reward-maximizing actions.

  • MDP formulation: The joint phase-shift control and trajectory-design problem is formulated as a Markov decision process before applying D-DQN.
  • MDP formulation: The MDP cycles through states, actions, rewards, and transitions, with actions influencing the next observed network state.
  • State and action design: The agent observes UAV and user coordinates plus RIS phase shifts, then selects UAV movement directions and reflecting-element phase-shift changes.
  • State and action design: Rewards or penalties reflect UAV energy consumption and network connectivity.
  • D-DQN algorithm: D-DQN selects actions to maximize long-term cumulative reward rather than reward at a particular timeslot.
  • D-DQN algorithm: The proposed algorithm uses value-based reinforcement learning because RIS phase shifts are modeled as discrete.Policy-based or actor-critic methods such as DDPG are suggested for continuous phase shifts.
  • D-DQN algorithm: The algorithm updates Q-values using replayed transitions and a target network, while its learning rate decays with episode count.
  • D-DQN algorithm: The learning rate is calculated as α(ne) = α0/1 + ηne in the stated algorithm.

1) States in the D-DQN model:

The D-DQN model represents network conditions through UAV, RIS, mobile-user, and power-allocation states, then selects control actions using rewards tied to rate satisfaction and energy consumption.

  • States: The state space includes RIS phase shifts, the UAV’s 2D coordinate, each mobile user’s 2D coordinate, and per-user transmit power.
  • Actions: The action space changes RIS phase shifts, the UAV’s movement direction and distance, and transmit power allocated to each mobile user.
  • Reward function: The reward function evaluates mobile-user transmit rates alongside the UAV’s energy consumption.
  • Reward function: Actions that reduce energy while satisfying users’ data rates receive positive rewards; actions increasing energy or violating rates receive penalties.
  • Policy behavior: The learned policy favors RIS phase-shift control over UAV repositioning unless users’ data-rate requirements cannot be satisfied.Maximizing long-term sum rewards is designed to minimize the UAV’s long-term energy consumption.
  • D-DQN training: The model uses CNN-based function approximation, experience replay, epsilon-greedy exploration, and a decaying learning rate.The decaying rate balances training speed against convergence to a local optimum and helps avoid oscillation.

C. Analysis of the Proposed Algorithm

The analysis discusses convergence of Q-learning, DQN, and D-DQN, emphasizing that decaying learning rates affect convergence speed while preserving the DQN algorithm’s convergence ability and optimality claims.

  • Convergence analysis: Conventional Q-learning is described as converging to the optimal solution, whereas DQN can converge to an optimal state when its neural network is sufficiently large.
  • Convergence analysis: The DQN-based algorithm is characterized as sub-optimal because reinforcement-learning optimality cannot be guaranteed.
  • Decaying learning rate: The decaying learning rate is intended to balance faster training, local-optimum convergence, and reduced oscillation.
  • Convergence analysis: The decaying learning rate does not affect the DQN algorithm’s convergence ability or optimality, but it influences convergence rate.
  • Computational complexity: The proposed algorithm’s complexity includes CNN computation and training complexity O(|S| · |A|), where |S| and |A| are the numbers of states and actions.

IV. SIMULATION RESULTS

Simulations evaluate convergence, RIS-assisted network performance, and RIS-NOMA versus RIS-OMA operation. The results report convergence with D-DQN, higher rates with RIS and trajectory design, and lower energy for RIS-assisted and rate-constrained designs.

  • Convergence rate: The conventional Q-learning algorithm fails to converge in the large-state UAV wireless-network problem, whereas D-DQN converges using neural-network function approximation.
  • Transmit rate: RIS assistance provides a higher achievable transmit rate at each timeslot than operation without RIS.
  • Transmit rate: Roaming users cause achievable rates to decrease as their distance from a static UAV increases, while UAV trajectory design slows this decline.
  • Energy objective: Optimizing energy consumption prioritizes limited UAV movement after users’ data-demand constraints are satisfied rather than pursuing higher throughput.
  • Energy objective: Maximizing energy efficiency achieves a higher rate but consumes far more energy than minimizing energy dissipation while satisfying all users’ data demands.
  • Energy consumption: RIS-NOMA consumes less energy than RIS-OMA, while operation without RIS consumes the most energy because the UAV performs more movement to form LoS links.

4) Impact of reflecting elements number:

The evaluation examines how reflecting elements, user load, decoding and power policies, UAV altitude, and channel modeling affect energy consumption in RIS-UAV networks. It also reports the D-DQN-based framework and its energy-saving comparison between RIS-NOMA and RIS-OMA.

  • Impact of reflecting elements number: More reflecting elements reduce the UAV's average energy consumption, whereas serving more mobile users increases it.With three clusters, the UAV consumes 10.3% more energy than with two clusters.
  • Impact of reflecting elements number: 11.7% less energy is consumed by RIS-NOMA than by RIS-OMA.The passage attributes this difference to NOMA's higher spectrum efficiency and higher mobile-user transmit rates, which make data-demand constraints more likely to be satisfied with fewer movement actions.
  • Impact of reflecting elements number: Dynamic decoding-order determination and power allocation reduce energy consumption compared with fixed policies.A fixed decoding order decreases NOMA spectrum efficiency and achievable rates, while fixed power allocation consumes more energy than optimal per-timeslot allocation.
  • Impact of reflecting elements number: RIS phase-shift design significantly reduces energy consumption compared with random phase shifts.The evaluation considers average UAV energy consumption in different cases, with both the UAV and mobile users moving.
  • Impact of reflecting elements number: At 200 m altitude, the UAV consumes more energy; higher altitude increases line-of-sight probability but also increases path loss from distance.The 3GPP channel-model and 3D-radio-map results are not identical, but their variation trends are the same.
  • Impact of reflecting elements number: The proposed framework jointly designs RIS phase shifts, UAV trajectory, dynamic NOMA decoding order, and power allocation while using D-DQN to minimize UAV energy consumption.The formulation includes propulsion-related energy consumption and satisfies each mobile user's data demand at every timeslot; D-DQN combines DQN with a decaying learning rate.
Loading 2010.02749v1…