RIS-assisted UAV Communications for IoT with Wireless Power Transfer Using Deep Reinforcement Learning

I. INTRODUCTION

• UAVs can extend wireless coverage and collect remote sensor data, but limited flight time and onboard processing constrain their use in difficult missions.Source: §I · p. 1
• IoT devices harvest energy in a downlink WPT phase for uplink WIT, while DRL agents learn through interaction without requiring pre-collected training data.Source: §I · p. 1

A. State-of-the-art

• Prior RIS research addresses beamforming and phase optimization, while RIS-assisted energy-harvesting protocols optimize transmit beamforming and RIS scheduling or phases for information-rate objectives –,,.Source: A. State-of-the-art · p. 2
• Prior UAV–RIS work reports stronger received signals, lower power consumption, and extended UAV flight time; studies optimize trajectory and RIS phases for rate or decoding-error objectives –[14].Source: A. State-of-the-art · p. 2
• DRL has been applied to RIS-assisted wireless optimization, including UAV trajectory and phase control for sum rate and fairness, secure communications, and received-SNR maximization [14], [20]–.Source: A. State-of-the-art · p. 2

B. Contributions

• The proposed harvest-then-transmit system pairs downlink energy harvesting with uplink data transmission, considering hovering and mobile UAVs with optimized scheduling and RIS phases.Source: B. Contributions · p. 2
• The optimization is formulated as an MDP, with DDPG and PPO proposed to solve the sum-rate maximization problem.Source: B. Contributions · p. 2
• RIS support establishes a better connection and significantly improves overall performance.Source: B. Contributions · p. 3

• A single-antenna UAV serves N randomly distributed ground IoT devices, while a wall-mounted RIS with K elements reflects signals to mitigate attenuation and path loss from surrounding obstacles.Source: §II · p. 3
• Communication uses two phases: the UAV transfers energy during the downlink for τT, then IoT devices transmit information to the UAV during the uplink for (1−τ)T, with T normalized to 1.Source: §II · p. 3
• The system model depicts UAV-assisted IoT wireless communications supported by a RIS.Source: Figure 1 · p. 3

A. Channel model

• The direct UAV–IoT link uses Rayleigh small-scale fading, modeled as complex Gaussian CN(0, 1), with a reference-distance path loss and exponent.The model denotes the direct-link channel gain and identifies β and κ1 as the reference-distance path loss and path-loss exponent, respectively.Source: A. Channel model · p. 3
• The UAV–RIS link is modeled as an air-to-air channel dominated by line-of-sight propagation, with an array response parameterized by angle of arrival.Its path-loss exponent, antenna separation, and carrier wavelength are also specified.Source: A. Channel model · p. 3
• The RIS–IoT link follows Rician fading, combining a deterministic line-of-sight component with a Rayleigh non-line-of-sight component.The model specifies a Rician factor and a path-loss exponent for this link.Source: A. Channel model · p. 3

B. Power transfer phase

• The signal received by each IoT device combines the UAV’s direct transmission with the signal reflected by the RIS.The UAV transmits symbol x at power P0, and the received signal includes complex Gaussian noise with variance α2.Source: B. Power transfer phase · p. 3
• The RIS phase-shift matrix Φt is diagonal, with each element’s phase shift φt_k ranging from 0 to 2π.The matrix contains the phase shifts of the RIS elements at time step t.Source: B. Power transfer phase · p. 3
• During the WPT phase, the UAV transfers energy to IoT devices over time span τ_t, and received power is characterized using efficiency η.The passage introduces a received-power expression for each device at time step t but does not provide its formula here.Source: B. Power transfer phase · p. 4

• IoT devices have no fixed energy source and use all harvested energy for the wireless information transmission phase.Source: C. Information transmission phase · p. 4
• The section formulates the UAV’s received SINR from each device and the devices’ sum-rate at each time step, with B denoting bandwidth.Source: C. Information transmission phase · p. 4
• The objective is to maximize sum-rate by optimizing the RIS phase-shift matrix, UAV trajectory, and energy-harvesting time, subject to flying-area and speed constraints.Source: C. Information transmission phase · p. 4

• The UAV hovers at the cluster center to extend operating time by reducing movement energy, trading off energy use against achievable sum-rate.Source: §III · pp. 4–5
• The MDP state comprises direct-plus-reflected channels for the IoT devices, and its actions select the energy-harvesting time and RIS phase shifts, with reward defined by total sum-rate.Source: §III · p. 4
• DDPG seeks a policy that jointly selects the energy-harvesting time and RIS phase shifts to maximize sum-rate for the hovering UAV’s downlink power-transfer and uplink information-transmission protocol.The RIS adjusts its phases during energy transfer and keeps them unchanged while devices transmit during the remaining time.Source: §III · p. 4
• DDPG uses actor and critic networks for policy selection and action evaluation, with experience replay and target networks to improve convergence speed and avoid excessive calculations.Training stores state-action-reward-next-state transitions in a finite replay buffer and updates the networks using stochastic gradient descent on mini-batches.Source: §III · p. 4

REINFORCEMENT LEARNING

• The joint optimization targets maximum expected reward by adjusting the UAV trajectory, IoT energy-harvesting schedule, and RIS phase shifts.The scheme is motivated by the UAV’s short flying time.Source: REINFORCEMENT LEARNING · p. 5
• The action space includes UAV velocity and flying angle, EH scheduling time, and the RIS phase shifts.Source: REINFORCEMENT LEARNING · p. 5
• After each action, the UAV moves to a new state, with its next position subject to environmental noise.Source: REINFORCEMENT LEARNING · p. 5
• UAV motion must remain inside the flying zone, keep velocity at or below vmax, and use a flying angle in [0, 2π].Source: REINFORCEMENT LEARNING · p. 5
• The agent iteratively uses received rewards and achieved CSI to update its policy and select UAV velocity, direction, EH scheduling, and RIS phases.Source: REINFORCEMENT LEARNING · p. 5
• The paper presents a DDPG-based technique for joint trajectory, EH-time, and RIS phase-shift optimization in Algorithm 1.M and T denote the maximum numbers of episodes and time steps, respectively.Source: REINFORCEMENT LEARNING · p. 5

V. PROXIMAL POLICY OPTIMIZATION TECHNIQUE FOR JOINT TRAJECTORY, EH TIME AND THE PHASE SHIFT OPTIMIZATION

• PPO trains a policy by stochastic gradient descent on mini-batches of transitions to seek a reward-maximizing policy for the joint optimization problem.The paper identifies the transition batch as (s_i, a_i, r_i, s_i+1).Source: §V · p. 5
• The PPO objective uses the current-to-previous policy probability ratio and an advantage function.Source: §V · p. 6
• A clipping surrogate is used to mitigate excessive policy modification during training [27].Source: §V · p. 6

VI. SIMULATION RESULTS

• The comparisons include hovering and mobile UAV variants, random RIS phase selection, random energy-harvesting time, and operation without RIS.Hovering methods optimize energy-harvesting time and RIS phase shifts; mobile methods jointly optimize trajectory, energy-harvesting scheduling, and RIS phase shifts.Source: §VI · p. 6
• For the hovering UAV across IoT-device counts, H-PPO has higher expected throughput than H-DDPG, RSS-HDDPG, and the two without-RIS techniques.The comparison uses K = 20 RIS elements and averages results over 1000 episodes per scheme.Source: §VI · pp. 6–7, Figure 2 · p. 7
• With N = 10, H-PPO’s sum-rate rises from 2.0 to 2.8 bits/s/Hz as RIS elements increase; H-DDPG slightly exceeds RSS and without-RIS schemes.Without RIS, PPO and DDPG reach similar results when optimizing only energy-harvesting time; random RIS phases can reflect signals away from destinations.Source: Figure 3 · p. 7, §VI · p. 7
• In the mobile-UAV comparison across IoT-device counts, F-DDPG achieves around 3.8 bits/Hz, while F-PPO becomes trapped at a local optimum.The passage attributes F-PPO’s weaker result to its on-policy approach and less random exploration during training.Source: Figure 4 · p. 7, §VI · p. 7
• F-DDPG outperforms the compared schemes at around 3.8 bits/s/Hz, with F-PPO next; joint optimization improves sum-rate over hovering and REH schemes.The joint optimization covers the UAV trajectory, IoT energy-harvesting time, and RIS phase shift.Source: §VI · p. 7, Figure 5 · p. 8

VII. CONCLUSION

• The proposed system combines downlink power transfer with uplink information transmission in RIS-assisted UAV communications.The design leverages UAV mobility and RIS flexibility for practical applications.Source: §VII · p. 8
• Two DRL techniques jointly optimize the UAV trajectory, IoT energy-harvesting time scheduling, and RIS phase-shift matrix to maximize network throughput.Source: §VII · p. 8
• The learned DRL systems can handle dynamic environments and satisfy some power restrictions and processing-time requirements in RIS-assisted UAV communications.Source: §VII · p. 8
• Future work will extend the study to distributed models and cooperative communications with multiple UAVs.Source: §VII · p. 8