PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid PPO-Based Multi-Objective Optimization for UAV-mounted STAR-RIS with Energy Harvesting

Anas Hussin, Khaled M. Rabie, Mahmoud M. Salim, Ali H. Muqaibel, Louai A. Al-Awami

arXiv:2610.28807v1eess.SYcs.SY

TL;DR

UAV-mounted STAR-RIS systems jointly face communication, energy-harvesting, and mobility decisions under coupled objectives and constraints. The paper proposes a hybrid PPO framework with a two-stage reward, reporting faster convergence and more stable training than PPO and TD3 while balancing communication and harvesting objectives across tested conditions.

Abstract

from arXiv · show

This paper investigates a multi-objective optimiza tion framework for UAV-mounted reconfigurable intelligent sur face (STAR-RIS)-assisted Sixth Generation (6G)-enabled Internet of Things (IoT) systems. The considered framework jointly supports communication enhancement and radio-frequency (RF) energy harvesting (EH) through adaptive STAR-RIS transmission and reflection control. A weighted multi-objective optimization problem is formulated to jointly optimize the base station (BS) beamforming vectors, STAR-RIS transmission and reflection coefficients, RF power-splitting factor, and UAV trajectory under quality-of-service (QoS), power, and mobility constraints. Due to the strong coupling among the communications, EH, and UAV mobility variables, the resulting problem is highly non convex and difficult to solve using conventional optimization methods. To address this challenge, a hybrid proximal policy optimization (H-PPO) framework is proposed. The developed hybrid actor-critic architecture combines a categorical policy for discrete UAV movement decisions with Beta-distributed policies for continuous communication-control variables. In addition, a two-stage reward mechanism is introduced to enforce QoS feasibility while balancing system sum rate, energy efficiency, and harvested energy. Simulation results demonstrate that the proposed H-PPO framework achieves faster convergence and more stable training behavior than proximal policy optimization (PPO) and twin delayed deep deterministic policy gradient (TD3) baselines. In particular, H-PPO achieves approximately 66% and 60% faster convergence than PPO and TD3, respectively, while maintaining more stable training behavior. Furthermore, the proposed framework effectively balances communication performance and EH under different QoS requirements, STAR RIS sizes, transmit power levels, and user densities.

I. INTRODUCTION

The introduction motivates joint UAV-mounted STAR-RIS communication and RF energy harvesting as a coupled optimization problem, then presents a hybrid PPO framework and reports improved convergence and training stability against PPO and TD3. Its motivation spans limitations in coverage, joint system design, and continuous-only control formulations.

1) STAR-RIS-Assisted Communications:

  • Traditional RIS reflects signals on one side, while STAR-RIS transmits and reflects to provide full-space coverage and improve spatial-resource utilization.The introduction also notes that fixed terrestrial STAR-RIS deployments face challenges with adaptable coverage and changing environments.Source: §I · p. 1

3) EH in Wireless Systems:

  • RF energy harvesting allocates received energy between harvesting and communication, creating a trade-off that couples system design and challenges conventional optimization.The introduction identifies increasing harvesting as reducing power available for signal reception.Source: §I · p. 2

2) UAV-Assisted Wireless Systems:

  • Prior UAV-assisted systems use mobility to improve communication, but many rely on conventional single-sided RIS or omit reconfigurable surfaces.This leaves the full-space coverage of aerial STAR-RIS underused in those designs.Source: 2) UAV-Assisted Wireless Systems · p. 2

4) DRL-Based Optimization for RIS Systems:

  • Most existing DRL approaches treat control as fully continuous, although UAV movement is naturally discrete or categorical and may be learned less efficiently that way.The cited review identifies this as a limitation of prior control formulations.Source: 4) DRL-Based Optimization for RIS… · p. 2

B. Motivations and Contributions

  • Existing studies often optimize subsets of system components, omit one or more of UAV mobility, STAR-RIS, and EH, or use single-objective or partially coupled designs.The introduction also identifies limited use of structured hybrid control in existing DRL approaches.Source: B. Motivations and Contributions · p. 3
  • The proposed unified framework jointly optimizes STAR-RIS configuration, UAV mobility, and energy harvesting under a multi-objective formulation.The contribution model includes beamforming, transmission and reflection coefficients, power splitting, and practical QoS, power, hardware, and mobility constraints; its hybrid PPO policy pairs categorical UAV actions with Beta-distributed continuous actions, while a two-stage reward first enforces QoS feasibility and then balances objectives.Source: B. Motivations and Contributions · p. 3
  • H-PPO outperforms PPO and TD3 in convergence speed, training stability, and overall multi-objective performance for UAV-mounted STAR-RIS systems.The reported evaluation jointly concerns communication performance and energy harvesting.Source: B. Motivations and Contributions · p. 3

II. SYSTEM MODEL

The system model uses a UAV-mounted STAR-RIS to serve transmission-side and reflection-side users through cascaded links. It defines the BS signal and channel geometry, with UAV position affecting the BS–STAR-RIS link.

II. SYSTEM MODEL

  • A UAV-mounted STAR-RIS serves transmission-side and reflection-side users through cascaded BS–STAR-RIS–user links, with direct BS–user links assumed unavailable due to blockage.The STAR-RIS has N passive elements, and the UAV flies at fixed altitude.Source: §II · p. 3

A. BS Signal Model

  • The BS uses linear precoding, with each user's information symbol weighted by its beamforming vector to form the transmitted signal.The beamforming vectors form the transmit covariance matrix, subject to the maximum available BS transmit-power budget.Source: A. BS Signal Model · p. 4

1) BS to STAR-RIS:

  • The BS–STAR-RIS channel model uses distance and elevation angle in its propagation description, and UAV position affects both quantities and the instantaneous link quality.The model combines large-scale fading with small-scale fading whose entries follow independent circularly symmetric complex Gaussian distributions.Source: 1) BS to STAR-RIS · p. 4

2) STAR-RIS to UT:

The STAR-RIS-to-user links use separate transmission- and reflection-side channel models, with Rician fading for the small-scale component. The STAR-RIS also splits incident RF power between harvesting and communication, creating a throughput–energy trade-off.

2) STAR-RIS to UT:

  • Transmission- and reflection-side users are modeled with separate STAR-RIS-to-user channels and path loss based on their distance from the UAV-mounted surface.The passage assigns separate channel vectors to each user group and specifies a path-loss coefficient and exponent.Source: 2) STAR-RIS to UT · pp. 4–5
  • The STAR-RIS-to-user small-scale fading follows a Rician model combining deterministic LoS and random scattering components.The Rician factor governs their relative strengths: larger values indicate stronger LoS, while smaller values imply scattering-dominated channels.Source: 2) STAR-RIS to UT · p. 5

C. Transmission and EH Model

  • The power-splitting coefficient ρ(t) allocates incident RF power to harvesting, while 1 − ρ(t) is used for communications.STAR-RIS reflection and transmission coefficients also obey an amplitude energy-conservation constraint.Source: C. Transmission and EH Model · p. 5
  • Increasing ρ(t) raises harvested energy but reduces effective communication gain, so maximizing harvesting alone can degrade throughput.Harvested energy also increases with incident RF power.Source: C. Transmission and EH Model · p. 5

D. UAV Mobility and Propulsion Energy Model

The UAV flies in the horizontal plane at fixed altitude, subject to a maximum-speed constraint. The section also models rotary-wing propulsion power and energy over each time slot.

D. UAV Mobility and Propulsion Energy Model

III. PERFORMANCE METRICS AND PROBLEM FORMULATION

The section defines throughput, power consumption, and energy efficiency as performance measures, then formulates a weighted objective over throughput, energy efficiency, and harvested energy. The optimization jointly selects communication and UAV controls subject to QoS, power, STAR-RIS, and mobility constraints.

A. Performance Metrics

  • The SNR-derived achievable throughput depends jointly on BS beamforming, STAR-RIS configuration, channel conditions, and UAV position.The system bandwidth is denoted by B.Source: A. Performance Metrics · p. 5
  • Instantaneous BS transmit power is defined from the users' beamforming vectors, and harvested RF power can offset system energy expenditure in the net-power model.Source: A. Performance Metrics · p. 5
  • Instantaneous energy efficiency is defined as a performance metric for the considered system.Source: A. Performance Metrics · p. 6

B. Problem Formulation

  • The weighted objective jointly maximizes system throughput, energy efficiency, and harvested energy while meeting every user's QoS requirement.The decision variables are BS beamforming, STAR-RIS transmission and reflection coefficients, RF power splitting, and UAV trajectory.Source: B. Problem Formulation · p. 6
  • The constraints enforce minimum throughput for both user groups, feasible power splitting and BS transmit power, STAR-RIS amplitude and phase limits, and UAV mobility and service-region boundaries.Source: B. Problem Formulation · p. 6
  • The λ weights set the original objective's priorities, whereas the RL reward weights combine min-max-rescaled metrics to avoid dominance from differing numerical scales.Source: B. Problem Formulation · p. 6

IV. PROPOSED HYBRID PPO-BASED METHODOLOGY

The paper addresses the coupled, nonconvex control problem with a PPO-based hybrid actor-critic that learns through environment interaction. Its state represents channel conditions and UAV position, while its policies distinguish discrete movement from continuous communication controls.

IV. PROPOSED HYBRID PPO-BASED METHODOLOGY

  • Strong coupling among communication, energy-harvesting, and mobility variables makes the optimization nonconvex and conventional iterative methods computationally expensive for online control.Time-varying user positions and UAV mobility also produce time-varying channel conditions.Source: §IV · p. 6
  • The hybrid PPO actor-critic uses a categorical policy for discrete UAV movement and Beta distributions for continuous STAR-RIS, beamforming, and power-splitting controls.The policy learns through interaction without explicitly decomposing the original optimization problem.Source: §IV · p. 6

A. Markov decision process(MDP) Formulation

  • The problem is represented as an MDP in which the agent observes a state, selects an action, receives a reward, and transitions to the next state.The MDP tuple includes state, action, transition distribution, reward, and discount factor.Source: A. Markov decision process(MDP)… · p. 6

1) State Space:

  • The state includes channel representations and normalized UAV position, giving the agent information about propagation conditions and UAV geometry.Source: 1) State Space · p. 6

2) Action Space:

The action space combines discrete UAV movement with continuous communication controls, while the reward first enforces QoS feasibility and then balances normalized objectives. A shared actor-critic representation supports separate policies for the two action types and state-value estimation.

2) Action Space:

  • The hybrid action pairs five discrete UAV movement choices with continuous communication controls, including STAR-RIS coefficients, beamforming variables, and RF power splitting.The discrete movement choices are left, right, forward, backward, and stay.Source: 2) Action Space · p. 7

3) Reward Function:

  • The reward first penalizes QoS violations, then evaluates feasible actions using normalized sum rate, energy efficiency, and harvested energy.The formulation uses a bounded, smooth QoS penalty and objective weights to balance metrics with different scales.Source: 3) Reward Function · p. 7

B. Hybrid Actor-Critic Network Architecture

  • A shared feature extractor feeds separate policy and value branches, providing a common latent representation for control and state-value estimation.The value network estimates the state-value function and provides a baseline for variance reduction and policy updates.Source: B. Hybrid Actor-Critic Network… · pp. 7–8
  • A categorical policy models discrete UAV movement, while independent Beta distributions model continuous communication controls within [0, 1].The hybrid log-probability and entropy combine contributions from the discrete and continuous policies.Source: B. Hybrid Actor-Critic Network… · pp. 7–8

C. Rollout Collection

The agent gathers on-policy transitions through fixed-horizon interaction, then uses generalized advantage estimation to compute advantages and return targets for policy optimization.

C. Rollout Collection

  • At each rollout step, the agent samples discrete and continuous actions, executes the hybrid action, and stores the resulting transition in a buffer.Data collection alternates with policy optimization over collected on-policy samples.Source: C. Rollout Collection · p. 8

D. Generalized Advantage Estimation

  • Generalized advantage estimation recursively computes advantages from temporal-difference residuals using the discount factor and GAE parameter.The return target is the advantage estimate plus the state-value estimate.Source: D. Generalized Advantage Estimation · p. 8
  • Advantages are normalized within each rollout using their sample mean and standard deviation to support optimization stability.The return target is formed before this normalization step.Source: D. Generalized Advantage Estimation · p. 8

E. PPO-Based Policy Update

The policy-update procedure uses PPO's clipped objective alongside value and entropy terms, with training performed in mini-batches and stopped early when approximate KL divergence exceeds a threshold.

E. PPO-Based Policy Update

  • The PPO clipped surrogate objective limits excessively large policy updates while preserving learning progress.The clipping coefficient controls the objective's clipping behavior.Source: E. PPO-Based Policy Update · p. 8
  • Each update recomputes log-probabilities, entropy, and values, then combines clipped-policy, value-loss, and entropy terms to update network parameters.The value-loss and entropy-bonus coefficients weight their respective terms.Source: E. PPO-Based Policy Update · p. 8
  • Training uses Adam with gradient clipping and breaks out of update epochs when approximate KL divergence exceeds a threshold.The trained hybrid policy is output after the training iterations.Source: E. PPO-Based Policy Update · p. 8

F. Computational Complexity

The paper characterizes H-PPO's computational cost in terms of network and action-space dimensions and evaluates it against PPO and TD3 using simulation parameters and algorithm hyperparameters.

F. Computational Complexity

  • H-PPO's per-step forward cost depends on state, hidden-layer, and action-space dimensions, while training cost also accounts for rollout length and PPO update epochs.The network includes a shared extractor, categorical movement head, Beta-distribution control heads, and a value head.Source: F. Computational Complexity · pp. 8–9
  • The hybrid-policy overhead scales linearly with discrete and continuous action dimensions, and deployment requires one forward pass through the trained policy.The paper concludes that the trained policy is suitable for online UAV-mounted STAR-RIS control.Source: F. Computational Complexity · p. 9

V. SIMULATION RESULTS AND DISCUSSION

  • Simulations examine convergence, QoS requirements, STAR-RIS element counts, BS transmit power, user counts, and reward-weighting effects, with agents trained for 10^6 interaction steps.Simulation parameters and DRL hyperparameters are summarized in Tables III and IV.Source: §V · p. 9, Table III · p. 9, Table IV · p. 9

A. Baseline Algorithms

  • PPO provides a clipped-objective on-policy comparison with relaxed continuous UAV actions, while TD3 represents an off-policy deterministic baseline with a structurally different algorithm.The PPO comparison shares H-PPO's optimizer and hyperparameter family; TD3 broadens evaluation across on-policy and off-policy paradigms.Source: A. Baseline Algorithms · p. 9
  • Fig. 3 presents training reward versus training steps for the different DRL algorithms.The caption identifies the plotted quantities but does not state a result.Source: Figure 3 · p. 10

B. Convergence Behavior

H-PPO converges faster and more stably than PPO and TD3, while its performance across system settings reflects trade-offs among communication, energy efficiency, and harvested energy. The conclusion presents it as a joint optimization framework for UAV mobility, STAR-RIS control, and RF energy harvesting.

B. Convergence Behavior

  • H-PPO stabilizes at rewards of 65–75 after about 10^5 steps, versus PPO’s 30–40 after 3×10^5 and TD3’s 35–45 after 2.5×10^5 steps.H-PPO exits negative rewards within approximately 2×10^4 steps; PPO remains there until nearly 2.2×10^5 steps, while TD3 oscillates.Source: B. Convergence Behavior · p. 10

C. Impact of key system parameters

  • As minimum-rate constraints rise, energy efficiency and sum-rate improve while harvested energy falls; H-PPO transitions more smoothly than PPO and TD3.TD3 degrades abruptly with EH collapse at high constraints, while PPO adapts later.Source: C. Impact of key system parameters · p. 10, Figure 4 · p. 10
  • Increasing STAR-RIS elements raises energy efficiency, sum-rate, and harvested energy, with close training–evaluation alignment across sizes.The reported gains are attributed to greater beamforming gains, spatial degrees of freedom, and RF-signal capture.Source: C. Impact of key system parameters · pp. 10–11, Figure 5 · p. 11
  • Higher transmit power increases sum-rate and harvested energy, but energy efficiency eventually declines as power consumption grows.The learned H-PPO policy adjusts control variables in response to this trend.Source: C. Impact of key system parameters · p. 11, Figure 6 · p. 11
  • With more users, energy efficiency and sum-rate increase while harvested energy remains nearly constant; training and evaluation remain close.The paper attributes the communication gains to multiuser diversity and says the policy prioritizes communication performance and QoS at high user density.Source: C. Impact of key system parameters · pp. 11–12, Figure 7 · p. 12
  • Increasing the EH reward weight improves harvested energy while energy efficiency and sum-rate remain almost stable when their weights are fixed.The paper describes this as improving one objective at the expense of others in the multi-objective setting.Source: C. Impact of key system parameters · p. 12

VI. CONCLUSION

  • The conclusion frames H-PPO as jointly optimizing UAV trajectory, STAR-RIS transmission and reflection, BS beamforming, and power splitting under QoS, power, and mobility constraints.Its hybrid policy combines categorical UAV movement decisions with Beta-distributed continuous communication controls, alongside a two-stage reward formulation.Source: §VI · p. 12
  • The paper identifies imperfect CSI, multi-UAV cooperation, distributed DRL, and hybrid RF/renewable energy harvesting as future research directions.Source: §VI · p. 12
Loading 2610.28807v1…