PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconfigurable Intelligent Surface Assisted Multiuser MISO Systems Exploiting Deep Reinforcement Learning

Chongwen Huang, Ronghong Mo, Chau Yuen

arXiv:2002.10072v1cs.ITcs.LG

TL;DR

The paper addresses joint beamforming and RIS phase-shift design for multiuser MISO systems, where the optimization is non-convex and the optimal solution is unknown. It uses DRL to obtain both designs simultaneously through reward-driven interaction, achieving comparable benchmark performance while learning progressively; neural-network settings affect performance and convergence.

  • Problem

    Jointly designing BS transmit beamforming and RIS phase shifts to maximize multiuser downlink MISO sum rate is a non-convex problem with unknown optimal solution.

  • Method

    A DRL algorithm uses trial-and-error environmental interactions and instant sum-rate rewards to jointly obtain continuous beamforming and phase-shift actions, using DDPG.

  • Results

    The proposed algorithm learns progressively and achieves comparable sum-rate performance with WMMSE and FP optimization with ZF benchmarks across considered scenarios.

  • Takeaways & Limitations

    The DRL framework offers a low-complexity, scalable approach that does not require explicit wireless-environment models or specific mathematical formulations.

Abstract

from arXiv · show

Recently, the reconfigurable intelligent surface (RIS), benefited from the breakthrough on the fabrication of programmable meta-material, has been speculated as one of the key enabling technologies for the future six generation (6G) wireless communication systems scaled up beyond massive multiple input multiple output (Massive-MIMO) technology to achieve smart radio environments. Employed as reflecting arrays, RIS is able to assist MIMO transmissions without the need of radio frequency chains resulting in considerable reduction in power consumption. In this paper, we investigate the joint design of transmit beamforming matrix at the base station and the phase shift matrix at the RIS, by leveraging recent advances in deep reinforcement learning (DRL). We first develop a DRL based algorithm, in which the joint design is obtained through trial-and-error interactions with the environment by observing predefined rewards, in the context of continuous state and action. Unlike the most reported works utilizing the alternating optimization techniques to alternatively obtain the transmit beamforming and phase shifts, the proposed DRL based algorithm obtains the joint design simultaneously as the output of the DRL neural network. Simulation results show that the proposed algorithm is not only able to learn from the environment and gradually improve its behavior, but also obtains the comparable performance compared with two state-of-the-art benchmarks. It is also observed that, appropriate neural network parameter settings will improve significantly the performance and convergence rate of the proposed algorithm.

I. INTRODUCTION

Massive-MIMO can impose high hardware cost, physical-size constraints, and power consumption, motivating RIS and DRL approaches for large-scale wireless optimization. This paper proposes jointly designing BS transmit beamforming and RIS phase shifts with DRL rather than alternating optimization.

  • I. INTRODUCTION: Massive-MIMO deployment is challenged by high hardware cost, constrained physical size, and increased power consumption at large antenna arrays.
  • I. INTRODUCTION: RIS reflecting arrays use many low-cost, low-energy elements whose reconfigurable phase shifts can enhance desired signals or reduce interference.
  • I. INTRODUCTION: Prior RIS-assisted MISO research largely addressed single-user systems or multiuser optimization with alternating components such as ZF beamforming and phase-shift search.
  • I. INTRODUCTION: DRL is suited to time-varying wireless channels because it can learn from environmental rewards without knowing the channel model or mobility pattern.
  • B. Contributions: The proposed method jointly outputs the transmit beamforming matrix and RIS phase shifts, using sum rate as the reward and DDPG for continuous actions.
  • B. Contributions: Simulations report stepwise learning, comparable performance with WMMSE and FP-with-ZF benchmarks, and sensitivity of convergence and performance to neural-network settings and low SNR.

II. SYSTEM MODEL AND PROBLEM FORMULATION

The paper models a multiuser MISO downlink assisted by a reflecting RIS and formulates joint beamforming and phase-shift optimization as a non-convex sum-rate problem.

  • System model: The system contains an M-antenna BS, one N-element reflecting RIS, and K single-antenna users with M ≥ K.K data streams are transmitted through the RIS, while direct BS–user transmissions are assumed negligible.
  • System model: The BS applies beamforming matrix G, while the RIS applies diagonal phase-shift matrix Φ to the impinging signals.The RIS phase entries have unit magnitude under ideal lossless reflection and continuous phase shifts.
  • System model: The received signals experience composite channel fading through the BS–RIS and RIS–user channels, without additional AWGN introduced by the passive RIS.The RIS reconfigures signal phases through an integrated micro-controller rather than decoding and encoding signals.
  • Problem formulation: The objective is to maximize ergodic sum rate C by jointly selecting G and Φ for a given CSI.The kth-user rate is Rk = log2(1 + ρk), and sum rate aggregates the user rates.
  • Problem formulation: The optimization is non-convex because of its objective and constraints, making exhaustive search impractical for large-scale networks.Conventional approaches typically alternate between optimizing G with fixed Φ and optimizing Φ with fixed G.

III. PRELIMINARY KNOWLEDGE OF DRL

This section introduces reinforcement-learning elements and explains how deep neural networks approximate value functions when state and action spaces are large.

  • RL fundamentals: Reinforcement learning learns actions through trial-and-error interaction with an environment, using states, actions, rewards, policies, and value functions.The agent observes state transitions and instant rewards after taking actions.
  • RL fundamentals: The state records observations, the action is a choice made by the agent, and the reward evaluates that choice at a given state.The policy gives action probabilities conditioned on the state.
  • Value functions: The state-action value function measures expected future discounted rewards resulting from taking an action in a state under policy π.The discount rate satisfies γ ∈ (0, 1].
  • Q-learning: Q-learning recursively searches for an optimal policy and can obtain the optimal Q function without exact reward or state-transition models.The Q function is updated using a learning rate α.
  • Deep reinforcement learning: Deep reinforcement learning uses neural networks to approximate value or policy functions for enormous state and action spaces.The parameter vector θ represents network weights and biases updated through stochastic optimization.
  • Deep reinforcement learning: Experience replay reduces temporal state correlation by updating the network with randomly sampled batches from a buffer of recent experiences.This differs from updating only from the latest state.

B. DDPG

DDPG addresses the proposed continuous state and action spaces with actor and critic networks that jointly approximate actions and Q values.

  • DDPG architecture: DDPG uses an actor network to map states to continuous actions and a critic network to evaluate those actions with the state.The actor avoids explicitly solving the non-convex action-maximization problem for the next state.
  • Network architecture: The proposed critic and actor are fully connected DNNs with one input layer, two hidden layers, and one output layer.Their input and output dimensions correspond to the relevant state/action representation and network output.
  • Network updates: The training actor is updated using the target critic’s gradient with respect to the action, directing actions toward higher Q values.This gradient-based update determines the preferred direction for action selection.
  • Network updates: Training and target critic networks are updated separately, with target-network parameters updated more slowly than training-network parameters.The target critic and target actor each have their own update rates.

IV. DRL BASED JOINT DESIGN OF TRANSMIT BEAMFORMING AND PHASE SHIFTS

The proposed joint-design algorithm uses DDPG to optimize transmit beamforming and RIS phase shifts through states, actions, and instant rewards.

  • Algorithm overview: The algorithm is driven by two DNNs together with the state s, action a, and instant reward r.The section first specifies the DNN structures and then describes these DRL components and the algorithm.

A. Construction of DNN

The paper uses fully connected deep neural networks for both critic and actor roles, with dimensions tied to the state and action spaces. Whitening, batch normalization, tanh activation, Adam optimization, and constraint-enforcing normalization support training.

  • Both the critic and actor networks are fully connected deep neural networks with one input layer, one output layer, and two hidden layers.
  • The critic input and output dimensions match the combined state-action cardinality, while the actor dimensions match the state and action cardinalities.
  • State whitening removes correlations among state entries before they enter either network.
  • Batch normalization is applied at hidden layers to accommodate changing layer-input distributions and support higher learning rates.
  • The networks use tanh activations and Adam optimizers with adaptive learning rates.
  • Output normalization enforces the transmit-power constraint for G and unit-modulus diagonal entries for Φ.

B. Algorithm Description

The algorithm uses an agent that observes channel information and previous actions to construct states, then iteratively trains critic and actor networks through episodic interaction. It returns the action with the best instantaneous reward after convergence or a step limit.

  • The agent is assumed to instantaneously collect channel information and use the previous G and Φ to construct the current state.
  • The experience replay buffer, critic network, actor-network parameters, and actions G and Φ are initialized before training.
  • Training runs for N episodes of up to T steps, terminating when the algorithm converges or reaches the maximum step limit.
  • The optimal Gopt and Φopt are selected as the action producing the best instantaneous reward.

1) State:

The algorithm constructs states from channel, power, received-signal, and previous-action information, separating complex quantities into real and imaginary components. It then stores transitions, updates critic and actor networks, and advances through episodes.

  • State: Algorithm 1 takes H1 and hk,2 for all k as input and outputs the optimal action a = {G, Φ}.
  • State: The state is initialized after collecting and preprocessing the channel information at the start of each episode.
  • State: At each step, the agent observes the next state and instantaneous reward, then stores the resulting transition in replay memory.
  • State: The critic Q-value and network losses are updated using sampled replay experiences, with periodic target-network updates and actor-network optimization.
  • State: The next state is fed to the DNN after each iteration, and the episode loops end after the prescribed steps.
  • State: Complex transmission and received powers are split into real and imaginary components, producing 2K power entries and 2K^2 received-power entries.
  • State: The state dimension is Ds = 2K + 2K^2 + 2N + 2MK + 2NM + 2KN, preserving information otherwise lost through absolute values.

2) Action:

The action jointly contains the base-station transmit beamforming matrix and RIS phase-shift matrix. The reward is the instantaneous sum-rate capacity under the current channels and actor-generated action.

  • Action: The action consists of G and Φ, with their real and imaginary parts separated for neural-network input.
  • Action: The action-space dimension is Da = 2MK + 2N.
  • Action: The instantaneous reward is the sum-rate capacity C(G(t), Φ(t), hk,2, H1) for the current channels and actor-generated action.

V. NUMERICAL RESULTS AND ANALYSIS

The proposed DRL algorithm is evaluated against WMMSE and FP-ZF benchmarks across transmit-power and RIS-size settings, then examined for convergence under varying system parameters. It achieves comparable sum-rate performance while showing sensitivity to transmit power and robustness to RIS element-number changes.

  • Setting and benchmarks: The simulations use randomly generated Rayleigh channel matrices with compensated large-scale path loss and shadowing effects.The evaluation focuses on the DRL framework for beamforming and phase-shift design.
  • Setting and benchmarks: WMMSE and FP optimization with ZF beamforming serve as the two state-of-the-art benchmarks.In their generic forms, both benchmarks are centralized, iterative methods requiring full up-to-date cross-cell CSI.
  • Comparisons with Benchmarks: The proposed DRL algorithm obtains comparable sum-rate performance to WMMSE and FP-ZF, with sum rates increasing as maximum transmit power Pt increases.This trend holds for both considered parameter sets: M = 32, N = 32, K = 32 and M = 8, N = 8, K = 8.
  • Comparisons with Benchmarks: Average sum rates increase with RIS element number N at Pt = 20dB, M = 64, K = 64, while the larger RIS incurs higher implementation complexity.The authors report that the DRL algorithm remains robust across these wider application scenarios and approaches optimal performance.
  • Impact of Pt on DRL: Rewards converge over time steps, with faster convergence at Pt = 5dB than Pt = 20dB because higher SNR produces larger instant-reward fluctuations.Starting from identity matrices, the DRL algorithm adjusts G and Φ through environmental feedback toward optimal solutions.

D. Impact of system settings

The DRL method is evaluated under changes in RIS element count, transmit power, learning rate, and decaying rate. These settings affect rewards, sum rate, performance, convergence, and sensitivity to neural-network hyperparameters.

  • RIS element number: Increasing RIS elements from N = 4 to 30 gradually increases average rewards without increasing DRL convergence time.The method is described as robust to this system-setting change.
  • Transmit power: Average sum rate increases with transmit power Pt as more power is allocated to the base station.
  • Learning rate: A learning rate of 0.001 achieves the best performance, although it converges more slowly than learning rates of 0.0001 and 0.00001.A learning rate of 0.01 has the worst performance.
  • Learning rate: Different learning rates have a great influence on DRL performance.The evaluated learning rates are listed as {0.01, 0.001, 0.0001, 0.0001} in the passage.
  • Hyperparameter sensitivity: DRL performance is very sensitive to DNN initialization and other hyperparameters, including minibatch size.The authors state that hyperparameters must be defined carefully for a given system setting.
Loading 2002.10072v1…