PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-hop RIS-Empowered Terahertz Communications: A DRL-based Hybrid Beamforming Design

Chongwen Huang, Zhaohui Yang, George C. Alexandropoulos, Kai Xiong, Li Wei, Chau Yuen, Zhaoyang Zhang, Merouane Debbah

arXiv:2101.09137v2eess.SPcs.ITcs.LG

TL;DR

THz links are constrained by severe attenuation and absorption that limit transmission distance and coverage. The paper proposes multi-hop RIS-assisted hybrid beamforming, using DRL with alternating-optimization initialization, and reports 50% more coverage range than benchmarks.

  • Problem

    Severe THz propagation attenuation and molecular absorption limit transmission distance and coverage, motivating improved multi-hop communication designs.

  • Method

    The paper deploys multiple passive RISs and jointly designs BS digital and RIS analog beamforming with a DRL algorithm initialized by two alternating-optimization methods.

  • Results

    50% more coverage range is achieved for THz communications compared with the considered benchmarks.

  • Takeaways & Limitations

    The proposed DRL-based method addresses the NP-hard hybrid beamforming problem, particularly when RIS-assisted THz signals experience multiple hops.

Abstract

from arXiv · show

Wireless communication in the TeraHertz band (0.1--10 THz) is envisioned as one of the key enabling technologies for the future sixth generation (6G) wireless communication systems scaled up beyond massive multiple input multiple output (Massive-MIMO) technology. However, very high propagation attenuations and molecular absorptions of THz frequencies often limit the signal transmission distance and coverage range. Benefited from the recent breakthrough on the reconfigurable intelligent surfaces (RIS) for realizing smart radio propagation environment, we propose a novel hybrid beamforming scheme for the multi-hop RIS-assisted communication networks to improve the coverage range at THz-band frequencies. Particularly, multiple passive and controllable RISs are deployed to assist the transmissions between the base station (BS) and multiple single-antenna users. We investigate the joint design of digital beamforming matrix at the BS and analog beamforming matrices at the RISs, by leveraging the recent advances in deep reinforcement learning (DRL) to combat the propagation loss. To improve the convergence of the proposed DRL-based algorithm, two algorithms are then designed to initialize the digital beamforming and the analog beamforming matrices utilizing the alternating optimization technique. Simulation results show that our proposed scheme is able to improve 50\% more coverage range of THz communications compared with the benchmarks. Furthermore, it is also shown that our proposed DRL-based method is a state-of-the-art method to solve the NP-hard beamforming problem, especially when the signals at RIS-assisted THz communication networks experience multiple hops.

I. INTRODUCTION

THz communications promise ultra-high-speed, low-latency links but suffer severe attenuation, absorption, and limited transmit power. RIS technology motivates multi-hop designs, while prior work largely leaves multi-hop multiuser RIS-assisted systems unresolved.

  • THz communications target ultra-high-speed and low-latency 6G links, but available mmWave bandwidth remains insufficient for Tbps rates.
  • Molecular absorption from water vapor and oxygen, along with milliwatt-scale transceiver power, further increases THz path loss.
  • RISs use programmable reflecting elements and phase shifts to focus desired signals, suppress interference, and support compact multi-RIS deployment.
  • A. Prior Works: Existing RIS-assisted THz studies address channel estimation, data-rate maximization, beamforming, resource allocation, coverage, or secrecy, generally under single-hop assumptions.
  • A. Prior Works: Multi-hop RIS systems remain rarely studied because relay methodologies do not directly apply to RIS reflection mechanisms, channel models, and unit-modulus constraints.

B. Contributions

The paper proposes a practical multi-hop RIS-assisted hybrid beamforming architecture and a DRL algorithm for its difficult joint optimization. Two alternating-optimization initializers improve DRL convergence, and simulations report a 50% coverage-range improvement over benchmarks.

  • Multiple passive RISs are deployed between the BS and users to combat THz propagation attenuation and improve coverage range.
  • The paper formulates a non-convex joint design of BS digital beamforming and RIS analog beamforming for multi-hop multiuser communication.
  • A DRL-based algorithm addresses the NP-hard joint beamforming problem in the presence of multi-hop signals and composite channel fading.
  • Two alternating-optimization methods initialize digital and analog beamforming matrices to improve convergence and mitigate local-optimal solutions.
  • 50% more coverage range is achieved for THz communications compared with the considered benchmarks.
  • The proposed DRL method is reported as a state-of-the-art solution for the NP-hard beamforming problem, especially with multiple hops.

II. SYSTEM MODEL AND PROBLEM FORMULATION

The section models THz multi-hop RIS-assisted links and formulates a practical hybrid beamforming architecture for extending coverage despite severe propagation losses. It jointly considers BS digital beamforming and RIS analog phase shifts for multiuser transmission.

  • A. Terahertz-Band Channel Model: THz links suffer high path loss from molecular absorption, especially involving water vapor and oxygen, while spreading and reflection losses further attenuate NLoS paths.
  • A. Terahertz-Band Channel Model: The channel model represents LOS and NLOS frequency responses using propagation delay, molecular absorption, spreading, and reflection components.
  • B. Proposed Multi-hop Scheme: Multiple programmable RISs are deployed across successive hops to collect more energy and improve the coverage range of THz communications.
  • B. Proposed Multi-hop Scheme: The system supports K simultaneous data streams from an M-antenna BS to K single-antenna users through up to N RIS hops, with direct BS-user paths also considered.
  • B. Proposed Multi-hop Scheme: The proposed architecture uses RISs as external, standalone analog beamformers, removing the BS’s internal analog precoder and reducing implementation cost.
  • B. Proposed Multi-hop Scheme: The channel model assumes frequency-flat fading, Rician-distributed links, and perfectly known CSI at the BS and users.
  • B. Proposed Multi-hop Scheme: The received-signal model combines cascaded RIS channels, the final RIS-user channel, the direct BS-user channel, BS beamforming, interference, and AWGN.
  • B. Proposed Multi-hop Scheme: Each RIS is modeled by a diagonal phase-shift matrix with unit-modulus entries and continuous phase resolution over [0, 2π).

C. Problem Formulation

The paper formulates multi-hop RIS beamforming as maximizing ergodic sum rate by jointly optimizing the BS digital beamforming matrix and RIS phase-shift matrices. This formulation is NP-hard and difficult for conventional optimization in large or variable-hop networks.

  • C. Problem Formulation: The objective is to combat THz propagation attenuation by maximizing the ergodic sum rate of the multi-hop RIS-assisted system.
  • C. Problem Formulation: The optimization variables are the BS digital beamforming matrix F and the RIS phase-shift matrices Φi for all deployed RISs.
  • C. Problem Formulation: The resulting problem is NP-hard because its objective is nontrivial and its constraints are non-convex.
  • C. Problem Formulation: Analytical solutions are nearly impossible for the multi-hop optimization, while exhaustive numerical search is impractical in large-scale networks.
  • C. Problem Formulation: Alternating-optimization approximations developed for single-hop RIS systems are difficult to apply when signals traverse multiple hops.
  • C. Problem Formulation: Because the prior number of RIS hops Ik is unknown, the paper proposes using DRL rather than directly solving the optimization mathematically.

III. FRAMEWORK OF THE PROPOSED DRL AND POLICY

The proposed framework models beamforming optimization as a reinforcement-learning problem and uses deep networks to handle the large or continuous state and action spaces. Its actor-critic design learns policies from environmental rewards and state transitions.

  • A. Framework of DRL: The DRL framework is used to design the transmit beamforming matrix and RIS phase-shift matrices in multi-hop THz networks.
  • A. Framework of DRL: An MDP represents the environment through states, actions, rewards, a policy, transition function, and Q-function.
  • A. Framework of DRL: The agent maximizes average rewards so that future cumulative rewards are considered alongside instantaneous rewards.
  • A. Framework of DRL: Because the hybrid beamforming problem has nearly infinite state and action spaces, a Q-table would require impractical storage and search complexity.
  • A. Framework of DRL: The framework approximates the Q-table with deep neural networks and uses an actor network for actions and a critic network for policy evaluation.
  • A. Framework of DRL: The critic updates the value-function approximation, while actor policy parameters are updated using gradients derived from the learning objective.
  • A. Framework of DRL: Training and target Q-function networks are maintained, with their differences defining the loss function and target values updated during learning.

B. Optimization Policy

The optimization policy uses DDPG to address continuous state and action spaces in hybrid beamforming. Its actor-critic updates use rewards, replayed experience, critic loss minimization, policy gradients, and target-network updates.

  • B. Optimization Policy: DDPG is selected because conventional DQN is suited to a few discrete spaces, whereas the beamforming problem has continuous state and action spaces.
  • B. Optimization Policy: The DDPG method combines deterministic policy gradients with an actor-critic reinforcement-learning framework.
  • B. Optimization Policy: The critic network is trained using the DDPG policy, while target-actor actions and target-critic gradients contribute to actor-network updates.
  • B. Optimization Policy: Algorithm 1 repeatedly collects states and actions, observes next states and rewards, stores experiences, samples minibatches, and updates network parameters.
  • B. Optimization Policy: The algorithm sets target functions, minimizes critic loss, updates the actor with policy gradients, and updates target networks using the specified equations.
  • B. Optimization Policy: The training procedure runs for Z episodes, with each episode containing T iterations.

IV. DRL-BASED DESIGN OF DIGITAL AND ANALOG BEAMFORMING

The proposed hybrid beamforming design uses DDPG with actor and critic networks to jointly control digital and RIS analog beamforming. Network normalization and power-related processing enforce the communication constraints during learning.

  • Network architecture: DDPG combines critic and actor neural networks for hybrid beamforming in multi-hop THz communication networks.The critic evaluates policies from states and actor-generated actions, while the actor learns states and outputs actions.
  • Network architecture: Both networks use fully connected DNNs with one input layer, two hidden layers, and one output layer.Their widths depend on the actions, while the final output layer is determined by the number of users.
  • Network implementation: Batch normalization with ReLU is placed between the hidden layers, and Adam is used for optimization with learning rate µ(t).The critic additionally uses tanh for negative inputs, while the state input is whitened to reduce adjacent-state correlation.
  • Constraint handling: The actor outputs actions to the critic and applies power modular normalization so the digital beamforming matrix satisfies the transmit-power constraint.The actor output is normalized according to Tr{FFH} = Pt before implementation.
  • Constraint handling: RIS phase shifts preserve amplitude because passive RIS elements satisfy |Φi(ni, ni)|^2 = 1 while changing the signal transmission direction.This constraint reflects that the RIS does not consume transmission power.

B. Proposed DRL Algorithm

The DRL agent represents channel and beamforming information as continuous states and actions, separating real and imaginary components for TensorFlow processing. Its learning process uses rewards based on transmission contributions and beamforming adjustments, with initialization and learning settings affecting convergence.

  • Initialization: The proposed algorithm initializes the replay buffer and action parameters using SVD and max-min SINR methods.These two initialization algorithms are introduced to improve the DRL learning process.
  • Convergence: The algorithm treats Fopt and Φi,opt as optimal when additional actions no longer increase the obtained rewards.The authors state that the method may nevertheless converge to suboptimal solutions.
  • Training procedure: The algorithm iterates over episodes, preprocesses states, applies DDPG, and feeds the resulting next state back to the agent.It stops when convergence is reached or the maximum number of iterations is attained.
  • State: The continuous state includes digital beamforming, channel information, and previous digital and analog beamforming actions.The state provides information used to select RIS and analog beamforming while learning interference patterns from historical data.
  • Action: The continuous action space consists of the BS digital beamforming matrix F and RIS analog beamforming matrices Φi.Their real and imaginary components are separated into distinct inputs, and the action dimension depends on system parameters.

3) Reward:

The reward and initialization design balances throughput-related gains with beamforming adjustments and uses SVD- or max-min-SINR-based starting points. The paper also identifies convergence sensitivity and constraints in these procedures.

  • Reward: The instantaneous reward combines throughput contributions with a penalty for adjusting the beamforming direction under prior information.It depends on channel information, user channels, and actor-generated actions.
  • Convergence: Initialization affects convergence, while gradient evolution and learning rate can cause divergence when their values are too large or too small.The paper investigates learning-rate effects in the simulation section.
  • Convergence: The DRL algorithm is not guaranteed to converge to the optimal solution, and its convergence speed strongly depends on initialization.Two methods are proposed to initialize the action variables for improved performance.
  • SVD initialization: The SVD-based initialization first selects unit-amplitude diagonal RIS phase matrices and then obtains transmit beamforming using zero forcing.The objective is to eliminate co-channel interference, requiring M ≥ K for the relevant null spaces to exist.
  • Max-min SINR initialization: The max-min SINR initialization iteratively solves fixed-point relations to obtain transmit beamforming and associated power coefficients.Its objective is to maximize the minimum SINR for a given RIS phase-shift matrix.

V. NUMERICAL RESULTS

The numerical evaluation considers a single-cell multi-hop RIS-assisted THz network with randomly deployed RISs and multiple single-antenna users. Simulations use a 0.12 THz carrier, 12 GHz bandwidth, and 10 Watt BS transmit power.

  • Simulation setup: The simulations evaluate the proposed DRL-based hybrid beamforming algorithm for multi-hop multiuser RIS-assisted THz communication networks.The study first specifies the communication model and simulation scenario before presenting results.
  • Simulation setup: The single-cell scenario contains one BS and randomly deployed RISs within a circular region of diameter 100 m.The RIS deployment is used to model the considered multi-hop THz communication environment.
  • System parameters: The BS has M = 8 antennas and the same number of RF chains, while K = {4, 32} users each have one antenna and RF chain.The user count is varied across the listed values.
  • System parameters: All N = 64 RISs use Ni = {64, 128} elements with element spacing of 2λ.The RISs share the same element count to reduce deployment and learning complexity.
  • Channel and power settings: The simulations use Rician small-scale fading, Friis-based large-scale fading, a 0.12 THz carrier, 12 GHz bandwidth, and 10 Watt BS transmit power.The Rician factors are set to KH = Kg = Kw = 1 unless otherwise specified.
  • Implementation: The algorithm is implemented in TensorFlow on a computer equipped with an i7-8700 CPU and Nvidia Geforce GTX 1080Ti GPU.Additional DRL parameter settings are reported in Table I.

3) Benchmarks:

The evaluation compares DRL-based multi-hop RIS beamforming with no-RIS, traditional alternating single-hop, and DRL-based single-hop benchmarks. Across transmission distances and system settings, multi-hop generally provides the strongest throughput and coverage, while reward convergence depends on power and RIS size.

  • Benchmark schemes: The evaluation uses three benchmarks: full-digital zero-forcing without RISs, traditional alternating optimization for single-hop RIS, and DRL-based single-hop beamforming.The multi-hop scheme with I = 1 is also treated as a special single-hop case.
  • Benchmark comparisons: The proposed DRL-based multi-hop scheme with I = 2 nearly always achieves the best throughput from 1 m to 20 m.The comparison uses M = 8, N_i = 128, and K = 32.
  • Benchmark comparisons: At 1 Gbps, the DRL-based two-hop scheme reaches about 50% greater distance than no-RIS ZF and 14% greater distance than the single-hop comparison.The cited comparison is made at the same throughput level.
  • Benchmark comparisons: The performance gap between the multi-hop and single-hop schemes increases as transmission distance grows, while traditional alternating optimization slightly outperforms DRL-based single-hop beamforming.The alternating method nevertheless performs much below the two-hop scheme.
  • Impact of system settings: Increasing RIS elements from 64 to 128 improves transmission distance by about 38%, with RIS size becoming especially important beyond 15 m.More users and larger RISs are also associated with better total throughput.
  • Impact of system settings: The sum-rate rewards converge over time; convergence is faster at P_t = 5W than at P_t = 30W because higher power produces larger instant-reward state spaces.The DRL agent uses these rewards to update beamforming matrices F and Φ_i toward a local optimum.

D. Impact of the Rician Factor

The evaluation examines how Rician factors and learning-rate choices affect throughput, convergence, and rewards in the proposed multi-hop RIS-assisted THz system. The paper concludes that the hybrid DRL-based design improves coverage range and addresses the NP-hard beamforming problem.

  • Rician-factor impact: The fourth Rician-factor case, with I = 2 and KH = Kg = Kw = 10, obtains the highest total throughput at the same transmission distance.The larger Rician factor increases throughput because transmission power is more focused on the LoS path, reducing scattering dissipation.
  • Rician-factor impact: Increasing the Rician factor from 1 to 10 increases total throughput and improves transmission distance significantly.The reported mechanism is reduced space scattering dissipation when the LoS component dominates small-scale fading.
  • Learning-rate impact: A learning rate of 0.001 achieves the best average rewards, whereas rates that are too large or too small produce lower rewards.The comparison considers constant learning rates including 0.01, 0.001, and 0.00001.
  • Learning-rate impact: Better rewards can increase convergence time because the larger state space may require more time to reach the optimal solution.The paper therefore emphasizes appropriate DRL parameter settings and tuning to improve performance and reduce convergence time.
  • Overall result: The proposed multi-hop RIS-assisted hybrid beamforming architecture improves THz coverage range by 50% compared with the considered benchmarks.The design jointly addresses digital beamforming at the BS and analog beamforming at RISs using a DRL-based algorithm with initialization methods for convergence.
  • Overall result: The proposed DRL-based method is reported as state of the art for the NP-hard beamforming problem when RIS-assisted THz signals experience multiple hops.The algorithm is described as having a standard formulation and low implementation complexity without requiring explicit wireless-channel knowledge.
Loading 2101.09137v2…