PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning of Beam Codebooks in Millimeter Wave and Terahertz MIMO Systems

Yu Zhang, Muhammad Alrabeiah, Ahmed Alkhateeb

arXiv:2102.11392v1cs.ITeess.SPmath.IT

TL;DR

Predefined beam codebooks may be poorly matched to the environment, user distribution, and hardware, while their many beams add training overhead. This paper uses deep reinforcement learning to adapt beam patterns from receive-power measurements without channel knowledge. In the reported NLOS evaluation, four learned beams outperform a 32-beam classical beamsteering codebook, and 16 beams with 4-bit phase shifters reach 80% of the EGC upper bound.

  • Problem

    Predefined codebooks may not fit the environment or hardware, and their large number of beams creates training overhead; unknown arrays and impairments also make channel information difficult to acquire.

    Source: §III · p. 8

  • Method

    A deep reinforcement learning framework adapts beam patterns and codebooks using receive-power measurements, without explicit channel, array-geometry, hardware, or user-position knowledge.

    Source: §I · p. 2, B. Contribution · p. 4

  • Results

    In NLOS, the learned solution surpasses a 32-beam classical beamsteering codebook with only 4 beams; with 16 beams and 4-bit phase shifters, it reaches 80% of the EGC upper bound.

    Source: B. Beam Codebook Learning · p. 27

  • Takeaways & Limitations

    The simulations show that learned codebooks can adapt to NLOS conditions and hardware impairments using receive-power measurements without channel knowledge.

    Source: A. Beam Pattern Learning · p. 25, B. Beam Codebook Learning · p. 27

Abstract

from arXiv · show

Millimeter wave (mmWave) and terahertz MIMO systems rely on pre-defined beamforming codebooks for both initial access and data transmission. Being pre-defined, however, these codebooks are commonly not optimized for specific environments, user distributions, and/or possible hardware impairments. This leads to large codebook sizes with high beam training overhead which increases the initial access/tracking latency and makes it hard for these systems to support highly mobile applications. To overcome these limitations, this paper develops a deep reinforcement learning framework that learns how to iteratively optimize the codebook beam patterns (shapes) relying only on the receive power measurements and without requiring any explicit channel knowledge. The developed model learns how to autonomously adapt the beam patterns to best match the surrounding environment, user distribution, hardware impairments, and array geometry. Further, this approach does not require any knowledge about the channel, array geometry, RF hardware, or user positions. To reduce the learning time, the proposed model designs a novel Wolpertinger-variant architecture that is capable of efficiently searching for an optimal policy in a large discrete action space, which is important for large antenna arrays with quantized phase shifters. This complex-valued neural network architecture design respects the practical RF hardware constraints such as the constant-modulus and quantized phase shifter constraints. Simulation results based on the publicly available DeepMIMO dataset confirm the ability of the developed framework to learn near-optimal beam patterns for both line-of-sight (LOS) and non-LOS scenarios and for arrays with hardware impairments without requiring any channel knowledge.

I. INTRODUCTION

Predefined single-lobe codebooks can require many training beams and may not suit NLOS users, unknown arrays, or hardware impairments. The paper proposes learning adaptive codebooks from receive-power measurements without explicit channel knowledge.

A. Prior Work:

  • Classical codebooks use many single-lobe beams, increasing training overhead, and may be suboptimal for NLOS users or unknown, uncalibrated arrays.Prior neural codebook learning still required full or partial channel knowledge.Source: §I · p. 2, A. Prior Work · pp. 3–4

B. Contribution:

  • The framework learns codebook beam patterns from receive-power measurements, adapting to the environment and hardware without explicit channel knowledge.It also targets hardware-constrained, quantized-phase beamforming.Source: §I · p. 2, B. Contribution · p. 4
  • The approach learns codebooks for surrounding users without requiring their positions or that they remain stationary during learning.SNR-based feature matrices support user clustering and assignment to parallel neural networks.Source: B. Contribution · pp. 4–5
  • The simulations report improved receive gain over classical beamsteering codebooks using receive combining gains rather than channel state information.The evaluation uses DeepMIMO scenarios and considers different codebook sizes, array geometries, and hardware impairments.Source: B. Contribution · p. 5

A. System Model

  • The system model uses a single-RF-chain base station with analog-only beamforming and a codebook of N beamforming or combining vectors.The phase shifters use r-bit quantized phases.Source: A. System Model · p. 5

B. Channel Model

The channel model represents propagation as multiple paths, while the hardware model allows arbitrary array geometry and phase mismatches. In evaluation, the paper focuses on non-uniform antenna spacing and phase mismatch.

B. Channel Model

  • The channel comprises L paths, each with a complex gain and angle of arrival, and its array response depends on geometry and hardware impairments.The array-response vector is defined using the adopted array and impairment model.Source: B. Channel Model · p. 6

C. Hardware Impairments Model

  • The model accounts for arbitrary array geometry and hardware imperfections, evaluating non-uniform antenna spacing and phase mismatch.Antenna positions and added phase shifts are treated as fixed but unknown random realizations.Source: C. Hardware Impairments Model · p. 7

III. PROBLEM DEFINITION

The section formulates beam-codebook design as maximizing average SNR over users while respecting phase-only, quantized-phase hardware constraints. Because channels may be unknown, it frames learning a beam pattern for one or similar users as a step toward codebook design.

  • The codebook objective is to maximize SNR averaged over the users the base station can serve.The formulation constrains each beam to phase-only weights with quantized phases.Source: §III · pp. 7–8
  • Unknown channels, array geometry, and hardware impairments can leave the base station with only beamforming gain or RSSI feedback.This makes the objective difficult to solve directly under the non-convex and discrete constraints.Source: §III · p. 8
  • The approach first learns an optimized pattern for one user or a group with similar channels, then addresses codebook design through clustering, assignment, and pattern learning.Source: §III · p. 9

IV. BEAM PATTERN LEARNING

Beam-pattern learning is a search over a huge finite discrete action space, so the paper uses a Wolpertinger architecture to make that search tractable. Its actor proposes a proto-action, which is mapped to nearby valid actions using K-nearest neighbors.

IV. BEAM PATTERN LEARNING

  • The Wolpertinger-based framework narrows the search over the huge discrete feasible set while aiming to avoid missing the optimal policy.For 32 antennas and 3-bit phase shifters, the passage reports over 7.9 × 1028 legitimate beamforming vectors.Source: §IV · p. 10

A. Wolpertinger Architecture Overview

  • Deep Q-networks become difficult to apply as action counts grow because their output dimension tracks the number of possible actions.The Wolpertinger architecture uses an actor-critic framework trained with DDPG and a KNN classifier for large finite discrete action spaces.Source: A. Wolpertinger Architecture Overview · p. 10

1) Actor Network:

  • The actor maps the state to a proto-action that may not belong to the finite discrete action set.That proto-action must be quantized to a legitimate action.Source: 1) Actor Network · p. 11

2) K-Nearest Neighbor:

  • The K-nearest-neighbor function returns the k actions in the discrete set closest to the proto-action by L2 distance.Source: 2) K-Nearest Neighbor · p. 11

3) Critic Network:

The critic evaluates candidate state-action pairs, while the actor is trained to maximize the critic’s predicted value. Target networks provide targets for training and are updated from the actor and critic parameters.

3) Critic Network:

  • The critic evaluates the k candidate state-action pairs and selects the action with the highest predicted Q value.Source: 3) Critic Network · p. 11

4) Network Update:

  • The actor is updated with the deep deterministic policy gradient to maximize the critic’s predicted Q value for a state.Source: 4) Network Update · p. 12
  • Target actor and critic networks calculate training targets and receive parameter updates from the actor and critic after a set number of iterations.The target parameters use a non-negative hyperparameter τ, usually far less than 1.Source: 4) Network Update · p. 12

B. DRL Based Beam Pattern Design

1) Reinforcement Learning Setup:

The learning setup represents the beamformer through its phase-shifter settings and rewards changes according to measured combining gain. During interaction, the base station relies on receive-power feedback, updates its threshold and prior-gain records, and trains the actor and critic from stored transitions.

1) Reinforcement Learning Setup:

  • The state is the vector of phase-shifter phases, and an action specifies the next state through element-wise phase changes.The phase vector maps to the corresponding beamforming vector.Source: 1) Reinforcement Learning Setup · pp. 12–13
  • The ternary reward compares current beamforming gain with both an adaptive threshold and the previous gain.The threshold starts at zero and is updated when a beam’s gain surpasses it; the best beam is recorded.Source: 1) Reinforcement Learning Setup · p. 13

2) Environment Interaction:

  • Because accurate channel state information is generally unavailable, the base station adjusts its beam using receive power or user-reported beamforming gain.Source: 2) Environment Interaction · p. 13
  • The base station evaluates average combining gain over the targeted user set, updates the threshold and previous gain, and stores each transition for learning.The critic and actor are updated using sampled minibatches and the target networks are periodically updated.Source: 2) Environment Interaction · p. 14

3) Exploration:

Exploration perturbs the actor’s predicted proto-action, with noise scheduled to move from broad exploration toward nearby phase choices. The resulting beam is quantized to a valid phase-shifter setting by nearest-neighbor lookup.

3) Exploration:

  • The exploration process starts from the actor’s proto-action predicted from the current beam state.Source: 3) Exploration · p. 14
  • Exploration noise is added element-wise to the actor’s proto-action, with temporally correlated Ornstein–Uhlenbeck samples and power decreasing over training.Early noise should reach any phase in Θ; near termination it should perturb phases only to adjacent choices.Source: 3) Exploration · pp. 14–15

4) Quantization:

  • The noisy proto-beam is quantized by a k = 1 nearest-neighbor lookup to produce a valid new beam.Each quantized phase is chosen by nearest-neighbor matching.Source: 4) Quantization · p. 15

5) Forward Computation and Backward Update:

The codebook-learning approach combines user clustering, cluster assignment, and beam-pattern learning; its clustering uses receive combining gains from sensing beams rather than explicit channel estimates. Feature transformation and k-means produce user groups that reflect persistent environmental structure.

5) Forward Computation and Backward Update:

  • The forward pass feeds the current and new beam states to the critic for Q-value computation, then updates actor and critic parameters in a backward pass.A pseudocode version is provided in Algorithm 1.Source: 5) Forward Computation and Backward… · p. 15

V. BEAM CODEBOOK LEARNING

  • The full codebook-learning pipeline has three stages: cluster users, assign clusters to DRL networks, and learn one beam pattern per network.The collection of learned beam patterns forms the codebook.Source: §V · pp. 15–16

A. User Clustering

  • Randomly sampled feasible sensing beams gather receive combining gains used to cluster users without explicitly estimating their channels.The gain measurements can be accumulated over users observed at different times because clustering reflects relatively static environmental scatterers.Source: A. User Clustering · pp. 16–17
  • Pairwise-difference features transform sensing-gain columns beyond the nonnegative orthant before k-means partitions users into N clusters.Directly clustering the original gain columns did not yield meaningful partitions in empirical evidence.Source: A. User Clustering · pp. 17–18
  • The learned clusters mainly reflect major environmental elements such as walls, buildings, and large trees, so they need not be retrained frequently.The clustering classifier can classify new users, and the relevant scatterers commonly change only over long periods.Source: A. User Clustering · p. 18

B. Cluster Assignment

When user clusters change, the method reassigns them to DRL networks so each network continues learning from a suitably similar cluster. It uses average beamforming gain as assignment suitability and solves a one-to-one matching problem with the Hungarian algorithm.

B. Cluster Assignment

  • Consistent assignments let each network refine its existing beam on a similar cluster rather than changing the beam completely.The suitability matrix compares each network’s temporarily best beam against each new channel cluster.Source: B. Cluster Assignment · pp. 18–19
  • The Hungarian algorithm finds a one-to-one cluster–network assignment that maximizes total suitability, represented by average beamforming gain.Each cost-matrix entry is the average gain of a temporarily best beam on a candidate channel cluster.Source: B. Cluster Assignment · p. 19
  • The assignment is a bijection between the temporarily best beams and the new channel groups.The optimization constrains the assignment matrix to be a permutation matrix.Source: B. Cluster Assignment · p. 19

C. Neural Network Update and Fine-Tuning

The approach keeps learning clusters and assigning users as they change, with each DRL network adapting its beam using gains from its assigned users. After training saturates, a perturb-and-quantize loop fine-tunes the beam without the more expensive agent-training process.

C. Neural Network Update and Fine-Tuning

  • Each DRL network adjusts its beam using receive combining gains from its assigned cluster, while clustering and assignment are repeated as users change over time.This setup is intended to account partially for environment dynamics rather than requiring all users to remain fixed during codebook learning.Source: C. Neural Network Update and Fine-Tuning · p. 20
  • The evaluation scenarios include outdoor LOS and indoor NLOS settings, selected for their respective likelihoods of LOS outdoors and NLOS indoors.Source: C. Neural Network Update and Fine-Tuning · p. 21, Figure 4 · p. 21
  • After agent training saturates, perturbing, quantizing, and evaluating the beam on its assigned users are repeated until beamforming gain saturates again.This fine-tuning avoids the relatively expensive agent-training process.Source: C. Neural Network Update and Fine-Tuning · p. 21

VI. EXPERIMENTS SETUP AND NETWORK TRAINING

The evaluation uses DeepMIMO LOS and NLOS scenarios and a DDPG-based learning architecture. In single-user LOS tests, learned beams approach the EGC upper bound, including with hardware impairments.

A. Communication Scenarios and Datasets

  • The DeepMIMO evaluation includes outdoor LOS at 60 GHz and indoor NLOS at 28 GHz, with additional LOS datasets varying antenna-spacing and phase-mismatch impairments.The impairment datasets vary the standard deviations of these two impairment sources.Source: A. Communication Scenarios and Datasets · pp. 21–22

B. Machine Learning Model Structure and Pre-processing

  • The DDPG model uses actor and critic networks: the actor maps an M-dimensional phase state to an M-dimensional action, and the critic estimates a scalar Q value from state and action.Each network has two hidden layers; actor layers have 16M neurons and critic layers 32M neurons, with ReLU activations.Source: B. Machine Learning Model Structure… · p. 22

A. Beam Pattern Learning

  • With 3-bit phase shifters, the learned LOS beam reaches almost 95% of the EGC upper bound without channel information, after exceeding 90% in fewer than 4 × 10^4 iterations.At iteration 100000, its main lobe has at least 10 dB gain over the second strongest side lobe; most side lobes are below −20 dB.Source: A. Beam Pattern Learning · p. 24
  • In the impaired-hardware LOS case, the learned beam exceeds 90% of the EGC upper bound in approximately 10^4 iterations, while classical beamsteering degrades substantially.The evaluation uses antenna-spacing standard deviation 0.1λ and phase-mismatch standard deviation 0.32π; the learned pattern captures dominant EGC lobes.Source: A. Beam Pattern Learning · p. 25

B. Beam Codebook Learning

The learned codebooks outperform classical beamsteering with fewer beams across the studied scenarios. Clustering and assignment split codebook learning into parallel tasks, while learned patterns adapt to LOS, NLOS, and impaired-hardware settings.

B. Beam Codebook Learning

  • The clustering-and-assignment approach decomposes codebook learning into N independent, parallel, lightweight tasks; in the stationary simulations, clustering and assignment run once.More frequent clustering and assignment are expected in more dynamic environments.Source: B. Beam Codebook Learning · p. 26
  • In LOS with an ideal array, six learned beams perform almost as well as a 32-beam classical codebook, and eight learned beams outperform it.Average beamforming gain increases monotonically with the number of learned beams.Source: B. Beam Codebook Learning · p. 26
  • In NLOS, four learned beams surpass the 32-beam classical codebook, while 16 beams with 4-bit phase shifters reach 80% of the EGC upper bound using receive combining gains.The learned NLOS beams have multiple lobes, matching signals arriving through reflections from the room’s side walls.Source: B. Beam Codebook Learning · pp. 27–28
  • As phase mismatch increases, the learned codebook maintains balanced performance, whereas the 32-beam classical codebook degrades drastically with increasing hardware impairment.The learned beams appear distorted in clean angular space but appear pointy in corrupted angular space.Source: B. Beam Codebook Learning · pp. 28–29

VIII. CONCLUSIONS AND DISCUSSIONS

  • Across all studied scenarios, learned codebooks outperform classical beamsteering with substantially fewer beams and approach ideal beam shapes and similar SNR performance.The conclusion describes evaluation with finite-resolution phase shifters and without channel knowledge.Source: §VIII · p. 29

Related papers

Loading 2102.11392v1…