PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantized Transformers for Massive MIMO Precoding with Automatic Resolution Tuning

Ghazal Kasalaee, Glodi Sala Mangituka, Ali Hasanzadeh Karkan, Jean-François Frigon, François Leduc-Primeau

arXiv:2610.04201v1eess.SP

TL;DR

Transformer precoders can improve Massive MIMO spectral efficiency, but their computational demands and large precision-search spaces complicate energy-efficient deployment. This paper applies differentiable automatic resolution tuning to jointly optimize model parameters and layer-wise precision. The resulting models can exceed WMMSE peak sum-rate while using substantially less compute energy at two deployment sites.

  • Problem

    Transformer precoders offer accuracy but have higher computational demands, and discrete precision search does not scale well to their large configuration spaces.

    Source: §I · p. 1

  • Method

    The paper uses differentiable automatic resolution tuning to jointly optimize weights, activations, quantization step sizes, and layer-wise bit-widths with a hardware-aware energy objective.

    Source: §I · p. 1, §III · p. 2, C. Automatic Resolution Tuning and… · p. 3

  • Results

    Transformer models exceed WMMSE (I = 25) peak sum-rate while consuming 125× less compute energy at Ericsson and 288× less at Sainte-Catherine.

    Source: A. Pareto Efficiency Analysis · p. 4

  • Takeaways & Limitations

    Across two ray-tracing channel models, the training pipeline produces Pareto-efficient models with little hyperparameter tuning.

    Source: §V · p. 5

Abstract

from arXiv · show

Deep learning precoders, particularly those based on Transformer architectures, offer superior spectral efficiency for Massive MIMO systems but may incur high computational cost. While existing mixed-precision quantization studies on convolutional neural networks establish important energy baselines, traditional discrete search methods fail to scale to the large parameter spaces of modern Transformers. To bridge this gap, we apply a differentiable precision learning technique to massive MIMO precoding. This approach autonomously and jointly optimizes weights, activations, quantization step sizes, and layer-wise bit-widths in a single training loop, directly optimizing for energy efficiency. We further expand this approach by incorporating Neural Architecture Search across various Transformer model sizes and evaluating the impact of random versus floating-point trained initializations on resolution tuning. Ultimately, we demonstrate that the proposed training method enables the deployment of highly compact Transformer precoders, improving energy efficiency by up to 288\(\times\) compared to the classical Weighted Minimum Mean Square Error algorithm at equal sum-rate performance in a dense downtown environment.

I. INTRODUCTION

The paper targets the energy and complexity costs of massive MIMO precoding, where Transformer precision configurations make discrete search difficult to scale. It proposes jointly learning model parameters and layer-wise precision, alongside a downlink system model optimized for sum-rate under a power constraint.

I. INTRODUCTION

  • Prior low-resolution precoders target DACs and phase shifters, while CNN quantization work uses fixed 8-bit precision and offline target generation,,,.Mixed-precision CNN baselines guided by NAS provide energy-efficiency references.Source: §I · p. 1
  • Transformers offer precoding accuracy but raise computational demands, while exhaustive bit-width search across multi-head attention layers is computationally prohibitive.Existing mixed-precision quantization baselines use CNNs, leaving a scalability gap for Transformer precision search.Source: §I · p. 1
  • The proposed automatic resolution tuning framework jointly learns weights, activations, quantization step sizes, and layer-wise bit-widths in one training run.It relaxes bit-width selection into a continuous variable and embeds a hardware-calibrated energy model in the loss.Source: §I · pp. 1–2

II. SYSTEM MODEL

  • The system is an FDD downlink where a base station with N_T antennas serves N_U single-antenna users using per-user precoding vectors.The channel vector for each user determines its received signal and SINR.Source: §II · p. 2
  • The system objective maximizes downlink sum-rate subject to a total transmit-power constraint.Sum-rate is defined as the sum of users' log2(1 + SINR_u).Source: §II · p. 2

A. Dataset Generation and Channel Modeling

The study constructs site-specific ray-tracing channel datasets for two Montreal deployment scenarios with contrasting propagation conditions. Both scenarios use a common antenna and transmitter setup, with users distributed around each base station.

A. Dataset Generation and Channel Modeling

  • The Ericsson scenario is an urban industrial site with 75% LOS users, whereas Sainte-Catherine is dense downtown with 75% NLOS links and greater blockage variability.Sainte-Catherine also has shorter link distances.Source: A. Dataset Generation and Channel… · p. 2
  • Both scenarios use an 8×8 half-wavelength-spaced uniform planar array at 2 GHz, with a 20 m-high transmitter operating at 20 W and 10 dB system loss.Users are placed 50–350 m from the base station.Source: A. Dataset Generation and Channel… · p. 2
  • Users are distributed in concentric circular patterns at 10° intervals, and ray tracing models urban multipath with up to 10 reflections from buildings and terrain.The channel datasets use site-specific environmental data from OpenStreetMap.Source: A. Dataset Generation and Channel… · p. 2

III. METHODOLOGY

The methodology combines a stacked Transformer precoder with differentiable, hardware-aware precision learning. Its energy model accounts for computation and data movement, and evaluates total network energy on continuous layer-wise precision choices.

III. METHODOLOGY

  • Automatic resolution tuning replaces discrete bit-configuration search with a differentiable framework that learns precision levels via gradient descent.The paper contrasts this approach with discrete NAS used in prior work.Source: §III · p. 2

A. Transformer Neural Architecture

  • The precoder maps concatenated real and imaginary CSI components to a complex beamforming matrix through an embedding layer, Transformer blocks, and a final linear projection.Each block uses multi-head self-attention and a SwiGLU feed-forward network, with residual connections and layer normalization.Source: A. Transformer Neural Architecture · p. 2
  • Five model capacities vary embedding dimension, attention heads, FFN width, and layer count to span approximately 0.5M to 25M parameters.The stated ranges are D = 64–256, H = 2–8, D_ff = 512–4096, and L = 2–8.Source: A. Transformer Neural Architecture · p. 2

B. Hardware-Aware Energy Modeling

  • The hardware energy model separates layer computation, weight transfer, and activation transfer, with compute and memory costs derived from MAC-energy estimates.MAC energy is calibrated against 45-nm CMOS measurements; main-memory access costs 2× EMAC and local-register access costs 1× EMAC.Source: B. Hardware-Aware Energy Modeling · p. 2
  • The model derives MHSA and SwiGLU operation counts and includes an element-wise overhead of 3N_A operations for normalization, bias, and activation.The layer-wise energy components are aggregated across the network and evaluated on continuous weight and activation precision vectors during training.Source: B. Hardware-Aware Energy Modeling · pp. 2–3

C. Automatic Resolution Tuning and Joint Optimization

The method optimizes task performance and hardware inference energy together by gradient descent, using differentiable layer-wise precision choices. It combines adjacent integer-bit quantizations to train with continuous precision variables.

C. Automatic Resolution Tuning and Joint Optimization

D. Training Protocol

Training proceeds through pretraining, joint optimization of model and precision variables, and discrete-precision fine-tuning. The procedure is repeated across λ values to map the Pareto front.

D. Training Protocol

  • Pretraining uses task loss alone for up to 300 epochs, establishing full-precision sum-rate and uniform 8-bit energy normalization anchors.Models use AdamW with cosine annealing; the learning rate depends on model size.Source: D. Training Protocol · p. 3
  • Joint training optimizes weights, quantizer step sizes, and continuous precision parameters for 120 epochs using the composite loss.Precision parameters start at 4 and use a fixed, model-size-independent learning rate; weights retain their size-dependent rate.Source: D. Training Protocol · p. 3
  • The continuous precisions are rounded and frozen, then weights and quantizer step sizes are fine-tuned on task loss for 30 epochs.The end-to-end procedure is repeated for λ ∈ {0.1, 0.5, 1, 5, 10, 25, 50}, producing a model for each value.Source: D. Training Protocol · p. 3

IV. NUMERICAL RESULTS

The evaluation compares Transformer precoders across five model capacities and seven λ settings with classical and CNN-based baselines. The figures describe the energy–sum-rate trade-off and learned layer-wise precision configurations.

IV. NUMERICAL RESULTS

  • Five Transformer footprints span Tiny (4 layers, D = 64, 0.2M parameters) to Large (8 layers, D = 256, 18.6M parameters).Each is evaluated at λ ∈ {0.1, 0.5, 1, 5, 10, 25, 50}.Source: §IV · p. 3
  • The baselines are RZF, WMMSE at 1, 5, 10, or 25 iterations, and an ART-trained CNN precoder.The CNN variants use λ = 1 for comparison.Source: §IV · p. 3
  • Fig. 1 plots sum-rate against inference energy across two sites, marking Pareto-front and dominated models alongside baselines.Fig. 2 reports learned weight and activation bit-widths and achieved sum-rates for six Pareto-front configurations.Source: Figure 1 · p. 4, Figure 2 · p. 4

A. Pareto Efficiency Analysis

The Pareto analysis finds Transformer precoders can exceed WMMSE’s peak sum-rate at substantially lower compute energy, with results varying by deployment site. It also reports that smaller models with moderate compression can outperform larger, more aggressively quantized alternatives at a fixed energy budget.

A. Pareto Efficiency Analysis

  • 125× less compute energy at Ericsson and 288× less at Sainte-Catherine accompanies Transformer sum-rate above WMMSE (I = 25) peak.These comparisons are shown in Fig. 1.Source: A. Pareto Efficiency Analysis · p. 4
  • At Ericsson, RZF slightly outperforms Transformers in energy consumption but delivers about 10% lower sum-rate than the best-performing precoders.The passage notes the gap could potentially shrink with a Transformer model smaller than Tiny.Source: A. Pareto Efficiency Analysis · p. 4
  • At Sainte-Catherine, the compressed Tiny model matches RZF throughput at 3.4× lower energy.Source: A. Pareto Efficiency Analysis · p. 4
  • At fixed energy, a smaller model with moderate compression consistently outperforms a larger, aggressively quantized alternative.For Base, increasing λ from 0.1 to 5 yields a 5.6× energy reduction.Source: A. Pareto Efficiency Analysis · p. 4
  • For Transformer models, λ = 0.1 almost always yields a Pareto-front model.The loss normalization makes one λ choice broadly applicable for generating Pareto-front models.Source: A. Pareto Efficiency Analysis · p. 4
  • Transformer precoders also support variable per-user rate requirements and maintain good performance with channel estimation errors.Source: A. Pareto Efficiency Analysis · p. 4

B. Bit-width Allocations

The learned allocations preserve higher precision in the first and last layers while often quantizing weights more aggressively than activations. The results illustrate layer-wise precision choices across Pareto-front configurations and deployment sites.

B. Bit-width Allocations

  • Fig. 2 presents learned layer-wise bit-widths for Pareto-front configurations across both deployment sites.The passage identifies the visual's scope but does not report a specific comparison from it.Source: B. Bit-width Allocations · p. 4
  • Fig. 3 concerns how the Phase 1 FP32 pretraining budget affects quantization robustness and minimum QAT epochs for specified thresholds.The caption does not state the outcomes of those comparisons.Source: Figure 3 · p. 5
  • Weights are often reduced to 1 or 2 bits, while the largest assigned resolutions are 8 bits for weights and 10 bits for activations.The optimization also retains higher precision in the first and last layers.Source: B. Bit-width Allocations · p. 5
  • The first and last layers retain higher precision, consistent with a common quantized-model training rule of thumb.This allocation is found autonomously by the optimization.Source: B. Bit-width Allocations · p. 5

C. FP32 Pretraining Budget

Sharing an FP32 pretraining phase across λ values reduces the overall training budget for multiple quantized models. The reported scratch-QAT comparison concerns a Small Transformer with λ = 5.

C. FP32 Pretraining Budget

  • About 300 QAT epochs are required for a Small Transformer trained from scratch at λ = 5 to converge.With FP32 pretraining, Fig. 3 reports minimum QAT epochs for thresholds of 90%, 92%, and 94% of RFP32.Source: C. FP32 Pretraining Budget · p. 5
  • Sharing FP32 pretraining across λ values reduces the overall training budget when training more than one quantized model.The passage motivates the two-phase training protocol with this shared-pretraining strategy.Source: C. FP32 Pretraining Budget · p. 5

V. CONCLUSION

The mixed-precision Transformer precoder achieves 288× greater energy efficiency than WMMSE at equivalent sum-rate in a dense downtown cell-site environment. The training pipeline jointly tunes weight and activation bit-widths and quantization step sizes while optimizing sum-rate and compute energy.

V. CONCLUSION

  • The training recipe combines automatic weight and activation bit-width tuning with learned quantization step sizes, optimizing sum-rate and compute energy.It replaces layer-wise bit-width search with automatic precision allocation, and evaluations on two ray-tracing channel models yield Pareto-efficient models with little hyperparameter tuning.Source: §V · p. 5
  • 288× greater energy efficiency than classical WMMSE is achieved at equivalent sum-rate in a dense downtown cell-site environment.This is the conclusion's reported mixed-precision model comparison.Source: §V · p. 5
Loading 2610.04201v1…