PaperlayerView exact revision on arXiv

Source-linked AI summary

Phase-Aware CNN for Real-Time 5G/6G Channel Estimation with Hardware-in-the-loop Validation

Javad Zolfaghari-Bengar, Rakibul Rony, Elisa Gomez-de-Lope, Alejandro Villena-Rodriguez, Abhinav Mahadevan, Nicolas Kourtellis

arXiv:2608.14676v1eess.SPcs.LG

TL;DR

Real-time 5G/6G channel estimation remains challenged by phase discontinuities, limited configuration generalization, and reliance on synthetic data. The paper proposes a lightweight CNN with sine–cosine phase encoding and testbed-derived evaluation, achieving lower magnitude and phase MSE than LS while supporting edge deployment.

  • Problem

    Deep-learning channel estimators still face phase discontinuities, limited generalization across UE and antenna configurations, and insufficiently realistic evaluation data.

  • Method

    The paper combines continuous sine–cosine phase features with a lightweight CNN and evaluates channel reconstruction using hardware-enabled O-RAN testbed data.

  • Results

    Magnitude MSE is 0.0105 versus 0.0860 for LS, while phase MSE is 0.7822 versus 1.9554 for LS.

  • Takeaways & Limitations

    The proposed model provides a deployment-oriented channel-estimation approach with stable reconstruction, cross-scenario generalization, and inference latency suitable for real-time edge use.

Abstract

from arXiv · show

In 5G/6G wireless systems, accurate and timely channel estimation is critical to ensure reliable communication under complex, fast-changing radio conditions. This work focuses on pilot-based channel estimation using deep learning to reconstruct both magnitude and phase across the full subcarrier grid, with particular emphasis on evaluation using emulated data collected from an end-to-end O-RAN testbed. The testbed includes hardware in the loop and controlled channel emulation to better reflect deployment conditions beyond pure software simulation. It addresses major limitations in classical estimators such as LS and MMSE, as well as deep learning-based approaches that struggle with phase prediction due to discontinuities at $\pm π$, poor generalization to different UE and antenna configurations, and computational inefficiency for real-time deployment. The proposed system combines a phase-aware input encoding using sine and cosine representations with a lightweight Convolutional Neural Network (CNN) architecture. This design achieves high accuracy, stable phase reconstruction, strong generalization across testbed-derived datasets, and real-time inference suitable for edge devices.

I. INTRODUCTION · II. RELATED WORKS · A. Classical Channel Estimation

The paper motivates phase-aware, pilot-based channel estimation for rapidly varying 5G/6G channels by identifying limitations in classical and deep-learning estimators. It proposes a lightweight sine–cosine CNN evaluated on testbed-derived data to improve stable magnitude and phase reconstruction across diverse environments.

  • I. INTRODUCTION: Accurate pilot-based channel estimation is fundamental for reliable 5G/6G communication, supporting robust demodulation, interference management, and beamforming under rapidly time-varying channels.Higher carrier frequencies, denser deployments, and diverse mobility profiles increase tracking demands on user equipment and base stations.
  • A. Classical Channel Estimation: LS is computationally efficient but performs poorly in low-SNR or interference-heavy environments because it is noise-sensitive and ignores channel statistics.These weaknesses are especially relevant in high-mobility and millimeter-wave scenarios.
  • A. Classical Channel Estimation: MMSE can improve accuracy by using channel and noise covariance information, but accurate covariance models are difficult to obtain under hardware impairments and changing propagation conditions.MMSE also depends heavily on covariance matrices that may be unavailable in practical deployments.
  • II. RELATED WORKS: Deep-learning estimators show promise in OFDM and MIMO scenarios but remain challenged by phase discontinuities at ±π, limited configuration generalization, and inference complexity.These issues restrict reliable deployment across diverse UE and antenna configurations and on edge devices.
  • I. INTRODUCTION: Existing studies rely extensively on synthetic statistical or geometry-based datasets that omit hardware nonlinearities, calibration offsets, quantization, mutual coupling, and implementation-specific RF impairments.Such effects significantly influence received pilots and channel responses.
  • I. INTRODUCTION: The proposed phase-aware CNN uses continuous sine–cosine phase features, a lightweight convolutional structure, and robust normalization for stable magnitude and phase reconstruction across diverse 5G channel environments.Its evaluation uses testbed-derived datasets, addressing limitations associated with purely synthetic data.
  • A. Classical Channel Estimation: MMSE requires high-dimensional matrix inversion, making it computationally prohibitive for large MIMO configurations despite extensions such as Bayesian MMSE and sparsity-aware estimation.These extensions still depend on assumptions that may not hold in practical systems.

B. Deep Learning-Based Channel Estimation

Deep-learning channel estimation has expanded beyond classical methods through diverse neural architectures and related physical-layer applications. However, reliance on synthetic data, limited robustness across conditions, and lack of hardware-enabled validation continue to constrain practical 5G/6G deployment.

  • Deep Learning-Based Channel Estimation: CNNs, RNNs, transformer-based networks, and hybrid architectures have been proposed for OFDM and MIMO channel estimation.These approaches provide flexible, data-driven alternatives to classical estimators.
  • Deep Learning-Based Channel Estimation: Existing deep-learning estimators struggle with phase discontinuities at ±π, domain shift, and generalization across channel models and antenna configurations.These challenges limit robustness beyond the conditions represented during training.
  • Deep Learning-Based Channel Estimation: Most related deep-learning studies rely heavily on synthetic datasets generated with simplified 3GPP or geometric channel models, which can impair generalization to real deployments.The passage identifies simulation-only training as a source of deployment mismatch.
  • Deep Learning-Based Channel Estimation: Meta-learning, reinforcement learning, and federated learning address generalization or channel-tracking needs but introduce computational cost, communication overhead, or instability under non-IID data.These methods broaden the design space while creating additional deployment challenges.
  • Deep Learning-Based Channel Estimation: Heavy dependence on simulated datasets and limited validation on hardware-enabled testbeds remain major limitations for practical 5G/6G deployment.The limitation applies to the broader deep-learning channel-estimation literature.

III. METHODOLOGY · A. CNN Model · B. Phase Representation

The methodology normalizes DMRS observations and uses a lightweight CNN to estimate the full MIMO channel response from structured spatial–frequency inputs. Phase is represented with sine and cosine components to avoid ±π discontinuities and support stable regression.

  • III. METHODOLOGY: Known TxDMRS pilots traverse RF impairments, multipath fading, and noise before RxDMRS is processed for channel estimation.The pipeline follows the uplink signal structure while avoiding intermediate analytical estimates associated with classical LS and MMSE methods.
  • III. METHODOLOGY: RxDMRS is power-normalized to match TxDMRS, producing a stable input that reflects propagation- and noise-induced distortions.This compensates for transmit-power and UE–BS link-budget variations before learning the full channel response.
  • A. CNN Model: The lightweight CNN captures spatial and frequency-domain correlations in MIMO channels while maintaining a memory footprint under 1 MB for edge deployment.Its layer configuration, including kernels, filter counts, activations, and tensor shapes, is summarized in Table II.
  • A. CNN Model: Sequential convolutions with residual connections learn hierarchical antenna- and pilot/subcarrier patterns using small antenna kernels and elongated frequency kernels.Batch normalization stabilizes feature distributions, improving convergence and reducing overfitting.
  • B. Phase Representation: The complex channel is parameterized by magnitude and phase, with phase encoded through sine and cosine components instead of raw wrapped values.This removes ±π discontinuities, preserves circular structure, and provides smooth regression targets.
  • B. Phase Representation: Magnitude, sin(θ), and cos(θ) form structured 3D tensors across antenna dimensions, subcarriers, and feature channels.These tensors expose spatial–frequency dependencies that encode multipath structure and frequency-selective fading patterns.
  • B. Phase Representation: During inference, phase is reconstructed from the predicted sine and cosine components, yielding a continuous circular representation for smoother regression.The representation avoids the artificial discontinuity at ±π found in direct phase prediction.

C. Robustness to Noise

The CNN improves noise robustness by suppressing uncorrelated noise through learned channel structure, while an SNR sweep evaluates estimation quality under injected additive Gaussian noise, including conditions constrained by hardware noise floors.

  • Noise robustness: The CNN suppresses uncorrelated noise by exploiting smoothness across subcarriers, spatial antenna correlation, and typical multipath signatures.These data-driven structural priors contrast with LS estimates, whose error variance is proportional to the noise variance.
  • Noise robustness: An SNR sweep isolates noise effects by holding ground-truth channels and transmitted pilots constant while varying additive Gaussian noise injected into RxDMRS.The evaluation is designed to assess whether estimation quality remains stable as SNR decreases.
  • Noise robustness: Hardware-derived data impose a non-negligible noise floor due mainly to non-ideal real RF components and commercial O-RAN equipment.This hardware constraint affects the noise conditions represented in the evaluation data.

IV. EXPERIMENTAL SETUP · A. Dataset and Testbed Description

The experiments use diverse 5G NR propagation scenarios and a hardware-in-the-loop O-RAN testbed to evaluate channel estimation under realistic RF and implementation effects. The setup spans varied UE placements, LOS conditions, antenna configurations, and dataset splits.

  • A. Dataset and Testbed Description: Three 3GPP channel models—RMa, UMa, and UMi—vary UE placement between indoor/outdoor deployments and propagation between LOS/NLOS conditions.These combinations support training and evaluation across diverse multipath and scattering characteristics.
  • A. Dataset and Testbed Description: The hardware-in-the-loop testbed integrates RF hardware, channel emulation, and AI processing, with two Keysight MXG N5182B generators emulating independent UEs transmitting uplink DMRS.A PROPSIM F8800B channel emulator applies imported channel impulse responses matching the selected scenarios.
  • A. Dataset and Testbed Description: Hardware-in-the-loop measurements capture calibration mismatches, quantization effects, synchronization imperfections, phase noise, and implementation-specific nonlinearities.The resulting evaluation reflects both propagation-induced channel variations and implementation-related distortions affecting received DMRS observations.
  • IV. EXPERIMENTAL SETUP: The wireless configuration uses 5G NR uplink at 3.5GHz with 100MHz bandwidth, 4096 FFT size, and 273 resource blocks.The listed setup defines the wireless-system parameters used throughout the experiments.
  • A. Dataset and Testbed Description: The setup uses DMRS ports [0, 2], DMRS Configuration Type I, single-symbol DMRS duration, and DMRS Type A position 2.These parameters specify the pilot configuration used for the uplink experiments.
  • A. Dataset and Testbed Description: The experiments use 300 ns delay spread, a 1 × 4 uplink MIMO configuration, 2 UEs, and 4 base-station receive antennas.The evaluated channel scenarios are RMa, UMa, and UMi under LOS/NLOS and indoor/outdoor conditions.
  • A. Dataset and Testbed Description: The dataset is divided into 12000 / 1000 / 2000 samples per scenario for training, validation, and testing.These splits are reported for each channel scenario.

B. Implementation details

The system was implemented in TensorFlow with GPU acceleration, jointly training magnitude and phase-related outputs using MSE. LS and LMMSE baselines used the same DMRS observations, with LMMSE serving as an oracle frequency-domain reference.

  • Model implementation: TensorFlow GPU acceleration supported training with joint MSE loss over magnitude, sin(θ), and cos(θ).Training used Adam with a learning rate of 0.001, batch size 32, and 50 epochs.
  • Model implementation: Approximately 30 ms per batch was measured for GPU inference in an unoptimized implementation.The latency is reported at batch level.
  • Baseline implementation: LS and LMMSE baselines were implemented using the same received DMRS observations as the CNN.LS divides received DMRS symbols by the corresponding transmitted DMRS sequence for each subcarrier, receive antenna, and UE.
  • Baseline implementation: The LMMSE baseline operates in the frequency domain using LS channel vectors and an empirical covariance matrix Rhh pooled across realizations, antennas, and UEs.Rhh is estimated from ground-truth channel realizations across the subcarrier axis.
  • Baseline limitations: Because Rhh uses available ground-truth channel realizations, LMMSE is an oracle frequency-domain reference rather than a fully deployable receiver implementation.It provides a strong second-order statistical benchmark.

V. RESULTS

The results section evaluates the phase-aware CNN estimator across quantitative metrics, qualitative comparisons, cross-scenario generalization, and noise robustness, including comparisons with least-squares estimation.

  • Evaluation scope: The experiments assess the proposed phase-aware CNN estimator using quantitative metrics, qualitative comparisons, cross-scenario generalization, and noise robustness.The evaluation is described as comprehensive and spans these four dimensions.
  • Experimental setup: The results include a dataset summary supporting the experimental evaluation.A dedicated dataset summary is provided in Table V.
  • Baseline comparison: The proposed architecture is experimentally compared with least squares.Table VI reports experimental results comparing the proposed architecture and least squares.

A. Quantitative Results

The proposed phase-aware CNN substantially improves channel estimation over classical LS, reducing both magnitude and phase MSE while reconstructing smoother profiles across the full frequency grid.

  • Benchmark comparison: The benchmark compares the proposed AI-based phase-aware CNN with classical Least Squares estimation using magnitude and phase MSE.The comparison evaluates subcarriers sampled at 1 in 8.
  • Magnitude and phase performance: 0.0105 magnitude MSE nearly matches an order-of-magnitude improvement over LS’s 0.0860 error.The CNN suppresses noise and reconstructs smoother magnitude profiles across the full frequency grid.

B. Qualitative Results · C. Generalization Results · D. Robustness Against Noise

The proposed CNN produces coherent channel magnitude and phase estimates, generalizes best when trained on multipath-rich UMa channels, and maintains the lowest MSE across SNR conditions, especially at low and moderate SNRs.

  • B. Qualitative Results: The CNN produces coherent magnitude and phase structures closely matching smooth multipath profiles, whereas LS estimates show noisy, erratic subcarrier fluctuations.The comparison is shown in Fig. 3.
  • C. Generalization Results: Models trained on UMa channels generalize more strongly to RMa and UMi scenarios than models trained on UMi LOS conditions.The richer multipath structure of UMa channels provides more diverse training samples.
  • C. Generalization Results: Limited propagation diversity in UMi LOS conditions reduces generalization to other environments.The results emphasize training on multipath-rich scenarios for robustness across real-world deployment conditions.
  • D. Robustness Against Noise: The CNN consistently achieves the lowest MSE across the evaluated SNR range, outperforming LS and the oracle frequency-domain LMMSE baseline.The MMSE curve uses the same hardware-derived channel realizations and received DMRS observations as the CNN.
  • D. Robustness Against Noise: The CNN’s performance gain is particularly pronounced at low and moderate SNRs, where it significantly outperforms both LS and MMSE.All methods show decreasing error as SNR increases.
  • D. Robustness Against Noise: The proposed model learns noise-resilient representations that remain effective under both noise-limited and high-SNR conditions.Robustness is evaluated across a wide range of SNR conditions.
  • C. Generalization Results: Cross-scenario MSE evaluation covers RMa, UMa, and UMi settings, with diagonal entries representing within-scenario testing and off-diagonal entries measuring generalization.Lower MSE values indicate better performance.

VI. CONCLUSION

The work presents a lightweight phase-aware CNN that combines magnitude features with sine–cosine phase encoding for stable, real-time 5G/6G channel estimation. Hardware-in-the-loop O-RAN validation shows strong generalization and noise robustness, supporting deployment in AI-native wireless systems.

  • Architecture: The lightweight CNN combines magnitude features with sine–cosine phase encoding to avoid phase discontinuities and enable stable, physically consistent learning.The architecture is designed for real-time edge deployment and outperforms classical LS and MMSE estimators in magnitude and phase accuracy.
  • Validation: Hardware-in-the-loop measurements from an end-to-end O-RAN testbed and controlled channel emulation validate the model across diverse 3GPP propagation scenarios.Generalization is particularly strong when training uses multipath-rich urban macro scenarios.
  • Robustness: The network consistently maintains low estimation error under degraded conditions by learning noise-resilient spatial–frequency representations across a wide SNR range.These findings position the approach as practical and deployment-ready for AI-native wireless communication systems.
Loading 2608.14676v1…