PaperlayerView exact revision on arXiv

Source-linked AI summary

Performance versus Complexity Study of Neural Network Equalizers in Coherent Optical Systems

Pedro J. Freire, Yevhenii Osadchuk, Bernhard Spinnler, Antonio Napoli, Wolfgang Schairer, Nelson Costa, Jaroslaw E. Prilepsky, Sergei K. Turitsyn

arXiv:2103.08212v2eess.SP

TL;DR

The paper addresses how neural-network equalizers for coherent optical channels trade performance against computational complexity. It compares conventional and newly combined architectures using Bayesian optimization, derives complexity in real multiplications per symbol, and evaluates them on numerical and experimental data. CNN+biLSTM performs best without a complexity constraint, while the three-layer MLP is best at relatively low complexity, including a 2.91 dB Q-factor improvement over conventional DSP in the experimental system.

  • Problem

    Existing NN channel equalizers require comparison across both performance and computational complexity, especially at identical complexity levels.

  • Method

    The study compares seven NN topologies, including CNN+MLP and CNN+biLSTM, after Bayesian optimization, and derives their real-multiplication complexity.

  • Results

    CNN+biLSTM is best without a complexity constraint, achieves a 2.91 dB Q-factor improvement over conventional DSP experimentally, and gives way to the simple MLP at relatively low complexity.

  • Takeaways & Limitations

    NN equalizer selection should account for computational complexity because the best-performing architecture changes when complexity is restricted.

Abstract

from arXiv · show

We present the results of the comparative analysis of the performance versus complexity for several types of artificial neural networks (NNs) used for nonlinear channel equalization in coherent optical communication systems. The comparison has been carried out using an experimental set-up with transmission dominated by the Kerr nonlinearity and component imperfections. For the first time, we investigate the application to the channel equalization of the convolution layer (CNN) in combination with a bidirectional long short-term memory (biLSTM) layer and the design combining CNN with a multi-layer perceptron. Their performance is compared with the one delivered by the previously proposed NN equalizer models: one biLSTM layer, three-dense-layer perceptron, and the echo state network. Importantly, all architectures have been initially optimized by a Bayesian optimizer. We present the derivation of the computational complexity associated with each NN type -- in terms of real multiplications per symbol so that these results can be applied to a large number of communication systems. We demonstrated that in the specific considered experimental system the convolutional layer coupled with the biLSTM (CNN+biLSTM) provides the highest Q-factor improvement compared to the reference linear chromatic dispersion compensation (2.9 dB improvement). We examine the trade-off between the computational complexity and performance of all equalizers and demonstrate that the CNN+biLSTM is the best option when the computational complexity is not constrained, while when we restrict the complexity to lower levels, the three-layer perceptron provides the best performance. Our complexity analysis for different NNs is generic and can be applied in a wide range of physical and engineering systems.

I. INTRODUCTION

The paper compares neural-network equalizers for coherent optical channels by jointly examining performance and computational complexity. It introduces CNN-based combinations, optimizes architectures with Bayesian optimization, and finds that the preferred topology depends on the complexity constraint.

  • I. INTRODUCTION: The paper frames NN equalizer selection as a trade-off between reducing BER and meeting the complexity requirements of practical implementation.The authors emphasize that suitability depends on the transmission system and implementation constraints.
  • I. INTRODUCTION: The benchmark is a dual-polarization 16-QAM link at 34.4 GBd over 9×50 km of TrueWave Classic fiber and 2 dBm launch power.The selected operating point places the system in a strongly nonlinear regime dominated by Kerr effects.
  • I. INTRODUCTION: CNN+biLSTM provides the best performance among the seven optimized NN topologies, while also having the highest complexity.The comparison uses Bayesian optimization separately for each architecture without complexity constraints.
  • I. INTRODUCTION: The study derives analytical complexity expressions in real multiplications per recovered symbol for each NN type.The expressions are intended to support comparisons across communication systems and NN equalizer designs.
  • I. INTRODUCTION: At relatively high complexity, CNN+biLSTM performs best, whereas the simple MLP outperforms advanced structures at lower complexity.This is the paper’s first comparison of different NN equalizers at identical computational-complexity levels.

II. A ZOO OF NEURAL NETWORK-BASED EQUALIZERS

This section describes the common data processing and training setup used to compare several NN equalizer architectures. The models use memory-bearing signal windows, standardized regression training, and reproducible implementation details.

  • II. A ZOO OF NEURAL NETWORK-BASED EQUALIZERS: The study reviews existing equalizers and introduces CNN+MLP and CNN+biLSTM as composite extensions of previously proposed NN configurations.The section presents the architectures before comparing their performance and complexity.
  • II. A ZOO OF NEURAL NETWORK-BASED EQUALIZERS: The NN input is a memory vector containing both polarizations’ real and imaginary components for a target symbol and 2N neighboring symbols.The memory window captures correlations introduced by chromatic dispersion and amplifier-induced channel memory.
  • II. A ZOO OF NEURAL NETWORK-BASED EQUALIZERS: All considered NNs use mean square error as the regression loss because alternative tested losses showed no noticeable benefit.The alternatives included mean absolute error, Huber loss, and Log-Cost loss.
  • II. A ZOO OF NEURAL NETWORK-BASED EQUALIZERS: Training uses Adam with learning rate 0.001 and up to 1000 epochs, with early stopping based on negligible loss changes over 150 epochs.BER is evaluated after every training epoch on an independently generated testing dataset.
  • II. A ZOO OF NEURAL NETWORK-BASED EQUALIZERS: The datasets use independently generated PRBS-32 sequences, and training data are shuffled to mitigate periodicity and overfitting.The training and evaluation sets contain 220 and 218 symbols, respectively.

A. A multi-layer perceptron

The MLP is a three-layer densely connected equalizer that jointly processes I/Q components and uses delayed inputs to represent channel memory. Bayesian optimization selects its layer-specific neuron counts and activation functions for the benchmark system.

  • A. A multi-layer perceptron: The MLP jointly processes both polarizations’ I/Q components and outputs the recovered real and imaginary parts of each symbol.This joint processing allows the equalizer to learn nonlinear phase and amplitude-related impairments.
  • A. A multi-layer perceptron: Delayed input samples provide the MLP with channel and device-induced memory information.The architecture uses time-delayed versions of the received signal to account for memory effects.
  • A. A multi-layer perceptron: Bayesian optimization identifies three MLP layers as optimal for the particular transmission scenario and optimizes the neurons and activation function in each layer.The paper uses the resulting three-layer MLP topology in its study.
  • A. A multi-layer perceptron: The three-layer MLP maps an input vector through three nonlinear hidden transformations and a final output transformation using weight matrices and biases.The matrix dimensions are determined by the input size, hidden-layer widths, and output size.

B. Long short-term memory NNs

LSTM equalizers use gated recurrent cells to model temporal channel memory, while the bidirectional form processes information from both sequence directions. This makes biLSTM suitable for intersymbol interference in coherent optical equalization.

  • B. Long short-term memory NNs: LSTM cells use input, forget, and output gates to control storage, erasure, and transmission of information across time steps.The cell state and hidden state carry information through the sequence.
  • B. Long short-term memory NNs: The LSTM cell updates its state from the current input and previous hidden state using trainable gate-specific weight matrices.The hidden-state size and input-feature count determine the relevant matrix dimensions.
  • B. Long short-term memory NNs: A biLSTM learns from both past-to-future and future-to-past sequence directions, making it suited to temporal signal processing.Its bidirectional structure is used to capture information across the memory interval.
  • B. Long short-term memory NNs: In optical channel equalization, biLSTM can handle intersymbol interference between preceding and following symbols.This is the stated advantage of bidirectional processing in the equalization context.
  • B. Long short-term memory NNs: The paper applies biLSTM to coherent optical equalization after prior work used it to compensate fiber nonlinearities in simulation.The cited prior approach also outperformed a low-complexity digital back-propagation method.

C. Echo state networks

The leaky-ESN uses a reservoir to preserve temporal signal features while training only its output weights. Its dynamic, leaking, and output layers process sequential inputs for optical equalization.

  • C. Echo state networks: The leaky-ESN preserves temporal features with relaxed training complexity and trains only the output weights.The reservoir and input weights remain fixed after initialization, while regression learns the output weights.
  • C. Echo state networks: A reservoir of interconnected neurons captures signal dynamics, followed by a static layer that integrates and leaks excitation over time.The dynamic layer forms the reservoir core, while the static layer accumulates inputs and loses excitation exponentially.
  • C. Echo state networks: The equalizer processes sequential input through dynamic, static, and output layers, with the output layer selecting states relevant to the task.The leaky-ESN used here has no output feedback connections.
  • C. Echo state networks: Prior optical-equalization studies reported a 2 dB Q-factor gain for a 64-QAM signal over 100 km using an ESN implemented in the optical domain.The cited prior work also applied a leaky-ESN digitally after analog-to-digital conversion.

D. Convolutional neural networks

The study uses convolution layers as feature-extracting preprocessors before MLP or biLSTM equalizers. It introduces CNN+MLP and CNN+biLSTM structures while restricting the convolution configuration and excluding deep CNNs.

  • D. Convolutional neural networks: CNN preprocessing is combined with either an MLP or biLSTM to form the CNN+MLP and CNN+biLSTM equalizers.The convolution layer extracts middle-level locally invariant features before subsequent processing.
  • D. Convolutional neural networks: The convolution layer applies filters across the input sequence to produce feature maps and a lower-dimensional representation retaining desirable characteristics.Its principal parameters are the filter count, kernel size, and activation function.
  • D. Convolutional neural networks: The study fixes convolution padding at 0, dilation at 1, and stride at 1, rather than investigating these additional parameters.Pooling is also omitted from the equalizer.
  • D. Convolutional neural networks: Deep CNNs are excluded because experiments found them weaker than CNN+MLP or CNN+biLSTM in the tested case.The convolution layers are therefore used only as preprocessing feature extractors.
  • D. Convolutional neural networks: The CNN+biLSTM architecture maps a four-feature time series through a 1D convolution, bidirectional LSTM, flattening layer, and two-neuron output.The two outputs represent the recovered symbol’s real and imaginary parts.

III. COMPUTATIONAL COMPLEXITY OF THE NN-BASED EQUALIZERS

The paper expresses equalizer complexity as real multiplications per recovered output symbol, using tensor dimensions and layer parameters. It derives generic costs for standalone and composite NN architectures under a real-time evaluation scope.

  • III. COMPUTATIONAL COMPLEXITY OF THE NN-BASED EQUALIZERS: Complexity is measured in real multiplications per recovered output symbol, while additions are neglected because multiplication is treated as more computationally costly.The analysis targets the evaluation phase rather than offline training.
  • III. COMPUTATIONAL COMPLEXITY OF THE NN-BASED EQUALIZERS: The framework parameterizes inputs by batch size, sequence length, and feature count, with four features and two outputs for the studied equalizers.Tensor-rank differences distinguish MLP inputs from ESN, biLSTM, and CNN inputs.
  • III. COMPUTATIONAL COMPLEXITY OF THE NN-BASED EQUALIZERS: For a three-layer MLP, complexity separates contributions from the input, hidden, and output layers using the neuron counts of each dense layer.The hidden-layer contribution includes consecutive products n1n2 and n2n3.
  • III. COMPUTATIONAL COMPLEXITY OF THE NN-BASED EQUALIZERS: A biLSTM with nh hidden units has complexity proportional to 2nsnh(4ni + 4nh + 3 + no), accounting for bidirectional processing across time steps.The expression aggregates input, recurrent, internal, and output multiplications.
  • III. COMPUTATIONAL COMPLEXITY OF THE NN-BASED EQUALIZERS: The ESN complexity includes input, dynamic reservoir, static leaking, and output operations repeated over all time steps.Its dynamic cost depends on reservoir size and connection sparsity.
  • III. COMPUTATIONAL COMPLEXITY OF THE NN-BASED EQUALIZERS: Composite CNN+MLP and CNN+biLSTM costs combine convolutional contributions with dense-layer or recurrent-layer contributions and transition corrections.The convolution output size depends on the sequence length and kernel size under the assumed valid configuration.

IV. PERFORMANCE VERSUS COMPUTATIONAL COMPLEXITY TRADE-OFF ANALYSIS

The trade-off analysis compares optimized equalizers by maximum Q-factor improvement and then evaluates six topologies at approximately matched computational complexity. This separates unconstrained performance from complexity-constrained performance.

  • IV. PERFORMANCE VERSUS COMPUTATIONAL COMPLEXITY TRADE-OFF ANALYSIS: The analysis first compares each optimized equalizer’s maximum Q-factor gain with its corresponding computational complexity.It then reduces the complexity of six NN topologies to compare performance at approximately equal complexity.
  • IV. PERFORMANCE VERSUS COMPUTATIONAL COMPLEXITY TRADE-OFF ANALYSIS: Equalizer performance is evaluated as improvement relative to the non-equalized case and against computational complexity.The matched-complexity analysis is intended to identify how optical performance depends on computational cost.

A. Experimental and numerical setups

The study evaluates NN equalizers using an experimental DP-16QAM coherent optical transmission setup and a corresponding numerical simulation. The NN receives the soft output of the regular DSP before the decision unit.

  • The experimental system transmits DP-16QAM at 34.4 Gbaud through 9×50 km of TWC fiber using DAC, electrical amplification, and a dual-polarization IQ modulator.
  • The NN input is the soft output of the regular DSP immediately before the decision unit.
  • At the receiver, coherent detection, 50-Gsamples/s sampling, chromatic-dispersion compensation, carrier-offset removal, training-based equalization, and offline DSP recover the signal.
  • The numerical setup mimics the experiment by solving the Manakov equations with split-step Fourier propagation, optical amplification, and amplified spontaneous-emission noise.

B. Optimized NN-based architectures

The equalizers are optimized with Bayesian optimization and compared using Q-factor gain, computational complexity, and training behavior. CNN+biLSTM achieves the strongest unconstrained performance in both numerical and experimental evaluations, while DBP remains less complex.

  • Bayesian optimization identifies each topology’s hyper-parameters, whose complexity is evaluated in real multiplications per recovered symbol.
  • 4.38 dB: CNN+biLSTM provides the largest numerical Q-factor improvement over conventional DSP and outperforms the other NN equalizers.Its numerical gain exceeds biLSTM by 0.05 dB, CNN+MLP by 0.47 dB, MLP by 1.4 dB, and ESN by 3.96 dB.
  • 2.91 dB: CNN+biLSTM improves experimental Q-factor over conventional DSP, exceeding biLSTM, CNN+MLP, MLP, and ESN equalizers.The corresponding gains over these equalizers are 0.15, 0.61, 0.96, and 2.33 dB, respectively.
  • Adding a convolutional layer reduces training epochs: CNN+biLSTM needs 169 versus 232 for biLSTM, while CNN+MLP needs 107 versus 753 for MLP experimentally.
  • DBP with 3 StPS is the least complex method, but CNN+biLSTM improves experimental Q-factor by 2.91 dB versus 1.32 dB for DBP.

C. Comparative analysis of different NN-based equalizers with the fixed computational complexity

Under fixed computational budgets, equalizer rankings change with complexity. CNN+biLSTM is strongest at high complexity, whereas MLP performs best at lower complexity levels, and excessive capacity can cause overfitting.

  • Experimental equalizers reach maximum gain near 10^7 RMpS, whereas simulations reach maximum performance near 10^6 RMpS.
  • Increasing complexity beyond the Bayesian-optimized level leaves gain nearly constant because oversized MLP capacity causes overfitting.
  • At high complexity, CNN+biLSTM performs best; from Topology 3 and below, the traditional MLP becomes the best-performing equalizer.
  • Reducing neurons, filters, and hidden units can place NN performance below DBP at comparable complexity; pruning and quantization are proposed alternatives.
  • ESN delivers the lowest gain, and increasing its neuron count can worsen equalization performance in high-noise nonlinear channels.

V. CONCLUSION

The study compares neural-network equalizers by performance and computational complexity, introducing CNN+MLP and CNN+biLSTM designs and deriving generic complexity expressions. In the benchmark system, CNN+biLSTM performs best without complexity constraints, whereas simple MLP equalizers become preferable at low complexity.

  • V. CONCLUSION: The study introduced CNN+MLP and CNN+biLSTM equalizers and compared them with existing neural-network methods using simulated and experimental data.The proposed architectures target coherent optical fiber-channel equalization.
  • V. CONCLUSION: The analysis derives real-multiplication complexity expressions for each neural-network type and argues that these results can apply beyond the benchmark system.The authors qualify the performance–complexity trade-off as specific to the studied system while considering the complexity analysis relatively generic.
  • V. CONCLUSION: CNN+biLSTM delivered the best performance in the benchmark system, although its optimized design had the highest complexity among the studied equalizers.The comparison used Bayesian-optimized structures, and pure biLSTM performance was only slightly lower.
  • V. CONCLUSION: At high complexity levels, CNN+biLSTM was the best-performing model, but reducing the allowed complexity made the simple MLP the best-performing structure.The transition reflects the different complexity demands of convolutional and recurrent components versus basic MLP operations.
Loading 2103.08212v2…