Source-linked AI summary
A Survey of Complex-Valued Neural Networks
Joshua Bassey, Lijun Qian, Xianfang Li
TL;DR
Most neural-network implementations use real numbers even when application data are naturally complex, motivating study of complex-valued neural networks. This paper surveys CVNNs across activations, optimization, representations, applications, and research challenges, concluding that the field remains emerging and requires further attention.
Problem
Most neural-network implementations use real numbers despite applications in which complex numbers occur naturally or by design, motivating investigation of complex-valued alternatives.
Method
The paper performs a comprehensive survey classifying CVNNs by activation functions, learning and optimization, input and output representations, applications, and future research directions.
Results
The review presents the state of the art, including CVNN applications in signal processing and computer vision and reported results such as 99.81% accuracy in low-SNR human-activity classification.
Takeaways & Limitations
CVNNs have potential in domains where data representations are naturally complex or complex by design, but remain an emerging field requiring more research attention.
Abstract
from arXiv · showhide
Artificial neural networks (ANNs) based machine learning models and especially deep learning models have been widely applied in computer vision, signal processing, wireless communications, and many other domains, where complex numbers occur either naturally or by design. However, most of the current implementations of ANNs and machine learning frameworks are using real numbers rather than complex numbers. There are growing interests in building ANNs using complex numbers, and exploring the potential advantages of the so-called complex-valued neural networks (CVNNs) over their real-valued counterparts. In this paper, we discuss the recent development of CVNNs by performing a survey of the works on CVNNs in the literature. Specifically, a detailed review of various CVNNs in terms of activation function, learning and optimization, input and output representations, and their applications in tasks such as signal processing and computer vision are provided, followed by a discussion on some pertinent challenges and future research directions.
I. INTRODUCTION
The paper surveys complex-valued neural networks (CVNNs), motivated by applications where complex numbers occur naturally or by design. It organizes prior work by activation functions, learning and optimization, representations, applications, and future challenges.
- Motivation: Complex numbers arise in telecommunications, robotics, bioinformatics, image processing, sonar, radar, and speech recognition.The paper highlights phase spectra as encoding fine-scale temporal dependencies.
- Motivation: CVNNs process information using complex-valued parameters and variables, with complex multiplication producing phase rotation and amplitude modulation.This can reduce the degree of freedom relative to an unconstrained real-valued formulation.
- Survey scope: The survey categorizes state-of-the-art CVNNs by activation functions, learning and optimization methods, input and output representations, and applications.It also discusses similarities, differences, advantages, limitations, challenges, and future research directions.
- Survey scope: The authors describe this work as the first comprehensive review dedicated solely to complex-valued neural networks.The paper surveys the literature and structures the discussion across Sections II–VII before concluding in Section VIII.
- Learning foundations: Research on complex-valued learning extends from LMS gradients over real and imaginary parts to Wirtinger-calculus gradients with respect to complex variables.Wirtinger calculus provides a framework for obtaining gradients of complex-valued functions.
B. Why Complex-Valued Neural Networks
Complex-valued neural networks are motivated by correlated real and imaginary signal components, meaningful phase information, and complex-domain operations that constrain model representations. The section also reviews activation-function choices and the continuous and discrete MVN mappings.
- Why complex-valued neural networks: Complex multiplication constrains synaptic-weighting freedom, so a CVNN is not equivalent to a two-dimensional real-valued neural network.The paper also notes that matrix representations can increase dimensions and parameters, potentially increasing overfitting tendency.
- Why complex-valued neural networks: Phase information contributes substantially to speech intelligibility and can help recover much of the information encoded in images.The paper presents phase as relevant from both signal-processing and biological perspectives.
- Why complex-valued neural networks: Complex conjugates can support key retrieval in holographic reduced representations, while unitary matrices provide spectral representations for recurrent networks.Unitary matrices generalize orthogonal matrices to the complex plane and are used in Unitary RNNs.
- Why complex-valued neural networks: Complex-valued models preserve correlations between real and imaginary components that real-valued models may ignore, especially for frequency-domain signals.The motivation is tied to phase changes represented as circular rotations and frequency-domain shifts.
- Activation functions: The MVN activation divides the complex plane into k equal sectors and maps the entire plane onto the unit circle.Its inputs, weights, and outputs are complex, with outputs given by kth roots of unity.
- Activation functions: As k →∞, the continuous-valued MVN maps the complex plane to the entire unit circle rather than a discrete subset.The mapping uses the weighted sum z and its modulus |z|.
- Activation functions: No consensus exists on when CVNNs should use holomorphic activations versus nonholomorphic functions resembling real-valued deep-learning activations.The complex domain cannot provide a bounded, complex-differentiable activation that is entire.
IV. OPTIMIZATION AND LEARNING IN CVNNS
CVNN learning minimizes complex-valued prediction error using loss functions suited to complex regression and classification. The literature organizes training into gradient-based and non-gradient-based approaches.
- Complex mean square loss is a non-negative real-valued measure that approaches zero as complex error magnitude decreases.It is used to quantify discrepancy between desired and predicted complex outputs.
- Logarithmic error represents magnitude and phase discrepancies explicitly and decreases monotonically toward zero as predictions approach targets.The formulation is suitable for complex-valued regression.
- For classification, CVNN outputs can be transformed into the real domain using a transform that need not be holomorphic.
- CVNN training generally uses either gradient descent with error backpropagation or error backpropagation without gradient descent.These two families are listed as the main learning approaches in the literature.
A. Gradient-based Approach
Gradient-based CVNN training backpropagates error through complex-valued neurons and updates complex weights using derivatives with respect to real and imaginary components or complex variables.
- Gradient-based Approach: Complex backpropagation derives weight updates by propagating output error backward through hidden neurons using the chain rule.The process parallels real-domain learning while accounting for complex activations, inputs, weights, and outputs.
- Gradient-based Approach: Complex networks using sigmoid activation were shown to solve XOR, while later derivations covered Cartesian split and non-holomorphic activation functions.
- Gradient-based Approach: Wirtinger Calculus computes gradients, Jacobians, and Hessians directly with respect to complex variables for holomorphic and nonholomorphic functions.This avoids computing derivatives separately with respect to real and imaginary components.
- Gradient-based Approach: A complex neuron produces output f(z_j)=u_j+iv_j from complex input z_j=x_j+iy_j and complex weights, with an optional complex bias.The real and imaginary parts are tracked explicitly during derivative calculation.
- Gradient-based Approach: Weight corrections are proportional to the negative gradient, with output-layer error terms propagated to hidden layers before updates are applied.
B. Non-Gradient-based Approach
Non-gradient CVNN learning uses error-correction rules rather than derivatives, including multi-valued neurons whose weights move along the unit circle.
- Non-Gradient-based Approach: Multi-valued neuron learning is derivative-free because its activation functions are nondifferentiable, so weight correction is determined directly by neuron error.For a single neuron, learning reduces to movement along the unit circle.
- Non-Gradient-based Approach: In the example MLMVN, T denotes the target, Y12 the output, and the listed weights, outputs, weighted sum, and error define the correction process.
- Non-Gradient-based Approach: Modified learning rules normalize hidden- and input-layer corrections to address erratic weighted-sum changes or hidden-neuron outputs varying around a constant.Output-layer error backpropagation is not normalized.
- Non-Gradient-based Approach: Training terminates when the global error satisfies a condition based on a prespecified learning precision ω.
- Non-Gradient-based Approach: The approach offers ease of implementation, avoids gradient-descent local-minimum problems, and supports hybrid networks combining discrete and continuous activation functions.
C. Training and hyperparameters optimization
CVNN training research has expanded beyond basic gradient methods to activation and normalization adaptations, second-order optimization, complex learning rates, and unitary recurrent weights, while regularization remains underdeveloped.
- Training and hyperparameters optimization: Researchers have analyzed adaptations of real-valued activation functions, weight initialization, and batch normalization for complex-valued networks.
- Training and hyperparameters optimization: Second-order complex optimization uses the complex Hessian, while other work models second-order complex statistics with linear approaches and introduces complex learning rates.
- Training and hyperparameters optimization: Unitary weight matrices address vanishing gradients in recurrent neural networks while improving time and space efficiency through orthogonal-matrix properties.
- Training and hyperparameters optimization: Regularization has received little attention beyond proposed noise-based methods and is identified as an open research problem.
V. INPUT AND OUTPUT REPRESENTATIONS IN CVNNS
CVNN input and output representations can be chosen in several forms, and their suitability depends on the application and the constraints imposed on the system. Experiments found that representation and activation choices affect performance and whether a closed-form solution exists.
- Representations: Inputs may be represented as amplitude-phase, complex, real-imaginary, or augmented complex vectors, with straightforward or analytic target representations.The analytic target uses the Hilbert transform for its imaginary part.
- Experimental comparison: The real-imaginary sigmoid model performed best among the tested combinations but diverged and performed poorly when trained on real inputs.
- Analytical consequences: Certain input-output combinations yield closed-form solutions, whereas amplitude-phase representation has no closed-form solution.
- Analytical consequences: When no closed-form solution exists, transforming the input, output, or internal representation can reduce solution complexity.
- Design choice: The best representation generally depends on the application and the level of constraint imposed on the system.
VI. APPLICATIONS OF CVNNS
CVNN applications are summarized across several domains, with signal processing receiving most of the reported attention because many natural and engineered signals are complex-valued. The survey identifies radio-frequency, audio, and image signals within this application landscape.
- Application landscape: The survey summarizes various CVNN applications in Table IV.
- Application landscape: CVNNs find most of their applications in signal processing, including radio-frequency signal, audio, and image processing.
A. Applications in Radio Frequency Signal Processing in Wireless Communications
Radio-frequency signal-processing research is a major focus of CVNN work, with applications spanning communications, beamforming, coherent optical systems, source separation, and radar noise reduction.
- Wireless communications: CVNNs have been applied to channel equalization, satellite communication equalization, adaptive beamforming, and coherent-lightwave networks.
- Signal processing: Reported signal-processing applications also include source separation and adaptive noise reduction in interferometric synthetic aperture radar.
B. Applications in Image Processing and Computer Vision
CVNNs have been applied to image processing and computer vision tasks ranging from optical flow and image restoration to recognition and classification. Reported results include improved or more efficient performance in several settings, with outcomes depending on the architecture and task.
- Applications: CVNNs have been used for optical flow, holographic movies, grayscale-image reconstruction, image deblurring, gene-expression classification, character recognition, gender recognition, and image classification.
- Image classification: On CIFAR10, a complex-valued VGG model performed slightly better than a real-valued VGG in training and testing accuracy while requiring fewer parameters.
- Sequential vision tasks: On randomly ordered MNIST pixels, a unitary-RNN outperformed a real-valued LSTM with fewer than one quarter as many parameters.
- Sequential vision tasks: The real-valued LSTM required 5 to 10 times as many epochs as the unitary RNN to reach convergence in the reported MNIST comparison.
- Image filtering: A complex neural network with multi-valued neurons produced very good nonlinear image filters when trained with as few as 400 images.
- Image filtering: Filtering results improved as more training images were added in the reported simulations.
C. Applications in Audio Signal Processing and Analysis
CVNNs have been applied across audio, radar, cryptography, biomedical signals, and computer vision, often exploiting complex-valued representations of phase and amplitude. Reported benefits include improved denoising, low-SNR radar classification, performance with limited labels, and more secure neural cryptography.
- CVNNs improved denoising of noise-corrupted waveforms compared with real-valued neural networks and supported audio source localization and codec enhancement.
- Complex deep networks were evaluated for music transcription on MusicNet and speech spectrum prediction on TIMIT, with imaginary components learned from real-valued datasets.
- 99.81% accuracy was achieved by a CV-CNN for human-activity classification from short-time-Fourier-transformed radar echoes, with better performance than other approaches at low SNR.
- A complex-valued GAN retained PolSAR amplitude and phase information and performed better than state-of-the-art real-valued approaches, especially with fewer annotated data.
- A CVTPM enabled two group keys to be exchanged in one neural-synchronization process and was shown to be more secure than a real network with the same architecture.
- The FDCCNN was reported to learn discriminative features for sleep-stage prediction from ECG signals even when the dataset was imbalanced.
VII. CHALLENGES AND POTENTIAL RESEARCH
CVNN research faces unresolved challenges in training, implementation, computational cost, recurrent memory use, and regularization. The survey identifies these constraints alongside methods and directions for continued development.
- Training and activation functions: CVNN training remains difficult because complex-valued activations cannot generally be both bounded and complex-differentiable, leaving alternative activation designs as an open research area.Some studies instead use activations differentiable independently with respect to real and imaginary components.
- Implementation infrastructure: Deep learning libraries are primarily optimized for real-valued arithmetic, creating a practical need for libraries specifically targeted at complex-valued computation.Real-imaginary sub-graph implementations enable existing frameworks, but remain fundamentally based on real-valued operations.
- Initialization: Complex weight initialization can model magnitude variance with a Rayleigh distribution while initializing phase uniformly, although phase-aware initialization remains a possible extension.The cited formulation shows weight variance depends on magnitude rather than phase.
- Recurrent networks: Unitary matrices mitigate vanishing and exploding gradients in complex recurrent networks, but long sequences still require storing all hidden states during backpropagation.An invertible nonlinear function could potentially reconstruct states during the backward pass and reduce memory requirements.
- Complexity and regularization: Complex parameters increase operations and real parameter counts, potentially improving expressiveness while increasing overfitting risk and making regularization important but unresolved.Complex multiplication can require up to four real multiplications and two real additions; conventional L2 regularization also does not directly transfer because phase information is lost.