Deep Complex Networks

Chiheb Trabelsi; Olexa Bilaniuk; Ying Zhang; Dmitriy Serdyuk; Sandeep Subramanian; João Felipe Santos; Soroush Mehri; Negar Rostamzadeh; Yoshua Bengio; Christopher J Pal

Source-linked AI summary

Deep Complex Networks

Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, João Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, Christopher J Pal

arXiv:1705.09792v4May 27, 2017cs.NEcs.LG

TL;DR

Deep learning lacks the building blocks needed to exploit complex-valued representations, despite their potential representational and memory benefits. The paper supplies those components for convolutional networks and convolutional LSTMs, finding competitive vision results and state-of-the-art audio-task results.

Problem
Deep learning largely uses real-valued operations, while complex-valued networks lack the frameworks needed to exploit their potential advantages.
Method
The paper formulates complex-valued building blocks, including complex batch normalization and weight initialization, and applies them to convolutional networks and convolutional LSTMs.
Results
Complex-valued models are competitive with real-valued architectures on vision tasks and achieve state-of-the-art results on MusicNet transcription and TIMIT speech spectrum prediction.
Takeaways & Limitations
The presented building blocks support competitive complex-valued networks across image, music transcription, and speech spectrum prediction tasks.
Takeaways & Limitations
Backpropagation requires cost functions and activations differentiable with respect to each complex parameter’s real and imaginary parts.

Abstract

from arXiv · show

At present, the vast majority of building blocks, techniques, and architectures for deep learning are based on real-valued operations and representations. However, recent work on recurrent neural networks and older fundamental theoretical analysis suggests that complex numbers could have a richer representational capacity and could also facilitate noise-robust memory retrieval mechanisms. Despite their attractive properties and potential for opening up entirely new neural architectures, complex-valued deep neural networks have been marginalized due to the absence of the building blocks required to design such models. In this work, we provide the key atomic components for complex-valued deep neural networks and apply them to convolutional feed-forward networks and convolutional LSTMs. More precisely, we rely on complex convolutions and present algorithms for complex batch-normalization, complex weight initialization strategies for complex-valued neural nets and we use them in experiments with end-to-end training schemes. We demonstrate that such complex-valued models are competitive with their real-valued counterparts. We test deep complex models on several computer vision tasks, on music transcription using the MusicNet dataset and on Speech Spectrum Prediction using the TIMIT dataset. We achieve state-of-the-art performance on these audio-related tasks.

1 INTRODUCTION

The paper introduces building blocks for deep complex-valued networks and applies them to convolutional architectures. Experiments find competitive image-classification performance and state-of-the-art results on MusicNet transcription and TIMIT speech spectrum prediction.

Complex representations are motivated by richer representational capacity, easier optimization, faster learning, better generalization, and noise-robust memory mechanisms.
The paper presents a general formulation of complex-valued deep-network building blocks for feed-forward convolutional networks and convolutional LSTMs.
The contributions include complex batch normalization, complex weight initialization, and a comparison of complex-valued ReLU-based activation functions.
State-of-the-art results are reported for MusicNet multi-instrument music transcription and TIMIT Speech Spectrum Prediction.
On CIFAR-10, CIFAR-100, and reduced-training-set SVHN*, complex-valued representations achieve performance competitive with respective real-valued architectures.

2 MOTIVATION AND RELATED WORK

The paper situates complex-valued networks within prior work on memory, recurrent models, biological representations, and signal processing. It identifies missing deep-learning tooling as the gap addressed by its framework.

Complex parameters have been studied for associative-memory retrieval, recurrent networks, biological representations, and signal-processing applications.
Complex weights and activations are proposed as a way to combine residual-network processing with associative-memory-like information retrieval and insertion.
Prior spectral convolution work parameterized kernels in the Fourier domain but avoided complex-valued convolutions by using real-valued spatial kernels.
Complex-valued neural networks predate deep-learning breakthroughs and have received renewed attention in more recent research.
The paper states that the absence of frameworks for complex-valued networks leaves a machine-learning tooling gap.

3 COMPLEX BUILDING BLOCKS

The paper develops a mathematical framework for complex-valued neural-network building blocks. Its representation stores real and imaginary components as separate real-valued entities while simulating complex arithmetic internally.

The framework lays down the mathematical components required to implement complex-valued deep neural networks.
A complex number z = a + ib is represented through logically distinct real-valued real and imaginary components.
Complex arithmetic is simulated internally using real-valued arithmetic on those separate components.

3.2 COMPLEX CONVOLUTION

Complex convolution is implemented by decomposing complex filters and inputs into real and imaginary parts. Distributivity then yields coupled real-valued convolutions for the output components.

The complex convolution uses a filter W = A + iB and input h = x + iy, with A, B, x, and y real-valued.
W ∗h = (A ∗x −B ∗y) + i (B ∗x + A ∗y) expresses the complex convolution through four real-valued convolutions.
The convolution operation is also represented in matrix notation for its real and imaginary components.

3.3 COMPLEX DIFFERENTIABILITY

Complex-valued networks can use ordinary real-valued backpropagation when costs and activations are differentiable with respect to each parameter’s real and imaginary parts. Restricting activations to holomorphic functions is therefore unnecessary, though holomorphic functions can reduce gradient computation.

Backpropagation requires differentiability with respect to the real and imaginary parts of each complex parameter.
Holomorphic activation functions restrict the available choices for complex-valued neural networks.
Non-holomorphic activations can be optimized with real-valued backpropagation using partial derivatives for real and imaginary parameters.
Holomorphic functions can require only two gradients instead of four because they share gradient values through the Cauchy-Riemann equations.

3.4 COMPLEX-VALUED ACTIVATIONS

The paper reviews complex-valued activations including modReLU, CReLU, and zReLU. These functions differ in whether they preserve phase, apply ReLUs componentwise, or satisfy the Cauchy-Riemann equations over restricted regions.

3.4.1 MODRELU: modReLU applies ReLU to a complex magnitude plus a learnable bias while preserving the input phase.It creates an inactive region around the origin and was used in unitary recurrent networks.
3.4.2 CRELU AND zRELU: CReLU applies separate ReLUs to the real and imaginary parts of a complex neuron.
3.4.2 CRELU AND zRELU: CReLU satisfies the Cauchy-Riemann equations when its real and imaginary parts are simultaneously strictly positive or strictly negative.
3.4.2 CRELU AND zRELU: The paper tests CReLU and zReLU in deep feed-forward complex networks, reporting their results in Table 6.4.
3.4.2 CRELU AND zRELU: zReLU is a ReLU-based complex activation that satisfies the Cauchy-Riemann equations except on two boundary sets.

3.5 COMPLEX BATCH NORMALIZATION

The paper extends batch normalization to complex values using two-dimensional whitening rather than separate mean and variance scaling. The procedure decorrelates real and imaginary components while retaining learnable complex shift and matrix-valued scaling parameters.

Complex batch normalization is proposed because standard batch normalization applies only to real values.Batch normalization is described as important for accelerating learning and, in some cases, optimizing deep networks.
Complex normalization whitens centered two-dimensional real-imaginary vectors using the inverse square root of their 2×2 covariance matrix.
Adding ϵI ensures positive definiteness and an analytically computable matrix inverse square root.
The procedure yields zero mean, unit covariance, and zero pseudo-covariance for the normalized complex values.
Normalization decorrelates the real and imaginary parts, reducing co-adaptation between the two components and the risk of overfitting.
Complex batch normalization uses a complex shift β with two learnable components and a positive semi-definite 2×2 scaling matrix γ with three learnable components.The reported initialization sets γrr and γii to 1/2, while γri and the real and imaginary parts of β are initialized to zero.

3.6 COMPLEX WEIGHT INITIALIZATION

The paper derives complex weight initialization criteria from Glorot and He schemes and parameterizes complex weights by magnitude and phase. Magnitudes use a Rayleigh distribution, while phases are sampled uniformly.

Proper initialization is important for reducing vanishing or exploding gradients, especially when batch normalization is absent.
A complex weight is represented in polar form as W = |W|e^iθ and in rectangular form as its real and imaginary components.The magnitude and phase are denoted by |W| and θ.
The paper defines complex-weight variance from the expectation of squared magnitude and the squared expectation.
The variance of a complex weight is computed from the variance and expectation of its Rayleigh-distributed magnitude.
Glorot initialization uses Var(W) = 2/(nin + nout), whereas He initialization uses Var(W) = 2/nin.The corresponding scale parameters are σ = 1/√(nin + nout) and σ = 1/√nin, respectively.
The initialization samples magnitude from a Rayleigh distribution and phase uniformly between −π and π.The experiments use a variant based on reshaped semi-unitary matrices to encourage independent kernels while respecting the chosen variance criterion.

3.7 COMPLEX CONVOLUTIONAL RESIDUAL NETWORK

The complex residual network largely follows the real-valued residual architecture while learning complex inputs and modifying stage-transition projections. Its design compares complex activations and normalization strategies under matched classification settings.

Architecture: The complex architecture largely matches the real residual network but learns the input’s imaginary component before complex processing.The learned imaginary component outperformed assuming a null imaginary part.
Activation functions: The classification comparison evaluates zReLU, modReLU, and CReLU across wide-shallow, deep-narrow, and in-between models.Most models use roughly 1.7M parameters, while modReLU models use roughly 2.5M.
Normalization: Normalization comparisons include naive complex batch normalization, complex batch normalization, and regular batch normalization for real and complex convolutions.The naive variant omits correlation handling in the complex unit.
Architecture: Stage transitions concatenate the final residual output with a same-width 1x1 convolution before 2x spatial subsampling.This differs from the real network’s projection, which doubles the feature filters while downsampling.

4 EXPERIMENTAL RESULTS

The experiments evaluate complex and real convolutional models on image classification, MusicNet transcription, and TIMIT speech spectrum prediction. Complex models are broadly competitive, with stronger results reported on the audio tasks.

Image recognition: Image experiments compare real and complex residual networks on CIFAR-10, CIFAR-100, and SVHN* using matched training procedures.The study varies width and depth under a fixed parameter budget and evaluates several activation functions.
Image recognition: On CIFAR-10 and SVHN*, real representations perform slightly better, while complex representations outperform real ones on CIFAR-100.Overall, the real and complex results are described as quite comparable.
Ablations: Ablations find that naive complex batch normalization often fails with NaNs, while CReLU converges and outperforms modReLU and zReLU.Five of six naive-normalization experiments failed, and modReLU or zReLU either failed to converge or performed substantially worse.
Automatic music transcription: The MusicNet evaluation uses 84 independent sigmoids because notes can fire simultaneously and compares raw-signal and frequency-spectrum inputs.The original audio is resampled from 44.1kHz to 11kHz for computational efficiency.
Automatic music transcription: The VGG-based deep complex MusicNet model reaches 72.9% average precision, compared with 69.6% for the deep real-valued model.The complex model uses significantly fewer parameters than its real counterpart and is reported as state of the art to the authors’ knowledge.
Speech spectrum prediction: Speech spectrum prediction compares real and complex convolutional LSTMs on TIMIT, with both models predicting the magnitude spectrum.The complex model uses 60 complex feature maps versus 84 real feature maps to match parameter counts.

5 CONCLUSIONS

The paper presents building blocks for training complex-valued neural networks and reports competitive image-classification results alongside state-of-the-art results for music transcription and speech spectrum prediction.

Conclusions: The work introduces complex batch normalization and complex weight initialization as key building blocks for complex-valued neural networks.It applies these components across complex convolutional architectures, including convolutional LSTMs.
Conclusions: Complex convolutional models achieve competitive image-classification results and state-of-the-art results on music transcription and speech spectrum prediction.The authors suggest extending complex-valued networks to more challenging generative models for audio and images.

6 APPENDIX

The appendix specifies how complex-valued convolutions, differentiability, normalization, and recurrent operations are implemented, while documenting their computational costs.

Complex convolution: Complex convolution is simulated with real-valued arithmetic by separately representing real and imaginary feature maps and kernels.The implementation uses real-valued convolutions among the real and imaginary components.
Complex differentiation: Holomorphic functions require complex differentiability at every point, characterized by the Cauchy–Riemann equations under continuous first partial derivatives.The equations are necessary, and become sufficient when the component functions have continuous first partial derivatives.
Complex differentiation: For a real-valued loss, the complex gradient combines derivatives with respect to the real and imaginary parts of each complex variable.The gradient is expressed as ∂L/∂ℜ(z) + i∂L/∂ℑ(z).
Computational complexity: 1030 MFLOPS versus 265 MFLOPS per example makes the complex architecture’s inference cost approximately quadruple the real-valued variant.The comparison ignores elementwise operations, which are described as a negligible fraction of neural-network floating-point operations.
Complex Convolutional LSTM: A Complex Convolutional LSTM replaces real convolutions with complex convolutions while retaining real-valued elementwise multiplication and separately applying activations to real and imaginary parts.The construction extends the real-valued convolutional LSTM equations to complex convolutional operations.
Complex standardization: Complex standardization subtracts the mean and applies the inverse square root of the covariance matrix to produce zero mean, unit covariance, and zero pseudo-covariance.Positive definiteness is ensured by adding ϵI through Tikhonov regularization.