PaperlayerView exact revision on arXiv

Source-linked AI summary

On Complex Valued Convolutional Neural Networks

Nitzan Guberman

arXiv:1602.09046v1cs.NE

TL;DR

CNNs are powerful but prone to overfitting, motivating task-informed regularization. The paper develops a complex-valued CNN, addresses its definition and training challenges, and finds comparable cell-detection performance with greater overfitting resilience, alongside difficult optimization.

  • Problem

    CNNs are vulnerable to overfitting, motivating model restrictions based on prior knowledge and raising the question of when complex-valued CNNs are useful classifiers.

  • Method

    The paper generalizes real CNNs to complex-valued inputs and weights, proposing solutions for complex ReLU, pooling, convolution, and optimization.

  • Results

    In cell detection, the complex network showed no overfitting while the real network suffered considerably, although only 20% of trials converged to a non-local minimum.

  • Takeaways & Limitations

    Complex CNNs form a regularized subclass suited to detecting meaningful phase structure, and the trained network detected phase structure typical of cell centers.

  • Takeaways & Limitations

    The complex network has phase ambiguity, and further data are needed to determine whether the observed accuracy pattern recurs across tasks and architectures.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) are the cutting edge model for supervised machine learning in computer vision. In recent years CNNs have outperformed traditional approaches in many computer vision tasks such as object detection, image classification and face recognition. CNNs are vulnerable to overfitting, and a lot of research focuses on finding regularization methods to overcome it. One approach is designing task specific models based on prior knowledge. Several works have shown that properties of natural images can be easily captured using complex numbers. Motivated by these works, we present a variation of the CNN model with complex valued input and weights. We construct the complex model as a generalization of the real model. Lack of order over the complex field raises several difficulties both in the definition and in the training of the network. We address these issues and suggest possible solutions. The resulting model is shown to be a restricted form of a real valued CNN with twice the parameters. It is sensitive to phase structure, and we suggest it serves as a regularized model for problems where such structure is important. This suggestion is verified empirically by comparing the performance of a complex and a real network in the problem of cell detection. The two networks achieve comparable results, and although the complex model is hard to train, it is significantly less vulnerable to overfitting. We also demonstrate that the complex network detects meaningful phase structure in the data.

1 Introduction

CNNs achieve strong computer-vision performance but remain vulnerable to overfitting. This work introduces complex-valued CNNs, analyzes their properties, and compares them with real networks on cell detection.

  • CNN overfitting motivates restricting models using prior knowledge about the task.
  • Complex numbers have been used to represent image properties and synchronization effects in neural-network-related work.
  • The paper generalizes CNNs to complex-valued inputs and parameters, addressing construction and optimization difficulties.
  • The complex and real networks achieve similar performance on cell detection, while the complex model is more resilient to overfitting but harder to optimize.

2 Backgroung

This background introduces supervised learning, neural networks, and CNNs, emphasizing empirical loss, overfitting, and image-specific architectural principles. CNNs combine convolution, ReLU, and pooling with later fully connected layers.

  • 2.1 Supervised Learning: Supervised learning fits a model to training examples of inputs and expected outputs, while loss measures performance on possible input-output pairs.
  • 2.1 Supervised Learning: The empirical loss minimization rule replaces population-level optimization with optimization over a finite sampled training set.
  • 2.1 Supervised Learning: Overfitting occurs when a function fits the training set but performs poorly over the broader input distribution.
  • 2.1 Supervised Learning: Choosing a hypothesis class trades off expressiveness, approximation error, estimation error, training-data requirements, and computational complexity.
  • 2.2 Convolutional Neural Networks: CNNs organize neurons as three-dimensional arrays and exploit locality, parameter sharing, and pooling to model natural images.
  • 2.2 Convolutional Neural Networks: Typical CNNs repeat convolution, ReLU, and pooling layers before collapsing spatial structure for fully connected and activation layers.

3 Motivations - Complex Numbers and Natural Images

The paper motivates complex-valued CNNs by combining complex representations of images with synchronization-sensitive neural signals, while targeting CNN overfitting through a restricted hypothesis class.

  • Complex-valued neural networks are motivated by the observation that real-valued data such as waves can be meaningfully represented in the complex domain.
  • Scattering networks provide image representations through cascaded wavelet convolutions and absolute-value operations, but their signals do not remain complex.
  • The proposed complex CNN loosens scattering networks’ fixed-kernel and post-convolution absolute-value constraints by learning kernels while keeping representations fully complex.
  • Prior synchronization models use complex phase factors so neural outputs depend on input rates and relative phases.
  • A complex CNN is proposed as a restricted model that may regularize image-related problems by incorporating prior structure into its computations.

4 Building a Complex Neural Network

The paper generalizes real-valued CNNs to complex inputs and weights, then addresses the mathematical and optimization difficulties introduced by complex functions.

  • The model generalizes real-valued CNNs to complex-valued inputs and weights.
  • The paper develops complex-function background, identifies construction and optimization difficulties, and derives complex-valued gradient descent and backpropagation.

4.1 Complex Calculus - Preliminaries

The preliminaries establish why complex optimization requires special treatment and introduce Wirtinger derivatives for expressing differentials and ascent directions.

  • Because the complex field lacks a meaningful total ordering, the network’s loss function must be real-valued.
  • Wirtinger derivatives treat a complex variable and its conjugate as independent variables for complex-domain differentiation.
  • The total differential of complex-valued functions can be expressed using derivatives with respect to a variable and its conjugate.
  • For real-valued functions, the steepest-ascent direction is obtained from the conjugate Wirtinger derivative.
  • This result supplies the optimization tool used later to adapt gradient-based methods to complex-valued CNNs.

4.2 Network Structure

Complex CNN building blocks must replace order-dependent real operations such as ReLU and max pooling, while preserving complex representations until real-valued outputs are required.

  • Complex CNNs generalize real CNN components, but ReLU, max pooling, and optimization require alternatives because complex numbers are not totally ordered.
  • 4.2.1 ReLU: A complex ReLU passes values inside a chosen sector containing the positive real ray and zeros values outside it.
  • 4.2.1 ReLU: The sector boundaries may be fixed or selected by cross-validation, but ReLU is not differentiable with respect to those boundaries during training.
  • 4.2.2 Pooling: Complex pooling can compare inputs through projections, with max-by-magnitude selecting the input having the largest magnitude.
  • 4.2.2 Pooling: Max-by-magnitude pooling matches real max pooling only for nonnegative inputs, and may differ when positive and negative real values coexist.
  • 4.2.2 Pooling: Softmax-based pooling offers smooth transitions among max, average, and minimum operations, with its parameter potentially learned during training.
  • A real-valued projection layer, often based on squared magnitude, converts complex network outputs for tasks requiring real-valued predictions.

4.3 Network Optimization - Complex Backpropagation

The paper adapts gradient descent and backpropagation to complex-valued networks by separating real and imaginary components and handling complex activation and pooling layers.

  • Complex gradient formulation: Complex-valued optimization represents weights and activations through their real and imaginary parts, enabling real multivariate gradient and Jacobian calculations.The notation writes complex variables as Z_n = X_n + ıY_n and W_n = A_n + ıB_n.
  • Backpropagation: Backpropagation retains forward and backward passes, computing the final gradient and propagating layer derivatives in reverse order using the chain rule.The algorithm outputs derivatives with respect to the weights across all layers.
  • Affine layers: For affine layers, complex multiplication expands into coupled real equations linking X_n and Y_n through A_n and B_n.The resulting Jacobians provide derivatives for both real and imaginary output components.
  • Activation layers: Complex activation functions are differentiated componentwise when their real and imaginary outputs are represented as u and v.The compact complex form is valid under differentiability conditions discussed by the authors.
  • Activation layers: The ReLU update can remain finite despite nondifferentiability on the real or imaginary axes, whereas projection activation is real-valued and nondifferentiable.The paper presents separate derivative handling for these cases.
  • Convolution and pooling: Convolution and pooling backpropagation are decomposed into reorganization, affine or columnwise operations, and reorganization layers.Reorganization changes locations without changing values, simplifying its derivative.

4.4 Complex Convolution as a Restricted Real Convolution method

A complex convolution can be implemented as a constrained real convolution over two-channel representations. This equivalence applies to convolution layers but not generally to activation and pooling layers.

  • 66?

4.5 Complex Convolution

Complex convolution measures phase alignment between an input patch and a conjugate kernel, producing strong responses when pointwise products synchronize.

  • A complex convolution maximizes the magnitude of the summed pointwise products, favoring kernels that align the input patch’s phase structure.Multiplication rotates each complex value, and aligned phases accumulate while mismatched phases cancel.
  • Different kernels can produce the same output magnitude with different angles, creating phase ambiguity in the model.The paper states that this ambiguity is addressed later.
  • The complex convolution output can be interpreted as a heat map of similarity between the conjugate kernel’s and input patch’s phase structures.
  • Figure 6 contrasts synchronized and unsynchronized products: equal mean magnitudes yield sums whose magnitudes differ by over 20 times.The synchronized upper-row product has a common phase, whereas the lower-row product lacks meaningful phase structure.
  • Complex CNNs are proposed as a regularized hypothesis class for problems containing informative phase structure.Candidate complex representations for images include gradients, optical flow, and wavelets, while Fourier representations do not preserve locality for CNNs.

5 Empirical Study - Cell Identification

The study evaluates complex and equivalent real CNNs for cell detection using gradient-based patches. Their accuracies are comparable, but the complex model is harder to optimize, less prone to overfitting, and learns cell-related phase structure.

  • 5.1 Experimental Details: The experiment classifies cell-containing patches from simulated fluorescence microscopy gradients, using Ix and Iy for the real network and Ix + iIy for the complex network.A patch is labeled “cell” when it contains at least 10 cell pixels.
  • 5.1 Experimental Details: The complex CNN uses two 5 × 5 convolution layers with activation and pooling layers, followed by a projection layer for real-valued labels.The experiment uses ReLU, magnitude max pooling, and |·|^2 projection; the real counterpart doubles channels and kernels.
  • 5.2 Comparison With a Real Network: The real and complex models achieve comparable accuracies, with the real model performing slightly better.The comparison is made after training each model to minimize training loss without regularization.
  • 5.2 Comparison With a Real Network: The real model overfits substantially, whereas the complex model’s training and test losses remain close, suggesting a regularization effect.In the real model, training loss nearly vanishes while test loss rises; the complex losses remain between the real model’s training and test losses.
  • 5.3 Optimization Difficulties: The complex network is difficult to train: only 4 of 20 repeated trials achieved training loss close to the best result.The real network produced similar results across all repeated trials, implicating the complex formulation in the optimization difficulty.
  • 5.4 Qualitative Analysis of Kernels: A learned first-layer kernel has a phase pattern resembling a cell center, and repeated trainings produce a similar kernel.After a global phase adjustment, the kernel closely matches the cell-center phase structure; its response contains mainly vectors with positive real and imaginary parts.

6 Conclusion and Future Work

The paper formulates complex-valued CNNs as a generalization of real CNNs and analyzes their restricted structure. In cell detection, the model shows promising regularization and phase-detection behavior, but optimization remains a major obstacle requiring broader evaluation.

  • 6 Conclusion and Future Work: The complex CNN generalizes real CNNs with complex inputs and weights, requiring solutions for comparison-based operations and a modified backpropagation procedure.The lack of order in the complex field makes operations such as ReLU and max pooling ill-defined.
  • 6 Conclusion and Future Work: A complex convolution is a constrained special case of a real convolution with twice as many parameters, making the model sensitive to meaningful phase structure.The theoretical analysis characterizes the complex model as a regularized subclass of CNNs.
  • 6 Conclusion and Future Work: In cell detection, the complex model avoids overfitting and detects phase structure typical of cell centers, while comparable trials converge only 20% of the time.The real network suffers considerable overfitting, whereas the complex network’s successful trials show promising results.
  • 6 Conclusion and Future Work: Further work should address optimization difficulties and evaluate complex networks on additional tasks and inputs, including optical flow and voice signals.Broader tasks are needed to clarify the importance of phase structure and the benefits of the model’s regularization capabilities.
Loading 1602.09046v1…