PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Quaternion Networks

Chase Gaudet, Anthony Maida

arXiv:1712.04604v3cs.NEcs.CV

TL;DR

Deep learning has largely focused on real-valued representations, while complex-valued networks suggest that richer numbers can support deeper models under fixed parameter budgets. This paper develops quaternion network components and evaluates them on classification and segmentation, finding improved convergence against real- and complex-valued networks with fewer parameters, especially for segmentation.

  • Problem

    Existing deep-learning work has focused mainly on real-valued numbers, leaving the benefits and required components of quaternion-valued networks less explored.

  • Method

    The paper develops quaternion convolution, weight initialization, and batch normalization, then trains quaternion models end to end on CIFAR-10, CIFAR-100, and KITTI road segmentation.

  • Results

    Quaternion models outperform real- and complex-valued networks on both classification datasets and KITTI segmentation while using fewer parameters; the segmentation margin is larger.

  • Takeaways & Limitations

    Deep quaternion networks provide a competitive lower-parameter alternative to real- and complex-valued networks across the evaluated classification and segmentation tasks.

Abstract

from arXiv · show

The field of deep learning has seen significant advancement in recent years. However, much of the existing work has been focused on real-valued numbers. Recent work has shown that a deep learning system using the complex numbers can be deeper for a fixed parameter budget compared to its real-valued counterpart. In this work, we explore the benefits of generalizing one step further into the hyper-complex numbers, quaternions specifically, and provide the architecture components needed to build deep quaternion networks. We develop the theoretical basis by reviewing quaternion convolutions, developing a novel quaternion weight initialization scheme, and developing novel algorithms for quaternion batch-normalization. These pieces are tested in a classification model by end-to-end training on the CIFAR-10 and CIFAR-100 data sets and a segmentation model by end-to-end training on the KITTI Road Segmentation data set. These quaternion networks show improved convergence compared to real-valued and complex-valued networks, especially on the segmentation task, while having fewer parameters

I. INTRODUCTION

Deep quaternion networks extend prior complex-valued approaches to hyper-complex representations, motivated by lower parameter costs and potentially richer processing of multidimensional signals. The paper contributes quaternion convolution, batch normalization, and weight initialization for testing these networks.

  • I. INTRODUCTION: Deep architectures need methods that improve gradient propagation while keeping parameter costs low.Batch normalization, shortcut paths, and ELU activations address training and vanishing-gradient challenges.
  • I. INTRODUCTION: Prior complex-valued networks improved learning speed and noise robustness, while quaternion networks remained comparatively underexplored.Earlier quaternion neural-network examples generally used quaternion inputs or outputs without quaternion-valued weights.
  • I. INTRODUCTION: The paper formulates and implements quaternion convolution, batch normalization, and weight initialization for deep quaternion networks.The work addresses the difficulty of quaternion normalization, including the lack of an analytic inverse square root for the relevant matrix.
  • I. INTRODUCTION: Quaternions can represent spatial transformations, rotations, and multidimensional signals, motivating their use for image and signal-processing networks.Quaternion filters couple information across axes and may treat RGB channels as a single entity.
  • I. INTRODUCTION: Quaternion representations may provide more efficient and robust memory mechanisms when combined with residual-network structures.This motivation follows the analogy between residual shortcut paths and associative memories.

III. QUATERNION NETWORK COMPONENTS

The paper develops the mathematical and representational foundations for quaternion networks, including their four-component algebra and real-valued matrix embedding. It then maps quaternion feature maps onto groups of real-valued channels for convolutional implementation.

  • A. Quaternion Representation: Quaternions consist of one real component and three imaginary components, represented as a + bi + cj + dk.Their multiplication is noncommutative, with distinct products among the i, j, and k axes.
  • A. Quaternion Representation: Quaternion arithmetic can be embedded injectively into 4 × 4 real matrices, enabling computation using real-valued operations.The paper uses one such representation, while noting that the representation is not unique.
  • Quaternion Network Components: A quaternion convolution layer with N feature maps requires N divisible by 4 and assigns successive quarters to the real, i, j, and k components.This channel arrangement implements quaternion values within a real-valued convolutional representation.

B. Quaternion Differentiability

Quaternion network operations are designed to remain differentiable with respect to all four quaternion components. Quaternion convolution combines real-valued matrices and vectors according to the noncommutative multiplication structure, coupling axes during filtering.

  • B. Quaternion Differentiability: Backpropagation requires the cost and activation functions to be differentiable with respect to the real, i, j, and k components.The paper provides a quaternion chain rule in its appendix.
  • B. Quaternion Differentiability: Quaternion convolution multiplies a quaternion filter W = A + iB + jC + kD by a quaternion vector h = w + ix + jy + kz.A, B, C, and D are real-valued matrices, while w, x, y, and z are real-valued vectors.
  • B. Quaternion Differentiability: The convolution can be implemented with a real-valued matrix whose structured entries encode quaternion multiplication.This preserves interactions among the four component axes during convolution.
  • B. Quaternion Differentiability: Quaternion convolution produces a distinct linear combination for each axis because each kernel axis interacts with each image axis.This differs from real-valued convolution, which multiplies corresponding channels independently.
  • B. Quaternion Differentiability: By reusing filters across axes and combinations, quaternion convolution treats RGB channels as a single entity and may extract cross-channel texture information.The paper frames this as a motivation for quaternion CNNs.

D. Quaternion Batch-Normalization

Quaternion batch normalization extends activation normalization to four-component values by whitening their joint covariance and then applying learned shift and symmetric scaling parameters. Cholesky decomposition supplies the whitening transform when direct inverse square-root calculation is inconvenient.

  • D. Quaternion Batch-Normalization: Real-valued batch normalization cannot directly normalize complex or hyper-complex activations to equal variance across components.Quaternion normalization therefore requires a multivariate treatment of the component statistics.
  • D. Quaternion Batch-Normalization: Quaternion whitening multiplies centered data by W, where W is obtained from a Cholesky decomposition involving the inverse covariance matrix V^-1.The covariance matrix contains pairwise covariances among the real, i, j, and k components.
  • D. Quaternion Batch-Normalization: The learned shift β is quaternion-valued, while the learned scaling γ is a symmetric matrix matching the covariance structure.The symmetric γ matrix has ten learnable parameters.
  • D. Quaternion Batch-Normalization: Quaternion batch normalization initializes γ diagonals to 1/4, its off-diagonal terms to 0, and all β components to 0.These initializations set the normalized value’s variance modulus to 1 before learning updates the parameters.

E. Quaternion Weight Initialization

The paper derives a quaternion weight-initialization scheme by relating quaternion variance to a four-degree-of-freedom magnitude distribution, then matching established Glorot and He variance targets.

  • Motivation: The scheme is motivated by the importance of proper weight initialization for convergence in deep networks.It follows the procedures of Glorot and Bengio and He et al.
  • Variance derivation: Quaternion weight variance is derived from the expected squared magnitude because symmetry around zero makes the mean-squared term vanish.The magnitude follows a four-DOF distribution used to express variance through σ.
  • Glorot initialization: Glorot initialization sets quaternion weight variance to 2/(n_in+n_out), where n_in and n_out are input and output units.The corresponding σ is obtained by equating this target with the quaternion variance expression.
  • He initialization: He initialization for ReLUs sets weight variance to 2/n_in, yielding σ = 1/√(2n_in) after matching the quaternion variance.This specializes the initialization to rectified linear units.
  • Sampling procedure: Quaternion weights are initialized by sampling magnitudes from the four-DOF distribution and angles uniformly between −π and π while enforcing the φ constraint.The magnitude uses σ selected according to the chosen initialization scheme.

IV. EXPERIMENTAL RESULTS

Experiments evaluate quaternion networks on CIFAR classification and KITTI road segmentation, using color-image benchmarks with class or pixel-level labels.

  • Benchmarks: Experiments cover CIFAR-10, CIFAR-100, and KITTI Road Estimation for image classification and image segmentation.CIFAR images are 32 × 32 color images; KITTI contains varying-size road scenes with corresponding label images.
  • Task structure: CIFAR images contain one labeled class, whereas KITTI provides per-pixel labels distinguishing road-related classes from non-road regions.The KITTI task uses binary classes as a simple segmentation test.
  • Training setup: All experiments were trained on a single Nvidia 980Ti.

A. Classification

The classification models use residual architectures with matched parameter budgets across real, complex, and quaternion networks, and quaternion models outperform the other representations on both CIFAR datasets.

  • Input representation: An additional residual block after the input learns imaginary or quaternion components because the datasets are real-valued.This block is used for the complex and hyper-complex networks.
  • Input representation: Color images can instead map grayscale to the real axis and RGB channels to the i, j, and k axes, eliminating the post-input learning block.
  • Parameter budget: Complex and quaternion networks halve and quarter the real network’s filters per layer, respectively, to maintain comparable parameter budgets.
  • Architecture: The models use three stages of residual blocks, with strided convolutions downsizing images between stages.Shallow and deep variants contain 2/1/1 and 10/9/9 residual blocks across the stages.
  • Results: Quaternion models outperform real- and complex-valued networks on both datasets with fewer parameters, but take roughly 50% longer to train.The reported training overhead is attributed to computationally intensive quaternion batch normalization.
  • Evaluation: Table I reports classification error on CIFAR-10 and CIFAR-100 together with total parameter counts.

B. Segmentation

The segmentation model adapts the classification architecture for large KITTI road images and predicts an input-sized road heatmap. Quaternion networks outperform real- and complex-valued networks by a larger margin than in classification while using fewer parameters.

  • Architecture: The KITTI model uses the small classification configuration with 2, 1, and 1 residual blocks across three stages.Strided convolutions are removed for segmentation, and only one model is used because of resource limitations.
  • Output: The final 1 × 1 convolution with sigmoid output produces a heatmap prediction the same size as the input.
  • Results: Quaternion outperformed real- and complex-valued networks by a larger margin than in classification, while using fewer parameters.Performance is measured using Intersection over Union on the KITTI benchmark.
  • Conclusion: The paper concludes that quaternion networks beat both real- and complex-valued networks with fewer parameters, but calls for testing on more segmentation datasets.

VII. APPENDIX

The appendix develops the quaternion Jacobian by representing quaternion variables through their real components and expressing the gradient componentwise.

  • A quaternion variable q is represented as q = a + i b + j c + k d, with real components a, b, c, and d.
  • The quaternion gradient is assembled from the real, i, j, and k derivatives of the real-valued loss.
  • A second quaternion variable g = m + i n + j o + k p is introduced to express q in terms of real-valued components.
  • Whitening transforms X into Z such that cov(Z) = I, thereby linearly decorrelating the input dimensions.
  • The Cholesky decomposition provides a suitable, though non-unique, method for finding the whitening matrix W.

C. Cholesky Decomposition

This section describes Cholesky decomposition as a factorization of a symmetric matrix into a lower-triangular matrix and its transpose, with entries determined row by row.

  • For a symmetric matrix A, Cholesky decomposition expresses A as A = LL′.
  • The factorization provides an efficient way to implement LU decomposition for symmetric matrices and find a matrix square root.
  • The lower-triangular factor L contains entries l_ki indexed by row k and column i.

D. 4 DOF Independent Normal Distribution

The appendix derives the distribution of the length of a four-dimensional vector with independent, zero-centered, normally distributed components by moving from its component densities to a polar representation.

  • The four-dimensional vector Y = (S, T, U, V) has independent components that are normally distributed and centered at zero.
  • The component density functions for S, T, U, and V share the same normal-density form.
  • The vector length is X = √(S^2 + T^2 + U^2 + V^2).
  • The cumulative distribution function of X is expressed using the four-dimensional sphere Hx.
  • The derivation then rewrites the integral in polar representation and differentiates the cumulative distribution function to obtain the probability density function of X.
Loading 1712.04604v3…