Spectral Representations for Convolutional Neural Networks

Oren Rippel; Jasper Snoek; Ryan P. Adams

Source-linked AI summary

Spectral Representations for Convolutional Neural Networks

Oren Rippel, Jasper Snoek, Ryan P. Adams

arXiv:1506.03767v1Jun 11, 2015stat.MLcs.LG

TL;DR

CNNs achieve strong results but require computationally efficient convolution and pooling methods. The paper uses spectral representations for pooling and filter parametrization, reporting better information preservation and 2-5 times faster convergence than standard spatial representation. It also identifies domain switching and limited spatial locality as remaining challenges.

Problem
CNNs achieve strong benchmark results, but their training requires computationally expensive convolutional kernels and pooling strategies can poorly preserve information.
Method
The paper introduces frequency-domain spectral pooling and complex-coefficient spectral parametrization of CNN filters.
Results
2-5 times faster convergence was observed for spectral filter parametrization across multiple CNN optimization tasks.
Takeaways & Limitations
Spectral representations support arbitrary-resolution pooling while retaining more information and provide a representation that facilitates CNN optimization.
Takeaways & Limitations
The DFT approach requires computationally intensive repeated transformations because nonlinearities are applied in the forward domain and lacks spatial locality for finite impulse response filtering.

Abstract

from arXiv · show

Discrete Fourier transforms provide a significant speedup in the computation of convolutions in deep learning. In this work, we demonstrate that, beyond its advantages for efficient computation, the spectral domain also provides a powerful representation in which to model and train convolutional neural networks (CNNs). We employ spectral representations to introduce a number of innovations to CNN design. First, we propose spectral pooling, which performs dimensionality reduction by truncating the representation in the frequency domain. This approach preserves considerably more information per parameter than other pooling strategies and enables flexibility in the choice of pooling output dimensionality. This representation also enables a new form of stochastic regularization by randomized modification of resolution. We show that these methods achieve competitive results on classification and approximation tasks, without using any dropout or max-pooling. Finally, we demonstrate the effectiveness of complex-coefficient spectral parameterization of convolutional filters. While this leaves the underlying model unchanged, it results in a representation that greatly facilitates optimization. We observe on a variety of popular CNN configurations that this leads to significantly faster convergence during training.

1 Introduction

The paper treats the frequency domain as both a computational tool and a representation for modeling and training CNNs. It introduces spectral pooling and spectral filter parametrization to preserve information and accelerate optimization.

Motivation: Discrete Fourier transforms can compute convolutions faster than direct spatial-domain methods, including for tiny filters.This motivates using spectral methods in CNNs beyond their computational efficiency.
Spectral representations: The frequency domain provides a representation for modeling and training CNNs, enabling applications that can be applied independently.Frequency decomposition analyzes inputs across length-scales of variation and supports spatially coherent data analysis.
Spectral parametrization: Spectral filter parametrization represents CNN filters as complex-valued frequency-domain maps without changing the underlying model.The frequency basis captures typical filter structure and provides more meaningful optimization directions.
Spectral parametrization: 2-5 times faster convergence was observed for spectral filter parametrization across multiple CNN optimization tasks.The reported result compares spectral parametrization with the standard spatial representation.
Spectral pooling: Spectral pooling reduces dimensionality by projecting inputs onto the frequency basis and truncating the representation.It is introduced as a pooling method for imposing a capacity bottleneck and facilitating computation.
Spectral pooling: Spectral pooling preserves more information for the same parameter count by removing higher frequencies that often encode noise.Typical natural-image power spectra concentrate power in lower frequencies.
Spectral pooling: Spectral pooling permits arbitrary output map dimensionality and supports randomized resolution as stochastic regularization.Resolution reduction can therefore be controlled as a function of network depth.

2 The Discrete Fourier Transform

The discrete Fourier transform decomposes spatial signals into frequency components and supports efficient convolution, approximation analysis, and gradient propagation. For real inputs, conjugate symmetry constrains the spectral representation.

The Discrete Fourier Transform: The 2D DFT decomposes a spatiotemporal signal into frequency components associated with different length-scales of variation.Its coefficients measure correlations with Fourier basis functions, while the FFT computes the transform in O(MN log(MN)) time.
Transform properties: The DFT is linear and unitary, so its inverse is the conjugate transform and Parseval’s theorem links spatial and frequency-domain ℓ2 losses.These properties make frequency-domain approximation errors and inverse-transform operations tractable.
Convolution using DFT: Convolution in the spatial domain corresponds to element-wise multiplication in the spectral domain.For inputs x and f, F(x ∗ f) = F(x) ⊙ F(f).
Conjugate symmetry constraints: Real-valued signals require conjugate symmetry in the frequency domain, leaving approximately half the map’s parameters unconstrained.Special real-valued coefficients occur at symmetry-fixed indices, while other coefficients occur in conjugate pairs.
Differentiation: Back-propagation through a DFT applies the inverse transform because the gradient is conjugated and the DFT is unitary.This describes gradient propagation through Fourier-transform layers.

3 Spectral Pooling

Spectral pooling reduces CNN map dimensionality by retaining a centered low-frequency block and reconstructing the result with an inverse DFT. It preserves information while allowing arbitrary output sizes and randomized resolution regularization.

Spectral Pooling: Spectral pooling computes a DFT, crops the centered frequency representation to H × W, handles symmetry corner cases, and applies an inverse DFT.The resulting pooled map has dimensionality H × W.
Back-propagation: The frequency truncation gradient is obtained by zero-padding gradient maps to the appropriate dimensions.Other back-propagation steps include redundancy removal, spectrum padding, and map recovery.
Computational considerations: In CNNs already using FFT-based convolution, spectral pooling can incur negligible additional computational cost because the DFT is already performed.Without FFT-based convolution, the DFTs are the computational bottlenecks of spectral pooling.
Information preservation: Spectral pooling preserves more information than max-pooling for the same number of degrees of freedom by eliminating low-amplitude higher frequencies.For spatially structured inputs, lower frequencies contain most spectral power, and Parseval’s theorem links this choice to minimized reconstruction ℓ2 distortion.
Information preservation: Unlike stride-based pooling, spectral pooling permits any arbitrary output dimensionality instead of reducing two-dimensional degrees of freedom by at least 75%.This enables gradual map-size reduction across network layers.
Randomized resolution: Randomized resolution reduction regularizes the network by sampling a truncation radius and removing frequencies outside the corresponding central square.The output map dimensionality can remain fixed while its effective resolution varies.

4 Spectral Parametrization of CNNs

The paper parameterizes CNN filters directly in the frequency domain while preserving the underlying model. This representation exploits filter structure and empirically accelerates optimization.

Spectral parametrization: Spectral parametrization learns CNN filters as complex-valued frequency-domain maps and obtains their spatial representations through the inverse DFT.The resulting spatial filters are then used in standard CNN convolution.
Spectral parametrization: The reparametrization changes only how the solution space is explored, not the underlying CNN model.It corresponds to a unitary transformation of the filters.
Leveraging filter structure: Frequency representations capture characteristic filter structure with less redundancy because many filters are localized in a narrow set of frequency components.The paper relates this structure to filters’ specific length-scales and orientations.
Optimization results: Adaptive element-wise optimizers can exploit the more meaningful axis alignment by making larger updates to fewer relevant spectral elements.The paper illustrates this mechanism quantitatively using Adam.

5 Experiments

Experiments evaluate spectral pooling and spectral parametrization across image classification and CNN optimization settings. Spectral pooling achieved competitive CIFAR results without data augmentation, while spectral parametrization produced non-negligible speedups across architectures and filter sizes.

Information preservation: Spectral pooling preserved better ImageNet reconstruction than max pooling at the same parameter count and allowed smooth selection of output dimensionality.Max pooling imposed a constraining lower bound and severe quantization, whereas spectral pooling supported arbitrary frequency truncation choices.
Classification: 8.6% on CIFAR-10 and 31.6% on CIFAR-100 were achieved by the optimal spectral pooling architecture without data augmentation.The authors describe these classification rates as competitive among approaches without data augmentation.
Classification: The optimized spectral-pooling configuration used the slowest tested map decay rate, γ = 0.85, with randomized resolution reduction.The selected constants were approximately α ≈ 0.30 and β ≈ 0.15.
Spectral parametrization: Optimization experiments compared spectral and spatial parametrizations across three CNN architectures, including the spectral pooling network.All networks used data augmentation, and the filters were initialized from identical spatial values.
Spectral parametrization: Non-negligible convergence speedup was observed even for tiny 3 × 3 filters.Training curves and speedup factors were reported for different architectures and filter sizes.

6 Discussion and remaining open problems

Spectral representations support flexible pooling and faster filter optimization, while important limitations remain in keeping networks entirely in the frequency domain and handling finite impulse response filtering.

Discussion: Spectral pooling permits any desired output dimensionality while retaining significantly more information than other pooling approaches.It reduces dimensionality by projecting onto the frequency basis and truncating the representation.
Discussion: Fourier functions provide a suitable basis for filter parametrization, yielding faster convergence during optimization.The underlying model is unchanged by the reparametrization.
Remaining open problems: Keeping an entire network in the frequency domain remains an open problem because nonlinearities require repeated transformations between domains.FFT and inverse FFT operations at each convolutional layer are described as computationally intensive.
Remaining open problems: The DFT has difficulty handling finite impulse response filtering because frequency projection uses global sums over the entire input.This creates perfect spatial locality without spectral locality, whereas the Fourier domain has the opposite locality pattern.
Remaining open problems: Wavelets are proposed as an intermediate representation combining spatial and spectral locality, with adaptive CNN learning identified as unexplored.The passage characterizes wavelets as a potentially effective middle ground between the two approaches.