PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Deep Convolutional Networks

Stéphane Mallat

arXiv:1601.04920v1stat.MLcs.CVcs.LG

TL;DR

Deep convolutional networks achieve strong results on high-dimensional problems, but the principles linking their architectures, weights, and nonlinearities to progressively powerful invariants remain complex. This paper introduces a mathematical framework centered on multiscale contractions, hierarchical symmetry linearization, and sparse separations, while identifying open requirements for a fuller theory.

  • Problem

    The paper addresses the limited understanding of how deep convolutional network architectures, weights, and nonlinearities compute progressively more powerful invariants in high-dimensional learning problems.

  • Method

    The paper develops a mathematical framework analyzing contraction and separation properties through multiscale contractions, linearized hierarchical symmetries, generalized convolutions, and sparse separations along network fibers.

  • Results

    The framework shows that invariant computations involve multiscale contractions, hierarchical symmetry linearization, and sparse separations.

  • Takeaways & Limitations

    The framework connects learned filters, local symmetries, invariances, and classification margins, and could help explain stable structures underlying transfer learning.

  • Takeaways & Limitations

    The analysis remains a framework requiring complexity measures, approximation theorems, and guaranteed convergence results for full mathematical understanding; local invariants also lose large-scale information without scale interactions.

Abstract

from arXiv · show

Deep convolutional networks provide state of the art classifications and regressions results over many high-dimensional problems. We review their architecture, which scatters data with a cascade of linear filter weights and non-linearities. A mathematical framework is introduced to analyze their properties. Computations of invariants involve multiscale contractions, the linearization of hierarchical symmetries, and sparse separations. Applications are discussed.

§1 Introduction

Deep convolutional networks address high-dimensional learning through cascades of convolutions and nonlinearities, while progressively contracting and linearizing variations associated with local symmetries. The paper frames their invariants through multiscale contractions, hierarchical symmetry linearization, and sparse separations.

  • §1 Introduction: Deep convolutional networks propagate inputs through linear convolutions and nonlinearities, typically across more than five layers.Their multilayer architecture differs from shallow ridge-function decompositions, whose analysis does not directly extend to deeper networks.
  • §1 Introduction: The paper identifies multiscale contractions, hierarchical symmetry linearization, and sparse separations as principles governing deep-network invariants.The authors present this conceptual basis as a first step toward a full mathematical understanding of convolutional network properties.
  • §1 Introduction: Supervised learning approximates functions from relatively few samples in very high-dimensional spaces, requiring strong regularity assumptions to reduce estimation complexity.For images and other large signals, the input dimension can exceed 10^6, while the sample count often grows only linearly with dimension.
  • §1 Introduction: A contractive representation Φ(x) reduces input variability while preserving distinctions between inputs with different function values.This separation-contraction trade-off must be adjusted to the properties of the target function f.
  • §1 Introduction: Deep networks progressively contract the space and linearize transformations along directions where f remains nearly constant.These directions are associated with groups of local symmetries.
  • §1 Introduction: General architectures filter and linearly combine channels at each layer, then apply contractive nonlinearities; scattering further emphasizes wavelet-based separation and linearization.The paper also studies how learned filters relate to symmetry groups and how sparsity supports separation.

§2 Linearization, Projection and Separability

The paper formulates dimensionality reduction as finding representations that preserve function values or class separation while making the target function easier to approximate linearly. It contrasts direct low-dimensional projection with higher-dimensional feature changes followed by regularized projection.

  • §2 Linearization, Projection and Separability: A representation Φ separates f when distinct function values remain distinct after transformation, with a Lipschitz condition strengthening separation for regression.The condition bounds output differences through representation distances and implies Lipschitz continuity of the induced function f0.
  • §2 Linearization, Projection and Separability: For classification, separation becomes a margin condition requiring a minimum representation-space distance between inputs from different classes.The paper also describes multiclass classification through one-versus-all binary problems.
  • §2 Linearization, Projection and Separability: A low-dimensional linear projection can separate f only when f remains constant along the discarded orthogonal directions.In most cases, the resulting dimension k cannot be much smaller than the original dimension d.
  • §2 Linearization, Projection and Separability: Linearization instead maps x into a feature space Φ(x), potentially of dimension d′ greater than d, where a low-dimensional projection can approximate f.This change of variables seeks directions along which f is constant or nearly constant.
  • §2 Linearization, Projection and Separability: When d′ exceeds the number of training samples q, the regression vector requires regularization, such as an lp norm penalty.The parameter p distinguishes sparse regressions for p ≤ 1 from kernel regressions at p = 2.
  • §2 Linearization, Projection and Separability: Classification approximates decision frontiers using a linear score sign(⟨Φ(x), w⟩), with w optimized against training error.This formulation relies on a representation that preserves class distinctions while enabling linear separation.

§3 Invariants, Symmetries and Diffeomorphisms

The paper treats invariance as preservation of f under groups of transformations and uses learned representations to linearize these transformations while maintaining separability. Translations motivate convolutional covariance, whereas diffeomorphisms require multiscale wavelet analysis.

  • §3 Invariants, Symmetries and Diffeomorphisms: A symmetry is an invertible operator g satisfying f(g.x) = f(x), and local symmetry groups preserve f within an input-dependent neighborhood.Continuous groups of operators with differential structure are called Lie groups.
  • §3 Invariants, Symmetries and Diffeomorphisms: Deep convolutional networks assume translations are local symmetries because convolutions are covariant to translations.For images, the translation group has only n = 2 generators, limiting its expressive symmetry constraints.
  • §3 Invariants, Symmetries and Diffeomorphisms: Image classification can also be locally invariant to small diffeomorphisms, which provide stronger constraints than translations but have an unknown local range.Some deformations preserve one digit class while changing another, so the valid symmetry range depends on f.
  • §3 Invariants, Symmetries and Diffeomorphisms: A change of variables Φ can locally linearize the action of group elements while keeping transformed distances controlled by the group displacement.For small transformations, Φ(x) − Φ(g.x) is closely approximated by a bounded linear operator of g.
  • §3 Invariants, Symmetries and Diffeomorphisms: Figure 1 depicts an image wavelet transform computed with a convolutional cascade over J = 4 scales and K = 4 orientations.The first arrows show the low-pass and four band-pass filters.

§4 Contractions and Scale Separation with Wavelets

Wavelet-based deep representations separate signal scales, remove oscillatory phase through nonlinearities, and average to obtain locally translation-invariant coefficients. Their contractions preserve distinctions especially for sparse coefficients, but large-scale averaging loses information unless scale interactions are captured.

  • Scale separation: Averaging over a scale 2^J provides local translation invariance and diffeomorphism stability, but removes variations above frequency 2^-J and can eliminate nearly all information at J = ∞.The averaging trade-off motivates retaining scale interactions for large-scale classification and regression.
  • Wavelet transform: Wavelet transforms separate signal variations across scales using multiscale filter convolutions, supporting the linearization of small diffeomorphisms.The transform combines low-pass averaging with band-pass wavelet filters at dilated scales.
  • Sparsity: Wavelet coefficients are mostly zero except for high-amplitude matches, creating sparse representations that are important for subsequent nonlinear contractions.The wavelets are chosen so the transform is contractive, invertible, and sparse.
  • Phase removal: The modulus removes wavelet oscillations before averaging, producing non-zero coefficients that are locally invariant at scale 2^J.A rectifier gives nearly the same result, up to a factor 2.
  • Contractions: For sparse coefficients, modulus and rectifier contractions reduce distances less when one compared coefficient is zero, helping preserve separation from other signals.This property supports reconstruction from scattering coefficients in sparse cases.
  • Scale separation: Local multiscale invariants lose large-scale structures because averaging discards information, so rich large-scale representations must also capture interactions across scales.The paper connects this challenge to complex multiscale interactions in physics.

§5 Deep Convolutional Neural Network Architectures

Deep convolutional architectures cascade linear operators and pointwise nonlinearities across translation-indexed layers and channels. Their convolutional structure preserves translation covariance, while increasing depth expands spatial support and learned filters are optimized for supervised prediction; some architectures remain vulnerable to perturbation amplification.

  • Layered architecture: A convolutional network computes each layer by applying a linear operator W_j to the previous layer, followed by a pointwise nonlinearity ρ.Layers are indexed by spatial position and channel, with spatial positions often subsampled.
  • Layered architecture: The nonlinearity ρ is contractive and may be a rectifier, sigmoid, or modulus, depending on the coefficient domain and architecture.The rectifier is ρ(α) = max(α, 0), while the modulus is ρ(α) = |α|.
  • Translation covariance: Translation covariance requires linear operators W_j to be translation-covariant, so each can be represented as a sum of convolutions.Translating the input translates the output.
  • Receptive-field growth: The cascade produces translation-covariant operators with progressively wider spatial supports as depth increases.With width-Δ filters and subsampling by 2, layer j has spatial scale Δ_j = 2^jΔ.
  • Architectural variants: Networks may add normalization, max pooling, bias subtraction, and soft thresholding, with thresholding increasing coefficient sparsity.These are described as side tricks rather than the core cascade.
  • Training: The output x_J = Φ_J(x) feeds a classifier, while supervised learning optimizes filter values to minimize training classification or regression error.The networks can contain more than 10^8 variables.
  • Stability and transfer: Some architectures amplify small input perturbations when cascaded operator norms exceed 1, although learned layers can also transfer across similar classification problems.The passage presents both instability and transferability as observed properties.

§6 Scattering on the Translation Group

The paper analyzes scattering transforms as simplified deep convolutional cascades, showing how multiscale filtering, contractions, and higher-order coefficients produce translation-invariant representations. It connects these representations to deformation stability, stationary-process statistics, reconstruction, sparsity, and classification.

  • Architecture: A deep network without channel combinations reduces to a convolution tree whose cascade can be represented using equivalent band-pass filters.Removing nonlinearities after low-pass filters reduces a depth-J cascade to m convolutions, while changing network filters changes the equivalent wavelets.
  • Architecture: The resulting operator ΦJx is a wavelet scattering transform built from multiscale convolutions, pointwise contractions, and averaging.Its coefficients are obtained by iterating convolutions and nonlinearities, with the final averaging filter providing translation-invariant outputs.
  • Scattering coefficients: Second-order coefficients capture interactions between variations at different scales, distances, and orientations that first-order coefficients miss.Because the nonlinearity is strongly contractive, coefficient amplitudes decrease quickly with order; for images and audio, energy is negligible for m ≥3.
  • Stability and invariance: Scattering representations are Lipschitz continuous to diffeomorphisms, and small deformations are thereby linearized over the coefficients.The result holds for modulus and rectifier nonlinearities and relies on commutation between wavelet transforms and diffeomorphisms.
  • Reconstruction and applications: Restricting classification vectors to orders m ≤2 supports strong results when intra-class variability is dominated by translations, small deformations, or ergodic stationary processes.The paper reports state-of-the-art results for handwritten digits, music and speech, and image textures using linear classifiers or scattering transforms.
  • Stationary processes: For stationary processes, spatial averaging estimates scattering moments, whose variance converges to zero under the weak assumption of slow mixing.Scattering moments can characterize complex multiscale properties of fractal and multifractal processes.
  • Reconstruction and applications: Inverse scattering reconstructs signals by gradient descent on coefficient discrepancies, but convergence is not guaranteed because the objective is non-convex.Reconstructions recover stationary signals with matching scattering moments; sparse wavelet coefficients can yield nearly perfect recovery up to translation, whereas non-sparse structures may be lost.

§7 Multiscale Hierarchical Convolutional Networks

The section generalizes scattering to hierarchical convolutional networks that organize local symmetries into fiberwise convolutions, while balancing contraction, linearization, and separation. Adapted filters, multiple fibers, and channel combinations preserve classification margins across increasingly complex transformations.

  • Hierarchical architecture: Hierarchical networks factor local symmetry groups across depth, replacing translation convolutions with generalized convolutions over progressively structured groups.The hierarchy begins with translations and can extend to larger local symmetry groups through semidirect-product factorizations.
  • Contraction and separation: Each layer contracts representations while preparing them for the next transformation, so consecutive operators are strongly dependent.The linear operator reduces space volume, while nonlinearities contract distances along class-preserving displacements.
  • Fiberwise computation: Filters transported along group fibers implement convolutions that can make representations invariant to local symmetries while preserving discriminative variation.Learned filters can capture large within-class variance along symmetry directions, which is then reduced by the next layer.
  • Multiscale linearization: Multiscale localized filters linearize small diffeomorphisms, including deformations associated with translations, rotations, and other hierarchical symmetries.Different filter supports are needed to handle deformations across scales.
  • Sparse separation: Multiple fibers preserve classification margins by separating support vectors, balancing dimension reduction along group fibers against an increasing number of specialized representations.The resulting sparse distributed codes encode invariant patterns and progressively more support vectors.

§8 Conclusion

The paper presents a mathematical framework for analyzing contraction and separation in deep convolutional networks, relating filters to local symmetries and sparse fiber representations. It positions the framework as an initial step that still requires complexity, approximation, and optimization theory.

  • Conclusion: The framework analyzes how deep convolutional networks contract variability while preserving classification margins through sparse separations along network fibers.Fibers combine invariance to symmetry groups with distributed pattern representations.
  • Conclusion: Network fibers may provide sufficiently stable invariant and distributed representations to help explain transfer learning.The paper presents this as a possibility within the proposed framework, not as a demonstrated guarantee.
  • Open problems: The framework remains incomplete without complexity measures, high-dimensional approximation theorems, and guaranteed convergence of filter optimization.These are identified as requirements for a fuller mathematical understanding of convolutional networks.
Loading 1601.04920v1…