PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling the Scattering Transform: Deep Hybrid Networks

Edouard Oyallon, Eugene Belilovsky, Sergey Zagoruyko

arXiv:1703.08961v2cs.CVcs.LG

TL;DR

The paper asks whether early learned CNN layers can be replaced by simpler non-learned operators that reduce variability while preserving discriminative information. It combines scattering representations with supervised local encoders or CNNs, achieving competitive large-scale recognition and strong results in unsupervised and small-sample settings.

  • Problem

    The paper investigates whether early CNN layers can be replaced by simpler non-learned operators that reduce variability while retaining discriminative information.

  • Method

    The paper combines a fixed scattering network with supervised 1 × 1 local encoders or standard CNN architectures such as ResNet.

  • Results

    The hybrid approach achieves AlexNet accuracy on ImageNet, similar accuracy to ResNet-18 with reduced depth, and state-of-the-art results for unsupervised CIFAR-10 features.

  • Takeaways & Limitations

    Pre-defined geometric features remain useful for interpretable and competitive visual recognition, especially in small-sample regimes.

  • Takeaways & Limitations

    The approach assumes that well-known geometric symmetries provide a suitable initialization, while discovering subsequent non-geometrical symmetries remains difficult.

Abstract

from arXiv · show

We use the scattering network as a generic and fixed ini-tialization of the first layers of a supervised hybrid deep network. We show that early layers do not necessarily need to be learned, providing the best results to-date with pre-defined representations while being competitive with Deep CNNs. Using a shallow cascade of 1 x 1 convolutions, which encodes scattering coefficients that correspond to spatial windows of very small sizes, permits to obtain AlexNet accuracy on the imagenet ILSVRC2012. We demonstrate that this local encoding explicitly learns invariance w.r.t. rotations. Combining scattering networks with a modern ResNet, we achieve a single-crop top 5 error of 11.4% on imagenet ILSVRC2012, comparable to the Resnet-18 architecture, while utilizing only 10 layers. We also find that hybrid architectures can yield excellent performance in the small sample regime, exceeding their end-to-end counterparts, through their ability to incorporate geometrical priors. We demonstrate this on subsets of the CIFAR-10 dataset and on the STL-10 dataset.

1. Introduction

The paper investigates whether early learned CNN layers can be replaced by fixed geometric representations while retaining discriminative information. It combines scattering transforms with supervised architectures to improve interpretability, data efficiency, and image-classification performance.

  • Motivation: Pre-defined geometric priors may improve low-sample performance, interpretability, and the spatial efficiency of learned modules.The authors identify these as potential advantages of hybrid representations combining fixed and learned features.
  • Motivation: Scattering transforms retain discriminative information while introducing geometric invariance and stability beyond traditional local descriptors such as SIFT.The paper presents scattering as a more generic representation for building further supervised modules.
  • Related work: The work contrasts scattering-based hybrids with prior fixed-representation pipelines that used Fisher-encoded SIFT and neural-network classifiers.The proposed approach instead combines scattering transforms with convolutional architectures.
  • Contributions: Shallow 1 × 1 convolutions achieve AlexNet accuracy on ImageNet while explicitly learning invariance to local rotations.These convolutions provide supervised local descriptors over scattering coefficients.
  • Contributions: Combining scattering with a reduced-depth ResNet obtains accuracy similar to ResNet-18 on ImageNet, while scattering substantially improves accuracy in limited-data settings.The paper evaluates these hybrid architectures on ImageNet and limited-data benchmarks.
  • Implementation: A highly efficient GPU implementation enables applying scattering on the fly when training very deep networks.The implementation and pretrained hybrid models are made available.

2. Scattering Networks and Hybrid Architectures

Scattering networks provide fixed, mathematically grounded representations that build translation-stable features through wavelet transforms, modulus nonlinearities, and averaging. Supervised local encoders or CNNs then learn from these representations while preserving locality and reducing the spatial burden of learned convolutions.

  • 2.1. Scattering Networks: The scattering transform uses fixed wavelet filters and local averaging to create a generic representation that is stable to geometric transformations.Its parameters generally do not need to adapt to dataset bias.
  • 2.1. Scattering Networks: Local averaging produces approximate invariance to translations smaller than 2J but discards high-frequency information needed for discrimination.The scattering construction addresses this loss by recovering information with wavelets.
  • 2.1. Scattering Networks: Wavelets select scale and angle, while modulus nonlinearities followed by averaging build nontrivial translation invariants.The first-order coefficients are formed as S1x = AJ|W1|x.
  • 2.1. Scattering Networks: Second-order scattering applies another wavelet transform after the first modulus, recovering high-frequency information lost during averaging and retaining only increasing scale paths.Higher-order scatterings are not computed because their energy is described as negligible.
  • 2.2. Cascading a supervised Deep architecture: Supervised CNNs are added because scattering performs well when variability is known but leaves a performance gap against supervised representations on complex image datasets.The learned network is intended to address non-geometrical intra-class variability.
  • 2.2. Cascading a supervised Deep architecture: The hybrid design treats roto-translation symmetry as a useful fixed initialization while avoiding learning these known symmetries in the earliest layers.The authors note that discovering later non-geometrical symmetries remains difficult.
  • 2.2. Cascading a supervised Deep architecture: Hybrid CNNs reduce the spatial dimensions on which convolutions are learned and can reduce sample complexity.The paper considers both Shared Local Encoders and standard CNNs such as ResNet on top of scattering.
  • 2.3. Shared Local Encoder for Scattering Representations: A Shared Local Encoder applies identical fully connected layers to each scattering descriptor using 1 × 1 convolutions, preserving non-overlapping locality.Scattering coefficients at each position describe a local neighborhood of size 2J, and fully connected layers are cascaded above the encoder for classification.

3. Local Encoding of Scattering

The SLE combines scattering coefficients with shallow supervised local encoders, achieving AlexNet-level ImageNet accuracy while revealing sparse, rotation-invariant structure in its first layer.

  • Architecture: The SLE uses a scattering representation followed by three 1×1 convolutional layers and fully connected layers for supervised local encoding.The scattering output is spatially organized into local descriptors, which the shared encoder processes at each position.
  • ImageNet evaluation: 20.4% Top 5 error and 43.0% Top 1 error were obtained on single-crop ILSVRC2012, with performance analogous to AlexNet.The result compares favorably with prior local encoding methods using SIFT and Fisher Vector pipelines.
  • ImageNet evaluation: 43.0% single-crop Top 1 error for SLE nearly matches AlexNet’s 43.1%, despite using six learned layers and constant 16×16 receptive fields.The fully connected layers provide the later spatial mixing needed by the architecture.
  • Local descriptor evaluation: The local SLE descriptor is also evaluated on Caltech-101, where raw scattering features achieve 62.8 ± 0.7 accuracy before learned features are considered.The benchmark uses 30 training samples per class and linear-SVM evaluation.
  • First-layer analysis: The first SLE layer inherits scattering structure and explicitly builds invariance to local rotations, with Fourier bases providing a natural operator basis.The analysis examines restrictions of F1 to zeroth-, first-, and second-order scattering coefficients.
  • First-layer analysis: Thresholding 80% of Fourier-domain coefficients to zero reduces ILSVRC2012 Top 1 and Top 5 performance by only 2% absolute without retraining.This supports a sparse approximation of the first layer in the frequency basis.
  • First-layer analysis: Energy concentrated in low angular frequencies indicates that F1 builds explicit invariance to local rotations.The left and right panels measure energy from Eq. 1 for the first- and second-layer operators.

4. Numerical performances of hybrid networks

Hybrid scattering networks achieve competitive or superior performance across large-scale, supervised, unsupervised, and limited-sample image-classification settings. Their strongest advantage appears in smaller-data regimes, where geometric priors improve results over comparable end-to-end CNNs.

  • Limited-Sample Experiments: The hybrid approach obtains substantial improvement over analogous end-to-end CNNs in limited-data evaluations on CIFAR-10 and STL-10.The experiments use decreasing CIFAR-10 subset sizes and a 10-fold STL-10 procedure.
  • Deep Hybrid CNNs on ILSVRC2012: 11.4% Top 5 error matches a comparable ResNet-18 at 11.1% while using fewer layers and similar parameter counts.The hybrid model uses 10 learned layers and 12.8M parameters versus 11.7M for the reference.
  • Hybrid Supervised and Unsupervised Representations on CIFAR-10: A scattering representation with a three-hidden-layer classifier sets a new state of the art using only unsupervised features on CIFAR-10.It outperforms methods using both learned and non-learned unsupervised features.
  • Hybrid Supervised and Unsupervised Representations on CIFAR-10: 93.1% accuracy is achieved by the supervised hybrid CIFAR-10 model, outperforming several benchmarks but trailing the original ResNet and Wide ResNet.The authors note that training and regularization procedures were largely inherited from networks trained directly on natural images.
  • CIFAR-10 Limited-Sample Experiments: Hybrid networks outperform purely CNN-based baselines particularly at 500 and 100 training samples in limited-sample CIFAR-10 experiments.The comparison averages accuracy over five different subsets.

5. Conclusion

The paper presents scattering-based hybrid networks as a competitive and interpretable alternative for large-scale recognition and limited-data classification. It argues that fixed geometric representations remain useful when combined with suitable learned modules.

  • Conclusion: Scattering-based hybrid networks achieve competitive large-scale recognition and state-of-the-art results in unsupervised CIFAR-10 and small-data CIFAR-10 and STL-10 settings.The conclusion specifically highlights ILSVRC2012, CIFAR-10, and STL-10 evaluations.
  • Conclusion: Pre-defined features remain useful for interpreting deep-learning techniques and may support more theoretically grounded and stable deep models.The authors present these as potential benefits when pre-defined features are combined with appropriate learning methods.
Loading 1703.08961v2…