PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured Receptive Fields in CNNs

Jörn-Henrik Jacobsen, Jan van Gemert, Zhongyu Lou, Arnold W. M. Smeulders

arXiv:1605.02971v2cs.CV

TL;DR

Limited training data make CNN learning difficult, and pre-training is less effective when the target domain differs from its source. The paper represents CNN receptive fields as learned combinations of a fixed filter basis; RFNNs perform strongly across small and medium datasets and outperform Scattering on large datasets.

  • Problem

    CNNs can overfit when training data are limited, while pre-training performance decreases when the target domain differs from the source domain.

    Source: §1 · p. 1, §2.2 · p. 2

  • Method

    RFNNs express receptive fields as linear combinations of a fixed filter basis and learn the combination weights, retaining CNN filter-learning flexibility.

    Source: §1 · pp. 1–2

  • Results

    RFNNs outperform CNNs by large margins when data are scarce, can outperform them on challenging medium datasets, and are superior to Scattering on large datasets.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The Gaussian derivative basis connects CNNs with multiscale image analysis and provides a smooth-feature prior, especially relevant where model insight and control matter.

    Source: §5 · p. 8

    The basis relies on the assumption that orders beyond the chosen maximum of four do not carry information meaningful to visual perception.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Learning powerful feature representations with CNNs is hard when training data are limited. Pre-training is one way to overcome this, but it requires large datasets sufficiently similar to the target domain. Another option is to design priors into the model, which can range from tuned hyperparameters to fully engineered representations like Scattering Networks. We combine these ideas into structured receptive field networks, a model which has a fixed filter basis and yet retains the flexibility of CNNs. This flexibility is achieved by expressing receptive fields in CNNs as a weighted sum over a fixed basis which is similar in spirit to Scattering Networks. The key difference is that we learn arbitrary effective filter sets from the basis rather than modeling the filters. This approach explicitly connects classical multiscale image analysis with general CNNs. With structured receptive field networks, we improve considerably over unstructured CNNs for small and medium dataset scenarios as well as over Scattering for large datasets. We validate our findings on ILSVRC2012, Cifar-10, Cifar-100 and MNIST. As a realistic small dataset example, we show state-of-the-art classification results on popular 3D MRI brain-disease datasets where pre-training is difficult due to a lack of large public datasets in a similar domain.

1. Introduction

The introduction proposes structured receptive fields to make CNN learning more effective on small datasets, where pre-training is less reliable when the target domain differs. It combines a fixed filter basis with learned filter combinations and reports strong results across small-data settings, including brain MRI classification.

1. Introduction

  • Pre-training is less effective when the target domain differs from the source, motivating parameter-space restrictions for learning from smaller datasets.CNNs generalize well to domains similar to their pre-training domain, but performance decreases significantly when moving away from it.Source: §1 · p. 1
  • RFNNs formulate filter learning as a linear decomposition onto a fixed basis, combining multiscale image analysis with learnable CNN filters.The chosen Gaussian family and its smooth derivatives provide the basis, while backpropagation learns the weights that form effective filters.Source: §1 · pp. 1–2
  • RFNNs combine Scattering’s fixed-basis prior with CNNs’ capacity to learn effective filter combinations, targeting both small and complex datasets.The introduction reports strength on small datasets where Scattering performs best and CNNs are weak, and on complex datasets where CNNs excel and Scattering is weak.Source: §1 · p. 2
  • The model achieves state-of-the-art classification results on two brain MRI datasets where pre-training is infeasible, outperforming all published ADNI results.The introduction identifies Alzheimer’s disease classification as the task.Source: §1 · p. 2

2. Related Work

The related work connects CNNs to multiscale image analysis and motivates restricting their parameter spaces. It contrasts learned CNN filters with fixed Scattering representations and other CNN design choices that constrain model functions.

2.1. Scale-space: the deep structure of images

  • Scale-space describes image structure through progressively larger smoothing scales, while Gaussian derivatives provide a complete, stable local basis; CNN pooling also builds representations across discrete resolutions.The paper uses scale-space theory to decompose images locally on a fixed multiscale filter basis.Source: §2.1 · p. 2

2.2. CNNs and their parameters

  • CNNs’ many unrestricted parameters make them data-hungry, and limited training data can lead to overfitting and poor generalization despite regularization.The paper notes that transfer learning from a similar domain is often necessary for competitive results in this setting.Source: §2.2 · p. 2

2.3. The Scattering representation

  • Scattering uses fixed wavelet cascades and can perform well on small datasets, whereas RFNNs learn effective combinations from a fixed basis for a compact task-specific representation.Scattering’s invariances are effective when known in advance; unknown variability can require exhaustive additional paths.Source: §2.3 · p. 3

2.4. Recent CNNs

  • Other CNNs restrict function spaces through global average pooling or by replacing 5x5 convolutions with two 3x3 layers, while the paper restricts free parameters without reducing filter size.The cited examples include Network in Network and GoogleNet [17] [34], and VGG [32].Source: §2.4 · p. 3

3. Deep Receptive Field Networks

The paper represents CNN filters through fixed Gaussian-derivative bases, connecting receptive-field learning to multiscale image analysis while retaining learnable filter combinations. It then uses this basis in a Network in Network architecture, reducing the number of spatial filters while retaining expressiveness.

3.1. Structured receptive fields

  • RFNNs treat images as functions in Scale-space and approximate arbitrary CNN filters with Taylor expansions represented by Gaussian-derivative responses.The derivative basis is complete in principle; the paper restricts its order based on the claim that higher orders do not carry visually meaningful information.Source: §3.1 · pp. 3–4

3.2. Transformation properties of the basis

  • Gaussian derivatives support dilation and steerable rotation from a minimal basis, and Scattering is a special case of RFNNs with fixed angles and scales.This connection allows Scattering concepts to be integrated into CNNs as hybrid architectures.Source: §3.2 · p. 4

3.3. Learning basis filter parameters

  • RFNNs learn weights that combine fixed Gaussian-derivative basis filters into effective filters, and compute convolution with the basis before recombining feature maps.The paper describes learning the weights by mini-batch stochastic gradient descent and backpropagation; linearity avoids explicitly computing effective filters.Source: §3.3 · pp. 4–5, Figure 3 · p. 4

3.4. The network

  • In RFNiN, fixed Gaussian-derivative basis layers replace spatial convolutions, followed by learned 1x1 recombinations; multiple scales are evaluated.The approach reduces the network’s 2D filters from 520,000 in standard NiN to 12–144 in RFNiN while retaining expressiveness.Source: §3.4 · p. 5

4. Experiments

The experiments examine RFNN design choices and compare performance across dataset sizes and domains. They include ImageNet subsets, MNIST, CIFAR-10 and CIFAR-100, and Alzheimer’s classification on brain MRI datasets.

4.1. Experiment 1: Model insight

  • On the 100 largest ILSVRC2012 classes, a third-order basis was sufficient, multiple scales improved performance, and RFNiN slightly outperformed NiN with the same setting.The champion RFNiN used 94 spatial filters versus NiN’s 520,000, though their parameter counts were relatively similar; the Gaussian basis also outperformed a freely learned basis in the refactorization comparison.Source: Table 1 · p. 6, §4.1 · p. 6, Table 2 · p. 7
  • The model analysis found that basis-filter weights decrease toward higher orders, while zero-order filters have relatively high weights in higher layers.The authors suggest this pattern hints at passing on scaled incoming features.Source: §4.1 · p. 6, Figure 4 · p. 6
  • The experiments compared Gaussian-basis RFNiN with refactorized NiN models and found the Gaussian basis advantageous, including when all layers were refactorized.The all-layer comparison used RFNiN-Scale 3rd-order against an otherwise identical NiN-factorize model with a freely learnable basis.Source: §4.1 · p. 6, Table 2 · p. 7

4.2. Experiment 2: Scattering and RFNNs

  • On reduced MNIST training sets, RFNN performed on par with Scattering even at the smallest sample size, while the gap between CNNs and predefined-filter networks widened as data decreased.On Cifar-10 and Cifar-100, RFNiN considerably improved on Roto-Translation Scattering, although RCNN performance was considerably higher.Source: §4.2 · p. 7, Table 3 · p. 7

4.3. Experiment 3: Limiting datasize

  • For 13k-image ILSVRC2012-10, the RFNiN–NiN gap increased to 8.0% from 2.4% on 130k-image ILSVRC2012-100, while the best RFNiN trailed NiN by 2.98% on ILSVRC2012-1000.The table reports that the gap between RFNiN and NiN increases for fewer classes; the larger RFNiN-v2 performs better on 1,000 classes, while v1 and v2 are similar on 100 and 10 classes.Source: §4.3 · p. 8, Table 4 · p. 8

4.4. Experiment 4: Small realistic data

  • On ADNI, RFNiN outperformed all published methods; on OASIS AD-126, it reached 80.26% accuracy versus 74.10% for a SIFT-based approach.The same three-layer NiN reached 84.21% accuracy, more than 10% worse than RFNiN; the ADNI comparison includes methods trained with more data.Source: §4.4 · p. 8, Table 5 · p. 8

5. Discussion

Structured receptive field networks perform strongly across scarce- and medium-data settings, and outperform Scattering on large datasets, while matching CNN performance on very large ILSVRC2012 remains open. Their scale-space filter basis regularizes toward smooth features and connects CNNs with multiscale image analysis.

  • RFNNs outperform CNNs by large margins on scarce data, can beat CNNs on challenging medium datasets, and surpass Scattering on large datasets.Source: §5 · p. 8
  • On two Alzheimer’s MRI datasets, RFNNs achieve competitive performance, including the best reported result on ADNI.Source: §5 · p. 8
  • A third-order Gaussian basis suffices across the evaluated datasets, while the method’s scale-space basis regularizes toward smooth features up to fourth order.The basis connects CNNs with multiscale image analysis and supports model insight and control.Source: §5 · p. 8
  • Matching CNN performance on the 1000-class ILSVRC2012 remains an open problem.Source: §5 · p. 8

Related papers

Loading 1605.02971v2…