PaperlayerView exact revision on arXiv

Source-linked AI summary

Making Convolutional Networks Shift-Invariant Again

Richard Zhang

arXiv:1904.11486v2cs.CVcs.LG

TL;DR

Modern convolutional networks lose shift-equivariance because common downsampling layers alias signals, making outputs sensitive to small input shifts. The paper inserts low-pass filtering before subsampling while retaining existing downsampling operations, and reports higher consistency, accuracy, and robustness. It also identifies open questions about high-frequency content and the mechanism behind improved generalization.

  • Problem

    Common downsampling layers ignore the Nyquist sampling criterion, so small input shifts can cause drastic changes in network outputs.

  • Method

    The paper inserts a low-pass filter between dense layer evaluation and subsampling, integrating anti-aliasing with existing downsampling operations such as max-pooling and strided-convolution.

  • Results

    The method achieves higher consistency across architectures and downsampling techniques, alongside increased classification accuracy and robustness to corruptions.

  • Takeaways & Limitations

    Anti-aliasing is a practical way to improve shift-equivariance while retaining existing downsampling strategies and can act as built-in shift-based data augmentation.

  • Takeaways & Limitations

    The method can lose high-frequency content important for conditional image generation, while the mechanism behind its observed generalization improvements remains insufficiently understood.

Abstract

from arXiv · show

Modern convolutional networks are not shift-invariant, as small input shifts or translations can cause drastic changes in the output. Commonly used downsampling methods, such as max-pooling, strided-convolution, and average-pooling, ignore the sampling theorem. The well-known signal processing fix is anti-aliasing by low-pass filtering before downsampling. However, simply inserting this module into deep networks degrades performance; as a result, it is seldomly used today. We show that when integrated correctly, it is compatible with existing architectural components, such as max-pooling and strided-convolution. We observe \textit{increased accuracy} in ImageNet classification, across several commonly-used architectures, such as ResNet, DenseNet, and MobileNet, indicating effective regularization. Furthermore, we observe \textit{better generalization}, in terms of stability and robustness to input corruptions. Our results demonstrate that this classical signal processing technique has been undeservingly overlooked in modern deep networks. Code and anti-aliased versions of popular networks are available at https://richzhang.github.io/antialiased-cnns/ .

1. Introduction

Modern convolutional networks can change drastically under small input shifts because common downsampling layers alias signals. The paper integrates low-pass filtering between dense evaluation and subsampling, preserving existing layers while improving stability and classification performance.

  • Motivation: Small input shifts can drastically change network outputs because max-pooling lacks the anti-aliasing capability of blurred downsampling.Network outputs may oscillate depending on input position.
  • Method: Low-pass filtering between dense max evaluation and subsampling augments rather than replaces max-pooling.The same placement also supports existing strided layers such as strided-convolution.
  • Results: Increased ImageNet classification accuracy is observed across architectures, despite concerns that filtering could degrade performance.The authors interpret the accuracy increase as effective regularization.
  • Validation: The method is compatible with max-pooling, average-pooling, and strided-convolution across multiple architectures and tasks.The paper validates the approach in image classification and image-to-image translation.
  • Results: The method stabilizes predicted probabilities under image shifts across networks and datasets, including AlexNet on ImageNet and VGG on CIFAR10.The figure contrasts chaotic baseline behavior with stabilized behavior from the proposed method.

2. Related Work

Prior work studied invariances, perturbation responses, and blurred downsampling, but commonly treated anti-aliased and max-pooling strategies as separate choices. This paper frames them as compatible and targets shift-equivariance with minimal additional computation.

  • Invariances and representations: Neural networks have long used local connectivity and weight sharing, while later work engineered additional invariances such as rotation, reflection, and scaling.Related analyses also examine which invariances emerge in learned representations.
  • Perturbation analysis: Quantitative studies measure representation or output changes under image transformations, geometric transforms, renderings, and adversarial perturbations.These works evaluate robustness through manually generated input changes.
  • Shift-equivariance: Literal shift-equivariance cannot hold after subsampling, although dense feature extraction can recover it at substantial computation and memory cost.The paper instead investigates blurring before subsampling with minimal additional computation.
  • Architectural modification: The proposed architectural modification better antialiases max-pooling, strided-convolution, and average-pooling.The paper presents these as common downsampling layers that can each be modified.
  • Downsampling strategies: Max-pooling became predominant because it performs better empirically than average pooling, despite blurred-downsampling's anti-aliasing benefits.Earlier work treated the two strategies as separate, preventing their combination.
  • Positioning: Convolutional Kernel Networks provided a translation-invariance-motivated architecture but lacked guidance for integrating Gaussian filtering with existing components.This paper addresses practical integration with any strided layer and reports performance increases on ImageNet.

3. Methods

The paper defines shift-equivariance and shift-invariance for convolutional feature extractors, then explains how downsampling breaks these properties. It proposes inserting low-pass filtering before subsampling while retaining max-pooling, strided convolution, or average pooling.

  • 3.1. Preliminaries: Shift-equivariance means shifting the input shifts the output equivalently, whereas shift-invariance means the representation remains identical after shifting.
  • 3.1. Preliminaries: Downsampling can violate shift-equivariance because subsampling discards phase information, while low-pass filtering suppresses high frequencies before sampling.The paper describes this as anti-aliasing, motivated by the sampling theorem.
  • 3.2. Anti-aliasing to improve shift-equivariance: Max-pooling can be decomposed into dense max evaluation followed by subsampling, isolating subsampling as the source of lost shift-equivariance.For the example signal, max-pooling produces [0, 1, 0, 1], while a shifted input produces [1, 1, 1, 1].
  • 3.2. Anti-aliasing to improve shift-equivariance: MaxBlurPool applies a low-pass filter between dense max evaluation and subsampling, producing representations that are closer across shifted inputs.The example outputs after filtering are [.5, 1, .5, 1] and [.75, .75, .75, .75].
  • 3.2. Anti-aliasing to improve shift-equivariance: The same anti-aliasing placement extends to strided layers, allowing existing strided-convolution operations to remain in the architecture.The method also treats stronger filtering as an alternative to ordinary average pooling.
  • 3.2. Anti-aliasing to improve shift-equivariance: The method supports normalized blur kernels from size 2 to 5, including rectangle, triangle, and binomial filters with increasing smoothing.The listed filters correspond respectively to box, bilinear, and Laplacian-pyramid-style filtering.

4. Experiments

The experiments evaluate anti-aliasing across classification and image-generation tasks, showing that filtering before downsampling improves shift-equivariance, accuracy, stability, and robustness while introducing a generation-quality tradeoff at stronger filtering.

  • CIFAR Classification: Across CIFAR, VGG progressively loses shift-equivariance after each max-pooling layer, with periodic-N equivariance persisting as subsampling accumulates.The periodicity factor N doubles after each subsampling operation.
  • CIFAR Classification: Anti-aliased MaxBlurPool better preserves internal shift-equivariance and produces more consistent classifications across shifts.The result holds for other filters and when training with augmentation.
  • ImageNet Classification: Across VGG, ResNet, DenseNet, and MobileNet-v2 on ImageNet, anti-aliased variants improve shift consistency and accuracy relative to baseline networks.The architectures use different downsampling strategies, enabling evaluation across network families.
  • ImageNet Classification: +0.8%, +1.7%, and +2.1% stability gains result from Rect-2, Tri-3, and Bin-5 filters, respectively, on ResNet50.Doubling layers to ResNet101 increases stability by +0.6%.
  • ImageNet Classification: +0.7% to +0.9% accuracy gains result from filtering ResNet50 without adding learnable parameters.The authors characterize this improvement as effective regularization.
  • Out-of-Distribution Robustness and Conditional Image Generation: Anti-aliasing improves stability and corruption robustness, while conditional image generation gains shift-equivariance but degrades in quality with sufficiently aggressive filtering.Generation quality typically holds with Tri-3 before degrading under stronger filtering, indicating a tradeoff between equivariance and quality.

5. Conclusions and Discussion

The paper restores shift-equivariance by integrating low-pass filtering into deep networks, producing higher consistency across architectures and downsampling techniques. Classification also shows boosts in accuracy and robustness, while broader effects on generalization remain an open direction.

  • Low-pass filtering restores shift-equivariance in modern deep networks, which lose it because downsampling layers ignore Nyquist sampling and alias.
  • The modification achieves higher consistency across architectures and downsampling techniques.
  • Classification experiments show boosts in accuracy and robustness after anti-aliasing.
  • The paper identifies better understanding of anti-aliasing’s effects on generalization as a future direction.

Supplementary Material

The supplementary material adds experiments covering CIFAR classification, expanded ImageNet results, and conditional image generation.

  • Supplementary experiments cover CIFAR classification, expanded ImageNet results, and conditional image generation.

A.1. Classification results

Supplementary classification experiments examine consistency, accuracy, filtering, learned filter smoothness, shift robustness, and computational cost across training settings and architectures. Anti-aliasing generally improves consistency, can preserve or improve accuracy, induces smoother filters, and reduces shift-related degradation.

  • Classification results: Without data augmentation, baseline classifications agree on random shifts only 88.1% of the time.
  • Classification results: MaxBlurPool increases consistency without training-time shifts, and larger filters produce more consistent output classifications.
  • Classification results: Shift-equivariance acts as built-in augmentation, with filtering increasing consistency while not destroying the signal or making learning harder.
  • Classification results: With data augmentation, Rect-2 raises consistency from 96.6 to 97.6 while accuracy changes from 93.8 to 93.7.
  • DenseNet results: In DenseNet, replacing average-pooling with blurred-pooling increases consistency and slightly improves accuracy; Bin-7 performs best for both.
  • Filter smoothness: Anti-aliased networks learn smoother convolutional filters than the baseline, with stronger low-pass filtering further decreasing Total Variation.
  • Timing analysis: Forward-pass overhead increases with filtering: kernel sizes 3, 5, and 7 add 5.5%, 7.6%, and 9.3% relative to baseline.
  • Shift robustness: Filtering reduces accuracy degradation under spatial and diagonal shifts, with Bin-7 remaining largely consistent across positions.

Classification variation distribution

Classification variation decreases after low-pass filtering, including when networks already use data augmentation, and stronger filters generally reduce variation further. Filtering also increases robustness to shift-based adversarial attacks.

  • Classification variation decreases immediately with a small 2 × 2 filter and continues decreasing as filter size increases.
  • Anti-aliasing reduces variation both without and with data augmentation, with stronger filtering further decreasing it.
  • Filtering increases robustness to shift-based adversarial attacks with and without data augmentation.
  • A maximum adversarial shift of 2 lets the adversary choose any of 25 positions within a 5 × 5 window.

Robustness to shift-based adversary

The study evaluates robustness to a shift-based adversary and finds that larger Binomial antialiasing filters improve robustness, including without adversarial training augmentation.

  • A max shift of 16 requires correct classification at all 1,024 positions.
  • Larger Binomial filters increase robustness to the shift-based adversary.
  • Bin-7 without augmentation outperforms the baseline with augmentation.
  • Larger Binomial filters also improve adversarial robustness when training uses augmentation.
  • These results demonstrate increased robustness to a shift-based adversarial attack.

A.2. Alternatives to MaxBlurPool

The alternatives study compares operation order, pooling combinations, and learned filters, while supporting MaxBlurPool as a methodology applicable beyond a single pooling layer.

  • Alternatives to MaxBlurPool: The alternatives include swapping max and blur, soft-gating max-pool with average-pool, and learning the blur filter.
  • Swapping max and blur: Blurring before max still improves shift-invariance over baseline, but performs worse than blurring after max before subsampling.
  • Softly gating between max-pool and average-pool: MixedPool improves over the MaxPool baseline, but soft weighting leaves some antialiasing capability unused.
  • Softly gating between max-pool and average-pool: Applying the same reduce-stride-then-BlurPool technique to MixedPool increases shift-invariance from 97.2 to 97.8.
  • Learning the blur filter: The paper presents antialiasing as a methodology rather than a pooling layer.
  • Learning the blur filter: Learned blur filters preserve accuracy but decrease consistency relative to fixed filters.

B. ImageNet Classification

Across classification, perturbation stability, corruption robustness, and image generation, antialiasing generally improves or preserves performance, while overly aggressive filtering can degrade generation quality.

  • Robustness to perturbations: Antialiasing reduces translation flip rate by 22.3% and reliably stabilizes 9 of 10 perturbation types.
  • Generalization to corruptions: Antialiasing reliably improves robustness for 13 of 15 corruptions, including noise, pixelation, and JPEG compression.
  • Generalization: The results indicate that antialiasing produces a smoother feature extractor that is more stable and robust to out-of-distribution perturbations.
  • Image-to-image translation: Generation quality is maintained with Rect-2 and Tri-3 filters but degrades with additional filtering.
  • ImageNet Classification: ImageNet classification consistency improves across all tested methods, and accuracy also improves with antialiasing.
  • Generalization to corruptions: Table 7 reports lower corruption error rates with antialiasing on most ImageNet-C corruptions.
Loading 1904.11486v2…