PaperlayerView exact revision on arXiv

Source-linked AI summary

Impact of Aliasing on Generalization in Deep Convolutional Networks

Cristina Vasconcelos, Hugo Larochelle, Vincent Dumoulin, Rob Romijnders, Nicolas Le Roux, Ross Goroshin

arXiv:2108.03489v1cs.CVcs.LG

TL;DR

The paper asks whether aliasing limits generalization and whether standard data augmentation can prevent it. It uses frequency analysis to place non-trainable low-pass filters in ResNet and EfficientNet, achieving substantial i.i.d. and o.o.d. gains without extra trainable parameters. The authors report state-of-the-art results on ImageNet-C and Meta-Dataset using default open-source settings.

  • Problem

    The paper investigates whether aliasing harms generalization and whether data augmentation alone can prevent it in standard convolutional architectures.

  • Method

    It uses frequency analysis to identify aliasing-critical locations and inserts non-trainable low-pass filters where architectures lack capacity to learn them.

  • Results

    The architectural changes substantially improve i.i.d. and o.o.d. generalization, including ImageNet-C and Meta-Dataset performance, without additional trainable parameters.

  • Takeaways & Limitations

    Anti-aliasing is simple, computationally inexpensive, complements data augmentation and smooth activations, and did not degrade performance in the reported experiments.

  • Takeaways & Limitations

    The paper’s hypotheses concern settings where aliasing may create brittle correlations or corrupt useful feature relationships, rather than establishing that every CNN task is affected.

Abstract

from arXiv · show

We investigate the impact of aliasing on generalization in Deep Convolutional Networks and show that data augmentation schemes alone are unable to prevent it due to structural limitations in widely used architectures. Drawing insights from frequency analysis theory, we take a closer look at ResNet and EfficientNet architectures and review the trade-off between aliasing and information loss in each of their major components. We show how to mitigate aliasing by inserting non-trainable low-pass filters at key locations, particularly where networks lack the capacity to learn them. These simple architectural changes lead to substantial improvements in generalization on i.i.d. and even more on out-of-distribution conditions, such as image classification under natural corruptions on ImageNet-C [11] and few-shot learning on Meta-Dataset [26]. State-of-the art results are achieved on both datasets without introducing additional trainable parameters and using the default hyper-parameters of open source codebases.

1. Introduction

The paper argues that stronger architectural priors matter when data are out of distribution or labels are limited. It studies aliasing and proposes anti-aliasing modifications that improve generalization on i.i.d. and o.o.d. benchmarks.

  • When training data cannot provide all necessary implicit knowledge, stronger architectural priors become an important research direction.
  • Aliasing is investigated as a source of reduced generalization under shifts in datasets’ spectral distributions.
  • Frequency analysis motivates simple anti-aliasing modifications for ResNet and EfficientNet, targeting locations where networks lack capacity to learn suitable filters.
  • The proposed architecture improves i.i.d. and o.o.d. benchmarks using open source codebases and default hyper-parameters.
  • It surpasses the stand-alone ImageNet-C state of the art in 9 of 15 categories, achieves the lowest clean ImageNet error, and reaches state of the art on Meta-Dataset among methods using all training sources.

2. Spectral Aliasing in Convolutional Networks

Aliasing arises when subsampling violates the Nyquist rate and can distort convolutional features. The paper tests whether networks can learn to prevent it, isolates its effects, and finds that architectural anti-aliasing benefits all spectral bands beyond data augmentation alone.

  • Subsampling below the Nyquist rate causes high frequencies to spill into lower frequencies, producing aliasing distortions in CNN feature maps.
  • Trainable filters may learn low-pass behavior only when they have sufficient spatial support and are positioned before the relevant subsampling operation.
  • Anti-aliasing improves data-augmentation results across a larger frequency range, whereas augmentation alone mainly improves low-frequency bands.
  • The paper asks whether aliasing affects generalization, whether networks learn anti-aliasing implicitly, and whether its effects can be separated from other architectural changes.
  • The authors hypothesize that aliasing can create brittle correlations and corrupt useful feature relationships across the spectrum.

3. Related Work

Prior work studies sampling sensitivity, augmentation, and learned anti-aliasing, but often introduces smoothing side effects or additional model capacity. The proposed approach combines architectural modifications with off-the-shelf augmentation and improves clean and corrupted accuracy together.

  • Zhang filters after a strided layer’s non-linearity, whereas this work filters immediately before subsampling to avoid smoothing intervening trainable features.
  • Zou et al. [31] add trainable nonlinear filtering components, making their method unsuitable for isolating aliasing from capacity increases and other confounds.
  • Data-augmentation robustness can be concentrated in particular spectral bands, while the proposed anti-aliasing approach targets broader spectral effects.
  • The combined method improves all 15 ImageNet-C corruption categories, outperforms in 9, and reaches 78.8% clean ImageNet accuracy versus 76.1% for.
  • Unlike methods trading clean against corrupted accuracy, the proposed method improves both i.i.d. and o.o.d. accuracy simultaneously.

4. Methods

The method identifies where CNNs cannot reliably learn low-pass filters, then inserts fixed filters before subsampling while preserving high-frequency information elsewhere. It applies this principle to anti-aliased ResNet and EfficientNet variants, selecting filter placement by considering architectural capacity and forward/backward side-effects.

  • Frequency analysis theory: Ideal low-pass filtering requires large spatial support, while smaller non-ideal filters trade reduced aliasing against information loss; stacks of small convolutions can approximate separable filters.A filter of size 1 cannot act as a low-pass filter, whereas filters larger than 1 can approximate one to varying degrees.
  • Optimal Placement and Confounding effects: Low-pass filtering should occur immediately before subsampling, preserving high-frequency information until it can cause aliasing rather than discarding it after a non-linearity.Filtering after the non-linearity can smooth newly produced high frequencies prematurely, whereas filtering before subsampling satisfies the band-limiting requirement at the relevant operation.
  • Optimal Placement and Confounding effects: The post-filter variant is preferred because its backward-pass smoothing better aligns with gradient up-sampling and produces superior empirical results in Table 1.Different placements introduce distinct forward- and backward-pass priors because subsampling and non-linearities make filter ordering non-commutative during backpropagation.
  • Aliasing critical paths: Aliasing critical paths are the operations between a subsampling layer and the nearest preceding non-linearity or skip connection, where last-chance low-pass filtering is required.These paths maintain or reduce frequency range but lack capacity to create new high frequencies, while upstream trainable layers may still filter more aggressively.
  • ResNets and EfficientNets: ResNet aliasing paths include initial strided convolution and pooling, strided residual-block paths, and strided skip connections, whose filter sizes determine anti-aliasing capacity.Strided skip connections use 1 × 1 filters, which lack the minimum size needed to represent a low-pass filter; a 7 × 7 trainable convolution is considered sufficient and therefore receives no added filter.
  • ResNets and EfficientNets: EfficientNet analysis targets its initial strided convolution and strided blocks, while the architecture’s smooth Swish activation is less likely than ReLU to introduce high-frequency content.EfficientNet is examined because its searched depth, width, and resolution directly affect aliasing, and its subsampling layers lack parallel skip paths.

5. Experimental Results

Experiments show that carefully placed anti-aliasing improves ImageNet accuracy, extends data augmentation across frequency bands, and strengthens robustness on ImageNet-C and Meta-Dataset.

  • 5.1. ImageNet: Strided skip connections are the ResNet components most severely affected by aliasing, while filtering the first large-kernel convolution can degrade performance through information loss.The ablations distinguish anti-aliasing gains from enlarged receptive fields and other side effects.
  • 5.1. ImageNet: 77.47% top-1 accuracy results from post-filtering all strided layers except the first large-kernel strided convolution, supporting selective anti-aliasing.The design targets critical paths lacking capacity to learn low-pass filters while preserving high-frequency information.
  • 5.2. Data-Augmentation: Anti-aliasing improves data-augmentation results across a broader frequency range: augmentation alone performs worst when mid-range bands are removed, whereas the combined model improves across all bins.The combined approach boosts the individual benefits of anti-aliasing and augmentation.
  • 5.3. ImageNet-C: 70.0% mCE and 22.5% clean error are reported for the anti-aliased ImageNet-C model, improving over.The table compares corruption error, mean corruption error, and clean error across models.
  • 5.3. ImageNet-C: 21.2% clean top-1 error and 64.9% mCE are achieved without sacrificing clean error while improving corruption robustness.The reported comparison specifically contrasts this result with.
  • 5.4. Meta-Dataset with SUR: The anti-aliased, smooth-activation SUR model achieves state-of-the-art average rank on Meta-Dataset among methods trained on all available training classes.Traffic Signs and MSCOCO are evaluation-only out-of-domain datasets in this benchmark.

6. Conclusion

The paper proposes simple anti-aliasing architectural changes grounded in sampling theory and reports gains in both i.i.d. and out-of-distribution generalization without added trainable parameters.

  • 6. Conclusion: Anti-aliasing architectural changes substantially improve i.i.d. and out-of-distribution generalization without additional trainable parameters.The authors report no setting in which anti-aliasing degraded performance and recommend it as a standard component of convolutional architectures.
  • 6. Conclusion: Low-pass filtering prevents subsampling aliasing by preserving lower-frequency components while suppressing spectral replicas.Subsampling replicates spectra, allowing frequencies from those copies to corrupt the original signal; low-pass filtering prevents this effect.
  • 6. Conclusion: Binomial filters provide finite-support low-pass filters, with larger supports attenuating cutoff frequencies more strongly but also removing more nearby high-frequency information.Smaller filters preserve more frequencies just below the cutoff, while larger filters provide stronger attenuation at the cutoff.
  • 6. Conclusion: The method’s filter design reflects a trade-off between reducing aliasing and retaining high-frequency information.The paper compares alternatives with different support sizes to balance computational cost, cutoff attenuation, and information preservation.

B. On Smooth activations

Smooth activations and anti-aliasing affect spectral representations differently: smooth activations mainly help lower-frequency bands, whereas anti-aliasing improves performance across bands and complements augmentation.

  • B. On Smooth activations: Nonlinear activations introduce high-frequency components when their inputs span the activation nonlinearity, requiring both positive and negative values here.The spectral changes are illustrated by comparing ReLU with GeLU and Swish on sinusoidal inputs.
  • B. On Smooth activations: Anti-aliasing improves performance across spectral bands, while smooth activations mainly improve lower-frequency bands.The combined Anti-aliasing + Rand Augmentation + Swish model combines these complementary effects.
  • B. On Smooth activations: Filtering high-frequency bands can leave images nearly indistinguishable from their originals, yet anti-aliased models correctly classify filtered examples across bands.The figure contrasts baseline errors with cases corrected by data augmentation or anti-aliasing across the 16 frequency intervals.
  • B. On Smooth activations: EfficientNet-B0 benefits from small non-trainable low-pass filters, although aliasing has a smaller impact there than in ResNet-50.The comparison uses equal 224×224 input resolution to avoid confounding from EfficientNet scaling.

D. ImageNet-C: Robustness to Natural Corruptions

On ImageNet-C, anti-aliasing improves robustness to natural corruptions while preserving clean accuracy and complements data augmentation and smooth activations.

  • D. ImageNet-C: Robustness to Natural Corruptions: Anti-aliasing the strided-skip connections alone surpasses Zhang’s model on both ImageNet and ImageNet-C, while the combined model improves further using fewer and smaller filters.The comparison is conducted under the same number of training epochs.
  • D. ImageNet-C: Robustness to Natural Corruptions: Anti-aliasing complements data augmentation and smooth activations, with their combination improving both ImageNet and ImageNet-C results.The combined model further improves results over the individual techniques in the out-of-distribution setting.

E. Additional experiments with Out-of-distribution Generalization

Anti-aliasing improves natural out-of-distribution generalization on ImageNet-R and ImageNet-V2, and its combination with data augmentation produces larger gains than either method alone.

  • E. Additional experiments with Out-of-distribution Generalization: Anti-aliased models outperform on ImageNet-R and ImageNet-V2 while maintaining similar ImageNet accuracy.ImageNet-R accuracy rises from 24.2 to 25.2 versus 24.1 for, while ImageNet-V2 reaches 65.2 versus 65.0 for and 64.6 for baseline.
  • E. Additional experiments with Out-of-distribution Generalization: On ImageNet-R, anti-aliasing improves accuracy from 24.2 to 25.2, compared with 24.1 for.The paper speculates that high-frequency patterns in ImageNet-R renditions may contribute to this result.
  • E. Additional experiments with Out-of-distribution Generalization: On ImageNet-V2, anti-aliasing reaches 65.2 accuracy, exceeding the 64.6 baseline and ’s 65.0.ImageNet-V2 measures generalization to a later collection policy for images similar to ImageNet.
  • E. Additional experiments with Out-of-distribution Generalization: Combining anti-aliasing with data augmentation yields larger accuracy gains on ImageNet-R and ImageNet-V2 than either method alone.The paper interprets the higher accuracy as improved out-of-distribution robustness.

F. Few-shot classification, Meta-Dataset, and SUR

Meta-Dataset evaluates few-shot classification across heterogeneous datasets and held-out classes, while SUR uses separate ResNet-18 backbones for eight training data sources. The anti-aliased SUR models improve test-episode accuracy by 3.75% absolute, including a 2.73% gain on out-of-domain tasks, with minor architectural changes.

  • Meta-Dataset: Meta-Dataset forms few-shot episodes from held-out classes, splitting sampled examples into support and query sets for training and evaluation.
  • Meta-Dataset: Meta-Dataset is more challenging than mini-ImageNet because it combines heterogeneous datasets whose classes are partitioned into training, validation, and test sets.
  • SUR: SUR trains separate ResNet-18 backbones for each of eight Meta-Dataset training sources and evaluates each with a nearest centroid classifier.
  • Experimental setup: The experiments fixed deterministic class-order sampling in SUR’s code, so reported baseline accuracies differ from the original paper, especially for Traffic Sign.
  • Experimental setup: SUR resizes native Meta-Dataset images to 84 × 84 using bilinear interpolation, and validation isolates preprocessing effects for each backbone’s corresponding validation classes.
  • Results: 74.80% average test-episode accuracy was obtained with anti-aliased skip connections and GELU, while filtering every downsampling operation reached 74.82%.The combined approach improved accuracy by 3.75% absolute, including 2.73% on out-of-domain tasks, using default hyper-parameters and minor architectural changes.
Loading 2108.03489v1…