PaperlayerView exact revision on arXiv

Source-linked AI summary

Global Filter Networks for Image Classification

Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, Jie Zhou

arXiv:2107.00645v2cs.CVcs.AIcs.LG

TL;DR

Self-attention and pure MLP vision models become quadratically more expensive as image resolution grows, limiting high-resolution feature processing. GFNet replaces self-attention with frequency-domain global filtering using FFT, element-wise multiplication, and inverse FFT, and achieves competitive accuracy and complexity trade-offs across ImageNet and downstream tasks.

  • Problem

    Self-attention and pure MLP vision models have quadratic complexity as the number of image tokens increases, making high-resolution features difficult to scale.

  • Method

    GFNet replaces the self-attention layer with 2D FFT, learnable frequency-domain global filters, and inverse FFT to mix spatial tokens with log-linear complexity.

  • Results

    GFNet is competitive with transformer-style models and CNNs in efficiency, generalization ability, robustness, and accuracy/complexity trade-offs across ImageNet and downstream tasks.

  • Takeaways & Limitations

    GFNet offers a computationally efficient alternative to vision transformers, MLP-like models, and CNNs for image classification and related evaluations.

Abstract

from arXiv · show

Recent advances in self-attention and pure multi-layer perceptrons (MLP) models for vision have shown great potential in achieving promising performance with fewer inductive biases. These models are generally based on learning interaction among spatial locations from raw data. The complexity of self-attention and MLP grows quadratically as the image size increases, which makes these models hard to scale up when high-resolution features are required. In this paper, we present the Global Filter Network (GFNet), a conceptually simple yet computationally efficient architecture, that learns long-term spatial dependencies in the frequency domain with log-linear complexity. Our architecture replaces the self-attention layer in vision transformers with three key operations: a 2D discrete Fourier transform, an element-wise multiplication between frequency-domain features and learnable global filters, and a 2D inverse Fourier transform. We exhibit favorable accuracy/complexity trade-offs of our models on both ImageNet and downstream tasks. Our results demonstrate that GFNet can be a very competitive alternative to transformer-style models and CNNs in efficiency, generalization ability and robustness. Code is available at https://github.com/raoyongming/GFNet

1 Introduction

GFNet addresses the quadratic complexity of self-attention and pure MLP vision models by replacing self-attention with learnable frequency-domain global filters. Experiments report competitive performance across ImageNet and downstream tasks.

  • Motivation: Self-attention and pure MLP models have computational complexity that grows quadratically with the number of image tokens, limiting high-resolution features.This can constrain downstream dense prediction applications such as detection and segmentation.
  • Contribution: GFNet replaces the vision-transformer self-attention sub-layer with a frequency-domain global filter layer based on Fourier operations.The architecture makes minimal modifications to vision transformers and learns spatial interactions from raw data without human priors.
  • Contribution: The global filters cover all frequencies, allowing GFNet to capture both long-term and short-term interactions among spatial locations.The filters are directly learned from the data rather than imposing handcrafted inductive biases.
  • Results: GFNet models outperform recent vision-transformer and MLP models on ImageNet, with hierarchical variants further enlarging the performance gap.The reported comparisons include DeiT, ResMLP, and gMLP.
  • Results: GFNet also performs well on transfer learning and semantic segmentation, supporting its competitiveness with transformer-style models and CNNs in efficiency, generalization, and robustness.These claims are reported as experimental conclusions across the evaluated tasks.

2 Related works

Related work spans vision transformers, MLP-like token mixers, and Fourier-transform methods for vision. GFNet is positioned as an efficient alternative that avoids the quadratic cost of self-attention and spatial MLPs.

  • Vision transformers: Vision transformers use self-attention to capture long-term dependencies and diverse interactions among spatial locations in vision tasks.Transformer architecture modifications often introduce inductive biases similar to those in CNNs.
  • GFNet positioning: GFNet targets the heavy O(L^2) self-attention operation with a more efficient global-filter operation while retaining broad spatial interaction.Its efficiency comparison includes self-attention and spatial MLP operations.
  • MLP-like models: MLP-Mixer, ResMLP, and gMLP replace self-attention with spatial token-mixing mechanisms, but their spatial MLPs remain quadratic in sequence length.The cited models differ in how they implement token mixing and normalization or gating.
  • Fourier methods: Fourier transforms have long been used in digital image processing and incorporated into deep-learning methods for frequency-based enhancement or CNN acceleration.Prior approaches use Fourier information for different vision tasks and computational purposes.

3 Method

GFNet mixes spatial tokens by transforming features to the frequency domain, applying learnable global filters, and transforming them back. FFT-based computation gives the layer O(L log L) complexity and supports hierarchical high-resolution architectures.

  • 3.1 Preliminaries: discrete Fourier transform: The discrete Fourier transform converts a discrete spatial signal into its frequency-domain representation, and inverse DFT reconstructs the original signal.FFT and IFFT algorithms provide efficient computation for these transformations.
  • 3.2 Global Filter Networks: The global filter layer computes 2D FFT, element-wise multiplication with a frequency-domain filter, and inverse FFT on token features.The implementation uses rfft2, multiplication by K, and irfft2 for real signals.
  • 3.2 Global Filter Networks: GFNet blocks combine global filtering with a feedforward network, followed by global average pooling and a linear classifier.The model projects non-overlapping image patches into L = HW tokens with dimension D.
  • 3.2 Global Filter Networks: The global filter is equivalent to a depthwise global circular convolution with filter size H × W, unlike local standard convolution.Its full spatial support avoids the locality constraint imposed by small convolution kernels.
  • 3.2 Global Filter Networks: Because real-input DFTs are conjugate symmetric, GFNet retains only half the frequency values while preserving the full frequency information.This reduces redundant computation and enables a reduced filter representation.
  • 3.2 Global Filter Networks: The global filter layer has O(L log L) complexity from FFT, O(L) element-wise multiplication, and O(L log L) inverse FFT.This is lower than the O(L^2) complexity of the compared vision-transformer and MLP token mixers.
  • 3.2 Global Filter Networks: GFNet differs from FNet by learning frequency filters and retaining the frequency-domain filtering formulation rather than directly adding the real spectrum to spatial tokens.The cited comparison also notes that FNet keeps only the real part of the spectrum.
  • 3.2 Global Filter Networks: GFNet’s log-linear complexity enables hierarchical architectures that begin with larger feature maps and progressively downsample them.The paper gives 56 × 56 as an example of a larger starting feature map.

4 Experiments

GFNet experiments evaluate ImageNet classification, hierarchical models, transfer learning, efficiency, accuracy-complexity trade-offs, robustness, and filter visualization. Across these settings, GFNet shows competitive or improved performance while supporting higher-resolution features and efficient computation.

  • ImageNet classification: GFNet-XS outperforms ResMLP-12 by 2.0% with slightly fewer FLOPs, while GFNet-Tiny exceeds DeiT-Ti by 2.4% and gMLP-Ti by 2.6%.GFNet-S also achieves better top-1 accuracy than gMLP-S and DeiT-S at comparable complexity.
  • Hierarchical architectures: GFNet hierarchical models outperform ResNet, RegNet, and PVT, while achieving similar performance to Swin with a simpler design.The comparison uses hierarchical architectures with comparable FLOPs and parameter counts.
  • Higher-resolution fine-tuning: GFNet can be fine-tuned from 224 × 224 to higher resolution with 30 epochs and achieve better performance.This addresses the fixed feature-resolution limitation described for MLP-like models.
  • Transfer learning: GFNet models generally perform well on CIFAR-10, CIFAR-100, Stanford Cars, and Flowers-102, outperforming ResMLP by a large margin.They achieve competitive performance with EfficientNet-B7, CNNs, and vision transformers.
  • Efficiency: The self-attention model runs out of memory beyond feature resolution 562, while GFNet's advantage increases as resolution grows.Efficiency is assessed using theoretical FLOPs, actual latency, and peak GPU memory with feature dimension 384 and batch size 32.
  • Accuracy-complexity trade-offs: GFNet achieves the best accuracy-complexity trade-off among the transformer-style models evaluated.The comparison is summarized in Figure 3 using ImageNet accuracy and model complexity.
  • Visualization: The learned global filters show clearer patterns in the frequency domain, and filters in the last layer particularly focus on low-frequency components.Their corresponding spatial-domain filters are less interpretable for humans.

5 Conclusion

GFNet is a conceptually simple, computationally efficient architecture for image classification that replaces transformer self-attention with frequency-domain filtering. Its log-linear token mixing supports favorable accuracy/complexity trade-offs against vision transformers, MLP-like models, and CNNs.

  • GFNet replaces the self-attention sub-layer with 2D FFT/IFFT operations and learnable global filters in the frequency domain.The filter-based token-mixing operation has log-linear complexity.
  • GFNet is highly efficient because its token mixing has log-linear complexity.
  • GFNet offers favorable accuracy/complexity trade-offs compared with vision transformers, MLP-like models, and CNNs.

A Discrete Fourier transform

This section introduces the discrete Fourier transform and discusses its derivation and properties.

  • The section elaborates on the derivation of the discrete Fourier transform.
  • The section explains properties of the discrete Fourier transform.
  • The discrete Fourier transform is treated as a central topic of the appendix section.

A.1 From Fourier transform to discrete Fourier transform

The appendix derives the discrete Fourier transform from the continuous Fourier transform through sampling, then extends the formulation from one dimension to two.

  • From Fourier transform to discrete Fourier transform: The Fourier transform converts a continuous signal from the time domain to the frequency domain.
  • From Fourier transform to discrete Fourier transform: Sampling converts a continuous signal into a sequence of discrete signal values using unit impulse functions.
  • From Fourier transform to discrete Fourier transform: For a finite-length discrete signal, the DTFT is sampled at frequencies ω_k = 2πk/N to obtain a sequence of DFT values.
  • From Fourier transform to discrete Fourier transform: The 2D DFT is obtained by performing one-dimensional DFTs on the two dimensions alternately.

A.2 Some properties of DFT

This section presents DFT properties used by GFNet, including conjugate symmetry for real signals and the frequency-domain interpretation of multiplication as circular convolution.

  • Some properties of DFT: The DFT of a real signal is conjugate symmetric, so half of its frequency coefficients contains the full frequency information.
  • Some properties of DFT: GFNet uses conjugate symmetry to reduce learnable parameters and redundant computation.
  • Some properties of DFT: Multiplication in the frequency domain is equivalent to circular convolution in the time domain.
  • Some properties of DFT: GFNet's global filter layer is equivalent to a depth-wise circular convolution with a filter matching the feature-map size.

B Implementation Details

The implementation uses controlled training settings, transfer-learning benchmarks, and comparisons across model complexity groups and resolutions. Additional figures examine layer-wise frequency behavior and resolution adaptation.

  • Model architectures: GFNet-XS uses the identical overall architecture to DeiT Small and ResMLP-12, modifying only selected sub-layers, the classifier, and residual connection.A single residual connection improves GFNet-XS ImageNet top-1 accuracy by 0.2%.
  • ImageNet experiments: GFNet models are trained for 300 epochs with AdamW, cosine learning-rate decay, warm-up, and gradient clipping.The initial learning rate is batch size / 1024 × 0.001, decayed to 1e−5.
  • Transfer learning: Transfer learning is evaluated on CIFAR-10, CIFAR-100, Stanford Cars, and Flowers-102 using ImageNet-pretrained weights and fine-tuning.The experiments follow commonly used settings and splits from prior work.
  • Resolution evaluation: Figure 6 evaluates ImageNet accuracy across resolutions for GFNet and DeiT without fine-tuning.The figure compares direct evaluation across different resolutions.
  • Semantic segmentation: Semantic segmentation comparisons on ADE20K group models into tiny, small, and base categories with similar computational costs.The reported mIoU uses Semantic FPN, while FLOPs are measured with 1024 × 1024 inputs.

C More Results & Analysis

Additional experiments show GFNet’s behavior on dense prediction, frequency filtering, and direct evaluation at unseen resolutions. The results indicate competitive segmentation performance and better adaptation across resolutions than DeiT-S.

  • Semantic segmentation: GFNet achieves competitive performance on ADE20K semantic segmentation across different levels of computational complexity.The evaluation uses hierarchical GFNet models for dense prediction.
  • Power distribution: Layer-wise power analysis provides a view of how global filters capture information across different frequency ranges.The analysis plots filter power for each layer.
  • Direct resolution adaptation: GFNet-S trained at 224 × 224 adapts to resolutions from 128 to 448 with less performance drop than DeiT-S.The comparison directly evaluates both models without retraining at each resolution.
  • Hierarchical filter visualization: Frequency-domain global filters are visualized for different stages and sizes in hierarchical GFNet-H-B.The visualization is provided in Figure 7.
Loading 2107.00645v2…