PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Frequency Filters As Efficient Global Token Mixers

Zhipeng Huang, Zhizheng Zhang, Cuiling Lan, Zheng-Jun Zha, Yan Lu, Baining Guo

arXiv:2307.14008v1cs.CV

TL;DR

Global token mixing is useful but costly, and existing efficient approaches can compromise expressiveness. The paper proposes AFF, which performs adaptive frequency filtering as global token mixing, and builds AFFNet around it. The authors report state-of-the-art accuracy-efficiency trade-offs across multiple vision tasks.

  • Problem

    Global-mixing Transformers, large-kernel CNNs, and MLPs are computationally costly, while efficient alternatives may sacrifice expressiveness and general-purpose global mixing for lightweight networks remains less explored.

    Source: §1 · p. 1

  • Method

    AFF transforms latent tokens into the frequency domain, applies learned instance-adaptive masks, and transforms the filtered representations back to perform global token mixing.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    AFFNet achieves state-of-the-art accuracy-efficiency trade-offs compared with other lightweight network designs across multiple vision tasks.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The authors hope this work inspires more interplay between conventional signal processing and deep learning.

    Source: §5 · p. 9

    The work focuses on vision, and its running speed remains below the theoretical upper limit because FFT/iFFT engineering optimization has not been fully considered.

    Source: §11 · p. 14

Abstract

from arXiv · show

Recent vision transformers, large-kernel CNNs and MLPs have attained remarkable successes in broad vision tasks thanks to their effective information fusion in the global scope. However, their efficient deployments, especially on mobile devices, still suffer from noteworthy challenges due to the heavy computational costs of self-attention mechanisms, large kernels, or fully connected layers. In this work, we apply conventional convolution theorem to deep learning for addressing this and reveal that adaptive frequency filters can serve as efficient global token mixers. With this insight, we propose Adaptive Frequency Filtering (AFF) token mixer. This neural operator transfers a latent representation to the frequency domain via a Fourier transform and performs semantic-adaptive frequency filtering via an elementwise multiplication, which mathematically equals to a token mixing operation in the original latent space with a dynamic convolution kernel as large as the spatial resolution of this latent representation. We take AFF token mixers as primary neural operators to build a lightweight neural network, dubbed AFFNet. Extensive experiments demonstrate the effectiveness of our proposed AFF token mixer and show that AFFNet achieve superior accuracy and efficiency trade-offs compared to other lightweight network designs on broad visual tasks, including visual recognition and dense prediction tasks.

1. Introduction

Global token mixing is effective but costly, and efficient designs still face a trade-off between efficiency and representational capacity. The paper introduces adaptive frequency filtering as an efficient, semantic-adaptive global mixer and uses it to build AFFNet.

1. Introduction

  • Transformers, large-kernel CNNs, and MLPs support global information fusion but impose computational costs that challenge practical deployment, especially on edge devices.The cited costs include self-attention, large kernels, and fully connected layers.Source: §1 · p. 1
  • Existing efficient token-mixing approaches can sacrifice expressiveness, while a general-purpose global mixer for lightweight networks remains less explored.The paper frames better accuracy-efficiency trade-offs for global-scope mixing as an open research need.Source: §1 · p. 1
  • The convolution theorem makes global token mixing equivalent to a large-kernel convolution in latent space and to an elementwise product in the Fourier domain.Fourier transforms of latent-space tokens enable this frequency-domain implementation.Source: §1 · pp. 1–2
  • Framing adaptive global mixing as frequency filtering reduces its complexity from O(N^2) to O(N log N) using FFTs.The filters are designed to vary with instances and across channels.Source: §1 · p. 2
  • AFF uses spatial 2D Fourier transforms, learned instance-adaptive frequency masks, elementwise filtering, and an inverse transform to produce global token mixing.The resulting operation is equivalent to convolution with a dynamic kernel spanning the latent representation’s spatial dimensions.Source: §1 · p. 2
  • AFFNet uses AFF as its primary neural operator and reports state-of-the-art accuracy-efficiency trade-offs across a broad range of vision tasks.The paper also compares AFF with related frequency-domain operators through theoretical analysis and empirical study.Source: §1 · p. 2

2. Related Work

The related work reviews token-mixing mechanisms and lightweight network designs, then distinguishes prior frequency-domain operations from the paper’s token-mixing focus.

2.1. Token Mixing in Deep Learning

  • CNNs, Transformers, and MLPs differ in their mixing weights and scope; the paper seeks a lightweight mixer that is efficient, semantic-adaptive, and global.The passage contrasts deterministic CNN and MLP weights with semantic-adaptive Transformer weights and their O(N^2) complexity.Source: §2.1 · p. 2

2.2. Lightweight Neural Networks

  • Lightweight designs improve CNN, Transformer, and MLP efficiency through architectural changes, reduced attention complexity, or restricted token-mixing regions.Examples include depthwise and pointwise convolutions, shuffle operations, linear-complexity attention, and stripe- or region-based MLP mixing.Source: §2.2 · pp. 2–3

2.3. Frequency-domain Deep Learning

  • Prior frequency-domain neural operations use varied filters and representations, but the paper distinguishes them by their modeling perspectives and architecture designs.The reviewed operations include convolution, trainable elementwise or matrix products, groupwise MLP layers, and spatial instance-adaptive masks.Source: §2.3 · p. 3

3. Method

The method frames token mixing as global convolution, introduces AFF as a frequency-domain implementation, analyzes other frequency-domain operators, and builds AFFNet around AFF blocks.

3.1. Unified Formulation of Token Mixing

  • CNN and MLP mixers use deterministic weights, while global attention has O(N^2) complexity; lightweight design seeks global, instance-adaptive mixing with few parameters and low computation.Source: §3.1 · p. 3

3.2. Adaptive Frequency Filtering Token Mixer

  • AFF uses FFT to obtain frequency representations, applies a learned instance-adaptive mask by elementwise multiplication, then inverse-transforms to produce global mixing equivalent to an X-adaptive depthwise kernel as large as X.The frequency-domain mask is implemented with group linear layers and a ReLU; the equivalent convolution uses circular padding.Source: §3.2 · pp. 4–5

3.3. Analysis

  • AFNO and FFC use deterministic nonlinearities, GFNet and FNO use fixed instance-shared masks, and DFF shares its mask across channels, unlike AFF's semantic-adaptive filtering.Source: §3.3 · p. 5

3.4. Network Architectures

  • AFFNet stacks AFF blocks, pairing MBConv with Layer Normalization for channel mixing and AFF for global mixing; AFFNet, AFFNet-T, and AFFNet-ET have 5.5M, 2.6M, and 1.4M parameters, respectively.Skip connections are used for channel and token mixing to facilitate training.Source: §3.4 · pp. 5–6

4. Experiments

The experiments evaluate AFFNet’s accuracy and efficiency across ImageNet-1K classification, object detection, semantic segmentation, and token-mixer ablations. Across these settings, the reported results favor AFFNet and its adaptive frequency filtering design.

4.1. Image Classification

  • 13.5%, 8.2%, and 14.9% faster than MobileViT-S/XS/XXS, AFFNet variants also outperform comparable-size lightweight networks in ImageNet-1K Top-1 accuracy and lead accuracy-efficiency trade-offs.The three throughput comparisons correspond, respectively, to AFFNet, AFFNet-T, and AFFNet-ET.Source: §4.1 · p. 6, Table 2 · p. 6

4.2. Object Detection

  • AFFNet detectors consistently outperform lightweight CNN- and transformer-based detectors in COCO mAP; AFFNet exceeds EdgeNext by 0.5% mAP with 0.6M fewer parameters.AFFNet-ET exceeds MobileViTv2-0.5 by 0.6% mAP with fewer parameters.Source: §4.2 · p. 7, Table 3 · p. 7

4.4. Ablation Study

  • AFF token mixers add 0.3M parameters (<6%) and 0.2G FLOPs (~15%) over Base while raising ImageNet-1K Top-1 accuracy by 1.9%; versus 3×3 convolution, they improve Top-1 by 1.2% at about half the parameters and FLOPs.Base. is the baseline without spatial token mixers.Source: Table 4 · p. 7, §4.4 · p. 7
  • At equal model complexity, frequency-domain AFF beats the same adaptive filtering without Fourier transforms by 1.4% Top-1 accuracy; original-domain filtering is weaker than convolutional token mixing.The comparison removes the Fourier and inverse Fourier transforms while retaining the other adaptive-filtering operations.Source: §4.4 · p. 7, Table 4 · p. 7
  • Frequency-domain AFF outperforms deterministic FFC/AFNO and fixed-mask GFN/FNO token mixers; Base.+FNO uses more than 25× AFF’s parameters, while replacing the Hadamard product with summation drops Top-1 by 1.0%.AFF also exceeds Base.+DFF by 0.5% with fewer parameters and FLOPs.Source: §4.4 · p. 8, Table 5 · p. 8
  • Sharing a pooled 1×H×W mask across channels drops accuracy by 0.5% at the same complexity, and reaches the same accuracy as Base.+DFF.The result supports channel-specific token mixing.Source: Table 6 · p. 8, §4.4 · p. 8
  • 0.4% Top-1 accuracy is lost with depthwise mask layers (Ngroup=C), while regular layers (Ngroup=1) gain 0.1% at 40% more parameters and 33.3% more FLOPs.Increasing the spatial kernel size does not further improve performance and increases model complexity.Source: Table 7 · p. 8, §4.4 · p. 8

5. Conclusion

The conclusion presents adaptive frequency filters as mathematically equivalent, efficient global token mixers and AFFNet as a lightweight backbone with strong accuracy-efficiency trade-offs across vision tasks. It also revisits frequency-domain operations to identify what matters for token mixing.

  • AFFNet achieves state-of-the-art accuracy-efficiency trade-offs compared with other lightweight network designs across multiple vision tasks.Source: §5 · pp. 8–9
  • Adaptive frequency filters provide mathematically equivalent global token mixing, which the AFF token mixer uses for low-cost adaptive mixing.Source: §5 · p. 8
  • The paper revisits frequency-domain neural operations to examine which design factors matter for token mixing.Source: §5 · p. 9
  • The authors hope this work will inspire more interplay between conventional signal processing and deep learning.Source: §5 · p. 9

6. Detailed Network Architectures

AFFNet has three hierarchical versions with different channel dimensions, built from a convolutional stem and three stages of MBConv and AFF blocks.

6. Detailed Network Architectures

  • AFFNet, AFFNet-T, and AFFNet-ET are three hierarchical backbone versions with different channel dimensions.Source: §6 · p. 12
  • The tokenization stem uses a stride-2 3×3 convolution followed by four kernel-size-3 MBConv layers.Source: §6 · p. 12
  • The main body has three stages, each with a stride-2 MBConv down-sampling layer and N_i AFF blocks, with N1=2, N2=4, and N3=3.Source: §6 · p. 12

7. Detailed Introduction for Dataset

The paper describes ImageNet-1k for classification, COCO for detection, instance segmentation, and keypoint detection, and ADE20k and VOC for segmentation-related evaluation.

7. Detailed Introduction for Dataset

  • ImageNet has over 1.2 million images and 1,000 object categories, and the models are trained on ImageNet-1k for image classification.Source: §7 · p. 12
  • MS-COCO contains more than 200,000 images and 80 categories, with bounding-box, mask, and keypoint annotations for detection, instance segmentation, and keypoint detection.Source: §7 · p. 12
  • ADE20k contains 20,210 indoor and outdoor scene images with pixel-level labels for 150 semantic categories.Source: §7 · p. 12
  • PASCAL VOC 2012 has more than 11,000 images across 20 object categories with pixel-level boundary and semantic annotations.Source: §7 · p. 12

8. Detailed Experiment Settings

Detailed experiment settings are provided for each task, covering model, data, and training configurations.

8. Detailed Experiment Settings

  • The detailed settings for different tasks cover model, data, and training configurations.Source: §8 · p. 12

9. More Experiment Results

AFFNet shows strong accuracy-and-speed comparisons on mobile and GPU, and performs consistently better than vanilla versions of additional downstream frameworks. Ablations favor FFT over wavelet and DCT, channel mixing before token mixing, and MBConv over FFN.

9.1. Quantitative Results

  • On an A100 GPU, AFFNet improves Top-1 accuracy by 0.4% and throughput by 179 image/s over EdgeNext-S; on iPhone, it improves Top-1 accuracy by 1.7% with 0.3 ms less latency than mobilevitv2.The iPhone latency is measured with CoreML at batch size 1; A100 throughput uses TorchScript at batch size 128.Source: §9.1 · p. 12, Table 10 · p. 14
  • AFFNet encoders perform consistently better than their vanilla versions in mAP@COCO and mIOU@ADE20K across additional downstream task frameworks.The frameworks use multi-stage or multi-scale feature interactions; AFFNet is evaluated as their encoder.Source: §9.1 · p. 12, Table 11 · p. 14
  • Wavelet filtering improves over the no-frequency-transform baseline but remains inferior to FFT, while DCT is slightly inferior to FFT.The authors attribute wavelet's limitation to local-space filtering and DCT's to a coarser basis and greater information loss.Source: §9.1 · pp. 13–14, Table 12 · p. 14
  • Channel-mixing first is slightly superior to token-mixing first, although the two orders deliver very close results.The comparison indicates a preference for within-token refinement before token mixing.Source: §9.1 · p. 14, Table 13 · p. 14
  • MBConv is a more powerful lightweight channel mixer than FFN at almost the same computational cost.MBConv adds a depth-wise 3×3 convolution between two linear layers; FFN uses two cascaded linear layers.Source: §9.1 · p. 14, Table 14 · p. 14

10. Visualization Results

AFFNet's qualitative detection and segmentation results show precise object localization and classification across diverse scales and complex backgrounds.

10. Visualization Results

  • AFFNet's dense-prediction results show precise localization and classification across diverse object scales and complex backgrounds.The authors interpret these results as evidence that AFF preserves spatial structure during token mixing.Source: §10 · p. 14

11. Limitations

AFFNet's current running speed remains below its theoretical upper limit because FFT/iFFT engineering optimization has not been fully considered; the work currently focuses on vision.

11. Limitations

  • AFFNet's running speed remains below its theoretical upper limit because FFT/iFFT engineering optimization has not been fully considered.The work currently focuses on the vision domain and anticipates extension to other research fields.Source: §11 · p. 14

Related papers

Loading 2307.14008v1…