PaperlayerView exact revision on arXiv

Source-linked AI summary

Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice

Peihao Wang, Wenqing Zheng, Tianlong Chen, Zhangyang Wang

arXiv:2203.05962v1cs.CVcs.LG

TL;DR

Deep ViTs face scaling difficulties, including attention collapse, patch uniformity, and a lack of rigorous analysis. The paper analyzes self-attention in the Fourier domain, proposes AttnScale and FeatScale, and reports up to 1.1% gains across ViT variants.

  • Problem

    Deep ViT scaling is limited by attention collapse and patch uniformity, while rigorous analysis of this scalability issue remains elusive.

  • Method

    The paper uses Fourier-domain analysis to characterize self-attention and proposes AttnScale and FeatScale to re-adjust low- and high-frequency components.

  • Results

    Up to 1.1%, 0.6% and 0.5% performance gains are reported for DeiT, CaiT, and Swin-Transformer, respectively.

  • Takeaways & Limitations

    The techniques help different ViT variants benefit from deeper architectures with little parameter overhead.

  • Takeaways & Limitations

    The theoretical token-distribution analysis assumes tokens lie inside a ball with radius γ > 0.

Abstract

from arXiv · show

Vision Transformer (ViT) has recently demonstrated promise in computer vision problems. However, unlike Convolutional Neural Networks (CNN), it is known that the performance of ViT saturates quickly with depth increasing, due to the observed attention collapse or patch uniformity. Despite a couple of empirical solutions, a rigorous framework studying on this scalability issue remains elusive. In this paper, we first establish a rigorous theory framework to analyze ViT features from the Fourier spectrum domain. We show that the self-attention mechanism inherently amounts to a low-pass filter, which indicates when ViT scales up its depth, excessive low-pass filtering will cause feature maps to only preserve their Direct-Current (DC) component. We then propose two straightforward yet effective techniques to mitigate the undesirable low-pass limitation. The first technique, termed AttnScale, decomposes a self-attention block into low-pass and high-pass components, then rescales and combines these two filters to produce an all-pass self-attention matrix. The second technique, termed FeatScale, re-weights feature maps on separate frequency bands to amplify the high-frequency signals. Both techniques are efficient and hyperparameter-free, while effectively overcoming relevant ViT training artifacts such as attention collapse and patch uniformity. By seamlessly plugging in our techniques to multiple ViT variants, we demonstrate that they consistently help ViTs benefit from deeper architectures, bringing up to 1.1% performance gains "for free" (e.g., with little parameter overhead). We publicly release our codes and pre-trained models at https://github.com/VITA-Group/ViT-Anti-Oversmoothing.

1 INTRODUCTION

The paper frames deep ViT scaling as a problem of feature degradation and addresses it with a Fourier-domain theory and two efficient scaling techniques. Experiments show gains across multiple ViT backbones.

  • 1 INTRODUCTION: Deeper vanilla ViTs can underperform shallower models, while deeper layers exhibit reduced patch diversity and rank collapse.Prior remedies are largely empirical, motivating a more principled analysis.
  • 1 INTRODUCTION: The paper establishes a Fourier-space analysis showing that cascaded self-attention behaves as repeated low-pass filtering.This connects spectral filtering with patch diversity loss and rank collapse.
  • 1 INTRODUCTION: AttnScale decomposes attention into low- and high-pass components and adaptively amplifies the high-pass component to enforce an all-pass filter.It operates directly on the calculated attention map.
  • 1 INTRODUCTION: FeatScale re-weights separate frequency bands in feature maps to selectively enhance high-frequency signals.Both techniques avoid explicit Fourier transformation and add little computational complexity.
  • 1 INTRODUCTION: Up to 1.1%, 0.6% and 0.5% performance gains are reported for DeiT, CaiT, and Swin-Transformer, respectively.The techniques are integrated with different ViT backbones without substantial extra machinery.

2 WHY VIT CANNOT GO DEEPER?

The paper analyzes ViT features in the Fourier domain and shows that self-attention progressively suppresses high-frequency information. This low-pass behavior explains why deeper ViTs lose feature expressiveness as attention operations accumulate.

  • ViT architecture: ViT processes image patches through stacked transformer blocks containing multi-head self-attention, feed-forward networks, normalization, and skip connections.The self-attention module aggregates information across image tokens, while multi-head attention combines multiple attention heads through a projection.
  • Fourier analysis: The Fourier analysis separates each channel signal into a Direct-Current component and a complementary high-frequency component.The paper defines DC[z] from the first Fourier coefficient and HC[z] from the remaining Fourier basis components.
  • Self-attention as low-pass filtering: Self-attention matrices produced by softmax are low-pass filters, independent of the input token features or the key/query matrices.Theorem 1 states that repeated application drives the ratio of high-frequency to DC energy to zero.
  • Self-attention as low-pass filtering: Composing distinctive self-attention matrices across ViT layers still acts like a low-pass filter, so layerwise recomputation does not remove the smoothing effect.The paper supports this conclusion with a corollary and visualizations of attention-map spectra.
  • Smoothing rate: The high-frequency component can be annihilated exponentially under a sufficient condition, while ∥WV∥2 < 1/√n guarantees monotonically decreasing high-frequency content.Theorem 3 bounds the smoothing rate, and its assumptions include token features lying within a ball of radius γ.
  • Other transformer components: The analysis extends beyond a single self-attention module to multi-head attention and other transformer building blocks when assessing whether they alleviate low-pass filtering.The paper derives these results from Theorem 3 and examines MSA, FFN, and residual connections.

Appendix C.1 that the convergence rate turns to σ1σ2H

The analysis shows that residual connections and feed-forward networks can slow high-frequency decay but cannot remove the underlying low-pass behavior of self-attention. This motivates frequency-selective scaling methods that separately amplify high-frequency components.

  • Residual connections: Residual connections can prevent high-frequency components from immediately diminishing to zero, but only by promoting their convergence rate.The skip path retrieves information that self-attention suppresses, although the high-frequency component remains diluted.
  • Feed-forward networks: A feed-forward network does not improve the original convergence rate when characterized by its Lipschitz constant.With a skip connection over the FFN, σ3 > 1 can make the upper bound of the high-frequency component non-contractive.
  • Limits of standard blocks: Multi-head attention, FFNs, and skip connections may slow convergence, but they amplify low- and high-frequency components indiscriminately and cannot promote high-frequency information separately.The MSA block therefore retains only low-pass representational power as depth increases.
  • Spectral interpretation: Self-attention remains a low-pass filter across attention-computation choices, with repeated application driving outputs toward the Direct-Current component.This spectral behavior parallels over-smoothing analyses in graph convolutional networks and specifies the rank-1 limit as the DC subspace.
  • AttnScale: AttnScale addresses the limitation by decomposing attention into low-pass and high-pass components and rescaling the high-pass part to form an all-pass filter.The method extracts L = 11T / n and combines it with a rescaled complementary component.
  • FeatScale: FeatScale instead decomposes MSA outputs into DC and high-frequency components and re-weights them channel-wise to prevent DC dominance.The components are computed efficiently without an explicit Fourier transform: DC uses a column average and HC uses X − DC[X].

5 EXPERIMENTS

Experiments evaluate AttnScale and FeatScale across DeiT, CaiT, and Swin-Transformer backbones on ImageNet under varied depth and training settings. Both methods improve multiple ViT variants, with gains reaching 1.1% and state-of-the-art results for some models.

  • Experiment Settings: Experiments use ImageNet and evaluate AttnScale and FeatScale with DeiT, CaiT, and Swin-Transformer under different depths and training modes.The ImageNet setup uses approximately 1.3M training images and 50k validation images.
  • Results: 1.0% top-1 accuracy gain is achieved by FeatScale on 12-layer DeiT, while AttnScale adds 0.9% with fewer than 100 extra parameters.For 24-layer DeiT, FeatScale improves top-1 accuracy by 0.8% and AttnScale by 0.6%.
  • Results: 0.6% performance gain is observed when AttnScale and FeatScale are integrated with CaiT, while both methods add around 0.5% accuracy on Swin-Transformer.Under fine-tuning, tens of epochs further improve performance by at least 0.2%.
  • Comparison with SOTA Models: Swin-Transformer combined with the proposed methods achieves state-of-the-art performance, while 24-layer CaiT-S variants outperform other pure transformers at low parameter cost.DeiT-S+FeatScale also outperforms ViT-B/16 and DeiT-S Distilled in the reported comparison.

A MORE PRELIMINARIES ON FOURIER ANALYSIS

This appendix formulates discrete Fourier analysis for real-valued one-dimensional and multi-channel signals, then defines DC and high-frequency components as complementary operators. These operators separate the constant signal component from the remaining frequencies.

  • Fourier Preliminaries: The appendix uses the discrete Fourier transform on real-valued signals, with a one-dimensional formulation because DC components are dimension-invariant.The transform maps R^n to C^n.
  • Fourier Preliminaries: Multi-channel signals are transformed independently channel by channel using the DFT matrix and its inverse.A matrix X ∈ R^(n×d) is treated as d channels of length n.
  • DC and High-Frequency Components: DC[·] retains only the first Fourier component and equals the averaging operator 11^T/n.The matrix form is DFT^-1 diag(1, 0, ···, 0) DFT.
  • DC and High-Frequency Components: The DC and high-frequency operators provide the decomposition used repeatedly in the paper’s subsequent proofs.They separate the constant component from the remaining signal components.
  • DC and High-Frequency Components: HC[·] retains all non-DC Fourier components and is the complementary operator I − 11^T/n.Its Fourier-domain mask is diag(0, 1, ···, 1).

B.1 PROOF OF THEOREM 1

Theorem 1 proves that a softmax self-attention matrix is a low-pass filter by analyzing its eigenvalues and Jordan form. Repeated application suppresses all modes except the all-one, DC-associated component.

  • Theorem 1: A softmax attention matrix is strictly positive and row-stochastic, so the all-one vector is an eigenvector with eigenvalue 1.Every row sums to one, yielding A1 = 1.
  • Eigenvalue Analysis: Perron-Frobenius theory makes eigenvalue 1 the unique largest eigenvalue, while every other eigenvalue has absolute value below 1.The matrix is rewritten in Jordan canonical form to analyze powers of A.
  • Jordan-Form Argument: Applying the attention matrix t times is analyzed through powers of its Jordan blocks, whose non-dominant terms decay as t increases.The proof uses f(x) = x^t and tracks diagonal and superdiagonal Jordan-block terms.
  • Limit: The dominant eigenvector is the all-one vector, identifying the limiting component with the DC subspace.The proof invokes the correspondence between λ1 = 1 and v1 = 1.
  • Limit: Orthogonality between DC and high-frequency terms is used when substituting the limiting expression into the original limit.This connects the spectral decomposition to the low-pass-filter conclusion.

B.2 PROOF OF COROLLARY 2

Corollary 2 extends the low-pass result from one attention matrix to a product of softmax attention matrices. The proof shows that the product remains a valid self-attention matrix by preserving row sums and strict positivity.

  • Corollary 2: A product of softmax attention matrices is treated as a candidate composite attention matrix to which Theorem 1 can be applied.The proof establishes the required matrix properties before invoking the theorem.
  • Conclusion: The product therefore qualifies as another self-attention matrix and is also a low-pass filter.This is the corollary’s conclusion from the preserved properties and Theorem 1.
  • Row-Stochasticity: Each factor has row sums equal to one, and induction shows that their product also has every row summing to one.The row-stochastic property is preserved through successive matrix multiplication.
  • Strict Positivity: Strict positivity is preserved under multiplication because every summand in each product entry is positive.Induction propagates positivity from the first factor through the full product.

B.3 PROOF OF THEOREM 3

This section proves bounds on how self-attention, multi-head attention, skip connections, and feed-forward networks suppress high-frequency feature components. It also connects the asymptotic behavior of attention maps to Markov-chain convergence.

  • Self-attention: Theorem 3 bounds the high-frequency output of a self-attention module using the attention logits, value projection, and input high-frequency magnitude.The theorem considers A = softmax(P), α = max_i,j|P_ij|, and SA(X) = AXW_V.
  • Proof setup: The proof decomposes X into its DC projection and complementary high-frequency component before bounding the latter.It writes X = DC[X] + HC[X], with DC[X] = 1z^T and H = HC[X].
  • DC component: The DC component is the least-squares optimal constant-frequency approximation, with β* = 1 minimizing the low-pass approximation objective.The auxiliary derivation identifies z* = X^T1/n as the minimizing average representation.
  • Markov interpretation: Self-attention maps can be viewed as Markov transition matrices whose positive entries imply convergence toward a unique stationary distribution.The contraction is described through the leading nontrivial eigenvalue λ ∈ (0,1).

D.2 REMARKS ON DONG ET AL. (2021)

This section examines assumptions and proof gaps in Dong et al. (2021), while noting that the paper’s own framework generalizes to several attention mechanisms. It identifies a missed bounded-difference assumption, unresolved derivation steps, and a counterexample to one claimed inequality.

  • Proof assumptions: The proof of Lemma A.3 implicitly requires max_i,j(E_ij − E_ij′) ≤ 1 for its Taylor-based upper bound.The authors reportedly confirmed this missed assumption after communication with the paper’s authors.
  • Proof assumptions: For Lemma A.1, the proof may assume entry-wise nonnegative R and W_V; entry-wise absolute values are proposed as a norm-preserving fix.The passage states that this change does not affect the relevant ℓ_1 and ℓ_∞ norms.
  • Proof gaps: The derivation of Eqn. (6)-(8) remains difficult to verify even after proposed changes to the proof.The authors were still communicating about this issue.
  • Counterexample: Eqn. (12) is disproved by a concrete matrix counterexample, suggesting additional constraints on E and r may be necessary.The example gives left-hand side 4 versus right-hand side 3.
  • Generalization: The paper’s Theorems 1 and 3 do not require prior knowledge of pre-softmax pairwise correlations, enabling extension beyond dot-product attention.The appendix discusses logistic and L2-distance attention and supplies corresponding α bounds.

F.1 DETAILS ON FIGURE 1

This section details experiments that compare measured high-frequency feature intensity with theoretical bounds and visualizes attention spectra and token-similarity behavior. The visualizations use DeiT checkpoints and ImageNet validation samples.

  • Figure 1 details: The experiment compares measured log high-frequency intensity against a parameter-derived upper bound across layers of a 12-layer DeiT-S checkpoint.The measured quantity is log(∥HC[X_l]∥_F / ∥X_0∥_F), while the bound uses γ_l∥HC[X_{l−1}]∥_F.
  • Figure 1 details: Without residual connections, the first two subfigures exhibit exponential convergence, consistent with Theorem 3.The visualization removes FFN or residual components from a pretrained model to isolate component effects.
  • Spectrum visualization: Attention spectra are computed as Λ = FAF^-1, with ∥Λ_i∥_2 used as the intensity of frequency band i.The visualization applies the Fourier-domain response to signal spectra from random ImageNet validation samples.
  • Spectrum visualization: Figure 5 arranges attention-spectrum visualizations with layers as rows and heads as columns.It covers 12 layers vertically and six heads horizontally.
  • Similarity metrics: Attention-map similarity averages pairwise cross-patch column similarity within each layer and across heads, excluding self-to-self similarity.Feature-map similarity analogously averages pairwise cosine similarity between distinct token rows.

G.1 FINE-TUNING EXPERIMENTS

This section describes fine-tuning and visualization experiments for AttnScale and FeatScale. It reports learned frequency reweighting behavior, attention-map changes, and evaluation protocols for CaiT and related analyses.

  • Fine-tuning protocol: CaiT fine-tuning runs AttnScale and FeatScale for 60 epochs against a simultaneously trained plain CaiT baseline.The fine-tuning uses learning rate 5 × 10^-5 and weight decay 5 × 10^-4, with the remaining recipe held consistent.
  • Evaluation: Table 3 encodes performance gains inside upward-arrow parentheses and distinguishes reported from reproduced accuracies.It evaluates fine-tuned AttnScale and FeatScale with CaiT.
  • AttnScale: AttnScale learns stronger high-pass amplification at deeper layers and produces more salient, higher-contrast attention than vanilla DeiT.The learned scaling weights increase with layer index, while the attention comparison uses a random ImageNet validation sample.
  • FeatScale: FeatScale assigns consistently larger reweighting factors to high-frequency components than to the DC term, with a larger gap in deeper layers.Its stated purpose is to counter dominance by the DC component.
  • Additional visualizations: Additional visualizations cover 24-layer attention spectra, 12/24-layer FeatScale high-frequency proportions, and cosine similarities with and without the proposed methods.These figures extend the qualitative analysis across depth and frequency behavior.
Loading 2203.05962v1…