Source-linked AI summary
SpectFormer: Frequency and Attention is what you need in a Vision Transformer
Badri N. Patro, Vinay P. Namboodiri, Vijay Srinivas Agneeswaran
TL;DR
Vision transformers have primarily used either multi-headed self-attention or spectral layers, motivating an analysis of whether combining them improves image representations. SpectFormer places spectral layers early and multi-headed attention deeper in the network, achieving strong ImageNet-1K results and consistent performance across transfer-learning and downstream tasks.
Problem
Vision transformers have used either multi-headed self-attention or spectral layers, leaving the value of combining both approaches to be investigated.
Method
SpectFormer combines initial Fourier-based spectral layers with deeper multi-headed self-attention layers and evaluates the design against alternative transformer architectures.
Results
85.7% top-1 accuracy on ImageNet-1K is reported for SpectFormer, alongside consistent performance on transfer learning, object detection, and instance segmentation.
Takeaways & Limitations
Combining spectral and attention layers yields consistently better performance than all-attention or all-spectral architectures within the reported evaluations.
Abstract
from arXiv · showhide
Vision transformers have been applied successfully for image recognition tasks. There have been either multi-headed self-attention based (ViT \cite{dosovitskiy2020image}, DeIT, \cite{touvron2021training}) similar to the original work in textual models or more recently based on spectral layers (Fnet\cite{lee2021fnet}, GFNet\cite{rao2021global}, AFNO\cite{guibas2021efficient}). We hypothesize that both spectral and multi-headed attention plays a major role. We investigate this hypothesis through this work and observe that indeed combining spectral and multi-headed attention layers provides a better transformer architecture. We thus propose the novel Spectformer architecture for transformers that combines spectral and multi-headed attention layers. We believe that the resulting representation allows the transformer to capture the feature representation appropriately and it yields improved performance over other transformer representations. For instance, it improves the top-1 accuracy by 2\% on ImageNet compared to both GFNet-H and LiT. SpectFormer-S reaches 84.25\% top-1 accuracy on ImageNet-1K (state of the art for small version). Further, Spectformer-L achieves 85.7\% that is the state of the art for the comparable base version of the transformers. We further ensure that we obtain reasonable results in other scenarios such as transfer learning on standard datasets such as CIFAR-10, CIFAR-100, Oxford-IIIT-flower, and Standford Car datasets. We then investigate its use in downstream tasks such of object detection and instance segmentation on the MS-COCO dataset and observe that Spectformer shows consistent performance that is comparable to the best backbones and can be further optimized and improved. Hence, we believe that combined spectral and attention layers are what are needed for vision transformers.
1. Introduction
SpectFormer adapts vision transformers by combining early Fourier-based spectral layers with deeper multi-headed self-attention. The architecture is empirically evaluated against attention-only, spectral-only, and hierarchical alternatives across classification and downstream tasks.
- Proposed architecture: SpectFormer uses spectral layers in initial stages and multi-headed self-attention in deeper layers to combine frequency-based and global representations.The spectral component transforms image tokens to the Fourier domain, applies learnable gating, and transforms them back.
- Empirical validation: SpectFormer outperforms LiT, GFNet, AFNO, and DeIT on ImageNet-1K according to the authors' comparisons.The comparisons include hierarchical, spectral, and complete multi-headed-attention transformer families.
- Motivation: The architecture is designed to address limitations of using only attention or only spectral layers, with a parameterized mixture of the two.The mixture can vary the number of spectral and attention layers through an α factor.
- Transfer learning: The proposed architecture obtains reasonable transfer-learning performance on CIFAR-10 and CIFAR-100.
- Downstream tasks: SpectFormer shows consistent performance on MS COCO object detection and instance segmentation tasks.
2. Related Work
Related work spans attention-based, spectral, and hierarchical vision transformers. SpectFormer is positioned as a Fourier-based hierarchical architecture combining localized spectral processing with later attention layers.
- SpectFormer variants: SpectFormer architecture specifications cover small, base, and large variants with stage-wise expansion ratios, spectral gating numbers, heads, and channel dimensions.
- Attention-based transformers: Vision transformers use multi-headed self-attention to mix image tokens and model global relationships.The related work includes vanilla and hierarchical transformer families such as DeiT, TNT, CaiT, CrossViT, Swin, and PVTv2.
- Spectral transformers: Spectral transformers replace or modify self-attention with Fourier or learned global-filter operations for token mixing.Examples include FNet, GFNet, AFNO, and Wave-ViT.
- Hierarchical transformers: Hierarchical transformers organize computation into stages to improve performance and capture multi-scale context.The literature includes PVT, Swin, Twins, and LiT.
- SpectFormer configurations: For vanilla SpectFormer, the configuration uses four spectral layers with α = 4 and L − α remaining attention layers.FLOPs are reported for 224 × 224 and 384 × 384 input sizes.
3. Method: Spectformer
SpectFormer combines initial spectral layers with deeper multi-headed attention layers, motivated by evidence that this ordering is more beneficial than all-attention, all-spectral, or inverse configurations. Its spectral blocks transform tokens into frequency space, apply learnable gating, and return them to physical space before attention-based processing.
- Mixed spectral-attention design: Initial spectral layers followed by multi-headed attention layers provide the more beneficial mixed configuration in the architecture study.The comparison included all-attention, all-spectral, and mixed arrangements with spectral layers placed initially or finally.
- Architecture: SpectFormer processes image patches through patch and positional embeddings, transformer blocks, and a classification head.The transformer block contains spectral layers followed by attention layers.
- Spectral layer: The spectral layer uses FFT, learnable frequency gating, and inverse FFT to capture localized frequency components before channel mixing.Layer normalization and an MLP follow the inverse transform, while spectral gating performs token mixing.
- Attention layer: The attention layer applies layer normalization, multi-headed self-attention for token mixing, another normalization layer, and an MLP for channel mixing.Its attention structure is similar to the DeIT attention architecture.
- Layer allocation: The α factor controls the number of spectral and attention layers, ranging from all-attention at α=0 to all-spectral at α=12.The staged architecture distributes SpectFormer blocks across four stages with varying spectral and attention compositions.
4. Experiments and Results
SpectFormer is evaluated through ablations, ImageNet classification, transfer learning, high-resolution fine-tuning, downstream detection and segmentation, and filter visualization. Across these evaluations, mixed spectral-attention models show strong or improved performance against spectral, attention, hybrid, hierarchical, CNN, and vision-transformer baselines.
- Evaluation design: The evaluation covers spectral-layer and α ablations, ImageNet-1K training, transfer learning, COCO detection and segmentation, and learned-filter visualization.Transfer-learning datasets include CIFAR-10, CIFAR-100, Oxford-IIIT flower, and Stanford Car.
- Spectral ablation: Fourier Gating Network outperforms Fourier Network, Wavelet Gating Network, and Fourier Neural Operator in the spectral-layer ablation.The ablation uses small-size staged networks.
- Model variants: All hierarchical SpectFormer models outperform the vanilla architecture and are reported as state of the art.The comparison includes small, base, and large hierarchical variants, with α = 4 for the model-size table.
- ImageNet-1K comparison: SpectFormer performs better than vanilla vision transformers, hybrid transformers, spectral transformers, and hierarchical attention architectures across the reported ImageNet-1K size comparisons.The comparisons include DeiT, FNet, FourierFormer, GFNet, PVT, Swin, LiT, and LiTv2.
- ImageNet-1K comparison: 2% better than LiTv1 for the small architecture and 3% better than GFNet-H-S are reported for SpectFormer.The passage also reports that SpectFormer performs better than DeiT.
- ImageNet-1K comparison: 83.1 top-1 accuracy is reported for SpectFormer-S without token labeling, exceeding Wave-ViT-s at 82.7 under the stated comparison.The enhanced SpectFormer variants use token labeling and MixToken, while SpectFormer-S is the degraded no-token-labeling version.
- State-of-the-art comparison: 85.1% top-1 accuracy for SpectFormer-B⋆ improves by 0.3% over Wave-ViT-B⋆, while 85.7% for SpectFormer-L⋆ improves by 0.2% over Wave-ViT-L⋆.These comparisons are reported under the Base and Large sizes, respectively.
- Downstream tasks: SpectFormer achieves comparative COCO performance across RetinaNet and Mask R-CNN while outperforming ResNet and the compared vanilla and hierarchical transformer models in AP.The experiments evaluate object detection and instance segmentation using an ImageNet-1K-pretrained backbone.
5. Conclusion
SpectFormer combines spectral and multi-headed attention layers, achieving consistently better performance than architectures using either approach alone. Its parameterized design also supports task-specific adaptations, and it reaches 85.7% top-1 accuracy on ImageNet-1K.
- SpectFormer combines spectral and multi-headed attention layers and shows consistently better performance than all-attention or all-spectral transformers.
- Its parameterized design allows different tasks to use different combinations of spectral and attention layers.
- 85.7% top-1 accuracy is achieved on ImageNet-1K.