Source-linked AI summary
FcaNet: Frequency Channel Attention Networks
Zequn Qin, Pengyi Zhang, Fei Wu, Xi Li
TL;DR
Channel attention typically compresses each channel into a scalar, leaving a fundamental representation problem because substantial information may be lost. FcaNet treats this as frequency-domain compression, proves that GAP is a special case of DCT, and introduces multi-spectral channel attention. It achieves state-of-the-art results across image classification, object detection, and instance segmentation while consistently outperforming SENet at the same parameter count and computational cost.
Problem
Channel attention uses one scalar to represent each channel, creating a fundamental compression problem because channel information can be substantially lost.
Method
FcaNet uses DCT-based multi-spectral channel attention to generalize channel compression in the frequency domain, with selected frequency components and one scalar per channel.
Results
FcaNet achieves state-of-the-art performance on image classification, object detection, and instance segmentation, consistently outperforming SENet with the same parameters and computational cost.
Takeaways & Limitations
The framework extends existing channel attention in the frequency domain and can be integrated into existing channel-attention implementations with only a few lines of code changes.
Takeaways & Limitations
The paper notes that constant normalization factors are removed from Eqs. 2 and 3 for simplicity, without affecting the reported results.
Abstract
from arXiv · showhide
Attention mechanism, especially channel attention, has gained great success in the computer vision field. Many works focus on how to design efficient channel attention mechanisms while ignoring a fundamental problem, i.e., channel attention mechanism uses scalar to represent channel, which is difficult due to massive information loss. In this work, we start from a different view and regard the channel representation problem as a compression process using frequency analysis. Based on the frequency analysis, we mathematically prove that the conventional global average pooling is a special case of the feature decomposition in the frequency domain. With the proof, we naturally generalize the compression of the channel attention mechanism in the frequency domain and propose our method with multi-spectral channel attention, termed as FcaNet. FcaNet is simple but effective. We can change a few lines of code in the calculation to implement our method within existing channel attention methods. Moreover, the proposed method achieves state-of-the-art results compared with other channel attention methods on image classification, object detection, and instance segmentation tasks. Our method could consistently outperform the baseline SENet, with the same number of parameters and the same computational cost. Our code and models will are publicly available at https://github.com/cfzd/FcaNet.
1. Introduction
FcaNet treats channel representation as a compression problem and extends channel attention into the frequency domain with multi-spectral DCT components. The method targets information loss from scalar channel representations and reports state-of-the-art performance at SENet-matched cost.
- Motivation: Channel attention commonly represents each channel with one scalar, making effective compression of the channel a fundamental difficulty.Global average pooling is widely used because it is simple and efficient, but its simplicity makes complex information difficult to capture.
- Motivation: DCT is introduced for channel compression because of its energy compaction, differentiability, and compatibility with CNN operations.The paper also identifies DCT as a generalization of global average pooling in which GAP corresponds to the lowest-frequency components.
- Method: FcaNet proposes multi-spectral channel attention that uses multiple limited DCT frequency components while retaining one scalar representation per channel.The framework includes LF, TS, and NAS criteria for selecting frequency components.
- Results: The method achieves state-of-the-art results on ImageNet and COCO with the same computational cost as SENet.The introduction reports ImageNet results in Figure 1.
2. Related Work
Prior work developed spatial, channel, and combined attention mechanisms for CNNs, while other studies applied attention to specific tasks. Frequency analysis has also been used in CNN processing, bandwidth reduction, compression, and pruning.
- Attention Mechanism in CNNs: SENet introduced channel attention by applying global average pooling and fully connected layers to calculate channel weights.Related methods subsequently combined spatial and channel attention in CNNs.
- Attention Mechanism in CNNs: CBAM, SRM, and GSoP addressed limitations of global average pooling using max pooling, standard-deviation pooling, or second-order pooling.CBAM describes GAP as producing a sub-optimal feature because of information loss.
- Task-Specific Attention: Attention mechanisms have also been extended to multi-label classification, saliency detection, visual explanation, and super-resolution.
- Frequency Domain Learning: Frequency analysis has been incorporated into CNNs for JPEG encoding, communication-bandwidth reduction, model compression, and pruning.
3. Method
FcaNet treats channel attention as a compression problem and replaces GAP-only compression with multi-spectral DCT components. It proves GAP is proportional to the lowest-frequency DCT component, then selects frequency components for channel groups using three criteria.
- Channel Attention and DCT: FcaNet models scalar channel representation as compression and uses DCT to represent channel information in the frequency domain.The method is motivated by DCT's energy-compaction property and its use in data compression.
- Multi-Spectral Channel Attention: Because GAP preserves only the lowest-frequency information, FcaNet generalizes channel compression to multiple DCT frequency components.The framework retains the lowest-frequency component while incorporating additional components to enrich the compressed channel representation.
- Multi-Spectral Channel Attention: The multi-spectral framework splits the input along channels, assigns each part a DCT frequency component, concatenates the compressed vectors, and computes channel attention.The resulting multi-spectral vector is passed through the channel-attention mapping and sigmoid.
- Frequency Component Selection: FcaNet provides three frequency-selection criteria: low-frequency selection, two-step Top-k selection, and neural architecture search.FcaNet-TS evaluates components individually before selecting the Top-k performers, while FcaNet-NAS searches components using continuous variables.
4. Experiments
Experiments show that multi-spectral frequency components improve channel attention across classification, detection, and segmentation while retaining SENet-like efficiency. The ablations support DCT-based compression and identify effective component-selection settings.
- Experimental scope: FcaNet evaluates ablations, frequency-component selection, fully learnable comparisons, complexity, implementation, and performance across three vision tasks.The experiments cover ImageNet classification, COCO object detection, and COCO instance segmentation.
- Frequency ablations: Lower frequencies perform best individually, while nearly all components except the highest remain within 0.5% Top-1 accuracy of GAP.This supports generalizing channel attention into the frequency domain rather than retaining only GAP’s lowest-frequency component.
- Frequency ablations: Multi-spectral attention significantly outperforms GAP-only attention; FcaNet-LF selects 2 components and FcaNet-TS selects 16 components for the reported method.These settings are chosen from the component-count ablation results.
- Learnability comparison: DCT initialization outperforms random initialization, and fixed DCT initialization also outperforms fully learnable channel-attention variants.The comparison treats the proposed method as the fixed-tensor, DCT-initialized FD setting.
- Efficiency and implementation: FcaNet has no extra parameters and adds only 0.04%, 0.13%, 0.11%, and 0.11% relative computational cost for ResNet-34, -50, -101, and -152.The DCT weights are pre-computed constants, and the method is described as having effectively the same cost as SENet.
- Efficiency and implementation: FcaNet can be integrated into arbitrary channel-attention methods with only a few code changes replacing GAP by multi-spectral 2D DCT compression.The DCT operation is implemented as element-wise multiplication followed by summation.
- Task results: FcaNet achieves the best performance nearly across ImageNet settings, COCO detection frameworks, and COCO instance segmentation comparisons.It outperforms ECANet by 0.9–1.3% AP in detection and GCNet by 0.5% AP in instance segmentation.
- Task results: FcaNet outperforms SENet with the same number of parameters and computational cost, and exceeds GSoPNet1 at matched computational cost.GSoPNet1 has higher performance in one comparison but costs 1.5 times more computationally.
5. Conclusion
The paper frames channel representation as a compression problem, proves that GAP is a special case of DCT, and proposes FcaNet’s multi-spectral attention module. FcaNet consistently outperforms SENet at equal parameter count and computational cost, while achieving state-of-the-art results across three vision tasks.
- FcaNet treats channel representation as a fundamental compression problem and generalizes channel attention in the frequency domain.
- The method proves that global average pooling is a special case of DCT and uses multi-spectral attention to extend existing channel attention.
- FcaNet explores frequency-component combinations and proposes three criteria for selecting components in its multi-spectral framework.
- FcaNet consistently outperforms SENet with the same number of parameters and computational cost.
- FcaNet achieves state-of-the-art performance on image classification, object detection, and instance segmentation compared with other channel attention methods.
- The method can be implemented in existing channel attention methods with only a few lines of code changes.