Source-linked AI summary
GhostNetV2: Enhance Cheap Operation with Long-Range Attention
Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Chao Xu, Yunhe Wang
TL;DR
Lightweight CNNs struggle to capture long-range dependencies, while conventional self-attention can make mobile inference impractical. The paper proposes hardware-friendly DFC attention and GhostNetV2, achieving 75.3% top-1 accuracy on ImageNet with 167M FLOPs versus GhostNetV1’s 74.5% at similar cost.
Problem
Lightweight convolutional models have weak long-range dependency modeling, while self-attention is computationally and operationally unfriendly for mobile deployment.
Method
DFC attention decomposes fully connected transformations into horizontal and vertical layers, and GhostNetV2 applies it to expanded GhostNet features.
Results
75.3% top-1 accuracy on ImageNet with 167M FLOPs surpasses GhostNetV1’s 74.5% with similar computational cost.
Takeaways & Limitations
GhostNetV2 achieves a better accuracy–speed trade-off for lightweight mobile vision models while capturing long-range spatial information.
Takeaways & Limitations
DFC attention is designed around long-range capture, deployment efficiency, and conceptual simplicity as explicit design requirements.
Abstract
from arXiv · showhide
Light-weight convolutional neural networks (CNNs) are specially designed for applications on mobile devices with faster inference speed. The convolutional operation can only capture local information in a window region, which prevents performance from being further improved. Introducing self-attention into convolution can capture global information well, but it will largely encumber the actual speed. In this paper, we propose a hardware-friendly attention mechanism (dubbed DFC attention) and then present a new GhostNetV2 architecture for mobile applications. The proposed DFC attention is constructed based on fully-connected layers, which can not only execute fast on common hardware but also capture the dependence between long-range pixels. We further revisit the expressiveness bottleneck in previous GhostNet and propose to enhance expanded features produced by cheap operations with DFC attention, so that a GhostNetV2 block can aggregate local and long-range information simultaneously. Extensive experiments demonstrate the superiority of GhostNetV2 over existing architectures. For example, it achieves 75.3% top-1 accuracy on ImageNet with 167M FLOPs, significantly suppressing GhostNetV1 (74.5%) with a similar computational cost. The source code will be available at https://github.com/huawei-noah/Efficient-AI-Backbones/tree/master/ghostnetv2_pytorch and https://gitee.com/mindspore/models/tree/master/research/cv/ghostnetv2.
1 Introduction
Lightweight CNNs target efficient mobile deployment, but convolution captures only local information while self-attention captures global dependencies at impractical cost. GhostNetV2 addresses this tension with hardware-friendly DFC attention and enhanced GhostNet features.
- Mobile deployment requires balancing model performance with actual inference speed and efficiency.
- Lightweight convolutional models have limited long-range dependency modeling, whereas self-attention is costly and hardware-unfriendly in practice.Self-attention introduces quadratic complexity and memory-intensive reshaping operations that increase latency on mobile devices.
- DFC attention uses decomposed horizontal and vertical fully connected layers to capture long-range spatial information efficiently.Stacking the two directional transformations produces a global receptive field while retaining implementation efficiency.
- GhostNetV2 enhances intermediate GhostNet features with DFC attention to combine local and long-range information.The architecture is designed to improve the accuracy–inference-speed trade-off over existing lightweight models.
2 Related Work
Lightweight network research seeks high performance with fast inference through efficient convolutional structures and hardware-aware architecture design. Related approaches reduce computation using separable or grouped operations, inverted bottlenecks, cheap operations, or neural architecture search.
- Designing lightweight architectures that jointly provide fast inference and high performance remains challenging.
- MobileNet reduces computation with depthwise separable convolutions, while MobileNetV2 adds residual inverted bottlenecks and MobileNeXt reconsiders bottleneck structure.
- ShuffleNet lowers pointwise-convolution cost through group convolution and channel shuffling, while GhostNet generates feature maps using cheap operations.
- Neural architecture search methods such as FBNet, MnasNet, and MobileNetV3 search architecture or hardware-aware design choices within defined spaces.
3 Preliminary
GhostNet reduces computation by generating intrinsic features with pointwise convolution and additional features through cheap operations, but this weakens spatial representation. Attention alternatives and deployment measurements motivate hardware-friendly long-range modeling.
- GhostNet first generates intrinsic features with a 1 × 1 convolution and then derives additional features from them using cheap operations.
- The Ghost module concatenates intrinsic and cheaply generated features to form the output feature.
- GhostNet’s representation ability is weakened because spatial interaction is limited for features produced by pointwise convolution.Only part of the features receives spatial information from cheap operations such as 3 × 3 depthwise convolution.
- A GhostNet block stacks two Ghost modules in an inverted bottleneck, using one for expansion and the other for channel reduction.
- Windowed or patch-based attention restricts computation spatially, but tensor reshaping and transposing can still increase latency in lightweight models.
- Adding self-attention to GhostNet increased practical mobile inference time by 2× despite adding only about 20% theoretical FLOPs.
4 Approach
GhostNetV2 introduces DFC attention to add long-range spatial modeling to lightweight CNNs without relying on deployment-unfriendly self-attention. It applies this mechanism to GhostNet’s expanded features, combining local and long-range information while reducing attention overhead through factorization and downsampling.
- DFC Attention: DFC attention decomposes fully connected transformations into horizontal and vertical operations, producing a global receptive field with reduced complexity.Sequential aggregation along the two spatial directions captures long-range dependence; the attention complexity is O(H^2W + HW^2).
- DFC Attention: DFC attention can be implemented with convolutions that avoid tensor reshaping and transposing operations, supporting efficient deployment on mobile hardware.Sequential depth-wise convolutions use kernels of 1 × KH and KW × 1, with theoretical complexity O(KHHW + KWHW), and are supported by TFLite and ONNX.
- Enhancing Ghost Module: The Ghost module and DFC branch operate in parallel, and their outputs are combined by element-wise multiplication to aggregate complementary information.The Ghost branch extracts feature information, while DFC attention generates a normalized attention map from a transformed input feature.
- Feature Downsampling: Downsampling the DFC branch to half height and width reduces its FLOPs by 75%, after which the attention map is upsampled to match the Ghost features.Average pooling and bilinear interpolation provide the default downsampling and upsampling operations; sigmoid is also applied on downsampled features for practical speed.
- GhostV2 Bottleneck: GhostNetV2 enhances the expanded features of the first Ghost module because this improves expressiveness more effectively than enhancing the output features.The enhanced features are then passed to the second Ghost module, preserving the inverted bottleneck structure while adding long-range pixel dependence.
- Experiments: GhostNetV2 is evaluated against lightweight models using ImageNet classification accuracy, parameter count, and FLOPs.Table 2 defines the comparison axes for the reported lightweight-model results.
5 Experiments
Experiments evaluate GhostNetV2 across classification, detection, segmentation, practical latency, and ablations. Results show improved accuracy and speed trade-offs, while ablations identify how DFC attention placement, scale, kernel size, and target features affect performance.
- Image Classification on ImageNet: 75.3% top-1 accuracy at 167M FLOPs surpasses GhostNet V1’s 74.5% at similar computational cost.
- Practical Inference Speed: 75.3% top-1 accuracy at similar 37 ms inference latency exceeds GhostNet V1’s 74.5%.Latency was measured on an ARM-based mobile phone using TFLite.
- Object Detection on COCO: 22.3% mAP at 340M FLOPs with 320×320 inputs exceeds GhostNet V1 by 0.5 mAP on MS COCO.
- Experiments with Other Models: DFC attention improves MobileNetV2 top-1 accuracy by 2.4%, exceeding CA’s 1.5% improvement.
- The Impact of Kernel Size in DFC Attention: 74.8% results from 1×3 and 3×1 kernels, while increasing kernel size to capture longer-range information significantly improves performance.
- The Location for Implementing DFC Attention: Applying DFC attention at any stage improves performance, while exhaustive location search could further adjust the accuracy–computational-cost trade-off.The default configuration applies DFC attention to all layers.
- The Impact of Scaling Function: Sigmoid and hard sigmoid improve performance, but applying them to large feature maps incurs long latency; placing scaling before up-sampling gives similar accuracy more efficiently.The default uses sigmoid before up-sampling.
- Enhancing Expressiveness or Capacity: Enhancing expanded features yields a 1.4% top-1 accuracy improvement at similar computational cost, exceeding enhancement of output features.Enhancing both feature types improves performance further but increases computational cost.
6 Conclusion
The paper proposes hardware-friendly DFC attention and the GhostNetV2 architecture for mobile applications. Extensive benchmark experiments support improved accuracy–speed trade-offs by capturing long-range spatial dependencies efficiently.
- DFC attention uses horizontal and vertical fully connected layers to capture long-range spatial dependencies with large directional receptive fields.
- GhostNetV2 combines computation-efficient, deployment-simple modules to improve the trade-off between accuracy and speed.
- Experiments on benchmark datasets including ImageNet and MS COCO validate GhostNetV2’s superiority.