PaperlayerView exact revision on arXiv

Source-linked AI summary

SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications

Abdelrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang, Fahad Shahbaz Khan

arXiv:2303.15446v2cs.CV

TL;DR

Self-attention’s quadratic complexity and matrix multiplications hinder real-time vision on resource-constrained mobile devices. SwiftFormer introduces efficient additive attention using linear operations and deploys it throughout the network, achieving state-of-the-art accuracy and mobile speed, including 78.5% ImageNet-1K accuracy at 0.8 ms on iPhone 14.

  • Problem

    Quadratic self-attention and expensive matrix multiplications limit transformer-based vision models for real-time deployment on resource-constrained mobile devices.

  • Method

    Efficient additive attention replaces matrix multiplication with element-wise multiplications and replaces explicit key-value interaction with query-key interactions followed by a linear transformation.

  • Results

    78.5% top-1 ImageNet-1K accuracy at 0.8 ms latency on iPhone 14 is achieved by SwiftFormer’s small model, while the models show state-of-the-art results across image classification, object detection, and segmentation.

  • Takeaways & Limitations

    The linear attention formulation enables efficient contextual modeling at all network stages and supports SwiftFormer models with strong accuracy-latency performance on mobile devices.

  • Takeaways & Limitations

    The discussed prior attention approximations can remain quadratic in the feature dimension, and dot-product operations may still be used between query and key matrices.

Abstract

from arXiv · show

Self-attention has become a defacto choice for capturing global context in various vision applications. However, its quadratic computational complexity with respect to image resolution limits its use in real-time applications, especially for deployment on resource-constrained mobile devices. Although hybrid approaches have been proposed to combine the advantages of convolutions and self-attention for a better speed-accuracy trade-off, the expensive matrix multiplication operations in self-attention remain a bottleneck. In this work, we introduce a novel efficient additive attention mechanism that effectively replaces the quadratic matrix multiplication operations with linear element-wise multiplications. Our design shows that the key-value interaction can be replaced with a linear layer without sacrificing any accuracy. Unlike previous state-of-the-art methods, our efficient formulation of self-attention enables its usage at all stages of the network. Using our proposed efficient additive attention, we build a series of models called "SwiftFormer" which achieves state-of-the-art performance in terms of both accuracy and mobile inference speed. Our small variant achieves 78.5% top-1 ImageNet-1K accuracy with only 0.8 ms latency on iPhone 14, which is more accurate and 2x faster compared to MobileViT-v2. Code: https://github.com/Amshaker/SwiftFormer

1. Introduction

Transformer-based vision models capture global context but are difficult to deploy on resource-constrained mobile devices because self-attention scales quadratically with image resolution. SwiftFormer addresses this bottleneck with efficient additive attention and achieves strong accuracy-latency results.

  • Quadratic self-attention complexity makes vision transformers impractical for low-powered mobile devices, while CNNs trade global interaction for computational efficiency.
  • Hybrid approaches place lightweight CNN modules in early high-resolution stages and self-attention in later low-resolution stages to balance speed and accuracy.
  • Efficient additive attention removes expensive matrix multiplications, uses query-key interactions followed by a linear transformation, and can operate at every network stage.
  • 78.5% top-1 ImageNet-1K accuracy and 0.8 ms iPhone 14 latency are achieved by SwiftFormer’s small model.
  • 83.0% accuracy and 1.9 ms latency are achieved by SwiftFormer’s large model, with the series reported to outperform MobileViT-v2 and EfficientFormer on accuracy-latency trade-offs.

2. Related Work

Prior efficient vision models reduce self-attention complexity through architectural approximations, token reduction, sparse or local attention, and separable operations. However, theoretical complexity reductions do not always translate into faster mobile inference.

  • Efficient CNNs: Efficient CNNs use techniques such as depth-wise separable convolutions, inverted residual blocks, pruning, low bit-width, and neural architecture search for mobile deployment.
  • Efficient CNNs: CNN-based methods are efficient on edge devices but rely on spatially local processing and lack global feature interaction.
  • Efficient Transformers: Efficient transformer methods reduce self-attention cost using sparse attention, downsampled keys and values, transposed attention, hashing, or low-rank factorization.
  • Efficient Transformers: Reformer reduces complexity from O(n2) to O(n log n), while LinFormer reduces it from O(n2) to O(n) through low-rank factorization.
  • Efficient Transformers: Despite theoretical reductions, some efficient attention methods remain inadequate for mobile inference speed; EfficientFormer therefore uses MHSA only in its last stage, whereas MobileViT-v2 uses separable self-attention.

3. Method

SwiftFormer combines efficient global context modeling with convolutional local representations throughout a hierarchical mobile vision architecture. Its efficient additive attention replaces costly attention interactions with linear operations, enabling use across all stages.

  • Attention Modules: Standard self-attention has O(n^2 · d) complexity because token-pair interactions and value aggregation use dot-product matrix multiplications.This quadratic scaling increases computational and memory demands as the number of tokens grows.
  • Attention Modules: Transpose self-attention reduces token-length scaling to O(n · d^2), but remains quadratic in feature dimension and still uses query-key dot products.It applies attention across channel dimensions rather than spatial dimensions.
  • Efficient Additive Attention: The proposed attention learns a global query by weighting and pooling Q, then element-wise multiplies the broadcasted query with K to form global context.The resulting context captures information from every token and is followed by a linear transformation.
  • Efficient Additive Attention: Efficient additive attention removes explicit key-value interactions, using query-key interactions and a linear transformation to compute global context.The design replaces expensive matrix multiplications with element-wise multiplications and a linear layer.
  • SwiftFormer Architecture: SwiftFormer uses Conv. Encoder blocks followed by SwiftFormer Encoder blocks at all four hierarchical stages, combining local convolutional and local-global representations.Stages operate at scales 1/4, 1/8, 1/16, and 1/32, with downsampling between consecutive stages.
  • SwiftFormer Architecture: SwiftFormer models are built without neural architecture search while targeting a better speed-accuracy trade-off than prior mobile vision designs.The architecture is based on EfficientFormer but changes token mixing and extends contextual modeling beyond the final stage.

4. Experiments

SwiftFormer is evaluated on ImageNet-1K, COCO, and ADE20K using mobile latency and task-specific accuracy metrics. Across classification and dense prediction, the models outperform prior lightweight backbones and achieve favorable accuracy-latency trade-offs.

  • Image classification: 79.2% top-1 accuracy at 1.1 ms latency is achieved by the EfficientFormer-L1 baseline on iPhone 14.Replacing its pool mixers with the proposed Conv. Encoder raises accuracy to 79.9% while maintaining the same latency.
  • Image classification: 1.6× faster and 1.4% higher top-1 accuracy are achieved by SwiftFormer-S compared with EfficientNet-b0.SwiftFormer-L3 also gains 4.5% over ResNet-50 and 0.9% over ConvNeXt-T in top-1 accuracy, with the reported latency comparisons.
  • Object detection and instance segmentation: 41.2 AP box and 38.1 AP mask are achieved by SwiftFormer-L1 for object detection and instance segmentation on COCO.These results surpass EfficientFormer-L1 by 3.3 AP box and the previous state of the art by 2.7 AP mask, respectively.
  • Qualitative results: Qualitative examples show SwiftFormer-L1 detecting and segmenting instances on COCO and segmenting indoor and outdoor scenes on ADE20K.The visualizations compare predicted outputs with the validation images or ground-truth masks described in the figure captions.
  • Semantic segmentation: 43.9 mIoU is achieved by the SwiftFormer-L3 semantic segmentation model on ADE20K, surpassing all previous methods.SwiftFormer-L1 achieves 41.4% mIoU, exceeding ResNet18, PoolFormer-S12, and EfficientFormer-L1.

5. Conclusion

SwiftFormer addresses the computational cost of self-attention in mobile vision by replacing expensive matrix multiplications with a linear formulation usable across network stages. It reports state-of-the-art results across classification, detection, and segmentation benchmarks, including qualitative advantages over EfficientFormer-L1 on COCO.

  • SwiftFormer replaces expensive self-attention matrix multiplications with an efficient additive formulation that is linear in the input tokens.The design also removes explicit key-value interaction from global-context computation.
  • SwiftFormer-L1 accurately detects and segments the example COCO objects that EfficientFormer-L1 misclassifies.The comparison covers dense prediction tasks including detection and instance segmentation.
  • The proposed attention can be used at all stages of the network, unlike previous efficient attention methods.

A. Architecture Details of SwiftFormer

SwiftFormer provides four model variants with stage-wise architectural specifications reported in Table 4. The architectures use a fixed convolutional encoder expansion ratio and manually selected widths and depths rather than neural architecture search.

  • Table 4 reports the resolution, channel count, and repeated-block count for SwiftFormer-XS, SwiftFormer-S, SwiftFormer-L1, and SwiftFormer-L3.
  • All SwiftFormer variants use an expansion ratio of 4 in the convolutional encoder.
  • The architectures are not built with neural architecture search, and their channels and blocks are selected for similar model size.

B. Additional Implementation Details

The classification experiments use a standardized 224×224 training resolution and large-scale multi-GPU training. Data augmentation is applied to improve model robustness.

  • 224×224 is used for training and reporting SwiftFormer accuracy to ensure fair comparison with baselines and previous methods.
  • Training uses a batch size of 2048 on eight A100 GPUs, with classification experiments averaging 36 hours of training.
  • Color jitter with a ratio of 0.4 and RandAugment are applied as data augmentations during training.The augmentations are used to enhance model robustness.

C. Additional Ablations

Additional ablations examine the effect of simplifying QKV interactions, while COCO error analysis compares SwiftFormer-L1 with EfficientFormer-L1 across object sizes and evaluation configurations.

  • QKV Interaction Ablation: 10% lower latency and 0.4% higher top-1 accuracy result when key-value interactions are replaced with a simple linear transformation.
  • QKV Interaction Ablation: The ablation results support simplifying QKV interactions within the efficient additive attention mechanism.
  • COCO Error Analysis: COCO error analysis compares precision-recall curves for all objects and large-sized objects across multiple evaluation configurations.The legend reports the area under each curve.
  • COCO Error Analysis: SwiftFormer-L1 provides consistent improvements over EfficientFormer-L1 in the COCO error analysis.

D. Error Analysis on COCO Dataset

SwiftFormer-L1 outperforms EfficientFormer-L1 in error analysis, with a particularly strong advantage for large objects.

  • SwiftFormer-L1 achieves better error-analysis results than the EfficientFormer-L1 baseline, especially for large-sized objects.The comparison covers all objects and large-sized objects using areas under the plotted curves.
  • 0.558 overall AP at IoU=0.75 for large objects rises to 0.793 with perfect localization in EfficientFormer-L1.The gap indicates the baseline has substantial localization-related headroom for large-sized objects.
  • Excluding background false positives is part of the error analysis for comparing the two models.

E. Qualitative Results

SwiftFormer provides qualitative detection and segmentation results across diverse scenes, while its architecture varies by resolution, channels, blocks, computation, and parameters.

  • Qualitative Results: SwiftFormer accurately localizes and segments objects in diverse scenes for instance segmentation and detection.
  • Architectures: SwiftFormer variants differ in output resolution, output channels C, block count N, GMACs, and parameter count.Downsampling between consecutive stages doubles the channel count and halves the resolution.

F. Discussion

Vision-transformer spatial mechanisms can create resolution sensitivity or inference overhead in dense prediction and mobile settings. SwiftFormer addresses these concerns by omitting positional encoding and attention biases while reporting downstream-task results.

  • Discussion: Attention bias can make dense-prediction models fragile because it is sensitive to input resolution.
  • Discussion: Typical positional encoding can slow inference on resource-constrained devices.
  • Discussion: SwiftFormer omits positional encoding and attention biases in its efficient additive attention mechanism to support fast inference.The models also show promising results in downstream tasks.
Loading 2303.15446v2…