Source-linked AI summary
EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers
Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, Brais Martinez
TL;DR
Mobile ViTs remain demanding for mobile devices and trail leading lightweight CNNs, motivating more efficient attention-based models. EdgeViTs address this gap with a local-global-local bottleneck and direct on-device evaluation, achieving competitive accuracy-efficiency tradeoffs and Pareto-optimal latency and energy results.
Problem
ViTs' quadratic attention makes them computationally demanding, while prior lightweight ViTs remain insufficient for mobile-device resource requirements and trail leading lightweight CNNs.
Method
EdgeViTs factorize self-attention into local aggregation, sparse delegate-token global attention, and local propagation using convolutions and standard optimized operations.
Results
Across three visual tasks, EdgeViTs match or surpass state-of-the-art lightweight CNNs, consistently outperform MobileViTs, and are Pareto-optimal for latency and energy tradeoffs.
Takeaways & Limitations
Direct latency and energy measurements show that lightweight attention-based models can compete with efficient CNNs in practical mobile-device tradeoffs.
Takeaways & Limitations
FLOPs and parameter counts are indirect efficiency proxies because device latency and energy depend on memory access, parallelism, and platform characteristics.
Abstract
from arXiv · showhide
Self-attention based models such as vision transformers (ViTs) have emerged as a very competitive architecture alternative to convolutional neural networks (CNNs) in computer vision. Despite increasingly stronger variants with ever-higher recognition accuracies, due to the quadratic complexity of self-attention, existing ViTs are typically demanding in computation and model size. Although several successful design choices (e.g., the convolutions and hierarchical multi-stage structure) of prior CNNs have been reintroduced into recent ViTs, they are still not sufficient to meet the limited resource requirements of mobile devices. This motivates a very recent attempt to develop light ViTs based on the state-of-the-art MobileNet-v2, but still leaves a performance gap behind. In this work, pushing further along this under-studied direction we introduce EdgeViTs, a new family of light-weight ViTs that, for the first time, enable attention-based vision models to compete with the best light-weight CNNs in the tradeoff between accuracy and on-device efficiency. This is realized by introducing a highly cost-effective local-global-local (LGL) information exchange bottleneck based on optimal integration of self-attention and convolutions. For device-dedicated evaluation, rather than relying on inaccurate proxies like the number of FLOPs or parameters, we adopt a practical approach of focusing directly on on-device latency and, for the first time, energy efficiency. Specifically, we show that our models are Pareto-optimal when both accuracy-latency and accuracy-energy trade-offs are considered, achieving strict dominance over other ViTs in almost all cases and competing with the most efficient CNNs. Code is available at https://github.com/saic-fi/edgevit.
1 Introduction
EdgeViTs target the gap between accurate but costly ViTs and efficient CNNs by evaluating lightweight vision models using practical on-device efficiency measures. The proposed family combines self-attention and convolutions to compete with lightweight CNNs in accuracy-efficiency tradeoffs.
- Motivation: ViTs' quadratic self-attention complexity makes computational cost important for mobile-oriented design.Prior efficient designs include hierarchical stages, locally grouped attention, and pooled attention.
- Practical requirements: FLOPs do not directly translate into device latency or energy because memory access, parallelism, and platform characteristics affect execution.The paper therefore treats latency and energy as direct efficiency metrics.
- Contribution: EdgeViTs use a lightweight local-global-local bottleneck combining depth-wise convolutions, sparse delegate-token attention, and transposed convolutions.The three operations aggregate local information, exchange long-range information, and propagate global context.
- Contribution: EdgeViTs are designed to compete with lightweight CNNs in accuracy-efficiency tradeoffs on mobile devices.The contribution is framed around practical deployment rather than proxy metrics alone.
- Results: Across three visual tasks, EdgeViTs match or surpass lightweight CNNs and consistently outperform recent MobileViTs in accuracy-efficiency tradeoffs.The evaluation includes on-device energy efficiency and reports Pareto-optimality across latency and energy tradeoffs.
2 Related Work
Prior work improves CNN and ViT efficiency through architectural changes, attention reduction, token reduction, and convolution-attention hybrids. MobileViTs specifically target mobile devices, but remain inferior to leading lightweight CNNs in on-device accuracy-efficiency tradeoffs.
- Efficient CNNs: Efficient CNN research includes depthwise separable convolutions, automated architecture design, pruning, dynamic computing, and low-bit-width techniques.These approaches target efficient architectures or alternative accuracy-efficiency tradeoffs for on-device deployment.
- Vision transformers: Efficient ViTs adopt hierarchical designs, relative or learnable positional mechanisms, and alternatives to standard multi-head self-attention.These changes address data efficiency, architecture design, and attention overhead.
- Efficient attention: Some ViTs reduce computation by down-sampling keys and values, while others dynamically prune or pool tokens during inference.Both strategies reduce the tokens involved in later computation.
- Hybrid designs: Convolution-attention hybrids combine convolutional designs and self-attention in settings ranging from convolutional stems to integrated operations.The related approaches explore complementarity between local convolutional processing and attention.
- Mobile ViTs: MobileViTs are dedicated to mobile devices but remain clearly inferior to leading lightweight CNNs in on-device accuracy-efficiency tradeoffs.This motivates further development of lightweight ViTs for mobile and edge deployment.
3 EdgeViTs
EdgeViTs replace full spatial self-attention with a four-stage architecture whose LGL blocks exchange local and global information through efficient operations. The design uses sparse delegate tokens for long-range attention while preserving interactions across the feature map.
- Overview: EdgeViTs use a hierarchical pyramid with progressively reduced spatial resolution and expanded channel dimensions across four stages.Stages contain transformer-based blocks operating on tensors of the same shape.
- Overview: Each LGL block combines local aggregation, sparse self-attention, local propagation, patch embedding, and a feed-forward network.The architecture overview identifies these components within the staged EdgeViT design.
- Local-Global-Local bottleneck: The LGL bottleneck enables information exchange between any token pair at lower compute cost than applying attention at every spatial location.The design exploits local redundancy while retaining global and local information flows.
- Local-Global-Local bottleneck: Local aggregation uses depth-wise and point-wise convolutions, while local propagation uses transposed convolutions to diffuse delegate information.These operations implement the local stages of the bottleneck with standard primitives.
- Local-Global-Local bottleneck: The global attention step samples one evenly distributed delegate token from each r×r window and attends only over those selected tokens.This differs from prior ViTs that involve all spatial tokens as queries.
- Local-Global-Local bottleneck: LGL first aggregates neighboring-token information, then applies global sparse attention among selected delegates, and finally propagates context to neighboring non-delegates.This sequence provides local-global-local information exchange.
- Implementation: LGL operators are implementation friendly because they use commonly available, highly optimized deep-learning operations.The prose identifies layer normalization, local aggregation, feed-forward networks, sparse attention, and local propagation as the main operators.
- Model family: EdgeViTs are built at 0.5G, 1G, and 2G computational complexities using four stages with progressively reduced spatial resolution.Stage down-sampling uses stride-2 convolutions, with a ×4 reduction in the first stage.
4 Experiments
Experiments evaluate EdgeViTs on ImageNet-1K, on-device latency and energy, ablations, and downstream detection, segmentation, and instance-segmentation tasks. Across these evaluations, EdgeViTs improve over lightweight ViTs and approach or compete with efficient CNNs under practical efficiency measures.
- ImageNet-1K classification: 3.9% and 2.3% improvements over PVT-v2-B0 and PVT-v2-B1 are achieved by EdgeViT-XXS and EdgeViT-S on ImageNet-1K, respectively.At similar GFLOPs complexity, EdgeViTs outperform other lightweight ViTs, while efficient CNNs retain a small accuracy-FLOPs advantage.
- On-device efficiency: 2.4% higher accuracy than MobileNet-V2 is achieved by EdgeViT-XXS while running slightly faster, whereas its 1.1% gain over MobileNet-V3 costs 9.8ms.EdgeViT-XS performs on par with EfficientNet-B0; these comparisons use direct latency rather than FLOPs or parameter counts.
- Energy efficiency: EdgeViTs dominate almost all other ViTs under the accuracy-energy Pareto comparison, with LeViT-384† the only stated exception.They compete with MobileNet-v3 and EfficientNet-B0, which are more efficient but less accurate, while MobileNet-v2 is dominated by its alternatives.
- Ablation study: 0.5% top-1 accuracy is gained by adding local propagation at only 0.4ms overhead, while convolutional local aggregation outperforms locally grouped self-attention.Ablations use EdgeViT-XXS alternatives rescaled to approximately 0.5GFLOPs for fair comparison.
- Downstream dense prediction: 1.5 AP, 1.7 APb, and 0.7 APm gains over PVTv2-B0 are obtained by EdgeViT-XXS on RetinaNet and Mask R-CNN, while EdgeViT-S gains 2.2, 3.0 APb, and 1.2 APm over PVTv2-B1.On ADE20K semantic segmentation, EdgeViT-XXS and EdgeViT-S exceed PVTv2-B0 and PVTv2-B1 by 2.5% and 3.4% mIoU, respectively.
5 Conclusion
The paper presents EdgeViTs as efficient ViTs designed for on-device deployment, with accuracy-efficiency trade-offs comparable to or better than efficient CNNs and ViTs.
- EdgeViTs achieve comparable or superior accuracy-efficiency trade-offs to state-of-the-art efficient CNNs and ViTs on generic visual recognition tasks.The evaluation uses on-device experiments emphasizing latency and energy-aware efficiency.
A.1 Computing Complexity
The LGL bottleneck reduces spatial-context computation by combining local aggregation, sparse global attention, and local propagation, avoiding the full quadratic cost of standard self-attention.
- The LGL bottleneck combines local aggregation, Sparse Global Self-Attention, and local propagation for spatial context modeling.Its total cost is the sum of these three components.
- O(k2hwc + h2w2/r4 c + r2hwc) is the total LGL cost, compared with O(h2w2c) for standard multi-head self-attention.The expressions describe spatial-context computation while omitting point-wise operations.
- The LGL bottleneck significantly reduces computation when k ≪ h, w and r > 1.The experiments use k = 3 and stage-wise subsampling rates r = (4,2,2,1).
A.2 Implementation Details
EdgeViTs are assembled from interchangeable local, global, and propagation components, with ablations replacing or removing these blocks under a matched-complexity setting.
- EdgeViT variants are built from the proposed component blocks according to the model schematic and configuration table.The implementation details describe ablations that replace or remove individual blocks.
- Removing local aggregation replaces separate local-global modeling with PVT spatial-reduced attention, while LA(LSA) substitutes local-grouped self-attention.Both variants test alternatives to the default local-context design.
- The default Center sampler is ablated with AvgPool2d() and MaxPool2d(), using kernel sizes equal to the sample rate.These variants replace the sampler with average or maximum pooling.
- Propagation ablations replace GlobalSparseAttn with PVT spatial-reduced attention or instantiate LocalProp as bilinear interpolation.The LocalAgg component is retained in the w/o LP variant.
- 0.5 GFLOPs is the matched-complexity target used by down-scaling the number of layers across variants.This normalization supports fair comparison among the ablation models.
A.3 Accuracy-Speed Pareto-Optimal Models
EdgeViTs are Pareto-optimal across accuracy-latency and accuracy-energy trade-offs, with evaluations conducted on specified mobile hardware.
- EdgeViTs are Pareto-optimal in both accuracy-latency and accuracy-energy trade-offs.The paper reports that they are comparable with efficient CNNs and significantly dominate prior ViTs in accuracy-speed comparisons.
- The ImageNet-1K trade-offs are measured on a Samsung Galaxy S21 for latency and a Snapdragon 888 Hardware Development Kit for energy.The figure highlights all three EdgeViT variants as Pareto-optimal.
A.4 Efficiency in detection/segmentation
The work evaluates EdgeViT backbones for detection and segmentation using inference time, energy, and efficiency, reporting higher efficiency than baselines. The evaluation covers 800 × 800 detection inputs and 512 × 512 segmentation inputs.
- A.4 Efficiency in detection/segmentation: EdgeViTs are evaluated as backbones for detection and segmentation using inference time, energy, and efficiency.The evaluation measures backbone efficiency rather than task accuracy.
- A.4 Efficiency in detection/segmentation: EdgeViTs demonstrate higher efficiency than the baselines in the detection and segmentation evaluations.The comparison is reported in Tables 7 and 8.
- A.4 Efficiency in detection/segmentation: Detection efficiency is reported at an input size of 800 × 800.
- A.4 Efficiency in detection/segmentation: Segmentation efficiency is reported at an input size of 512 × 512.