Source-linked AI summary
EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications
Muhammad Maaz, Abdelrahman Shaker, Hisham Cholakkal, Salman Khan, Syed Waqas Zamir, Rao Muhammad Anwer, Fahad Shahbaz Khan
TL;DR
EdgeNeXt addresses the difficulty of deploying accurate vision models on resource-constrained edge devices. It combines convolutional and efficient self-attention encoders, achieving favorable accuracy–compute trade-offs across classification, detection, and segmentation tasks, including 79.4% top-1 ImageNet-1K accuracy with 5.6M parameters.
Problem
Accurate CNN and ViT models often overlook the computational efficiency needed for low-latency operation on resource-constrained mobile platforms.
Method
EdgeNeXt combines adaptive depth-wise convolution with split depth-wise transpose attention that applies self-attention across feature channels.
Results
EdgeNeXt delivers favorable accuracy–compute trade-offs across classification, detection, and segmentation, including 79.4% top-1 ImageNet-1K accuracy with 5.6M parameters and 1.3G MAdds.
Takeaways & Limitations
The proposed hybrid design provides an efficient way to jointly model local and global information for mobile vision tasks.
Abstract
from arXiv · showhide
In the pursuit of achieving ever-increasing accuracy, large and complex neural networks are usually developed. Such models demand high computational resources and therefore cannot be deployed on edge devices. It is of great interest to build resource-efficient general purpose networks due to their usefulness in several application areas. In this work, we strive to effectively combine the strengths of both CNN and Transformer models and propose a new efficient hybrid architecture EdgeNeXt. Specifically in EdgeNeXt, we introduce split depth-wise transpose attention (STDA) encoder that splits input tensors into multiple channel groups and utilizes depth-wise convolution along with self-attention across channel dimensions to implicitly increase the receptive field and encode multi-scale features. Our extensive experiments on classification, detection and segmentation tasks, reveal the merits of the proposed approach, outperforming state-of-the-art methods with comparatively lower compute requirements. Our EdgeNeXt model with 1.3M parameters achieves 71.2% top-1 accuracy on ImageNet-1K, outperforming MobileViT with an absolute gain of 2.2% with 28% reduction in FLOPs. Further, our EdgeNeXt model with 5.6M parameters achieves 79.4% top-1 accuracy on ImageNet-1K. The code and models are available at https://t.ly/_Vu9.
1 Introduction
EdgeNeXt addresses the challenge of achieving accurate, low-latency vision models on resource-constrained mobile platforms by combining CNN and Transformer strengths. It reports favorable accuracy–compute trade-offs across mobile vision comparisons.
- Resource-constrained applications require recognition that is both accurate and low latency.
- Existing lightweight CNNs efficiently encode local details but do not explicitly model global pixel interactions.
- Self-attention enables global interaction but typically introduces slow inference and substantial computation for mobile deployment.
- The figure compares MAdd operations, top-1 ImageNet-1K accuracy, and parameter counts to assess compute–accuracy trade-offs.
- EdgeNeXt achieves 79.4% top-1 ImageNet-1K accuracy with 5.6M parameters and 1.3G MAdds, requiring 35% less MAdds than MobileViT.
2 Related Work
Prior mobile vision models mainly optimize CNN efficiency or combine CNNs and Transformers, but lightweight designs continue to face a global-interaction or self-attention bottleneck. EdgeNeXt targets this trade-off with efficient channel-wise attention and depth-wise convolutions.
- Lightweight CNN families and hardware-aware NAS improve mobile speed–accuracy trade-offs but lack global interaction between pixels.
- Vision Transformers achieve competitive visual recognition results but are difficult to deploy on edge devices because multi-headed self-attention is computationally costly.
- MobileViT combines CNN and Transformer advantages, yet its multi-headed attention remains an efficiency bottleneck with quadratic complexity in input size.
- EdgeNeXt combines depth-wise convolutions with transpose attention to obtain an accuracy–speed trade-off for mobile vision tasks.
3 EdgeNeXt
EdgeNeXt is a hierarchical hybrid architecture built from adaptive convolutional and split depth-wise transpose-attention encoders. Its design mixes multi-scale spatial processing with efficient channel-wise global representation learning.
- SDTA Encoder: Transpose attention computes channel-wise cross-covariance rather than spatial dot products, producing C×C attention with linear complexity in the input spatial dimension.
- Overall Architecture: The first stage downsamples images to 1/4 resolution with 4 × 4 strided convolution, while later stages use 2 × 2 strided convolutions.
- Overall Architecture: EdgeNeXt uses a stage-wise architecture with four hierarchical feature scales and convolutional encoders paired with SDTA encoders.
- Convolution Encoder: The convolution encoder uses adaptive depth-wise kernels for spatial mixing, pointwise convolutions for channel mixing, normalization, GELU, and a skip connection.
- SDTA Encoder: The SDTA encoder splits H×W×C inputs into subsets, applies hierarchical depth-wise convolutions, and receives features from previous splits.
- SDTA Encoder: After attention, two 1×1 convolutions, layer normalization, and GELU generate nonlinear output features.
4 Experiments
EdgeNeXt is evaluated on ImageNet-1K classification, COCO detection, and Pascal VOC segmentation, with comparisons emphasizing accuracy, compute, and latency across model variants.
- Evaluation setup: ImageNet-1K, COCO, and Pascal VOC provide the classification, detection, and segmentation evaluation benchmarks.The reported metrics are top-1 accuracy, mAP at IoU 0.50–0.95, and mIoU, respectively.
- Image classification: EdgeNeXt achieves a better accuracy-versus-compute trade-off than lightweight convolutional, transformer-based, and hybrid models on ImageNet-1K.The comparison considers parameter count and MAdds.
- Image classification: 71.2% top-1 accuracy is achieved by EdgeNeXt-XXS with 1.3M parameters, while outperforming MobileViT variants with fewer MAdds and faster edge-device inference.The model-size comparison covers S, XS, and XXS variants.
- Additional classification experiments: Knowledge-distilled EdgeNeXt-S reaches 81.1% top-1 ImageNet accuracy, while EdgeNeXt-B uses ImageNet-21K-P pretraining for large-scale evaluation.The EdgeNeXt-B comparison reports accuracy-versus-compute trade-offs against ConvNeXt and MobileViT-V2.
- Inference on edge devices: EdgeNeXt-XXS is approximately 8% faster than MobileViT-XXS on Jetson Nano and approximately 34% faster on an A100 GPU.All models are converted to TensorRT engines and evaluated in FP16 mode for the Jetson Nano comparison.
5 Ablations
Ablations identify SDTA encoders, adaptive kernels, branching, positional encoding, and stage placement as important design choices, while activation and normalization trade accuracy for latency.
- SDTA encoder and adaptive kernels: Replacing SDTA encoders with convolution encoders reduces accuracy by 1.1%, while fixing every kernel size to 7 reduces accuracy by 0.4%.These results support SDTA encoders and adaptive kernel sizes in the proposed design.
- SDTA components: Removing adaptive branching or positional encoding slightly decreases accuracy, indicating that both components contribute to the SDTA module.The ablation is reported in Table 7.
- Hybrid design: Using one SDTA encoder as the last block in the last three stages provides the optimal accuracy-latency trade-off.Progressively adding SDTA encoders in the last three stages improves accuracy with some inference-latency loss.
- SDTA placement: Placing the global SDTA encoder at the end of each stage is generally more beneficial than placing it at the start.The authors report this placement result in Table 10.
- Activation and normalization: Using Hard Swish and batch normalization instead of GELU and layer normalization greatly reduces latency while slightly lowering accuracy.The activation and normalization choice is presented as a speed-accuracy trade-off.
6 Qualitative Results
Qualitative results show EdgeNeXt detection and segmentation models operating across objects presented in various views.
- Qualitative evaluation: EdgeNeXt detects and segments objects in various views in qualitative detection and segmentation results.The results are shown in Figures 4 and 5.
7 Conclusion
EdgeNeXt combines convolution with efficient self-attention to model local and global information while maintaining low computational cost. Experiments show promising performance and generalization across classification, detection, and segmentation tasks.
- EdgeNeXt combines convolution and efficient self-attention encoders to jointly model local and global information with efficient parameters and MAdds.The hybrid design targets vision tasks where transformer self-attention creates substantial computational overhead.
- EdgeNeXt detection results qualitatively show effective object localization and classification across diverse COCO validation scenes.The detection model is trained on COCO with 80 detection classes.
- EdgeNeXt segmentation results qualitatively provide high-quality masks on unseen COCO images after training on Pascal VOC.The segmentation setup uses 20 Pascal VOC classes and visualizes predicted masks, overlays, and class color encodings.
- Experimental results demonstrate promising performance for different EdgeNeXt variants, indicating effectiveness and generalization ability.The reported evaluation spans image classification, object detection, and semantic segmentation tasks.