Source-linked AI summary
Rethinking Mobile Block for Efficient Attention-based Models
Jiangning Zhang, Xiangtai Li, Jian Li, Liang Liu, Zhucun Xue, Boshen Zhang, Zhengkai Jiang, Tianxin Huang, Yabiao Wang, Chengjie Wang
TL;DR
Lightweight vision models lack a recognized IRB-like infrastructure for attention-based designs, while CNNs face accuracy limits and attention models face high computation. The paper unifies IRB with Transformer components into MMB, derives iRMB, and builds EMO using only iRMBs. Across classification and dense-prediction benchmarks, EMO achieves competitive accuracy-efficiency trade-offs, including faster mobile execution than EdgeNeXt.
Problem
Lightweight CNNs face accuracy limits, attention models incur high MHSA computation, and few methods explore simple IRB-like infrastructure for attention-based models.
Method
The paper unifies IRB, MHSA, and FFN into a one-residual Meta Mobile Block, then derives iRMB and builds EMO using only iRMBs.
Results
EMO achieves competitive results across classification and dense prediction, including 71.5, 75.1, and 78.4 Top-1 for EMO-1M/2M/5M and faster mobile execution than EdgeNeXt.
Takeaways & Limitations
A single simple iRMB-based infrastructure supports lightweight attention models across downstream tasks while balancing parameters, FLOPs, and accuracy.
Takeaways & Limitations
The authors leave more complex operators, higher-resolution inputs, NAS, distillation, larger pretraining data, and stronger training strategies for future work because of limited computational power.
Abstract
from arXiv · showhide
This paper focuses on developing modern, efficient, lightweight models for dense predictions while trading off parameters, FLOPs, and performance. Inverted Residual Block (IRB) serves as the infrastructure for lightweight CNNs, but no counterpart has been recognized by attention-based studies. This work rethinks lightweight infrastructure from efficient IRB and effective components of Transformer from a unified perspective, extending CNN-based IRB to attention-based models and abstracting a one-residual Meta Mobile Block (MMB) for lightweight model design. Following simple but effective design criterion, we deduce a modern Inverted Residual Mobile Block (iRMB) and build a ResNet-like Efficient MOdel (EMO) with only iRMB for down-stream tasks. Extensive experiments on ImageNet-1K, COCO2017, and ADE20K benchmarks demonstrate the superiority of our EMO over state-of-the-art methods, e.g., EMO-1M/2M/5M achieve 71.5, 75.1, and 78.4 Top-1 that surpass equal-order CNN-/Attention-based models, while trading-off the parameter, efficiency, and accuracy well: running 2.8-4.0x faster than EdgeNeXt on iPhone14.
1. Introduction
Lightweight vision models must balance accuracy with parameters and FLOPs, but CNNs face accuracy limits while attention models incur high computational costs. The paper proposes a unified lightweight infrastructure that extends IRB-like design to attention-based models and supports efficient downstream models.
- Motivation: Mobile models target storage- and computation-restricted applications by reducing parameters and FLOPs.Depth-wise separable convolution became a fundamental CNN component for subsequent lightweight models.
- Motivation: CNN-only models remain accuracy-limited by the natural inductive bias of static convolutions.The paper identifies advancing beyond the Inverted Residual Block as one core direction.
- Motivation: Attention models improve over CNNs through dynamic modeling but face quadratic MHSA computation, especially with large feature maps.Prior approaches reduce this cost through linear-complexity variants, lower spatial resolution, channel rearrangement, or local windows.
- Motivation: Existing efficient hybrid models improve the accuracy-efficiency trade-off but often use complex structures or multiple hybrid modules that hinder optimization.This motivates seeking an attention-based IRB-like infrastructure using only basic operators.
- Approach: The paper unifies IRB, MHSA, and FFN into a one-residual Meta Mobile Block and instantiates an iRMB containing depth-wise convolution and improved EW-MHSA.A ResNet-like EMO is then built using only iRMBs for lightweight model design.
- Results: EMO-1M/2M/5M reach 71.5, 75.1, and 78.4 Top-1 while competing with CNN- and Transformer-based models under lightweight constraints.The reported experiments also include dense-prediction results and FLOPs comparisons with MobileViTv2.
2. Methodology: Induction and Deduction
The paper abstracts a one-residual Meta Mobile Block from IRB, MHSA, and FFN, then deduces iRMB and uses it uniformly to build EMO under usability, uniformity, effectiveness, and efficiency criteria.
- Design criteria: The design criteria prioritize simple implementation, few core modules, strong classification and dense-prediction performance, and reduced parameters and calculations.These criteria correspond to usability, uniformity, effectiveness, and efficiency.
- Induction: The Meta Mobile Block unifies IRB-like structure with Transformer FFN and MHSA using expansion ratio λ and an efficient operator F.It uses channel expansion, feature enhancement through F, channel shrinkage, and one residual connection.
- Induction: Unlike two-residual MetaFormer, the Meta Mobile Block contains one sub-module and one residual connection, reducing memory access and deployment cost.The paper contrasts its efficiency-oriented induction with MetaFormer’s high-performance Transformer/MLP-oriented induction.
- Deduction: The iRMB instantiates F with efficient window MHSA and depth-wise convolution, combining dynamic long-distance interaction with efficient local-feature modeling.Its design addresses channel-expansion costs and the quadratic pixel cost of naive MHSA.
- Macro design: EMO is a ResNet-like four-phase model composed only of iRMBs, while iRMB itself uses standard convolution and multi-head self-attention and supports down-sampling through stride.MHSA is activated only in stages 3 and 4, with shutdown treated as a special case of the iRMB structure.
- Empirical design choices: The component ablation reaches 78.4 Top-1 when EW-MHSA and DW-Conv are both used, and reversing their order reduces performance by 0.6.The ablation indicates that iRMB instantiation and operator ordering affect model performance.
3. Experiments
Experiments evaluate EMO on ImageNet-1K, COCO2017, ADE20K, throughput, and ablations, showing strong accuracy-efficiency trade-offs and device-friendly speed.
- Image Classification: EMO-2M achieves 75.1 Top-1 with 439M FLOPs, while EMO-5M reaches 78.4 accuracy with 5.1M parameters and 903M FLOPs.EMO-6M reaches 79.0 Top-1 with 961M FLOPs after increasing the fourth-stage channel count.
- Object Detection: EMO-1M achieves 22.0 mAP with 0.6G FLOPs and 2.3M parameters using SSDLite, improving 2.1 points over MobileViT with 66% of its FLOPs.EMO-5M reaches 27.9 mAP using 1.8G FLOPs, 53% of MobileViT-S's 3.4G.
- Semantic Segmentation: DeepLabv3 with EMO-1M/2M/5M obtains 33.5/35.3/37.8 mIoU, surpassing MobileViTv2 by +1.6/+0.6/+0.6 points with fewer parameters and FLOPs.The same conclusion holds when EMO is used with PSPNet.
- Throughput: EMO-1M is 116% faster than EdgeNeXt-XXS on CPU and 20% faster on GPU at the same FLOPs.On iPhone14, EMO variants run 2.8×, 3.9×, and 4.80× faster than EdgeNeXt.
- Ablation and Analysis: EMO-5M is robust to drop-path rates from 0 to 0.1, with accuracy fluctuations within 0.2, while batch sizes of 512 or less degrade performance.A drop-path rate of 0.05 gives a slightly better result.
- Ablation and Analysis: DW-Conv and MHSA account for 4.6%/4.1% and 13.8%/14.6% of EMO's parameters/FLOPs, indicating a relatively balanced model.Feature-similarity analysis finds DW-Conv favors short-distance correlations, EW-MHSA adds long-distance correlations, and iRMB combines both.
4. Related Work
Related work advances efficient CNNs and hybrid models, but balancing parameters, computation, accuracy, and usability remains an open design challenge.
- Efficient CNN Models: Efficient CNNs reduce parameters and computation through techniques such as 1x1 filters, channel reduction, asymmetric convolutions, and depth-wise separable convolution.These designs established fundamental components for later lightweight CNNs.
- Efficient CNN Models: Current transformer-based efficient approaches often require elaborate complex modules, limiting model mobility and usability.The paper identifies balancing parameters, computation, and accuracy in easy-to-use mobile models as an unresolved problem.
5. Conclusion and Future Works
The paper presents MMB, iRMB, and EMO as a unified lightweight attention-based design, while identifying several directions for improving efficiency-model performance.
- Conclusion: The study unifies efficient IRB and Transformer components through Meta Mobile Block, then derives iRMB and builds EMO using only iRMB for downstream tasks.Experiments across several datasets support the approach's reported superiority.
- Future Work: The authors leave complex operators, higher-resolution inputs, NAS, distillation, larger pretraining data, and stronger training strategies for future work.These attempts were deferred because of limited computational power.