PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Mobile Block for Efficient Attention-based Models

Jiangning Zhang, Xiangtai Li, Jian Li, Liang Liu, Zhucun Xue, Boshen Zhang, Zhengkai Jiang, Tianxin Huang, Yabiao Wang, Chengjie Wang

arXiv:2301.01146v4cs.CV

TL;DR

Lightweight CNNs depend on IRB, while attention models face high MHSA costs at large feature sizes. The paper abstracts MMB, derives iRMB for EMO, and reports that EMO variants outperform MobileViTv2 in detection and segmentation.

  • Problem

    Lightweight CNNs need stronger blocks beyond IRB, while MHSA incurs high computation costs at large feature-channel counts and resolutions.

    Source: §1 · p. 1

  • Method

    The paper abstracts a unified Meta Mobile Block from IRB and Transformer modules, then derives iRMB and builds EMO using only iRMB.

    Source: §2.2 · p. 3, §2.3 · p. 4

  • Results

    EMO variants outperform MobileViTv2 in detection and segmentation; EMO-1M with DeepLabv3 obtains 33.5 mIoU.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study provides experimental findings on designing attention-based lightweight models for downstream tasks.

    Source: §5 · p. 9

    Current computational limits left additional operators, higher-resolution inputs, NAS, distillation, larger-dataset training, and stronger training strategies for future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

This paper focuses on developing modern, efficient, lightweight models for dense predictions while trading off parameters, FLOPs, and performance. Inverted Residual Block (IRB) serves as the infrastructure for lightweight CNNs, but no counterpart has been recognized by attention-based studies. This work rethinks lightweight infrastructure from efficient IRB and effective components of Transformer from a unified perspective, extending CNN-based IRB to attention-based models and abstracting a one-residual Meta Mobile Block (MMB) for lightweight model design. Following simple but effective design criterion, we deduce a modern Inverted Residual Mobile Block (iRMB) and build a ResNet-like Efficient MOdel (EMO) with only iRMB for down-stream tasks. Extensive experiments on ImageNet-1K, COCO2017, and ADE20K benchmarks demonstrate the superiority of our EMO over state-of-the-art methods, e.g., EMO-1M/2M/5M achieve 71.5, 75.1, and 78.4 Top-1 that surpass equal-order CNN-/Attention-based models, while trading-off the parameter, efficiency, and accuracy well: running 2.8-4.0x faster than EdgeNeXt on iPhone14.

1. Introduction

The paper unifies IRB and Transformer components to derive a simple attention-based mobile block and build EMO. It reports competitive ImageNet classification and downstream detection and segmentation results.

  • MobileNet’s depth-wise separable convolution [20] and MobileNetv2’s IRB [54] established lightweight CNN infrastructure, but CNN accuracy still needs improvement.Source: §1 · p. 1
  • Attention models improve over CNNs but MHSA’s quadratic computation impedes deployment, while hybrid models can add complex structures or multiple modules.The paper identifies a need for a lightweight IRB-like attention infrastructure built from basic operators.Source: §1 · pp. 1–2
  • The authors abstract a one-residual Meta Mobile Block (MMB) from IRB [54] and Transformer [66] modules, then derive iRMB and an EMO built only from iRMBs.iRMB combines depth-wise convolution and improved EW-MHSA to model short- and long-distance dependencies.Source: §1 · p. 2
  • EMO-1M/2M/5M reach 71.5/75.1/78.4 Top-1 over current SoTA CNN-/Transformer-based models.Source: §1 · p. 2
  • SSDLite-equipped EMO-1M/2M/5M score 22.0/25.2/27.9 mAP with 2.3M/3.3M/6.0M parameters and 0.6G/0.9G/1.8G FLOPs.The paper reports these scores exceed MobileViTv2 [50] by +0.8/+0.6/+0.1, with FLOPs decreased by 33%/50%/62%.Source: §1 · p. 2
  • EMO-1M/2M/5M obtain 33.5/35.3/37.98 mIoU with 5.6M/6.9M/10.3M parameters and 2.4G/3.5G/5.8G FLOPs, surpassing MobileViTv2 by +1.6↑/+0.6↑/+0.8↑ with much lower FLOPs.Source: §1 · p. 2

2. Methodology: Induction and Deduction

The methodology frames efficient mobile vision design around usability, uniformity, effectiveness, and efficiency, then derives a one-residual block that unifies lightweight CNN and Transformer components. It instantiates this design as iRMB and builds EMO from iRMB blocks alone.

2.1. Criteria for General Efficient Model

  • The design criteria are usability, uniformity, effectiveness, and efficiency; EMO aims to satisfy them without complicated operations while supporting classification and dense prediction.The paper characterizes MobileViT [50] [67] as having higher FLOPs and slightly complex modules, and EdgeNeXt [47] and EdgeViT [52] as using elaborate basic blocks.Source: §2.1 · p. 2

2.2. Meta Mobile Block

  • MMB abstracts IRB, MHSA, and FFN as a one-residual module: channel expansion, an efficient operator, and channel shrinkage are followed by a skip connection.Its expansion ratio λ and operator F allow it to instantiate different modules; unlike two-residual MetaFormer, it uses one sub-module to extend IRB for efficient model design.Source: §2.2 · p. 3

2.3. Micro Design: Inverted Residual Mobile Block

  • iRMB instantiates F as cascaded MHSA and convolution, using unexpanded X for Q and K and expanded Xe for V to reduce attention cost.The design pairs local feature modeling and Transformer-like long-distance interactions with Window-MHSA and DW-Conv; MHSA and convolution can be switched on or off independently.Source: §2.3 · p. 4

2.4. Macro Design of EMO for Dense Prediction

  • EMO uses only iRMB blocks in a ResNet-like four-phase model, with gradually increasing expansion rates and channel counts; MHSA is enabled only in stages 3 and 4.DW-Conv supports down-sampling without positional embeddings, and the module uses standard convolution and multi-head self-attention without other complex operators.Source: §2.4 · pp. 4–5
  • 78.4 Top-1 is achieved when iRMB uses both EW-MHSA and DW-Conv, the best result in the reported component ablation.The ablation also reports 73.5 Top-1 with neither component and gains of +3.1 and +4.1 when adding DW-Conv and EW-MHSA, respectively.Source: §2.4 · p. 5, Table 5 · p. 5
  • Switching DW-Conv and EW-MHSA order causes a -0.6↓ performance drop, so EMO-5M uses EW-MHSA first by default.The authors also report that parallelizing the two operators yields 78.1 (-0.3↓) Top-1 with 5.1M parameters and 964M FLOPs (+63M↑than EMO-5M), while throughput slows by about -7%↓.Source: §2.3 · p. 4, §2.4 · p. 5

3. Experiments

EMO achieves strong ImageNet-1K classification and downstream detection and segmentation results, while experiments examine its speed, design choices, and feature behavior. The results support a trade-off between accuracy and efficiency across lightweight configurations.

3.1. Image Classification

  • 71.5 Top-1 for EMO-1M beats MobileNetv3-L-0.50 by +2.7 and MobileViTv2-0.5 by +1.3; EMO-2M, EMO-5M, and EMO-6M score 75.1, 78.4, and 79.0 Top-1.EMO-5M uses 5.1M parameters and 903M FLOPs; EMO-6M uses 961M FLOPs. The paper also reports EMO-6M at 79.0 Top-1 with fewer parameters than EfficientNet-B1 (7.8M) and EfficientFormer-L1 (12.3M).Source: §3.1 · p. 5, Table 7 · p. 6, §3.3 · p. 7

3.2. Downstream Tasks

  • 22.0 mAP with 0.6G FLOPs and 2.3M parameters for SSDLite+EMO-1M is +2.1 over MobileViT; SSDLite+EMO-5M reaches 27.9 mAP with 53% of MobileViT-S FLOPs.For RetinaNet, EMO reports +2.6 AP over ResNet-50 and +1.7 AP over PVTv2-B0.Source: Table 8 · p. 6, Table 9 · p. 6, §3.2 · p. 6
  • 33.5/35.3/37.8 mIoU for DeepLabv3 with EMO-1M/2M/5M exceeds MobileViTv2 by +1.6/+0.6/+0.6, respectively.The paper reports consistent conclusions for PSPNet and more accurate, stable DeepLabv3 segmentation examples than MobileViTv2, including bathtub, sand, and baseball-field scenes.Source: §3.2 · pp. 6–7, Table 10 · p. 7

3.3. Extra Ablation and Explanatory Analysis

  • 2.8×, 3.9×, and 4.80× faster on iPhone14 than EdgeNeXt, EMO-1M also boosts GPU speed by +20% and CPU speed by +116% at similar FLOPs.Throughput comparisons use AMD EPYC 7K62 CPU and V100 GPU at 224×224 resolution and batch size 256.Source: Table 11 · p. 7, §3.3 · p. 7
  • Replacing BN with LN in all stages lowers throughput from 1,693 to 952 for only +0.2 accuracy, while limited LN use is enough to avoid unstable NaNs.The default uses LN in stages 3 and 4; the authors conclude LN is necessary for Attention-based EMO but need only be used in a few stages.Source: §3.3 · pp. 7–8, Figure 6 · p. 8
  • Applying MHSA at every stage gives 78.8 Top-1 versus 78.4 for EMO-5M with last-two-stage MHSA, but raises FLOPs from 903M to 992M; drop path 0.05 and batch sizes 1,024 or 2,048 are suitable.Accuracy fluctuates within 0.2 over drop path rates [0, 0.1]; batch sizes ≤512 degrade performance, while larger batches saturate and require more hardware.Source: Figure 6 · p. 8, §3.3 · p. 8
  • DW-Conv and MHSA account for 4.6%/4.1% and 13.8%/14.6% of parameters/FLOPs, respectively, and EMO's parameter and FLOP proportions are balanced.Feature similarities show DW-Conv captures short-distance correlations, EW-MHSA adds long-distance correlations, and iRMB combines both for a larger receptive field; Grad-CAM shows EMO focusing more accurately on salient objects than ResNet and MPViT.Source: Figure 8 · p. 8, Figure 9 · p. 8, §3.3 · pp. 8–9

4. Related Work

Related work reduces CNN cost through efficient convolutions and architecture search, integrates convolution into vision Transformers to add local inductive bias, and develops lighter Transformer models. The paper identifies a need for simpler mobile models that balance parameters, computation, and accuracy.

Efficient CNN Models

Hugging Vision Transformer with CNN

  • Vision Transformer work includes ResNet-like pyramids for dense prediction [42] and convolution integrations in FFN, positional embedding, attention, and QKV calculations.The passage notes that omitting 2D convolution may increase optimization difficulty and harm accuracy by removing local inductive bias.Source: §4 · p. 9

Efficient Transformer Improvements

  • Existing efficient Transformers use varied lightweight or hybrid modules, but their complexity limits mobility and usability, leaving the parameters–computation–accuracy balance unresolved.The paper instead studies a lightweight model based on an improved one-residual attention block.Source: §4 · p. 9

5. Conclusion and Future Works

The paper introduces Meta Mobile Block and derives iRMB for lightweight attention-based EMO models, reporting broad experimental support. It identifies architectural and training extensions for future work, deferred because of computational limits.

5. Conclusion and Future Works

  • The work unifies efficient IRB and Transformer components in Meta Mobile Block, derives iRMB, and builds attention-based EMO models using only iRMB for downstream tasks.The authors also report experiments across several datasets and detailed studies of the method.Source: §5 · p. 9
  • Future work will test transposed channel attention, multi-scale Res2Net [14], and efficient Performer [7], alongside higher-resolution inputs, NAS, distillation, ImageNet-21K training, and stronger augmentations or strategies [47] [28].The authors defer these experiments because of limited computational power.Source: §5 · p. 9

Related papers

Loading 2301.01146v4…