Source-linked AI summary
RepViT: Revisiting Mobile CNN From ViT Perspective
Ao Wang, Hui Chen, Zijia Lin, Jungong Han, Guiguang Ding
TL;DR
Lightweight ViTs offer strong mobile performance, but their architectural differences from lightweight CNNs remain insufficiently examined. The paper modernizes MobileNetV3-L with efficient ViT designs to create RepViT, a pure lightweight CNN, which outperforms state-of-the-art lightweight ViTs and CNNs across vision tasks with favorable latency.
Problem
Architectural differences between lightweight ViTs and CNNs remain insufficiently examined despite lightweight ViTs’ mobile performance advantages and deployment challenges.
Method
RepViT incrementally modernizes MobileNetV3-L by incorporating efficient lightweight ViT architectural designs into a pure lightweight CNN.
Results
RepViT outperforms state-of-the-art lightweight ViTs and CNNs across vision tasks, including over 80% top-1 accuracy at 1.0 ms latency on an iPhone 12.
Takeaways & Limitations
The results highlight the prospect of pure lightweight CNNs as strong mobile models and establish RepViT as a baseline for lightweight edge deployment research.
Abstract
from arXiv · showhide
Recently, lightweight Vision Transformers (ViTs) demonstrate superior performance and lower latency, compared with lightweight Convolutional Neural Networks (CNNs), on resource-constrained mobile devices. Researchers have discovered many structural connections between lightweight ViTs and lightweight CNNs. However, the notable architectural disparities in the block structure, macro, and micro designs between them have not been adequately examined. In this study, we revisit the efficient design of lightweight CNNs from ViT perspective and emphasize their promising prospect for mobile devices. Specifically, we incrementally enhance the mobile-friendliness of a standard lightweight CNN, \ie, MobileNetV3, by integrating the efficient architectural designs of lightweight ViTs. This ends up with a new family of pure lightweight CNNs, namely RepViT. Extensive experiments show that RepViT outperforms existing state-of-the-art lightweight ViTs and exhibits favorable latency in various vision tasks. Notably, on ImageNet, RepViT achieves over 80\% top-1 accuracy with 1.0 ms latency on an iPhone 12, which is the first time for a lightweight model, to the best of our knowledge. Besides, when RepViT meets SAM, our RepViT-SAM can achieve nearly 10$\times$ faster inference than the advanced MobileSAM. Codes and models are available at \url{https://github.com/THU-MIG/RepViT}.
1. Introduction
Lightweight ViTs can outperform lightweight CNNs but remain difficult to deploy on mobile hardware, while architectural differences between the families are underexamined. RepViT revisits MobileNetV3-L using lightweight ViT designs and reports strong accuracy-latency results across vision tasks.
- Lightweight models target strong vision performance with reduced computational cost for deployment on resource-constrained mobile devices.
- ViTs often become too large and latent for mobile deployment, while aggressively shrinking them can make their performance inferior to lightweight CNNs.
- Lightweight ViTs improve mobile accuracy-latency trade-offs, but inadequate hardware support and high-resolution latency remain practical challenges.
- Despite structural similarities, differences in block, macro, and micro designs between lightweight ViTs and CNNs have not received sufficient examination.
- RepViT modernizes MobileNetV3-L with efficient lightweight ViT designs, producing a pure lightweight CNN in a ViT-like MetaFormer structure.
- Over 80% top-1 accuracy is achieved with 1.0 ms latency on an iPhone 12, while RepViT-SAM delivers nearly 10× faster inference than MobileSAM.
2. Related Work
CNNs offer useful locality and deployment advantages, whereas heavyweight ViTs require substantial resources. Related work has made ViTs more mobile-friendly, motivating RepViT’s direct reconsideration of lightweight CNN design from a ViT perspective.
- CNNs are computationally expensive in standard form, motivating techniques such as separable convolutions to make them suitable for resource-constrained mobile devices.
- ViTs achieve strong recognition performance but commonly require substantial computation and memory, limiting their suitability for mobile devices.
- Lightweight ViTs use hybrids, efficient attention, and dimension-consistent designs to improve mobile efficiency and performance.
- RepViT addresses overlooked architectural distinctions between lightweight CNNs and ViTs by integrating ViT designs into lightweight CNNs.
3. Methodology
The methodology progressively modernizes MobileNetV3-L for real mobile latency, aligning training and redesigning block, macro, and micro architectures. The resulting choices improve the accuracy-latency trade-off without relying on distillation in the reported modernization process.
- 3.1. Preliminary: Actual on-device latency is used instead of FLOPs or model size because it supports more accurate evaluation and fairer mobile comparisons.
- 3.1. Preliminary: MobileNetV3-L measures 1.01 ms under the mobile latency benchmark.
- 3.1. Preliminary: The training recipe is aligned with lightweight ViTs using AdamW, cosine scheduling, augmentation, and regularization, while knowledge distillation is initially excluded.
- 3.2. Block design: The RepViT block separates token and channel mixers by repositioning depthwise convolution and using structural re-parameterization.
- 3.2. Block design: Setting the channel-mixer expansion ratio to 2 reduces latency to 0.65 ms, while widening stages yields 73.5% top-1 accuracy at 0.89 ms latency.
- 3.2. Block design: The redesigned block outperforms direct expansion-ratio and width adjustments to the original MobileNetV3 block: 73.0% top-1 accuracy at 0.91 ms versus 73.5% at 0.89 ms.
- 3.3. Macro design: Macro optimization replaces MobileNetV3-L’s complex stem with early convolutions and deepens the downsampling layer to address mobile latency and information loss.
- 3.4. Micro design: Cross-block SE placement uses SE in alternating blocks, reaching 77.4% top-1 accuracy with 0.87 ms latency.
4. Experiments
Experiments show that RepViT delivers strong accuracy–latency trade-offs for image classification, SAM-based segmentation, object detection, instance segmentation, and semantic segmentation on mobile-oriented benchmarks.
- Image Classification: RepViT consistently achieves state-of-the-art performance across model sizes, including 3.0% and 2.0% top-1 accuracy gains over EfficientFormerV2-S0 and FastViT-T8 at similar latency.RepViT-M1.1 also improves over EfficientFormerV2-S1 by 1.7% performance.
- Image Classification: 83.7% accuracy with 2.3 ms latency is achieved by RepViT-M2.3, while pure lightweight CNNs outperform state-of-the-art lightweight ViTs on mobile devices.The reported latency is measured on mobile-oriented hardware settings described in the experiments.
- Image Classification: 2.7% accuracy gain at 1.0 ms latency is achieved over MobileOne-S1 without knowledge distillation; RepViT-M2.3 also reduces latency by 34.3% versus PoolFormer-S36.The latter comparison pairs a 1.1% performance improvement with latency reduction from 3.5 ms to 2.3 ms.
- RepViT meets SAM: RepViT-SAM runs smoothly on iPhone 12 while MobileSAM and ViT-B-SAM fail to run, and is nearly 10× faster than MobileSAM on Macbook M1 Pro.The latency comparison uses the standard 1024×1024 resolution and Core ML Tools.
- RepViT meets SAM: RepViT-SAM outperforms MobileSAM and ViT-B-SAM across zero-shot edge detection, zero-shot instance segmentation, and SegInW, with comparable ODS and OIS to ViT-H-SAM on edge detection.The comparison covers BSDS500, COCO, and SegInW benchmarks.
- Downstream Tasks: RepViT improves downstream-task accuracy and latency trade-offs, including higher APbox and APmask, nearly 2× faster detection, and favorable mIoU–latency results.RepViT-M1.1 exceeds EfficientFormer-L1 by 1.9 APbox and 1.8 APmask; RepViT-M2.3 is nearly 2× faster than EfficientFormerV2-L while increasing mIoU by 0.9.
- Ablation Studies: Removing structural re-parameterization consistently reduces RepViT performance, while alternative SE-layer placement provides a more advantageous accuracy–latency trade-off.The SE placement comparison evaluates removing SE layers and placing them in every block.
5. Conclusion
RepViT revisits lightweight CNN design by incorporating lightweight ViT architectural principles, producing a family of pure CNNs for resource-constrained mobile devices. It outperforms existing lightweight ViTs and CNNs across vision tasks while maintaining favorable performance–latency trade-offs.
- RepViT incorporates lightweight ViT architectural designs into a new family of lightweight CNNs for resource-constrained mobile devices.
- RepViT outperforms existing state-of-the-art lightweight ViTs and CNNs across various vision tasks.
- The results highlight the promising prospect of pure lightweight CNNs for mobile devices.