Source-linked AI summary
MobileNetV4 -- Universal Models for the Mobile Ecosystem
Danfeng Qin, Chas Leichner, Manolis Delakis, Marco Fornoni, Shixin Luo, Fan Yang, Weijun Wang, Colby Banbury, Chengxi Ye, Berkin Akin, Vaibhav Aggarwal, Tenghui Zhu, Daniele Moro, Andrew Howard
TL;DR
Mobile devices require a balance between accuracy and efficiency under computational constraints. MobileNetV4 addresses this with UIB, Mobile MQA, refined NAS, and distillation, achieving mostly Pareto-optimal performance across mobile hardware and 87% ImageNet-1K accuracy at 3.8ms Pixel 8 EdgeTPU latency.
Problem
Mobile devices impose computational constraints that make balancing neural-network accuracy and efficiency a significant challenge.
Method
MobileNetV4 combines the flexible UIB block, Mobile MQA, refined NAS recipes, and a novel distillation approach to design efficient mobile models.
Results
MobileNetV4 models achieve mostly Pareto-optimal performance across diverse mobile hardware, while MNv4-Hybrid-L reaches 87% top-1 ImageNet-1K accuracy at 3.8ms Pixel 8 EdgeTPU latency.
Takeaways & Limitations
MobileNetV4 provides a model suite designed to run efficiently across mobile CPUs, GPUs, DSPs, and specialized accelerators.
Takeaways & Limitations
Roofline modeling assumes software implementation has no impact on workload performance, so its predictions can differ from performance on real devices.
Abstract
from arXiv · showhide
We present the latest generation of MobileNets, known as MobileNetV4 (MNv4), featuring universally efficient architecture designs for mobile devices. At its core, we introduce the Universal Inverted Bottleneck (UIB) search block, a unified and flexible structure that merges Inverted Bottleneck (IB), ConvNext, Feed Forward Network (FFN), and a novel Extra Depthwise (ExtraDW) variant. Alongside UIB, we present Mobile MQA, an attention block tailored for mobile accelerators, delivering a significant 39% speedup. An optimized neural architecture search (NAS) recipe is also introduced which improves MNv4 search effectiveness. The integration of UIB, Mobile MQA and the refined NAS recipe results in a new suite of MNv4 models that are mostly Pareto optimal across mobile CPUs, DSPs, GPUs, as well as specialized accelerators like Apple Neural Engine and Google Pixel EdgeTPU - a characteristic not found in any other models tested. Finally, to further boost accuracy, we introduce a novel distillation technique. Enhanced by this technique, our MNv4-Hybrid-Large model delivers 87% ImageNet-1K accuracy, with a Pixel 8 EdgeTPU runtime of just 3.8ms.
1 Introduction
MobileNetV4 addresses the accuracy–efficiency challenge on mobile devices with UIB, Mobile MQA, refined NAS, and distillation. The resulting models are designed for broad hardware efficiency, with strong accuracy and latency results.
- Architectural innovations: UIB unifies Inverted Bottleneck, ConvNext, FFN, and ExtraDW designs through optional depthwise convolutions.The block provides flexibility in spatial and channel mixing and can extend the receptive field.
- Architectural innovations: Over 39% inference speedup is achieved by Mobile MQA on mobile accelerators relative to Multi-Head Attention.
- Search and distillation: The two-phase NAS approach separates coarse and fine-grained searches, while offline distillation data reduces reward-measurement noise and improves model quality.
- Model suite: MNv4 models achieve mostly Pareto-optimal performance across CPUs, DSPs, GPUs, and specialized accelerators.The evaluated comparison includes hardware-specific exclusions and compatibility limitations for some baselines and MNv4-Hybrid models.
2 Related Work
Related work improves mobile efficiency through efficient convolutional networks, hybrid convolution-attention models, efficient attention approximations, and hardware-aware NAS. These approaches target different combinations of architectural efficiency, attention cost, and device-specific optimization.
- Mobile Convolutional Networks: Mobile convolutional networks improve efficiency through depthwise separable convolutions, inverted residuals, depthwise operations, lightweight attention, and reparameterized branches.
- Efficient Hybrid Networks: Efficient hybrid networks combine convolutions with attention or transformer components to capture long-range dependencies and fuse complementary features.
- Efficient Attention: Efficient attention methods reduce the cost of self-attention through linear-complexity approximations or downsampling of attention inputs.
- Hardware-aware Neural Architecture Search: Hardware-aware NAS automates model design under device constraints using latency tables, reinforcement learning, or differentiable search.
3 Hardware-Independent Pareto Efficiency
MobileNetV4 uses roofline analysis to design models that remain mostly Pareto-optimal across hardware with different compute and memory bottlenecks. Its architecture allocates compute- and memory-intensive operations where each contributes most across the ridge-point range.
- Roofline analysis: The roofline model estimates latency from operational intensity and hardware peak compute throughput and memory bandwidth.It identifies whether a workload is compute-bound or memory-bound, with the slower operation approximately determining latency.
- Pareto efficiency: MobileNetV4 is mostly Pareto-optimal from ridge points of 0 to 500 MACs/byte, matching the real hardware measurements reported elsewhere.The figure’s analytically derived charts connect latency and accuracy trade-offs across this range.
- Roofline analysis: Ridge Point is the ratio of PeakMACs to PeakMemBW and represents the minimum operational intensity needed for maximum performance.The analysis sweeps RP from 0 MAC/byte to 500 MACs/byte to cover expected hardware bottlenecks.
- Modeling assumption: The common total-MAC latency proxy corresponds to a roofline model with RP = 0, assuming zero memory time for every layer.This simplification does not represent all hardware bottlenecks.
- Modeling limitation: Roofline analysis assumes software implementation does not affect workload performance, so it can overestimate techniques with complex memory access on real devices.The paper gives pruning as an example of a technique that may perform much better in the roofline model than on hardware.
- Architecture across hardware: Large Conv2D layers are costly on low-ridge-point hardware but add inexpensive capacity on high-ridge-point hardware, while FC and depthwise layers show the reverse trend.MobileNetV4 places MAC-intensive Conv2Ds near the beginning and memory-intensive FC layers near the end of the network.
4 Universal Inverted Bottlenecks
The Universal Inverted Bottleneck is a searchable block that unifies several efficient network micro-architectures while allowing spatial mixing, receptive-field, and utilization trade-offs. Its optional depthwise convolutions give NAS flexibility to tailor blocks to different efficiency targets.
- UIB overview: UIB is an efficient-network building block that adapts to varied optimization targets while remaining simple enough for Neural Architecture Search.Figure 4 presents its block structure.
- UIB structure: UIB extends the Inverted Bottleneck with optional depthwise convolutions before expansion and between expansion and projection.NAS selects which depthwise operations to include.
- Unified micro-architectures: UIB unifies the original Inverted Bottleneck, ConvNext, and FFN blocks and introduces the ExtraDW variant.ExtraDW combines ConvNext-Like and Inverted Bottleneck benefits while increasing depth and receptive field inexpensively.
- Search efficiency: The UIB search space shares more than 95% of parameters between instantiations, keeping SuperNet-based NAS manageable.Pointwise expansion and projection components are shared, while depthwise operations remain searchable.
- Design flexibility: At each stage, UIB trades off spatial and channel mixing, receptive-field size, and computational utilization.The paper reports that Table 1 compares accuracy and latency across searches using IB, ConvNext-Like, and full UIB blocks.
5 Mobile MQA
Mobile MQA is an accelerator-optimized attention block that reduces memory-bandwidth demands and speeds attention on mobile hardware. Spatial reduction further improves efficiency with minimal accuracy loss.
- Mobile MQA speeds attention by over 39% on mobile accelerators relative to Multi-Head Attention.
- Sharing keys and values across query heads reduces memory-bandwidth requirements when batched tokens are few relative to feature dimensions.
- Spatial Reduction Attention downscales key and value resolution while retaining high-resolution queries.
- Stride-2 depthwise-convolution spatial reduction adds over 20% efficiency with a -0.06% accuracy change.
6 Design of MNv4 Models
MNv4 combines hardware-aware design principles, flexible UIB blocks, Mobile MQA, and an enhanced two-stage NAS process. These choices target Pareto-efficient models across diverse mobile hardware.
- Design philosophy: MNv4 prioritizes simplicity and hardware compatibility to approach Pareto-optimal performance across diverse mobile platforms.
- UIB blocks: UIB lets NAS tune spatial and channel mixing, receptive fields, and computational utilization across network stages.
- Mobile MQA: Mobile MQA provides straightforward attention designed for efficient deployment in hybrid models.
- NAS recipe: The two-stage NAS separates coarse filter-size search from fine-grained depthwise-layer configuration.
- NAS recipe: An offline JFT distillation dataset reduces NAS reward sensitivity, while extending training to 750 epochs yields deeper, higher-quality models.
- Model construction: MNv4-Conv models use NAS-optimized UIB blocks, while MNv4-Hybrid models interlace Mobile MQA and UIB blocks.
7 Results
MNv4 models are mostly Pareto-optimal across CPUs, DSPs, GPUs, and specialized accelerators in ImageNet classification and COCO object detection evaluations. Their advantages vary by hardware, with strong CPU and EdgeTPU results and broad cross-platform coverage.
- Experimental setup: The evaluation measures ImageNet-1K validation Top-1 accuracy and latency across representative mobile CPUs, DSPs, GPUs, Apple Neural Engine, and Google EdgeTPU hardware.
- ImageNet classification: MNv4 models achieve mostly Pareto-optimal performance across CPUs, DSPs, GPUs, Apple Neural Engine, and Google EdgeTPU.
- ImageNet classification: MNv4 is roughly 2x faster than MobileNetV3 on CPUs and 2x as fast as MobileNetV3 on EdgeTPUs at comparable accuracy.
- ImageNet classification: MNv4-Conv models remain the top performers on DSP, while MNv4-Hybrid models perform well on CPUs and accelerators.
- Object detection: COCO object detection evaluation uses RetinaNet with MNv4 medium backbones, a 256-d FPN, depth-separable decoder convolutions, and 600 training epochs.
8 Enhanced distillation recipe
The enhanced distillation recipe targets higher accuracy for mobile models without increasing latency. It combines dynamic dataset mixing with additional in-domain JFT data, improving student performance over individual augmentation datasets.
- Motivation: Distillation can increase mobile-model accuracy without increasing latency, complementing architectural improvements.
- Dynamic Dataset Mixing: Dynamic Dataset Mixing combines multiple datasets with diverse augmentation strategies instead of using a fixed augmentation sequence.
- Dynamic Dataset Mixing: 84.4% student accuracy from D1 + D2 exceeds D2 at 84.1% and D1 at 83.8%.
- JFT Data Augmentation: The recipe adds in-domain, class-balanced JFT-300M data by resampling 130K images per class, totaling 130M images.
9 Conclusion
MobileNetV4 combines new building blocks, improved NAS, distillation, and hardware-aware analysis to deliver efficient models across the mobile ecosystem. The resulting suite is mostly Pareto optimal across diverse mobile hardware, with high-end accuracy reaching 87% at 3.8 ms on Pixel 8 EdgeTPU.
- 9 Conclusion: MobileNetV4 combines UIB, Mobile MQA, improved NAS recipes, and a novel distillation approach for efficient mobile models.UIB, Mobile MQA, and refined NAS target efficiency across devices, while distillation further improves accuracy.
- 9 Conclusion: 87% ImageNet-1K accuracy is achieved at 3.8 ms Pixel 8 EdgeTPU latency with the distilled high-end MNv4 model.The conclusion identifies this result as a new state-of-the-art.
- 9 Conclusion: MNv4 models are mostly Pareto optimal across mobile CPUs, GPUs, DSPs, and specialized accelerators.The paper presents this cross-hardware characteristic as absent from other tested models.
- 9 Conclusion: The search space uses fixed initial and head layers while NAS selects UIB blocks and instantiations across the remaining stages.The design prioritizes higher computational density in early layers because pointwise convolutions can have low operational intensity at high resolution.
- 9 Conclusion: Restricting the search to components with well-correlated cross-device cost models makes EdgeTPU latency optimization yield universally efficient models.The paper supports this relationship through later hardware analyses.
F Additional Roofline Analysis
The additional roofline analysis evaluates MNv4 across operational-intensity regimes and compares predictions with measured hardware performance. It also examines Mobile MQA optimization, showing that reordering Einsum indices removes unnecessary transposes and substantially reduces latency.
- Additional Roofline Analysis: Roofline sweeps extend the analysis across small, medium, and large MNv4-Conv models from MACs-only to accelerator-like memory-bandwidth bottlenecks.The sweep covers ridge points from 0.0 to 500.0 MACs/byte and includes measured-latency, roofline, and MAC-count correlations.
- Additional Roofline Analysis: Roofline models capture the relative ordering of model families near Pareto frontiers on each hardware target, while target-specific nuance remains.Table 16 and Fig. 6 compare empirically fitted roofline predictions with measured hardware behavior.
- Small Models: MobileNetV4-Conv-S reaches 73.80% roofline operations at 0.00 MACs/byte, compared with 65.65% for MobileNetV1-0.5x, 66.00% for MobileNetV2-0.5x, and 69.20% for MobileNetV3-0.5x.These values represent the MACs-only end of the roofline sweep.
- Medium Models: MobileNetV4-Conv-Medium reaches 79.9% roofline operations at 0.00 MACs/byte, compared with 79.40% for MobileOne-S4.The comparison is reported for the MACs-only roofline condition.
- Einsum Optimization: Mobile MQA reduces attention-block latency by threefold on Pixel 7 and by 13.8% on Pixel 8 through Einsum index and weight-tensor reordering.The changes eliminate unnecessary transposes and require only a few lines of code.
- Einsum Optimization: Mobile MQA also provides an almost threefold training speedup on Google Tensor v4 with the tf.keras training framework.The optimization addresses inefficiencies caused by tensor transposes in Einsum-based implementations.
H Implementation details of our distillation recipe
The distillation recipe creates offline teacher soft labels from JPEG-compressed augmented images and trains students with a restricted augmentation and regularization setup. It improves efficiency beyond accuracy, including faster training and reduced hyper-parameter tuning.
- Distillation dataset: The offline distillation dataset records EfficientNet L2’s softmax probabilities for 1,000 classes after augmenting, JPEG-compressing, decoding, and inferring each image.EfficientNet L2 has 480 M parameters, 290 G MACs, and 87.5% top-1 ImageNet-1K accuracy.
- Student training: Student training disables augmentation except left-right flips and uses dropout only in the final fully connected layer for regularization.The student is trained with teacher-soft-label cross-entropy, AdamW, and cosine learning-rate scheduling with warm-up.
- Efficiency benefits: 2.5x faster training is achieved with distillation than with ImageNet-1K training.MNv4-Conv-Large takes 2.3 hours for 400 epochs with batch size 16k on 128 TPU v5e.
- Efficiency benefits: Distillation provides significant relief from hyper-parameter tuning in addition to improving model accuracy.The passage presents this as another benefit of the distillation training procedure.