PaperlayerView exact revision on arXiv

Source-linked AI summary

MobileOne: An Improved One millisecond Mobile Backbone

Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, Anurag Ranjan

arXiv:2206.04040v2cs.CV

TL;DR

MobileOne examines why FLOPs and parameter count may fail to predict mobile latency and analyzes architectural and optimization bottlenecks through on-device benchmarking. It introduces a re-parameterizable backbone with simple inference-time structure, achieving sub-millisecond latency and strong results across classification, detection, and segmentation. The paper concludes that MobileOne offers efficient, general-purpose performance on mobile devices, while remaining less accurate than large models.

  • Problem

    FLOPs and parameter count may not correlate well with latency when efficient neural networks are deployed on mobile devices.

  • Method

    The paper benchmarks networks on an iPhone12, analyzes latency bottlenecks, and uses train-time over-parameterization with inference-time re-parameterization to design MobileOne.

  • Results

    MobileOne achieves significant latency improvements while maintaining accuracy across image classification, object detection, and semantic segmentation, including variants running under 1 ms on an iPhone12.

  • Takeaways & Limitations

    MobileOne provides a general-purpose efficient backbone that is effective on mobile devices and also performs efficiently on desktop CPUs and GPUs.

  • Takeaways & Limitations

    Although state-of-the-art among efficient architectures, MobileOne’s accuracy lags large models.

Abstract

from arXiv · show

Efficient neural network backbones for mobile devices are often optimized for metrics such as FLOPs or parameter count. However, these metrics may not correlate well with latency of the network when deployed on a mobile device. Therefore, we perform extensive analysis of different metrics by deploying several mobile-friendly networks on a mobile device. We identify and analyze architectural and optimization bottlenecks in recent efficient neural networks and provide ways to mitigate these bottlenecks. To this end, we design an efficient backbone MobileOne, with variants achieving an inference time under 1 ms on an iPhone12 with 75.9% top-1 accuracy on ImageNet. We show that MobileOne achieves state-of-the-art performance within the efficient architectures while being many times faster on mobile. Our best model obtains similar performance on ImageNet as MobileFormer while being 38x faster. Our model obtains 2.3% better top-1 accuracy on ImageNet than EfficientNet at similar latency. Furthermore, we show that our model generalizes to multiple tasks - image classification, object detection, and semantic segmentation with significant improvements in latency and accuracy as compared to existing efficient architectures when deployed on a mobile device. Code and models are available at https://github.com/apple/ml-mobileone

1. Introduction

MobileOne addresses the weak relationship between conventional efficiency metrics and mobile latency by analyzing on-device bottlenecks and designing a re-parameterizable backbone. Its variants achieve sub-millisecond inference and strong accuracy across image classification, detection, and segmentation.

  • Motivation: FLOPs and parameter count may not correlate well with mobile latency because memory access cost and parallelism affect inference efficiency.Skip-connections and branching can incur substantial memory-access costs.
  • Approach: MobileOne analyzes architectural and optimization bottlenecks by benchmarking networks on an iPhone12 and separating train-time and inference-time architectures.Train-time linear over-parameterization is re-parameterized into simpler inference structures.
  • MobileOne: MobileOne variants run under 1 ms on an iPhone12 while achieving state-of-the-art accuracy within efficient architectures.The inference structure removes branches and skip-connections, reducing memory-access costs and permitting wider layers.
  • Results: 2.3% better top-1 accuracy than EfficientNet-B0 is achieved on ImageNet at similar latency, while MobileOne is 5× faster than MobileViT-S on image classification.These comparisons are reported alongside broader latency and accuracy evaluations across efficient models.
  • Results: MobileOne generalizes to object detection and semantic segmentation while outperforming recent efficient models.The paper also reports released networks, code, and an iOS benchmarking application.

2. Related Work

Related mobile backbones optimize different proxies for efficiency, including parameters, FLOPs, compound scaling, and transformer-inspired designs. Platform-specific operators and re-parameterizable structures represent complementary strategies for improving mobile models.

  • Efficiency metrics: SqueezeNet and MobileViT optimize parameter count, while MobileNets, MobileNeXt, ShuffleNet-V1, GhostNet, and MixNet focus on FLOPs.EfficientNet and TinyNet study compound scaling of depth, width, and resolution.
  • Transformer-inspired models: MobileFormer and MobileViT adapt transformer-like ideas to mobile platforms, with MobileFormer optimizing FLOPs and MobileViT optimizing parameter count.These models are presented as recent mobile alternatives to efficient CNNs.
  • Platform considerations: MobileNet-V3 introduces Hard-Swish for a specific platform, but scaling such specialized activation functions across platforms may be difficult.This motivates designs based on operators broadly available across platforms.
  • Re-parameterization: ExpandNets, ACNet, and DBBNet use drop-in convolution replacements, while RepVGG uses re-parameterizable skip connections that simplify at inference.These approaches provide context for MobileOne’s train-time branches and inference-time re-parameterization.

3. Method

The paper evaluates mobile-network latency against FLOPs, parameter count, activations, and architectural blocks, then uses these findings to design MobileOne with separate training and inference structures. Its re-parameterizable, over-parameterized training blocks collapse into branch-free inference blocks, while training also uses model scaling and scheduled regularization.

  • Latency Analysis: Latency is only moderately correlated with FLOPs and weakly correlated with parameter count for efficient mobile architectures.Several higher-parameter models have lower latency, and convolutional models are faster than transformer counterparts at similar FLOPs and parameter counts.
  • Key Bottlenecks: Activation functions can produce drastically different latencies in otherwise identical 30-layer convolutional networks.The paper attributes much of this difference to synchronization costs from functions such as SE-ReLU, Dynamic Shift-Max, and DynamicReLU.
  • Key Bottlenecks: Multi-branch architectures and synchronization-heavy blocks increase mobile runtime through memory-access and synchronization costs.MobileOne therefore removes branches at inference and limits Squeeze-Excite usage to larger models.
  • MobileOne Architecture: MobileOne uses depthwise and pointwise convolutions with re-parameterizable skip and replicated branches during training, then removes all branches through re-parameterization at inference.The trivial over-parameterization factor k is varied from 1 to 5 and tuned across variants.
  • Re-parameterization: Batch normalization is folded into preceding convolutional branches, and the resulting branch weights and biases are summed to form the inference convolution.For skip connections, batch normalization is folded into an identity 1x1 convolution before padding and aggregation.
  • Model Scaling: MobileOne scales depth and width while avoiding input-resolution scaling because higher resolution increases FLOPs and memory consumption.Its depth scaling uses shallower early stages because those stages operate at larger input resolutions and are slower.
  • Optimization: Weight decay is annealed with a cosine schedule during training rather than removed completely, alongside cosine learning-rate scheduling and progressive learning.The paper evaluates these training settings through ablations on MobileOne-S2.

4. Experiments

MobileOne is evaluated across image classification, object detection, and semantic segmentation using standard datasets and latency-grouped comparisons. The experiments report strong accuracy–latency performance, including substantial gains over efficient baselines.

  • Image classification: ImageNet-1K evaluation compares models without distillation, grouped by latency on mobile devices.The validation set contains 50,000 images from 1,000 classes, and MobileOne models are evaluated at 224 × 224 resolution.
  • Image classification: 79.4% top-1 accuracy at 1.86ms makes MobileOne-S4 approximately 38× faster than MobileFormer at similar accuracy.MobileFormer reaches 79.3% top-1 accuracy with 70.76ms latency, while MobileOne-S4 reaches 79.4% with 1.86ms latency.
  • Knowledge distillation: MobileOne-S0 has 0.4M fewer inference parameters and 2.8% better top-1 accuracy than MobileNetV3-Small on ImageNet-1K under distillation.Train-time overparameterization enables distillation to improved performance despite similar or smaller inference parameter counts.
  • Object detection: The best MobileOne detector outperforms MNASNet by 27.8% and the best MobileViT model by 6.1% on MS-COCO mAP.MobileOne is used as the backbone feature extractor for an SSDLite detector at 320×320 input resolution.
  • Semantic segmentation: MobileOne is also evaluated as the backbone of Deeplab V3 for semantic segmentation on Pascal VOC and ADE20K.VOC models use the augmented Pascal VOC dataset, while ADE20K models use the same hyperparameters and augmentations.

VOC ADE20k

The paper extends its evaluation to robustness and micro-architecture benchmarks, where MobileOne generally maintains competitive or superior accuracy–efficiency trade-offs. Results include stronger out-of-distribution performance and improved accuracy at comparable latency.

  • Robustness benchmarks: MobileOne significantly outperforms efficient architectures on ImageNet-R and ImageNet-Sketch out-of-distribution benchmarks.It is less robust to corruption than MobileNetV3-L, but outperforms MobileNetV3-L on the out-of-distribution benchmarks and exceeds several other baselines on both benchmark types.
  • Micro architectures: MobileOne variants can outperform previous state-of-the-art micro architectures despite their extremely low FLOPs and parameter counts.The comparison highlights that architectural choices optimized for FLOPs or parameter count do not always produce lower-latency models.
  • Robustness benchmarks: Robustness results report mean corruption error for ImageNet-C and top-1 accuracy for the other robustness datasets.Lower is better for mean corruption error, while higher is better for top-1 accuracy.
  • Micro architectures: MobileOne-µ1 is 2× smaller and has 6.3% better top-1 accuracy than TinyNet-E at similar latency.Across the micro-architecture comparison, MobileOne models have similar latency to TinyNets while achieving lower parameter counts and better top-1 accuracy.

5. Discussion

The discussion frames MobileOne as a general-purpose mobile backbone whose direct latency measurements and re-parameterizable structures support efficient performance across hardware and tasks. It also identifies a boundary: accuracy still trails large models.

  • Discussion: MobileOne is proposed as an efficient general-purpose backbone for image classification, object detection, and semantic segmentation.The model scaling strategy uses re-parameterizable structures to attain state-of-the-art performance while remaining efficient on mobile devices and desktop CPUs.
  • Discussion: Direct mobile latency measurements show that latency may not correlate well with parameter count or FLOPs in the efficient regime.The paper analyzes latency bottlenecks from architectural components and empirically studies re-parameterizable structures as an optimization strategy.
  • Limitations and Future Work: MobileOne models are state-of-the-art within efficient architectures, but their accuracy lags large models.Future work targets improving the accuracy of these lightweight models and extending the backbone to additional computer-vision applications.

B.1. Additional Benchmarks

Additional benchmarks test MobileOne across TPU, mobile-device accuracy–latency plots, and progressive training settings. The results indicate that its accuracy–latency trade-off extends beyond the primary iPhone 12 evaluation.

  • Additional hardware: MobileOne achieves a state-of-the-art accuracy–latency trade-off on the Pixel-6 TPU.The paper compares existing architectures ported to the TPU with MobileOne, alongside CPU, desktop GPU, and iPhone 12 evaluations.
  • Benchmark visualizations: Figures 5–7 plot top-1 accuracy against iPhone 12 latency, zoomed-out accuracy–latency results, and top-1 accuracy against mAP.These figures enlarge the main paper’s Figure 1 panels.
  • Training settings: Progressive training settings increase image resolution and augmentation strength over training, with AutoAugment used for MobileOne-S2, S3, and S4.All models are trained from scratch for 300 epochs using SGD with momentum and an effective batch size of 256.

C.2. Analysis of Training Recipes

The analysis compares original and shared training recipes across efficient models, finding that MobileOne reaches competitive accuracy with fewer regularization techniques while recipe transfers vary by architecture.

  • Training runs reproduce original recipes, MobileOne’s recipe, and augmented recipes adding EMA, Progressive Learning, and Annealing Weight Decay.All models are trained for 300 epochs using the Timm library for comparison.
  • MobileOne uses less regularization than EfficientNet, MobileNetV3-L, and MixNet-S while reaching competitive accuracies.Applying the MobileOne recipe to those competitors does not improve their performance.
  • MobileNetV1 improves 1% and MobileNetV2 ×1.4 improves 0.5% when Progressive Learning and Annealing Weight Decay augment baseline recipes.The reported gains occur with the baseline recipes plus both additions.
  • MobileOne training runs are stable and achieve similar performance across two random seeds.

D. Micro Architectures

The micro-architecture analysis samples very small MobileOne variants and uses trivial over-parameterization to train them competitively, with larger benefits for smaller models.

  • Micro variants are sampled to be significantly smaller in parameter count rather than optimized for FLOPs.Trivial over-parameterization is used to train these architectures to competitive accuracies.
  • Additional over-parameterization branches benefit smaller variants more than larger variants.The experiments report consistent improvements for smaller variants, using the main-paper hyperparameters.

E.2. Qualitative Results

The qualitative and benchmark results show MobileOne variants achieving strong ImageNet accuracy at low mobile latency, with comparisons spanning training recipes and object detection.

  • 79.4 top-1 accuracy at 1.86 ms is reported for MobileOne-S4, while MobileOne-S3 reaches 78.1 at 1.53 ms.Both variants use CosLR + EMA + AA + PL + AWD.
  • EfficientNet-B0 reaches 77.8 top-1 accuracy at 1.72 ms with its reproduced recipe plus Progressive Learning and Annealing Weight Decay.
  • 77.4 top-1 accuracy at 1.18 ms is reported for MobileOne-S2, while MobileOne-S1 reaches 75.9 at 0.89 ms.The S1 result uses CosLR + EMA + PL + AWD.
  • MobileNetV2 ×1.4 reaches 76.2 top-1 accuracy at 1.36 ms with its reproduced recipe plus Progressive Learning and Annealing Weight Decay.
  • MobileOne-S2-SSDLite and MobileNetV2-SSDLite have similar latency, while MobileOne detects small and large objects more effectively in qualitative examples.Reported examples include potted plants, a dog and frisbee, and a blurry tennis racket and ball.

F.2. Qualitative Results

The semantic-segmentation examples compare MobileOne with MobileViT-S-DeepLabV3 and report better handling of foreground objects and fine details.

  • MobileOne performs better than MobileViT-S-DeepLabV3 in the qualitative semantic-segmentation results.The comparison includes examples involving background confusion, fine details, and foreground objects.
  • MobileOne resolves fine details such as a horse’s leg and tiny birds in the reported examples.
  • MobileOne segments large foreground objects in close-up views and small objects in the qualitative comparisons.
Loading 2206.04040v2…