Source-linked AI summary
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen
TL;DR
Mobile vision models often require more computation and memory than mobile and embedded devices can support. MobileNetV2 addresses this with inverted residuals and linear bottlenecks, improving ImageNet performance across a wide range and, with SSDLite, achieving 20× less computation and 10× less parameters than YOLOv2 for COCO detection.
Problem
Modern state-of-the-art neural networks often require computational resources beyond the capabilities of mobile and embedded devices.
Method
MobileNetV2 uses inverted residuals with linear bottlenecks, expanding compressed inputs, filtering them with depthwise convolutions, and projecting them back to low-dimensional representations.
Results
MobileNetV2 improves ImageNet performance across a wide range and, with SSDLite, outperforms real-time COCO detectors while using 20× less computation and 10× less parameters than YOLOv2.
Takeaways & Limitations
The architecture enables highly efficient mobile models with memory-efficient inference using standard operations available in neural-network frameworks.
Takeaways & Limitations
Although t-way splitting leaves multiply-add counts unchanged, replacing one matrix multiplication with several smaller ones can hurt runtime performance in existing implementations.
Abstract
from arXiv · showhide
In this paper we describe a new mobile architecture, MobileNetV2, that improves the state of the art performance of mobile models on multiple tasks and benchmarks as well as across a spectrum of different model sizes. We also describe efficient ways of applying these mobile models to object detection in a novel framework we call SSDLite. Additionally, we demonstrate how to build mobile semantic segmentation models through a reduced form of DeepLabv3 which we call Mobile DeepLabv3. The MobileNetV2 architecture is based on an inverted residual structure where the input and output of the residual block are thin bottleneck layers opposite to traditional residual models which use expanded representations in the input an MobileNetV2 uses lightweight depthwise convolutions to filter features in the intermediate expansion layer. Additionally, we find that it is important to remove non-linearities in the narrow layers in order to maintain representational power. We demonstrate that this improves performance and provide an intuition that led to this design. Finally, our approach allows decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis. We measure our performance on Imagenet classification, COCO object detection, VOC image segmentation. We evaluate the trade-offs between accuracy, and number of operations measured by multiply-adds (MAdd), as well as the number of parameters
1. Introduction
MobileNetV2 addresses the resource demands of modern neural networks with a mobile-tailored architecture that reduces operations and memory while retaining accuracy. Its core innovation is the inverted residual with linear bottleneck, which expands, depthwise-filters, and linearly projects compressed representations.
- Modern state-of-the-art networks often require computational resources beyond many mobile and embedded devices.
- The proposed architecture significantly decreases operations and memory needed for mobile computer vision models while retaining the same accuracy.
- The inverted residual with linear bottleneck expands a low-dimensional representation, filters it with a lightweight depthwise convolution, and projects it back linearly.
- The module reduces inference memory footprint by never fully materializing large intermediate tensors, lowering main-memory access on embedded hardware.
2. Related Work
Prior work improved neural-network efficiency through architecture search, training methods, pruning, connectivity learning, and altered convolutional connectivity. MobileNetV2 instead seeks the simplest possible network design by using intuition about how neural networks operate, complementing optimization-based architectural search.
- Efficiency-oriented architecture research: Recent efficiency research has combined manual architecture design and training improvements with hyper-parameter optimization, network pruning, and connectivity learning.These efforts followed substantial improvements over early architectures such as AlexNet and VGGNet.
- Efficiency-oriented architecture research: Other approaches modify internal convolutional connectivity through structures such as ShuffleNet or by introducing sparsity.These methods target the connectivity structure of convolutional blocks.
- Algorithmic architectural search: Genetic algorithms and reinforcement learning opened a direction for algorithmic architectural search, but the resulting networks can become very complex.The paper presents its approach as complementary to this line of work.
- Paper positioning: This paper uses intuition about neural-network operation to guide the simplest possible network design.Its stated goal is to develop better intuition about how neural networks operate rather than rely only on optimization methods.
3. Preliminaries, discussion and intuition
The section motivates MobileNetV2’s efficient convolutions and inverted residual bottlenecks through computational savings and an analysis of information preservation under ReLU. It argues for linear bottlenecks and shortcuts between bottlenecks, with expansion ratios greater than 1 separating network expressiveness from capacity.
- Efficient convolutions: Depthwise separable convolutions factorize standard convolution into depthwise filtering and 1 × 1 pointwise convolution, reducing computation while retaining similar empirical performance.For k = 3, MobileNetV2 reports computational cost 8 to 9 times smaller than standard convolutions with only a small reduction in accuracy.
- Manifold intuition: ReLU can preserve complete information about an input manifold only when the manifold lies in a low-dimensional subspace of the activation space.When the ReLU output retains non-zero volume, the corresponding mapping is limited to a linear transformation; channel collapse otherwise loses information.
- Linear bottlenecks: Linear bottleneck layers are therefore inserted into convolutional blocks because nonlinearities in narrow layers can destroy information and hurt performance.The paper treats the manifold as low-dimensional and uses linear layers to avoid excessive information loss in bottlenecks.
- Inverted residuals: Inverted residual blocks connect bottlenecks with shortcuts, rather than connecting expanded representations, because bottlenecks contain the necessary information while expansion accompanies nonlinear transformation.The inverted design improves gradient propagation, is more memory efficient, and works slightly better in the reported experiments.
- Expansion ratio: Expansion ratios greater than 1 are most useful, enabling the network’s expressiveness to be studied separately from its capacity.An expansion ratio below 1 corresponds to a classical residual convolutional block, while an inner-layer depth of 0 yields the identity through the shortcut.
4. Model Architecture
MobileNetV2 uses bottleneck depth-separable convolutions with residuals, organized into an initial convolution and 19 residual bottleneck layers. Its design fixes key implementation choices while exposing expansion rate, resolution, and width multiplier for accuracy–efficiency trade-offs.
- Architecture: The architecture begins with a 32-filter fully convolutional layer followed by 19 residual bottleneck layers.The bottleneck block is a depth-separable convolution with residuals.
- Implementation: ReLU6, 3 × 3 kernels, dropout, and batch normalization are used for robustness and training.ReLU6 is selected because of its robustness under low-precision computation.
- Expansion: Expansion rates between 5 and 10 produce nearly identical performance curves, with smaller models favoring smaller rates and larger models favoring larger rates.The main experiments use expansion factor 6; a 64-channel input therefore expands to 384 channels.
- Trade-off hyper parameters: The primary network uses width multiplier 1 and 224 × 224 input resolution, costing 300 million multiply-adds and 3.4 million parameters.Input resolution and width multiplier are tunable to target different accuracy/performance trade-offs, with explored resolutions from 96 to 224.
- Implementation: For width multipliers less than one, applying the multiplier to every layer except the last convolutional layer improves performance for smaller models.This is an implementation difference from [27].
5. Implementation Notes
MobileNetV2’s bottleneck residual blocks support memory-efficient inference by scheduling tensor computations and treating internal convolutions as disposable. A t-way channel split further reduces intermediate storage, with t=2–5 balancing memory savings against runtime efficiency.
- Memory scheduling: Inference implementations build a directed acyclic compute hypergraph and choose computation orders to minimize stored intermediate tensors.The optimization considers plausible orders and tensor sizes during computation.
- Bottleneck memory: Treating a bottleneck residual block as one operation makes memory use depend mainly on bottleneck tensors rather than larger internal tensors.For trivial parallel structures, the memory bound simplifies to the maximum combined input and output size across operations.
- Bottleneck memory: The bottleneck operator’s memory requirement can be as low as |s2k| + |s′2k′| + O(max(s2, s′2)).The block composes linear input and output transformations around a per-channel nonlinear transformation.
- Channel splitting: Using a t-way split keeps only one intermediate block of size n/t in memory, or a single channel when n = t.This relies on the inner transformation being per-channel and the surrounding non-per-channel operators having a substantial input-to-output size ratio.
- Channel splitting: 2–5 is the most helpful range for t because splitting preserves multiply-add count while significantly reducing memory and retaining most optimized operator efficiency.Larger numbers of smaller matrix multiplications can hurt runtime through increased cache misses; framework-level optimization may improve this trade-off.
6. Experiments
The experiments compare MobileNetV2 with prior mobile architectures across ImageNet classification, COCO object detection, and PASCAL VOC semantic segmentation, while evaluating efficiency through parameters and Multiply-Adds. They also show that inverted residual connections and linear bottlenecks improve performance, and introduce SSDLite as a computationally efficient mobile detector.
- ImageNet classification: MobileNetV2 is compared with MobileNetV1, ShuffleNet, and NASNet-A on ImageNet, using Multiply-Adds and Pixel 1 running time as efficiency measures.ShuffleNet results are omitted because efficient group convolutions and shuffling were unsupported in the evaluation setup.
- COCO object detection: SSDLite replaces SSD prediction-layer convolutions with depthwise separable convolutions, substantially reducing parameter count and computational cost.Each replacement uses a depthwise convolution followed by a 1 × 1 projection.
- COCO object detection: 20× more efficient and 10× smaller, MobileNetV2 SSDLite still outperforms YOLOv2 on COCO while achieving the highest accuracy among the three compared detectors.The evaluation uses 320 × 320 inputs and reports mAP, parameters, and Multiply-Adds.
- Mobile semantic segmentation: MobileNetV2 and MobileNetV1 feature extractors are evaluated with DeepLabv3 on PASCAL VOC 2012 using mIOU, with experiments varying extractors, simplified heads, and inference strategies.Multi-scale and left-right flipped inputs significantly increase Multiply-Adds and are unsuitable for on-device applications.
- Architectural analysis: Shortcuts connecting bottleneck layers perform better than shortcuts connecting expanded layers, and linear bottlenecks improve performance despite being less powerful than nonlinear bottlenecks.The experiments support the view that nonlinearities destroy information in low-dimensional spaces.
7. Conclusions and future work
The paper presents a simple, efficient mobile architecture that improves ImageNet performance and outperforms real-time COCO detectors in accuracy and model complexity. Its convolutional block separates network expressiveness from capacity, motivating further research.
- Conclusions: The architecture enables highly efficient mobile models through memory-efficient inference and standard neural-network operations.Its basic building unit is designed for mobile applications.
- Conclusions: On ImageNet, the architecture improves the state of the art across a wide range of performance points.
- Conclusions: 20× less computation and 10× less parameters than YOLOv2 are achieved by combining the architecture with SSDLite for COCO object detection.The combination also outperforms state-of-the-art realtime detectors in accuracy and model complexity.
- Future work: The convolutional block separates network expressiveness, encoded by expansion layers, from capacity, encoded by bottleneck inputs.Exploring this separation is identified as an important direction for future research.
A. Bottleneck transformation
The section argues that ReLU can destroy information in narrow representations, whereas sufficiently expanded bottlenecks can preserve information while retaining nonlinear transformations. It formalizes invertibility conditions and connects expansion size to both information preservation and representational complexity.
- Motivation: Narrow ReLU transformations can exploit nonlinearity only at the cost of information loss, while sufficiently wide expansions can remain highly nonlinear and invertible.The operator is A ReLU(Bx), with B of size m × n and A of size n × m; the contrast is between m ≤ n and n ≪ m.
- Invertibility condition: ReLU(Bx) is uniquely invertible when its output has at least n non-zero values corresponding to n linearly independent rows of B.If fewer than n coordinates are non-zero, the corresponding system is underdetermined and has infinitely many solutions.
- Information preservation: When m ≫ n, only a small fraction of Bx values need to be positive for ReLU(Bx) to be invertible.The section further states that sufficiently large expansion layers make the fraction of collapsed space small, so ReLU(Bx) preserves information with high probability.
- Empirical validation: Empirical activation patterns remain above the invertibility thresholds in all but two fully trained layers, despite substantially increased standard deviation after training.At initialization, activation patterns concentrate around half positive channels; the threshold is the number of positive dimensions exceeding the input-space dimension.
- Representational power: L ReLU layers with n neurons and bottleneck expansion pn can map pn^L input volumes to the same output region, linking expansion size to greater function complexity.This construction is stated for any integer L ≥ 1 and p > 1, with input volumes linearly isomorphic to [0, 1]^n.
B. Semantic segmentation visualization results
Figure 8 presents MobileNetV2 semantic segmentation visualizations on the PASCAL VOC 2012 validation set and identifies output stride 16 with single-scale input 1 as a favorable FLOPS–accuracy trade-off.
- Evaluation setting: The visualizations evaluate MobileNetV2 semantic segmentation on the PASCAL VOC 2012 validation set.The figure reports results using output stride (OS), single-scale input (S), and multi-scale plus flipped inputs (MS+F).
- Evaluation setting: Multi-scale inputs use scales {0.5, 0.75, 1, 1.25, 1.5, 1.75} together with left-right flipped inputs.This configuration is labeled MS+F in the figure.
- FLOPS–accuracy trade-off: Output stride = 16 with single input scale = 1 attains a good trade-off between FLOPS and accuracy.The figure identifies this setting as the preferred balance between computational cost and segmentation accuracy.