Source-linked AI summary
MobileDets: Searching for Object Detection Architectures for Mobile Accelerators
Yunyang Xiong, Hanxiao Liu, Suyog Gupta, Berkin Akin, Gabriel Bender, Yongzhe Wang, Pieter-Jan Kindermans, Mingxing Tan, Vikas Singh, Bo Chen
TL;DR
MobileDets addresses the challenge of finding accurate, efficient object detectors for diverse mobile accelerators. The method enlarges neural architecture search with strategically placed regular convolutions and directly searches for detection models, producing superior quality-latency trade-offs across hardware platforms.
Problem
Finding models with suitable quality-compute trade-offs is difficult because higher-capacity networks consume more resources and manually designed architectures can become sub-optimal across changing hardware platforms.
Method
MobileDets uses latency-aware neural architecture search for object detection over a search space combining inverted bottlenecks with flexible regular-convolution sequences, including fused inverted bottlenecks.
Results
MobileDets achieve superior detection quality-latency trade-offs across mobile CPUs, EdgeTPUs, DSPs, and edge GPUs, including 1.7 mAP over MobileNetV3+SSDLite at comparable CPU latency.
Takeaways & Limitations
Regular convolutions can improve mobile object-detection accuracy-latency trade-offs when neural architecture search places them appropriately, challenging inverted-bottleneck-only search spaces.
Abstract
from arXiv · showhide
Inverted bottleneck layers, which are built upon depthwise convolutions, have been the predominant building blocks in state-of-the-art object detection models on mobile devices. In this work, we investigate the optimality of this design pattern over a broad range of mobile accelerators by revisiting the usefulness of regular convolutions. We discover that regular convolutions are a potent component to boost the latency-accuracy trade-off for object detection on accelerators, provided that they are placed strategically in the network via neural architecture search. By incorporating regular convolutions in the search space and directly optimizing the network architectures for object detection, we obtain a family of object detection models, MobileDets, that achieve state-of-the-art results across mobile accelerators. On the COCO object detection task, MobileDets outperform MobileNetV3+SSDLite by 1.7 mAP at comparable mobile CPU inference latencies. MobileDets also outperform MobileNetV2+SSDLite by 1.9 mAP on mobile CPUs, 3.7 mAP on Google EdgeTPU, 3.4 mAP on Qualcomm Hexagon DSP and 2.7 mAP on Nvidia Jetson GPU without increasing latency. Moreover, MobileDets are comparable with the state-of-the-art MnasFPN on mobile CPUs even without using the feature pyramid, and achieve better mAP scores on both EdgeTPUs and DSPs with up to 2x speedup. Code and models are available in the TensorFlow Object Detection API: https://github.com/tensorflow/models/tree/master/research/object_detection.
1. Introduction
MobileDets address the difficulty of balancing detection quality and computation on diverse, changing edge-device hardware. The approach expands NAS beyond inverted bottlenecks and produces models with improved latency–accuracy trade-offs across mobile platforms.
- Higher-capacity networks often improve performance but consume more resources, complicating deployment under limited edge-device inference budgets.
- Manual lightweight-architecture design is time-consuming and can become sub-optimal as new hardware platforms with different performance characteristics emerge.
- Regular convolutions can outperform depthwise convolutions on some accelerators, including running 3× faster on an EdgeTPU despite 7× more FLOPS.
- MobileDet expands the search space with flexible full-convolution sequences alongside inverted bottlenecks and uses latency-aware NAS for object detection across mobile hardware.
- 1.9 mAP on CPUs, 3.7 mAP on EdgeTPUs, 3.4 mAP on DSPs, and 2.7 mAP on edge GPUs are the reported MobileDet gains over MobileNetV2 at comparable inference latencies.
- MobileDets achieve state-of-the-art quality–latency trade-offs across mobile CPUs, EdgeTPUs, DSPs, and edge GPUs.
2. Related Work
Mobile detection research combines efficient one-stage heads and mobile backbones with hardware-aware architecture search. MobileDets differ by searching the backbone with full-convolution-based choices rather than relying only on depthwise-separable designs or feature-pyramid search.
- One-stage detectors use a single network pass and are therefore suited to efficient inference on edge devices.
- SSDLite is a lightweight SSD variant commonly paired with MobileNetV2 or MobileNetV3 for mobile detection baselines.
- NAS work has expanded from channel tuning and classification architectures toward direct searches for object-detection models with better latency–accuracy trade-offs.
- MnasFPN searches a mobile-friendly feature-pyramid head that heavily uses depthwise-separable convolutions, while its backbone remains outside the search.
- On Pixel-4 DSPs, IBN-only NAS adds 1.6 mAP over handcrafted MobileNetV2, while the MobileDet search space adds another 1.6 mAP.
- MobileDets search the backbone with full-convolution-based building blocks, targeting platforms including mobile CPUs, DSPs, and EdgeTPUs.
3. Revisiting Full Convolutions for Mobile Search Spaces
The paper expands mobile architecture search beyond inverted bottlenecks by introducing regular-convolution building blocks whose configurations are selected by NAS. These layers target accelerator-specific latency-accuracy trade-offs while retaining flexible channel expansion and compression.
- Search-space design: The MobileDet search space augments IBNs with flexible full-convolution sequences for mobile object detection.The search space targets mobile CPUs, DSPs, EdgeTPUs, and edge GPUs.
- Motivation: IBN-only search spaces may be suboptimal on accelerators because depthwise-convolution efficiency does not consistently track FLOPS or parameter count.A regular convolution can run 3× as fast as its depthwise counterpart on EdgeTPU despite having 7× more FLOPS.
- Fused inverted bottleneck layers: Fused inverted bottlenecks replace the initial pointwise-plus-depthwise sequence with a K × K regular convolution followed by a 1 × 1 projection.The regular convolution expands channels from C1 to s × C1, with s > 1, before projection to C2.
- Tucker convolution layers: Tucker layers generalize bottlenecks by allowing NAS to choose separate intermediate channel dimensions around a K × K regular convolution.They use input compression s < 1 and output compression e < 1 before restoring C2 output channels.
- Decomposition connections: The proposed layer variants are connected to CP and Tucker decompositions, linking their sequential structures to approximations of regular convolutions.The paper explicitly names the new building blocks Tucker convolution layers because of their Tucker-decomposition connection.
4. Architecture Search Method
The architecture search jointly trains a one-shot model and controller to optimize a platform-aware reward, using a learned latency surrogate for efficient device-specific search. The cost model is trained from sampled architectures benchmarked on each target hardware platform.
- Search algorithm: TuNAS constructs a one-shot model containing the search choices and a controller that selects architectures optimizing a platform-aware reward.The one-shot model and controller are trained together during search.
- Search objective: The reward combines architecture mAP with a latency-cost penalty relative to a specified budget.The cost c(M) is inference latency, c0 is the cost budget, and τ < 0 balances accuracy against cost.
- Cost models: The latency cost model is a linear regression over layer type and input/output channel-size features.Its reported fidelity across platforms is r2 ≥0.99.
- Efficient evaluation: During search, mAP is estimated from a small mini-batch and latency is supplied by a device-trained regression surrogate.Several thousand randomly sampled architectures are benchmarked once per hardware platform before search.
5. Experiments
Experiments evaluate latency-aware object-detection NAS across mobile CPUs, EdgeTPUs, DSPs, and GPUs, comparing search spaces and hardware-targeted models. MobileDets improve accuracy-latency trade-offs, especially on accelerators where regular convolutions are useful.
- Experimental setup: Experiments use COCO detection, 320×320 images, mAP, and real accelerator latency, with separate architecture-search and retraining stages.Benchmarks use TF-Lite for CPU, EdgeTPU, and DSP, while GPU models use TensorRT.
- Search spaces: The evaluated search spaces expand from IBN-only to IBN+Fused and IBN+Fused+Tucker, while searching layer widths and compression ratios.Fused convolutions add searchable kernel sizes and expansion factors; Tucker layers add searchable input and output compression ratios.
- Mobile CPU: +1.7 mAP over MobileNetV3+SSDLite is achieved on mobile CPUs at comparable latency, while full convolutions remain quality-neutral there.Detection-specific NAS provides the CPU gain, and the experiments find no evidence that fused IBNs or Tucker bottlenecks are required for high-quality CPU models.
- EdgeTPU: +3.7 mAP over MobileNetV2+SSDLite is achieved on EdgeTPUs at comparable latency, with IBN+Fused+Tucker adding 0.8 mAP and reducing latency by 7% versus IBN-only.Regular convolutions improve the accuracy-latency trade-off even though IBN-only retains the best accuracy-MAdds trade-off.
- DSP: Including regular convolutions improves DSP search results from 26.9 mAP at 12.2 ms to 28.5 mAP at 11.9 ms.The resulting MobileDet reaches 28.5 mAP at 12.3 ms and outperforms MobileNetV2+SSDLite by 3.2 mAP at comparable latency.
- Cross-hardware behavior: Searched EdgeTPU and DSP architectures transfer mutually, whereas accelerator-specialized architectures do not transfer well to mobile CPUs.MobileDets use regular convolutions extensively on EdgeTPU and DSP, especially early in the network where depthwise convolutions tend to be less efficient.
6. Conclusion
The paper concludes that depthwise inverted bottlenecks should not be the only building blocks for edge vision models. MobileDets use strategically placed full convolutions selected through NAS to improve detection accuracy-latency trade-offs across mobile accelerators.
- Conclusion: Full convolutions can substantially improve accuracy-latency trade-offs on several accelerators when NAS places them at suitable network positions.The resulting MobileDets outperform prior art across multiple hardware platforms.
- Conclusion: MobileDets achieve superior detection results across mobile CPUs, EdgeTPUs, DSPs, and edge GPUs.The conclusion presents object detection as the case study for revisiting depthwise inverted bottlenecks.
A. Appendix
The appendix describes the relationship between MobileDet search-space building blocks and the linear structure of Tucker and CP decomposition.
- Appendix: The appendix relates MobileDet building blocks to the linear structure of Tucker and CP decomposition.It provides further details rather than introducing a new experimental result.
A.1. Connections with Tucker/CP decomposition
The MobileDets search space connects inverted bottlenecks and proposed layer variants to approximate convolution through CP and Tucker decomposition. It combines fused convolution layers, Tucker layers, and IBNs.
- CP decomposition: The inverted bottleneck structure is equivalent to the sequential approximate evaluation of a regular convolution using CP decomposition, aside from nonlinearities.
- CP decomposition: CP decomposition approximates convolution with sequential pointwise and depthwise mappings, linking this structure to inverted bottlenecks.The sequence includes a 1 × 1 pointwise convolution, spatial depthwise convolutions, and a final pointwise convolution.
- MobileDets search space: The MobileDets search space mixes fused convolution layers and Tucker layers with inverted bottlenecks.The paper names the expansion operation a fused convolution layer and the compression operation a Tucker layer.