PaperlayerView exact revision on arXiv

Source-linked AI summary

Mobile-Former: Bridging MobileNet and Transformer

Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, Zicheng Liu

arXiv:2108.05895v3cs.CVcs.LG

TL;DR

Efficient vision models struggle to combine MobileNet’s local processing with transformer global interaction at low computational budgets. Mobile-Former addresses this with parallel branches, few learnable global tokens, and a lightweight bidirectional bridge, achieving strong classification and detection results. Its practical limitations include resolution-dependent latency and parameter inefficiency in image classification.

  • Problem

    Low-FLOP vision models need to encode both local processing and global interaction efficiently.

  • Method

    Mobile-Former parallelizes MobileNet and transformer branches, using up to six randomly initialized learnable tokens and lightweight cross attention for bidirectional fusion.

  • Results

    Mobile-Former outperforms efficient CNNs and vision transformers in low-FLOP classification and improves object detection over MobileNetV3 and DETR.

  • Takeaways & Limitations

    A thin transformer and lightweight bridge can improve representation capability while maintaining low computational cost across classification and detection.

  • Takeaways & Limitations

    Mobile-Former can be slower than MobileNetV3 at smaller image resolutions and is parameter-inefficient for classification because of its heavy head.

Abstract

from arXiv · show

We present Mobile-Former, a parallel design of MobileNet and transformer with a two-way bridge in between. This structure leverages the advantages of MobileNet at local processing and transformer at global interaction. And the bridge enables bidirectional fusion of local and global features. Different from recent works on vision transformer, the transformer in Mobile-Former contains very few tokens (e.g. 6 or fewer tokens) that are randomly initialized to learn global priors, resulting in low computational cost. Combining with the proposed light-weight cross attention to model the bridge, Mobile-Former is not only computationally efficient, but also has more representation power. It outperforms MobileNetV3 at low FLOP regime from 25M to 500M FLOPs on ImageNet classification. For instance, Mobile-Former achieves 77.9\% top-1 accuracy at 294M FLOPs, gaining 1.3\% over MobileNetV3 but saving 17\% of computations. When transferring to object detection, Mobile-Former outperforms MobileNetV3 by 8.6 AP in RetinaNet framework. Furthermore, we build an efficient end-to-end detector by replacing backbone, encoder and decoder in DETR with Mobile-Former, which outperforms DETR by 1.1 AP but saves 52\% of computational cost and 36\% of parameters.

1. Introduction

Mobile-Former addresses the challenge of combining efficient local processing with global interaction by parallelizing MobileNet and a transformer connected through a bidirectional bridge. It achieves strong low-FLOP performance across classification and detection while introducing explicit efficiency and parameter limitations.

  • Vision transformers’ gains diminish below 1G FLOPs, while MobileNet-based CNNs remain dominant below 300M FLOPs because of efficient local processing.
  • Mobile-Former parallelizes MobileNet and transformer branches, connecting them with a two-way bridge for local-global feature communication.Mobile extracts local features with inverted bottleneck blocks, while Former uses learnable tokens for global features.
  • The bridge and Former consume less than 20% of total computational cost while improving representation capability.The design uses very few transformer tokens and a lightweight bridge to exchange local and global features.
  • 77.9% top-1 accuracy is achieved on ImageNet at 294M FLOPs, outperforming MobileNetV3 and LeViT in the reported comparison.
  • 8.6 AP is gained over MobileNetV3 in RetinaNet, while a Mobile-Former DETR variant gains 1.1 AP over DETR with 52% fewer FLOPs and 36% fewer parameters.The DETR comparison uses the same number of object queries: 100.
  • The authors note that optimal network width and height are outside this work’s scope.

2. Related Work

Related work spans lightweight CNNs, vision transformers, and hybrid CNN-transformer designs. Mobile-Former differs by parallelizing MobileNet and a transformer with bidirectional cross attention, targeting the low-FLOP regime.

  • Lightweight CNNs reduce computation through depthwise-pointwise, group, microfactorized, dynamic, and other efficient operators.
  • Vision transformers improve global visual processing, with later methods addressing training data, resolution, hierarchical structure, and attention locality.
  • Prior hybrid models combine convolution and transformers through convolutional stems, attention modifications, or serial integration.
  • Mobile-Former instead parallelizes MobileNet and transformer branches with bidirectional cross attention and reports efficiency and effectiveness at low FLOPs.

3. Our Method: Mobile-Former

Mobile-Former parallelizes MobileNet and a thin transformer, connecting local feature maps and global tokens through a lightweight bidirectional cross-attention bridge. Its blocks combine efficient local processing, global interaction, dynamic activation, and low-cost scaling across architectures and FLOP budgets.

  • 3.1. Overview: Mobile-Former parallelizes MobileNet and transformer components and connects them with bidirectional cross attention.Mobile extracts local features from images, while the transformer processes a few learnable global tokens.
  • 3.1. Overview: M ≤6 randomly initialized tokens represent global image priors, reducing transformer computation relative to vision transformers.The tokens are denoted Z ∈ R^M×d, where M and d are the token count and dimension.
  • 3.1. Overview: The Mobile→Former bridge fuses local feature maps into global tokens using lightweight cross attention at Mobile’s low-channel bottleneck.Key and value projections are removed from the Mobile side, while Former-side projections are retained.
  • 3.1. Overview: The Mobile←Former bridge fuses global tokens into local features, using local features as queries and global tokens as keys and values.The query projection is removed from the Mobile side to reduce computation, while Former-side key and value projections remain.
  • 3.2. Mobile-Former Block: Each Mobile-Former block contains Mobile and Former sub-blocks plus Mobile→Former and Mobile←Former cross-attention modules.The block updates local feature maps X and global tokens Z for the next block.
  • 3.2. Mobile-Former Block: The Mobile sub-block replaces ReLU with dynamic ReLU whose parameters are generated from the first Former output token.It otherwise follows an inverted bottleneck design with 3×3 depthwise convolutions.
  • 3.2. Mobile-Former Block: The Former sub-block is a standard transformer block with multi-head attention and FFN, using FFN expansion ratio 2 and post-layer normalization.Former is processed between the Mobile→Former and Mobile←Former bridges.
  • 3.2. Mobile-Former Block: Former and the two-way bridge consume less than 20% of a block’s total computational cost, while Mobile consumes most computation at O(HWC^2).Former complexity is O(M^2d + Md^2), and the bridge complexity is O(MHWC + MdC).

4. Efficient End-to-End Object Detection

Mobile-Former builds an efficient end-to-end detector by using Mobile-Former blocks in both the backbone and head, with separate global tokens and object queries across multiple resolutions. It also adds spatial-aware dynamic ReLU and adaptive query positions while achieving higher detection accuracy than DETR at substantially lower computation.

  • Backbone–Head architecture: Mobile-Former uses separate tokens in its backbone and head: six global tokens for the backbone and 100 object queries for the head.The head’s object queries are generated similarly to DETR.
  • Backbone–Head architecture: The head employs multiscale resolutions, progressively refining all object-query representations from coarse to fine.This avoids manually allocating objects across scales by size as in FPN.
  • Detection results: 1.1 AP: the end-to-end Mobile-Former detector reaches 43.1 AP versus DETR’s 42.0 AP while using 41.4G versus 86G FLOPs.The Mobile-Former detector uses a Mobile-Former backbone and nine Mobile-Former blocks in the head.
  • Spatial-aware dynamic ReLU: Spatial-aware dynamic ReLU uses all global tokens to generate position-specific parameters instead of deriving spatial-shared parameters from only the first token.The token-to-position attention is obtained by normalizing Mobile→Former cross attention along the tokens.
  • Adapting position embedding in head: The detection head adapts each query’s position embedding after every block from its feature embedding, allowing positions to change with content.The adaptation function uses two MLP layers with ReLU between them.

5. Experimental Results

Experiments evaluate Mobile-Former on ImageNet classification and COCO detection, showing that its parallel MobileNet–transformer design improves accuracy while keeping computation low. Ablations attribute this efficiency and effectiveness to compact global tokens, lightweight bridging, and complementary components.

  • Experimental setup: Mobile-Former is evaluated on ImageNet classification and COCO object detection, including comparisons with efficient CNNs, vision transformers, RetinaNet backbones, and DETR.ImageNet uses 1000 classes with 1,281,167 training images and 50,000 validation images; classification models use 224×224 inputs.
  • ImageNet classification: 77.9% top-1 accuracy at 294M FLOPs outperforms LeViT’s 76.6% at 305M FLOPs without distillation or extended training.Mobile-Former is trained for 450 epochs, whereas LeViT uses teacher distillation and 1000 epochs.
  • ImageNet classification: Mobile-Former clearly outperforms CNNs and vision-transformer variants across the evaluated accuracy–FLOP range.The comparison includes GhostNet, Swin, and DeiT, with Swin and DeiT implemented from 100M to 2G FLOPs.
  • Ablations: Adding Former and the two-way bridge costs 12% of computation and gains 2.6% top-1 accuracy over Mobile alone, while dynamic ReLU adds 1.0%.The bridge fuses local and global features, and dynamic-ReLU parameters are generated from the first global token.
  • Ablations: A single global token reaches 77.1% top-1 accuracy, while six tokens reach 0.7% additional improvement and further tokens stop improving performance.The token dimension is 192; compact global tokens are identified as a key efficiency contributor.
  • Ablations: Six tokens of dimension 192 allocate 35M/294M, or 12%, of computation to Former and the bridge.Accuracy improves from 76.8% to 77.8% as token dimension increases from 64 to 192, then converges.
  • Ablations: Removing FFN drops top-1 accuracy by 0.3%, while replacing multi-head attention with position-mixing MLP drops accuracy from 77.8% to 77.3%.The MLP is more efficient but static rather than adaptive to different input images.
  • Object detection: Mobile-Former outperforms MobileNetV3 and ShuffleNetV2 by 8.3+ AP under similar computational cost in RetinaNet.It also achieves higher AP than ResNet and transformer variants with significantly fewer backbone FLOPs.

6. Limitations and Discussions

Mobile-Former’s efficiency depends on the deployment setting: its latency advantage over MobileNetV3 disappears at smaller image resolutions, and its classification parameter count is inflated by the head.

  • Inference latency: Mobile-Former is slower than MobileNetV3 as image resolution decreases, despite faster inference on large images.The overhead comes from resolution-independent Former and bridge projections whose PyTorch implementations are less efficient than convolution.
  • Parameter efficiency: The Mobile-Former-294M classification head uses 4.6M of 11.4M parameters, or 40% of the total.This parameter inefficiency is mitigated in object detection because the image-classification head is removed.

7. Conclusion

Mobile-Former parallelizes MobileNet and a transformer, connecting them with a two-way bridge to combine local processing and global interaction. The design improves accuracy while reducing computational cost across image classification and object detection.

  • Mobile-Former parallelizes MobileNet and transformer components with a two-way bridge for communication.Mobile extracts local features, while the transformer models global interaction.
  • The design improves accuracy and saves computational cost compared with efficient CNNs and vision transformer variants in the low FLOP regime.The paper reports gains on both image classification and object detection.
  • An end-to-end Mobile-Former detector outperforms DETR while using substantially fewer computations and parameters.

A. Mobile-Former Architecture

Mobile-Former models combine lightweight MobileNet blocks, a compact transformer-based head, and efficient upsampling and downsampling components. The model and detector variants are manually designed across multiple scales and computational budgets.

  • Model variants: Seven manually designed Mobile-Former variants span 508M–52M, with the smallest 26M model replacing 1×1 convolutions by group convolution.The models serve image classification or object-detection backbones.
  • Mobile-Former blocks: Downsampled Mobile-Former blocks use a stride-two depthwise convolution before costly pointwise convolutions at lower resolution.The downsampling variant contains four convolutional layers rather than three.
  • Training: Training learning rate, weight decay, and dropout increase with model size to prevent overfitting.These hyperparameters are specified for ImageNet classification models.
  • Detection heads: End-to-end detector heads use 100 object queries of dimension 256 and vary in Mobile-Former depth and number of scales.The largest head uses nine blocks over three scales, while smaller heads use fewer blocks over two scales.
  • Detection heads: Detector heads project input features to 256 channels and use bilinear upsampling, backbone-feature addition, and depthwise-pointwise convolution.

B. More Experimental Results

Additional experiments examine latency, kernel size, and attention choices across classification and detection settings. Mobile-Former’s overhead is relatively larger at low resolution, while several design choices preserve efficiency or improve detection performance.

  • Inference latency: Resolution-independent Former and bridge projections create relatively large overhead at small image sizes, which becomes negligible as resolution grows.The authors attribute this partly to less efficient PyTorch implementations than convolution and identify implementation optimization as future work.
  • Inference latency: Mobile-Former-214M is slower than MobileNetV3 Large at low resolution but becomes faster above 750×750 despite similar FLOPs.The comparison uses single-thread, batch-size-one inference on an Intel Xeon CPU.
  • Kernel-size ablation: Increasing the Mobile depthwise-convolution kernel from 3×3 to 5×5 provides negligible gain, supporting the use of smaller kernels.The authors attribute the limited gain to global features enlarging Mobile’s receptive field.
  • Attention ablation: Adding multi-head attention to five low-resolution detection blocks increases performance by 0.6 AP with 15% additional computational cost.The improvement is especially large for large-object detection, increasing APL from 58.7 to 60.4.

C. Visualization

Visualizations show that the two-way bridge progressively changes token focus, separates foreground from background, and reduces attention diversity across tokens at higher levels.

  • Global-token focus: Global-token focus shifts from edges and corners to connected regions, foreground or background, and finally discriminative regions across network levels.The first token’s attention is visualized over pixels in Mobile→Former.
  • Foreground-background separation: At middle layers, Mobile←Former attention separates foreground and background through different token contributions.At block 8, background pixels attend more to the first token while foreground pixels attend more to the last token.
  • Attention diversity: Attention patterns are more diverse across tokens at low levels than at high levels.The six tokens differ at blocks 3 and 5, become similar at block 8, and largely converge by block 12.
Loading 2108.05895v3…