PaperlayerView exact revision on arXiv

Source-linked AI summary

MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features

Shakti N. Wadekar, Abhishek Chaurasia

arXiv:2209.15159v2cs.CVcs.AIcs.LG

TL;DR

MobileViTv1’s fusion block creates a complex learning task and scaling challenges for lightweight vision models. MobileViTv3 changes the fusion design to simplify feature integration and scaling, improving results over MobileViTv1 and MobileViTv2 across ImageNet-1K and downstream vision tasks.

  • Problem

    MobileViTv1 provides lightweight CNN–ViT models, but its fusion block creates a complex learning task and scaling challenges.

  • Method

    MobileViTv3 replaces and augments the fusion design to combine input, local CNN, and global ViT features while enabling wider model scaling.

  • Results

    MobileViTv3 models outperform corresponding MobileViTv1 and MobileViTv2 models across ImageNet-1K, ADE20K, COCO, and PascalVOC2012 evaluations.

  • Takeaways & Limitations

    The proposed block supports higher accuracy at similar parameters and FLOPs across lightweight classification, segmentation, and detection models.

  • Takeaways & Limitations

    Higher FLOPs than other mobile CNNs may remain an issue for edge devices.

Abstract

from arXiv · show

MobileViT (MobileViTv1) combines convolutional neural networks (CNNs) and vision transformers (ViTs) to create light-weight models for mobile vision tasks. Though the main MobileViTv1-block helps to achieve competitive state-of-the-art results, the fusion block inside MobileViTv1-block, creates scaling challenges and has a complex learning task. We propose changes to the fusion block that are simple and effective to create MobileViTv3-block, which addresses the scaling and simplifies the learning task. Our proposed MobileViTv3-block used to create MobileViTv3-XXS, XS and S models outperform MobileViTv1 on ImageNet-1k, ADE20K, COCO and PascalVOC2012 datasets. On ImageNet-1K, MobileViTv3-XXS and MobileViTv3-XS surpasses MobileViTv1-XXS and MobileViTv1-XS by 2% and 1.9% respectively. Recently published MobileViTv2 architecture removes fusion block and uses linear complexity transformers to perform better than MobileViTv1. We add our proposed fusion block to MobileViTv2 to create MobileViTv3-0.5, 0.75 and 1.0 models. These new models give better accuracy numbers on ImageNet-1k, ADE20K, COCO and PascalVOC2012 datasets as compared to MobileViTv2. MobileViTv3-0.5 and MobileViTv3-0.75 outperforms MobileViTv2-0.5 and MobileViTv2-0.75 by 2.1% and 1.0% respectively on ImageNet-1K dataset. For segmentation task, MobileViTv3-1.0 achieves 2.07% and 1.1% better mIOU compared to MobileViTv2-1.0 on ADE20K dataset and PascalVOC2012 dataset respectively. Our code and the trained models are available at: https://github.com/micronDLA/MobileViTv3

1 INTRODUCTION

MobileViTv3 improves the lightweight MobileViT family by simplifying feature fusion and enabling scaling while maintaining similar parameter and FLOP budgets. The resulting models improve performance across classification, segmentation, and detection benchmarks.

  • Motivation: MobileViTs target resource-constrained mobile devices while supporting classification, detection, and segmentation tasks.The work focuses on models with parameter budgets of 6M or less.
  • Contribution: The paper proposes a simpler fusion of input, local CNN, and global ViT features to address MobileViT-block challenges.The changes target both fusion complexity and architectural scaling.
  • Results: 2% and 1.9% accuracy gains are reported for MobileViTv3-XXS and MobileViTv3-XS over corresponding MobileViTv1 models on ImageNet-1K.The comparisons use similar parameters and FLOPs.
  • Results: 2.1% and 1.0% gains are reported for MobileViTv3-0.5 and MobileViTv3-0.75 over corresponding MobileViTv2 models on ImageNet-1K.The new models add the proposed fusion block to MobileViTv2 architectures.

2 RELATED WORK

Related work combines CNN locality and ViT long-range modeling to improve vision performance and efficiency. This includes general-purpose ViTs, efficient CNNs, and hybrid architectures designed for constrained devices.

  • Vision Transformers: ViT introduced transformer architectures from natural-language processing into vision, while later variants improved training, attention, or computational efficiency.Examples include DeiT, XCiT, SwinT, ViL, and Focal-transformer.
  • CNNs: CNN families such as ResNet and DenseNet use residual or dense connections to support deep networks and strong benchmark performance.Efficient CNN variants also target reduced parameters and FLOPs.
  • Hybrids: Hybrid models combine CNN local information with ViT self-attention to capture both local structure and long-range dependencies.MobileViT and MobileFormer specifically target constrained environments such as mobile devices.

3 NEW MOBILEVIT ARCHITECTURE

MobileViTv3 introduces four block-level changes that simplify fusion and reduce scaling costs. These changes support wider model variants and improve accuracy relative to MobileViTv1 under comparable resource budgets.

  • MobileViTv3 block: MobileViTv3 replaces the MobileViTv1 fusion design with four architectural changes and also adds a fusion block to MobileViTv2.Figure 2 compares the MobileViTv1, MobileViTv2, and MobileViTv3 modules.
  • Fusion block: Replacing the fusion 3x3 convolution with 1x1 convolution simplifies location-independent feature fusion and reduces width-scaling costs.The original convolution combines features from multiple locations and incurs large parameter and FLOP increases when channels grow.
  • Fusion block: MobileViTv3 concatenates local and global features in fusion, then adds input features through a residual connection.The input-feature residual contributes a 0.6% accuracy gain in the ablation study.
  • Local representation: A depthwise 3x3 convolution replaces the local block’s standard 3x3 convolution to reduce parameters with limited accuracy impact.The change is presented as a parameter–accuracy trade-off.
  • Scaling: MobileViTv3-S, XS, and XXS scale by increasing layer width, with architecture details and resource comparisons reported against MobileViTv1.The proposed changes enable the scaled variants.
  • Scaling: 2%, 1.9%, and 0.9% accuracy gains are reported for MobileViTv3-XXS, XS, and S over corresponding MobileViTv1 models on ImageNet-1K.The table caption states the comparisons for the three scaled variants.

4 EXPERIMENTAL RESULTS

The experiments evaluate MobileViTv3 across classification and lightweight-model comparisons on ImageNet-1K, using parameter and FLOP-matched baselines. MobileViTv3 achieves leading accuracy across several parameter ranges while retaining resource efficiency.

  • Experimental setup: MobileViTv3 models are evaluated on ImageNet-1K alongside ViT, hybrid, and lightweight CNN models under parameter and FLOP budgets.Comparisons focus mainly on models around 6M parameters or less and models above 6M parameters with approximately 2 GFLOPs or less.
  • Comparison with ViTs and hybrid models: 72.33% accuracy makes MobileViTv3-0.5 the best model reported in the 1–2M-parameter range.MobileViTv3-XXS also achieves the best accuracy in this range among the listed models.
  • Comparison with ViTs and hybrid models: 76.7% Top-1 accuracy gives MobileViTv3-XS the highest reported result in the 2–4M-parameter range.This is 3.9% higher than Mini-DeiT-Ti, 4.5% higher than XCiT-N12, and 6.2% higher than PVTv2-B0.
  • Comparison with ViTs and hybrid models: MobileViTv3-S attains the highest accuracy in the 4–8M-parameter range.It is 0.7% better than XCiT-T12 despite using a simple training recipe and 300 epochs, versus distillation, advanced augmentation, and 400 epochs for XCiT-T12.
  • Comparison with lightweight CNNs: MobileViTv3-0.5 and MobileViTv3-XXS achieve 72.33% and 70.98% accuracy, respectively, among lightweight CNN comparisons in the 1–2M range.MobileViTv3-0.5 improves by over 2.5% versus the listed MobileNetv3, ShuffleNetv2, and ESPNetv2 models.
  • Comparison with lightweight CNNs: MobileViTv3-XS and MobileViTv3-S improve accuracy by over 4% and more than 2%, respectively, against the listed lightweight CNN baselines in their parameter ranges.EfficientNet-B1 reaches 79.1% with 1.3x more parameters and 2.6x fewer FLOPs than MobileViTv3-S, which reaches 79.3%.

4.2 SEGMENTATION

MobileViTv3 is evaluated for semantic segmentation on PASCAL VOC 2012 and ADE20K using DeepLabv3 integration and mean Intersection over Union. It outperforms corresponding MobileViTv1 and MobileViTv2 models across the reported comparisons.

  • Implementation details: MobileViTv3 is integrated with DeepLabv3 for PASCAL VOC 2012 segmentation, with performance reported using mean Intersection over Union.Training uses 50 epochs; MobileViTv3-S, XS, and XXS use a batch size of 48, while corresponding MobileViTv1 models use 128.
  • Results: 80.04% mIOU is achieved by MobileViTv3-1.0 on PASCAL VOC 2012, 1.1% above MobileViTv2-1.0.MobileViTv3-XS is 1.6% better than MobileViTv1-XS, and MobileViTv3-0.5 surpasses MobileViTv2-0.5 by 1.41%.
  • Results: 2.07%, 1.73%, and 1.64% are the reported ADE20K improvements of MobileViTv3-1.0, 0.75, and 0.5 over their MobileViTv2 counterparts.The ADE20K models use the same stated training hyperparameters as the corresponding MobileViTv2 models.
  • Results: MobileViTv3 models outperform corresponding MobileViTv1 and MobileViTv2 models on PASCAL VOC 2012 despite using a lower training batch size of 48 versus 128.The comparison is reported for models trained under the stated segmentation settings.

4.3 OBJECT DETECTION

MobileViTv3 is evaluated for COCO object detection using pretrained backbones in an SSDLite network. The reported results show gains over MobileViTv1 and lightweight CNN baselines, with additional comparisons to heavier CNNs.

  • Implementation details: MobileViTv3 uses pretrained backbones in SSD, with standard SSD-head convolutions replaced by separable convolutions to form SSDLite for COCO detection.The COCO evaluation uses 117K training images and 5K validation images.
  • Results: MobileViTv3-XS improves by 0.8% mAP over MobileViTv1-XS and by 2.6% mAP over MNASNet on COCO.The comparison includes lightweight CNN models, while a separate table reports comparisons with heavier CNNs.
  • Results: MobileViTv3-XS and MobileViTv3-1.0 surpass MobileViTv1-XS and MobileViTv2-1.0 by 0.8% and 0.5% mAP, respectively.The reported parameter counts for MobileViT models refer to the encoder or backbone architecture only.

4.5 ABLATION STUDY OF OUR PROPOSED MOBILEVITV3 BLOCK

The ablation study evaluates four MobileViTv3 block changes and shows that simplifying fusion, adding a residual connection, and using depthwise convolution improve accuracy while enabling scaling. After scaling to similar parameters and FLOPs, MobileViTv3 models outperform MobileViTv1 baselines.

  • Ablation setup: The ablation evaluates four proposed MobileViTv3 block changes incrementally against MobileViTv1-S and defines the fully modified unscaled model as MobileViTv3-S(unscaled).The study records Top-1 ImageNet-1K accuracy after adding changes one by one.
  • Fusion changes: 1.1% improvement follows from replacing the fusion block’s 3x3 convolution with a 1x1 convolution.The change is intended to simplify fusion of local and global features.
  • Fusion changes: A residual connection raises MobileViTv3-S(unscaled) to a 1.6% gain over MobileViTv1-S, adding 0.6% over the preceding change.The residual connection adds input features to the fusion-block output.
  • Local representation: Depthwise convolution preserves a 1.3% gain over baseline while reducing parameters and FLOPs, with only a 0.3% drop from the preceding change.This change supports scaling the MobileViTv3 block.
  • 300-epoch comparison: 77.5% Top-1 accuracy is achieved by MobileViTv3-S(unscaled), outperforming the 75.6% MobileViTv1-S baseline by 1.9% with 22.7% fewer parameters and 18.6% fewer FLOPs.Both models were trained for 300 epochs with batch size 192.
  • Scaled models: 79.3% accuracy is achieved by scaled MobileViTv3-S with similar parameters and FLOPs, while MobileViTv3-XS and XXS surpass their MobileViTv1 counterparts by 1.9% and 2.0%.The models are scaled and trained with batch size 384.

5 DISCUSSION AND LIMITATIONS

MobileViTv3 targets mobile deployment by improving accuracy under comparable resource budgets and by examining latency and throughput. However, its higher FLOPs can remain problematic for edge devices because of the self-attention module.

  • Discussion: MobileViTv3 improves accuracy while maintaining the baseline’s memory and computation, and it can improve latency and throughput with minimal accuracy impact.The comparison covers MobileViTv1 and MobileViTv2 models and measures parameters, FLOPs, latency, and throughput.
  • Limitations: Higher FLOPs remain a limitation for edge devices, inherited from the vision-transformer self-attention module.The authors identify optimization of self-attention as future work.
  • Future directions: ImageNet-1K accuracies could potentially improve with training batch size 1024, and the fusion of input, local, and global features could be explored in other hybrid architectures.These observations define directions for further evaluation and transfer.

A.1 OBJECT DETECTION ON COCO DATASET

The appendix presents COCO validation-image object detection examples using SSD-Lite with MobileViTv3-S and MobileViTv3-1.0 backbones, including challenging visual cases.

  • Detection setup: Figures 4 and 5 show COCO validation object detection using SSD-Lite with MobileViTv3-S and MobileViTv3-1.0 backbones.The examples include blurred people and complex backgrounds.

A.2 SEMANTIC SEGMENTATION ON PASCALVOC2012 DATASET

The appendix presents PascalVOC2012 semantic-segmentation outputs from Deeplabv3 using MobileViTv3-S and MobileViTv3-1.0 backbones, with inputs, predictions, and overlays shown in sequence.

  • Segmentation setup: Figures 6 and 7 show PascalVOC2012 validation segmentation using Deeplabv3 with MobileViTv3-S and MobileViTv3-1.0 backbones.Each sequence moves from the input image to the segmentation output and then its overlay on the input.
  • Model variants: The figure panels compare segmentation visualizations for MobileViTv3-S and MobileViTv3-1.0.The captions identify the model backbone used for each result set.
Loading 2209.15159v2…