PaperlayerView exact revision on arXiv

Source-linked AI summary

CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applications

Tianfang Zhang, Lei Li, Yang Zhou, Wentao Liu, Chen Qian, Jenq-Neng Hwang, Xiangyang Ji

arXiv:2408.03703v2cs.CV

TL;DR

ViTs offer global context, but their computational demands and complex operators challenge mobile and real-time deployment. CAS-ViT uses convolutional additive attention in a hybrid lightweight network family, reporting results across classification and segmentation tasks alongside detection. The authors present it as balancing performance and efficient deployment.

  • Problem

    ViT’s computational complexity and complex operators complicate resource-constrained deployment, while low-complexity networks have not achieved a satisfactory balance between efficiency and accuracy.

    Source: §I · p. 1

  • Method

    CAS-ViT uses a convolutional additive token mixer with spatial and channel interactions in hybrid blocks, avoiding matrix multiplication and Softmax.

    Source: §I · p. 2, B. Convolutional Additive Token Mixer · p. 4, C. Convolutional Additive… · p. 5

  • Results

    CAS-ViT reports results across classification, object detection and instance segmentation, and semantic segmentation, including 45.0% and 43.6% mIoU for its M and T models.

    Source: 2) Results · pp. 6, 8, 9

  • Takeaways & Limitations

    The authors conclude that the lightweight models show performance across several vision tasks while targeting efficient inference and convenient deployment.

    Source: §I · p. 2, §VI · p. 10

    Convolution layers retain inductive bias, and CAS-ViT may be slightly less effective on large-scale datasets and large parametric models.

    Source: §V · p. 10

Abstract

from arXiv · show

Vision Transformers (ViTs) mark a revolutionary advance in neural networks with their token mixer's powerful global context capability. However, the pairwise token affinity and complex matrix operations limit its deployment on resource-constrained scenarios and real-time applications, such as mobile devices, although considerable efforts have been made in previous works. In this paper, we introduce CAS-ViT: Convolutional Additive Self-attention Vision Transformers, to achieve a balance between efficiency and performance in mobile applications. Firstly, we argue that the capability of token mixers to obtain global contextual information hinges on multiple information interactions, such as spatial and channel domains. Subsequently, we propose Convolutional Additive Token Mixer (CATM) employing underlying spatial and channel attention as novel interaction forms. This module eliminates troublesome complex operations such as matrix multiplication and Softmax. We introduce Convolutional Additive Self-attention(CAS) block hybrid architecture and utilize CATM for each block. And further, we build a family of lightweight networks, which can be easily extended to various downstream tasks. Finally, we evaluate CAS-ViT across a variety of vision tasks, including image classification, object detection, instance segmentation, and semantic segmentation. Our M and T model achieves 83.0\%/84.1\% top-1 with only 12M/21M parameters on ImageNet-1K. Meanwhile, throughput evaluations on GPUs, ONNX, and iPhones also demonstrate superior results compared to other state-of-the-art backbones. Extensive experiments demonstrate that our approach achieves a better balance of performance, efficient inference and easy-to-deploy. Our code and model are available at: \url{https://github.com/Tianfang-Zhang/CAS-ViT}

I. INTRODUCTION

ViTs capture global context, but their computational and operator demands complicate mobile deployment, while CNNs trade restricted receptive fields for efficient computation. CAS-ViT proposes lightweight networks intended to balance performance and deployment efficiency.

A. Efficient Vision Transformers

  • ViT’s global modeling complements CNNs’ lower computational complexity, motivating hybrid approaches for efficient vision tasks.The introduction describes token mixers as capturing long-range dependencies and global context.Source: §I · p. 1, A. Efficient Vision Transformers · p. 2

I. INTRODUCTION

  • ViT’s quadratic-complexity self-attention and complex operators challenge mobile deployment, while CNNs’ restricted receptive fields can limit feature extraction and performance.The paper frames efficient deployment together with powerful performance as an unresolved need for mobile and real-time applications.Source: §I · p. 1
  • Prior token-mixer work refines self-attention or uses heterogeneous designs, including pool token mixers and context vectors to accelerate inference.The cited approaches include feature shift, carrier tokens, sparse attention, and linear attention.Source: §I · p. 1
  • CAS-ViT combines a token mixer using spatial and channel interactions with a hybrid block architecture, forming a lightweight network family for downstream tasks.The proposed CATM avoids matrix multiplication and Softmax; the CAS block uses CATM in each block.Source: §I · p. 2

A. Efficient Vision Transformers

  • Efficient ViT research refines token mixers to address self-attention’s quadratic complexity and combines CNN and Transformer components to compensate for limitations in local information processing.Examples include spatial reduction in PVT and windowed attention in Swin.Source: A. Efficient Vision Transformers · p. 2

B. Efficient Token Mixer

Efficient token mixers seek to reduce computation while retaining useful feature interactions. The paper situates CATM among prior mixer designs and introduces the methods used to explain its distinctions and the CAS-ViT architecture.

B. Efficient Token Mixer

  • Token-mixer research includes MSA variants that retain Q–K multiplication and designs that reduce complexity through approaches such as linear attention.Examples include combining global and local dependencies, channel integration, and increasing attention heads.Source: B. Efficient Token Mixer · pp. 2–3
  • Prior heterogeneous mixers simplify attention in different ways: MobileViTv2 uses a context vector, while SwiftFormer removes Value and uses weighted feature sums.The paper motivates its design through interactions across spatial and channel domains.Source: B. Efficient Token Mixer · p. 3

III. METHODS

  • The paper’s methods section reviews MSA and its variants, then presents CATM and the overall CAS-ViT network architecture.The stated focus is CATM’s distinctions and advantages over traditional mechanisms.Source: §III · p. 3

A. Overview of Self-Attention and Variants

The overview contrasts standard self-attention with separable and swift variants that use reduced feature representations or context vectors. Figure 3 also depicts a four-stage classification backbone whose blocks are stacked within each stage.

A. Overview of Self-Attention and Variants

  • Standard self-attention models relationships between positions, while separable self-attention reduces the feature metric to a vector and aggregates a context vector across spatial positions.The separable variant calculates context scores with Softmax.Source: A. Overview of Self-Attention and… · p. 3
  • Swift self-attention reduces the keys to two, weights tokens using coefficients derived from transformed Q, then aggregates spatial context with K.The passage identifies normalized Query and a linear transformation in its formulation.Source: A. Overview of Self-Attention and… · p. 3
  • The classification backbone has four downsampling stages, with Ni blocks stacked in each stage.The figure also shows the block architecture in its lower panel.Source: Figure 3 · p. 4

B. Convolutional Additive Token Mixer

CATM replaces pairwise token affinity with spatial and channel interactions, then combines context scores from Query and Key additively. Its convolutional design avoids matrix multiplication and Softmax while retaining linear complexity in input size.

B. Convolutional Additive Token Mixer

  • The design targets global context through interactions across spatial and channel domains, rather than relying on pairwise Query comparisons with quadratic complexity.The paper notes that convolutional stacking and multiple information interactions can achieve a similar receptive-field effect to attention.Source: B. Convolutional Additive Token Mixer · pp. 4–5
  • CATM forms spatial attention with a 3×3 convolution, 1×1 projection, and Sigmoid, then combines spatial and channel interactions into Φ(x).The channel interaction uses a 1×1 convolution to integrate information between channels; the stacked operations produce Φ(x).Source: B. Convolutional Additive Token Mixer · p. 4
  • CATM defines similarity as the sum of context scores from Query and Key, integrating them additively to avoid high-complexity matrix multiplication.The additive combination is described as retaining useful information.Source: B. Convolutional Additive Token Mixer · p. 4
  • CATM has linear complexity with input size and comprises QKV separation convolution, Φ(Q), Φ(K), and a final mapping layer Γ.Spatial and channel interactions are implemented with depthwise convolution and Sigmoid activation, using a 3×3 kernel.Source: B. Convolutional Additive Token Mixer · p. 5

C. Convolutional Additive Self-attention Block

The CAS block is the network’s basic structure and is stacked at each stage. It combines an integration subnet, CATM, and MLP with residual shortcuts.

C. Convolutional Additive Self-attention Block

  • Each CAS block combines an integration subnet, CATM, and MLP, with residual shortcuts.The block is stacked in each stage of the network.Source: C. Convolutional Additive… · p. 5
  • Three ReLU-activated 3×3 depthwise convolutions in the integration subnet enlarge the receptive field, integrate local information, and provide positional encoding.CATM is used in all blocks at every network stage; the convolution-based block is described as balancing computational efficiency and deployment.Source: C. Convolutional Additive… · p. 5

D. Network Architecture

CAS-ViT uses a convolutional stem followed by four encoding stages, with patch-embedding downsampling between stages. The model family varies stage channel counts and block counts.

D. Network Architecture

  • The stem downsamples an H × W × 3 image to H/4 × W/4 × C1 using two consecutive 3×3 stride-2 convolutions.The stem and resulting feature-map dimensions are specified across the architecture description.Source: D. Network Architecture · p. 5
  • Four encoding stages use patch-embedding downsampling between stages, producing feature maps at H/8, H/16, and H/32 spatial scales.Each stage contains Ni stacked blocks, and the feature-map size remains constant within a stage.Source: D. Network Architecture · p. 5
  • CAS-ViT variants are formed by varying the channel counts Ci and block counts Ni across stages.Table I presents network configurations for the CAS-ViT variants.Source: D. Network Architecture · p. 5, Table I · p. 5

IV. EXPERIMENTS

The experiments describe ImageNet-1K training and evaluation across classification and robustness benchmarks. CAS-ViT-M/T report higher ImageNet-1K top-1 accuracy than several compared backbones, while the XS/S variants are described as efficient under limited computation.

1) Implementation Details:

  • ImageNet-1K training uses 224×224 inputs for 300 epochs, followed by 30-epoch finetuning at 384×384; training uses no additional data.RegNetY-16GF is used as a distillation teacher, and the models are trained from scratch before finetuning.Source: 1) Implementation Details · p. 5
  • Throughput is measured on V100 GPUs, an Intel Xeon CPU using ONNX, and an iPhone X neural engine using CoreML.The classification implementation uses PyTorch and runs on 16 V100 GPUs.Source: 1) Implementation Details · p. 5

2) Results:

  • 83.0% and 84.1% top-1 accuracy after further training for CAS-ViT-M/T, surpassing SwiftFormer by 3.2% and 1.1%, respectively.The reported initial accuracies are 82.8% and 83.9%; CAS-ViT-M also exceeds MobileViTv2-2.0 by 1.8%, and CAS-ViT-T exceeds Swin-B by 0.6% with 24% fewer parameters.Source: Table II · p. 6, 2) Results · p. 6
  • The XS and S variants are reported to combine parameter count and FLOPs efficiently while maintaining performance under limited computation.The supplied robustness table caption identifies ImageNet-A/R/Sketch top-1 accuracy and ImageNet-O AUPR as its comparison metrics, with higher values better.Source: 2) Results · p. 6, Table III · p. 6

3) Computational Efficiency:

CAS-ViT is evaluated for computational efficiency across GPU, ONNX, and mobile platforms, and for robustness on additional ImageNet benchmarks. The reported comparisons emphasize accuracy and computational trade-offs, while noting that some methods have higher throughput.

3) Computational Efficiency:

  • CAS-ViT is deployed and evaluated for throughput on GPU, ONNX, and Apple neural-engine platforms.The provided Table IV caption specifies FLOPs measured at 800 × 1280 image size.Source: 3) Computational Efficiency · p. 6, Table IV · p. 7
  • Under similar parameter counts, CAS-ViT-S is reported to have faster throughput and 2.7% higher performance than EMO-5M across platforms; CAS-ViT-M has 3.1% higher performance than GCViT-XXT.Against EfficientViT and MobileViTv2, throughput falls somewhat short, while the paper describes CAS-ViT as more favorable in the performance trade-off.Source: 3) Computational Efficiency · pp. 6–7, Table IV · p. 7
  • Across platforms, CAS-ViT is reported to maintain competitive computational efficiency, with higher or comparable performance at lower computational overhead in Flops and parameter counts.The Table IV caption identifies COCO detection and instance-segmentation performance and states that the reported results achieve better performance with less computational overhead.Source: 3) Computational Efficiency · pp. 6–7, Table IV · p. 7

4) Effective Robustness:

  • On ImageNet-A, CAS-ViT-M and T score 22.1% and 29.6%; on ImageNet-Sketch, CAS-ViT-T is 0.2% more accurate than EfficientFormer-L7 using 26.4% of its parameters and 35.3% of its computational effort.The robustness evaluation uses models without finetuning or processing at 224×224 image size.Source: 4) Effective Robustness · p. 7

B. Object Detection and Instance Segmentation

The evaluation uses ImageNet-pretrained CAS-ViT backbones for COCO object detection and instance segmentation, and also describes a separate ADE20K semantic-segmentation setup. On COCO, the reported comparisons show strong AP performance against several backbones.

1) Implementation Details:

  • COCO detection and instance-segmentation models use ImageNet-1K-pretrained backbones, RetinaNet or Mask R-CNN, and a 12-epoch finetuning schedule.Images are resized with short side 800 pixels and long side no more than 1333 pixels.Source: 1) Implementation Details · pp. 7–8

2) Results:

  • The COCO evaluation compares detection and segmentation AP across small, medium, and large scales alongside parameters and FLOPs.The supplied passage describes the evaluation scope but does not provide the scale-specific values.Source: 2) Results · p. 8
  • With similar parameter count, CAS-ViT-T reaches 43.5/39.6 AP box/mask with Mask R-CNN, exceeding SwiftFormer-L3 by 0.8/0.5.The paper also reports advantages over ConvNet and Swin-T, and CAS-ViT-S exceeds EfficientFormer-L1 by 1.9/1.3 with Mask R-CNN.Source: 2) Results · p. 8

1) Implementation Details:

  • The ADE20K semantic-segmentation setup uses Semantic FPN with frozen normalization layers and ImageNet-1K-pretrained weights.Training runs for 40K iterations with batch size 32 and 512×512 image crops; the supplied table caption identifies the results as ADE20K with Semantic FPN and FLOPs at 512 × 512.Source: 1) Implementation Details · p. 8, Table VI · p. 8

2) Results:

The results report efficient ADE20K semantic segmentation, broader heatmap coverage, and qualitative detection and segmentation differences from PoolFormer. An ablation study also compares CATM and alternative similarity interactions.

2) Results:

  • 45.0% and 43.6% mIoU are reported for CAS-ViT-M/T on ADE20K, outperforming recent methods including PVT, EfficientFormer, and SwiftFormer with fewer parameters.The paper describes the models as achieving a balance of computational efficiency and segmentation accuracy despite relatively lower parameters and FLOPs.Source: 2) Results · pp. 8–9

1) Heatmap:

  • Compared with PoolFormer and SwiftFormer, CAS-ViT heatmaps are reported to cover a wider range while localizing key information accurately.The authors relate this wider information association to convolution and report that it preserves global perceptual ability to a great extent.Source: 1) Heatmap · p. 9

2) Detection and Segmentation Results:

  • Compared with PoolFormer, the qualitative examples report fewer false detections, better detection of edge or shaded objects, and more complete bounding boxes for large objects.The examples include a skateboard and cat for false detections, a bus for edge and shading, and elephants for bounding-box completeness.Source: 2) Detection and Segmentation Results · p. 9, Figure 5 · p. 9

2) Results:

  • The ablation study compares CATM with other modules and the similarity function with alternative interactions.The supplied caption identifies the study scope but does not report its outcomes.Source: Table VII · p. 9

E. Ablation Study

The ablations favor CATM over simpler mixer alternatives and show the value of spatial and channel interactions. The conclusion reports deployment across vision tasks while noting weaker effectiveness at larger scales.

E. Ablation Study

  • Pooling reduces FLOPs but causes significant performance loss, whereas CATM achieves higher accuracy with lower FLOPs than W-MSA.Source: E. Ablation Study · pp. 9–10
  • Omitting spatial and channel interactions decreases results by 0.45% and 0.21%, respectively; full interaction performs similarly across tested arrangements, so their order is not particularly required.Source: E. Ablation Study · p. 10

V. LIMITATION AND FUTURE WORK

  • Extensive convolution retains inductive bias and supports faster convergence than vanilla Transformers, but CAS-ViT is slightly less effective on large-scale datasets and large models.The authors identify larger datasets and model scales as future work.Source: §V · p. 10

VI. CONCLUSION

  • CAS-ViT maintains high accuracy while facilitating efficient mobile deployment and inference across image classification, object detection, and instance and semantic segmentation.Source: §VI · p. 10

Related papers

Loading 2408.03703v2…