Source-linked AI summary
Lite Vision Transformer with Enhanced Self-Attention
Chenglin Yang, Yilin Wang, Jianming Zhang, He Zhang, Zijun Wei, Zhe Lin, Alan Yuille
TL;DR
Lightweight vision transformers can produce inconsistent local dense predictions, motivating stronger attention in shallow, narrow networks. LVT addresses this with CSA for low-level features and RASA for high-level features, and demonstrates strong performance across recognition and segmentation tasks while remaining subject to lightweight-model capacity limits.
Problem
Lightweight vision transformers still suffer from inconsistent and incorrect local dense predictions, while their performance is limited by parameter count and model depth.
Method
LVT is a four-stage lightweight transformer backbone using CSA in the first stage and RASA in the last three stages for low- and high-level features.
Results
LVT demonstrates strong performance against previous mobile methods on ImageNet recognition, ADE20K semantic segmentation, and COCO panoptic segmentation.
Takeaways & Limitations
LVT provides a lightweight vision-transformer backbone with enhanced self-attention mechanisms for mobile vision tasks.
Takeaways & Limitations
As a lightweight model, LVT has weaker representation power than models with larger numbers of parameters.
Abstract
from arXiv · showhide
Despite the impressive representation capacity of vision transformer models, current light-weight vision transformer models still suffer from inconsistent and incorrect dense predictions at local regions. We suspect that the power of their self-attention mechanism is limited in shallower and thinner networks. We propose Lite Vision Transformer (LVT), a novel light-weight transformer network with two enhanced self-attention mechanisms to improve the model performances for mobile deployment. For the low-level features, we introduce Convolutional Self-Attention (CSA). Unlike previous approaches of merging convolution and self-attention, CSA introduces local self-attention into the convolution within a kernel of size 3x3 to enrich low-level features in the first stage of LVT. For the high-level features, we propose Recursive Atrous Self-Attention (RASA), which utilizes the multi-scale context when calculating the similarity map and a recursive mechanism to increase the representation capability with marginal extra parameter cost. The superiority of LVT is demonstrated on ImageNet recognition, ADE20K semantic segmentation, and COCO panoptic segmentation. The code is made publicly available.
1. Introduction
LVT targets compact mobile vision models with two specialized self-attention mechanisms: CSA for low-level features and RASA for high-level features. It is evaluated across recognition and segmentation tasks.
- Motivation: LVT is designed as a compact transformer backbone for mobile applications, balancing performance and model size.It follows a four-stage structure and has a parameter size similar to MobileNetV2 and PVTv2-B0.
- Convolutional Self-Attention: CSA inserts local self-attention into a 3 × 3 convolution kernel to enrich low-level features in LVT’s first stage.It combines dynamic kernels with learnable filters rather than merging convolution with global self-attention.
- Recursive Atrous Self-Attention: RASA combines multi-scale similarity computation with recursion to increase high-level representation capacity without additional parameters.Its Atrous Self-Attention component captures multi-scale context, while the recursive formulation uses weight sharing.
- Evaluation: LVT is evaluated on ImageNet recognition, ADE20K semantic segmentation, and COCO panoptic segmentation as a generalized vision backbone.The paper reports strong performance compared with previous mobile methods across these tasks.
- Architecture: LVT uses CSA in the first stage and RASA in the last three stages to process low- and high-level features, respectively.The backbone retains a standard four-stage design while assigning the two attention mechanisms to different feature levels.
2. Related Work
Prior vision-transformer work explores tokenization, positional encoding, multi-scale processing, and efficient attention. LVT instead investigates recursive self-attention with multi-scale query information for a lightweight general backbone.
- Vision Transformer: Vision transformers represent images as sequences of embedded patches and have been extended through distillation, recursive token aggregation, smaller tokens, and dynamic position encoding.The cited methods target training, tokenization, or positional-embedding improvements.
- Multi-scale Processing: Other vision-transformer methods investigate multi-scale processing through local and global attention, convolutional position embedding, or cross-attention across feature scales.These approaches address representation across different spatial or feature resolutions.
- LVT’s Position: Unlike the cited approaches, LVT uses a recursive self-attention layer with multi-scale query information in a lightweight vision-transformer backbone.The related-work passage characterizes this as improving mobile-model performance effectively.
3. Lite Vision Transformer
LVT is a lightweight four-stage vision transformer that assigns CSA to early low-level processing and RASA to later high-level processing. CSA combines local self-attention with convolution, while RASA adds multi-scale context and recursive computation with limited parameter overhead.
- LVT architecture: LVT uses a four-stage architecture, with a 3 × 3 CSA layer in the first stage and global-kernel RASA layers in the last three stages.The stages progressively reduce resolution from stride-4 to stride-32.
- Convolutional Self-Attention: In a 3 × 3 window, CSA produces 3 × 3 outputs through nine input-dependent weighted summations, whereas convolution produces a 1 × 1 output.The shared BMM operation preserves the convolutional projection while self-attention supplies dynamic summation weights.
- Recursive Atrous Self-Attention: ASA builds multi-scale queries with shared-kernel depth-wise convolutions at dilation rates 1, 3, and 5, then uses self-calibrated scale weights in similarity computation.The resulting similarity map weights the summation of values across spatial locations.
- Convolutional Self-Attention: CSA integrates local self-attention into a convolution kernel to combine input-dependent kernels with learnable filters for low-level features.Its formulation generalizes convolution and self-attention, with both methods appearing as special cases under particular weight settings.
- Recursive Atrous Self-Attention: RASA combines ASA with recursive self-attention, using a hidden state and a recursion depth of two to increase representation capacity without adding projection parameters.The method sets the projection weights to one and uses ASA as the nonlinear activation function.
- Evaluation: LVT is evaluated on ImageNet classification, ADE20K semantic segmentation, and COCO panoptic segmentation, with reported gains on mobile segmentation benchmarks.The architecture table describes CSA and RASA as enhanced self-attention layers for low- and high-level features, respectively.
4. Experiments
LVT is evaluated as a compact vision backbone across ImageNet recognition, ADE20K semantic segmentation, and COCO panoptic segmentation. The experiments report strong recognition and segmentation performance under mobile-model constraints.
- ImageNet Classification: The ImageNet experiments constrain the encoder to fewer than 3.5M parameters and compare LVT with standard mobile and ResNet50-scale models.The encoder is emphasized because it serves as the backbone for detection and segmentation.
- ImageNet Classification: LVT demonstrates high performance for ImageNet recognition under the reported mobile-oriented encoder-size setting.
- COCO Panoptic Segmentation: COCO panoptic segmentation evaluates LVT through a task that jointly covers object recognition, detection, localization, and segmentation.The experiments use the COCO 2017 split with 118K training images and 5K validation images.
5. Ablation Studies
The ablation studies examine recursion depth and the separate contributions of CSA and RASA. Performance increases dramatically with two recursion iterations, while both attention mechanisms contribute to improvement.
- Recursion Depth: Performance increases dramatically with two recursion iterations on ImageNet classification, so the main experiments use LVT R2 for efficiency.
- CSA and RASA Contributions: Both CSA and RASA significantly contribute to performance improvement in the ablation experiments.The comparison uses VOLO as the base network and evaluates contributions on ImageNet and ADE20K.
6. Conclusion
LVT combines CSA and RASA in a lightweight transformer backbone for low- and high-level features. The paper reports strong performance across recognition, semantic segmentation, and panoptic segmentation, while noting the limitation of weaker representation power than larger models.
- Conclusion: LVT uses CSA in the first stage and RASA in the last three stages to process low- and high-level features.
- Conclusion: LVT demonstrates strong performance compared with previous mobile methods on visual recognition, semantic segmentation, and panoptic segmentation.
- Limitations: LVT has weaker representation power than models with large numbers of parameters because it is designed as a lightweight model.The stated focus is mobile models, with future work including scaling LVT to larger backbones.