PaperlayerView exact revision on arXiv

Source-linked AI summary

ResT: An Efficient Transformer for Visual Recognition

Qinglong Zhang, Yubin Yang

arXiv:2105.13677v5cs.CV

TL;DR

Existing Transformer backbones face low-level feature, computational, attention-head, and fixed-scale representation limitations in visual recognition. ResT addresses these issues with a hierarchical multi-scale backbone using efficient attention, spatial-attention positional encoding, and convolutional patch embedding. It is validated across image classification and downstream tasks, with ResT-Small reaching 79.6% Top-1 accuracy on ImageNet-1k at similar model size to ResNet-18 and PVT-Tiny.

  • Problem

    Existing Transformer backbones have difficulty extracting low-level features, incur quadratic multi-head self-attention costs, limit per-head embedding dimensions, and use fixed-scale tokens and positional encodings.

  • Method

    ResT is a hierarchical multi-scale Transformer backbone combining efficient attention, spatial-attention positional encoding, and overlapping convolutional patch embedding.

  • Results

    ResT is validated on ImageNet-1k, object detection, and instance segmentation, while ResT-Small achieves 79.6% Top-1 accuracy on ImageNet-1k at similar model size to ResNet-18 and PVT-Tiny.

  • Takeaways & Limitations

    The experiments demonstrate ResT's effectiveness and generalization ability as a strong backbone for visual recognition and dense prediction.

Abstract

from arXiv · show

This paper presents an efficient multi-scale vision Transformer, called ResT, that capably served as a general-purpose backbone for image recognition. Unlike existing Transformer methods, which employ standard Transformer blocks to tackle raw images with a fixed resolution, our ResT have several advantages: (1) A memory-efficient multi-head self-attention is built, which compresses the memory by a simple depth-wise convolution, and projects the interaction across the attention-heads dimension while keeping the diversity ability of multi-heads; (2) Position encoding is constructed as spatial attention, which is more flexible and can tackle with input images of arbitrary size without interpolation or fine-tune; (3) Instead of the straightforward tokenization at the beginning of each stage, we design the patch embedding as a stack of overlapping convolution operation with stride on the 2D-reshaped token map. We comprehensively validate ResT on image classification and downstream tasks. Experimental results show that the proposed ResT can outperform the recently state-of-the-art backbones by a large margin, demonstrating the potential of ResT as strong backbones. The code and models will be made publicly available at https://github.com/wofmanaf/ResT.

1 Introduction

ResT is introduced as an efficient, multi-scale Transformer backbone addressing limitations of existing CNN and Transformer architectures for visual recognition. It combines hierarchical processing with efficient attention, flexible positional encoding, and convolutional patch embedding, and reports 79.6% Top-1 accuracy on ImageNet-1k at similar model size to ResNet-18 and PVT-Tiny.

  • Background: CNNs and Transformers are the two common backbone families, both constructing representations by stacking multiple blocks.CNNs typically use bottleneck blocks, whereas Transformer blocks combine multi-head self-attention with feed-forward networks.
  • Background: Transformer backbones capture long-distance information and dynamically adapt their receptive fields according to image content.
  • Motivation: Existing Transformers struggle with low-level features, quadratic attention cost, limited per-head embedding dimensions, and fixed-scale tokens and positional encodings.These limitations affect structures such as edges and corners, computational overhead, attention-head interactions, and dense prediction tasks.
  • ResT: ResT follows a ResNet-like pipeline with a stem, four hierarchical stages, and a classification head for constructing multi-scale feature maps.Each stage contains patch embedding, positional encoding, and multiple Transformer blocks.
  • ResT: ResT uses overlapping convolutional patch embedding, spatial-attention positional encoding, and efficient attention to address fixed-scale processing and Transformer overheads.The patch embedding hierarchically expands channels while reducing spatial resolution, while spatial attention is conditioned on local neighborhoods.
  • Results: 79.6% Top-1 accuracy on ImageNet-1k is achieved by ResT-Small at a similar model size to ResNet-18 and PVT-Tiny.

2 ResT

ResT is a hierarchical vision Transformer that combines efficient attention, convolutional multi-scale patch embedding, and flexible positional encoding. These components target Transformer limitations in computational cost, low-level feature extraction, fixed token lengths, and positional encoding flexibility.

  • 2.5 Classification Head: ResT’s stages combine patch embedding, positional encoding, and efficient Transformer blocks at specific spatial resolutions and channel dimensions.The classification architecture includes four models—ResT-Lite, ResT-Small, ResT-Base, and ResT-Large—using global average pooling and a linear classifier.
  • 2.1 Rethinking of Transformer Block: Standard Transformer blocks use MSA and FFN with residual connections and layer normalization, while MSA models global relationships between token sequences.MSA produces queries, keys, and values through multiple head-specific projections; FFN expands and then reduces embedding dimensions.
  • 2.1 Rethinking of Transformer Block: MSA has quadratic computation in spatial or embedding dimensions, and each head sees only a subset of embedding dimensions.These limitations create training and inference overhead and may impair performance when per-head embeddings are short.
  • 2.2 Efficient Transformer Block: ResT replaces MSA with EMSA, which reduces token-map resolution using depth-wise convolution, projects keys and values from the reduced map, and models interactions across attention heads.Instance normalization after the softmax restores the diversity of multi-head attention, while the outputs are concatenated and linearly projected.
  • 2.4 Positional Encoding: ResT constructs positional encoding as spatial attention, allowing variable-length encodings conditioned on the input rather than fixed learned position parameters.The PA module uses a 3 × 3 depth-wise convolution followed by a sigmoid to produce pixel-wise weights, and it can be embedded into patch embedding.

3 Experiments

ResT is evaluated on ImageNet-1k classification, COCO detection, and instance segmentation, alongside ablations of its stem, attention reduction, head interaction, normalization, and positional encoding. Across benchmarks, ResT improves over comparable Transformer and CNN backbones, while ablations support its main design choices.

  • Benchmark evaluation: ResT is evaluated on ImageNet-1k classification and COCO object detection and instance segmentation using established benchmark settings.The experiments compare ResT with state-of-the-art Transformer and ConvNet backbones and include ablations of its design elements.
  • Object detection: +3.6 box AP is obtained by ResT-Small over PVT-T for RetinaNet on COCO, while ResT-Base exceeds PVT-S by +1.6 box AP.The reported comparisons use models with similar computation cost or slightly larger model size.
  • Instance segmentation: +2.9 box AP and +2.1 mask AP are obtained by ResT-Small over PVT-T for Mask RCNN on COCO val2017.For larger models, ResT-Base gains +1.2 box AP and +0.9 mask AP over PVT-S.
  • Ablation study: Removing Instance Normalization degrades Top-1 accuracy by 0.9%, while removing both the 1 × 1 convolution and normalization causes a 1.16% drop.The ablation attributes the normalization effect to preserving diversity among attention heads.
  • Ablation study: PA positional encoding improves Top-1 accuracy by 0.84% over GL, while removing PA reduces accuracy from 72.88% to 71.54%.The results indicate that positional encoding is important and that spatial attention can serve as positional encoding.

4 Conclusion

ResT is presented as a multi-scale Transformer backbone that produces hierarchical feature representations for dense prediction. Its efficient attention compresses memory, models interaction among heads while preserving diversity, and uses spatial attention for arbitrary-sized inputs.

  • 4 Conclusion: ResT produces hierarchical feature representations for dense prediction as a multi-scale Transformer backbone.Its design compresses standard MSA memory, models interaction between multi-heads while preserving diversity, and redesigns positional encoding as spatial attention.

A Appendix

The appendix supplies related work and additional experimental results that complete the main paper's experimental sections.

  • A Appendix: The appendix provides related work and additional experimental results to complete the main paper.

A.1 Related Work

Related work contrasts CNNs’ local-information aggregation with Transformers’ global self-attention and describes how vision Transformers represent images as patch-token sequences.

  • Convolutional Networks: ResNet backbones use residual bottlenecks built from 1 × 1, 3 × 3, and 1 × 1 convolution layers.
  • Convolutional Networks: CNNs emphasize parameter sharing and local aggregation, whereas self-attention provides a global receptive field and aggregates global information.
  • Vision Transformers: Vision Transformers treat images as patch sequences processed by encoders composed of multi-head self-attention and feed-forward network blocks.
  • Vision Transformers: ViT uses fixed-size patch tokenization, while T2T-ViT identifies that simple tokenization misses local structures such as edges and lines.
  • Positional Encoding: Positional encoding is essential because Transformer self-attention cannot distinguish token order across spatial positions.

A.2 Visualization and Interpretation

The paper analyzes EMSA attention diversity and visualizes ResT attention, showing how convolution and instance normalization affect head behavior while heatmaps adapt to image content.

  • Analysis on EMSA: EMSA diversity is measured from attention maps before convolution, after convolution, and after instance normalization across 1,000 ImageNet-1k validation images.
  • Visualization: Figure 5 visualizes attention maps from the last blocks of stage 4 in ResT-Lite.
  • Analysis on EMSA: The 1 × 1 convolution models interactions among heads but impairs joint attention across representation subsets at different positions.
  • Analysis on EMSA: Instance normalization restores the diversity ability lost after the 1 × 1 convolution.
  • Interpretability: Group-CAM heatmaps compare ResT-Lite with ResNet-50 on six ImageNet-1k validation images.
  • Interpretability: ResT-Lite adaptively produces heatmaps according to image content.

A.3 More Experiments

Additional experiments evaluate EMSA against standard MSA and test layer normalization in object detection, reporting efficiency and downstream accuracy effects.

  • EMSA versus MSA: EMSA achieves better Top-1 performance than MSA on ResT-Lite while both versions use almost the same number of parameters.
  • EMSA versus MSA: Table 9 compares the performance and efficiency of MSA and EMSA on ResT-Lite.
  • Object Detection: Layer normalization improves RetinaNet by an average +0.8 box AP in downstream object detection.

A.4 Discussions

The discussion defines GL and explains the controlled ablation setting used to assess ResT components and reduce the influence of training tricks.

  • Mathematical Definition of GL: GL splits an input x with spatial and channel dimensions into g non-overlapping groups.
  • Mathematical Definition of GL: Each GL group is transformed by a separate linear operation, with W_i denoting that operation’s weight.
  • Mathematical Definition of GL: The transformed group outputs are concatenated into y, and ResT sets g equal to the input channel dimension d_m.
  • Ablation Study Settings: The ablation study uses ResNet-style data augmentation and hyperparameters to investigate ResT components without strong augmentation or training tricks.
  • Ablation Study Settings: The authors argue that this controlled setting addresses, to some extent, doubts that Vision Transformer gains over CNNs mainly arise from strong augmentation and training tricks.
Loading 2105.13677v5…