PaperlayerView exact revision on arXiv

Source-linked AI summary

Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention

Xuran Pan, Tianzhu Ye, Zhuofan Xia, Shiji Song, Gao Huang

arXiv:2304.04237v1cs.CV

TL;DR

Global attention’s quadratic computation is costly for high-resolution vision, while sparse and window patterns can compromise local features or add design constraints. Slide Attention implements local attention with depthwise convolutions and deformed shifting. The authors report improved performance across Vision Transformer models and hardware, including an iPhone 12 speed comparison.

  • Problem

    Global attention has quadratic complexity, while sparse and window attention can compromise local feature learning or require handcrafted designs.

    Source: §1 · pp. 1–2

  • Method

    Slide Attention reframes Im2Col as row-wise feature shifts, replaces shifts with depthwise convolutions, and adds deformed shifting to relax fixed key/value positions.

    Source: §1 · p. 2

  • Results

    Slide Attention is reported to improve over baselines across tasks, models, and hardware, with Swin-small outperforming vanilla Swin-base and achieving 1.7x inference speedup on iPhone 12.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The module is reported to work across a variety of Vision Transformers and hardware devices while offering a better trade-off between computation efficiency and model performance.

    Source: §6 · p. 8

    Increasing the Slide Attention window size yields only marginal improvements, and a better performance-efficiency trade-off for larger windows remains future work.

    Source: F. Limitations · p. 10

Abstract

from arXiv · show

Self-attention mechanism has been a key factor in the recent progress of Vision Transformer (ViT), which enables adaptive feature extraction from global contexts. However, existing self-attention methods either adopt sparse global attention or window attention to reduce the computation complexity, which may compromise the local feature learning or subject to some handcrafted designs. In contrast, local attention, which restricts the receptive field of each query to its own neighboring pixels, enjoys the benefits of both convolution and self-attention, namely local inductive bias and dynamic feature selection. Nevertheless, current local attention modules either use inefficient Im2Col function or rely on specific CUDA kernels that are hard to generalize to devices without CUDA support. In this paper, we propose a novel local attention module, Slide Attention, which leverages common convolution operations to achieve high efficiency, flexibility and generalizability. Specifically, we first re-interpret the column-based Im2Col function from a new row-based perspective and use Depthwise Convolution as an efficient substitution. On this basis, we propose a deformed shifting module based on the re-parameterization technique, which further relaxes the fixed key/value positions to deformed features in the local region. In this way, our module realizes the local attention paradigm in both efficient and flexible manner. Extensive experiments show that our slide attention module is applicable to a variety of advanced Vision Transformer models and compatible with various hardware devices, and achieves consistently improved performances on comprehensive benchmarks. Code is available at https://github.com/LeapLabTHU/Slide-Transformer.

1. Introduction

Global self-attention is costly for high-resolution vision, while sparse and window attention can compromise local features or constrain model design. Slide Attention replaces inefficient local-attention implementation with convolution-based operations and deformed shifting, and is reported to improve performance across models and hardware settings.

1. Introduction

  • Global self-attention’s quadratic complexity creates excessive computation costs for high-resolution vision inputs.The paper identifies this as a central obstacle to adapting Transformers to vision.Source: §1 · p. 1
  • Sparse global attention can miss local features, while window attention can hinder cross-window communication and require extra design choices.Source: §1 · p. 2
  • Slide Attention reinterprets Im2Col row-wise, substitutes shifts with depthwise convolutions, and uses deformed shifting to relax fixed key/value positions.The deformed shifting module uses re-parameterization to increase model capacity while preserving inference efficiency.Source: §1 · p. 2
  • Experiments cover image classification, semantic segmentation, and object detection under five Vision Transformer models.Source: §1 · p. 2
  • Slide Attention is reported to improve over all baselines, and Swin-small with Slide Attention outperforms vanilla Swin-base with 1.7x inference speedup on iPhone 12.The paper also reports efficiency on devices without CUDA support, including Metal Performance Shader.Source: §1 · p. 2

2. Related Works

Related work adapts Vision Transformers through changes to model structure and attention patterns. Local attention offers a different route, but existing implementations trade efficiency against hardware portability.

2.1. Vision Transformer

  • PVT and DAT use sparse key/value locations, MViT uses pooling, and Swin and CSwin constrain attention to designed window patterns.Source: §2.1 · pp. 2–3

2.2. Local Attention

  • Local attention methods include block-neighborhood attention and CUDA-kernel implementations, while Im2Col-based methods are inefficient and CUDA kernels limit use on CUDA-free devices.The passage names HaloNet, SAN, and NAT as examples of approaches to local attention.Source: §2.2 · p. 3

3. Overview of Self-Attention

The overview frames attention design as a trade-off between feature extraction and computational efficiency. It introduces multi-head self-attention, contrasts attention patterns, and explains why local attention remains difficult to implement efficiently.

3.1. Multi-Head Self-Attention

  • An MHSA block computes attention across M heads and concatenates their outputs; each query’s receptive field determines its corresponding key/value pairs.The formulation defines σ as SoftMax and d as the channel dimension of each head.Source: §3.1 · p. 3

3. Overview of Self-Attention

  • Self-attention’s flexibility increases computation complexity, motivating attention patterns that inject inductive biases to improve the performance-efficiency trade-off.Source: §3.2 · p. 3, §3 · p. 3

3.2. Attention Patterns

  • Sparse global attention shares selected key/value pairs across queries, window attention can isolate nearby features across window edges, and local attention uses each query’s neighbors.Local attention combines local inductive bias and translation-equivariance with query-centric attention and little human design.Source: §3.2 · p. 3

3.3. Local Attention Implementation

  • Im2Col sampling breaks data locality and is time-consuming, while CUDA-kernel implementations restrict use on devices without CUDA support.Figure 3 contrasts column-based sampling, row-based shifting, and depthwise-convolution substitution; the paper reports that Im2Col local attention is less favorable in efficiency and performance, while CUDA approaches offer comparable speed but only marginal improvements.Source: §3.3 · pp. 3–4, Figure 3 · p. 4

4. Method

The method recasts local attention’s Im2Col sampling as directional shifts, implements those shifts with depthwise convolutions, and adds a learnable deformed path. Together, these designs form Slide Attention, a flexible local-attention module intended for varied Transformer architectures and hardware.

4. Method

  • Slide Attention combines efficient depthwise-convolution shifts with a learnable deformed shifting module to provide efficient, flexible local attention.The module is presented as a plug-in with little restriction on architecture design.Source: §4 · p. 4, §4.4 · p. 6

4.1. New Perspective on Im2Col

  • The row-based view represents the key/value matrix as k^2 rows, each corresponding to shifting the input in a direction, and yields the same local windows as Im2Col.For k = 3, the construction shifts features in nine directions, flattens them into rows, then concatenates them in columns.Source: §4.1 · p. 5

4.2. Shift as Depthwise Convolution

  • For a local window of size k, k^2 depthwise convolutions can replace feature shifts and be reduced to one group convolution.This implementation avoids inefficient slicing and can benefit from optimized convolution operations on many hardware platforms.Source: §4.2 · p. 5

4.3. Deformed Shifting Module

  • A parallel path with learnable kernels relaxes fixed key/value positions into deformed features formed from local features.At inference, re-parameterization merges the designed-shift and learnable paths into one convolution while maintaining inference efficiency.Source: §4.3 · pp. 5–6, Figure 4 · p. 5

4.4. Implementation

  • The module is implemented as a plug-in for multiple modern vision Transformer architectures and used at their early stages while the rest of each block stays unchanged.The paper reports implementations on Nvidia GPU, Metal Performance Shader, and iPhone 12.Source: §4.4 · p. 6

5. Experiments

Experiments evaluate Slide Attention on ImageNet-1K classification, ADE20K segmentation, and COCO detection, compare it with other local-attention modules, and test runtime and design choices. Across these settings, the reported results show improvements, including stronger small-object detection and favorable runtime trade-offs.

5.1. ImageNet-1K Classification

  • In classification results, the method consistently improves over baseline models under comparable FLOPs or parameters; based on PVT, it achieves 0.5% higher performance with 60% FLOPs.PVTv2- and Swin-based models also achieve comparable performance with 60%–70% of competitive baselines’ FLOPs.Source: §5.1 · pp. 6–7

5.2. ADE20K Semantic Segmentation

  • On ADE20K, Slide Attention improves performance in SemanticFPN and UperNet, supporting its use across segmentation frameworks.ADE20K has 20K training and 2K validation images.Source: §5.2 · p. 7

5.3. COCO Object Detection

  • Across experiments with both 1x and 3x schedules and different detection heads, the model performs better under all settings and improves small-object detection by up to 4.5%.Source: §5.3 · p. 7

5.4. Comparison with Other Local Attentions

  • Against SASA, Slide Attention achieves better results; against CUDA-based SAN and NAT, it reports 0.5%–1.3% higher performance at comparable inference speed.The comparison uses Swin Transformer and NAT structures; Table 4 reports FPS measured on a single RTX3090 GPU.Source: §5.4 · p. 8, Table 4 · p. 8

5.5. Inference Time

  • Across tested devices, Slide Attention improves the runtime–performance trade-off, with speedups up to 2.3× and 3.7×–3.9× against Im2Col while maintaining higher performance.CUDA-based approaches could not be implemented on the tested non-CUDA devices, so those comparisons use the Im2Col-based approach.Source: §5.5 · p. 8, Figure 6 · p. 8

5.6. Ablation Study

  • Ablations indicate that Slide Attention performs better when applied at early Transformer stages, and that its module designs contribute to performance or efficiency.The stage study uses Swin-Tiny; the authors relate the early-stage finding to prior work on convolution in Vision Transformers.Source: §5.6 · p. 8, Table 5 · p. 8

6. Conclusion

The conclusion presents Slide Attention as an efficient local-attention module built from common convolution operations. Experiments support its use across Vision Transformer models and hardware, with a better computation-efficiency and performance trade-off.

6. Conclusion

  • Slide Attention replaces Im2Col with depthwise convolutions and adds deformed shifting to deliver local attention with efficiency, flexibility, and generalizability.The experiments show adoption across varied Vision Transformers and hardware, with a better trade-off between computation efficiency and model performance.Source: §6 · p. 8

Supplementary Material

For fair comparisons, the supplementary model configurations replace only early-stage self-attention blocks with Slide Attention. Other blocks, training configurations, and model width and depth remain unchanged.

A. Model Architectures

  • The supplementary comparisons substitute Slide Attention only for baseline self-attention blocks at early stages, keeping remaining blocks, training configurations, width, and depth unchanged.The five architectures are PVT, PVTv2, Swin Transformer, CSwin Transformer, and NAT.Source: A. Model Architectures · p. 9

B. Dataset and Training Setup

The setup describes training and evaluation configurations for ImageNet, COCO, and ADE20K, including dataset splits, schedules, and input sizes.

B. Dataset and Training Setup

  • ImageNet uses 1.28 million training and 50,000 validation images; baseline training follows original configurations with AdamW for 300 epochs and DeiT data augmentation.CSwin-Transformer additionally follows its original EMA setting.Source: B. Dataset and Training Setup · p. 9
  • COCO training uses 80K samples and 35K for validation, with 12-epoch 1x and, for several models, 36-epoch 3x schedules.The reported main-paper mAP results use input size (3, 1333, 800).Source: B. Dataset and Training Setup · p. 9
  • ADE20K training uses 20K images and validation uses 2K; training crops are 512×512 and testing resizes the shorter side to 512 pixels.SemanticFPN trains for 80K iterations and UperNet for 160K iterations.Source: B. Dataset and Training Setup · p. 9

C. Additional Downstream Experiments

Slide Attention shows consistent improvements on semantic segmentation and COCO detection benchmarks, with larger gains on small objects and a better runtime-performance trade-off than baselines.

C. Additional Downstream Experiments

D. Full Classification Results

The appendix provides full ImageNet classification results across baseline model sizes and examines how local-attention window size affects performance and runtime.

D. Full Classification Results

  • Table 7 provides full ImageNet-1K classification comparisons across baseline model sizes, including FLOPs, parameters, and accuracy.The main paper shows only representative classification results.Source: D. Full Classification Results · p. 10

E. Ablation Study on the Window Size

  • Increasing the window size brings only marginal performance improvements while greatly increasing inference time on iPhone 12 and RTX 3090.The ablation uses Swin-T and Swin-S, replacing self-attention in the first two stages; the authors use k = 3 in the main paper.Source: E. Ablation Study on the Window Size · p. 10
  • The authors identify finding a better performance-efficiency trade-off for larger windows as a future direction.They describe the ineffectiveness of increasing window size as a direction for further investigation.Source: E. Ablation Study on the Window Size · p. 10

F. Limitations

The authors identify the performance-efficiency trade-off for larger local-attention windows as an unresolved limitation and future research direction.

F. Limitations

  • Larger windows yield only marginal performance improvements, leaving a better performance-efficiency trade-off for future investigation.Figure 8 compares runtime across window sizes on iPhone 12 and RTX 3090.Source: F. Limitations · p. 10, Figure 8 · p. 11
  • Figure 8 presents window-size runtime comparisons on iPhone 12 and an RTX 3090 GPU.Source: Figure 8 · p. 11

Related papers

Loading 2304.04237v1…