PaperlayerView exact revision on arXiv

Source-linked AI summary

DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition

Jiayu Jiao, Yu-Ming Tang, Kun-Yu Lin, Yipeng Gao, Jinhua Ma, Yaowei Wang, Wei-Shi Zheng

arXiv:2302.01791v1cs.CV

TL;DR

Vision Transformers face a trade-off between global attention's quadratic cost and local attention's limited receptive fields. DilateFormer exploits locality and sparsity in shallow layers through multi-scale dilated attention, achieving strong performance across vision tasks with reduced computation.

  • Problem

    Global attention models arbitrary patch dependencies at quadratic cost, whereas local attention reduces cost but has small attended receptive fields.

  • Method

    DilateFormer uses sliding-window dilated attention with multiple dilation rates in shallow stages and global multi-head self-attention in deeper pyramid stages.

  • Results

    DilateFormer achieves state-of-the-art performance across ImageNet-1K classification, COCO detection and instance segmentation, and ADE20K semantic segmentation.

  • Takeaways & Limitations

    The model uses locality and sparsity to reduce redundant shallow-layer interactions while retaining multi-scale and deeper global modeling.

Abstract

from arXiv · show

As a de facto solution, the vanilla Vision Transformers (ViTs) are encouraged to model long-range dependencies between arbitrary image patches while the global attended receptive field leads to quadratic computational cost. Another branch of Vision Transformers exploits local attention inspired by CNNs, which only models the interactions between patches in small neighborhoods. Although such a solution reduces the computational cost, it naturally suffers from small attended receptive fields, which may limit the performance. In this work, we explore effective Vision Transformers to pursue a preferable trade-off between the computational complexity and size of the attended receptive field. By analyzing the patch interaction of global attention in ViTs, we observe two key properties in the shallow layers, namely locality and sparsity, indicating the redundancy of global dependency modeling in shallow layers of ViTs. Accordingly, we propose Multi-Scale Dilated Attention (MSDA) to model local and sparse patch interaction within the sliding window. With a pyramid architecture, we construct a Multi-Scale Dilated Transformer (DilateFormer) by stacking MSDA blocks at low-level stages and global multi-head self-attention blocks at high-level stages. Our experiment results show that our DilateFormer achieves state-of-the-art performance on various vision tasks. On ImageNet-1K classification task, DilateFormer achieves comparable performance with 70% fewer FLOPs compared with existing state-of-the-art models. Our DilateFormer-Base achieves 85.6% top-1 accuracy on ImageNet-1K classification task, 53.5% box mAP/46.1% mask mAP on COCO object detection/instance segmentation task and 51.1% MS mIoU on ADE20K semantic segmentation task.

I. INTRODUCTION

Vanilla ViTs capture arbitrary patch dependencies but incur quadratic cost, while local attention is cheaper yet has limited receptive fields. DilateFormer targets this trade-off by exploiting locality and sparsity in shallow-layer attention with multi-scale dilated attention.

  • Global ViTs model dependencies between arbitrary image patches but incur quadratic computational cost.
  • Local attention reduces computational cost but limits modeling to small neighborhoods and weakens long-range dependency modeling.
  • Shallow ViT attention exhibits locality and sparsity, making many distant-patch interactions redundant for semantic modeling.
  • SWDA sparsely selects patches in a surrounding sliding window, while MSDA captures dependencies at multiple scales using different dilation rates across heads.
  • DilateFormer uses MSDA in shallow stages and global multi-head self-attention in deeper stages within a pyramid architecture.

A. Global Attention in Vision Transformers

Prior vision Transformers reduce global-attention cost through local or sparse mechanisms, but often neglect data-dependent sparsity or require complex multi-scale designs. DilateFormer instead uses dilation-based multi-scale interaction within self-attention.

  • Local-window Transformers reduce computational cost by restricting self-attention to fixed or structured neighborhoods.
  • Several approaches consider locality but omit sparsity, while uniformly sparse methods approximate the global receptive field.
  • Existing multi-scale Transformer designs use complex embedding layers, attention blocks, convolution branches, or extra branches, adding parameters or computation.
  • Dilated attention introduces multiple dilation rates at the same semantic level within one self-attention operation to model multi-scale interaction.
  • Unlike single-scale fixed-dilation attention, DilateFormer assigns different dilation rates to different heads to capture and fuse multi-scale semantic features.

III. MULTI-SCALE DILATED TRANSFORMER

The model implements sparse local attention through SWDA and extends it with multi-scale heads in MSDA. A pyramid DilateFormer combines MSDA for shallow stages with ordinary MHSA for deeper stages.

  • A. Sliding Window Dilated Attention: SWDA computes self-attention for each query using sparsely selected keys and values in a centered sliding window.
  • A. Sliding Window Dilated Attention: MSDA splits feature channels across heads and applies SWDA with different dilation rates to extract multi-scale semantic information.
  • A. Sliding Window Dilated Attention: The default 3 × 3 kernel with r = 1, 2 and 3 produces attended receptive fields of 3 × 3, 5 × 5 and 7 × 7.
  • A. Sliding Window Dilated Attention: The dilation rate r controls sparsity by spacing selected keys and values within the sliding window.
  • A. Sliding Window Dilated Attention: SWDA uses zero padding at feature-map edges and preserves the feature-map size while modeling local and sparse dependencies.

B. Multi-Scale Dilated Attention

MSDA extracts multi-scale semantic information by applying sparse windowed attention across heads with different dilation rates. The resulting head outputs are concatenated and linearly aggregated, reducing attention redundancy without extra computational cost.

  • B. Multi-Scale Dilated Attention: MSDA projects a feature map into queries, keys, and values, then applies SWDA across heads with different dilation rates.Each head processes a channel slice at its own scale.
  • B. Multi-Scale Dilated Attention: The outputs from all heads are concatenated and sent to a linear layer for feature aggregation.
  • B. Multi-Scale Dilated Attention: Different head dilation rates aggregate semantic information at multiple scales within the attended receptive field.
  • B. Multi-Scale Dilated Attention: MSDA reduces self-attention redundancy without complex operations or extra computational cost.

C. Overall Architecture

DilateFormer uses a pyramid architecture that assigns MSDA to low-level stages and ordinary MHSA to high-level stages. Its blocks also include convolutional positional encoding, MLP layers, and overlapping tokenization.

  • C. Overall Architecture: The first two DilateFormer stages use MSDA, while the latter two stages use ordinary MHSA.
  • C. Overall Architecture: The overlapping tokenizer uses multiple zero-padded 3 × 3 convolution modules, with output resolution controlled by stride size.
  • C. Overall Architecture: Convolutional positional encoding is implemented as a zero-padded depth-wise convolution with a 3 × 3 kernel.
  • C. Overall Architecture: Each block adds an MLP consisting of two linear layers, channel expansion ratio 4, and one GELU activation.
  • C. Overall Architecture: DilateFormer provides Tiny, Small, and Base variants with different model capacities.

IV. EXPERIMENTS

The experiments evaluate DilateFormer across ImageNet-1K classification, COCO detection and instance segmentation, and ADE20K semantic segmentation. Results emphasize competitive accuracy with lower computational cost and parameter counts.

  • IV. EXPERIMENTS: DilateFormer is evaluated on ImageNet-1K classification, COCO detection and instance segmentation, and ADE20K semantic segmentation.
  • IV. EXPERIMENTS: Without extra training data, DilateFormer variants achieve comparable or better ImageNet-1K performance with fewer model parameters.
  • IV. EXPERIMENTS: ImageNet-1K experiments follow DeiT and PVT training strategies, using AdamW, 300 epochs, cosine learning-rate decay, batch size 1024, and weight decay 0.05.
  • IV. EXPERIMENTS: 83.3% top-1 accuracy: Dilate-S surpasses Swin-T by 2.0% and ViL-S by 1.3% on ImageNet-1K at 224 resolution.
  • IV. EXPERIMENTS: 3.2 GFLOPs and 82.1% accuracy: Dilate-T is comparable to ViL-S, Focal-T, and PVT-L without extra assistance or high-resolution finetuning.

B. Object Detection and Instance Segmentation on COCO

On COCO, DilateFormer variants outperform recent Transformers for both object detection and instance segmentation across Mask R-CNN frameworks. The experiments use ImageNet-1K-pretrained backbones and compare multiple training schedules.

  • B. Object Detection and Instance Segmentation on COCO: COCO2017 provides 118K training images, 5K validation images, and 20K test images for evaluation.
  • B. Object Detection and Instance Segmentation on COCO: The experiments use Mask R-CNN and Cascade Mask R-CNN with ImageNet-1K-pretrained DilateFormer backbones.
  • B. Object Detection and Instance Segmentation on COCO: DilateFormer variants outperform recent Transformers on both object detection and instance segmentation in two COCO frameworks.
  • B. Object Detection and Instance Segmentation on COCO: For Mask R-CNN 1×, DilateFormer surpasses Swin Transformer by 2.8–3.6% box mAP and 2.5–2.6% mask mAP at comparable settings.
  • B. Object Detection and Instance Segmentation on COCO: 49.9% box mAP and 43.7% mask mAP: Dilate-B achieves these scores with Mask R-CNN under the 3× + MS schedule.

D. Ablation Studies

The ablations evaluate sparse and local attention patterns, dilation scales, stage placement, tokenization, inference, and visualization. Results support using multi-scale dilated attention in shallow stages to balance accuracy, receptive field, and computation.

  • SWDA vs. other sparse and local patterns: 82.1% Top-1 accuracy shows SWDA outperforms DC, DDC, WASS and SWA across ImageNet-1K, COCO, and ADE20K comparisons.SWDA exceeds DC by +0.4%, +1.4%/+0.6%, and +1.69%; DDC by +0.3%, +0.6%/+0.3%, and +0.94%; WASS by +0.3%, +0.8%/+0.5%, and +1.18%; and SWA by +0.3%, +0.5%/+0.1%, and +2.21%.
  • Dilation scale: The multi-scale dilation setting [2] [3] reaches 82.1% Top-1 accuracy, outperforming single-scale settings because it provides richer information.The dilation rates are selected to model locality and sparsity without the redundant interaction associated with overly large receptive fields.
  • MSDA vs. other block settings: 82.1% Top-1 accuracy with MSDA exceeds shifted-window local attention at 81.7% using fewer FLOPs and exceeds global attention by 0.3% with half the FLOPs.MSDA also surpasses global attention with spatial reduction by 0.5% without extra parameters.
  • Stage setting: 82.2% Top-1 accuracy is obtained when MSDA is used only in stage1, while accuracy decreases to 80.5% as MSDA occupies more stages.Using MSDA in stages1 and2 is the default despite 0.1 percentage point lower accuracy because it reduces FLOPs by 0.35G.
  • Overlapping tokenizer/downsampler: +0.4% Top-1 accuracy is achieved with the overlapping tokenizer/downsampler, indicating the main improvement does not rely on these modules.The comparison replaces overlapping modules with simple nonoverlapping convolutional tokenizers or downsamplers.
  • Comparisons of real running times: With comparable parameters and FLOPs, DilateFormers have comparable FPS and better performance than current state-of-the-art models.FPS and peak memory are measured from forward passes with batch size 256 on a single A100 GPU.
  • Grad-CAM visualization: Grad-CAM indicates Dilate-Tiny locates target objects more effectively and attends to semantic areas more continuously and completely than DeiT-Tiny and Swin-Tiny.The visualization associates this behavior with better classification performance.
  • Attention-map visualization: Shallow ViT-Small attention maps show high-attention patches sparsely scattered around each query patch, while other patches receive low attention scores.This visualization supports the sparse local interaction pattern used by the proposed attention mechanism.

V. CONCLUSION

DilateFormer uses MSDA to exploit locality and sparsity in shallow-layer self-attention while aggregating multi-scale information efficiently. The paper reports strong representations and state-of-the-art results across classification, detection, and segmentation tasks.

  • DilateFormer provides powerful and general representations for various vision tasks.
  • MSDA considers locality and sparsity in shallow-layer self-attention to aggregate semantic multi-scale information and reduce redundancy without extra computational cost.
  • State-of-the-art performance is reported on ImageNet-1K classification, object detection, and semantic segmentation.

MORE ABLATION STUDIES

The kernel-size ablation examines how the attended receptive field affects DilateFormer accuracy and computational cost. Larger kernels slightly improve performance but increase FLOPs, motivating the default 3 × 3 kernel.

  • Larger SWDA kernels slightly improve performance through a larger attended receptive field but increase FLOPs.
  • The default SWDA kernel size is 3 × 3.
Loading 2302.01791v1…