PaperlayerView exact revision on arXiv

Source-linked AI summary

CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention

Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, Wei Liu

arXiv:2108.00154v2cs.CVcs.LG

TL;DR

Existing vision transformers lack effective interactions between features at different scales and may sacrifice fine-grained information when reducing attention cost. CrossFormer combines cross-scale embeddings, long-short distance attention, and dynamic position bias to address these issues. It outperforms other vision transformers across classification, detection, and segmentation tasks, with especially substantial gains on dense prediction.

  • Problem

    Existing vision transformers fail to build interactions among different-scale features because embeddings are single-scale and some efficient attention mechanisms sacrifice small-scale features.

  • Method

    CrossFormer combines Cross-scale Embedding Layer, Long Short Distance Attention, and Dynamic Position Bias to provide cross-scale features, efficient attention, and variable-size position handling.

  • Results

    CrossFormer outperforms other state-of-the-art vision transformers on image classification, object detection, instance segmentation, and semantic segmentation, with particularly substantial gains on dense prediction tasks.

  • Takeaways & Limitations

    CrossFormer is a versatile vision architecture that accommodates variable-sized inputs and is especially effective for dense prediction tasks.

Abstract

from arXiv · show

Transformers have made great progress in dealing with computer vision tasks. However, existing vision transformers do not yet possess the ability of building the interactions among features of different scales, which is perceptually important to visual inputs. The reasons are two-fold: (1) Input embeddings of each layer are equal-scale, so no cross-scale feature can be extracted; (2) to lower the computational cost, some vision transformers merge adjacent embeddings inside the self-attention module, thus sacrificing small-scale (fine-grained) features of the embeddings and also disabling the cross-scale interactions. To this end, we propose Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). On the one hand, CEL blends each embedding with multiple patches of different scales, providing the self-attention module itself with cross-scale features. On the other hand, LSDA splits the self-attention module into a short-distance one and a long-distance counterpart, which not only reduces the computational burden but also keeps both small-scale and large-scale features in the embeddings. Through the above two designs, we achieve cross-scale attention. Besides, we put forward a dynamic position bias for vision transformers to make the popular relative position bias apply to variable-sized images. Hinging on the cross-scale attention module, we construct a versatile vision architecture, dubbed CrossFormer, which accommodates variable-sized inputs. Extensive experiments show that CrossFormer outperforms the other vision transformers on image classification, object detection, instance segmentation, and semantic segmentation tasks. The code has been released: https://github.com/cheerss/CrossFormer.

1 INTRODUCTION

CrossFormer addresses vision transformers’ limited cross-scale interactions with cross-scale embeddings, long-short distance attention, and dynamic position bias for variable-sized inputs. Experiments across four vision tasks show superior performance, especially on dense prediction tasks.

  • Motivation: Existing vision transformers use equal-sized patches and costly attention approximations, limiting interactions between fine-grained and coarse-grained features.Self-attention has O(N^2) computational and memory cost, while some substitutes sacrifice small-scale features or long-distance dependencies.
  • Method: Cross-scale Embedding Layer constructs each embedding by projecting and concatenating patches sampled with multiple kernel sizes.CEL appears at the start of each pyramid stage and supplies cross-scale features to subsequent self-attention.
  • Method: Long Short Distance Attention divides attention into short-distance and long-distance counterparts, reducing cost while retaining small- and large-scale features.SDA and LDA appear alternately in consecutive CrossFormer blocks.
  • Method: Dynamic Position Bias generates relative position bias from embeddings’ relative distances, accommodating variable image or group sizes.This extends relative position representations beyond fixed-size inputs.
  • Results: CrossFormer handles image classification, object detection, instance segmentation, and semantic segmentation, outperforming other state-of-the-art vision transformers on all four tasks.The reported gains are particularly substantial for object detection and segmentation.

2 BACKGROUND

Vision transformers adapt transformer architectures to visual inputs and use pyramid structures or attention substitutes to manage computational costs. Position representations provide information that self-attention alone does not preserve.

  • Vision Transformers: ViT and DeiT transfer the original transformer to vision tasks, while later models introduce pyramid structures to reduce the number of patches.The background discusses ViT, DeiT, PVT, HVT, Swin, and ViTAE as representative vision transformers.
  • Substitutes of Self-attention: Vanilla self-attention costs O(N^2), so vision transformers approximate it for large inputs such as object detection and segmentation.Swin restricts attention locally, whereas PVT and Twins share keys and values across adjacent embeddings.
  • Position Representations: Position representations compensate for transformers’ permutation invariance by encoding where embeddings occur.Absolute position embeddings and relative position bias are presented as alternative approaches.

3 CROSSFORMER

CrossFormer combines cross-scale embedding with alternating short- and long-distance attention in a four-stage pyramid architecture. Dynamic position bias extends relative position representations to variable image or group sizes.

  • Architecture: CrossFormer uses four pyramid stages, each beginning with a CEL and continuing with CrossFormer blocks containing LSDA and an MLP.CEL generates cross-scale embeddings from the preceding stage or input image.
  • Cross-scale Embedding Layer: CEL samples patches with multiple kernel sizes, projects them, and concatenates corresponding patches into each embedding.The first CEL uses 4 × 4, 8 × 8, 16 × 16, and 32 × 32 kernels with a shared 4 × 4 stride; padding is used when necessary.
  • Cross-scale Embedding Layer: CEL allocates lower projected dimensions to larger kernels and higher dimensions to smaller kernels because convolutional cost grows with K^2D^2.This allocation controls the total CEL computational budget while retaining multiple scales.
  • Long Short Distance Attention: LSDA alternates SDA and LDA across blocks: SDA groups adjacent G × G embeddings, whereas LDA groups embeddings sampled at interval I.Both attention variants use vanilla self-attention after grouping, and the grouping strategy preserves local and distant interactions.
  • Long Short Distance Attention: Cross-scale embeddings support LDA because adjacent large-scale patches provide context for relating embeddings whose small-scale patches are non-adjacent.This connects the LDA grouping pattern with the multi-scale construction of CEL embeddings.
  • Dynamic Position Bias: DPB generates each relative position bias from two embeddings’ coordinate distance, avoiding the fixed-size bound of conventional RPB.Its MLP has three fully connected layers, D/4-dimensional intermediate layers, and a scalar output; DPB can handle variable image or group sizes and has an O(G^2) implementation.
  • Variants of CrossFormer: CrossFormer backbones can be fine-tuned across tasks with different G or I because these settings do not change weight-tensor shapes.Detection and segmentation models may use larger-image configurations in their first two stages while reusing classification backbones.

4 EXPERIMENTS

Experiments evaluate CrossFormer across classification, detection, instance segmentation, semantic segmentation, and ablations. CrossFormer consistently performs strongly against vision-transformer baselines, with especially pronounced gains on dense prediction tasks.

  • Experimental settings: CrossFormer is evaluated on image classification, object detection, instance segmentation, and semantic segmentation using competitive vision-transformer baselines.The experiments use ImageNet, COCO 2017, and ADE20K benchmarks with task-specific detection and segmentation heads.
  • Image classification: CrossFormer achieves the highest ImageNet accuracy with parameters and FLOPs comparable to state-of-the-art vision transformers.It outperforms DeiT, PVT, and Swin by at least 1.2% absolute accuracy on small models; on large models, it reaches 84.0% versus RegionViT’s 83.3%.
  • Object detection and instance segmentation: CrossFormer outperforms all compared architectures on both detection and instance segmentation tasks for small and base model sizes.The performance gain over other architectures becomes sharper as the model size increases.
  • Semantic segmentation: CrossFormer’s semantic-segmentation advantage increases with model size, reaching a 3.1% absolute IOU gain over Twins-SVT-L.CrossFormer-T is 1.4% higher on IOU than Twins-SVT-B, while CrossFormer-B is 3.1% higher than Twins-SVT-L.
  • Ablation studies: Cross-scale embeddings improve performance by 1% absolute over single-scale embeddings, while different kernel-size combinations yield similar accuracies.CrossFormer reaches 82.5% versus 81.5% with single-scale embeddings, while tested cross-scale combinations range from 82.3% to 82.5%.
  • Ablation studies: Long-short distance attention exceeds PVT-like and Swin-like alternatives by at least 0.6% accuracy, while DPB matches RPB accuracy at negligible extra cost and supports variable sizes.DPB and RPB both outperform APE by 0.4% absolute accuracy; residual connections in DPB do not improve performance.

5 CONCLUSIONS

CrossFormer is a vision transformer built around cross-scale attention through CEL and LSDA, with dynamic position bias extending relative position representations to arbitrary input sizes. Experiments report superior performance across representative vision tasks, particularly dense prediction.

  • 5 CONCLUSIONS: CrossFormer combines Cross-scale Embedding Layer and Long Short Distance Attention to enable cross-scale attention.CEL and LSDA are identified as the architecture’s core ingredients.
  • 5 CONCLUSIONS: Dynamic position bias makes relative position bias applicable to any input size.This extends the flexibility of position representations beyond fixed-size inputs.
  • 5 CONCLUSIONS: CrossFormer achieves superior performance over other state-of-the-art vision transformers on several representative vision tasks.The conclusion particularly highlights improvements on object detection and segmentation.
  • 5 CONCLUSIONS: The reported detection and segmentation improvements indicate that CEL and LSDA are together essential for dense prediction tasks.The conclusion links the architecture’s cross-scale attention design with its gains on dense prediction.

A CROSSFORMER

The LSDA pseudocode separates embeddings into short-distance and long-distance attention groups, then restores their original spatial arrangement using reshape and permute operations.

  • A CROSSFORMER: Both SDA and LDA require only reshape and permute operations and can be implemented in about ten lines of code.The pseudocode describes these operations as the implementation of both attention variants.
  • A CROSSFORMER: LSDA groups the input tensor differently for short-distance attention (SDA) and long-distance attention (LDA).The input tensor has shape (H, W, D), with G denoting the group size.
  • A CROSSFORMER: SDA reshapes embeddings into local G × G groups before attention.The grouping uses H // G and W // G spatial partitions.
  • A CROSSFORMER: LDA reshapes embeddings to connect positions separated across spatial intervals while retaining G × G attention groups.The resulting sequence contains H * W // (G ** 2) groups, each with G ** 2 embeddings.
  • A CROSSFORMER: After attention, LSDA ungroups the embeddings and reshapes them back to (H, W, D).SDA and LDA use corresponding permute operations during restoration.

A.2 DYNAMIC POSITION BIAS (DPB)

Dynamic Position Bias (DPB) generates relative position biases from embedding distances, allowing position representations to adapt when image or group sizes vary.

  • A.2 DYNAMIC POSITION BIAS (DPB): For a group of size G × G, relative coordinate differences range from 1 − G to G − 1 in both axes.The coordinates themselves satisfy 0 ≤ x, y < G.
  • A.2 DYNAMIC POSITION BIAS (DPB): DPB constructs a matrix B̂_i,j by evaluating DPB over all relative offsets in a (2G − 1) × (2G − 1) grid.The matrix is indexed using offsets from 1 − G through G − 1.
  • A.2 DYNAMIC POSITION BIAS (DPB): The computational complexity of constructing B̂ is O(G2).The bias matrix B is then drawn from B̂.
  • A.2 DYNAMIC POSITION BIAS (DPB): When G is fixed, B̂ and B remain unchanged during testing, so DPB is equivalent to relative position bias in that case.This permits computing both matrices once for fixed image or group sizes.

A.3 VARIANTS OF CROSSFORMER FOR DETECTION AND SEGMENTATION

CrossFormer provides detection and segmentation backbones that retain the classification architecture while changing early-stage grouping parameters for dense prediction tasks.

  • A.3 VARIANTS OF CROSSFORMER FOR DETECTION AND SEGMENTATION: CrossFormer variants for object detection, instance segmentation, and semantic segmentation are listed as dense-prediction backbones.The example input size in the architecture table is 1280 × 800.
  • A.3 VARIANTS OF CROSSFORMER FOR DETECTION AND SEGMENTATION: The dense-prediction architectures match the image-classification architectures except for different G and I values in the first two stages.G and I denote group size and interval for SDA and LDA, respectively.
  • A.3 VARIANTS OF CROSSFORMER FOR DETECTION AND SEGMENTATION: Changing G and I does not change weight-tensor shapes, enabling direct fine-tuning of ImageNet-pretrained backbones on other tasks.The passage states this remains possible even when dense-prediction tasks use different settings.
  • A.3 VARIANTS OF CROSSFORMER FOR DETECTION AND SEGMENTATION: Table 9 reports object-detection results on COCO val 2017, including allocated GPU memory.The table covers RetinaNet and Mask-RCNN detection heads.

B.1 OBJECT DETECTION

For object detection, smaller grouping parameters achieve higher AP, but the gain is marginal; larger settings provide a better performance–memory trade-off.

  • B.1 OBJECT DETECTION: Smaller (G, I) achieves higher AP than larger settings, but the performance gain is marginal.This comparison is reported for object detection with RetinaNet and Mask-RCNN heads.
  • B.1 OBJECT DETECTION: The configuration (G1 = 14, I1 = 16, G2 = 14, I2 = 8) is judged to offer a better trade-off between performance and cost.The authors associate this choice with the larger settings used in Table 8.

B.2 SEMANTIC SEGMENTATION

Semantic segmentation experiments compare two (G, I) configurations and report similar memory costs, with CrossFormers‡ showing advantages under semantic FPN.

  • The two (G, I) configurations have almost identical memory costs, unlike the object-detection experiments.
  • With semantic FPN, CrossFormers‡ achieve higher IOU than CrossFormers, 46.4 versus 46.0.
  • With semantic FPN, CrossFormers‡ require fewer FLOPs than CrossFormers, 209.8G versus 220.7G.
Loading 2108.00154v2…