PaperlayerView exact revision on arXiv

Source-linked AI summary

XCiT: Cross-Covariance Image Transformers

Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, Hervé Jegou

arXiv:2106.09681v2cs.CVcs.LG

TL;DR

Vision transformers offer global token interactions but self-attention scales quadratically, limiting long sequences and high-resolution images. XCiT addresses this with channel-wise cross-covariance attention and combines it with local patch interaction and feed-forward modules. It reports competitive classification, strong dense-prediction results, and effective self-supervised learning across several vision benchmarks.

  • Problem

    Quadratic time and memory complexity of token self-attention limits transformer application to long sequences and high-resolution images.

  • Method

    XCiT replaces token self-attention with cross-covariance attention over feature channels, then combines XCA with local patch interaction and feed-forward modules.

  • Results

    XCiT is competitive with vision transformers for classification and outperforms ResNet and transformer backbones on dense prediction, while also improving DINO-based self-supervised performance over a DeiT backbone.

  • Takeaways & Limitations

    Linear token complexity and fixed-channel attention make XCiT suitable for high-resolution and variable-resolution vision tasks across classification, detection, segmentation, and self-supervised learning.

  • Takeaways & Limitations

    The covariance interpretation assumes centered features, although centering is not required for the relation between covariance and Gram matrices.

Abstract

from arXiv · show

Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens ,i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high-resolution images. We propose a "transposed" version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images. Our cross-covariance image transformer (XCiT) is built upon XCA. It combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including image classification and self-supervised feature learning on ImageNet-1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k.

1 Introduction

XCiT replaces token self-attention with cross-covariance attention, reducing complexity from quadratic to linear in the number of tokens. The resulting models target high-resolution and variable-resolution vision tasks while retaining competitive performance across classification, dense prediction, and self-supervised learning.

  • Motivation: O(w^2h^2) complexity makes conventional self-attention prohibitive for high-resolution vision tasks such as detection and segmentation.The quadratic cost arises from the number of input tokens or image patches.
  • Method: XCA attends over feature channels rather than tokens, using a key-query cross-covariance matrix and linear complexity in the number of tokens.This transposed interaction replaces explicit full pairwise token interactions.
  • Architecture: XCiT combines XCA with local patch interaction and feed-forward modules in a columnar architecture.The layer uses three main blocks: XCA, LPI, and FFN, with LayerNorm and residual connections.
  • Scalability: XCA attends to a fixed number of channels, making XCiT more robust to test-time resolution changes and variable-size images.This property supports processing images with more than thousand pixels in each dimension.
  • Results: 86.0% top-1 accuracy is achieved by XCiT-L24 on ImageNet, outperforming comparable CaiT-M24 and NFNet-F2 counterparts.The classification models use a simple columnar architecture with constant resolution across layers.
  • Results: 48.5% mAP for object detection, 43.7% mAP for instance segmentation, and 48.4% mIoU for semantic segmentation demonstrate strong dense-prediction performance.The models outperform ResNet and multiple transformer-based backbones; XCiT also achieves 80.9% top-1 accuracy with DINO self-supervised learning.

2 Related work

Related work addresses transformer depth, spatial structure, efficient attention, high-resolution vision, and data-dependent layers. These approaches modify architecture, attention patterns, resolution, or filtering to improve representation or efficiency.

  • Deep vision transformers: Deep vision transformer work addresses training instability and optimization through LayerScale and class-attention layers.These mechanisms support deeper models and decouple patch-feature learning from classification feature aggregation.
  • Spatial structure in vision transformers: Spatial-structure methods use overlapping or progressively reduced patches, pixel-level intra-patch processing, convolutional initialization, or multistage resolution reduction.The cited approaches target stronger spatial representations or more efficient visual processing.
  • Efficient attention: Efficient-attention methods reduce quadratic cost using local, strided, axial, or adaptive patterns, or by approximating attention through token-dimension projection.These methods alter the span or approximation of the self-attention matrix.
  • Transformers for high-resolution images: High-resolution vision transformers reduce complexity through pyramidal architectures or pooling across spatial and temporal dimensions.These designs target object detection, segmentation, and video recognition.
  • Data-dependent layers: Data-dependent layers generate or apply filters based on feature-derived information, including dynamic filtering and data-dependent 1×1 convolutions.XCiT is related to this line through its data-dependent weight matrix.

3 Method

XCiT replaces token-wise self-attention with cross-covariance attention over feature channels, then combines it with local patch interaction and transformer components. This design reduces token-dependent complexity and supports efficient processing of variable-resolution, high-resolution images.

  • XCiT architecture: XCiT constructs its transformer from cross-covariance attention, local patch interaction, feed-forward, and class-attention modules.The columnar architecture maintains spatial resolution across layers; local patch interaction uses depth-wise 3×3 convolutions, while class attention aggregates patch embeddings for classification.
  • Cross-covariance attention: Cross-covariance attention operates along feature channels rather than tokens, replacing explicit pairwise token interactions with a key-query cross-covariance matrix.The operation computes attention from the cross-covariance matrix and produces output token embeddings as convex combinations of the corresponding value features.
  • Cross-covariance attention: XCA applies ℓ2-normalization to queries and keys and introduces a learnable temperature to control attention-weight sharpness and improve stability.Normalization bounds cross-covariance entries within [−1,1], while temperature scaling permits sharper or more uniform softmax distributions.
  • Complexity: O(Nd^2/h) time and O(d^2/h+Nd) memory make cross-covariance attention linear in the number of tokens, unlike token self-attention’s O(N^2d) time and O(hN^2+Nd) memory.This token-scaling property addresses the quadratic cost of forming an N×N attention matrix for images with many patches.

4 Experimental evaluation

XCiT is evaluated across classification, self-supervised learning, detection, instance segmentation, and semantic segmentation, with strong results and robustness to resolution changes. Ablations identify XCA and convolutional components as important design choices.

  • Benchmark coverage: XCiT is evaluated on ImageNet classification, DINO self-supervised learning, COCO detection and instance segmentation, and ADE20k semantic segmentation.The experiments use standard benchmarks and compare XCiT with convolutional and transformer-based backbones.
  • Resolution robustness: XCiT is more robust to train-test resolution changes than DeiT and remains effective when processing high-resolution images.This adaptability supports its use for dense prediction tasks, where image resolutions and object sizes vary.
  • Ablations: Removing XCA reduces accuracy by 6.1%, compared with a 1.2% reduction after removing the local patch interaction layer.Convolutional patch projection and local patch interaction improve accuracy and accelerate training; query-key normalization provides stability across configurations.

5 Conclusion

The paper concludes that XCiT replaces quadratic token self-attention with feature-wise cross-covariance attention, yielding a generic vision architecture for high-resolution and variable-resolution settings. It maintains strong performance across classification, dense prediction, and self-supervised learning.

  • Conclusion: XCiT replaces token self-attention with attention over feature dimensions, eliminating expensive quadratic attention maps.The architecture uses cross-covariance attention as its core component.
  • Conclusion: XCiT combines state-of-the-art-level image classification with robustness to changing image resolutions similar to convolutional networks.The conclusion presents this as a property of the overall architecture rather than a single benchmark result.
  • Conclusion: XCiT provides strong performance as a backbone for object detection, instance segmentation, and semantic segmentation.The conclusion describes dense prediction as a central demonstrated application.
  • Conclusion: XCiT matches state-of-the-art self-supervised learning results with less compute.The conclusion identifies self-supervised learning as another supported use of the architecture.

A.2 Approximate attention models in ViT with DeiT training

The appendix compares efficient attention variants with full attention in a DeiT-S backbone. All studied variants lose accuracy relative to the baseline, although their efficiency trade-offs differ.

  • Approximate attention comparison: All evaluated efficient self-attention variants show a clear performance drop relative to the DeiT-S full-attention baseline.The comparison covers spatial reduction attention, Linformer, efficient attention, and axial attention after 300 training epochs.
  • Approximate attention comparison: Axial attention has the strongest performance among the studied efficient variants, with a 1.5% accuracy drop versus the baseline.Efficient attention has a larger 3.6% drop, while Linformer cannot process variable-length sequences.
  • Approximate attention comparison: Efficient attention offers linear complexity and a better trade-off than the other approximate methods, but remains 3.6% below full attention.The appendix contrasts this efficiency with its measured accuracy loss.

A.3 Training and testing with varying resolution

The resolution analysis examines how training and testing resolutions interact for vision transformers. It motivates handling varying resolutions because several vision tasks contain objects with different sizes.

  • Motivation: Handling varying image resolutions is important for image segmentation, detection, and retrieval, where objects of interest may have very different sizes.The appendix analyzes the corresponding train-test resolution trade-off for DeiT.
  • Resolution analysis: Table A.2 reports the trade-off between training and testing resolutions for DeiT, including multi-scale training conditions.MS denotes multi-scale training, where models see images at different resolutions during training.

B Additional details of training and our architecture

The architecture uses a 64-dimensional intermediate 2D sinusoidal positional encoding and adapts XCiT for multiscale feature extraction. Its XCA implementation transposes, normalizes, correlates, temperature-scales, and mixes channel features.

  • Positional encoding: XCiT encodes each x and y coordinate with 32 sinusoidal dimensions, concatenating them into a 64-dimensional intermediate positional representation.Each coordinate uses cosine and sine functions at 16 frequencies each before projection to the transformer working space.
  • Multiscale features: For 12-layer models, FPN features come from layers 4, 6, 8, and 12; for 24-layer models, they come from layers 8, 12, 16, and 24.The extracted features are re-scaled to make XCiT compatible with FPN detection and segmentation methods.
  • Training details: Training configurations specify stochastic-depth and LayerScale initialization hyperparameters for each model.These settings are listed in Table B.1, including the stochastic-depth drop rate d_r and LayerScale initialization ϵ.
  • XCA implementation: The XCA operation reshapes projected queries, keys, and values, then transposes them so channel dimensions precede token dimensions.This produces tensors with shape (B, h, C, N) before cross-covariance attention is computed.
  • XCA implementation: XCA applies ℓ2 normalization across tokens, computes the key-query cross-covariance matrix, scales it with learnable temperatures, and applies softmax.The resulting d × d attention map mixes channels rather than directly forming token-token attention.
  • XCA implementation: The attention map multiplies values to mix channels per token, after which the output is permuted, reshaped, projected, and returned.The pseudocode describes XCA as requiring normalization, temperature parameters, and transposition relative to token self-attention.

D Additional results

Additional experiments vary image and patch resolutions and evaluate transfer learning across several datasets. XCiT models show competitive transfer performance against strong transformer and convolutional baselines.

  • Additional image and patch sizes: Additional ImageNet-1k results evaluate XCiT with 384×384 images and 16×16 patches, and with 224×224 images and 8×8 patches.These combinations are reported in Table D.1 as top-1 accuracy evaluations.
  • Transfer learning: XCiT models pretrained on ImageNet-1k are fine-tuned on CIFAR-10, CIFAR-100, Flowers-102, Stanford Cars, and iNaturalist.The transfer-learning experiments assess performance across diverse downstream image datasets.
  • Transfer learning: Across these transfer-learning datasets, XCiT provides competitive performance against ViT-B, ViT-L, DeiT-B, and EfficientNet-B7.The comparison is reported in Table D.2 against strong transformer and convolutional baselines.

D.3 Image Retrieval

The image-retrieval setting addresses the scalability of transformer descriptors at high resolution. XCiT is presented as scaling linearly with the number of image pixels, supporting high-resolution retrieval models.

  • Motivation: Vision-based retrieval methods commonly extract image features from high-resolution images, traditionally aggregating local descriptors and more recently using convolutional networks.The passage frames retrieval as an application where image resolution is important.
  • Motivation: Vision transformers have shown promise for retrieval, but their scalability with image resolution is limited compared with convolutional networks.The cited context states that ViT models do not scale well as image resolution increases.
  • XCiT scalability: XCiT models scale linearly with the number of pixels and therefore do not suffer from the stated high-resolution scalability limitation.This is presented as the retrieval-relevant scalability property of XCiT.

D.3.1 Datasets and evaluation measure

The retrieval evaluation uses landmark and object datasets, measures mean average precision, and compares global and local representations extracted from pretrained XCiT models.

  • Evaluation: Retrieval performance is measured as mean average precision by searching query images against a database of images.This evaluation procedure applies to each benchmark.
  • Datasets: The retrieval datasets include Holidays, Oxford, and Paris6k, covering objects, scenes, buildings, landmarks, and monuments.Oxford evaluation uses the revisited benchmark’s medium and hard settings because technique ordering was unchanged on easy measures.
  • Image representations: The study extracts three types of image representations from pretrained XCiT models: CLS-token descriptors, VLAD descriptors, and AMSK descriptors.The local-feature approach treats transformer output patches as inputs to methods developed for local descriptors or CNN features.
  • Implementation: For representation extraction, models are fine-tuned at 384×384 resolution and use a default retrieval resolution of 768.The latter is compared with a resolution of 800 used for ResNet in prior work.

D.3.3 Experimental setting: Image retrieval with models pretrained on Imagenet1k only

Using ImageNet-1k-pretrained models, the experiments examine retrieval performance, resolution scaling, self-supervision, and inference efficiency. XCiT benefits from higher resolution and patch-based descriptors while maintaining favorable scalability.

  • Scaling resolution: XCiT performance improves steadily as resolution increases to 768, while gains begin to saturate at 1024 after fine-tuning at 384.The experiments therefore retain 784 as the pivot resolution.
  • Self-supervision: Self-supervised XCiT features outperform supervised features on Holidays but underperform them on ROxford.
  • Impact of Image description: XCiT-VLAD performs significantly better than the class token for image retrieval, with AMSK further improving results toward the state of the art on Holidays.The experiments report this despite using an off-the-shelf aggregation technique without retrieval-specific fine-tuning.
  • Impact of Image description: Compared with ResNet-50 under a comparable setting, XCiT is slightly worse on ROxford but significantly better on Holidays.
  • Inference efficiency: XCiT provides favorable inference scalability, with processing time scaling linearly with resolution and strong peak-memory behavior relative to other models.The comparison includes token-attention vision transformers and ResNet-50, using batch size 64 on a 32GB V100 GPU.
  • XCA representation: Higher-magnitude query and key patch embeddings correspond to more salient image regions in the XCA representation.XCA computes a cross-covariance matrix from queries and keys normalized across patches, whose entries represent cosine similarities influenced by patch magnitude.
Loading 2106.09681v2…