Source-linked AI summary
Co-Scale Conv-Attentional Image Transformers
Weijian Xu, Yifan Xu, Tyler Chang, Zhuowen Tu
TL;DR
Image Transformers must balance global contextual modeling with computational cost and sufficient image detail. CoaT combines cross-scale attention with efficient convolutional attention, achieving strong ImageNet classification and demonstrated applicability to detection and instance segmentation, while incurring higher latency and FLOPs than similar-sized Swin Transformers.
Problem
High-resolution self-attention has quadratic cost, while patch tokenization can limit image-detail modeling; Transformer classifiers also retain design gaps relative to CNNs.
Method
CoaT maintains Transformer encoder branches at separate scales with cross-scale attention and uses convolutional relative position encoding in efficient factorized attention.
Results
CoaT attains strong ImageNet classification results against similar-sized CNNs and Transformer classifiers, with downstream applicability demonstrated for object detection and instance segmentation.
Takeaways & Limitations
CoaT provides a multi-scale, contextual image-Transformer backbone applicable to image classification and downstream detection and segmentation tasks.
Takeaways & Limitations
CoaT generally has higher latency and FLOPs than similar-sized Swin Transformers, partly because its parallel groups and sequential operations are computationally demanding.
Abstract
from arXiv · showhide
In this paper, we present Co-scale conv-attentional image Transformers (CoaT), a Transformer-based image classifier equipped with co-scale and conv-attentional mechanisms. First, the co-scale mechanism maintains the integrity of Transformers' encoder branches at individual scales, while allowing representations learned at different scales to effectively communicate with each other; we design a series of serial and parallel blocks to realize the co-scale mechanism. Second, we devise a conv-attentional mechanism by realizing a relative position embedding formulation in the factorized attention module with an efficient convolution-like implementation. CoaT empowers image Transformers with enriched multi-scale and contextual modeling capabilities. On ImageNet, relatively small CoaT models attain superior classification results compared with similar-sized convolutional neural networks and image/vision Transformers. The effectiveness of CoaT's backbone is also illustrated on object detection and instance segmentation, demonstrating its applicability to downstream computer vision tasks.
1. Introduction
Transformer-based image classifiers are promising but still face computational and design gaps relative to established CNNs. CoaT addresses these gaps with cross-scale modeling and efficient convolutional attention, achieving strong ImageNet results.
- Motivation: Transformers have shown encouraging results in computer vision, but their self-attention is more computationally demanding than convolution.Self-attention dynamically computes affinities between every pair of tokens, whereas convolution uses learned weights fixed during testing.
- Motivation: Transformer image classifiers still lag behind well-developed CNNs and commonly use a single fixed-size image grid.This fixed patching can limit modeling of details within each patch.
- CoaT: CoaT maintains encoder branches at separate scales while enabling attention across scales through serial and parallel blocks.The blocks support fine-to-coarse, coarse-to-fine, and cross-scale image modeling.
- CoaT: CoaT's conv-attention realizes relative position embeddings with convolutions in a factorized attention module for improved computation efficiency.The design targets the higher cost of vanilla self-attention while retaining contextual modeling.
- Results: CoaT achieves state-of-the-art ImageNet classification against competitive CNNs and outperforms competing Transformer-based image classifiers.The paper reports these results for relatively small CoaT models and presents the comparison in Figure 1.
2. Related Works
CoaT builds on Transformer image classifiers while addressing their single-grid design through co-scale modeling and convolutional attention. Its approach combines multi-scale representation learning with efficient relative-position modeling.
- Transformer image classifiers: ViT and DeiT established Transformer-based image classification, but both rely on a single image grid with fixed patch size.ViT additionally requires extra training data for its reported ImageNet performance, while DeiT uses training strategy and distillation to remove that requirement.
- CoaT motivation: CoaT is motivated by the representation benefits of multi-scale modeling and the connection between relative position encoding and convolution.These observations motivate its co-scale and conv-attentional designs.
- Multi-scale modeling: Prior CNN architectures implement increasingly rich scale interactions, from fine-to-coarse paths to simultaneous fine and coarse representations.The related work contrasts CNNs, U-Net, and HRNet as precedents for multi-scale modeling.
- Conv-attention: CoaT's conv-attention combines efficient factorized attention with depthwise convolutional relative and general position encoding.The design is discussed in relation to LambdaNets and CPVT.
3. Revisit Scaled Dot-Product Attention
Scaled dot-product attention projects image tokens into queries, keys, and values before computing weighted value aggregation. For high-resolution images, its quadratic complexity motivates patch tokenization and CoaT's co-scale, conv-attentional alternative.
- Scaled dot-product attention: Transformers project each input token into query, key, and value vectors, producing Q, K, and V representations for the sequence.The learned projections map X ∈ R^N×C into Q, K, V ∈ R^N×C.
- Scaled dot-product attention: Scaled dot-product attention computes softmax(QK^T/√C)V, weighting value vectors according to query-key affinities.The operation aggregates information from the full token sequence.
- Vision Transformer inputs: Vision Transformers represent an image as a class token plus flattened image tokens, giving sequence length N = HW + 1.Here H and W are the feature-map spatial dimensions.
- Computational limitation: Full-image attention becomes impractical at high resolution because it requires O(N^2) space and O(N^2C) time.Patch tokenization reduces sequence length but coarse patches can limit detail modeling, motivating CoaT's co-scale mechanism.
4. Conv-Attention Module
CoaT combines factorized attention with convolutional position encodings to reduce attention cost while restoring local positional relationships. Its conv-attentional module uses efficient convolution-like operations for relative position encoding and feeds the resulting representations into subsequent processing.
- Factorized Attention Mechanism: Factorized attention reduces sequence-length complexity by computing key–value interactions before combining them with queries.The factorization has O(NC′ + NC + CC′) space and O(NCC′) time complexity, both linear in sequence length N.
- Factorized Attention Mechanism: CoaT’s factorized attention uses the identity for ϕ and softmax for ψ, with projected channels C′ = C and explicit scaling by 1/√C.Its resulting complexity is O(NC + C^2) space and O(NC^2) time; it is a generalized attention mechanism rather than a direct approximation of scaled dot-product attention.
- Factorized Attention Mechanism: The factorized attention produces a global data-dependent linear transformation for each query feature vector through L = softmax(K)ᵀV.Without positional encoding, identical query vectors receive identical outputs, limiting awareness of locally nearby feature differences.
- Co-scale Architecture: The CoaT architecture uses serial blocks for reduced-resolution processing and parallel blocks for communication across multiple scales.Serial blocks down-sample and tokenize features before conv-attention and feed-forward layers, while parallel blocks support fine-to-coarse, coarse-to-fine, and cross-scale interaction.
- Convolutional Relative Position Encoding: CoaT adds convolutional relative position encoding by simplifying EV and computing it with depthwise convolution over image tokens.The convolution operates on reshaped image tokens, while the class token receives a zero positional-relative contribution in the final concatenation.
- Convolutional Position Encoding: CoaT also inserts depthwise convolution into input features and concatenates the resulting position-aware features back with them as convolutional position encoding.This general position encoding complements the convolutional relative position encoding inside each conv-attentional module.
5. Co-Scale Conv-Attentional Transformers
CoaT combines multi-scale encoder branches with cross-scale interaction through serial and parallel blocks, while conv-attention models relationships efficiently. The architecture uses feature interpolation for co-scale communication in the reported models.
- Co-Scale Mechanism: CoaT maintains separate encoder branches at multiple scales while enabling fine-to-coarse, coarse-to-fine, and cross-scale information flow.Serial and parallel blocks realize the co-scale mechanism.
- Co-Scale Mechanism: Parallel groups support direct cross-layer attention and attention with feature interpolation for cross-scale interaction.Feature interpolation is adopted for better empirical performance and is used in the final CoaT-Lite and CoaT models.
- Model Architecture: CoaT-Lite uses serial blocks in a fine-to-coarse pyramid and classifies images from the CLS token produced by the final serial block.The serial blocks progressively down-sample image features into lower-resolution representations.
- Model Architecture: CoaT combines serial and parallel blocks, passing multi-scale features and corresponding CLS tokens into three parallel blocks before aggregating CLS tokens for classification.The parallel group processes features from three scales.
- Experiments: CoaT and CoaT-Lite report strong ImageNet classification results against similarly sized ConvNet and ViTNet methods.The models are evaluated using parameter-budget comparisons and top-1 accuracy on the ImageNet validation set.
6. Experiments
Experiments evaluate CoaT on ImageNet classification and COCO detection and instance segmentation, including comparisons, ablations, and computational-cost measurements. Results show strong accuracy and downstream performance, while co-scale and combined position encodings improve performance but increase computational demands.
- Image Classification: CoaT and CoaT-Lite show strong ImageNet top-1 accuracy compared with similarly sized ConvNet and ViTNet methods.The comparison uses the ImageNet validation set and groups architectures under different parameter budgets.
- Object Detection and Instance Segmentation: CoaT models show clear performance advantages over ResNet, PVT, and Swin backbones for COCO detection and instance segmentation under both 1× and 3× settings.The evaluations use Mask R-CNN and Cascade Mask R-CNN frameworks.
- Object Detection and Instance Segmentation: 3.9% improvement in average precision is achieved by the CoaT backbone over Deformable DETR with ResNet-50 on COCO.The comparison uses the Deformable DETR framework with comparable backbone settings.
- Ablation Study: 77.5% top-1 accuracy is obtained when CRPE and CPE are combined in CoaT-Lite, outperforming variants without or with only one position encoding.The ablation is conducted with the CoaT-Lite Tiny architecture on ImageNet-1K validation.
- Ablation Study: Co-scale with feature interpolation improves both image classification and Mask R-CNN detection over CoaT without co-scale, while also outperforming direct cross-layer attention.The feature-interpolation variant is reported to have lower computational complexity and higher accuracy.
- Computational Cost: CoaT generally achieves higher accuracy than similarly sized Swin Transformers but incurs larger latency and FLOPs.The authors attribute latency overhead possibly to operations that are not running in parallel.
7. Conclusion
CoaT develops cross-scale attention and efficient conv-attention operations for Transformer-based image classification. The models achieve strong ImageNet classification results and demonstrate applicability to downstream detection and instance segmentation.
- CoaT combines cross-scale attention with efficient conv-attention operations in a Transformer-based image classifier.The conclusion identifies both mechanisms as core components of the model.
- CoaT achieves strong ImageNet classification results and demonstrates applicability to object detection and instance segmentation.The conclusion covers both classification and downstream computer vision tasks.