Contextual Transformer Networks for Visual Recognition

Yehao Li; Ting Yao; Yingwei Pan; Tao Mei

Source-linked AI summary

Contextual Transformer Networks for Visual Recognition

Yehao Li, Ting Yao, Yingwei Pan, Tao Mei

arXiv:2107.12292v1Jul 26, 2021cs.CVcs.AIcs.LGcs.MM

TL;DR

Visual self-attention often learns attention from isolated query-key pairs and underuses neighboring-key context. The paper introduces CoT, which combines static key-context encoding with dynamic contextualized self-attention in a unified block that can replace ResNet 3×3 convolutions. Across image recognition, object detection, and instance segmentation, CoTNet outperforms several state-of-the-art backbones, including ResNeSt.

Problem
Existing visual self-attention mainly relies on independent query-key interactions, leaving rich contextual information among neighboring keys under-exploited.
Method
CoT contextually encodes neighboring keys with a 3×3 convolution, uses the contextualized keys and queries to learn dynamic self-attention, and replaces ResNet 3×3 convolutions to form CoTNet.
Results
CoTNet outperforms several state-of-the-art backbones across image recognition, object detection, and instance segmentation, including a 0.9% absolute top-1 error reduction against ResNeSt-101 on ImageNet.
Takeaways & Limitations
A unified block that combines key-context mining with self-attention can strengthen visual representations while serving as an alternative to standard convolutions with favorable parameter budgets.

Abstract

from arXiv · show

Transformer with self-attention has led to the revolutionizing of natural language processing field, and recently inspires the emergence of Transformer-style architecture design with competitive results in numerous computer vision tasks. Nevertheless, most of existing designs directly employ self-attention over a 2D feature map to obtain the attention matrix based on pairs of isolated queries and keys at each spatial location, but leave the rich contexts among neighbor keys under-exploited. In this work, we design a novel Transformer-style module, i.e., Contextual Transformer (CoT) block, for visual recognition. Such design fully capitalizes on the contextual information among input keys to guide the learning of dynamic attention matrix and thus strengthens the capacity of visual representation. Technically, CoT block first contextually encodes input keys via a $3\times3$ convolution, leading to a static contextual representation of inputs. We further concatenate the encoded keys with input queries to learn the dynamic multi-head attention matrix through two consecutive $1\times1$ convolutions. The learnt attention matrix is multiplied by input values to achieve the dynamic contextual representation of inputs. The fusion of the static and dynamic contextual representations are finally taken as outputs. Our CoT block is appealing in the view that it can readily replace each $3\times3$ convolution in ResNet architectures, yielding a Transformer-style backbone named as Contextual Transformer Networks (CoTNet). Through extensive experiments over a wide range of applications (e.g., image recognition, object detection and instance segmentation), we validate the superiority of CoTNet as a stronger backbone. Source code is available at \url{https://github.com/JDAI-CV/CoTNet}.

1. Introduction

CNNs provide locality and translation equivariance but struggle with long-range dependencies, while existing visual self-attention often models isolated query-key pairs. CoT addresses this by combining local key-context mining with self-attention and forms CoTNet backbones that improve recognition, detection, and segmentation results.

Motivation: CNN convolutions impose spatial locality and translation equivariance, but their limited receptive fields hinder modeling global or long-range dependencies.Long-range interaction supports numerous computer-vision tasks.
Motivation: Existing visual self-attention mainly learns attention from independent query-key pairs, under-exploiting contextual information among neighboring keys.This limitation motivates contextualized attention over 2D feature maps.
Contextual Transformer: CoT first contextualizes neighboring keys with a 3×3 convolution, then combines the contextualized keys with queries to learn dynamic self-attention within one architecture.The design captures static local context and uses it to guide dynamic attention without an additional context-mining branch.
Contextual Transformer Networks: CoTNet replaces ResNet 3×3 convolutions without increasing parameter and FLOP budgets, producing a unified Transformer-style backbone.The block is designed as an alternative to standard convolutions in existing ResNet architectures.
Results: 0.9% absolute top-1 error reduction on ImageNet, 1.5% mAP improvement on COCO detection, and 0.7% mAP improvement on COCO instance segmentation were reported against ResNeSt.These results span image recognition, object detection, and instance segmentation.

2. Related Work

Vision backbones evolved from convolutional designs toward self-attention to capture longer-range interactions, including global and local variants. CoT extends this direction by explicitly modeling neighboring-key context within a unified architecture.

Convolutional Networks: ConvNet development progressed through deeper and increasingly structured convolutional architectures, including AlexNet, VGG, GoogleNet, ResNet, ResNeXt, and DenseNet.These designs aimed to strengthen visual representation capacity through depth, aggregation, and cross-layer connections.
Self-attention in Vision: Vision self-attention adapts Transformer-style long-range dependency modeling to spatial feature maps, but global attention can scale poorly.Local self-attention over patches limits parameter and computation costs.
Contextual Transformer: CoT differs from conventional self-attention by explicitly modeling rich contexts among neighboring keys while unifying context mining and attention learning.The unified design maintains a favorable parameter budget.

3. Our Approach

The Contextual Transformer (CoT) block extends local self-attention by using contextualized neighboring keys to guide attention, then fuses static and dynamic contextual representations. CoT serves as a convolutional replacement in ResNet-style backbones, forming CoTNet variants with comparable computational budgets.

Motivation: Conventional local self-attention learns query-key relations independently and therefore does not exploit contextual information among neighboring keys.The local relation matrix is formed from pairwise relations within each k × k neighborhood, but the relations are learned over isolated query-key pairs.
Contextual Transformer Block: CoT first applies k × k group convolution to neighboring keys, producing K1 as a static contextual representation.The input keys and queries are set to X, while values are projected through Wv.
Contextual Transformer Block: CoT aggregates values with the learned attention matrix to form dynamic context K2, then fuses K1 and K2 into the block output.K2 captures dynamic feature interactions, while K1 captures static local context.
Contextual Transformer Networks: CoTNet directly replaces 3×3 convolutions throughout ResNet-50, retaining a slightly smaller parameter and FLOP count than ResNet-50.The CoTNet-50 construction replaces the 3×3 convolutions in the res2, res3, res4, and res5 stages.
Contextual Transformer Networks: CoTNeXt-50 uses a 2×48d template and requires 1.2× more parameters and 1.01× more FLOPs than ResNeXt-50.This construction replaces the 3×3 kernels in ResNeXt-50 group convolutions while keeping computational cost similar.

4. Experiments

Experiments evaluate CoTNet on ImageNet, object detection, and instance segmentation under default and advanced training settings. Across these applications, CoTNet generally improves recognition, accuracy–inference tradeoffs, and downstream performance while retaining favorable computational budgets.

Experimental scope: Experiments evaluate CoTNet for ImageNet recognition and transfer its pretrained backbones to COCO object detection and instance segmentation.ImageNet models are trained from scratch, then evaluated as pretrained backbones for downstream tasks.
ImageNet recognition: Under default ImageNet training, CoTNet and CoTNeXt outperform comparable ConvNet and attention-based backbones across top-1 and top-5 accuracy with favorable parameter budgets.The comparison groups models by 50-layer and 101-layer depth.
ImageNet recognition: 82.1% and 83.2% top-1 accuracy are achieved by CoTNeXt-50 and CoTNeXt-101 in advanced training, improving over the best competitors by 1.0% and 0.9%.The advanced setup uses longer training and stronger augmentation and regularization.
Efficiency: 0.6% higher top-1 accuracy and 2.75× faster inference are obtained by SE-CoTNetD-152 (320) than EfficientNet-B6.CoTNet models consistently achieve better top-1 accuracy with less inference time across both training setups.
Ablation studies: 79.2% top-1 accuracy is reached by the full CoT block, exceeding static context, dynamic context, and linear fusion variants in the CoTNet-50 ablation.The results support complementarity between static and dynamic contexts and benefit from dynamic fusion via attention.
Ablation studies: Replacing more ResNet-50 stages with CoT blocks generally improves performance while slightly decreasing parameter count and FLOPs.Stages are progressively replaced in the order res2→res3→res4→res5.
Downstream transfer: Pretrained CoTNet models consistently improve COCO object detection and instance segmentation performance over comparable ConvNet and attention-based backbones across most reported IoU thresholds.For instance segmentation, CoTNet-50 leads on most metrics despite smaller input size and fewer fine-tuning epochs than BoTNet-50.

5. Conclusions

CoT block uses contextual information among input keys to guide self-attention learning, while CoTNet replaces ResNet 3×3 convolutions with this unified block for visual representation learning.

CoT block captures static context among neighboring keys before using it to trigger self-attention that mines dynamic context.
The unified architecture combines context mining and self-attention learning in a single block to strengthen visual representation.
CoT blocks can replace standard convolutions in ResNet architectures while retaining a favorable parameter budget.
CoTNet replaces ResNet or ResNeXt 3×3 convolutions with CoT blocks and is pretrained on ImageNet.
Experiments on COCO show that CoTNet-pretrained visual representations generalize to object detection and instance segmentation.