PaperlayerView exact revision on arXiv

Source-linked AI summary

UniFormer: Unifying Convolution and Self-attention for Visual Recognition

Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, Yu Qiao

arXiv:2201.09450v3cs.CV

TL;DR

Visual recognition must handle local redundancy and complex global dependency, which CNNs and ViTs address only partially. UniFormer unifies convolution and self-attention through local shallow-layer and global deep-layer relation aggregation. It reports strong results across image and video tasks, including 86.3 top-1 accuracy on ImageNet-1K without extra training data and 2-4× higher throughput for an efficient variant.

  • Problem

    Visual representation learning must address large local redundancy and complex global dependency in images and videos.

  • Method

    UniFormer combines convolution-style local token affinity in shallow layers with self-attention-style global token affinity in deep layers within a concise transformer format.

  • Results

    UniFormer achieves strong performance across image classification, video classification, detection, segmentation, and pose estimation, including 86.3 top-1 accuracy on ImageNet-1K without extra training data.

  • Takeaways & Limitations

    The unified backbone provides accuracy-computation trade-offs across image and video tasks, while its efficient variant reaches 2-4× higher throughput than recent lightweight models.

  • Takeaways & Limitations

    In shallow layers, local token affinity is represented by a learnable parameter matrix that depends only on relative position.

Abstract

from arXiv · show

It is a challenging task to learn discriminative representation from images and videos, due to large local redundancy and complex global dependency in these visual data. Convolution neural networks (CNNs) and vision transformers (ViTs) have been two dominant frameworks in the past few years. Though CNNs can efficiently decrease local redundancy by convolution within a small neighborhood, the limited receptive field makes it hard to capture global dependency. Alternatively, ViTs can effectively capture long-range dependency via self-attention, while blind similarity comparisons among all the tokens lead to high redundancy. To resolve these problems, we propose a novel Unified transFormer (UniFormer), which can seamlessly integrate the merits of convolution and self-attention in a concise transformer format. Different from the typical transformer blocks, the relation aggregators in our UniFormer block are equipped with local and global token affinity respectively in shallow and deep layers, allowing to tackle both redundancy and dependency for efficient and effective representation learning. Finally, we flexibly stack our UniFormer blocks into a new powerful backbone, and adopt it for various vision tasks from image to video domain, from classification to dense prediction. Without any extra training data, our UniFormer achieves 86.3 top-1 accuracy on ImageNet-1K classification. With only ImageNet-1K pre-training, it can simply achieve state-of-the-art performance in a broad range of downstream tasks, e.g., it obtains 82.9/84.8 top-1 accuracy on Kinetics-400/600, 60.9/71.2 top-1 accuracy on Sth-Sth V1/V2 video classification, 53.8 box AP and 46.4 mask AP on COCO object detection, 50.8 mIoU on ADE20K semantic segmentation, and 77.4 AP on COCO pose estimation. We further build an efficient UniFormer with 2-4x higher throughput. Code is available at https://github.com/Sense-X/UniFormer.

✦ 1 INTRODUCTION R

Visual recognition must address both local redundancy and complex global dependency. UniFormer combines convolution-style local aggregation with self-attention-style global aggregation, achieving strong accuracy-computation trade-offs across image and video tasks.

  • Motivation: Images and videos contain substantial local redundancy and complex global dependency, creating simultaneous efficiency and representation-learning challenges.Local visual content is often similar within neighborhoods, while targets in distant regions may have dynamic relations.
  • Motivation: CNN convolution reduces local redundancy through small-neighborhood aggregation, but its limited receptive field hinders global-dependency learning.The cited examples use neighborhoods such as 3×3 or 3×3×3.
  • Motivation: ViT self-attention captures global dependency, yet shallow-layer attention often compares all tokens even when only neighboring tokens contribute to local representations.Visualization of DeiT and TimeSformer shows spatial attention focusing mostly on local regions and temporal attention on adjacent frames.
  • UniFormer: UniFormer uses local token affinity in shallow layers and global token affinity in deep layers within a concise transformer format.Its blocks contain Dynamic Position Embedding, Multi-Head Relation Aggregator, and Feed-Forward Network modules.
  • Results: 86.3 top-1 accuracy is achieved on ImageNet-1K without extra training data, while ImageNet-1K-pretrained UniFormer-B reaches 82.9/84.8 top-1 on Kinetics-400/600.The backbone is also evaluated on Something-Something, COCO detection, ADE20K segmentation, and COCO pose estimation.
  • Results: 2-4× higher throughput is reported for the efficient UniFormer than recent lightweight models.The paper reports this efficient variant alongside broad image and video recognition results.

2 RELATED WORK

Prior vision backbones are dominated by CNNs and ViTs, while subsequent work explores combining their complementary operations. UniFormer extends this direction toward accuracy-computation trade-offs across image and video recognition.

  • CNNs: CNNs became a dominant framework for image understanding and were adapted to video recognition as video applications expanded.The related work describes extensive CNN development from AlexNet onward and subsequent attempts to apply CNNs to video.
  • ViTs: ViTs model visual-token relations with attention to capture long-term dependencies, but vanilla ViT relies on sufficient training data and careful augmentation.Related approaches address patch embedding, data-efficient training, efficient self-attention, and multi-scale architectures.
  • Hybrid backbones: Hybrid CNN-ViT research combines convolution through patch stems, position embeddings, feed-forward layers, projections, or MBConv modules.The cited passage also notes analogous combinations for video understanding using global attention and related components.
  • UniFormer: UniFormer uses transformer-style local and global token relations to achieve better accuracy-computation trade-offs across image and video tasks.This framing connects the paper's unified design with the broader effort to combine CNN and ViT strengths.
  • Efficient models: Lightweight CNNs target platforms with limited computing resources, whereas lightweight ViT design is described as less fully investigated.Examples include MobileNets, ShuffleNets, and EfficientNets, followed by recent lightweight ViT work.

3 METHOD

UniFormer presents a transformer-format backbone that combines convolution-like local aggregation in shallow layers with self-attention-like global aggregation in deep layers. Its DPE and MHRA modules target flexible position encoding, reduced redundancy, and long-range dependency modeling across image and video inputs.

  • 3.1 Overview: UniFormer uses DPE, MHRA, and FFN modules within a transformer-format block for image and video representation learning.Image inputs are treated as videos with one frame, while the block is designed to reduce computational redundancy and capture complex dependency.
  • 3.2 Multi-Head Relation Aggregator: MHRA reshapes input features into tokens, applies relation aggregation across multiple heads, and fuses the resulting head representations.Each relation aggregator encodes token context and summarizes it using a learned token-affinity matrix.
  • 3.2.1 Local MHRA: In shallow layers, local MHRA learns token affinity from a small neighborhood using a learnable parameter matrix based on relative positions.This design reflects the observation that adjacent shallow-layer tokens vary subtly, making dynamic affinity unnecessary there.
  • 3.2.1 Local MHRA: Local MHRA can be implemented as a PWConv-DWConv-PWConv sequence, connecting it to MobileNet-style channel-separated convolution while retaining transformer structure.The local affinity operates on each output channel after pointwise transformation, and a final pointwise transformation fuses the heads.
  • 3.2.2 Global MHRA: In deep layers, global MHRA computes token affinity from content similarity across the full spatiotemporal token space to learn long-range relations.The global formulation uses distinct query and key transformations and can be instantiated as spatiotemporal self-attention.
  • 3.3 Dynamic Position Embedding: DPE uses depthwise convolution with zero padding to encode position information, support arbitrary input shapes, and provide a lightweight position mechanism.The spatiotemporal version handles video position information, while zero padding helps tokens progressively access absolute-position cues.

4 FRAMEWORK

UniFormer backbones progressively combine local and global relation aggregation across four stages, then adapt these backbones for video and dense prediction tasks.

  • Backbone Design: UniFormer backbones hierarchically stack local and global blocks across four stages for computation-accuracy balance.Local blocks occupy the first two stages, while global blocks occupy the last two.
  • Backbone Design: The image backbone uses local relation aggregation through PWConv-DWConv-PWConv and global relation aggregation through multi-head self-attention.The local depthwise convolution uses a 5×5 spatial size for image classification.
  • Video Classification: Video adaptation preserves the four-stage structure while inflating all 2D convolution filters into 3D filters and using spatiotemporal attention.The video model uses 3×3×3 and 5×5×5 kernels for DPE and local MHRA, respectively.
  • Dense Prediction: Dense prediction requires adaptation because high-resolution inputs make direct backbone use computationally expensive.The framework therefore focuses on modifying Stage3 relation aggregation.
  • Dense Prediction: For dense prediction, windowed global MHRA reduces Stage3 computation, while hybrid groups combine three window blocks with one global block.A predefined window such as 14×14 lowers computation but can reduce performance without global interaction.

5 TOWARDS LIGHTWEIGHT UNIFORMER

The lightweight UniFormer reduces global token-interaction cost with an Hourglass block that shrinks and later recovers tokens while preserving their spatiotemporal structure.

  • Hourglass UniFormer Block: The Hourglass UniFormer block saves MHRA and FFN computation through adaptive token shrinking and recovering.It is used in Stages 3 and 4 of the lightweight architecture.
  • Token Shrinking: A score token estimates visual-token importance from attention similarity averaged across heads.The resulting importance values guide which tokens are retained or fused.
  • Token Shrinking: Low-importance tokens are fused into one weighted representative, reducing X from C×THW tokens to C×M with M ≪ THW.The reduced tokens then enter global MHRA and FFN.
  • Token Recovering: After global MHRA and FFN, representative tokens are replicated to recover the original spatiotemporal structure.This recovery allows convolutional dynamic position encoding in the next block.
  • Lightweight Architecture: The lightweight variants reduce depth, width, or resolution while retaining the broader UniFormer architecture.UniFormer-XS and UniFormer-XXS use explicitly specified stage depths and channel widths.

6 EXPERIMENTS

Experiments evaluate UniFormer across image, video, and dense vision benchmarks, reporting strong accuracy and computation results under ImageNet-1K training or pre-training.

  • Experimental Scope: Experiments cover ImageNet-1K, four video datasets, COCO detection and segmentation, and ADE20K semantic segmentation.The study also includes ablations of UniFormer design choices.
  • Image Classification: 83.4% top-1 accuracy at 4.2G FLOPs lets UniFormer-S† surpass RegNetY-4G, Swin-T, CSwin-T, and CoAtNet under the reported comparisons.The respective accuracy gains are 3.4%, 2.1%, 0.7%, and 1.8%.
  • Image Classification: 86.3% top-1 accuracy is achieved by UniFormer-L when fine-tuned on 384×384 ImageNet images.UniFormer-L also uses only 21% of LV-ViT-M's FLOPs and 61% of VOLO-D3's FLOPs while achieving higher accuracy under the same settings.
  • Video Classification: 82.9% versus 82.8% top-1 accuracy on Kinetics-400 is reported for UniFormer-B32f versus ViViT-L, with 16.7× fewer computation.The comparison uses ImageNet-1K pre-training for UniFormer and larger pre-training datasets for the cited baselines.
  • Video Classification: 61.0% and 71.2% top-1 accuracy establish the best reported UniFormer results on Something-Something V1 and V2, respectively.The V1 result is 4.2% higher than TDNEN, while the V2 result is 1.6% higher than Swin-B.
  • Video Classification: UniFormer significantly outperforms previous state-of-the-art methods on UCF101 and HMDB51, indicating transfer to relatively small video datasets.The passage notes that these datasets are relatively small and their performances have largely saturated.

6.3 Object Detection and Instance Segmentation

UniFormer improves COCO object detection and instance segmentation results over CNN and Transformer backbones while using hybrid Stage3 blocks for high-resolution inputs.

  • Mask R-CNN: UniFormer variants outperform CNN and Transformer backbones with Mask R-CNN on COCO detection and instance segmentation.The hybrid Stage3 style uses a window size of 14 to reduce training cost.
  • Mask R-CNN: 7.0–7.6 box mAP and 6.7–7.2 mask mAP gains over ResNet are reported under the 1× schedule.These comparisons use comparable settings.
  • Cascade Mask R-CNN: +0.3 box mAP and +0.3 mask mAP over CSwin-S are achieved by UniFormer-B under the 3× multi-scale schedule.The model also surpasses larger Swin-B and Focal-B backbones in the reported comparison.

6.4 Semantic Segmentation

UniFormer is evaluated for semantic segmentation on ADE20K using Semantic FPN and UperNet frameworks under established training settings. It improves mIoU over Swin Transformer with similar model sizes, while global MHRA further improves results at higher computational cost.

  • Experimental settings: Semantic segmentation experiments use ADE20K with Semantic FPN and UperNet frameworks.Training follows PVT settings for Semantic FPN and Swin Transformer settings for UperNet.
  • Results: +4.7/+2.5 mIoU is achieved by UniFormer-Sh32/Bh32 over Swin Transformer with Semantic FPN at similar model sizes.
  • Results: +2.5/+1.9 mIoU and +2.7/+1.2 MS mIoU improvements are achieved with UperNet.These comparisons are reported for UniFormer-Sh32/Bh32 against Swin Transformer.
  • Results: Global MHRA consistently improves segmentation results but increases computation cost.

6.6 Light-Weight UniFormer

The lightweight UniFormer variants target favorable accuracy-throughput trade-offs across image and video classification and dense prediction. They outperform strong lightweight baselines while retaining a concise design without complicated lightweight-specific additions.

  • Classification: UniFormer-XS192 achieves 1.7× higher throughput than EfficientNet-B3 with similar ImageNet performance.
  • Classification: UniFormer-XXS192 provides 0.6% higher accuracy and 16% higher throughput than MobileFormer on ImageNet.
  • Video classification: UniFormer-XXS150×16f achieves 9.3% higher performance and 16% higher throughput than MoViNet-A0 on Kinetics-400.
  • Video classification: UniFormer-XS192×32f runs 4.2× faster and attains 5.4% higher accuracy than X3D-S on Kinetics-400.
  • Dense prediction: UniFormer-XS improves over PVTv2-B1 by 2.7 box mAP and 2.1 mask mAP on COCO, and by +1.9 mIoU on ADE20K.

6.7 Ablation Studies

Ablations examine UniFormer structures, video transfer and sampling choices, testing strategies, and H-UniFormer designs. Results support combining local and global aggregation, joint spatiotemporal learning, and dataset-specific evaluation choices.

  • Model designs: Dynamic position embedding improves top-1 accuracy by +0.5% on ImageNet and +1.7% on Kinetics-400.The reported benefit is attributed to preserving spatial and temporal order.
  • Model designs: Using only local MHRA reduces accuracy by -1.0% on ImageNet and -2.1% on Kinetics-400 because long-term dependency is not captured.
  • Model designs: Using global MHRA in all stages causes a -4.0% drop on Kinetics-400 with heavier computation.The passage associates this with redundant attention, overfitting, and weaker detailed representation extraction.
  • Transfer learning: Joint spatiotemporal attention is more powerful than separate attention for Kinetics-400 transfer learning.
  • Transfer learning: +1.8% top-1 accuracy is obtained on Something-Something V1 by pre-training from ImageNet through Kinetics-400 instead of ImageNet alone.
  • Inflating methods: Inflating convolutional filters to 3D produces similar Kinetics-400 results but improves Something-Something V1 performance.
  • Sampling stride: With multi-clip testing, frame stride 4 performs best despite larger strides often improving single-clip results.
  • Testing strategies: Multi-clip testing is preferred for Kinetics, whereas multi-crop testing is better for Something-Something.The choice reflects scene coverage for Kinetics and motion capture for Something-Something.

6.8 Visualizations

Visualizations compare attention behavior across UniFormer structural variants and show qualitative results across downstream tasks. The comparisons highlight the differing object focus and granularity produced by global-only and local-only aggregation.

  • Attention visualization: Global-only aggregation struggles to focus on key objects because it compares token similarities across all layers.The examples identify the mountain and skateboard as key objects.
  • Attention visualization: Local-only aggregation produces coarse and inaccurate attention without global information.
  • Downstream-task visualization: Qualitative visualizations across downstream tasks demonstrate the effectiveness of UniFormer backbones.

7 CONCLUSION

UniFormer unifies convolution and self-attention in a concise transformer format for efficient visual recognition, using different relation aggregators across network depth. Experiments report broad state-of-the-art performance with less training cost.

  • UniFormer unifies convolution and self-attention in a concise transformer format to address redundancy and dependency in visual recognition.
  • Local MHRA is used in shallow layers to reduce computation, while global MHRA is used in deep layers to learn global token relations.
  • Extensive experiments demonstrate UniFormer's modeling capacity across a broad range of vision tasks with less training cost.
Loading 2201.09450v3…