PaperlayerView exact revision on arXiv

Source-linked AI summary

VOLO: Vision Outlooker for Visual Recognition

Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, Shuicheng Yan

arXiv:2106.13112v2cs.CV

TL;DR

ViTs lagged behind leading CNNs partly because they encoded fine-level features and contexts into tokens inefficiently. VOLO introduces Outlooker attention and a two-stage architecture for fine-level encoding followed by global aggregation, achieving leading classification and segmentation results without extra training data.

  • Problem

    ViTs remained below state-of-the-art CNNs on ImageNet without extra data, while fine-level token encoding was identified as a performance-limiting gap.

  • Method

    VOLO uses lightweight Outlooker attention for efficient fine-level token encoding before transformer-based global information aggregation.

  • Results

    VOLO establishes new state-of-the-art results across ImageNet classification and semantic segmentation benchmarks, including 87.1% ImageNet top-1 accuracy and 84.3% Cityscapes mIoU.

  • Takeaways & Limitations

    The reported results support fine-level feature learning as a valuable direction for attention-based visual recognition models.

Abstract

from arXiv · show

Visual recognition has been dominated by convolutional neural networks (CNNs) for years. Though recently the prevailing vision transformers (ViTs) have shown great potential of self-attention based models in ImageNet classification, their performance is still inferior to that of the latest SOTA CNNs if no extra data are provided. In this work, we try to close the performance gap and demonstrate that attention-based models are indeed able to outperform CNNs. We find a major factor limiting the performance of ViTs for ImageNet classification is their low efficacy in encoding fine-level features into the token representations. To resolve this, we introduce a novel outlook attention and present a simple and general architecture, termed Vision Outlooker (VOLO). Unlike self-attention that focuses on global dependency modeling at a coarse level, the outlook attention efficiently encodes finer-level features and contexts into tokens, which is shown to be critically beneficial to recognition performance but largely ignored by the self-attention. Experiments show that our VOLO achieves 87.1% top-1 accuracy on ImageNet-1K classification, which is the first model exceeding 87% accuracy on this competitive benchmark, without using any extra training data In addition, the pre-trained VOLO transfers well to downstream tasks, such as semantic segmentation. We achieve 84.3% mIoU score on the cityscapes validation set and 54.3% on the ADE20K validation set. Code is available at \url{https://github.com/sail-sg/volo}.

1. Introduction

VOLO addresses ViTs’ remaining ImageNet gap by efficiently encoding fine-level features and contexts before global aggregation. Its attention-based architecture achieves leading classification and segmentation results without extra training data.

  • ViTs still lagged behind state-of-the-art CNNs on ImageNet without extra data, with CaiT reaching 86.5% versus NFNet-F5’s 86.8%.
  • Fine-level feature encoding limits ViT performance, while finer tokenization increases self-attention complexity quadratically.
  • Outlooker efficiently enriches tokens with fine-level information using linear projections instead of expensive dot-product attention.
  • VOLO uses fine-level Outlookers before coarse-level self-attention to combine expressive local representations with global information aggregation.For 224 × 224 inputs, it uses 8 × 8 patches for 28 × 28 fine-level tokens before 14 × 14 coarse modeling.
  • 87.1% top-1 accuracy on ImageNet was achieved by the 296M-parameter VOLO, alongside 90.6% on ImageNet-ReaL and 78.0% on ImageNet-V2.The results were reported without extra training data and described as new SOTA performance on all three classification benchmarks.
  • 84.3% mIoU on Cityscapes and 54.3% on ADE20K demonstrate strong transfer to semantic segmentation.The Cityscapes result was 0.3% above SegFormer-B5, while the ADE20K result improved on Swin Transformer’s 53.5%.

2. Method

VOLO uses Outlookers to encode fine-level token information before Transformers aggregate global information. Outlook attention generates local weights efficiently and densely aggregates neighboring value representations.

  • Network Architecture: VOLO uses two stages: Outlookers generate fine-level token representations, then Transformer blocks aggregate global information.Patch embedding modules map inputs to designed token shapes at the beginning of each stage.
  • Outlooker: Outlooker combines outlook attention for spatial encoding with an MLP for inter-channel interaction.LayerNorm is applied before the residual Outlooker operation.
  • Outlook Attention: Outlook attention generates local weights from token features and avoids the dot-product attention computation through efficient linear projections.It measures local token similarities while using a reshaping operation instead of Query-Key matrix multiplication.
  • Outlook Attention: Outlook attention projects each token into outlook weights and value representations, then reshapes and normalizes local weights for aggregation.The weights are generated through linear projections, reshaped to K^2 × K^2, and passed through Softmax.
  • Outlook Attention: Dense aggregation sums weighted values contributing to the same spatial location from different local windows.This local, dense aggregation is intended to encode fine-level information efficiently.
  • Multi-Head Outlook Attention: Multi-head outlook attention splits outlook weights and value embeddings into N segments, computes attention separately per head, and concatenates the outputs.Each head has dimension C_N satisfying C_N × N = C.
  • Discussion: For C = 384, K = 3, and N = 6, outlook attention satisfies NK^4 < 2C and is described as more computationally efficient.The comparison is made for a normal configuration of the attention mechanism.
  • Network Architecture: VOLO tokenizes 224 × 224 images into 8 × 8 patches for fine-level processing before downsampling and applying Transformers to coarser tokens.The fine-level tokens are arranged at 28 × 28, while global self-attention operates at 14 × 14.

3. Experiments

Experiments show that VOLO consistently outperforms prior models across ImageNet classification and downstream segmentation, while ablations validate the contributions and scaling behavior of Outlookers.

  • Main Results: 84.2% top-1 accuracy is obtained with VOLO-D1 at 224 resolution, increasing to 85.2% after finetuning at 384 resolution.VOLO-D1 uses 26.6M parameters.
  • Main Results: VOLO-D3 improves ImageNet-V2 accuracy by 0.8% over the previous best result while using 86M rather than 356M parameters.VOLO-D5 further reaches 78.0% on ImageNet-V2.
  • Ablation Analysis: Replacing the LV-ViT-S baseline with smaller patches and Outlookers raises ImageNet accuracy from 83.3% to 83.9%, while further changes and 384-resolution finetuning reach 85.2%.Changing the patch size from 16 × 16 to 8 × 8 initially provides a 0.4% gain; adding two more Outlookers raises performance to 83.9%.
  • Ablation Analysis: Classification performance saturates at four Outlookers, while increasing Outlooker head count helps only slightly and stops improving beyond six heads.The default is six heads for 384 hidden dimensions and twelve heads for 768 hidden dimensions.
  • Downstream Segmentation: 84.3% mIoU on Cityscapes and 54.3% mIoU on ADE20K demonstrate strong transfer to semantic segmentation.The Cityscapes result is 0.3% above SegFormer-B5, while the ADE20K result improves on 53.5% by Swin Transformer.

4. Related Work

Related work progresses from CNNs to attention-based and transformer-based models, while VOLO adds fine-level token encoding through Outlooker alongside long-range dependency modeling.

  • CNN-based models: CNN classifiers evolved from stacked convolutions and poolings toward deeper architectures using skip connections and specialized multi-path building blocks.
  • Transformer-based models: Transformers enable long-range dependency modeling through self-attention and have been applied successfully to image classification.
  • Transformer-based models: ViT initially required large-scale pretraining, while DeiT and T2T-ViT addressed data efficiency.
  • VOLO: Outlooker differs from convolution-based hybrid architectures by using local pair-wise token similarities to encode fine-level features and spatial context.

5. Conclusions

VOLO establishes strong results across image classification and semantic segmentation, with its performance attributed to outlook attention’s dense dynamic aggregation of fine-level features.

  • Conclusions: VOLO outperforms CNN- and Transformer-based models across image classification and semantic segmentation experiments.
  • Conclusions: The model’s performance superiority is attributed to outlook attention dynamically aggregating fine-level features in a dense manner.
  • Conclusions: The authors identify better fine-level feature learning as a direction for follow-up research and plan to investigate other applications.
Loading 2106.13112v2…