PaperlayerView exact revision on arXiv

Source-linked AI summary

TransNeXt: Robust Foveal Visual Perception for Vision Transformers

Dai Shi

arXiv:2311.17132v3cs.CVcs.AI

TL;DR

Efficient Vision Transformers can suffer insufficient information mixing from depth degradation in residual stacking, motivating a design closer to biological foveal vision. The paper combines Aggregated Attention and Convolutional GLU in the TransNeXt backbone, which achieves state-of-the-art results across visual tasks and model sizes, including 84.0% ImageNet accuracy for TransNeXt-Tiny with 69% fewer parameters than ConvNeXt-B.

  • Problem

    Depth degradation can limit information mixing in efficient Vision Transformers that rely on stacking, producing unnatural visual perception.

  • Method

    TransNeXt combines biomimetic Aggregated Attention as a token mixer with Convolutional GLU as a channel mixer, using local and learnable-token interactions for visual modeling.

  • Results

    TransNeXt achieves state-of-the-art performance across classification, detection, segmentation, and robustness evaluations, including 84.0% ImageNet accuracy for TransNeXt-Tiny with 69% fewer parameters than ConvNeXt-B.

  • Takeaways & Limitations

    The results support combining foveal information aggregation with local feature-based channel attention in a robust visual backbone.

Abstract

from arXiv · show

Due to the depth degradation effect in residual connections, many efficient Vision Transformers models that rely on stacking layers for information exchange often fail to form sufficient information mixing, leading to unnatural visual perception. To address this issue, in this paper, we propose Aggregated Attention, a biomimetic design-based token mixer that simulates biological foveal vision and continuous eye movement while enabling each token on the feature map to have a global perception. Furthermore, we incorporate learnable tokens that interact with conventional queries and keys, which further diversifies the generation of affinity matrices beyond merely relying on the similarity between queries and keys. Our approach does not rely on stacking for information exchange, thus effectively avoiding depth degradation and achieving natural visual perception. Additionally, we propose Convolutional GLU, a channel mixer that bridges the gap between GLU and SE mechanism, which empowers each token to have channel attention based on its nearest neighbor image features, enhancing local modeling capability and model robustness. We combine aggregated attention and convolutional GLU to create a new visual backbone called TransNeXt. Extensive experiments demonstrate that our TransNeXt achieves state-of-the-art performance across multiple model sizes. At a resolution of $224^2$, TransNeXt-Tiny attains an ImageNet accuracy of 84.0%, surpassing ConvNeXt-B with 69% fewer parameters. Our TransNeXt-Base achieves an ImageNet accuracy of 86.2% and an ImageNet-A accuracy of 61.6% at a resolution of $384^2$, a COCO object detection mAP of 57.1, and an ADE20K semantic segmentation mIoU of 54.7.

1. Introduction

The paper argues that stacking-based information exchange can degrade in efficient Vision Transformers and proposes biomimetic token and channel mixers to improve visual perception and robustness. Combined in TransNeXt, these components achieve strong results across classification, detection, segmentation, and robustness evaluation.

  • Motivation: Residual-block stacking may provide less effective cross-layer information exchange than expected, contributing to insufficient information mixing in deep efficient Vision Transformers.The paper relates this depth degradation effect to unnatural visual artifacts in efficient ViT models.
  • Contributions: Pixel-focused attention uses per-pixel, nearest-neighbor attention to mimic foveal vision and continuous eye movement while providing convolution-like visual priors.Its dual-path design combines fine-grained local attention with broader information aggregation.
  • Contributions: Convolutional GLU adds channel attention based on nearest-neighbor image features, providing a channel mixer intended to improve local modeling and robustness with fewer FLOPs than convolutional feed-forward layers.The design connects gated channel attention with image-local feature modeling.
  • Results: 84.0% ImageNet-1K accuracy is achieved by TransNeXt-Tiny, surpassing ConvNeXt-B with 69% fewer parameters.The paper also reports strong results for detection, segmentation, and robustness across model sizes.
  • Contributions: Aggregated attention combines QKV, LKV, and QLV attention mechanisms within one mixer, with additional computation accounting for only 0.2%-0.3% of the model.This diversifies affinity-matrix generation beyond query-key similarity.

2. Related Work

Related work covers Vision Transformers, learnable-token attention variants, and biomimetic vision models. These approaches differ in how they generate affinity matrices and represent foveal visual perception.

  • Vision transformers: Vision Transformer models divide images into non-overlapping patches, project them into token sequences, and process them with transformer encoders.ViT uses self-attention as a token mixer and MLP layers as channel mixers.
  • Non-QKV attention variants: Non-QKV attention variants use learnable tokens to replace queries or keys, generating affinity matrices that are not based solely on conventional query-key interactions.The cited examples include Involution and VOLO, which replace the original keys with learnable tokens.
  • Biomimetic vision modeling: Biomimetic vision models seek higher acuity near visual focus and lower acuity farther away, but window-partitioned approaches treat edge and center tokens differently.The paper identifies window-wise processing as a mismatch with continuous foveal vision.

3. Method

The method combines pixel-focused and aggregated attention with learnable-token variants, length-scaled cosine attention, and Convolutional GLU to provide local-global perception and efficient channel mixing.

  • Pixel-focused Attention: Pixel-focused attention combines query-centered sliding-window attention with pooling attention to provide fine-grained local perception and coarse-grained global awareness.The sliding window contains k^2 pixels, while pooled features contain H_pW_p tokens.
  • Pixel-focused Attention: ‘Activate and Pool’ projects and activates features before adaptive average pooling to reduce information loss during spatial downsampling.Replacing PVTv2-li’s downsampling module improved CIFAR-100 top-1 accuracy from 68.1% to 70.4%.
  • Aggregating Diverse Attentions: Learnable query embeddings, learnable keys, and query-dependent positional attention diversify affinity generation beyond input query-key similarity.Learnable query embeddings are added to traditional queries, while learnable tokens generate dynamic positional information for local attention.
  • Aggregating Diverse Attentions: Aggregated Attention combines these attention mechanisms with multi-scale extrapolation techniques into an enhanced form of pixel-focused attention.The paper names this enhanced mechanism Aggregated Attention (AA).
  • Overcoming Multi-scale Image Input: With fixed pooling dimensions, both PFA and AA have linear complexity in input sequence length, and a minimal 3 × 3 window reduces computation and memory.The pooling paths give each query a global receptive field, reducing the need for larger sliding windows.
  • Feature Analysis: Convolutional GLU adds a minimal 3 × 3 depthwise convolution before GLU gating, creating channel attention based on nearest-neighbor features.The resulting mixer is described as simpler and more robust than ConvFFN, with lower stated computational structure.

4. Experiment

Experiments evaluate TransNeXt on ImageNet classification, robustness, multi-scale inference, object detection, segmentation, and component ablations. Across these settings, the model shows strong accuracy, robustness, dense-prediction performance, and scaling behavior.

  • ImageNet-1K classification: 84.0% ImageNet-1K top-1 accuracy is achieved by TransNeXt-Tiny with 69% fewer parameters than ConvNeXt-B.TransNeXt-Micro reaches 82.5% while using 55% fewer parameters than FocalNet-T(LRF).
  • Robustness: 61.6% ImageNet-A accuracy is achieved by TransNeXt-Base at 384^2, outperforming ConvNeXt-L by 10.9% with 45% of its parameters.TransNeXt-Small reaches 58.3% and outperforms ConvNeXt-L by 7.6%.
  • Object detection and instance segmentation: 57.1 box mAP is achieved by TransNeXt-Base under 5-scales DINO detection, approaching Swin-L at 57.2.TransNeXt-Tiny reaches 55.1 AP_b under 4-scales with only 14% of ConvNeXt-L’s backbone parameters.
  • Semantic segmentation: 54.7 mIoU is achieved by TransNeXt-Base under Mask2Former semantic segmentation.TransNeXt-Small reaches 54.1 mIoU, surpassing Swin-B at 53.9.
  • Multi-scale inference: TransNeXt outperforms pure convolutional solutions in both normal and linear modes for multi-scale image inference.Length scaling becomes significant at 640^2, while interpolated relative position biases substantially reduce performance; extrapolative log-CPB is effective.
  • Ablation studies: 0.8%, 0.9%, and 0.7% ImageNet-1K accuracy gains result from replacing ConvFFN with ConvGLU, SRA with PFA, and PFA with aggregated attention, respectively.The corresponding ImageNet-A gains are 4.3%, 3.4%, and 3.0%.

5. Conclusion

The paper combines Aggregated Attention and Convolutional GLU into TransNeXt, a robust visual backbone for classification, detection, and segmentation. It reports state-of-the-art performance, strong multi-scale inference, and substantial CUDA acceleration.

  • Conclusion: TransNeXt combines Aggregated Attention as a biomimetic foveal token mixer with Convolutional GLU as a gated channel-attention mixer.The resulting backbone targets classification, detection, and segmentation.
  • Conclusion: TransNeXt achieves state-of-the-art performance across classification, detection, and segmentation tasks.Its multi-scale inference performance highlights advantages over large-kernel strategies for depth degradation.
  • Conclusion: 103.4% training acceleration and 60.5% inference acceleration are achieved with the provided CUDA implementation.More detailed experimental data and discussions are included in the appendix.

A. Equivalent Form of Pixel-Focused Attention

Pixel-focused attention can be written as attention over concatenated keys and values from its sliding-window and pooling paths. Although this form is mathematically concise, implementation separates the paths to reduce memory pressure while preserving equivalence.

  • Equivalent formulation: K_concat and V_concat concatenate keys and values from the sliding-window path ρ(i,j) and pooling path σ(X).The concatenated tensors combine the two attention paths before computing the output.
  • Equivalent formulation: The pixel-focused output applies one softmax to Q_(i,j)K_concat^T divided by sqrt(d), plus positional bias, then multiplies V_concat.Using the concatenated similarity result in the same softmax preserves mathematical equivalence between the two forms.
  • Practical implementation: Separately computing the two attention paths avoids two large temporary tensors created by explicitly merging keys and values.The concatenated implementation increases memory usage and memory-access pressure, slowing model speed.

B. Comparative Analysis of Human Vision and Attention Visualization

The paper relates pixel-focused attention to human foveal and peripheral vision, using a small sliding window alongside full-image pooling. Visualization shows these paths retain distinct but complementary perceptual preferences.

  • Human vision: Human foveal vision provides high sensitivity over 1 to 2 degrees, while peripheral vision offers a broader but less precise receptive field.Saccades let the eye process multiple fields of view and integrate their information through transsaccadic perception.
  • Pixel-focused attention: The sliding-window path simulates foveal vision and continuous eye movements, while the pooling path simulates peripheral vision with full-image perception.Their features overlap and compete in the same softmax, allowing detailed and broad context to be integrated.
  • Attention visualization: At the central position, sliding-window attention focuses on textures, fluff, and sharp edges, whereas pooling attention maintains a reliable grasp of broader image content.The visualization presents these contrasting preferences within the same attention head.

C.2. Training Settings for ImageNet-1K

The supplied passages mainly specify ImageNet-1K training regularization and optimization settings, while also including model configuration and ablation context.

  • Training settings: Training uses Random Augmentation, Mixup, CutMix, Random Erasing, Label Smoothing, and DropPath.
  • Training settings: The optimizer is AdamW with gradient clipping norm 1.0, weight decay 0.05, and an initial learning rate of 10^-3.
  • Training settings: Training warms up for 5 epochs from a learning rate of 10^-6 before cosine scheduling.
  • Training settings: ImageNet-1K evaluation uses center cropping for images below 384 × 384 resolution.
  • Ablation context: Query embedding improves ImageNet-1K validation and ImageNet-V2 performance but reduces performance on ImageNet-A, ImageNet-R, and ImageNet-Sketch.

D.2. Detailed Data of Multi-scale Inference

The supplied passages describe multi-scale inference behavior, relative-position-bias choices, and ablations of Convolutional GLU, window size, and model architecture.

  • Multi-scale inference: Linear-complexity inference is possible when Hp and Wp remain fixed independently of input size.
  • Multi-scale inference: Length-scaling begins to affect attention at 640^2 resolution, when sequence-length variation exceeds 8×.
  • Relative position bias: Interpolation of relative position bias at 640^2 causes significant performance degradation, while log-CPB supports extrapolation to unknown resolutions.
  • Relative position bias: Removing aggregated-attention relative position bias decreases performance by 0.8%, whereas replacing it with dynamic bias decreases performance by 0.6%.
  • Model architecture: Replacing aggregated attention with multi-head self-attention decreases performance by 0.5% in stage 3 and an additional 0.1% in stage 2.
  • Model architecture: A fully aggregated-attention micro model achieves 82.6% ImageNet-1K accuracy at 256^2 resolution.

D.7. CUDA Implementation

The supplied passages compare CUDA and native PyTorch implementations and describe throughput conditions and remaining efficiency limitations.

  • Implementation: Native PyTorch sliding-window extraction creates a large temporary tensor and increases memory-access pressure.
  • Implementation: The CUDA implementation computes QK similarity and aggregates values to address the overhead of native unfold-based extraction.
  • Limitations: The current native-CUDA implementation remains less efficient than models benefiting from highly optimized dense GPU operators.
  • Evaluation protocol: Throughput is measured on one V100 with FP16 and batch size 64, while training time and memory use are measured on 8× A100s with automatic mixed precision.

E. Downstream Experimental Results

The supplied passages identify downstream evaluation tables for COCO detection and instance segmentation and ADE20K semantic segmentation.

  • COCO: Table 12 reports COCO object detection and instance segmentation results using the Mask R-CNN 1× schedule.
  • COCO: Table 13 compares COCO object detection results using the DINO method.
  • ADE20K: Table 14 compares ADE20K semantic segmentation with UperNet under multi-scale and flip augmentation, including interpolation and extrapolation settings.
  • ADE20K: Table 15 compares ADE20K semantic segmentation results using Mask2Former.

F. Visualization Based on Effective Receptive Field

The paper uses Effective Receptive Field visualizations to compare information aggregation across model stages and datasets. TransNeXt-Tiny shows broader, smoother third-stage coverage and a visual pattern associated with natural visual perception.

  • Multi-stage ERF comparison: ERF visualizations compare four encoder stages across TransNeXt-Tiny and seven other models.The compared models are ConvNeXt-T, Swin-T, CSWin-T, Focal-T, MaxViT-Tiny, BiFormer-S, and SLaK-T.
  • Multi-stage ERF comparison: TransNeXt-Tiny outperforms seven other models in ERF coverage at the third stage, exhibiting a more natural and smoother visual perception.The comparison covers multi-stage outputs shown in Figure 8.
  • Multi-stage ERF comparison: The broader third-stage ERF coverage partially explains TransNeXt’s performance advantage in detection and segmentation tasks, which rely more heavily on lower-stage outputs.This connects the visualization to the reported downstream-task advantage.
  • Cross-dataset ERF comparison: The ERF analysis supports a local-global modeling approach that the paper describes as more akin to biological vision.The paper presents this observation as validation of TransNeXt’s biomimetic design and natural visual perception.
  • Cross-dataset ERF comparison: Figure 9 compares fourth-stage ERFs for TransNeXt-Tiny, ConvNeXt-T, and Swin-T across clean, adversarial, sketch, and corrupted-image datasets.ImageNet-C visualizations include corruption severity levels 1, 3, and 5, with each ERF averaged over 5000 images at 224 × 224 resolution.
Loading 2311.17132v3…