PaperlayerView exact revision on arXiv

Source-linked AI summary

LocalViT: Analyzing Locality in Vision Transformers

Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, Michele Magno, Luca Benini, Luc Van Gool

arXiv:2104.05707v2cs.CV

TL;DR

Vision transformers model global interactions well but lack locality for exchanging information within nearby regions. This paper systematically evaluates locality mechanisms and inserts one into the feed-forward network, finding gains across design choices and transformer architectures.

  • Problem

    Vision transformers lack an efficient mechanism for modelling local dependencies between nearby pixels alongside self-attention's global connectivity.

  • Method

    The paper adds 2D depth-wise convolution followed by a non-linear activation to the feed-forward network and evaluates activation, layer placement, and expansion-ratio choices.

  • Results

    Locality-enhanced transformers outperform their baselines, and the mechanism generalizes across five vision transformer architectures.

  • Takeaways & Limitations

    Locality is a broadly applicable design component for vision transformers, with lower-layer placement and feed-forward expansion identified as important choices.

Abstract

from arXiv · show

The aim of this paper is to study the influence of locality mechanisms in vision transformers. Transformers originated from machine translation and are particularly good at modelling long-range dependencies within a long sequence. Although the global interaction between the token embeddings could be well modelled by the self-attention mechanism of transformers, what is lacking is a locality mechanism for information exchange within a local region. In this paper, locality mechanism is systematically investigated by carefully designed controlled experiments. We add locality to vision transformers into the feed-forward network. This seemingly simple solution is inspired by the comparison between feed-forward networks and inverted residual blocks. The importance of locality mechanisms is validated in two ways: 1) A wide range of design choices (activation function, layer placement, expansion ratio) are available for incorporating locality mechanisms and proper choices can lead to a performance gain over the baseline, and 2) The same locality mechanism is successfully applied to vision transformers with different architecture designs, which shows the generalization of the locality concept. For ImageNet2012 classification, the locality-enhanced transformers outperform the baselines Swin-T, DeiT-T, and PVT-T by 1.0%, 2.6% and 3.1% with a negligible increase in the number of parameters and computational effort. Code is available at https://github.com/ofsoundof/LocalViT.

I. INTRODUCTION

Vision transformers model global relationships effectively but lack efficient local information exchange. LocalViT investigates adding locality within the feed-forward network and tests its design choices and generalization across transformer architectures.

  • Motivation: Vision transformers capture global dependencies through self-attention but lack a mechanism for modelling local dependencies between nearby pixels.
  • Method: LocalViT introduces locality into the feed-forward network using depth-wise convolution, inspired by the structural similarity between feed-forward networks and inverted residual blocks.Image tokens are rearranged into a 2D feature map for local processing, while the class token bypasses the feed-forward network.
  • Findings: Locality alone improves baseline transformer performance, while activation choice, lower-layer placement, and larger feed-forward expansion can further improve classification accuracy.The empirical conclusions identify four properties of the investigated locality mechanism.
  • Experimental analysis: The controlled study examines the effects of depth-wise convolution, activation function, layer placement, and hidden-dimension expansion ratio.
  • Generality: The same locality mechanism generalizes across five vision transformers, including DeiT, Swin, T2T-ViT, PVT, and TNT.The reported architectures span different transformer designs.

II. RELATED WORK

Transformer models originated in machine translation and use attention to aggregate information across an entire input sequence. Their application has since expanded toward vision and robotics.

  • Origins: Transformers were introduced for machine translation and are especially effective at modelling long-range dependencies between sequence elements.
  • Applications: Subsequent work adapted transformers to vision and robotics, including approaches that learn mappings from language.
  • Scope: The related work situates vision transformers within broader efforts to extend transformer models beyond machine translation.

B. Locality vs. global connectivity

CNNs provide local information aggregation, while transformers provide global token connectivity; this paper examines how to combine these complementary properties efficiently in vision transformers.

  • CNNs aggregate local information through sliding-window convolutions, while transformers model long-range dependencies through self-attention.
  • Depth-wise convolution is parameter- and computation-efficient because it operates independently on each channel.
  • Vision-transformer images are converted into patch-token sequences, allowing self-attention to expand the effective receptive field across the whole image.
  • The feed-forward network applies two fully connected transformations position-wise, with an expanded hidden dimension for richer feature representations.
  • Rearranging tokens into a 2D lattice recovers token proximity, enabling the feed-forward network to use convolutional operations for locality.

B. Locality

The proposed locality mechanism augments the feed-forward network with depth-wise convolution, using the similarity between transformer feed-forward networks and inverted residual blocks.

  • The transformer locality gap arises because 1×1 convolution lacks adjacent-pixel interaction while self-attention models global rather than local dependencies.
  • The design adapts the inverted residual structure by adding depth-wise convolution between hidden-dimension expansion and squeezing operations.
  • The proposed module reshapes tokens into a 2D feature map, applies two 1×1 convolutions and a depth-wise convolution, then reshapes the result back into tokens.
  • Activation functions are treated as an important design choice, with ReLU6, h-swish, squeeze-and-excitation, efficient channel attention, and combinations evaluated.

C. Class token

Because the class token prevents exact spatial rearrangement with image tokens, the method separates it before the locality-enhanced feed-forward network and concatenates it afterward.

  • A trainable class token is added to the image-token embedding and exchanges information with image tokens through self-attention for classification.
  • Depth-wise convolution requires separating the class token from the image tokens before rearranging the image tokens into a feature map.
  • The transformed image tokens are concatenated with the class token after the feed-forward network.
  • Although excluded from the feed-forward network, the class token still participates in self-attention-based information exchange and aggregation.

IV. EXPERIMENTAL RESULTS

The experiments first evaluate locality in DeiT-T, then test design choices and generalization across several vision-transformer architectures.

  • Experiments begin by testing depth-wise-convolution locality in DeiT-T before evaluating activation, placement, and expansion-ratio choices.The study then applies locality-enhanced variants to T2T-ViT, PVT, TNT, and Swin transformer architectures.

A. Implementation details

Locality is introduced across five vision transformers with architecture-specific considerations, using reduced TNT and Swin variants for faster experiments and a shared ImageNet2012 protocol.

  • Locality is introduced into DeiT, Swin, T2T-ViT, PVT, and TNT, with modifications adapted to their differing architectures.The T2T module is also modified because it contains transformer blocks with feed-forward networks.
  • TNT-T reduces embedding dimension from 384 to 192, while Swin-M reduces third-stage transformer blocks from 6 to 2.These smaller variants are created for faster experiments.
  • ImageNet2012 provides 1.28M training images and 50K validation images across one thousand classes.Training follows the DeiT protocol with 224 × 224 crops, cross-entropy, label smoothing, weight decay, and AdamW.
  • Validation uses center crops, and experiments run on 8 NVIDIA TITAN RTX GPUs.

B. Influence of the locality

Controlled experiments assess depth-wise-convolution locality and expansion ratio, finding accuracy gains with little added computational or parameter cost.

  • Table I investigates locality brought by depth-wise convolution, while Table II examines non-linear activation choices with γ fixed at 4.
  • 1.5% and 3.0% Top-1 accuracy gains result from adding locality at γ = 4 and γ = 6, respectively.Compared with the baseline, parameter growth is marginal and computational growth negligible.
  • 0.3% and 1.2% Top-1 accuracy gains occur without depthwise convolution at γ = 4 and γ = 6, respectively.These changes introduce no additional parameters or computation compared with DeiT-T.

C. Activation functions

Activation functions, channel-attention modules, locality placement, and feed-forward expansion are evaluated as complementary design choices for LocalViT-T.

  • Activation functions: Replacing ReLU6 with h-swish increases the Top-1 accuracy gain over baseline from 1.5% to 2.2%.Adding ECA provides a further 0.1% improvement while introducing only 60 parameters.
  • Activation functions: Reducing the SE reduction ratio from 192 to 4 increases the Top-1 accuracy gain from 2.6% to 3.6%.The computational complexity remains almost unchanged while the parameter count increases.
  • Activation functions: Table III compares locality in all, low, middle, and high transformer layers, while Table IV investigates feed-forward hidden-layer expansion ratios.
  • Activation functions: The experiments use h-swish combined with SE, retaining 4 channels after the squeeze operation as a balance between parameters and accuracy.The paper concludes that local information is important in vision transformers and that efficient modules can be introduced into them.

D. Placement of locality, expansion ratio, and discussion

Locality placement and feed-forward expansion substantially affect LocalViT performance, while locality generalizes across transformer designs and improves object localization. The added complexity can reduce throughput, especially for PVT-T and Swin-T.

  • Placement of locality: Moving locality from lower to higher stages decreases accuracy, showing that local information is especially important in lower layers.Enabling locality progressively from lower layers improves performance, because aggregated local information can propagate upward.
  • Expansion ratio: Increasing the feed-forward expansion ratio from 1 to 4 raises Top-1 accuracy from less than 70% to nearly 75%, while almost doubling model complexity.The expansion ratio γ provides a trade-off between network performance and model complexity.
  • Generalization across architectures: LocalViT improves classification across DeiT, T2T-ViT, TNT, PVT, and Swin transformer designs, with gains of 2.6%, 0.8%, 2.3%, and 3.1% reported for several baselines.The reported gains are 2.6% and 1.0% over DeiT-T and DeiT-S, 0.8% over T2T-ViT-7, 2.3% for LocalViT-TNT, and 3.1% over PVT-T.
  • CNN comparison: Locality-enhanced transformers can be comparable to or outperform CNNs in pairwise comparisons across several model families.Examples include LocalViT-T versus MobileNetV2, LocalViT-S versus ResNet-50, and LocalViT-PVT versus DenseNet-169.
  • Feature-map comparison: LocalViT produces better object localization in the compared feature maps than transformers without the locality mechanism.The comparison uses last-layer feature maps from DeiT and LocalViT on ImageNet inputs.
  • Discussion: Locality increases inference cost: throughput decreases by less than 10% for LocalViT versus DeiT-T and TNT-T, but by up to 24% versus PVT-T and Swin-T.The throughput measurements were conducted on a single NVIDIA TITAN Xp GPU.

V. CONCLUSION

The paper introduces locality into transformer feed-forward networks using 2D depth-wise convolutions and evaluates how design choices affect performance. Applying the mechanism to five vision transformers supports its generality.

  • V. CONCLUSION: The proposed locality mechanism adds 2D depth-wise convolutions followed by a non-linear activation function to the transformer feed-forward network.Token embeddings are rearranged into a 2D feature map, with the class token split before and concatenated after the enhanced feed-forward network.
  • V. CONCLUSION: Controlled studies examine activation function, layer placement, and expansion ratio as factors influencing locality-mechanism performance.The mechanism is applied successfully to five different vision transformers, supporting its generality.
Loading 2104.05707v2…