PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Connection between Local Attention and Dynamic Depth-wise Convolution

Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, Jingdong Wang

arXiv:2106.04263v5cs.CV

TL;DR

Local attention improves efficient vision transformers, but its relationship to depth-wise convolution and the roles of connectivity, weight sharing, and dynamic weights require clarification. The paper reframes local attention as a channel-wise locally connected layer, compares it with dynamic depth-wise convolution, and evaluates replacements for Swin Transformer. Across ImageNet classification, COCO detection, and ADE segmentation, DWNet performs comparably or slightly better with lower computation complexity, while the analysis identifies weight sharing and dynamic weights as important regularization mechanisms.

  • Problem

    The paper examines how local attention’s sparse connectivity, weight sharing, and dynamic weights relate to depth-wise convolution and contribute to efficient vision models.

  • Method

    The paper rewrites local attention as a channel-wise spatially locally connected layer with dynamic connection weights and replaces Swin Transformer’s local attention with static or dynamic depth-wise convolution.

  • Results

    DWNet achieves comparable or slightly higher performance than Swin Transformer on ImageNet classification, COCO object detection, and ADE semantic segmentation with lower computation complexity.

  • Takeaways & Limitations

    The results suggest that local vision transformers benefit from sparse connectivity, weight sharing, and dynamic weights as complementary regularization mechanisms.

  • Takeaways & Limitations

    Local attention loses spatial-order information when collecting keys and values as a set, with positional embeddings only partially remedying this loss.

Abstract

from arXiv · show

Vision Transformer (ViT) attains state-of-the-art performance in visual recognition, and the variant, Local Vision Transformer, makes further improvements. The major component in Local Vision Transformer, local attention, performs the attention separately over small local windows. We rephrase local attention as a channel-wise locally-connected layer and analyze it from two network regularization manners, sparse connectivity and weight sharing, as well as weight computation. Sparse connectivity: there is no connection across channels, and each position is connected to the positions within a small local window. Weight sharing: the connection weights for one position are shared across channels or within each group of channels. Dynamic weight: the connection weights are dynamically predicted according to each image instance. We point out that local attention resembles depth-wise convolution and its dynamic version in sparse connectivity. The main difference lies in weight sharing - depth-wise convolution shares connection weights (kernel weights) across spatial positions. We empirically observe that the models based on depth-wise convolution and the dynamic variant with lower computation complexity perform on-par with or sometimes slightly better than Swin Transformer, an instance of Local Vision Transformer, for ImageNet classification, COCO object detection and ADE semantic segmentation. These observations suggest that Local Vision Transformer takes advantage of two regularization forms and dynamic weight to increase the network capacity. Code is available at https://github.com/Atten4Vis/DemystifyLocalViT.

1 INTRODUCTION

The paper reframes local attention as a channel-wise, spatially local layer with dynamic weights and connects it to depth-wise convolution. Across ImageNet classification, COCO detection, and ADE segmentation, DWNet matches or slightly exceeds Swin Transformer with lower computation complexity.

  • Local attention partitions images into small windows and performs attention within each window, improving memory and computation efficiency for vision tasks.
  • The analysis characterizes local attention through sparse connectivity, weight sharing, and dynamic weight prediction.Positions connect only within local windows; weights are shared across channels or channel groups and predicted per image instance.
  • Local attention and depth-wise convolution share channel-wise, spatially local connectivity but differ in weight sharing and dynamic-weight computation.Depth-wise convolution shares kernels across spatial positions, whereas attention shares weights across channels and computes them from pairwise query-key dot products.
  • Replacing Swin Transformer’s local attention with depth-wise convolution or its dynamic variants yields DWNet while preserving the overall architecture.
  • DWNet achieves comparable or slightly higher performance than Swin Transformer on ImageNet classification, COCO object detection, and ADE semantic segmentation with lower computation complexity.The ablations associate model capability with weight sharing and dynamic weight mechanisms.

2 CONNECTING LOCAL ATTENTION AND DEPTH-WISE CONVOLUTION

The paper analyzes local attention as a channel-wise, spatially locally connected operation whose weights are dynamically computed, then compares its connectivity and regularization properties with depth-wise convolution. Both use local, channel-separated connectivity, but they differ in how weights are shared and generated.

  • Local attention: Local attention aggregates values from the query’s local window using attention weights computed from query-key relationships.The window contains Nk = Kw × Kh positions, and multi-head attention repeats the process over subvectors before concatenating outputs.
  • Local attention: The formulation ignores query, key, and value linear projections for presentation, while vision implementations may use different projections of the same feature.
  • Regularization view: The paper studies sparse connectivity, weight sharing, and dynamic weight as regularization forms affecting complexity, parameter count, and model capability.Sparse connectivity reduces model complexity; weight sharing reduces parameters; dynamic weights specialize connections per instance.
  • Local attention: Local attention is a channel-wise, spatially locally connected layer with dynamically computed weights.Its connectivity pattern is illustrated in Figure 1(c).
  • Local attention: Its aggregation can be written as element-wise multiplication with a weight vector formed from single-head or multi-head attention weights.
  • Local attention: Local attention connects each position only to Nk positions in its window and has no cross-channel connections.Each output channel depends on corresponding input channels in the window, while attention weights can incorporate information across channels within a head.
  • Local attention: Attention weights are shared across channels in single-head attention or within channel groups in multi-head attention.
  • Translation equivalence: Local attention’s translation equivalence depends on whether keys and values change under translation: sparse windows yield block-wise equivalence, whereas dense windows yield translation equivalence.

3 EXPERIMENTAL STUDY

The experiments compare Swin-style local attention with static, depth-wise, and dynamic depth-wise convolution under matched architectures and training settings. Across recognition tasks, depth-wise variants achieve comparable performance with lower complexity, while weight sharing and dynamic weights provide distinct benefits.

  • Experimental setup: The study evaluates local attention and depth-wise convolution on ImageNet classification, COCO object detection, and ADE semantic segmentation using Swin-compatible settings.DWNet variants follow Swin Transformer structure and training/evaluation protocols.
  • ImageNet classification: 14.2% fewer parameters and 15.5% lower computation are reported for tiny DWNet models, while base models reduce the two costs by 15.9% and 16.2%, respectively.These reductions are reported for depth-wise convolution-based networks relative to the corresponding Swin models.
  • ImageNet classification: Local attention and depth-wise convolution perform on par within a 0.1 top-1 and real-accuracy difference for tiny and base ImageNet models.The two dynamic DWNets perform higher in the tiny-model case.
  • Weight sharing: Proper channel-wise weight sharing helps both local attention and local MLP, while position-wise sharing substantially improves depth-wise convolution.For local MLP, channel sharing reduces parameters; for depth-wise convolution, position sharing significantly increases performance.
  • Dynamic weight: Dynamic weights improve both local attention and depth-wise convolution, with Swin’s tiny and base models rising from 80.3% to 81.3% and 82.2% to 83.3%, respectively.The static local MLP is compared with dynamic Swin on ImageNet classification.
  • Dynamic weight: Inhomogeneous dynamic weight computation reaches 81.8 versus 81.4 for attention-based computation, although this difference disappears for large models and detection tasks.The paper attributes the difference partly to vector-based versus set-based window representations.

4 RELATED WORK

Related work frames the paper around three design dimensions: sparse connectivity, weight sharing, and dynamic weight computation. It positions the contribution as an explanation of why lightweight depth-wise convolution can match Transformer performance in vision.

  • Sparse connectivity: Sparse connectivity reduces channel-domain redundancy through depth-wise and group convolutions in established architectures such as MobileNet, ShuffleNet, and ResNeXt.These schemes remove or restrict cross-channel connections.
  • Sparse connectivity: Point-wise convolution has no cross-spatial connections, whereas larger-kernel convolutions and local attention connect positions within local neighborhoods.The related-work comparison distinguishes spatial and channel connectivity patterns.
  • Weight sharing: Convolution primarily shares weights across spatial positions, while attention and token-mixer MLPs share weights across channels.The paper uses this distinction to compare local attention with depth-wise convolution.
  • Dynamic weight: Dynamic convolution methods predict homogeneous weights or region- and position-specific weights, while ViT-style attention learns dynamic connection weights for each position.The paper situates its dynamic depth-wise convolution within these two families.
  • Convolution versus Transformer: Unlike concurrent NLP work that mainly compares convolution and Transformer performance, this paper analyzes their parity through sparse connectivity, weight sharing, and dynamic weight.The stated goal is to explain why the architectures perform on par for vision tasks.

5 CONCLUSION

The conclusion identifies local attention and dynamic depth-wise convolution as closely related through local, channel-sparse connectivity, while their weight-sharing patterns explain key efficiency and performance differences.

  • Sparse connectivity: Local attention benefits from local connections and the absence of cross-channel connections, matching the two sparse-connectivity forms of dynamic depth-wise convolution.The conclusion presents these as shared structural properties.
  • Weight sharing: Channel-wise sharing in local attention reduces attention-weight complexity and slightly improves performance, whereas position-wise sharing in depth-wise convolution improves performance.The conclusion contrasts the principal benefits of the two sharing patterns.

A RELATION GRAPH

The paper represents neural layers through sparse connectivity, weight sharing, and dynamic weight, using these dimensions to relate MLPs, convolutions, and local attention. Local ViT and depth-wise convolution share local channel-wise sparsity but differ mainly in which dimension shares weights.

  • Connectivity: MLP is fully connected, whereas convolution and separable MLP impose structured sparsity on the connection matrix.Convolution connects local spatial neighborhoods; separable MLP separates spatial and channel mixing.
  • Connectivity: Depth-wise separable convolution combines spatial mixing through depth-wise convolution with channel mixing through point-wise 1 × 1 convolution.The depth-wise component is the spatially local operation, while the point-wise component mixes channels.
  • Dynamic weights: Vision Transformer dynamically predicts the spatial-mixing weights, while Local Vision Transformer restricts each output neuron to a local window.Local ViT is therefore a spatially sparser dynamic separable MLP.
  • Weight sharing: Depth-wise convolution and Local ViT have similar channel-wise local connectivity, but depth-wise convolution shares kernel weights across spatial positions whereas attention shares weights across channels.This distinction is the central relation between the two operators.
  • Matrix view: The connection matrix can be organized by channels or positions, with repeated block matrices expressing weight sharing across channels or channel groups.The paper also describes separable MLP as a Kronecker-product approximation of the connection matrix.

C LOCAL ATTENTION VS CONVOLUTION: DYNAMIC WEIGHTS

The paper contrasts dynamic convolution and dot-product attention by how they preserve spatial order and compute local weights. It then combines convolutional and attention mechanisms to obtain two-level dynamic weighting.

  • Dynamic weight computation: Dynamic convolution predicts position-specific weights from features while preserving the ordered arrangement of positions within the local window.Inhomogeneous dynamic convolution predicts weights separately for each position using the feature at the window center.
  • Dynamic weight computation: Dot-product attention predicts weights from feature interactions at corresponding positions and can be viewed as a two-level dynamic scheme.The dynamic component is represented by Θd, while relative position embeddings provide an additional static component.
  • Convolutional attention: Convolutional attention is designed to combine dynamic convolution with dot-product attention while retaining spatial order information and two-level dynamic weight prediction.The framework is presented as a combination of the two weighting mechanisms.
  • Convolutional attention: Pre-convolutional attention first applies convolution to representations and then performs dot-product attention.The convolution may use shared kernel weights across channels, although non-shared weights are also allowed.
  • Spatial order: The convolutional representation differentiates positions by assigning corresponding window offsets to consistent rows, reducing the need for separate positional embeddings.The passage explicitly connects this positional encoding effect to the use of convolution.

D ARCHITECTURE DETAILS

The experiments construct DWNet variants by preserving Swin Transformer’s overall architecture while replacing local attention with depth-wise convolution and dynamic depth-wise convolution.

  • Network construction: DWNet-T and DWNet-B follow Swin Transformer’s overall structure and replace local self-attention with depth-wise convolution using the same window size.The corresponding dynamic and inhomogeneous dynamic variants are also constructed.
  • Architecture comparison: The architecture comparison covers Swin Transformer and DWNet for tiny models, with the base model obtained by changing stage depth and channel dimensions.The architecture details are summarized in Table 8.
  • Dynamic variants: Dynamic depth-wise convolution generates instance-dependent connection weights rather than using only fixed learned kernels.The homogeneous variant obtains a pooled feature vector and uses linear projections to generate kernels shared across spatial positions.
  • Evaluation scope: Table 9 compares ResNet, HRNet, Mixer, ResMLP, gMLP, ViT, DeiT, Swin, DWNet, and dynamic DWNet variants on ImageNet classification.The table accompanies the module-level comparison of sparse connectivity, weight sharing, and dynamic weight.

E SETTING DETAILS

The experiments use Swin Transformer’s training settings to compare models fairly across ImageNet classification, COCO object detection, and ADE semantic segmentation.

  • ImageNet pretraining: ImageNet pretraining uses Swin Transformer’s settings, including 224 × 224 inputs, AdamW, 300 epochs, cosine decay, warm-up, and an eight-GPU batch size of 1024.The models also use the same augmentation and regularization strategies as Swin Transformer.
  • COCO object detection: COCO object detection uses Cascade Mask R-CNN with Swin Transformer’s multi-scale training and optimization settings.The training schedule uses 36 epochs and batch size 16.
  • ADE semantic segmentation: ADE semantic segmentation uses UPerNet and follows Swin Transformer’s optimization, iteration, warm-up, augmentation, and multi-GPU settings.The setup uses 160,000 iterations and 1,500 warm-up iterations.
  • Ablation settings: The static Local MLP baseline removes key and query projections, dot-product, and softmax, replacing dynamic attention weights with learned static parameters shared across images.This isolates the effect of dynamic weight prediction.
  • Resolution study: Depth-wise convolution models are additionally retrained at 384 × 384 resolution using 7 × 7 and 12 × 12 windows.The retraining uses a learning rate of 10^-5, weight decay of 10^-8, and 30 epochs.

F ADDITIONAL EXPERIMENTS AND ANALYSIS

Additional experiments test locality, pretraining, normalization, alternative local-attention architectures, larger images, and squeeze-and-excitation. Across these settings, the results support comparable performance, benefits from locality and dynamic modules, and sensitivity to window-size handling.

  • Locality-based sparsity facilitates ImageNet-1K training beyond sparsity between channels and spatial positions.
  • Dynamic-convolution DWNets achieve performance comparable to Swin Transformer after ImageNet-22K pre-training across classification, detection, and segmentation.The comparisons use the same training settings as Swin Transformer.
  • The experiments compare Swin and DWNet using layer normalization, batch normalization, and centering calibrated batch normalization on ImageNet classification.
  • Depth-wise convolution is also evaluated by replacing local self-attention in SVT and Vision Outlooker in VOLO.
  • With 7 × 7 windows, DWNet performs better after fine-tuning on 384 × 384 images, whereas a 12 × 12 upsampled kernel performs worse than 7 × 7.The authors suspect that upsampling kernel weights is a poor fine-tuning starting point; Swin improves with the larger window.
  • DWNet benefits from squeeze-and-excitation, while Swin Transformer does not; the reason may lie in optimization.

G POTENTIAL STUDIES

The potential studies identify computation balance, normalization and architecture combinations, dynamic-weight extensions, and convolution-style MLP weights as directions for extending the analysis. They also describe how depth-wise convolution and local attention differ in sharing and dynamic-weight design.

  • Depth-wise convolution contributes about 2% of computation in its architecture, so 1 × 1 convolutions dominate the complexity.Suggested remedies include grouped 1 × 1 convolution, channel-wise weighting, or additional depth-wise convolutions.
  • Swin Transformer incurs slightly higher FLOPs for a 7 × 7 window than for 12 × 12 because of greater padding.
  • The comparison of local attention and depth-wise convolution is extended to VOLO and SVT architectures.
  • Dynamic weights are explored for the spatial component in Swin Transformer and dynamic depth-wise convolution, while Lite-HRNet studies them for 1 × 1 convolution.The paper identifies dynamic weighting for both spatial and point-wise components as an open direction.
  • Convolution-style spatial-mixing MLP weights could support larger images and downstream tasks with different image sizes.
Loading 2106.04263v5…