PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking and Improving Relative Position Encoding for Vision Transformer

Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, Hongyang Chao

arXiv:2107.14222v1cs.CV

TL;DR

The paper addresses unclear and controversial evidence about whether relative position encoding works effectively for vision transformers and whether it can replace absolute encoding. It reviews existing methods, analyzes their design factors, and proposes efficient 2D image RPE methods. The proposed methods improve DeiT-S by 1.5% top-1 Acc and DETR-ResNet50 by 1.3% mAP over original models without adjusting hyperparameters or settings.

  • Problem

    Relative position encoding is effective in natural language processing, but its efficacy and suitability for 2D vision data remain unclear and controversial.

  • Method

    The paper reviews existing RPE methods, analyzes directional, contextual, interaction, and computational factors, and proposes four efficient image RPE methods for vision transformers.

  • Results

    1.5% top-1 Acc for DeiT-S and 1.3% mAP for DETR-ResNet50 are achieved over original models without adjusting hyperparameters or settings.

  • Takeaways & Limitations

    Relative position encoding can replace absolute encoding for image classification, while absolute encoding remains necessary for object detection because pixel position matters for localization.

  • Takeaways & Limitations

    The paper identifies extending the method to high-resolution semantic segmentation and non-pixel inputs such as point clouds as future work.

Abstract

from arXiv · show

Relative position encoding (RPE) is important for transformer to capture sequence ordering of input tokens. General efficacy has been proven in natural language processing. However, in computer vision, its efficacy is not well studied and even remains controversial, e.g., whether relative position encoding can work equally well as absolute position? In order to clarify this, we first review existing relative position encoding methods and analyze their pros and cons when applied in vision transformers. We then propose new relative position encoding methods dedicated to 2D images, called image RPE (iRPE). Our methods consider directional relative distance modeling as well as the interactions between queries and relative position embeddings in self-attention mechanism. The proposed iRPE methods are simple and lightweight. They can be easily plugged into transformer blocks. Experiments demonstrate that solely due to the proposed encoding methods, DeiT and DETR obtain up to 1.5% (top-1 Acc) and 1.3% (mAP) stable improvements over their original versions on ImageNet and COCO respectively, without tuning any extra hyperparameters such as learning rate and weight decay. Our ablation and analysis also yield interesting findings, some of which run counter to previous understanding. Code and models are open-sourced at https://github.com/microsoft/Cream/tree/main/iRPE.

1 Sun Yat-sen University 2 Microsoft Research Asia

Transformers require explicit positional information because self-attention is invariant to sequence ordering, while vision raises unresolved questions about adapting relative position encoding from 1D sequences to 2D spatial data. The paper reviews existing methods, analyzes their design factors, and introduces efficient image-specific relative position encodings with reported gains across classification and detection.

  • Motivation: Self-attention cannot capture token ordering on its own, so transformers need explicit positional representations for structured inputs.Absolute encodings represent individual token positions, whereas relative encodings model pairwise distances and relations.
  • Motivation: Vision-specific RPE remains uncertain because language methods target 1D sequences, while images and videos contain highly structured 2D spatial relationships.The paper specifically questions whether naive 1D-to-2D extension is suitable and whether directional information matters.
  • Approach: The paper reviews existing methods and analyzes direction, context, query-key-value interactions, relative embeddings, and computational cost to guide RPE design.These factors support a comprehensive empirical analysis of relative position encoding in vision transformers.
  • Approach: O(n^2d) computational cost is reduced to O(nkd), where k ≪ n, through an efficient relative encoding implementation.The implementation is intended for high-resolution images whose token counts can be very large.
  • Results: Four image RPE methods improve DeiT-S by 1.5% top-1 Acc and DETR-ResNet50 by 1.3% mAP over their original models without hyperparameter or setting adjustments.The methods are described as efficient, simple, and pluggable into self-attention layers.
  • Results: Relative encoding can replace absolute encoding for image classification, whereas absolute encoding remains necessary for object detection because pixel position matters for localization.The paper also reports that relative encodings may be shared across attention heads with comparable contextual-mode performance and equal MACs to unshared encodings.

3. Method

The paper reviews prior relative position encodings and introduces efficient 2D image RPE methods that model direction, distance, and interactions with self-attention embeddings. The proposed design uses bias or contextual modes, a piecewise distance function, and an efficient implementation for vision transformers.

  • Previous Relative Position Encoding Methods: Prior methods differ in how they incorporate relative positions into queries, keys, values, and attention biases.The review covers Shaw’s key encoding, Transformer-XL’s query biases, Huang’s query-key interactions, and SASA’s horizontal and vertical 2D decomposition.
  • Bias Mode and Contextual Mode: The proposed iRPE framework separates input-independent bias mode from input-dependent contextual mode.Bias mode uses a trainable scalar, whereas contextual mode uses a trainable vector that interacts with queries, keys, or values.
  • A Piecewise Index Function: A piecewise index function preserves differentiated treatment of long-range relative distances instead of assigning all distances beyond a threshold the same encoding.The paper contrasts piecewise mapping with clipping and uses the function to index shared relative-position encodings.
  • 2D Relative Position Calculation: The 2D relative-position methods include undirected Euclidean and Quantization mappings and directed Cross and Product mappings.The Cross method computes horizontal and vertical encodings separately and summarizes them; the methods use finite buckets to share encodings.
  • Efficient Implementation: The efficient implementation reduces relative-position encoding cost from O(n^2d) to O(nkd), where k ≪ n.It pre-computes shared indexed encodings in O(nkd), then assigns them to token pairs with a lower-cost O(n^2) operation.

4. Experiments

Experiments analyze design choices in image RPE and show consistent gains across ImageNet classification and COCO detection, with low computational overhead. Results highlight the roles of direction, contextual encoding, query/key interactions, bucket count, and task-dependent position information.

  • Analysis on Relative Position Encoding: Directed methods generally outperform undirected methods on vision transformers, indicating that relative direction matters for structured image inputs.Cross and Product outperform Euclidean and Quantization methods in the reported comparison.
  • Analysis on Relative Position Encoding: Contextual relative position encoding outperforms bias encoding because it adapts the encoding to input features rather than keeping it static.
  • Analysis on Relative Position Encoding: Sharing relative position encodings across attention heads is nearly cost-free in contextual mode but significantly reduces accuracy in bias mode.Both shared and unshared contextual schemes achieve an average top-1 accuracy of 80.9%; the authors therefore adopt sharing for parameter savings.
  • Analysis on Relative Position Encoding: Piecewise and clip functions perform similarly for image classification, whereas piecewise encoding is better for high-resolution object detection with long sequences.The piecewise function can distinguish positions beyond the bucket threshold, while clipping assigns them the same encoding.
  • Analysis on Relative Position Encoding: Top-1 accuracy rises from 79.9 to 80.9 before 50 buckets, after which further improvement is insignificant for DeiT-S with a 14 × 14 feature map.The authors identify 50 buckets as a balance between computational cost and accuracy in this setting.
  • Comparison on Image Classification and Object Detection: Position encoding has task-dependent effects: relative encoding matches or exceeds absolute encoding in classification, while absolute encoding is better for DETR detection.The authors conjecture that classification benefits from local information, whereas detection benefits from an absolute-position prior for locating objects.
  • Comparison on Image Classification and Object Detection: The proposed encoding improves DeiT by 1.5%/1.0%/0.6% for DeiT-Ti/DeiT-S/DeiT-B and improves DETR by 1.3mAP and 1.7mAP at 150 and 300 epochs.The DeiT gains use relative encoding only on keys, while adding it to queries and values yields further improvements; the method adds at most 1% computational cost with efficient implementation.
  • Visualization: Visualization suggests that shallow transformer blocks emphasize neighboring patches, while higher blocks lose this pattern after accumulating local information.The authors interpret RPE as injecting a Conv-like locality bias into otherwise globally attending transformer layers.

5. Related Work

The paper situates vision transformers within transformer-based modeling and reviews absolute and relative position encoding, including their use in vision tasks.

  • Vision Transformers: Vision transformers split images into patches, add position encodings, and process the resulting tokens with transformer architectures.DeiT and ViT use patch embeddings with absolute position encodings, while DETR applies position encodings to transformer features from a CNN backbone.
  • Open Question: Relative position encoding was established for language modeling, but its suitability for spatially structured 2D images remains unclear.The paper highlights open questions about extending 1D methods to 2D and about the importance of directional information.

6. Conclusions and Remarks

The paper concludes that its 2D relative position methods improve vision transformers with little added complexity, while analysis identifies task-specific and architectural design principles.

  • Conclusions: The proposed methods improve both image classification and object detection while adding negligible computational complexity.
  • Task-Specific Findings: Relative position encodings can replace absolute encodings for image classification, but absolute encodings remain necessary for object detection.The detection requirement is linked to predicting object locations.
  • Design Principles: The analysis recommends sharing encodings across heads, modeling positional directivity, and using relative encoding to emphasize local patches in shallow layers.Shared contextual encodings achieve comparable performance to non-shared ones while saving parameters.
  • Future Work: Future work will extend the method to high-resolution tasks such as semantic segmentation and non-pixel inputs such as point clouds.

—— Supplementary Material ——

The supplementary material expands methodological details, adds experiments, and reports implementation-related analyses for the proposed relative position encoding methods.

  • Additional Experiments: The supplement provides additional details for several main-text sections and adds experiments testing the effectiveness and generality of iRPE.
  • Implementation Analysis: It also includes visualizations, weight-initialization details, computational-complexity analysis, and inference-time comparisons.The visualizations cover Euclidean, Quantization, Cross, and Product methods.

1. Visualization of 2D Relative Position

The visualization compares four 2D relative position encoding schemes by showing how offsets are assigned to shared or distinct buckets around reference positions.

  • Visualization Setup: The visualizations use red stars for reference positions, with colors indicating buckets whose relative positions share encodings.The DeiT visualization uses 14 × 14 patch positions and considers top-left and center reference locations.
  • Euclidean Method: Euclidean encoding is undirected because positions with equal Euclidean distance share a bucket.For example, offsets (−1, 0) and (1, 0) receive the same encoding.
  • Quantization Method: Quantization improves Euclidean bucketing by mapping distinct distances to integer levels, increasing buckets for adjacent positions.It addresses cases where nearby offsets with different distances otherwise share a Euclidean bucket.
  • Cross Method: Cross encoding is directed, combining separate horizontal and vertical encodings that depend on relative distances and directions.Offsets sharing an axis distance share the corresponding horizontal or vertical encoding.
  • Product Method: Product encoding is directed and assigns independent encodings to offsets even when their horizontal or vertical distances match.It is more efficient than Cross because it requires no extra addition operation.

2. Weight Initialization

Relative position weights are initialized to zero, with no observed performance difference from normal-distribution initialization; weight decay is omitted because its effect is negligible.

  • Relative position weights are initialized with zero, and normal-distribution initialization produces no performance difference.
  • Weight decay is not applied to relative position encoding weights because its effect on final performance is negligible.

3. Computation Complexity

Shared and unshared relative position encodings have the same MACs because broadcast operations preserve equivalent computational complexity across attention heads.

  • Shared and unshared relative position encodings have identical MACs across attention heads.For bias mode, both schemes cost O(hn^2); for contextual mode, both cost O(hndk).
  • In bias mode, broadcast addition costs O(hn^2), while contextual mode uses broadcast multiplication costing O(hndk).

4. Injecting Previous RPE Methods into DeiT

The comparison adapts prior 1D relative position encoding methods to 2D images using the authors’ Product method, while reporting runtime and memory for the DeiT-S reference model.

  • The DeiT-S reference model without RPE processes 1,096 images/s and uses 8,930 Mb of memory.
  • Prior 1D relative position encoding methods are adapted to 2D images using the authors’ Product method.The piecewise function g(x) is replaced with clip function h(x), and encoding weights are shared across attention heads.
  • Shaw’s, Transformer-XL’s, and Huang’s relative position encodings are evaluated as DeiT-S variants.

5. Training and Test Settings of DETR

The supplied experiments use the original DETR training protocol and also report iRPE results on Swin-T and fine-grained transfer-learning datasets, including runtime observations.

  • DETR follows its original training protocol with an ImageNet-pretrained ResNet-50 backbone and frozen BatchNorm layers.Transformer blocks use Xavier initialization; images are resized within the stated shortest- and longest-side limits, with flipping and random cropping.
  • Contextual RPE on QKV improves Swin-T to 81.9% top-1 accuracy on ImageNet.The original Swin-T without RPE obtains 80.5%, while RPE bias mode adds 0.8%.
  • DeiT-B with iRPE on keys reaches 93.4% on Stanford Cars and 84.9% on CUB200 2011.These exceed the original DeiT-B results of 92.1% and 83.4% by 1.3 and 1.5 percentage points, respectively.
  • Inference runtime and memory are measured on an Nvidia V100 GPU with batch size 128, and iRPE on keys is reported as more effective.
Loading 2107.14222v1…