Source-linked AI summary
Conditional Positional Encodings for Vision Transformers
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Chunhua Shen
TL;DR
Vision Transformers need positional information, but conventional encodings restrict longer inputs and can conflict with translation equivalence. The paper introduces locally conditioned CPE through PEG and builds CPVT, which shows stronger performance, arbitrary-resolution generalization, and negligible cost.
Problem
Conventional positional encodings limit longer-sequence handling and weaken translation equivalence, while positional information remains necessary for vision Transformers.
Method
CPE dynamically generates positional encodings from local token neighborhoods through PEG and is used to build CPVT.
Results
CPVT achieves stronger performance, generalizes to arbitrary input resolutions, and improves segmentation and detection performance for pyramid Transformers.
Takeaways & Limitations
The proposed encodings naturally process longer input sequences, preserve desired translation equivalence, and add negligible computational cost.
Takeaways & Limitations
The stated patch-partitioning assumption requires image height and width to be divisible by the patch size.
Abstract
from arXiv · showhide
We propose a conditional positional encoding (CPE) scheme for vision Transformers. Unlike previous fixed or learnable positional encodings, which are pre-defined and independent of input tokens, CPE is dynamically generated and conditioned on the local neighborhood of the input tokens. As a result, CPE can easily generalize to the input sequences that are longer than what the model has ever seen during training. Besides, CPE can keep the desired translation-invariance in the image classification task, resulting in improved performance. We implement CPE with a simple Position Encoding Generator (PEG) to get seamlessly incorporated into the current Transformer framework. Built on PEG, we present Conditional Position encoding Vision Transformer (CPVT). We demonstrate that CPVT has visually similar attention maps compared to those with learned positional encodings and delivers outperforming results. Our code is available at https://github.com/Meituan-AutoML/CPVT .
1 INTRODUCTION
Vision Transformers offer flexible self-attention but positional encodings can limit sequence-length generalization and translation equivalence. The paper introduces conditional positional encodings and CPVT, reporting stronger performance and arbitrary-resolution generalization.
- Self-attention captures long-distance information but is permutation-invariant, so Transformers require positional encodings to represent token order.
- Absolute positional encodings can restrict longer-sequence inference and disrupt the desired translation equivalence in vision tasks.
- CPE dynamically generates positional encodings from local token neighborhoods rather than using predefined, input-agnostic encodings.
- CPE provides stronger explicit bias toward translation equivalence, which the paper identifies as important for improving Transformer performance.
- CPVT builds vision Transformers on CPE and achieves better performance than previous vision Transformers.
- CPE generalizes to arbitrary input resolutions and boosts segmentation and detection performance for pyramid Transformers.
2 RELATED WORK
Prior positional-encoding methods differ in whether they encode absolute or relative position and whether they are fixed or learnable. Related approaches address sequence length, efficiency, or richer positional information with different trade-offs.
- Absolute positional encodings add fixed sinusoidal or learnable vectors to inputs, with learnable encodings jointly optimized with model parameters.
- Relative positional encodings represent distances between tokens and can support translation equivariance and sequences longer than those seen during training.
- Other approaches include complex-value embeddings, rotary embeddings, and continuous dynamical models for encoding absolute or relative position.
- Convolutional methods have modeled local relations in speech recognition and machine translation, whereas this work processes two-dimensional vision images.
3 VISION TRANSFORMER WITH CONDITIONAL POSITION ENCODINGS
The paper motivates conditional positional encodings as a way to retain positional information while addressing fixed-length and translation-equivalence limitations. PEG generates encodings from local 2D neighborhoods and supports CPVT, with GAP offering a translation-invariant classification variant.
- 3.1 MOTIVATION: Learnable absolute encodings limit sequences longer than training lengths and make vision Transformers non-translation-equivariant by assigning unique vectors to patches.
- 3.1 MOTIVATION: Removing positional encodings permits longer sequences but reduces DeiT-tiny ImageNet top-1 accuracy from 72.2% to 68.2%.
- 3.1 MOTIVATION: Interpolation can adapt positional encodings to longer sequences but requires fine-tuning to avoid a marked performance drop.
- 3.2 CONDITIONAL POSITIONAL ENCODINGS: The proposed design characterizes local relationships, enabling longer-sequence generalization while preserving translation equivariance and providing absolute position through boundary awareness.
- 3.2 CONDITIONAL POSITIONAL ENCODINGS: PEG reshapes flattened tokens into 2D image space and applies a local function, efficiently implemented with a 2D convolution, to produce conditional encodings.
- 3.3 CONDITIONAL POSITIONAL ENCODING VISION TRANSFORMERS: CPVT follows ViT and DeiT architectures while replacing their positional encodings with conditional encodings generated by PEG.
- 3.3 CONDITIONAL POSITIONAL ENCODING VISION TRANSFORMERS: CPVT-GAP replaces the class token with global average pooling, which is inherently translation-invariant and improves image-classification performance with CPE.
4 EXPERIMENTS
Experiments evaluate CPVT on ImageNet, higher resolutions, global average pooling, and broader vision-transformer settings. CPVT generalizes without fine-tuning and improves performance while adding little computational overhead.
- Generalization to higher resolutions: CPVT-Ti improves from 73.4% to 74.2% when evaluated at 384×384 after training at 224×224, while DeiT-tiny falls from 72.2% to 71.2%.The CPVT-versus-DeiT gap increases at larger input resolutions.
- Global average pooling: 74.9% top-1 accuracy is achieved by CPVT-Ti with GAP on ImageNet, outperforming DeiT-tiny by 2.7%.GAP also reduces computation by removing class-token interactions with image patches.
- Efficiency: CPVT-Ti adds 1,728 parameters with a 3×3 depth-wise PEG, compared with 37,632 learnable positional-encoding parameters in DeiT-tiny.The reported extra computation is 0.34M FLOPs versus 2.1G total model FLOPs.
- Performance comparison: CPVT models obtain better top-1 accuracy than DeiT with similar throughput, including further gains when inputs are upscaled without fine-tuning.The GAP variant is reported as a new state-of-the-art for vision Transformers.
- Performance comparison: 75.9% accuracy is obtained by CPVT with hard distillation, exceeding DeiT-tiny by 1.4%.RegNetY-160 is used as the teacher.
- Pyramid transformer architectures: PEG is applied to PVT and Swin, where it improves ImageNet accuracy by 3.1% for PVT-tiny and 1.15% for Swin-tiny.The method is also evaluated on semantic segmentation and object detection tasks.
5 ABLATION STUDY
Ablations test whether PEG contributes positional information, where it is inserted, how many PEGs are used, and how it compares with other encodings. Results support PEG’s positional role and favor repeated placement across early encoder blocks.
- Positional encoding or merely a hybrid?: 71.3% top-1 accuracy with PEG applied only to Q and K exceeds DeiT-tiny without positional encoding at 68.2%.The result is close to 72.4% when PEG is applied to Q, K, and V.
- Positional encoding or merely a hybrid?: 71.3% accuracy from a randomly initialized, fixed 3×3 PEG is 3.1% higher than DeiT without positional encoding.The comparison attributes the improvement to positional information rather than learned added parameters.
- Positional encoding or merely a hybrid?: 72.3% accuracy from a fixed learned PEG is close to the 72.4% achieved by a learnable PEG.Fixing the learned PEG during training still helps optimization in this experiment.
- PEG position in CPVT: PEG reaches approximately 72.4% when placed at positions [0, 3], with position 0 outperforming position -1 because of the receptive-field difference.Position -1 is before the first encoder, whereas position 0 follows the first encoder block.
- PEG depth and placement: 73.4% accuracy is obtained by inserting one PEG after each of the first five encoders, outperforming stacking PEGs within one block.A four-layer PEG reaches 72.9%, while combining a single PEG with learnable absolute encoding improves the baseline by 0.5%.
- Comparisons with other positional encodings: A learnable absolute encoding plus relative positional encoding reaches 72.4% on ImageNet but scales less well to untrained resolutions than CPE.The paper presents this as a major difference between the combination and CPE.
- Comparisons with other positional encodings: 72.7% for repeated learnable encodings is 0.7% lower than the PEG 0-5 configuration.The comparison supports continued benefits from PEG stacking and the use of relative information through PEGs.
6 CONCLUSION
CPVT dynamically generates positional encodings from local token neighborhoods, achieving stronger performance while naturally processing longer sequences and preserving a desired translation-equivalence bias. The method is easy to implement and incurs negligible cost.
- CPVT dynamically generates position encodings from the local neighbors of each input token.
- The positional encodings can naturally process longer input sequences and keep the desired translation equivalence in vision tasks.
- The positional encodings achieve stronger performance than previous positional encodings in extensive experiments.
- The method is easy to implement and comes with negligible cost.
- Translation-equivariance means that output feature maps are translated equally with the input signal, but CPVT is not strictly translation-equivariant because of zero padding.
B.1 ARCHITECTURE VARIANTS OF CPVT
CPVT variants share architectures with corresponding DeiT and ViT models, while zero padding supplies useful absolute positional information. Multiple PEG insertions improve accuracy up to a point, after which additional insertions become incremental.
- Architecture variants: CPVT-B matches ViT-B and DeiT-B, while CPVT-S and CPVT-Ti match DeiT-small and DeiT-tiny architectures.
- Zero-padding ablation: Removing zero padding reduces CPVT-S ImageNet performance to 70.5%, indicating that zero padding and absolute positional information matter for object classification.
- Multiple PEGs: 73.4% top-1 accuracy is achieved by the tiny model with five PEG positions, surpassing DeiT-tiny by 1.2%.
- Multiple PEGs: 80.5% is achieved by CPVT-S with multiple PEG positions.
- Multiple PEGs: Adding more PEGs helps up to a level, after which additional positions provide only incremental gains.
B.5 CLASSFICATION EVALUATION OF SWIN WITH PEG
Adding PEG to Swin improves ImageNet classification accuracy for Swin-Tiny.
- 82.25% top-1 accuracy is achieved by Swin-Tiny with PEG, improving over 81.10% by 1.15%.
B.6 EVALUATION ON SEGMENTATION AND DETECTION
PEG improves PVT performance across segmentation and detection tasks, while the simplest depth-wise convolution form is sufficient and preserves the transformer-based design.
- Semantic segmentation: PEG boosts PVT-tiny by 3.1% mIoU on ADE20K semantic segmentation.
- Object detection: PEG improves PVT-tiny by 2.0% mAP under COCO’s standard 1× schedule and by 2.4% mAP under the 3× schedule.
- PEG design: Large 7×7 kernels and dense convolutions provide limited improvement, while stacking more depth-wise convolution layers is not significantly better.
- PEG design: A single depth-wise 3×3 convolution is used as the simplest PEG implementation and achieves performance on par with or better than recent methods.
- Comparison to Lambda Networks: CPVT’s transformer-based positional encoding is contrasted with Lambda Networks’ 2D relative positional encoding, whose RPE2D-d128 configuration obtains 68.7%.
D.3 QUALITATIVE ANALYSIS OF CPVT
Attention visualizations show that CPVT with PEG learns locality patterns similar to DeiT with original positional encodings, unlike DeiT without positional encoding.
- For a 224×224 image represented by 196 patches, the study visualizes each attention head’s 196×196 score matrix to assess positional information.The analysis compares normalized attention weights across encoder blocks and heads.
- DeiT with original positional encodings attends strongly to each patch’s local neighbors and weakly to distant patches.The diagonal and nearby elements receive stronger attention, indicating learned locality.
- Removing positional encodings causes DeiT patches to produce similar attention weights and fail to attend preferentially to nearby patches.
- CPVT with PEG learns an attention pattern similar to DeiT with original positional encodings, indicating that PEG provides position information implicitly.
- Across several encoder blocks, DeiT and CPVT share diagonally aligned locality patterns, although projections into different Q and K subspaces can shift the pattern slightly.
D.4 COMPARISON WITH OTHER APPROACHES
The paper compares CPVT with convolutional, hybrid, and pyramid vision Transformer approaches on ImageNet validation data. CPVT-S-GAP slightly outperforms ConViT-S while using fewer parameters and FLOPs, and CPVT also exceeds a convolutional-projection DeiT variant.
- CPVT-S-GAP outperforms ConViT-S by 0.2% with 4M fewer parameters and 0.8G fewer FLOPs.The comparison is conducted on the ImageNet validation set.
- The comparison groups CvT, ConViT, and CoAtNet into plain and pyramid models, adapting CPVT to the PVT and Swin pyramid frameworks.The adaptations support fair comparison across model categories.
- A CvT-flavored DeiT using one convolutional projection reaches 70.6% top-1 accuracy on ImageNet validation, below the paper’s 72.4% result.The model is trained from scratch under strictly controlled settings with the convolutional projection inserted at position 0.
- Tables 15 and 16 report ImageNet-1k validation comparisons at 224×224 resolution.