Source-linked AI summary
Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, General
Christoffer Koo Øhrstrøm, Rafael I. Cabral Muchacho, Yifei Dong, Filippos Moumtzidellis, Ronja Güldenring, Florian T. Pokorny, Lazaros Nalpantidis
TL;DR
Vision position encodings adapted from language may incompletely capture vision characteristics. PaPE addresses this with a parabola-based encoding derived from five principles, and experiments report broad generality plus strong classification extrapolation. Its main limitation is moderately increased resource usage and additional parameters dependent on m.
Problem
Language-derived position encodings adapted to vision may incompletely account for vision-specific characteristics, motivating a vision-centric position encoding.
Method
PaPE and PaPE-RI are parabola-based vision position encodings derived from translation invariance, rotation invariance, distance decay, directionality, and context awareness.
Results
Across 8 datasets and 4 vision modalities, PaPE matches the best baseline on 5 datasets and exceeds all baselines on 2; classification extrapolation surpasses all baselines by at least 10.5% at resolution 1024^2.
Takeaways & Limitations
PaPE is presented as a general vision position encoding with strong classification extrapolation across the evaluated settings.
Takeaways & Limitations
PaPE has moderately increased resource usage and additional parameters, both dependent on m.
Abstract
from arXiv · showhide
We propose Parabolic Position Encoding (PaPE), a parabola-based position encoding for vision modalities in attention-based architectures. Given a set of vision tokens-such as from videos, event camera streams, images, or point clouds-our objective is to encode their positions while accounting for the characteristics of vision modalities. Prior works have largely extended position encodings from 1D-sequences in language to nD-structures in vision, but only with partial account of vision characteristics. We address this gap by designing PaPE from principles distilled from prior work: translation invariance, rotation invariance (PaPE-RI), distance decay, directionality, and context awareness. Extrapolation experiments on ImageNet-1K show how PaPE extrapolates remarkably well, improving in absolute terms by up to 10.5\% over the next-best encoding. Generality experiments on 8 datasets across 4 modalities show that PaPE is a general vision position encoding, as PaPE matches the best baseline on 5 datasets and exceeds all on 2 datasets. Code is available at https://github.com/DTU-PAS/parabolic-position-encoding.
1 Introduction & Related Works
PaPE is a vision-specific position encoding motivated by characteristics incompletely covered by language-derived adaptations. It combines five design principles and reports strong extrapolation and broad cross-modality performance.
- Vision transformers often adapt language position encodings to nD structures, but these adaptations may incompletely cover vision-specific characteristics.
- PaPE is designed from translation invariance, rotation invariance, distance decay, directionality, and context awareness.
- PaPE uses parabola-based position encoding for vision modalities, with PaPE-RI providing a rotation invariant version.
- PaPE uses separate query-key transformations, making it compatible with efficient attention kernels.
- 10.5%: PaPE surpasses all baselines by at least this amount at resolution 1024^2 for classification extrapolation.
- 8 datasets across 4 vision modalities: PaPE matches the best baseline on 5 datasets and exceeds all baselines on 2.
2 Preliminaries
Attention measures token similarity through query-key dot products, while positional information is added because attention is permutation invariant. Position vectors represent token locations in modality-specific dimensions.
- Queries, keys, and values are derived from token embeddings through learnable matrices Wq, Wk, and Wv.
- Sij = ⟨qi, kj⟩: attention measures token similarity using dot products between query-key pairs.
- Because attention is permutation invariant, positional information must encode the arrangement of tokens.
- Position vectors ri ∈ R^p represent token locations, with p = 2 for images and p = 3 for point clouds.
3 Design Principles
The paper identifies five properties for vision position encodings: translation invariance, task-dependent rotation invariance, distance decay, directionality, and context awareness.
- Translation invariance: Translation invariance makes encoding depend on relative arrangements rather than absolute global locations.
- Rotation invariance: Rotation invariance is useful when object orientation should not affect prediction, but it is task-dependent because orientation can carry motion information.
- Distance decay: Distance decay makes nearby tokens interact more strongly than distant tokens, biasing attention toward local interactions.
- Directionality: Directionality modulates attention according to geometric direction, including above, below, left, right, and diagonal relations.
- Context awareness: Context awareness lets the model adapt decay and direction strength to token content, supporting both local neighborhoods and long-range interactions.
4 Parabolic Position Encoding 1
PaPE models relative token positions with sums of parabolas whose coefficients are content-dependent, separating distance, direction, and semantic contributions. PaPE-RI imposes constraints for rotation invariance, while position-aware query-key transformations preserve efficient-kernel compatibility.
- Parabolic Position Encoding: PaPE reshapes attention into a sum of m parabolas using projected relative positions as dependent variables.
- Parabolic Position Encoding: The learnable projection Wp maps relative positions, while Wa and Wb derive parabola coefficients from token content.
- PaPE-RI: PaPE-RI enforces rotation invariance by setting bi = 0 and constraining Wp, but this removes directionality.
- Design principles: Negative quadratic coefficients penalize distances away from the origin, producing distance decay.
- Design principles: Content-derived direction coefficients align attention with preferred directions, making PaPE directional and context-aware.
- Parabolic Position Encoding: PaPE’s attention score separates distance, direction, and semantic terms.
- Efficient attention kernels: Separate position-aware query-key transformations preserve standard attention computation and provide plug-and-play compatibility with efficient kernels.
- Efficient attention kernels: The compatibility construction increases query and key dimensionality by p^2 + 2p + 2, adding 10 dimensions for p = 2 and 17 for p = 3.
5 Experiments
Experiments evaluate PaPE across extrapolation, generality, robustness, and component ablations. PaPE extrapolates strongly, performs broadly across modalities, and benefits from directionality, distance decay, and context awareness.
- Extrapolation: PaPE improves accuracy by up to 10.5% over the next-best encoding at resolution 1024^2 after training at 224^2.The evaluation uses no additional training or model modifications at higher resolutions.
- Generality: Across 8 datasets and 4 vision modalities, PaPE achieves the highest average score of 66.3 versus RoPE’s 65.3.PaPE matches the best baseline on 5 datasets, exceeds all baselines on 2, and underperforms only on ScanNet.
- Spatio-temporal performance: PaPE delivers its largest absolute accuracy gain on UCF101, reaching 49.5 versus RoPE’s 43.9, and exceeds RoPE by 1.2 mAP on GEN1.It also ties sinusoidal encodings for top accuracy on DvsGesture.
- Modality-specific findings: Rotation-invariant encodings do not outperform rotation-variant alternatives in the point-cloud experiments.The authors suggest that strict rotation invariance may trade away representation flexibility.
- Out-of-distribution generalization: PaPE does not degrade more than baselines on ImageNetV2 and ImageNet-Renditions evaluated without additional training.It ties RoPE on ImageNetV2 and performs best on ImageNet-Renditions.
- Ablations: Removing distance decay, directionality, or context awareness causes an accuracy drop of around 2%, while removing softplus yields 77.5 accuracy.Adding Wp contributes a further 0.3 accuracy improvement.
6 Limitations & Future Work
PaPE’s main limitation is moderately increased resource usage and additional parameters, both dependent on m.
- PaPE incurs moderately increased resource usage and additional parameters that depend on m.Future work targets lowering m, removing Wp, and integrating PaPE into efficient attention kernels.
7 Conclusions
PaPE is introduced as a principled, vision-centric position encoding, with PaPE-RI providing rotation invariance under stated parameter conditions. The appendix derives this property through quadratic and linear-term proofs.
- 7 Conclusions: PaPE and PaPE-RI are vision-centric position encodings derived from prior principles and compatible with efficient attention kernels.The stated principles include translation and rotation invariance, distance decay, directionality, and context awareness.
- 7 Conclusions: 10.5% is the minimum absolute improvement over all baselines reported for classification extrapolation at resolution 1024^2.The contribution passage reports this as the principal extrapolation result.
- 7 Conclusions: PaPE-RI satisfies rotation invariance when the specified conditions on b_i, a_iℓ, and W_p hold.The proof reduces position-dependent terms and uses R^T R = I_p for rotations R ∈ SO(p).
- 7 Conclusions: The appendix verifies the quadratic and linear terms separately before completing the proof of the position-encoding identities.The quadratic-term argument uses symmetry of M, while the linear-term proof expands the query-key dot product block by block.
B Extrapolation Details
The extrapolation analysis compares encodings with and without position interpolation across resolutions. PaPE remains the strongest classification-extrapolation encoding in the best-configuration comparison, despite interpolation being detrimental to it.
- B Extrapolation Details: Training exposes positions [1, 14], whereas resolution 1024^2 expands the test-time range to [1, 64] without position interpolation.This follows from training at 224^2 with patch size 16 and testing at higher resolutions.
- B Extrapolation Details: Position interpolation rescales positions by 224/R to map higher-resolution test positions back into the training range.At R = 448, the scaling factor is 0.5.
- B Extrapolation Details: PaPE remains the best classification-extrapolation encoding even when interpolation is applied to encodings that benefit from it.The best-performing comparison uses interpolation for nD-sincos, RoPE, and RoPE-Mixed, but not for nD-ALiBi, LookHere, PaPE, or PaPE-RI.
C Additional UCF101 Results
Additional UCF101 experiments confirm PaPE’s leading accuracy over RoPE under both the original and parameter-matched evaluations. The reported differences are statistically significant, while efficiency measurements indicate a moderate resource increase for one PaPE setting.
- C Additional UCF101 Results: PaPE’s advantage over RoPE on UCF101 is statistically significant at p=0.011 across the official 3-fold cross-validation results.The additional results report noticeable gains on all folds and provide fold-level accuracies, means, and standard deviations.
- C Additional UCF101 Results: Parameter-matched experiments retain PaPE’s highest score over RoPE on UCF101, with a statistically significant difference at p=0.026.Baselines and PaPE-RI were retrained on UCF101 and DvsGesture to match PaPE’s parameters.
- C Additional UCF101 Results: 7% to 20% is the reported resource-usage increase for PaPE with m = 50 on 224^2 ImageNet-1K images.Resource usage is averaged over 1000 samples with batch size 1.
E Efficiency
PaPE adds moderate computational and parameter overhead while retaining practical resource and timing profiles for vision transformers.
- PaPE increases effective head size by p^2 + 2p + 2 and introduces the positional parameters W_a, W_b, and W_p.The resource analysis varies m for ViT-B/16 on ImageNet-1K at resolution 224^2 with batch size 1.
- 7%–20%: PaPE increases resource usage for the m = 50 configuration compared with the lowest-resource baseline.For m = 64, the relative increase is 9%–23%.
- 1%–16%: PaPE increases the parameter footprint of a standard 86.4M-parameter ViT-B.The authors characterize this as a moderate parameter increase.
- 0.6–2.4 ms: PaPE increases training step time, while inference time rises by 0.1–0.2 ms over the fastest baselines.The relative increases are 5%–19% for training and 7%–14% for inference, which may matter in some applications.
F Model Analysis
PaPE’s decomposed attention separates positional and semantic contributions for model analysis, while its polynomial formulation supports context-dependent positional functions and efficient-kernel compatibility.
- Model Analysis: PaPE enables separate analysis of positional and semantic information in attention heads.The score z quantifies their relative importance, and top-attended keys are selected using a cumulative attention threshold τ.
- Model Analysis: Most heads balance position and semantics, while specialized heads cluster in early layers L1–L5.Examples include position-focused L2H2 and semantics-driven L1H10, using τ = 80%.
- Polynomial Generalizations of PaPE: PaPE can be viewed as the first instance of multivariate polynomials on relative positions with context-dependent coefficients.The paper leaves broader polynomial generalizations for future work.
- Efficient Attention Compatibility: PaPE’s polynomial terms can be separated into feature vectors for efficient attention kernels when coefficients factor across query and key content.In PaPE, the key-side content function is set to 1, so coefficients depend on the query content.
- Polynomial Generalizations of PaPE: Multivariate polynomial extensions can represent bilinear forms such as (r_j − r_i)^T B(r_j − r_i), which the former family cannot represent.Here B can be any matrix in R^(p×p).
- nD-ALiBi and PaPE-RI: PaPE-RI subtracts a content-dependent squared-distance matrix, whereas nD-ALiBi subtracts a distance matrix with constant head-wise scaling.Both induce translation and rotation invariance, but only PaPE-RI is compatible with efficient attention kernels.
I Training Details
The experiments use shared Transformer settings and dataset-specific configurations across video, event-camera, image, detection, and point-cloud tasks.
- Common Configuration: All models use AdamW with a OneCycle learning-rate schedule and cosine decay, except UniTR on nuScenes, which uses Adam.Unless otherwise specified, Transformer settings match ViT-B in layers, heads, dimensionality, and head size.
- Video and Event-Camera Tasks: UCF101 uses ViViT-style 2-tubelets, up to 5 samples per video, RandAugment, HorizontalFlip, and official 3-fold cross-validation.Reported test performance is the mean across the official splits.
- Video and Event-Camera Tasks: DvsGesture and GEN1 use Spiking Patches with σ = 256 and T = 100 ms for asynchronous, spatially sparse event tokens.Temporal positions are rescaled by 1/50000 on DvsGesture and 1/100000 on GEN1; GEN1 samples the most recent 500K events.
- Common Configuration: The hyperparameters are shared across position encodings for each dataset.
- Image and Detection Tasks: ImageNet-1K uses a stratified 99%/1% training-validation split, uniform initial class probabilities, and augmentations including RandAugment and MixUp.The official validation split serves as the test set.
- Image and Detection Tasks: COCO combines a ViT-B backbone with a ViTDet neck and YOLOv10 head, while ScanNet and ModelNet40 use Point Transformer V3 without pretrained initialization.COCO drops shear and rotation augmentation because they corrupt ground-truth bounding boxes.