Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding

Yang Li; Si Si; Gang Li; Cho-Jui Hsieh; Samy Bengio

Source-linked AI summary

Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding

Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, Samy Bengio

arXiv:2106.02795v3Jun 5, 2021cs.LGcs.AIcs.CV

TL;DR

Transformers require positional information because attention is order-invariant, while existing encodings can inadequately capture multi-dimensional spatial relationships. The paper introduces learnable Fourier features modulated by an MLP, and reports consistent improvements across multi-dimensional spatial tasks, including better generalization to unseen positions. It also identifies open questions about modeling complex spatial relations and combining positional and content embeddings.

Problem
Attention-only models need positional encodings, but existing approaches may inadequately capture multi-dimensional spatial relationships or generalize to unseen positions.
Method
The method maps multi-dimensional positions through learnable Fourier features and an MLP, producing continuous encodings whose parameters do not grow with sequence length.
Results
Across image generation, object detection, image classification, and sparse UI spatial modeling, the method consistently outperforms benchmark positional encodings.
Takeaways & Limitations
The representation generalizes significantly better to unseen positions and can capture spatial relationships beyond independently encoded dimensions.
Takeaways & Limitations
The method is evaluated with positional and content embeddings combined by addition, leaving concatenation for these image tasks as future work.

Abstract

from arXiv · show

Attentional mechanisms are order-invariant. Positional encoding is a crucial component to allow attention-based deep model architectures such as Transformer to address sequences or images where the position of information matters. In this paper, we propose a novel positional encoding method based on learnable Fourier features. Instead of hard-coding each position as a token or a vector, we represent each position, which can be multi-dimensional, as a trainable encoding based on learnable Fourier feature mapping, modulated with a multi-layer perceptron. The representation is particularly advantageous for a spatial multi-dimensional position, e.g., pixel positions on an image, where $L_2$ distances or more complex positional relationships need to be captured. Our experiments based on several public benchmark tasks show that our learnable Fourier feature representation for multi-dimensional positional encoding outperforms existing methods by both improving the accuracy and allowing faster convergence.

1 Introduction

Transformers’ attention is order-invariant, so positional encoding is essential for representing sequence and spatial order. The paper targets flexible, learnable encoding for multi-dimensional positions.

Attention-only Transformers lack the ordinal information that recurrent and convolutional architectures capture through sequential or spatial dependencies.
Positional encoding maps token positions or pixel coordinates into vectors that are combined with content representations for attention computation.
Fixed sinusoidal encodings use predefined wavelengths that may not capture task-dependent positional information.
The paper designs a learnable function that maps multi-dimensional positions through Fourier features and an MLP.
The encoding function is inductive, parameter-efficient, applicable to arbitrary test positions, and composable across subsets of dimensions.

2 Background

Existing positional encodings trade flexibility, spatial structure, and generalization. The paper motivates learnable Fourier features because they can encode multi-dimensional positions through shift-invariant relationships tied to Euclidean distance.

2.1 Positional Encoding: Self-attention uses dot-product similarity between content-and-position representations, making positional encoding relevant to attention strength.
2.1 Positional Encoding: Discrete position embeddings can model complex relationships but may generalize poorly to positions rarely encountered during training.
2.2 Encoding Multi-Dimensional Spatial Positions: For 2D inputs, independently encoding and concatenating dimensions can produce directionally distorted spatial similarities.
2.2 Encoding Multi-Dimensional Spatial Positions: Learned embeddings can model complex spatial relations but incur overhead as the number of unique positions grows and require adjustments for changed image sizes.
2.2 Encoding Multi-Dimensional Spatial Positions: The paper seeks a holistic multi-dimensional encoding that provides inductive bias and learnability while generalizing to unseen positions.
2.3 Fourier Features: Fourier features approximate a Gaussian shift-invariant kernel whose inputs are multi-dimensional positions, motivating their use for spatial encoding.

3 Learnable Fourier Features Positional Encoding

The proposed encoding maps multi-dimensional positions into Fourier features, modulates them with an MLP, and projects them for attention-based computation. Learnable parameters provide continuous, length-independent representations with spatial inductive bias.

The method maps an M-dimensional position into a K-dimensional feature vector for downstream attention computation.
Learnable Fourier Features: Fourier features represent positions using concatenated sinusoidal functions whose trainable projection controls feature orientation and wavelength.
Learnable Fourier Features: The dot product of Fourier feature vectors depends on the positional difference x−y, giving the representation shift invariance.
Learnable Fourier Features: Normal initialization makes the Fourier-feature dot product approximate a Gaussian kernel over the original positions.
Learnable Fourier Features: The resulting representation introduces an inductive bias toward L2 distances.
MLP Layer: An MLP modulates the Fourier features before projection and combination with content embeddings.
The learnable parameters are independent of sequence length, and the function accepts arbitrary positions such as those from images with different resolutions.
For complex spatial relations, coordinate dimensions can be partitioned into groups that share the encoding pipeline before concatenation.

4 Experiments

Experiments across image generation, object detection, image classification, and widget captioning compare Learnable-Fourier+MLP with existing positional encodings. The method generally improves convergence, accuracy, generalization to unseen positions, and captioning metrics.

Image Generation: Learnable-Fourier+MLP outperforms image-generation baselines in convergence speed and final accuracy, measured by lower bits per dim.Reformer evaluates positional encodings on held-out ImageNet 64x64 data; Figure 3 reports means and 95% confidence intervals from three repeats.
Object Detection: Learnable-Fourier+MLP shows a more pronounced benefit when DETR must detect objects at unseen image dimensions, widening the performance gap between Embed-2D and other methods.The benchmark reserves image sizes 736, 768, and 800 for validation and disables position normalization, requiring generalization to positions absent during training.
Image Classification: Learnable-Fourier+MLP achieves a 74.5% Precision@1 on ImageNet validation, compared with 73.6% for Embed1D in ViT-B/16.The models are trained for 90 epochs on ImageNet using a 14 × 14 grid of 16×16 patches.
Widget Captioning: In widget captioning, Learnable-Fourier+MLP outperforms Embed-4D on BLEU-1, BLEU-2, ROUGE, and CIDEr while using fewer parameters.Embed-4D uses 5.11M parameters, whereas the proposed method uses 5.07M; the 4/1 variant is significantly better than Embed-4D on BLEU-1, ROUGE, CIDEr, and METOER.
Widget Captioning: Widget-captioning results favor grouping bounding-box coordinates as 2/2 or 4/1 over 1/4, indicating that more complex distances than L2 may need to be modeled.The 4/1 and 2/2 variants generally do not differ significantly, except that 4/1 exceeds 2/2 on CIDEr with p = 0.042.
Ablations: Ablations show that combining Learnable-Fourier features with an MLP performs better than either component alone, while the MLP is crucial in widget captioning.Fixed-Fourier+MLP-2/2 performs best among fixed-feature variants, and Learnable-Fourier-2/2 performs worse than corresponding variants on every metric.

5 Discussion

The experiments favor holistic multi-dimensional positional encoding with Fourier features and an MLP, especially for generalizing to unseen positions, while leaving important design questions open.

Flattened-sequence encodings such as Embed-1D and Sine-1D do not perform well on image tasks.
Holistic multi-dimensional representations often outperform independently encoded dimensions that are later concatenated.
An MLP generally improves Fourier-feature positional encoding before it is mixed with content embedding.MLP alone performs competitively for object detection but poorly for UI modeling with sparse spatial structures.
The proposed encoding generalizes significantly better to unseen widget positions than baseline methods.The method represents positions continuously and preserves shift-invariant relative positional relationships during learning.
Future work should examine concatenating positional encoding with content embedding instead of using addition.
The method’s spatial assumptions remain incomplete because rectangular UI objects can have alignment, overlap, and domain-specific relationships beyond Euclidean distance.The paper identifies more direct modeling of these complex spatial relationships as future work.

6 Conclusion

The paper introduces learnable Fourier features for multi-dimensional positional encoding and evaluates them across spatial tasks, reporting consistent gains over benchmark methods.

The paper presents learnable Fourier features as a novel approach to positional encoding.
The approach is evaluated on image generation, object detection, image classification, and sparse spatial-structure modeling in user interfaces.
The encoding joins content representation through concatenation or element-wise addition, with positional encoding required for strong accuracy in Transformer-like models.
The positional encoding function maps multi-dimensional positions into vectors using Fourier features passed through an MLP.Its initialization makes encoding inner products approximate Euclidean distances, while learning adapts the representation to the task.

B.1 PE Analysis for Image Generation Tasks

Image-generation analysis compares initialized and trained positional similarity, showing that training broadens the learned spatial relationship and that the MLP produces cleaner structure.

Training makes positional relationships less concentrated than the initially seeded Fourier features, enlarging the similarity region around each position.The analysis uses dot-product similarity on a 64 × 64 image and compares initialization with the representation after 100K training steps.
Learnable-Fourier positional similarity is visualized for representative image positions including corners and the center.The analyzed positions are Top-Left, Top-Right, Center, Button-Left, and Bottom-Right on the image grid.
Removing the MLP modulator produces a less clean learned positional encoding than using the MLP.

B.2 PE Analysis for Object Detection Tasks

Object-detection visualizations compare several positional encodings on a 42 × 42 grid, highlighting distinct spatial biases and the relatively stable ball-shaped similarity of Learnable-Fourier+MLP.

B.2 PE Analysis for Object Detection Tasks: The comparison evaluates positional similarity for five representative locations on DETR’s 42 × 42 grid.The locations are Top-Left, Top-Right, Center, Button-Left, and Bottom-Right.
B.2 PE Analysis for Object Detection Tasks: Embed-2D slowly learns greater similarity for closer positions but favors orthogonal directions because it concatenates independently embedded dimensions.
B.2 PE Analysis for Object Detection Tasks: MLP positional similarity is skewed toward bottom and right directions, and its dot-product intensity is not normalized like cosine similarity.Because MLP lacks shift invariance, the analyzed pattern may not generalize across the full grid.
B.2 PE Analysis for Object Detection Tasks: Sine-2D exhibits a cross-shaped similarity pattern, with position normalization concentrating similarity toward the center.
B.2 PE Analysis for Object Detection Tasks: Learnable-Fourier+MLP mostly maintains a ball-shaped similarity pattern throughout training.

B.3 PE Analysis for Widget Captioning Tasks

The widget captioning task models positions as four-coordinate bounding boxes, producing complex and increasingly skewed positional relationships. KL regularization improves distributional symmetry and accuracy for one Fourier-feature variant, but has little apparent effect on image-based tasks.

Positional relationships: Widget captioning positions are four-coordinate bounding boxes, making positional relationships more complex than in image generation.The analysis treats point-wise similarity as a building block for bounding-box similarity.
Positional relationships: The learned positional relationship becomes more concentrated and skewed toward the anti-diagonal during training.This behavior is observed for Learned-Fourier+MLP 2/2 initialized with γ = 100.
KL regularization: KL regularization constrains Wr toward a zero-centered Gaussian, helping maintain symmetry across positional directions.The regularizer is added to the overall training loss.
Results: KL regularization clearly improves accuracy for Learned-Fourier+MLP 2/2 in widget captioning, but appears to have little effect on image generation and object detection.The experiment uses α = 1.

C Additional Ablation Studies

The ablations compare learnable Fourier features with multidimensional sinusoidal alternatives and examine MLP augmentation and Fourier-weight initialization. Learnable frequencies perform better than the tested fixed-frequency baseline, while MLPs are broadly beneficial and normal initialization can outperform uniform initialization.

Multidimensional sinusoidal encoding: The multidimensional sinusoidal ablation combines vertical and horizontal positions using manually determined Transformer-style frequencies.The encoding uses sine and cosine of a linear combination of the two coordinates.
Multidimensional sinusoidal encoding: Transformer MD-Sine performs poorly on the Reformer ImageNet64 task, and adding an MLP still leaves it below Learnable Fourier.Learnable Fourier features avoid the need to manually tune fixed combination frequencies.
MLP augmentation: MLP augmentation often improves existing sinusoidal or embedding-based positional encodings and provides consistently good results across tasks.The widget captioning example specifically reports improved overall accuracy for Sine-4D with an MLP.
Fourier-weight initialization: In object detection, initializing Wr from a uniform [0, 1] distribution performs worse than initializing it from a normal distribution.The ablation also compares models with and without MLP layers.

D Hyperparameters & Parameter Sizes

The implementation uses task-specific Fourier and MLP hyperparameters while keeping parameter counts broadly comparable across positional-encoding variants. Learnable Fourier features with an MLP are slower than embedding and sinusoidal alternatives in Reformer, whereas runtime differences are negligible in the other reported tasks.

Parameter sizes: Learnable-Fourier+MLP uses roughly the same number of trainable parameters as Embed-2D in Reformer.The broader Reformer parameter comparison is summarized in Table 7.
Reformer hyperparameters: Reformer Fourier-based methods use |F| = 768, |H| = 32, D = 768, and γ = 1.0, with jointly mapped vertical and horizontal positions.The MLP applies LayerNorm before dense projections, and G = 1 models the two spatial coordinates jointly.
DETR configuration: DETR Learnable-Fourier+MLP uses γ = 1.0 and a 2 × 256 GeLU MLP, while DETR variants have roughly equal trainable parameter counts.The parameter comparison is summarized in Table 8.
Widget-captioning hyperparameters: The Fourier-based widget-captioning variants use G = 1, 2, 4 for 1/4, 2/2, and 4/1 position groupings, respectively, with γ = 100.The experiments also use |F| values of 128, 64, and 32 across these grouping variants.
Computational complexity: Reformer training runs at 1.86 steps/second for Embed2D, 1.81 for Sine2D, and 1.22 for Learnable-Fourier+MLP.Runtime impact is negligible for object detection and widget captioning because other training computation dominates.

E Unseen Position Distribution in the Widget Captioning Dataset

The widget-captioning test set includes many unseen positions, most of which lie inside the convex hull of training positions. This distribution is visualized after mapping the unseen positions to two dimensions with PCA.

Unseen-position distribution: 2685 of 2692 unseen widget-captioning test positions lie inside the convex hull of the training positions.The paper uses this distribution to characterize unseen positions in the test set.
Visualization: The unseen positions are mapped to two dimensions with PCA and plotted together with the training-position convex hull.Figure 16 visualizes the resulting distribution.