Source-linked AI summary
Position Information in Transformers: An Overview
Philipp Dufter, Martin Schmitt, Hinrich Schütze
TL;DR
Transformers lack sensitivity to input order even though word order is essential to language, motivating systematic study of position information. This survey compares existing approaches with unified notation and concludes that the field remains broad, useful for application-aware selection, and not fully understood.
Problem
Transformers are invariant to input reordering, whereas language is sequential and word order affects meaning; existing position methods lacked a broad systematic overview.
Method
The survey provides a unified notation and theoretical, dimensional comparison of over 30 Transformer position models, covering sequential, tree, graph, and crosslingual approaches.
Results
The survey organizes absolute and relative position methods and reports findings including improved training speed and language-modeling perplexity for Shortformer, and large zero-shot translation improvements after reducing position propagation.
Takeaways & Limitations
Application characteristics should inform position-encoding selection, while unified comparison can expose common components and blind spots in current research.
Takeaways & Limitations
Many aspects and differences among position-information models remain not fully understood.
Abstract
from arXiv · showhide
Transformers are arguably the main workhorse in recent Natural Language Processing research. By definition a Transformer is invariant with respect to reordering of the input. However, language is inherently sequential and word order is essential to the semantics and syntax of an utterance. In this article, we provide an overview and theoretical comparison of existing methods to incorporate position information into Transformer models. The objectives of this survey are to (1) showcase that position information in Transformer is a vibrant and extensive research area; (2) enable the reader to compare existing methods by providing a unified notation and systematization of different approaches along important model dimensions; (3) indicate what characteristics of an application should be taken into account when selecting a position encoding; (4) provide stimuli for future research.
1 INTRODUCTION
Transformers support many NLP tasks but are invariant to input reordering, despite language depending on sequence and word order. This survey organizes and theoretically compares diverse position-information methods to clarify their properties and application-relevant selection criteria.
- Motivation: Transformers are central to modern NLP, supporting tasks including machine translation and language modeling through stacked self-attention and feed-forward layers.Layer normalization and residual connections are additional components.
- Motivation: Without position information, a Transformer cannot distinguish an ordered sentence from a multiset of the same units.For example, “the cat chases the dog” differs semantically from an unordered collection of its words.
- Survey scope: Existing methods inject absolute or relative position information through position embeddings, attention-matrix manipulation, or recurrent preprocessing.The literature contains a broad range of approaches for Transformer-based NLP models.
- Survey scope: The survey provides a theoretical comparison of over 30 position models using unified notation and dimensions such as learnable-parameter count.Its goal is analysis and identification of common components and research blind spots, rather than selecting a single best method.
- Contributions: The authors aim to showcase the field’s breadth, enable method comparison, guide application-aware position-encoding selection, and stimulate future research.These objectives are presented as four complementary contributions.
2 BACKGROUND
A Transformer maps sequences through repeated attention-based encoder layers, but its core operations are permutation-equivariant and therefore do not encode word order. Position information is consequently needed to make sequential distinctions available to the model.
- Notation: The paper represents scalars, vectors, and matrices with distinct notation and uses tmax for maximum sequence length.Positions use r, s, t, while vector components use i, j.
- Transformer architecture: A Transformer encoder layer combines self-attention with a feed-forward layer, alongside layer normalization and residual connections.A full Transformer is a composition of l such layers.
- Transformer architecture: Input units such as characters, subwords, or words are converted to vectors through a vocabulary embedding lookup, then padded or truncated to tmax.The resulting embedding matrix supplies the Transformer input.
- Order invariance: For any permutation of the input, the Transformer preserves the corresponding permutation of its output because attention and remaining operations are permutation-equivariant or position-wise.Position information breaks this order invariance so sequential distinctions can affect the computation.
- Transformer architecture: Encoder-decoder Transformers mask future positions in the decoder’s attention matrix, while encoder-only models such as BERT use the Transformer for masked language modeling.The decoder also integrates encoder outputs in sequence-to-sequence settings.
3 RECURRING CONCEPTS IN POSITION INFORMATION MODELS
Position information models recur around absolute versus relative references and three integration locations: the input, attention matrices, or before the output. Adding position embeddings and modifying attention matrices are distinct but closely linked approaches.
- ABSOLUTE VS. RELATIVE POSITION ENCODING: Absolute positions encode a unit’s location within a sentence, whereas relative positions encode its location relative to other units.Relative positions can better preserve decisive relationships when an inserted word shifts absolute positions but changes meaning only slightly.
- ADDING POSITION EMBEDDINGS: Adding position embeddings feeds T(U + P) into the Transformer, combining unit embeddings with a position-information matrix.The first-layer transformation subsequently applies normalization and feed-forward operations to the position-enriched representation.
- MODIFYING ATTENTION MATRIX: Modifying the attention matrix adds absolute or relative position biases directly to attention interactions.The resulting matrix can represent unit-unit, unit-position, and position-position interactions, omitting the attention scaling factor in the notation.
- APE AND MAM: APE and MAM are distinguished because APE learns position embeddings, while MAM directly modifies attention using scalar biases or multipliers.APE can modify attention as one of its effects, making the two approaches highly interlinked despite the distinction.
- INTEGRATION: Position information can be integrated into the input, at each attention matrix, or directly before the output.Information added at the beginning affects only the first layer and must be propagated upward indirectly.
4 CURRENT POSITION INFORMATION MODELS
The survey uses “position information model” for methods integrating position information and distinguishes position encodings from their numerical embedding vectors. It organizes these models by reference point and topic.
- TERMINOLOGY: A position information model is a method that integrates position information into a Transformer.A position encoding is a position ID or relative distance, while a position embedding is the numerical vector associated with that encoding.
- SYSTEMATIZATION: The survey systematizes position information models along reference point and topic.Topic describes a paper’s main subject, while reference point specifies how position encodings are defined.
4.1 SEQUENTIAL
Transformer position methods encode sequence order through absolute or relative information, using embeddings, attention modifications, recurrent preprocessing, and other parameterizations. The survey compares these approaches across mechanisms and reports task-dependent empirical differences.
- Absolute position encodings: Absolute position embeddings add a learned position matrix to unit embeddings before Transformer processing.FLOATER instead models positions with a continuous dynamic function, with sinusoidal embeddings as a special case.
- Absolute position encodings: Shortformer adds position embeddings to keys and queries but not values, allowing cached position-independent value representations.The method attends to outputs from previous subsequences and reintroduces position information in each layer.
- Absolute position encodings: Position-sensitive unit embeddings make each unit representation depend on its position and yield large improvements across Transformer, static-embedding, LSTM, and CNN experiments.This approach replaces simple summation with a position-dependent embedding function.
- Relative position encodings: Relative-position methods modify attention interactions, including key/query terms, position biases, and direction- or distance-aware functions.Some formulations omit position-position interactions, while others directly model pairwise position interactions or use shared parameters across heads.
- Relative position encodings: On several GLUE tasks, the last two Shaw et al. methods perform best, while combining relative and absolute embeddings boosts GLUE performance in another study.Other studies report improved natural-language-understanding performance from combining relative-position models and improved text classification with a lightweight headwise parameterization.
- Relative position encodings: RNN-based position encoding reaches comparable performance to position embeddings, performs better on longer sequences, and improves further when combined with relative position embeddings.The resulting combination is called RR-Transformer.
- Relative position encodings: Stochastic Positional Encoding approximates relative-position interactions with correlated Gaussian-process cross-covariances, making them available to linear-complexity Transformers.The authors report improvements over absolute position encodings on tasks involving long-range dependencies.
4.2 SINUSOIDAL
Sinusoidal encodings provide fixed absolute position information with theoretical relative-distance structure, while later variants adapt them for recurrence, variance, continuity, rotation, and direction awareness.
- Sinusoidal position embeddings: Sinusoidal position embeddings are fixed absolute encodings that remain unchanged during training and can handle arbitrary sequence lengths.They are parameter-efficient because they introduce no learned position parameters.
- Theoretical properties: Sinusoidal position embeddings have dot products determined by relative distance, but the basic construction is unaware of direction.TENER modifies attention to preserve distance structure while distinguishing directions.
- Direction-aware variants: TENER reports massive performance increases for named entity recognition compared to standard Transformer models.Its attention design uses per-head and per-layer learnable parameters and omits the scaling factor after finding better performance.
- Recurrent and adaptive variants: The Universal Transformer recurrently reapplies shared Transformer weights and can halt or continue computation independently at each position.Its position information can be interpreted as adding sinusoidal embeddings at each layer.
- Variance and continuous variants: Maximum variances position embeddings address the claim that sinusoidal variance differs greatly between small and large positions.The proposed remedy uses a hyperparameter for which the authors claim suitable values satisfy k > 1000.
- Rotary variants: Rotary position embeddings rotate unit representations according to position and match learned absolute embeddings on Chinese data.The authors claim benefits for long sequences.
4.3 GRAPHS
Graph position models encode tree or arbitrary-graph structure through absolute or relative representations, but their assumptions and evaluation settings limit direct comparisons.
- Hierarchies (trees): Structural position representations compute absolute and relative positions from dependency-tree distances rather than sequential positions.Relative SPR additionally uses node order, making it suitable for dependency graphs but incompatible with other tree structures.
- Hierarchies (trees): SPR performs best when combined with sequential position information, whereas using SPR alone as the sentence representation causes a large performance drop.Absolute SPR uses distance to the dependency-tree root; relative SPR combines root distances with sequential order.
- Hierarchies (trees): Tree position encodings represent each node by its full root-to-node path, assigning unique positions rather than only encoding path length.TPE can be parameter-free or combine multiple path encodings with learned weights before the first Transformer layer.
- Arbitrary graphs: Cai and Lam add absolute SPR to exploit hierarchical structure in AMR graphs, leaving the relative impact of each position model unclear.Their graph representations aggregate shortest-path label information with a bidirectional GRU.
- Arbitrary graphs: Arbitrary-graph encodings can represent relative position using shortest-path lengths, edge-label sequences, or direction-aware path representations.Graformer uses shortest-path lengths to avoid the computational overhead of encoding edge-label sequences.
- Arbitrary graphs: Graformer explicitly models disconnected graphs with ∞ and omits sequential position information.Its performance is difficult to compare with other approaches because it was not evaluated on the same tasks.
4.4 DECODER
Decoder position methods incorporate target-length information or predict output positions to control sequence length and support non-autoregressive decoding.
- Length-controlled decoding: Length-difference and length-ratio position embeddings incorporate a specified output length into decoder position representations.The length constraint is the gold-standard output length in the described setup.
- Length-controlled decoding: Length-controlled decoding effectively controls output length, and later work adds noise to or predicts the target length.These approaches focus on the decoder, although they could theoretically also be used in the encoder.
- Non-autoregressive decoding: PNAT predicts target sentence length and a permutation, then reorders decoder position embeddings to enable effective non-autoregressive decoding.The model achieves performance improvements in machine translation.
4.5 CROSSLINGUAL
Crosslingual position information is challenging because languages differ in unit ordering, and studies report both benefits from language-aware encodings and harms from excessive order information.
- Crosslingual position sharing: Languages differ in constituent ordering and in how fixed that ordering is, raising the question of whether position information should be shared across languages.English is described as SVO, while other languages permit different or more variable orderings.
- Crosslingual position sharing: Language-specific position embeddings help most transferred languages but appear harmful for some languages.The experiments concern learned absolute position embeddings when transferring monolingual models to multiple languages.
- Crosslingual reordering: Crosslingual position embeddings integrate original and reordered source positions and improve machine translation performance.The reorderings are obtained before integrating both position-encoding variants into the model.
- Crosslingual transfer: Removing a middle-layer residual connection to reduce position-information propagation yields large improvements in zero-shot translation.This result is reported in the context of machine translation.
- Crosslingual transfer: Reducing source-language order information improves crosslingual transfer in the reported zero-shot setting.The study reduces order information through local convolutions, source-order shuffling, or frozen multilingual position embeddings.
4.6 ANALYSIS
Comparative studies find that position-encoding performance depends on task, sequence length, training setting, and injection design. Table 2 organizes the surveyed approaches by shared methodological criteria to facilitate comparison.
- Empirical comparisons: Absolute and relative encodings show similar machine-translation performance, but relative encodings are superior for long sentences.The number of learnable parameters can often be reduced without performance loss.
- Empirical comparisons: RNN and DiSAN achieve similar word-reordering performance, whereas Transformer performs worse; after machine-translation training, Transformer performs best.
- Empirical comparisons: Absolute position embeddings are favorable for classification, while relative embeddings perform better for span prediction.This finding comes from an empirical comparison of 13 position-embedding variants.
- Systematization: Table 2 groups position-information methods by reference point, injection method, learning, recurrence, length generalization, and parameter count.The grouping is intended to make comparisons easier and commonalities faster to identify.
- Empirical comparisons: An argued superiority of relative position embeddings may result from adding them to each attention head rather than from the encoding type itself.The comparison also investigates segment embeddings and parameter sharing across heads and layers.
5 CONCLUSION
The conclusion presents position information as an established but incompletely understood research area. It identifies empirical task comparisons, task-specific word-order importance, and richer text structure as open questions.
- Conclusion: The survey’s unified notation and systematic comparison are intended to foster understanding and stimulate new ideas about position information.
- Open questions: Empirical comparisons of current position-information models across different tasks remain an open question.Existing analyses provide many insights, but many aspects and differences remain incompletely understood.
- Open questions: For some tasks, treating sentences as bags of words may be sufficient because removing position embeddings causes only marginal performance drops.The conclusion calls for identifying tasks where position information is essential.
- Open questions: Future work should investigate whether position-information models can represent tables, document layout, list enumerations, and sentence order.It remains open whether existing methods can integrate these structures or new methods are required.