Source-linked AI summary
Modeling Localness for Self-Attention Networks
Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, Tong Zhang
TL;DR
Self-attention captures global dependencies but can underemphasize useful local context. The paper adds a learnable Gaussian bias to attention, applying localness mainly in lower layers, and reports consistent translation improvements across Chinese-English and English-German tasks.
Problem
Self-attention can overlook neighboring relations because its weighted averaging distributes attention across all signals, despite the value of local context.
Method
The paper predicts a Gaussian bias’s central position and dynamic window from intermediate representations, incorporates it into attention, and applies localness modeling to lower layers.
Results
The approach consistently improves translation performance across WMT17 Chinese⇒English and WMT14 English⇒German tasks and model variations.
Takeaways & Limitations
Query-specific window prediction yields the best result among the proposed strategies, while learned bias scopes increase with layer depth.
Takeaways & Limitations
The approach remains to be validated on other tasks, including reading comprehension, language inference, and stance classification.
Abstract
from arXiv · showhide
Self-attention networks have proven to be of profound value for its strength of capturing global dependencies. In this work, we propose to model localness for self-attention networks, which enhances the ability of capturing useful local context. We cast localness modeling as a learnable Gaussian bias, which indicates the central and scope of the local region to be paid more attention. The bias is then incorporated into the original attention distribution to form a revised distribution. To maintain the strength of capturing long distance dependencies and enhance the ability of capturing short-range dependencies, we only apply localness modeling to lower layers of self-attention networks. Quantitative and qualitative analyses on Chinese-English and English-German translation tasks demonstrate the effectiveness and universality of the proposed approach.
1 Introduction
Self-attention captures long-range dependencies but can overlook neighboring relations because attention is distributed across all signals. The paper models localness with a learnable Gaussian bias and evaluates it on translation tasks.
- Self-attention directly attends to all signals, enabling long-range dependency modeling but potentially dispersing attention across neighboring relations.The paper motivates locality modeling as a way to retain global dependency capture while improving attention to local context.
- The proposed localness model predicts a central position and dynamic window from intermediate representations as a learnable Gaussian bias.The central position represents the locality center, while the window represents its scope.
- The Gaussian bias is incorporated into the original attention distribution to produce a revised distribution emphasizing expected local context.
- Localness modeling is applied to lower layers because lower layers tend to capture short-range dependencies, while higher layers capture longer-range dependencies.Multi-head attention can still attend to local regions centered at different positions, preserving complete input-sequence information.
- Experiments on WMT14 English⇒German and WMT17 Chinese⇒English consistently improve translation performance over the strong TRANSFORMER baseline.The approach is also complementary to relative position encoding, and combining them can further improve translation performance.
2 Background
Attention mechanisms dynamically select related representations, while self-attention computes pairwise token attention to capture long-range dependencies. The paper motivates adding locality because global weighted averaging can weaken neighboring-word relations.
- Attention dynamically selects related representations and supports generation tasks such as machine translation and image captioning.
- Self-attention computes attention weights between every pair of tokens in one sequence, directly modeling long-range dependencies.It uses parallel computation and dependency modeling, unlike recurrent alternatives described in the passage.
- At layer l, hidden states attend to layer l−1 states after those states are transformed into queries, keys, and values.The first layer is the word embedding layer.
- Self-attention models global dependencies without considering distance, but weighted averaging over all positions can inhibit neighboring-word relations.
- Modeling neighboring positions can capture phrasal patterns; for example, attending from “Bush” to “held” also emphasizes “a talk” to form “held a talk”.
3 Localness Modeling
The approach models localness with a learnable Gaussian bias that predicts where and over what scope attention should focus, then incorporates this bias into self-attention. It supports flexible window strategies, assigns distinct biases across heads, and investigates how localness modeling integrates with multi-layer TRANSFORMER networks.
- 3.1 Localness Modeling as a Gaussian Bias: A Gaussian bias predicts a central position and window scope from intermediate self-attention representations, then revises the original attention distribution to emphasize expected local context.The central position specifies the locality center, while the window specifies its scope.
- 3.1 Localness Modeling as a Gaussian Bias: The Gaussian bias is added to attention logits, which through softmax is equivalent to weighting the original attention distribution toward nearby positions.The bias values lie between 0 and negative infinity, producing weights between 1 and 0 after exponentiation.
- 3.3 Window Size Prediction: Three window-selection strategies are considered: fixed, layer-specific, and query-specific windows, with the latter two providing more flexible context scopes.The fixed strategy uses a constant window, whereas layer-specific and query-specific strategies derive window sizes from layer or query information.
- 3.3 Window Size Prediction: The query-specific strategy shares a hidden-state transformation with central-position prediction but uses distinct projections for the two scalars, reducing parameters and computation relative to the layer-specific model.The shared hidden state reflects the interdependence of local center and window size.
- 3.4 Incorporating into TRANSFORMER: In multi-head self-attention, each head receives a distinct Gaussian bias so heads can attend to different positions and representation subspaces.The approach assigns separate parameters to predict the central position and window size for each head.
- 3.4 Incorporating into TRANSFORMER: Localness modeling improves more significantly with multi-head attention than with single-head attention, supporting the complementarity between diverse head-level attention and local context modeling.The reported comparison appears in the analysis of multi-head versus single-head attention.
4 Experiments
Experiments evaluate localness modeling across window strategies, encoder layers, attention types, translation tasks, and model variations. Results favor flexible query-specific windows, lower-layer encoder application, and consistent gains across WMT17 Zh⇒En and WMT14 En⇒De.
- Experimental Setup: Experiments use WMT17 Chinese⇒English and WMT14 English⇒German translation tasks with Transformer Base and Big models.The Zh⇒En training data contains about 20.62 million sentence pairs, while both Base and Big configurations are evaluated.
- Window Prediction Strategies: All proposed window prediction strategies improve performance over the baseline, while layer-specific and query-specific windows outperform fixed windows.The flexible strategies adapt local context according to layer or query information and avoid handcrafted window-size parameters.
- Window Prediction Strategies: +0.07 and +0.23 BLEU points are achieved by query-specific prediction over fixed-window modeling on Zh-En and En-De validation sets, respectively.The differences are not always significant, but the flexible strategy consistently outperforms its fixed counterpart across language pairs.
- Layers to be Applied: Applying localness to part of the encoder layers consistently outperforms applying it to all layers in training speed and translation quality.Lower layers benefit more from local context, so subsequent experiments use the lower three layers.
- Attention Networks to be Applied: Localness modeling on decoder self-attention and encoder-decoder attention marginally improves or can harm translation quality, so it is applied to the lower three encoder layers.The main setting uses query-specific window prediction on the lower three encoder layers.
- Main Results: Localness consistently improves translation across language pairs and model variations, and combining it with relative position encoding further improves performance.The approach is reported as efficient, universal, and complementary to relative position encoding.
5 Analysis
The analyses examine how localness modeling interacts with multi-head and multi-layer attention, and whether it improves phrasal-pattern capture. Results show benefits from diverse head-specific scopes, broader scopes in higher layers, and stronger performance on larger n-grams.
- Compatibility with Multi-Head Attention: Multi-head attention gains more from localness modeling than single-head attention, improving by +0.70 versus +0.13.
- Compatibility with Multi-Head Attention: Head-specific windows show that different heads select diverse scopes, with middle layers combining local and global information.The head-specific model assigns each head a learnable unified window size, and the learned distributions vary across heads and layers.
- Analysis on Multi-Layer Attention: Except for the first layer, higher layers favor larger local scopes, matching a progression from short-range to beyond-phrase dependencies.
- Analysis on Multi-Layer Attention: The first layer uses large scopes despite lacking contextualized embeddings, and localness modeling there remains valid.
- Analysis on Phrasal Pattern: Localness modeling underperforms the baseline on unigram translations but consistently improves larger granularities, especially with query-specific scopes on 4-grams to 8-grams.The larger-n-gram improvement is attributed to stronger phrasal-information capture and greater query-specific flexibility.
6 Related Work
Related work establishes self-attention as a mechanism for global dependency modeling while showing that explicit locality and phrase modeling can improve translation. The paper extends these ideas with flexible local scopes rather than fixed restrictions.
- Self-attention directly captures long-distance dependencies and supports parallel dependency modeling in neural machine translation.
- Prior work improves self-attention through relative positions, directional masks, multidimensional features, or explicit locality restrictions.
- Combining relative position encoding with localness modeling further improves performance because the two models are complementary.
- The paper's flexible strategies for placing and zooming local scopes yield better results than fixed-window approaches used in prior work.
7 Conclusion
The paper models local context in self-attention with a learnable Gaussian bias and finds that flexible, especially query-specific, scopes improve translation across English-German and Chinese-English tasks. Learned scopes generally expand in higher layers, while broader validation and stronger localness techniques remain future directions.
- A learnable Gaussian bias enhances local-context capture in self-attention, with query-specific scope prediction achieving the best result through greater flexibility.
- Experiments on English-German and Chinese-English translation demonstrate the proposed approach's effectiveness and universality.
- Higher layers learn larger local scopes, consistent with prior findings about layer-dependent dependency modeling.
- The approach remains to be validated on other tasks and combined with linguistic knowledge or additional Transformer techniques.