Source-linked AI summary
Convolutional Self-Attention Networks
Baosong Yang, Longyue Wang, Derek Wong, Lidia S. Chao, Zhaopeng Tu
TL;DR
SANs can disperse attention across all elements, and conventional multi-head attention does not exploit interactions between head features. CSANs add parameter-free local convolution and interactions across adjacent heads; machine-translation experiments report gains over Transformer and prior locality methods.
Problem
SANs can overlook neighboring-element relations and phrasal patterns, while separate attention heads fail to exploit useful interactions across subspaces.
Source: §1 · p. 1Method
Results
Across three language-pair translation tasks, CSANs consistently improve over Transformer and outperform prior locality methods in translation quality and training efficiency.
Source: §1 · p. 1Takeaways & Limitations
The analyses suggest that locality benefits SANs, cross-head feature interaction can further improve performance, and dynamic weights can outperform fixed weights for local feature extraction.
Source: §6 · p. 5
Abstract
from arXiv · showhide
Self-attention networks (SANs) have drawn increasing interest due to their high parallelization in computation and flexibility in modeling dependencies. SANs can be further enhanced with multi-head attention by allowing the model to attend to information from different representation subspaces. In this work, we propose novel convolutional self-attention networks, which offer SANs the abilities to 1) strengthen dependencies among neighboring elements, and 2) model the interaction between features extracted by multiple attention heads. Experimental results of machine translation on different language pairs and model settings show that our approach outperforms both the strong Transformer baseline and other existing models on enhancing the locality of SANs. Comparing with prior studies, the proposed model is parameter free in terms of introducing no more parameters.
1 Introduction
CSANs address SANs’ limited treatment of neighboring elements and interactions across attention heads with a parameter-free convolutional approach. The paper reports improved translation performance across language pairs and gains over prior locality methods.
1 Introduction
- SANs capture dependencies regardless of distance by attending to all elements, and multi-head attention applies attention in multiple representation subspaces.The paper identifies these properties as strengths of SANs and multi-head attention.Source: §1 · p. 1
- SANs’ attention over all elements can disperse attention and overlook neighboring-element relations and phrasal patterns, while separate attention heads miss useful cross-head interactions.The paper presents these as two limitations motivating CSANs.Source: §1 · p. 1
- CSANs use parameter-free 1D convolution to focus attention on neighboring elements and extend convolution across an attention-head axis to enable local feature interaction between adjacent subspaces.The paper describes the 1D operation as restricting attention to a local window and the 2D operation as extending the area across heads.Source: §1 · p. 1
- Across WMT14 English-to-German, WMT17 Chinese-to-English, and WAT17 Japanese-to-English, CSANs consistently improve over Transformer and boost translation quality and training efficiency over prior locality methods.The comparison to Transformer is reported across language pairs; the comparison to prior locality methods covers translation quality and training efficiency.Source: §1 · p. 1
2 Multi-Head Self-Attention Networks
Multi-head SANs transform an input sequence into query, key, and value representations, then apply attention in multiple subspaces. The head outputs are concatenated into the final representation.
2 Multi-Head Self-Attention Networks
- Given input sequence X, the model transforms it into queries Q, keys K, and values V.The projections use trainable matrices, and the resulting representations are split into H subspaces.Source: §2 · p. 2
- The query, key, and value representations are divided among H subspaces, each with its own attention computation.The paper denotes each subspace with an h index.Source: §2 · p. 2
- The final output representation O concatenates the outputs generated by the multiple attention models.Each attention model operates on a separate subspace.Source: §2 · p. 2
3 Approach
The approach uses convolution to strengthen local token dependencies and to model interactions among attention heads. Its 1D and 2D variants respectively restrict attention to a local token window and extend that window across adjacent heads.
3 Approach
- The approach targets both neighboring-token dependencies and interactions between features extracted by different attention heads.Source: §3 · p. 2
3.1 Locality Modeling via 1D Convolution
- 1D convolution restricts attention to a fixed, position-centered window of M + 1 tokens, and locality is applied only in lower SAN layers.Lower-layer distance-aware local information is intended to complement higher-layer distance-agnostic global information.Source: §3.1 · pp. 2–3
3.2 Attention Interaction via 2D Convolution
- Two-dimensional convolution lets SANs build relevance between elements across adjacent attention heads, enabling flexible extraction of local features from different subspaces rather than restricting extraction to a single head.Source: §3.2 · p. 3
4 Related Work
Prior work enhanced SANs with recurrence or contextual modeling, explored locality through CNNs, attention biases, positional masks, or relative position encodings, and studied multi-head feature diversity and aggregation. The paper positions its approach as parameter-free locality modeling by revising attention scope and as a way to model interactions among heads.
Self-Attention Networks
- Chen et al. (2018) and Hao et al. (2019) complemented SANs with recurrence, while Yang et al. (2019) modeled contextual information.Source: §4 · p. 3
- Yu et al. (2018) added CNN layers to fuse local information, while Yang et al. (2018) and Guo et al. (2019) used parametric localness biases.The paper describes its own approach as introducing no new parameters and contrasts its attention-scope revision with positional masks and relative position encoding.Source: §4 · p. 3
- Shaw et al. (2018) modeled locality with relative position embeddings, whereas this approach revises attention scope; the authors report better translation quality and training efficiency under a shared framework.Source: §4 · p. 3
Multi-Head Attention
- Prior multi-head work explored distinct linguistic features, head diversity, and aggregation; this paper instead models interactions among heads to exploit their distributed local properties.Source: §4 · p. 3
5 Experiments
Experiments report consistent translation improvements across three language pairs and better performance than prior locality approaches. They also examine window and head settings and phrase-translation accuracy.
5 Experiments
- Experiments use WMT14 En⇒De, WMT17 Zh⇒En, and WAT17 Ja⇒En, applying the method to the encoder’s lowest three of six layers in Base and Big settings.Training used about 4.5 million, 20.62 million, and 2 million sentence pairs, respectively.Source: §5 · pp. 3–4
5.1 Effects of Window/Area Size
- A window size of 11 is best for 1D-CSANs; adding attended heads improves 2D-CSANs, but translation quality drops as head count rises.Figure 2 varies window size for 1D-CSANs and attended head numbers for 2D-CSANs, with the 2D window dimension fixed at 11.Source: §5.1 · p. 4, Figure 2 · p. 4
5.2 Comparison to Related Work
- On En⇒De, CSANs outperform existing locality methods; dynamic local weights beat fixed CNN weights, and the parameter-free design only marginally affects training efficiency.The comparison re-implemented prior approaches under the same framework.Source: Table 1 · p. 4, §5.2 · p. 5
5.3 Universality of The Proposed Model
- The model consistently improves translation performance across En⇒De, Zh⇒En, and Ja⇒En, with 2D-Convolution in the base setting comparable to TRANSFORMER-BIG.Table 2 reports results on WMT14 En⇒De, WMT17 Zh⇒En, and WAT17 Ja⇒En.Source: §5.3 · p. 5, Table 2 · p. 5
5.4 Accuracy of Phrase Translation
- Both variants outperform the baseline on larger n-gram granularities, while dependencies among heads can complement localness modeling.Figure 3 reports performance improvement across n-grams, with “Gap of BLEU” defined as improvement over the baseline.Source: §5.4 · p. 5, Figure 3 · p. 5
6 Conclusion
The paper concludes that parameter-free convolutional self-attention enhances neighboring-feature extraction across heads. Its analyses support locality modeling and cross-head interaction, while identifying other sequence tasks for future validation.
6 Conclusion
- The parameter-free convolutional self-attention model enhances neighboring-feature extraction across multiple heads.Source: §6 · p. 5
- The analyses suggest that locality benefits SANs, cross-head feature interaction can further improve performance, and dynamic weights are superior to fixed counterparts to some extent.Source: §6 · p. 5
- The authors identify reading comprehension, language inference, semantic role labeling, sentiment analysis, and sentence classification as tasks for future validation.Source: §6 · p. 5