Mask Attention Networks: Rethinking and Strengthen Transformer

1 Introduction

• SAN's unrestricted mask supports long-range relationships, whereas FFN's identity mask confines each token to its own information.The paper describes these as complementary specialties in text representation.Source: §1 · pp. 1–2
• Static-mask self-attention can give unrelated tokens with a shared neighbor large attention scores, adding noise and diverting attention from relevant neighbors.Source: §1 · p. 2
• Mask Attention Networks (MANs) unify SAN and FFN as special cases with static masks, and the paper argues these masks are insufficient for localness modeling.Source: §1 · pp. 1–2
• The paper introduces DMAN to model localness and proposes the sequential structure DMAN→SAN→FFN to combine the layers' different strengths.Source: §1 · p. 2
• The proposed model outperforms the original Transformer on machine translation and abstract summarization.Source: §1 · p. 2

2 Model

• The section reviews Transformer, formulates MANs, analyzes localness limitations, defines DMAN, and describes how the three layers are combined.The proposed sequence is DMAN→SAN→FFN.Source: §2 · p. 2, §2.5 · p. 4

2.1 Transformer

• Transformer’s two sublayers are SAN, which applies attention across token pairs, and FFN, which transforms each token independently.The FFN uses two affine transformations with a pointwise nonlinearity.Source: §2.1 · p. 3

2.2 Mask Attention Networks

• SAN’s all-ones mask and FFN’s identity mask are two static-mask special cases of Mask Attention Networks.The MAN formulation applies a mask matrix to the attention computation.Source: §2.2 · p. 3

2.3 Deficiency of SAN and FFN in Localness Modeling

• SAN allows attention to all tokens, while FFN processes each token independently; the paper identifies both as limited in localness modeling.SAN can give substantial attention to non-neighbors, while FFN cannot access other tokens or their neighbors.Source: §2.3 · pp. 3–4

2.4 Dynamic Mask Attention Network

• DMAN replaces a fixed 0/1 neighborhood gate with a soft mask that incorporates query tokens, relative distance, attention head, and layer.Its mask parameters include a scalar for relative positions and trainable parameters associated with heads and layers.Source: §2.4 · p. 4

2.5 Collaboration of Mask Attention Networks

• The proposed DMAN→SAN→FFN sequence models localness, then globalness, and finally self-evolution.DMAN, SAN, and FFN are described respectively as specializing in local structure, global semantics, and self-processing.Source: §2.5 · p. 4

3.1.1 Machine Translation

• Machine-translation evaluation uses IWSLT14 German-to-English and WMT14 English-to-German, with small, base, and big model settings compared against Transformer.IWSLT14 contains about 153K/7K/7K training, validation, and test sentence pairs; WMT14 contains about 4.5M training pairs.Source: §3.1.1 · p. 5, §3.2.1 · p. 6

3.1.2 Abstract Summarization

• Abstractive summarization is evaluated on CNN/Daily Mail and Gigaword using a six-layer encoder and decoder, without copy, coverage, or reinforcement learning.The evaluation metric is ROUGE F1.Source: §3.1.2 · p. 5, §3.2.2 · p. 6

3.2.1 Machine Translation

• 1.6, 1.8, and 2.0 BLEU points: the model beats Transformer small on IWSLT14 De-En and Transformer base and big on WMT14 En-De.The WMT14 base model also exceeds the reported Transformer big result with much less parameters.Source: Table 1 · p. 5, §3.2.1 · p. 6
• The model improves BLEU by 1.7 points over Convolutional Transformer and 1.2 points over Local Transformer.The authors describe these comparisons as validating their approach to localness modeling.Source: §3.2.1 · p. 6

3.2.2 Abstractive Summarization

• CNN/Daily Mail gains are 1.4 ROUGE-1, 2.2 ROUGE-2, and 1.2 ROUGE-L; Gigaword gains are 0.7, 0.5, and 0.7, respectively.Both datasets are compared against Transformer.Source: §3.2.2 · p. 6

4.1 Investigate Stacking Methods for Different Sublayers

• C#5 achieves the best performance among the tested collaboration strategies, and its DMAN→SAN→FFN order outperforms the inverse DMAN→SAN comparison.The authors report that the least BLEU improvement for configurations with DMAN over those without it is 0.2.Source: §4.1 · pp. 6–7, Table 3 · p. 7
• DMAN and SAN configurations outperform those without either module, consistent with their reported localness and globalness strengths complementing one another.C#4 and C#5 are both better than C#3 and C#2.Source: §4.1 · p. 6

4.2 Static Mask and Dynamic Mask

• Both fixed-boundary SMAN strategies outperform Transformer but fall behind DMAN, which can model different query-token neighborhoods more flexibly.SMAN1 sets boundary b to L/2; SMAN2 sets b to 4.Source: §4.2 · p. 7, Table 4 · p. 7

4.3 Analysis of DMAN in Localness Modeling

• Within window size 2, DMAN’s summed attention scores are 50% higher than SAN’s across layers 1, 3, and 6, with a fivefold gap in layer 1.The analysis averages attention scores across heads and across the IWSLT14 De-En test set.Source: §4.3 · pp. 7–8, Table 5 · p. 8

5 Related Work

• Prior Transformer work addresses recurrence and context, modifies attention weights with relative-position information or learned spans, and combines convolution with self-attention for local and long-range structures.Examples include relative-position methods by Shaw et al. (2018) and Dai et al. (2019), and multi-scale representation learning by Zhao et al. (2019).Source: §5 · p. 8
• Other work merges or restructures attention and FFN, including persistent memory vectors in Sukhbaatar et al. (2019b) and Macaron layers in Lu et al. (2019).The paper situates DMAN as a localness-focused supplement to SAN and FFN within Mask Attention Networks.Source: §5 · p. 8

• The proposed model achieves better performance than the original Transformer in neural machine translation and abstract summarization.Source: §6 · p. 8
• SAN and FFN are reformulated as special cases of MANs with static masks, and the paper analyzes their deficiencies in localness modeling.Source: §6 · p. 8
• DMAN is derived from MANs for better local-structure modeling and is arranged sequentially with SAN and FFN as DMAN→SAN→FFN.Source: §6 · p. 8
• Future work considers incorporating structure information or external knowledge, such as dependency trees, through MAN mask matrices.Source: §6 · p. 8