Source-linked AI summary
Multi-Head Attention with Disagreement Regularization
Jian Li, Zhaopeng Tu, Baosong Yang, Michael R. Lyu, Tong Zhang
TL;DR
Multi-head attention lacks a mechanism ensuring that its heads capture distinct features. The paper introduces three disagreement regularizations targeting subspaces, attended positions, and output representations, and finds consistent translation improvements across language pairs, with Transformer-BASE approaching Transformer-BIG performance at nearly twice the training speed.
Problem
Multi-head attention can attend to different subspaces and positions, but does not guarantee that different heads capture distinct features.
Method
The paper adds three disagreement regularizations that diversify head subspaces, attended positions, or output representations, individually or in combination.
Results
The approach consistently improves translation performance across English⇒German and Chinese⇒English tasks; Transformer-BASE achieves comparable performance with Transformer-BIG while training nearly twice faster.
Takeaways & Limitations
Disagreement regularization encourages attention heads to learn distinct features and is effective across the evaluated translation language pairs.
Takeaways & Limitations
Combining different disagreement regularizations does not further improve translation performance and may make training more difficult because their guidance can overlap.
Abstract
from arXiv · showhide
Multi-head attention is appealing for the ability to jointly attend to information from different representation subspaces at different positions. In this work, we introduce a disagreement regularization to explicitly encourage the diversity among multiple attention heads. Specifically, we propose three types of disagreement regularization, which respectively encourage the subspace, the attended positions, and the output representation associated with each attention head to be different from other heads. Experimental results on widely-used WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness and universality of the proposed approach.
1 Introduction
The paper addresses the lack of a mechanism ensuring that multi-head attention captures distinct features by introducing disagreement regularization. It proposes three regularization types and reports consistent translation improvements across language pairs.
- Motivation: Multi-head attention can attend jointly to different representation subspaces at different positions, but its heads are not guaranteed to capture distinct features.The paper introduces disagreement regularization to explicitly encourage diversity among heads.
- Approach: The disagreement regularization term serves as an auxiliary objective guiding the related attention component during training.
- Approach: Three regularization types target input subspaces, attended positions, and output representations, and can be used individually or in combination.The subspace and output terms maximize cosine distances, while the position term disperses attended positions through element-wise multiplication of attention matrices.
- Evaluation: The approach is evaluated on Transformer models for English⇒German and Chinese⇒English translation tasks.
- Results: Transformer-BASE with disagreement regularization achieves comparable performance with Transformer-BIG while training nearly twice faster.
2 Background: Multi-Head Attention
Multi-head attention models relevance between representations by transforming queries, keys, and values into multiple learned subspaces. Parallel attention heads produce output states that are concatenated into the final state.
- Attention mechanism: Attention models the strength of relevance between representation pairs, allowing one representation to build a direct relation with another.
- Multi-head attention: Multi-head attention jointly attends to different representation subspaces at different positions, allowing heads to capture different context.In the illustrated two-head example, the heads attend to different positions.
- Multi-head attention: The model transforms queries, keys, and values into H subspaces using different learnable linear projections.The projected query, key, and value representations are associated with individual attention heads.
- Multi-head attention: H attention functions operate in parallel to produce output states, and the states are concatenated into the final state.Each head has an attention distribution used to produce its output.
3 Approach
The approach adds an auxiliary disagreement objective to multi-head attention and proposes three regularizers targeting subspaces, attended positions, and head outputs. These terms can be applied individually or combined to encourage diversity among attention heads.
- Framework: The method augments the standard training objective with a disagreement term that guides attention components to capture different features.The regularization acts like L1 and L2 terms without introducing new parameters.
- Disagreement Regularization: Three regularizers target projected subspaces, attended positions, and output representations, either individually or in combination.The subspace and output terms maximize cosine distances, while the position term disperses attended positions through element-wise matrix multiplication.
- Disagreement on Subspaces: Subspace disagreement enlarges the average negative cosine similarity among projected value vectors from different heads.The method uses cosine distance because it does not depend on the vectors’ absolute values.
- Disagreement on Attended Positions: Position disagreement measures similarity between heads’ attention matrices using summed element-wise multiplication to disperse attended positions.This formulation adapts an agreement-regularization idea by encouraging alignment disagreement rather than alignment agreement.
- Disagreement on Attended Positions: Squared element-wise subtraction underperformed multiplication in preliminary experiments, motivating the selected position-disagreement formulation.The authors report this comparison as consistent with prior results.
- Disagreement on Outputs: Output disagreement directly regularizes each head’s output by maximizing differences measured with negative cosine similarity.It applies the same distance principle used for subspace disagreement to the output representations.
4 Related Work
The related work positions disagreement regularization alongside methods that control or expand the features represented by multiple attention heads. It is presented as complementary to these alternatives.
- Multi-Head Attention: Prior work weighted attention heads instead of simply concatenating them because standard multi-head attention lacks effective control over head influence.This motivates approaches that regulate how multiple heads contribute to the final representation.
- Multi-Head Attention: Other work extended scalar relevance scores to multidimensional attention weights to model multiple features in attention networks.The paper describes its disagreement-based approach as complementary to this feature-modeling direction.
- Evaluation Context: The evaluation compares regularization effects using encoder self-attention, with training speed measured in steps per second.Table 1 is framed as an effect comparison rather than a related-work result.
5 Experiments
Experiments on WMT17 Chinese⇒English and WMT14 English⇒German evaluate disagreement regularization across terms, attention networks, and model settings. The approach consistently improves translation performance, while analyses show that output and subspace diversity are more useful than positional diversity in several settings.
- Experimental setup: Experiments use WMT17 Chinese⇒English and WMT14 English⇒German translation tasks with TRANSFORMER Base and Big models.The Chinese⇒English corpus contains 20M sentence pairs and the English⇒German corpus contains 4M sentence pairs.
- Effect of regularization terms: +0.65 BLEU improves the Output regularization model over the vanilla TRANSFORMER baseline on the Zh⇒En task.All individual disagreement regularization terms outperform the baseline, while the Position term is less effective than the other two.
- Effect of regularization terms: Combining different regularization terms does not further improve translation performance, possibly because their guidance overlaps and complicates training.The failure to improve is reported for the combinations evaluated in Rows 5–7.
- Effect on different attention networks: +0.72 BLEU improves performance when disagreement regularization is applied to all three attention networks.The training speed decreases by 12%, which the authors describe as acceptable given the performance improvement.
- Main results: Nearly twice faster training makes TRANSFORMER-BASE with disagreement regularization comparable to TRANSFORMER-BIG across the evaluated language pairs.The paper reports consistent translation improvements for both Base and Big TRANSFORMER models.
- Quantitative analysis: Output disagreement approaches 1.0 under Output regularization, indicating that output vectors become almost perpendicular by the cosine-distance measure.The disagreement score for each individual component increases with its corresponding regularization term.
- Quantitative analysis: The standard model’s attended-position disagreement is only 0.007, and remains very low from encoder layers 2 through 6.This pattern suggests that different attended positions may not be the essential strength of multi-head attention for translation.
6 Conclusion
The paper proposes disagreement regularization to encourage distinct features across multi-head attention heads. Experiments across language pairs validate the effectiveness of the proposed approaches, with potential extensions to other tasks and techniques.
- Conclusion: The proposed disagreement regularizations encourage different attention heads to learn distinct features.The conclusion describes these regularizations as augmentations to the multi-head attention model.
- Conclusion: Experimental results across language pairs validate the effectiveness of the proposed approaches.The authors identify reading comprehension, language inference, and combination with other techniques as potential extensions.