PaperlayerView exact revision on arXiv

Source-linked AI summary

Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection

Guangxiang Zhao, Junyang Lin, Zhiyuan Zhang, Xuancheng Ren, Qi Su, Xu Sun

arXiv:1912.11637v1cs.CLcs.LG

TL;DR

Self-attention can model long-term dependencies but may extract irrelevant context, weakening attention focus. The paper introduces Explicit Sparse Transformer, which uses explicit top-k selection to concentrate attention on relevant states, and reports better performance than vanilla Transformer across three tasks with comparable or better results and faster training and testing than prior sparse-attention methods.

  • Problem

    Self-attention can model long-term dependencies but may assign attention to irrelevant context, reducing focus on relevant information.

  • Method

    Explicit Sparse Transformer explicitly selects the top-k most contributive states before forming a sparsified attention distribution.

  • Results

    Explicit Sparse Transformer outperforms vanilla Transformer across three NLP tasks and achieves comparable results to previous sparse-attention methods with faster training and testing.

  • Takeaways & Limitations

    Explicit selection concentrates attention on relevant components and is reported to improve alignment effects while reducing the cost of sparse attention relative to prior methods.

  • Takeaways & Limitations

    The comparison omits sparse-attention methods based on local attention constraints because they do not show performance on neural machine translation.

Abstract

from arXiv · show

Self-attention based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks. Self-attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context. To tackle the problem, we propose a novel model called \textbf{Explicit Sparse Transformer}. Explicit Sparse Transformer is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments. Extensive experimental results on a series of natural language processing and computer vision tasks, including neural machine translation, image captioning, and language modeling, all demonstrate the advantages of Explicit Sparse Transformer in model performance. We also show that our proposed sparse attention method achieves comparable or better results than the previous sparse attention method, but significantly reduces training and testing time. For example, the inference speed is twice that of sparsemax in Transformer model. Code will be available at \url{https://github.com/lancopku/Explicit-Sparse-Transformer}

1 INTRODUCTION

The paper proposes Explicit Sparse Transformer to address vanilla Transformer's diffuse attention by explicitly selecting the most relevant context states. Experiments across three NLP tasks report better performance than vanilla Transformer, while comparisons with prior sparse-attention methods report faster training and testing with comparable results.

  • Motivation: Vanilla Transformer assigns attention credits broadly across context, including irrelevant words, which reduces focus on relevant information.For “tim,” the relevant words are described as “heart” and nearby words, while “him” receives irrelevant credit.
  • Method: Explicit Sparse Transformer selects the k most contributive states so attention becomes more concentrated than in vanilla Transformer.The method uses explicit top-k selection to remove distraction from irrelevant context positions.
  • Contribution: The proposed model is designed to enhance Transformer attention concentration through explicit selection.
  • Experiments: Experiments cover Neural Machine Translation, Image Captioning, and Language Modeling, with better performance than vanilla Transformer across all three tasks.
  • Experiments: Compared with previous sparse-attention methods, the proposed method reports comparable results with substantially faster training and testing.

2 EXPLICIT SPARSE TRANSFORMER

Explicit Sparse Transformer replaces dense attention with explicit top-k selection, retaining the most contributive elements before normalization. This concentrates attention while preserving the Transformer framework and extending to self- and context-attention.

  • 2 EXPLICIT SPARSE TRANSFORMER: Explicit Sparse Transformer sparsifies attention by selecting only the most contributive elements and removing irrelevant information.The method is designed to improve concentration relative to conventional attention.
  • 2 EXPLICIT SPARSE TRANSFORMER: The model computes attention scores from query, key, and value representations derived through learnable linear transformations of the source context.For self-attention, Q, K, and V are formed as Q = WQx, K = WKx, and V = WVx.
  • 2 EXPLICIT SPARSE TRANSFORMER: Top-k masking selects the k largest scores in each row, recording a row-specific threshold and assigning lower scores negative infinity.The threshold for row i is the k-th largest score, ti.
  • 2 EXPLICIT SPARSE TRANSFORMER: Explicit selection differs from dropout because it deterministically preserves high-scoring components and can simplify computation when k is small.The passage gives k = 8 as an example of a usually small selection size.
  • 2 EXPLICIT SPARSE TRANSFORMER: After masking, softmax normalization makes the retained scores probabilities, while scores below the top-k threshold receive probabilities approximating zero.The resulting normalized distribution is used to compute the self-attention output representation.
  • 2 EXPLICIT SPARSE TRANSFORMER: The output representation is the expectation of the value under the sparsified attention distribution, and the mechanism also extends to context attention.For context attention, the query is derived from decoding states rather than the source context.

3 RESULTS

Explicit Sparse Transformer improves performance across neural machine translation, image captioning, and language modeling, while its sparse attention remains faster than prior methods.

  • Neural Machine Translation: Explicit Sparse Transformer outperforms vanilla Transformer on En-De, En-Vi, and De-En translation tasks.It reaches 29.4 BLEU on En-De, 31.1 on En-Vi, and 35.6 on De-En.
  • Image Captioning: Explicit Sparse Transformer improves image captioning results on the COCO Karpathy test split across BLEU-4, METEOR, and CIDEr.The reported gains over the implemented Transformer are +0.4 BLEU-4, +0.3 METEOR, and +0.7 CIDEr.
  • Language Modeling: Explicit Sparse Transformer-XL outperforms Transformer-XL on the enwiki8 language-modeling test set.Table 3 compares the sparsified Transformer-XL with state-of-the-art baselines.

4 DISCUSSION

The discussion evaluates Explicit Sparse Transformer against prior sparse-attention methods, studies the choice and training role of k, and examines whether sparsification improves alignment. It reports competitive performance with substantially faster inference and more focused attention.

  • 4.1 COMPARISON WITH OTHER SPARSE ATTENTION METHODS: Comparable results with prior sparse-attention methods accompany inference speeds 2x faster than sparsemax and 10x faster than Entmax-alpha.The comparison uses the IWSLT 2014 De-En dataset, batch size 128 for inference, and FP-16 training.
  • 4.1 COMPARISON WITH OTHER SPARSE ATTENTION METHODS: Prior local-attention methods are not compared in Table 4 because they do not report neural machine translation performance.
  • 4.2 HOW TO SELECT A PROPER K?: Figure 3 analyzes k on IWSLT En-Vi and De-En, with “inf” denoting attention to all positions as in the original Transformer.
  • 4.2 HOW TO SELECT A PROPER K?: For k ∈ {4, 8, 16, 32}, setting k to 8 achieves consistent improvements over the Transformer baseline on En-Vi.Across tested k values, performance generally rises and then falls as k increases; experiments use three initializations and mean validation BLEU.
  • 4.3 DO THE PROPOSED SPARSE ATTENTION METHOD HELPS TRAINING?: Adding sparsification only during training improves performance by 0.3 BLEU scores on the IWSLT En-Vi validation set.The authors interpret this improvement as evidence that vanilla Transformer may be overparameterized and that sparsification encourages model simplification.
  • 4.4 DO THE EXPLICIT SPARSE TRANSFORMER ATTEND BETTER?: Attention visualizations show that Explicit Sparse Transformer focuses on several relevant positions, whereas vanilla Transformer can concentrate on the last source token or disperse attention.The reported examples associate sparse attention with stronger alignment for generated phrases and improved extraction of relevant source-side information.

5 RELATED WORK

Related work includes local, hard, block, and sparsemax-style attention methods. Explicit Sparse Transformer differs by selecting contributive states without blocking sentences, while retaining long-distance dependencies.

  • RELATED ATTENTION METHODS: Earlier attention research includes local attention, hard attention, and related extensions intended to enhance attention mechanisms.
  • SPARSE TRANSFORMER METHODS: Child et al. use local and block attention to sparsify Transformers, whereas this approach avoids sentence blocking and still captures long-distance dependencies.
  • SPARSE ATTENTION METHODS: Sparsemax variants improve machine translation tasks, motivating empirical comparison with the proposed sparse attention method.

6 CONCLUSION

The conclusion presents Explicit Sparse Transformer as a model that concentrates attention on contributive components and outperforms vanilla Transformer across three NLP tasks. Qualitative analyses associate this improvement with alleviated top-layer attention problems and better alignment.

  • CONCLUSION: Explicit Sparse Transformer makes vanilla Transformer attention more concentrated on the most contributive components.
  • CONCLUSION: The model outperforms vanilla Transformer across three different NLP tasks.
  • CONCLUSION: Qualitative analyses identify a top-layer attention problem in vanilla Transformer that Explicit Sparse Transformer alleviates with improved alignment effects.

A.1.1 ATTENTION MECHANISM

The attention mechanism maps a query and key-value pair to an output through attention scores and normalization. In NMT, encoder states provide keys and values, while decoder states provide queries.

  • ATTENTION MECHANISM: Attention maps a query and a key-value pair to an output using an attention score function and softmax normalization.
  • NMT FORMULATION: In NMT, encoder output states form K and V, while decoder output states form Q.The sequence lengths of Q and K/V may differ, and d denotes the state dimension.
  • NMT FORMULATION: The attention formulation uses f for computing attention scores.

A.1.2 TRANSFORMER

Transformer uses self-attention to model context, but soft attention can distribute weight across irrelevant words. Explicit Sparse Transformer addresses this by selecting the most contributive states to concentrate attention.

  • Self-attention derives query, key, and value representations as linear transformations of the input.
  • Multi-head attention splits computation across multiple heads so different input parts can be processed individually.The common practice described uses 8 heads for the base model and 16 for the large model.
  • The outputs of all attention heads are concatenated and passed through a linear transformation for the final multi-head attention output.
  • Soft attention may assign weights to many words less relevant to the query, reducing concentration during information extraction.
  • Explicit Sparse Transformer improves concentration by selecting only the k most contributive states for attention.

A.2 EXPERIMENTAL DETAILS

The experiments evaluate Explicit Sparse Transformer across translation, image captioning, and language modeling, while the method masks attention scores below each row’s k-th largest value. The implementation follows Transformer defaults and includes a resource-based scope limitation.

  • Evaluation: Experiments cover neural machine translation, image captioning, and language modeling using task-specific preprocessing, training, and evaluation settings.Translation uses BLEU, image captioning reports BLEU-4, METEOR, and CIDEr, and language modeling uses BPC.
  • Top-k selection: The masking function retains scores at least as large as each row’s k-th largest value and replaces lower scores with −∞.This implements explicit top-k selection before normalization.
  • Back-propagation: The mask is treated as constant during back-propagation, while the subsequent normalization produces normalized scores.
  • Back-propagation: The softmax operation remains differentiable, allowing gradients to be calculated for the top-k selection process.
  • Implementation: The proposed method is presented as easy to implement and plug into Transformer, with Figure 5 showing single-head self-attention code.
  • Implementation: The study does not implement the large Explicit Sparse Transformer-XL because of limited TPU resources.
Loading 1912.11637v1…