PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-Aware Self-Attention Networks

Baosong Yang, Jian Li, Derek Wong, Lidia S. Chao, Xing Wang, Zhaopeng Tu

arXiv:1902.05766v1cs.CL

TL;DR

Self-attention calculates relationships between representations without contextual information, though context can help model dependencies. This paper contextualizes query and key transformations using internal global and deep representations. It reports consistent translation improvements across language pairs and model variations, with only a marginal speed decrease.

  • Problem

    Self-attention calculates dependencies between representations without contextual information, despite evidence that context can enhance modeling dependencies among neural representations.

    Source: Introduction · p. 1, Motivation · p. 2

  • Method

    The approach contextualizes query and key transformations using internal representations that encode global, deep, or combined context, avoiding external resources.

    Source: Introduction · p. 1, Context-Aware Self-Attention Model · p. 2

  • Results

    The approach consistently improves translation performance across WMT language pairs and model variations over the TRANSFORMER baseline, with only a marginal speed decrease.

    Source: 2) deep context · p. 1, Main Results · p. 5, Conclusion · p. 7

  • Takeaways & Limitations

    The analyses find that deep and global approaches are complementary and that the model flexibly handles contextual information for different representations.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Self-attention model have shown its flexibility in parallel computation and the effectiveness on modeling both long- and short-term dependencies. However, it calculates the dependencies between representations without considering the contextual information, which have proven useful for modeling dependencies among neural representations in various natural language tasks. In this work, we focus on improving self-attention networks through capturing the richness of context. To maintain the simplicity and flexibility of the self-attention networks, we propose to contextualize the transformations of the query and key layers, which are used to calculates the relevance between elements. Specifically, we leverage the internal representations that embed both global and deep contexts, thus avoid relying on external resources. Experimental results on WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness and universality of the proposed methods. Furthermore, we conducted extensive analyses to quantity how the context vectors participate in the self-attention model.

Introduction

Self-attention networks capture long-range dependencies, but their attention calculations omit contextual information that can help model relationships among neural representations. The paper contextualizes query and key transformations using internal global and deep representations, improving translation performance across two WMT tasks.

Introduction

  • Self-attention networks capture long-range dependencies by attending directly to signals, but their dependency calculations do not account for contextual information.The input is treated as a bag of word tokens, with each token attending over that set.Source: Introduction · p. 1
  • Tu et al. (2017) and Zhang et al. (2017) enhanced attention components with internal contextual representations, while Wang et al. (2017) and Voita et al. (2018) used external representations summarizing previous source sentences.Source: Introduction · p. 1
  • The method uses internal representations to contextualize query and key transformations, avoiding external resources while aiming to preserve self-attention networks’ simplicity and flexibility.The strategies use global, deep, or combined deep-global context.Source: Introduction · p. 1
  • The proposed contextualization consistently improves translation performance over a strong TRANSFORMER baseline on WMT14 English⇒German and WMT17 Chinese⇒English, with only a marginal speed decrease.Analyses also find that function words require more context than content words.Source: 2) deep context · p. 1

Background

Self-attention networks use attention to capture dependencies between sequence elements, including long-range dependencies. The paper identifies a gap in this calculation: query-key similarity does not incorporate contextual information.

Background

  • Self-attention networks attend across a sequence and can capture long-range dependencies more directly than RNNs.They calculate attention weights between pairs of tokens in a sequence.Source: Background · pp. 1–2
  • The input states are transformed into queries, keys, and values, and the output is constructed by attending to the input states; this work uses dot-product attention, described as faster and more space-efficient than additive attention.The passage also identifies d as the scaling factor.Source: Background · p. 2

Motivation

  • Self-attention networks can directly capture dependencies between hidden states, but query-key similarity is controlled by trained parameter matrices without useful contextual information.The paper illustrates this gap with the word pair “talk” and “Sharon,” whose relevance is calculated without context.Source: Motivation · p. 2

Approach

The approach contextualizes query and key transformations with internal representations, preserving self-attention’s flexibility while incorporating sequence-wide and lower-layer information. It offers global, deep, and combined context strategies, with learned factors regulating context contributions.

Context-Aware Self-Attention Model

  • Contextualizing query and key transformations with internal context vectors strengthens self-attention without requiring a specific attention function.The approach propagates context by addition to avoid significantly increasing computation.Source: Approach · p. 2, Context-Aware Self-Attention Model · p. 2

Choices of Context Vectors

  • Global context mean-summarizes the input layer, while deep context concatenates lower layers to expose their syntactic and semantic information.Deep context is intended to fuse information captured at different layers.Source: Global Context · pp. 2–3
  • Without the learned λQ and λK factors, translation quality drops by 5.23 BLEU, and the authors attribute this to large query and key magnitudes hindering convergence.The factors also regulate the magnitude of the contextualized query and key representations.Source: Global Context · p. 2, Context-Aware Self-Attention Model · p. 2
  • For deep context, the method uses dense connections rather than normalized linear-combination weights, with distinct element-wise controls for greater modeling flexibility.The projection matrices combine the lower-layer states, while the method assigns a distinct scalar to each element.Source: Global Context · p. 3
  • Deep-global context combines global summaries across layers, intended to provide linguistic biases from lexical through syntactic to semantic levels.Multiple context vectors can also be concatenated; decoder-side global summaries use only visible forward representations at each step.Source: Deep-Global Context · p. 3

Experiments

Experiments on WMT14 En⇒De and WMT17 Zh⇒En evaluate context-vector strategies in TRANSFORMER. Context-aware models improve translation performance across tasks, while the results vary with where context-awareness is applied.

Setup

  • The study evaluated WMT14 English⇒German and WMT17 Chinese⇒English; Base-model variations were tested on En⇒De, while overall performance was evaluated with Big models on both translation tasks.The training sets contained about 4.56 million sentence pairs for En⇒De and about 20.62 million for Zh⇒En.Source: Setup · p. 4

Ablation Study on the Context Vector

  • Combining deep-global context with deep context improves encoder-side results beyond either strategy alone, while all tested encoder context strategies outperform the baseline.The authors interpret the gains as evidence that the two context types improve self-attention in different aspects.Source: Applied to Encoder · p. 4
  • On decoder-side self-attention, deep-global context plus deep context improves performance more than deep-global context alone.The authors note that global and lower-layer representations remain useful for decoder-side self-attention.Source: Applied to Decoder · p. 4
  • Applying context-awareness to both encoder and decoder slightly harms translation quality versus encoder-side models; the authors attribute the drop to contextual information in top-layer encoder representations.The conventional encoder-decoder attention uses those top-layer representations.Source: Applied to Both Encoder and Decoder · p. 4

Main Results

  • Context-aware models consistently improve performance across language pairs and model sizes; TRANSFORMER-BASE with context achieves performance comparable to TRANSFORMER-BIG with nearly twice the training speed and half the parameters.Table 2 compares WMT17 Zh⇒En and WMT14 En⇒De test results; its ⇑ marker denotes significance over the conventional self-attention counterpart at p < 0.01.Source: Table 2 · p. 5, Main Results · p. 5

Analysis

The analyses trace how deep and global context are used across layers, query/key roles, encoder/decoder sides, word classes, and sentence lengths.

Deep Context vs. Global Context

  • Deep-context weights stay near 0.5, giving current-layer and historical-layer information equivalent importance; the authors report that using historical context benefits translation quality.This supports their claim that conventional self-attention's weighted averaging does not fully capture contextual richness.Source: Stable Necessity of Deep Context · p. 5
  • Global-context weights fall in higher layers, indicating that lower layers require more global information as that information accumulates with depth.The authors consider global context more beneficial to lower layers for modeling semantic meanings.Source: The Lower Layer, The More Global… · pp. 5–6
  • Global-context weights are usually higher for keys than queries; Table 3 results show greater benefit from keys, while contextualizing queries can still improve performance.Table 3 reports BLEU scores on the En⇒De validation set for context integrated into queries and keys.Source: Keys Required More Global Information · p. 6, Table 3 · p. 6

Source Context vs. Target Context

  • Target-side context vectors receive lower weights than source-side vectors, and decoder-side context does not further improve translation quality as expected.The authors attribute this to useful context already embedded in top-layer encoder representations and conveyed through encoder-decoder attention.Source: Source Context vs. Target Context · p. 6, Figure 3 · p. 6

Linguistic Analysis

  • Function words require more deep-global contextual information than content words, according to the learned context weights grouped by part of speech.The paper attributes this pattern to function-word representations benefiting more from contextual information.Source: Analysis on Part-of-Speech · p. 6, Figure 4 · p. 6
  • Across ten sentence-length groups, the approaches outperform the baseline in almost all; global strategies lead above 20 words, while deep context leads on shorter sentences.Figure 5 plots the BLEU-score gap between the proposed model and the baseline by input sentence length.Source: Analysis on Long Sentences · pp. 6–7, Figure 5 · p. 7

Related Work

Prior work explores contextualized attention and self-attention limitations, while this paper focuses on adding internal context to self-attention networks.

Related Work

  • Earlier attention work enhanced query or memory with internal context, while other work used external context; deep modeling of syntax and semantics also benefits multilayer networks.The paper positions its contribution as improving self-attention networks with contextual information.Source: Related Work · p. 7
  • Unlike prior RNN-based approaches or methods requiring external resources, this work contextualizes self-attention networks using internal representations.The paper contrasts its approach with methods using previous source-sentence embeddings.Source: Related Work · p. 7
  • Prior studies report limitations in conventional self-attention and explore local attention, direct connections, relative positions, and external contextual representations.The cited work includes approaches using previous source-sentence embeddings.Source: Related Work · p. 7

Conclusion

The paper concludes that deep and global context strategies complement each other and improve self-attention across translation language pairs. Its analyses describe how context use varies across representations.

Conclusion

  • Deep and global approaches are complementary, and experiments across language pairs demonstrate the proposed approach's effectiveness and universality.The analyses also report that the model flexibly handles contextual information for different representations.Source: Conclusion · p. 7
  • The authors identify validation on other tasks, linguistic context techniques, and combinations with other methods as directions for future work.Suggested tasks include reading comprehension, language inference, and stance classification.Source: Conclusion · p. 7
Loading 1902.05766v1…