Source-linked AI summary
Attention Strategies for Multi-Source Sequence-to-Sequence Learning
Jindřich Libovický, Jindřich Helcl
TL;DR
Multi-source sequence-to-sequence learning needs attention mechanisms that can represent differing roles among input sequences. The paper proposes flat and hierarchical attention combination strategies and evaluates them on multimodal translation and automatic post-editing. Both strategies achieve competitive scores, while hierarchical combination learns faster than the other strategies.
Problem
The paper addresses limited explicit modeling of the differing importance of multiple input sequences in multi-source sequence-to-sequence learning.
Method
The paper combines attention over multiple encoders either jointly across all hidden states using flat attention or factorized over individual encoders using hierarchical attention.
Results
The proposed flat and hierarchical strategies achieve competitive scores on multimodal translation and automatic post-editing, with hierarchical combination learning faster than the other strategies.
Takeaways & Limitations
Flat and hierarchical attention combination can be applied to multimodal translation and automatic post-editing while making attention allocation across encoders explicitly interpretable.
Abstract
from arXiv · showhide
Modeling attention in neural multi-source sequence-to-sequence learning remains a relatively unexplored area, despite its usefulness in tasks that incorporate multiple source languages or modalities. We propose two novel approaches to combine the outputs of attention mechanisms over each source sequence, flat and hierarchical. We compare the proposed methods with existing techniques and present results of systematic evaluation of those methods on the WMT16 Multimodal Translation and Automatic Post-editing tasks. We show that the proposed methods achieve competitive results on both tasks.
1 Introduction
The paper studies sequence-to-sequence learning with multiple input sequences and a single recurrent decoder. It proposes interpretable flat and hierarchical strategies for combining attention across sources.
- Multi-source sequence-to-sequence learning uses multiple input sequences, possibly from different modalities, with one recurrent decoder attending to individual encoder states.
- Existing approaches do not explicitly model different source importance for the decoder.
- In multimodal translation, captions may provide primary information while images support output disambiguation; in post-editing, source text may matter mainly when translations contain errors.
- The paper proposes two interpretable attention-combination strategies: flat and hierarchical attention.
- The study evaluates the proposed strategies on multimodal translation and automatic post-editing tasks.
2 Attentive S2S Learning
Attention lets an RNN decoder access and weight encoder information at each decoding step. The paper describes standard attention and a sentinel extension adapted for GRU decoders.
- Attention estimates a probability distribution over encoder hidden states at each decoding step and uses it to compute a weighted-average context vector.
- Standard attention computes energies, attention distributions, and context vectors from decoder and encoder states.
- Projection matrices map decoder and encoder states into a common space, while a weight vector scores dimensions in that space.
- The paper adapts the sentinel-gate extension from LSTM attention to GRU units for its experiments.
- The sentinel contributes an additional energy and vector without depending on any encoder hidden state.
- The decoder can use attention to choose between encoder information and its own state when the input is not relevant.
3 Attention Combination
The section addresses how a multi-encoder decoder combines attention information when source sequences may differ in modality and dimensionality. It presents flat and hierarchical combinations that explicitly model encoder-level attention.
- Multi-encoder sequence-to-sequence models require a decoder to combine attention information collected from individual encoders.Existing concatenation approaches attend to each encoder independently and leave their combination to later network layers.
- Both strategies explicitly compute how much attention is paid to each encoder at every decoding step.This makes the role of individual source sequences interpretable rather than resolving their combination only implicitly.
- Flat Attention Combination: Flat attention projects all encoder hidden states into a shared space and computes one joint attention distribution over them.Encoder-specific projections handle differing vector spaces and dimensionalities, while attention parameters are partly shared across encoders.
- Flat Attention Combination: Flat attention uses shared-space projections for hidden states, with encoder-specific projection matrices and experimentally varied parameter sharing.The projected context vectors are combined using attention weights; the experiments also vary whether the projection spaces are shared and whether the sentinel is included.
- Hierarchical Attention Combination: Hierarchical attention first computes each encoder’s context vector independently, then attends over the projected context vectors.A second attention mechanism combines encoder-level contexts, optionally including the sentinel.
4 Experiments
The experiments evaluate flat and hierarchical attention combination alongside established setups on multimodal translation and automatic post-editing. Results show fast or competitive performance, with task-specific differences in model configuration and gains.
- The proposed attention strategies are evaluated on multimodal translation and automatic post-editing tasks.
- 4.1 Multimodal Translation: Hierarchical attention without a sentinel achieves the best multimodal translation results and fastest convergence, while flat attention eventually performs similarly.Sharing projection matrices does not improve over the concatenation baseline and substantially slows training.
- 4.1 Multimodal Translation: Multimodal models do not surpass the textual baseline of BLEU 33.0.
- 4.1 Multimodal Translation: Conditional GRU units improve performance by about 1.5 BLEU points on average, except concatenation, which drops by almost 5 BLEU points.The authors hypothesize that concatenation requires learning implicit attention combination at multiple network locations.
- 4.2 Automatic MT Post-editing: The automatic post-editing setup uses edit-operation targets to transform machine translations into references and avoids conditional GRUs because of overfitting on the small dataset.The dataset contains 12,000 training, 2,000 validation, and 1,000 test triplets from the IT domain.
- 4.2 Automatic MT Post-editing: Attention-combination models slightly but significantly improve over the unchanged MT baseline of HTER 24.8, with no significant differences among strategies.
5 Related Work
Related work covers multi-source and multimodal sequence-to-sequence architectures, including concatenated inputs, separate decoders, and shared representations. Reported comparisons show that architectural choices and additional data influence performance across tasks.
- Prior APE work concatenates source and target inputs into one sequence, yielding an attention mechanism similar to flat combination but limited to sequential data.
- The strongest WMT16 APE system averages outputs from two separate sequence-to-sequence models, with its largest improvement attributed to additional training-data generation.
- A related multimodal system sums context vectors from textual and visual encoders after assuming their hidden states share a vector space.That system reports BLEU 27.82 versus BLEU 32.50 for its text-only setup.
- Hierarchical attention visualizations show encoder and sentinel weights by column, with the image encoder activated for content words despite low overall importance.
- Later Multi30k work reports BLEU 37.1 for a multimodal model against a stronger monomodal baseline of BLEU 33.7, while attending only to text encoder states.
6 Conclusions
The paper introduces flat and hierarchical strategies for combining attention across multiple encoders. Both compute a joint distribution over encoder hidden states and achieve competitive results on multimodal translation and automatic post-editing.
- The paper introduces two attention-combination strategies for multi-source sequence-to-sequence learning.
- Both methods compute a joint distribution over the hidden states of all encoders.
- Flat and hierarchical attention combination can be applied to multimodal translation and automatic post-editing while maintaining competitive scores against previously used techniques.
- The proposed combinations work with conditional GRU decoder units, and hierarchical combination learns faster than the other strategies.