Source-linked AI summary
Monotonic Chunkwise Attention
Chung-Cheng Chiu, Colin Raffel
TL;DR
Soft attention is computationally expensive and unsuitable for real-time transduction, while hard monotonic attention sacrifices expressivity. MoChA applies soft attention over adaptively selected chunks, retaining online linear-time decoding; it achieves state-of-the-art online speech recognition, matches offline soft attention, and improves document summarization over monotonic attention.
Problem
Soft attention has quadratic time and space cost and cannot produce outputs online, while hard monotonic attention limits alignment expressivity and underperforms soft attention.
Method
MoChA combines hard monotonic chunk-endpoint selection with soft attention over preceding memory chunks and supports standard backpropagation training.
Results
MoChA achieves state-of-the-art online speech recognition, matches offline soft attention, and provides a 20% relative improvement over monotonic attention on document summarization.
Takeaways & Limitations
MoChA provides soft alignment flexibility together with online and linear-time decoding across the evaluated sequence transduction tasks.
Takeaways & Limitations
Soft attention remains unsuitable for online transduction because computing each context vector requires all encoder hidden states and has O(TU) time and space cost.
Abstract
from arXiv · showhide
Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction. To address these issues, we propose Monotonic Chunkwise Attention (MoChA), which adaptively splits the input sequence into small chunks over which soft attention is computed. We show that models utilizing MoChA can be trained efficiently with standard backpropagation while allowing online and linear-time decoding at test time. When applied to online speech recognition, we obtain state-of-the-art results and match the performance of a model using an offline soft attention mechanism. In document summarization experiments where we do not expect monotonic alignments, we show significantly improved performance compared to a baseline monotonic attention-based model.
1 INTRODUCTION
Soft attention is flexible but requires quadratic decoding and cannot operate online. MoChA addresses these constraints by combining monotonic chunk selection with soft attention, while improving over hard monotonic attention.
- Soft attention assigns probabilities over every memory entry at each output timestep, enabling arbitrary input-output alignments.The context vector is a probability-weighted average of the memory.
- O(TU) decoding time and space make soft attention unsuitable for very long sequences and real-time transduction.It must process the entire input before producing output.
- Hard monotonic attention enables online, linear-time decoding by scanning memory left-to-right and stopping at a selected entry.Its context vector is hard-assigned to the attended memory entry.
- Hard monotonicity limits expressivity relative to soft attention, and its models previously lagged behind standard soft attention.
- MoChA selects chunk endpoints with hard monotonic attention, then applies soft attention within preceding chunks.This retains online and linear-time benefits while allowing soft alignments.
2 DEFINING MOCHA
MoChA combines hard monotonic scanning with soft attention over adaptively selected memory chunks. It preserves online, linear-time decoding while enabling more flexible alignments and efficient standard-backpropagation training.
- 2.2 STANDARD SOFT ATTENTION: Soft attention computes each context vector from all encoder hidden states, requiring O(TU) decoding time and space and preventing online transduction.The decoder must have processed the entire input sequence before producing output.
- 2.3 MONOTONIC ATTENTION: Hard monotonic attention scans memory from the previous attended index and stops when a Bernoulli decision selects an entry.The selected index determines the current context vector and permits a single pass over the memory.
- 2.3 MONOTONIC ATTENTION: The expected attention distribution can be computed efficiently using cumulative sums and products in parallel, allowing standard backpropagation training.The recurrence is transformed into a first-order linear difference equation before applying its parallelizable solution.
- 2.3 MONOTONIC ATTENTION: Hard monotonic attention enables online linear-time decoding but restricts each output to one memory entry and enforces strictly monotonic alignment.Its sampling and hard assignment also prevent direct backpropagation, motivating expected-context training.
- 2.4 MONOTONIC CHUNKWISE ATTENTION: MoChA adds soft attention over small memory chunks preceding the location selected by hard monotonic attention.Chunk locations are chosen adaptively, and the chunk attention uses a separate ChunkEnergy function.
- 2.4 MONOTONIC CHUNKWISE ATTENTION: MoChA’s chunkwise soft attention adds only a constant factor w to runtime, retains online decoding, and recovers hard monotonic attention when w = 1.In the speech recognition experiments, adding the second attention energy function increased total parameters by about 1%.
3 EXPERIMENTS
Experiments evaluate MoChA on online speech recognition and document summarization, testing both its natural monotonic setting and its ability to handle local reorderings.
- Experimental setup: MoChA was evaluated on online speech recognition and document summarization, with the baseline model architecture and training approach held constant.The experiments changed only the attention mechanism to isolate its performance effect.
- Online speech recognition: 20% relative improvement over prior state-of-the-art was reported for online speech recognition on the WSJ corpus.Across eight trials, MoChA had a lower best WER but a slightly higher mean WER than soft attention; the difference in means was not statistically significant.
- Online speech recognition: MoChA matched standard offline soft attention performance in online speech recognition while using a length-2 soft-attention chunk.Its alignments were roughly similar to those of the other attention mechanisms.
- Online speech recognition: A 0.3% WER decrease from enlarging the monotonic baseline’s energy function was not significant and was dwarfed by MoChA’s gains.The parameter increase from MoChA’s independent ChunkEnergy function was about 1% in the speech-recognition model.
- Document summarization: Nearly 8 ROUGE-1 points were lost with hard monotonic attention on document summarization, while MoChA approximately halved the gap to soft attention using w = 8.The task requires strong reordering, unlike the roughly monotonic alignments expected in speech recognition.
4 RELATED WORK
Related approaches support online or chunked transduction, but MoChA differs through adaptively positioned chunks bounded by an independent hard monotonic attention mechanism.
- Chunk-based transduction: Neural Transducer performs attentive transduction over equally sized, non-overlapping pre-segmented chunks and marginalizes possible end-of-sequence locations.MoChA also attends over chunks, but its chunk locations are adaptive.
- Chunk-based transduction: A fixed window around the previous timestep’s maximal attention index also yields chunkwise soft attention, but MoChA sets boundaries with independent hard monotonic attention.This distinction led the prior approach to use a very large chunk size.
- Non-attentive online models: CTC, RNN transducer, segment-to-segment transduction, and segmental RNN models support online settings without attention because their decoders do not directly condition on the input sequence.They are distinguished from attentive sequence-to-sequence models by this decoder property.
- Hard monotonic alignments: Other hard-monotonic approaches use reinforcement learning, separately computed target alignments, or strictly diagonal alignments.The authors suggest these approaches may also benefit from chunkwise attention.
5 CONCLUSION
The paper concludes that MoChA combines adaptive chunkwise soft attention with online, linear-time decoding and improves performance across speech recognition and summarization settings.
- Contribution: MoChA performs soft attention over adaptively located input chunks.The mechanism retains online and linear-time decoding while allowing local input-output reorderings.
- Results: MoChA achieved state-of-the-art performance on online speech recognition and substantially outperformed hard monotonic attention on document summarization.The summarization result tests the model in a task without monotonic input-output alignment.
- Future work: Future work includes applying MoChA to approximately monotonic tasks and allowing chunk size w to vary adaptively.The authors also provide an example online implementation.
A EXPERIMENT DETAILS
The appendix provides details of the experiments and notes that all experiments were implemented using TensorFlow.
- Experiment details: The appendix describes specifics about the experiments carried out in section 3.It states that all experiments used TensorFlow.
A.1 ONLINE SPEECH RECOGNITION
The speech-recognition implementation uses convolutional feature processing, a unidirectional LSTM decoder, and Adam training with a scheduled learning rate.
- Speech features are represented as 80-coefficient mel-scaled spectrograms with delta and delta-delta coefficients, then processed by two convolutional layers.Each layer uses 3×3 filters, 2×2 stride, and 32 filters, followed by batch normalization and ReLU.
- The decoder is a unidirectional LSTM with 256 hidden units, using symbol embeddings and the attention context as inputs.The attention energy function has hidden dimensionality d = 128, and the output layer receives the context vector concatenated with the decoder state.
- Training uses Adam with β1 = 0.9, β2 = 0.999, and ϵ = 10−6, while the learning rate drops tenfold after 600,000, 800,000, and 1,000,000 steps.This schedule improved the soft-attention baseline and MoChA but hurt hard monotonic attention, whose reported result came from prior work.
A.2 DOCUMENT SUMMARIZATION
The summarization system reimplements a pointer-generator with learned word embeddings and recurrent encoder-decoder components, trained on inputs truncated to 400 words.
- The summarization model uses one-hot inputs from a 50,000-word vocabulary mapped to 512-dimensional learned embeddings.Its encoder is a bidirectional LSTM with 512 hidden units, while its decoder is a unidirectional LSTM with 1024 hidden units.
- The attention mechanisms use hidden dimensionality d = 1024, with output words embedded into a learned 1024-dimensional representation.
- Training uses Adam with initial learning rate 0.0005, batch size 64, gradient clipping at 5, and input sequences truncated to a maximum of 400 words.The learning rate is halved every 10,000 steps after 50,000 steps until reaching 0.00005.
B SPEED BENCHMARK
A synthetic benchmark isolates attention-mechanism speed across increasing input and output lengths, showing that MoChA scales linearly while soft attention scales roughly quadratically.
- The benchmark measures attention speed alone over randomly generated encoder and decoder states with dimensionality 256.Input and output lengths T and U vary together from 10 to 100, and each timing is averaged over 100 trials.
- Soft attention exhibits roughly quadratic time complexity, whereas MoChA exhibits linear time complexity as T and U increase.
- MoChA’s complexity increases linearly with chunk size w, so its speed advantage grows with larger T and U and relatively small w.
- For T, U = 10 and w = 8, MoChA and soft attention have similar speed because the chunk effectively spans the entire memory.
C MONOTONIC ADAPTIVE CHUNKWISE ATTENTION (MATCHA)
MAtChA uses variable-sized chunks bounded by consecutive monotonic-attention locations and supports online, linear-time decoding, but its training and decoding costs are substantially higher than MoChA’s.
- Limitations: MAtChA did not outperform MoChA on the tested tasks, so the authors recommend against using it in its current form.
- Test-time decoding: MAtChA performs soft attention over variable-sized memory chunks whose boundaries are set by locations attended to by monotonic attention.At test time, the chunk begins at the previous monotonic-attention location rather than a fixed offset.
- Test-time decoding: If the current and previous monotonic-attention locations coincide, MAtChA assigns all attention to the current memory entry.
- Attention computation: MAtChA’s attention distribution sums over possible previous and current monotonic-attention locations before computing softmax attention over the resulting chunk.
- Parallel computation: A dynamic program computes MAtChA terms in parallel using cumulative sums and cumulative products.The relevant partial-sum and partial-product operations can be parallelized for each output timestep.
- Limitations: MAtChA requires O(UT^2) decoding memory, compared with O(UT) for standard soft attention, monotonic attention, and MoChA.