Source-linked AI summary
Neural Machine Translation in Linear Time
Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, Koray Kavukcuoglu
TL;DR
ByteNet addresses limitations of recurrent sequence models with a convolutional encoder-decoder architecture that preserves source resolution and dynamically handles differing sequence lengths. It runs in linear time, achieves state-of-the-art results in character-level language modelling and translation, and learns latent alignments that mirror expected token alignments.
Problem
Recurrent sequence models are inherently serial and require signals to traverse long paths between tokens, complicating parallel computation and learning long-range dependencies.
Method
ByteNet uses dilated one-dimensional convolutional encoder and decoder networks, stacking the decoder over a resolution-preserving encoder representation and dynamically unfolding it for source and target sequences of different lengths.
Results
ByteNet has linear running time, reaches state-of-the-art character-level language modelling and character-to-character translation performance, and its latent structure mirrors expected token alignments.
Takeaways & Limitations
The results support convolutional sequence models as scalable alternatives to recurrent models for character-level language modelling and machine translation.
Takeaways & Limitations
The proposed design assumes source representations remain linear in source-string length and that short signal paths help learning long-range dependencies.
Abstract
from arXiv · showhide
We present a novel neural network for processing sequences. The ByteNet is a one-dimensional convolutional neural network that is composed of two parts, one to encode the source sequence and the other to decode the target sequence. The two network parts are connected by stacking the decoder on top of the encoder and preserving the temporal resolution of the sequences. To address the differing lengths of the source and the target, we introduce an efficient mechanism by which the decoder is dynamically unfolded over the representation of the encoder. The ByteNet uses dilation in the convolutional layers to increase its receptive field. The resulting network has two core properties: it runs in time that is linear in the length of the sequences and it sidesteps the need for excessive memorization. The ByteNet decoder attains state-of-the-art performance on character-level language modelling and outperforms the previous best results obtained with recurrent networks. The ByteNet also achieves state-of-the-art performance on character-to-character machine translation on the English-to-German WMT translation task, surpassing comparable neural translation models that are based on recurrent networks with attentional pooling and run in quadratic time. We find that the latent alignment structure contained in the representations reflects the expected alignment between the tokens.
1. Introduction
ByteNet addresses the scaling and memorization limitations of recurrent and compressed encoder-decoder translation models with a resolution-preserving, dynamically unfolded convolutional architecture. It achieves state-of-the-art results in character-level language modelling and character-to-character translation while retaining linear running time.
- Existing neural translation models either run super-linearly or compress the source into a fixed-size representation, making both drawbacks worse for longer sequences.
- ByteNet stacks a target decoder on the source encoder while preserving temporal resolution, avoiding a fixed-size source representation.
- Dynamic unfolding lets the decoder process source and target sequences of different lengths efficiently.
- ByteNet uses fixed-depth one-dimensional CNNs with increasing dilation, while masked decoder convolutions prevent access to future target tokens.
- Its running time is linear in source and target length, and encoder training/decoding plus decoder training can run efficiently in parallel.The linear-time bound holds up to a constant c ≈ log d, where d is the desired dependency-field size.
- The decoder reaches 1.31 bits/character on the Hutter Prize Wikipedia task and ByteNet achieves BLEU scores of 22.85 and 25.53 on WMT English-German 2014 and 2015 test sets.The paper reports that ByteNet outperforms recurrent language models and a comparable GNMT system on character-level translation.
2. Neural Translation Model
Neural translation models estimate target-string distributions conditioned on source strings. The paper emphasizes parallel sequence computation, resolution-preserving representations, and short signal paths as architectural desiderata.
- A neural translation model estimates a distribution over target strings conditioned on a source string.
- The target distribution becomes tractable through a product of conditional probabilities over target tokens.
- Parallel operations along sequence length can reduce computation time.
- Desiderata: The source representation should scale linearly with source length rather than have constant size, avoiding an additional memorization step.
- Desiderata: Short paths between input and output tokens may propagate signals better and help learn long-range dependencies.
3. ByteNet
ByteNet stacks a decoder over a resolution-preserving encoder and dynamically unfolds it to handle differing sequence lengths. Its dilated convolutions expand receptive fields while retaining high representational bandwidth.
- 3. ByteNet: Figure 3 contrasts decoder residual blocks using ReLUs with residual multiplicative blocks, while the experiments assign them to translation and language modelling, respectively.
- 3.1. Encoder-Decoder Stacking: ByteNet stacks the decoder on the encoder representation to preserve temporal resolution and maximize representational bandwidth.
- 3.2. Dynamic Unfolding: Dynamic unfolding connects encoder and decoder networks that process sequences of different lengths by adapting the decoder to the target length.
- 3.2. Dynamic Unfolding: The target-length upper bound is chosen as a linear function of source length, with a = 1.20 and b = 0 for English-to-German translation.
- 3.4. Masked One-dimensional Convolutions: Masked one-dimensional convolutions prevent future target tokens from affecting current predictions, while dilation enlarges the receptive field exponentially with network depth.
4. Model Comparison
The comparison evaluates translation architectures by running time, sequence-resolution preservation, network type, and path length. ByteNet combines linear running time with preserved resolution and short paths.
- Compared models: The analysis includes convolutional and recurrent translation models, including two recurrent ByteNet variants that are not evaluated experimentally.The comparison covers RCTM, RNN Enc-Dec, attentional pooling, Grid LSTM, and Extended Neural GPU models.
- Comparison criteria: The comparison separates computation time, source and target network type, and resolution preservation into explicit criteria.CNNs can run in parallel along sequence length, whereas RNNs are recurrent.
- Running time: ByteNet, Recurrent ByteNets, and RNN Enc-Dec are the only compared networks with linear running time.The comparison treats linear time as holding up to a constant c.
- Resolution and complexity: RNN Enc-Dec has linear running time but does not preserve source-sequence resolution, while several resolution-preserving models have quadratic running time.The resolution-preserving quadratic-time models include RCTM 2, RNN Enc-Dec Att, Grid LSTM, and Extended Neural GPU.
- Path properties: ByteNet’s dilated convolutions connect any two source or target tokens through a small number of layers tied to network depth.For character sequences, the resulting paths are sublinear in token distance.
5. Character Prediction
The ByteNet decoder is evaluated as a character-level language model on the Hutter Prize Wikipedia benchmark. Its dilated residual architecture yields a strong test-set result.
- Dataset: The Hutter Prize Wikipedia task uses 90 million bytes for training, 5 million for validation, and 5 million for testing, with a 205-character vocabulary.The evaluation follows the standard dataset split.
- Architecture: The decoder uses 30 residual blocks arranged in six five-block sets with dilation rates 1, 2, 4, 8, and 16.A masked kernel size of 3 produces a receptive field of 315 characters.
- Result: 1.31 bits/character is achieved by the ByteNet decoder on the Wikipedia test set.The other listed results use variants of recurrent LSTM models.
6. Character-Level Machine Translation
The full ByteNet performs character-to-character English-to-German translation without explicit word or morpheme segmentation. It achieves leading benchmark performance and produces translations illustrating reordering and transliteration.
- Task setup: The evaluation uses character sequences for both English inputs and German outputs, with no explicit word or morpheme segmentation.The English and German vocabularies contain 296 and 323 characters, respectively.
- Architecture: The translation model uses 30 residual blocks in both encoder and decoder, with repeated dilation rates of 1, 2, 4, 8, and 16.The encoder and masked target kernels both have size 3.
- Decoding: Padding to multiples of 50, additional 20% source padding, bucketed batching, and beam search with size 12 support efficient decoding and training.The beam search uses total candidate likelihood and accepts only candidates ending with an end-of-sentence token.
- Qualitative analysis: Generated translations highlight reordering and transliteration phenomena, while character-level outputs can make post-processing unnecessary in principle.Gradient-based visualization is used to inspect prediction sensitivity to source and target inputs.
- Results: The ByteNet achieves the highest NewsTest 2014 performance among character-level and subword-level neural systems and the best published NewsTest 2015 results.On NewsTest 2014, it ranks below only word-piece GNMT among word-level systems.
7. Conclusion
The paper concludes that ByteNet combines linear-time translation, reduced reliance on memorization, and short signal paths. It reports state-of-the-art character-level language modelling and translation, with latent representations mirroring expected token alignments.
- Contributions: ByteNet has linear running time, decouples translation from memorization, and provides short signal propagation paths.These are presented as the model’s central architectural properties.
- Language modelling: The ByteNet decoder is reported as a state-of-the-art character-level language model that outperforms recurrent neural language models.The decoder is based on a convolutional neural network.
- Translation: ByteNet achieves state-of-the-art character-to-character machine translation and excellent general results while maintaining linear running-time complexity.The paper also describes the model as a generalization of the RNN Enc-Dec architecture.
- Representations: The latent structure learned by ByteNet mirrors the expected alignment between tokens in the sentences.This conclusion concerns the representations learned by the model.