Source-linked AI summary
Language Modeling with Gated Convolutional Networks
Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier
TL;DR
Language modeling has relied on recurrent networks for their ability to capture unbounded context, but their sequential computation limits parallelization. This paper replaces recurrent connections with gated temporal convolutions and reports state-of-the-art WikiText-103 performance, competitive Google Billion Word results, and improved efficiency.
Problem
Recurrent language models capture long dependencies but process sequence positions sequentially, motivating a parallelizable alternative for large-scale language modeling.
Method
The paper uses stacked gated temporal convolutions to build hierarchical finite-context representations and applies gated linear units within the convolutional network.
Results
The model achieves state-of-the-art on WikiText-103, competitive results on Google Billion Word, and 38.1 test perplexity versus 39.8 for a comparable LSTM.
Takeaways & Limitations
Gated convolutional networks can model sufficiently large contexts for strong language-modeling performance while exploiting parallel computation.
Abstract
from arXiv · showhide
The pre-dominant approach to language modeling to date is based on recurrent neural networks. Their success on this task is often linked to their ability to capture unbounded context. In this paper we develop a finite context approach through stacked convolutions, which can be more efficient since they allow parallelization over sequential tokens. We propose a novel simplified gating mechanism that outperforms Oord et al (2016) and investigate the impact of key architectural decisions. The proposed approach achieves state-of-the-art on the WikiText-103 benchmark, even though it features long-term dependencies, as well as competitive results on the Google Billion Words benchmark. Our model reduces the latency to score a sentence by an order of magnitude compared to a recurrent baseline. To our knowledge, this is the first time a non-recurrent approach is competitive with strong recurrent models on these large scale language tasks.
1. Introduction
Language modeling has shifted from sparse classical n-grams to neural models, with LSTMs as the recurrent state of the art for long dependencies. The paper proposes gated convolutions as a parallelizable finite-context alternative that achieves strong benchmark results.
- LSTMs represent the prevailing language-modeling approach and can theoretically model arbitrarily long dependencies.
- The proposed gated convolutional networks stack convolutions to represent large contexts and extract increasingly abstract hierarchical features.
- Convolutional networks can process all input words simultaneously, unlike recurrent networks whose sequential dependencies prevent parallelization across a sequence.
- Gated linear units provide a linear gradient path while retaining nonlinear capabilities, addressing vanishing gradients in deep architectures.
- Gated convolutional networks outperform comparable LSTMs on Google Billion Word and achieve state-of-the-art results on WikiText-103.
2. Approach
The approach replaces recurrent connections with gated temporal convolutions that build finite-context representations without temporal dependencies. Stacked layers, causal padding, embeddings, residual blocks, and adaptive softmax form the language model.
- The model replaces recurrent connections with gated temporal convolutions, allowing parallel computation across words.
- Convolutional processing uses finite context, while stacked layers provide sufficiently large contexts for strong practical performance.
- Word embeddings are arranged as a sequence and passed through hidden convolutional layers to produce contextual representations.
- Causal convolutions prevent each position from accessing future words by shifting inputs and zero-padding the sequence beginning.
- Each gated layer combines a linear convolutional projection with sigmoid gates through element-wise multiplication.
- Adaptive softmax assigns greater capacity to frequent words and less capacity to rare words for large-vocabulary prediction.
3. Gating Mechanisms
The paper uses output-only gates to control information flow through convolutional layers. Its gated linear units preserve a nonscaled gradient path, enabling faster convergence and better perplexities than the compared gated-tanh mechanism.
- Convolutional networks do not suffer from recurrent timestep memory problems in the same way, so output gates control which information propagates through layers.
- Gated linear units couple linear units to gates, retaining nonlinear capabilities while allowing gradients to propagate without scaling.
- The gated-tanh gradient gradually vanishes as layers are stacked because tanh′(X) and σ′(X) downscale it.
- The gated linear unit has a gradient path without downscaling for activated gates, functioning as a multiplicative skip connection.
- Experiments find that gated linear units converge faster and reach better perplexities than the compared gating schemes.
4. Experimental Setup
The experiments evaluate gated convolutional language models on two large-scale benchmarks, using perplexity and GPU-based training with several optimization and architecture-search choices.
- Datasets: Two public datasets are used: Google Billion Word contains nearly one billion tokens, while WikiText-103 contains over 100M tokens and preserves consecutive sentences.Google Billion Word uses shuffled single-sentence sequences; WikiText-103 uses entire paragraphs, enabling larger contexts.
- Evaluation: Models are evaluated by computing perplexity on the standard held-out test portion of each dataset.
- Implementation: The models are implemented in Torch and trained primarily on single Tesla M40 GPUs, with larger models using an 8-GPU data-parallel setup.Workers compute separate gradient portions that are summed using Nvidia NCCL.
- Optimization: Training uses Nesterov momentum, gradient clipping, and weight normalization to increase convergence speed and stability.Nesterov momentum adds parameter-sized memory but substantially accelerates convergence with minimal additional computation.
- Optimization: Gradient clipping accelerates convergence even though the models are non-recurrent, supporting its interpretation through general trust-region methods rather than only recurrent networks.
- Model Selection: Hyperparameters are selected by random-search cross-validation, varying residual blocks, embedding size, hidden units, and kernel width.The reported search ranges include 1–10 residual blocks, 128–256-dimensional embeddings, 128–2048 units, and kernel widths of 3–5.
- Model Selection: Larger models generally perform better, while optimizer hyperparameters are straightforward to find and remain relatively stable across datasets.The setup uses Kaiming initialization, learning rates sampled from [1., 2.], momentum 0.99, and clipping 0.1.
5. Results
Across large-scale benchmarks, gated convolutional networks outperform comparable recurrent models while maintaining strong computational efficiency. Their results also show that GLU gating and finite context are sufficient for strong language modeling performance.
- 38.1 test perplexity versus 39.8 for a comparable LSTM on Google Billion Word, using the same GPUs and adaptive softmax.
- The largest GCNN reaches 31.9 test perplexity versus 30.6 for a much larger LSTM, while using 8 GPUs for 2 weeks instead of 32 GPUs for 3 weeks.
- On WikiText-103, the GCNN outperforms LSTMs despite conditioning on much longer documents, showing that its finite context is sufficient for strong results.
- The GCNN with bottlenecks improves responsiveness by 20 times while maintaining high throughput at approximately 43.9 perplexity.
- GLUs converge faster and reach lower perplexity than alternative activation mechanisms on WikiText-103 and Google Billion Word.
- Returns from larger contexts diminish drastically beyond 40 words, and strong performance can be achieved with contexts as short as 30 tokens.
6. Conclusion
The paper introduces a gated convolutional language model that builds hierarchical input representations to capture long-range dependencies. It achieves state-of-the-art on WikiText-103 and competitive Google Billion Word results with significantly fewer resources.
- Gated convolutional networks replace recurrent connections with a novel gating mechanism for language modeling.
- Hierarchical convolutional representations make it easier to capture long-range dependencies, while fixed-depth processing eases learning.
- The model achieves a new state of the art on WikiText-103.
- The model achieves competitive results on the Google Billion Word benchmark with significantly fewer resources.