Source-linked AI summary
Quasi-Recurrent Neural Networks
James Bradbury, Stephen Merity, Caiming Xiong, Richard Socher
TL;DR
RNN timestep dependencies limit parallelism and make long sequences difficult to process efficiently. QRNNs address this by combining parallel convolutions with minimalist recurrent pooling, and they outperform equal-hidden-size LSTMs across tasks while running substantially faster.
Problem
RNN timestep dependencies limit parallelism and make long sequences difficult to process efficiently.
Method
QRNNs alternate convolutional layers that operate across timesteps with minimalist recurrent pooling that operates across channels.
Results
QRNNs outperform equal-hidden-size LSTM-based models across word- and character-level tasks while running substantially faster.
Takeaways & Limitations
QRNNs can serve as building blocks for long-sequence tasks that were previously impractical with traditional RNNs.
Takeaways & Limitations
Attention’s first step is quadratic in sequence length, despite being faster in practice than the model’s linear and convolutional layers.
Abstract
from arXiv · showhide
Recurrent neural networks are a powerful tool for modeling sequential data, but the dependence of each timestep's computation on the previous timestep's output limits parallelism and makes RNNs unwieldy for very long sequences. We introduce quasi-recurrent neural networks (QRNNs), an approach to neural sequence modeling that alternates convolutional layers, which apply in parallel across timesteps, and a minimalist recurrent pooling function that applies in parallel across channels. Despite lacking trainable recurrent layers, stacked QRNNs have better predictive accuracy than stacked LSTMs of the same hidden size. Due to their increased parallelism, they are up to 16 times faster at train and test time. Experiments on language modeling, sentiment classification, and character-level neural machine translation demonstrate these advantages and underline the viability of QRNNs as a basic building block for a variety of sequence tasks.
1 INTRODUCTION
QRNNs combine CNN-like parallel computation with RNN-like sensitivity to sequence order, targeting the limited parallelism and long-sequence scaling of recurrent models. Across three language tasks, they outperform strong LSTM baselines while reducing computation time.
- RNNs model sequential data effectively but each timestep depends on the previous hidden-state output, limiting parallelism.
- CNNs apply time-invariant filters in parallel across sequence windows, improving parallelism and scaling to long sequences.
- QRNNs alternate parallel convolutional computation with recurrent pooling, combining timestep parallelism with dependence on sequence order.
- QRNN models outperform strong LSTM baselines on sentiment classification, language modeling, and character-level machine translation while reducing computation time.
2 MODEL
A QRNN layer uses masked convolutions to compute candidate features and gates in parallel, followed by simple channel-wise recurrent pooling. The architecture supports several pooling variants and extensions, including attention, regularization, dense connections, and encoder–decoder models.
- 2 MODEL: QRNN layers combine convolutional and pooling subcomponents, parallelizing convolutions across sequence positions and pooling across minibatch and feature dimensions.
- 2 MODEL: Masked convolutions produce candidate vectors using only current and preceding inputs, preventing access to future timesteps.
- 2 MODEL: Separate masked convolutions generate candidate vectors and sigmoid gates, while larger filter widths capture higher n-gram features.
- 2 MODEL: Pooling variants use forget and output gates to mix states across timesteps independently by channel, with negligible computation from their simple recurrence.
- 2 MODEL: A QRNN layer performs input-dependent pooling followed by a gated linear combination of convolutional features, and stacked layers increase modeling capacity.
- 2.1 VARIANTS: QRNN extensions include zoneout-style regularization, dense connections, and encoder–decoder attention for sequence-to-sequence tasks.
- 2.1 VARIANTS: The encoder–decoder variant supplements decoder convolutions with the final encoder state and computes soft attention over encoder hidden states.
- 2.1 VARIANTS: The first attention step is quadratic in sequence length, although its parallel dot-product scoring is faster in practice than the model’s linear and convolutional layers.
3 EXPERIMENTS
Across sentiment classification, language modeling, and character-level translation, QRNN models were evaluated against LSTM baselines and showed competitive or better predictive performance alongside substantial speed advantages.
- 3.1 SENTIMENT CLASSIFICATION: 3.2x faster IMDb training per epoch was observed for QRNNs than NVIDIA cuDNN LSTMs, with up to 16x gains for specific batch sizes and sequence lengths.The comparison concerns optimized LSTM and QRNN implementations; Figure 4 provides broader speed comparisons.
- 3.1 SENTIMENT CLASSIFICATION: QRNN hidden-state changes were visible and interpretable on IMDb examples, with channel-wise pooling delaying direct interaction until the next QRNN layer.The visualization includes resets and recovery of hidden states in response to phrases within a review.
- 3.2 LANGUAGE MODELING: On PTB language modeling, the QRNN was highly competitive with LSTMs, and the QRNN without zoneout strongly outperformed medium LSTMs without recurrent dropout.With zoneout p = 0.1, the QRNN required no early stopping and achieved competitive perplexity to a variational LSTM.
- 3.2 LANGUAGE MODELING: QRNN training on PTB was substantially faster than standard and optimized cuDNN LSTMs because recurrent layers were no longer the computational bottleneck.For the tested 10,000-word vocabulary, softmax and optimization overhead took equal or greater time than the QRNN recurrent components.
- 3.3 CHARACTER-LEVEL NEURAL MACHINE TRANSLATION: In character-level IWSLT German–English translation, the QRNN outperformed an equal-sized character-level attentional LSTM and almost matched a word-level attentional baseline.The table reports translation performance using BLEU and training speed in hours per epoch.
4 RELATED WORK
QRNNs relate to strongly-typed recurrent and hybrid convolutional–recurrent models while combining convolutional parallelism with recurrent context. Their encoder–decoder design also shares parallelism and path-length properties with ByteNet.
- Strongly-typed recurrent models: QRNNs are related to strongly-typed recurrent architectures, but attention and skip-connections make some QRNNs outside the strongly typed family.The comparison distinguishes QRNN variants by filter size, pooling choice, and activation functions.
- Hybrid convolutional–recurrent models: Hybrid convolutional–recurrent models typically use convolutions to generate features before feeding them into recurrent layers for text classification or translation.Examples include word-level and character-level convolutions followed by LSTMs or highway networks.
- Encoder–decoder models: QRNN encoder–decoder models share ByteNet’s favorable parallelism and path-length properties for character-level machine translation.ByteNet is described as using residual convolutions over binary trees to support parallelism, linear-time complexity, and short paths.
5 CONCLUSION
QRNNs combine convolutional parallelism with recurrent handling of long-distance context, yielding faster and more accurate sequence modeling than equal-hidden-size LSTM-based models. The authors conclude that these advantages persist across tasks and representation levels, supporting QRNNs as building blocks for long-sequence tasks.
- Core conclusion: QRNNs combine parallel convolutional computation for context-invariant information with recurrent processing for long-distance context.This design aims to retain contextual information without giving up the parallelism of convolutional models.
- Core conclusion: QRNNs achieve better predictive accuracy than equal-hidden-size LSTM-based models while using fewer parameters and running substantially faster.The conclusion attributes the combined advantage to exploiting both convolutional and recurrent properties.
- Implications: QRNNs can serve as building blocks for long-sequence tasks that were previously impractical with traditional RNNs.The authors also note that CNN and RNN extensions often apply directly to QRNNs and that channel independence improves hidden-state interpretability.
APPENDIX
The appendix describes a modified beam-search ranking criterion that applies length normalization during decoding. It includes ordinary probability ranking and length-normalized ranking as endpoint cases and can be evaluated incrementally.
- Beam search ranking criterion: The appendix introduces a modified log-probability ranking criterion for translation beam search.The criterion is explicitly presented as the ranking method used in the translation experiments.
- Beam search ranking criterion: The criterion uses α for length normalization and defines Ttrg as the source sentence length plus five characters.wi denotes the ith output character.
- Normalization settings: At α = 0, the criterion reduces to ordinary beam search with probabilities.This is the unnormalized endpoint of the proposed ranking scheme.
- Normalization settings: At α = 1, the criterion yields beam search with probabilities normalized by length up to the target length.The appendix presents this as the length-normalized endpoint.
- Decoding procedure: The ranking criterion can be computed at intermediate beam-search timesteps, eliminating separate reranking of complete hypotheses.This supports incremental use during decoding rather than only after hypotheses finish.