Source-linked AI summary
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, Quoc V. Le
TL;DR
Recurrent reading-comprehension models can be slow because of sequential processing. QANet replaces recurrent encoders with convolution and self-attention, achieving large speedups and reaching 84.6 F1 on SQuAD with augmented data.
Problem
Recurrent reading-comprehension models achieve strong results but are often slow for training and inference because of their sequential nature.
Method
QANet uses a fully feed-forward encoder built from convolutions and self-attention, combined with attention and backtranslation-based data augmentation.
Results
84.6 F1 score on the SQuAD test set exceeded the best published result of 81.8, while QANet was 3x to 13x faster in training and 4x to 9x faster in inference.
Takeaways & Limitations
QANet's speedup allows training with more data, and augmented-data training improves SQuAD accuracy beyond the best published result.
Takeaways & Limitations
Backtranslation quality and diversity remain improvable, and paraphrases longer than the maximum training sequence length may be cut off.
Abstract
from arXiv · showhide
Current end-to-end machine reading and question answering (Q\&A) models are primarily based on recurrent neural networks (RNNs) with attention. Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs. We propose a new Q\&A architecture called QANet, which does not require recurrent networks: Its encoder consists exclusively of convolution and self-attention, where convolution models local interactions and self-attention models global interactions. On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference, while achieving equivalent accuracy to recurrent models. The speed-up gain allows us to train the model with much more data. We hence combine our model with data generated by backtranslation from a neural machine translation model. On the SQuAD dataset, our single model, trained with augmented data, achieves 84.6 F1 score on the test set, which is significantly better than the best published F1 score of 81.8.
1 INTRODUCTION
QANet removes recurrent encoders from reading comprehension, using convolutions and self-attention to improve speed while preserving accuracy. Data augmentation through paraphrasing further improves SQuAD performance.
- 1 INTRODUCTION: Recurrent models are slow because sequential processing increases training and inference costs, limiting rapid experimentation, larger-dataset use, and real-time deployment.These limitations motivate replacing recurrence in machine reading comprehension systems.
- 1 INTRODUCTION: QANet replaces recurrent encoders with convolutions and self-attention to make machine comprehension faster.The architecture separately encodes queries and contexts, models their interactions with attention, and decodes answer-span boundaries.
- 1 INTRODUCTION: 3x to 13x faster training and 4x to 9x faster inference were achieved on SQuAD while maintaining equivalent accuracy to recurrent models.QANet matched BiDAF's 77.0 F1 score after 3 hours, compared with 15 hours for BiDAF.
- 1 INTRODUCTION: Convolution captures local text structure, whereas self-attention learns global interactions between word pairs.This division of labor motivates the feed-forward encoder design.
- 1 INTRODUCTION: Backtranslation augments training data by paraphrasing examples, increasing both the number of instances and phrasing diversity.The method translates English sentences into another language and then back into English.
- 1 INTRODUCTION: 84.6 F1 score on the SQuAD test set exceeded the best published result of 81.8.This result was obtained by QANet trained with augmented data.
2 THE MODEL
QANet is a feedforward reading-comprehension model that replaces recurrent encoders with convolution and self-attention while retaining standard attention and span prediction components. Its architecture combines local and global modeling, shared encoder structures, and a task-specific output layer.
- Encoder architecture: QANet uses convolutional and self-attention mechanisms instead of recurrent networks in its embedding and modeling encoders.The model is organized into encoder blocks combining convolution, self-attention, and feed-forward layers.
- Encoder architecture: Convolutions capture local text structure, while self-attention models global interactions between words.This combination is the central design rationale for the feedforward encoder.
- Encoder architecture: Encoder sublayers use depthwise separable convolutions, multi-head self-attention, feed-forward layers, layer normalization, residual connections, and positional encodings.The same encoder-block design is reused with varying numbers of convolutional layers, and weights are shared in specified encoder groups.
- Context-query attention: The context-query attention layer constructs query-aware context representations from similarities between context and query words.It computes context-to-query attention from a row-normalized similarity matrix and additionally uses query-to-context attention.
- Output layer: The output layer predicts start and end probabilities for answer spans, whose score is the product of the two position probabilities.Training minimizes the averaged negative log probabilities at the true start and end indices; inference selects a span with s ≤ e.
3 DATA AUGMENTATION BY BACKTRANSLATION
The paper augments reading-comprehension training data by backtranslating text through a pivotal language, generating paraphrases while preserving question-answer structure. For SQuAD, this approach improves data quality and diversity, though paraphrase quality remains a limiting factor.
- Data augmentation: Backtranslation enriches training data by translating English text into another language and back, producing paraphrases with more varied phrasing.The method uses neural machine translation models and is intended to increase the number and diversity of training instances.
- SQuAD procedure: For SQuAD, the question remains unchanged while documents are paraphrased and new answers are extracted from the resulting documents.The procedure creates new triples (d′, q, a′) through document paraphrasing followed by answer extraction.
- Paraphrase generation: Using beam decoding with k = 5 yields 25 paraphrase choices per sentence, after which one paraphrase is randomly selected for each sentence.The document is formed by independently paraphrasing its sentences and replacing each original sentence with a selected alternative.
- Answer extraction: Answer extraction identifies a paraphrased answer by comparing character-level 2-gram scores between candidate words and the original answer boundaries.The highest-scoring candidate is selected as the new answer a′ in the paraphrased sentence.
- Limitations: Paraphrase quality and diversity remain important constraints, with long paraphrases potentially truncated by the model’s maximum training sequence length.The authors suggest better translation models, sampling during beam search, and paraphrasing questions and answers as possible improvements.
4 EXPERIMENTS
Experiments evaluate QANet on SQuAD and TriviaQA, measuring accuracy, speed, data augmentation, and robustness. QANet matches or exceeds strong baselines while providing substantial speedups over recurrent models.
- Accuracy: 76.2/84.6 EM/F1 on SQuAD test significantly outperforms the best documented result of 73.2/81.8.
- Speed: 3 to 13 times faster in training and 4 to 9 times faster in inference than recurrent models on SQuAD.
- Speed: 4.3 and ... QANet is faster than BiDAF when obtaining the same performance on SQuAD.The supplied passage truncates the second speedup value.
- Ablation study and analysis: Almost 3 percentage points of both F1 and EM are lost when convolutions are removed, while self-attention contributes 1.4/1.3 EM/F1.The ablation attributes these effects to complementary local and global modeling.
- Ablation study and analysis: 0.5 F1 is gained by doubling training data with English-French-English paraphrases, and another 0.2 by adding equally sized English-German-English data.
- Ablation study and analysis: 0.5/0.3 EM/F1 drops when augmented-data sampling changes from (1:1:1) to (1:2:1), suggesting that back-translated data can be noisy.
- Robustness study: On adversarial SQuAD, QANet is on par with Mnemonic and significantly better than other compared models.The authors associate this robustness with training on augmented data.
- Experiments on TriviaQA: 3 to 11 times faster in training and 3 to 9 times faster in inference on TriviaQA than recurrent counterparts.
5 RELATED WORK
Earlier reading-comprehension systems predominantly used recurrent networks, whose sequential computation limited parallelism and complicated long-dependency modeling. Prior acceleration attempts retained recurrent components or sacrificed accuracy, while existing augmentation methods did not diversify syntax.
- RNN-based reading-comprehension models process tokens sequentially, preventing parallel computation and making long dependencies difficult to model.
- Skipping irrelevant tokens had been explored for simpler tasks, but its suitability for complicated question answering remained unclear.
- Earlier fast Q&A approaches retained RNNs or removed context-query attention, with the latter reporting EM 68.4 and F1 77.1.
- QANet was presented as the first model to combine self-attention and convolutions, achieving a significant gain of 2.7 F1.
- Earlier SQuAD augmentation by type swapping improved accuracy but preserved the original data's syntactic structure.
6 CONCLUSION
QANet is a fully feedforward reading-comprehension model that removes recurrent networks from the encoder. It combines convolutions, attention, linear layers, and layer normalization, surpasses published SQuAD results, and benefits from backtranslation-based augmentation.
- QANet removes recurrent networks from the encoder and uses separable convolutions, attention, linear layers, and layer normalization.
- QANet surpasses the best published SQuAD results while running up to 13/9 times faster than competitive recurrent models for training/inference iterations.
- Backtranslation-based data augmentation produces significant gains by paraphrasing questions and contexts through translation to and from another language.