Source-linked AI summary
Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data
Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, Jingming Liu
TL;DR
GEC must correct errors while preserving much of the source, and limited labeled data constrains high-accuracy learning. The paper introduces a copy-augmented architecture, pre-trains it with denoising auto-encoding on unlabeled data, and adds multi-task learning; combined, these methods reach 61.15 F0.5 on CoNLL-2014, exceeding the previous state of the art by +4.9 F0.5 points.
Problem
GEC differs from translation because it changes only several source words, while achieving high accuracy remains challenging with insufficient labeled training data.
Method
The paper enables direct copying of unchanged and out-of-vocabulary source words, pre-trains the architecture with denoising auto-encoders on unlabeled data, and adds token- and sentence-level multi-task learning.
Results
61.15 F0.5 on the CoNLL-2014 test set exceeds the previous state-of-the-art system by +4.9 F0.5 points when denoising auto-encoders and multi-task learning are combined.
Takeaways & Limitations
The approach outperforms recently published state-of-the-art GEC systems by a large margin on the reported evaluation.
Takeaways & Limitations
The approach remains short of human-level reliability because GEC includes complex error types, particularly subjective and culturally characterized errors.
Abstract
from arXiv · showhide
Neural machine translation systems have become state-of-the-art approaches for Grammatical Error Correction (GEC) task. In this paper, we propose a copy-augmented architecture for the GEC task by copying the unchanged words from the source sentence to the target sentence. Since the GEC suffers from not having enough labeled training data to achieve high accuracy. We pre-train the copy-augmented architecture with a denoising auto-encoder using the unlabeled One Billion Benchmark and make comparisons between the fully pre-trained model and a partially pre-trained model. It is the first time copying words from the source context and fully pre-training a sequence to sequence model are experimented on the GEC task. Moreover, We add token-level and sentence-level multi-task learning for the GEC task. The evaluation results on the CoNLL-2014 test set show that our approach outperforms all recently published state-of-the-art results by a large margin. The code and pre-trained models are released at https://github.com/zhawe01/fairseq-gec.
1 Introduction
GEC corrects grammatical errors while usually preserving most source words, motivating a copy-augmented architecture. The paper combines copying with unlabeled-data pre-training and multi-task learning to address limited labeled data and improve performance.
- GEC detects and corrects grammatical errors in text, a task receiving increased attention with the growing number of English language learners.
- More than 80% of words can be copied from the source sentence, distinguishing GEC from translation that changes only several source words.
- The proposed architecture directly copies unchanged and out-of-vocabulary words from source tokens, the first reported neural copying mechanism for GEC.
- Denoising auto-encoders pre-train the copy-augmented model on unlabeled One Billion Benchmark data to alleviate insufficient labeled training data.
- Token-level labeling and sentence-level copying are added as auxiliary tasks to further improve the copy-augmented architecture’s GEC performance.
- Copying supports a small vocabulary, strengthens generation by separating unchanged output, and can recall more GEC edits than simply resolving unknown words.
- 61.15 F0.5 on the CoNLL-2014 test set exceeds state-of-the-art systems by +4.9 F0.5 points when denoising auto-encoders and multi-task learning are combined.
2 Our Approach
The approach augments a Transformer GEC model with source-token copying, combining generation and copying to handle unchanged and out-of-vocabulary words. The resulting architecture is designed specifically for GEC’s largely extractive character.
- 2.1 Base Architecture: The Transformer encoder-decoder maps learner sentences to corrected sentences using self-attention and encoder-decoder attention.
- 2.1 Base Architecture: The decoder predicts each next target word from the source tokens and its target hidden state.The generation distribution is obtained by applying softmax to the target hidden state and word-embedding matrix.
- 2.2 Copying Mechanism: Copying enables the model to reproduce unchanged and out-of-vocabulary source words directly while making the generating component more powerful.
- 2.2 Copying Mechanism: The copy mechanism was applied to GEC for the first time in this work.
- 2.2 Copying Mechanism: The copy-augmented model combines fixed-vocabulary generation with a copy distribution over source input tokens.Its final distribution mixes generation and copying, extending the effective vocabulary with words in the source sentence.
3 Pre-training
The paper pre-trains its copy-augmented sequence-to-sequence model on unlabeled text with denoising auto-encoding, and compares this with decoder-only pre-training. Artificial corruption creates reconstruction pairs resembling GEC corrections.
- 3.1 Denoising Auto-encoder: Denoising auto-encoding pre-trains the copy-augmented model on the unlabeled One Billion Word Benchmark.The method is intended to address insufficient labeled GEC data.
- 3.1 Denoising Auto-encoder: The corruption process deletes tokens, adds tokens, replaces words randomly, and shuffles word positions.Each listed operation is applied with the probabilities or positional-noise setting specified in the training procedure.
- 3.1 Denoising Auto-encoder: With abundant artificial data, the sequence-to-sequence model learns to reconstruct inputs while trusting most, but not all, source tokens.
- 3.1 Denoising Auto-encoder: The authors relate corrupted-to-clean reconstruction pairs to GEC pairs because both map imperfect sentences toward perfect sentences.
- 3.2 Partial Pre-training: A partially pre-trained variant initializes the decoder as a language model while leaving the encoder and encoder-decoder attention randomly initialized.Tied encoder-decoder embeddings mean most model parameters are pre-trained except those components.
4 Multi-Task Learning
The paper explores two GEC auxiliary tasks: token-level correctness labeling and sentence-level copying. These tasks jointly train the model to identify edits and copy more aggressively when sentences are correct.
- 4 Multi-Task Learning: The multi-task setup explores two auxiliary tasks intended to improve GEC performance.
- 4.1 Token-level Labeling Task: Token-level labeling assigns each source token a right or wrong label based on whether it aligns identically with a target token.
- 4.1 Token-level Labeling Task: The encoder predicts each token’s correctness label through an affine transformation followed by softmax.
- 4.1 Token-level Labeling Task: Token-level labeling explicitly provides source-token correctness information to the encoder for later use by the decoder.
- 4.2 Sentence-level Copying Task: The sentence-level copying task encourages more copying when the input sentence appears entirely correct.
- 4.2 Sentence-level Copying Task: Training balances sampled correct and edited sentence pairs, removing decoder attention to encoder outputs for correct inputs to boost copying.
5 Evaluations
The evaluation uses public labeled GEC corpora, the unlabeled One Billion Word Benchmark, and CoNLL-2014 as the main test set. Models use a Transformer-based setup with specified optimization, loss-weighting, and decoding procedures.
- 5.1 Datasets: The labeled training data comprise NUCLE, Lang-8, and FCE, while the One Billion Word Benchmark supplies unlabeled data.CoNLL-2013 is used for development and CoNLL-2014 for testing; JFLEG is another evaluation set.
- 5.1 Datasets: CoNLL-2014 and CoNLL-2013 use M2 scores, whereas JFLEG uses the GLEU metric.The datasets and metrics are selected to support comparison with prior GEC systems.
- 5.2 Implementation Details: The implementation uses the Transformer in the public FAIR Sequence-to-Sequence Toolkit.The model has 6 encoder and decoder layers, 8 attention heads, 512-dimensional embeddings and hidden states, and 97M parameters.
- 5.2 Implementation Details: Models are optimized with Nesterov’s Accelerated Gradient and evaluated on the development set after every epoch.The reported settings include learning rate 0.002, weight decay 0.5, momentum 0.99, and minimum learning rate 10^-4.
- 5.2 Implementation Details: The training objective includes edit-weighted maximum likelihood, scaling the loss of changed words by Λ.Λ is set to 3 for denoising auto-encoder training and between 1 and 1.8 for GEC models.
- 5.2 Implementation Details: Decoding uses beam size 12 and length-normalized scores, with language-model reranking applied only for JFLEG evaluation.No reranking is used for CoNLL-2014 evaluation.
5.3 Experimental Results
The copy-augmented architecture surpasses prior systems on CoNLL-2014, and combining copying with denoising auto-encoding and multi-task learning yields the strongest reported result. Comparisons with a system using more non-public labeled data are not direct.
- Comparison with Prior Systems: 56.42 F0.5 is achieved by the copy-augmented architecture on CoNLL-2014 without reranking or pre-training, outperforming previous architectures.Prior systems commonly used reranking and some used partially pre-trained parameters.
- Comparison with Prior Systems: 61.15 F0.5 is achieved when denoising auto-encoders and multi-task learning are combined with the model.This exceeds the previous state-of-the-art system by +4.9 F0.5 points.
- Comparison with Prior Systems: The comparison with Ge et al. (2018) is not direct because that system used non-public corpora and 3.6 times more labeled training data.The paper reports that its CoNLL-2014 and JFLEG results are nevertheless very close to theirs.
- Evaluation Measures: Table 4 reports M2 scores for CoNLL-2014 and GLEU scores for JFLEG, including effects of language-model reranking and four-model ensembles.DA denotes denoising auto-encoder pre-training.
5.4 Ablation Study
Ablation studies show gains from copying, pre-training, and sentence-level copying objectives. Copying improves performance beyond handling out-of-vocabulary words, while pre-training improves both early initialization and final scores.
- 5.4.1 Copying Ablation Results: 54.67 F0.5 is achieved with copying versus 48.07 for the Transformer baseline, a +6.6 absolute increase.Most improvement comes from out-of-vocabulary words copied rather than predicted as UNK.
- 5.4.1 Copying Ablation Results: 1.62 F0.5 points remain after all UNK edits are ignored, with most of the benefit coming from increased recall.This indicates copying contributes beyond direct UNK-word handling.
- 5.4.2 Pre-training Ablation Results: 58.8 F0.5 is obtained with denoising auto-encoder pre-training, compared with 54.67 without pre-training, while partially pre-training the decoder reaches 57.21.Both precision and recall improve after pre-training.
- 5.4.2 Pre-training Ablation Results: 48.89 versus 17.19 F0.5 after one labeled-data fine-tuning epoch shows a large early advantage for the pre-trained model.Without fine-tuning, the pre-trained model reaches 31.33 F0.5.
- 5.4.3 Sentence-level Copying Task Ablation Results: The sentence-level copying task is designed to encourage no edits for correct input sentences.The evaluation samples 500 correct Wikipedia sentences and 500 error sentences from CoNLL-2013.
- 5.4.3 Sentence-level Copying Task Ablation Results: After adding the sentence-level copying task, α_copy changes from 0.44/0.45 to 0.81/0.57 for correct/error sentence sets.The authors interpret this as improved distinction between correct and erroneous sentences.
5.5 Attention Visualization
The attention visualization separates copying from generation: copying attends locally to the next word, while generation attends more broadly to nearby words and sentence-ending information. The paper links this division to handling long dependencies and global information.
- Attention Roles: Copying attention focuses its weights on the next word in good order, whereas generation attends more to nearby words and the sentence end.The visualization compares copying attention alignment with encoder-decoder attention alignment.
- Attention Roles: Generation attends to broader information to find long dependencies, while copying handles the local copying pattern.The paper describes this division as separating copying work from generation work.
6 Discussion
The discussion examines recall across grammatical error types, showing strong performance on mechanical agreement errors but a substantial weakness on subjective or culturally dependent errors.
- Error-type analysis: The analysis reports recall across the nine most frequent error types, grouping the remaining 19 types into a final table row.The figure illustrates how copy and encoder-decoder attention focus on source tokens when predicting corrected words.
- Discussion: The systems perform well on definite mechanical errors but remain far behind humans on subjective errors with cultural characteristics.
7 Related Work
Earlier GEC systems used classifiers and statistical or neural translation architectures, while recent work increasingly combines neural modeling with auxiliary information and model-independent methods.
- Earlier approaches: Early GEC research developed error-specific classifiers and hybrid systems before treating correction as a statistical machine translation problem.
- Neural approaches: Neural approaches included sequence labeling for error detection, word- and character-level hybrid modeling, and convolutional encoder-decoder networks.
- Neural approaches: Deep RNN and transformer encoder-decoder models improved results when combined with model-independent methods for neural GEC.
- State of the art: The reported state-of-the-art system used fluency-boost learning and inference but relied on non-public training data, complicating comparisons.
8 Conclusions
The paper concludes by combining a copy-augmented GEC architecture, full unlabeled-data pre-training, and auxiliary multi-task learning. It reports a large-margin improvement over the state-of-the-art system while acknowledging that human-level reliability remains unresolved.
- Conclusions: The architecture directly copies unchanged and out-of-vocabulary words from source tokens and is fully pre-trained with large-scale unlabeled data using denoising auto-encoders.
- Conclusions: The method introduces two auxiliary tasks for multi-task learning.
- Conclusions: The approach outperforms the state-of-the-art automatic GEC system by a large margin, but automatic correction is not yet as reliable as human correction.