Source-linked AI summary
Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, Kenneth Heafield
TL;DR
Neural GEC had not matched phrase-based SMT, raising whether low-resource neural MT methods could close the gap. The paper adapts those methods and establishes trustability guidelines, obtaining better-than-state-of-the-art neural systems across CoNLL-2014 and JFLEG. Its conclusions are qualified by optimizer instability and architecture-sensitive transfer-learning pretraining.
Problem
Neural GEC systems lagged behind phrase-based SMT and hybrid systems on established benchmarks.
Method
The paper adapts low-resource neural MT and SMT-based GEC techniques, including BPE preprocessing, source noise, domain adaptation, a GEC-specific objective, transfer learning, and ensembling.
Results
More than 10% M2 improvement over the previous best neural GEC system was achieved on CoNLL-2014, while JFLEG improvement over the previous best pure neural system was 5.9% GLEU.
Takeaways & Limitations
Combined model-independent methods allow pure neural GEC systems to outperform prior state-of-the-art systems on CoNLL-2014 and achieve the best reported JFLEG results.
Takeaways & Limitations
Neural GEC results can vary by about 3 M2 points across initializations, and transfer-learning pretraining must be adapted to the model architecture to exploit every shared parameter.
Abstract
from arXiv · showhide
Previously, neural methods in grammatical error correction (GEC) did not reach state-of-the-art results compared to phrase-based statistical machine translation (SMT) baselines. We demonstrate parallels between neural GEC and low-resource neural MT and successfully adapt several methods from low-resource MT to neural GEC. We further establish guidelines for trustable results in neural GEC and propose a set of model-independent methods for neural GEC that can be easily applied in most GEC settings. Proposed methods include adding source-side noise, domain-adaptation techniques, a GEC-specific training-objective, transfer learning with monolingual data, and ensembling of independently trained GEC models and language models. The combined effects of these methods result in better than state-of-the-art neural GEC models that outperform previously best neural GEC systems by more than 10% M$^2$ on the CoNLL-2014 benchmark and 5.9% on the JFLEG test set. Non-neural state-of-the-art systems are outperformed by more than 2% on the CoNLL-2014 benchmark and by 4% on JFLEG.
1 Introduction
Neural GEC lagged behind phrase-based and hybrid systems despite neural MT’s broader success. The paper frames GEC as low-resource MT and adapts methods from that setting to surpass prior neural and non-neural systems.
- Motivation: Phrase-based and hybrid systems remained ahead of pure neural GEC systems on established benchmarks.The best pure neural systems were several percent behind.
- Motivation: Low-resource MT results show NMT quality starts low on small corpora but surpasses SMT at about 15 million words.With increasing data, NMT also beats SMT paired with a large in-domain language model.
- Motivation: GEC’s limited error-annotated resources motivate treating it as a low-resource or mid-resource machine translation problem.The paper argues that low-resource neural MT techniques should therefore transfer to neural GEC.
- Contribution: The paper adapts low-resource neural MT and SMT-based GEC methods to make neural GEC catch up to and outperform SMT-based systems.The approach includes a trustable setup and model-independent improvements.
- Results: More than 10% M2 improvement over the previous best neural GEC system was achieved on CoNLL-2014.The system also improved over comparable SMT and state-of-the-art hybrid systems, while achieving 5.9% GLEU improvement on JFLEG over the previous best pure neural system.
2 A trustable baseline for neural GEC
The paper defines a reproducible neural GEC baseline using public data, standardized preprocessing, and explicit evaluation practices. It also identifies optimizer instability and explains how domain adaptation, source noise, and ensembling affect results.
- Baseline and evaluation: The baseline combines SMT-based GEC insights with neural MT recommendations to support trustable neural GEC results.The experiments use public resources and report CoNLL M2 and, where appropriate, JFLEG GLEU.
- Baseline and evaluation: The experiments use NUCLE and Lang-8 NAIST while excluding non-public CLC to preserve comparability with prior systems.The authors strongly discourage training on CLC without contrastive results excluding it.
- Preprocessing: BPE subwords address NMT’s large-vocabulary problem, using 50,000 units trained on error-annotated data.The paper presents this as its first use of BPE subwords for GEC and reverses segmentation before evaluation.
- Optimizer instability: Neural GEC results vary by about 3 M2 points across differently initialized runs, making improvements below 2 or 3 points doubtful.The paper recommends reporting multiple independently trained models; JFLEG GLEU appears more stable than CoNLL M2.
- Ensembling: Ensembles bias predictions toward precision, producing fewer but better corrections and potentially lowering M2 when baseline models are weak.The authors argue this bias can help when combined with methods designed to increase recall.
3 Adaptations for GEC
The authors adapt low-resource MT and domain-adaptation techniques to make neural GEC more specific and effective, including source noise, error-rate adaptation, tied embeddings, and edit-weighted training.
- Source-side noise: Source-word dropout adds stronger corruption during training, encouraging the model to trust the source less and apply corrections more freely.The method drops a source word’s full embedding vector with probability p_src.
- Domain adaptation: Domain adaptation oversamples the in-domain NUCLE corpus ten times and removes correct sentences until its error rate reaches 15%.NUCLE has 6% erroneous tokens, whereas the CoNLL-2013 test set has about 15%.
- Tied embeddings: Tying source, target, and output embeddings shares information across the model and reduces its parameter count.In monolingual GEC, the source and target vocabularies are mostly equal, making embedding tying natural.
- Edit-weighted MLE: Edit-weighted MLE scales the loss for target tokens differing from aligned source tokens by a factor Λ, emphasizing corrective edits.The alignments are computed for each sentence pair with fast-align.
- Edit-weighted MLE: Λ = 3 works best among tested values on the CoNLL-2013 development set, while the +Edit-MLE system reaches 50.95 M2 on CoNLL.The weighting parameter affects the precision/recall trade-off and requires task-specific tuning.
- Results: 50.95 M2 surpasses a phrase-based SMT system with a large domain-adapted language model by 1% M2, making it the first neural system to beat that baseline.This result is reported for the +Edit-MLE system on the CoNLL benchmark.
4 Transfer learning for GEC
The paper transfers monolingual knowledge into neural GEC through pretrained embeddings and decoder parameters, with positive effects in both edit-weighted and standard training settings.
- Transfer learning: Transfer learning initializes low-resource neural models from parameters trained on high-resource or monolingual data.The paper applies this strategy to GEC using word embeddings and a GRU-based language model.
- Pre-training embeddings: Word2vec embeddings are inserted into the tied source, target, and output embedding layers, while remaining model parameters are initialized randomly.The adaptation is called +Pretrain-Emb.
- Pre-training the decoder: A GRU-based language model initializes the decoder’s corresponding parameters, including embedding layers, while remaining parameters are initialized randomly.The resulting configuration is called +Pretrain-Dec.
- Results: Pre-training has significantly positive effects with and without the edit-weighted MLE objective.These comparisons are summarized in Table 6 for the CoNLL benchmark.
- Results: 53.3% M2 on CoNLL-2014 matches the reported 53.14% M2 state-of-the-art result and exceeds the highest neural GEC system by 8% M2.The comparison is made against Chollampatt and Ng (2017) and Ji et al. (2017), respectively.
5 Ensembling with language models
The authors combine neural GEC models with an independently trained neural language model, reusing monolingual data without additional GEC training. All tested configurations benefit, though gains for the pretrained model are small.
- Language-model integration: The method reuses the neural language model trained for decoder pre-training, avoiding additional training.Separate language-model integration is attractive because monolingual data can be exploited without retraining the GEC model.
- Scoring: The correction score averages four translation-model probabilities with a weighted language-model probability and normalizes by sentence length.The language-model weight α is selected on development data by linear search over [0, 2] in steps of 0.1.
- Results: All tested configurations benefit from language-model ensembling, although gains for the pretrained model are rather small.Table 7 reports these results for three intermediate configurations.
6 Deeper NMT models
The paper tests whether its model-independent neural GEC techniques transfer to deeper and alternative NMT architectures. Deep RNN and Transformer models improve over the shallow model, with architecture-specific pre-training adaptations required.
- The section extends the model-independent methods to deeper or different NMT architectures.The investigated architectures are a deep RNN and a Transformer.
- Deep RNN: The deep RNN stacks four GRU blocks in the encoder and eight in the decoder within single-layer recurrent networks.
- Transformer: The Transformer uses six encoder and six decoder attention blocks with 512-dimensional embeddings and 2048-dimensional feed-forward filters.
- Training settings: Training the deep models uses synchronous SGD, warm-up for 16,000 iterations, inverse square-root decay, and averaging of the best eight checkpoints.Dropout is increased to 0.3 for the deep architectures, while source-word dropout remains unchanged.
- Pre-training: Pre-training must be modified for larger architectures to maximize the number of shared pre-trained parameters.Decoder-only language models retain Transformer decoder self-attention while removing encoder-dependent components.
- Results: Both deep models significantly improve over the shallow model, and the Transformer reaches the paper’s best CoNLL-2014 test-set result.Language-model ensembling is ineffective under CoNLL M2 but produces strong JFLEG GLEU improvements.
7 A standard tool set for neural GEC
The paper identifies a standard toolkit of methods for reliable, state-of-the-art neural GEC across CoNLL and JFLEG. Combining these methods raises pure neural performance beyond prior neural and phrase-based systems, while increased depth is architecture-dependent rather than model-independent.
- Standard tool set: Reliable neural GEC methods include source-embedding dropout, edit-weighted training, monolingual pre-training, model and language-model ensembling, domain/error adaptation, and increased depth.
- Standard tool set: Domain and error adaptation target a specific benchmark, while increasing model depth is explicitly not model-independent.
- CoNLL-2014 results: More than 10% M2 improvement over pure neural systems on CoNLL-2014 also surpasses the prior state-of-the-art hybrid phrase-based system by 2%.
- CoNLL-2014 results: A pure neural system outperforms the strong pure phrase-based SMT system when both are similarly adapted to GEC.
- JFLEG results: 5.9% GLEU improvement over the previous best pure neural system establishes the new state of the art on JFLEG.Without monolingual data, the reported improvement is 4.5% GLEU.