Source-linked AI summary
An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction
Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, Kentaro Inui
TL;DR
GEC pseudo-data training lacks consensus about how pseudo data should be generated and incorporated. This study evaluates those configuration choices through extensive experiments and identifies effective settings, achieving state-of-the-art results on CoNLL-2014 and BEA-2019 without changing model architecture.
Problem
GEC research lacks consensus on pseudo-data generation, seed-corpus selection, and optimization settings despite pseudo data being important for training.
Method
The study conducts extensive experiments comparing pseudo-data generation methods, seed corpora, and JOINT versus PRETRAIN optimization settings.
Results
F0.5 = 65.0 on CoNLL-2014 and F0.5 = 70.2 on BEA-test are achieved with the best reported configuration, without modifying model architecture.
Takeaways & Limitations
Effective GEC configurations combine Gigaword-based pseudo data with JOINT or BACKTRANS (NOISY) PRETRAIN settings, depending on the training regime.
Abstract
from arXiv · showhide
The incorporation of pseudo data in the training of grammatical error correction models has been one of the main factors in improving the performance of such models. However, consensus is lacking on experimental configurations, namely, choosing how the pseudo data should be generated or used. In this study, these choices are investigated through extensive experiments, and state-of-the-art performance is achieved on the CoNLL-2014 test set ($F_{0.5}=65.0$) and the official test set of the BEA-2019 shared task ($F_{0.5}=70.2$) without making any modifications to the model architecture.
1 Introduction
GEC commonly adapts neural machine translation, but limited genuine parallel data motivates pseudo-data augmentation. The study examines unresolved choices for generating and incorporating pseudo data and reports strong benchmark performance.
- Motivation: GEC is widely formulated as machine translation, treating ungrammatical sentences as sources and grammatical sentences as targets.This formulation enables the use of neural MT architectures such as encoder-decoder models.
- Motivation: The largest publicly available GEC parallel corpus, Lang-8, contains only two million sentence pairs, while encoder-decoder models require substantial training data.
- Motivation: Pseudo-data augmentation has therefore been studied intensively to expand GEC training data.
- Research gap: Researchers lack consensus on pseudo-data generation, seed-corpus selection, and optimization settings.
- Contribution: The study investigates these choices through extensive experiments and achieves state-of-the-art performance without modifying model architecture.The model surpasses prior single-model results and most ensemble results, then improves further with task-specific techniques.
2 Problem Formulation and Notation
The paper formalizes GEC as learning conditional probabilities from ungrammatical-to-grammatical sentence pairs, while examining pseudo-data generation, seed-corpus choice, and optimization strategy. It compares JOINT training with pseudo-data pretraining followed by fine-tuning.
- Task definition: GEC training data D consists of pairs of ungrammatical source sentences X and grammatical target sentences Y.
- Objective: The model optimizes parameters Θ by minimizing a conditional negative-log-likelihood objective over the training data.The conditional probability p(Y|X, Θ) represents the probability of grammatical target Y given source X.
- Pseudo data: Pseudo data Dp are generated from grammatical sentences in a seed corpus T and incorporated alongside genuine parallel data Dg.
- Experimental aspects: The study compares pseudo-data generation methods, seed corpora, and optimization settings because their effects on GEC performance lack established consensus.
- Optimization settings: JOINT concatenates genuine and pseudo data, whereas PRETRAIN first optimizes on pseudo data before fine-tuning on genuine data.
3 Methods for Generating Pseudo Data
The paper compares reverse-model methods that generate ungrammatical sentences with DIRECTNOISE, which corrupts grammatical sentences directly. Its variants differ in how reverse-model outputs are produced or how noise is injected.
- Comparison: The experiments compare these three pseudo-data generation methods under a common GEC evaluation setup.
- Backtranslation: Backtranslation trains a reverse model to generate ungrammatical sentences from grammatical inputs, then pairs each output with its input as pseudo data.
- Backtranslation: BACKTRANS (NOISY) adds scaled random noise to beam-hypothesis scores at each decoding step.The noise is uniformly sampled, and β_random controls its scale; setting β_random = 0 recovers standard backtranslation.
- Backtranslation: BACKTRANS (SAMPLE) generates ungrammatical sentences by sampling from the reverse model’s distribution during decoding.
- Direct noise: DIRECTNOISE injects noise directly into grammatical sentences rather than using a reverse model.For each token, it probabilistically selects operations such as masking with a placeholder token.
4 Experiments
The experiments compare pseudo-data generation methods, seed corpora, and optimization settings using an off-the-shelf EncDec model across several GEC datasets. They identify effective configurations and report state-of-the-art benchmark performance after adding shared-task techniques.
- Experimental setup: The experiments use an off-the-shelf EncDec model without task-specific architecture, tuning and evaluating each method or setting on validation data.Datasets include BEA-valid, BEA-test, CoNLL-2014, and JFLEG; reported scores generally average five trials with different random seeds.
- Pseudo-data generation: BACKTRANS (NOISY) and BACKTRANS (SAMPLE) achieve competitive F0.5, while DIRECTNOISE achieves the best F0.5 among the compared generation methods.The study retains BACKTRANS (NOISY) and DIRECTNOISE; BACKTRANS (NOISY) also decodes ungrammatical sentences 1.2 times faster than BACKTRANS (SAMPLE).
- Seed corpus: Gigaword consistently outperforms Wikipedia and SimpleWiki, although seed-corpus differences in F0.5 are small when |Dp| = 1.4M.DIRECTNOISE with Gigaword achieves the best F0.5 among all configurations.
- Optimization setting: Effective configurations combine JOINT with Gigaword, avoid excessively large pseudo-data quantities in JOINT, or use PRETRAIN with large BACKTRANS (NOISY) pseudo data.The authors compare JOINT and PRETRAIN across pseudo-data scales, using Wikipedia for the optimization-setting experiments.
- Optimization setting: 70M BACKTRANS (NOISY) pseudo data followed by BEA-train fine-tuning is the best available approach after combining PRETRAIN with relatively small DIRECTNOISE data fails to improve BEA-valid performance.This configuration is referred to as PRETLARGE.
- Comparison with top models: PRETLARGE+SSE+R2L achieves F0.5 = 65.0 on CoNLL-2014 and F0.5 = 69.8 on BEA-test, while adding SED raises BEA-test to F0.5 = 70.2.SED decreases performance on CoNLL-2014 and JFLEG, suggesting sensitivity to the BEA validation domain used for fine-tuning.
5 Conclusion
The study identifies effective pseudo-data configurations for grammatical error correction and achieves state-of-the-art performance on two benchmark test sets.
- Gigaword as the seed corpus and BACKTRANS (NOISY) pretraining were found effective for incorporating pseudo data.
- The proposed settings achieved state-of-the-art performance on the CoNLL-2014 and BEA-2019 test sets.
- DIRECTNOISE algorithm: DIRECTNOISE generates pseudo data by applying token-level keep, mask, deletion, or insertion operations to grammatical sentences.
B BEA-2019 Workshop Official Dataset
The BEA-2019 Workshop official dataset comprises four learner-English corpora and is publicly available.
- The official dataset contains FCE, Lang-8, NUCLE, and W&I+LOCNESS corpora.
- The BEA-2019 dataset is publicly available through the workshop website.
C Data Preparation Process
The BEA-train data are tokenized, filtered, and segmented into subwords using a specified preprocessing pipeline.
- BEA-train is tokenized with spaCy, and identical source-target sentence pairs are removed.
- Byte-pair encoding is applied to both source and target text with 8,000 merge operations.
D Hyper-parameter Settings
The paper reports separate hyper-parameter tables for JOINT and PRETRAIN optimization.
- Table 6 reports hyper-parameters for JOINT optimization.
- Table 7 reports hyper-parameters for PRETRAIN optimization.
E Mask Probability of DIRECTNOISE
The study varies DIRECTNOISE’s masking probability while fixing the other operation probabilities, finding slight improvement as µmask increases within the tested range.
- µkeep = 0.2 was fixed, while µinsertion and µdeletion were set to (1 − µkeep − µmask)/2.
- Increasing µmask from 0.1 to 0.5 slightly improved performance on BEA-valid.
- µmask = 0.5 was selected for the subsequent experiment.
F Noise Strength of BACKTRANS (NOISY)
The study evaluates BACKTRANS (NOISY) across noise strengths and finds that noise is necessary for useful pseudo data, with βrandom = 6 selected as the best tested value.
- βrandom = 0 produced worse performance than the baseline on BEA-valid.The authors attribute this to a reverse model that becomes too conservative without noise, generating insufficient grammatical errors for useful teaching signal.
- βrandom = 6 was the best value for BACKTRANS (NOISY).
G Examples of Noisy Sentences
The paper illustrates noisy sentences produced by BACKTRANS (NOISY) and DIRECTNOISE, including how DIRECTNOISE outputs change as µmask varies.
- Figure 4 presents examples of noisy sentences generated by BACKTRANS (NOISY) and DIRECTNOISE.
- The examples contrast original grammatical sentences with outputs from both pseudo-data generation methods.
- Figure 5 shows DIRECTNOISE examples under different µmask values, with N/A denoting the original text.
- PRETRAIN uses pseudo data before fine-tuning on genuine parallel data, and Figure 6 reports models using pretraining only.