Source-linked AI summary
Corpora Generation for Grammatical Error Correction
Jared Lichtarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, Simon Tong
TL;DR
Sequence-to-sequence grammatical error correction is constrained by limited parallel training data. The paper generates GEC corpora from Wikipedia revision edits and round-trip translation, then uses iterative decoding; Lang-8 fine-tuning allows a single model to surpass the state of the art on CoNLL-2014 and JFLEG.
Problem
GEC has limited parallel data for sequence-to-sequence training, and Lang-8, the largest publicly available parallel corpus, contains only about 25M words.
Source: §1 · p. 1Method
Results
Fine-tuning models trained on the generated data on Lang-8 allows a single model to surpass the state of the art on CoNLL-2014 and JFLEG.
Source: §1 · p. 2Takeaways & Limitations
Wikipedia revisions can provide valuable GEC training data, though minimal filtering leaves noise and may make fine-tuning on clean in-domain data important for good performance.
Source: §9 · pp. 8–9
Abstract
from arXiv · showhide
Grammatical Error Correction (GEC) has been recently modeled using the sequence-to-sequence framework. However, unlike sequence transduction problems such as machine translation, GEC suffers from the lack of plentiful parallel data. We describe two approaches for generating large parallel datasets for GEC using publicly available Wikipedia data. The first method extracts source-target pairs from Wikipedia edit histories with minimal filtration heuristics, while the second method introduces noise into Wikipedia sentences via round-trip translation through bridge languages. Both strategies yield similar sized parallel corpora containing around 4B tokens. We employ an iterative decoding strategy that is tailored to the loosely supervised nature of our constructed corpora. We demonstrate that neural GEC models trained using either type of corpora give similar performance. Fine-tuning these models on the Lang-8 corpus and ensembling allows us to surpass the state of the art on both the CoNLL-2014 benchmark and the JFLEG task. We provide systematic analysis that compares the two approaches to data generation and highlights the effectiveness of ensembling.
1 Introduction
Limited parallel data constrains sequence-to-sequence GEC, so the paper develops two Wikipedia-based corpus-generation strategies. Fine-tuning models trained on the resulting data on Lang-8 allows a single model to surpass the state of the art on CoNLL-2014 and JFLEG.
1 Introduction
- Sequence-to-sequence GEC faces data scarcity: the largest publicly available parallel corpus, Lang-8, contains only about 25M words.The introduction motivates generating parallel data from publicly available Wikipedia revision histories.Source: §1 · p. 1
- Minimal filtering of Wikipedia revision edits produces a large, noisy corpus of about 4B tokens, containing real corrections alongside semantic changes, imperfect corrections, ignored errors, and spam.The authors describe this as their first strategy, in contrast to synthesizing errors in clean text.Source: §1 · p. 1
- The second strategy corrupts clean Wikipedia sentences through round-trip translation, producing relatively clean errors limited to those translation models tend to make.Both approaches draw on Wikipedia’s broad range of topics.Source: §1 · pp. 1–2
- Fine-tuning models trained on the generated data on Lang-8 yields additional improvements, allowing a single model to surpass the state of the art on CoNLL-2014 and JFLEG.The paper also compares training on combined data with ensembling models trained on separate sources.Source: §1 · p. 2
2 Data Generation from Wikipedia Revision Histories
The revision-history method forms source–target pairs from consecutive Wikipedia snapshots, then filters and samples them to create a large but noisy GEC corpus. It also introduces character-level spelling errors and reduces the share of unchanged examples.
2 Data Generation from Wikipedia Revision Histories
- Consecutive page snapshots form source–target pairs, but pages over 64Mb are discarded and revisions are downsampled to log1.5(n) pairs per page, reducing data 20-fold.This sampling limits the influence of large, frequently edited pages.Source: §2 · p. 2
- Aligned text is probabilistically cut into examples, with the older snapshot as source and the newer snapshot as target.Non-aligned sequences are skipped when making cuts.Source: §2 · p. 2
- The source receives randomly selected character deletions, insertions, replacements, or adjacent-character transpositions at a rate of 0.003 per character.This corruption is intended to help the model learn spelling as well as grammar correction.Source: §2 · p. 2
- Examples longer than 256 word-pieces are removed, and identity examples are downsampled by 99% to leave 3.8% unchanged pairs in the final dataset.Despite this processing, most extracted examples are identical before downsampling, and the shown edits are mostly not grammatical corrections.Source: §2 · p. 2
3 Data Generation from Round-trip Translations
The round-trip method uses unchanged Wikipedia sentences as targets and translates them through bridge languages to create noisy source sentences. Its errors are relatively clean but cover only part of the domain of real-world errors.
3 Data Generation from Round-trip Translations
- Unchanged Wikipedia sentences become targets, and round-trip translation through a bridge language produces their corrupted source sentences.The authors use French, German, Japanese, and Russian, chosen as high-resource languages relatively dissimilar from one another, with a competitive machine translation system.Source: §3 · pp. 2–3
- The corpus is augmented with 2.5% identity translations, character-level spelling corruptions, and common-error corruptions mined from short, close-edit-distance Wikipedia revisions.The mined edits exclude numbers and capitalization; their rules are probabilistically applied to translated text.Source: §3 · p. 3
- The generated corpus matches the Wikipedia Revision corpus in size, while unchanged Wikipedia targets are less likely to contain poor grammar, misspellings, or spam.Targets were left unchanged for at least one Wikipedia revision.Source: §3 · p. 3
- Round-trip errors are relatively clean but represent only a subset of real-world errors, whereas revision data likely covers real-world grammatical errors more broadly but contains substantial noise.Figure 2 shows examples from both corpora; Table 2 shows round-trip examples by bridge language.Source: §3 · p. 4
4 Iterative Decoding
Iterative decoding makes multiple incremental corrections by repeatedly searching for high-confidence rewrites. It improves performance over single-shot decoding for revision-trained models, but not for models trained on round-trip data; Lang-8-finetuned models improve with iteration.
4 Iterative Decoding
- Iterative decoding addresses inputs with multiple errors by accumulating individually high-confidence edits until no more edits are suggested.In the illustrated example, the model generally applies fewer edits as it approaches a final sentence requiring no edits.Source: §4 · p. 4
- Each iteration runs beam search and permits a non-identity rewrite only when its negative log probability cost is below the identity translation’s cost times a preset threshold.A stricter threshold can be used than in single-shot decoding because a low-confidence change may be selected in a later iteration.Source: §4 · p. 4
- Iterative decoding significantly improves over single-shot decoding for models trained on noisy Wikipedia revisions, but not for models trained on relatively clean round-trip data.All models fine-tuned on Lang-8 improve with iterative decoding.Source: §4 · p. 4
5 Model
The paper uses a Transformer sequence-to-sequence model for GEC, trained on Wikipedia-derived data and fine-tuned on Lang-8.
5 Model
- The model is a Transformer with six encoder and decoder layers, eight attention heads, and 1024-dimensional embeddings.Each layer has a position-wise feed-forward network of size 4096; training uses Adafactor and subword tokenization.Source: §5 · p. 4
- Wikipedia-corpus training runs for five epochs with batches of approximately 64,000 word pieces.The learning rate is 0.01 for the first 10,000 steps, then decays with the inverse square root of the step count.Source: §5 · pp. 4–5
- Models are fine-tuned on Lang-8 for 50 epochs at a constant learning rate of 3 × 10−5, stopping before development-set overfitting.Source: §5 · p. 5
6 Experiments
The experiments compare evaluation settings, Wikipedia revision and round-trip translation corpora, and ways of combining them. Fine-tuning and ensembling improve the reported systems, which outperform earlier systems on both benchmarks.
6.1 Evaluation
- They report CoNLL-2014 dev/test performance using M2 F0.5 and JFLEG dev/test performance using GLEU+; CoNLL decoding threshold and iterations are selected on a training-set subset, while JFLEG threshold uses dev data.JFLEG dev and test sets are spell-checked, unlike CoNLL sets. Ensembles decode four separately trained, fine-tuned identical Transformers using geometric averaging.Source: §6.1 · p. 5
6.2 Data from Wikipedia Revisions
- Wikipedia revision parameter variants produce modest performance variation, and their union yields no benefit; fine-tuning improves all models, which outperform Lang-8-only training.Source: §6.2 · p. 6, Table 6 · p. 6
6.3 Round Trip Translations
- Japanese bridge-language data performs best on CoNLL-2014, while pooled round-trip data performs best on JFLEG.The authors attribute the Japanese result to error-pattern similarity with CoNLL-2014 and report that pooling dilutes that similarity.Source: §6.3 · p. 6, Table 7 · p. 6
6.4 Combining Data Sources
- For combined Wikipedia edits and round-trip data, single-model CoNLL-2014 performance is unchanged, while the ensemble improves.Across each corpus type, the ensemble also outperforms a single model trained on all that corpus's data.Source: §6.4 · p. 6, Table 8 · p. 7
6.5 Comparison with Other systems
- The revisions-trained single system outperforms previous systems on CoNLL-2014 and JFLEG, and the ensemble improves on that single-system result.Source: §6.5 · p. 7
7 Error Analysis
Lang-8 fine-tuning makes Wikipedia-trained corrections more conservative, while the two training sources show different strengths across error categories. The analysis also reports benchmark results using additional data beyond the standard datasets.
7 Error Analysis
- Lang-8 fine-tuning makes Wikipedia-trained models more conservative, favoring grammatical corrections over out-of-scope editorial edits.The paper describes fine-tuning as adaptation from Wikipedia editing to GEC.Source: §7 · p. 7
- F0.5 61.3 and GLEU 62.4 were Tao et al. (2018b)'s results using non-public data, versus 62.8 and 65.0 with additional fine-tuning data.Source: §7 · p. 7
- The round-trip ensemble performs better on prepositions and pronouns, while the revisions ensemble is better on morphology and orthography.Their overall F0.5 scores are similar.Source: §7 · p. 7
8 Related Work
Prior GEC work includes phrase-based and neural systems, while this paper extends neural pretraining with large Wikipedia-derived parallel data. It also uses round-trip translation to generate training corpora rather than corrections from a translation lattice.
8 Related Work
- The approach extends Transformer GEC by pretraining on parallel data from Wikipedia revisions or round-trip translations, and follows work reporting gains from iterative decoding.The passage describes the public-data CoNLL-2014 state of the art as F0.5 56.25 from Grundkiewicz and Junczys-Dowmunt (2018).Source: §8 · p. 7
- Back-translation introduces noise using a clean-to-noisy translation model, which itself requires parallel data scarce in GEC; round-trip translation instead uses bridge translations.Source: §8 · p. 8
- Unlike Madnani et al. (2012) and D´esilets et al. (2009), this work uses round-trip translation to generate a large parallel training corpus for neural GEC.Source: §8 · p. 8
9 Discussion
The discussion presents Wikipedia-derived corpora as promising research avenues, while emphasizing multilingual potential, flexibility in choosing round-trip translation data, and the value of combining diverse sources through ensembles.
9 Discussion
- The paper presents two contrasting approaches for generating large parallel corpora from the same publicly available data source.Both approaches are described as promising avenues for further GEC research.Source: §9 · p. 8
- Revision-trained models can be valuable for GEC and may extend it to languages without existing GEC corpora; small amounts of clean, in-domain fine-tuning may be crucial for good performance.The paper reports experiments only with English-language GEC corpora for round-trip translation, not revision-based models.Source: §9 · pp. 8–9
- Table 11 reports F0.5 across error categories on the CoNLL-2014 test set.The caption identifies the table's scope but supplies no category-level values.Source: Table 11 · p. 9
- More filtering may reduce noise, while reported revision-trained models with and without fine-tuning provide baselines for evaluating smaller, cleaner revision datasets.The authors used minimal filtration to keep the dataset large and the process simple.Source: §9 · p. 9
- Round-trip translation was tested only for English, but any clean text source could support parallel GEC data, including domain-specific corpora; translation-quality effects remain to assess.The authors give patents as an example of a possible domain-specific source.Source: §9 · p. 9
- Pooling two diverse data sources used to train competitively performing models on the same task can degrade performance, motivating the search for useful dataset partitions that can be combined through ensembles.The authors suggest comparing partition-based ensembling with alternatives such as special tokens and meta-learning.Source: §9 · p. 9
- The authors released extraction scripts as a resource for GEC and other NLP tasks that use edit histories, including sentence rewriting and text simplification.The cited examples are sentence rewriting (Botha et al., 2018) and text simplification.Source: §9 · p. 9