Source-linked AI summary
Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models
Felix Stahlberg, Shankar Kumar
TL;DR
Synthetic GEC data can lack diversity or produce simplistic errors. This paper conditions corruption on ERRANT error tags and matches synthetic tag frequencies to a target distribution, reporting benchmark gains and native-English adaptation that can surpass real mixed-proficiency training data.
Problem
Existing synthetic GEC methods can lack diversity or produce simplistic errors rather than the broad range of errors made by human writers.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Tag conditioning provides more precise control over synthetic errors than round-trip translation or tag-independent back-translation.
Source: §6 · p. 8Constrained decoding often fails in large-scale experiments when the selected tag is incompatible with the source sentence.
Source: §4.2 · p. 5
Abstract
from arXiv · showhide
Synthetic data generation is widely known to boost the accuracy of neural grammatical error correction (GEC) systems, but existing methods often lack diversity or are too simplistic to generate the broad range of grammatical errors made by human writers. In this work, we use error type tags from automatic annotation tools such as ERRANT to guide synthetic data generation. We compare several models that can produce an ungrammatical sentence given a clean sentence and an error type tag. We use these models to build a new, large synthetic pre-training data set with error tag frequency distributions matching a given development set. Our synthetic data set yields large and consistent gains, improving the state-of-the-art on the BEA-19 and CoNLL-14 test sets. We also show that our approach is particularly effective in adapting a GEC system, trained on mixed native and non-native English, to a native English test set, even surpassing real training data consisting of high-quality sentence pairs.
1 Introduction
The paper uses error tags to control synthetic GEC corruption, addressing simplistic outputs and matching synthetic error frequencies to a target domain. Its synthetic pre-training data improves benchmark results and can outperform high-quality real data when adapting to native English.
1 Introduction
- The corruption models generate an ungrammatical sentence from a clean one given an error-type tag, with inference-time FST constraints providing another way to force a tag.The approach uses tags such as SPELL and SVA to control the generated errors.Source: §1 · pp. 1–2
- Error tags steer corruption toward specific patterns found in GEC corpora, mitigating the simplistic errors often produced by untagged models.Many error types require more complex rewrites, and matching tag frequencies can target a specific domain.Source: §1 · p. 1
- Synthetic pre-training data reaches state-of-the-art results on BEA-test (74.9 F0.5) and CoNLL-14 (68.3 F0.5); native-English adaptation can surpass fine-tuning on high-quality real data.The synthetic data follows a desired tag distribution, such as the development-set distribution.Source: §1 · p. 2
2 Tagged Corruption Models
Tagged corruption models generate ungrammatical sentences from clean inputs conditioned on one of 25 ERRANT error tags. The section describes input-tag training and constrained Seq2Edits decoding, with examples illustrating the range of generated errors.
2 Tagged Corruption Models
- The core model generates an ungrammatical sentence from a clean input conditioned on one of the 25 ERRANT error-type tags.The tags are supported by the automatic annotation toolkit ERRANT.Source: §2 · p. 2
- One training option prepends an ERRANT tag to the clean sentence and trains a model to generate its ungrammatical counterpart.If a pair has multiple tags, it is duplicated for each unique tag, potentially helping the model learn co-occurring error categories.Source: §2 · p. 2
- Seq2Edits can instead be constrained during beam search with an FST that forces a chosen error tag without supplying tags in the input sequence.The described FST variants differ in which other tags they allow before or after the required tag.Source: §2 · pp. 2–3
- A tagged Seq2Edits model demonstrates a broad variety of realistic errors across the ERRANT tags.Table 1 shows outputs for the example clean sentence “There were a lot of sheep.”Source: Table 1 · p. 2, §2 · p. 3
3 Synthetic Data Generation with Tagged Corruption Models
The section compares three ways to assign error tags and generate synthetic sentence pairs, with the goal of matching a desired tag distribution across the training set.
Offline-Optimal
- Offline-Optimal frames tag assignment as constrained optimization, matching the observed tag distribution to the desired distribution with minimum-cost flow.The task is an instance of maximum weighted bipartite matching.Source: §3 · p. 3
Offline-Probabilistic
- Offline-Probabilistic draws tags from the desired distribution, then samples sentence pairs likely to contain each tag.Its derivation assumes equal probability across samples and equal tag likelihood given a source sentence.Source: §3 · p. 3
Online
- Online sampling does not guarantee that every original training sentence is included, though this may matter less with a large example pool.This is a contrast with Offline-Optimal.Source: §3 · p. 4
- The Online method draws each sentence’s tag from the desired distribution, generates its corrupted target, and runs in Θ(N).Unlike the offline methods, it avoids running the corruption model for every tag–sentence combination, which requires Ω(N|T|) runtime.Source: §3 · p. 4
4 Results
The experiments show that tagged synthetic corruption improves GEC performance, supports native-English adaptation, and scales to large pre-training data. The final system sets reported state-of-the-art scores on BEA-test and CoNLL-14.
4.1 Training Setup
- The setup evaluates BEA with span-based ERRANT F0.5, CoNLL with M2 F0.5, and JFLEG with GLEU; correction models use a two-stage fine-tuning recipe after pre-training.C4200M is generated by applying corruption methods to 200M C4 sentences; final correction models are fine-tuned first on Lang-8, then BEA+FCE.Source: §4 · p. 4, §4.1 · p. 4
4.2 Synthetic vs. Real Parallel Data
- Tagged corruptions mitigate simplistic outputs from untagged models by generating specific error patterns found in actual GEC corpora.The experiments use tags drawn from the BEA-dev distribution; the chosen unconstrained tagged Seq2Edits model was retained for later experiments as practical at scale.Source: §4.2 · p. 5
- 42.9 F0.5 on native-English N2 using native-speaker tags surpasses the 42.1 real-data score, while A/B/C tag distributions perform similarly across most test sets.The experiments estimate proficiency-dependent tag distributions from one BEA-dev split and evaluate on disjoint splits.Source: Table 5 · p. 5, §4.2 · p. 6
- 48.0 F0.5 from the best tagged Seq2Edits method exceeds untagged back-translation’s 42.4 and approaches the 50.4 real-parallel-data score.Offline-Optimal tends to underperform Offline-Probabilistic and Online in constrained Seq2Edits experiments; the authors hypothesize that non-optimal tag–sentence pairing increases error diversity and usefulness.Source: §4.2 · p. 5
- Native English has more PUNCT and SPELL errors, while DET errors are more common in non-native text.Figure 3 compares BEA-dev tag distributions for non-native CEFR levels A, B, and C with native English.Source: Figure 3 · p. 6, §4.2 · p. 6
4.3 The C4200M Synthetic Data Set
- Figure 4 uses C4200M in pre-training; the text reports final F0.5 improving to 52.1, while all three pre-training sets yield 32.4 after pre-training and 52.9 after second fine-tuning.With WikiEdits target sentences held constant, tagged corruption outperforms untagged corruption and round-trip translation.Source: Figure 4 · p. 6, §4.3 · p. 7, Table 7 · p. 7
- 74.9 F0.5 on BEA-test and 68.3 F0.5 on CoNLL-14 are state-of-the-art scores for the final ensemble.The gains use vanilla Transformer correction models pre-trained with C4200M, without changing the GEC architecture.Source: Table 8 · p. 8, §4.3 · p. 8
5 Related Work
Prior GEC work generates synthetic errors through heuristics, back-translation, and other methods. This paper builds on that landscape with error-tag-guided corruption that does not require changes to model architecture or training procedure.
5 Related Work
- Prior synthetic GEC data methods include random word- or character-level insertion, substitution, deletion, or shuffling, spell checkers, training-data edits, back-translation, decoding noise, sampling instead of beam search, and fluency-boost learning.Source: §5 · p. 8
- This work’s tagged corruption is inspired by Wan et al. (2020), while adding tags to the input is simpler than their latent-representation approach and requires no architecture or training-procedure changes.Tags for back-translation in machine translation were also explored by Caswell et al. (2019).Source: §5 · p. 8
6 Conclusion
The paper introduces tag-conditioned corruption models to generate diverse, realistic grammatical errors and uses them to create synthetic GEC data. Distribution-matched synthetic fine-tuning can outperform real mixed-proficiency data for native English, while C4200M pre-training yields state-of-the-art results on two standard test sets.
6 Conclusion
- Tag-conditioned corruption models generate a wide range of realistic grammatical errors, with error tags enabling more precise control than round-trip translation or tag-independent back-translation.The method generates errors from clean sentences conditioned on an error type tag.Source: §6 · p. 8
- Fine-tuning on synthetic data matching a native English error-tag distribution can outperform fine-tuning on genuine parallel data from mixed proficiency levels.Source: §6 · p. 8
- Pre-training vanilla Transformer GEC models on the 200M-sentence C4200M corpus yields state-of-the-art performance on BEA-test and CoNLL-14.Source: §6 · p. 8
A The Seq2Edits Model
Seq2Edits models sequence transduction as autoregressively predicted edit operations over source spans. Here, the model is used for corruption, with finite-state constraints forcing a specified error tag.
A The Seq2Edits Model
- Seq2Edits represents sequence transduction as edit operations applied to source spans, which are either copied or replaced.Source: A The Seq2Edits Model · p. 12
- Each edit is represented by an error tag, a source-span end position, and replacement tokens; the start position is implicit from the previous edit.Source: A The Seq2Edits Model · p. 12
- A modified Transformer with multiple target tapes predicts the edit tuples autoregressively; in this work, each edit represents an artificial error rather than a correction.Source: A The Seq2Edits Model · p. 12
- Finite-state transducers constrain the error-tag tape to force generation of a specific error type.Source: A The Seq2Edits Model · p. 12
B Example Corruptions
Explicit error tags diversify corruption-model outputs that otherwise tend to be dull and monotonous. The examples and distribution comparison illustrate that tagged corruption covers more complex error types than untagged corruption.
B Example Corruptions
- Explicit error tags control and diversify corruption-model outputs, addressing the dull, monotonous outputs of conventional untagged models.Source: B Example Corruptions · p. 12
- In the examples, the untagged model deletes a determiner, while the tagged model produces a wider variety of human-like writing errors.Source: B Example Corruptions · p. 12
- The comparison supports broader coverage of complex error types by tagged corruption, whereas untagged corruption mainly produces simplistic punctuation errors.Source: B Example Corruptions · p. 12
C Example Corrections
Example corrections illustrate how C4200M pre-training can produce more fluent outputs and handle meaning and long-range changes. The examples also show that radical rewrites may subtly alter semantics.
C Example Corrections
- The C4200M-pretrained system tends to produce more fluent corrections, which the authors associate with training on more English text from diverse sources.Source: C Example Corrections · p. 12, Table 10 · p. 15
- In one example, C4200M changes “stock up on traffic” to “stuck in traffic,” reflecting the model's learned preference for a semantically plausible phrase.Source: C Example Corrections · p. 12
- The system can make radical changes, including long-range reorderings, to improve fluency across domains, but these edits can subtly change meaning.Source: C Example Corrections · p. 12