Source-linked AI summary
A Simple Recipe for Multilingual Grammatical Error Correction
Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, Aliaksei Severyn
TL;DR
Multilingual GEC models must correct errors across languages while prior training setups can be difficult to replicate and build upon. The paper combines language-agnostic synthetic pre-training with large multilingual models, reports new state-of-the-art benchmark results in four languages, and releases cleaned LANG-8 data that simplifies training.
Problem
Prior best GEC models required multiple training stages, making them difficult to replicate and build upon.
Method
The paper adapts mT5 with a fully unsupervised, language-agnostic correction objective using automatically corrupted grammatical sentences, followed by supervised fine-tuning.
Results
The approach reports new state-of-the-art GEC benchmark results in the four studied languages.
Takeaways & Limitations
The released cleaned LANG-8 dataset enables competitive GEC training with a simplified setup and supports distillation into smaller, more efficient models.
Takeaways & Limitations
Fine-tuning and inference with the largest models require multi-GPU/TPU infrastructure.
Abstract
from arXiv · showhide
This paper presents a simple recipe to train state-of-the-art multilingual Grammatical Error Correction (GEC) models. We achieve this by first proposing a language-agnostic method to generate a large number of synthetic examples. The second ingredient is to use large-scale multilingual language models (up to 11B parameters). Once fine-tuned on language-specific supervised sets we surpass the previous state-of-the-art results on GEC benchmarks in four languages: English, Czech, German and Russian. Having established a new set of baselines for GEC, we make our results easily reproducible and accessible by releasing a cLang-8 dataset. It is produced by using our best model, which we call gT5, to clean the targets of a widely used yet noisy lang-8 dataset. cLang-8 greatly simplifies typical GEC training pipelines composed of multiple fine-tuning stages -- we demonstrate that performing a single fine-tuning step on cLang-8 with the off-the-shelf language models yields further accuracy improvements over an already top-performing gT5 model for English.
1 Introduction
GEC addresses grammatical errors in text, but multilingual progress is constrained by scarce, uneven, and noisy training data and complicated synthetic-data pipelines. The paper proposes language-agnostic pretraining with large multilingual models and releases a cleaned dataset to simplify training.
- GEC improves the grammaticality and readability of user-generated text, particularly benefiting non-native speakers, children, and people with language impairments.
- LANG-8 covers 80 languages but has highly skewed coverage, with Japanese and English exceeding one million sentence pairs while only ten languages exceed 10,000.Its uncontrolled collection also introduces unnecessary paraphrasing and erroneous or incomplete corrections.
- Synthetic GEC data methods often require language-specific tuning, complicating replication, development, and fair comparison across models.The paper notes that multiple fine-tuning stages further make previous best results difficult to reproduce and extend.
- The paper uses a language-agnostic pretraining objective with mT5 and synthetic corruptions, training one model across all 101 languages before supervised fine-tuning where data are available.The approach avoids language-specific priors during synthetic pretraining.
- The released CLANG-8 dataset cleans LANG-8 targets with gT5, enabling off-the-shelf T5 models fine-tuned only on CLANG-8 to outperform models trained on original LANG-8 data and simplify training.The paper also reports language-agnostic pretraining and model scaling as contributions to state-of-the-art GEC results.
2 Model
The model builds on multilingual T5 and adapts its span-prediction framework for GEC through sentence-level synthetic corruption. The corruption recipe is deliberately language-agnostic so it can be applied across all 101 languages.
- mT5 is a multilingual Transformer encoder-decoder pretrained on 101 languages, and this work uses its 600M-parameter base and 13B-parameter xxl variants.
- Because mT5 span prediction does not directly perform GEC and was trained on paragraphs, the paper splits paragraphs into sentences and creates corrupted sentence-target pairs.
- Synthetic corruption operations drop or swap token and character spans, insert characters, change word case, and alter the first character of a word.The operations are applied to grammatical sentences to mimic correction data.
- About 2% of examples remain uncorrupted so the model learns that grammatical inputs can be left unchanged.
- The recipe avoids sophisticated language-specific corruption methods because they are difficult to apply consistently across all 101 languages.Character insertions are drawn from the same passage to avoid inserting characters from another alphabet.
3 gT5: Large Multilingual GEC Model
The gT5 training regime combines multilingual GEC pretraining with supervised fine-tuning on language-specific datasets. The most computationally expensive sequential regime performs best, and the 11B model reaches new state-of-the-art results in all evaluated languages except English before further refinement.
- gT5 is fine-tuned on FCE and W&I for English, AKCES-GEC for Czech, Falko-MERLIN for German, and RULEC-GEC for Russian.
- Results: Table 1 reports the sizes of the datasets used to fine-tune gT5 across the evaluated languages.
- Training Regime: The sequential regime that first trains on GEC pretraining data until convergence and then fine-tunes on supervised data gives the best results despite being most computationally expensive.
- Results: Table 2 reports F0.5 scores, using M 2 for CoNLL-14, Russian, Czech, and German and ERRANT for the BEA test.Asterisks denote ensemble models.
- Results: The 11B xxl model produces new state-of-the-art results on all evaluated languages except English, while the base model is inferior to current state-of-the-art systems.The paper attributes the base model’s weaker performance to insufficient capacity for covering all 101 languages.
4 CLANG-8: Cleaned LANG-8 Corpus
CLANG-8 cleans noisy LANG-8 corrections using model-generated targets, enabling competitive GEC training with simpler pipelines. Experiments show gains across English, German, and Russian settings, though outcomes vary by language and dataset size.
- Corpus construction: CLANG-8 replaces noisy LANG-8 targets with corrections generated by a supervised gT5 xxl model.The original LANG-8 targets often contain unnecessary paraphrasing and erroneous or incomplete corrections.
- English results: CLANG-8 improves English F0.5 scores over LANG-8 on both CoNLL-14 and BEA test across model sizes.Larger models consistently outperform smaller siblings, including comparisons between xl and xxl.
- Error analysis: CLANG-8 improves all evaluated BEA error types except orthographic and punctuation errors.The comparison covers the top five error types reported in Table 5.
- Multilingual results: For German and Russian, CLANG-8 contains 114K and 45K examples; performance increases with model size, but Russian mT5 does not match state-of-the-art performance.The authors attribute part of the Russian gap to the smaller CLANG-8 dataset.
- Dataset release: The released CLANG-8 dataset contains 2.4M English, 114k German, and 45k Russian examples, while Czech is excluded because it would provide only 2k examples.The dataset is intended to support smaller, more practical models through distillation of gT5 xxl knowledge.
5 Conclusion
The paper reports new state-of-the-art GEC results in four studied languages using language-agnostic pretraining of large multilingual models. Releasing cleaned LANG-8 data enables easier and more accurate training of smaller models.
- Conclusion: The approach achieves new state-of-the-art results on GEC benchmarks in the four studied languages.The paper’s setup uses language-agnostic pretraining of large multilingual language models.
- Conclusion: The released cleaned LANG-8 dataset supports distillation into smaller, more efficient GEC models.It also enables easier and more accurate training of GEC models.