Source-linked AI summary
GECToR -- Grammatical Error Correction: Tag, Not Rewrite
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, Oleksandr Skurzhanskyi
TL;DR
GECToR addresses deployment limitations of sequence-generation GEC by recasting correction as sequence tagging with custom transformations and staged training. Its best taggers achieve state-of-the-art benchmark results while running up to 10 times faster than Transformer-based seq2seq systems.
Problem
Transformer-based seq2seq GEC systems have strong benchmark performance but suffer from slow inference and interpretability challenges that complicate real-world deployment.
Method
GECToR uses a pretrained Transformer encoder, custom token-level grammatical transformations, and three training stages from synthetic pretraining through errorful and mixed fine-tuning.
Results
F0.5 of 65.3/66.5 on CoNLL-2014 test and 72.4/73.6 on BEA-2019 test is achieved by the best single-model/ensemble tagger, with inference up to 10 times as fast as Transformer-based seq2seq systems.
Takeaways & Limitations
A faster, simpler, and more efficient GEC system can be developed using sequence tagging, pretrained Transformer encoders, custom transformations, and three-stage training.
Abstract
from arXiv · showhide
In this paper, we present a simple and efficient GEC sequence tagger using a Transformer encoder. Our system is pre-trained on synthetic data and then fine-tuned in two stages: first on errorful corpora, and second on a combination of errorful and error-free parallel corpora. We design custom token-level transformations to map input tokens to target corrections. Our best single-model/ensemble GEC tagger achieves an $F_{0.5}$ of 65.3/66.5 on CoNLL-2014 (test) and $F_{0.5}$ of 72.4/73.6 on BEA-2019 (test). Its inference speed is up to 10 times as fast as a Transformer-based seq2seq GEC system. The code and trained models are publicly available.
1 Introduction
GECToR reframes grammatical error correction as sequence tagging to address deployment issues in neural sequence-generation systems. It combines custom grammatical transformations, staged training, and pretrained Transformer encoders.
- Motivation: NMT-based GEC systems achieve strong benchmark performance but are inconvenient to deploy because of slow inference, demand, and limited interpretability.The passage lists slow inference speed, unspecified demand, and the need for additional functionality to explain corrections.
- Approach: GECToR simplifies GEC from sequence generation to sequence tagging with three training stages.The stages are synthetic-data pretraining, errorful-corpus fine-tuning, and joint errorful/error-free fine-tuning.
- Contributions: Custom g-transformations replace regular-token prediction and improve the generalization of the GEC sequence tagging system.These are token-level edits designed for grammatical error correction.
- Contributions: Fine-tuning is split into errorful-only training followed by training on a small, high-quality dataset containing both errorful and error-free sentences.This decomposition is presented as a distinct contribution of the system.
- Contributions: XLNet and RoBERTa encoders outperform ALBERT, BERT, and GPT-2 in the reported experiments.The comparison concerns pretrained Transformer encoders incorporated into the sequence tagging system.
2 Datasets
The system uses synthetic data for pretraining, learner corpora for fine-tuning, and established benchmark sets for evaluation.
- Training data: 9M parallel sentences with synthetically generated grammatical errors are used for pretraining stage I.This synthetic corpus supports the pretraining stage.
- Training data: Fine-tuning stages II and III use NUCLE, Lang-8, FCE, Cambridge Learner Corpus, and Write & Improve + LOCNESS.These are learner-English datasets used for the two fine-tuning stages.
- Evaluation data: Evaluation uses the CoNLL-2014 test set with the official M2 scorer and BEA-2019 development and test sets with ERRANT.The passage specifies the benchmark splits and evaluation tools.
3 Token-level transformations
GECToR maps each source token to a target subsequence through custom token-level transformations, enabling grammatical edits within a limited tag vocabulary.
- Transformation design: Custom transformations T(x_i) recover target text by applying edits to source tokens.They increase correction coverage for common errors including spelling, noun number, subject-verb agreement, and verb form.
- Transformation design: The tag vocabulary contains basic KEEP, DELETE, APPEND, and REPLACE transformations plus 29 token-independent g-transformations.The default vocabulary size is 5000, including 1167 APPEND and 3802 REPLACE transformations.
- Transformation design: G-transformations encode operations such as changing case, merging adjacent tokens, splitting tokens, and modifying noun or verb morphology.Examples include singular/plural conversion and changes in regular or irregular verb form, number, or tense.
- Preprocessing: Preprocessing aligns source tokens with target subsequences, finds fitting transformations, and retains one transformation per source token.The alignment selects subsequences by minimizing a modified Levenshtein distance that treats successful g-transformations as zero distance.
- Coverage: The default tag vocabulary size of 5000 is chosen as a heuristic compromise between grammatical-error coverage and model size.Table 2 compares coverage for basic transformations alone with coverage for all transformations across vocabulary sizes.
4 Tagging model architecture
The tagging model uses a pretrained BERT-like Transformer encoder and linear softmax layers to detect and tag errors at the token level.
- Architecture: The architecture stacks a pretrained BERT-like Transformer encoder with two linear layers and softmax outputs.The first linear layer performs error detection and the second performs error tagging.
- Inference: The system can iteratively retag modified sequences because some corrections depend on others.Most corrections are typically made during the first two iterations.
- Architecture: Token-level processing uses the first subword representation from each encoder tokenization scheme.RoBERTa uses BPE, BERT uses WordPiece, and XLNet uses SentencePiece.
5 Iterative sequence tagging approach
GECToR applies predicted token-level transformations to an input sentence and iteratively retags the modified sequence to address interdependent corrections.
- 5 Iterative sequence tagging approach: Predicted transformations T(x_i) are applied to each source token x_i to produce the modified sentence.The tagger predicts a tag-encoded transformation for every input token before sentence modification.
- 5 Iterative sequence tagging approach: Iterative retagging handles corrections that depend on earlier corrections by repeatedly transforming the newly modified sequence.Most corrections occur during the first two iterations, while successive iterations generally produce fewer corrections.
- 5 Iterative sequence tagging approach: Table 4 reports cumulative corrections and CoNLL-2014 test scores as the number of iterations increases.The table evaluates the best single model across iteration counts.
- 5 Iterative sequence tagging approach: Table 5 evaluates GECToR (XLNet) after each training stage and after inference adjustments.Its caption frames the comparison around training stages and inference tweaks.
6 Experiments
Experiments test the three-stage training procedure, pretrained encoders, inference adjustments, model combinations, and runtime on standard GEC benchmarks.
- Training: The system is pretrained on synthetic errorful sentences, fine-tuned on errorful-only sentences, then fine-tuned on mixed errorful and error-free sentences.The paper reports that using both fine-tuning stages is crucial for performance.
- Training: Two fine-tuning stages with and without error-free sentences are crucial for performance.The stages respectively use errorful-only data and a subset containing errorful and error-free sentences.
- Inference: Inference tuning adds a positive $KEEP bias and a sentence-level minimum error-probability threshold to increase precision by trading off recall.The hyperparameters were found by random search on BEA-dev.
- Results: 65.3 and 72.4 are the best single-model F0.5 scores on CoNLL-2014 test and BEA-2019 test, respectively.The best single model is GECToR (XLNet).
- Results: 66.5 and 73.6 are the best ensemble F0.5 scores on CoNLL-2014 test and BEA-2019 test, respectively.The ensemble averages output probabilities from BERT, RoBERTa, and XLNet single models.
- Speed: Inference is up to 10 times as fast as a state-of-the-art Transformer sequence-to-sequence system.The comparison uses NVIDIA Tesla V100 hardware and batch size 128; tagging is naturally parallelizable because corrections are not predicted one by one.
7 Conclusions
The paper concludes that sequence tagging with a pretrained Transformer encoder, custom transformations, and three-stage training yields a fast and effective GEC system.
- 7 Conclusions: The proposed system combines sequence tagging, a pretrained Transformer encoder, custom transformations, and three-stage training.The authors characterize the resulting GEC system as faster, simpler, and more efficient.
- 7 Conclusions: 65.3/66.5 and 72.4/73.6 are the single-model/ensemble F0.5 scores on CoNLL-2014 test and BEA-2019 test, respectively.The paper reports state-of-the-art GEC results with inference speed up to 10 times that of Transformer-based seq2seq systems.
A Appendix
The appendix provides the token-level transformation inventory and tables documenting training-stage performance and inference-tuning values.
- A Appendix: Table 9 lists token-level transformations using a core transformation and a transformation suffix.The table defines the notation used for transformation tags.
- A Appendix: Table 10 reports GECToR (RoBERTa) performance after each training stage and inference tweak, alongside the best single GECToR (XLNet) model.The XLNet results are referenced from Table 5.
- A Appendix: Table 11 lists inference-tweaking values found by random search on BEA-dev.These values correspond to the inference adjustments used in the experiments.