Source-linked AI summary
Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction
Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, Kentaro Inui
TL;DR
The paper examines how to incorporate MLMs into encoder-decoder GEC models when GEC inputs differ from MLM pre-training data. It fine-tunes BERT on GEC-related data before using its outputs as additional GEC features, and reports state-of-the-art results on BEA-2019 and CoNLL-2014.
Problem
The paper addresses how encoder-decoder GEC models can benefit from MLMs when GEC inputs differ substantially from MLM pre-training corpora.
Method
The method fine-tunes BERT with GEC data or grammatical error detection and uses its output as additional features in the encoder-decoder GEC model.
Results
The fine-tuned feature-based approach is most effective in most evaluated GEC corpora, and the best model achieves state-of-the-art results on BEA-2019 and CoNLL-2014.
Takeaways & Limitations
Fine-tuning BERT for GEC-oriented information provides an effective way to use MLM representations in encoder-decoder GEC models.
Abstract
from arXiv · showhide
This paper investigates how to effectively incorporate a pre-trained masked language model (MLM), such as BERT, into an encoder-decoder (EncDec) model for grammatical error correction (GEC). The answer to this question is not as straightforward as one might expect because the previous common methods for incorporating a MLM into an EncDec model have potential drawbacks when applied to GEC. For example, the distribution of the inputs to a GEC model can be considerably different (erroneous, clumsy, etc.) from that of the corpora used for pre-training MLMs; however, this issue is not addressed in the previous methods. Our experiments show that our proposed method, where we first fine-tune a MLM with a given GEC corpus and then use the output of the fine-tuned MLM as additional features in the GEC model, maximizes the benefit of the MLM. The best-performing model achieves state-of-the-art performances on the BEA-2019 and CoNLL-2014 benchmarks. Our code is publicly available at: https://github.com/kanekomasahiro/bert-gec.
1 Introduction
The paper asks whether encoder-decoder GEC models can benefit from masked language models despite differences between GEC inputs and MLM pre-training data. It evaluates several BERT-integration strategies and proposes fine-tuning BERT on GEC data before using its output as additional features.
- GEC commonly uses encoder-decoder models, motivating investigation of whether MLMs such as BERT can improve this architecture.
- Initialization can damage pre-trained representations when sequence-to-sequence tasks require substantial task-specific fine-tuning.
- The study evaluates BERT initialization, direct BERT feature fusion, and a combined approach for encoder-decoder GEC models.
- The proposed approach fine-tunes BERT on GEC corpora and supplies its output as additional features to the GEC model.
- The fine-tuned feature-based method is most effective across most evaluated GEC corpora, and the best model reaches state-of-the-art results on BEA-2019 and CoNLL-2014.
2 Related Work
Prior work incorporates MLMs into GEC through re-ranking, filtering, or encoder-decoder integration, while this paper focuses on adapting MLM representations for encoder-decoder GEC.
- MLMs have improved GEC as re-rankers or filtering tools, and encoder-decoder GEC models can be combined with these pipeline methods.
- Existing work also uses MLM outputs as contextual embeddings and addresses mismatches between pre-training representations and downstream task distributions.
- The paper extends this mismatch perspective to the gap between grammatically correct raw corpora and GEC corpora.
3 Methods for Using Pre-trained MLM in GEC Model
The methods compare BERT initialization, feature fusion, and fusion after GEC-oriented fine-tuning. The proposed fine-tuning targets the mismatch between clean MLM pre-training data and erroneous GEC inputs.
- The paper studies three strategies: initializing the GEC model with BERT, adding BERT outputs as features, and adding outputs from GEC-fine-tuned BERT.
- BERT initialization: BERT initialization transfers pre-trained weights to a GEC encoder-decoder model but cannot initialize from parameters learned on pseudo-data.
- BERT-fuse: BERT-fuse encodes the input with BERT, then provides BERT representations alongside the input to the GEC encoder.
- BERT-fuse: Attention models combine GEC encoder states with BERT outputs, after which feedforward processing produces the encoder representations used by the decoder.
- BERT-fuse Mask and GED: Because BERT-fuse may not match erroneous GEC inputs, the paper further trains BERT on GEC data or fine-tunes it for grammatical error detection.
4 Experimental Setup
The experiments train and evaluate GEC and fine-tuned BERT models across multiple datasets and metrics, comparing BERT variants with non-BERT, pseudo-data, and ensemble settings.
- Train and Development Sets: GEC models use W&I-train, NUCLE, FCE-train, and Lang-8, with W&I-dev for development; BERT fine-tuning uses W&I-train, NUCLE, and FCE-train.
- Evaluation: Evaluation uses W&I-test, CoNLL-2014, FCE-test, and JFLEG with ERRANT, M2, and GLEU metrics, respectively.
- Training: Training uses development-set evaluation, learning-rate reduction after stalled loss, and stopping at a minimum learning rate or 30 epochs.
- BERT Training: The experiments use BERT-Base cased for BERT-fuse mask and GED training and evaluate the model on development data.
- Pseudo-data: Additional experiments combine BERT-fuse variants with a pseudo-data-pre-trained GEC model generated using probabilistically injected character errors.
- Re-ranking: The study also uses right-to-left re-ranking, rescoring n-best hypotheses with combined left-to-right and right-to-left generation probabilities.
5 Results
BERT consistently improves GEC accuracy, with fine-tuned feature-based variants outperforming initialization in nearly all cases. BERT-fuse GED achieves state-of-the-art single-model results on BEA-2019 and CoNLL-2014, while ensembling improves CoNLL14 by 0.2 points.
- BERT consistently improves the accuracy of the GEC model over the w/o BERT baseline.
- BERT-fuse, BERT-fuse mask, and BERT-fuse GED outperform BERT-init in almost all cases.
- BERT-fuse GED outperforms the other models in all cases except FCE-test in the second evaluation setting.
- BERT-fuse GED achieves state-of-the-art results with a single model on BEA2019 and CoNLL14.
- The ensemble model improves state-of-the-art results by 0.2 points on CoNLL14.
6 Analysis
The analysis examines fine-tuned BERT representations and error-type performance. Fine-tuning separates correct from incorrect word representations, while BERT-fuse GED improves performance across the evaluated error types.
- Hidden Representation Visualization: The analysis visualizes hidden representations from vanilla and GED-fine-tuned BERT using frequently mistaken words and matched correct cases.
- Hidden Representation Visualization: Vanilla BERT does not distinguish correct and incorrect clusters, whereas fine-tuned BERT places them on different sides.
- Hidden Representation Visualization: Fine-tuned BERT preserves information about correct word usage while incorporating grammatical-error information.
- Performance for Each Error Type: ERRANT is used to measure F0.5 for each error type.
- Performance for Each Error Type: BERT-fuse GED performs better than w/o BERT for all evaluated error types, including the five most frequent types in W&I-dev.
7 Conclusion
The paper concludes that BERT-fuse GED is among the most effective methods when BERT is fine-tuned on GEC corpora. Future work will address catastrophic forgetting to assess whether BERT-init can become effective.
- BERT-fuse GED is one of the most effective techniques when BERT is fine-tuned with GEC corpora.
- Future work will investigate methods for using BERT-init while addressing catastrophic forgetting.