PaperlayerView exact revision on arXiv

Source-linked AI summary

Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study

Tao Ge, Furu Wei, Ming Zhou

arXiv:1807.01270v5cs.CLcs.AI

TL;DR

Neural GEC models are limited by sparse corrected training pairs and imperfect single-round correction. This paper introduces fluency boost learning and inference, and its convolutional seq2seq system reaches human-level performance on both CoNLL-2014 and JFLEG.

  • Problem

    Conventional neural GEC models learn from insufficient corrected sentence pairs, limiting their generalization.

  • Method

    The approach generates diverse fluency boost sentence pairs during training and incrementally improves sentences through multi-round inference.

  • Results

    The combined convolutional seq2seq system reaches human-level performance on both benchmarks, achieving 75.72 F0.5 on CoNLL-2014 10 annotation and 62.42 GLEU on JFLEG.

  • Takeaways & Limitations

    Fluency boosting improves the base seq2seq system and supports state-of-the-art GEC performance across CoNLL-2014 and JFLEG.

  • Takeaways & Limitations

    The report does not provide a detailed comparison and analysis of different fluency boost learning and inference methods.

Abstract

from arXiv · show

Neural sequence-to-sequence (seq2seq) approaches have proven to be successful in grammatical error correction (GEC). Based on the seq2seq framework, we propose a novel fluency boost learning and inference mechanism. Fluency boosting learning generates diverse error-corrected sentence pairs during training, enabling the error correction model to learn how to improve a sentence's fluency from more instances, while fluency boosting inference allows the model to correct a sentence incrementally with multiple inference steps. Combining fluency boost learning and inference with convolutional seq2seq models, our approach achieves the state-of-the-art performance: 75.72 (F_{0.5}) on CoNLL-2014 10 annotation dataset and 62.42 (GLEU) on JFLEG test set respectively, becoming the first GEC system that reaches human-level performance (72.58 for CoNLL and 62.37 for JFLEG) on both of the benchmarks.

1 INTRODUCTION

The introduction identifies limited training coverage and single-round inference as weaknesses of neural GEC, then proposes fluency boosting to expand learning examples and iteratively improve corrections.

  • 1 INTRODUCTION: Conventional seq2seq GEC models train on limited corrected pairs, so slight input changes can expose generalization failures.The introduction also motivates iterative inference because single-round decoding may not perfectly correct a sentence.
  • 1 INTRODUCTION: Fluency boost learning creates additional error-corrected sentence pairs from less fluent model outputs, expanding training instances and improving generalization.Generated pairs are added during later training epochs when the target sentence is more fluent than the source.
  • 1 INTRODUCTION: Fluency boost inference incrementally corrects sentences through multiple inference rounds while proposed edits continue improving fluency.Multi-round correction can clarify context for remaining grammatical errors.
  • 1 INTRODUCTION: 75.72 F0.5 on CoNLL-2014 10 annotation and 62.42 GLEU on JFLEG made the combined system the first to reach human-level performance on both benchmarks.The approach combines fluency boosting with convolutional seq2seq models.

2 BACKGROUND: NEURAL GRAMMATICAL ERROR CORRECTION

Neural GEC typically uses an attention-based encoder-decoder to map raw sentences to grammatically correct sentences through learned probabilistic sequence mappings.

  • 2 BACKGROUND: NEURAL GRAMMATICAL ERROR CORRECTION: An attention-based encoder-decoder seq2seq model edits a raw sentence into its grammatically correct counterpart.The background relates this setup to neural machine translation.
  • 2 BACKGROUND: NEURAL GRAMMATICAL ERROR CORRECTION: The error-correction model learns P(xc|xr) from error-corrected sentence pairs using maximum likelihood estimation.Model parameters Θcrt are optimized to maximize the likelihood of corrected sentences given raw inputs.
  • 2 BACKGROUND: NEURAL GRAMMATICAL ERROR CORRECTION: During decoding, the model selects an output sequence through beam search to maximize its sequence probability.The supplied notation identifies the selected output as xo.

3 FLUENCY BOOST LEARNING

Fluency boost learning expands GEC training beyond scarce original correction pairs by generating meaning-preserving, less-fluent sources paired with fluent targets. The paper develops back-, self-, and dual-boost strategies, with filtering and data-size controls, and extends them to native text.

  • 3 FLUENCY BOOST LEARNING: Fluency boost learning addresses limited correction-pair availability by generating additional pairs whose targets improve source fluency without changing meaning.This exploits a GEC-specific fluency boost condition rather than relying only on original error-corrected pairs.
  • 3.1 BACK-BOOST LEARNING: Back-boost trains an error-generation model, samples n-best outputs for correct sentences, and retains sufficiently less-fluent candidates as training sources.The correction model then learns from both original pairs and sampled fluency boost pairs.
  • 3.1 BACK-BOOST LEARNING: A fluency threshold filters unnecessary edits, while the number of generated pairs is capped so they do not overwhelm original correction pairs.The threshold is set slightly above 1.0, and the sampled boost-pair set is constrained not to exceed the original-pair set.
  • 3.2 SELF-BOOST LEARNING: Self-boost reuses the correction model’s own non-gold, less-fluent predictions as disfluency candidates and expands those candidates across training epochs.Unlike back-boost, it generates candidates by editing raw sentences rather than correct sentences.
  • 3.3 DUAL-BOOST LEARNING: Dual-boost combines back- and self-boost perspectives to generate more diverse disfluency candidates for correction-model training.Its motivation is that greater candidate diversity can provide more helpful training instances.
  • 3 FLUENCY BOOST LEARNING: The learning strategies can also incorporate massive native text data, extending their use beyond the original error-corrected corpus.The paper states that native data has proved useful for GEC.

4 FLUENCY BOOST INFERENCE

Fluency boost inference repeatedly edits a sentence while each proposed output improves fluency, addressing cases where one inference pass cannot fully correct multiple errors. Round-way correction alternates right-to-left and left-to-right models so their error-specific strengths complement each other.

  • 4.1 MULTI-ROUND ERROR CORRECTION: Multi-round inference repeatedly feeds each hypothesis back into the correction model until the next output no longer improves fluency.This is enabled by GEC’s same-language source and target, allowing sentences to be edited more than once.
  • 4.1 MULTI-ROUND ERROR CORRECTION: Right-to-left decoding is better suited to some article errors, whereas left-to-right decoding is more likely to handle subject–verb agreement, so the two directions complement one another.The figure presents this division of error-type strengths as the basis for correcting more errors than an individual model.
  • 4.1 MULTI-ROUND ERROR CORRECTION: Round-way correction successively applies right-to-left and left-to-right seq2seq models instead of progressively using only one model.The approach is an advanced form of fluency boost inference.

5 EXPERIMENTS

Experiments evaluate the fluency-boost approach using established learner and native-English data, standard CoNLL and JFLEG benchmarks, and comparisons with prior GEC systems. The combined system achieves strong benchmark performance, including human-level results on both datasets, while inference improves fluency-related performance at a precision cost.

  • 5.1 DATASET AND EVALUATION: The experiments use public learner corpora, 2,865,639 additional non-public Lang-8 pairs, 61,677,453 Wikipedia sentences, and CoNLL-2014 and JFLEG evaluation sets.CoNLL evaluation uses F0.5, while JFLEG evaluation uses GLEU; spelling errors are preprocessed with a public spell checker.
  • 5.3 EXPERIMENTAL RESULTS: 75.72 F0.5 on CoNLL-10 and 62.42 GLEU on JFLEG make the combined system the first to reach human-level performance on both benchmarks.The comparison includes human performance on CoNLL-10 and JFLEG.
  • 5.3 EXPERIMENTAL RESULTS: 61.34 on CoNLL-2014, 76.88 F0.5 on CoNLL-10, and 61.41 GLEU on JFLEG are achieved after adding fluency boost learning to the base convolutional seq2seq model.Adding fluency boost inference then raises JFLEG to 62.42 GLEU, although CoNLL F0.5 scores drop.
  • 5.3 EXPERIMENTAL RESULTS: Fluency boost inference raises recall from 36.30 to 40.18 on CoNLL-2014 and from 50.31 to 53.15 on CoNLL-10, while precision falls from 74.12 to 68.45 and from 88.56 to 84.71.Because F0.5 weights precision twice as heavily as recall, these precision losses reduce CoNLL F0.5.
  • 5.3 EXPERIMENTAL RESULTS: Fluency boost learning improves precision, recall, F0.5, and GLEU over the base model, and improves recall for most CoNLL-2014 error types.The error-type analysis supports the use of diverse generated errors during training.
  • 5.3 EXPERIMENTAL RESULTS: Round-way correction combines left-to-right and right-to-left models because each direction performs better on different error types, improving recall for most types.The right-to-left model is stronger on ArtOrDet, whereas the left-to-right model is stronger on SVA.

6 RELATED WORK

Related GEC work includes classifier-based and machine-translation-based systems, followed by growing use of neural seq2seq models. The paper focuses on fluency-boost learning and inference rather than surveying grammatical error detection or evaluation in detail.

  • 6 RELATED WORK: Advanced GEC systems have traditionally been classifier-based or machine-translation-based, while recent work increasingly uses seq2seq models.The cited studies span both earlier approaches and newer neural GEC systems.
  • 6 RELATED WORK: The paper proposes dynamic fluency-boost data augmentation alongside training, unlike models trained only with original error-corrected data.This positions the contribution within the shift toward seq2seq GEC.
  • 6 RELATED WORK: Research on grammatical error detection and GEC evaluation is acknowledged but not introduced in detail because it is not closely related to this work’s contributions.The paper distinguishes these areas from its focus on GEC modeling.

7 CONCLUSION

The system combines fluency boost learning and inference with a convolutional seq2seq GEC model. This mechanism achieves state-of-the-art results and human-level performance on both CoNLL-2014 and JFLEG.

  • The convolutional seq2seq GEC system reaches state-of-the-art results and human-level performance on both CoNLL-2014 and JFLEG.Its fluency boost learning and inference mechanism underpins the reported performance.
  • Fluency boost learning generates diverse error-corrected sentence pairs during training by exploiting error-corrected and native data.These additional pairs improve performance over the base seq2seq model.
  • Fluency boost inference progressively improves sentence fluency through round-way correction.The approach uses iterative correction as part of inference.
Loading 1807.01270v5…