Source-linked AI summary
JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction
Courtney Napoles, Keisuke Sakaguchi, Joel Tetreault
TL;DR
GEC research lacks a broad, fluency-oriented gold standard beyond a single minimal-edit learner corpus. The paper introduces JFLEG, collects four holistic corrections for 1,511 learner sentences, and evaluates four leading systems with fluency-sensitive and minimal-edit metrics. JFLEG reveals differing system rankings and shows that current systems handle some closed-class errors while missing spelling and long-range context-dependent errors.
Problem
GEC evaluation has relied heavily on one learner population and minimal edits, which do not necessarily produce fluent or native-sounding text.
Method
JFLEG adds four human-written holistic fluency corrections to 1,511 GUG sentences and evaluates four leading systems using GLEU and M2.
Results
System rankings vary considerably between fluency and minimal-edit evaluation; neural systems make more changes, while systems often leave errors unaddressed.
Takeaways & Limitations
JFLEG provides a gold standard for assessing both grammaticality and fluency, while exposing strengths and gaps in current GEC systems.
Takeaways & Limitations
With only four systems, the evaluation yields no conclusive recommendation about the best metric for fluency corrections.
Abstract
from arXiv · showhide
We present a new parallel corpus, JHU FLuency-Extended GUG corpus (JFLEG) for developing and evaluating grammatical error correction (GEC). Unlike other corpora, it represents a broad range of language proficiency levels and uses holistic fluency edits to not only correct grammatical errors but also make the original text more native sounding. We describe the types of corrections made and benchmark four leading GEC systems on this corpus, identifying specific areas in which they do well and how they can improve. JFLEG fulfills the need for a new gold standard to properly assess the current state of GEC.
1 Introduction
GEC progress is constrained by reliance on a single minimal-edit corpus representing one learner population. JFLEG addresses this gap with holistic corrections intended to make sentences grammatical and natural-sounding.
- Progress is limited because GEC systems have largely been developed and evaluated on the CoNLL–2014 dataset.
- The CoNLL–2014 corpus represents one learner population and contains minimal edits that may not make sentences fluent or native-sounding.
- Table 1 contrasts a grammatical but awkward minimal-edit correction with a more extensive fluency correction.
- JFLEG adds holistic fluency corrections to learner sentences and benchmarks four leading GEC systems against this broader standard.
2 GEC corpora
Existing GEC corpora differ in source populations, annotation practices, and whether they support minimal or fluency edits. JFLEG provides holistic rewrites with reliable annotations and sentence-level grammaticality scores.
- Existing corpora span learner essays, internet text, and technical writing, with substantial differences in proficiency and annotation procedures.
- NUCLE and FCE use constrained minimal edits, whereas Lang-8 and AESW allow fluency edits without error-code restrictions.
- JFLEG contains four holistic corrections per sentence and does not restrict edits to minimal error spans or coded errors.
- JFLEG offers more reliable corrections than Lang-8 through perfect alignments and screened editors, and more extensive corrections than AESW.
- JFLEG is the only corpus described here that pairs corrections with sentence-level grammaticality scores for uncorrected text.
3 The JFLEG corpus
JFLEG extends the GUG learner corpus with four human fluency corrections for each of 1,511 sentences. Its annotations show that substantive rewrites are common, though many awkward sentences remain insufficiently revised.
- JFLEG contains 1,511 GUG development and test sentences, each corrected four times by qualified U.S. participants.
- The corpus withholds 747 sentences as a blind test set for community evaluation.
- Mean Levenshtein distance between originals and corrections exceeded that of existing corpora by more than twofold.
- 81% of sampled originals contained at least one error, and 68% of those received error-free annotations.
- 69% of annotations contained minimal edits, 25% contained a fluency edit, and 17% contained both.
- Fewer than half of awkward sentences received fluency edits, potentially explaining why many corrected sentences remained awkward.
4 Evaluation
The evaluation compares four unmodified MT-based GEC systems on JFLEG using GLEU, M2, and human rankings, finding that fluency-oriented evaluation changes relative system performance. Neural systems perform best on fluency measures, while metric rankings do not support a conclusive choice of evaluation metric.
- Systems and metrics: Four leading GEC systems, all using some form of machine translation, were evaluated without modifications on JFLEG.The systems were CAMB14, CAMB16, AMU, and NUS.
- Systems and metrics: GLEU evaluates n-gram overlap while penalizing unchanged n-grams that human corrections modify, whereas M2 scores aligned word- and phrase-level edits with F0.5.Human rankings were additionally collected because both metrics had previously been evaluated only on CoNLL-2014.
- System comparisons: The two highest human-ranked systems were also the two highest GLEU systems, but GLEU reversed the order of the bottom two and M2 reversed both pairs.Human rankings covered six versions of each of 150 JFLEG sentences and were inferred with TrueSkill.
- System comparisons: 49.5 was AMU's highest M2 score, while CAMB16 ranked third with 39.9 despite being best on the fluency corpus.CAMB16 and NUS were substantially better than the other two systems according to both metrics, and neural systems achieved approximately 80% of human performance according to M2.
- Evaluation limitations: No conclusive recommendation about the best metric for fluency corrections can be drawn because only four systems were compared and several rankings lacked significant differences.There was no significant difference between the metric rankings, and human rankings showed no significant difference among three systems.
- Edit-distance analysis: Neural systems achieved the highest GLEU scores for nearly all edit distances and generated the most sentences with larger edit distances.CAMB14 had the most consistent GLEU scores, while AMU's scores above LD 6 were erratic because it produced few such outputs.
5 Qualitative analysis
The qualitative analysis shows that current GEC systems make fewer and narrower corrections than humans, especially for fluency, spelling, and context-dependent errors.
- Humans make significantly more fluency and minimal edits than any evaluated system.
- Neural systems make more changes and produce fewer erroneous sentences, but systems often correct only one or two errors while leaving others.
- CAMB16 changes the meaning of 15 sentences, showing a fluency-related trade-off despite strong output quality by other measures.
- In context-heavy examples, systems fail to resolve long-range meaning, and CAMB16’s attempted correction can remain nonsensical.
6 Conclusions
The paper presents JFLEG as a corpus for evaluating grammaticality and fluency, then benchmarks four systems to identify their strengths and remaining errors. Results vary considerably between fluency and minimal-edit evaluation, while current systems still trail human proofreading.
- JFLEG is a new corpus for developing and evaluating GEC systems with respect to fluency as well as grammaticality.
- System performance varies considerably between fluency-corpus and minimal-edit-corpus evaluation, supporting the need for a new evaluation dataset.
- Current systems handle some closed-class targets but ignore many spelling mistakes and long-range context-dependent errors.
- Neural methods make more fluency edits than other systems, potentially at the expense of preserving input meaning.
- The benchmark indicates substantial remaining distance from human proofreader performance and identifies specific directions for future improvement.