PaperlayerView exact revision on arXiv

Source-linked AI summary

JELV: A Judge of Edit-Level Validity for Evaluation and Automated Reference Expansion in Grammatical Error Correction

Yuhao Zhan, Yuqing Zhang, Jing Yuan, Qixiang Ma, Zhiqi Yang, Yu Gu, Zemin Liu, Fei Wu

arXiv:2511.21700v2cs.CL

TL;DR

Limited reference diversity leaves valid GEC corrections undercounted in evaluation and narrows correction patterns represented in training. The paper introduces JELV, benchmarks it on PEVData, and applies it to evaluation and reference expansion. The authors report state-of-the-art human-judgment correlation and performance gains from retraining on expanded references.

  • Problem

    GEC datasets often have only one or two references, limiting coverage of valid corrections in evaluation and training.

    Source: §1 · p. 1

  • Method

    JELV assesses edits against grammaticality, meaning preservation, and fluency criteria, using PEVData and supporting evaluation and reference expansion.

    Source: §1 · p. 2

  • Results

    JELV’s evaluation metric achieves state-of-the-art correlation with human judgments, and retraining on its expanded corpus yields clear GEC performance improvements.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    JELV offers implicit and explicit reference-expansion approaches to address reference scarcity in evaluation and GEC training.

    Source: §1 · p. 2

    About 10% inconsistency is attributed mainly to subtle fluency edits that can require expert discussion to resolve.

    Source: JELV2.0: Classifier Model Distillation · p. 4

Abstract

from arXiv · show

Existing Grammatical Error Correction (GEC) systems suffer from limited reference diversity, leading to underestimated evaluation and restricted model generalization. To address this issue, we introduce the Judge of Edit-Level Validity (JELV), an automated framework to validate correction edits from grammaticality, faithfulness, and fluency. Using our proposed human-annotated Pair-wise Edit-level Validity Dataset (PEVData) as benchmark, JELV offers two implementations: a multi-turn LLM-as-Judges pipeline achieving 90% agreement with human annotators, and a distilled DeBERTa classifier with 85% precision on valid edits. We then apply JELV to reclassify misjudged false positives in evaluation and derive a comprehensive evaluation metric by integrating false positive decoupling and fluency scoring, resulting in state-of-the-art correlation with human judgments. We also apply JELV to filter LLM-generated correction candidates, expanding the BEA19's single-reference dataset containing 38,692 source sentences. Retraining top GEC systems on this expanded dataset yields measurable performance gains. JELV provides a scalable solution for enhancing reference diversity and strengthening both evaluation and model generalization.

1 Introduction

Sparse GEC references can undercount acceptable corrections and constrain the correction patterns represented in training. The introduction defines validity across grammaticality, meaning preservation, and fluency, then presents JELV, PEVData, and evaluation and reference-expansion applications.

1 Introduction

  • Most GEC datasets have only one or two references, so valid corrections may be undercounted and narrow correction patterns may constrain generalization.Prior reference expansions are manual, which limits scalability and diversity.Source: §1 · p. 1
  • JELV defines an edit as valid only when it satisfies grammatical correctness, meaning preservation, and fluency improvement.This standard extends Minimal Edit’s grammaticality and faithfulness criteria by also requiring improved fluency.Source: §1 · pp. 1–2
  • PEVData benchmarks edit validity using expert judgments of source–hypothesis pairs that differ only in a single edit span.The dataset targets hypothesis edits initially classified as false positives under reference-based evaluation.Source: §1 · p. 2
  • JELV combines a multi-turn LLM-as-Judges pipeline with over 90% accuracy and a distilled DeBERTa classifier with over 85% precision on valid edits.Both implementations are validated against human labels on PEVData.Source: §1 · p. 2
  • JELV’s evaluation pipeline achieves state-of-the-art correlation with human judgments, while retraining on its expanded corpus yields measurable GEC performance gains.The expansion generates candidates for BEA19’s 38,692 single-reference sentences and uses JELV to retain valid edits.Source: §1 · p. 2
  • For evaluation, JELV reclassifies misjudged false positives as true positives, decouples remaining false positives, and incorporates fluency scoring.For explicit expansion, it filters LLM-generated candidates before retraining GEC systems.Source: §1 · p. 2

2 Related Work

GEC evaluation includes reference-based, reference-less, and LLM-based metrics, while meta-evaluation benchmarks compare metric scores with human judgments. The cited work contrasts their trade-offs and benchmark reliability.

GEC Evaluation

  • Reference-based metrics can penalize valid corrections absent from references, reference-less methods may lack transparency and inherit model biases, and LLM-based metrics can be costly and unstable.Meta-evaluation benchmarks assess how well GEC metrics correlate with human judgments.Source: §2 · p. 2
  • GJG15 was later found problematic, casting doubt on conclusions based on it, while SEEDA evaluates corrections at both edit and sentence levels.SEEDA’s two granularities are presented as making it more reliable.Source: §2 · p. 2
  • Reference-less evaluation uses statistical and language models, whereas reference-based evaluation compares system outputs with human references.LLM-based metrics use LLMs as scorers.Source: §2 · p. 2

3 Validity Judgment

JELV defines valid edits through grammaticality, faithfulness, and fluency, then introduces PEVData and two automated judges. The LLM pipeline exceeds 90% accuracy, while the distilled DeBERTa model offers a faster alternative.

3.1 Criteria for Validity

  • An edit is valid only when it meets grammaticality, faithfulness, and fluency criteria simultaneously, assuming text outside the edit span is correct.The framework distinguishes overcorrections on already grammatical text from valid edits to ungrammatical text that are absent from references.Source: §3.1 · pp. 2–3

3.2 PEVData: Dataset Curation

  • PEVData contains 2,797 single-edit pairs—1,459 valid and 1,338 invalid—curated from CoNLL14, ArgRewrite, and JFLEG.Pairs isolate one differing edit span; experts judged pairs with surrounding sentence context, while JFLEG pairs drawn from its reference set were accepted as valid without annotation.Source: §3.2 · p. 3

3.3 JELV: Developing an Automated Judge

  • JELV automates edit-validity assessment using a high-accuracy LLM-as-judges pipeline and a faster distilled DeBERTa classifier, evaluated against PEVData.The workflow separates expert annotation for PEVData from the LLM evaluation stream whose labels are distilled into JELV2.0.Source: §3.3 · p. 3, Figure 3 · p. 4

JELV1.0: LLM-as-Judges Pipeline

  • JELV1.0 uses explanations, two-shot in-context examples, surrounding-sentence context, and sequential review by DS-V3, GPT-4o, and Qwen-Max.Qwen-Max provides the final validity labels; the in-context memory uses 50 PEVData edits excluded from the test set.Source: JELV1.0: LLM-as-Judges Pipeline · p. 3
  • JELV1.0’s combined strategies exceed 90% accuracy; its third-turn Qwen-Max reaches 0.9134 accuracy and 0.8707 F0.5 on PEVData.Accuracy improves as explain-then-annotate, in-context learning, and context-aware prompting are added; each subsequent LLM also outperforms its predecessor.Source: JELV1.0: LLM-as-Judges Pipeline · pp. 3–4, Figure 4 · p. 4

JELV2.0: Classifier Model Distillation

  • JELV2.0 achieves 85.25% precision, 82.24% F0.5, and 78.91% accuracy on the test set.The lightweight DeBERTa classifier uses GPT-2 probability and SBERT semantic similarity as auxiliary features; roughly 10% inconsistency is mainly associated with subtle fluency edits that can require expert discussion.Source: JELV2.0: Classifier Model Distillation · p. 4

4 Evaluation

The evaluation metric combines JELV-based false-positive reclassification, differentiated penalties for remaining false positives, and GPT-2 fluency scoring. On SEEDA, it achieves state-of-the-art correlations with human judgments across nearly all evaluation levels and granularities.

4.1 Method

  • JELV2.0 reclassifies initially flagged false positives judged valid; remaining false positives are separated into non-overcorrections and overcorrections, with α weighting overcorrections.The generalized precision and F-score provide the edit-level component; GPT-2 probabilities yield sentence-level fluency, combined with it using γ.Source: §4.1 · p. 5

4.2 Experiments

  • The metric’s hyperparameters are tuned on one SEEDA domain and evaluated on the other, with system- and sentence-level meta-evaluations against human judgments.The domains contain 163 genetic and 228 social-media source sentences.Source: §4.2 · pp. 5–6

4.3 Result and Analysis

  • The metric achieves state-of-the-art human-judgment correlations across nearly all evaluation levels and granularities on SEEDA’s social-media test set.The genetic-domain test results show similarly strong performance across most levels and granularities.Source: §4.3 · p. 6, Table 2 · p. 6
  • Ablations show that removing JELV reclassification, false-positive decoupling, or fluency integration reduces sentence-level correlation with human judgments.The comparisons disable reclassification, set α=1, or set γ=0, respectively.Source: §4.3 · p. 6, Figure 6 · p. 6
  • The optimal α is larger for edit-level than sentence-level evaluation in most settings, while γ is larger for “+Fluent corr.” than “Base.”These patterns hold across the genetic, social-media, and overall SEEDA domains.Source: §4.3 · p. 6, Figure 7 · p. 6

5 Reference Expansion and Retraining

The authors generate correction candidates for BEA19’s single-reference sentences and use JELV2.0 to filter them before retraining GEC systems. Training on expanded data improves performance for most systems, with further gains from filtering.

Generation

  • Four LLMs generate correction candidates from BEA19’s 38,692 single-reference source sentences, using each sentence and its reference as a one-shot prompt example.The average candidates per sentence are 8.74 for Llama 4, 6.82 for GPT-4o, 4.06 for Qwen, and 2.41 for DS-V3.Source: §5 · pp. 6–7

Filtering

  • JELV2.0 retains only candidate edits judged valid, reducing BEA-train’s average corrections per sentence from 11.54 to 3.90 and its maximum from 104 to 40.The pipeline pairs each candidate correction with its source sentence before classifier evaluation.Source: §5 · p. 7, Figure 8 · p. 7

Retraining

  • Expanded and JELV-filtered BEA19 training yields the highest performance for most systems and significantly outperforms the raw baseline (both tests p < 0.01).Unfiltered expansion also improves over baseline for most systems, while JELV filtering delivers further gains.Source: §5 · p. 7, Table 3 · p. 7
  • The eight retrained systems are evaluated on CoNLL14 using M2, CLEME, and JELV-based F(x), with scores averaged over two retraining runs.The three training variants are raw BEA19, unfiltered expansion, and expansion with JELV validation.Source: §5 · p. 7, Table 3 · p. 7

6 Conclusions

JELV automates edit-validity assessment and supports both evaluation and reference expansion. The authors report state-of-the-art human-judgment correlation for their evaluation metric and improved GEC performance after retraining on expanded data.

6 Conclusions

  • JELV combines false-positive reclassification, false-positive decoupling, and fluency scoring for evaluation, and generation-then-filtering for reference expansion.The authors report state-of-the-art correlation with human judgments and clear GEC performance improvements after retraining on expanded data.Source: §6 · p. 7

Ethical Statement

The ethical statement describes the annotators and data used. The related-work discussion notes that reference expansion has relied on substantial human effort and does not scale easily.

Ethical Statement

  • Three expert annotators with English-teaching, proofreading, and linguistics backgrounds manually judged edit validity and were compensated at market rates.The authors state that, to their knowledge, the data and annotations raise no ethical concerns.Source: Ethical Statement · p. 8

A.1 GEC Reference Expansion

  • Prior manual reference expansions include eight CoNLL14 annotations, 2.3 references per MuCGEC sentence, and three references per sentence in a Russian corpus, but require substantial human effort.The passage argues for an automated, scalable method to generate diverse valid edits.Source: §A.1 · p. 8

A.2 GEC Evaluation Metrics

GEC evaluation metrics include reference-based, reference-less, and LLM-based approaches, each with trade-offs. Meta-evaluation compares metric scores with human judgments, though dataset choice and evaluation granularity can affect conclusions.

A.2 GEC Evaluation Metrics

  • Reference-based metrics can penalize valid non-reference corrections, while reference-less metrics may lack interpretability and inherit model biases.LLM-based metrics offer natural-language scoring but face explainability, instability, bias, and computational-cost challenges.Source: §A.2 · p. 8

A.3 Meta-evaluation

  • Meta-evaluation correlates metric scores with human judgments, commonly using GJG15 and SEEDA, but GJG15 has been found problematic and sensitive to evaluation granularity.SEEDA separates edit-based SEEDA-E and sentence-based SEEDA-S evaluations.Source: §A.3 · p. 8

B Dataset Collection

The dataset collection samples single-edit pairs from CoNLL14, ArgRewrite, and JFLEG, using expert re-annotation for selected edits and existing references for JFLEG.

CoNLL14

  • 1,118 edits unique to CoNLL14’s eight-reference extension were selected for expert validity annotation.The official test set provides two references per sentence; the extension adds corrections, some of which are considered unreliable.Source: B Dataset Collection · p. 8

ArgRewrite and JFLEG

  • 844 ArgRewrite revisions with at least five “Better” votes were selected for expert re-annotation, while 835 single-edit JFLEG references were treated as valid.Prior “Better” labels do not guarantee that an edit meets all three validity criteria; JFLEG was designed for fluency evaluation and supplies four references per source sentence.Source: B Dataset Collection · p. 8

C Ablation Study for JELV2.0’s Features

The JELV2.0 ablation examines its final embedding choices, GPT-2 and SBERT; the reported qualitative result supports including each component.

C Ablation Study for JELV2.0’s Features

D Meta-evaluation on SEEDA’s Test Set of Genetic Domain

The paper reports that its metric retains state-of-the-art human correlations across most evaluation levels and granularities on SEEDA’s genetic-domain test set. The accompanying passages also specify tuned hyperparameters, JELV1.0 judging prompts, and candidate-generation instructions.

D Meta-evaluation on SEEDA’s Test Set of Genetic Domain

  • The metric achieves state-of-the-art correlations with human judgments across most evaluation levels and granularities on SEEDA’s genetic-domain test set.The passage describes the result as confirming robustness across different domains.Source: D Meta-evaluation on SEEDA’s Test Set… · p. 9

E Optimal Hyperparameters

  • Optimal hyperparameters α and γ are tuned for different evaluation settings across domains.The supplied passage does not state their values.Source: E Optimal Hyperparameters · p. 9

G.1 LLM-as-Judges Pipeline (JELV1.0)

  • JELV1.0’s prompts require grammaticality, faithfulness, and fluency, with validity assigned only when all three criteria are satisfied.The later-turn prompt checks whether an earlier analysis and binary prediction correctly assess those criteria and revises them when needed.Source: §G.1 · pp. 9–10

G.2 Candidate Correction Generation

  • The candidate-generation prompt requests all possible corrections that preserve intended meaning and improve naturalness and readability, or “ONLY one reference!” if none qualify.It also requires grammatical correctness and allows clarity or expression improvements that preserve the source’s intended meaning.Source: §G.2 · p. 10

Related papers

Loading 2511.21700v2…