PaperlayerView exact revision on arXiv

Source-linked AI summary

CLEME2.0: Towards Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction

Jingheng Ye, Zishan Xu, Yinghui Li, Linlin Song, Qingyu Zhou, Hai-Tao Zheng, Ying Shen, Wenhao Jiang, Hong-Gee Kim, Ruitong Liu, Xin Su, Zifei Shan

arXiv:2407.00934v2cs.CL

TL;DR

GEC evaluation metrics have given limited attention to interpretability, despite its relevance to identifying system weaknesses. CLEME2.0 disentangles four system aspects in a reference-based metric, and experiments report high human-judgment correlations and robustness across reference datasets.

  • Problem

    GEC metrics have focused on alignment with human judgments, while their interpretability—their capacity to disclose system characteristics—has received less emphasis.

    Source: §1 · p. 1

  • Method

    CLEME2.0 is a reference-based metric that scores hit-correction, wrong-correction, under-correction, and over-correction as distinct system aspects.

    Source: §1 · p. 2

  • Results

    Experiments report consistently high correlations on GJG15 and SEEDA and robustness across six reference datasets with disparate annotation styles.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The separated scores characterize system features through distinct edit types, including different kinds of false-positive edits.

    Source: §1 · p. 2

    The paper's experiments do not thoroughly evaluate interpretability, which the authors say remains unverified.

    Source: Limitation · p. 9

Abstract

from arXiv · show

The paper focuses on the interpretability of Grammatical Error Correction (GEC) evaluation metrics, which received little attention in previous studies. To bridge the gap, we introduce **CLEME2.0**, a reference-based metric describing four fundamental aspects of GEC systems: hit-correction, wrong-correction, under-correction, and over-correction. They collectively contribute to exposing critical qualities and locating drawbacks of GEC systems. Evaluating systems by combining these aspects also leads to superior human consistency over other reference-based and reference-less metrics. Extensive experiments on two human judgment datasets and six reference datasets demonstrate the effectiveness and robustness of our method, achieving a new state-of-the-art result. Our codes are released at https://github.com/THUKElab/CLEME.

1 Introduction

GEC metrics have prioritized alignment with human judgments over interpretability, limiting how clearly they expose system weaknesses. CLEME2.0 separates four system aspects and adds edit weighting, with experiments reporting high correlations and robustness across reference sets.

1 Introduction

  • The paper identifies limited interpretability as a gap in GEC metric research, where human-judgment alignment has received greater attention.It frames interpretability as disclosing system characteristics to help identify weaknesses.Source: §1 · p. 1
  • CLEME2.0 distinguishes hit-correction, wrong-correction, under-correction, and over-correction, linking each aspect to a different type of edit.Hit-correction corresponds to TP, wrong-correction to FPne, under-correction to FN, and over-correction to FPun.Source: §1 · p. 2
  • Separating necessary false positives (FPne) from unnecessary false positives (FPun) distinguishes wrongly modified errors from incorrectly positioned edits.The former indicate wrong corrections, while the latter indicate over-corrections.Source: §1 · pp. 1–2
  • Similarity-based and LLM-based edit weighting assign weights per edit to capture contextual semantics beyond surface-form similarity.The paper motivates weighting by the differing significance of edits such as punctuation and content-word corrections.Source: §1 · p. 2
  • Experiments report consistently high correlations on GJG15 and SEEDA, and robustness across six reference datasets with differing annotation styles.The authors also report extensive experiments and analyses confirming effectiveness and robustness.Source: §1 · p. 2

2 Related Work

Prior GEC evaluation work includes reference-based and reference-less metrics, each with different trade-offs. Figure 2 summarizes CLEME2.0's sequence from edit extraction to an integrated score.

2 Related Work

  • M2 identifies optimal edit sequences using F0.5, ERRANT improves edit extraction but remains language-dependent and biased in multi-reference evaluation, and CLEME provides unbiased F0.5 scores.PT-M2 combines PT-based and existing GEC metrics for higher correlations with human judgments.Source: §2 · p. 2
  • Figure 2 depicts CLEME2.0 extracting and categorizing edits, computing four scores, then integrating them into an overall score through edit weighting.The categorized hypothesis edits are TP, FN, FPne, and FPun.Source: Figure 2 · p. 3
  • Reference-less metrics can align well with human judgments, but their dependence on trained models limits interpretability and poses latent risks.The passage lists approaches using grammaticality, regression, language-model, and BERT-based estimators.Source: §2 · p. 3

3 Method

CLEME2.0 extracts aligned edits, separates them into four interpretable scores, and combines the scores into an overall evaluation. Edit weights can reflect either similarity-based edit significance or LLM-assigned importance.

3.1 Edit Extraction

  • Source, hypothesis, and reference sentences are aligned into equal-length chunk sequences, with chunks marked unchanged, corrected, or dummy.A chunk is a basic edit unit.Source: §3.1 · p. 3

3.2 Disentangled Scores

  • TP, FPne, FPun, and FN distinguish matching corrections, mismatched corrections, edits absent from references, and missed reference edits.TP, FPne, and FN are treated as necessary edits; FPun is unnecessary, and conventional metrics merge FPne and FPun as FP.Source: §3.2 · p. 4
  • Hit, wrong, and under-correction scores divide TP, FPne, and FN edits, respectively, by all necessary reference edits.Over-correction is the proportion of FPun edits among all hypothesis corrected or dummy edits.Source: §3.2 · p. 4

3.3 Comprehensive Score

  • A weighted sum combines the four disentangled scores, with higher hit-correction preferred and the other scores treated inversely.Each trade-off factor is between 0 and 1, and the four factors sum to 1.Source: §3.3 · p. 4

3.4 Edit Weighting

  • Similarity-based weighting uses PTScore to assign edit-specific weights that prioritize edits with more significant meaning and quality changes.PTScore simulates a partially accurate source by replacing a source chunk with its hypothesis chunk; larger absolute weights indicate greater edit significance, while positive and negative weights distinguish beneficial and wrong corrections.Source: §3.4 · p. 5
  • LLM-based weighting prompts Llama-2-7B to assign each edit a weight from 1 to 5, with higher values marking more critical edits.Source: §3.4 · p. 5

4 Experiments

Experiments compare CLEME2.0 with GEC metrics against human rankings and across reference datasets, then examine weighting choices and scoring behavior.

4.1 Experimental Settings

  • Experiments use GJG15 for classical systems and SEEDA, focused mainly on neural systems, with human rankings correlated using Pearson and Spearman coefficients.The evaluation also tests six reference datasets with varied annotation styles and annotator counts.Source: §4.1 · p. 5

4.2 Results of GJG15 Ranking

  • CLEME2.0-sim-ind achieves the highest average corpus correlations on GJG15, while SentCLEME2.0-sim-dep and SentCLEME2.0-sim-ind lead on sentence-level Pearson and Spearman correlations, respectively.CLEME2.0-sim-dep closely follows CLEME2.0-sim-ind at corpus level; unweighted variants also achieve comparable correlations to other metrics.Source: Table 1 · p. 6, §4.2 · p. 7
  • Sentence-level metrics outperform corpus-level counterparts: system-level rankings weight each sample equally regardless of edits, while corpus-level metrics emphasize samples with more edits, widening the gap from human evaluation.SentPT-M2 outperforms our approach on CoNLL-2014 but performs worse on BN-10GEC, E-Minimal, and NE-Fluency, indicating a lack of robustness.Source: §4.2 · p. 7
  • Similarity-based weighting generally outperforms unweighted variants, although weighted and unweighted results are comparable on E-Minimal and NE-Minimal.The authors suggest minimal but crucial annotations on those datasets limit weight variation and the efficacy of edit weighting.Source: §4.2 · p. 7

4.3 Results of SEEDA Ranking

  • CLEME2.0 consistently achieves the best outcomes on both SEEDA datasets, and improves across indicators as most metrics’ correlations decline from classical to neural systems.SEEDA-Edit results exceed SEEDA-Sentence results, which the authors attribute to SEEDA-Edit’s greater detail and closer fit with CLEME2.0.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 Results of LLM-based Weighting

  • LLM-based weighting performs less favorably than similarity-based weighting, which the authors link partly to its coarse 1-to-5 edit-weight scale versus the continuous [0, 1] scale.Their approach uses open-source Llama-2-7B, unlike cited work using GPT-4 and GPT-3.5 with specific prompt engineering.Source: Table 4 · p. 8, §4.4 · p. 8

5 Analysis

The analysis illustrates how edit weighting distinguishes corrections by their meaning and influence, examines correlations for the disentangled scores, and compares evaluation runtimes.

5.1 Case Study

  • Chunk 5’s meaning-changing wrong correction receives more weight than Chunk 3, while other examples distinguish minor corrections and over-corrections.Chunk 2 has the highest edit weight (0.056); Chunk 8 introduces a significant pronoun error, whereas Chunk 6 leaves the original meaning unchanged.Source: §5.1 · p. 8

5.2 Ablation Study

  • Hit-correction and under-correction show moderate correlations, while wrong-correction correlates negatively and over-correction has small corpus-level positive and minimal sentence-level negative correlations.The authors caution that wrong-correction’s negative correlations do not mean it has no impact: its trade-off factor is substantial, and evaluating it alone might favor highly confident edits.Source: Table 5 · p. 8, §5.2 · p. 9

5.3 Efficiency

  • 88.4 seconds is the minimum combined duration for ERRANT, CLEME, and CLEME2.0: 33.4 seconds for extraction, 20.7 for chunking, and 34.3 for weighting.PT-M2 took several hours in the reported evaluation, so the authors did not report a precise runtime.Source: §5.3 · p. 9, Table 6 · p. 9

6 Conclusion

CLEME2.0 is an interpretable GEC evaluation metric that highlights four system aspects and incorporates edit weighting. The paper reports that experiments and analyses confirm its effectiveness and robustness.

6 Conclusion

  • CLEME2.0 highlights four key aspects of GEC systems.The conclusion does not name the aspects in this passage.Source: §6 · p. 9
  • The metric incorporates edit weighting techniques to address semantic subtleties that traditional reference-based metrics face difficulty recognizing.Source: §6 · p. 9
  • The paper reports that extensive experiments and analyses confirm CLEME2.0’s effectiveness and robustness.Source: §6 · p. 9

Limitation

The paper identifies unverified efficiency beyond English and limited evidence across languages and text domains, and says CLEME2.0’s interpretability remains unverified.

Limitation

  • CLEME2.0’s efficiency beyond English remains unverified, and the authors call for evaluation datasets covering a range of languages and text domains.The reference sets used in the experiments stem from the CoNLL-2014 shared task.Source: Limitation · p. 9
  • The paper’s experiments primarily assess correlations between automatic metrics and human judgments, not a thorough analysis of interpretability.Source: Limitation · p. 9
  • The interpretability of CLEME2.0 remains unverified; the authors identify human evaluation experiments as future work.Source: Limitation · p. 9

Ethics Statement

The paper describes its dataset sources, edit-weighting prompt, human-ranking data and procedures, ranking algorithms, and reference-set statistics.

Ethics Statement

  • The experiments use publicly available reference datasets and GEC metrics, report no sensitive data, and state that datasets and baselines match their intended purposes.Source: Ethics Statement · pp. 9–10

A LLM-based Edit Weighting

B.1 Human Rankings

  • GJG15 ranks 12 CoNLL-2014 systems using Expected Wins and TrueSkill from eight native speakers’ rankings; SEEDA includes edit- and sentence-level ratings for 12 systems and two human corrections (Kobayashi et al., 2024b).SEEDA’s overall rankings also use Expected Wins and TrueSkill based on pairwise judgments from three native English speakers.Source: §B.1 · pp. 14–15

B.2 Ranking Algorithms

  • Expected Wins aggregates expected victories against opponents using skill-based win probabilities, whereas TrueSkill updates Bayesian skill distributions and ranks participants using a conservative skill estimate.TrueSkill also supports team matches and draws; a stated conservative estimate is μ − 3σ.Source: §B.2 · p. 15

B.3 Statistics of Reference Datasets

  • Table 7 presents statistics for all reference sets used in the experiments.Source: §B.3 · p. 15

B.4 Baseline Metrics

The section surveys reference-based GEC metrics used as baselines and describes cross-validation for selecting CLEME2.0 trade-off factors.

B.4 Baseline Metrics

  • Table 8 reports GJG15 ranking correlations with unchanged-reference results and reductions relative to Table 1, highlighting the highest and second-highest scores.The reference-set statistics table notes that ERRANT is used for edit extraction and CLEME for chunk extraction.Source: Table 8 · p. 16, Table 7 · p. 16
  • M2 and ERRANT extract edits by matching annotations or refining edit extraction, while PT-M2 adds pre-trained language-model knowledge to score edits.M2 uses Levenshtein overlap with gold annotations; ERRANT uses a linguistically refined Damerau-Levenshtein algorithm.Source: §B.4 · p. 16
  • CLEME and SentCLEME provide multi-reference scores and support either correction-dependence or correction-independence assumptions.For SEEDA, the evaluation adds metrics following Kobayashi et al. (2024b).Source: §B.4 · p. 16
  • Other baselines use error-correction difficulty, GPT-2 complexity and ranking ratios, BERT-based criteria, quality and similarity models, or n-gram matching.These include GoToScorer, Scribendi Score, SOME, IMPARA, and GLEU.Source: §B.4 · pp. 16–17

B.5 Details of Determining Trade-off Factors

  • Trade-off factors are selected by repeatedly holding out one of six GJG15 reference sets after grid-searching factors on the other five.The search uses 0.05 intervals and selects factors with the highest average correction on the held-out references.Source: §B.5 · p. 17

C.1 Results of Full References

Across reference sets, CLEME2.0 maintains high or comparable correlations, and the accompanying analyses examine weighting choices and average correlations.

C.1 Results of Full References

  • CLEME2.0 achieves highest or comparable correlations across reference sets; many metrics decline on E-Minimal, while three sentence-level metrics turn negative on NE-Minimal.The negative correlations are reported for SentERRANT, SentCLEME-dep, and SentCLEME-ind.Source: §C.1 · p. 17

C.2 Case Study of LLM-based Weighting

  • In the LLM-weighted examples, Llama-2 assigns high or low weights to edits, with examples illustrating under-correction and over-correction.The authors speculate that the adopted LLM's small scale impairs its ability to distinguish errors of varying levels.Source: §C.2 · p. 17

C.3 Extra Results of CLEME with Similarity Weighting

  • Similarity weighting beats length weighting for corpus-level CLEME, while the reverse holds for SentCLEME; both outperform unweighted settings, but CLEME2.0 remains stronger.The paper attributes the latter comparison to differences in design and scoring frameworks.Source: §C.3 · p. 18, Table 10 · p. 18

C.4 Average Correlations.

  • All tested parameter configurations yield positive average correlations, and similarity weighting raises global human correlation.Corpus-level metrics generally have higher averages, though optimally parameterized sentence-level metrics can outperform them.Source: §C.4 · p. 18, Table 11 · p. 18

C.5 Details Results on CoNLL-2014

The CoNLL-2014 results illustrate how CLEME2.0 separates system performance into edit-related dimensions that can inform system selection.

C.5 Details Results on CoNLL-2014

  • For CAMB, CLEME2.0-dep reports a top hit-correction score of 0.271.This corresponds to about 27.1% of its edits being accurate.Source: §C.5 · p. 18, Table 12 · p. 19
  • CAMB's wrong-correction score is 0.194, its under-correction score is 0.534, and its over-correction score is 0.470.The passage interprets these as 19.4% incorrect edits, 53.4% overlooked grammatical errors, and 47.0% unnecessary edits.Source: §C.5 · p. 18, Table 12 · p. 19
  • Users may prioritize a minimal under-correction score in high-stakes settings, even if that means accepting some unnecessary edits.The paper also says the detailed scores can help developers and researchers pinpoint aspects needing improvement.Source: §C.5 · p. 18

Related papers

Loading 2407.00934v2…