Source-linked AI summary
ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark
Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, Michael Lyu
TL;DR
The paper addresses uncertainty about ChatGPT’s ability to revise text and correct grammatical errors by evaluating it on CoNLL2014 against Grammarly and GECToR. ChatGPT performs worse on automatic metrics, especially for long sentences, while human evaluation finds fewer under- and mis-corrections but more over-corrections.
Problem
It remains unclear how capable ChatGPT is at revising text and correcting grammatical errors, motivating evaluation of whether it is a good GEC tool.
Source: §1 · p. 1Method
The study evaluates ChatGPT on CoNLL2014 against Grammarly and GECToR using automatic metrics and human annotation of system outputs.
Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 4Results
Takeaways & Limitations
The results show that relying solely on automatic metrics has limitations for assessing GEC systems.
Source: §4 · p. 5The evaluation uses only 100 randomly selected CoNLL-2014 test sentences and one prompt, without in-context learning methods.
Source: Limitations and Future Works · p. 5
Abstract
from arXiv · showhide
ChatGPT is a cutting-edge artificial intelligence language model developed by OpenAI, which has attracted a lot of attention due to its surprisingly strong ability in answering follow-up questions. In this report, we aim to evaluate ChatGPT on the Grammatical Error Correction(GEC) task, and compare it with commercial GEC product (e.g., Grammarly) and state-of-the-art models (e.g., GECToR). By testing on the CoNLL2014 benchmark dataset, we find that ChatGPT performs not as well as those baselines in terms of the automatic evaluation metrics (e.g., $F_{0.5}$ score), particularly on long sentences. We inspect the outputs and find that ChatGPT goes beyond one-by-one corrections. Specifically, it prefers to change the surface expression of certain phrases or sentence structure while maintaining grammatical correctness. Human evaluation quantitatively confirms this and suggests that ChatGPT produces less under-correction or mis-correction issues but more over-corrections. These results demonstrate that ChatGPT is severely under-estimated by the automatic evaluation metrics and could be a promising tool for GEC.
1 Introduction
The paper asks how well ChatGPT corrects grammatical errors, evaluating it against Grammarly and GECToR on CoNLL2014. It reports weaker automatic scores, especially on long sentences, alongside a different pattern in the kinds of errors found by human evaluators.
1 Introduction
- The study addresses uncertainty about ChatGPT’s GEC ability by testing it on CoNLL2014 against Grammarly and GECToR.The authors describe it as the first study of ChatGPT’s GEC ability.Source: §1 · p. 1
- ChatGPT performs worse than Grammarly and GECToR on automatic evaluation metrics, particularly on long sentences.The paper also reports that human evaluation found fewer under-corrections and mis-corrections, but more over-corrections.Source: §1 · p. 1
- ChatGPT often changes phrase surface expressions or sentence structures beyond one-by-one corrections while preserving grammatical correctness.Source: §1 · p. 1
- Human evaluation finds that ChatGPT makes fewer under-corrections and mis-corrections but more over-corrections.The paper frames this as evidence that automatic evaluation metrics alone can be limiting when assessing GEC systems.Source: §1 · p. 1
2 Background
The background introduces ChatGPT and defines grammatical error correction, then describes common GEC error categories and benchmark datasets.
2.1 ChatGPT
- ChatGPT is described as an OpenAI chatbot powered by large language models.Source: §2.1 · p. 2
2.2 Grammatical Error Correction
- GEC covers spelling, punctuation, grammatical, and word-choice errors, which the background groups broadly as omission, replacement, and insertion errors.The background also notes that JFLEG includes holistic fluency edits to make text more native-sounding, while BEA-2019 represents a wider range of English levels and abilities.Source: §2.2 · p. 2
3 ChatGPT for GEC
On CoNLL2014, ChatGPT scores below Grammarly and GECToR on automatic metrics, especially for long sentences, but its edits often preserve grammaticality while changing expression. Human evaluation finds fewer under-corrections and mis-corrections but more over-corrections from ChatGPT.
3.1 Experimental Setup
- On 100 sequentially sampled CoNLL2014 sentences, the study compares manual-prompt ChatGPT, Grammarly, and RoBERTa-based GECToR using Precision, Recall, and F0.5.ChatGPT was queried manually because of instability; the GECToR implementation uses a pretrained RoBERTa model.Source: §3.1 · pp. 2–3
3.2 Results and Analysis
- Grammarly achieves the highest F0.5 score, while ChatGPT has the highest recall and GECToR the highest precision.The authors suggest that ChatGPT corrects more errors, GECToR leaves more errors uncorrected, and Grammarly balances the two metrics more stably.Source: §3.2 · p. 3
- ChatGPT's performance gap with Grammarly narrows on short sentences but widens substantially on long sentences under existing evaluation metrics.The 100 test sentences were divided into three equally sized length categories.Source: §3.2 · p. 3, Table 3 · p. 3
- ChatGPT changes phrase expression and sentence structure more readily than the minor edits made by GECToR and Grammarly, while maintaining grammatical correctness.Examples include changing “chances” to “opportunities” and recasting a phrase about family diseases.Source: §3.2 · p. 4, Table 4 · p. 4
- Grammarly's further correction adds negligible improvement to ChatGPT's output, whereas it improves GECToR by +2.1 F0.5 and +16.5 Recall.The authors interpret this as evidence that ChatGPT generates correct sentences while GECToR leaves errors.Source: §3.2 · p. 4, Table 5 · p. 4
- Human evaluation finds that ChatGPT has the fewest under-corrections and fewer mis-corrections than GECToR, but produces more over-corrections.Two annotators assessed 20 sampled sentences for under-, mis-, and over-correction.Source: §3.2 · p. 4, Table 6 · p. 4
- Grammarly's slight +0.5 F0.5 improvement mainly comes from punctuation, and undoing punctuation corrections raises F0.5 by +0.0015.The authors note that Grammarly's punctuation modifications are not always correct.Source: §3.2 · p. 5
4 Conclusion
The conclusion reports that ChatGPT trails Grammarly and GECToR on CoNLL2014 automatic metrics, yet often makes grammatical edits beyond one-by-one corrections. Human evaluation finds fewer under- and mis-corrections but more over-corrections, motivating caution about relying only on automatic metrics.
4 Conclusion
- On CoNLL2014 automatic metrics, ChatGPT performs worse than Grammarly and GECToR.The comparison is reported for the CoNLL2014 benchmark dataset.Source: §4 · p. 5
- ChatGPT changes surface expressions and sentence structure beyond one-by-one corrections while maintaining grammatical correctness.Human evaluation finds fewer under-correction or mis-correction issues but more over-corrections from ChatGPT.Source: §4 · p. 5
- The authors argue that relying solely on automatic metrics limits assessment of GEC performance and describe ChatGPT as a potentially valuable GEC tool.Source: §4 · p. 5
Limitations and Future Works
The study’s limitations are its narrow evaluation data, reliance on a single ChatGPT prompt without in-context learning, and use of only three evaluation metrics.
More Datasets
- The evaluation uses only the CoNLL-2014 test set and randomly selects 100 sentences.The authors plan to evaluate on more datasets.Source: Limitations and Future Works · p. 5
More Prompt and In-context Learning
- The study uses one ChatGPT prompt and omits in-context learning methods, which may underestimate ChatGPT’s full potential.Examples the authors identify include demonstration examples and chain-of-thought.Source: Limitations and Future Works · p. 5
More Evaluation Metrics
- The evaluation is limited to Precision, Recall, and F0.5, with broader metrics proposed for future work.The authors mention pretraining-based metrics as one possible addition.Source: Limitations and Future Works · p. 5