Source-linked AI summary
Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation
Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, Yue Zhang
TL;DR
ChatGPT’s GEC capabilities were insufficiently studied comprehensively, so this paper evaluates zero-shot and few-shot CoT prompting across multilingual and document-level benchmarks. It finds strong error detection, fluent corrections, and multilingual promise, but also over-correction and weaknesses in several document-level error types.
Problem
The paper addresses the lack of comprehensive evidence about ChatGPT’s genuine grammatical error correction capabilities.
Method
The study evaluates zero-shot and few-shot CoT prompting across five official test sets in three languages and three English document-level GEC test sets.
Results
ChatGPT shows strong error detection, human-like fluency, and advantages in multilingual and low-resource GEC settings.
Takeaways & Limitations
The findings indicate potential for multilingual GEC and improved performance through chain-of-thought prompting.
Takeaways & Limitations
ChatGPT tends toward casual edits even though practical applications such as language education may require minimal corrections.
Abstract
from arXiv · showhide
ChatGPT, a large-scale language model based on the advanced GPT-3.5 architecture, has shown remarkable potential in various Natural Language Processing (NLP) tasks. However, there is currently a dearth of comprehensive study exploring its potential in the area of Grammatical Error Correction (GEC). To showcase its capabilities in GEC, we design zero-shot chain-of-thought (CoT) and few-shot CoT settings using in-context learning for ChatGPT. Our evaluation involves assessing ChatGPT's performance on five official test sets in three different languages, along with three document-level GEC test sets in English. Our experimental results and human evaluations demonstrate that ChatGPT has excellent error detection capabilities and can freely correct errors to make the corrected sentences very fluent, possibly due to its over-correction tendencies and not adhering to the principle of minimal edits. Additionally, its performance in non-English and low-resource settings highlights its potential in multilingual GEC tasks. However, further analysis of various types of errors at the document-level has shown that ChatGPT cannot effectively correct agreement, coreference, tense errors across sentences, and cross-sentence boundary errors.
1 Introduction
The paper addresses limited comprehensive evidence about ChatGPT’s grammatical error correction capabilities by evaluating zero-shot and few-shot chain-of-thought strategies. Results indicate strong error detection, fluent corrections, multilingual potential, and recurring over-correction and document-level weaknesses.
- Motivation: The study fills a research gap by comprehensively investigating ChatGPT’s GEC capabilities beyond a preliminary evaluation of 300 CoNLL14 sentences.The paper evaluates ChatGPT across multiple languages, test sets, and document-level settings.
- Approach: Zero-shot and few-shot CoT prompting are designed to evaluate and improve ChatGPT’s GEC performance through in-context learning.The evaluation covers five official test sets in three languages and document-level English GEC.
- Findings: ChatGPT has strong error detection but lower Precision and F0.5 than current SOTA systems, while achieving remarkably superior Recall.Its elevated modification freedom may produce superfluous changes.
- Findings: ChatGPT shows human-like fluency on JFLEG, with minimal difference from SOTA models and performance surpassing human evaluations.This result supports its potential for fluent grammatical correction.
- Findings: Zero-shot CoT gives ChatGPT advantages in non-English and low-resource settings over Transformer models trained from scratch.The result highlights potential for multilingual GEC.
- Findings: Human evaluations find greater fluency alongside over-correction, while longer sentences show increasing under-correction as ChatGPT follows minimal edits.These tendencies reveal a trade-off between fluent revision and conservative correction.
- Limitations: Document-level GEC remains weak for agreement, coreference, cross-sentence tense errors, and cross-sentence boundary errors.The paper associates these weaknesses with limitations in processing excessively long sentences.
2 Experimental Setup
The evaluation spans multilingual sentence-level and English document-level GEC benchmarks, compares ChatGPT with established systems, and combines automatic and human assessment methods.
- Datasets: The experiments use five official sentence-level GEC test sets across English, German, and Chinese, plus three English document-level test sets.The sentence-level sets include CoNLL14, BEA19, NLPCC18, and Falko-MERLIN; document-level sets include FCE, BEA19, and CoNLL14.
- Systems: Comparisons include Transformer-base, GECToR, T5, TagGEC, MultiEnc-dec, and Grammarly across sentence-level and document-level tasks.The selected systems represent mainstream or state-of-the-art baselines for the relevant settings.
- Metrics: Automatic evaluation uses M2 Scorer for CoNLL14, Falko-MERLIN, and NLPCC18, ERRANT for BEA19, and GLUE for JFLEG.Document-level evaluation uses the official BEA19 shared-task ERRANT scorer after constructing document-level references.
- Human Evaluation: Human evaluation combines automatic human-performance comparisons with manual annotation by three annotators on CoNLL14 outputs.This design addresses potential bias from limited reference corrections in automatic evaluation.
3 Experiments
The experiments compare zero-shot and few-shot chain-of-thought prompting for ChatGPT on English and non-English grammatical error correction tasks. CoT prompting improves performance over normal zero-shot prompting, while English results show strong recall and fluency-oriented correction.
- Prompting Design: Zero-shot CoT significantly outperforms normal zero-shot prompting on the CoNLL14 and BEA19 English test sets.
- Evaluation Setup: The study evaluates zero-shot CoT and 1-, 3-, and 5-shot CoT strategies across English, German, and Chinese GEC test sets.
- English GEC Results: ChatGPT surpasses existing SOTA systems in Recall on CoNLL14 and BEA19, despite lower Precision and F0.5 scores.
- English GEC Results: Under 3-shot CoT, ChatGPT’s GLEU score on JFLEG falls 1.2 points short of SOTA and exceeds T5-large by 0.7 points.
- Prompting Effects: Few-shot CoT generally outperforms zero-shot CoT, but performance tends to decrease when more than five in-context examples are used.
- Multilingual Evaluation: The non-English evaluation includes German Falko-MERLIN and Chinese NLPCC18, with German treated as a low-resource GEC task because its training dataset is smaller.
De (Falko-MERLIN) Zh (NPLCC18)
ChatGPT shows strong recall in German and Chinese GEC, outperforming Transformer models trained from scratch, but its precision and F0.5 remain lower than stronger systems. In document-level English GEC, it similarly favors recall while prompting and long-document constraints limit overall performance.
- German and Chinese GEC: ChatGPT outperforms SOTA systems in Recall on German and Chinese GEC but scores significantly lower in Precision and F0.5.This pattern mirrors the English sentence-level results and suggests broad error-detection ability alongside excessive modifications.
- German and Chinese GEC: ChatGPT surpasses Transformer models trained from scratch on German and Chinese evaluation metrics, indicating potential for low-resource and multilingual GEC.The German setting is low-resource because its training dataset is much smaller than English and Chinese.
- German and Chinese GEC: Few-shot CoT performs worse than zero-shot CoT for Chinese GEC, unlike the pattern observed in English and German.The authors hypothesize that English-centric modeling and Chinese lexical complexity may contribute, while more effective example selection could help.
- Document-level GEC: On three English document-level test sets, zero-shot CoT retains good Recall but has significantly lower Precision and F0.5 than Transformer-based comparison systems.The comparison includes CoNLL14, FCE, and BEA19 document-level evaluations.
4 Human Evaluation and Analysis
Human evaluations find that ChatGPT produces highly fluent corrections and detects many errors, but its fluency is accompanied by over-correction and limited adherence to minimal edits. Document-level analysis identifies especially poor handling of cross-sentence boundary errors and possible limitations in contextual consistency.
- Automatic Human Evaluation: ChatGPT’s best few-shot CoT result is only 0.85 F0.5 points below human-level evaluation on CoNLL14.The result remains below two mainstream SOTA models in the human evaluation.
- Automatic Human Evaluation: On JFLEG, ChatGPT exceeds the human-level score by 1.15 GLEU points and T5 large by 0.7 GLEU points.The authors interpret this as evidence that ChatGPT’s corrections are highly fluent and natural.
- Manual Human Evaluation: Manual evaluation assesses Fluency, Minimal Edits, Over-Correction, and Under-Correction using three postgraduate student evaluators.The evaluation covers 50 short, 50 medium-length, and 50 long CoNLL14 sentences.
- Manual Human Evaluation: ChatGPT achieves significantly better fluency across short, medium, and long sentences but is less inclined than other systems to use minimal edits, especially for short sentences.Its adherence to minimal edits improves as sentence length increases.
- Manual Human Evaluation: ChatGPT over-corrects more than the other evaluated systems while making fewer under-corrections.The combination indicates freer and more diverse corrections, consistent with its strong fluency scores.
- Document-level Error Analysis: Cross-sentence boundary errors produce declines of 14.8 F0.5 in CONJ and 18.3 F0.5 in PUNCT for ChatGPT.The authors associate this underperformance with possible limitations in contextual memory, consistency, and coherence across sentences.
5 Conclusion
The study evaluates ChatGPT for grammatical error correction across multilingual sentence-level and English document-level settings, finding strong fluency and error detection but important precision and document-level weaknesses.
- ChatGPT was evaluated on five official test sets in three languages and three English document-level GEC test sets.
- ChatGPT demonstrates strong error detection and human-like fluency despite poor precision and F0.5 scores.
- ChatGPT shows potential for multilingual GEC, particularly in non-English and low-resource settings.
- Document-level analysis finds poor correction of agreement, coreference, cross-sentence tense, and cross-sentence boundary errors.
Limitations
The study is limited by its focus on ChatGPT, its tendency toward fluent but casual edits, and randomly selected few-shot examples rather than sentence-specific demonstrations.
- The study examines ChatGPT but not other mainstream LLMs, including other GPT-3.5 models and GPT-4.
- ChatGPT produces fluent, casual edits, while some applications such as language education may require minimal corrections.
- Few-shot prompts use randomly selected development-set examples rather than examples specifically designed for each sentence.
A.1 Examples of zero-shot settings
The appendix illustrates the prompting configurations used for ChatGPT's zero-shot and zero-shot chain-of-thought settings.
- Figure 3 presents a zero-shot ChatGPT setting, while Figures 4 and 5 present zero-shot chain-of-thought settings.
A.2 GEC System Output Examples
The output examples compare GEC systems and show that ChatGPT often trades minimal editing for greater fluency through over-correction.
- ChatGPT tends to over-correct short and medium-length sentences instead of adhering strictly to the minimum editing principle.
- The resulting corrected sentences exhibit higher fluency despite ChatGPT's over-correction behavior.
- For long sentences, ChatGPT may replace a grammatical contrast marker such as “while” with “and,” weakening the contrasting relationship.
- Table 10 records evaluator scores for fluency, minimal edits, over-correction, and under-correction across short, medium, and long sentences.