PaperlayerView exact revision on arXiv

Source-linked AI summary

GrammarGPT: Exploring Open-Source LLMs for Native Chinese Grammatical Error Correction with Supervised Fine-Tuning

Yaxin Fan, Feng Jiang, Peifeng Li, Haizhou Li

arXiv:2307.13923v2cs.CL

TL;DR

Native Chinese grammatical error correction remains insufficiently explored with open-source LLMs, despite closed-source models showing strong capabilities. GrammarGPT combines ChatGPT-generated and human-annotated data with error-invariant augmentation for instruction tuning, and significantly outperforms the SOTA system using much less data.

  • Problem

    The potential of open-source LLMs for native Chinese grammatical error correction remains unexplored.

    Source: §1 · p. 2

  • Method

    GrammarGPT builds about 1k hybrid parallel examples, augments named entities with similar ones, and fine-tunes open-source LLMs with instruction tuning.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    GrammarGPT significantly outperforms the SOTA system while using fine-tuning data that is 1200x smaller.

    Source: §1 · p. 3, §4.4 · p. 9

  • Takeaways & Limitations

    The results demonstrate the potential of open-source LLMs for native Chinese grammatical error correction.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

Grammatical error correction aims to correct ungrammatical sentences automatically. Recently, some work has demonstrated the excellent capabilities of closed-source Large Language Models (LLMs, e.g., ChatGPT) in grammatical error correction. However, the potential of open-source LLMs remains unexplored. In this paper, we introduced GrammarGPT, an open-source LLM, to preliminary explore its potential for native Chinese grammatical error correction. The core recipe of GrammarGPT is to leverage the hybrid dataset of ChatGPT-generated and human-annotated. For grammatical errors with clues, we proposed a heuristic method to guide ChatGPT to generate ungrammatical sentences by providing those clues. For grammatical errors without clues, we collected ungrammatical sentences from publicly available websites and manually corrected them. In addition, we employed an error-invariant augmentation method to enhance the ability of the model to correct native Chinese grammatical errors. We ultimately constructed about 1k parallel data and utilized these data to fine-tune open-source LLMs (e.g., Phoenix, released by The Chinese University of Hong Kong, Shenzhen) with instruction tuning. The experimental results show that GrammarGPT outperforms the existing SOTA system significantly. Although model parameters are 20x larger than the SOTA baseline, the required amount of data for instruction tuning is 1200x smaller, illustrating the potential of open-source LLMs on native CGEC. Our GrammarGPT ranks $3^{rd}$ on NLPCC2023 SharedTask1, demonstrating our approach's effectiveness. The code and data are available at \url{https://github.com/FreedomIntelligence/GrammarGPT}.

1 Introduction

Native Chinese grammatical errors are fluent and subtle, while open-source LLMs remain underexplored for correcting them. GrammarGPT addresses this gap with hybrid data, augmentation, and instruction tuning, outperforming the SOTA system with far less data.

  • Native Chinese grammatical errors are often fluent and habit-consistent but grammatically incorrect, making them harder to detect than learner errors.Source: §1 · p. 2
  • GrammarGPT explores open-source LLMs for native Chinese grammatical error correction through supervised fine-tuning.Source: §1 · p. 2
  • The hybrid dataset combines ChatGPT-generated examples for clue-based errors with manually corrected web examples for errors without clues.Source: §1 · p. 2
  • The contribution includes constructing a hybrid dataset through ChatGPT generation and manual annotation for native Chinese grammatical errors.Source: §1 · p. 3
  • The method augments parallel data by substituting named entities with similar ones, helping the model focus on unchanged grammatical errors.Source: §1 · pp. 2–3
  • GrammarGPT significantly outperforms the SOTA system while using instruction-tuning data that is only 1/1200 as large.Source: §1 · p. 3

2 Related Work

Prior CGEC research uses sequence editing or sequence-to-sequence paradigms, while instruction tuning adapts LLMs through task-guided data. GrammarGPT follows the sequence-to-sequence and hybrid-data directions for native CGEC.

2.1 Grammatical Error Correction

  • Sequence editing predicts insertion, deletion, or substitution labels iteratively, whereas sequence-to-sequence models treat CGEC as monolingual translation.Source: §2.1 · p. 4
  • CGEC studies combine the complementary sequence-editing and sequence-to-sequence paradigms through ensembles, while this paper adopts sequence-to-sequence instruction tuning.Source: §2.1 · p. 4

2.2 Instruction Tuning for LLMs

  • Instruction tuning improves generalization by training models on many instruction-guided tasks, with data commonly sourced from ChatGPT, human annotation, or both.Source: §2.2 · p. 4
  • Prior work uses ChatGPT-generated data to distill reasoning or synthesis capabilities into smaller or task-specific models.Source: §2.2 · pp. 4–5
  • Other studies directly convert supervised datasets into instructions for fine-tuning LLaMA on financial sentiment analysis and information extraction.Source: §2.2 · p. 5
  • Hybrid instruction-tuning datasets combine ChatGPT/GPT-4 outputs with human or domain data, an approach GrammarGPT applies to native CGEC.Source: §2.2 · p. 5

3 Methods

GrammarGPT constructs native-Chinese grammatical-error data through complementary automatic and manual methods, augments it by replacing named entities, and formats the pairs for instruction tuning.

ChatGPT-generated Data

3.1 Hybrid Dataset Construction

3.2 Error-invariant Data Augmentation

  • Error-invariant augmentation replaces named entities with similar ones so the model focuses on unchanged grammatical errors rather than specific nouns.Source: §3.2 · p. 7

3.3 Instruction Tuning

  • Instruction tuning uses a task prefix, task description, ungrammatical input, and grammatical output.Source: §3.3 · p. 7

4 Experiments

GrammarGPT is evaluated on a 500-sample native-CGEC validation set with M2 scoring and outperforms the SOTA baseline using roughly 1k training pairs, while ablations clarify dataset and augmentation effects.

4.1 Datasets

  • The evaluation uses a 500-sample NLPCC2023 validation set and word-level and char-level M2 Precision, Recall, and F0.5 scores.Source: §4.1 · p. 8, §4.2 · p. 8
  • The 1061-sample training set contains roughly 35% manually annotated data and 65% ChatGPT-generated data.Source: §4.1 · p. 8

4.4 Experimental Results

  • S2S BART trained on the 1k hybrid dataset reaches 17.57 Word-level and 18.16 Char-level F0.5, comparable to its 1.2M-data baseline.Source: §4.4 · p. 9
  • GrammarGPT achieves 32.56 and 35.84 F0.5 on Word-level and Char-level evaluation with only about 1k fine-tuning samples, nearly doubling baseline performance and ranking 3rd on the official test set.Source: §4.4 · p. 9, Table 5 · p. 9

4.5 Ablation Study

  • ChatGPT-generated data consistently outperforms human-annotated data, partly because annotation is smaller and clue-free errors are harder to correct.Source: §4.5 · p. 10
  • Error-invariant augmentation substantially improves Recall and F0.5 but only slightly improves Precision by emphasizing grammatical errors over named entities.Source: §4.5 · p. 10

5 Conclusion

GrammarGPT fine-tunes an open-source LLM on a small hybrid dataset for native Chinese grammatical error correction, combining ChatGPT-generated and human-annotated data with error-invariant augmentation.

  • GrammarGPT uses approximately 1k hybrid parallel samples, error-invariant augmentation, and instruction tuning to improve native Chinese grammatical error correction.Source: §5 · p. 10
Loading 2307.13923v2…