PaperlayerView exact revision on arXiv

Source-linked AI summary

Larger Context Window, Fewer Overcorrections: Optimizing Prompts and Batching for Minimal-Edit Grammatical Error Correction

Kateryna Karpo, Artem Chernodub

arXiv:2609.10810v1cs.CL

TL;DR

Prompted LLMs struggle with minimal-edit GEC because they rewrite well-formed spans, while fine-tuning is infrastructure-heavy. The paper combines taxonomy-based instructions, batching, and LLM-assisted prompt optimization, achieving 78.32 F0.5 on BEA-2019 and narrowing the fine-tuned-model gap to 0.38 points. The results support prompt-based GEC as a practical alternative, though reproducibility and deployment scope remain constrained.

  • Problem

    Prompted LLMs systematically overcorrect in minimal-edit GEC, while fine-tuning remains infrastructure-heavy despite stronger performance.

  • Method

    The paper combines taxonomy-based minimal-edit instructions, batched inference, and model-specific LLM-assisted prompt optimization.

  • Results

    78.32 F0.5 on BEA-2019 places the best prompt within 0.38 points of the best fine-tuned single model.

  • Takeaways & Limitations

    Careful prompt design and context management can bring general-purpose LLMs to fine-tuned GEC quality without fine-tuning infrastructure.

  • Takeaways & Limitations

    The study relies mostly on opaque commercial APIs and evaluates each configuration once without reporting variance, limiting strict reproducibility.

Abstract

from arXiv · show

Minimal-edit Grammatical Error Correction (GEC) is a challenging task for zero- and few-shot prompted Large Language Models (LLMs), which systematically overcorrect and degrade $F_{0.5}$ by rewriting well-formed spans. While fine-tuning provides an effective solution, it imposes substantial infrastructure demands. We introduce a prompt-based approach that closes the gap to fine-tuned models through three advances in GEC prompting methodology. First, we introduce taxonomy-based instructions to enforce minimal-edit constraints with a comprehensive list of grammatical error rules, equipping the LLM with a bounded, metric-aligned scope of correctable edits, which benefits the strongest models while remaining model-dependent overall. Second, we show that batching multiple uncorrected sentences into a single input context acts as a targeted regularizer against overcorrection, systematically reducing the edit rate across diverse LLM families; we hypothesize this arises from attention dilution effect induced by the bounded capacity of self-attention scores. Finally, LLM-assisted Prompt Optimization refines these instructions. Powered by Gemini 3.1-Pro, our prompt achieves $F_{0.5}=78.32$ on the BEA-2019 test set - establishing a new prompt-based SOTA while shrinking the gap to the fine-tuned single-model SOTA (Staruch et al., 2025) to a mere $0.38$ points. Code, prompts, and outputs are publicly available.

1 Introduction

Minimal-edit GEC preserves original phrasing while correcting targeted errors, but prompted LLMs often overcorrect well-formed spans. The paper addresses this gap with taxonomy-based instructions, batching, and prompt optimization, reaching 78.32 F0.5 on BEA-2019.

  • Motivation: Minimal-edit GEC corrects grammatical, spelling, and punctuation errors while preserving original phrasing, unlike fluency-oriented rewriting.This distinction matters in educational applications that aim to preserve learner intent.
  • Motivation: Prompted LLMs systematically overcorrect by rewriting well-formed spans through synonym swaps, stylistic polishing, or clause rephrasings.These changes can degrade the precision-weighted F0.5 objective.
  • Results: 78.32 F0.5 on BEA-2019 establishes a new prompt-based state of the art, within 0.38 points of the best fine-tuned single model.The approach also reaches 67.08 F0.5 on CoNLL-2014.
  • Contributions: Taxonomy-based instructions enumerate grammatical error categories to constrain models to a bounded, metric-aligned edit scope.The taxonomy productively constrains the strongest models but can act as noise for others.
  • Contributions: Batching multiple sentences in one context reduces word-level edit rates and consistently improves F0.5, which the authors hypothesize reflects attention dilution.The paper combines batching with LLM-assisted prompt optimization that iteratively adopts revisions improving dev-set F0.5.

2 Related Work

Prior prompt-based GEC work explored wording, examples, detection, optimization, and decoding aggregation, but prompt-based systems still lagged fine-tuned models. This paper positions batching as a new regularizer against overcorrection and uses human-guided prompt optimization.

  • Prompt-based GEC: Prompt wording can steer LLMs between minimal-edit and fluency-edit regimes, motivating increasingly structured strategies for controlling edits.Later work added detection-correction, syntax-aware example selection, and edit-level voting.
  • Prompt-based GEC: Prompt-based minimal-edit GEC still lagged supervised state of the art despite richer examples, detection prompts, and automatic optimization.Fine-tuning remained the mainstream route to enforcing minimal-edit behavior.
  • Batching: Batching is introduced as a targeted regularizer that uses input-context expansion to reduce overcorrection rather than merely improve inference throughput.The paper distinguishes this use from prior batching work and relates it to attention-dilution effects.
  • Prompt optimization: The paper’s prompt optimization is semiautomatic: a Claude Code skill rewrites the entire prompt under researcher feedback rather than operating fully autonomously.This procedure differs from earlier automatic prompt-optimization applications to GEC.

3 Proposed Method

The method combines minimal-edit instructions, explicit error taxonomies, batched inference, and model-specific LLM-assisted prompt optimization. Optimization uses targeted diagnoses and greedy validation to refine prompts differently across model families.

  • Method: The framework combines GEC taxonomy-grounded prompts, batched inference, and LLM-assisted prompt optimization.The strongest manual prompt combines minimal-edit constraints, few-shot examples, and taxonomy guidance before optimization.
  • Prompting strategies: The taxonomy prompt lists 25 ERRANT categories with labels, glosses, and examples, making edits outside those categories explicitly off-task.It addresses ambiguity left by minimal-edit instructions that do not specify what counts as an error.
  • Qualitative analysis: Manual prompts lengthen as constraints and examples accumulate, whereas optimized prompts vary by model and may compress or expand the longest manual baseline.Prompt length is measured in cl100k_base tokens.
  • Batching: Batched inference supplies numbered, newline-separated sentences and returns a corresponding numbered list, enabling one-to-one evaluation after splitting outputs.The authors advocate batching as the default deployment mode for prompted minimal-edit GEC.
  • LLM-assisted Prompt Optimization: Prompt optimization diagnoses dev-set errors, proposes one targeted constraint or exemplar change, and adopts it only when overall and targeted F0.5 criteria improve.The greedy loop continues until gains plateau.
  • Qualitative analysis: Optimized prompts diverge by model: proprietary runs favor precision-first constraints, while Qwen3-8B retains the taxonomy and demonstrations with limited wording changes.This model dependence supports tailoring prompts to model families rather than assuming one universal instruction.

4 Experimental setup

The study evaluates prompted commercial LLMs and one open-weight baseline on BEA-2019 and CoNLL-2014 using standard GEC metrics. Its pipeline uses structured JSON outputs to preserve sentence-to-prediction alignment.

  • Datasets and metrics: Final evaluation uses 1,312 CoNLL-2014 test sentences with M2 scoring and 4,477 BEA-2019 test sentences with ERRANT v3.0.0 via CodaBench.Development uses 34,308 BEA-2019 training sentences and 4,384 development sentences.
  • Models: The study evaluates six commercial LLMs from OpenAI, Anthropic, and Google across high- and medium-capacity tiers, plus Qwen3-8B as an open-weight baseline.The tiers separate model capacity from prompting and batching effects.
  • Inference pipeline: A LiteLLM router enforces structured JSON outputs validated by a Pydantic schema, producing bijective sentence-to-prediction mappings.The schema removes free-form post-processing from the inference pipeline.

5 Experiments

Across prompting, batching, and optimization experiments, stricter edit constraints reduce overcorrection and improve precision-weighted GEC performance, culminating in near-parity with fine-tuned systems.

  • 5.1 Prompting strategies: Minimal-edits prompting improves F0.5 by 1.1–10.9 points for nearly all models but lowers recall by 6–22 points.Few-shot examples recover part of the lost recall and add 1.3–3.0 F0.5 points across most models.
  • 5.1 Prompting strategies: Taxonomy instructions help the strongest models but act as noise for weaker ones, making prompt structure substantially model-dependent.The best manual prompt reaches 59.28 F0.5 for Claude Opus 4.6 and 57.54 for Gemini 3.1-Pro.
  • 5.2 Batching Effect: Batching shifts sentences toward zero edits and suppresses heavy rewrites, while precision rises and recall falls as batch size increases.For Gemini 3.1-Pro, untouched sentences increase from 33.8% to 50.5%; for Qwen3-8B, they nearly double from 43.8% to 80.7%.
  • 5.2 Batching Effect: F0.5 peaks at B=15 for the high-capacity models and B=5 for Qwen3-8B, reflecting a trade-off between precision gains and recall loss.The batching effect also generalizes to medium-capacity models, whose optima shift to B=30–B=60.
  • 5.4 Final Evaluations: Gemini 3.1-Pro gains 16.4 F0.5 points when taxonomy rules and batching jointly replace vanilla zero-shot prompting.Precision increases by 20.81 points under the joint constraints.

6 Conclusion

The framework combines minimal-edit instructions with batched inference to reduce overcorrection and narrow the gap between prompted and fine-tuned GEC. It achieves prompt-based state-of-the-art results, including F0.5 = 78.32 on BEA-2019 and F0.5 = 67.08 on CoNLL-2014.

  • 6 Conclusion: Batched inference layered on minimal-edit instructions systematically reduces edit rates across LLM families while lowering inference cost and latency.The authors characterize batching as a targeted regularizer against overcorrection.
  • 6 Conclusion: Taxonomy instructions constrain the edit space productively mainly for the strongest models, making their benefit model-dependent.
  • 6 Conclusion: F0.5 = 78.32 on BEA-2019 with Gemini 3.1-Pro places the prompt within 0.38 points of the best fine-tuned single model.The authors describe this as a new state of the art for prompt-based minimal-edit GEC.
  • 6 Conclusion: The framework reaches F0.5 = 67.08 on CoNLL-2014 alongside its BEA-2019 result.

Limitations

The study’s conclusions are bounded by possible benchmark exposure, opaque and variable commercial systems, limited optimization coverage, and restricted evaluation settings. Additional constraints concern metric coverage, the unconfirmed batching mechanism, and deployment costs and conditions.

  • Limitations: Public training data and benchmark splits may have been encountered during pretraining, so results may not measure generalization to completely unseen text.
  • Limitations: Commercial API opacity, silent provider updates, single-run evaluations, and one open-weight baseline limit reproducibility and systematic model comparisons.
  • Limitations: Prompt optimization covers medium-capacity and one open-weight model, leaving transfer from high-capacity siblings unresolved.
  • Limitations: Evaluations use English learner essays, leaving transfer to morphologically richer, lower-resource, professional, and fluency-oriented settings unclear.
  • Limitations: F0.5-centered evaluation omits human evaluation and GLEU, while the self-attention dilution explanation remains an unconfirmed empirical conjecture.Longer prompts also increase inference costs and latency, and shuffled independent sentences may not represent topical deployment streams.

Ethical Considerations

The paper discloses the use of several AI systems for drafting, editing, and proofreading. The authors state that they reviewed the generated text and accept responsibility for the final content.

  • Ethical Considerations: ChatGPT, Claude, Gemini, and Grammarly were used for drafting, editing, and proofreading.
  • Ethical Considerations: The authors reviewed all AI-generated text before finalizing the paper.
  • Ethical Considerations: The authors take full responsibility for the final content.

A Prompts

The appendix lists prompts for English minimal-edit GEC, progressing from vanilla zero-shot instructions to minimal-edit, few-shot, taxonomy-based, and combined prompt variants. All variants instruct the model to preserve meaning and style while making only necessary corrections.

  • A Prompts: The appendix replaces <input_text> with either one sentence or a batched block of sentences at inference time.
  • A.1 Vanilla zero-shot prompt: The vanilla zero-shot prompt asks for a corrected sentence or an unchanged copy when no errors exist.
  • A.2 Minimal-edits zero-shot prompt: The minimal-edits zero-shot prompt requires the smallest corrections while prohibiting paraphrase, stylistic changes, and fluency improvements.
  • A Prompts: Across the prompt variants, the instructions preserve original meaning and style rather than encouraging general rewriting.
  • A.3 Minimal-edits few-shot prompt: The minimal-edits few-shot prompt adds BEA training examples as guidance without permitting example copying.
  • A.3 Minimal-edits few-shot prompt: The listed few-shot examples illustrate corrections involving grammar, number, articles, and related learner-error patterns.
  • A.4 Minimal-edits zero-shot + taxonomy prompt: The taxonomy zero-shot prompt limits correction to 25 listed grammatical error types and leaves unclear errors unchanged.
  • A.5 Minimal-edits few-shot + taxonomy prompt: The taxonomy few-shot prompt combines the 25-type scope with minimal-edit rules and few-shot BEA examples.

A.6 Minimal-edits few-shot + taxonomy + optimized

The prompts combine minimal-edit constraints with explicit error-type rules, punctuation safeguards, and few-shot examples. Additional rules target ambiguous comma splices and tense consistency while preserving unchanged sentences when errors are unclear.

  • Taxonomy-based instructions: The taxonomy prompt constrains corrections to named grammatical and spelling error types while requiring the smallest possible edit and preserving meaning and style.It also instructs the model to leave unclear or error-free sentences unchanged and return plain text only.
  • Punctuation safeguards: The prompt adds a comma-splice rule that replaces commas joining two independent clauses only when the second clause begins with a clearly identified subject and finite verb.It explicitly excludes dependent clauses, fragments, list continuations, lowercase continuations, and non-pronoun openings.
  • Tense safeguards: A past-narrative rule changes a later present-tense verb to simple past when the sentence clearly refers to the same past event, while preserving present tense for general truths.
  • Few-shot guidance: Few-shot examples demonstrate input-to-correction behavior, including unchanged sentences and targeted fixes for agreement, punctuation, determiners, and word forms.
  • Punctuation safeguards: Sentence-initial comma rules target introductory clauses and adverbial phrases only at sentence starts, without adding stylistic commas mid-sentence.

B Inference Pipeline and Structured Output

The inference pipeline sends each sentence, or numbered batch, through a structured-output LLM call validated against a strict Pydantic-derived JSON schema. Configuration and recovery paths are designed to preserve deterministic sentence-to-prediction mapping and replayability.

  • Inference flow: Each sentence is processed through one LLM call in single-sentence mode, while batching packs multiple numbered sentences into one call before applying the same pipeline.
  • Structured output: A strict Pydantic response schema requires corrected-sentence fields and closes object properties, replacing free-form output handling with validated structured responses.
  • Reproducibility: Decoding parameters are read from the run YAML and copied with outputs, enabling bit-for-bit replay under the same provider model snapshot.
  • Provider recovery: Provider incompatibilities trigger a JSON-object retry, while a single-field recovery path extracts the corrected sentence from malformed JSON for evaluation continuity.
  • Batch alignment: The batch payload assigns each source a stable 1-based identifier and requires one result for every SENTENCE_ID, preserving one-to-one source alignment.

C LLM-assisted Prompt Optimization Skill

The prompt-optimization skill iteratively diagnoses category-level errors, proposes one high-precision change, evaluates it, and logs attributable results. Its workflow preserves lineage and rejects changes whose targeted category does not move as predicted.

  • Optimization loop: Each optimization iteration follows a diagnose-propose-register-evaluate-log cycle and changes exactly one prompt variable so performance deltas remain attributable.
  • Diagnosis: Diagnosis regenerates ERRANT type-level category metrics, ranks false negatives or false positives, and inspects 10–20 real examples before proposing a rule.The workflow uses type-level categories because they surface actionable rules.
  • Targeted proposals: Proposals must be narrow, human-readable, and high precision, with explicit expected true-positive gains, false-positive risks, and falsifying observations.The skill rejects additions that cannot exceed roughly 80% precision on the inspected diagnostic sample.
  • Registration: New prompt variants are registered with lineage metadata, imported into the prompt registry, and paired with a configuration that changes only prompt_name.
  • Evaluation and adoption: A candidate becomes the new parent only when F0.5 improves and the targeted category moves as predicted; otherwise the workflow retains the parent or reruns with another seed.

D Batching Sweep and Edit-Count Distributions: Numerical Results

The batching sweep evaluates precision, recall, ERRANT F0.5, word edit distance, and Scribendi score across models and batch sizes, alongside per-sentence edit-count distributions. Edit counts are computed from spaCy-tokenized source and output sentences using word-level Levenshtein distance and grouped into four buckets.

  • Batching sweep: Batching results report Precision, Recall, ERRANT F0.5, word edit distance, and Scribendi score for each model and batch size.The sweep covers B ∈{1, 2, 5, 15, 30, 60, 120} under the minimal-edits few-shot + taxonomy prompt.
  • Edit-count distributions: Per-sentence edits are measured as word-level Levenshtein distance between spaCy-tokenized source and prediction, then grouped into 0, 1–2, 3–5, and 6+ edits.Table 5 summarizes the same 4,384-sentence evaluation using mean and maximum edit counts.
  • Batching sweep: Figure 4 compares edit-count distributions with precision, recall, F0.5, and Scribendi trends for Qwen3-8B and three high-capacity models across shared batch sizes.Open rings identify per-model maxima for the reported metrics, while edit counts are shown in four buckets.
  • Edit-count distributions: Across five of six models, larger batches shift edit-count mass toward zero edits and reduce the 6+ and 3–5 buckets.Claude Sonnet 4.6 is the exception, with a rising 6+ tail and redistribution from the 1–2 bucket toward both lower and higher edit counts.
Loading 2609.10810v1…