PaperlayerView exact revision on arXiv

Source-linked AI summary

Clause Encounters of the Third Kind: Can LLMs Replace Language Teachers?

Kristina Šekrst, Ana Kovačić

arXiv:2608.16286v1cs.CL

TL;DR

The paper asks whether LLMs can provide the corrective feedback and methodological explanations required for language teaching. It evaluates model and prompting choices, finding that domain-aware prompting substantially improves correction while instructional scaffolding remains unresolved.

  • Problem

    The paper examines whether LLMs can provide corrective feedback and methodological explanations that support language teaching.

  • Method

    The study evaluates multiple LLMs on English learner errors while varying technical parameters and prompting approaches.

  • Results

    Domain-aware prompting improved GPT-4o performance from 4/19 to 17/19, exceeding differences between models in the main evaluation.

  • Takeaways & Limitations

    The findings make scaffolding quality an open problem for LLM-based language teaching.

  • Takeaways & Limitations

    The evaluation covered twenty sentences, including five testing ESP terminology, within a single disciplinary domain.

Abstract

from arXiv · show

While various organizations now actively encourage LLM use in classrooms, we still lack rigorous, systematic evaluations of how well these models actually perform the fundamental tasks of language pedagogy. This paper examines whether state-of-the-art LLMs can deliver the kind of corrective feedback and methodological explanations that language learners need. The study tests multiple large language models on their ability to identify, correct, and explain common learner mistakes in English, by systematically varying model parameters to investigate how these technical adjustments affect output quality, pedagogical clarity, and consistency, along with using retrieval-augmented generation to query methodological data. The evaluation employs automated metrics (GLEU, BERTScore) but also human expert judgments to capture dimensions that purely computational measures miss: linguistic nuance, cultural sensitivity, and instructional appropriateness. While models demonstrate impressive surface-level correction abilities, their explanations often lack the terminological and domain knowledge that effective language teaching requires, suggesting that current enthusiasm for AI-assisted language learning may be outpacing our understanding of these systems' actual pedagogical competence.

Introduction

LLMs increasingly perform language-education and learner-support tasks, while adoption has outpaced evidence that they can provide accurate, nuanced, and pedagogically appropriate instruction. This study compares major model families on English learner-error feedback and examines how prompting, parameter variation, and retrieval affect educational usefulness.

  • Study design: The study compares GPT, Claude, and Gemini on identifying, correcting, and explaining common English learner errors in a 20-sentence ESP dataset.The sentences represent frequent grammatical and ESP-related problems drawn from empirical learner mistakes.
  • Experimental scope: The evaluation tests prompt engineering, generation-parameter variation, and retrieval-augmented generation for effects on output quality, pedagogical clarity, and consistency.RAG supplements general model training with external instructional materials, while the findings are intended to inform specialized educational tools and deployment practices.
  • Evaluation: Because GLEU and BERTScore cannot adequately measure linguistic nuance, cultural sensitivity, or instructional appropriateness, the study combines automated metrics with expert human judgment.These limitations are especially consequential in ESP, where pedagogical adequacy requires expert assessment.

Teaching with large language models

Research reviews find that LLMs support personalized, interactive, and feedback-rich language learning, but also produce hallucinated explanations, uneven cultural sensitivity, and integrity concerns. The literature largely catalogs benefits and risks separately, leaving unclear the conditions for effective and responsible pedagogical use.

  • Reviews of ChatGPT in ESL and EFL contexts identify personalized learning, interactive practice, and dynamic feedback alongside hallucinated explanations, uneven cultural sensitivity, and academic integrity pressures.
  • The literature often documents AI’s benefits and risks side by side without analyzing how those risks constrain effective classroom use.
  • Teachers and educators report efficiency and expected gains in vocabulary, grammar, communication, lesson planning, feedback generation, and assessment, while retaining concerns about integrity, privacy, and accuracy.
  • Responsible implementation requires structured educator training, prompt-engineering competence, and responsible-integration frameworks, especially in developing countries.
  • ChatGPT can act as interlocutor, content provider, teaching assistant, or evaluator, but productive use depends on teachers curating outputs and guiding student interaction.

Technical challenges

LLMs generate language by sampling probable next tokens without distinguishing truth from probability, allowing fluent but incorrect or fabricated outputs. Sampling parameters and automated metrics shape evaluation, but they cannot capture key dimensions of pedagogical adequacy, making human judgment essential.

  • Model limitations: Because LLMs lack an internal distinction between truth and probability, fluent outputs may be factually incorrect, internally inconsistent, or fabricated.Correct and incorrect outputs arise through the same generation process.
  • Generation parameters: Temperature, top-k, and top-p settings control the sampling space, trading predictable and coherent outputs against lexical diversity.Higher temperature increases access to lower-ranked tokens, while top-p adapts the candidate set to distribution concentration.
  • Automated evaluation: GLEU measures grammatical-correction conformity to natural usage, whereas BERTScore measures semantic similarity through contextual embeddings.Their complementary focus reflects the possibility that differently worded explanations convey the same meaning.
  • Evaluation limitations: GLEU and BERTScore cannot assess explanation accuracy, learner-level scaffolding, or sensitivity to cultural, professional, and occupational context.These metrics therefore miss pedagogical dimensions beyond correction and semantic similarity.
  • Human evaluation: Systematic training-data biases can reinforce stereotypes or overlook learners’ linguistic and cultural diversity, so human judgment remains essential.The limitation concerns both bias and the adequacy of outputs for actual learner populations.

Language mistakes and teaching to correct them

This section frames learner errors as essential evidence of developing linguistic systems and argues that effective correction requires elicitation, negotiation, and learner-generated repair. It highlights a pedagogical risk of LLM-mediated writing: polished output can conceal unchanged competence and weaken diagnostic learning.

  • Corrective feedback: Corrective feedback most engages learners when elicitation leads to uptake through negotiation rather than immediately supplying the correct form (Lyster & Ranta, 1997; Ellis, 2003, p. 80).Such interaction requires learners to draw on their own linguistic resources and notice gaps between intended meaning and available forms (Swain, 2005).
  • Corrective feedback: LLM feedback typically reformulates errors immediately, often without elicitation, learner repair, or metalinguistic explanation.This contrasts with instructor diagnostic work, which depends on active communication and knowledge of instructional dynamics.
  • Methodological implications: Terminology detection rose from 21% to 89% when models received explicit information about the discipline, topic, and expected register.The domain-aware prompt condition supplied this contextual information without learner-specific context.
  • Learning risks: When learners delegate writing to LLMs, surface-level accuracy can conceal unchanged underlying competence because difficult forms are bypassed rather than produced.This discrepancy may remain unnoticed by both students and teachers, removing incentives for learners to modify their responses.
  • ESP learning contexts: In ESP contexts, directly supplied terminology may produce appropriate forms without the repeated exposure and associative learning that builds lexical networks.This motivated testing ESP materials, where errors reflect unfamiliarity with professional vocabulary and conventions.

Case study

The case study found that current LLMs reliably corrected the grammatical errors tested, but ESP terminology and register required domain-aware prompting and remained uneven across models. Prompt specificity, encoding teacher expertise as text, improved terminology detection more effectively than model selection, parameter tuning, or retrieval alone.

  • Case study: Every tested model achieved at least 97% grammatical-correction accuracy, with nearly identical outputs across GPT-4o parameter configurations for most items.The residual errors were possible alternative corrections, while agreement between evaluators exceeded 90%.
  • Case study: ESP terminology detection ranged from 11% for GPT-4o to 79% for Claude Opus and Gemini Pro, while larger models outperformed smaller counterparts despite similar grammar scores.Claude Opus detected 15 terms versus Claude Sonnet’s 5, and Gemini Pro detected 15 versus Gemini Flash’s 11.
  • Case study: Models sometimes silently corrected terminology without explaining its disciplinary inappropriateness, solving immediate communication problems while missing opportunities for metalinguistic learning.Claude Opus 4.6 and GPT-5 provided explanations approximating ESP instruction, whereas GPT-4o default feedback treated lay paraphrases as acceptable.
  • Case study: A domain-aware prompt increased GPT-4o’s term detection from 4/18 to 17/18 by specifying the discipline, topic, and expected register.These prompt specifications encoded the ESP instructor’s disciplinary and pedagogical knowledge as text, activating terms absent under the generic prompt.
  • Case study: Retrieval augmentation reached 18/18 only with GPT-5, while alternative configurations scored 15/18 and smaller 200-word chunks consistently performed worse.Poorly tuned retrieval could introduce noise, reduce detection below prompt-only performance, or produce phrasing too technical for classroom use.

Transparency and accountability

LLM-generated language feedback can fail silently because learners and instructors often cannot evaluate its adequacy or trace the configuration that produced it. Responsible deployment therefore requires transparency, traceability, and institutional policies addressing feedback quality alongside academic integrity.

  • Learners cannot reliably distinguish adequate from inadequate feedback because silent failures are indistinguishable from success without the domain knowledge the tool is meant to provide.
  • Whether feedback is adequate can turn on an invisible prompt decision, allowing students submitting identical sentences to receive categorically different feedback.In consumer interfaces, learners generally cannot inspect or change the system prompt or determine whether domain-specific terminology was activated.
  • The more consequential risk is corrupted learning from confident but terminologically vacuous feedback that students have no means of evaluating.
  • Responsible deployment requires transparency about evaluation limits, including disclosure that domain-specific terminology is not configured unless explicitly prompted, which current models do not do by default.
  • Traceability requires instructors to access the prompts, parameters, and model versions producing feedback, while institutional policies must address feedback quality alongside academic integrity.Without access to the feedback chain, instructors cannot assess whether outputs meet instructional standards; interfaces such as ChatGPT also do not expose the parameter adjustments used in the study.

Concluding remarks

Current LLMs can reliably correct standard English grammar, but they cannot replace language teachers because their strongest abilities are least central to acquisition while their key instructional limitations require expert guidance.

  • Concluding remarks: 97% accuracy or higher on standard morphosyntactic errors was achieved by every tested model, robustly across parameter variation, model scale, and three providers.This supports delegating grammatical error correction to current-generation LLMs without meaningful loss of accuracy.
  • Concluding remarks: ESP terminology detection ranged from 11% to 79% across models, but the spread nearly disappeared when prompts specified the discipline, topic, and register.The terminology was present in training data but required instructor-directed activation.
  • Concluding remarks: LLM-generated tests produced high accuracy quickly, yet recognition-based tasks provide fewer opportunities for the effortful integration underlying interlanguage restructuring.The ChatGPT-generated test was completed in a fraction of the time, with half the cohort achieving perfect scores (Schachter (1974)).
  • Concluding remarks: LLMs automate grammatical correction, surface reformulation, and recognition-based items, but lack domain-sensitive detection, deep-processing tasks, and feedback enabling noticing or elicitation.Expertise determines whether outputs are genuinely productive: generic prompts can yield grammatically correct but terminologically imprecise feedback, marking the distance between a tool and a teacher.

Appendix A: Evaluation sentences​

The evaluation set contains 20 ESP-oriented English sentences designed to test grammatical correction and terminological or stylistic appropriateness. Sentences 1–15 contain deliberate grammatical errors, while sentences 16–20 use lay paraphrases of biotechnical terminology related to lycopene.

  • Dataset composition: The dataset comprises 15 learner-like grammatical errors and five well-formed but terminologically and stylistically inappropriate sentences for target discourse communities.The final five sentences use lay paraphrases of domain-specific biotechnical terminology drawn from instructional materials on lycopene.
  • Category 2: Article usage: Sentences 6–10 test article usage in scientific contexts, including missing, indefinite, and definite articles before researchers, substances, instruments, and academic roles.Examples reference experiments, bacteria, lycopene, tomatoes, MRI scans, vitamin D, biochemistry, and university research.
  • Category 4: ESP-specific terminology and register: Sentences 16–20 assess ESP-specific terminology and register through lay descriptions of tomato nutrition, lycopene, absorption, and health effects.The sentences paraphrase domain concepts such as fat-soluble pigments, crystallization, heat processing, fat-assisted use, and harmful atoms or cholesterol buildup.

Appendix B: Standardized evaluation prompt … D.1 Model configurations

The appendices standardize corrective-feedback evaluation, extend it with domain-aware and retrieval-augmented prompts, and document model configurations used in the study. The evaluation varies instructional context, retrieved materials, and model-generation parameters while preserving structured outputs.

  • Appendix B: Standardized evaluation prompt: The standardized prompt asks models to identify learner errors, provide a corrected sentence, and explain each correction for future application.It frames the evaluator as an experienced ESP and academic-writing instructor and uses a tripartite correction, identification, and explanation structure.
  • Appendix B: Standardized evaluation prompt: The tripartite response structure reflects effective corrective feedback described by Lyster and Ranta (1997), applied to twenty evaluation sentences through a Python 3.14 script.The {sentence} placeholder was replaced with each sentence listed in Appendix A.
  • Appendix C: Domain-aware and RAG evaluation prompts: Domain-aware prompting specifies the discipline, topic, and expected register to activate instructional terminology that the generic prompt may fail to elicit.The design corresponds these contextual elements to information an ESP instructor would bring to feedback.
  • Appendix C: Domain-aware and RAG evaluation prompts: RAG adds course-relevant textual chunks to test whether instructional-source retrieval can compensate for knowledge unavailable from model training alone.The retrieval corpus contains 390 chunks of 800 words with 200-word overlap, derived from four course-relevant lecture materials; ten similar chunks were retrieved per sentence using text-embedding-3-small.
  • Appendix C: Domain-aware and RAG evaluation prompts: Preliminary testing found that smaller chunks and lower retrieval depths produced incomplete terminological coverage, motivating the 800-word, k=10 configuration.Appendix D.5 is described as confirming that only this configuration matched or exceeded the relevant comparison.
  • Appendix C: Domain-aware and RAG evaluation prompts: The domain-aware prompt targets food-science ESP writing by flagging lay or informal language, requiring domain-specific terminology, and enforcing academic or scientific register.The RAG condition supplies reference material on lycopene and food science for identifying lay paraphrases and standard scientific equivalents.
  • D.1 Model configurations: GPT-4o was tested at temperatures 0.0, 0.3, 0.7, and 1.0 and top-p values 0.5, 0.9, and 1.0, while GPT-5.2 fixed both at 1.Anthropic’s API did not permit simultaneous temperature and top-p specification.
  • D.1 Model configurations: The study used default API model versions including gpt-4o, gpt-5, claude-opus-4-6, claude-sonnet-4-6, gemini-3-pro-preview, and gemini-3-flash-preview.Models were run from Feb 1 to Feb 22, 2026, with no specific subversions pinned and negligible differences reported.

D.2 Cross-model results

Table 3 presents cross-model results ranked by ESP detection, with GPT-5’s ESP score reported out of 15. The evaluation notes one missing output, defensible alternative corrections, and hidden reasoning overhead in GPT-5 token counts.

  • Table 3 reports cross-model results ranked by ESP detection, with GPT-5’s ESP score measured out of 15.
  • One sentence produced no output in the last recorded run, limiting the evaluable terms.
  • Residual grammar misses were defensible alternatives, including “The researcher” for “A researcher” and restructuring around “affected.”
  • GPT-5 token counts include hidden reasoning overhead.

D.3 Reasoning Model Behavior · D.4 GPT-4o parameter sensitivity · D.5 Retrieval-augmented generation

Reasoning-model outputs were highly sensitive to token budgets, while GPT-4o corrections were mostly stable except for greater variation on ESP sentences. Retrieval-augmented GPT-5 matched or exceeded the domain-aware prompt baseline only with 800-word chunks and k=10 retrieval.

  • D.3 Reasoning Model Behavior: At 1,500 tokens, GPT-5 returned empty responses for 15 of 20 sentences, but 19 of 20 completed at 4,096 tokens and all 20 at 8,192.The 1,500-token failures consumed the full allocation on internal reasoning.
  • D.3 Reasoning Model Behavior: GPT-5’s hidden reasoning consumed an estimated 73%–91% of tokens, averaging 87% on ESP sentences.Gemini Pro at 1,500 tokens truncated 5 of 20 responses.
  • D.3 Reasoning Model Behavior: Gemini Pro leaked reasoning traces in two cases at 1,500 tokens, including teacher-persona guidance and a correction rationale.The surfaced traces included tone instructions and reasoning that “suggests” was correct for the neither…nor sentence.
  • D.4 GPT-4o parameter sensitivity: On ESP sentences, parameter variation produced visible effects: sentence 20 had eight distinct corrections, sentence 19 had six, and sentence 19’s variants increased with temperature.Sentence 19 had one variant at temperature 0.0 and three at 1.0.
  • D.5 Retrieval-augmented generation: The retrieval experiment used 390 corpus chunks with text-embedding-3-small and retrieved 10 chunks per sentence by cosine similarity; GPT-5 was not tested with the generic prompt.The corpus drew on Rao, Young, and Rao (2021), Pokorný, Yanishlieva, and Gordon (2001), and lecture materials, with similarity scores ranging from 0.55–0.65.
  • D.5 Retrieval-augmented generation: 18/18 matched or exceeded the domain-aware prompt baseline, compared with 17/18, only for GPT-5 retrieval using 800-word chunks and k=10.With 200-word chunks, performance was 15/18 regardless of retrieval depth, while 800-word chunks at k=5 did not reach the same result.

D.6 Automated metrics

Automated evaluation used GLEU and BERTScore across 339 responses against references containing full target ESP terminology. Results show that scores can diverge from terminology detection because preserved phrasing or extensive rewriting affects metric overlap.

  • Evaluation setup: GLEU and BERTScore were computed for all 339 responses against gold-standard references incorporating full target ESP terminology.The metrics were applied to references designed to include the complete terminology target.
  • Automated metric results: Sonnet’s ESP GLEU scores were 0.017 and 0.042, with BERTScore 0.835, aligning with responses that judged sentences correct and proposed no corrections.These results correspond to sentences 16 and 18.
  • Automated metric results: GPT-4o’s relatively high ESP GLEU despite detecting only 2/19 terms resulted from overlap in preserved non-terminological phrasing.Thus, GLEU could appear favorable without corresponding terminology detection.
  • RAG condition metrics: GPT-5’s low GLEU despite strong ESP term detection of 17/18 reflects extensive sentence rewriting in the retrieval-augmented condition.Table 6 reports metrics for ESP sentences evaluated with attached course materials and Category 4 examples.
Loading 2608.16286v1…