PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation

Maja Stahl, Leon Biermann, Andreas Nehring, Henning Wachsmuth

arXiv:2404.15845v1cs.CL

TL;DR

Manual essay feedback is labor-intensive, and whether LLMs can provide helpful feedback remains unclear. This paper compares prompting strategies for feedback and AES, including a joint approach. The results indicate that feedback generation improves scoring, while essay scoring has little impact on generated feedback.

  • Problem

    Manual essay feedback limits individualized support, and AES often provides scores without explaining them or guiding essay improvement.

    Source: §1 · pp. 1–2

  • Method

    The paper compares zero- and few-shot LLM prompting strategies for essay feedback and AES, studying the tasks jointly.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Explaining predicted scores through generated feedback improves AES performance, while essay scoring ultimately has little impact on generated feedback.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Manual evaluation finds the generated essay feedback helpful for improving writing, while the impact of scoring on that feedback remains low.

    Source: §1 · p. 2

    The feedback approaches rely on detailed rubrics, which may not always be available and could reduce transferability to other essay datasets.

    Source: §7 · p. 9

Abstract

from arXiv · show

Individual feedback can help students improve their essay writing skills. However, the manual effort required to provide such feedback limits individualization in practice. Automatically-generated essay feedback may serve as an alternative to guide students at their own pace, convenience, and desired frequency. Large language models (LLMs) have demonstrated strong performance in generating coherent and contextually relevant text. Yet, their ability to provide helpful essay feedback is unclear. This work explores several prompting strategies for LLM-based zero-shot and few-shot generation of essay feedback. Inspired by Chain-of-Thought prompting, we study how and to what extent automated essay scoring (AES) can benefit the quality of generated feedback. We evaluate both the AES performance that LLMs can achieve with prompting only and the helpfulness of the generated essay feedback. Our results suggest that tackling AES and feedback generation jointly improves AES performance. However, while our manual evaluation emphasizes the quality of the generated essay feedback, the impact of essay scoring on the generated feedback remains low ultimately.

1 Introduction

Individual essay feedback can help students improve, but the effort of writing it manually limits its practical availability. This study explores LLM prompting for essay feedback and scoring, finding that feedback generation can improve scoring while scoring has little impact on feedback.

1 Introduction

  • Manual feedback is time- and labor-intensive, making individualized guidance difficult to provide despite its potential to help students recognize and overcome weaknesses.Automated feedback could offer guidance at students’ own pace and convenience.Source: §1 · p. 1
  • Most AES approaches provide a score without explaining why it was predicted or guiding students in improving their essays.Source: §1 · pp. 1–2
  • Prior feedback-comment generation addresses grammatical errors at the sentence level, while feedback on complete essays remains relatively unexplored.Source: §1 · p. 2
  • LLMs are evaluated with varied prompting strategies for zero- and few-shot essay feedback, including investigation of how AES and feedback generation affect one another.The study also compares prompting strategies for AES and proposes an LLM-based automatic evaluation strategy for feedback helpfulness.Source: §1 · p. 2
  • Explaining a predicted essay score through generated feedback improves scoring performance on the ASAP dataset, while essay scoring has little impact on feedback.Manual evaluation found the generated feedback helpful for improving essay writing; an LLM-based helpfulness judge correlated well with human judgments.Source: §1 · p. 2

2 Related Work

Prior work has extensively studied automated essay scoring and feedback on sentence-level grammatical errors, but whole-essay feedback has received little attention. This paper situates its work at the intersection of feedback quality and scoring.

2 Related Work

  • AES research commonly targets holistic scores, while assessing multiple essay traits remains challenging; approaches also vary between full-data and few- or zero-shot settings.Prior work includes fine-tuned and prompted approaches, with one self-supervised method outperforming fine-tuned BERT in a one-shot setting.Source: §2 · pp. 2–3
  • Explaining trait-score features can offer insight, but the pedagogical quality and writing impact of scores without accompanying feedback remain questionable.Source: §2 · p. 3
  • Educational NLP has generated explanations for sentence-level grammatical errors, but complete-essay feedback addressing broader improvement needs has received little attention.The cited prior essay-feedback work used zero-shot Chain-of-Thought prompting and received higher human helpfulness judgments than standard prompting.Source: §2 · p. 3
  • This work compares prompting strategies for essay feedback and examines how jointly addressing feedback and AES may benefit each task.Its stated aim is specific, elaborate feedback that assesses the current state and instructs students how to improve.Source: §2 · p. 3

3 Approach

The approach jointly explores essay scoring and feedback generation through in-context learning, varying prompt format, task instructions, and example count. Its task instructions also test different task orders and reasoning styles.

3 Approach

  • The study varies prompt patterns, task instructions, and in-context example counts to examine scoring and feedback generation jointly.The prompt dimensions include context and layout, the tasks and their order, and the number of examples.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Prompt Patterns

  • Base prompts provide general context and layout, while persona prompts assign the model a teacher’s assistant, educational researcher, or creative writing mentor role.The persona patterns aim to implicitly define the expected output type.Source: §3.1 · pp. 3–4, Table 1 · p. 4

3.2 Task Instruction Types

  • Task instructions compare scoring-only, feedback-only, and combined tasks, including scoring before feedback or feedback before scoring.The combined task orders test how scoring and feedback generation relate to one another.Source: §3.2 · p. 4
  • The instructions also test score explanations and two reasoning variants: zero-shot Chain-of-Thought and rubric-guided, stepwise analysis before feedback and scoring.Scoring→Explanation asks for an explanation after a score; Feedback_dCoT→Scoring analyzes essay quality with the rubric, generates feedback, and then scores.Source: §3.2 · pp. 4–5

3.3 In-Context Learning

  • For in-context learning, the study compares zero-shot, one-shot, and few-shot prompts using essays paired with scores and score justifications.Few-shot prompts select essays across score levels and are limited to 5,120 characters.Source: §3.3 · pp. 5–6

4 Data

The study uses the ASAP dataset for essay scoring and as input for feedback generation, because no parallel feedback dataset is available. ASAP contains 12,980 essays scored by two human raters.

4 Data

  • ASAP contains 12,980 essays written by students in grades 7–10, each scored manually by two raters.The essays are divided into eight sets.Source: §4 · p. 6
  • The eight ASAP essay sets differ in prompt, scoring range, and rubric, whose descriptions cover quality levels and typical essay elements.Source: §4 · p. 6
  • Because no parallel dataset exists for essay feedback generation, the study uses ASAP essays as input and evaluates generated feedback without supervision.Source: §4 · p. 6

5 Evaluation

The evaluation compares prompting strategies for essay scoring and feedback generation, then assesses feedback helpfulness automatically and manually. Feedback-first instructions perform well for scoring, while feedback is generally rated helpful; scoring’s effect on feedback remains limited.

5.1 Essay Scoring

  • Feedback_dCoT→Scoring and Explanation→Scoring achieve the highest mean QWK (.553), and feedback-first variations generally outperform scoring-first counterparts.The prompting approaches perform competitively with AES-Prompt; examples aid scoring, with one-shot slightly outperforming few-shot.Source: §5.1 · p. 7, Table 6 · p. 7, Table 7 · p. 7

5.2 Essay Feedback Generation

  • Feedback-only instructions receive the highest average helpfulness scores, while feedback-first variants tend to rate above scoring-first counterparts.Both evaluators favor persona prompts: Mistral’s top score is ER (8.26), and Llama-2’s is CWM (7.48).Source: §5.2 · pp. 7–8, Table 8 · p. 8, Table 9 · p. 8
  • More in-context examples modestly improve feedback helpfulness when scoring precedes feedback.The manual study sampled 24 feedback texts from essay set 4; Feedback led all five helpfulness statements, and all instruction types averaged above neutral.Source: §5.2 · pp. 8–9
  • Manual ratings identify explaining why errors are wrong and student suitability as the feedback aspects with most room to improve.Average interannotator agreement across the four groups is Krippendorff’s α = 0.44.Source: Table 11 · p. 8, §5.2 · p. 9
  • Mistral’s automatic overall-helpfulness scores correlate with manual judgments at 0.61, whereas Llama-2’s scores correlate with none of the statements.The evaluation uses LLM-assigned helpfulness ratings because no existing automatic metric assesses generated essay feedback.Source: §5.2 · pp. 7, 9, Table 12 · p. 9

6 Conclusion

The paper compares prompting strategies for joint essay feedback generation and automated essay scoring. Feedback is judged helpful, and feedback-first prompting benefits scoring, but scoring has little impact on feedback helpfulness.

6 Conclusion

  • Prompting LLMs to generate feedback before scoring benefits essay scoring, while scoring has little impact on feedback helpfulness.The generated feedback is deemed helpful for students by automatic and manual evaluation.Source: §6 · p. 9

7 Limitations

The work’s prompting approaches have limitations in performance, data dependence, and real-world validation.

7 Limitations

  • The prompting approaches’ performance on essay scoring and feedback generation remains improvable.Source: §7 · p. 9
  • Feedback approaches depend on detailed rubrics, whose absence could reduce transferability to other essay datasets.The rubrics include quality-level descriptions, typical essay elements, and reasoning behind example scores.Source: §7 · p. 9
  • Although evaluation suggests feedback is helpful, whether student writers perceive it that way remains unclear.The authors identify real-world use of the generated feedback as pending.Source: §7 · p. 9

A Task Instructions

The appendix presents task-instruction templates that vary whether scoring, explanation, or feedback comes first, and whether reasoning is prompted explicitly.

A Task Instructions

B Automatic Helpfulness Assessment

The automatic helpfulness evaluation prompt asks an evaluator to judge teacher feedback for an essay. It defines helpfulness through error explanation and repair guidance, then requests a score from 1 to 10.

B Automatic Helpfulness Assessment

C Generated Essay Feedback

Table 13 presents generated feedback for one student essay under three task instruction types, as compared in the manual annotation study.

C Generated Essay Feedback

Related papers

Loading 2404.15845v1…