Source-linked AI summary
Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing
Norah Almousa, Shayan Peyghambari Oskoui, Raquel Coelho, Gayle Rogers, Xiang Lorraine Li, Diane Litman
TL;DR
Evaluations of LLM feedback have largely overlooked what feedback focuses on and whether that focus adapts like expert teachers’ feedback. The paper builds FEEDTYPE and compares teacher and LLM feedback across writing courses. Most LLMs cover most feedback types but do not match teacher distributions or adaptive patterns.
Problem
Feedback focus and whether LLMs adapt it across draft stages and student performance, as expert teachers do, remain underexamined.
Source: §1 · pp. 1–2Method
The authors refine a seven-type feedback-focus taxonomy, annotate teacher and LLM feedback across three university writing courses, and evaluate six LLMs under three prompting strategies.
Source: §1 · p. 2Results
Takeaways & Limitations
FEEDTYPE is released as a public corpus of annotated teacher and LLM feedback to support future research on pedagogical alignment.
Source: §6 · p. 9Feedback-adaptivity analysis is limited to one composition course taught by a single instructor, so whether the patterns generalize to other contexts and instructors remains open.
Source: §7 · p. 9
Abstract
from arXiv · showhide
We investigate whether state-of-the-art large language models (LLMs) generate feedback that reflects the pedagogical practices of expert teachers in terms of feedback focus and adaptivity. Previous evaluation efforts have examined feedback characteristics, its impact on learning, and its target, yet the focus of feedback and its adaptivity remains largely overlooked. To bridge this gap, we adopt and refine Narciss's taxonomy into seven feedback focus types to annotate teacher and LLM-generated feedback across three university writing courses. We release FeedType, a benchmark containing annotated teacher and LLM feedback from six LLMs under three prompting strategies. We assess the coverage and distribution of feedback focus types, and examine whether LLMs adapt their feedback across draft stages and student performance levels as an expert instructor does. Our findings show that while most LLMs cover most feedback focus types, they fail to reflect teacher feedback distributions and show varying levels of adaptivity, with none matching the teachers' adaptive behavior. We believe FeedType will support future research on pedagogical alignment in LLM feedback generation.
1 Introduction
The paper examines a neglected dimension of LLM feedback evaluation: what feedback focuses on and whether that focus adapts to draft stage and student performance. It introduces FEEDTYPE to compare teacher and LLM feedback using a shared taxonomy, finding that LLMs do not reproduce teacher distributions or adaptive behavior.
1 Introduction
- LLMs have shown promise across writing contexts, but prior educational research finds they can fail to capture expert teachers’ pedagogical practices.For example, prior tutoring work found ChatGPT tends to give direct answers where expert teachers guide students with hints.Source: §1 · p. 1
- Feedback focus and its adaptivity remain underexamined, despite teachers varying focus to address different aspects of student writing and student needs.The study asks which feedback types LLMs generate and whether they adapt across draft stages and student performance levels as expert teachers do.Source: §1 · pp. 1–2
- The study addresses whether LLMs cover teachers’ feedback types, match their feedback-type distributions, and adapt focus across draft stages and student performance.The authors frame these as three research questions about comparison with expert teachers.Source: §1 · p. 2
- FEEDTYPE annotates teacher feedback across three university writing courses and uses the same taxonomy to annotate feedback generated by six LLMs under three prompting strategies.This shared annotation enables direct comparisons of feedback focus and analysis of adaptation to student performance and draft stage.Source: §1 · p. 2
2 Related Work
Prior work spans sentence- and essay-level feedback generation, while paragraph-level feedback has received little attention. Existing evaluations cover several feedback dimensions, but this paper focuses on feedback focus and its adaptivity.
LLMs for Feedback Generation on Writing
- Prior work has explored feedback generation at sentence and complete-essay levels, while paragraph-level feedback has received little attention despite being common in expert teaching.The paper generates paragraph-level feedback across L2 learner writing, composition, and fiction, supplying the full essay while targeting specific paragraphs.Source: LLMs for Feedback Generation on Writing · p. 2
Evaluation of LLM-Generated Feedback
- Prior evaluations have examined feedback quality, content, targets, and diversity; this work adds feedback-focus distributions and their adaptivity as alignment dimensions.It compares teacher and LLM distributions using population-level evaluation approaches used in prior educational research.Source: Evaluation of LLM-Generated Feedback · p. 2
L2 Learner Writing
The paper treats feedback adaptivity as changes in feedback focus and releases FEEDTYPE to support analysis across draft stages and student performance levels.
L2 Learner Writing
- Feedback adaptivity refers to adjustments in feedback focus, and prior work had not examined whether LLMs show similar adaptive patterns.The paper considers changes across draft stages and learner performance levels.Source: L2 Learner Writing · p. 3
- FEEDTYPE supports analysis of feedback focus and adaptivity across draft stages and student performance levels.The corpus is released with the components the authors identify as necessary for this analysis.Source: L2 Learner Writing · p. 3
- FEEDTYPE jointly provides scores, revisions, teacher and automatic feedback, and feedback-focus annotations across writing domains.The authors describe this combination as needed to analyze focus and adaptivity across draft stages and student performance levels.Source: L2 Learner Writing · p. 3
3 FEEDTYPE & Annotation Procedure
The authors construct FEEDTYPE from three university writing courses, classify paragraph-level teacher feedback with a refined seven-type taxonomy, and use validated LLM annotations to scale labeling.
3 FEEDTYPE & Annotation Procedure
- The corpus samples first-year college essays and teacher feedback from composition, ESL, and fiction courses in one semester.Composition and ESL students submitted essays with multiple drafts, while fiction students submitted essays without multiple drafts; each paragraph receives one teacher comment.Source: §3 · pp. 3–4
- The adapted Narciss taxonomy separates feedback content focus into seven types: Task Constraints, Concepts, Elaboration, Clarification, Mistakes, Praise, and Other.The authors retained relevant types, refined others, and introduced types to emphasize explanatory feedback.Source: §3 · p. 4
- Two authors refined the coding manual over four pilot rounds, with the final round reaching Cohen’s Kappa κ = 0.774.Each round independently labeled 50 teacher comments and associated paragraphs, then used disagreements to revise definitions and instructions.Source: §3 · p. 4
- Gemini 2.0 Flash was selected to scale feedback classification after achieving κ = 0.78 with human annotators, passing the Alt-Test, and matching conclusions from human labels.The authors evaluated it using a 100-instance agreement dataset and a separate 400-instance test set for conclusion consistency.Source: §3 · p. 5
4 Experiments
The experiments generate feedback with six LLMs under three prompting conditions and evaluate its alignment with teacher feedback through focus-type coverage and distribution similarity. They test adaptivity across draft stages and student performance levels.
4.1 Generating Feedback with LLMs
- Six open- and closed-source models, including education-specialized LearnLM, generate the feedback evaluated in the experiments.The models span different sizes and types.Source: §4.1 · p. 5
- Each model is prompted zero-shot, with focus-category names and definitions, or few-shot with category examples.Prompts also include the assignment rubric and task prompt, the full essay and target paragraph, draft version, and prior feedback for revision and final drafts.Source: §4.1 · p. 5
4.2 Evaluation Methods
- Alignment is assessed by the share of teacher feedback focus types each model covers and the Jensen-Shannon divergence between model and teacher focus-type distributions.Coverage counts types with at least one model-generated instance; JSD ranges from 0 for identical distributions to 1 for completely different distributions.Source: §4.2 · p. 5
- Fisher’s Exact Test compares feedback-type proportions between initial and revision drafts and between revision and final drafts, using p < 0.05 to reject independence.It is used because some models provide zero instances of certain focus types, which chi-square tests do not accept.Source: §4.2 · pp. 5–6
- For performance-level adaptivity, Fisher’s Exact Test compares the distribution of each feedback focus type between high- and low-performing students..Source: §4.2 · p. 6
5 Results
LLMs covered many feedback focus types, but their feedback distributions and adaptation patterns differed from teachers'. LearnLM was closest to the teacher distribution in zero-shot prompting, while teachers changed more focus types across draft stages and performance groups.
5 Results
- Prompting helped Mistral-24B produce Praise and examples helped Qwen-70B, but Gemini still lacked Praise and category guidance made LearnLM miss Other.Gemini also began missing Other under the prompting strategies, in addition to Praise.Source: §5 · p. 6
- Zero-shot LearnLM, Llama-7B, and Mistral-7B achieved full coverage; Gemini and Mistral-24B lacked Praise, while Qwen-70B lacked Other.Full coverage includes feedback that addresses multiple focus types in one comment or is not categorizable under the seven types.Source: Table 3 · p. 6, §5 · p. 6
- Gemini, Qwen-70B, and Mistral-7B assigned over 50% of feedback to Elaboration, while Llama-7B assigned over 60% to Mistakes.LearnLM and Mistral-24B had no focus type above 50%; teachers distributed feedback more evenly.Source: §5 · pp. 6–7, Figure 2 · p. 7
- LearnLM had the lowest zero-shot JSD (0.1344); prompting offered limited gains, and zero-shot remained closest for most models.Prompt 2 increased JSD for most models; Prompt 3 improved alignment only for Mistral-7B (0.2398).Source: §5 · p. 6, Table 4 · p. 7
- Teachers changed five focus types across both draft transitions, versus three for Mistral-24B, the closest model pattern.All models showing draft-stage adaptivity changed Mistakes and Elaboration, the focus types they produced most frequently.Source: §5 · p. 7, Table 5 · p. 8
- In the composition course, teachers differed by performance group in Task Constraints and Praise; Llama-7B and Mistral-7B showed no significant differences.Other models differed in Clarification, Mistakes, or Elaboration; except for Gemini, each significant model focus was its most frequent type.Source: §5 · p. 8
6 Conclusion
The study finds that LLMs cover most feedback focus types but do not match teacher distributions or adaptive behavior. It releases FEEDTYPE, a corpus of annotated teacher and LLM feedback across three writing courses.
6 Conclusion
- Most LLMs cover most feedback focus types but fail to reflect teacher feedback distributions, and none matches teachers’ adaptive behavior.Across draft stages, teachers show significant changes in five types, while the best-performing model shows changes in three; some models show none.Source: §6 · p. 8
- Across draft stages, teachers show significant changes in five feedback focus types, while the best-performing model shows significant changes in only three.Some models show no significant changes at all.Source: §6 · p. 8
- For student performance, some models adapt only in the feedback focus types they produce most frequently.FEEDTYPE is a public corpus of annotated teacher and LLM feedback across three writing courses.Source: §6 · p. 9
7 Limitations
The study’s adaptivity analysis is limited to one composition course taught by one instructor. It examines paragraph-level feedback and three prompting strategies, leaving other feedback formats and approaches for future study.
7 Limitations
- Feedback adaptivity is analyzed only for the composition course taught by a single instructor.Whether the observed patterns generalize to other writing contexts and instructors remains a question for future work.Source: §7 · p. 9
- The study focuses on paragraph-level feedback, not holistic essay-level feedback or other feedback levels.Future research could examine how feedback distributions differ across formats.Source: §7 · p. 9
- The three prompting strategies do not establish how fine-tuned models or models with explicit pedagogical instructions would align.The authors suggest future work investigate whether targeted fine-tuning can elicit more adaptive and teacher-like feedback.Source: §7 · p. 9
A.1 Data Selection
The data selection, taxonomy, annotation rules, and annotator checks define how the study compares teacher and LLM feedback focus. The selections restrict performance analysis to paragraphs with exactly one teacher comment.
A.1 Data Selection
- Students were grouped by median grade separately for each essay and draft stage, then paragraphs with exactly one feedback comment were selected.Because selection followed the median split, excluding paragraphs with multiple comments produced unequal counts of high- and low-performing students.Source: §A.1 · p. 14
A.2 Example Teacher Feedback in a Paragraph
- Table 8 provides examples of student paragraphs across draft stages alongside teacher feedback and its feedback-type annotation.Source: §A.2 · p. 14
B Narciss’s Taxonomy and Our Adaptations
- The adapted taxonomy retains Task Constraints, Concepts, and Mistakes; adapts procedural guidance and correct responses as Elaboration and Clarification; and adds Praise and Other.It excludes simple feedback components and Metacognition, which was rarely observed in the dataset.Source: B Narciss’s Taxonomy and Our Adaptations · p. 14
C Annotation Guidelines
- Annotators assigned each comment to its best-fitting single category, using Other when it fit none or could reasonably fit multiple categories.When categories overlapped, Task Constraints outranked Mistakes, Clarification outranked Elaboration, and corrective content outranked Praise.Source: C Annotation Guidelines · pp. 14–15
D Evaluating LLMs as Annotators
- The annotator evaluation checks agreement on a reference set, whether an LLM can substitute for human annotators, and whether full-set conclusions match human-annotation conclusions.Source: D Evaluating LLMs as Annotators · p. 15
D.1 Traditional Metrics
The study compares LLM feedback annotations with human labels using agreement and accuracy measures. Gemini 3 Flash and GPT-4-Turbo show human-level alignment in classification, outperforming two other tested models.
D.1 Traditional Metrics
- After training, annotators achieved Cohen’s Kappa of κ = 0.68 on 40 new feedback items selected from earlier pilot rounds.The three annotators then labeled 100 feedback items; majority labels served as the reference for agreement and accuracy calculations.Source: §D.1 · p. 16
- LLM annotations were evaluated against human majority labels using Cohen’s κ, and accuracy was also measured on the gold-standard sample.The sample contained 100 feedback items, with at least 10 per category except for the less frequent Concepts category.Source: §D.1 · pp. 15–16
- GPT-4-Turbo shows human-level alignment and outperforms LLaMA-3.1 8B and Qwen3-14B in the classification task.Source: §D.1 · p. 16
D.2 Alt-Test
The appendix tests annotation trustworthiness against individual human annotators and checks whether Gemini captures meaningful feedback patterns. It also documents the prompts and examples used for feedback generation and classification.
D.2 Alt-Test
- The Alt-Test compares each LLM with individual annotators using leave-one-out consensus; at ϵ = 0.1, Gemini Flash and GPT-4-Turbo with 10 few-shot examples reached ω = 0.67.Gemini 3 Flash was selected as annotator because it cost less than GPT-4-Turbo.Source: §D.2 · pp. 16–17, Table 11 · p. 17
D.3 LLM Annotation Validity
- Human and Gemini annotations tracked rising Praise and falling Task Constraints across drafts, and lower Praise but more Mistakes for lower-performing students.Gemini generally reflected human trends, though category percentages were not always identical.Source: Table 12 · p. 18, Table 13 · p. 18, §D.3 · pp. 18–19
E.1 Using LLMs for Feedback Generation
- Generation prompts supplied the essay, target paragraph, rubric, and draft stage; later-draft prompts also included prior feedback, with categories and examples varying by condition.The three conditions were zero-shot, zero-shot with category definitions, and few-shot with category definitions and examples.Source: §E.1 · p. 19, Table 14 · p. 20
E.2 Examples of Feedback Generation
- Table 15 presents one paragraph with feedback from teachers and multiple models.Source: §E.2 · p. 19, Table 15 · p. 21
F Using an LLM as an Annotator
- The annotator-classification prompts included seven category definitions, the rubric, target paragraph, and feedback; few-shot prompts used 85 examples, with at least 10 per category except Concept.The Concept category was relatively rare in the dataset.Source: F Using an LLM as an Annotator · p. 19, Table 14 · p. 20