Source-linked AI summary
Using Generative AI and Multi-Agents to Provide Automatic Feedback
Shuchen Guo, Ehsan Latif, Yifan Zhou, Xuan Huang, Xiaoming Zhai
TL;DR
GenAI feedback for student science responses can over-praise or infer beyond students’ work. This study compares AutoFeedback, a generator paired with a validator and reviser, with a single agent and reports fewer instances of both errors.
Problem
GenAI feedback can over-praise students or infer beyond their responses, while research evaluating and improving these feedback flaws remains limited.
Source: §1 · p. 2Method
AutoFeedback uses one agent to generate feedback and another to examine and revise it, and its feedback is compared with a single agent’s.
Source: §1 · p. 2Results
AutoFeedback reduced over-praise by 14.17% and over-inference by 20.21% compared with single-agent feedback.
Source: §6 · p. 11Takeaways & Limitations
Multi-agent feedback can improve GenAI feedback quality by addressing over-praise and over-inference in the evaluated setting.
Source: §6 · p. 11Agent 2 did not always detect these problems accurately, and some identified issues remained unresolved in final feedback.
Source: §7 · p. 12
Abstract
from arXiv · showhide
This study investigates the use of generative AI and multi-agent systems to provide automatic feedback in educational contexts, particularly for student constructed responses in science assessments. The research addresses a key gap in the field by exploring how multi-agent systems, called AutoFeedback, can improve the quality of GenAI-generated feedback, overcoming known issues such as over-praise and over-inference that are common in single-agent large language models (LLMs). The study developed a multi-agent system consisting of two AI agents: one for generating feedback and another for validating and refining it. The system was tested on a dataset of 240 student responses, and its performance was compared to that of a single-agent LLM. Results showed that AutoFeedback significantly reduced the occurrence of over-praise and over-inference errors, providing more accurate and pedagogically sound feedback. The findings suggest that multi-agent systems can offer a more reliable solution for generating automated feedback in educational settings, highlighting their potential for scalable and personalized learning support. These results have important implications for educators and researchers seeking to leverage AI in formative assessments, offering a pathway to more effective feedback mechanisms that enhance student learning outcomes.
1 INTRODUCTION
Automatic feedback could provide tailored, real-time support, but GenAI feedback can over-praise or infer beyond student work. This study tests a two-agent system against a single agent on constructed science responses.
1 INTRODUCTION
- Automatic feedback may offer tailored, real-time support, including when instructors cannot provide individualized feedback simultaneously to students with diverse needs.The paper situates its study in formative assessment and student learning support.Source: §1 · p. 2
- Over-praise and over-inference can mislead students, undermine motivation, and impair opportunities to improve learning.Over-praise is excessive positivity misaligned with performance; over-inference draws conclusions not supported by the response.Source: §1 · p. 2
- AutoFeedback pairs one agent that generates feedback with another that examines and revises it, and the study compares it with a single agent.The evaluation focuses on reducing over-praise and over-inference in feedback on students’ written explanations of scientific phenomena.Source: §1 · p. 2
2 AI-BASED AUTOMATIC FEEDBACK FOR LEARNING
The section traces automatic feedback from established approaches to GenAI, while emphasizing the need for feedback that is accurate, constructive, and responsive to students’ work.
2.1 Significance of Learning Feedback
- Effective feedback addresses learning goals, current performance, and next steps, but meeting diverse students’ needs through regular personal contact is challenging.The section presents automatic feedback as a potential source of real-time, personalized learning support.Source: §2.1 · p. 3
2.2 Automatic Feedback Generation
- Traditional answer matching struggles with unexpected responses, while NLP/ML can assess constructed work but requires substantial data, resources, and technical expertise.GenAI may be more accessible and less dependent on extensive datasets, yet its feedback quality still needs attention.Source: §2.2 · p. 3
2.3 Quality of GenAI-generated Automatic Feedback: Over-Praise and Over-Inference
- Reported GenAI feedback shortcomings include correctness, suggestions, diagnostic value, and tone; comparisons also found weaker improvement guidance and language-sensitive performance.ChatGPT feedback was less satisfying than educators’ feedback in Greek.Source: §2.3 · p. 4
- Over-praise mismatches encouragement with student performance, while over-inference attributes conclusions unsupported by the response; both can mislead learners.The section notes risks to learning and self-evaluation, including for students receiving indiscriminate praise (Black & Wiliam, 1998; Irvin et al., 2021; H. I. Lee et al., 2017).Source: §2.3 · p. 4
- Research evaluating and improving GenAI-generated feedback remains limited: most studies compare it with human feedback, and fewer address improving feedback based on identified issues than feedback generation, particularly for English learners.This study focuses on over-praise and over-inference and proposes AutoFeedback, a multi-agent solution for improving feedback on constructive responses.Source: §2.3 · p. 4
3 MULTI-AGENTS FOR AUTOMATIC FEEDBACK
The section introduces multi-agent systems as collaborators with specialized roles, then presents AutoFeedback as a workflow for generating and checking student feedback.
3.1 Multi-Agents
- Multi-agent systems divide work among autonomous agents that may specialize by role; educational examples include adaptive learning, tutoring, and classroom simulation.The section describes these systems as useful for complex educational processes involving different aspects of learning (Tang et al., 2023).Source: §3.1 · pp. 4–5
3.2 Multi-Agent for Automatic Feedback
- AutoFeedback uses a generation agent followed by a validation agent that checks feedback for issues such as over-praise and over-inference.Figure 1 presents the architecture in relation to a single-agent system; the workflow passes generated feedback for validation and revision when needed.Source: §3.2 · pp. 5–6, §3.1 · p. 5
- The framework is presented as a potential way to improve feedback effectiveness and assessment efficiency in large-scale educational settings.The proposed approach combines generative language understanding with systematic review of outputs.Source: §3.2 · p. 6
4 METHODS
The methods evaluate single- and multi-agent feedback on a balanced sample of middle-school science responses, using iteratively developed prompts and human coding of two targeted feedback errors.
4.1 Item and Student Responses
- 240 responses—120 Beginning and 120 Proficient—were randomly selected from 845 human-scored responses, with 30 additional responses used for prompt development.The assessment asked students to model how thermal-energy transfer affects particle motion or temperature.Source: §4.1 · p. 6
4.2 Experiments
- The single-agent baseline used Agent 1 alone, while AutoFeedback passed its output to Agent 2 for review and revision; both agents used GPT-4o.Responses were processed independently and in parallel to simulate multiple student submissions.Source: §4.2 · pp. 6–7
4.2.1 Agent 1: Feedback Generator
- Agent 1’s prompt was iteratively developed and reviewed, combining a teacher role, task, item rubric and response, learning context, and feedback criteria.Researchers tested and revised the prompt with 30 sampled responses until reaching saturation; criteria included understandable, balanced, actionable, and encouraging feedback.Source: §4.2.1 · p. 7
4.2.2 Agent 2: Feedback Validation and Modifier
- Agent 2 evaluates Agent 1’s feedback and revises it when needed, using the item and context plus descriptions of over-praise and over-inference.Its task prompt asks whether revision is needed and requests reasons.Source: §4.2.2 · p. 7
4.3 Data Analysis
- Two assistant professors coded feedback for over-praise and over-inference; after 100% inter-rater reliability on 30% of cases, one rater coded the rest.The resulting human analysis was used to compare single-agent and multi-agent performance.Source: §4.3 · p. 8
5 FINDINGS
Single-agent feedback exhibited over-praise and over-inference, while AutoFeedback significantly reduced feedback with these problems. Some issues nevertheless remained or were not fully corrected.
5.1 Feedback Generated by Single Agent
- Single-agent feedback was labeled over-inference in 27.20%, over-praise in 15.42%, and both in 9.58% of cases.The reported coding results are presented in Table 1.Source: §5.1 · p. 8
- Examples show praise for gibberish, inferred temperature reasoning absent from an answer, and inferred mastery despite a missing connection to the assessed concept.In the over-praise example, AutoFeedback revised feedback after its second agent flagged praise given to a gibberish response.Source: §5.1 · p. 8, §5.2 · p. 9
5.2 Feedback Generated by AutoFeedback
- AutoFeedback generated significantly less feedback with over-praise and/or over-inference than the single-agent system (p < 0.001).Table 1 presents the comparison; the authors report that some multi-agent responses still had these issues or were not fully revised.Source: §5.2 · pp. 9, 11, Table L · p. 9
- With AutoFeedback, over-praise fell from 15.42% to 1.2%, over-inference to 7.08%, and feedback with both problems to 0.83%.For over-praise, the system identified the issue and revised the feedback to better align it with the student response.Source: §5.2 · pp. 9–10
- Some multi-agent responses still had over-praise or over-inference; Agent 2 sometimes detected problems but did not fully revise them or evaluated feedback incorrectly.Source: §5.2 · p. 11
6 CONCLUSIONS AND DISCUSSION
AutoFeedback reduced over-praise and over-inference relative to a single agent, using one agent to generate feedback and another to identify issues and revise it. The discussion situates this result amid concerns about GenAI feedback quality and prior educational uses of multi-agent systems.
6 CONCLUSIONS AND DISCUSSION
- Over-praise decreased by 14.17% and over-inference by 20.21% compared to a single agent; feedback containing both issues (9.58%) was reduced by 8.75%.These comparisons are reported for AutoFeedback versus a single agent.Source: §6 · p. 11
- Over-praise can misalign praise with performance and may reduce motivation, while over-inference inaccurately evaluates performance and can mislead learning or distort self-perception.The discussion also cites studies associating over-praise with poorer school performance and higher depression, and notes LLM accuracy and language limitations as possible sources of over-inference.Source: §6 · p. 11
- Prior comparisons found human feedback higher quality than ChatGPT on four of five criteria and rated useful by 88% of participants versus 59% for ChatGPT feedback.The five criteria included criteria-based feedback, clear improvement directions, accuracy, prioritization of essential writing features, and supportive tone.Source: §6 · p. 11
- The authors attribute improved feedback quality to agents specializing in collaborative generation, review, and revision, which they say provides more accurate feedback than a single agent.They frame quality feedback free of over-praise and over-inference as the system’s intended end product.Source: §6 · p. 11
- AutoFeedback pairs an agent that generates feedback with one that identifies issues and revises it, and the authors describe this as the first GenAI multi-agent study for constructive science responses.The discussion contrasts this focus with prior multi-agent educational systems and a multiple-choice feedback approach.Source: §6 · pp. 11–12
- Earlier educational multi-agent applications included adaptive learning, recommender systems, and intelligent tutoring systems; the cited multiple-choice approach did not provide automatic feedback as generally understood.The authors suggest that feedback detection and revision strategies could potentially address other feedback-quality issues.Source: §6 · p. 12
7 LIMITATIONS AND FUTURE STUDY
The authors identify remaining feedback-quality issues and study-scope constraints, and call for broader assessment coverage and evaluation in authentic educational settings.
7 LIMITATIONS AND FUTURE STUDY
- Agent 2 did not always detect over-praise and over-inference accurately, and some identified issues remained unresolved in final feedback.The authors say these limitations warrant further work to improve automatic feedback.Source: §7 · p. 12
- The study used student responses to only one item, and the authors say this limited information made it hard for ChatGPT to generate truly personalized learning strategies.They note that highly personalized suggestions are difficult for human educators as well.Source: §7 · p. 12
- The authors call for more assessment items and research on automatic feedback’s real-world impact on student learning in authentic educational settings.They also recommend further exploration of prompt engineering and multi-agent design.Source: §7 · p. 12
APPENDIX
The appendix captions identify prompts for two agents on a sample assessment item.
APPENDIX
- The appendix includes a sample-assessment-item prompt for Agent 1 and a prompt for Agent 2.The captions identify the prompts but do not describe their contents.Source: APPENDIX · pp. 13–14