Source-linked AI summary
Beyond Score Accuracy: Examining the Diagnostic Quality of LLM-Generated Structured Assessment in Higher Education
Xi Zhao, Xinyue Jiao, Zhen Xu
TL;DR
Structured LLM grading can appear analytically detailed, but its diagnostic quality has received limited scrutiny. Using JorGPT responses scored by instructors and three commercial LLMs, the study finds redundant rubric dimensions, less frequent misconception detection, and domain variation in grading alignment.
Problem
Whether LLMs' structured rubric scores and feedback provide independent diagnostic information and appropriately identify misconceptions and calibrate tone remains underexamined.
Source: §1 · p. 2Method
The study compares instructor and three commercial LLM evaluations of JorGPT student answers, analyzing score alignment, rubric dimensions, and feedback content and tone.
Source: §2.1 · p. 2, §3.2 · p. 5, §3.3 · p. 5Results
Despite moderate-to-strong grade correlations (r = 0.785 to 0.815), rubric dimensions are redundant (VIF up to 45; inter-dimension r up to 0.99), and misconception detection is lower (5 to 7% vs. 15 to 31%).
Source: §4 · pp. 7–8Takeaways & Limitations
LLM structured scores can resemble analytic rubrics while functioning more like holistic assessments, and feedback may identify omissions without identifying student misunderstandings.
Source: §4 · p. 8The study covers one institution and software engineering course; its rule-based role classification, VADER tone analysis, and specific generation of three LLMs constrain generalizability.
Source: §4 · p. 8
Abstract
from arXiv · showhide
As Large Language Models (LLMs) are increasingly adopted for automated grading and feedback in higher education, their structured outputs, including multi-dimensional rubric scores, detailed feedback comments, and improvement suggestions, create an appearance of thorough analytic evaluation. This study examines whether these outputs deliver what they appear to offer. Using the JorGPT dataset of 3,041 student responses to 50 open-ended computer science questions, scored by both human instructors and three commercial LLMs, we identify three systematic discrepancies between the apparent and actual quality of LLM-generated grading and feedback. The sub-dimension scores are highly correlated (r = 0.82-0.99, VIF up to 45), providing redundant rather than independent diagnostic information. The textual feedback rarely detects student misconceptions (5-7% vs. 15-31% for teachers), functioning as a coverage checklist rather than a diagnostic instrument. The feedback tone remains uniformly positive regardless of response quality, lacking the severity modulation observed in human feedback. Additionally, grading accuracy varies significantly by knowledge domain, with procedural topics most reliable. These findings provide empirically grounded guidance on which aspects of LLM-generated grading and feedback can be relied upon and which require continued human oversight.
1. Introduction
LLMs are increasingly used for grading and feedback, and score-level alignment with human ratings has shown promise. This study asks whether structured scores and comments provide independent, diagnostically useful information and whether their content and tone differ from instructors'.
1. Introduction
- Feedback quality depends on addressing errors rather than merely indicating correctness, while tone and severity shape how students interpret and act on feedback.Programming misconceptions may persist when feedback does not address underlying conceptual errors.Source: §1 · p. 2
- Evidence on LLM assessment has emphasized score alignment, while diagnostic specificity, misconception identification, and tonal calibration in structured grading remain underexamined.Prior work also reports chatbot feedback with irrelevant or incorrect suggestions and sentiment bias across demographic attributes.Source: §1 · pp. 1–2
- Structured outputs combine multiple rubric sub-dimension scores, totals, and detailed comments, resembling expert analytic assessments whose dimensions are intended to provide distinct diagnostic information.Whether LLM outputs provide comparably independent and diagnostically meaningful information has received limited empirical scrutiny.Source: §1 · p. 2
- The study asks whether LLM grades align with instructor grades across knowledge domains, whether sub-dimension scores offer independent information, and how feedback content and tone compare with instructors'.Source: §1 · p. 2
2. Data and Method
The study analyzes JorGPT student answers scored by instructors and three commercial LLMs, using quantitative comparisons of grades and rubric dimensions alongside semantic-role and sentiment analyses of feedback.
2.1 Dataset
- JorGPT contains 3,041 authentic answers to 50 open-ended computer science questions from undergraduate assessments at one Spanish university, with instructor grades and qualitative comments.Responses were anonymized; collection received ethics review and informed consent, and the dataset covers computer science and software engineering topics.Source: §2.1 · p. 2
- Three commercial LLMs independently scored each response using zero-shot prompts and a unified schema, receiving the question, student answer, and instructor-defined ideal answer but not human grades or other responses.Their JSON outputs include four rubric dimensions, a computed total score, and a qualitative comment.Source: §2.1 · pp. 2–3
- The authors classified the 50 questions into six knowledge domains through independent review of question content and ideal answers to support topic-level analysis.The domains are TDD & Testing, Docker & DevOps, Vaadin & Frontend, Spring Boot & REST, JPA & Database, and Architecture & Integration.Source: §2.1 · p. 3
2.2 Analysis Approach
- The analysis compares LLM total scores with instructor grades using correlation, absolute error, and signed bias, then tests domain differences with Kruskal-Wallis and corrected pairwise Mann-Whitney U tests.These metrics characterize agreement rather than causation.Source: §2.2 · p. 3
- OLS models test whether three continuous rubric dimensions predict instructor grades, excluding binary learning_transfer, while VIF and bivariate correlations assess redundancy and individual associations.Source: §2.2 · p. 3
- Feedback comments are classified for praise, missing content, misconception detection, and improvement suggestions, with rule-based patterns checked on a stratified sample.VADER sentiment is compared by student grade band and feedback source; word count is also compared.Source: §2.2 · pp. 3–4
3. Results
LLM grades correlate moderately to strongly with instructor grades, but accuracy varies by knowledge domain. Their rubric dimensions largely overlap, while feedback less often diagnoses misconceptions and shows less tone variation than instructor feedback.
3.1 Overall Grading Alignment and Domain Variation
- Gemini had the lowest MAE (1.22) and near-zero bias (+0.06), while DeepSeek under-graded (bias = – 0.55) and Qwen was lenient (bias = +1.28).All three models had moderate-to-strong correlations with instructor grades (r = 0.785–0.815).Source: §3.1 · p. 4, Table 1 · p. 4
- TDD & Testing was most reliably graded (MAE = 1.05–1.35, r = 0.84–0.87), while Architecture & Integration and Spring Boot & REST had the highest errors.Qwen’s Spring Boot performance was particularly poor (MAE = 2.36, bias = +2.27); alignment varied significantly across domains for all three LLMs (Kruskal-Wallis p < 10⁻⁶).Source: §3.1 · p. 4, Figure 3 · p. 5
3.2 Independence of Sub-Dimension Scores
- Sub-dimension scores were highly redundant: pairwise correlations ranged from 0.82 to 0.99, and Qwen’s Coverage and Justification VIFs were 44.6 and 41.1.Coverage & Accuracy was the dominant predictor across models (β = 0.61–1.08, all p < 0.001); DeepSeek’s Terminology coefficient was negative despite a positive bivariate correlation with human grades.Source: §3.2 · p. 5, Table 2 · p. 5
3.3 Feedback Content and Tone
- Human instructors flagged misconceptions in 15.0% of feedback, versus 4.8 to 7.4% for LLMs; for low-scoring responses, the figures were 31.5% versus 5 to 7%.The comparison is reported in the semantic-role distribution in Table 3.Source: §3.3 · p. 6, Table 3 · p. 6
- For students scoring 0 to 2, LLMs included praise in 47 to 57% of feedback and suggestions in 60 to 77%, compared with 22.6% and 3.3% for instructors.The authors characterize LLM feedback as more of a coverage checklist than a diagnosis of the nature and source of errors.Source: §3.3 · p. 6
- For students scoring 0 to 2, instructor sentiment averaged –0.07 while LLM sentiment ranged from +0.17 to +0.26; instructor sentiment rose to +0.45 for grades 8 to 10, unlike LLM sentiment.Instructor feedback averaged 59 words, compared with 23 to 28 words for LLMs; only teacher feedback length correlated positively with student grade (r = 0.17, p < 10⁻²⁰).Source: §3.3 · pp. 6–7
4. Discussion
LLM grading and feedback show important gaps beyond score alignment: rubric sub-dimensions are largely redundant, and feedback often misses misconceptions while remaining uniformly positive. Grading reliability also varies by domain, and the study’s single-institution, single-course setting limits generalizability.
4. Discussion
- All three LLMs correlate moderately to strongly with human grades (r = 0.785 to 0.815), but grading alignment varies significantly by domain.Procedural topics such as TDD are more reliably assessed than design-thinking topics such as architecture; the authors suggest screening use in procedural domains and continued human assessment where evaluative judgment is required.Source: §4 · p. 7
- Sub-dimension scores are highly collinear (VIF up to 45; inter-dimension r up to 0.99), limiting their value as independent indicators of student strengths or weaknesses.The structured format can create unwarranted confidence in diagnostic specificity.Source: §4 · pp. 7–8
- LLMs detect misconceptions in 5–7% of low-scoring responses, versus 15–31% for instructors, and instead tend to evaluate coverage.Students may learn what they omitted without learning what they misunderstood.Source: §4 · p. 8
- LLM feedback remains uniformly positive regardless of response quality, unlike human feedback, which modulates tone and severity.The absence of tonal modulation may make it harder for students to recognize inadequate responses.Source: §4 · p. 8
- The findings are constrained by data from one institution and one software-engineering course, limiting generalizability to other disciplines and educational contexts.The study also notes that rule-based semantic-role classification may miss subtle expressions, VADER may not capture nuanced tone, and newer LLMs may behave differently.Source: §4 · p. 8