PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Scale AI Grading of Handwritten Physics Assessments: Score Agreement and Olympiad Team Selection Outcomes

Praveen Pathak, Siddharth Tiwary, Charudatt Kadolkar, Vijay Singh, David Rakestraw, Shirish Pathare, Anwesh Mazumdar

arXiv:2608.20521v1physics.ed-phcs.AIcs.CY

TL;DR

Handwritten physics grading raises the question of whether AI can agree with official scores when assessments determine high-stakes outcomes. This study evaluated rubric-based AI grading and found strong total-score agreement and recovery of the official five-student Olympiad team.

  • Problem

    Evidence remained open on whether AI grading can agree with examiners in handwritten physics assessments where small score differences affect rankings, selection, and medals.

    Source: §I · p. 1

  • Method

    The study compared two rounds of AI grading with official scores across handwritten physics assessments, using the same detailed rubrics while withholding human marks from the AI.

    Source: §I · p. 2, B. AI grading procedure and grading… · p. 3

  • Results

    AI grading showed strong agreement with official total scores and recovered the same top-five group as human examiners for OE2, though it ranked students differently.

    Source: §VI · p. 13

  • Takeaways & Limitations

    AI is best used as a second reader or targeted audit tool under examiner control, with detailed rubrics and review of flagged or borderline cases.

    Source: §VI · p. 14

    A separate human–human regrading experiment was outside the study’s scope, so the analysis used moderated official scores as its human reference.

    Source: §VI · p. 13

Abstract

from arXiv · show

Multimodal AI can read handwritten physics solutions, but high-stakes grading requires agreement with official scores and outcomes. This study evaluated GPT-5.5-based grading on 10364 scanned pages from 520 handwritten submissions by 416 unique candidates or students across three assessments: a national Physics Olympiad theory examination, the final Olympiad selection camp with theory and experiment components, and a university quantum-mechanics examination. Each submission was graded twice by AI using the official rubrics. The second round used revised page-by-page and evidence-location instructions developed after first-round disagreement analysis. During grading, AI did not see official human marks or AI--human comparisons. Total-score correlations with official marks were high (0.91--0.97). For the final Olympiad selection, AI recovered the same five-student team as official grading. The second round improved aggregate question-part agreement, especially where first-round disagreements were larger. The main difficulty remained exact partial-credit grading, especially in experimental work. Reliable AI grading therefore depends on detailed rubrics and should be used as a second reader or audit tool under examiner control.

I. INTRODUCTION · II. DATA AND METHODS · A. Assessments and reference scores

The study tests whether rubric-guided AI can agree with official grading across handwritten physics assessments, including high-stakes Olympiad selection and a university examination. It considers both score agreement and the consequences of using scores for selection.

I. INTRODUCTION

  • Prior physics studies found useful grading but retained item-level errors and human-review needs, while mathematics studies report strong agreement alongside item and visual-representation limits [8] [9] [12] [13] [14].Scoring also depends on prompt design, rubric, item context, and model choice.Source: §I · p. 1
  • Olympiad grading is consequential because small score differences can affect rankings, team selection, and medals, while examiners assess reasoning as well as final answers.A correct answer may follow faulty reasoning, and an incorrect answer may still show a sound method.Source: §I · p. 1
  • The assessments span Olympiad theory, Olympiad theory and experiment, and university quantum mechanics, covering derivation, data analysis, diagrams, and conceptual reasoning.OE1 identifies a top cohort for the next stage; OE2 helps select the International Physics Olympiad team.Source: §I · pp. 1–2

II. DATA AND METHODS · A. Assessments and reference scores

  • AI grading occurred retrospectively after Olympiad results and university grades were released, leaving official results unchanged; OE2 rankings combine theory and experiment at 60:40.Normalized OE2 totals are equivalent to 0.6T + 0.4E; analyses compare grading rounds and assess total-score, selection, question-part, and question-type agreement.Source: §I · p. 2, A. Assessments and reference scores · p. 2
  • The Olympiad program progresses from 10 292 participants in the OE1 year to about 350 at the next stage, about 35–40 at camp, and a final team of five.OE1 and OE2 come from different years because of submission availability and scanning history, and test two high-stakes stages of the same selection system.Source: A. Assessments and reference scores · p. 2
  • The study processed 10 364 scanned pages; QM comprised 40 handwritten submissions, and all exam submissions were handwritten.OE1 and OE2 each represent one year and stage of the Olympiad program, while QM is an end-of-semester university examination.Source: A. Assessments and reference scores · p. 2
  • Final official scores, checked by a second examiner and settled before release, serve as the reference; OE1 permits regrading requests, while OE2 and QM allow score review with examiners.Because OE1 and OE2 are selection-oriented, the analysis also considers rank and top-group agreement.Source: A. Assessments and reference scores · p. 2

B. AI grading procedure and grading rounds

AI grading used anonymized handwritten submissions and the same official rubrics as human grading. The study included two full grading rounds and a focused refinement stage prompted by Round I disagreements.

Grading procedure

  • The AI received exam questions, solutions, rubrics, and anonymized submissions, with visible human scores and comments erased before grading.Rubrics specified partial credit, common errors, alternative valid paths, and error-carried-forward rules; the main comparison used GPT-5.5 Thinking at high effort.Source: B. AI grading procedure and grading… · p. 3

Round I

Focused refinement

  • Focused refinement tested clearer physics and scoring instructions on three repeated or large Round I disagreements, one each from QM, OE1, and OE2.These diagnostic runs examined whether more explicit instructions could reduce AI–official score differences.Source: B. AI grading procedure and grading… · p. 3

Round II

  • Round II used revised instructions, refined rubrics for three parts, and page-by-page grading that located credited evidence in long submissions.The AI saw no Round I or human scores, comments, totals, ranks, selection status, or AI–human comparisons; output also recorded confidence labels and review flags.Source: B. AI grading procedure and grading… · p. 3

C. Evaluation measures · III. AGREEMENT AND SELECTION RESULTS · A. Total-score agreement

AI and official graders showed high agreement on total scores across the three assessments, with the revised workflow reducing some score differences. The evaluation also defines the comparison unit and score-difference measures used in the results.

C. Evaluation measures

  • The analysis counted 7,058 AI–official comparisons, each defined as one student response to one official question part or subpart with its own maximum mark and grading scheme.Table III defines the quantities used in the analysis; D and MAD are percentages of the relevant maximum possible score unless raw marks are explicitly stated.Source: C. Evaluation measures · p. 3, Table III · p. 4

A. Total-score agreement

  • RII used the full revised workflow—page-by-page checking, evidence notes, stricter permitted-score checks, confidence and review fields, and clearer instructions for selected questions.The three focused refinements discussed later covered about 3.6% of the combined official rubrics, so RI–RII differences reflect the full revised workflow rather than those questions alone.Source: A. Total-score agreement · p. 4
  • Total-score MAD fell from 7.1% to 4.8% for OE1 and from 9.4% to 3.8% for QM; RI–RII changes were smaller for OE2, which already had strong agreement.MAD measures the size of the remaining score difference.Source: A. Total-score agreement · p. 4, Figure 1 · p. 5
  • In RI, D was positive in all four total-score comparisons, indicating general AI over-awarding; RII reduced the positive shift in OE1 and QM, consistent with earlier concerns.D is reported as a percentage of the relevant maximum possible score unless raw marks are stated.Source: A. Total-score agreement · p. 4, Table III · p. 4
  • Pearson correlations were 0.91–0.97 in RI and 0.93–0.96 in RII, with student scores lying close to the equal-score diagonal across the assessments.Fig. 1 shows OE1, QM, and the OE2 theory and experiment components; each plotted point represents one student.Source: A. Total-score agreement · p. 4, Figure 1 · p. 5
  • Residual plots show RI differences more often above zero and RII less shifted upward, with differences spread across scores rather than confined to low scores or near a cutoff.The fitted slopes are visual guides and should not be over-interpreted because the plotted difference includes the human score.Source: Figure 2 · p. 5, A. Total-score agreement · p. 5

B. Top-cohort and selection agreement

Outcome checks show AI recovered the selected five-student Olympiad team and closely matched released course grades, while exact ranks near the OE1 cutoff still required human grading.

OE1 cohort selection

  • 31/40 and 40/50: Each OE1 round recovered that many students from the human top 40 and top 50, while top-10 overlap rose from 3/10 in RI to 7/10 in RII.This agreement supports identifying the top group, but human grading remains necessary for exact ranks near a cutoff.Source: B. Top-cohort and selection agreement · p. 6

OE2 team selection

  • 5/5: In both rounds, AI recovered the same five students as the human-selected OE2 team, though their internal ranking differed.Both rounds also recovered 9/10 of the human top 10 and 19/20 of the human top 20; the final ranking combined five exam components per candidate.Source: B. Top-cohort and selection agreement · p. 6

QM course grades

OE2 AI-mode checks

  • 5/5: Pro Standard also recovered the human OE2 top-five group, with 9/10 top-10 and 19/20 top-20 overlap.The Pro Extended pilot covered only six selected OE2 submissions with large earlier disagreements; it modestly reduced some, especially in theory, while over-awarding persisted. Earlier work reported run-to-run grading variation, and mode checks were not repeated for OE1 or QM.Source: B. Top-cohort and selection agreement · p. 6

C. Question-part agreement and partial credit · IV. STRENGTHS AND DISAGREEMENTS IN AI GRADING

RII improved question-part agreement over RI, including exact matches and broad credit-band recognition, but exact partial-credit scoring remained the hardest case. The comparisons use official question parts, with differences measured in raw points and interpreted alongside normalized MAD and credit-band analyses.

C. Question-part agreement and partial credit

  • Question-part differences were measured as d = |AI −Human| in raw points, with 0.5-point increments and agreement bands considered alongside normalized MAD and credit-band analyses.Maximum marks differed across parts, so the d bands should be interpreted with those additional analyses.Source: C. Question-part agreement and partial… · p. 6
  • Across all 7058 official question parts, RII raised exact agreement with human scores from about 63% under RI to about 70%, while reducing differences greater than one point from about 13% to 7%.Source: C. Question-part agreement and partial… · p. 6
  • In RII, AI kept 80% of human-zero parts at zero, 86% of human-partial parts in the partial-credit range, and 87% of human-full parts at full credit.Partial-credit parts accounted for 40% of available points.Source: C. Question-part agreement and partial… · p. 7, Figure 4 · p. 7
  • For human-partial parts, exact agreement rose from 24.6% in RI to 32.8% in RII, while MAD among parts where both graders awarded partial credit fell from 0.92 to 0.72 raw marks.Exact partial-credit scoring remained the most difficult case despite these improvements.Source: C. Question-part agreement and partial… · p. 7
  • When AI moved a human-zero part into partial credit, its average award was about one raw mark, and this occurred less often in RII than RI.RII improved both broad category recognition and partial-credit calibration.Source: C. Question-part agreement and partial… · p. 7
  • The section next examines representative successes, experimental grading, differences across question types, and recurring disagreement patterns.Source: C. Question-part agreement and partial… · p. 7

A. Examples of physics-specific grading · B. Experimental grading

The examples show AI grading physics evidence across handwritten derivations, sketches, tables, and graphs, while experimental marks still require human review when components must be physically consistent.

A. Examples of physics-specific grading

B. Experimental grading

  • OE2 experimental grading involved tables, calculations, graphs, fit lines, uncertainty estimates, and conclusions, with RII recording components more separately than some broad official marks.The comparison used corresponding human markings in the official grading scheme.Source: B. Experimental grading · p. 8
  • AI often recognized experimental tables, plotted points, fit and limiting lines, slopes, and reported values, but struggled when marks depended on consistency among data range, fit, uncertainty, and conclusion.Human review remains important for judging that consistency.Source: B. Experimental grading · pp. 8–9
  • The AI evaluated graph evidence beyond mere presence, identifying fit and limiting lines and agreeing with partial credit when plotted points lacked a usable fit, slope, or uncertainty construction.Source: B. Experimental grading · p. 9, Figure 7 · p. 9

C. Agreement across question types

Agreement extended across numerical, conceptual, derivation, diagram, and data/graph tasks, and RII had lower MAD than RI for every question type. Exact partial-credit differences remained sensitive to whether responses met specific rubric conditions.

C. Agreement across question types

  • Each official question part was assigned one primary question-type label to compare OE1, OE2, and QM; for OE2 experimental sections, the label reflected the main task.The labels were broad and captured the main type of judgment required.Source: C. Agreement across question types · p. 9, Table V · p. 9
  • Across derivations, numerical answers, conceptual reasoning, diagrams, and data/graph tasks, AI scores generally rose with human scores, but conceptual reasoning had the greatest RI over-scoring.Figure-data and figure-diagram tasks were sensitive to whether the visual or experimental evidence met the specific grading condition.Source: C. Agreement across question types · p. 9
  • MAD was lower in RII for every question type across 7,058 official question parts.The figure reports MAD as mean absolute difference relative to each part’s maximum possible score.Source: C. Agreement across question types · p. 9, Figure 8 · p. 10
  • RII gains extended beyond explicitly refined questions, with some of the largest improvements in figure-data and figure-diagram parts and gains also in numerical, conceptual, and derivation parts.Reviewed cases suggest page-by-page checking and clearer credit conditions help when scoring depends on evidence in a specific response part.Source: C. Agreement across question types · pp. 9–10

D. Recurring sources of disagreement · V. IMPROVING AND REVIEWING AI GRADES · A. Focused rubric refinement

Focused, physics-specific rubric refinements reduced AI–official score disagreements in selected cases, while reviewed disagreements had identifiable causes. The findings support stating scoring conditions, deductions, acceptable alternatives, and carried-forward-error rules explicitly before AI grading.

D. Recurring sources of disagreement

A. Focused rubric refinement

  • The focused test regraded three problem cases—QM question 3(b), OE1 Q2, and OE2 T2 Q1(f)—with question-specific instructions, and explicit physics and scoring conditions reduced several disagreements.Source: A. Focused rubric refinement · pp. 10–11
  • For an OE2 thermodynamic-cycle question, explicit branch-curvature and temperature conditions stopped AI from awarding shape credit to the identified incorrect-curvature cases.For the Fig. 9 submission, the AI noted that the upper branch had the wrong curvature and a process arrow was inconsistent.Source: A. Focused rubric refinement · p. 11, Figure 9 · p. 11
  • In the Grover example, a plausible circuit-style sketch lacked the required two-dimensional rotation diagram and operator action; the revised rubric credited the qualitative idea but rejected the sketch as the required diagram.Source: A. Focused rubric refinement · p. 11, Figure 10 · p. 11
  • For the Grover-rotation example, refinement reduced MAD from 1.1 to 0.4 marks, raised Pearson’s r from 0.70 to 0.82, exact agreement from 15% to 45%, and agreement within 0.5 mark from 35% to 80%.RII matched the official score of 0.5 out of 3 points, compared with RI’s 2 out of 3.Source: A. Focused rubric refinement · pp. 11–12, Figure 11 · p. 12
  • For 72 OE1 Q2 submissions selected for large Round I differences, MAD fell from 38% to 17% of the part’s maximum score and Pearson’s r rose from 0.58 to 0.83.The refined rubric required the relevant physical criterion for each option and limited credit for unsupported conclusions.Source: A. Focused rubric refinement · p. 12, Figure 11 · p. 12
  • Reliable AI grading depends on detailed, physics-specific rubrics that explicitly state mark-awarding conditions, common deductions, acceptable alternatives, and carried-forward-error rules.Source: A. Focused rubric refinement · p. 12

B. Confidence and human-review flags

In RII grading, high-confidence question parts were consistently closer to official scores, but confidence flags still missed substantial disagreements. The findings support using flags to prioritize human review, not to replace examiner judgment.

B. Confidence and human-review flags

  • MAD% was lower for high-confidence parts in every comparison: OE1 6.0% vs 17.0%, QM 9.8% vs 28.2%, and OE2 theory 5.3% vs 27.1% and experiment 6.9% vs 25.6%.Fig. 12 compares high-confidence parts with medium- or low-confidence parts.Source: B. Confidence and human-review flags · p. 12, Figure 12 · p. 13
  • Of 1724 high-confidence OE2 parts accepted without review, 87 still differed from official scores by more than one point, supporting triage rather than automatic acceptance.The OE2 analysis included 1898 official question parts: 1482 theory and 416 experiment; the study evaluated the AI’s own review flags without a separately trained routing model.Source: B. Confidence and human-review flags · p. 13

VI. DISCUSSION AND CONCLUSIONS

AI grading showed strong agreement with official scores and recovered the official OE2 top-five group, while QM grades were mostly reproduced exactly. The study recommends AI as a second reader or audit aid under detailed rubrics and examiner control.

VI. DISCUSSION AND CONCLUSIONS

  • 34 of 40 released QM grades were reproduced exactly in RII, and all 40 were within one grade step; both OE2 rounds recovered the official top five, though rankings differed.For OE1, AI identified most of the larger top group in both rounds.Source: §VI · p. 13
  • Official scores were moderated, but the study did not include human–human regrading, limiting direct comparison of AI–official and human–official differences.For 20 physics responses scored by four instructors, reported human-only ICCs were 0.88 with a holistic rubric and 0.94 with a checklist; mid-level responses scored 0.28 under the holistic rubric and 0.89 with the checklist [28]. Independent second-human grading on a representative subset is a proposed next step.Source: §VI · p. 13
  • AI often read handwritten work, recognized correct physics, and located relevant evidence, but was more willing than examiners to award partial credit for incomplete or visually plausible work.This tendency was especially evident in conceptual reasoning, diagrams, and experimental graph or data tasks, where human examiners rely more on context and judgment.Source: §VI · p. 13
  • Explicit physics credit conditions improved targeted grading, but RII was revised after inspecting RI disagreements and needs confirmation on an untouched examination or holdout set.OE2 total-score agreement alone could conceal differences in marks for individual parts; its RI–RII changes were smaller and mixed, although question-part and experimental analyses improved.Source: §VI · p. 13
  • High-confidence responses had smaller average score differences across all examinations, but confidence and review fields should prioritize examiner review rather than replace it.Some high-confidence responses still differed from official marks, so examiner checking remains necessary before final acceptance.Source: §VI · p. 13
  • Page-by-page grading with evidence locations helped long submissions, auditability, and agreement, but increased AI grading time.OE1 grading typically involved about 20 graders over about 2.5 days, while OE2 involved about 25 people over four days. Token-use and processing-time records were incomplete, and human time logs were unsuitable for direct comparison, so these figures provide workload context rather than a measured time comparison.Source: §VI · pp. 13–14
  • Reliable use requires explicit rubrics and examiner review; the more intensive Pro modes fit best as targeted audits, while AI can serve as a second reader and aid to selection decisions.Thinking High already reproduced the OE2 top-five outcome, and Pro Standard and limited Pro Extended checks left the grading picture largely unchanged; their outcome-level performance was comparable to Thinking High.Source: §VI · p. 14

DATA AND MATERIALS STATEMENT

Anonymized scores, task categories, prompts, and analysis scripts may be shared with approval, while raw handwritten submissions remain confidential.

DATA AND MATERIALS STATEMENT

  • Anonymized scores by question part, task categories, prompts, and analysis scripts may be shared subject to approval from relevant authorities and institutions, but raw handwritten submissions remain confidential.Confidentiality is due to student privacy and examination confidentiality.Source: DATA AND MATERIALS STATEMENT · p. 14
Loading 2608.20521v1…