PaperlayerView exact revision on arXiv

Source-linked AI summary

EDU-CIRCUIT-HW: Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutions

Weiyu Sun, Liangliang Chen, Yongnuo Cai, Huiru Xie, Yi Zeng, Ying Zhang

arXiv:2602.00095v3cs.CVcs.AIcs.CY

TL;DR

MLLMs still lack reliable evaluation for university STEM handwritten solutions, whose intertwined handwriting, mathematics, diagrams, and reasoning are poorly represented by existing benchmarks and incompletely probed by downstream tasks. The paper introduces EDU-CIRCUIT-HW and evaluates recognition fidelity together with auto-grading using expert-verified transcriptions and grading reports, revealing pervasive latent recognition errors that worsen under finer grading criteria. A case study uses these error patterns to support more robust grading with limited human intervention.

  • Problem

    Existing benchmarks and downstream evaluations inadequately capture MLLMs’ understanding of authentic university STEM handwritten solutions and may leave recognition errors unobserved.

  • Method

    The paper introduces EDU-CIRCUIT-HW and evaluates MLLM recognition and downstream auto-grading using expert-verified transcriptions, grading reports, error taxonomy, and diagnostic workflows.

  • Results

    The evaluation reveals pervasive latent recognition errors beneath grading outcomes, with failures becoming more detrimental as grading granularity increases.

  • Takeaways & Limitations

    Reliable MLLM-based student-solution understanding remains a distant goal, while identified error patterns can support detection and suppression of unseen recognition failures.

  • Takeaways & Limitations

    The dataset focuses on a university circuit-analysis course, so its conclusions may not fully generalize to handwritten tasks with substantially different diagram modalities.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) hold significant promise for revolutionizing traditional education and reducing teachers' workload. However, accurately interpreting unconstrained STEM student handwritten solutions with intertwined mathematical formulas, diagrams, and textual reasoning poses a significant challenge due to the lack of authentic and domain-specific benchmarks. Additionally, current evaluation paradigms predominantly rely on the outcomes of downstream tasks (e.g., auto-grading), which often probe only a subset of the recognized content, thereby failing to capture the MLLMs' understanding of complex handwritten logic as a whole. To bridge this gap, we release EDU-CIRCUIT-HW, a dataset consisting of 1,300+ authentic student handwritten solutions from a university-level STEM course. Utilizing the expert-verified verbatim transcriptions and grading reports of student solutions, we simultaneously evaluate various MLLMs' upstream recognition fidelity and downstream auto-grading performance. Our evaluation uncovers an astonishing scale of latent failures within MLLM-recognized student handwritten content, highlighting the models' insufficient reliability for auto-grading and other understanding-oriented applications in high-stakes educational settings. As a potential solution, we present a case study demonstrating that leveraging identified error patterns to preemptively detect and correct recognition errors, while requiring only minimal human intervention (e.g., routing 3.3% of assignments to human graders and the remainder to the GPT-5.1 grader), can effectively enhance the robustness of the deployed AI-enabled grading system. Code and dataset are available in this GitHub repo: https://gt-learning-innovation.github.io/CIRCUIT_EDU_HW_ACL.

1 Introduction

EDU-CIRCUIT-HW addresses the difficulty of evaluating MLLMs on authentic university-level STEM handwritten solutions, where intertwined handwriting, mathematics, diagrams, and reasoning challenge recognition. It evaluates recognition fidelity alongside downstream grading and finds latent errors that can remain hidden under coarse assessment criteria.

  • University-level STEM handwritten solutions combine unconstrained handwriting, mathematical derivations, and hand-drawn diagrams into a challenging visual language.
  • Existing benchmarks often isolate visual elements, simplify diagrams, or target lower-difficulty K–12 mathematics, limiting insight into realistic college STEM understanding.
  • Downstream evaluations can miss recognition errors when those errors do not affect the specific task outcome, even though they may matter for other applications.
  • The dataset combines near-verbatim transcriptions, grading reports, error taxonomy, and cascading-impact analysis to assess recognition and auto-grading together.
  • Recognition errors are pervasive beneath grading outcomes, remaining dormant under coarse criteria but becoming more detrimental as grading granularity increases.
  • EDU-CIRCUIT-HW provides 1300+ authentic university-level STEM solutions for evaluating MLLM visual understanding in real-world educational scenarios.

2 The EDU-CIRCUIT-HW Dataset

EDU-CIRCUIT-HW is a university-level circuit-analysis dataset designed to evaluate handwritten solution recognition and downstream auto-grading. It contains authentic student submissions with separate observation and test sets, including expert-verified recognition for the observation set.

  • EDU-CIRCUIT-HW contains 1334 handwritten homework solutions from 29 undergraduate students in a university circuit-analysis course.The problems range from basic circuit concepts to first- and second-order circuit analyses.
  • The observation set contains 513 solutions with manually verified grades and recognition results, including transcriptions of handwritten content and descriptions of non-textual elements.
  • The test set contains 821 solutions with ground-truth grades but no expert-verified recognition.

3 Handwritten Solution Recognition and Automated Grading Framework

This framework recognizes handwritten solutions, detects discrepancies against expert-verified transcriptions, categorizes recognition errors, and evaluates their effects on rubric-based auto-grading.

  • 3.1 Handwritten Solution Recognition: MLLM recognition results are compared with expert-proofread transcriptions to identify discrepant sentences and equations as recognition errors.Experts manually check recognition results against original handwritten images and rectify errors in Markdown files.
  • 3.1.1 LLM-as-a-Judge Method Validation: Over 186 samples from three MLLMs are used to validate automated error detection against expert annotations at sample and item levels.Item-level evaluation uses precision, recall, and F1, while sample-level evaluation checks whether any recognition error is present.
  • 3.1.2 Taxonomy on Recognition Errors: Recognition errors are organized into four categories to analyze their distribution and reveal weaknesses across different MLLM recognitions.The taxonomy supports automated categorization of discrepant items and fine-grained analysis of recognition capability.
  • 3.2 Downstream Task: Automated Grading: The framework is designed to examine how upstream recognition errors affect downstream auto-grading and help reduce their negative effects.The downstream analysis treats auto-grading as the target task for studying recognition-error impact.
  • 3.2 Downstream Task: Automated Grading: Auto-grading uses recognized student solutions, problem context, reference solutions, and problem-specific rubrics to assess performance against expert-verified grading reports.The rubric covers five assessment perspectives, and violations can incur separate score deductions.
  • 3.2 Downstream Task: Automated Grading: Grading agreement is measured through Binary Agreement, Type Agreement, and Point Agreement, with Point Agreement requiring exact error types and deductions.These levels progressively assess mistake detection, error-type consistency, and exact point deductions.

4 Experiments

The experiments evaluate MLLM handwriting recognition and its downstream grading impact using fine-grained error analysis, revealing that task-level grading can conceal substantial recognition failures.

  • Experiment Setup: Six MLLMs and an expert-transcription oracle are evaluated as recognizers, with GPT-5.1 serving as the grader across all settings.The evaluation includes five closed-source commercial models, one open-source model, and a graduate-grader comparison.
  • Metrics: SER measures the proportion of solutions containing recognition errors, while AEC measures the average item-level error count per solution.Grading is assessed through binary, type, and point agreement with expert reports.
  • Recognition and Grading Results: 37.62% SER for Gemini-3-Preview corresponds to only 7.60% EIR, showing that downstream grading can mask many recognition errors.As SER and EIR rise toward Qwen3-VL-8B-Thinking, agreement gaps increase from 12.86% for binary agreement to 17.16% for type and 17.35% for point agreement.
  • Recognition and Grading Results: Stricter type- and point-level grading exposes larger weaknesses in MLLMs, while human graduate grading remains more consistent across rubric granularities.MLLMs can outperform the graduate grader on binary agreement, but even the strongest MLLMs lag behind on finer-grained agreement.
  • Recognition Error Taxonomy: Symbolic and Character errors are most prevalent and have EIRs around 20%, whereas Diagrammatic and Textual & Logical errors frequently have EIRs below 10%.Qwen3-VL-8B-Thinking records 98 Diagrammatic errors, compared with 103 for Gemini-2.5-Pro; commercial models generally perform better on Textual & Logical errors.

5 Case Study

The case study integrates recognition-error diagnostics into a human-in-the-loop regrading workflow that selectively reassesses solutions after initial grading.

  • Regrading Workflow: Previously identified recognition-error patterns prompt an LLM to detect potential errors in test-set solutions representing unseen deployment scenarios.Solutions receiving point deductions in the first round are subsequently regraded.
  • Results: Table 6 reports performance and routing statistics before and after regrading, including the percentages assigned to LLMs and teaching assistants.These statistics quantify the trade-off between automated reassessment and human review.
  • Deployment Design: The regrading module is designed to limit intervention because recognition errors primarily produce false-positive penalties and point deductions affect a small fraction of solutions.GPT-5.1 implements the grader, re-grader, and recognition-error detector.

6 Conclusions

The paper introduces EDU-CIRCUIT-HW as a real-world benchmark for evaluating MLLM recognition and grading, and validates human-in-the-loop mitigation of recognition failures.

  • Contributions: EDU-CIRCUIT-HW contains over 1,300 authentic university-level STEM handwritten student solutions with expert-verified transcriptions and grading reports.The benchmark evaluates both visual understanding and its cascading impact on downstream auto-grading.
  • Findings: The evaluation uncovers latent recognition errors that stricter and more professional rubrics may expose, creating challenges for reliable MLLM-based educational applications.The conclusion frames these errors as a significant deployment challenge rather than only a recognition-measurement issue.
  • Mitigation: A human-in-the-loop grading workflow mitigates recognition failures and significantly enhances system robustness with minimal human intervention.The paper presents this workflow as a response to the benchmark’s identified reliability challenges.

7 Data Scope: Beyond Circuit Analysis

Although EDU-CIRCUIT-HW comes from a circuit-analysis course, its solutions require interleaved reasoning across multiple foundational STEM disciplines.

  • Scope: The dataset is not limited to isolated circuit notation because its problems require handwriting, diagrams, mathematical derivations, and textual reasoning together.The authors explicitly position the dataset as broader than a narrow subset of STEM challenges.
  • Example 1: One example combines KCL, KVL, a second-order differential equation, Laplace-domain characteristic equations, and exponential-term solutions.The solution determines inductor current for t > 0 in a circuit containing resistors, a capacitor, an inductor, and a voltage source.
  • Scope: The authors state that these examples indicate the benchmark results are not limited to circuit analysis alone.The clarification is intended to guide future researchers’ understanding and use of the dataset.
  • Example 2: A second example requires three complex-valued mesh-current equations in matrix form, combining circuit analysis with linear algebra.The task uses phasor and impedance quantities to determine three mesh currents.

8 Limitations

The study’s conclusions are bounded by its circuit-analysis setting, its focus on auto-grading, and potentially subjective expert grading. The authors identify broader domains, downstream tasks, and assessment protocols as future extensions.

  • Scope: The dataset centers on a university-level circuit analysis course, so findings may not generalize to handwritten solutions with substantially different diagram modalities.Abstract mathematical diagrams, such as complex geometric graphs, are underrepresented.
  • Downstream tasks: The propagation analysis primarily evaluates auto-grading and does not examine other downstream tasks such as visual question answering.Other tasks may have different sensitivities to recognition inaccuracies.
  • Assessment: Grading labels and rubrics were provided by a small number of doctoral experts, leaving potential subtle subjective biases in grading scores.The authors attempted to improve consistency through detailed criteria and verification grounded in mathematical logic and domain expertise.
  • Future work: The authors frame broader subject domains, additional downstream tasks, and alternative assessment protocols as opportunities for future work.These extensions correspond to the study’s stated scope boundaries.

9 Ethics Considerations

The ethics and related-work discussion describes challenges in recognizing complex handwritten STEM work and notes the need for targeted analysis of recognition errors. It also situates the work among OCR, error-correction, and auto-grading research.

  • Recognition challenges: Complex deductions, auxiliary plots, and uncommon symbol combinations degrade handwriting-recognition performance and increase the need for manual verification.Standard text and simple one-line mathematical expressions are easier for OCR and visual-understanding techniques to process.
  • Related approaches: Prior handwriting-text-recognition work includes sequential detect-then-rectify methods and external-LLM correction of mathematical formulas using grammar and semantic context.The cited examples include Greek text recognition and mathematical-formula correction.
  • Error detection: Error-detection methods include BERT-based and confidence-based approaches, but their effectiveness remains underexplored for complex student handwriting under practical constraints.The discussion emphasizes minimizing API calls while maintaining high-recall error detection.
  • Auto-grading: Existing auto-grading agents use foundation models for handwritten submissions, but their opacity and relatively lower performance make them unreliable at the current stage.The discussion contrasts direct end-to-end grading with alternative auto-grading platforms.

B Dataset Details and Ethics Statements

EDU-CIRCUIT-HW combines authentic handwritten circuit-analysis solutions, reference materials, expert transcriptions, and categorized recognition-error records. The appendix documents the collection and transcription procedures through examples and taxonomy prompts.

  • Dataset components: The dataset contains 1334 authentic handwritten solutions, 62 reference answers and rubrics, 513 near-verbatim expert transcriptions, and categorized recognition-error records.The solutions were collected from a university-level STEM course during Spring 2025.
  • Error taxonomy: Recognition errors are categorized with a four-type taxonomy covering Symbol & Character, Structural & Notational, Diagrammatic, and Textual & Logical errors.Figure 5 presents the prompt used to assign errors to these categories.
  • Collection boundary: Some digitally submitted assignments were excluded, so not every student has handwritten solutions for every question.The collection and grading reports were gathered during Spring 2025.
  • Gold-standard construction: Experts reviewed Gemini-2.5-Pro transcriptions against original handwritten images to produce gold-standard results, then categorized identified errors using the taxonomy.The categorized errors will also be released across evaluated models to support further research.
  • Illustrative examples: Figures illustrate the rectification process by contrasting original submissions, initial model transcriptions, and expert-corrected versions.Recognition errors are marked in red and corrections in green.

D.1 Automated LLM-Enabled Recognition Error Detector

The automated detector compares MLLM transcriptions with expert-verified references, while the grading pipeline transcribes submissions before rubric-based evaluation. A regrading module handles suspected recognition errors and routes uncertain cases to humans.

  • Recognition error detection: The detector uses expert-verified transcriptions as gold-standard references for identifying discrepancies in MLLM outputs.Its inputs include target content and the corresponding expert-verified label content.
  • Detector outputs: The detector outputs sample recognition errors, corrections, and detailed evidence by comparing target transcriptions with ground truth.The output structure mirrors the demonstration example.
  • Detector prompting: A one-shot example pairs a GPT-5.1 transcription with expert-rectified ground truth from another model, exposing the detector to representational variability.The example was selected from the test set to prevent data leakage.
  • Vanilla grading pipeline: The vanilla grading pipeline first transcribes handwritten solutions with an MLLM, then evaluates the structured text using an LLM-based grader.GPT-5.1 performs rubric-based scoring and a second dialogue round generates deduction justifications and the final report.
  • Regrading module: The regrading module uses empirical error criteria to detect likely transcription failures and instructs the regrader to disregard flagged content.This module was developed for the case study using recognition errors identified in the observation set.
  • Human escalation: Uncertain cases are referred to a human grader when the detector cannot distinguish a recognition error from a genuine student mistake.The case study uses the course’s doctoral teaching assistant for final evaluation.
  • Model settings: Experiments use official MLLM APIs with fixed temperature settings where applicable, and results are based on one run per model because deployment typically controls inference costs.GPT-5.1 uses its official temperature setting of 1, while other models use temperature 0.

D.5 Examples about Recognition Error Taxonomy

This section organizes recognition errors into four categories—Symbolic & Character, Structural & Notational, Diagrammatic, and Textual & Logical—and illustrates them with examples and detection procedures.

  • Recognition Error Taxonomy: Recognition errors are categorized as Symbolic & Character, Structural & Notational, Diagrammatic, or Textual & Logical.Examples are provided in Tables 8–11, including textual and logical errors in Table 11.
  • Representative Examples: The examples include student solutions combining physics, differential equations, calculus, linear algebra, and complex operations to solve circuit-current problems.Figures 10 and 11 show representative handwritten solutions involving these mathematical and physical foundations.
  • Representative Examples: Transcription examples compare MLLM outputs with expert-rectified versions, marking recognition errors and their corrections in red and green.Figures 12–15 illustrate this comparison for Gemini-2.5-Pro and GPT-5.1 transcriptions.
  • Error Detection: The recognition error detector compares MLLM transcriptions with expert-verified gold standards using one-shot examples and reports item-level errors with rectifications and explanations.Figure 19 shows six corrected item-level errors, while Figure 20 provides explanatory reference examples.
  • Grading and Regrading: The grading workflow uses rubric-based deduction prompts, a second-round explanation step, and regrading that accounts for flagged recognition errors.Figures 21–24 describe the rubric, deduction generation, deduction localization and explanation, and GPT-5.1 regrading prompts.
  • Error Detection: The detector classifies each issue as a handwriting recognition error, a student mistake, or uncertain, with uncertain cases reviewed by a human expert during regrading.Question-specific context such as the problem statement and final answer supports detection, and uncertain cases are reviewed by a doctoral teaching assistant.
Loading 2602.00095v3…