Source-linked AI summary
Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math
Dingjie Song, Tianlong Xu, Yi-Fan Zhang, Hang Li, Zhiling Yan, Xing Fan, Haoyang Li, Lichao Sun, Qingsong Wen
TL;DR
Authentic handwritten scratchwork is difficult to analyze because it combines ambiguous visual information with varied reasoning, while existing systems largely focus on textual answers or producing correct solutions. The paper introduces ScratchMath, a benchmark and dataset for explaining and classifying student errors, then evaluates 16 MLLMs and finds persistent gaps from human experts, with proprietary and reasoning models performing better in key settings. The dataset is limited by its collection from Chinese students on one online education platform.
Problem
Existing educational NLP and MLLM benchmarks provide limited evidence on diagnosing and explaining errors in authentic handwritten mathematical scratchwork.
Method
ScratchMath provides 1,720 student samples with Error Cause Explanation and Error Cause Classification tasks, using human-machine annotation and expert review.
Results
Evaluating 16 MLLMs reveals significant gaps from human experts, with proprietary models outperforming open-source models and reasoning models showing promise for explanation.
Takeaways & Limitations
ScratchMath exposes limitations in current MLLMs’ visual processing and logical understanding for diagnosing student errors.
Takeaways & Limitations
All samples come from Chinese students using a single online education platform, constraining generalizability across languages, demographics, and educational contexts.
Abstract
from arXiv · showhide
Assessing student handwritten scratchwork is crucial for personalized educational feedback but presents unique challenges due to diverse handwriting, complex layouts, and varied problem-solving approaches. Existing educational NLP primarily focuses on textual responses and neglects the complexity and multimodality inherent in authentic handwritten scratchwork. Current multimodal large language models (MLLMs) excel at visual reasoning but typically adopt an "examinee perspective", prioritizing generating correct answers rather than diagnosing student errors. To bridge these gaps, we introduce ScratchMath, a novel benchmark specifically designed for explaining and classifying errors in authentic handwritten mathematics scratchwork. Our dataset comprises 1,720 mathematics samples from Chinese primary and middle school students, supporting two key tasks: Error Cause Explanation (ECE) and Error Cause Classification (ECC), with seven defined error types. The dataset is meticulously annotated through rigorous human-machine collaborative approaches involving multiple stages of expert labeling, review, and verification. We systematically evaluate 16 leading MLLMs on ScratchMath, revealing significant performance gaps relative to human experts, especially in visual recognition and logical reasoning. Proprietary models notably outperform open-source models, with large reasoning models showing strong potential for error explanation. All evaluation data and frameworks are publicly available to facilitate further research.
1 Introduction
ScratchMath addresses the lack of multimodal methods for diagnosing errors in authentic handwritten mathematical scratchwork. It introduces benchmark tasks and evaluates MLLMs, finding substantial gaps from human experts and advantages for proprietary and reasoning models.
- Motivation: Authentic handwritten scratchwork complicates automated feedback through ambiguous symbols, complex spatial layouts, and individualized solution strategies.These challenges matter because scratchwork reflects students’ cognitive processes and can support personalized teaching interventions.
- Research gap: Existing educational NLP mainly analyzes textual answers, while current MLLMs generate correct answers more often than they diagnose student errors.Recent multimodal benchmarks also emphasize structured inputs, error classification, or localization rather than detailed explanations of authentic scratchwork.
- Contribution: ScratchMath provides 1,720 Chinese student mathematics samples for Error Cause Explanation and Error Cause Classification across seven error types.The dataset covers primary and middle school students and uses human-machine annotation with expert labeling, review, and verification.
- Evaluation: The study evaluates 16 leading MLLMs and reports significant gaps from human experts, especially in visual recognition and logical transitions in multistep solutions.Proprietary models outperform open-source models, while large reasoning models show particular promise for explanation.
- Contributions: The work contributes a multimodal educational benchmark, a publicly released authentic-scratchwork dataset, and an evaluation of MLLM capabilities and limitations.These contributions are intended to support research on multimodal error detection and explanation.
2 Related Work
Related work spans AI tutoring, multimodal mathematical reasoning, and handwritten expression recognition, but ScratchMath targets a less-addressed problem: explaining reasoning failures in authentic handwritten solutions.
- AI tutoring: AI-tutor research studies pedagogical alignment, adaptive feedback, and learner-data summarization, while handwritten grading remains difficult on authentic scratchwork.Few studies explicitly pinpoint and explain precise reasoning failures in such solutions.
- Multimodal reasoning: Multimodal reasoning benchmarks show that advanced models can overlook important visual details in diagrams, handwritten derivations, and other visual reasoning tasks.Specialized methods have not fully resolved these interpretation challenges.
- Handwritten mathematics: Handwritten Mathematical Expression Recognition focuses on symbol-level notation accuracy, whereas ScratchMath diagnoses reasoning errors using visual content and mathematical logic.The two goals therefore require different capabilities despite both involving handwritten mathematics.
3 The ScratchMath Benchmark
The ScratchMath benchmark represents each student problem with textual, solution, answer, and scratchwork-image components, supporting explanation and classification of error causes. Its dataset combines filtering, MLLM pre-annotation, expert validation, and quality screening.
- Task Definition: Each benchmark instance contains a problem statement, reference answer, reference solution, student answer, and scratchwork image.This structure connects the student’s written work with the correct solution and answer for error analysis.
- Tasks and Evaluation: Error Cause Explanation produces an open-ended reason for an error, while Error Cause Classification assigns a category from a predefined taxonomy.Explanation quality is assessed with an LLM-as-a-Judge framework, whereas classification requires exact class matching under accuracy.
- Error Taxonomy: The taxonomy contains seven categories: Procedural, Calculation, Logical Reasoning, Transcription, Problem Comprehension, Conceptual Knowledge, and Attention and Detail Errors.All seven error types appear across both primary- and middle-school problems.
- Dataset Construction: The benchmark construction pipeline collects and filters student data, generates preliminary annotations with an MLLM, validates them through expert teachers, and performs post-annotation screening.Filtering removes low-quality or incomplete entries, while indeterminate cases are discarded before finalization.
- Dataset Statistics: The final dataset contains 1,720 problems, including 1,479 primary-school and 241 middle-school problems, spanning multiple mathematical topics.Primary problems emphasize foundational areas, while middle-school problems include equations, functions, and advanced algebraic concepts.
4 Experiments
The experiments benchmark 16 MLLMs on ScratchMath using standardized prompting and task-specific evaluation, with LLM-as-a-Judge for ECE and exact accuracy for ECC. Proprietary and larger models generally perform better, but substantial gaps remain relative to human performance.
- 4 Experiments: 16 MLLMs spanning open-source and proprietary models were evaluated on ScratchMath.The benchmark covers diverse model sizes and architectures, including 10 open-source and 6 proprietary models.
- 4 Experiments: Standardized prompts, temperature 0, and a 2048-token output limit were used for reproducible comparisons.Additional Chain-of-Thought experiments showed some improvements on ECC performance.
- 4 Experiments: ECE was evaluated with an LLM-as-a-Judge framework, while ECC required exact agreement with the annotated class.Manual verification found 88.6% judge accuracy versus 91.4% human-human agreement on 70 sampled ECE cases.
- 4.2 Main Results: Proprietary models consistently outperform open-source models, yet both remain substantially below human performance.The authors attribute the proprietary-model advantage partly to more diverse training data and emphasize ScratchMath’s difficulty.
- 4.2 Main Results: Larger models generally perform better, reasoning models are especially strong on ECE, and ECC is harder across models.The comparison includes representative model error examples for ECE in Table 4.
- 4.2 Main Results: Primary-level performance is unexpectedly lower than middle-school performance on ECC.The authors suggest that less structured and harder-to-interpret primary-level handwriting may complicate precise classification.
5 Further Analysis
Further analyses examine model failures, error categories, mathematics topics, and grade levels. They show that visual recognition, reasoning reconstruction, error-type complexity, and handwriting structure all shape performance.
- 5.1 RQ1: Challenges in Error Identification: Visual processing, formatting interpretation, and inference of student reasoning are recurring failure modes in ECE.The qualitative cases classify errors as Visual Recognition Failure, Formatting Misinterpretation, and Misaligned Misinterpretation.
- 5.1 RQ1: Challenges in Error Identification: o4-mini failures most often involve OCR or image recognition, reasoning reconstruction, and speculative over-inference.In a 100-case analysis, Qwen2.5-VL-7B additionally showed 22% hallucination errors and 17% model calculation errors.
- 5.2 RQ2: Impact of Problem Type on Performance: On ECC, o4-mini and Gemini 2.0 Flash Thinking perform well in most categories but struggle with Logical Reasoning and Calculation Errors.Procedural and Transcription errors are also challenging across models, while category disparities suggest differing problem complexity.
- 5.2 RQ2: Impact of Problem Type on Performance: Proprietary models show strong, stable performance across mathematics topics, whereas open-source models vary by topic.Skywork-R1V is stronger in Statistics and Probability and Applied Mathematics but weaker in Equations and Functions.
- 5.3 RQ3: Impact of Difficulty on Model Performance: ECE performance trends slightly downward with increasing grade level, while ECC performance generally improves.Middle-school scratchwork contains clearer sequential steps and more standardized notation than elementary responses, facilitating classification.
- 5.3 RQ3: Impact of Difficulty on Model Performance: Proprietary models consistently outperform open-source models across all grades.The analysis uses the top three open-source and proprietary models for both ECE and ECC.
6 Conclusion and Future Work
ScratchMath exposes limitations in current MLLMs for diagnosing student errors and motivates models more closely aligned with educators’ analytical processes. The study’s generalizability is constrained by its single-source Chinese student dataset.
- 6 Conclusion and Future Work: ScratchMath is presented as a comprehensive multimodal benchmark that exposes current MLLM limitations in student-error diagnosis.The authors identify closer alignment with educators’ analytical processes as an urgent need.
- 6 Conclusion and Future Work: All samples came from Chinese students using one online education platform, constraining generalizability across languages, demographics, and educational contexts.Future work proposes broader populations and settings, explicit error-type prediction, improved visual recognition, and step-by-step reasoning alignment.