PaperlayerView exact revision on arXiv

Source-linked AI summary

SketchJudge: A Diagnostic Benchmark for Grading Hand-drawn Diagrams with Multimodal Large Language Models

Yuhang Su, Mei Wang, Yaoyao Zhong, Guozhang Li, Shixing Li, Yihan Feng, Hua Huang

arXiv:2601.06944v1cs.CVcs.AI

TL;DR

MLLMs have been evaluated extensively on visual understanding, but grading ambiguous hand-drawn diagrams and diagnosing their errors remains underexplored. SketchJudge introduces a four-domain benchmark with realistic student sketches and structured annotations, and experiments show that state-of-the-art models still struggle with core grading competencies. The benchmark therefore supports diagnostic study while indicating that educational deployment requires validation and oversight.

  • Problem

    Existing MLLM benchmarks largely assume clean visual inputs and solver roles, leaving grading of ambiguous hand-drawn diagrams and fine-grained error diagnosis underexplored.

  • Method

    SketchJudge evaluates MLLMs on 1,015 hand-drawn student responses across geometry, physics, charts, and flowcharts using correctness labels and domain-specific error taxonomies.

  • Results

    State-of-the-art MLLMs still struggle with robustness to stylistic variation, structural and topological distinctions, and consistent conceptual-error attribution.

  • Takeaways & Limitations

    SketchJudge frames current MLLMs as research tools rather than replacements for human educators in high-stakes grading.

  • Takeaways & Limitations

    The labor-intensive annotation pipeline constrains dataset expansion and taxonomy updates, while borderline error categories retain residual ambiguity.

Abstract

from arXiv · show

While Multimodal Large Language Models (MLLMs) have achieved remarkable progress in visual understanding, they often struggle when faced with the unstructured and ambiguous nature of human-generated sketches. This limitation is particularly pronounced in the underexplored task of visual grading, where models should not only solve a problem but also diagnose errors in hand-drawn diagrams. Such diagnostic capabilities depend on complex structural, semantic, and metacognitive reasoning. To bridge this gap, we introduce SketchJudge, a novel benchmark tailored for evaluating MLLMs as graders of hand-drawn STEM diagrams. SketchJudge encompasses 1,015 hand-drawn student responses across four domains: geometry, physics, charts, and flowcharts, featuring diverse stylistic variations and distinct error types. Evaluations on SketchJudge demonstrate that even advanced MLLMs lag significantly behind humans, validating the benchmark's effectiveness in exposing the fragility of current vision-language alignment in symbolic and noisy contexts. All data, code, and evaluation scripts are publicly available at https://github.com/yuhangsu82/SketchJudge.

1 Introduction

SketchJudge addresses the gap between clean-image solver benchmarks and the harder task of grading ambiguous, hand-drawn STEM diagrams. It introduces a four-domain benchmark with structured error diagnosis and evaluates MLLMs’ diagram-level reasoning.

  • Motivation: Hand-drawn STEM sketches challenge clean-image assumptions through ambiguity, stylistic diversity, and unstructured visual information.These inputs include geometric constructions, free-body diagrams, and flowcharts used to express complex reasoning.
  • Motivation: Existing benchmarks mainly evaluate MLLMs as solvers, overlooking grading that identifies conceptual errors in student solutions.The grading role is especially relevant in education, where models should augment rather than replace human thinking.
  • Motivation: Effective diagram grading requires perceptual parsing, structural understanding, semantic mastery, and reasoning about solution correctness.The benchmark frames grading as a multilayered process that mirrors human cognition.
  • Benchmark contribution: 1,015 hand-drawn student answers across 300 questions and four domains form SketchJudge, with binary correctness labels and 5–7 expert-validated error types per domain.The domains are geometry, physics, charts, and flowcharts.
  • Benchmark contribution: SketchJudge is presented as the first diagnostic benchmark, to the authors’ knowledge, for grading hand-drawn diagrammatic answers across four task types.Its fine-grained taxonomy supports structured diagnosis beyond simple correctness judgments.
  • Benchmark contribution: Systematic experiments with state-of-the-art MLLMs reveal persistent limitations in diagram-level reasoning.The experiments provide insights for future research.

2 Related Work

Related multimodal benchmarks increasingly test reasoning, realism, and integrated capabilities, but typically use clean inputs and solver-oriented evaluation. SketchJudge combines freehand student diagrams, grading, and fine-grained error diagnosis.

  • Benchmark gap: Most multimodal benchmarks use clean, standardized visuals and rarely evaluate grading behavior on noisy, freehand diagrams.Table 1 compares freehand inputs, student-style responses, grading, error diagnosis, and reference diagrams across related benchmarks.
  • Benchmark gap: SketchJudge fills the combined gap by targeting freehand sketches, diagram grading, and fine-grained error diagnosis.Its emphasis is on hand-drawn, structure-rich student diagrams rather than answer matching alone.
  • Evaluation paradigms: Existing benchmarks largely follow a solver paradigm that matches model outputs directly against ground-truth answers.Judge-based evaluation instead assesses response quality across dimensions such as correctness, helpfulness, and consistency.
  • Educational context: Educational assessment remains distinct from realism-oriented multimodal datasets because student responses are often messy, hand-drawn, and structurally irregular.Educational datasets still predominantly focus on text-based answers.

3 Benchmark Design

SketchJudge models classroom grading with question–reference–answer instances, two reference conditions, and annotations for correctness and domain-specific errors. Its design preserves realistic sketch variation while addressing perceptual, structural, and semantic grading challenges.

  • 3.1 Overview of SketchJudge: Each instance contains a natural-language question Q, clean reference diagram R, and one or more hand-drawn student answers A.The sketches preserve construction traces, uneven strokes, and minor skew to mirror classroom-style submissions.
  • 3.1 Overview of SketchJudge: The WithRef setting supplies R, whereas NoRef withholds it and tests reasoning directly over Q and A.Both settings produce correctness judgments and fine-grained error-type diagnoses.
  • 3.2 Data collection: The benchmark covers geometry, physics, charts, and flowcharts, using mostly real-world questions plus some synthetic chart and flowchart problems.Synthetic construction controls structural complexity and data diversity.
  • 3.2 Data collection: Clean references come from accepted solutions, programmatic chart rendering, or manual authoring, with disagreements resolved by consensus among three raters.Problem statements and references are independently verified.
  • 3.3 Annotation Protocol: Ground truth combines binary correctness with one or more domain-specific error types for incorrect responses.This enables diagnostic evaluation beyond correctness alone.
  • 3.3 Annotation Protocol: Taxonomies are induced from annotator-written descriptions, refined with subject-matter experts, and applied by GPT-4o with human verification and correction.The two-stage protocol combines bottom-up category induction with expert-informed labeling.
  • 3.4 Challenges in Diagram Grading: Grading is difficult because sketches vary perceptually and stylistically, while valid solutions can differ in layout or orientation yet remain topologically and semantically correct.Correctness therefore cannot rely on reference-image similarity alone.
  • 3.4 Challenges in Diagram Grading: Fine-grained grading must distinguish acceptable imprecision from localized conceptual or procedural errors that alter semantic meaning.Examples include missing or incorrect labels, minor procedural mistakes, and small inaccuracies.

4 Experiments

SketchJudge evaluates MLLMs as diagram graders through binary correctness and fine-grained error recognition, using zero-shot experiments across models and diagnostic settings. Results show persistent gaps to human performance, model-dependent grading biases, and limited reliability of prompting and structural diagnosis.

  • Main Results: 83.33% human accuracy versus 78.42% for GPT-5 shows that diagram grading remains challenging for current MLLMs.Gemini-2.5-Flash reaches 77.74%, while most open-source models fall in the 53%–69% range.
  • Main Results: Closed-source models generally outperform open-source models in binary grading accuracy, while open-source systems show lower accuracy and greater cross-domain variance.Multiple closed-source models exceed 73% overall accuracy, whereas robust vision–symbol alignment and cross-domain generalization remain difficult for open-source models.
  • Main Results: 66.42 human ebF1 versus 61.13 for GPT-5 indicates that fine-grained error recognition remains below the human reference.Gemini-2.5-Flash reaches 58.30%, and the strongest model varies across domains.
  • Strictness and Leniency: Models exhibit substantially different strictness and leniency, with some rejecting correct sketches and others accepting plausible-looking incorrect diagrams.GPT-5 and Gemini-2.5-Flash are closer to balanced grading but still make errors in both FNR and FPR.
  • Effect of Reference Answers: Reference diagrams improve accuracy for most models, but their effects on error-type recognition are less uniform.The results suggest references aid correctness verification more consistently than fine-grained error assignment.
  • Effect of Prompt Design: CoT prompting consistently hurts the three tested models, while rubric prompting produces modest, model-dependent changes.For Gemini-2.5-Flash, CoT lowers Acc from 77.74% to 76.16% and raises FNR from 0.263 to 0.351.
  • Per-Error-Type Recall Analysis: Error types linked to salient local cues are easier to recognize than categories requiring global structure or latent intent.Recall is higher for connection-related physics errors, chart-type mistakes, and some flowchart errors, while concept-target, scaling, strategy-level, presentation, and omission errors remain difficult.

5 Conclusion and Outlook

SketchJudge evaluates whether MLLMs can grade messy hand-drawn diagrams rather than merely solve diagram-based problems. Results show persistent weaknesses in stylistic robustness, structural reasoning, and conceptual error attribution, while highlighting applications and boundaries for future educational use.

  • SketchJudge tests diagram grading across stylistic variation, structural distinctions, and conceptual error attribution.These competencies extend beyond recognizing visual patterns to reasoning about diagram meaning.
  • Current MLLMs still struggle with core grading competencies and often behave more like perceptual pattern recognizers than reasoning-aware graders.
  • SketchJudge supports future work on tool-augmented and agentic multimodal reasoning for constructing, generating, or verifying diagrams.The benchmark is positioned as a testbed for models that plan, externalize intermediate steps, and interact with tools.
  • Improved diagram-level understanding could support formative assessment and personalized feedback, but high-stakes deployment requires validation, oversight, and human–AI collaboration.
  • The benchmark’s labor-intensive annotation pipeline constrains dataset expansion and taxonomy updates, while borderline categories retain residual ambiguity.The paper also limits its current scope to controlled educational-style diagram tasks requiring further robustness and calibration work for broader settings.

A SketchJudge Task Details

SketchJudge combines collected educational problems with a constrained synthetic subset and organizes hand-drawn diagram tasks across geometry, physics, charts, and flowcharts. Each domain targets distinct structural, semantic, and representation challenges under drawing noise.

  • Data screening: All collected samples were manually screened to remove personally identifying information and inappropriate content.
  • Data sources and construction: SketchJudge draws problems from web-collected exam or textbook items and GPT-4o-generated synthetic charts and flowcharts.Synthetic generation used interactive, multi-turn prompting with domain-specific constraints.
  • Geometry: Geometry tasks assess construction and spatial reasoning despite imprecise angles, ambiguous auxiliary marks, multiple valid constructions, and annotation errors.
  • Physics: Physics tasks combine geometric fidelity with semantic correctness in circuits, vectors, and force analysis, including direction, polarity, connections, and physical principles.
  • Charts: Chart tasks require mapping numerical values to bar, line, or pie encodings, with errors involving chart choice, associations, scaling, omissions, and conventions.The chart split contains 54% bar, 25% line, 19% pie, and 2% other charts.
  • Flowcharts: Flowchart tasks depend on global procedural topology and decision logic, including correct steps, connections, branches, symbols, and labels.

B Error Category Taxonomy

SketchJudge uses a fine-grained taxonomy of error types spanning all four task categories. The taxonomy supports multi-label annotation of diagram errors.

  • The taxonomy covers error types across geometry, physics, charts, and flowcharts and forms the basis of multi-label annotation.

C Prompt Templates

The appendix documents taxonomy, evaluation, and prompting procedures for SketchJudge. Models are evaluated under standardized input configurations and structured-output requirements, with additional rubric and chain-of-thought variants.

  • C Prompt Templates: The appendix collects prompts for taxonomy induction, error labeling, and multimodal model evaluation.
  • C.1 Prompts for Taxonomy Induction and Annotation: GPT-4o clusters textual error descriptions and maps them to finalized error types without access to the corresponding sketches.
  • C.2 Prompt Templates; C.3 Task Settings: Evaluation prompts define the grading task, available inputs, and required structured outputs across four canonical input configurations.
  • C.3 Task Settings; C.5 Rubric-Based Prompt Templates; C.6 Chain-of-Thougth Prompt Templates: Baseline prompts provide minimal grading instructions, while rubric-based and chain-of-thought variants add explicit criteria or stepwise reasoning guidance under the same output constraints.
  • C.4 Baseline Prompt Templates: Figures 7–10 specify the baseline model role, inputs, task setting, and JSON output for the four cases.
  • D Evaluated Models: Table 19 evaluates open-weight and closed-source MLLMs in a zero-shot setup with identical prompts and deterministic decoding at temperature = 0.
  • Output Robustness and Format Compliance: Rare formatting deviations were handled by deterministic JSON extraction, validation, and manual correction, with negligible reported impact on performance.
  • C Prompt Templates: The appendix includes error-taxonomy tables for geometry, physics, flowcharts, and charts, plus a mapping from full names to shortened labels.

E Metric Definitions

The evaluation framework defines metrics for binary grading and fine-grained error-type recognition, with prompts, input configurations, and model metadata documented for reproducibility.

  • Binary grading metrics measure overall correctness and class balance, while multi-label metrics assess fine-grained error-type recognition.
  • The appendix documents taxonomy-induction and labeling prompts, four canonical input configurations, rubric templates, reasoning templates, and evaluated MLLMs.

E.1 Binary Grading Metrics

Binary grading treats each submission as correct or incorrect and reports accuracy-oriented metrics alongside false-negative and false-positive rates to diagnose grading bias.

  • Binary grading assigns y = 1 to gold-correct answers and y = 0 to gold-incorrect answers, comparing these labels with model predictions.
  • Accuracy is the overall proportion of correct decisions across true positives, true negatives, false positives, and false negatives.
  • MacroF1bin averages class-specific F1 scores so performance on correct and incorrect answers is balanced.F1pos measures recognition of gold-correct answers, while F1neg measures recognition of gold-incorrect answers.
  • False-negative rate diagnoses over-strictness, whereas false-positive rate diagnoses over-leniency in grading decisions.
  • MCC summarizes binary prediction quality on a scale from −1 for inverse prediction to 1 for perfect prediction, with 0 indicating random performance.

E.2 Error-Type Recognition Metrics

Error-type recognition evaluates whether models identify domain-specific mistakes after incorrectness is established, using global labels, instance-level F1, and per-class recall.

  • Domain-specific error taxonomies are merged into a namespaced global label set for recognizing fine-grained errors in incorrect answers.
  • MacroF1err and MicroF1err summarize error-label recognition using respectively label-level averaging and globally aggregated counts.
  • Example-based F1 measures overlap between predicted and gold error sets for each instance and averages the resulting scores.It is computed only when both gold annotation and model prediction indicate incorrectness.
  • Per-class diagnostic recall measures recovered instances relative to all gold instances of an error class, using only jointly incorrect submissions.Interquartile ranges of per-model recalls quantify cross-model consistency independently of class frequency.
  • Without reference diagrams, overall performance drops substantially because models must solve the task while interpreting noisy hand-drawn diagrams.
  • Performance differences between hand-drawn and electronic sketches are relatively small across most models, with inconsistent modality effects.

F.4 Exact vs. Alternative Correct Answers

The exact-versus-alternative analysis tests whether models recognize correct diagrams that differ structurally or strategically from the reference, rather than relying on visual matching.

  • F.4 Exact vs. Alternative Correct Answers: Correct answers are divided into reference-aligned sketches and reference-divergent sketches that remain valid despite structural or strategic differences.
  • F.4 Exact vs. Alternative Correct Answers: Most models perform better on reference-aligned answers, with larger drops on reference-divergent solutions in Physics and Flowchart.Reported gaps include Claude-3.7-Sonnet differences of 52.47 in Physics and 47.91 in Flowchart, and an o3 difference of 29.46 in Physics.
  • F.4 Exact vs. Alternative Correct Answers: Alternative correct solutions require structural-equivalence judgments, such as topological consistency or physically equivalent layouts, under freehand noise.Charts often preserve geometric cues through rigid conventions, reducing reliance on canonical matching.
  • F.4 Exact vs. Alternative Correct Answers: Qualitative examples compare model grading responses with gold annotations and show how different error types appear in student sketches.
Loading 2601.06944v1…