Source-linked AI summary
OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios
Zinuo Guo, Min Zhang, Bo Jiang
TL;DR
Realistic handwritten OCR remains insufficiently evaluated, especially for multilingual text, writer errors, and complex formulas. OmniHandwritingOCR benchmarks these settings across 77.57K labeled images and thirteen systems, finding limited aggregate performance, ranking shifts, and substantial degradation with structural complexity.
Problem
Existing OCR evaluations insufficiently cover realistic handwriting, including multilingual text, writer errors, fact-preserving transcription, and complex multi-line formulas.
Method
OmniHandwritingOCR unifies multilingual handwritten text and difficulty-stratified formula evaluation across six subtasks and twelve subsets, using standardized scoring to diagnose model behavior.
Results
Aggregate performance remains limited, model rankings vary across language and formula settings, and structural complexity causes substantial degradation, with Kimi-VL-A3B-Instruct reaching 93.33 F1 on single-line formulas.
Takeaways & Limitations
The benchmark exposes task-specific robustness and hallucinated corrections as important failure modes for generative handwritten OCR systems.
Takeaways & Limitations
The benchmark covers offline image-based handwriting only, and LATEX-style formula scoring can penalize semantically equivalent expressions written in different markup forms.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) are increasingly used as OCR systems in document and knowledge-processing pipelines, but their ability to faithfully read real handwriting remains underexplored. Existing OCR benchmarks focus largely on printed text or clean single-line inputs, leaving limited coverage of realistic handwritten OCR scenarios such as multilingual handwriting, writer errors, and structurally complex mathematical expressions. We introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating MLLMs and OCR systems on handwritten OCR. It covers handwritten text recognition and handwritten mathematical expression recognition across six subtasks and twelve subsets, totaling 77.57K labeled images from public datasets and newly collected student writings. A key component is a difficulty-stratified multi-line formula corpus designed to test robustness under increasing structural complexity. We evaluate thirteen open- and closed-source systems with five complementary metrics under a unified protocol. Results show that current systems remain far from faithful transcription: performance drops sharply on complex multi-line formulas, model rankings vary across language and formula settings, and several generative models hallucinate plausible but visually unsupported corrections. OmniHandwritingOCR provides a challenging testbed for diagnosing language, content, structural, and visual-grounding failure modes of multimodal models in handwritten OCR scenarios.
1 Introduction
OmniHandwritingOCR is introduced as a diagnostic benchmark addressing gaps in handwritten OCR evaluation for MLLM/OCR systems. It combines diverse handwriting and formula settings with difficulty stratification and unified evaluation to expose language, content, structural, and visual-grounding failures.
- Motivation: MLLMs increasingly serve as OCR systems within information and knowledge-processing pipelines, alongside specialized OCR engines.OCR outputs support indexing, search, retrieval-augmented generation, educational analytics, and document-level knowledge extraction.
- Evaluation gaps: Existing evaluations insufficiently test multilingual handwriting, real student work, fact-preserving transcription, and complex multi-line formulas.CROHME and MathWriting are dominated by single-line expressions, limiting coverage of structurally complex handwritten mathematics.
- Evaluation gaps: Aggregate OCR scores can conceal omissions, repaired student mistakes, and fabricated content that drifts from visual evidence into plausible correction.The benchmark motivation emphasizes identifying where models fail and how performance changes with structural complexity.
- Benchmark design: OmniHandwritingOCR contains 77.57K image-label pairs spanning English text, Chinese text, single-line formulas, and easy, medium, and hard multi-line formulas.The benchmark analyzes language, content type, structural complexity, and visual grounding using public and newly collected sources.
- Contributions: The benchmark integrates HTR and HMER across six subtasks and twelve subsets, with difficulty-stratified multi-line formulas targeting complex real student work.A unified evaluation setting applies task-specific normalization and tokenization rules for fair comparison across general-purpose MLLMs.
2 Related Work
Prior work has advanced handwritten text and mathematical expression recognition through increasingly capable sequence, structural, and Transformer-based models, but existing benchmarks remain limited for realistic, complex handwritten OCR. OmniHandwritingOCR addresses this gap by unifying HTR and HMER evaluation under a common stress-test protocol, emphasizing difficult multi-line formulas and faithful transcription.
- Handwritten Text Recognition: HTR has progressed from HMM-based recognizers to CNN-RNN-CTC, multidimensional recurrent, attention-based, and Transformer-based models such as TrOCR.These approaches have improved line-level and document-level transcription while remaining important baselines for handling handwritten variation.
- Handwritten Text Recognition: Real handwritten text challenges recognition beyond isolated characters through inconsistent spacing, writer-specific abbreviations, touching strokes, insertions, deletions, and line-layout changes.Public datasets including IAM, RIMES, READ-BAD, GNHK, CASIA-HWDB, and SCUT-HCCDoc cover varied languages and document settings, but many samples are cleaner than educational handwriting.
- Handwritten Mathematical Expression Recognition: HMER requires both symbol identification and recovery of two-dimensional relations, motivating grammar-based, trajectory-based, sequence, tree, graph, and structure-aware Transformer approaches.Errors such as missing braces, misplaced superscripts, or incorrect fraction scopes can change the recovered expression, so structural fidelity matters alongside symbol accuracy.
- Handwritten Mathematical Expression Recognition: Existing HMER benchmarks standardize evaluation but remain dominated by single-line expressions or relatively controlled layouts, limiting coverage of long, noisy, multi-line handwritten derivations.A model may recognize isolated symbols or short formulas yet fail to preserve alignment, multi-step dependencies, cases, fractions, superscripts, or crossed-out content.
- Benchmark Gap and OmniHandwritingOCR: OmniHandwritingOCR unifies HTR and HMER as a stress test for general handwritten OCR engines, with special emphasis on difficulty-stratified multi-line formulas and evaluation under one protocol.The unified setting exposes effects from language, mathematical, and instruction-following priors while helping distinguish genuine visual grounding from other influences.
3 OmniHandwritingOCR
OmniHandwritingOCR is designed as a diagnostic benchmark combining broad handwritten text and formula coverage with difficulty-aware analysis and standardized, faithful transcription. Its construction combines public datasets with newly collected student writings, challenge-oriented filtering, structural stratification, and expert annotation.
- Benchmark design: More than 77K samples span English text, Chinese text, and mathematical formulas, with multi-line formulas systematically stratified by difficulty.The design targets analysis across languages, layouts, and structural complexity rather than aggregate OCR scores alone.
- Benchmark design: OmniHandwritingOCR combines handwritten text recognition and mathematical expression recognition under a shared protocol while representing realistic mixed-content scenarios.The benchmark is intended to expose model behavior beyond a single clean text line.
- Filtering and stratification: 46K high-quality public samples were distilled from approximately 160K by discarding examples that Qwen2.5-VL-72B recognized easily using normalized edit distance.The filtering process deliberately selects samples on which the model performs poorly to increase evaluation difficulty.
- Data collection: 86K images were segmented from over 6,000 authentic student mathematics answer sheets, alongside 874 student Chinese compositions targeting complex real-world multi-line samples.These private data sources address gaps in existing resources and contribute challenging handwritten material.
- Filtering and stratification: 30K private mathematics samples retain core solution sections and emphasize multi-line difficult formulas, whose subsets show monotonic increases in length, LATEX command density, and non-empty lines.These model-agnostic statistics validate that the difficulty split reflects observable structural complexity as well as recognition difficulty.
- Dataset composition: 38,088 handwritten multi-line formula samples make the benchmark substantially more focused on structural complexity than comparable benchmarks.The benchmark uniquely combines handwritten text with handwritten mathematical expressions to simulate mixed-content scenarios.
4 Experiments
The experiments evaluate 13 open- and closed-source MLLM and OCR systems under a unified zero-shot transcription protocol, diagnosing performance across languages, handwriting types, and formula complexity. Results show task-conditioned rankings, severe degradation on structurally complex formulas, and visually unsupported generative corrections.
- Experimental setup: The benchmark evaluates 13 methods, comprising 7 general-purpose MLLMs and 6 specialized OCR models, using a unified zero-shot transcription protocol.Each system receives one image and a short task-neutral instruction without retrieval, demonstrations, or dataset-specific context.
- Overall results: Qwen3-VL-8B achieves the strongest aggregate score, but specialized OCR models remain competitive on particular metrics and subsets.Nanonets-OCR2-3B achieves the best overall OCR-model score and lower CER/WER than other specialized OCR baselines.
- Task-conditioned results: Model rankings are task-conditioned: Qwen2.5-VL-72B leads Chinese-text CER/WER, Kimi-VL-A3B-Instruct leads single-line and easy multi-line formulas, and Nanonets-OCR2-3B leads many English-text OCR metrics.Formula rankings change again on harder multi-line subsets, with Qwen3-VL-8B leading ml-medium and Qwen2.5-VL-72B leading ml-hard.
- Formula complexity: 86.92 to 70.45 F1: Qwen3-VL-8B declines from ml-easy to ml-hard, while its CER worsens from 24.95 to 57.08.Across the benchmark, average label length rises from 156.9 to 488.8 characters and average LATEX command count from 4.1 to 25.7 as formula difficulty increases.
- Failure modes: Generative models exhibit visually unsupported corrections, including repairing writer mistakes, inserting plausible steps, dropping difficult symbols, and adding Markdown or explanatory formatting.The evaluation penalizes corrections, altered formulas, missing content, or added reasoning when unsupported by the image.
5 Data Release, Ethics, and Limitations
OmniHandwritingOCR is designed for reproducible offline evaluation with public benchmark materials and deterministic scoring, while protecting educational data and preserving visible writer errors. Its scope is limited by offline image inputs, markup-sensitive formula scoring, and a model-dependent difficulty measure.
- Data release and reproducibility: The benchmark will release images, labels, split manifests, and evaluation scripts, using anonymized identifiers for newly collected educational data.The scripts implement normalization, tokenization, output-length caps, and metric computation.
- Ethics: Private educational handwriting samples are used only for recognition evaluation, with personally identifying metadata removed and no demographic labels required.The annotation policy preserves visible writer errors for faithful transcription without judging writers’ ability or correctness.
- Limitations: The benchmark excludes online pen trajectories and focuses on offline image-based handwriting.Its formula scoring uses LATEX-style textual comparison, which can penalize semantically equivalent expressions in different markup forms.
- Limitations: The difficulty score includes a model-performance component, so it is an evaluation stratification tool rather than an intrinsic human difficulty measure.Model-agnostic statistics are included to support interpretation.
6 Conclusion
OmniHandwritingOCR introduces a diagnostic benchmark and unified protocol for handwritten OCR that integrates multilingual text with single-line and difficulty-stratified multi-line formulas. Evaluation across thirteen systems shows limited aggregate performance and varying model rankings across language and formula settings.
- Benchmark scope: OmniHandwritingOCR combines multilingual handwritten text with single-line and difficulty-stratified multi-line formulas to cover capabilities underrepresented in existing OCR evaluations.The benchmark is paired with a unified evaluation protocol for MLLM/OCR systems.
- Evaluation findings: Thirteen evaluated systems achieve limited aggregate performance on handwritten OCR.The conclusion reports this finding across the benchmark’s unified evaluation protocol.
- Evaluation findings: Model rankings vary across language and formula settings.This variation is reported alongside limited aggregate performance in the evaluation of thirteen systems.
GenAI Usage Disclosure
Generative AI tools supported the research in two roles: vision-language models handled pre-annotation, while coding assistants aided experiment-related code development.
- Vision-language models were used for pre-annotation, as described in Section 3.
- GenAI-based coding assistants assisted with code development for conducting the experiments.
- Generative AI tools therefore contributed to both data preparation and experimental implementation.The disclosed roles were pre-annotation and assistance with experiment-related code development.