Source-linked AI summary
WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans
Jun Zhang, Qiao Zhao, Cheng Cui, Jianying Qu, Zhongkai Sun, Jianwen Yang, Changda Zhou, ZhuoXin Liu, Shubin Han
TL;DR
Handwritten document understanding remains poorly characterized because existing benchmarks omit important structures and do not explain recognition failures. WildHandBench addresses this gap with broad handwritten-document coverage and Prior-Driven Error analysis, finding that models remain below calibrated humans and rely more on language priors in their errors.
Problem
Existing benchmarks provide limited evidence about handwritten document understanding because they overlook handwritten tables, real-world degradation, and why recognition errors occur.
Method
WildHandBench evaluates 500 handwritten documents across three structures, four language settings, and nine scenarios, using PDE to quantify prior-driven errors.
Results
Models remain below humans overall, with the best model at 71.85% versus 77.09% for humans, and model errors more often prior-driven.
Takeaways & Limitations
The benchmark shows that handwritten understanding remains unsolved and that conventional accuracy metrics miss systematic reliance on language priors.
Takeaways & Limitations
The 500-sample benchmark is not a population-level estimate, and its small formula and table subsets limit statistical power for distinguishing highly competitive models.
Abstract
from arXiv · showhide
While the top model on OmniDocBench now reaches 96.34% overall on printed-document parsing, the ability of current models to handle challenging handwritten documents remains largely uncharacterized. Existing benchmarks focus on isolated text or formulas, overlook handwritten tables and real-world degradation, and report aggregate accuracy without explaining why models fail. We present WildHandBench, a benchmark containing 500 handwritten documents across three structures (free text, tables, formulas), four languages, and nine real-world scenarios. We introduce a Prior-Driven Error (PDE) metric that quantifies whether errors originate from language priors rather than visual evidence. Evaluating 18 state-of-the-art models together with calibrated human baselines, we find: (1) the best model achieves only 71.85% overall; (2) humans outperform all models yet the gap is narrow (77.09% vs. 71.85%); and (3) model errors are qualitatively different from human errors -- 63-91% of model errors are prior-driven versus only 49% for humans, exposing systematic reliance on language priors that conventional accuracy metrics cannot capture.
Introduction
WildHandBench argues that strong printed-document results do not characterize handwritten document understanding, where models face ambiguous writing, irregular structures, and uncertainty. It addresses limited benchmark coverage and unexplained failures with a broader benchmark, PDE analysis, and evaluation against humans.
- Motivation: 96.34% overall on OmniDocBench reflects performance on printed documents, leaving handwritten understanding insufficiently characterized.The paper argues that printed documents typically have regular layouts and clear visual appearances.
- Motivation: Handwritten document understanding combines ambiguous handwriting, irregular structures, visual-evidence discrimination, and uncertainty reasoning.The paper treats handwriting as a distinct document understanding challenge rather than simply a harder OCR problem.
- Benchmark Gap: Existing benchmarks largely omit handwritten tables, naturally written formulas, real-world degradation, and analysis of why recognition errors occur.They primarily evaluate isolated free-text recognition and aggregate accuracy.
- Contributions: 500 handwritten documents span free text, tables, formulas, four language settings, and nine real-world scenarios in WildHandBench.The benchmark is designed to address gaps in evaluation coverage.
- Contributions: PDE quantifies whether recognition errors arise from language priors rather than visual evidence, extending analysis beyond conventional accuracy metrics.It targets hallucination-like recognition behaviors.
- Findings: 71.85% overall is the strongest model result, while humans reach 77.09%; 63–91% of model errors are prior-driven versus 49% for humans.The evaluation covers 18 state-of-the-art models and calibrated human baselines.
Related Work
Prior work is fragmented across isolated handwritten recognition tasks and aggregate document benchmarks. WildHandBench targets the resulting gap by unifying handwritten structures and measuring prior-driven errors on naturally occurring documents.
- Handwritten Document Understanding: Existing handwritten datasets are highly task-specific, with most focusing on line-level recognition under relatively controlled settings.Examples include IAM, RIMES, CASIA-HWDB, SCUT-EPT, and SCUT-HCCDoc.
- Benchmarking Document Understanding: Holistic document benchmarks broaden evaluation beyond isolated OCR but primarily report aggregate recognition accuracy.This limits insight into why handwritten documents remain challenging.
- Prior-Driven Errors: Existing prior-driven-error evaluations are restricted to handwritten mathematical expressions or synthetically perturbed printed documents.They do not cover naturally occurring handwritten documents across multiple structures.
- Positioning: WildHandBench is presented as the first benchmark to measure prior-driven errors across naturally occurring handwritten free text, tables, and formulas.It combines unified coverage with calibrated human baselines.
WildHandBench: Benchmark Design
WildHandBench is a 500-image benchmark organized across languages, document structures, and real-world scenarios, constructed from offline and Internet sources with multi-stage annotation and verification.
- Benchmark Scope: 500 handwritten document images cover four language settings, three structures, and nine representative real-world scenarios.The structures are free text, tables, and formulas; the benchmark also includes realistic degradation.
- Benchmark Scope: Free text comprises 367 samples, tables 81, and formulas 52 images containing 278 annotated formula regions.Their reported shares are 73.4%, 16.2%, and 10.4%, respectively.
- Benchmark Positioning: WildHandBench is described as the first unified benchmark covering handwritten free text, tables, and formulas with real-scenario degradation and calibrated human baselines.The claim is made relative to representative prior benchmarks.
- Data Sources: Documents come from voluntarily contributed offline manuscripts and publicly available Internet resources.Offline sources support reliable transcription through writers’ access to original content, while online resources provide contextual annotation information.
- Quality Control: Automatic similarity scanning and manual inspection remove duplicates and resolve ambiguous collection cases.These steps precede the annotation stages.
- Annotation Pipeline: Three construction stages use independent MLLM pre-annotation, human ground-truth annotation, and second-annotator verification with consensus adjudication.Annotations use Markdown, LATEX, and HTML <table> formats, plus six difficulty dimensions.
Evaluation Protocol
The evaluation combines structure-specific recognition metrics, an error-attribution metric, and calibrated human baselines. PDE isolates the share of mismatches attributable to language priors rather than visual perception.
- Recognition Metrics: Structure-specific metrics evaluate free text with Edit, formulas with CDM, and tables with TEDS.CDM matches rendered character bounding boxes, while TEDS measures table structure and textual fidelity through HTML.
- Prior-Driven Error: PDE distinguishes visually grounded recognition failures from fluent but visually unsupported outputs that rely on language priors.This separates failure modes that require different modeling improvements.
- Prior-Driven Error: PDE scores mismatched aligned segments as prior-driven when model output perplexity is lower than ground truth; unsupported inserted content is always prior-driven.Alignment is character-level for free text and formulas and cell-level for tables.
- Prior-Driven Error: The reported overall PDE is the arithmetic mean of per-category PDE rates for text, tables, and formulas.For inserted content, ground-truth perplexity is treated as infinite.
- Human Baselines: Humans use the same images, output formats, and metrics as models, but perform single-pass recognition without auxiliary resources.This creates a calibrated comparison between human and machine performance.
Experiments
WildHandBench evaluates current vision-language models and humans on handwritten documents, revealing a persistent performance gap and distinct prior-driven failure patterns. Tables are especially difficult, while model PDE rates vary substantially across architectures and structures.
- Evaluation Setup: 18 models were evaluated under an identical post-processing pipeline, with calibrated human baselines providing a reference for recognition and error analysis.Models included proprietary, general-purpose open-source, and OCR-focused open-source VLMs.
- Overall Results: 71.85% overall was achieved by the strongest model, while models exceeding 90% on OmniDocBench dropped substantially on handwritten documents.OCR-focused models also fell behind general VLMs on WildHandBench despite their strength on printed-document benchmarks.
- Results by Structure: 60.44 was the best TEDS score for tables, indicating nearly 40% structural and content mismatch and the smallest cross-model variance among functional models.Text recognition showed the widest spread across models, while Gemini 3.1 Pro led all three categories and overall.
- Human Baseline: 77.09% was achieved by humans versus 71.85% by the best MLLM, a 5.24-point overall advantage for the human baseline.The modest gap indicates that wild handwriting challenges both humans and models.
- Error Pattern Analysis: 49.27% was the human PDE rate, below every evaluated model’s minimum of 63.28%, showing qualitatively different error distributions.Models more often generated fluent but visually unsupported outputs, whereas humans tended toward conservative partial transcriptions.
- Error Pattern Analysis: 87–98% of formula errors were prior-driven, compared with 52–84% for text and 42–90% for tables.Among models with functional visual encoders, overall PDE rates ranged from 63% to 79% without a simple correlation to accuracy.
- Metric Limitation: PDE characterizes error type rather than error quantity, and its classification depends on the Qwen3-8B-Base reference language model.A high prior-driven rate therefore indicates more systematic errors, not necessarily lower reliability, and another reference model may shift boundary cases.
Discussion
WildHandBench shows that handwritten document understanding remains unsolved, with errors often driven by language priors rather than visual evidence and important limits on benchmark scope and interpretation.
- 71.85% overall is the best model performance on WildHandBench, below the over-90% performance models achieve on OmniDocBench.
- 63–79% of errors among functional models are prior-driven, compared with 49% for humans.Prior-driven errors are fluent, confident, and unsupported by visual evidence.
- Prior-driven errors matter especially in medical records, financial documents, and legal archives, where they are particularly difficult to detect.The findings motivate optimizing visual grounding and calibrated uncertainty alongside recognition accuracy.
- 500 samples do not provide a population-level estimate, while the 52-image formula and 81-image table subsets limit statistical power.The benchmark focuses primarily on Chinese and English handwriting, with future expansion planned for language coverage and scale.
- Source bias and subjective transcription in some scenarios constrain the benchmark and make the human baseline a calibrated reference rather than an absolute upper bound.All annotators are native Chinese speakers, which may underestimate human performance on highly cursive or stylistically diverse English handwriting.
- PDE classifications can vary with the external reference language model, and higher PDE indicates error type rather than lower reliability.PDE measures the proportion of errors attributed to language priors rather than visual perception, not overall error quantity.
Conclusion
WildHandBench jointly evaluates handwritten free text, tables, and formulas using recognition metrics, human baselines, and Prior-Driven Error analysis. Results show a persistent performance gap and qualitatively different model errors.
- The best model reaches 71.85% versus 77.09% for humans, while 63–79% of model errors are prior-driven versus 49% for humans.
Supplementary Material
The supplementary material adds dataset construction details, qualitative PDE examples, and complete evaluation specifications beyond the main paper.
- Supplementary Sections A–C cover dataset construction, qualitative PDE examples, and complete evaluation details.The evaluation details include the full model list, prompt templates, and inference configuration.
A. Dataset Details
The dataset-details section expands the main paper's construction description with distributions, scenario counts, annotation, and image-processing specifications.
- Additional details include per-source counts, joint category–language distributions, per-scenario counts, and annotation and image-processing specifications.
A.1 Data Collection Sources
WildHandBench draws handwritten samples from offline contributions and internet sources, then organizes them across document structures, languages, scenarios, and independently rated difficulty dimensions.
- Collection Sources: 500 samples come from two channels: 141 offline handwritten manuscripts and 359 internet-collected handwritten images.Offline contributions concentrate on education and medical domains, while internet sources broaden coverage to long-tail genres.
- Dataset Distribution: The benchmark’s category–language composition is documented through a joint-distribution table, while scenario coverage is documented through per-scenario counts.The main paper reports marginal category and language counts separately; Tables 3 and 4 provide joint and scenario-level distributions.
- Difficulty Annotation: Five reviewers independently score six difficulty dimensions per sample, and the averaged ratings support fine-grained analysis but not aggregate benchmark scores.Per-dimension reviewer scores are averaged across reviewers.
- Annotation Format: Annotations support Markdown free text, LATEX formulas, and HTML tables, with formula polygons and structural table markup preserving irregular layouts.Each formula polygon is paired with one LATEX transcription; table annotations include rowspan, colspan, and header hierarchy.
C. Complete Evaluation Details
The evaluation standardizes prompts, output formats, decoding, retries, and human testing across models while applying structure-specific recognition metrics. It also calibrates human baselines under model-comparable conditions.
- Models and Prompts: The study evaluates the complete list of models using standard OmniDocBench prompts for general VLMs and default settings for OCR-focused models and document parsers.No user-supplied prompt overrides the stated defaults.
- Output Formats: Text outputs are converted to Markdown, formulas to LATEX with inline or block delimiters, and tables to HTML wrapped in table tags.The prompt also requires preserving document structure and recognizing text without guessing or inferring.
- Runtime Controls: All models use deterministic decoding at temperature 0.0, up to three retries with exponential backoff, model-dependent timeouts, and an 8192-token output limit where configurable.Images above 4000 px are proportionally resized, and model outputs are stripped of Markdown code fences without other normalization.
- Human Baseline: Humans receive the same images as models, perform single-pass recognition without auxiliary resources, use identical target formats, and are evaluated with the same metrics.The protocol prohibits revisions, second attempts, and consultation of predictions, transcripts, discussions, or experts.
- Interpretation: The human protocol provides a calibrated model-comparable reference rather than an absolute upper bound on human performance.This scope boundary qualifies how human–model comparisons should be interpreted.