OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models

1 Introduction

• Liu et al. report excellent zero-shot OCR by large multimodal models in the wild without OCR-domain-specific training.Their capabilities on text-related visual tasks remain important to understand because these models infer context from multiple data sources but may struggle with complex cross-modal relationships.Source: §1 · p. 1
• Even Gemini and GPT4V struggle with blurry, handwritten, multilingual, and handwritten mathematical text, and favor common words over random letter sequences.The authors report significant gaps versus domain-specific methods across text-related tasks.Source: §1 · p. 2
• The study evaluates 14 large multimodal models on Text Recognition, Scene Text-Centric VQA, Document-Oriented VQA, Key Information Extraction, and Handwritten Mathematical Expression Recognition.Source: §1 · p. 2

2.1 Large Multimodal Models

• Prior LMMs bridge vision and language, align modalities, and improve OCR or visual detail through specialized training and image-processing designs.BLIP2 uses a Querying Transformer; Flamingo [13] and OpenFlamingo add gated cross-attention, while LLaVA uses GPT-4-generated instruction data. LLaVAR, BLIVA, MiniGPT4V2, UniDoc, Docpedia, Monkey, and TextMonkey [30] explore text-rich training, visual features, task identifiers, instruction tuning, frequency-domain input, detailed captions with high-resolution architecture, and window attention with token resampling, respectively.Source: §2.1 · p. 2

2.2 Benchmarks for Large Multimodal Models

• Existing LMM evaluations offer limited OCR testing, and multiple-choice or True/False formats cannot accurately assess word recognition; OCRBench provides 1,000 corrected QA pairs across five tasks.The study evaluates LMMs on five prominent text-related tasks; OCRBench consists of manually filtered and corrected question-answer pairs.Source: §2.2 · p. 2

3.1 Evaluation Metric and Evaluation Dataset

• The evaluation counts an answer as correct when the ground truth appears in the LMM output, excludes answers shorter than four symbols, and samples 3,000 instances from some large datasets.The criterion addresses explanatory output that makes exact matching or ANLS unsuitable for zero-shot evaluation.Source: §3.1 · p. 2
• The evaluation spans text recognition, scene-text VQA, document VQA, KIE, and HMER, using datasets including IIIT5K, STVQA, DocVQA, SROIE, and HME100K.The recognition suite also includes semantic and shuffled-character non-semantic text sets, each with 3,000 images; ESTVQA is split into Chinese and English subsets.Source: §3.1 · pp. 2–4

3.2 Results

• LMMs perform comparably to supervised state-of-the-art models on regular, irregular, occluded, and artistic text, but lag on handwritten, Chinese, handwritten-string, and non-semantic text.In WordArt, InstructBLIP2 and BLIVA even outperform the supervised state-of-the-art model.Source: Table 1 · p. 4, §3.2 · p. 4
• Smaller input resolutions yield poorer results on scene-text VQA, document VQA, and KIE, while LMMs also struggle with HMER’s messy characters, spatial structure, LaTeX representation, and scarce training data.Monkey’s 1344×896 input capability is associated with enhanced performance on those fine-grained tasks.Source: §3.2 · pp. 4–5

3.3 OCRBench

• Gemini achieves the highest OCRBench score, GPT4V ranks second, and Monkey trails only those two models.GPT4V refused to provide results for 84 OCRBench images; the reported benchmark also shows that GPT4V and Gemini struggle with HMER, unclear images, handwritten and non-semantic text, and following task instructions.Source: Table 3 · p. 5, §3.3 · p. 6

3.2 Results

• LMM accuracy on non-semantic text falls by an average of 57.0% versus semantic text, while the scene-text-recognition SOTA method drops around 4.6%.The authors attribute the larger LMM decline to reliance on semantic understanding rather than direct character recognition; low ORAND performance further supports this finding.Source: §3.2 · p. 5
• LMMs perform 51.9% below supervised SOTA on handwritten text, and the paper reports a notable ESTVQA Chinese-English gap alongside limited Chinese proficiency.Monkey performs better in Chinese scenarios, which the authors associate with substantial Chinese training of its language model and visual encoder.Source: §3.2 · p. 5

3.3 OCRBench

• OCRBench combines five OCR task components into 1,000 question-answer pairs, manually verifies and corrects answers, and recommends improving fine-detail and character recognition and developing multilingual training data.For KIE, the benchmark adds a prompt directing models to answer using image text; the authors offer pretrained visual-language alignment and possible OCR exposure in multimodal training as plausible explanations for OCR ability, while noting unsupervised pairs cannot match fully supervised data.Source: §3.3 · pp. 5–7

4 Future work

• MiniCPMV-2.6 scored 852 on OCRBench and Mini-Monkey scored 802, underscoring model potential and an opportunity for open-source OCR research.The passage notes that most models’ data is not entirely open-source and many models remain proprietary.Source: §4 · p. 8
• OCRBench lacks multilingual documents, texts from diverse scenarios, and text-detection tasks; future work will expand the benchmark to include them.Source: §4 · p. 8

Findings

• LMMs show promising OCR results, especially in text recognition, and attain SOTA performance on some datasets, yet gaps from domain-specific supervised methods remain.The conclusion notes that specialized methods tailored to each task remain essential while using substantially less computational resources and data.Source: §5 · p. 8

Benchmark

• OCRBench serves as a benchmark for evaluating multimodal models’ OCR capabilities and supports their development in the OCR field.Source: §5 · p. 8

Future directions

• Future work will explore LMMs across more scenarios, more complex tasks, and multiple languages toward more intelligent and versatile OCR solutions.Source: §5 · p. 8

A Summary of the Evaluation Benchmarks.

• OCRBench contains 1,000 questions across five tasks, and all answers were manually filtered and corrected.The benchmark covers Text Recognition, Scene Text-centric VQA, Doc-oriented VQA, Key Information Extraction, and Handwritten Mathematical Expression Recognition.Source: A Summary of the Evaluation Benchmarks · p. 15, Table 5 · p. 15
• Text Recognition contains 300 images across six subcategories, while Scene Text-centric VQA and Doc-oriented VQA each contain 200 questions across four datasets.The six recognition subcategories each contribute 50 images; each VQA dataset contributes 50 questions.Source: A Summary of the Evaluation Benchmarks · p. 15, Table 5 · p. 15
• Key Information Extraction includes 200 questions from SROIE, FUNSD, and POIE, and handwritten mathematical expression recognition includes 100 HME100k images.The extraction questions are distributed as 67 from SROIE, 66 from FUNSD, and 67 from POIE.Source: A Summary of the Evaluation Benchmarks · p. 15, Table 5 · p. 15

• Most tested LMMs have input size limited to 224 or 336 and perform poorly on Doc-oriented VQA and KIE, while higher-resolution models perform better on these tasks.Source: B Summary of the Models · p. 15

C Supervised SOTA

• PARSeq [110] is the OCR text-recognition reference; TPS-ResNet is used for ReCTS, AttentionHTR for IAM, and Yu for ORAND-CAR-2014.PARSeq is trained on ST [111] and MJ [112] synthetic data and tested directly on real datasets. ReCTS is evaluated with Normalized Edit Distance as a reference; IAM's word error rate of 8.76 is expressed as 91.24 word accuracy. Yu uses prototype learning with feature-level sliding windows.Source: C Supervised SOTA · p. 16
• GIT [14] is the SOTA reference for STVQA and OCR-VQA, while Mia achieves SOTA on TextVQA using T5 [116].GIT uses an image encoder and text decoder in one language-modeling task; Mia uses masked language modeling and relative position prediction to align object features and scene text.Source: C Supervised SOTA · p. 16
• BAIDU-DI achieves DocVQA SOTA with ERNIE-Layout [117] and DocPrompt; Fang, DUBLIN [119], and Liu [120] are references for ESTVQA, InfographicVQA, and ChartQA.Fang uses cross-modal attention and a transformer decoder; because it reports no separate Chinese and English results, its average performance is used as reference. Liu [120] uses the DEPLOT modality-conversion module.Source: C Supervised SOTA · p. 16
• ERNIE-Layout [117], StrucTexT [121], Kuang [60], and Li [122] are the respective SOTA references for FUNSD, SROIE, POIE, and HME100K.Their approaches use reading-order correction, segment-token alignment with visual-language and box-direction pretraining, contrastive learning between OCR and information extraction, and joint HMER-symbol-counting optimization, respectively.Source: C Supervised SOTA · p. 16