Source-linked AI summary
OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning
Ling Fu, Zhebin Kuang, Jiajun Song, Mingxin Huang, Biao Yang, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, Zhang Li, Guozhi Tang, Bin Shan, Chunhui Lin, Qi Liu, Binghong Wu, Hao Feng, Hao Liu, Can Huang, Jingqun Tang, Wei Chen, Lianwen Jin, Yuliang Liu, Xiang Bai
TL;DR
Existing OCR benchmarks leave challenging abilities such as text localization, handwritten extraction, and logical reasoning underexplored despite strong text-recognition results. OCRBench v2 addresses this gap with a broad, multi-task benchmark and private test set, finding substantial weaknesses in current LMMs across harder OCR capabilities.
Problem
Existing benchmarks underexplore text localization, handwritten content extraction, and logical reasoning in LMMs, despite strong reported text-recognition performance.
Method
OCRBench v2 evaluates eight OCR capabilities across 23 tasks and 31 scenarios using 10,000 human-validated pairs, six metrics, and a private 1,500-image test set.
Results
Most evaluated LMMs score below 50 on harder capabilities such as text spotting, element parsing, mathematical calculation, and knowledge reasoning.
Takeaways & Limitations
OCRBench v2 provides broad baselines and identifies critical limitations and factors affecting current LMM OCR performance.
Takeaways & Limitations
Existing benchmarks remain limited in task diversity, instruction complexity, and structured output formats, with some questions answerable without visual input.
Abstract
from arXiv · showhide
Scoring the Optical Character Recognition (OCR) capabilities of Large Multimodal Models (LMMs) has witnessed growing interest. Existing benchmarks have highlighted the impressive performance of LMMs in text recognition; however, their abilities in certain challenging tasks, such as text localization, handwritten content extraction, and logical reasoning, remain underexplored. To bridge this gap, we introduce OCRBench v2, a large-scale bilingual text-centric benchmark with currently the most comprehensive set of tasks (4x more tasks than the previous multi-scene benchmark OCRBench), the widest coverage of scenarios (31 diverse scenarios), and thorough evaluation metrics, with 10,000 human-verified question-answering pairs and a high proportion of difficult samples. Moreover, we construct a private test set with 1,500 manually annotated images. The consistent evaluation trends observed across both public and private test sets validate the OCRBench v2's reliability. After carefully benchmarking state-of-the-art LMMs, we find that most LMMs score below 50 (100 in total) and suffer from five-type limitations, including less frequently encountered text recognition, fine-grained perception, layout perception, complex element parsing, and logical reasoning. The project website is at: https://99franklin.github.io/ocrbench_v2/
1 Introduction
Existing OCR benchmarks do not fully test LMMs on diverse, complex visual-text tasks, while preliminary evaluations reveal failures in localization, handwriting extraction, and reasoning. OCRBench v2 addresses this gap with broad task coverage, human-validated data, systematic model evaluation, and analysis of factors affecting OCR performance.
- Motivation: Existing text-focused datasets often lack diversity and complexity, and some questions can be answered without visual input.These limitations reduce their suitability for evaluating LMM reasoning on scene text.
- Motivation: 96.4% accuracy on DocVQA and 88.8% on OCRBench by Qwen2.5-VL contrast with near-human DocVQA performance and motivate broader evaluation.DocVQA human performance is reported as 98.1%.
- Motivation: Preliminary tests show state-of-the-art LMMs fail on text localization, handwritten content extraction, or document-based logical reasoning.These failures indicate gaps in detailed visual perception for accurate localization, recognition, and contextual understanding.
- OCRBench v2: OCRBench v2 evaluates eight core OCR abilities through 23 tasks, 31 diverse scenarios, 10,000 human-validated instruction-response pairs, and six evaluation metrics.The benchmark is designed for complex, practical OCR scenarios.
- OCRBench v2: A private test set of 1,500 additional text-images is reserved to validate model generalization.The set is collected and labeled independently from the public benchmark.
- Contributions: The paper systematically evaluates commercial and open-source LMMs and analyzes generalization, robustness, and complex visual-textual relations.This establishes broad OCR baselines while examining factors affecting model capabilities.
2 Related Work
Related work includes OCR-enhanced LMMs and text-centric benchmarks spanning documents, charts, infographics, and scene text. As task diversity grows, existing datasets often require specialized handling, making unified scalable evaluation difficult.
- OCR-Enhanced LMMs: OCR-enhanced LMMs use text-centric instruction tuning or increased input resolution to improve OCR-related abilities.Instruction-tuning approaches are restricted to low-resolution inputs, while higher resolution increases computational demands.
- Benchmarks: Text-centric benchmarks target document understanding, chart reasoning, infographic interpretation, scene text comprehension, and broader multi-task evaluation.Examples include DocVQA, ChartQA, Infographics VQA, TextVQA, OCRBench, CONTEXTUAL, and SEED-Bench-2-Plus.
- Benchmarks: Expanding evaluation across diverse text-oriented tasks makes unified and scalable assessment difficult because existing datasets often require task-specific handling.This challenge is illustrated in Figure 2.
3 Why Do We Need OCRBench v2?
Existing OCR benchmarks are limited in task diversity, instruction complexity, and multimodal coverage. OCRBench v2 addresses these gaps with a unified, challenging framework spanning diverse scenarios while focusing on single-image text tasks.
- Existing benchmarks often remain narrow in scope, task diversity, instruction complexity, and structured output formats for LMM evaluation.
- OCRBench v2 combines 23 tasks under 8 core capabilities, enabling systematic model comparison and revealing reasoning–recognition trade-offs.
- The benchmark covers 31 scenarios, spans poorly handled OCR capabilities, uses complex human-authored instructions, and includes structured outputs.
- A private evaluation set is included to help prevent overfitting and training contamination.
- OCRBench v2 focuses on single-image text tasks rather than multi-image tasks, while multi-page evaluation requires separate long-sequence modeling benchmarks.
- The private test set contains 1,500 manually collected text-rich images with human-annotated labels across 23 tasks.
4 Benchmark Construction
OCRBench v2 constructs a multi-capability benchmark by organizing text-centric tasks, diverse data, tailored prompts, manual verification, and category-specific metrics. Its statistics indicate rich OCR content and more diverse, less redundant questions than OCRBench.
- Task Description: OCRBench v2 organizes text-reading evaluation into eight core capability areas with specific sub-tasks addressing text comprehension and interpretation.
- Task Description: Text recognition, referring, spotting, relation extraction, element parsing, calculation, visual text understanding, and knowledge reasoning cover complementary OCR abilities.
- Task Description: Visual text understanding evaluates document classification, diagram QA, and cognition VQA for semantic interpretation of image-embedded text.
- Task Description: Knowledge reasoning includes science QA, app-agent interactions, ASCII-art classification, translation, and reasoning VQA requiring non-visible-answer inference.
- Dataset Construction: The dataset combines 81 manually screened academic datasets with additional private data to cover 31 typical scenarios.
- Instruction Formatting: Task-specific prompts support LMM-compatible instructions, structured outputs, and normalized coordinates for localization evaluation.
- Data Quality: Approximately 1% of annotation errors were corrected through manual review of public-data instructions.
- Statistics: OCRBench v2 shows higher Average Entropy, Type-Token Ratio, and Average Variability Index than OCRBench, indicating more diverse and structurally varied questions.
5 Results and Findings
OCRBench v2 evaluations show that LMMs handle some basic OCR capabilities but remain weak on text localization, structured parsing, and reasoning. Results on public and private data reveal recurring limitations in rare-text recognition, spatial and layout perception, element analysis, and logical reasoning.
- Main Results: Most LMMs score below 50 on text spotting and element parsing, despite stronger performance on text recognition and visual text understanding.Text spotting failures involve imprecise localization and recognition.
- Main Results: Public and private test sets show similar trends, with unsatisfactory performance in text referring, text spotting, element parsing, mathematical calculation, and knowledge reasoning.Closed-source LMMs outperform open-source counterparts on the private data.
- Main Findings: 79.1% accuracy on high-frequency texts drops to 46.7% on low-frequency texts for InternVL3-14B.Low-frequency examples include occluded text, CAPTCHA, and dot-matrix text.
- Main Findings: 78.3% response accuracy on position-based VQA contrasts with only 12.9% IoU for answer-region localization by InternVL3-14B.The model can roughly identify an answer’s location but struggles to output its exact region.
- Main Findings: A 55.7% performance drop occurs when InternVL3-14B processes rotated DocVQA images, falling from 90.9% to 35.2%.The results expose weaknesses with overlapping, handwritten, and rotated text layouts.
- Main Findings: InternVL3-14B declines from 94.4% accuracy in unpaired entity matching to 84.9% in key information extraction.Performance further degrades on element-parsing tasks requiring structured outputs.
- Main Findings: Human-rated accuracies vary across reasoning categories, from 56.5% for calculation to 83.0% for visual-text understanding.The evaluated categories also include common sense reasoning, pattern recognition, and expert knowledge.
6 Conclusion
The paper introduces OCRBench v2 as a broad benchmark for evaluating OCR capabilities in LMMs. It combines diverse tasks, scenarios, metrics, and public and private data to analyze current model limitations and support future research.
- Conclusion: OCRBench v2 covers 23 tasks across 31 diverse scenarios and assesses eight core capabilities for text-oriented visual understanding.The benchmark includes 10,000 high-quality QA pairs and six evaluation metrics.
- Conclusion: A private test set of 1,500 manually labeled images supports robust generalization evaluation.The benchmark is used to analyze representative LMMs and identify factors affecting OCR performance.
A Technical Appendices and Supplementary Material
The supplementary material provides experimental, methodological, statistical, visualization, and broader-impact resources for OCRBench v2. It also includes a dedicated discussion of limitations.
- Supplementary Contents: The supplement includes comparisons with text-centric expert models, data collection details, task definitions, benchmark statistics, evaluation metrics, and experimental settings.It also documents the compute resources used for evaluation.
- Supplementary Contents: Additional materials report LMM evaluation results and discuss factors affecting OCR capabilities.The supplementary sections include visualization samples for task examples and failure cases.
- Supplementary Contents: The supplement contains discussions of broader impacts and limitations.These topics are listed as dedicated supplementary sections.
A.1 Comparison with LMMs and Text-centric Expert Models
Comparisons with specialized text recognizers, text spotters, and GOT position LMMs as strong in general text recognition but weaker in text spotting and document parsing. These comparisons clarify where general-purpose multimodal models retain gaps relative to specialized systems.
- Comparison with Expert Models: LMMs are compared with CRNN, ABINet, ASTER, MASTER, and SVTR on text recognition.The comparison uses five representative LMMs and reports results in Table 6.
- Comparison with Expert Models: LMMs show promising text recognition but notable room for improvement in text spotting compared with ABCNet models and TESTR.The text spotters were fine-tuned with TotalText.
- Comparison with Expert Models: LMMs have advantages in general text recognition, whereas GOT performs better on document parsing.The comparison evaluates GOT against representative LMMs.
A.2 Data Collection
OCRBench v2 assembles task-specific data from diverse public benchmarks, private collections, and manually annotated or re-annotated sources. The collection spans recognition, parsing, reasoning, localization, extraction, and translation tasks.
- Text recognition: Text recognition combines datasets covering regular, scene, handwritten, CAPTCHA, digit-string, and non-semantic text images.Sources include ICDAR2013, SVT, IIIT5K, IAM, CAPTCHA datasets, and others.
- Localization: Localization tasks use established scene-text datasets with official or derived bounding-box annotations for grounding, positional VQA, and text spotting.Text spotting requires simultaneous localization and transcription of all text instances.
- Information extraction: Information extraction and mapping draw on document, receipt, form, and entity-linking datasets, including private photographed-receipt data.Extraction outputs key-value JSON, whereas mapping pairs supplied entity keys with their values.
- Parsing and mathematics: Parsing tasks cover tables, charts, documents, formulas, and mathematical questions using specialized benchmarks and private formula data.The sources include MMTab, OneChart, DoTA, HME100K, MathVerse, and related datasets.
- Reasoning and multimodal understanding: Cognition, reasoning, science, diagram, classification, agent, ASCII-art, and translation tasks combine many multimodal datasets to broaden task coverage.The collection also includes private exam-paper, receipt, movie-poster, and website-screenshot data.
A.3 Task Definitions
OCRBench v2 defines a broad set of text-centric tasks spanning recognition, parsing, counting, question answering, reasoning, localization, extraction, and translation. The tasks target both direct visual reading and higher-level perception or reasoning.
- Text Recognition: Text recognition evaluates reading across regular, irregular, artistic, handwritten, digit-string, non-semantic, occluded, dot-matrix, and CAPTCHA text.This breadth targets recognition across diverse image-text conditions.
- Localization and extraction: Fine-grained recognition, full-page OCR, grounding, positional VQA, and spotting test text comprehension, transcription, localization, or their combination.Positional VQA requires answers and normalized coordinates, while spotting requires localization and content for all text.
- Structured parsing: Parsing tasks convert tables to Markdown or HTML, charts to JSON, documents into text plus complex elements, and formulas into LaTeX.The document task includes charts, tables, and formulas; formula recognition covers mathematical and chemical formulas.
- Additional capabilities: Additional tasks assess counting, diagram relationships, document categories, science, app understanding, ASCII-art abstraction, and Chinese-English translation.Counting covers character frequency and word counts; ASCII-art classification targets pattern recognition and visual abstraction.
- Question answering: Cognition VQA answers questions whose information is explicitly present in the image, whereas reasoning VQA requires logical inference beyond directly visible answers.This split distinguishes fundamental text-centric question answering from reasoning over visual information.
A.4 Additional Statistics of OCRBench v2
OCRBench v2 reports coverage across 31 image scenes and a balanced distribution of textual content across its 23 tasks. The statistics are intended to support broad and adequate task analysis.
- Scene coverage: 31 classic scene categories organize the dataset according to image context.The scene-specific image counts are reported in Table 9.
- Task statistics: 23 tasks are distributed across public English and Chinese data with adequate textual information in each task.The benchmark also reports average OCR text lines per task for both languages.
A.5 Evaluation Metrics
OCRBench v2 evaluates text-centric LMM abilities with task-specific metrics spanning structured parsing, localization, extraction, long reading, counting, and VQA. Results show strong basic OCR performance but persistent weaknesses in localization, complex elements, and reasoning, with consistent private-test trends.
- Parsing: TEDS measures structural similarity between predicted and ground-truth trees for image-to-structure parsing tasks.Identical trees receive a TEDS of 1, while increasing structural differences make the score approach 0.
- Localization: IoU measures overlap between predicted and ground-truth bounding boxes for text referring and spotting.The score compares the intersection area with the union area of the two regions.
- Extraction and long reading: F1 evaluates relation extraction from predicted and ground-truth key-value pairs, while long reading averages BLEU, METEOR, F1, and normalized edit distance.These metrics assess matching quality and text similarity across paragraph- or page-length outputs.
- Long reading: Normalized edit distance measures the operations needed to transform a predicted string into the ground truth, normalized by the longer string length.A value of 0 indicates identical strings, whereas 1 indicates completely different strings.
- Results: Most models score below 50 points overall, performing better on understanding, recognition, and extraction than on referring, spotting, parsing, and calculation.Private-test trends remain consistent with public-test results, while private data generally produces lower scores.