Source-linked AI summary
GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts
Amir Hossein Kargaran, Nafiseh Nikeghbal, Jana Diesner, François Yvon, Hinrich Schütze
TL;DR
Existing OCR benchmarks cover only a narrow range of Unicode scripts, leaving generalization across the writing systems encoded by Unicode insufficiently measured. GlotOCR Bench evaluates OCR models across 158 scripts and finds that current systems largely fail on low-resource scripts, while unfamiliar scripts trigger hallucinated text in familiar scripts.
Problem
Existing OCR benchmarks evaluate mainly Latin, CJK, and a small number of other scripts, leaving OCR performance across the full breadth of Unicode unevaluated.
Method
GlotOCR Bench renders real multilingual texts as clean and degraded images across 158 Unicode scripts and evaluates open-weight and proprietary vision-language OCR models across resource tiers.
Results
Fewer than 7.7% of low-resource sentences achieve character error rate below 5% for the best model, and models hallucinate fluent text in familiar scripts rather than failing silently.
Takeaways & Limitations
OCR generalization is restricted to a handful of scripts, performance broadly tracks script-level pretraining coverage, and unfamiliar scripts elicit characters from known scripts.
Takeaways & Limitations
The benchmark excludes randomly generated character sequences for rare scripts because they lack the linguistic validity required for evaluation.
Abstract
from arXiv · showhide
Optical character recognition (OCR) has advanced rapidly with the rise of vision-language models, yet evaluation has remained concentrated on a small cluster of high- and mid-resource scripts. We introduce GlotOCR Bench, a comprehensive benchmark evaluating OCR generalization across 100+ Unicode scripts. Our benchmark comprises clean and degraded image variants rendered from real multilingual texts. Images are rendered using fonts from the Google Fonts repository, shaped with HarfBuzz and rasterized with FreeType, supporting both LTR and RTL scripts. Samples of rendered images were manually reviewed to verify correct rendering across all scripts. We evaluate a broad suite of open-weight and proprietary vision-language models and find that most perform well on fewer than ten scripts, and even the strongest frontier models fail to generalize beyond thirty scripts. Performance broadly tracks script-level pretraining coverage, suggesting that current OCR systems rely on language model pretraining as much as on visual recognition. Models confronted with unfamiliar scripts either produce random noise or hallucinate characters from similar scripts they already know. We release the benchmark and pipeline for reproducibility. Pipeline Code: https://github.com/cisnlp/glotocr-bench, Benchmark: https://hf.co/datasets/cis-lmu/glotocr-bench.
1 Introduction
Existing OCR benchmarks cover only a narrow range of scripts, leaving Unicode-wide generalization largely unevaluated. GlotOCR Bench addresses this gap with broad script coverage and reports severe failures on unfamiliar, low-resource scripts.
- Existing OCR benchmarks largely evaluate Latin, CJK, and a small number of mid-resource scripts rather than the full breadth of Unicode.
- Unicode encodes 172 scripts, but there is no systematic picture of where current OCR models succeed or fail across them.
- GlotOCR Bench spans 158 Unicode scripts using clean and degraded image variants from carefully curated multilingual text.
- The benchmark evaluates open-weight and frontier OCR models with character error rate and Acc@0/Acc@5 stratified by script and resource level.
- Current OCR generalization is restricted to a handful of scripts, tracks script-level pretraining coverage, and often produces familiar-script hallucinations on unfamiliar scripts.
- The benchmark, rendering pipeline, evaluation code, and per-model results are publicly released.
2 Related Work
Prior OCR benchmarks and low-resource datasets have expanded multilingual coverage unevenly, but script diversity is rarely the primary evaluation axis. Existing resources remain limited by narrow script sets, language-focused design, restricted access, or limited task scope.
- Major OCR benchmarks concentrate on Latin and Chinese, while CC-OCR covers eleven languages and Reasoning-OCR focuses on reasoning from OCR cues.
- The closest multilingual works broaden coverage, but one is not publicly available and neither treats script coverage as the primary evaluation axis.
- Low-resource OCR datasets and adaptation methods typically cover small script or language subsets, including 24 scripts in CAMIO and four writing systems in OmniOCR.
3 GlotOCR Bench
GlotOCR Bench constructs script-diverse OCR data from real multilingual text and renders each sample in clean and degraded forms. The pipeline broadens coverage while enforcing script validity and reproducible rendering choices.
- The benchmark provides two image variants per sentence across 158 scripts: clean white-background renderings and degraded aged-document simulations.
- It samples up to 100 sentences per script, with larger samples for Latin and selected mid-resource scripts, totaling 16,375 sentences.
- Text is assembled from GlotLID v3 and other public sources, prioritizing 30–100-character sentences for broad language–script coverage.
- Newly gathered text is verified with GlotScript, and randomly generated character sequences are excluded because they lack linguistic validity for evaluation.
- Fonts are selected through script-aware metadata and rendered with HarfBuzz shaping and FreeType rasterization, producing uniformly LTR or RTL images.
- The dataset is publicly released for evaluation only, while the rendering pipeline is separately released under Apache 2.0.
4 Evaluation Setup
The evaluation uses zero-shot OCR inference on native-resolution images and compares open-weight and proprietary systems with character- and script-level metrics. Acc@5 measures near-perfect transcription, while ScriptAcc diagnoses script identification and cross-script hallucination.
- All models are evaluated zero-shot with prompts requesting tagged transcriptions and no commentary, using images at their native rendered resolution.
- Table 2 reports macro-averaged A@0 and A@5 across high-, mid-, and low-resource script tiers, identifying best and second-best systems.
- CER is normalized character-level Levenshtein distance, while Acc@k is the fraction of sentences with CER ≤k/100 for k∈{0,5}.
- ScriptAcc measures whether the output uses the correct script regardless of transcription accuracy, helping diagnose cross-script hallucination.
5 Results
OCR performance is strong for Latin and some mid-resource scripts but collapses across low-resource scripts, with errors driven by script coverage, visual complexity, and cross-script hallucination. Script-aware hints provide limited overall benefit, while degradation harms performance across resource tiers.
- Resource-tier performance: Acc@5 reaches 61.9% overall for Gemini 3.1 Flash-Lite, but the three best models achieve only 7.7% on low-resource scripts.The benchmark covers High (Latin), Mid (9 scripts), and Low (148 scripts) tiers.
- Resource-tier performance: Acc@5 drops 27.6 percentage points from High (87.6%) to Mid (60.0%) and 57.7 points from Mid (60.0%) to Low (2.3%).The mid-to-low transition is substantially steeper than the high-to-mid decline.
- Within-script variance: Per-language performance varies substantially: Arabic degrades most severely, while Cyrillic can achieve medians comparable to Latin.Arabic errors reflect visually similar characters, optional diacritics, and orthographic variation across languages.
- Script recognition and OCR: ScriptAcc is a weak but early indicator of Acc@5: Arabic suffers intra-script variation, Hebrew cross-script confusion, and Japanese exceeds Latin despite combining three writing systems.These deviations identify failure modes beyond script recognition alone.
- Script-aware hinting: 125 of 149 scripts show no change under GPT-4.1 script-aware hinting, for a mean gain of only +0.7 percentage points.Hani, Cyrillic, and Thai benefit selectively from reduced character ambiguity, while low-resource scripts remain largely unsolved.
- Failure modes: Cross-script hallucination dominates failures: models assign only 12.5% of predictions to the correct script and hallucinate across scripts at 68.4%.Models overwhelmingly confabulate in recognizable wrong scripts rather than abstaining.
6 Conclusion
GlotOCR Bench evaluates OCR generalization across 158 Unicode scripts using clean and degraded images rendered from real multilingual texts. Across 14 vision-language models, performance is strong on Latin but collapses for most low-resource scripts, while hallucinated familiar-script text and limited hinting expose training coverage as a key bottleneck.
- GlotOCR Bench spans 158 Unicode scripts and includes clean and degraded image variants rendered from real multilingual texts.
- 14 open-weight and proprietary vision-language models perform strongly on Latin but degrade substantially on mid-resource scripts.
- Fewer than 7.7% of sentences from the 148 low-resource scripts meet the character error rate below 5% for even the best model.
- Models hallucinate fluent text in familiar scripts, while script-aware hinting provides only marginal transcription-accuracy gains.
A Rendering Pipeline Details
The rendering pipeline produces clean and degraded OCR images while supporting script-aware shaping, rasterization, and bidirectional text. Degradation simulates physical-document and digitization artifacts through geometric, photometric, and morphological transformations validated by human inspection.
- Images use HarfBuzz for text shaping and FreeType for glyph rasterization, with mixed bidirectional sentences excluded and all text uniformly LTR or RTL.
- Clean variants render 48px text on a plain white 1000px-wide canvas with 40px padding and up to ±1° random rotation.
- Degraded variants begin with randomly cropped scanned-paper textures and rotations of up to ±2° to simulate aged documents and page tilt.
- The degradation pipeline combines elastic deformation, Gaussian noise, ink dropout, intensity scaling, texture noise, spacing changes, morphology, line jitter, and page-curl displacement.
- These transformations model document-capture artifacts and OCR augmentations, and were human-validated across Latin, Greek, Cyrillic, and Arabic scripts.
B Per-Script Results
Per-script results report Acc@5 and ScriptAcc for each evaluated model, alongside sentence counts and output-script diagnostics. Additional tables identify scripts that all models fail to recognize or transcribe.
- Table 4 lists scripts with zero ScriptAcc for every model, together with benchmark sentence counts and the two most frequent output scripts.
- Tables 5 and 6 report per-script Acc@5 and ScriptAcc percentages for all evaluated models, with sentence counts and frequently observed output scripts.
- Table 4 identifies scripts for which all models obtain zero in both Acc@5 and ScriptAcc, meaning no model identifies those scripts.