PaperlayerView exact revision on arXiv

Source-linked AI summary

CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognition

Sina J. Semnani, Han Zhang, Xinyan He, Merve Tekgürler, Monica S. Lam

arXiv:2509.19768v1cs.CLcs.CV

TL;DR

Historical documents are difficult to recognize because their languages, scripts, layouts, and condition vary, despite their value as cultural records. The paper builds CHURRO-DS and fine-tunes CHURRO for historical text recognition, outperforming Gemini 2.5 Pro on printed and handwritten test text while being more cost-effective.

  • Problem

    Historical text recognition is challenging because documents vary in language, script, layout, and condition, although they are valuable primary sources.

    Source: §1 · p. 1

  • Method

    The authors curate CHURRO-DS and fine-tune a compact vision-language model for historical text recognition.

    Source: §5 · pp. 8–9

  • Results

    82.3% (printed) and 70.1% (handwritten): CHURRO surpasses Gemini 2.5 Pro by 1.4% and 6.5%, respectively, while being 15.5 times more cost-effective.

    Source: §5 · p. 9

  • Takeaways & Limitations

    CHURRO-DS provides a diverse resource for benchmarking and improving VLMs on historical texts.

    Source: §5 · p. 9

    CHURRO-DS underrepresents some languages and includes no languages native to the African continent.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Accurate text recognition for historical documents can greatly advance the study and preservation of cultural heritage. Existing vision-language models (VLMs), however, are designed for modern, standardized texts and are not equipped to read the diverse languages and scripts, irregular layouts, and frequent degradation found in historical materials. This paper presents CHURRO, a 3B-parameter open-weight VLM specialized for historical text recognition. The model is trained on CHURRO-DS, the largest historical text recognition dataset to date. CHURRO-DS unifies 155 historical corpora comprising 99,491 pages, spanning 22 centuries of textual heritage across 46 language clusters, including historical variants and dead languages. We evaluate several open-weight and closed VLMs and optical character recognition (OCR) systems on CHURRO-DS and find that CHURRO outperforms all other VLMs. On the CHURRO-DS test set, CHURRO achieves 82.3% (printed) and 70.1% (handwritten) normalized Levenshtein similarity, surpassing the second-best model, Gemini 2.5 Pro, by 1.4% and 6.5%, respectively, while being 15.5 times more cost-effective. By releasing the model and dataset, we aim to enable community-driven research to improve the readability of historical texts and accelerate scholarship.

1 Introduction

Historical documents are valuable primary sources, but their varied languages, scripts, layouts, and degradation make recognition difficult. The paper introduces CHURRO-DS and a fine-tuned VLM that outperforms evaluated VLMs on historical text recognition.

1 Introduction

  • Historical recognition is complicated by degradation, marginalia, rubrication, illumination, inconsistent spelling, and varied printed fonts.Handwriting adds diverse styles and calligraphic traditions, as well as thousands of historical abbreviations.Source: §1 · pp. 1–2
  • Existing historical OCR commonly depends on collection-specific annotated training data, while standard VLM benchmarks primarily target modern documents.The authors identify reliance on annotated training data as a significant bottleneck.Source: §1 · p. 2
  • CHURRO-DS unifies over 150 datasets into a diverse historical-text resource spanning 99,491 documents and 46 language clusters.Its pages pair images with full-text annotations and cover material from the 3rd century BC to the 20th century.Source: §1 · p. 2

We introduce

  • 82.3% (printed) and 70.1% (handwritten): CHURRO exceeds all evaluated open and commercial VLMs by at least 1.4% and 6.5%, respectively.The model is fine-tuned from a 3-billion-parameter VLM on CHURRO-DS.Source: CHURRO, a low-cost open-weight VLM… · p. 2
  • The study benchmarks VLMs and OCR systems, finding substantial room to improve zero-shot VLM recognition of historical documents.The best commercial VLM averages 80.9% (printed) and 63.6% (handwritten), while the best evaluated open-weight VLM averages 72.7% and 54.5%, respectively.Source: CHURRO, a low-cost open-weight VLM… · p. 2

2 Related Work

Prior document benchmarks and VLM research largely address modern documents, short outputs, or limited language coverage. CHURRO-DS extends page-level text recognition toward curated, human-annotated historical transcriptions across many languages.

2.1 Document Understanding Benchmarks

  • Modern-document benchmarks cover layout, classification, retrieval, and multimodal tasks, but text-recognition benchmarks often focus on short outputs and modern English or Chinese documents.Some page-level OCR studies evaluate noisy LLM-annotated data or only English and Chinese; CHURRO-DS contributes human-annotated multilingual page-level transcriptions with explicit reading order.Source: §2.1 · pp. 2–3

2.2 Vision-Language Models

  • VLM research spans contrastive pretraining, instruction-tuned models, variable-resolution images, localization, document layout, and proprietary vision-capable models.Recent work also extends explicit reasoning capabilities of LLMs to vision.Source: §2.2 · p. 3

3 The CHURRO-DS Dataset

CHURRO-DS standardizes historical page images and transcriptions into a benchmark and training resource for printed and handwritten text. Its curation combines selection criteria, annotation harmonization, metadata, reading-order processing, and balanced evaluation splits.

3 The CHURRO-DS Dataset

  • CHURRO-DS turns printed and handwritten historical pages into a benchmark and training resource by standardizing each page as one text string in reading order.This page-level image-to-text format enables straightforward VLM evaluation.Source: §3 · p. 3

3.1 Dataset Selection

  • Selected corpora must predate the mid-20th century, permit research use, and provide human-annotated gold text for entire pages.The collection also includes previously untapped resources without bounding boxes and the authors' publicly available Ottoman Turkish HTR dataset.Source: §3.1 · p. 4

3.2 Dataset Curation

  • Curation standardizes transcriptions to diplomatic text in proper reading order, using automated transformations and manual verification.VLM-assisted reading-order outputs were manually spot-checked, with more than 98% of checked reading orders accurate; corrections to identified annotation issues were reviewed and validated by the authors.Source: §3.2 · p. 4
  • The authors use a VLM and manual review to assign language metadata, grouping languages into clusters based on ISO macrolanguage classifications.Scripts are classified using ISO 15924 and Glottolog, yielding 14 scripts from five families.Source: §3.2 · p. 4

3.3 Dataset Preparation

  • Preparation removes pages with fewer than 30 tokens and approximately 15,000 near-duplicates, then resizes large images while preserving aspect ratios.Seventeen language clusters occur only in training because examples are limited for some languages.Source: §3.3 · p. 5
  • Balanced validation and test sets sample 60 pages per language cluster and document type, leaving the remaining pages for training.Only 29 of 46 clusters appear in evaluation sets, which each contain 1,170 samples; training contains 97,151 samples.Source: §3.3 · p. 5

4 Experiments and Results

The experiments compare fine-tuned and zero-shot VLMs with OCR systems on printed and handwritten historical documents, using normalized Levenshtein similarity. CHURRO leads on both subsets; the error analysis highlights reading-order mistakes, hallucinations, and annotation and scan-related issues.

4.1 Experimental Setup

  • The study evaluates zero-shot VLMs and OCR systems, then fine-tunes 3B Qwen 2.5 VL on CHURRO-DS; hybrid OCR–VLM and oracle systems are also assessed.The hybrid uses Azure OCR bounding boxes and reading order before VLM transcription; the oracle selects the highest-scoring model per example.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 Evaluation Metrics

  • Normalized Levenshtein similarity is one minus character edit distance divided by the longer string length, averaged separately for printed and handwritten documents.Arabic text receives additional diacritic and hamza normalization.Source: §4.2 · p. 6

4.3 Evaluation Results on Printed Documents

  • On print, fine-tuning improves Qwen 2.5 VL (3B) by 14.5% on average; CHURRO beats Gemini 2.5 Pro on 12 of 18 languages and matches it on Czech.Chinese print is weak, which the authors attribute to only 6 training examples for that language.Source: §4.3 · p. 6
  • For print, Gemini 2.5 Pro leads proprietary VLMs at 80.9%, while Azure OCR leads OCR systems at 71.9%; the Azure–Gemini hybrid trails both components overall.Among open-weight VLMs, NuMarkdown reaches 72.7%; the hybrid improves over Azure alone for Bengali and Romanian. Model size does not clearly correlate with performance within families.Source: §4.3 · p. 6

4.4 Evaluation Results on Handwritten Texts

  • On handwriting, CHURRO beats Gemini 2.5 Pro on 18 of 21 languages; the Azure–Gemini hybrid does not outperform either component, and VLMs outperform OCR systems across both subsets.Gemini 2.5 Pro averages 63.6%; the best open-weight VLM, Qwen 2.5 VL (72B), averages 54.5%. Sanskrit, Khmer, and Hebrew remain difficult even for the oracle.Source: §4.4 · pp. 6–7

4 Experiments and Results

  • CHURRO achieves the highest average performance on both subsets: 82.3% for printed documents and 70.1% for handwritten texts.The reported scores are normalized Levenshtein similarity.Source: §4.3 · p. 6, §4.4 · p. 6, §4.2 · p. 6

4.5 Error Analysis

  • In 50 predictions, major reading-order errors occur in 42% of zero-shot Qwen 2.5 VL (3B) outputs versus 16% for CHURRO, especially on multi-column pages and East Asian top-to-bottom scripts.The oracle reaches only 26.1% for Sanskrit, 32.9% for Khmer, and 59.1% for Hebrew; Greek and Turkish also challenge all evaluated zero-shot models.Source: §4.5 · pp. 7–8, §4.4 · p. 7
  • In a 50-example review, 36% of zero-shot Qwen 2.5 VL (3B) predictions exhibited major hallucinations; recurring minor errors included confusing visually similar characters and scan-curvature errors near the spine.Two of the 50 examples had gold-standard text that omitted parts of the page.Source: §4.5 · p. 8

5 Conclusion

CHURRO is a compact model fine-tuned for historical text recognition, supported by CHURRO-DS, a diverse page-level dataset. It outperforms the best-performing commercial VLM reported in the conclusion on printed and handwritten text.

5 Conclusion

  • CHURRO is a compact vision-language model fine-tuned from Qwen 2.5 VL for historical text recognition.CHURRO-DS was curated to support its training and evaluation.Source: §5 · p. 8
  • CHURRO-DS unifies 155 corpora and nearly 100,000 pages spanning 22 centuries across 46 language clusters.The dataset is described as a resource for benchmarking and improving VLMs on historical texts.Source: §5 · p. 9
  • 82.3% (printed) and 70.1% (handwritten) normalized Levenshtein similarity let CHURRO surpass Gemini 2.5 Pro’s 80.9% and 63.6%, respectively.The conclusion reports that CHURRO is 15.5 times more cost-effective and improves 14.5% and 27.2% over its base model.Source: §5 · p. 9
  • Even imperfect model outputs can serve as first drafts that reduce scholars’ correction effort and make historical documents accessible to broader audiences.The passage includes readers without specialized training in historical scripts among those who may benefit.Source: §5 · p. 9

Limitations

CHURRO-DS still underrepresents some languages, reflecting a scarcity of annotated historical datasets for lower-resource languages. The authors also tested only standard supervised fine-tuning.

Limitations

  • CHURRO-DS underrepresents some languages because annotated historical datasets are scarce for many lower-resource languages.The dataset currently includes no languages native to the African continent.Source: Limitations · p. 9
  • CHURRO-DS currently includes no languages native to the African continent.This is a stated boundary of the dataset’s current coverage.Source: Limitations · p. 9
  • The experiments used only standard supervised fine-tuning; the authors identify more advanced training methods as future work.The stated limitation concerns the training methods explored in this setting.Source: Limitations · p. 9

Ethical Considerations

The authors report that no crowdsourcing was conducted and that dataset licenses permit research use. They do not anticipate potential harm from using the dataset.

Ethical Considerations

A.1 Dataset Statistics

The appendix describes dataset statistics through figures and language/script counts, then records model identifiers, hyperparameters, and the zero-shot prompt location. It specifies the training setup and input-size limits for fine-tuning.

A.1 Dataset Statistics

  • Figure 3 plots the distribution of gold-text token lengths on a log-scale x-axis, with a dashed line marking the median.Source: Figure 3 · p. 19, §A.1 · p. 19
  • Figure 4 plots each CHURRO-DS image by width and height; large images are resized within 2500 × 2500 pixels while preserving aspect ratio.Source: Figure 4 · p. 19

A.2 Language and Script Statistics

  • Tables 3 and 4 report the number of CHURRO-DS examples by language and script.Source: §A.2 · p. 19

B.1 Model IDs

  • Table 5 lists experiment model IDs; closed-model IDs follow provider APIs, open-weight IDs come from HuggingFace, and models are referred to by total parameter count.The passage notes this parameter-count convention is used even when an official model ID differs.Source: §B.1 · p. 19

B.2 Hyperparameters

  • VLM outputs use temperature 0 and a single run; fine-tuning uses 5 epochs, an effective batch size of 128, and a 5 × 10^-5 learning rate.The experiments resize inputs to at most 5,120 image patches of 28 × 28 pixels, keeping image and text tokens below 25,000 per example.Source: §B.2 · p. 20

B.3 Prompt Used for Zero-Shot Evaluation of VLMs

  • The zero-shot VLM evaluation prompt is shown in Table 6.Source: §B.3 · p. 20

C Additional Experiments

Additional experiments find that accuracy generally declines as page text grows; the example outputs illustrate common error categories using zero-shot Qwen 2.5 VL and CHURRO.

C Additional Experiments

  • Accuracy generally decreases as the amount of text on a page increases.Figure 5 plots normalized Levenshtein similarity against the number of tokens in the gold transcription.Source: C Additional Experiments · p. 20

D Example System Outputs

  • The example outputs compare gold text with zero-shot Qwen 2.5 VL and fine-tuned CHURRO across common error categories.The examples are presented side by side with the correct gold output.Source: D Example System Outputs · p. 20

D.1 Reading Order-Related Errors in Vertical Script Layouts

On a Classical Chinese page with vertical, right-to-left text, CHURRO preserves the reading order while the zero-shot VLM produces an incoherent transcription with omissions and rearrangements.

D.1 Reading Order-Related Errors in Vertical Script Layouts

  • CHURRO’s transcription closely matches the ground truth, preserving reading order, line segmentation, and content without hallucinated tokens.Most discrepancies are minor normalization of historical glyph variants or substitutions of rare characters not fully represented in modern Unicode.Source: §D.1 · p. 20
  • The zero-shot VLM produces a fragmented transcription with hallucinated markup, omissions, and rearrangements.The page uses vertical top-to-bottom text and right-to-left column order.Source: §D.1 · p. 20
  • The authors attribute the zero-shot errors to a layout assumption of left-to-right, top-to-bottom reading rather than the page’s traditional format.This misalignment degrades both recognition accuracy and semantic structure.Source: §D.1 · p. 20

D.2 Recognition Challenges with Small Characters

The examples show that CHURRO handles dense small characters and difficult reading orders more successfully than zero-shot VLMs, while errors also include stereotype-driven hallucinations and repetitive degeneration.

D.2 Recognition Challenges with Small Characters

  • Figure 5 compares document length with normalized Levenshtein similarity for Gemini 2.5 Pro and CHURRO.Gemini 2.5 Pro is shown on the left and CHURRO on the right.Source: Figure 5 · p. 21
  • Figure 6 exemplifies major zero-shot VLM mistakes on a document with unusual reading order.The accompanying example is a Classical Chinese page with vertical text and right-to-left columns.Source: §D.2 · p. 20, Figure 6 · p. 21
  • CHURRO transcribes a dense Classical Chinese page with large and small characters while preserving reading order; the zero-shot VLM misses most small text.CHURRO’s remaining mismatches are minor and attributed to character encoding limitations.Source: §D.2 · pp. 20, 22

D.3 Reading Order in Western Languages

  • On a Western postcard, CHURRO recognizes most text but reverses its order because it misses that the text begins on the right and continues at the top left.Source: §D.3 · p. 22

D.4 Major Hallucinations from Stereotypes

  • A zero-shot VLM turns partial recognition of “India” and “queen” into a stereotyped but factually mistaken story about travel to Calcutta.The passage notes that statistically likely historical patterns can still produce factual mistakes when used to generate text.Source: §D.4 · p. 22

D.5 Repetitive Degeneration

  • A zero-shot VLM repeats a conventional German letter-opening sentence over a dozen times after recognizing the document as a letter.On a Latin manuscript, both zero-shot and fine-tuned models show degeneration; the manuscript remains difficult even for CHURRO.Source: §D.5 · p. 22

D.6 Errors Due to Historical Script Changes

The paper notes that historical language differences can include script changes, using transitional Romanian as an example of a nineteenth-century alphabet that combined Cyrillic and Latin characters.

D.6 Errors Due to Historical Script Changes

  • Transitional Romanian combines Cyrillic and Latin characters, and zero-shot VLMs default to Cyrillic spellings when they fail to recognize the Latin characters in a nineteenth-century printed book.These mistakes are drastically reduced in CHURRO.Source: §D.6 · p. 22

D.7 Strict Evaluation

This appendix distinguishes orthographic normalization from recognition errors, documents model failures, and catalogs the historical-text resources unified for CHURRO-DS. The resources span varied handwriting, medieval multilingual manuscripts, administrative records, and multiple document domains, including Sanskrit.

D.7 Strict Evaluation

  • Modernizing German sharp S to standard “ss” is treated as orthographic normalization, while figures show zero-shot failures on small text and repetitive degeneration, plus mistakes by CHURRO on a Danish postcard.The captions also document repetitive degeneration by both models on a Latin manuscript and a zero-shot failure to recognize Latin characters in a Romanian book.Source: §D.7 · p. 22, Figure 7 · p. 23, Figure 8 · p. 23

E List of 155 Sources Used to Create CHURRO-DS

Corpus of Spanish Documents Prior to 1700 (CODEA) (Research Group Texts for the His-

  • The catalogue includes Spanish documents from the 12th–17th centuries, historical Arabic books, Romanian texts mixing Latin and Cyrillic, and Dutch handwriting materials.The Spanish corpus description cites 1,500 documents; the Romanian transitional-script texts cover 1833–1864, and the Dutch materials include VOC and notarial pages.Source: Corpus of Spanish Documents Prior to… · pp. 30–32
  • The cited resources span Bentham handwriting with variable image quality and crossed-out text, plus Notre-Dame registers written by at least 18 hands, more than 98% in Latin, in cursive, for daily administration.The Faithful Transcriptions dataset covers 181 pages and 8.952 text lines from 12 manuscripts in German, Dutch, and Latin.Source: Corpus of Spanish Documents Prior to… · p. 34

ICDAR2017 Competition on Handwritten Text Recognition on the READ Dataset (Maheshwari et al., 2022a)

This section surveys historical text-recognition datasets and resources across varied collections and document traditions. It covers newspapers and manuscripts spanning diverse languages, scripts, periods, formats, and recognition challenges.

ICDAR2017 Competition on Handwritten Text Recognition on the READ Dataset (Maheshwari et al., 2022a)

NewsEye / READ OCR training dataset from Swedish Newspapers (18th, 19th, early 20th C.)

TRANSCRIPTIONS OF THE INTERNATIONAL CONGRESS OF ETHNOGRAPHIC

Handwritten Paleographic Greek Text Recognition: A Century-Based Approach (Platanou et al., 2022)

  • The Barocci collection contains 244 volumes dated from the 8th to 17th centuries, with examples of Greek minuscule and cursive writing.Its varied writing styles make the collection broader than a single script style.Source: Handwritten Paleographic Greek Text… · p. 43
  • EPARCHOS derives from a Greek codex dated around 1500–1530, written by two hands and containing common minuscule abbreviations, logograms, and conjunctions.The dataset covers texts by Hierocles, Matthaeus Blastares, and Michael Psellos.Source: Handwritten Paleographic Greek Text… · pp. 44–45
  • A human-annotated OCR dataset provides scanned images and OCR texts from 19th- and 20th-century letterpress reprints of Hussite-era documents.Source: Handwritten Paleographic Greek Text… · p. 46
  • Khmer palm-leaf recognition is complicated by vowels placed around consonants, stacked consonant forms, overlapping lines, and visually ambiguous characters.Some character distinctions depend on small holes or short strokes.Source: Handwritten Paleographic Greek Text… · pp. 46–47
  • The Finlam La Liberté dataset contains 1,500 issues of a French newspaper published from 1925 to 1928.It is listed with an MIT License.Source: Handwritten Paleographic Greek Text… · p. 47
  • The Osman Agha dataset contains 243 images of an 18th-century Ottoman manuscript and its Arabic-script transcription, aligned at page level.The manuscript memoir was completed in 1724 and survives as a single authorial copy held at the British Library; research use is permitted.Source: Handwritten Paleographic Greek Text… · p. 47
  • American Stories comprises newspaper scans from the Library of Congress’s Chronicling America collection.The passage lists an Apache License 2.0.Source: Handwritten Paleographic Greek Text… · p. 47
Loading 2509.19768v1…