PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comparative Evaluation of Digitization Pipelines for Historiographical Sources

Marina Gómez Rey, Patricia Callejo, Mario Muñoz-Organero, Carlos Alario-Hoyos

arXiv:2608.24976v1cs.DL

TL;DR

Historical-document OCR errors can propagate into retrieval and generation systems, but evidence on extraction pipelines for heterogeneous historiographical sources remains limited. This study evaluates thirteen pipelines and finds that Marker’s end-to-end parsing performs best overall, while LLM post-correction often degrades accurate extractions.

  • Problem

    Historical documents’ complex layouts, typography, degradation, and mixed content types create OCR errors that can propagate into factually incorrect RAG outputs.

  • Method

    The study systematically evaluates thirteen extraction pipelines on a stratified corpus of Visigothic historiographical sources using character- and word-level accuracy against manual ground truth.

  • Results

    98.70% CER accuracy overall is achieved by Marker, while conventional OCR degrades on complex layouts and scans, embedded-text extraction fails on scans, and LLM correction often degrades accurate outputs.

  • Takeaways & Limitations

    End-to-end parsing is the most reliable approach for heterogeneous historical collections, while LLM post-correction requires validation before large-scale use.

  • Takeaways & Limitations

    The corpus is limited to Spanish historiographical texts on one topic, so generalisation to other languages, domains, and periods requires validation.

Abstract

from arXiv · show

Purpose: The digitization of historical documents presents fundamental challenges for modern information retrieval and Artificial Intelligence (AI) systems. Optical character recognition (OCR) errors in source corpora propagate through retrieval-augmented generation (RAG) pipelines, compromising the factual accuracy of generated outputs. Methods: This study presents a systematic evaluation of PDF-to-text extraction pipelines applied to historiographical secondary sources on the Visigothic period. We assess thirteen distinct approaches spanning three methodological families: direct extraction, Large Language Model (LLM) post-correction, and chunk-and-extract. Documents are stratified into five categories based on production method and visual complexity. Performance is measured using character error rate (CER) and word error rate (WER) against manually corrected ground truth. Results: Results demonstrate that direct extraction with Marker achieves superior performance (98.70% CER accuracy; 97.71% WER accuracy overall), while conventional OCR pipelines exhibit substantial degradation on scanned documents and complex layouts. Embedded-text extraction performs well on digital PDFs but fails on scanned documents. LLM post-correction does not provide systematic improvements and frequently degrades accurate extractions. Conclusion: End-to-end document parsing is the most reliable approach for heterogeneous historical collections. Document characteristics such as scan quality, layout complexity, and the presence of embedded text layers have a significant impact on extraction accuracy. LLM-based post-correction should not be assumed beneficial by default and requires validation before large-scale application.

1 Introduction

Historical-document digitization errors can propagate through retrieval and generation, undermining the reliability of AI systems using historical sources. This study evaluates thirteen PDF-to-text pipelines on heterogeneous Visigothic historiographical sources and examines how document characteristics and LLM post-correction affect extraction quality.

  • Motivation: OCR substitutions, segmentation failures, and layout corruption can cause RAG systems to miss relevant chunks or pass corrupted content to generation models.Historical materials commonly contain multi-column layouts, archaic orthography, physical degradation, and mixed content types that produce these errors.
  • Motivation: “Liuva 11” instead of “Liuva II” illustrates how OCR misrecognition of a Roman numeral can produce a confident but factually incorrect historical answer.Conversational AI systems may present outputs confidently regardless of source quality, making digitization errors difficult for non-experts to detect.
  • Research gap: Existing solutions do not reliably digitize complex historiographical documents, while systematic comparative evaluations of heterogeneous historical corpora remain limited.Prior work has explored neural OCR, vision-language models, LLM-based post-correction, hybrid layout analysis, and end-to-end document parsing.
  • Study design: The study evaluates thirteen extraction pipelines on a stratified Visigothic corpus spanning poor scans, modern digitizations, native digital PDFs, and complex layouts.The corpus also includes Latin terms, proper nouns, and Roman numerals to stress-test OCR systems beyond standard benchmarks.
  • Contributions: Document production method, scan quality, and layout complexity systematically affect extraction quality, while LLM-based post-correction can degrade performance under standard edit distance metrics.These findings have direct implications for workflow design and tool selection.

2 Related Work

Related work traces OCR’s evolution toward neural and unified document-processing architectures while emphasizing persistent challenges from historical document characteristics. It also examines LLM-based OCR correction and the cascading effects of document quality and positioning on RAG reliability.

  • OCR systems: OCR has progressed from rule-based pattern matching to neural architectures that combine convolutional recognition with recurrent sequence modeling.Contemporary approaches substantially improve accuracy over traditional methods, although performance varies across document types and quality levels.
  • Historical documents: Historical OCR is challenged by non-standard typography, physical degradation, and archaic linguistic forms.Prior work also addresses data scarcity through convolutional segmentation and recurrent recognition models for 19th-century German Fraktur.
  • Document structure: Document-processing systems have evolved from whitespace rectangle-based structure detection toward unified architectures for layout detection, table recognition, and reading-order determination.
  • LLM post-correction: LLM-based OCR post-correction produces improvements that depend on base OCR quality and language characteristics.Constraining outputs to valid lexical items can reduce hallucination while preserving correction capability.
  • RAG reliability: OCR errors cascade through RAG pipelines, corrupting retrieval and generation, while document type and prompt position influence RAG effectiveness.Document positioning also affects LLM attention distribution and accuracy.

3 Methodology

The study evaluates thirteen PDF-to-text pipelines on a heterogeneous corpus of Visigothic historiographical sources, using manually transcribed ground truth. Documents are stratified by scan quality, layout complexity, and embedded-text availability to test distinct extraction challenges.

  • Corpus construction: The corpus comprises 14 documents, yielding 15 document units, distributed across five categories of three documents each.The sources are Spanish academic publications on the Visigothic kingdom (415–721 CE), selected to represent challenges in historical digitization.
  • Document categorization: The five categories cover annotated low-quality scans, complex multi-column layouts, clean scans, digital PDFs, and documents without embedded text layers.These categories distinguish visual degradation, reading-order complexity, ideal embedded-text conditions, and image-only documents requiring OCR.
  • Ground truth preparation: Ground truth consists of manual transcriptions of approximately 15 pages per document, saved as plain text after editing.Manual transcription was used to maximize accuracy while maintaining a representative workload.
  • Pipeline design: Thirteen pipelines are compared across three methodological families: direct extraction, LLM post-correction, and chunk-and-extract.All pipelines process the same corpus and are evaluated against the manually corrected ground truth.
  • Direct extraction: Direct extraction uses PyMuPDF, Docling, and Marker, with Docling tested in three configurations.The PyMuPDF-based implementation converts PDFs to Markdown and depends on embedded text, while Docling supports layout analysis and OCR capabilities.

2. Docling

Docling configures Tesseract OCR with LSTM-based recognition, preprocessing, layout analysis, and region segmentation. Its performance typically declines on noisy images, complex layouts, and unusual fonts.

  • 2. Docling: Docling configures Tesseract with OCR enabled, using LSTM-based recognition, preprocessing, layout analysis, and region segmentation.
  • 2. Docling: Performance typically degrades on noisy images, complex layouts, or unusual fonts.

3. Docling

The section describes document-parsing and correction pipelines that combine layout-aware extraction, OCR options, paragraph-level LLM post-correction, and recursive image-based chunking. These approaches target consistent handling of digital, scanned, and complex-layout documents while controlling extraction errors and context length.

  • Marker: Marker uses an end-to-end Surya-based pipeline to convert documents into structured formats.It integrates document understanding rather than chaining separate extraction and processing tools.
  • Marker: Marker’s three stages extract content, refine reading structure, and render the final format.Processing includes layout detection, text recognition, table parsing, equation identification, span merging, hyphenation correction, and reading-order establishment.
  • LLM post-correction: Paragraph-level Qwen3 correction was prompted to preserve content while fixing localized OCR errors.Paragraph-wise processing manages context lengths and the instructions prohibit invention, rewriting, summarization, and extra output.
  • LLM post-correction: Five post-correction pipelines combined distinct direct extraction methods with the same LLM correction step.The combinations used PyMuPDF, Docling without OCR, Docling with Tesseract, Docling with EasyOCR, or Marker, each followed by LLM processing.
  • Chunk-and-extract: The chunk-and-extract method first segments page images into text blocks before extraction.Pages were converted with PyMuPDF and OpenCV at 350–450 DPI, while recursive chunking addressed oversized blocks and multiple-column layouts.

1. Chunk

The chunk pipeline segments documents into blocks for vision-based or OCR extraction, addressing layout detection before word-level recognition. The study evaluates established parsing and OCR tools using standard edit-distance metrics with document-balanced macroaverages.

  • Chunk-based extraction: Each extracted block is submitted to the 8B-parameter MiniCPM-V vision model, which first checks for text before returning the extracted content.The prompt instructs the model to return “NO TEXT” when no text is present.
  • Chunk-based extraction: EasyOCR is applied to well-segmented blocks because chunking addresses its main limitation, layout detection, while retaining its word-level extraction strength.The method expects block-level processing to improve reliability by separating layout determination from text recognition.
  • Tool selection: The evaluation selected tools based on open-source ecosystem adoption and established use as Document AI and vision-language-model baselines.Included examples are Tesseract, EasyOCR, PyMuPDF, Docling, MiniCPM-V, and Qwen3-8B.
  • Tool selection: Tesseract represents the traditional LSTM OCR baseline, whereas EasyOCR represents deep-learning scene-text recognition using CRAFT detection.The passage reports over 72,000 GitHub stars for Tesseract and approximately 29,000 for EasyOCR.
  • Evaluation metrics: CER and WER are computed with Levenshtein edit distance, while CER Accuracy and WER Accuracy are reported as 1 − CER and 1 − WER.The metrics count substitutions, deletions, and insertions at character or word level; macroaverages weight documents equally.

4 Results

Marker achieves the strongest overall extraction accuracy across the evaluated document types, while performance varies substantially by document characteristics and pipeline family. Embedded-text methods fail without text layers, LLM post-correction often degrades results, and chunk-based approaches perform poorly or exhibit hallucination-related errors.

  • Overall performance: 98.70% CER accuracy and 97.71% WER accuracy make Marker the best overall pipeline.Marker substantially outperforms all alternatives in macro-averaged accuracy across document types.
  • Overall performance: 89.06% CER accuracy and 82.28% WER accuracy make Docling + Tesseract the strongest conventional OCR approach.Docling + EasyOCR follows with 85.39% CER and 78.07% WER.
  • Document-type effects: 0% accuracy on Type 5 documents causes embedded-text extraction approaches to fail when no embedded text layer exists.For Types 1–4, PyMuPDF achieves 84.05% CER accuracy and 79.43% WER accuracy, while Docling (No OCR) achieves 83.42% CER and 77.85% WER.
  • Pipeline-family effects: LLM post-correction reduces Docling + Tesseract from 89.06% CER accuracy to 78.77% CER accuracy, while chunk-based systems show poor or hallucination-prone results.Chunk + MiniCPM-V achieves 9.36% CER accuracy with negative accuracy values on several document types; Chunk + Qwen3-VL reaches 47.76% CER accuracy.
  • Document-type effects: 98.13% CER accuracy and 96.62% WER accuracy show Marker remains robust on multi-column complex layouts, where PyMuPDF reaches 57.27% CER accuracy.Forcing OCR through Docling improves complex-layout performance to 84.66% CER accuracy with Tesseract and 82.39% CER accuracy with EasyOCR.
  • Document-type effects: 99.56% CER accuracy and 98% WER accuracy are achieved by Marker on clean scans, while conventional OCR and direct extraction also perform strongly.Docling + Tesseract reaches 95.41% CER accuracy, Docling + EasyOCR 91.97% CER accuracy, and Docling No OCR 91.93% CER accuracy.

5 Discussion

The discussion finds that end-to-end parsing with Marker is the most robust approach across heterogeneous historical documents, whereas conventional OCR, embedded-text extraction, and vision-language pipelines exhibit condition-dependent or severe failures. LLM post-correction generally reduces edit-distance accuracy, and the findings require broader validation beyond the Spanish historiographical corpus and current metrics.

  • Pipeline families: Marker substantially outperforms alternatives and maintains over 97% CER accuracy across all document categories.Its end-to-end layout detection and text recognition appear to generalize across production methods and document quality levels.
  • Document heterogeneity: Conventional OCR degrades with poor scan quality and complex layouts, while embedded-text extraction succeeds only when high-quality text layers exist.Multi-column documents trigger segmentation, substitution, and layout errors; true scans can produce empty or fragmentary outputs.
  • Pipeline families: −153.60% CER for Chunk + MiniCPM-V on Type 1 documents shows that vision-language pipelines can introduce more erroneous characters than the reference contains.Even Chunk + Qwen3-VL reaches only 47.76% CER overall, remaining well below conventional OCR baselines.
  • Document heterogeneity: 0% accuracy for embedded-text approaches on Type 5 documents demonstrates that PyMuPDF and basic Docling configurations can silently fail on true scans.For digitally born Type 4 PDFs, Docling achieves 96.54% CER regardless of OCR activation, indicating that OCR is redundant when clean embedded text exists.
  • LLM post-correction: LLM post-correction degrades all evaluated base extraction systems under edit-distance metrics rather than systematically improving OCR quality.Hallucinations and normalization of archaic spellings, punctuation, or unusual phrasing can turn accurate source text into metric errors; readability-oriented applications may evaluate such changes differently.
  • Limitations: The study’s conclusions require validation across other languages, domains, and periods because the corpus is limited to Spanish historiographical texts on one topic.The labour-intensive ground-truth process limits corpus size, and edit-distance metrics may not align with downstream needs such as semantic search.

6 Conclusion

The conclusion identifies end-to-end parsing as the most reliable strategy for heterogeneous historical collections, while showing that document characteristics materially affect extraction accuracy. It also warns that digitization errors can propagate through retrieval and generation systems, producing confident but erroneous outputs.

  • Performance and reliability: 98.70% CER accuracy overall was achieved by Marker, demonstrating superior and stable performance for end-to-end document parsing.The evaluation found substantial variation across extraction approaches and document types.
  • Performance and reliability: Conventional OCR degrades on complex layouts and degraded scans, while embedded-text extraction succeeds only on digital documents and fails on true scans.These limitations contribute to performance differences across document types.
  • Implications for AI systems: OCR quality is a critical bottleneck for digitized historical collections because systematic digitization errors propagate through retrieval and generation pipelines.The conclusion links this problem to expanding conversational AI and RAG architectures.
  • Practical guidance: End-to-end parsing should be preferred for heterogeneous historical collections, with workflows adapted to scan quality, layout complexity, and embedded text layers.The conclusion recommends adapting workflows rather than applying one pipeline uniformly.
Loading 2608.24976v1…