Source-linked AI summary
RefLAM: A Reference-Grounded Line Annotation Pipeline for Historical Arabic Manuscripts
Mohamed Guechaoui, Mohamed Diaa Zellagui, Souleyman Chaib, Sahraoui Dhelim
TL;DR
Line-level Arabic manuscript HTR data is scarce because manual annotation does not scale, while prior automatic alignment lacks the needed layout coverage and correctness guarantee. RefLAM combines segmentation, structured MLLM OCR, normalization, and reference-grounded fuzzy alignment under continuous human review. It yields AraMS-28k at manuscript scale and supports downstream HTR training, with a 75× annotation speedup and validated confidence-100 alignment rule.
Problem
Line-level labelled corpora for historical Arabic manuscripts remain scarce, while prior annotation approaches rely on non-scalable manual work or lack multi-script, two-zone layouts and provable correctness guarantees.
Method
RefLAM uses page segmentation, structured MLLM OCR, diacritic-agnostic normalization, and fuzzy alignment to ground OCR lines in contiguous reference spans under human review.
Results
75× throughput over manual annotation produced AraMS-28k with 14 books, 3,043 pages, 27,971 main-text lines, and 629 margin lines; the confidence-100 rule had no corpus counterexample.
Takeaways & Limitations
RefLAM provides validated, auditable line-level ground truth whose released corpus supports practical downstream HTR training.
Takeaways & Limitations
RefLAM requires a clean transcription beforehand, approximately 70% of margin lines could not be confidently anchored, and its aligner is greedy rather than globally optimal.
Abstract
from arXiv · showhide
Existing approaches to building line-level Arabic handwritten-text-recognition (HTR) training data either rely on fully manual annotation, which does not scale, or on automatic OCR-to-reference alignment methods not yet extended to multi-script, two-zone (main-plus-margin) manuscript layouts with a provable correctness guarantee. We present RefLAM (Reference-grounded Line Annotation for Manuscripts), a pipeline converting manuscript page images and clean transcriptions into validated, line-level ground truth without sacrificing human oversight. RefLAM couples a deep-learning page-segmentation model with a multimodal large language model (MLLM) for structured OCR and a diacritic-agnostic fuzzy alignment engine that grounds each OCR line in a contiguous span of the reference text, with a character-level confidence score in $[0,100]$. A perfect score is provably equivalent to character-for-character identity of the normalised strings (the Confidence-100 rule), verified with no counterexample across the released corpus. A reviewer can thus trust a perfect score, confirming most lines at a glance rather than retyping them, so annotation becomes triaged, with attention concentrated on uncertain alignments. Across 7 fully page-validated books we measured a 75$\times$ throughput gain over manual annotation (3,000 vs. 40 lines/hr); applying the same guarantee to 7 further books, we retained 16,533 confidence-100 main-text lines within one week, excluding sub-100 lines rather than manually correcting them. Using RefLAM, we release AraMS-28k: 14 historical Arabic manuscript books, 3,043 pages, and 27,971 main-text and 629 margin-line annotations with bounding boxes, layout labels, and insertion anchors for 191 margin entries (30.4%). We also finetune Muharaf-pretrained baselines (including HATFormer) on AraMS-28k and report CER results confirming its practical utility for downstream HTR training.
1 Introduction
Historical Arabic manuscripts contain a large machine-unreadable corpus, while scarce line-level labelled data limits HTR development. RefLAM addresses this annotation bottleneck with a scalable, human-reviewed, reference-grounded pipeline and a provable confidence guarantee.
- The Arab-Islamic manuscript corpus spans more than a millennium, but much of it remains unrendered as machine-readable text.
- Arabic manuscript HTR models are datahungry, yet line-level labelled corpora remain scarce.
- 75× speedup over manual annotation is achieved by RefLAM while preserving human review at every confidence level.
- The confidence-100 rule makes the speedup safe by providing a provable alignment guarantee.
2 Related Work
Existing Arabic manuscript datasets and OCR-alignment systems leave gaps in scale, layout annotation, human review, and correctness guarantees. RefLAM combines automated reference alignment with multi-script, two-zone layout handling and formally characterized confidence.
- Arabic Manuscript Datasets: Public line-level datasets of genuine historical handwritten Arabic manuscripts remain scarce because transcription and verification are labor-intensive.Existing resources vary in size and annotation scope, including missing main/margin distinctions or manually produced labels.
- Arabic Manuscript Datasets: No prior work combines verified automated line-level ground truth, main/margin layout tags, and a provable confidence criterion.
- Reference-Grounded OCR Alignment: ACDC aligns noisy Kraken output with clean editions and reports a 19.6% absolute character-accuracy improvement without manual transcription.Unlike RefLAM, ACDC uses heuristic quality criteria and provides no individual human review or layout structure annotation.
- Reference-Grounded OCR Alignment: RefLAM extends fuzzy reference alignment from paragraphs to lines and adds multi-script, two-zone Arabic layout tags plus a provably exact confidence threshold.
- OCR and Segmentation: MLLM OCR failure modes are routed to detailed review because noisy outputs receive lower similarity against the reference.
3 The RefLAM Pipeline
RefLAM converts page images and clean transcriptions into auditable line-level annotations through segmentation, structured MLLM OCR, normalization, page anchoring, and fuzzy alignment. The design treats OCR as noisy evidence while using the reference text to localize and validate each line under human review.
- Pipeline Design: Human oversight, throughput improvement, and auditability are non-negotiable constraints governing RefLAM’s five-stage pipeline.Each line remains traceable to its reference span, and reproducibility depends on fixed configuration and cached MLLM responses.
- Segmentation: The Kraken segmentation model supplies polygonal line contours that RefLAM converts to axis-aligned bounding boxes while retaining polygons for mask generation.A human reviewer verifies the segmentation output before subsequent processing.
- MLLM Structured OCR: Gemini receives whole page images, emits structured lines with margin markers, and is treated as a noisy label rather than ground truth.Hallucinated content fails alignment and is routed to detailed review.
- Normalization: Canonical normalization strips diacritics and other specified variation so OCR and fully vocalized references can be compared consistently.The dataset preserves both raw OCR and ground-truth fields for future diacritisation-restoration research.
- Page Anchoring: Page anchoring locates each page in the book-level reference by comparing its first five normalized main OCR lines with reference windows.The partial-ratio score is used because the page signature may occur as a substring of a larger reference region.
- Line-Level Fuzzy Alignment: After page anchoring, each OCR line is matched to a contiguous reference span using an LCS-based character similarity and a greedy monotonic pointer.The pointer advances past each matched span, preserving reading order.
- Validation and Sensitivity: Removing page anchoring increases alignment errors by a factor of 3–5, while monotonic pointer advance improves line-order coherence.Parameters were tuned on initial books and then frozen for the remainder of construction.
3.7 The Confidence-100 Rule
The Confidence-100 rule proves that a perfect alignment score means character-for-character identity after normalisation, while corpus auditing found no incorrect perfect-score lines.
- Theorem: C = 100 implies character-for-character identity between the normalised OCR line and matched reference span.The proof follows because the maximal LCS-based similarity can occur only when both normalised strings have equal length and are identical.
- Scope: The guarantee remains valid despite diacritisation and letter-variant differences absorbed by the normalisation operator.It certifies normalised-string identity, not equality of the raw OCR and reference strings.
- Margin lines: For margin lines, a perfect partial score identifies the correct text within a contiguous reference window, although repeated short formulae may yield multiple offsets.The text is certified even when the exact location is not unique.
- Empirical verification: The rule was manually audited across all 14 books, with zero errors found among confidence-100 lines.The audit displayed the page image, OCR text, and matched reference span side by side.
- Human oversight: Confidence-100 lines still undergo human inspection; the rule accelerates review rather than automatically accepting annotations.Reviewers use a rapid comparison path, while lower-confidence lines receive detailed review.
3.8 Human Review
RefLAM keeps human review in the loop for every line, using rapid visual confirmation for perfect alignments and detailed adjudication for uncertain ones.
- Review protocol: Every line, regardless of confidence, is reviewed by a human before entering the released dataset.The browser-based review application presents the relevant page and alignment information for inspection.
- Rapid review: Confidence-100 lines are checked through a rapid side-by-side comparison of the page image, segmented box, and reference span.The confirmation typically takes approximately 1–2 seconds.
- Detailed review: Sub-100 lines receive detailed review, and reviewers may accept, correct, or reject the proposed match.
4 The Resulting Corpus: AraMS-28k
RefLAM produces AraMS-28k, a multi-script corpus of historical Arabic manuscript lines with layout, geometry, reference alignments, confidence scores, and selected margin insertion anchors.
- Corpus scale: 14 historical Arabic books comprise AraMS-28k, covering 3,043 pages, 27,971 main-text lines, and 629 margin lines.The books include thirteen hand-copied manuscripts across Naskh, Ruq‘ah, and Maghrebi scripts, plus one lithographed edition.
- Annotations: Each line includes a main/margin layout label, geometry, aligned reference transcription with confidence, and a coarse orientation estimate.
- Margin structure: 191 of 629 margin lines, approximately 30%, receive human-assigned insertion anchors when their attachment point is unambiguous.Other margin lines retain null anchors when they are non-referential or lack a clear attachment point.
- Validation: Construction used 548 fully page-validated pages and 2,495 line-validated pages, releasing only confidence-100 main-text lines from the latter phase.
5 Analysis
Confidence scores are strongly bimodal and vary substantially by book, while low-confidence cases arise from scan degradation, unusual script forms, and margin–main overlap.
- Confidence distribution: Most page-validated lines fall either in the 90–100 confidence bin or below 50, with comparatively few in the 60–89 range.The distribution suggests disruptions often cause sharp similarity drops rather than gradual degradation.
- Review implication: Intermediate-confidence lines, roughly 60–99, require detailed review rather than rapid confirmation.
- Release criterion: For line-validated books, C=100 is 100% by construction because only perfect-score main lines were retained for release.
- Error analysis: Scan degradation, unusual script forms, and margin–main overlap are representative causes of low-confidence alignments.These cases remain in the corpus with sub-100 scores and review metadata for filtering or further study.
6 Validation and Throughput
RefLAM’s two-phase validation retains provably exact high-confidence lines while concentrating manual effort on uncertain cases, yielding substantially higher throughput and usable downstream HTR training data.
- Validation: 548 pages and 11,438 lines in seven page-validated books were inspected completely by two independent reviewers, with C=100 lines empirically error-free.
- Validation: 16,533 main-text lines from seven line-validated books were retained only when they achieved Confidence-100; sub-100 lines were excluded rather than corrected.
- Throughput: 40 lines per person-hour was the manual baseline, versus about 3,000 confidence-100 lines per person-hour when reviewers only confirmed segmentation.
- Throughput: 75× was the measured internal speedup over manual annotation under the same conditions on the same page-validated books.
- Baseline HTR Results: Under identical finetuning conditions, Kraken achieved lower overall weighted CER than HATFormer: 23.31% versus 26.74%.
- Baseline HTR Results: The script-specific CER ordering tracks proximity to the Muharaf pretraining distribution rather than the amount of AraMS-28k supervision.
7 Conclusion
The conclusion presents RefLAM as a reference-grounded route to validated manuscript line annotations, centered on a provable confidence guarantee and demonstrated corpus-scale output. It also states key scope boundaries and future extensions.
- Conclusion: RefLAM treats MLLM OCR as a noisy hypothesis and clean reference text as the anchor for validated, line-level ground truth.
- Conclusion: The confidence-100 rule forces character-for-character identity of normalized strings and was verified across the full corpus without a counterexample.
- Conclusion: AraMS-28k contains 28,600 line annotations from 14 books and 3,043 pages, produced in one calendar month by two annotators.
- Limitations: RefLAM requires a clean transcription before annotation, and approximately 70% of margin lines could not be confidently anchored.
- Limitations: The line-level aligner is greedy rather than globally optimal, while the MLLM implementation relies on a proprietary model.
- Future Work: Future work includes open-source MLLM replacement, Persian and Ottoman Turkish extensions, additional HTR benchmarks, continuous angle regression, and automated margin linking.
Data and Code Availability
The project releases AraMS-28k, HTR-ready derivatives, and the RefLAM pipeline and review tool under a noncommercial share-alike license, with integrity checks for Zenodo archives.
- Resources: AraMS-28k images and JSONL are released at approximately 2.05 GB through Zenodo and a GitHub mirror.
- Resources: AraMS-28k-HTR provides approximately 2.42 GB of pre-built PNG line crops and .gt.txt files derived automatically from AraMS-28k.
- Resources: The pipeline and browser-based review tool are released through the Reflam-pipeline GitHub repository.
- License: All resources are released under CC BY-NC-SA 4.0.
- Integrity: SHA-256 checksums are supplied in each Zenodo archive for integrity verification.
A RefLAM MLLM OCR Prompt (v2.3)
The v2.3 prompt instructs the MLLM to transcribe handwritten Arabic pages exactly while distinguishing main text from margin text and preserving physical line order. It adds strict output constraints to reduce hallucination.
- Layout: The prompt distinguishes two page zones: regular main-text lines and margin text, which may be sideways or squeezed in.
- Output Rules: Main text must be output as one line per physical line from top to bottom, without numbering.
- Output Rules: The model must output exact Arabic text without summarization, commentary, English writing, introductions, or extra text.
- Output Rules: Margin lines must receive a margin tag and follow all main lines in the output.
- Version History: Version 2.3 added strict output constraints intended to minimize hallucination.