PaperlayerView exact revision on arXiv

Source-linked AI summary

When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study

Moshiur Farazi, Firoj Alam, Abderrahmane Maaradji, Zakaria Maamar, Hamdy Mubarak, Wajdi Zaghouani

arXiv:2608.22366v1cs.CV

TL;DR

Reliable recognition of Arabic and Islamic manuscripts remains underexplored, especially for deciding when VLMs should replace or complement OCR. The paper evaluates OCR, general and Arabic-specialized VLMs, and OCR-conditioned correction across eight datasets, finding that no single approach dominates. OCR conditioning helps when the prior is recoverable but can hurt when it is script-mismatched or systematically misleading, motivating adaptive routing.

  • Problem

    Zero-shot VLM behavior on Islamic manuscript images remains underexplored, particularly in comparison with traditional OCR and OCR-conditioned correction across document settings.

  • Method

    The paper conducts a cross-domain zero-shot evaluation of traditional OCR, general-purpose and Arabic-specialized VLMs, and OCR-conditioned correction across eight Arabic datasets.

  • Results

    No single system is uniformly best: OCR-conditioned correction improves recognition when the OCR prior remains readable and recoverable, but degrades performance when it is systematically misleading.

  • Takeaways & Limitations

    An adaptive OCR-VLM workflow should route pages using script, OCR-prior recoverability, length diagnostics, and failure-mode indicators.

  • Takeaways & Limitations

    The study is a zero-shot deployment analysis using reproducible local inference, and its OCR-prior recoverability analysis is mainly conducted at the dataset level.

Abstract

from arXiv · show

Vision-language models (VLMs) are increasingly being used for document understanding, yet their role in Arabic and Islamic manuscript recognition remains underexplored. To address such a gap in this paper, we evaluate traditional OCR, general-purpose VLMs, Arabic-specialized VLMs, and OCR-conditioned VLM correction across eight Arabic text datasets spanning historical manuscripts, aged printed books, clean print, multi-domain documents, and handwriting. The results show that no single approach dominates across setups. On line-level historical manuscripts, VLMs are close to Tesseract; on page-level manuscript images, they perform better; and in several settings, an OCR-conditioned corrector improves over both standalone OCR and standalone VLMs. The central finding is an OCR-prior recoverability principle: OCR conditioning helps when the OCR output remains visually and textually recoverable, providing anchors that the VLM can refine against the image. It improves recognition on aged print, clean print, mixed-domain Arabic, and some Naskh manuscripts, but degrades performance when the prior is script-mismatched or systematically misleading, as in Maghribi manuscripts and realistic student handwriting. Additional diagnostics show that Arabic VLM-OCR is sensitive to diacritics, preprocessing, generation budget, and repetition loops. These findings support an adaptive OCR-VLM workflow that routes pages according to script, OCR-prior recoverability, length diagnostics, and failure-mode indicators.

1 Introduction

The paper examines when VLMs should complement or replace OCR for Arabic and Islamic manuscript recognition. Across eight datasets, it finds that OCR-conditioned correction is useful when the OCR prior is recoverable, while standalone VLMs are safer when that prior is misleading.

  • Motivation: Arabic manuscript recognition remains difficult because cursive writing, diacritics, degradation, script variation, and complex layouts affect visual and linguistic interpretation.These conditions include marginalia, rubrication, colophons, faded ink, and traditions such as Naskh and Maghribi.
  • Study scope: VLMs are evaluated against traditional OCR, Arabic-specialized models, and OCR-conditioned correction across eight distinct Arabic text datasets.The datasets span historical manuscripts, aged print, clean print, multi-domain documents, and handwriting.
  • Research questions: The study asks how zero-shot VLMs compare with OCR and supervised HTR, when OCR correction helps, how preprocessing affects recognition, which failure modes matter, and how prompt sensitivity affects metadata extraction.The five research questions cover recognition, correction, preprocessing, failure modes, and structured extraction.
  • Main findings: VLMs are competitive with traditional OCR in several manuscript settings but are not yet replacements for supervised HTR.On line-level Muharaf, Qwen2.5-VL-7B is close to Tesseract after diacritics normalisation; on page-level Madinah, VLMs outperform Tesseract.
  • OCR-conditioned correction: On Muharaf, Tesseract→Qwen reduces CER from 69.2% to 60.6%, while correction also improves SARD, KITAB-Bench, and QNL Books.The corrector provides little benefit or hurts on RASAM 2, KHATT, and Students Essays when the OCR prior is unreliable or script-mismatched.
  • Deployment implications: The findings support routing workflows using script type, OCR-prior recoverability, CER, length ratio, repetition, and qualitative inspection.Standalone VLMs are safer when OCR is distorted, script-mismatched, or dominated by systematic errors.

2 Background and Related Work

Arabic manuscript recognition combines difficult script, page, and image conditions with limited training data. Related VLM and OCR research shows progress, but zero-shot cross-dataset behavior and OCR-conditioned correction remain insufficiently characterized.

  • Arabic script: Arabic manuscript pages contain cursive, context-sensitive writing whose positional forms make character segmentation ambiguous and error-prone.Letter shapes vary by isolated, initial, medial, and final position.
  • Arabic script: Dots and diacritics carry lexical and semantic information but are difficult to recognize when manuscripts are faded, noisy, or degraded.In Qur’anic and scholarly manuscripts, diacritics may be part of the text rather than optional decoration.
  • Manuscript conditions: Historical scripts, multi-zone layouts, specialized content, and physical damage limit transfer from clean modern Arabic to manuscript pages.Relevant complications include Naskh, Maghribi, marginal glosses, colophons, bleedthrough, stains, tears, and binding damage.
  • Existing recognition systems: Traditional OCR uses preprocessing, layout analysis, recognition, and post-processing, whereas supervised HTR can be much stronger when labelled training data is available.Tesseract remains a practical Arabic OCR reference point, while systems such as HATFormer and RASAM 2 demonstrate the value of domain-specific training.
  • Data constraints: Limited expert transcriptions make collection-specific supervised HTR expensive and difficult to scale across varying scripts, layouts, and degradation.This constraint motivates zero-shot and weakly supervised alternatives before large annotation efforts are possible.
  • VLM research gap: OCR-free and end-to-end VLMs have broadened document understanding, but Arabic-specific systems still largely target modern print, controlled handwriting, synthetic data, or general documents.Historical Arabic zero-shot behavior remains underexplored, especially across traditional OCR, VLM recognition, and OCR-conditioned correction.
  • OCR correction: Prior OCR-correction studies show that correction can degrade historical OCR, particularly when the textual anchor is weaker than the reference available to the corrector.This motivates testing correction across scripts and document settings rather than assuming that post-OCR correction is uniformly beneficial.
  • Positioning: The paper positions itself as jointly evaluating zero-shot VLM recognition, OCR-conditioned correction, diacritics sensitivity, and cross-dataset OCR-prior recoverability for historical Arabic manuscripts.The stated gap is the absence of prior work combining these evaluation dimensions.

3 Datasets

The paper evaluates eight Arabic text datasets as distinct deployment settings spanning manuscripts, handwriting, printed Arabic, and multi-domain documents. Two unpublished in-house collections extend coverage to aged print and realistic student handwriting.

  • Dataset coverage: Eight datasets cover historical manuscripts, modern handwriting, multi-domain Arabic OCR, synthetic print, aged books, and student handwriting.The evaluation treats datasets separately because each represents a different deployment condition.
  • Manuscripts: Muharaf and Madinah test historical manuscript recognition, while RASAM 2 tests transfer to Maghribi-script manuscripts.Muharaf uses 1,334 line images, Madinah uses 40 page images, and RASAM 2 uses 159 page images with line-level ground truth.
  • Handwriting: KHATT and Students Essays assess modern handwriting under controlled and realistic conditions, respectively.Students Essays contains 200 essays with variation in letter shape, spacing, tilt, erasures, and corrections.
  • Printed and mixed-domain text: SARD represents clean printed Arabic, KITAB-Bench provides 3,760 samples across 13 subsets, and QNL Books represents aged printed Arabic.QNL Books contains 199 evaluable pages from 100 Arabic books, with yellowing, ink degradation, and scanning artefacts.
  • Scope boundary: Sample-level metadata exists for Students Essays but is not analysed in this paper.
  • In-house collections: The two in-house collections broaden evaluation to aged print and realistic student handwriting and were not publicly released during evaluation.Students Essays uses verbatim references as primary ground truth, while corrected references are supplementary.

4 Experimental Design

The experiments compare traditional OCR, standalone VLMs, and OCR-conditioned correction in a zero-shot deployment study across varied Arabic document settings. Evaluation combines recognition metrics with preprocessing, error-taxonomy, and metadata-pilot analyses.

  • Research questions: The evaluation maps five research questions to standalone recognition, OCR-conditioned correction, preprocessing, failure modes, and metadata extraction.The metadata pilot studies prompt sensitivity rather than full extraction accuracy.
  • Evaluation setting: The comparison is zero-shot rather than a supervised HTR benchmark, targeting systems deployable without collection-specific training.Supervised Arabic HTR may perform better when labelled target-collection data is available.
  • OCR-conditioned pipeline: The OCR-corrector first runs Tesseract, then gives its noisy output and the source image to a VLM for faithful correction.The image lets the model check the textual prior, while the prior supplies textual context that may reduce hallucination.
  • Preprocessing: Preprocessing experiments on Muharaf compare eight input conditions, including raw, binarised, contrast-enhanced, denoised, and sharpened images.
  • Recognition metrics: CER and WER are edit distances normalised by reference length, reported with and without Arabic diacritics.Corpus-level scores are length-weighted, and main CER comparisons use bootstrap 95% confidence intervals with 10,000 iterations.
  • Diagnostics and limitations: The error taxonomy describes substitutions, diacritics, omissions, and hallucinations, but its length-ratio heuristics understate moderate mismatches.The metadata pilot reports per-field non-null rates and schema compliance without expert field-level ground truth.

5 Results

Results vary by input granularity and document setting: VLMs are close to Tesseract on line-level Muharaf but outperform it on page-level Madinah. The study also reports substantial zero-shot errors and cautions that the page comparison is confounded by dataset differences.

  • Line-level recognition: 69.2% CER for Tesseract, 70.4% for Qwen, and 70.8% for Fanar-2 were reported on diacritics-stripped Muharaf lines.The Tesseract–Qwen difference was −1.3 with a 95% CI of [−8.8, +5.0], which was not statistically significant.
  • Line-level recognition: Zero-shot VLMs were close to Tesseract on Muharaf, although absolute error rates remained high.QARI-OCR obtained 167.2% normalised CER, indicating severe domain mismatch and hallucinated-text errors.
  • Page-level recognition: 39.6% CER for Fanar-2 and 40.7% for Qwen beat Tesseract’s 51.0% CER on Madinah pages.The comparison is cautious because Madinah contains only 40 pages from eight books and differs from Muharaf in content, image quality, and annotation unit.
  • Page-level recognition: The study interprets the Madinah result as evidence that VLMs may benefit from page context, including surrounding text, layout cues, and visual continuity.The line-versus-page comparison does not isolate page context because the datasets also differ in other respects.

5.2 OCR-Conditioned Correction and Input Ablations

OCR-conditioned correction helps when noisy OCR remains recoverable, but misleading or script-mismatched priors can hurt. Gains vary across datasets, models, and samples.

  • OCR-conditioned correction: The corrector output is closer to Tesseract than standalone Qwen in 71.4% of 1,198 Muharaf lines with non-empty OCR.Mean normalized distance is 133.8% for the corrector versus 178.1% for standalone Qwen.
  • Input ablations: Both image and OCR prior are necessary: shuffling the prior raises CER from 60.6% to 81.6%, while withholding the image raises it to 73.9%.The prior supplies a textual cue, while the image grounds correction.
  • Cross-dataset recoverability: Across datasets, correction improves SARD, KITAB-Bench, Muharaf, and QNL Books but gives limited gains or degrades recognition on RASAM 2, KHATT, and Students Essays.Script compatibility, error type, and corrector-model choice matter beyond the Tesseract–VLM CER gap.
  • Cross-dataset recoverability: KITAB-Bench correction reaches 32.7% CER, a 15.5% relative improvement over Qwen and an 18.7% improvement over Tesseract.Excluding khattparagraph, CER is 17.8%, a 24.3% relative improvement over the better standalone method.
  • Cross-dataset recoverability: QNL Books achieves 15.5% CER with Tesseract→Fanar-2, a 19.3% relative improvement over the best standalone method.This is the strongest single-dataset gain, while the hypothesis about Arabic-specialized tuning is based on one dataset.

5.4 Diacritics Sensitivity

Retaining diacritics substantially penalizes VLM recognition, whereas Tesseract’s CER changes little largely because it omits tashkeel. Diacritics-stripped CER is therefore the main cross-system comparison, with raw CER still relevant when fidelity matters.

  • Diacritics sensitivity: Qwen’s CER increases by 16.3% and Fanar-2’s by 16.9% when diacritics are retained.The average VLM diacritics penalty is 16.6%, compared with a 0.4% CER change for Tesseract.
  • Diacritics sensitivity: Tesseract’s low diacritics penalty reflects omission of tashkeel rather than faithful preservation.The same cross-dataset pattern is largest on KHATT and Muharaf and smaller on SARD.
  • Reporting choice: Raw CER should still be reported when tashkeel fidelity matters.Diacritics-stripped CER is used as the main cross-system comparison.

5.5 Impact of Preprocessing

On Muharaf line images, sharpening is the only tested preprocessing operation that improves Qwen recognition; other operations degrade performance by damaging fine Arabic-script cues.

  • Preprocessing effect: Sharpening reduces Qwen CER from 70.4% to 67.2% on Muharaf line images.The sharpened Qwen result becomes better than standalone Tesseract, while sharpening Tesseract changes CER only to 69.5%.

5.6 Decoding-Length and Repetition Analysis

Arabic VLM-OCR is sensitive to generation limits and decoding behavior. SARD exposes truncation from short budgets, while Students Essays exposes repetition loops that can dominate corpus CER.

  • Decoding failures: Longer output limits can also enable repetition loops, so page-level VLM-OCR requires both sufficient length and repetition control.These are distinct decoding risks across SARD and Students Essays.
  • Generation length: A 512-token limit produces a truncation artefact on SARD, with Qwen at 36.3% CER and prediction-to-reference ratios falling to 0.48 on long-reference fonts.The affected fonts are Arial and Calibri.
  • Generation length: Increasing max_new_tokens to 2,048 lowers Qwen CER from 36.3% to 3.8% and corrector CER from 36.7% to 2.6%.The corrector slightly outperforms Tesseract after the longer generation budget.
  • Decoding failures: Students Essays reaches 199.0% corpus CER for Qwen because 28% of outputs are excessively long repetitive predictions.Constrained decoding corrects 51 of 56 Qwen failures and lowers corpus CER to 66.6%.

5.7 Zero-Shot Metadata Extraction

The metadata pilot tests whether prompt design changes zero-shot VLM extraction behavior on 16 Madinah manuscript pages. A permissive prompt systematically fills fields, while calibrated prompting suppresses most unsupported content extraction.

  • Evaluation setup: The pilot evaluates metadata extraction on 16 Madinah pages selected from the beginnings and ends of eight books.The selection targets locations where title pages and colophons often occur.
  • Prompt behavior: A naive prompt produces non-null responses for every metadata field on every page, often copying transcribed body text instead of metadata.This is identified as compliance-bias failure mode.
  • Prompt behavior: Under the naive prompt, the non-null rate is 100% for every field.Table 18 defines non-null rate as how often the model attempts an extraction.
  • Prompt behavior: A calibrated prompt returns language and script type for all pages while reducing content-field non-null rates to 0%–18.8%.The remaining outputs may be correct or residual compliance bias, which cannot be separated without expert annotation.
  • Interpretation: The pilot supports a prompt-sensitivity finding rather than a metadata extraction accuracy estimate.The absence of expert field-level ground truth prevents distinguishing correct residual outputs from compliance bias.

5.8 Failure Taxonomy

The Muharaf failure taxonomy contrasts conventional OCR errors with VLM-specific risks. Tesseract is dominated by substitutions and omissions, whereas VLM errors shift toward diacritics and hallucination.

  • Error profiles: Tesseract errors on Muharaf are dominated by substitution and omission.The taxonomy is heuristic rather than a complete decomposition of CER.
  • Error profiles: Substitution and omission account for 97.2% of Tesseract’s attributed errors.Table 19 normalizes the four heuristic error categories to sum to 100%.
  • Error profiles: VLM errors shift toward diacritics and hallucination rather than following Tesseract’s dominant substitution-and-omission profile.Figure 8 reports the distributional contrast on Muharaf.

6 Discussion

The discussion frames Arabic manuscript recognition as adaptive system selection rather than choosing one universally best model. OCR-conditioned correction helps when the OCR prior is recoverable, while standalone VLMs are safer for misleading or script-mismatched priors.

  • Adaptive selection: No single recognition system is uniformly best across the evaluated document settings.The paper compares traditional OCR, standalone VLMs, and OCR-conditioned correction across distinct deployment conditions.
  • Input granularity: Full-page or region-level inputs can exploit context, while line crops remain useful for controlled evaluation.Page context may include neighboring text, layout cues, and visual continuity.
  • Deployment boundary: Supervised HTR remains stronger when labelled target-collection data exists, whereas zero-shot VLM-OCR addresses settings where such transcription is costly or unavailable.This defines the deployment boundary for the zero-shot approach.
  • OCR-prior recoverability: OCR-conditioned correction helps when the OCR prior preserves readable structure that the VLM can refine against the image.It improves on Muharaf, KITAB-Bench, SARD after increasing output length, and QNL Books, but fails on RASAM 2, KHATT, and Students Essays.
  • OCR-prior recoverability: Shuffling the OCR prior or removing the image degrades correction, showing that the method uses both textual and visual evidence.The prior supplies partial text cues, while the image validates and corrects them.
  • Preprocessing: Sharpening improves Qwen2.5-VL-7B on Muharaf, whereas binarisation, denoising, contrast enhancement, and combined pipelines can degrade recognition.Aggressive enhancement may damage dots, diacritics, and thin connecting strokes.
  • Evaluation diagnostics: VLM evaluation should track diacritics-stripped and raw CER alongside length ratios, hallucination, truncation, and repetition indicators.Diacritics, output limits, and decoding loops can materially affect observed errors beyond aggregate CER.
  • Limitations and future work: The study is a zero-shot deployment analysis, not an exhaustive benchmark, and excludes frontier API models such as GPT-4o, Claude, and Gemini.Its recoverability analysis is mainly dataset-level; future work should develop page- or region-level routers.

7 Conclusion

Across eight Arabic text datasets and more than 7,700 samples, the paper finds that OCR, standalone VLMs, and OCR-conditioned correction each suit different settings. The central principle is to use correction when OCR remains recoverable and standalone VLMs when the prior is misleading.

  • Conclusion: Across eight Arabic text datasets and more than 7,700 samples, no single system is uniformly best.Traditional OCR, standalone VLMs, and OCR-conditioned correction perform best in different settings.
  • Conclusion: OCR-conditioned correction improves recognition when OCR output remains readable enough to guide the VLM, but degrades performance when the prior is systematically misleading.Controlled ablations show that both the OCR prior and source image are needed.
  • Conclusion: The study identifies deployment-critical risks including diacritics sensitivity, selective preprocessing effects, output-length truncation, repetition loops, and metadata hallucination.These diagnostics motivate routing decisions beyond a single recognition metric.
  • Conclusion: The findings provide a practical decision framework for collections where supervised HTR training data is costly or unavailable.The framework routes recognition according to document-setting and OCR-prior behavior.
Loading 2608.22366v1…