PaperlayerView exact revision on arXiv

Source-linked AI summary

Does a Modern-Handwriting Warm-Up Help Historical Arabic OCR? A Reproducible, Compute-Matched Evaluation on Muharaf and KHATT

Sumaih Almarshad, Maram Alamri, Dona Aloraini, Fares Altuwaim, AlJawharh AlOtaibi, Reem Alyabis, Rayah Aldawsari

arXiv:2608.22316v1cs.LGcs.CV

TL;DR

Whether modern Arabic handwriting warm-up helps historical Arabic HTR is unclear because conclusions often rely on one implementation and comparison. This paper tests stability across varied implementations and a compute-matched three-seed experiment, finding unstable pilot effects but a small negative effect under controlled conditions.

  • Problem

    The reliability of intermediate-domain training for historical Arabic HTR remains uncertain because pilot implementations differ along several axes at once.

  • Method

    The study repeats one transfer ablation across four configurations, then controls warm-up data, budgets, optimization, and seeds in a compute-matched experiment.

  • Results

    Effects ranged from −17.64 to +14.52 CER points across four runs, while compute-matched KHATT warm-up was 2.42 CER points worse than a same-domain control.

  • Takeaways & Limitations

    Modern-handwriting warm-up shows no stable pilot benefit and a small negative effect under the controlled configuration, rather than a general property of the method.

  • Takeaways & Limitations

    The four implementations were not controlled replications, so the study establishes instability but cannot decompose which differing axes caused it.

Abstract

from arXiv · show

Whether an intermediate stage of modern Arabic handwriting helps or hurts historical Arabic HTR is usually decided from one implementation and one comparison, too thin a basis for a claim either way. We test stability by running the same nominal ablation four times, letting the base checkpoint, encoder-freezing strategy, epoch budget, precision, and learning-rate schedule vary as they naturally did during development, while holding the normalization, scorer, and interval estimation fixed. Each run compares intermediate training on modern handwriting (KHATT) then fine-tuning on historical manuscripts (Muharaf) against fine-tuning on Muharaf directly. Across the four runs the estimated effect swings from -17.64 to +14.52 CER points and reverses sign. The two extremes are exactly the two runs with an identifiable confound (a fivefold lower learning rate in one; a checkpoint of undisclosed provenance in the other); the two clean runs land at -0.25 and +0.94, i.e. no effect. A tight interval from one implementation says nothing about the next. We then run a compute-matched experiment with identical budgets over three seeds: KHATT warm-up is +2.42 CER points worse than a matched same-domain control (95% interval [+0.60, +4.25]); the part of that gap specific to the handwriting domain is only about 0.6 points a small negative effect under this configuration, not a universal result. We release a SaudiHeritage-OCR package with the normalizer, interval scorer, a verified KHATT decoder, experimental manifests, VLM baselines, and an edition-alignment protocol, so the result can be checked independently. The Al-Mahd inscription line is held strictly out and is not offered as a benchmark.

1 Introduction

The paper asks whether modern Arabic handwriting warm-up improves historical Arabic HTR under limited annotation and compute, and shows that implementation choices can dominate the apparent transfer effect. Across four configurations the effect reverses sign, while a compute-matched follow-up finds a small negative effect for KHATT warm-up in one configuration and the authors release infrastructure for independent checking.

  • Motivation: Historical Arabic text is growing faster than it can be read, motivating automated HTR for cultural-heritage preservation and digitization.Transformer systems reach single-digit CERs on historical Arabic manuscripts, while VLMs transcribe Arabic without task-specific training.
  • Research question: The central question is whether intermediate KHATT training helps when a model is subsequently fine-tuned on smaller historical Muharaf data.The question arises under limited annotation and compute, where modern handwriting is larger and more available than the historical target.
  • Methodological implication: A single implementation’s confidence interval does not capture implementation variability, so the two sources of uncertainty should be reported separately.The authors treat cross-implementation variability as a finding and provide a reproducible way to test transfer strategies on Saudi historical material.
  • Implementation sensitivity: −17.64 to +14.52 CER points: four separately configured KHATT→Muharaf runs reverse sign, with the two clean frozen-encoder runs at −0.25 and +0.94.The extremes correspond to identifiable confounds: a fivefold lower staged-arm learning rate and a checkpoint of undisclosed provenance.
  • Compute-matched follow-up: +2.42 CER points: across three seeds, KHATT warm-up was worse than a compute-matched same-domain control, with interval [+0.60, +4.25].The domain-specific portion against a disjoint real-target slice was about 0.6 points, framed as a small negative effect for this configuration rather than a general claim.
  • Reproducible infrastructure: The release includes the normalizer, scorer, interval and paired-difference code, verified KHATT decoder, manifests, VLM utilities, and edition-alignment utilities.The Al-Mahd material remains held out and is not presented as a powered benchmark.

2 Related Work

Arabic OCR progressed from brittle segmentation pipelines and handcrafted models to learned end-to-end representations, while historical Arabic remains especially challenging. Prior work suggests that intermediate training transfers conditionally, motivating matched and reproducible evaluation across modern handwriting and historical manuscripts.

  • Foundations: Classical Arabic OCR used sequential segmentation, feature extraction, and handcrafted recognition, but cursive joins, contextual shapes, ligatures, and diacritics make segmentation brittle.These difficulties intensify on handwritten and historical pages.
  • End-to-End OCR: 8.6% CER is HATFormer’s reported result on Muharaf and the strongest published number on that benchmark.HATFormer extends the TrOCR end-to-end Transformer template to historical Arabic manuscripts.
  • Transfer Learning: Intermediate training can help or hurt, with transfer governed by intermediate–target task and domain similarity rather than by extra data alone.Continued-pretraining work similarly reports reliable in-domain benefits but limited or negative effects from off-domain continuation.
  • Reproducibility: Reported machine-learning gains often fail to survive implementation, tuning-budget, or seed changes, while unmatched tuning can favor the more-attended condition.This paper frames its unstable Arabic HTR schedule ablation as a domain-specific instance of that reproducibility concern.
  • Datasets and Benchmarks: KHATT is the standard modern Arabic handwriting benchmark, whereas Muharaf contains 24,495 expertly transcribed historical text-line images.KITAB-Bench finds open-source VLMs trailing proprietary models zero-shot, and existing benchmarks do not hold Saudi epigraphic material.

3 Data

The study keeps Muharaf, KHATT, and Al-Mahd strictly separated by role: historical target, intermediate corpus, and out-of-domain qualitative inspection. It documents reproducible corpus preparation while withholding Al-Mahd from quantitative evaluation.

  • Corpus roles and preparation: Muharaf contains 24,495 historical Arabic manuscript line images, with 22,091 training, 1,069 validation, and 1,334 test lines after excluding empty transcriptions.The original partition is retained; 22,092 is the pre-exclusion figure because one training instance has an empty transcription.
  • Corpus roles and preparation: KHATT is used only for intermediate training and never for final evaluation.Its coded-column transcriptions were decoded using the official KHATT v1.0 lookup table, covering 100% of codes in both training and validation CSVs.
  • Corpus roles and preparation: Al-Mahd is reserved for out-of-domain qualitative inspection and is not used for training or fine-tuning.The held-out material consists of line crops from early Islamic Hijazi inscriptions documented in the 2026 Al-Mahd survey.
  • Corpus roles and preparation: N = 1 line is available from Al-Mahd, so the study makes no quantitative claim about manuscript-to-stone transfer.Post-editing is avoided because corrected readings may echo model errors.

4 Methodology · 4.1 Normalization policy · 4.2 Scoring with uncertainty, and what an interval settles

The methodology applies one shared normalization policy and scorer, then distinguishes line-, seed-, and implementation-level uncertainty. This framework shows that implementation-level variation can greatly exceed the uncertainty visible within a single run.

  • 4 Methodology: The methodology evaluates references, predictions, and edition strings with one shared normalizer and scorer across experiments.The normalizer includes Unicode NFKC, tatweel removal, diacritic stripping, whitespace collapse, and orthographic unification.
  • 4.1 Normalization policy: Every string undergoes Unicode NFKC, tatweel removal, diacritic stripping, whitespace collapse, and alef–hamza unification.Raw and normalized scores are reported side by side.
  • 4.1 Normalization policy: Orthographic unification is an explicit modeling choice that lowers CER by several points relative to diacritic stripping alone.The policy targets genuinely ambiguous orthographic variation in historical handwriting.
  • 4.2 Scoring with uncertainty, and what an interval settles: CER and WER are computed from per-line Levenshtein edits and aggregated at corpus level.This establishes the common scoring basis for all reported comparisons.
  • 4.2 Scoring with uncertainty, and what an interval settles: Scores use bootstrap 95% confidence intervals, with 1,000 resamples over lines and 2,000 for the paired comparison in Section 6.4.Paired-delta intervals on shared lines account for strong per-line correlation between systems.
  • 4.2 Scoring with uncertainty, and what an interval settles: The analysis separates line-level, seed-level, and implementation-level uncertainty because a line-level interval excluding zero cannot settle the latter two.Implementation-level variation comes from separately configured runs using the same nominal protocol.
  • 4.2 Scoring with uncertainty, and what an interval settles: 32.16 CER points is the implementation-level range, versus 1.22 for I1’s paired line-level interval and 0.94 for its three-seed spread.The implementation-level range therefore dominates the two within-implementation quantities shown in Figure 2.

4.3 Readers

The section defines “readers” as OCR models that transcribe manuscript line images and evaluates zero-shot, VLM-LoRA, and TrOCR-family readers. The TrOCR comparison includes four separately configured builds spanning single-stage and staged training, with potential optimism for two community-checkpoint systems because training-data overlap with Muharaf cannot be ruled out.

  • Readers are OCR models evaluated by their transcription of manuscript line images, not human readers.
  • The zero-shot floor uses Qwen2.5-VL-7B-Instruct in 4-bit NF4 quantization with Arabic prompts for verbatim single-line transcription.The per-sample generation cap scales to reference length; the 4-bit setup is reported as a commodity-GPU floor rather than the model ceiling.
  • VLM-LoRA adapts the same VLM with rank 16, α = 32, dropout 0.05, q/v projection LoRA, a frozen vision tower, and staged KHATT→Muharaf training.The loss is masked to the assistant span.
  • Four separately configured TrOCR vision-encoder-decoder builds run both single-stage (M1) and staged (M2) conditions.Two use the RayR1/trocr-base-arabic-handwritten community checkpoint, while two build readers from a ViT encoder and AraBERT.
  • Absolute CER for the two community-checkpoint readers may be optimistic because their model-card training data are undisclosed and overlap with Muharaf cannot be ruled out.

4.4 Preprocessing · 4.5 Compute-matched warm-up matrix (I5) · 4.6 Domain-distance measurement

The study standardizes image preprocessing, isolates warm-up data source in a compute-matched matrix, and measures domain distance using frozen encoder embeddings and text 3-gram distributions. The matrix contrasts KHATT warm-up with same-domain, no-warm-up, and in-domain-real controls under fixed corpus size and optimization budgets.

  • 4.4 Preprocessing: CLAHE, Hough deskewing, aspect-preserving resizing, and quality filtering form a shared pipeline for heterogeneous line images.Manifests are integrity-checked, and image–transcription pairs are spot-checked before use.
  • 4.5 Compute-matched warm-up matrix (I5): The compute-matched follow-up varies only the warm-up corpus domain between arms, addressing disagreement among pilot runs that changed several factors simultaneously.This design separates the data-schedule question from checkpoint provenance.
  • 4.5 Compute-matched warm-up matrix (I5): All arms use a reader trained from generic pretrained weights, with ImageNet-initialized ViT-base and AraBERT components joined through VisionEncoderDecoderModel.This prevents undocumented prior exposure to KHATT or Muharaf from checkpoint provenance.
  • 4.5 Compute-matched warm-up matrix (I5): Every warm-up corpus is subsampled to 9,497 lines, and every warm-up arm receives Nw = 1,200 steps with a fixed target budget Nt.The corpus size uses a frozen seed and matches the smallest available corpus, KHATT.
  • 4.5 Compute-matched warm-up matrix (I5): The matrix includes A+ same-domain Muharaf control, B KHATT warm-up, A0 no-warm-up reference, and E best-case in-domain-real control.A0 is explicitly not compute-matched because it skips warm-up and receives fewer optimization steps.
  • 4.6 Domain-distance measurement: Each warm-up corpus is compared with the target split on two independent domain-distance axes.The measurement is designed to quantify domain gap rather than rely on intuition.
  • 4.6 Domain-distance measurement: Image distance uses unbiased RBF-kernel MMD2 on frozen base-encoder embeddings from up to 2,000 images per corpus, with a 200-fold bootstrap 95% interval.Kernel bandwidth follows the median pairwise-distance heuristic, and the encoder is frozen before fine-tuning.
  • 4.6 Domain-distance measurement: Text distance uses Jensen–Shannon divergence between character 3-gram frequency distributions.Both distance axes are computed from the frozen, pre-fine-tuning base encoder framework described for the measurement procedure.

4.7 Statistical protocol · 4.8 Edition alignment: a protocol for corpora without line-level labels

The statistical protocol pre-registers one primary contrast, corrects secondary tests, and uses paired seed-level intervals while acknowledging that domain-distance monotonicity cannot be tested with only two levels. The edition-alignment protocol uses expert editions to derive labels, is released but not yet run at scale, and therefore has no audited precision or verified labels under its measured definition.

  • 4.7 Statistical protocol: The primary contrast is B vs. A+, comparing KHATT warm-up with a compute-matched no-domain-shift control, and it is reported uncorrected.A0 vs. A+ and E vs. A+ are secondary contrasts.
  • 4.7 Statistical protocol: Secondary contrasts use paired t-tests with joint Holm-Bonferroni correction at α = 0.05 to control false-positive inflation.The correction applies when adding secondary arms.
  • 4.7 Statistical protocol: Seed-level contrasts use a paired Studentt interval over per-seed CER differences.
  • 4.7 Statistical protocol: Monotonicity between ∆CER and measured domain distance is not tested because only two distance levels, B and E, are available.A Spearman ordering test requires at least three distance levels.
  • 4.8 Edition alignment: a protocol for corpora without line-level labels: The alignment protocol exploits published scholarly editions whose expert-verified wording lacks correspondence to specific page lines.It is designed for heritage manuscript corpora without line-level labels.
  • 4.8 Edition alignment: a protocol for corpora without line-level labels: Each line-crop prediction is matched to normalized edition text using boundary-aware partial-ratio alignment with score threshold 82 and minimum prediction length 6.The adopted label is the edition wording over the aligned span, never the reader’s output, repairing naive prefix matching.
  • 4.8 Edition alignment: a protocol for corpora without line-level labels: The protocol is specified, implemented, and released in SaudiHeritage-OCR but was not run at scale because the target Saudi manuscript corpus was not acquired.No audited precision is reported.
  • 4.8 Edition alignment: a protocol for corpora without line-level labels: The paper does not describe these labels as verified in the protocol’s measured sense, avoiding an unsupported claim.

5 Experimental Setup

The experiments use explicitly controlled training and evaluation configurations across VLM-LoRA and TrOCR lanes, with compute-matched matrix arms sharing identical settings across seeds. Reported safeguards address selection bias, fixed target budgets, and full-test evaluation without checkpoint or seed cherry-picking.

  • VLM-LoRA lane: 1,000-line training subsets, 200-line evaluation samples, and 1,000-resample bootstrap intervals define the controlled VLM-LoRA reader lane.The lane uses learning rate 10−4, 4-bit NF4, 3 epochs per stage, effective batch 8, and seeds {42, 43, 44}.
  • TrOCR lanes: TrOCR uses learning rate 5 × 10−5 by default, beam width 4, and maximum target length 128 tokens, with multi-seed evaluation only for I1 and M1/M2.I1 evaluated M1 with seeds {42, 43, 44} and M2 with {42, 43}; remaining implementations report single runs.
  • Reporting safeguards: Earlier best-of-n selection chose the lowest normalized CER per condition, so Table 9 instead reports means over completed seeds and identifies single-run results.The paper reports this deviation rather than hiding it because best-of-n selection biases point estimates.
  • Matrix controls: Identical training hyperparameters across every matrix arm and seed make uniformity the experiment’s central control.The arm-E warm-up slice is carved once from Muharaf train and kept disjoint from the shared target split; Table 5 defines the warm-up sources and Table 6 lists the shared hyperparameters.
  • Fixed target budget: Nt = 3,000 target steps was frozen for every arm and seed from a single pilot, despite validation loss still edging down at the 2,500-step cutoff.The pilot selected best step s∗ = 2,500 using validation loss 5.521 → 5.239 → 5.095 → 5.043 → 5.042.
  • Evaluation: Every arm and seed is evaluated once on the full 1,334-line Muharaf test set, without test-set early stopping or best-of-n over checkpoints or seeds.CER and WER use the single shared normalizer.

6 Results

The results show that historical Arabic OCR performance is highly sensitive to evaluation and implementation choices: zero-shot output is unusable, the warm-up effect reverses across implementations, and compute-matched testing finds only a small negative effect under one configuration. Qualitative errors are fluent substitutions, especially on ornate and numeral-dense lines, so corpus-level metrics require inspection.

  • Zero-shot baseline: WER exceeds 100% in the zero-shot condition because fluent, heavily diacritized Arabic is largely unrelated to the source.On the held-out Al-Mahd line, the floor reaches 60.71% CER / 83.33% WER, but N = 1 makes that result illustrative only.
  • Evaluation sensitivity: 2.1 normalized CER points and 10.7 raw points differ between two runs that vary only in generation control.This calibrates how much reported OCR variation can arise outside model training.
  • Warm-up ablation: 32.16 CER points span the four nominal ablation runs, with the warm-up effect reversing sign across implementations.The two extremes are the confounded runs, while the reported clean runs show no effect.
  • Warm-up ablation: +14.52 CER points is the largest warm-up penalty, but it comes from I1’s unequal target-stage learning rates of 5 × 10−5 versus 1×10−5.A paired bootstrap gives ∆CER = +14.58 points with 95% CI [+13.97, +15.19], while the same base checkpoint and corpora in I2 contradict the result.
  • Compute-matched experiment: +2.42 CER points separates KHATT warm-up from the compute-matched same-domain control, with a paired seed-level 95% interval of [+0.60, +4.25].Across three shared seeds, this is interpreted as a small negative effect under this configuration rather than a universal claim.

7 Discussion

The discussion’s clearest conclusion is methodological: single-implementation schedule findings can reverse under another reasonable implementation, so the broader training-schedule question remains open. Within the controlled I5 configuration, KHATT warm-up shows only a small negative effect, while several confounds and design limits constrain interpretation.

  • Evidence and interpretation: Single-implementation schedule findings can reverse under another reasonable implementation, so the interval may answer a narrower question than the claim it supports.The divergence is interpreted through interval-first evaluation rather than dismissed as noise.
  • Practical implications: Published single-implementation schedule results should not allocate annotation or compute budgets; the broader question remains open across architectures and training regimes.The operational recommendation applies to practitioners adapting readers to Saudi heritage material.
  • Controlled result: Within the controlled I5 configuration, KHATT warm-up has a small negative effect, while its domain-specific component is small under the appropriate control.Cross-implementation results suggest the effect may depend on the operating point and adaptation choices.
  • Confounds: Asymmetric stabilization adjustments—such as lowering learning rate, shortening the target stage, or freezing a component—can confound staged-versus-baseline comparisons while remaining invisible in results tables.These adjustments may be made for engineering reasons and applied only to the staged arm.
  • Methodological contribution: A single normalizer makes the four runs comparable, and the interval-first apparatus makes their divergence interpretable rather than merely noisy.The discussion presents a third uncertainty level and recommends independent re-im
  • Limitations: The four implementations establish instability but cannot decompose its causes because they vary along several axes, while undisclosed checkpoint provenance limits the use of I1 and I2 absolute CER.Contamination threatens absolute levels more than within-run ∆CER, motivating reliance on the compute-matched matrix for directional claims.

8 Conclusion and Future Work

Intermediate-domain training effects varied sharply across implementations, while the two runs without the identified confound showed essentially no effect. The study therefore emphasizes compute matching, multi-seed evaluation, consistent preprocessing, and explicit reporting of implementation-level variation, alongside expanded Saudi epigraphic evaluation and ground-truth efforts.

  • Conclusion: −17.64 to +14.52 CER points: four configured runs produced effects that changed direction under a common normalizer and interval-based scorer.The study examined the reliability of intermediate-domain training for historical Arabic HTR through repeated implementation and controlled experimentation.
  • Conclusion: The two runs without the asymmetric target-stage learning-rate confound showed essentially no effect.The asymmetric learning-rate cut occurred in the run with the largest observed penalty and is the most concrete candidate explanation for divergence.
  • Conclusion: 2.42 CER points: across three shared seeds, KHATT warm-up was worse than the compute-matched same-domain control.The compute-matched experiment tested whether instability persisted after major training variables were controlled.
  • Conclusion: The framework emphasizes compute matching, multi-seed evaluation, consistent preprocessing, and explicit reporting of implementation-level variation.The conclusion states that single-implementation confidence intervals do not capture implementation-level variation.
  • Future Work: Future work will expand the Al-Mahd evaluation set, establish expert-from-scratch gold transcriptions, audit edition alignment on larger Saudi manuscript collections, and investigate restoration of damaged or incomplete ins.These efforts aim to measure Saudi epigraphic performance without model-assisted post-editing and build reliable line-level ground truth.

Code and Data Availability

The SaudiHeritage-OCR package is publicly available upon publication with the paper’s normalization, scoring, interval, and KHATT-decoding utilities. Third-party corpora are retrieved from their original sources rather than redistributed, and the held-out Al-Mahd line is not released as a benchmark.

  • Package contents: SaudiHeritage-OCR is publicly available upon publication and includes the Section 4.1 normalizer, CER/WER scorer, confidence-interval utilities, and verified KHATT v1.0 decoder.The decoder achieves 100% code coverage on the training and validation CSVs of the commonly mirrored KHATT release.
  • Data access: Muharaf and KHATT are not redistributed; the package retrieves them from their original sources and reproduces the released-manifest splits.
  • Data access: The Al-Mahd inscription material is not released as a benchmark because it is a single held-out line with a post-edited reference reading and edition-related ownership constraints.The passage also notes that distributing one line as an evaluation set would invite the kind of underpo…
Loading 2608.22316v1…