PaperlayerView exact revision on arXiv

Source-linked AI summary

Combining Synthetic and Real Data for Low-Resource Historical OCR: A Manchu Case Study

Yan Hon Michael Chung, Hanlin Wang

arXiv:2609.11495v1cs.LG

TL;DR

Low-resource historical OCR must combine synthetic data with limited real supervision while addressing the synthetic-to-real gap. Using Manchu as a case study, the paper evaluates recognizers and training regimes, finding that real images raise accuracy to 95.09–96.28%, while complementary recognizers and dictionary-guided voting reach 98.27%.

  • Problem

    Historical OCR lacks sufficient annotated data, and synthetic images do not fully reproduce real documents’ typography, variation, degradation, and capture conditions.

  • Method

    The study evaluates four recognizers and four synthetic-real training regimes on Manchu word images using a fixed archival benchmark and common checkpoint-selection protocol.

  • Results

    Real-image configurations reach 95.09–96.28% WA versus at most 87.92% for synthetic-only training, while recognizer combination raises ARCH-test WA to 98.27%.

  • Takeaways & Limitations

    Real historical supervision is central, synthetic supplementation varies by recognizer, and complementary errors can improve OCR through voting with dictionary-based adjudication.

  • Takeaways & Limitations

    The conclusions rely on a fixed 753-image benchmark with lexical overlap and single training runs, limiting discrimination among strong configurations and generalization of run-sensitive claims.

Abstract

from arXiv · show

Manchu, now critically endangered, was one of the principal languages of the Qing empire (1636-1912), and its extensive archival record is increasingly digitized but remains difficult to search and analyze at scale. Previous work showed that vision-language models (VLMs) trained only on synthetic Manchu word images can reach 87.4% word accuracy on real Qing manuscripts and prints, leaving a substantial synthetic-to-real gap. This study examines how synthetic and real historical training data should be combined for low-resource OCR. Using 60,000 synthetic and 20,306 real historical word images, we evaluate three pretrained VLMs and a compact convolutional recurrent neural network (CRNN) under four regimes: synthetic-only, real-only, joint synthetic-real, and sequential synthetic-to-real training, following a common checkpoint-selection and archival evaluation protocol. Introducing real training images raises the leading configurations to between 95.09% and 96.28% word accuracy, while no synthetic-only configuration exceeds 87.92%. Synthetic supplementation substantially improves all three VLMs, whereas its marginal effect for the CRNN is sensitive to the training objective. Joint and sequential training yield broadly similar archival accuracy under the tested practical pipelines. A compact CRNN also reaches the leading performance range once real images are available, showing that model scale alone does not determine recognition accuracy. Finally, complementary errors among strong recognizers allow voting to raise accuracy to 98.27% without additional training, while an eighteenth-century Manchu dictionary provides a principled rule for adjudicating disagreements.

1 Introduction

This study asks how synthetic and real historical images should be combined for low-resource Manchu OCR, motivated by archival inaccessibility and a persistent synthetic-to-real gap. It evaluates multiple recognizers and training regimes, finding that real supervision substantially improves accuracy and that voting further raises performance.

  • Motivation: Manchu archives are increasingly digitized but remain difficult to search and analyze because much material is available only as scanned images.Reliable OCR is presented as a step toward searchable, indexable, computationally usable collections.
  • Research question: The study examines whether training-data composition affects pretrained VLMs and a task-specific recognizer differently in a low-resource historical setting.It combines 60,000 synthetic images with 20,306 annotated real images and compares synthetic-only, real-only, joint, and sequential training.
  • Study and findings: 95.09–96.28% word accuracy is reached by leading configurations using real training images, whereas no synthetic-only configuration exceeds 87.92%.The study evaluates three pretrained VLMs and a compact CRNN under four training regimes.
  • Training regimes: Synthetic supplementation improves all three VLMs, but its marginal effect for the CRNN depends on the training objective, while joint and sequential training perform broadly similarly.These conclusions apply to the tested practical pipelines.
  • Ensemble result: Voting among strong recognizers raises archival word accuracy to 98.27% without additional training, using an eighteenth-century dictionary to adjudicate disagreements.The gain is attributed to complementary recognizer errors and dictionary-based attestation.

2 Background and Related Work

Manchu OCR addresses an important but difficult archival setting: vertically written historical documents are increasingly digitized yet often remain image-only. Prior work demonstrates both strong performance from authentic data and synthetic-to-real transfer, while leaving the best way to combine supervision unresolved.

  • Manchu script: Manchu’s vertical alphabetic script, positional letter forms, continuous word units, and document variation create distinctive OCR challenges.The study evaluates against original Manchu-script ground truth while using modified Möllendorff romanization for lexical lookup where needed.
  • Manchu sources and access: Manchu records contain historically important evidence, but digitized collections remain difficult to search because they are often available as page images rather than machine-readable text.Reliable OCR is framed as necessary for computational access at scale.
  • Research gap: Existing research shows both strong authentic-data performance and synthetic transfer, but the interaction between these supervision sources remains unclear.The present study targets this gap using a smaller authentic corpus and a separate archival benchmark.
  • Synthetic data: Synthetic data address OCR annotation scarcity, but cannot fully reproduce historical typography, degradation, handwriting variation, or capture conditions.This mismatch produces a persistent synthetic-to-real gap.
  • Related training research: Prior studies have not systematically compared synthetic-only, real-only, joint, and sequential training across pretrained VLMs and a task-specific OCR recognizer under one historical evaluation.The Manchu case study directly tests whether training-data effects transfer consistently across model classes.
  • Complementary methods: Recognizer combination and lexical information provide established routes for improving historical OCR beyond a single model’s prediction.Earlier work used aligned hypotheses, confidence voting, dictionary constraints, and lexicon-based post-correction.

3 Dataset

The dataset combines synthetic Manchu word images, an annotated real historical corpus, validation splits, and a separate archival test set under a common input format. The design supports controlled comparison of training regimes while preserving an unchanged benchmark for archival transfer.

  • Data overview: The experimental design uses synthetic and real historical images for training and validation, plus a separate archival test set, with SCI-val used for checkpoint selection.Training-regime labels specify which images are used for parameter optimization.
  • Synthetic corpus: 60,000 synthetic training images and 15,000 synthetic validation images are taken unchanged from a curated ManchuOCR subset generated from a 130,917-entry lexicon.The source corpus uses multiple Manchu typefaces.
  • Real historical corpus: The real SCI-DB corpus contains 24,280 word images from Manchu books printed between 1733 and 1867, with 2,428 unique words and ten samples per word originally.The images were extracted from the National Library of China’s Series of Rare Ancient Books in Manchu and Chinese.
  • Preprocessing: Normalization maps images to a 480 × 64 canvas using grayscale or inversion, denoising, contrast enhancement, resizing or rotation, and padding.Real vertical crops are rotated during normalization, while synthetic words are rendered horizontally.
  • Real-data split: After removing 615 unusable images, the real corpus yields 20,306 training images and 3,359 validation images, with most word labels represented in the held-out validation split.The split holds out one or two images for most of 2,306 word labels.
  • Archival test set: ARCH-test contains 753 word crops from seven Qing-period handwritten and printed sources and is retained unchanged for direct comparison with prior synthetic-only results.The benchmark uses the same 480 × 64 input format and includes both represented and unrepresented lexical items relative to training vocabularies.

4 Experimental Design

The study compares four ways to combine synthetic and real Manchu training images across three pretrained VLMs and a compact CRNN, using common checkpoint selection and archival evaluation. It reports word accuracy and CER after selecting checkpoints on SCI-val and reserving ARCH-test for final comparison.

  • Recognition Models: The model set comprises LLaMA, Pixtral, Qwen, and a fixed compact CRNN with a 9-layer CNN, 4-layer bidirectional LSTM, and CTC output.
  • Training Regimes: Four regimes compare synthetic-only, real-only, joint synthetic–real, and sequential synthetic-to-real training across each recognizer.JOINT shuffles both corpora in one run, whereas SEQ continues real-data training from a SYN checkpoint.
  • Training Regimes: JOINT and SEQ use the same data sources but differ in how real images are introduced, so their comparison is between practical pipelines rather than a controlled training-order test.SEQ warm-start provenance and total updates differ slightly across recognizers.
  • Checkpoint Selection: Checkpoints are selected by highest SCI-val word accuracy, with validation CER and then earlier training step resolving ties; ARCH-test is reserved for final evaluation.The best observed checkpoint is used because SCI-val performance is not always monotonic.
  • Evaluation Metrics: Word accuracy measures exact word matches, while CER is micro-averaged Levenshtein distance divided by total reference length after whitespace stripping.Both metrics are computed over the evaluation set’s word images, references, and predictions.
  • Statistical Comparison: ARCH-test comparisons use 95% bootstrap intervals from 1,000 resamples, with paired comparisons over the same 753 images.The paired intervals are conditional on the fixed ARCH-test corpus rather than estimates of generalization to new document sources.

5 Results

Adding real historical training images substantially improves archival Manchu OCR and changes which recognizers lead. Joint and sequential pipelines perform similarly overall, while their effects vary by model and evaluation domain.

  • Effect of Real Historical Training Data: 95.09–96.28% ARCH-test word accuracy is achieved by seven configurations using real training images, whereas no synthetic-only configuration exceeds 87.92%.The leading configurations include LLaMA-SEQ at 96.28% and CRNN-REAL at 95.88%.
  • Effect of Real Historical Training Data: LLaMA-SYN reaches 87.92% ARCH-test word accuracy, closely reproducing the earlier 87.4% synthetic benchmark.
  • Joint versus Sequential Training: SEQ has higher ARCH-test point estimates than JOINT for all four recognizers, but every paired 95% confidence interval includes zero.The differences are 0.27 percentage points for CRNN, 0.66 for Pixtral, 1.20 for LLaMA, and 1.59 for Qwen.
  • Joint versus Sequential Training: CRNN-SEQ retains 76.91% SYN-val accuracy versus 99.39% for CRNN-JOINT, despite similar ARCH-test performance.Sequential real-data fine-tuning reduces synthetic-domain performance without a corresponding archival gain in this comparison.
  • Recognizer Differences: Training-data composition changes model rankings: LLaMA and Pixtral lead under SYN, CRNN leads under REAL, and three models converge at 95.09–95.48% under JOINT.Qwen remains lower under JOINT at 85.39%.
  • Recognizer Differences: Moving from REAL to JOINT raises accuracy by 3.85 points for LLaMA, 5.98 for Pixtral, and 14.48 for Qwen, while CRNN changes by −0.40 points under the released recipe.The CRNN ordering is implementation-sensitive according to a corrected-objective robustness check.

6 Residual Error Analysis and Dictionary-Guided Voting

The three strong recognizers make complementary errors, enabling voting to raise ARCH-test accuracy without retraining. QWJ dictionary attestation supplies a deterministic rule for disagreements, but residual errors remain when candidates are shared, unattested, or misranked.

  • 6.1 Residual Errors and Recognizer Complementarity: Only seven ARCH-test images are misrecognized by all three recognizers, while 67 of 73 union errors contain non-identical outputs suitable for adjudication.The Venn diagram tracks incorrectness overlap, not identical transcriptions; six images receive the same incorrect transcription from all three.
  • 6.2 Ensemble Construction: The ensemble combines LLaMA-SEQ, Pixtral-SEQ, and CRNN-JOINT, selected solely on SCI-val rather than ARCH-test performance.Qwen is excluded because its best SCI-val configuration performs more than three percentage points below the other models.
  • 6.2 Dictionary-Guided Adjudication: QWJ adjudication returns consensus on agreement, prefers attested candidates on disagreement, and uses vote count plus SCI-val priority for ties.When no candidate is attested, the rule falls back to majority voting under the same priority and has no trainable parameters.
  • 6.3 Voting Performance and Residual Errors: 98.27% ARCH-test word accuracy reduces errors from 28 to 13 through dictionary-guided voting, correcting 16 errors while introducing one.The gain is supported by complementary recognizer errors; 60 of 67 disagreement cases are resolved correctly, while six shared incorrect outputs remain wrong.
  • 6.3 Voting Performance and Residual Errors: Plain majority voting also reaches 98.27% ARCH-test accuracy, so QWJ contributes a principled disagreement rule rather than an additional benchmark score.QWJ provides particular value when all three outputs differ or no majority exists.
  • 6.3 Voting Performance and Residual Errors: The 13 residual errors comprise six identical-error cases and seven unresolved disagreements, including absent correct forms and one incorrectly losing attested candidate.Five unresolved cases involve inflected or derived forms absent from the QWJ token lexicon.

7 Discussion

Real historical supervision brings compact CRNNs into the leading performance range and makes training-data composition recognizer-dependent. Complementary recognizer errors enable a no-retraining ensemble gain, while QWJ primarily provides principled adjudication rather than extra ARCH-test accuracy.

  • 7.1 Real Data and Model Scale: Real images bring a compact CRNN into the same leading ARCH-test performance range as much larger VLMs, so model scale alone does not reliably predict accuracy.The best CRNN and VLM results differ by less than half a percentage point, within the reported paired uncertainty.
  • 7.2 Synthetic-Real Composition: Synthetic supplementation improves all three VLMs but slightly decreases CRNN performance under the released training recipe, making its value recognizer- and implementation-dependent.REAL-to-JOINT gains are 3.85 points for LLaMA, 5.98 for Pixtral, and 14.48 for Qwen.
  • 7.2 Synthetic-Real Composition: Joint and sequential training show no generally superior schedule because every paired confidence interval includes zero, despite slightly higher sequential point estimates.Operational considerations such as second-stage training and checkpoint hand-off may therefore influence the choice.
  • 7.3 Complementarity and Lexical Adjudication: A heterogeneous ensemble raises ARCH-test accuracy from 96.28% to 98.27% without additional training because strong recognizers make complementary errors.Errors fall from 28 for the best single recognizer to 13, with only seven images misrecognized by all three.
  • 7.3 Complementarity and Lexical Adjudication: QWJ attestation does not increase ARCH-test accuracy beyond plain majority voting, but it supplies a reproducible rule when no majority exists or candidates remain plausible.Its role is therefore specific to principled disagreement adjudication rather than the ensemble’s overall benchmark gain.
  • 7.3 Complementarity and Lexical Adjudication: Historical lexical resources can be added after recognition as an adjudication layer, although their usefulness remains bounded by lexical coverage.This connects period dictionaries and similar structured resources with modern recognition pipelines without retraining the underlying models.

8 Limitations and Future Work

The conclusions are constrained by a small fixed archival benchmark, single-run training, one real-data corpus, word-level evaluation, and QWJ’s historically specific lexical coverage. Future work requires broader corpora, repeated runs, page-level evaluation, and more transferable lexical resources.

  • Benchmark Scope: The fixed 753-image ARCH-test limits discrimination among leading configurations and does not quantify generalization to new sources or vocabulary.Its confidence intervals are conditional on this corpus; a larger, more diverse archival test set would address the boundary.
  • Training-Data Scope: All real training images come from SCI-DB, so the experiment does not establish whether synthetic-real patterns transfer across more varied historical corpora.Broader data should vary script style, document type, and image quality.
  • Run-to-Run Variation: Single training runs leave stochastic variation in initialization, data order, LoRA training, and checkpoint selection unmeasured.For the CRNN, objective sensitivity is confounded with changes in the realized training trajectory.
  • Evaluation Scope: Word-level evaluation assumes regions are already identified and therefore excludes page layout, segmentation, and end-to-end document transcription.Deployment across archival collections still requires extending the workflow to page-level OCR.
  • Lexical-Resource Scope: QWJ-guided adjudication is bounded by a historically favorable token lexicon that misses some inflected and derived forms and can prefer an attested incorrect candidate.Transfer to genres, periods, and lexical domains outside its tradition is therefore not established.

9 Conclusion

Real historical supervision is the clearest determinant of strong Manchu OCR, while recognizer diversity and lexical resources further reduce residual errors.

  • 95.09–96.28% word accuracy was achieved by several real-data configurations, whereas no synthetic-only configuration exceeded 87.92%.A compact CRNN also reached this leading range, so model scale alone did not determine recognition accuracy.
  • Synthetic supplementation substantially improved all three VLMs, but its effect on the CRNN varied with the training objective.The experiments found no universal advantage for joint over sequential training.
  • 98.27% ARCH-test word accuracy was reached by combining three strong recognizers without additional training.The gain primarily reflects complementary residual errors, with an eighteenth-century Manchu dictionary adjudicating disagreements.
  • Synthetic data can bootstrap recognition, real images should be added when available, and compact or heterogeneous recognizers should remain viable candidates.Existing lexical resources can further reduce residual error.

A.1 VLM Training

The VLM experiments use a shared fine-tuning and inference recipe, with training budgets and sequential warm starts specified separately.

  • All 12 VLM cells share one fine-tuning and inference recipe.Inference settings are identical for sweep-time SCI-val scoring and final ARCH-test scoring.
  • The JOINT budget equals the combined SYN and REAL budgets, while each training image receives the same number of passes under both regimes.The stated budgets are 25,100 = 18,750 + 6,350 and 502,000 = 375,000 + 127,000.

A.2 CRNN Training

The CRNN architecture and baseline hyperparameters are held constant across data-composition cells, with a robustness check correcting the loss input.

  • The CRNN architecture is identical across all four CRNN cells and is inherited unchanged from Chung and Choi (2026).
  • The four CRNN cells differ only in training-data composition, using an otherwise identical training configuration.
  • 91.50–96.28% versus 57.64% ARCH-test word accuracy was observed after adding log softmax before CTCLoss, preserving the conclusion that real-supervised training outperforms synthetic-only training.The robustness retraining left the remaining configuration identical, although run-to-run randomness was not controlled.

B Full Results and Statistical Comparisons

The full evaluation reports all 16 selected configurations, paired archival comparisons, and checkpoint-selection sensitivity for tied validation peaks.

  • Table B.1 reports word accuracy and CER for all 16 selected configurations across SYN-val, SCI-val, and the 753-image ARCH-test set.Rows are ordered by ARCH-test word accuracy, and each row uses the SCI-val-peak checkpoint selected for that configuration.
  • Table B.2 gives paired ARCH-test comparisons using word-accuracy differences, discordant-item counts, and paired bootstrap intervals.Ensemble rows use the dictionary-guided ensemble.
  • 95.48% ARCH-test word accuracy was obtained by the selected CRNN-JOINT checkpoint, compared with 96.15–96.41% for its other tied checkpoints.The formal selection rule remains fixed and the overall conclusion is unchanged, but the spread affects interpretation of the CRNN JOINT–SEQ comparison.
  • All selected and alternate tied-peak checkpoints lie inside the leading band’s confidence intervals.
Loading 2609.11495v1…