Source-linked AI summary
Reading the Registers of Honorius III without Manual Line Transcription: Forced-Aligned HTR, End to End, Modular or Hybrid
Tamas Kovacs, Daniel Luger
TL;DR
Line-level expanded transcription is scarce for Honorius III’s heavily abbreviated registers, limiting machine-readable study of this historical source. The paper aligns existing line-break-free expert transcriptions to manuscript images and trains hybrid HTR; its best single model reaches 4.7% CER and 8.7% WER on alignable test lines.
Problem
Expanded line-level ground truth is scarce for heavily abbreviated papal registers, although historical analysis requires machine-readable text.
Source: §1 · p. 1Method
Results
4.7% CER and 8.7% WER are achieved by the best single hybrid, while guarded ROVER reaches 8.4% WER on alignable test lines.
Source: §4 · p. 5Takeaways & Limitations
Existing expert transcriptions without line breaks can become training data without annotation cost, while target conventions determine whether initials are reproduced as written.
Source: §5 · p. 5Results come from one register volume and held-out pages of the same hands, with test pages restricted to versos.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
Unlocking medieval archives at scale is bottlenecked by the cost of line-level diplomatic transcriptions and manual abbreviation expansion. Focusing on the registers of Pope Honorius III (1216-1227) over 5,000 letters surviving in full only as manuscript images - we show that existing, line-break-free expert transcriptions (edition drafts) suffice to train end-to-end hand-written text recognition (HTR) that jointly transcribes and expands heavily abbreviated Latin. Using forced alignment on 860 unaligned edition entries across 394 pages of Reg. Vat. 9, we automatically bootstrap 13,122 training lines without manual alignment of segmented lines. While the generic CATMuS baseline struggles on this domain (18.9% CER) and a modular space-free HTR + word-segmentation pipeline (cf. Camps et al. 2021) plateaus at 15.6% WER, our from-scratch hybrid Conformer - combining CTC with bidirectional subword attention decoders, iterative self-realignment and n-best LM rescoring - achieves 4.7% CER and 8.7% WER (8.4% WER with ROVER system combination), nearly halving errors on clause-final words (22.7% to 12.2% WER). These figures are measured on the test lines that align with the transcription (86% of all test lines for the best hybrid). Furthermore, we address a tension between paleographic fidelity and editorial normalization: training on targets that preserve scribal name initials reduces the WER against the manuscript text on the 31 affected test lines from 13.3% to 9.2%, at a small cost against the expanded transcription (WER +0.3 points, CER +0.4). Finally, extensive ablations show that none of an auxiliary diplomatic CTC head, a copy mechanism, a permutation decoder or masked-image pretraining improves on direct end-to-end supervision, while strict padding-mask propagation and per-line beam termination make inference batch-invariant.
1 Introduction
Honorius III’s registers are historically important but remain unedited and difficult to make machine-readable because dense script and pervasive abbreviation complicate HTR. The paper uses existing expert transcriptions to build expanded-text recognizers and compare end-to-end, modular, and hybrid approaches.
1 Introduction
- 47% of Reg. Vat. 9 words are abbreviated, while graphemic transcription leaves case endings and syllable counts undetermined.Expanded line-level ground truth is scarce, despite the need for machine-readable text in historical analysis.Source: §1 · p. 1
- Earlier Honorius-register HTR reported 19.18% CER on 39 pages and 15.90% CER with a model trained on 9,000 expert lines of other material.Later ICR work used expert-transcribed whole lines on about 20 folios; residual errors included unseen abbreviations and inconsistent target expansions.Source: §1 · p. 1
- Camps et al. (2021) found a diplomatic-recognition, segmentation, and normalization chain better than unsegmented expanded output, but its behavior with more expanded data was untested.On 1,524 training lines, the chain reached 31.81% WER against 54.68%.Source: §1 · p. 2
- 13,122 silver lines are bootstrapped from existing transcriptions, and expansion-aware recognizers improve on earlier work on the registers.The study also compares end-to-end, modular, and hybrid recognition and tests targets that preserve written name initials.Source: §1 · p. 2
2 Data and Silver Lines
The study uses 394 pages of Reg. Vat. 9 and two line-break-free expert transcriptions, then locates entry spans on manuscript lines by forced alignment. This produces silver training lines and a page-based train, development, and test split.
2 Data and Silver Lines
- 394 page images and two expert transcriptions provide 203,819 Latin words, with abbreviations expanded and no line breaks.The sources cover regnal year 1 and the first half of year 2; apparatus and editorial notes are removed, and parenthesized expansions are joined to their words.Source: §2 · p. 2
- 13,122 silver lines are accepted from spans of at least 15 characters with normalized distance at most 0.25.Kraken segments pages, CATMuS reads lines, and semi-global edit distance locates each line in the folio’s transcription; 82.4% of lines pass the broader acceptance threshold of 0.35.Source: §2 · p. 2
- 99.98% of accepted lines are placed correctly in simulation at 15% CER with half the words abbreviated.The split uses 315 training, 40 development, and 39 test pages, yielding 10,525 training and 1,313 development lines; test pages are versos and development pages rectos.Source: §2 · p. 2
- Targets inherit editorial expansions of name initials, while footnote marks and editorial notes are excluded.A lexicon check estimates transcription slips at 0.13–0.46% of reference words.Source: §2 · p. 2
3 Recognizers and Variants
The study compares pretrained and from-scratch recognizers with modular, language-model, and attention-decoder variants. It also defines evaluation measures for expanded-text accuracy, line coverage, and preservation of clause-final cursus.
3 Recognizers and Variants
- C is a from-scratch Conformer-CTC recognizer, while A is the published CATMuS model and B fine-tunes A on silver lines.The variants share data, split, seed, optimizer, and early stopping; C uses greedy CTC decoding.Source: §3 · p. 2
- The hybrid adds jointly trained attention decoders to the masked encoder, while LM fusion adds a Kneser–Ney word trigram model to CTC prefix beam search.The four decoder variants include plain hybrid, copy-gate, permutation, and diplomatic-head models; training text excludes entries touching held-out pages.Source: §3 · p. 2
- Table 1 compares published results on papal registers and related material with this study’s recognizers, but the comparison is indicative because test sets, targets, and metrics differ.The table concerns held-out pages from the same register for this study’s recognizers.Source: Table 1 · p. 3
- The modular route removes spaces before character tagging, while sigla-as-written targets retain name initials as they appear on the page.The segmenter is trained on Honorius text, Innocent III text, or a word-bigram model in addition; the NER head labels characters using rule-derived labels.Source: §3 · p. 3
- Silver v2 re-aligns training pages with hybrid readings, then tests rotary/depthwise kernels, right-to-left decoding, BPE targets, and LM rescoring.The guarded ROVER replaces a primary hybrid word only when two other systems propose the same different word; choices are fixed on development pages.Source: §3 · p. 3
- CER and WER use Levenshtein distances on the reading layer against expanded transcription, with WER counting reference words not reproduced.Significance is assessed by paired bootstrap over 39 test pages; headline measures cover recognizer-dependent aligned lines, not every page line.Source: §3 · p. 3
- Cursus is assessed at clause endings using a rule-based classifier that identifies final-word stress patterns in the expert text and recognizer output.The passage defines α as noncanonical endings read as canonical and β as canonical endings read as noncanonical.Source: §3 · pp. 3–4
4 Results
The hybrid’s final configuration achieves the strongest reported results, while experiments show gains from language modeling and iterative decoding, and expose trade-offs between modular segmentation and transcription conventions.
Accuracy and earlier work
- Learning expansion from expanded targets is decisive: C reaches 7.1% CER and 14.7% WER, while B reaches 8.3% CER and 18.3% WER; registers, test sets, and metrics differ from prior work.With 1,052 training lines, C reaches 11.5% CER and 28.9% WER, and its learning curve has not flattened at 10,525 lines.Source: §4 · p. 4
Where the errors are
- 22.7% of clause-final words are wrong, while abbreviated and fully written words are misread at similar rates, 14.4% versus 15.0%.Words outside the training vocabulary are wrong in 26.9% of cases, compared with 13.6% for other words; 2.0% of word errors are pure boundary errors.Source: §4 · p. 4
Variants
- The internal language model lowers hybrid WER from 14.7% to 12.5%, while external word modeling reaches 13.0% and masked retraining 14.3%.The copy gate, diplomatic head, permutation decoder, and masked-image pretraining do not improve on their reference systems.Source: §4 · p. 4
Modular route and sigla as written
- 15.6% WER is the best reported result for the tested modular route, which still trails end-to-end C; re-segmenting C gives 16.2%.Papal Latin improves the segmenter from 20.8% to 15.6% WER, but it splits unseen names, whose OOV WER is above 32%.Source: §4 · p. 4
Iterative alignment, bidirectional and subword decoding
- 8.7% WER and 4.7% CER for the best single hybrid; guarded ROVER reaches 8.4% WER and clause-final WER falls to 12.2%.Paired bootstrap differences against the first hybrid are −3.7 WER points and −1.7 CER points for the best single hybrid, with p < 0.0001.Source: Table 2 · p. 4, §4 · p. 5
Names and sigla
- 9.2% WER on the 31 affected test lines against the sigla reference, down from 13.3% with the hybrid trained on expanded name initials.Against the expanded transcription, sigla-as-written targets raise WER from 12.5% to 12.8% and CER from 6.4% to 6.8%; the jointly trained named-entity head adds no measurable gain.Source: §4 · p. 5
5 Discussion and Conclusion
The paper concludes that line-break-free expert transcriptions can supply training data without manual line alignment, and that end-to-end recognition can learn word spacing with sufficient data. Its machine-readable output remains a starting point for expert review, not a substitute for an edition.
5 Discussion and Conclusion
- Line-break-free expert transcriptions become training data without annotation cost, and recognizer-based realignment adds lines the graphemic reading could not place.The discussion says data volume explains most of the accuracy.Source: §5 · p. 5
- The tested space-free modular route loses once the recognizer has enough data to learn word spacing; the diplomatic chain from Camps et al. remains untested.The paper identifies the output convention as a choice to fix before training: abbreviations expanded, name initials as written.Source: §5 · p. 5
- Bidirectional attention decoding repairs clause-final inflections, while preserving initials as written makes output follow the scribe at a small cost against edited text.The paper reports no measurable gain from adding a named-entity head.Source: §5 · p. 5
- Batch-invariant inference requires deterministic recognition: unmasked convolutions and batch-wide beam stopping make output depend on the batch.Source: §5 · p. 5
- Errors remain concentrated at clause ends and in names, so the machine-readable registers need calibrated review and are a starting point, not a substitute for an edition.Source: §5 · p. 5
Limitations
Results are limited to one register volume and held-out pages of the same hands; most models use one seed, and the headline figures cover lines aligned with the transcription.
Limitations
- All results use one volume and held-out pages of the same hands, with test pages only on versos and development pages only on rectos.The earlier-work comparison uses published figures from different test sets and metrics.Source: Limitations · p. 5
- Most models use a single seed, so differences of a few tenths may reflect run-to-run variation; a second seed confirmed the key hybrid’s 10.6% WER.Its CER was 5.1% and 5.4% across the two seeds, against 12.5% WER for the first hybrid.Source: Limitations · p. 5
- The diplomatic modular route requires diplomatic ground truth, which is not yet available for these registers.Silver-v2 realignment used pre-batch-invariant decoded readings only to select spans; the permutation decoder is simplified, and pretraining used training-page lines because other volume pages are reserved for calibration.Source: Limitations · p. 5
- The headline figures cover only test lines that align with the transcription, and the NER head is evaluated against rule-derived rather than expert labels.No named-entity annotation exists for the registers; the head’s gain over sigla-as-written targets alone is small and not significant on the 31 affected lines.Source: Limitations · p. 6
A Training Details
The appendix specifies the recognizer and decoding recipes, reports variants that did not improve test WER, and identifies Table 3’s corpus and split counts.
A Training Details
- The hybrid uses attention decoders and n-best rescoring, while its recipe also includes a right-to-left decoder, character/BPE/word language-model rescoring, and optional NER or MWER variants.The appendix also specifies masked-image pretraining and reports that minimum-Bayes-risk decoding did not improve development WER: 7.49% against 7.45%.Source: A Training Details · pp. 7–8
- The listed variants did not improve test WER over their references: auxiliary CTC changed masked C from 14.3% to 14.6%, while the hybrid model soup remained at 10.6%.The table also reports CER changes for these and other variants; vertical attention pooling diverged twice, and checkpoint averaging gained at most 0.1 development-CER points.Source: A Training Details · p. 8
B Corpus and Silver Lines
- Table 3 reports transcription, recognized-line, accepted-line, and silver-line counts by transcription and split.Accepted lines use distance ≤0.35; silver lines require at least 15 characters and distance ≤0.25.Source: Table 3 · p. 8
C Error Bases
The section defines distinct line, compared-text, and page error bases; it also specifies the scopes of the recognizer-comparison and learning-curve tables.
C Error Bases
- 7.8% line CER for C exceeds its 7.3% CER on lines of at least 15 characters, mainly because of short lines.Compared-text CER averages pages for accepted lines of at least 15 characters; page CER also counts marginalia, rubrics, and reading-order errors.Source: C Error Bases · p. 8
- Table 4 compares recognizers on three error bases, with WER aligned per page and hybrid scores measured against the expanded transcription.Its aligned share is based on the 2,004 test lines; line CER, compared CER, and WER use accepted lines of at least 15 reference characters.Source: Table 4 · p. 8
D Learning Curve
- Table 5 reports C’s early-stopping epochs, best development CER, and test CER and WER when trained on random samples of silver training lines.Source: Table 5 · p. 8
E Errors by Position
The section defines clause-final tokens and the rule used to classify a word as abbreviated.
E Errors by Position
- Clause ends are reference tokens ending in ., ?, ;, or !, excluding abbreviation points and register truncations.The exclusions include initials, numerals, Kal., Id., etc., and usque.Source: E Errors by Position · p. 8
- A word counts as abbreviated when the CATMuS reading at the same position contains an abbreviation sign.Source: E Errors by Position · p. 8
F Names and Sigla
The section defines how name initials are handled and evaluates a rule-derived named-entity head. Its F1 measures reproduction of rule labels, not agreement with expert annotation.
F Names and Sigla
- A name is a capitalized word that does not open a clause, excluding specified Latin forms and chancery, calendar, or ecclesiastical titles.Place adjectives ending in -ensis count as places; other names count as persons.Source: F Names and Sigla · pp. 8–9
- The head's F1 measures reproduction of rule-derived labels from decoder states, not agreement with an expert.Source: F Names and Sigla · p. 9
- F1 = 0.78 for the rule, 0.69 for the head, 0.73 for their union, and intersection precision 0.85 on development recognized text.The rule's advantage is partly circular because reference labels come from applying it to expert text.Source: F Names and Sigla · p. 9
G Reproducibility
Batch layout could change both Conformer outputs and hybrid beam-search results. Propagating padding masks and stopping each line's beam independently makes inference effectively batch-invariant.
G Reproducibility
- Padding changed up to 810 of 2,004 C test lines across batch layouts; masking every convolutional and normalization layer removed the dependence without retraining.A model trained with mask propagation gives identical metrics for every batch layout.Source: G Reproducibility · p. 9
- With original hybrid beam stopping, 402 of 2,004 test lines changed across batch layouts and WER ranged from 11.62% to 12.25%; per-line stopping left at most 3 lines changed.The best single hybrid had only 1 changed line under per-line stopping.Source: G Reproducibility · pp. 9–10
- Table 8 compares changed lines and metrics across width-grouped or random batches, including original versus per-line hybrid beam stopping.Source: Table 8 · p. 10