Source-linked AI summary
Text Restoration of Ancient Documents with Language Models
Shibingfeng Zhang, Edoardo Caraffa, Annafelicia Zuffrano, Maddalena Modesti, Giovanni Colavizza
TL;DR
The paper asks whether language models can restore missing text in damaged ancient manuscripts, a task traditionally requiring paleographic expertise. It compares models and realistic known- and unknown-length scenarios, finding that restoration is not fully automatable but can assist paleographers, especially for formulaic sections and when lacuna length is known.
Problem
The study addresses the need to restore missing or illegible ancient-text passages and determine how reliably language models can perform this task.
Method
The study systematically compares pretrained and large language models across realistic restoration scenarios, including character-length-aware and length-unknown settings, using decoding strategies to address tokenization mismatches.
Results
Text restoration cannot yet be fully automated; formulaic sections are considerably easier than non-formulaic sections, and overall hit-rate@1 drops by 25.85% without lacuna-length information.
Takeaways & Limitations
The resulting models and methods can support paleographers as assistive tools, with selective deployment and greatest confidence in formulaic contexts.
Takeaways & Limitations
The proposed models are constrained by the relatively small notarial-document dataset, motivating reliance on Latin-pretrained resources.
Abstract
from arXiv · showhide
Purpose - This study investigates the feasibility of restoring missing text caused by physical lacunae in damaged ancient manuscripts using language models. Methodology - The study proposes different scenarios to replicate real-world conditions. Language models of different architectures are applied according to their suitability to each scenario. We also propose several decoding strategies that further enhance performance and address the discrepancy between lacuna boundaries and the models' tokenization schemes. Findings - The results reveal that text restoration of these documents cannot be fully automated, but it can serve as a useful tool to assist paleographers in their work. Model performance varies greatly depending on which structural part of the document needs to be restored and whether the character length of missing text is available. Originality - This is the first study and to analyze model performance on formulaic and non-formulaic content and the impact of lacuna length awareness in manuscript restoration. Both are recurring challenges in paleographers' manual restoration work. Through systematic comparison and both qualitative and quantitative analysis of different models' performance under varying settings, this study offers a guideline for developing assistive tools to support paleographers.
1 Introduction
The study examines whether language models can restore missing ancient-text passages and develops assistive tools in collaboration with paleography and diplomatics. It compares architectures and experimental settings, including unknown lacuna lengths, to identify restoration practices and support scholars.
- The dataset contains around 1,200 previously unseen Late Medieval Bolognese notarial documents, enabling comparisons of formulaic and non-formulaic content.The documents’ unseen status supports a fair comparison, while their structure makes the distinction transferable to other historical traditions.
- Several language-model architectures and sizes are evaluated across multiple experimental settings to identify suitable restoration practices.
- The study investigates whether language models can reliably reconstruct missing or illegible content in ancient texts.
- The study compares strategies for reconciling character-level lacuna lengths with language-model tokenization, including unknown-output-length scenarios.
- Systematic experiments and analysis across models and text categories provide guidance for restoration in under-resourced historical domains.
- Fine-tuned language models are provided as practical tools to support paleographers and diplomatists.
2 Related Work
Prior work combines traditional restoration practices with domain-adapted language models, while large language models remain largely unexplored for historical text restoration. This study extends the field by comparing known- and unknown-length settings and incorporating prompting practices inspired by paleography.
- Historical text restoration traditionally relies on paleographic and diplomatic analysis, including analogical comparison and statistical evidence from related textual production.
- Recent computational approaches commonly adapt general-use pretrained models to specialized historical domains, often assuming the missing-text length is known.
- Large-scale language models had not, to the authors’ knowledge, been explored for historical text restoration, motivating few-shot prompting experiments.
- Prompts draw on manual restoration practices by supplying task-specific examples and word concordances as contextual information.
- The study investigates both known-length and unknown-length restoration to examine the trade-off between estimating lacuna size and restoration accuracy.
- The study considers Latin pretrained models, including LatinBERT and other generative models, alongside specialized systems such as Aeneas.
3 Data
The dataset comprises medieval Latin notarial documents from Bologna, organized by documentary type and functional component. Their contrasting formulaic and non-formulaic sections support fine-grained evaluation of restoration difficulty.
- The corpus contains 1,184 Latin notarial documents produced in Bologna between the early eleventh and late thirteenth centuries.The collection spans approximately 280 years.
- The documents include transactions such as sales, donations, exchanges, concessions, emphyteutic leases, obligations, and renunciations, as well as testamentary acts.
- Professional diplomatists transcribed the documents digitally to ensure high accuracy and consistency in the representations.
- Notarial documents comprise three functional components—protocol, text body, and eschatocol—with rogatio serving as a summary note for later extended drafting.
- Protocol and eschatocol are highly formulaic, whereas text body and rogatio are less standardized because they reflect individual transactions.
4 Experimental design & Methodologies
The experiments reproduce practical manuscript-restoration conditions by comparing settings with and without lacuna-length information. They evaluate language-model restoration using historical document examples and distinguish accuracy from implementation practicality.
- The experimental design aims to develop tools that automate or facilitate paleographers’ reconstruction work under realistic conditions.
- Length-known setting: The length-known setting supplies the estimated number of missing characters together with the surrounding textual context.
- Figure 1 presents a 1132 notarial document together with its English translation as an example of the source material.
- Length-unknown setting: The length-unknown setting supplies only the text surrounding the lacuna, without an estimate of missing-text length.
- Length-known restoration is expected to produce more precise and controllable predictions, whereas length-unknown restoration is easier to implement and more broadly applicable.In the dataset, reliable length estimation is unavailable for 6.8% of damages.
4.1 Dataset Partitions and Constructing the Test Set
The test set is constructed by partitioning documents, generating sliding-window samples, and injecting synthetic lacunae into intact test regions. Damage lengths are sampled from trimmed empirical distributions and balanced between short and long spans.
- Evaluation uses artificially damaged intact test regions rather than naturally lacunose passages, avoiding uncertain paleographical restorations as ground truth.
- Documents are divided into training, development, and test sets using an 8:1:1 ratio, with samples generated by sliding windows.
- Synthetic lacuna lengths are sampled from the empirical distribution of real training-set damages after trimming values outside the 15th–85th percentiles.
- The filtered length pool is split at a median of 7 characters into short and long pools for robustness analysis.
- For each text window, the pipeline draws ⌈n/2⌉ spans from the short pool and ⌊n/2⌋ from the long pool, rejecting overlaps with natural lacunae or previously selected spans.
4.2 Length-known Setting
The length-known setting supplies the estimated missing-text length alongside surrounding context and selects models able to support Latin, controllable output, and reproducible open use. Character-level tokenization provides direct length control, while subword models require boundary-aware decoding strategies.
- The length-known setting provides the estimated character length of the lacuna together with its surrounding context.
- Selected models must support controllable output length, Latin, and open-source or open-weight reproducible use.
- LaBERTa, PhilBERTa, and Aeneas are selected for the length-known setting based partly on how their tokenization supports length control.
- Aeneas uses character-level tokenization, making the number of decoded tokens correspond one-to-one with the restored text’s character length.
- LaBERTa and PhilBERTa formulate restoration as masked language modeling, but byte-pair tokens can span multiple characters.
- The proposed decoding expands lacunae to nearby token boundaries, estimates token composition, and iteratively decodes using prefix conditioning and suffix matching.
- Few-shot general-purpose LLMs are included for a no-training-data scenario, with word concordances supplied as a corpus-based analogue to manual restoration practice.
4.3 Length-unknown Setting
The length-unknown setting withholds the missing text’s character length and evaluates whether models can restore content from context alone. It uses specialized models and few-shot LLMs, with tokenization-aware decoding retained where applicable.
- The length-unknown setting assumes that only surrounding context is available and the missing text’s character length is unknown.
- LaTa, PhilTa, and Aeneas are selected for the length-unknown setting, while LLaMA-70B Instruct and deepseek-v4-pro are evaluated few-shot.
- For Aeneas, the unknown-length condition changes the lacuna marker from “-” to “#” while leaving the architecture and initial weights unchanged.
- LaTa and PhilTa expand lacunae to token boundaries and use prefix conditioning with suffix-matching bonuses, without restricting output length.
- LLMs are tested for robustness by removing the supplied lacuna character length while keeping the prompt, temperature, and other hyperparameters unchanged.
5 Results
Results were evaluated across repeated runs, model types, lacuna lengths, functional components, and prompting conditions. Performance generally favored shorter lacunae and known lengths, while restoration difficulty varied substantially by document component and model.
- Evaluation protocol: Five-run means with 95% confidence intervals were reported for all experiments, including independently trained fine-tuned checkpoints.LLM experiments were repeated five times; fine-tuned models used five checkpoints with different random seeds.
- Length-known setting: Aeneas outperformed other systems across most metrics, except Overlap Score and HR@10 for long lacunae, where its difference from LaBERTa was non-significant.Overall performance was higher for short lacunae and declined as lacuna length increased.
- Functional components: Fine-tuned models restored protocol sections most easily and rogatio sections most difficultly, while LaBERTa and multilingual PhilBERTa achieved very similar results.LaBERTa sometimes held a slight advantage over PhilBERTa.
- Length-unknown setting: Best Hit-rate@1 fell from 0.72 for short and 0.29 for long lacunae with known lengths to 0.42 and 0.18, respectively, when lengths were unknown.The unknown-length setting significantly increased restoration difficulty.
- Length-unknown setting: In the unknown-length setting, Aeneas led on short lacunae, LaTa led on long lacunae for CER and Hit-rate@1, and DeepSeek achieved the best CER and Overlap scores overall.DeepSeek outperformed LaTa and PhilTa on short lacunae but fell behind LaTa on long lacunae for Overlap Score and HR@N.
- Functional components: DeepSeek outperformed all other systems on eschatocol restoration with statistically significant advantages in CER, Overlap Score, and HR@1.Across functional components, protocol remained easier and rogatio harder for fine-tuned models.
- Prompting ablation: Combining few-shot examples with concordance evidence consistently produced DeepSeek’s best performance in both known- and unknown-length settings.Few-shot prompting mainly improved CER, whereas concordance evidence mainly improved hit rates; combined gains covered all evaluated aspects.
6 Discussion
Restoration difficulty depends on lacuna length, document component, and how plausibility is evaluated. Expert review shows that automatic metrics can underrate useful alternatives to the ground truth.
- Performance patterns: Short lacunae are consistently easier to restore than long lacunae in both length-known and length-unknown settings.Longer spans require more decoding steps, creating more opportunities for divergence from the ground truth.
- Performance patterns: Formulaic protocol and eschatocol sections, together with text body, are easier for every model to restore than rogatio.The authors attribute this pattern partly to the greater frequency of formulaic components in training material.
- Performance patterns: Rogatio has the highest lexical richness and Hapax Legomena-to-token ratio, yet multilingual training did not provide the expected advantage in this non-formulaic context.This lexical profile helps explain why rogatio presents a distinct restoration challenge.
- Qualitative evaluation: Paleographers evaluated grammaticality and contextual suitability separately when assessing whether model predictions were plausible restorations.Grammaticality concerns Latin morphological and syntactic compatibility, whereas context concerns capturing the information required by the surrounding passage.
- Qualitative evaluation: 13.3%–24.9% of non-matching predictions were judged plausible alternatives across the evaluated models and length settings.The proportions were 23.6%, 18.8%, and 13.3% in the length-known setting, and 24.9%, 20.3%, and 15.3% in the length-unknown setting.
- Model-specific tendencies: Aeneas performed well on proper names, kinship relations, and short lacunae, while LaBERTa and LaTa showed strengths on common nouns and material goods.LaBERTa occasionally produced grammatically and contextually acceptable restorations when other models failed.
7 Conclusion
The study evaluates language models for restoring damaged Latin notarial manuscripts across architectures and realistic restoration settings. It concludes that restoration is not yet fully automatable, but can assist paleographers when applied selectively.
- Conclusion: The study systematically compares pre-trained models with different architectures and general-purpose LLMs across real-world restoration scenarios.The evaluation addresses both technical challenges in historical NLP and practical needs of paleographers.
- Conclusion: Text restoration cannot yet be fully automated, but it can serve as a valuable assistive tool for paleographers.Formulaic sections are easier to reconstruct than non-formulaic sections, supporting selective deployment.
- Conclusion: 25.85%: overall hit-rate@1 drops by this amount when lacuna length becomes unavailable.Length-known and length-unknown settings are compared directly, while useful hypotheses can still be generated without length information.
- Conclusion: The work contributes a systematic unknown-length analysis, component-level performance comparison, and guidance on architectures and decoding strategies.Fine-tuned models and methodologies are made available to support paleographers.
A Prompt
The prompt supplies lacuna length information only in the length-known setting, and the displayed input text is truncated for space in the paper.
- Prompt: Lacuna length is provided to the language model only in the length-known setting.
- Prompt: The prompt figure shows an input text that is truncated for limited space in the paper.