PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging External Knowledge for Historical Document Restoration via Retrieval-Augmented Large Language Models

Gabeen Kim, Kyeongpil Kang

arXiv:2607.21936v1cs.CL

TL;DR

Historical document restoration is hindered by damage and by the difficulty of recovering named entities from local context alone. ARI combines pretrained LLM knowledge with retrieval-augmented generation, outperforming existing baselines overall and especially on externally dependent named entities while showing practical value for domain experts.

  • Problem

    Existing restoration methods rely mainly on local document context, limiting their ability to recover named entities that require external historical knowledge.

  • Method

    ARI uses a pretrained large language model with retrieval-augmented generation and historical metadata to restore damaged characters in Korean archives.

  • Results

    ARI consistently outperformed existing baselines overall, with especially notable gains for named entities, on ground-truth tests and expert evaluations of real-world damaged data.

  • Takeaways & Limitations

    The framework provides practical value for domain experts and may support broader use of historical archives by reducing restoration time and cost.

  • Takeaways & Limitations

    A potential discrepancy may remain between the constructed datasets and the intrinsic characteristics of real-world damaged documents.

Abstract

from arXiv · show

Historical documents act as invaluable knowledge archives but often suffer from illegibility due to physical deterioration and damage. While existing restoration methods based on masked language modeling effectively utilize local context, they struggle to restore named entities that require external historical knowledge. To address this limitation, we introduce a novel framework for historical document restoration that leverages large language models with retrieval-augmented generation (RAG). By combining the implicit knowledge of pre-trained LLMs with explicitly retrieved external context, our model ARI effectively mitigates the challenge of inferring context-dependent proper nouns. Extensive experiments on Korean historical documents demonstrate that our approach significantly outperforms baselines, achieving substantial gains in restoring both general characters and named entities. Furthermore, comprehensive evaluations including expert assessments confirm that ARI serves as a practical tool for domain experts, promising to accelerate the analysis of historical records.

1 Introduction

Historical document restoration is hindered by physical damage and by the difficulty of recovering named entities without external knowledge. The proposed LLM-based retrieval-augmented framework combines implicitly learned historical context with explicitly retrieved information to address this limitation.

  • Challenges: Historical manuscripts are vulnerable to environmental damage and degradation because they were commonly written on less durable materials than modern paper.Exposure to light can cause discoloration and structural deterioration.
  • Challenges: 41.9K characters across 11.1K JRS documents remain damaged or unrecognizable.
  • Limitations of Existing Methods: Masked language modeling methods rely only on the damaged document, limiting their ability to infer named entities that require external historical knowledge.The passage highlights proper nouns as a key case where external information is crucial.
  • Proposed Framework: The proposed framework uses an LLM-based retrieval-augmented generation approach to combine implicitly learned historical context with explicitly retrieved external information.The pre-trained LLM learned indirectly from extensive datasets, including historical knowledge and documents available on the web.

2 Related Work

Historical document restoration has progressed from deep-learning correction and masked language modeling toward causal LLMs augmented with retrieved external knowledge. Prior methods rely heavily on internal document context, limiting restoration of information such as named entities.

  • Deep-learning restoration: Earlier restoration methods used deep learning to correct typographical errors, predict missing characters, and jointly perform restoration and translation.Pythia achieved 73.5% top-20 accuracy on missing-character prediction in Greek epigraphy.
  • Limitations of prior methods: Single-document input restricts prior models to internal context, causing shortcomings when restoration requires external knowledge, especially for named entities.The related work identifies external data sources as necessary for effectively restoring damaged historical documents.
  • Masked language modeling: Masked language modeling is widely used because recovering masked tokens resembles repairing damaged text, but BERT-based approaches predict tokens nonautoregressively.BERT uses a bidirectional Transformer Encoder trained by masking random training-data positions.
  • LLMs and retrieval augmentation: Large language models provide implicit knowledge from extensive archives, while retrieval-augmented generation supplies relevant external documents as context through lexical- or embedding-based retrieval.RAG can incorporate information absent from pre-training and alleviate hallucinations.
  • Proposed direction: The proposed approach combines causal language modeling, retrieved relevant documents, and fine-tuning to improve historical document restoration using external knowledge.The causal model is intended to capture contextual dependencies of damaged characters, while fine-tuning optimizes use of external information.

3 Dataset

The study uses the Hanja-based AJD and JRS corpora, enriched with temporal metadata and named entities, to train and evaluate restoration. Its datasets combine real corrupted documents with synthetic-corruption benchmarks targeting overall restoration and named-entity recovery.

  • Corpora: AJD and JRS are Hanja document corpora with temporal metadata and named-entity annotations from the National Institute of Korean History.Metadata includes year, month, date, personal names, organizations, and book titles.
  • Evaluation datasets: Documents containing real-world corrupted parts were strictly held out for human evaluation.
  • Damage analysis: Only 4 instances (0.7%) were misclassified among entities manually inspected in 200 randomly sampled documents, supporting the reliability of Gemini-2.5-Pro labels.Gemini-2.5-Pro predicted the named-entity type for each damaged character.
  • Data splits: The dataset contains 2.02M training documents and 10K documents each for validation and testing, with duplicates across splits removed to prevent leakage.The training corpus also served as the RAG retrieval source and was used to train the model and baselines.
  • Evaluation datasets: DRand applies synthetic corruptions at random positions for overall restoration, whereas DNE corrupts named entities to test restoration requiring external knowledge.Both evaluation datasets pair partially corrupted inputs with their ground-truth originals.

4 Proposed Methods

The proposed restoration framework combines chronological metadata, few-shot prompting, and retrieval-augmented generation to supply external knowledge for damaged-character restoration. It evaluates baseline models and develops ARI alongside a scratch-trained BERT-Res baseline.

  • Framework overview: The framework evaluates baseline restoration models, improves accuracy using metadata and RAG, and presents ARI alongside BERT-Res trained from scratch.The framework overview includes task descriptions, examples, damaged-document context, and temporal metadata as LLM inputs.
  • Prompt design: The base prompt explicitly identifies each corrupted character’s position and extracts the corresponding restored character from the output.This positional format addresses the autoregressive nature of most pretrained language models.
  • Baseline evaluation: Larger Qwen3 and Gemini-2.5 models generally achieved higher restoration accuracy, while thinking mode improved named-entity restoration but slightly reduced random-character performance.Gemini-2.5-Pro and Sonnet 4.5 surpassed other models on characters masked at random positions.
  • Metadata integration: Adding temporal identifiers such as the reigning king, year, month, and day improved reconstruction accuracy, demonstrating the importance of temporal grounding.The metadata integration was evaluated through restoration results reported in Table 3.
  • Few-shot prompting and RAG: Few-shot format exemplars reduced formatting errors, while RAG retrieved the top 20 relevant documents and used external context to improve Qwen3 32B restoration.Embedding-based retrieval and BM25 significantly outperformed random selection, although random references still improved performance over using no references.
  • Retrieval deduplication: Deduplication peaked at the 80% similarity threshold, whereas performance declined above 85% and reached its lowest point at the 100% exact-match threshold.Removing highly similar retrieved documents preserves diverse external knowledge and avoids duplicate-driven bias.

5 Experimental Results

ARI-32B achieved the strongest overall restoration performance by combining retrieval-augmented external knowledge with fine-tuning, particularly for named entities. Expert and qualitative evaluations further supported its accuracy and usefulness, while performance declined under larger temporal gaps.

  • Overall performance: Named entity restoration was substantially harder than general character restoration, but external knowledge enabled untuned LLMs to outperform BERT-Res on named entities.Qwen3 32B also outperformed its 8B variant, indicating that model scale affects performance.
  • Overall performance: ARI-32B achieved the highest overall restoration performance among the evaluated open-source and proprietary models.It combines retrieval-augmented external knowledge with fine-tuning, whereas BERT-Res relies on intra-document context and performs better on general characters than Gemini-2.5-Pro.
  • Named entity categories: ARI-32B outperformed BERT-Res and Gemini-2.5-Pro for LOC and PER entities, while Gemini-2.5-Pro performed best on POH entities.The POH advantage was attributed to Gemini-2.5-Pro’s benefit from broad knowledge of widely shared Confucian classics.
  • Expert evaluation: ARI-32B outperformed Gemini-2.5-Pro and BERT-Res in expert-evaluated Top-1 and Top-10 accuracy and achieved higher nDCG@10 and win ratio.Its ranking and candidate diversity reduced experts’ search cost when identifying correct characters.
  • Qualitative evaluation: ARI-32B was the only model to restore several person and geographic names correctly in qualitative examples, although BERT-Res sometimes exactly restored general text.These results indicate that external knowledge particularly improves restoration of proper nouns.
  • Temporal generalization: ARI-32B remained robust under moderate temporal shifts but declined as the temporal gap between retrieved and target documents widened.The temporal-shift experiment used Goryeosa documents with AJD and JRS references spanning the 14th to 19th centuries.

6 Conclusion

The paper introduces a retrieval-augmented generation framework that improves restoration of damaged historical documents, especially named entities requiring external knowledge. By reducing restoration time and cost, the framework may help reveal obscured information and broaden historical archive use.

  • Framework contribution: The proposed RAG framework enables large language models to use external knowledge when restoring damaged historical documents.This framework targets restoration cases that cannot be solved using only local document context.
  • Experimental conclusion: The framework consistently improves overall restoration performance, with especially notable gains for named entities dependent on external knowledge.The conclusion highlights named-entity recovery as the area with the strongest improvement.
  • Practical impact: By reducing restoration time and cost, the framework could reveal previously obscured information and encourage wider use of historical archives.The stated practical impact extends beyond restoration accuracy to the accessibility and use of historical records.

Limitations

The study’s constructed datasets may not fully match real-world damaged documents despite statistical alignment and expert validation. Computational limits also restricted inputs to 4,096 tokens, filtering approximately 1.7% of data, while the approach currently uses only text modality.

  • Dataset limitations: Constructed datasets may differ from the intrinsic characteristics of real-world damaged documents despite statistical alignment and human-expert validation.The datasets were aligned using characteristics such as damaged-character frequency and evaluated with actual damaged documents.
  • Computational and modality limitations: 4,096 tokens was the maximum model input length, causing approximately 1.7% of data exceeding this limit to be filtered.The limitation resulted from computational constraints.
  • Computational and modality limitations: The current approach relies solely on the text modality despite leveraging external knowledge and textual context.Future work aims to extend the context window for longer documents and improve performance.

A Training Dataset Construction and Training Settings · B Experiments

The study varied DNE–DRand mixtures within a fixed 100K-sample training set and evaluated restoration performance. ARI models were trained with specified large-scale optimization, schedule, batch, sequence-length, and token-budget settings.

  • A Training Dataset Construction and Training Settings · B Experiments: Increasing DNE proportions consistently improved named-entity restoration while reducing random-character restoration performance.Each configuration mixed DNE and DRand, with the DNE proportion varied across settings.
  • A Training Dataset Construction and Training Settings: The total training-set size was fixed at 100K samples across all mixture configurations.The mixtures comprised DNE and DRand with varying DNE proportions.
  • A Training Dataset Construction and Training Settings: ARI-32B was trained on eight HGX H200 GPUs using FSDP and the Lion-8Bit optimizer.The implementation used FSDP and Lion-8Bit for distributed training and optimization.
  • A Training Dataset Construction and Training Settings: ARI-32B used cosine scheduling with a peak learning rate of 6 × 10−6 and a warmup ratio of 0.05.Training lasted two epochs with a global batch size of 64 and a maximum sequence length of 4,096 tokens.
  • A Training Dataset Construction and Training Settings: ARI-32B training comprised 16.06 billion training tokens.The reported configuration used two epochs, a global batch size of 64, and a maximum sequence length of 4,096 tokens.
  • A Training Dataset Construction and Training Settings: ARI-8B was trained for 10 epochs with a global batch size of 1,024 and a maximum sequence length of 2,048 tokens.This configuration required approximately 12 hours of training.

B.1 Analysis of Retrieval Strategies

The analysis finds that BM25-only retrieval is optimal for Hanja restoration, indicating lexical similarity is more useful than semantic similarity. Performance generally improves with more retrieved documents but slightly declines at 24, motivating a 20-document configuration for efficiency and stability.

  • Retrieval composition: BM25-only retrieval with a 16:0 BM25-to-Gemini-Embedding ratio achieved the best harmonic mean, 35.3%.The study varied the composition of 16 retrieved documents between BM25 and Gemini-Embedding.
  • Retrieval composition: The BM25-only result suggests lexical similarity is more important than semantic similarity for character-level restoration.This interpretation follows the observed superiority of the BM25-only configuration in the retrieval-ratio comparison.
  • Retrieved-document count: Performance generally improved on DNE and DRand as more documents were retrieved, but declined slightly at 24 documents.The number-of-documents analysis examined performance trends across different retrieval counts.
  • Retrieved-document count: ARI models were configured to use 20 retrieved documents to balance computational efficiency and performance stability.This configuration was selected after observing the retrieval-count performance trends.

B.2 Top-k Accuracy during Training · C Byte-level Constrained Decoding for Top-K Candidate Selection

ARI-32B’s Top-1, Top-5, and Top-10 restoration accuracies on DRand and DNE improve throughout training. Byte-level constrained decoding addresses fragmented Hanja representations by forming valid character sequences and ranking candidates for user presentation.

  • B.2 Top-k Accuracy during Training: Top-1, Top-5, and Top-10 accuracies of ARI-32B are measured on DRand and DNE at each training step.These measurements are presented in Figure 11.
  • B.2 Top-k Accuracy during Training: ARI-32B’s restoration performance improves as training progresses.The improvement is reported for the training-step accuracy curves on DRand and DNE.
  • C Byte-level Constrained Decoding for Top-K Candidate Selection: Frequent Chinese characters are typically mapped to single tokens by the Byte-level BPE tokenizer.The tokenizer is attributed to Wang et al. (2020).
  • C Byte-level Constrained Decoding for Top-K Candidate Selection: Hanja characters are often absent from the tokenizer’s vocabulary and are decomposed into sequences of multiple byte tokens, typically 2–3.This fragmentation creates the decoding issue addressed during restoration.
  • C Byte-level Constrained Decoding for Top-K Candidate Selection: For the i-th damaged character, constrained decoding continues after an initial byte token until a valid character sequence is formed.The procedure is illustrated in Figure 12.
  • C Byte-level Constrained Decoding for Top-K Candidate Selection: Candidate scores are calculated by averaging the log probabilities of their constituent byte tokens.This scoring follows formation of a valid character sequence.
  • C Byte-level Constrained Decoding for Top-K Candidate Selection: The top-K candidate options are presented to the user after constrained decoding and score calculation.The passage specifies top-K presentation as the default candidate-selection outcome.

D Expert Evaluation · E Prompt Format

The expert evaluation uses a web interface to assess contextual coherence, factual validity, and model preference for restorations. The final prompt combines task instructions, context-specific requirements, structured input/output, reference documents, and few-shot examples.

  • D Expert Evaluation: Evaluators select all contextually coherent and factually valid candidates generated by different models for each damaged document.They may select multiple answers in the first question.
  • D Expert Evaluation: Evaluators then select the single most suitable model, directly measuring model preference.This is the second question in the expert-evaluation process.
  • E Prompt Format: The system prompt has four components: Task, Requirements, Input & Output, and Example Input & Output.These components define the restoration objective, contextual guidelines, interaction format, and few-shot examples.
  • E Prompt Format: The task requires restoring each [Dn] to exactly one original Hanja character.Each damaged token corresponds to exactly one character.
  • E Prompt Format: The requirements instruct the model to use the document’s overall context and meaning rather than treating damaged tokens in isolation.The prompt tailors restoration guidelines to the damaged document’s context.
  • E Prompt Format: Inputs include the document, metadata, and related documents, while outputs use a JSON mapping from each [Dn] to its restored Hanja character.Related documents are always present in the actual dataset, even when omitted from the concise prompt display.
  • E Prompt Format: The prompt supplies reference documents and few-shot examples spanning dated historical documents and multi-character restorations.Examples demonstrate mappings such as [D1] to 李 and [D1][D2] to 兵 and 曹.
Loading 2607.21936v1…