PaperlayerView exact revision on arXiv

Source-linked AI summary

Mobilizing the Archive: Bitext Extraction at Scale from Archival Documents for Extremely Low-resource Languages

Chutong Meng, Alec J. Bateman, Antonios Anastasopoulos

arXiv:2610.40233v1cs.CL

TL;DR

Many languages lack sufficient web bitext, while linguistic archives contain potential parallel data but are heterogeneous and difficult to process at scale. The paper combines OCR, a vision language model, and postprocessing to extract examples from archival PDFs; extracted examples improve few-shot translation, with reported average gains of 15.78 and 6.94 chrF2++ into low- and high-resource languages.

  • Problem

    Most of the world’s languages lack sufficient web data and parallel sentences, while heterogeneous archival resources are challenging to process at scale.

    Source: §1 · p. 1

  • Method

    The pipeline uses OCR to recover text, a VLM to identify bitext from visual layout and OCR text, then cleans and filters the alignments.

    Source: §1 · p. 1, §4 · p. 3

  • Results

    15.78 and 6.94 chrF2++ average gains were observed for translation into LRLs and HRLs, respectively; extracted bitext improved Qwen performance in all languages except dgo.

    Source: §6.1 · p. 6

  • Takeaways & Limitations

    Across more than 20 languages, few-shot LLM translation with extracted archival bitext substantially improves low-resource MT over zero-shot translation.

    Source: §7 · p. 8

    Orthography mismatches remain challenging: language-specific normalization requires manual rules, and OCR can struggle with diacritics and visually similar characters.

    Source: Limitations · p. 9

Abstract

from arXiv · show

While mining parallel sentences from web pages has proven effective for low-resource machine translation, the majority of the world's languages have little to no web presence. What they do have, though, are resources that are the result of centuries of efforts by linguists, such as grammars, dictionaries, and other documentation materials. We propose extracting translation examples from such linguistic resources, which offer parallel data for languages under-represented online and could potentially scale to thousands more languages. We develop a pipeline to achieve this, leveraging an Optical Character Recognition (OCR) model as well as a vision language model (VLM), to ensure identified content to be aligned with visual layout information. We demonstrate the utility of the extracted bitexts for bootstrapping translation systems across 20 languages, both by training machine translation models and by applying in-context learning for large language models. For in-context learning, we observe improvements of +15.78 and +6.94 chrF2++ over zero-shot translation for into low- and high-resource languages, respectively.

1 Introduction

Most languages lack the web data and parallel sentences commonly used to train machine translation models, while archival linguistic resources offer a much broader but smaller source of bitext. The paper develops an OCR- and VLM-based pipeline for extracting examples from heterogeneous PDFs and applies it across 20 languages.

1 Introduction

  • Most languages lack sufficient web data and parallel sentences, while archival collections could extend bitext coverage from around 200 languages to thousands.The DReaM corpus covers more than 4,000 languages, though resources often contain only a few thousand parallel sentences.Source: §1 · p. 1
  • Extracting structured data at scale is difficult because prior approaches depend on machine-readable resources or human interpretation of varied textual and visual cues.These approaches are difficult to generalize to previously unseen resources with diverse layouts.Source: §1 · p. 1
  • The pipeline applies OCR to recover text, then uses a vision language model to identify parallel sentences from OCR output and page images.The VLM stage addresses layout signals such as interlinear glossed text, adjacent translations, and parallel columns.Source: §1 · pp. 1–2
  • 20 languages yielded an average of 2,578 extracted bitext examples per language.The VLM uses both OCR output and page images to identify parallel sentences, because visual structure can signal parallelism.Source: §1 · p. 2

2 Related Work

Prior work has used linguistic resources and different extraction procedures to support translation, but methods often rely on structured inputs or manual effort. This paper focuses on constructing usable bitext from heterogeneous archival resources with minimal manual intervention.

2.1 Linguistic Resources Based MT

  • Linguistic resources have been used with LLMs to translate unseen languages, but converting heterogeneous, poorly structured materials into usable data remains a challenge.Tanzer et al. (2024) is cited as prior work showing grammar books and other linguistic resources can improve MT for unseen languages.Source: §2.1 · p. 2
  • Prior resource-based MT studies use grammar books, dictionaries, OCR, formatting cues, or manual extraction, with some extraction procedures unspecified.Examples include Tanzer et al. (2024), Aycock et al. (2025), Zhang et al. (2024), Hus and Anastasopoulos (2024), Merx et al. (2024), Zhang et al. (2025), and Pei et al. (2025).Source: §2.1 · p. 2
  • Unlike work focused on resource types and prompting strategies, this paper targets resource construction and extraction from archival materials with minimal manual intervention.Setiawan et al. (2026b) combine OCR with an LLM for PDFs but evaluate one language; Setiawan et al. (2026a) use OCR and a VLM for dictionary entries, not bitext or downstream MT.Source: §2.1 · p. 2

3 Languages and Status

The study selects extremely under-resourced languages with existing test sets and separates languages by NLLB support to approximate differing web-presence and pretraining conditions.

3 Languages and Status

  • The study selects extremely under-resourced languages previously studied so the method can be evaluated with existing test sets.Table 1 summarizes the languages and their resource status.Source: §3 · p. 3
  • Languages supported by NLLB are treated as having at least some online monolingual text and as likely to have appeared in frontier LLM pretraining.The distinction is used to separate languages with and without web presence.Source: §3 · p. 3
  • Languages without NLLB support approximate a setting where a model translates a previously unseen language using only archival translation examples.Source: §3 · p. 3

4 Method

The method extracts and filters bitext from archival PDFs using OCR, visual-language identification, and postprocessing. It also evaluates OCR models and the quality of extracted sentence pairs.

4 Method

  • The pipeline applies multilingual OCR, uses a VLM to identify bitext from visual cues, then cleans and filters the resulting alignments.Source: §4 · p. 3

4.1 OCR

  • dots.ocr performs best and supports the broadest range of scripts, but retrying when output exceeds the length limit reduces its CER from 7.88% to 5.66%.The OCR models are evaluated on synthetic multilingual PDFs rendered from FLORES devtest sentences, with macro-averaged CER across languages.Source: §4.1 · pp. 3–4

4.2 Vision Language Model

  • The VLM takes a page image and OCR text as input, identifying candidate bitext pairs from visual cues while using specialized OCR for text recovery.Books are assumed to primarily contain an LRL described in an HRL; language identification and sentence counts are used to select matching books.Source: §4.2 · p. 4

4.3 Postprocessing

  • Raw extracted pairs can contain annotations, word-level alignments, and incorrect matches, so the pipeline applies language-agnostic cleaning and language-specific orthographic normalization.The authors note that conventional bitext-mining pipelines also require extensive cleaning and filtering.Source: §4.3 · p. 4

4.3.1 Language-agnostic Cleaning and Filtering

  • Language-agnostic filtering removes IGT annotation symbols and pairs with extreme length mismatch or source-target similarity above 0.9, and checks language identification.Source: §4.3.1 · p. 4

4.3.2 Language-specific Orthography Normalization

  • Language-specific normalization reconciles archival orthographies with modern evaluation conventions through diacritic removal, transliteration, or orthography conversion.Some normalization remains manual; OCR can fail on combining low lines, which are not restored for git.Source: §4.3.2 · pp. 4–5
  • Manual verification may still be needed to reconcile archival orthographies with current community practices, rather than to fix bitext extraction itself.Source: §4.3.2 · p. 5

4.4 Intrinsic Evaluation

  • 3.098 was the average QE score for extracted bitext, compared with 3.723 for the test set.A score of 3 indicates sentences are generally equivalent in meaning, with only minor differences permitted under the XSTS scale predicted by BLASER 2.0-QE.Source: §4.4 · p. 5, Table 3 · p. 5

5 MT Experiments for Extrinsic Evaluation

The experiments assess whether extracted archival bitext helps both few-shot LLM translation and MT fine-tuning. They evaluate 20 languages using language-appropriate test sets and compare against zero-shot or out-of-the-box baselines.

5.1 Training and Evaluation Data

  • Evaluation covers 20 languages, using FLORES for 11, SIGMORPHON shared-task data for 6, and previously curated test sentences for 3.BOUQuET is an additional evaluation set for the nine supported languages.Source: §5.1 · p. 5
  • To prevent train–test overlap, extracted pairs are removed when either side has a Levenshtein similarity ratio above 0.9 with any development or test sentence.SIGMORPHON test sentences are detokenized with Moses; other test sets remain unchanged.Source: §5.1 · p. 5

5.2 Experiment Setup

  • Qwen few-shot examples are retrieved by ranking candidate sentences against the input with chrF2++; the best-scoring pairs are selected.An ablation found that 512 examples performed best.Source: §5.2 · p. 6
  • The fine-tuning setup uses nllb-200-distilled-1.3B with LoRA adapters and trains one multilingual model on all extracted data using temperature-based sampling.The authors cite the tiny dataset sizes as the reason for using a single multilingual model.Source: §5.2 · p. 6
  • Few-shot Qwen translation is compared with zero-shot translation, while nllb fine-tuning is compared with its out-of-the-box model where supported.For unsupported low-resource-to-high-resource translation, proxy source-language codes are used.Source: §5.2 · p. 6
  • Scores are reported as spBLEU and chrF2++ after lowercasing and removing punctuation except apostrophes; paired bootstrap resampling tests significance.spBLEU uses the flores200 tokenizer.Source: §5.2 · p. 6

6 Results and Analysis

Extracted bitext improves few-shot Qwen translation across both low- and high-resource target languages, while traditional alignment methods struggle with archival text. The analyses also compare book examples with in-domain examples and test the pipeline on Creole languages.

6.1 Main Results

  • Extracted bitext raises Qwen chrF2++ by 15.78 into LRLs and 6.94 into HRLs on average, except for dgo.The dgo result is likely related to its automatically derived orthography normalization rules.Source: §6.1 · p. 6, Table 5 · p. 6, Table 6 · p. 7
  • Fine-tuning nllb does not improve translation into LRLs, likely because of limited training data, but improves translation into HRLs by 3.59 chrF2++ on average.The authors conclude that the extracted bitext provides useful supervision even at this small scale.Source: §6.1 · p. 6

6.2 A Failed Attempt: Bitext Extraction with Existing Sentence Alignment Methods

  • Embedding-based alignment remains constrained by high xSIM error rates for unseen languages, including 34% for Mambai and 52% for Kalamang, despite SONAR performing best overall.The alignment-method evaluation uses mgm-eng, which has a relatively low SONAR xSIM error rate; performance is expected to be worse for higher-error languages.Source: §6.2 · p. 7, Table 7 · p. 7
  • Traditional sentence alignment performs poorly on OCR text: preprocessing errors include failed sentence segmentation and language identification, while VLM alignment uses page images to infer boundaries and correspondences.On 182 manually aligned Mambae sentence pairs, VLM achieved the highest F1; it can miss sentence-level alignments on pages dominated by word-level alignments.Source: Table 8 · p. 7, §6.2 · p. 8

6.3 Using In-domain Parallel Sentences for Prompting

  • In-domain examples outperform book examples for most languages, raising average chrF2++ from 30.40 to 38.81 into LRLs and by 4.09 into HRLs.Book examples perform better for git, mgm, mnc, and ntu; the authors suggest more available book examples and possible test-set overlap as explanations.Source: §6.3 · p. 8, Table 9 · p. 8

6.4 Broader Applicability: CreoleMT

  • On 27 Kreyòl-MT Creole languages, extracted bitext raises average chrF2++ from 16.87 to 35.71 into Creole and from 34.54 to 48.79 into English.This evaluation uses language-agnostic cleaning and filtering, without language-specific orthographic normalization.Source: §6.4 · p. 8

7 Conclusion

The paper introduces a VLM-based pipeline that extracts parallel sentences from archival linguistic resources by using their visual structure. Across more than 20 languages, extracted bitext improves few-shot LLM translation over zero-shot translation.

7 Conclusion

  • The proposed pipeline extracts parallel sentences from archival linguistic resources by leveraging their visual structure.The approach targets data construction from archival resources with minimal manual intervention.Source: §7 · p. 8
  • Across more than 20 languages, few-shot translation with an LLM using extracted bitext substantially improves low-resource MT over zero-shot translation.The conclusion presents archival linguistic resources as a source of parallel data.Source: §7 · p. 8
  • The conclusion frames the extracted examples as useful for bootstrapping translation systems in languages underrepresented online.The paper demonstrates this through both LLM in-context learning and MT-model training in its experiments.Source: §7 · p. 8

Limitations

Archival resources pose orthographic and content-selection constraints for extracting useful parallel data. The related-work discussion explains why conventional bitext mining and sentence alignment may not fit grammar books or unseen, extremely low-resource languages.

Limitations

  • Orthography mismatches remain difficult: normalization still requires manually designed language-specific rules, and OCR can struggle with diacritics and visually similar characters encoded as different Unicode code points.The paper gives Latin and Cyrillic ‘a’ as an example of visually similar characters with different encodings.Source: Limitations · p. 9
  • Some archival books use historical spellings or contain little parallel text, so careful resource selection is important for relevant, high-quality MT examples.The collection includes books focused mainly on phonology or grammatical description.Source: Limitations · p. 9

A Related Work on Bitext Alignment

  • Grammar books combine examples with descriptive text, making document-level sentence alignment unsuitable; bitext mining is more applicable but multilingual representations may generalize poorly to unseen, extremely low-resource languages.Bitext mining typically depends on multilingual sentence embeddings, while sentence alignment assumes parallel documents.Source: A Related Work on Bitext Alignment · pp. 12–13

B Script-level OCR Results

The section reports OCR retry gains, describes prompt and orthography procedures, and tests whether normalizing special characters changes few-shot MT results.

B Script-level OCR Results

  • Retrying decoding at progressively higher temperatures after length-limit repetition substantially improves Hang, Hant, and Hebr script recognition.The retry temperatures are 0.3, 0.5, and 0.8, following default-temperature decoding at 0.1.Source: B Script-level OCR Results · p. 13

C Prompts

  • The bitext-extraction and few-shot experiments use separate instruction, system, and user prompt templates.Figure 4 shows the extraction instruction; Figures 5 and 6 show the few-shot system and user prompts.Source: C Prompts · p. 13

D Orthography Normalization for Main Results

D.1 Normalizing Special Letters for both Example and Test Sentences

  • Removing diacritics and the Combining Low Line character from both examples and test sentences yields very similar results for the affected languages.The authors infer that errors involving these characters have only limited impact on MT performance in these experiments.Source: §D.1 · p. 13

E Applying Existing Alignment Methods

The section compares alignment approaches for grammar-book text and reports intrinsic and few-shot evaluations. The reported gains vary with the model and number of in-context examples.

E Applying Existing Alignment Methods

  • The baseline pipeline splits OCR text into language-specific documents, then applies sentence mining, Vecalign, or dictionary-and-length-based Hunalign alignment.Hunalign uses dictionary entries that may not be readily available in practice; Vecalign is run page by page with only 1-to-1 alignments.Source: E Applying Existing Alignment Methods · pp. 13–14

F.1 Intrinsic Evaluation for Unseen Languages

  • BLASER 2.0-QE shows a relatively small quality gap between extracted and test examples for unsupported languages, but the authors treat these scores only as a rough reference.They exclude these scores as a primary evaluation metric because the QE model may be unreliable for unsupported languages.Source: §F.1 · p. 14

F.2 Few-shot Experiments using Llama

  • Llama-3.1-8B few-shot translation improves most directions, with average gains of 7.20 chrF2++ into LRLs and 2.33 chrF2++ into HRLs.The authors note benefits even with this considerably smaller, less powerful model.Source: §F.2 · p. 14

F.3 Optimize the Number of Bitext Examples

  • Llama-3.1-8B performance plateaus at approximately 8 examples, while Qwen3.6-27B continues improving through the tested range up to 512.The authors use 8-shot prompting for Llama and 512-shot prompting for Qwen; beyond 512 examples, gains diminish while memory and inference costs become prohibitive.Source: §F.3 · p. 14

F.4 spBLEU Scores for the Main Results

These appendix results report few-shot and fine-tuning outcomes across translation settings, followed by training and computation details. The reported few-shot gains coexist with a remaining gap to gold in-domain examples on BOUQuET.

G Details of NLLB fine-tuning

  • NLLB fine-tuning uses rank-16 LoRA adapters on all linear modules, adds language tokens for languages not originally supported, and trains for 7,500 steps.The setup uses a batch size of 32, eight gradient-accumulation steps, and a 1 × 10^-4 learning rate; the final checkpoint is evaluated.Source: G Details of NLLB fine-tuning · pp. 15–16
  • Temperature-based sampling upsamples lower-resource languages, with T set to 5.The sampling probability is defined for each language using its number of training sentence pairs and the temperature.Source: G Details of NLLB fine-tuning · p. 15

H Computation Resources

  • All experiments run on one A100-80GB GPU; OCR takes minutes to hours per book, VLM extraction about 20 minutes, and MT 30 minutes to 36 hours per direction.The time ranges depend on pages, script, input sentences, and number of bitext examples.Source: H Computation Resources · p. 16

F.4 spBLEU Scores for the Main Results

  • Few-shot MT improves over zero-shot by 8.96 spBLEU into LRLs and 5.07 spBLEU into HRLs; fine-tuning nllb hurts originally supported languages but helps HRL translation for newly supported languages.Source: §F.4 · p. 14, Table 14 · p. 16, Table 15 · p. 18

F.5 Evaluation Results on BOUQuET

  • On BOUQuET, book examples improve Qwen3.6-27B by 9.33 chrF2++ into LRLs and 3.31 chrF2++ into HRLs, while a gap remains to the gold-example topline.That remaining gap is 10.02 chrF2++ into LRLs and 3.34 chrF2++ into HRLs.Source: §F.5 · p. 15, Table 16 · p. 19

F.6 Evaluation Results on Kreyòl-MT

Related papers

Loading 2610.40233v1…