PaperlayerView exact revision on arXiv

Source-linked AI summary

Climbing the Ladder: Automating Data Pipelines and Reproducing Baselines in Low-Resource Settings

Donato Meoli, Eryk Wdowiak

arXiv:2610.30374v1cs.CL

TL;DR

Sicilian is included in multilingual translation models, but its available training text is noisy and non-standardized. The paper rebuilds a Sicilian NMT pipeline and adapts pretrained NLLB with staged training. The pretrained model is much stronger than the from-scratch baseline, while its multilingual adapter can degrade directions on which it was not trained.

  • Problem

    NLLB-200 includes Sicilian, but its mined non-standard web text leaves the model with a non-standard orthography, making training-data quality a central concern.

    Source: §1 · p. 2

  • Method

    The paper rebuilds journal-text extraction and alignment, then compares small from-scratch models with LoRA-adapted NLLB, including staged training on back-translations and curated literary text.

    Source: §1 · p. 2

  • Results

    NLLB-1.3B scores 29.00 BLEU on scn→en out of the box; staged training reaches 32.40 BLEU on scn→en and 21.72 BLEU on en→scn, improving both directions over the bidirectional adapter.

    Source: §8 · p. 9

  • Takeaways & Limitations

    Staged training improves both translation directions, with larger gains into Sicilian than from Sicilian.

    Source: §8 · p. 9

    On FLORES-200, the trilingual adapter lowers it→en from 34.42 to 27.14 BLEU, limiting its use as a general multilingual model.

    Source: §8 · p. 12

Abstract

from arXiv · show

The massively multilingual translation systems that now include Sicilian learned it from noisy, non-standardised web text. To obtain better translation quality, we redevelop the small neural machine translation (NMT) system that Wdowiak customized for Sicilian. We rebuild on maintained software the extraction of the parallel text from the PDFs of the bilingual journal Arba Sicula, whose sentences are aligned by a language-agnostic sentence encoder with thresholds tuned against an existing hand alignment and then filtered for the typical errors of automatic alignment. Furthermore, the tokenization and the desinence-biased subword segmentation of the original system are ported to a Perl-free implementation, and all the results are measured on a frozen test set drawn from the journal. We compare small Transformers trained from scratch with the pretrained NLLB model adapted by low-rank fine-tuning, in both directions and with Italian as a closely related bridge language, and we adapt to the pretrained model the reverse-training recipe of the original system, i.e., a stage on forward-scored back-translations of the web Sicilian followed by a stage on the curated literary text. The pretrained model is by far the better of the two, and the staged training improves the translation into Sicilian more than the translation from it. On the test set of the original system, and in the tokenized space in which it was scored, the staged model with a larger adapter and more back-translations is above the reverse-training recipe in the directions that involve English and below it between Sicilian and Italian. Yet, a multilingual adapter degrades the directions on which it was not trained, which limits its use as a general multilingual model.

1 Introduction

Sicilian translation has shifted from a lack of coverage to a problem of noisy, inconsistent training data. This work rebuilds Wdowiak’s Sicilian NMT pipeline with maintained tools and modern pretrained models.

1 Introduction

  • Noisy web text has left NLLB-200 with a non-standard Sicilian orthography, making training-data quality the central problem despite Sicilian’s inclusion in the model.Extreme orthographic diversity can also prevent consistent generation and lead to foreign words.Source: §1 · pp. 1–2
  • The paper rebuilds Wdowiak’s labor-intensive Sicilian NMT effort using maintained, mostly CPU-only tooling and seeks better quality with pretrained models.The earlier system used a small, regularized Transformer and later added staged reverse training.Source: §1 · p. 2
  • Prior work describes low-rank adapters, related-language bridges, and back-translation as approaches for low-resource translation [9] [2] [5] [10] [37] [26].The paper frames these methods in relation to the challenges described by Koehn and Knowles [11].Source: §1 · p. 2
  • A LaBSE-based aligner extracts journal sentence pairs, with thresholds tuned against hand alignments and the output filtered for remaining errors.The automatically aligned pairs are deduplicated and combined with hand-aligned sheets, which cover only a fraction of each issue’s pairs.Source: §1 · p. 2
  • The pipeline ports the original tokenization and desinence-biased segmentation to Python, adds a lemma source factor to the from-scratch baseline, and adapts NLLB-200 with LoRA.The adapted reverse-training recipe uses forward-scored web Sicilian back-translations followed by curated literary text; evaluation also reproduces the original test set and tokenized scoring space.Source: §1 · p. 2

2 Background

The background presents NMT as conditional sequence generation and the Transformer as its standard parameterization. For limited data, it motivates smaller, regularized models and subword units suited to Sicilian morphology.

2 Background

  • NMT encodes source sentence x and models target sentence y as a left-to-right conditional probability, trained by parallel-corpus log-likelihood and decoded with beam search.The decoder factorizes the probability of y into next-token probabilities conditioned on the preceding target tokens and x.Source: §2 · p. 3
  • The Transformer replaces recurrence with scaled dot-product attention computed across parallel heads [32].The attention operation uses query, key, and value matrices.Source: §2 · p. 3
  • With little data, smaller models, high dropout, and small subword vocabularies help address overfitting; BPE shares word-form statistics and supports morphologically rich Sicilian [25] [27].The paper also uses linguistic input features [24].Source: §2 · p. 3

3 Data Pipeline

The data pipeline extracts text from Arba Sicula PDFs, aligns Sicilian and English sentences, cleans the resulting pairs, and assembles web and curated training sets. A frozen literary test set supports consistent evaluation.

3.1 Extraction from PDFs

  • Digital PDFs are read with PyMuPDF, while scanned issues through AS18 and AS21 are excluded from training because their OCR text lacks vowel accents.Zero-width spaces in some born-digital issues are detected from glyph positions and removed.Source: §3.1 · p. 3

3.2 Sentence alignment

  • LaBSE sentence embeddings feed a monotonic dynamic program with 1–1, 1–2, and 2–1 alignment moves, plus penalized skips; thresholds were tuned against hand alignment.The chosen thresholds are τpage = 0.50 and τsent = 0.40; the aligner exactly reproduces 715 pairs, with recall 0.54 and precision 0.63.Source: §3.2 · p. 4

3.3 Cleaning

  • Cleaning removes layout residue and pairs with length imbalance, contents-page leaders, OCR-like characters, or excessive non-dictionary English words, leaving 15,465 deduplicated pairs in the digital archive.The length-ratio cutoff was tightened to 1.6 after pairs between 1.6 and 2.0 showed lower mean LaBSE similarity than those below 1.6.Source: §3.3 · p. 4

3.4 Datasets

  • The Web training set combines an initial journal extraction with Sicilian data curated from NLLB and WikiMatrix, while the staged-training set replaces web text with curated sources.The journal sheets and automatic alignment are complementary because hand-aligned sheets cover only 10–20% of an issue’s pairs.Source: §3.4 · pp. 4–5
  • A frozen test set of 1,000 Arba Sicula pairs provides a literary standard absent from FLORES; training data are guarded against overlap with test sentences.The guard removed 794 pairs from the second training set; without it, staged-training results would have been inflated.Source: §3.4 · p. 5

4 Standardization

Sicilian has a long-standing literary standard despite lacking one official orthography, but collected data also include dialectal and Corsican material. The paper links surface-form similarity both to alignment and to cross-language contamination risks.

4 Standardization

  • Sicilian lacks a single official orthography, yet a common literary standard has existed for several hundred years and offers a basis for refinement.Most Sicilian sentences in the NLLB dataset were written in that standard.Source: §4 · p. 5
  • About one-third of NLLB’s collected Sicilian sentences were dialectal, some were Corsican, and very few had valid translations.The authors suspect seed data inconsistent with the literary standard caused these quality issues.Source: §4 · p. 5

4.1 Why Sicilian lives in the Romance neighbourhood

  • Literary Sicilian’s surface similarity to Italian supports retrieval: LaBSE finds the correct English translation for Sicilian in 99% of cases, versus 100% for Italian.Among non-English nearest translations, Italian is selected for Sicilian 49% of the time, compared with 21% for French.Source: §4.1 · p. 6
  • Surface-form cues create an asymmetry: Italianate Sicilian can retrieve the right English sentence, while Corsican-like seed data can lead to wrong Sicilian generation.The authors attribute this to multilingual models encoding script and surface form more than deeper linguistic structure [33].Source: §4.1 · p. 6
  • Literary prose aligns more readily than dialect poetry: mean cosine is 0.76 for prose and 0.64 for poetry, with 91% and 80% of pairs above τsent = 0.4, respectively.Merging up to three poetry verses raises mean cosine by about 0.11, but this correction is not part of the production aligner.Source: §4.1 · p. 6
  • Forward-scoring English and Italian back-translations of Sicilian helps select sentences written in standard Sicilian.This yields 549,486 English-to-Sicilian and 689,840 Italian-to-Sicilian simulated back-translations.Source: §4.1 · p. 6

5 Tokenization and Subword Segmentation

The section ports the Sicilian tokenizer and desinence-biased segmentation to Python, then evaluates normalization choices for NLLB. It adopts light “std” normalization for later experiments.

5 Tokenization and Subword Segmentation

  • BLEU is 30.67 on raw text, 30.79 with “std” and 30.61 with “full”; “std” is adopted despite its 0.1 BLEU edge being within repeated-run noise.The corresponding chrF scores are 56.31, 56.49 and 56.36, respectively; “full” is no better than raw text.Source: §5 · p. 7
  • The Napizia tokenizer ASCII-folds accents and uncontracts Sicilian forms; joint scn–en BPE is biased toward textbook desinences to keep morphological endings as units, a combination called “lever B.”The desinences are appended to the Sicilian side when learning merges; Wdowiak’s example “Carinisi are dogs!” becomes “car@@ in@@ isi are dogs !”.Source: §5 · p. 7
  • The from-scratch tokenizer is reimplemented in Python, and its output matches the original Perl version except for added dash and ellipsis normalization.The normalization rules and the two normalization levels (“std” and “full”) are described in the passage.Source: §5 · pp. 6–7

6 Models

The paper compares a small from-scratch Transformer with LoRA-adapted NLLB-200, including a staged recipe that trains on filtered back-translations before curated literary text.

6.1 From-scratch baseline

  • The from-scratch baseline is a roughly 6.7M-parameter Transformer trained on CPU; lever B and a lemma source factor, lever D, are added in sequence.The lemma factor is summed into the source embedding; its small lemma vocabulary generalizes across inflected forms, with unknown lemmas assigned to missing forms.Source: §6.1 · p. 7

6.2 Pretrained model

  • NLLB-200’s 1.3B model is adapted with LoRA by learning low-rank updates to frozen attention projections, with adapters trained for Sicilian–English or four Sicilian–English/Italian directions.The bidirectional adapter is also used to translate unaligned English sentences into Sicilian; a further Sicilian-to-English adapter is trained on selected synthetic pairs and web data.Source: §6.2 · pp. 7–8

6.3 Staged training

  • Staged training selects high-scoring, filtered back-translations for a second stage, then fine-tunes the same adapter on curated text across Sicilian–English and Sicilian–Italian directions.The second stage uses the best 100,000 pairs per file where available; the final curated-text stage runs for two epochs.Source: §6.3 · p. 8
  • A curriculum variant prepends one epoch on the 200,000 worst-scoring pairs per file, then uses the same two stages as the base recipe.The coarse stage includes dialectal, Italian and Corsican text that the later stages filter out.Source: §6.3 · p. 8

7 Evaluation

Evaluation uses a frozen literary test set and sacreBLEU, reporting BLEU and chrF. The paper also scores NLLB on the original system’s hand-selected test lines in the original tokenizer’s space.

7 Evaluation

  • Sockeye systems are scored in tokenized space, while NLLB systems are scored on raw text with Sicilian normalized as in training, except for raw-input rows.The source-copy floor is 5.18 BLEU in tokenized space and 5.44 on raw text, so the two table groups are not strictly comparable.Source: §7 · pp. 8–9
  • Table 2 reports BLEU/chrF results on the frozen literary test set; its upper rows use web-set training, while staged rows use back-translations and curated-set training.Source: §7 · p. 8, Table 2 · p. 9
  • NLLB is also evaluated on 121 hand-selected trilingual lines from AS38 and AS39, with BLEU computed in the original system’s Napizia-tokenized space.The paper reports results both on all lines and on those absent from each model’s training data.Source: §7 · p. 9

8 Results

The staged NLLB model matches or exceeds the reverse-training system in most reported directions, while the final ablation variant’s results vary by direction. NLLB substantially outperforms the reproduced from-scratch baseline, but the multilingual adapter reduces performance on directions it did not train.

Results on the frozen test set

  • 29.00 BLEU on scn→en out of the box rises to 31.27 with the bidirectional web adapter, while en→scn rises from 9.90 to 19.19.Adding 7,498 back-translated pairs brings scn→en to 31.60 BLEU.Source: §8 · p. 9

Staged training

  • 32.40 BLEU on scn→en and 21.72 on en→scn improve over the bidirectional adapter by 1.13 and 2.53 BLEU, respectively.The staged run changes both the training data and back-translation use, so their individual effects are not isolated.Source: §8 · pp. 9–10

Ablation of the staged training

  • 49.25 versus 48.36 BLEU into Sicilian and 48.77 versus 48.00 into Italian are the final ablation model’s only gains over reverse training; it loses 0.7–1.8 BLEU in the other directions.The final model is selected as best on validation, but is below reverse training in both Sicilian–Italian directions.Source: §8 · pp. 10–11, Table 3 · p. 11

Italian directions

  • 43.47 BLEU on scn→it and 26.97 on it→scn show strong Italian-pair performance, but the multilingual adapter lowers scn↔en scores to 30.75 and 17.49 BLEU.The bidirectional adapter scores 31.27 and 19.19 BLEU on those English directions.Source: Table 4 · p. 11, §8 · p. 11

Forgetting of the untrained directions

  • 34.42 to 27.14 BLEU on it→en and 29.42 to 28.77 on en→it show that the trilingual adapter also degrades directions on which it was not trained.Effects on other NLLB language directions are not quantified against the base model.Source: Table 4 · p. 11, §8 · p. 12

Relation to the state of the art

  • The comparison with published scores is not fully even: the earlier systems selected checkpoints on the test set, while the staged runs were scored there only after fixed-epoch training.The earlier system’s training data excluded the test lines but included other text from the same articles; the staged runs included test lines in training.Source: §8 · p. 12
  • 45.07 versus 45.1 BLEU into Sicilian and 61.49 versus 61.4 from Italian into Sicilian put the staged model on par with reverse training; it exceeds that system by 7.8 BLEU from Sicilian into English and 2.0 and 14.3 between English and Italian, but trails by 3.06 from Sicilian into Italian.These comparisons use the second author’s test lines absent from the staged models’ training data.Source: §8 · p. 12, Table 5 · p. 13

Reproduction of the original baseline

  • 15.97 versus 25.1 BLEU on en→scn and 16.65 versus 29.1 on scn→en leave the reproduced baseline well below the published scores.The reproduction used scratch training and validation-based stopping, while the published system used a warm start and stopped on the test set; its runs had not converged at the 20-epoch cap.Source: §8 · p. 13, Table 6 · p. 13

9 Conclusion and Future Work

The rebuilt staged NLLB system outperforms the from-scratch baseline on the paper’s test set, while results on the original system’s test set favor it over reverse training in directions involving English but not between Sicilian and Italian. Future work includes enlarging and refining the in-domain data and extending the pipeline to related varieties.

9 Conclusion and Future Work

  • 36.01 BLEU on scn→en and 26.34 on en→scn were reached by the best staged model on the paper’s test set.The model used three times the back-translations, a larger adapter, longer sequences, and WikiMatrix it–scn pairs.Source: §9 · p. 14
  • On the second author’s test set, the staged model scored 49.25 BLEU into Sicilian and 59.34 from it, 4.2 and 10.7 BLEU above his reverse-training system.It remained below that system only between Sicilian and Italian.Source: §9 · p. 14
  • Retraining the original baseline with its configuration did not reach its published scores, while adding the new alignment to its sheets raised it by 4.0 to 6.5 BLEU.Source: §9 · p. 14
  • Future work includes merging short poetry verses, collecting unaligned Sicilian, quality-weighting journal text, and testing alternative back-translation sources.The paper also proposes limiting forgetting by mixing in a large multi-parallel corpus and applying the pipeline to Calabrese and Salentino.Source: §9 · p. 14
  • The pipeline and evaluation code are public, but copyright prevents release of the journal text, and the hand-aligned sheets are not public.Source: §9 · p. 14
Loading 2610.30374v1…