Source-linked AI summary
Yo-ByT5: Efficient and High-Fidelity Diacritic Restoration for Yor\`ub\'a
Ahmad Samuel Gali, Shamsuddeen Hassan Muhammad
TL;DR
Yorùbá diacritics help disambiguate words, but are often omitted, limiting downstream NLP. This paper fine-tunes byte-level ByT5-small as Yo-ByT5 and evaluates it against public diacritisers and an open-weight LLM. It matches the top subword baseline’s accuracy while using about half the parameters and shows higher input text fidelity.
Problem
Yorùbá often appears without diacritics that disambiguate words, and this plain-text form degrades downstream NLP performance.
Source: §1.1 · p. 1Method
The paper fine-tunes byte-level ByT5-small for Yorùbá ADR and evaluates it against public models under a unified benchmark protocol.
Source: §3.1 · p. 3, §6 · p. 5Results
Yo-ByT5 matches the top subword baseline’s accuracy and alters base input text in 0.84% of errors, versus 26.42% for mT5-base.
Source: §6 · p. 5, §5.3 · p. 5Takeaways & Limitations
Text fidelity differentiates models beyond DER alone, and the authors release code, outputs, and dataset patches to support reproducible comparisons.
Source: §6 · p. 5The authors identify the need for a larger, dedicated Yorùbá diacritic-restoration dataset.
Source: Limitations · p. 6
Abstract
from arXiv · showhide
Yor\`ub\'a is a widely spoken tonal language that depends on diacritics to avoid lexical ambiguity. However, it is often written without these diacritics, thereby hindering downstream Natural Language Processing (NLP) tasks. In this paper, we introduce Yo-ByT5, a byte-level Automatic Diacritic Restoration (ADR) model fine-tuned from ByT5-small. We evaluate Yo-ByT5 alongside five publicly released Yor\`ub\'a ADR models and one open-weight large language model (LLM) on the YAD benchmark under a consistent protocol. Our results demonstrate that Yo-ByT5 matches the performance of the strongest existing model, mT5-base, with a DER of 10.14% and a CER of 3.48%. Furthermore, it exhibits superior text fidelity despite using approximately half the parameter count of mT5-base. We also release our training code and model outputs, as well as call for the development of a larger, purpose-built benchmark for Yor\`ub\'a diacritic restoration.
1 Introduction
Yorùbá diacritics disambiguate words, but are often omitted, creating a challenge for downstream NLP. The paper presents byte-level Yo-ByT5 and a unified evaluation of Yorùbá diacritic restorers.
1.1 The problem
- Yorùbá diacritics distinguish lexical readings, yet casual writing often omits them, and unmarked text degrades downstream NLP performance.In one corpus, 85% of words carry diacritics, and 32% of unmarked word types have at least two valid diacritised counterparts.Source: §1.1 · p. 1
1.2 Modelling choices for the task
- Byte-level models process raw text without fixed subword vocabularies, avoiding out-of-vocabulary issues and complex preprocessing steps.The paper notes that about 66% of mT5-base parameters are allocated to vocabulary embeddings and output matrices (Xue et al., 2021, 2022).Source: §1.2 · pp. 1–2
1.3 This paper
- The paper introduces Yo-ByT5, a ByT5-small-based byte-level Yorùbá ADR model, and evaluates it with five public diacritisers and an open-weight language model.The evaluation covers CER, WER, DER, and WDER, examines decoding and text fidelity, and releases code and per-sentence outputs.Source: §1.3 · p. 2
1.4 Contributions
- The contributions include an open-weight model and training procedure, a seven-model YAD comparison, a text-fidelity analysis, released code and outputs, and a call for a larger purpose-built benchmark.The authors also release a small patch for the YAD benchmark discovered during evaluation.Source: §1.4 · p. 2
2 Related Work
Prior Yorùbá ADR work spans neural models, corpora, and benchmarks, while LLM diacritisation results vary. This paper situates Yo-ByT5 within those approaches and compares it with specialized models and an LLM.
2.1 Yorùbá diacritic restoration
- Orife (2018) framed Yorùbá ADR as sequence-to-sequence machine translation, while later work addressed weak generalisation to colloquial text, contractions, and loan words.Orife et al. (2020) built a larger multi-domain corpus and added a news test set for modern, non-literary text.Source: §2.1 · p. 2
2.2 Corpora and benchmark data
- MENYO-20k is a multi-domain Yorùbá–English corpus with complete diacritisation, while existing Nigerian-language NLP often repurposes resources rather than building new annotated datasets (Inuwa-Dutse, 2025).The YAD benchmark draws on MENYO-20k, JW300, and the Yorùbá Bible.Source: §2.2 · p. 2
2.3 Model architectures
- Yo-ByT5 fine-tunes ByT5-small, which replaces subword vocabularies with raw UTF-8 bytes, for Yorùbá ADR.The paper also describes mT5 as a multilingual T5 variant pretrained on Common Crawl spanning 101 languages.Source: §2.3 · pp. 2–3
2.4 Large language models for diacritisation
- Prior LLM ADR results vary by model and language: some outperform specialized baselines, while smaller models often hallucinate or degrade text (Nad˘a¸s and Dio¸san, 2025; Toyin et al., 2025).This study benchmarks specialized Yorùbá diacritisers, including Yo-ByT5, against a 120B-parameter open-weight LLM on YAD.Source: §2.4 · p. 3
3 Yo-ByT5
Yo-ByT5 uses byte-level ByT5-small for Yorùbá diacritic restoration and is trained on a combined Yorùbá corpus. The authors audit overlap with YAD and report no meaningful leakage.
3.1 Architecture and rationale
- The byte-level ByT5-small model restores Yorùbá tone marks and underdots, treating diacritisation as a character- and byte-level operation.The underlying pretrained model has 299,637,760 parameters.Source: §3.1 · p. 3
3.2 Training data
- Training combines 10,070 MENYO-20k sentences and 36,243 Bible segments for 46,313 sentences, with separate validation and internal test sets.The held-out sets contain 5,305 validation and 6,633 internal test sentences.Source: §3.2 · p. 3
3.3 Training procedure
- Training uses two fine-tuning phases, each running four epochs with sequences capped at 1,024 bytes; the second phase starts from the first-phase checkpoint and produces the released model.Both phases use AdamW, linear decay, weight decay 0.01, gradient clipping at 0.5, and an effective global batch size of 64; learning rates are 2e-4 then 1e-4.Source: §3.3 · p. 3
3.4 Contamination check
- The training-data audit found only two short-string overlaps with YAD dev and test, both also present in YAD’s official training split, so the authors judge leakage not meaningful.The greeting occurs in four test and 11 dev sentences; a boilerplate header occurs in eight dev sentences.Source: §3.4 · p. 3
4 Experimental Setup
The evaluation compares Yo-ByT5 with five publicly released Yorùbá diacritisers and an open-weight LLM on YAD, using deterministic decoding and multiple scoring and error-analysis measures.
4.1 Benchmark
- YAD’s 3,330-sentence test split, drawn from MENYO-20k, is the evaluation benchmark.The model comparison includes Yo-ByT5, five public Yorùbá ADR models, and gpt-oss:120b as an open-weight LLM baseline.Source: §4.1 · p. 3, §4.2 · p. 3
4.2 Models compared
- The seven-model comparison covers Yo-ByT5, five public Yorùbá diacritisers, and one open-weight LLM baseline.The listed public models are mT5-base, two omowe-T5 variants, ByT5-small menyo, and mT5-small; the LLM is evaluated with one-shot prompting.Source: §4.2 · p. 3, Table 2 · p. 4
4.3 Decoding
- Deterministic decoding supports reproducibility; greedy and beam decoding are both evaluated because weaker models can shift by up to 10.5 CER points.The two strongest models change by at most 0.12 points across decoding strategies.Source: §4.3 · p. 3
4.4 Metrics
- The evaluation reports eight metrics, including WER, CER, DER, WDER, DER-underdot, DER-tone, BLEU, and ChrF.WER and CER use edit distance, while DER measures incorrect diacritics on reference letters.Source: §4.4 · pp. 3–4
- Words are aligned on unmarked base forms before diacritic scoring, and outputs and references are Unicode-normalized; 3.56% of YAD underdots required codepoint correction.The correction patch is released with the code.Source: §4.4 · p. 4
4.5 Error typology and text fidelity
- The error typology distinguishes tone and underdot errors from lexical substitutions, deletions, and insertions.Text fidelity is measured as the share of total errors that are lexical substitutions, deletions, or insertions; lower values indicate higher fidelity.Source: §4.5 · p. 4
- Text-altering rate measures lexical substitutions, deletions, and insertions as a share of all errors; lower rates indicate higher fidelity.It is reported for all seven models using greedy decoding.Source: §4.5 · p. 5
4.6 Environment
- Evaluation runs on a single NVIDIA T4 GPU with pinned software versions.The listed versions are Python 3.12.13, torch 2.10.0+cu128, transformers 4.57.6, and sacrebleu 2.6.0.Source: §4.6 · p. 5
5 Results and Discussion
On YAD, Yo-ByT5 and mT5-base are the leading Yorùbá ADR models, with DER performance within the margin of error. Yo-ByT5 also preserves input text more reliably, while decoding strategy particularly affects weaker models.
5.1 Overall comparison
- Table 3 reports all seven models’ metrics on YAD under one decoding and scoring protocol, except for gpt-oss:120b.The table is sorted by DER.Source: §5.1 · p. 5
5.2 Yo-ByT5 and mT5-base: parity at a fraction of the size
- Yo-ByT5 and mT5-base lead the YAD comparison, with DER results within the margin of error; Yo-ByT5 uses roughly half mT5-base’s parameters.Yo-ByT5 is better on CER, while mT5-base is better on word diacritic error rate and slightly better on WER; the DER gap was tested by resampling the test set 10,000 times.Source: §5.2 · p. 5, §5.1 · p. 5
5.3 Error analysis
- Yo-ByT5 alters base input text in 0.84% of its errors, versus 26.42% for mT5-base and 59.79% for ByT5-small menyo.Tone-only errors outnumber underdot-only errors for every model, and omowe-T5 menyo’s middle-ranking DER accompanies among-the-worst CER and WER.Source: §5.3 · p. 5
5.4 Open-weight language model baseline
- The 120B-parameter LLM scores 28.42% DER versus 10.14% for Yo-ByT5, ranking third on CER and fifth on DER.Its text fidelity falls from 53.4% of sentences under 60 characters to 3.0% of those with 240 or more characters.Source: §5.4 · p. 5
5.5 Decoding strategy
- Greedy and beam decoding differ by at most 0.12 CER points for the two most accurate models, while beam search recovers up to 10.5 CER points for the weakest public models.The authors caution that fixing one decoding strategy risks misranking weaker models.Source: §5.5 · p. 5
6 Conclusion
Yo-ByT5 is a byte-level Yorùbá diacritic restoration model that matches the top subword baseline with about half its parameters and higher input-text fidelity. The analysis highlights fidelity as a distinction that DER alone misses.
6 Conclusion
- Text fidelity differentiates models in ways that DER alone does not capture.Figure 2 categorizes word errors into mark classes and text-altering classes across seven models.Source: §6 · p. 5, Figure 2 · p. 6
- The authors release code and model outputs to support future research.Source: §6 · p. 5
- Yo-ByT5 matches the top subword baseline while using about half its parameters and preserving input text with superior fidelity.The comparison is on the YAD benchmark; Table 3 covers the corrected test set and notes that gpt-oss:120b uses one-shot decoding rather than beam search.Source: §6 · p. 5, Table 3 · p. 6
Limitations
The evaluation has limits in word alignment, LLM coverage, training-register balance, and benchmark scope. The authors call for broader evaluation and a larger, purpose-built Yorùbá diacritic-restoration benchmark.
Limitations
- One-to-one word alignment can slightly affect scores when a model splits or merges a word, counting it as a substitution plus a gap.Source: Limitations · p. 6
- The LLM comparison covers one model at one size, one prompt, and one run, although outputs can vary across runs.Source: Limitations · p. 6
- About 78% of training data is biblical text, and prior work indicates that over-reliance on biblical corpora can limit generalisation to colloquial text.The evaluation uses multi-domain YAD rather than scripture; a more balanced training corpus may improve informal-text performance.Source: Limitations · p. 6
- The YAD benchmark has 3,330 sentences; the authors call for a larger benchmark focused on diacritic restoration and covering more contexts.Source: Limitations · p. 6