The Ongiini-Eval-OW Benchmark: A Concept Paper for the Planned Benchmarking of Machine Translation and Large Language Models on Oshindonga and Oshikwanyama

1 Introduction

• The authors report no public machine-translation benchmark for Oshiwambo and no coverage of either standardized dialect in major commercial services, open multilingual models, or Masakhane checkpoints.Frontier LLMs produce Oshiwambo-like output, but its quality has not been benchmarked.Source: §1 · p. 1
• Ongiini-Eval-OW is a planned 600-item English–Oshindonga and English–Oshikwanyama benchmark for machine translation and large-language-model output.Its first public release is targeted for Q4 2026; the 600-item v1.0 build is in production.Source: §1 · pp. 1–2
• Two independent translators will provide native-speaker references for all 600 source items, with 30 items translated by both to measure inter-translator agreement.The overlap is intended to publish agreement as a variance measurement rather than assume uniform reference quality.Source: §1 · p. 2
• Eleven phenomenon tags target known low-resource MT failure modes, with at least 30 items per tag for roughly nine items per tag in the 180-item blind split.The paper presents this as a working minimum for slice-level comparison.Source: §1 · p. 2
• Roughly 30% of source items derive from production WhatsApp queries, paraphrased and scrubbed of personally identifiable information.The dataset is intended to reflect questions Namibian users ask digital assistants, rather than an encyclopaedic projection.Source: §1 · p. 2
• The scoring protocol combines chrF++, BLEU, and COMET-22 with a 50-item human-evaluation round and published Krippendorff’s α.Automated metrics are computed over a published JSONL submission format; human ratings cover adequacy and fluency.Source: §1 · p. 2

2.1 The languages

• Oshiwambo comprises eight mutually intelligible Bantu dialects; Oshindonga and Oshikwanyama are the two with standardized written forms.Oshikwanyama is more numerous in both Namibia and Angola; the 2011 Namibian census recorded it in 21.2% of households and Oshindonga in 15.1%.Source: §2.1 · p. 2

2.2 The coverage gap

• An audit found no Oshiwambo coverage in major translation systems, and whether frontier LLM output is accurate, fluent, register-appropriate, or useful remains an empirical question.The audit used direct inspection of open-system artifacts and public language lists for closed services.Source: §2.2 · p. 2

2.3 Why this matters

• Oshiwambo is spoken at home by roughly half of Namibian households, and the paper frames the lack of a translation layer as a digital-inclusion concern.The paper cites Joshi et al. (2020), who classify Oshindonga as “Left-Behind” in their resource-tier taxonomy.Source: §2.3 · pp. 2–3

2.4 What changes when a benchmark exists

• The authors describe a benchmark as a measurable baseline, a pressure point on vendors’ opaque coverage decisions, and a repeatable methodology for other low-resource language communities.They also identify it as a scaffold for follow-on work on adjacent Namibian languages.Source: §2.4 · p. 3

3.1 Composition

• The planned v1.0 dataset comprises 600 English source items, each paired with reference translations in both Oshindonga and Oshikwanyama, compared with an existing internal 423-item v0.1 build.The revision addresses v0.1’s thin per-phenomenon slices and over-dominance of chat/conversational register; v1.0 gives every phenomenon 30+ items and bulks up formal, community, and religious registers.Source: §3.1 · p. 3, §3 · p. 3

3.2 Stratification

• Eleven phenomenon tags each have at least 30 items, targeting low-resource MT failure modes and roughly nine examples per tag in the 180-item blind split.Items may carry multiple phenomenon tags.Source: §3.2 · p. 4, Table 4 · p. 5

3.3 Splits

• The deterministic 30% blind split contains 180 items stratified by phenomenon, length, and domain; the remaining 420-item development set supports iteration.All systems are evaluated on every item; the split determines which scores are reported as headline, and both subsets are planned for publication.Source: §3.3 · p. 4

3.5 Safety and provenance

• Mining safeguards include removing personal information before review, paraphrasing mined queries, translating from English without machine proposals, and capping contributions at three items per user.The published provenance label is mined_paraphrased, rather than raw real-mined text.Source: §3.1 · p. 3, §3.5 · p. 4

3.6 Inter-translator agreement

• Two independent translators will each translate a designated 30-item overlap set per dialect to measure agreement using chrF++ similarity and qualitative disagreement coding.Both translators’ references are treated as valid; the set exposes reference variance rather than adjudicating one translation as correct.Source: §3.6 · p. 5

4.1 Launch model matrix

• The launch matrix spans American, European, and Chinese systems, including frontier proprietary models and open-weight models run locally where possible.The authors distinguish systems they can access and run from those invited through external submissions.Source: §4.1 · pp. 5–6

Meta Muse Spark

• Muse Spark’s apparently stronger Oshiwambo output is an anecdotal observation from a handful of prompts, without metrics or a held-out set.The authors commit to evaluating Muse Spark 1.1 at launch if API access is available, otherwise via submission.Source: Meta Muse Spark · p. 6

4.2 Prompting protocol

• LLMs use a single public zero-shot prompt without in-context examples, while MT systems use native APIs or a related-Bantu fallback code where needed.The stated rationale for zero-shot prompting is fairness across systems with uneven few-shot support and alignment with end-user use.Source: §4.2 · pp. 6–7

4.3 Automated metrics

• chrF++ is the primary metric, BLEU is secondary for legacy comparability, and COMET-22 is tertiary and only a suggestive source-conditioned signal because its training data excludes Oshiwambo.Scores are computed by a public script and reported overall and across phenomenon, length, domain, and split slices.Source: §4.3 · p. 7

4.4 Human evaluation

• A 50-item stratified human-evaluation sample will be rated for adequacy and fluency by approximately 5–10 native speakers per dialect per item.Volunteer raters are self-selected toward Ongiini supporters; the design discloses this bias and uses identity blinding, randomized order, calibration items, and possible external raters.Source: §4.4 · p. 7

4.5 Reporting matrix and statistical power

• Headline claims use chrF++ on the 180-item blind split; per-phenomenon results are interpretive rather than leaderboard rankings.The authors say differences below roughly three chrF++ points may be statistically indistinguishable and will publish confidence intervals; per-phenomenon slices typically contain about nine items.Source: §4.5 · p. 8

Dataset and scripts

• The dataset is targeted for publication on Hugging Face under CC-BY-4.0, while the evaluation scripts are MIT-licensed.The repository scaffolding already matches the intended dataset target.Source: §5 · p. 8

Scripts

• The listed mining, curation, build, translation-fill, and export scripts have run end-to-end on the internal v0.1 build and will be adapted for v1.0.The scripts are intended to support the dataset pipeline.Source: §5 · p. 8

Baselines

• Claude Opus 4.7 and Gemma 4 26B outputs on v0.1 validate the pipeline; v1.0 baselines will be recomputed with Claude Opus 5 and Gemma 4 26B before release.The recomputed baselines are planned as the first leaderboard rows.Source: §5 · p. 8

Versioning and citation

• The first public release will be pinned as v1.0, schema-compatible updates will increment minor versions, and schema-breaking changes will increment major versions.All releases are to be archived, and a Zenodo DOI is planned at first formal publication.Source: §5 · p. 8

6.1 Submit a model

• Contributors can run systems on all 600 English sources and submit JSONL outputs with a model card for validation, scoring, and attributed leaderboard inclusion.Build-phase submissions through Q4 2026 are planned for the launch leaderboard; eligible submissions before Q2 2027 may receive an offer of co-authorship on the eventual paper.Source: §6.1 · p. 8

6.2 Review the reference translations

• Native speakers can review translations, submit alternatives or disagreements, and optionally provide dialect and regional demographics for the published methodology.Accepted alternatives are planned for integration into a minor dataset version, with reviewers credited in the acknowledgments.Source: §6.2 · p. 9

6.3 Propose phenomena and items

• Contributors can propose tagged items for under-represented areas including Namibian-Afrikaans code-switching, proverbs, tone-affecting honorifics, and discourse markers.Accepted items are planned for a future dataset version, with contributors credited in release notes.Source: §6.3 · p. 9

6.4 Roadmap

• The paper points to Table 6 for the roadmap from this concept paper to the first peer-reviewed publication.The passage does not specify the roadmap steps.Source: §6.4 · p. 9

7 Conclusion

• The planned benchmark covers English↔Oshindonga and English↔Oshikwanyama machine translation and large-language-model output.Its design includes native-speaker references from two independent translators, phenomenon-tagged stratification, and an inter-translator agreement set.Source: §7 · p. 9
• A reproducible protocol will score systems with chrF++, BLEU, and COMET-22, supplemented by a focused human-evaluation round.The conclusion identifies the human-evaluation round as focused but does not specify its size here.Source: §7 · p. 9
• The dataset is in production and targeted for first public release in Q4 2026, with an updated paper planned alongside it.The authors invite model submissions, reference review, and phenomenon-coverage contributions during the build phase.Source: §7 · p. 9

Limitations

• Two translators cover the dataset, with a 30-item overlap for agreement reporting; this is weaker than a panel of three or more raters.An open translation-review contribution path is the stated mitigation.Source: Limitations · p. 9
• The protocol omits back-translation validation, which the authors identify as a more rigorous approach that future versions may add on a sample.Budget instead went toward broader phenomenon coverage and the inter-translator agreement set.Source: Limitations · p. 9
• Conversational chat is 38% of the dataset; long-form, literary, and scientific registers are under-represented, and the benchmark targets sentences and short-paragraph cohesion.The longest items are 19+-word multi-sentence paragraphs; document-level coherence is not benchmarked.Source: Limitations · pp. 9–10

Anticipated reviewer critiques and our responses

• The authors acknowledge that N = 600 is small, prioritizing useful deployment-register items and phenomenon coverage over raw size, with the contribution pipeline intended to support growth.They defend per-slice comparisons by giving each phenomenon at least 30 items and making 30% of the dataset blind; per-slice scores are reported with confidence intervals and not as headline rankings.Source: Limitations · pp. 9–10, Anticipated reviewer critiques and our… · pp. 10–11

Limitations

• The benchmark is text-only and English-source-only, excluding direct tests of pronunciation, tone, prosody, dialect-to-dialect translation, and third-language pivots.Adjacent Namibian languages face the same coverage gap but are future work, not part of the benchmark’s commitment.Source: Limitations · p. 10
• The full 600-item v1.0 build remains in production; final agreement figures and populated leaderboard results are deferred to the Q4 2026 release.The internal v0.1 build has 423 items and has been used to validate the pipeline.Source: Limitations · p. 10

Anticipated reviewer critiques and our responses

• Deployment-derived items are paraphrases inspired by real-traffic distributions, not raw user text, because verbatim queries cannot be released for privacy reasons.The authors describe these paraphrases as the closest publishable proxy to the deployment surface available.Source: Anticipated reviewer critiques and our… · p. 10

Ethics Statement

• Production-query source items are paraphrased and scrubbed of personal identifiers before review; verbatim user text is never released.A per-user cap of three derived items limits the influence of any one user’s speech patterns.Source: Ethics Statement · p. 11
• Both reference translators are compensated at market rates, and their names appear in the acknowledgments and dataset citation file with consent.The 30-item agreement set publishes both translators’ work side by side rather than adjudicating one reference as correct.Source: Ethics Statement · p. 11
• The dataset and paper use CC-BY 4.0 and the code uses MIT, with attribution to the benchmark and named translators framed as a counterweight to proprietary extraction.Source: Ethics Statement · p. 11
• Human-evaluation raters come from a self-selected volunteer community of about 40 Oshiwambo speakers already engaged with the project.The authors disclose this selection-bias risk and describe blinding, randomized presentation, calibration items, demographic reporting, and possible external raters as mitigations.Source: Ethics Statement · p. 11

A Phenomenon tag definitions

• The tag set is designed to distinguish failure modes, including honorific and address-register choices, noun-class agreement, context-dependent polysemy, and cohesion across sentences.The multi-sentence tag covers 2–4 sentence mini-paragraphs and tests pronoun reference, tense consistency, and topic chaining.Source: A Phenomenon tag definitions · p. 13
• Other tags test negation scope, preservation of numbers and dates, named entities, and distinctions among tense and aspect.The negation tag includes single, double, and scope-ambiguous constructions; the numbers tag tests preservation of numerical content while translating context.Source: A Phenomenon tag definitions · p. 13
• The remaining tags cover code-switching, pronoun coreference, and idioms whose literal translation would be non-idiomatic.The code-switch tag includes English loanwords in otherwise-Oshiwambo sentences and the reverse.Source: A Phenomenon tag definitions · p. 13

B Schema reference

• The TSV schema assigns each item a stable ID, length bucket, domain, phenomenon tags, provenance, English source, and references for both dialects.It also allows alternate references and translator notes, with alternate references available for the 30-item agreement set.Source: B Schema reference · pp. 13–14
• A deterministic stratified split marks 180 items as blind and the remaining items as development data; the schema also flags the agreement set.The blind split uses seed 42, and separate JSONL files are available for blind and development subsets.Source: B Schema reference · p. 14

C Prompting templates

• All frontier and open-weight LLMs use the same zero-shot template, with the dialect and English source substituted and no system-prompt variation.MT-system templates are system-specific because of API differences and are documented per system at publication.Source: C Prompting templates · p. 14

D Human-eval rater rubric

• Human raters score adequacy and fluency on five-point scales, with anchor examples and third-reader adjudication when raters differ by at least two points.Adequacy concerns preservation of the English source meaning; fluency is rated independently as naturalness to a fluent speaker.Source: D Human-eval rater rubric · p. 14

• Contributions treat Oshindonga and Oshikwanyama as living, evolving languages of millions of speakers, not extractable resources.Source: E Contributor code of conduct · p. 14
• Reference translations and reviewer contributions are credited at contributors’ discretion, and anonymous contributions are supported.Source: E Contributor code of conduct · p. 14
• Submissions must not contain machine-translated content; references and reviewer alternates must be native-speaker work, not MT post-edits.Source: E Contributor code of conduct · p. 14
• Accepted contributions use CC-BY-4.0 for data and MIT for code, consistent with the existing dataset and scripts.Source: E Contributor code of conduct · p. 14
• Discussions, code reviews, and dataset reviews are conducted with patience and care.Source: E Contributor code of conduct · p. 14