PaperlayerView exact revision on arXiv

Source-linked AI summary

EnSiTa - A Trilingual Multi-Domain Parallel Dataset and Benchmark for Domain-Specific Machine Translation

Surangika Ranathunga, Nisansa de Silva, Aloka Fernando, Kavindu Warnakulasuriya, Isuru Wijesiri, Menan Velayuthan, Charitha Rathnayaka, Thivaharan Varatharajan, Sajeevi Silva, Piumi Kandanaarachchi, Uthayasanker Thayasivam

arXiv:2609.29511v1cs.CL

TL;DR

Low-resource machine translation has limited parallel data, especially in specialised domains. EnSiTa provides a multilingual, multi-domain dataset and benchmark for English, Sinhala, and Tamil, which the paper uses to study domain-specific MT. Its results offer practical guidance on data selection, transfer, and continued fine-tuning under limited budgets.

  • Problem

    Parallel data for low-resource languages is scarce or absent in specialised domains, where machine translation also lags behind high-resource languages.

    Source: §1 · p. 1

  • Method

    EnSiTa provides professionally post-edited training data and manually translated test sets across domains for English, Sinhala, and Tamil, supporting experiments on domain-specific MT.

    Source: §1 · p. 1

  • Results

    The study finds that useful domain-specific MT strategies depend on model state: curated in-domain data, domain divergence, and continued fine-tuning have different implications for pre-trained and from-scratch systems.

    Source: §6 · p. 11

  • Takeaways & Limitations

    For practitioners with limited budgets, the results provide guidance on choosing in-domain versus out-of-domain data and on when domain fit or additional parallel data matters.

    Source: §6 · p. 11

    Identifying the exact domain of the datasets is out of scope; the study adopts a corpus-source definition that can encompass differences in topic, genre, style, or formality.

    Source: §3.4 · p. 4

Abstract

from arXiv · show

Machine Translation (MT) for low-resource languages remains far behind that of high-resource languages, and the gap is widest in specialised domains, where parallel data is scarce or entirely absent. We present EnSiTa, a trilingual multi-domain parallel dataset and benchmark for English, Sinhala and Tamil. EnSiTa provides human post-edited training data for seven domains, plus manually translated test sets for those and one additional domain, all produced by professional translators under a multi-year, rigorously quality-controlled process. Using this dataset, we conduct an extensive study of domain-specific MT for all six language directions, fine-tuning a from-scratch Transformer, a pre-trained translation model (NLLB-600M), and decoder-only LLMs (Gemma 3 family, 1B-12B, and TranslateGemma) across training-data sizes, model scales, and in-domain, cross-domain, multilingual and multi-domain settings. To the best of our knowledge, this is the most extensive systematically documented multi-domain parallel data creation and benchmarking effort for low-resource MT. Our data and models will be publicly released.

1 Introduction

EnSiTa provides professionally post-edited training data and manually translated test sets across multiple domains for English, Sinhala, and Tamil. The paper uses these resources to study domain-specific MT and presents the dataset and experiments as a systematic contribution to low-resource MT.

1 Introduction

  • The dataset contains 200k+ professionally post-edited training sentence pairs across three language pairs and 10k+ manually created test pairs.Training and test data cover seven domains, with test data for an eighth, LLM Jailbreak.Source: §1 · p. 1
  • The study evaluates translation systems across domain-specific settings, including training-data size, model scaling, cross-domain transfer, and multilingual and multi-domain fine-tuning.It uses 11 test sets and compares a from-scratch sequence-to-sequence model, NLLB-600, and Gemma 3-1B instruct.Source: §1 · p. 1
  • The authors describe EnSiTa as the first low-resource MT study to release per-domain training data and manually created test sets at this scale alongside systematic domain experiments.They state that the data and models will be publicly released.Source: §1 · p. 2

2 Related Work

Prior low-resource MT resources and evaluations often cover few domains, lack per-domain training data at scale, or do not test cross-domain transfer. For Sinhala, Tamil, and English, the paper situates EnSiTa amid ongoing MT development and limited clean, publicly available parallel data.

2.1 MT Datasets for LRLs

  • Earlier low-resource MT datasets often focused on one domain or lacked domain-specific experiments; other multi-domain resources lacked per-domain training data at scale or cross-domain evaluation.MENYO-20k provides six test domains, while IN22 provides human-translated tests across 13 domains but only about 80 sentences per domain.Source: §2.1 · p. 2

2.2 Domain-specific NMT Evaluation

  • Prior domain-specific low-resource MT evaluations generally covered fewer than five domains, including the comprehensive study of mBART domain performance and divergence.Other cited evaluations used fine-tuned pre-trained encoder-decoder models such as mBART.Source: §2.2 · p. 2

2.3 Sinhala-Tamil-English MT

  • Sinhala is low-resource and morphologically rich, while Tamil is mid-resource, agglutinative, and has limited publicly available resources for Sri Lankan Tamil.Tamil dialectal variation is also reported to hinder robust NLP applications.Source: §2.3 · p. 2
  • MT is important in Sri Lanka for education, health, and government communication, and prior Sinhala–Tamil work has progressed from SMT to NMT and pre-trained translation models.The cited work reports that fine-tuned NLLB outperforms fine-tuned mBART.Source: §2.3 · pp. 2–3
  • The authors report that earlier Sinhala–Tamil–English research did not publicly release its developed corpora, while available web-mined corpora are extremely noisy.They identify FLORES and MultiMWP as human-curated public resources covering all three language pairs, respectively open-domain and math-word-problem data.Source: §2.3 · p. 3

3 Multi-Domain Dataset Preparation

EnSiTa combines professionally post-edited training corpora with manually translated tests for English, Sinhala, and Tamil, drawing on public parallel and domain-specific monolingual sources. Its ten-domain statistics show varied sentence-length profiles and substantial differences in divergence across domains and language directions.

3.1 Data Sources and Preparation

  • Public parallel data were filtered and ranked, while monolingual material was translated into Sinhala and Tamil; web-mined and machine-translated sentences were human post-edited.NLLB and OpenSubtitles data were deduplicated on English and scored with LASER-3; the top 30,000 pairs with at least three words were selected.Source: §3.1 · p. 3

3.2 Test Set Creation

  • Test sets were manually translated, including an additional LLM Jailbreak set, and the authors checked for overlap across data splits.The experiments also use existing Bible and Government corpora with separately provided test and validation sets.Source: §3.2 · p. 3

3.3 Post-Editing & Review Process

  • More than 110 translators contributed over three years, with screened translators, domain-specific guidelines, continuous review, and feedback.For Sinhala–Tamil data, the team aligned corresponding Sinhala and Tamil corpora using English as a pivot, with further trilingual refinement when translators were available.Source: §3.3 · pp. 3–4

3.4 Dataset Statistics & Analysis

  • The corpora span 10 domains and total 200k+ training, 8k+ validation, and 11k+ test sentence pairs.Bible, Government, and FLORES+ data are borrowed from prior research.Source: §3.4 · p. 4
  • Jailbreak has the most irregular English sentence-length distribution; Literature and OpenSubtitles favor shorter utterances, while Literature also includes single-word and long sentences.Government and Health trend toward longer sentences; NLLB, Bible, Wikipedia, and News most resemble normal distributions.Source: §3.4 · pp. 4–5, Figure 1 · p. 5
  • JSD places NLLB near OpenSubtitles, Literature, News, and Jailbreak (0.12–0.18), while Health/Bible/Literature and Bible/Government diverge (0.34–0.40).Source: §3.4 · p. 5, Figure 2 · p. 5
  • Directional KL divergence exposes language-direction differences that bounded JSD flattens, including high Bible EnSi and EnTa divergence and relatively low Government divergence in those directions.The authors relate Government’s lower divergence to domain terms that were created as loanwords or calques in Sinhala and Tamil.Source: §3.4 · pp. 5–6

4 Experiments

The experiments vary training-data size, model type, and domain setting, with separate comparisons of larger models and multilingual or multi-domain fine-tuning. The setup uses fixed LoRA recipes for Gemma 3 scale comparisons, but a lighter recipe and separate zero-shot baseline for TranslateGemma.

4.1 Experiment Plan

  • The core plan compares Fairseq, NLLB-600M, and Gemma 3 1B across six training sizes and in-domain versus out-of-domain evaluation.Training sizes range from 1k to 25k sentence pairs; out-of-domain evaluation tests on a domain different from the training domain.Source: §4.1 · p. 6
  • Larger Gemma 3 and TranslateGemma models are compared in a government-domain ablation, limited to that domain to keep the experiment tractable.The tested variants are Gemma 3 4B and 12B, and TranslateGemma 4B and 12B.Source: §4.1 · p. 6
  • Multilingual, multi-domain, and combined settings use samples of 1,000 sentence pairs per domain for each language pair.These experiments use Fairseq, Gemma 12B, and NLLB; the multi-domain setting also includes TranslateGemma 12B.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 Experiment Setup

  • LoRA recipes are fixed across Gemma 3 sizes, whereas TranslateGemma uses a lighter recipe and is assessed against its own zero-shot baseline.The authors say the lighter recipe is intended to limit erosion of TranslateGemma’s reinforcement-learned translation ability, so the comparison is not like-for-like with base Gemma 3.Source: §4.2 · p. 7

5 Automatic Evaluation

The evaluation compares domain-specific MT across training sizes, model types, language directions, and transfer settings. It finds that curated in-domain data benefits pretrained models, while domain divergence, Sinhala generation, and model scale shape performance.

5 Automatic Evaluation

  • Degenerate Gemma-1B outputs at or below 1.5 chrF++ still receive mean COMET 0.40 scores, up to 0.54, motivating chrF++ for discussion.The authors describe COMET scores as inconsistent and unreliable.Source: §5 · p. 7

5.1.1 In-domain

  • At 5k, Literature and OpenSubtitles are hardest for the from-scratch Transformer, while Maths and Bible are easiest; token budget, not lexical difficulty, dominates domain variation.OpenSubtitles averages 7.7 English words per sentence, versus 25.2 in Bible and 26.1 in Health.Source: §5.1.1 · pp. 7–8

5.2 Language-wise Analysis

  • Across trained in-domain results, target-language means are English 42.7, Tamil 37.9 and Sinhala 31.3 chrF++, locating the largest asymmetry in generating Sinhala.Degenerate outputs occur in 159 vanilla Transformer cases and 73 Gemma-1B cases, with every Gemma-1B collapse into Sinhala; si-ta beats ta-si in 17 of 19 matched-content cells, by up to +15.9 chrF++.Source: Figure 4 · p. 8, Figure 5 · p. 9, §5.2 · p. 10

5.3 Effect of model choice

  • 57.1 chrF++ for Gemma-12B versus 56.0 for NLLB-600M on Government at 25k; fine-tuned Gemma-3 scores rise with scale, but zero-shot rank can differ.Gemma-12B scores below Gemma-1B and Gemma-4B zero-shot on low-resource directions; fine-tuning restores the expected scale order.Source: Figure 5 · p. 9, §5.3 · p. 11

5.1.1 In-domain

  • NLLB-600M scores 47.0 chrF++ zero-shot, gains 4.6 points with 1k in-domain pairs and 3.9 more at 25k, and at 1k beats competitors at full size in all nine domains.Its zero-shot average covers nine trainable domains and all language directions.Source: §5.1.1 · p. 9

5.1.2 Out-domain

  • Out-domain results are mostly lower than corresponding in-domain results, and domain divergence is more predictive of transfer loss for the vanilla Transformer than for pretrained models.For Government, NLLB-trained data transfers better than Bible data, especially with the vanilla Transformer.Source: §5.1.2 · p. 9

5.4 Multilingual vs Multi-Domain

  • A dedicated in-domain corpus of about 25k generally beats a double-size multilingual and multi-domain mix, while combining data helps domains with fewer than 5k pairs.The experiments merged corpora without accounting for domain divergence; selecting domains by divergence could yield better gains, but was not exhaustively tested.Source: Figure 7 · p. 10, §5.4 · p. 11

5.1.2 Out-domain

  • NLLB loses 7.2 chrF++ on FLORES+ as Bible training data grows from 1k to 25k, while less-divergent open-web, News, OpenSubtitles and Wikipedia yield gains.Across eight domains, erosion correlates with mean divergence (rank correlation -0.71); Gemma-1B improves on FLORES+ by +2.5 to +22.5. For the vanilla Transformer, JSD/result correlation is r=-0.56, and the effect is less visible in pretrained models.Source: §5.1.2 · pp. 9–10

6 Conclusion

EnSiTa provides a trilingual, eight-domain benchmark and uses it to study domain-specific MT across data sizes, model architectures, scale, and transfer. Its findings offer practical guidance on data selection and fine-tuning for low-resource systems.

6 Conclusion

  • EnSiTa is a parallel dataset and benchmark for English, Sinhala and Tamil, covering all six translation directions and eight domains.The study spans training-data size, model architecture and scale, cross-domain transfer, and multilingual and multi-domain fine-tuning.Source: §6 · p. 11
  • For pretrained translation models, a small curated in-domain set is on average worth more than any amount of out-of-domain data, especially when translating into English.Source: §6 · p. 11
  • From-scratch models need volume before domain fit matters, while domain divergence predicts transfer loss more for them than for pretrained models.Source: §6 · p. 11
  • Continued fine-tuning on a divergent domain erodes competent models’ general translation ability; planned next steps include document-level translation, additional domains, and human evaluation.The dataset, fine-tuned models, and evaluation outputs will be released.Source: §6 · p. 11

A Training Configuration Details

The configuration uses fixed Gemma 3 settings for model-scale comparisons and a lighter, capability-preserving TranslateGemma recipe for a separate adaptation question. It specifies prompts, training-data mixtures, and implementation settings, while limiting cross-family comparisons.

A Training Configuration Details

  • LoRA fine-tuning freezes pretrained weights and trains low-rank adapters; the NLLB-600M and Fairseq experiments ran separately on an institutional GPU cluster.Gemma 3 adapters are applied to language-model projections, with the vision encoder frozen for 4B/12B checkpoints; 8-bit AdamW was used after 32-bit AdamW produced NaN gradients early in training.Source: A Training Configuration Details · p. 17
  • Base Gemma 3 examples use a source-and-target-language chat prompt, while TranslateGemma retains its native structured translation prompt.Training loss is masked to target-text tokens; training and evaluation use identical prompts in both model families.Source: A Training Configuration Details · p. 17
  • The multi-domain mixture samples 1,000 pairs per domain, while per-domain multilingual adapters use 1,000 pairs per direction.The combined multilingual and multi-domain adapter uses 49,018 examples across all six directions.Source: A Training Configuration Details · p. 17
  • The Gemma 3 recipe is fixed across 1B, 4B, and 12B variants, with effective batch size held at 32.TranslateGemma uses a lighter recipe to limit erosion of its pre-existing translation tuning.Source: Table 3 · p. 18
  • TranslateGemma uses a deliberately lighter adaptation recipe and is assessed against its own zero-shot baseline, not as a controlled comparison with base Gemma 3.The recipes address different research questions; changing rank, learning rate, and epochs constrains updates to limit erosion of translation ability.Source: A Training Configuration Details · pp. 17–18

B Training Materials

Translators received video recordings and guideline documents to familiarize themselves with the task and consult during it. The materials include a sample task sheet and the English–Sinhala Wikipedia guideline document.

B Training Materials

Related papers

Loading 2609.29511v1…