PaperlayerView exact revision on arXiv

Source-linked AI summary

COILD: An Indic-Centric Parallel Corpus and Benchmark for Machine Translation Across Indian Languages

Kshetrimayum Boynao Singh, Nitin Kumar Mishra, Palash Pratim Dutta, Atai Waris Khan, Aparna Kaushik, Avinash Kumar, Deeksha, Deepak Kumar, Saroj Kumar Jha, Saloka Sengupta, Anansa Roy, Umalatha Kannoth, Saifulla Samar, Meena Sharma, Manpreet Kaur, Jyoti Sharma, Ashwini Vaidya, Muralikrishna SN, Md Shad Akhtar, Poonam Bansal, Amita Dev, Sanasam Ranbir Singh, Samit Bhattacharya, Tanmoy Chakraborty, Asif Ekbal

arXiv:2609.28826v1cs.CL

TL;DR

Indian-language MT lacks high-quality Indic-centric parallel corpora and benchmarks that reflect Indian linguistic characteristics and practical domains. COILD contributes a human-verified corpus and domain-focused benchmark, and fine-tuning two multilingual models on it improves translation quality across most language pairs and domains. The resource is also positioned for future multilingual LLM research, though its language and document-level coverage remain bounded.

  • Problem

    Indian-language MT has limited high-quality Indic-centric parallel corpora and benchmarks, while English-pivot or non-Indian-origin resources may miss Indian linguistic characteristics and applied domains.

    Source: §1 · pp. 1–2

  • Method

    COILD constructs an Indic-centric human-translated and verified corpus from Indian-language sources, adds an expert-verified domain benchmark, and tests fine-tuning IndicTrans2-Distilled and NLLB-200.

    Source: §1 · p. 2, §3.5 · p. 6

  • Results

    Fine-tuning IndicTrans2-Distilled and NLLB-200 on COILD consistently improves translation quality across most language pairs, with gains across all eight domains and particularly strong gains in Governance, Judiciary, and Science & Technology.

    Source: §6 · p. 10

  • Takeaways & Limitations

    COILD is presented as a training and evaluation resource for Indian-language MT and as a potential resource for multilingual LLM pretraining, instruction tuning, and domain adaptation.

    Source: §6 · p. 11, §7 · p. 11

    COILD covers 20 language pairs and eight domains rather than India’s full linguistic diversity, and sentence-level data limits evaluation of document-level translation phenomena.

    Source: Limitations · p. 11

Abstract

from arXiv · show

Machine translation (MT) for Indian languages remains constrained by the limited availability of high-quality, Indic-centric parallel corpora and evaluation benchmarks. Existing multilingual resources are largely constructed from English-pivot content and often fail to capture the linguistic diversity, cultural complexity, and domain-specific characteristics of Indian languages. We present COILD, an Indic-centric parallel corpus comprising over 1.16 million human-translated and human-verified sentence pairs, covering 20 Indian language pairs across the Indo-Aryan, Dravidian, Tibeto-Burman, and Austro-Asiatic language families. The corpus is built entirely from original Indian language sources collected from licensed repositories spanning eight domains with direct real-world applicability. Furthermore, we introduce a domain-centric benchmark comprising 2,000 expert-verified sentences to enable consistent multilingual and cross-lingual evaluation across Indian language pairs. To validate the effectiveness of COILD, we fine-tune two representative multilingual neural machine translation models, IndicTrans2-Distilled and NLLB-200. Experimental results demonstrate consistent improvements across language pairs, domains, automatic evaluation metrics, and human evaluation, highlighting the effectiveness of high-quality Indic-centric supervision. COILD provides a valuable training and evaluation resource for advancing multilingual machine translation and future multilingual language models for Indian languages.

1 Introduction

COILD addresses gaps in Indian-language machine translation resources: English-pivot data and benchmarks can miss Indian linguistic and cultural characteristics and practical domains. It combines an Indic-centric corpus and domain-specific benchmark with experiments showing improvements from COILD fine-tuning.

1 Introduction

  • Existing English-pivot corpora and benchmarks can underrepresent Indian-language characteristics and applied domains, limiting how well they reflect real-world Indian contexts.The introduction identifies lexical, morphological, syntactic, cultural, and domain-specific representation gaps.Source: §1 · pp. 1–2
  • Fine-tuning IndicTrans2 and NLLB-200 on COILD improves translation quality across nearly all evaluated directions, while analyses examine pivot choice and automatic-metric reliability.Human evaluation also identifies limitations in widely used automatic metrics for Indian languages.Source: §1 · p. 2
  • COILD contains over 1.16 million human-translated and human-verified pairs from original Indian-language sources, with Hindi and Tamil serving as family-specific pivots.Its dual-pivot design covers Indo-Aryan, Tibeto-Burman, and Austro-Asiatic languages via Hindi and Dravidian languages via Tamil.Source: §1 · p. 2
  • The authors introduce an expert-verified benchmark across eight real-world domains for evaluation of Indian-language machine translation.The benchmark is held out from training and supports evaluation across language pairs and domains.Source: §1 · p. 2
  • The framework proceeds from dual-pivot corpus construction to model fine-tuning and then automatic, domain-wise, and human evaluation.The evaluation includes five automatic metrics and human ratings of adequacy, fluency, and overall quality.Source: Figure 1 · p. 3

2 Related Work

Prior Indian-language corpora expanded data availability, but English-centric construction and mined content can weaken direct Indic representation. Existing benchmarks offer standardised evaluation yet provide limited coverage of Indian domains; COILD responds with human-verified Indic-centric data and a domain-focused benchmark.

2.1 Parallel Corpora for Indian Languages

  • Existing large corpora expanded scale, but English-mediated pairs, web mining, and non-Indian-origin content can introduce indirect alignments, noise, and weak representation of Indian expressions.The related work describes Samanantar, NLLB, and BPCC as extending data volume and language coverage while noting these shared characteristics.Source: §2.1 · p. 3
  • COILD instead uses original Indian-language content and human translation with expert verification to preserve natural linguistic characteristics.Its coverage includes eight domains.Source: §2.1 · p. 4

2.2 Machine Translation Benchmarks

  • Existing multilingual benchmarks are largely English-origin and general-purpose, with limited domain-wise analysis of real-world Indian applications.COILD introduces 2,000 expert-verified sentences across eight application domains for consistent evaluation across Indian language pairs.Source: §2.2 · p. 4

2.3 Multilingual Machine Translation for Indian Languages

  • Architectural advances in multilingual MT do not remove the need for high-quality parallel data and realistic Indian-domain evaluation resources.The related work notes that such corpora and benchmarks remain scarce for many Indian language pairs.Source: §2.3 · p. 4

2.4 Our Contribution

  • COILD combines more than 1.16 million human-verified pairs across 20 language pairs with a 2,000-sentence, eight-domain benchmark.The authors fine-tune IndicTrans2 and NLLB-200 and report consistently improved translation performance.Source: §2.4 · p. 4

3 The COILD Corpus

COILD is built from original Indian-language sources through human translation and layered quality checks, with a separate expert-verified benchmark designed for aligned multilingual and cross-lingual evaluation.

3.1 Corpus Design Principles

  • The corpus design prioritises Indic-origin sources, manual translation, domain diversity, and human quality control.Translations are manually produced without machine translation and undergo multiple rounds of verification.Source: §3.1 · p. 5

3.2 Source Collection and Validation

  • Licensed and public Indian-language materials are processed into sentence-level text, including OCR and manual cleanup where scanned sources require it.Experts validate grammar and spelling, remove duplicates and incomplete or ambiguous sentences, filter personal information, and retain contextually complete material.Source: §3.2 · pp. 5–6

3.3 Translator Recruitment and Selection

  • Recruitment sought translators proficient in Hindi or Tamil and the corresponding target language, but qualified candidates were difficult to find for low-resource languages.An open call across states and language communities aimed to minimise regional and dialectal bias; some applicants also lacked annotation-platform computer skills.Source: §3.3 · p. 6
  • Candidates passed background and language assessment followed by expert evaluation of translation adequacy, fluency, grammar, and terminology.Only applicants who passed both stages were onboarded.Source: §3.3 · p. 6

3.4 Human Translation Workflow

  • Native target-language speakers translated assigned batches manually under guidelines to preserve meaning, discourse, register, and domain terminology while favouring natural expression.Translations were submitted for linguistic review after each batch.Source: §3.3 · p. 6, §3.4 · p. 6

3.5 Multi-stage Quality Assurance

  • Independent experts review every translation batch for adequacy, fluency, terminology, and grammar, using a 20% TER-based edit-rate threshold to accept or return batches.Returned batches undergo revision cycles, and accepted translations receive domain-expert validation for terminology, context, and relevance.Source: §3.5 · p. 6

3.6 Benchmark Construction

  • A 2,000-sentence evaluation benchmark comprises expert-verified parallel sentences, created exclusively by in-house consortium experts; shared sentence identifiers align the same source sentence across supported languages, enabling multilingual and cross-lingual evaluation.Source: §3.6 · p. 7

4 Experimental Setup

The experiments test COILD as a training resource rather than propose a new architecture, fine-tuning two complementary multilingual models on the corpus. Evaluation uses the held-out COILD benchmark, automatic metrics, and domain-wise analysis.

4 Experimental Setup

  • The experiments assess whether COILD improves multilingual MT, benefits models with different training paradigms, and supports evaluation across Indian language pairs and domains.The study fine-tunes representative models and evaluates them on the held-out COILD benchmark.Source: §4 · p. 7

4.1 Models

  • IndicTrans2-Distilled represents an Indic-focused multilingual model, while NLLB-200 provides a general-purpose system trained on multilingual data.Both are fine-tuned from official pretrained checkpoints using COILD.Source: §4.1 · pp. 7–8

4.2 Training Data

  • The training corpus contains more than 1.16 million human-verified pairs across 20 language pairs, four language families, and eight application domains.Each language pair is fine-tuned independently to isolate COILD supervision while preserving pretrained multilingual representations.Source: §4.2 · p. 8

4.3 Implementation Details

  • Fine-tuning follows each model’s recommended settings, retains its original vocabulary and tokenization, and selects checkpoints by validation chrF++.Training uses mixed precision (FP16); Table 2 is identified as the source of the hyperparameter configuration, but its values are not given in the supplied passage.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Evaluation Protocol

  • Evaluation uses the 2,000-sentence aligned benchmark for multilingual and cross-lingual comparisons across eight domains.Five metrics are reported: BLEU, chrF++, TER, BERTScore, and COMET; domain-wise results assess generalisation across application scenarios.Source: §4.4 · p. 8

5 Results

Fine-tuning COILD improved translation across most language pairs for both models, across application domains, and in human evaluation. Gains were strongest for lower-resource directions and selected specialized domains.

5.1 Overall Translation Performance

  • Fine-tuning improved both models across nearly all language pairs, with larger gains for lower-resource pairs and smaller positive gains for stronger baselines.The overall comparison covers BLEU, chrF++, TER, BERTScore, and COMET.Source: §5.1 · pp. 8–9, Figure 3 · p. 9

5.2 Domain-wise Performance Analysis

  • Both models improved across all eight domains, with multiple pairs showing significant BLEU gains in each; strongest gains were in Governance, Judiciary, and Science & Technology.Education improved stably across nearly all pairs; Agriculture, Tourism, Climate, and Healthcare also benefited, though some directions gained less.Source: §5.2 · pp. 9–10

5.3 Human Evaluation

  • Two native annotators rated adequacy and fluency on five-point scales; most pairs improved, with a few modest decreases, and human judgments aligned with automatic gains.Larger gains were particularly evident in several low-resource directions.Source: §5.3 · p. 10
  • Urdu→Hindi under IndicTrans2 gained 2.35 points in Adequacy and 2.32 in Fluency, its largest reported human-evaluation increase.NLLB’s listed gains included Hindi→Sindhi (2.41/2.25), Hindi→Konkani (2.13/1.94), and Tamil→Kannada (1.26/0.57) for Adequacy/Fluency.Source: §5.3 · p. 10

6 Discussion

COILD fine-tuning improved translation across two different multilingual models, supporting the importance of training-data quality alongside architecture. Domain-wise and human evaluations also support the gains, while the experiments use established NMT baselines.

6 Discussion

  • Fine-tuning COILD improved translation across most language pairs for IndicTrans2-Distilled and NLLB-200, suggesting its gains are not specific to one architecture.The results support the importance of training-data quality alongside model architecture.Source: §6 · p. 10
  • Both models improved across all eight application domains, with particularly strong gains in Governance, Judiciary, and Science & Technology.The discussion links these gains to accurate translation of domain-specific terminology and domain-balanced resources.Source: §6 · p. 10
  • Independent native-language assessments reported improvements in adequacy, fluency, and overall quality, aligning with automatic evaluation.The agreement suggests the gains reflect translation-quality improvements rather than metric-specific optimization.Source: §6 · p. 11
  • The experiments use IndicTrans2-Distilled and NLLB-200 rather than LLMs; the authors identify COILD as relevant to future multilingual LLM pretraining, instruction tuning, and domain adaptation.The authors chose reproducible NMT baselines to isolate the dataset’s contribution.Source: §6 · p. 11

7 Conclusion

COILD combines a large human-translated corpus with an expert-verified, domain-centric benchmark. Fine-tuning two multilingual models improved translation across language pairs, domains, and evaluation types.

7 Conclusion

  • COILD contains over 1.16 million human-translated and human-verified sentence pairs across 20 Indian language pairs and eight real-world domains.Its benchmark comprises 2,000 expert-verified sentences for multilingual and cross-lingual evaluation.Source: §7 · p. 11
  • The authors position COILD as a reusable resource for Indian-language MT and future multilingual LLM pretraining, instruction tuning, domain adaptation, and many-to-many translation.The reported fine-tuning results improve across language pairs, application domains, automatic metrics, and human evaluation.Source: §7 · p. 11

Limitations

COILD’s language and domain coverage is incomplete, and its sentence-level construction limits evaluation of document-level translation phenomena. The experiments also leave multilingual LLM evaluation and subtle regional variation unresolved.

Limitations

  • Coverage is limited to 20 language pairs and eight domains, leaving many regional dialects and minority languages unrepresented.Sentence-level construction also limits evaluation of discourse coherence, context-dependent translation, and coreference resolution.Source: Limitations · p. 11
  • Experiments cover IndicTrans2-Distilled and NLLB-200, not recent multilingual LLMs, and subtle linguistic variations or regional preferences may remain.The authors intend to expand the corpus and benchmark to improve coverage, domain diversity, and linguistic representation.Source: Limitations · p. 11

Ethics Statement

COILD's data collection and evaluation followed a human-centric process, with permissioned sources, privacy and content screening, and reviewed human translations.

Ethics Statement

  • Sources came from public or licensed Indian resources with necessary permissions, and were screened for personal information, offensive content, and inappropriate material.Qualified native speakers were recruited through a structured process and compensated for their work.Source: Ethics Statement · p. 11
  • Qualified native speakers manually translated the material, and translations received multiple rounds of linguistic and domain-specific review for accuracy, fluency, and consistency.Source: Ethics Statement · p. 11
  • Two native annotators independently conducted the human evaluation for each translation direction.Source: Ethics Statement · p. 11

Institute-wise Contributor

COILD’s guidelines were shaped by linguistic analysis and language-specific expertise, then applied through translation, review, and standardization.

Institute-wise Contributor

A Translation Guidelines

  • COILD developed standardized translation guidelines with language experts and faculty to support linguistic accuracy, cultural appropriateness, and semantic fidelity.Workshops with language-specific colleges helped resolve points of disagreement in translation rules.Source: A Translation Guidelines · p. 13

A.1 Linguistic Analysis

  • Contrastive analysis of Hindi-centric and Tamil-centric language pairs examined grammatical, morphological, syntactic, typological, agreement, honorific, and lexical differences to inform the guidelines.Source: §A.1 · p. 13

A.2 Translation Rules and Language Conventions

  • Language-specific rules cover grammar, named entities, terminology, foreign and compound words, idioms, abbreviations, numerals, punctuation, quality, and domain-specific recommendations.Recommendations address legal, healthcare, education, agriculture, governance, and science content.Source: §A.2 · p. 13

A.3 Translation and Post-editing

  • Native translators followed predefined guidelines, favoring natural, culturally appropriate translations over literal wording when literal phrasing sounded unnatural.Internal language-specific research associates reviewed and post-edited translations for grammar, lexical consistency, and readability.Source: §A.3 · p. 14

A.4 Quality Assurance and Validation

  • Native linguists and domain experts independently reviewed translations for accuracy, structure, fluency, terminology, and cultural appropriateness, resolving inconsistencies through discussion.Final decisions were made by mutual consent and incorporated into the corpus guidelines over time.Source: §A.4 · p. 14

A.5 Final Standardization

  • Validated translations were standardized across language pairs for formatting and consistent treatment of punctuation, named entities, acronyms, numerals, and technical terminology.The account also notes updates to replaced terms across languages.Source: §A.5 · p. 14

B Human Annotation Guidelines for Evaluation

The evaluation protocol combines native-speaker ratings of adequacy and fluency with qualitative remarks across translation directions, while figures and a table present baseline-to-finetuned comparisons.

B Human Annotation Guidelines for Evaluation

  • The five-point human-rating scale assesses adequacy and fluency for translations from baseline and fine-tuned IndicTrans2 and NLLB-200 systems.Adequacy concerns preservation of source meaning and intent; fluency concerns grammatical correctness, readability, naturalness, and word choice.Source: B Human Annotation Guidelines for… · p. 14
  • For each of 40 directions, evaluation sampled 50 sentences from each of eight domains; two native-speaker annotators independently rated each system output.Scores for each criterion were averaged across the two annotators.Source: B Human Annotation Guidelines for… · p. 14

B.2 Qualitative Remarks

  • Qualitative remarks recorded translation issues and improvements in semantic preservation, fluency, terminology, grammar, and error correction across the evaluation protocol.The same protocol was applied across all 40 translation directions.Source: §B.2 · p. 14
  • Table 3 compares baseline and fine-tuned human ratings; positive and negative deltas are color-coded, while Odia evaluation remains underway.Source: Table 3 · p. 15
  • Figure 4 organizes per-domain BLEU changes by translation direction, enabling comparison of fine-tuning gains across domains and directions.Source: Figure 4 · p. 16
  • The captions identify Figures 5 and 6 as plots of overall baseline and fine-tuned performance on the COILD Benchmark dataset, for IndicTrans2 and NLLB, respectively.Source: Figure 5 · p. 16, Figure 6 · p. 17

Related papers

Loading 2609.28826v1…