Source-linked AI summary
BM25-Augmented Many-Shot Translation for Low-Resource North-Eastern Indian Languages
Aashish Dhawan, Christopher Driggers-Ellis, Dzmitry Kasinets, Christan Grant, Daisy Zhe Wang
TL;DR
Several North-Eastern Indian languages have almost no prior machine-translation work. This paper evaluates a training-free BM25- and Gemini 2.5 Flash-based system across English and eleven languages, outperforming competitors by small measurable margins.
Problem
Several North-Eastern Indian languages, including Nyishi, Karbi, and Tagin, have almost no prior machine-translation work.
Method
A training-free system uses BM25-retrieved parallel examples and Gemini 2.5 Flash to translate across 22 English–language directions.
Results
In 19 translation tasks, the contrastive submission achieved the best BLEU score in nine and the best ChrF++ score in eleven.
Takeaways & Limitations
The results isolate retrieval-augmented machine translation from vision and reproduce the team’s earlier success with small, measurable competitive margins.
Takeaways & Limitations
Retrieval-bank quality varies widely, with limited and often out-of-domain parallel data for Category 2 languages.
Abstract
from arXiv · showhide
This paper describes the University of Florida Gators submission to the WMT26 Low-Resource Indic Language Translation shared task. We adapt the retrieval-augmented many-shot translation pipeline from our AmericasNLP 2026 system to translate between English and eleven North-Eastern Indian languages in both directions. At inference time, BM25 retrieves the most similar parallel examples from a language-specific training bank, and Gemini 2.5 Flash translates the input conditioned on these examples. No model fine-tuning is involved. Training banks combine official WMT26 data with publicly available corpora such as Samanantar and prior WMT shared task releases. A grid search over retrieval count r and development exemplar count d across all 22 language-direction pairs selects the best configuration for each submission.
1 Introduction
The paper presents a BM25-plus-Gemini translation system for WMT26’s eleven North-Eastern Indian languages, including several with little prior machine-translation work. It adapts an earlier retrieval-augmented pipeline by removing image captioning and translating English source sentences directly.
- Task scope: WMT26 covers eleven North-Eastern Indian languages spanning Indo-Aryan, Austroasiatic, Tibeto-Burman, and English-based creole families.The task includes Assamese, Khasi, eight Tibeto-Burman languages, and Nagamese.
- Task scope: Nyishi, Karbi, and Tagin have almost no prior machine-translation work.
- Prior system: The system replaces fine-tuning with BM25 retrieval of relevant parallel examples in a long Gemini 2.5 Flash prompt for translation.This approach followed earlier work in which synthetic-data augmentation of mBART produced modest, domain-sensitive ChrF++ gains.
- WMT26 adaptation: For WMT26, the pipeline drops VLM image captioning and feeds English source sentences directly into the BM25-plus-Gemini translation process.Per-language prompts, the r,d hyperparameter sweep, and scoring infrastructure are retained.
2 Background
The WMT Low-Resource Indic shared task expanded from four language pairs in 2023 to eleven in 2026, spanning moderate- and very-limited-data categories. Earlier submissions mainly used fine-tuned multilingual models, whereas this submission uses BM25-selected demonstrations with Gemini API calls at inference time.
- Task setting: The WMT Low-Resource Indic shared task grew from four language pairs to eleven between 2023 and 2026.Languages are divided into Category 1, with moderate data, and Category 2, with very limited data.
- Task setting: Category 1 includes Assamese, Mizo, Khasi, Manipuri, and Nyishi, while Category 2 includes Bodo, Kokborok, Karbi, Nagamese, and Tagin.Manipuri appears in both Bengali script and Meitei Mayek.
- Prior work: Previous editions were dominated by fine-tuned multilingual models combined with backtranslation or transfer from higher-resource Indic languages.Examples include IndicTrans2, mBART, and NLLB-200.
- This submission: This submission instead allocates its translation budget to Gemini API calls and uses BM25 for example selection at inference time.The approach follows retrieval-augmented in-context translation work using parallel demonstrations.
3 System Description
The system uses BM25 retrieval over deduplicated, filtered multilingual banks to augment Gemini 2.5 Flash prompts for bidirectional translation. It searches retrieval and development-exemplar configurations across all 22 language-direction pairs, then applies script-aware post-processing.
- Retrieval-bank construction: Retrieval banks merge WMT26 data with selected WMT23 and public parallel corpora, using SHA-256 deduplication, Unicode normalization, length-ratio filtering, and optional LaBSE filtering.Bodo receives the largest external augmentation, while data collection emphasizes resource-constrained Category 2 languages.
- Retrieval: BM25 Okapi indexes the source side appropriate to translation direction, retrieves top-r pairs, skips exact matches, and deduplicates by 60-character normalized prefixes.English is indexed for En→X, whereas the Indic-language side is indexed for X→En, resolving an earlier directionality bug.
- Configuration selection: The prompt combines d word-overlap-ranked development exemplars with retrieved pairs, while a 4×4 grid over r and d is evaluated across all 22 directions.The grid uses r ∈ {0, 20, 40, 80}, d ∈ {0, 10, 20, 49}, and 50 held-out development sentences per configuration.
- Generation: Gemini 2.5 Flash generates translations with temperature 0, disabled thinking, a 512-token maximum, and language-specific prompts covering script, word order, and morphology.A two-key round-robin API pool with quota failover manages rate limits.
- Post-processing and architecture: Post-processing removes prefixes, normalizes Unicode, validates expected script blocks, and records 4 of 21,599 empty outputs caused by unresolved Gemini safety-filter retries.The architecture removes the AmericasNLP vision-captioning stage, adds bidirectional support, and merges external corpora into retrieval banks.
4 Languages
The shared task covers eleven North-Eastern Indian language tracks, eight of them Tibeto-Burman, alongside Assamese, Khasi, and Nagamese. Most languages use SOV order, while Khasi is the principal SVO exception; Manipuri is represented in two scripts.
- Language composition: Eight of the eleven language tracks are Tibeto-Burman, while Assamese, Khasi, and Nagamese are non-Tibeto-Burman.Assamese is Indo-Aryan, and Nagamese is an Assamese-lexified creole spoken across Nagaland.
- Word order: Most languages predominantly follow Subject–Object–Verb order, consistent with clause-final patterns across the region.Khasi, an Austroasiatic language, is the principal exception, with basic Subject–Verb–Object order.
- Scripts and tracks: Manipuri appears in two tracks using Bengali script and Meitei Mayek, representing scripts of the same language rather than genetically distinct languages.The paper refers to the Meitei Mayek track as Meitei elsewhere.
5 Experimental Setup
Experiments run on the University of Florida HiPer-Gator cluster using a CPU-only Gemini API pipeline. Development exemplars come from held-out training pairs, and a pilot grid search selects the configuration with the greatest ChrF score.
- Experimental Infrastructure: The pipeline runs CPU-only on HiPer-Gator’s hpg-default partition in a Python venv, while GPU nodes and mamba environments support NLLB-200 data generation.Evaluation uses SacreBLEU to measure ChrF and ChrF++.
- Development and Selection: The last 500 pairs or fewer from each language’s training data form development exemplars D, and each (r, d) cell is evaluated on n = 50 sentences.The submitted configuration achieves the greatest ChrF score.
6 Results
The results evaluate the method on Category 1 and Category 2 languages in both En→X and X→En directions. Performance is reported using ChrF, ChrF++, and organizer-published BLEU results.
- Development results: Table 3 reports contrastive performance for each language pair and translation direction on augmented development datasets before submission.The evaluation covers both Category 1 and Category 2 languages in En→X and X→En tasks.
- Development results: ChrF determines which retrieval-count r and development-exemplar-count d configurations are submitted, while ChrF++ is included for direct comparison.Both metrics are used to measure performance in the augmented development evaluation.
- Test results: Table 4 presents shared-task test performance for each language and translation direction using organizer-published BLEU and ChrF++ measurements.Additional test-time metrics are reported in the shared task findings paper.
7 Ablations
The ablations show that domain-matched retrieval, orthography, data filtering, and source-side indexing materially affect BM25-based RAG performance. Retrieval helps especially for En→X, while its benefit is weaker or absent for X→En under revised configurations.
- Synthetic augmentation: Domain match contributes more strongly than sheer volume, so MultiScript30k synthetic augmentation is excluded from all final retrieval banks.Synthetic data had been prepared for three of the eleven languages but was ultimately excluded.
- Script effect: 62.34 dev ChrF++ for Bengali-script Manipuri versus 36.86 dev… for Meitei Mayek shows a substantial orthography-associated gap in En→Mni translation.For X→En, performance drops from 67.86 to 52.85 ChrF++, representing roughly 25- and 15-point gaps for the same spoken language.
- Data Cleaning: 18 of 22 tasks change optimal r, d configurations after near-duplicate filtering and reranking by ChrF++, revealing strong sensitivity to data cleaning.The revised search finds r = 0 in seven of eleven X→En tasks, d = 49 in nine, and r = 80 in nine of eleven En→X tasks.
- Data Cleaning: r = 80 is optimal for most En→X languages, whereas r = 0 in X→En indicates retrieval may not help when generating a high-resource language.Karbi and Nagamese use r = 40 and have the smallest corpora, possibly limiting retrieval strength under extreme resource scarcity.
- Source-Side Retrieval Index: Correcting the X→En BM25 index to the Indic source side raises optimal retrieval from r = 0 for As→En and Kha→En under the fixed English index.The corrected index yields the configurations reported in Tables 3 and 4, indicating that retrieval finally improves translation for these languages.
8 Discussion
The discussion reports broad dev-to-test performance decay, especially for Bodo and Kokborok, while finding that corpus size and curated development exemplars matter more than retrieval-pair counts or retrieval count alone. It also identifies correlations and domain-similarity hypotheses that require further examination.
- Development-to-test differences: Nearly 30 and 40 points separate development and test performance for Bodo and Kokborok-related directions, particularly En→Bodo and Trp→En.The paper observes some dev-to-test decay for nearly every language and direction, with the largest differences involving Bodo and Kokborok.
- Development-to-test differences: Thinner or reversed development-to-test gaps may indicate closer domain alignment between the shared-task test and development sets, but this hypothesis requires dataset examination.The proposed explanation does not account for sizable gains from development to test data.
- Corpus size and augmentation: No significant correlation links retrieval-pair counts with ChrF++ or BLEU, while augmentation ratio correlates positively with dev En→X performance (r = 0.650, p < 0.05, m > 0).The correlation analysis uses least-squares linear regression with Wald t-tests.
- Retrieval and exemplar counts: For Category 2 languages, increasing curated development exemplars from d = 0 to d = 10 yields more gain than increasing retrieval count from r = 0 to r = 80.A significant positive correlation appears between dev d values and En→X ChrF++ (r = 0.625, p < 0.05, m > 0), but not between r and En→X performance.
- Retrieval and exemplar counts: 72.14 ChrF++ is reported in the search-grid analysis, while increasing r produces relatively flat gains for languages with fewer than 10,000 exemplars, including Karbi (899) and Tagin (4,421).Karbi is the only language with r < 80 among the contrastive En→X submissions, supporting the conclusion that larger retrieval corpora better reward higher r.
9 Conclusion
The Gators submission presents a training-free BM25- and Gemini 2.5 Flash-based system for translating between English and eleven Northeastern Indian languages. Across 19 translation tasks, it achieves the best BLEU score in nine and the best ChrF++ score in eleven, with small but measurable margins over competing submissions.
- System: The system uses BM25 retrieval and Gemini 2.5 Flash without training to translate between English and eleven Northeastern Indian languages.It adapts the AmericasNLP 2026 system while removing vision and constructing retrieval banks from different parallel text.
- Coverage: The submission covers all 22 language–direction pairs and uses a grid search alongside other ablations.The grid search spans every pair in both directions.
- Analysis: The study demonstrates the importance of domain matching for synthetic data augmentation and fixes a critical bug in BM-25 index construction.These changes address issues carried over from the previous shared task system.
- Results: In 19 translation tasks, the contrastive submission achieves the best BLEU score for nine and the best ChrF++ score for eleven.The submissions consistently outperform the competition by small yet measurable margins.
Limitations
The system is limited by dependence on Gemini 2.5 Flash, uneven retrieval-bank quality, and development metrics that overstate generalization because the development data shares the training domain.
- API dependence: Gemini 2.5 Flash API dependence caused burst-level 429 errors that consumed SLURM job windows and required careful scheduling with a 2-key API pool.These failures affected completion of all 22 directions’ grid cells during the pilot grid search.
- API dependence: 0.018% of test sentences (4 of 21,599) produced empty outputs because Gemini safety filters blocked specific political content.The blocked outputs occurred among the test sentences.
- Retrieval-bank quality: Retrieval-bank quality varies widely because Category 2 languages have limited parallel data and external corpora are often out-of-domain.For Nagamese, the only external corpus consisted of Bible translations; adding data does not help when BM25 cannot surface it.
- Evaluation: Development metrics overstate performance because the split shares its domain with training data and evaluates only 50 sentences per grid cell.The split uses the last 500 or fewer training pairs, and Guaraní dev scores roughly halved on the official test set in AmericasNLP; this paper reports drops for all languages except Bodo.
Appendix
The appendix provides supplementary implementation details for Gemini 2.5 Flash prompting and retrieval-hyperparameter selection. It also directs readers to public GitHub resources for prompt specifications and reproducible grid-search experiments.
- Appendix: The appendix covers Gemini 2.5 Flash prompt engineering and the r, d grid search used across experiments and language directions.These topics are organized in Sections 9.1 and 9.2.
- Prompting: Prompts target correct scripts and encodings for En→X translation and provide applicable guidance on Indic word order, tonality, and agglutination.Tables 6 and 7 document prompting strategies for Category 1 and Category 2 languages, respectively.
- Grid search: The search evaluates 16 (r, d) combinations from r ∈ {0, 20, 40, 80} and d ∈ {0, 10, 20, 49} on n = 50 WMT26 samples.ChrF and ChrF++ are recorded for each tuple, with ChrF designated as the primary metric.
- Grid search: Each language-direction submission uses the optimal r, d configuration selected through the primary ChrF grid search.The search is conducted for each language and translation direction.
- Reproducibility: Rather than tabulating 352 or 704 grid-search cells, the appendix refers readers to a GitHub bash script that can recreate the experiments using WMT26 data.The larger count reflects 22 translation tasks and, for 704 cells, a second search’s results.