Source-linked AI summary
Statistical Machine Translation Systems of English-Pnar Language Pair : Some Insights of the Emperical Study
Edawanbiang Dhar Surmila Thokchom, Thoudam Doren Singh
TL;DR
Pnar lacks digital corpora and NLP resources, and its English–Pnar language pair had no prior computational MT study. The paper constructs a Wyrta-based parallel corpus and evaluates phrase-based SMT configurations, finding benchmark translation performance and direction-dependent effects of reordering and MERT. Remaining errors include morphological OOVs, long-distance reordering, and Khasi code-mixing.
Problem
Pnar is an under-resourced language with scarce computational resources, and no prior computational MT research had been reported for English–Pnar.
Method
The study builds a Wyrta-based parallel corpus and compares phrase-based SMT systems using Moses, GIZA++, KenLM, lexicalized reordering, and MERT.
Results
The best systems achieve BLEU 14.97 for Pnar→English and BLEU 11.16 for English→Pnar, establishing the first published baseline.
Takeaways & Limitations
Lexicalized reordering improves Pnar→English by +3.73 BLEU, while MERT degrades BLEU in both directions under the low-resource setting.
Takeaways & Limitations
Residual errors arise from morphological OOV words, long-distance reordering, and Khasi code-mixing.
Abstract
from arXiv · showhide
Pnar, an Austroasiatic language spoken by approximately 0.4 million people in the Jaintia Hills of Meghalaya, lacks the digital corpora and natural language processing (NLP) resources. This paper presents the first machine translation study for the English and Pnar language pair. Using articles collected from the Wyrta newspaper, we built a parallel corpus comprising of 10,234 sentences and trained phrase-based statistical machine translation (SMT) systems the models using 9,563 parallel corpora under three configurations for each direction using Moses, GIZA++ , KenLM, varying lexicalized reordering and minimum error rate training (MERT) tuning. The models are evaluated on a held out test set of 371 sentences, the best performing system achieves a BLEU score of 14.97 (chrF2: 33.42, TER: 77.60) for Pnar to English and 11.16 (chrF2: 31.38, TER: 93.51) for English to Pnar, establishing the first quantitative benchmark for this language pair. Lexicalized reordering improves translation quality by 3.73 BLEU points for Pnar to English, reflecting the structural shift from the source language's SOV word order to the target language's SVO order, whereas MERT tuning degrades BLEU performance under low resource conditions. Finally, we analyze the remaining translation errors, including morphological out of vocabulary (OOV) words, long-distance reordering and Khasi code mixing and discuss future directions toward neural and multilingual machine translation for Pnar.
1 Introduction
Pnar is a severely under-resourced Austroasiatic language that remains largely unexplored in computational research. The paper situates translation technology as a response to Pnar’s scarcity of computational resources.
- Pnar is a severely under-resourced language spoken by the Jaintia community in Meghalaya.
- Despite linguistic proximity to Khasi, Pnar remains largely unexplored in computational research.
- Pnar’s limited computational resources reflect broader challenges faced by low-resource languages in natural language processing.
- Developing language technologies is presented as supporting Pnar’s preservation, digital access, education, and communication.
2 Related Work
Prior low-resource MT work has addressed several Northeast Indian languages, but Pnar remains outside systematic computational investigation. Its distinct word order, morphology, code-mixing, and scarce digital resources motivate a dedicated study.
- Low-resource MT research has covered Manipuri, Mizo, Bodo, and Khasi, but not Pnar.
- Pnar differs from previously studied languages through SOV order, agglutinative verbal morphology, and frequent Khasi code-mixing.
- Pnar also faces acute scarcity of digital resources, including limited monolingual text.
3 Dataset Preparation
The corpus was assembled from contemporary Wyrta newspaper content and processed through a standard SMT preparation pipeline. Cleaning and splitting produced separate training, development, and held-out test sets.
- Pnar data were collected from the Wyrta newspaper, covering community news, governance, cultural events, sports, and health.
- Preprocessing lowercased and tokenised text, normalised punctuation, and manually corrected the 500 most frequent tokens.
- Sentences exceeding 80 tokens were removed to improve alignment quality.
- The cleaned corpus contains 9,563 training sentences, 300 development sentences, and 371 held-out test sentences.
4 Experimental Setup
The study trains phrase-based SMT systems that combine translation, language, alignment, reordering, and tuning components. The architecture searches for target sentences using combined model scores, while lexicalized reordering addresses Pnar–English word-order divergence.
- The Moses-based pipeline uses GIZA++ alignment, phrase extraction, and a 5-gram KenLM target-language model.
- The SMT formulation searches for the most probable target sentence using translation-model and language-model probabilities.
- The decoder uses beam search over the combined model score to produce an optimal hypothesis.
- Lexicalized Reordering: Lexicalized reordering models monotone, swap, discontinuous-left, and discontinuous-right phrase orientations.
- Configurations and Tuning: MERT tunes log-linear feature weights on development data to optimise BLEU, with up to 25 iterations or convergence.
- Configurations and Tuning: The experiments vary lexicalized reordering and MERT tuning across three SMT configurations per translation direction.
5 Experiments and Results
Experiments compare phrase-based SMT configurations for English–Pnar translation, examining lexicalized reordering, MERT tuning, statistical significance, and residual errors. Lexicalized reordering substantially benefits Pnar→English, whereas MERT tuning harms BLEU under the low-resource setting.
- Evaluation setup: Tables 3(a) and 3(b) report translation results for all configurations on the Pnar→English and English→Pnar test sets.
- Lexicalized reordering: +3.73 BLEU and +5.98 chrF2 are gained by lexicalized reordering for Pnar→English, reflecting the required SOV→SVO transformation.The corresponding TER improvement is −14.10.
- Lexicalized reordering: Lexicalized reordering improves Pnar→English but produces a marginal −1.02 BLEU change for English→Pnar.The with-reordering English→Pnar system produces genuine Pnar output and is treated as the better baseline despite the score comparison.
- MERT tuning: MERT tuning decreases BLEU by −2.50 for Pnar→English and −3.23 for English→Pnar while marginally improving chrF2.The tuned systems over-generate, suggesting degenerate weight optimization under the small-corpus setting.
- Statistical significance: +3.72 BLEU is the statistically significant Pnar→English reordering gain, with a 95% CI of [3.16, 4.30] and p < 0.001.The MERT decrease is also significant at −2.50 BLEU, with a 95% CI of [−3.24, −1.75] and p < 0.001.
- Error analysis: The remaining failure modes are OOV tokens, long-distance reordering, Khasi code mixing, and morphological complexity.Agglutinative morphology and complex verb-final constructions exceed phrase-table coverage in the 9,563-sentence training corpus.
6 Conclusion and Future Work
This work establishes the first English–Pnar machine translation study and published baseline, using a parallel corpus and phrase-based SMT configurations to quantify translation performance. Lexicalized reordering substantially improves Pnar→English, while MERT tuning degrades BLEU under the current data scale.
- Contributions: The study establishes the first published English–Pnar translation baseline using BLEU, chrF2, and TER.It uses 9,563 sentence pairs from the Wyrta newspaper and compares six phrase-based SMT configurations.
- Findings: 3.73 BLEU points: lexicalized reordering improves Pnar→English translation by modeling Pnar’s SOV-to-English SVO structural transformation.
- Findings: 14.97 BLEU: the best Pnar→English system, compared with 11.16 BLEU for the best English→Pnar system.
- Findings: MERT tuning consistently degrades BLEU performance under the current low-resource data scale.The authors suggest that limited development data makes MERT prone to overfitting.