Source-linked AI summary
MuRIL: Multilingual Representations for Indian Languages
Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, Partha Talukdar
TL;DR
Existing multilingual language models perform suboptimally on Indian languages, especially with limited representation and transliterated text. MuRIL addresses this gap with Indian-language corpora, translated and transliterated pairs, and dedicated pre-training. It significantly outperforms mBERT across XTREME tasks and demonstrates efficacy on transliterated test sets.
Problem
Multilingual language models underrepresent Indian languages and inadequately handle Indian-language text transliterated into Latin script.
Method
MuRIL is trained on Indian-language corpora using MLM for monolingual text and TLM for translated and transliterated document pairs.
Results
MuRIL significantly outperforms mBERT on all tasks in XTREME and demonstrates efficacy on transliterated test sets.
Takeaways & Limitations
MuRIL supports better technologies and applications for Indian languages within the evaluated cross-lingual and transliterated settings.
Abstract
from arXiv · showhide
India is a multilingual society with 1369 rationalized languages and dialects being spoken across the country (INDIA, 2011). Of these, the 22 scheduled languages have a staggering total of 1.17 billion speakers and 121 languages have more than 10,000 speakers (INDIA, 2011). India also has the second largest (and an ever growing) digital footprint (Statista, 2020). Despite this, today's state-of-the-art multilingual systems perform suboptimally on Indian (IN) languages. This can be explained by the fact that multilingual language models (LMs) are often trained on 100+ languages together, leading to a small representation of IN languages in their vocabulary and training data. Multilingual LMs are substantially less effective in resource-lean scenarios (Wu and Dredze, 2020; Lauscher et al., 2020), as limited data doesn't help capture the various nuances of a language. One also commonly observes IN language text transliterated to Latin or code-mixed with English, especially in informal settings (for example, on social media platforms) (Rijhwani et al., 2017). This phenomenon is not adequately handled by current state-of-the-art multilingual LMs. To address the aforementioned gaps, we propose MuRIL, a multilingual LM specifically built for IN languages. MuRIL is trained on significantly large amounts of IN text corpora only. We explicitly augment monolingual text corpora with both translated and transliterated document pairs, that serve as supervised cross-lingual signals in training. MuRIL significantly outperforms multilingual BERT (mBERT) on all tasks in the challenging cross-lingual XTREME benchmark (Hu et al., 2020). We also present results on transliterated (native to Latin script) test sets of the chosen datasets and demonstrate the efficacy of MuRIL in handling transliterated data.
1 Why MuRIL?
MuRIL is a multilingual language model built specifically for Indian languages to address weaknesses in existing multilingual systems. It uses Indian-language data and translated or transliterated document pairs, and outperforms mBERT across XTREME tasks while handling transliterated inputs.
- MuRIL is a multilingual language model specifically built for Indian languages.
- MuRIL is trained on large Indian-language text corpora and augmented with translated and transliterated document pairs.
- Existing multilingual models show significant performance differences between English and other Indian-language NER test sets.
- MuRIL significantly outperforms mBERT on all tasks in the cross-lingual XTREME benchmark.
- MuRIL demonstrates efficacy on test sets containing Indian-language text transliterated from native scripts to Latin.
2 Model and Data
MuRIL combines Indian-language data, an upsampled training distribution, an India-focused vocabulary, and MLM/TLM pre-training. Its data and vocabulary are designed to represent Indian languages and transliterated text more effectively.
- MuRIL supports 17 languages, comprising 16 Indian languages and English.The Indian languages include Assamese, Bengali, Gujarati, Hindi, Kannada, Kashmiri, Malayalam, Marathi, Nepali, Oriya, Punjabi, Sanskrit, Sindhi, Tamil, Telugu, and Urdu.
- MuRIL uses MLM on monolingual documents and TLM on translated and transliterated document pairs.MLM is unsupervised and uses monolingual text, whereas TLM is supervised and uses parallel data.
- Monolingual data comes from Common Crawl OSCAR and Wikipedia, while translated and transliterated pairs provide parallel training data.Translated sources include PMINDIA and in-house translations; transliterated sources include Dakshina and indic-trans.
- Upsampling adjusts each language’s monolingual Wikipedia representation using a multiplier based on its original token count.The multiplier uses all 17 languages and sets α to 0.3; the upsampled token count is m_i ∗n_i.
- MuRIL learns a cased WordPiece vocabulary from upsampled data, with a final vocabulary size of 197,285.The vocabulary preserves case to avoid stripping accents and accounts for Indian-language and transliterated words.
- mBERT has a higher fertility ratio than MuRIL because Indian languages have little representation in its vocabulary and transliterated words are not accounted for.A higher fertility ratio means more sub-words per word and can lead to loss of semantic meaning.
3 Evaluation
MuRIL is evaluated for cross-lingual understanding in a zero-shot XTREME setting, using English training data and Indian-language test sets. Results compare MuRIL with mBERT on native-script and transliterated test sets, with identical fine-tuning hyperparameters.
- Evaluation setup: Zero-shot evaluation fine-tunes models on English labeled data and evaluates them on Indian-language test sets.The goal is cross-lingual understanding across XTREME tasks.
- Evaluation setup: XTREME covers 40 typologically diverse languages, 12 language families, and 9 syntax- or semantics-focused tasks.
- Native-script results: MuRIL significantly outperforms mBERT on all XTREME datasets for the supported Indian languages.The reported values are averages over test sets for those supported languages.
- Transliterated results: Transliterated evaluation converts Indian-language test sets from native scripts to Latin and reports average performance across the resulting test sets.
- Evaluation setup: The same fine-tuning hyperparameters are used for evaluating mBERT and MuRIL.For TyDiQA-GoldP, SQuAD English training data is added; Tatoeba uses pooled last-layer outputs without fine-tuning.
4 How to use MuRIL?
MuRIL is made available through TensorFlow Hub and Hugging Face, with preprocessing support for the encoder. The released resources are intended to support applications for Indian languages.
- Available resources: The MuRIL encoder is released on TensorFlow Hub with detailed usage instructions.
- Available resources: A preprocessing module converts raw text into the encoder’s expected input format.
- Available resources: The pretrained MuRIL model, including its MLM layer for masked word predictions, is released on Hugging Face.
- Intended use: The released tools are intended to aid the development of technologies and applications for Indian languages.
A Pre-training Data Statistics
The appendix reports upsampled token counts for each language and corpus in Table 4.
- Data statistics: Table 4 reports upsampled token counts for each language and corpus.
B Detailed Results
Detailed results are reported separately for each supported language across the XTREME datasets, including native-script and transliterated test sets.
- Native-script results: Per-language results are reported for PANX, UDPOS, XNLI, Tatoeba, XQuAD, MLQA, and TyDiQA-GoldP.These results appear in Tables 5–10.
- Transliterated results: Detailed transliterated test-set results are reported for PANX, UDPOS, XNLI, and Tatoeba.These results appear in Tables 11–14.
C Analysis
MuRIL’s sample predictions show stronger use of context across named entities, sentiment, and question answering, including transliterated and code-mixed inputs. The examples contrast these behaviors with mBERT’s reliance on individual words or surface forms.
- Named Entity Recognition: MuRIL uses context to classify ambiguous named entities, while mBERT often relies on misleading location words.MuRIL labels Atlanta Falcons as ORG and Shirdi’s Sai Baba as PER; mBERT predicts LOC in both cases.
- Named Entity Recognition: MuRIL captures entity meaning from surrounding qualifiers, correctly recognizing Bajirao Mastani as a film.
- Named Entity Recognition: MuRIL correctly classifies misspelled words in the analyzed examples.
- Sentiment Analysis: MuRIL correctly predicts positive sentiment in a Hindi sentence containing an English loanword and negation, and in a transliterated sentence.The examples include “account” with negation and the transliterated sentence about Ramu and the film’s pace.
- Question Answering: In question answering, MuRIL resolves cross-script references and treats “bank ki paribhasha” as one entity whose definition varies across countries.It connects the Hindi translation of “Greek” in context with its transliteration in the question.