PaperlayerView exact revision on arXiv

Source-linked AI summary

AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages

Anoop Kunchukuttan, Divyanshu Kakwani, Satish Golla, Gokul N. C., Avik Bhattacharyya, Mitesh M. Khapra, Pratyush Kumar

arXiv:2005.00085v1cs.CL

TL;DR

Indic languages lack large, publicly available monolingual corpora, despite their large speaker populations. The paper presents a 2.7-billion-word corpus for 10 languages and associated NLP resources. The authors report that its embeddings outperform publicly available embeddings on many tasks.

  • Problem

    Indic languages lack large, publicly available monolingual corpora despite being spoken by more than a billion people.

    Source: §1 · p. 1

  • Method

    The paper compiles contemporary, broad-topic corpora primarily from news and Wikipedia and trains FastText embeddings, alongside related language resources.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    The 2.7-billion-word IndicNLP corpus covers 10 languages, and its derived resources outperform other pre-trained embeddings and corpora on many NLP tasks.

    Source: §6 · p. 5

  • Takeaways & Limitations

    The authors hope public availability of the corpus will accelerate NLP research for Indian languages by enabling further resources and solutions.

    Source: §6 · p. 5

    Expanding major Indian-language collections to at least 1 billion words and building richer or multilingual representations and representative-task benchmarks remain work in progress.

    Source: §6 · p. 5

Abstract

from arXiv · show

We present the IndicNLP corpus, a large-scale, general-domain corpus containing 2.7 billion words for 10 Indian languages from two language families. We share pre-trained word embeddings trained on these corpora. We create news article category classification datasets for 9 languages to evaluate the embeddings. We show that the IndicNLP embeddings significantly outperform publicly available pre-trained embedding on multiple evaluation tasks. We hope that the availability of the corpus will accelerate Indic NLP research. The resources are available at https://github.com/ai4bharat-indicnlp/indicnlp_corpus.

1 Introduction

Indic languages lack large, publicly available monolingual corpora despite their broad use and linguistic diversity. This work builds corpora and related NLP resources for multiple Indian languages.

1 Introduction

  • Pre-trained representations can support NLP tasks and multilingual transfer, but their quality is impacted by monolingual corpus size.Large, diverse monolingual corpora are commonly used to train them.Source: §1 · p. 1
  • Indic languages are spoken by more than a billion people, yet lack large, publicly available monolingual corpora.They include 8 of the 20 most spoken languages and about 30 languages with more than a million speakers.Source: §1 · p. 1
  • Indic languages span four language families, with Indo-Aryan and Dravidian spoken by 96% of India's population.The languages are generally SOV and morphologically rich, and their long-term contact has produced linguistic convergence.Source: §1 · p. 1
  • The work contributes corpora for 10 languages across Indo-Aryan and Dravidian, FastText embeddings, classification datasets for 9 languages, and unsupervised morphanalyzers for 10.Each corpus has at least 100 million words except Oriya.Source: §1 · pp. 1–2

2 Related Work

The related-work discussion describes limited organized monolingual corpora for Indian languages and existing embeddings trained on relatively limited resources.

2 Related Work

  • Existing corpus sources include EMILLE/CIIL, small Indian-language Wikipedias, Leipzig collections, and language-specific corpora for Hindi and Urdu.EMILLE/CIIL spans 14 languages with 92 million words; the largest Indian-language Wikipedia, Hindi, has 40 million words.Source: §2 · p. 2
  • Processed CommonCrawl data for 2013–2016 had few good sentences for most Indian languages except Hindi, Tamil, and Malayalam, at around 50 million words.The authors report that corpus size likely had not changed significantly based on more recent CommonCrawl statistics.Source: §2 · p. 2
  • Polyglot and FastText provide Indian-language embeddings trained on Wikipedia, while FastText also offers embeddings trained on Wikipedia plus CommonCrawl.Embeddings had also been trained for many Indian languages using limited corpora.Source: §2 · p. 2

3 IndicNLP Corpus Creation

IndicNLP builds contemporary, broad-topic corpora primarily from news and Wikipedia, then processes and deduplicates the collected text. Hindi and Tamil are the largest collections, while Oriya is the smallest.

3 IndicNLP Corpus Creation

  • The corpus draws primarily on news websites and Wikipedia to reflect contemporary Indic-language use across a wide range of topics.Additional sources include Leipzig, WMT NewsCrawl, WMT CommonCrawl, and HindEnCorp for specified languages.Source: §3 · p. 2
  • Hindi and Tamil are the largest collections, Oriya is the smallest, and the other languages each have 100–200 million words.Bengali has around 100 million words; the authors would like to increase that collection.Source: Table 1 · p. 2, §3 · p. 3
  • Processing includes article extraction, quality filters, Unicode canonicalization, sentence splitting, and tokenization tailored to Indic text.The final collection is deduplicated with a 128-bit Murmurhash and shuffled.Source: §3 · pp. 2–3
  • CommonCrawl contributes 55% of Tamil and 35% of Malayalam data, while most other languages' data come from the authors' crawls.The Hindi corpus compiles existing sources.Source: §3 · p. 3

4 IndicNLP Word Embeddings

IndicNLP embeddings are trained on the new corpora and evaluated on intrinsic, text-classification, and bilingual lexicon induction tasks. They outperform baselines on average in word similarity and analogy, on nearly all classification datasets and languages, and in bilingual lexicon induction except Bengali.

4 IndicNLP Word Embeddings

  • The evaluation compares IndicNLP embeddings with FastText embeddings trained on Wikipedia and on Wikipedia plus CommonCrawl.Tasks include word similarity, word analogy, text classification, and bilingual lexicon induction.Source: §4 · p. 3

4.1 Training Details

  • The embeddings are 300-dimensional FastText skipgram models trained on each IndicNLP corpus for 10 epochs.Training uses a window size of 5, minimum token count of 5, and 10 negative examples; FastText incorporates character n-grams.Source: §4.1 · p. 3

4.2 Word Similarity & Analogy Evaluation

  • IndicNLP embeddings outperform baseline embeddings on average in word similarity and Hindi word analogy evaluations.Word similarity covers seven Indian languages; Table 2 reports the similarity and analogy results.Source: §4.2 · p. 3, Table 2 · p. 3

4.3 Text Classification Evaluation

  • IndicNLP embeddings outperform baseline embeddings on nearly all text-classification datasets and languages.The evaluation includes news-topic, headline-topic, and sentiment classification, including the new IndicNLP News Category dataset.Source: §4.3 · p. 4, Table 4 · p. 4, Table 5 · p. 4

4.4 Bilingual Lexicon Induction

  • IndicNLP bilingual embeddings significantly outperform baseline bilingual embeddings, except Bengali, using MUSE and in-house English–Telugu bilingual dictionaries and CSLS distance to search the 200k most frequent target-language words during inference.Table 6 reports Accuracy@1 for bilingual lexicon induction.Source: §4.4 · p. 4, Table 6 · p. 4

5 Unsupervised Morphology Analyzers

The paper trains unsupervised Morfessor analyzers from monolingual corpora and tests their usefulness in phrase-based SMT between Indic languages.

Motivation and SMT evaluation

  • The paper frames morphology analysis as a way to factor words into morphemes amid large-vocabulary data sparsity and limited analyzer coverage or quality.The SMT evaluation uses the ILCI parallel corpus, with about 50,000 sentences per Indian-language pair.Source: §5 · pp. 4–5

Results

  • Morph-level SMT using the IndicNLP-trained analyzer significantly outperforms the word-level model and beats the comparison analyzer in most cases.The comparison analyzer is from Kunchukuttan and Bhattacharyya (2016) and was trained on data containing the parallel corpus itself, which may make it more tuned to the task.Source: §5 · p. 5

Training

  • The analyzers are trained with Morfessor 2.0 on word types occurring at least five times, without using their frequencies.This configuration is recommended when annotated data is unavailable for tuning.Source: §5 · p. 5

6 Summary and Future Work

The paper presents a large general-domain corpus and derived resources for Indic NLP, reporting that they outperform other pre-trained embeddings and corpora on many tasks. It plans further corpus expansion and additional representations and benchmarks.

Summary

  • IndicNLP presents a 2.7-billion-word corpus across 10 Indian languages, alongside embeddings, morphanalyzers, and text-classification benchmarks.The paper reports that resources derived from the corpus outperform other pre-trained embeddings and corpora on many NLP tasks.Source: §6 · p. 5

Future Work

  • The authors are working to expand the collection to at least 1 billion words for major Indian languages.Planned work also includes BERT and ELMo representations, multilingual representations, and benchmark datasets for representative tasks.Source: §6 · p. 5
  • The planned representations and benchmarks are described as work in progress, while the authors hope corpus availability will accelerate Indian-language NLP research.Source: §6 · p. 5

A Publicly Available Text Classification Datasets

The paper lists publicly available datasets used in its text-classification experiments and points to a table of their statistics.

Datasets

  • The experiments use review and sentiment datasets, news-article classification datasets, a headlines dataset, and the Soham Bengali news corpus.The listed sources include IIT-Patna Movie and Product reviews, ACTSA, IIT-Bombay sentiment analysis, BBC News, iNLTK Headlines, and Soham Bengali News.Source: A Publicly Available Text… · pp. 6–7

Dataset statistics

  • Table 8 summarizes statistics for the publicly available datasets, with N denoting the number of classes.Source: Table 8 · p. 7

Some notes on the above mentioned public datasets

The dataset notes describe merging review datasets, simplifying their class labels, and limiting Telugu evaluation to the news-line data because other domains had few data points.

Related papers

Loading 2005.00085v1…