Source-linked AI summary
Word Similarity Datasets for Indian Languages: Annotation and Baseline Systems
Syed S. Akhtar, Arihant Gupta, Avijit Vajpayee, Arjit Srivastava, M. Shrivastava
TL;DR
Word-similarity evaluation resources were lacking for computationally resource-poor Indian languages, despite the task's use in evaluating word representations. The paper constructs manually annotated datasets for six languages from translated English datasets and reports baseline evaluations, including better performance by FastText with OOV handling than FastText alone.
Problem
Word-similarity test datasets were largely unavailable for Indian languages, limiting evaluation resources for word representations in these computationally resource-poor languages.
Source: §1 · p. 1Method
The paper translates English word-similarity datasets into six Indian languages and has native speakers rate the translated pairs.
Source: §4.1 · p. 2, §4.2 · p. 2Results
FastText w/ OOV performed better than FastText in all experiments, with character-based models generating embeddings for unseen words.
Source: §6 · p. 3Takeaways & Limitations
The six datasets provide evaluation resources intended to support fast comparison and development of word-representation techniques for the covered languages.
Source: §1 · p. 1
Abstract
from arXiv · showhide
With the advent of word representations, word similarity tasks are becoming increasing popular as an evaluation metric for the quality of the representations. In this paper, we present manually annotated monolingual word similarity datasets of six Indian languages - Urdu, Telugu, Marathi, Punjabi, Tamil and Gujarati. These languages are most spoken Indian languages worldwide after Hindi and Bengali. For the construction of these datasets, our approach relies on translation and re-annotation of word similarity datasets of English. We also present baseline scores for word representation models using state-of-the-art techniques for Urdu, Telugu and Marathi by evaluating them on newly created word similarity datasets.
1 Introduction
Word similarity benchmarks evaluate word representations by correlating human judgments of semantic similarity with vector-based comparisons, but such datasets were largely unavailable for Indian languages. The paper creates datasets for six Indian languages and reports baseline scores for word-representation models.
- Word similarity evaluates representations by correlating human-assigned semantic similarity with corresponding word vectors.The paper describes the task as computationally efficient and notes applications including word sense disambiguation, machine translation evaluation, question answering, and lexical substitution.Source: §1 · p. 1
- Word-similarity evaluation was largely limited to English, and no such test datasets existed for resource-poor Indian languages including Marathi, Punjabi, and Telugu.Source: §1 · p. 1
- The paper introduces six Indian-language word-similarity datasets and baseline scores, aiming to provide evaluation resources for languages ranked among the world's 25 most spoken.The authors say the datasets would allow fast, efficient comparison and promote better techniques using word representations for these languages.Source: §1 · p. 1
2 Datasets
The paper trains word embeddings with Skip-gram, CBOW, and FastText and evaluates them on newly created similarity datasets. Corpus descriptions are provided for Marathi, Urdu, and Telugu; the evaluation uses Spearman correlation and assigns zero vectors to missing words.
- The models use Skip-gram, CBOW, and FastText with 300 dimensions, minimum count 5, and negative sampling.Source: §2 · p. 2
- Training corpora comprise 27 million Marathi tokens, 95 million Urdu tokens, and 11 million Telugu tokens.The sources are the IIT-Bombay monolingual corpus, Jawaid et. al. (2014), and a Telugu Wikidump, respectively.Source: §2 · p. 2
- The six test sets contain 97–163 word pairs, and scores are Spearman ρ × 100 between human ratings and embedding cosine similarity.Missing words receive zero vectors; the set sizes are Urdu 100, Marathi 104, Telugu 111, Punjabi 143, Gujarati 163, and Tamil 97.Source: §2 · p. 2
- Word2vec embeddings are trained on the training set using the parameters described for the models.Source: §2 · p. 2
3 Related Work
Prior word-similarity resources and multilingual adaptations provide the context for the paper's dataset construction. RG-65 and WordSim-353 are the English datasets used as its bases.
- English word-similarity resources include WordSim-353, MC-30, Simlex-999, and RG-65; RG-65 is described as an old, popular benchmark.The cited sources are WordSim-353, MC-30, Simlex-999, and RG-65.Source: §3 · p. 2
- RG-65 served as a basis for word-similarity datasets in French, German, Portuguese, Spanish, and Farsi.The cited works are Joubarne and Inkpen (2011), Zesch and Gurevyc (2006), Granada et al. (2014), and Camacho-Collados et al. (2015).Source: §3 · p. 2
- Reported inter-annotator agreement was 0.81 for German, 0.71 for Portuguese, 0.83 for Spanish, and 0.88 for Farsi; French agreement was not calculated.The paper's datasets use RG-65 and WordSim-353 as their bases, with their agreement reported later.Source: §3 · p. 2
4 Construction of Monolingual Word Similarity datasets
Dataset construction combines translation by native speakers fluent in English with similarity ratings from native speakers of each target language. The authors report good evaluator agreement except for relatively weaker agreement on Tamil.
4.1 Translation
- Three target-language native speakers fluent in English translated the English word pairs, with a third annotator resolving disparities between the first two.The annotators finalized translated pairs without repetitions, following an approach similar to Camacho-Callados et al. (2015).Source: §4.1 · p. 2
4.2 Scoring
- Eight native speakers per language rated each word pair's meaning similarity from 0 to 10, and the averaged ratings showed good agreement except for weaker Tamil agreement.The instructions distinguished similarity from relatedness and were based on the SemEval Cross-Level Semantic Similarity task.Source: §4.2 · p. 2
5 Evaluation
The evaluation considers how consistently annotators assign word-similarity ratings and uses Fleiss’ Kappa to assess agreement across the six datasets.
5.1 Inter Annotator Agreement (IAA)
- Annotators may interpret sentence and word meanings differently, so annotation can remain subjective even with defined guidelines.Inter-annotator agreement measures how consistently annotators make the same decision.Source: §5.1 · pp. 2–3
5.1.1 Fleiss’ Kappa
- Fleiss’ Kappa measures agreement among a fixed number of raters classifying items, relative to the agreement expected by chance.Source: §5.1.1 · p. 3
- Fleiss’ Kappa was calculated for all six word-similarity datasets, with Table 1 presenting the scores.Source: §5.1.1 · p. 3, Table 1 · p. 3
6 Result and Analysis
FastText with OOV handling performed better than FastText in all experiments; the authors relate this to character-based embeddings handling unseen words. The tested word pairs include some rarity.
- CBOW, Skipgram, and FastText-SG baselines were evaluated on the Urdu, Marathi, and Telugu datasets in Tables 2–4.Source: §6 · p. 3, Table 2 · p. 3, Table 3 · p. 3
- The word-similarity sets contain some rare pairs, and the trained models encountered unseen word pairs during evaluation.Source: §6 · p. 3
- FastText w/ OOV performed better than FastText in all experiments, which the authors attribute to character models generating embeddings for unseen words.Source: §6 · p. 3
7 Future Work
Future work proposes expanding word-similarity resources and improving word representations, including by building more training corpora.
- The authors describe this work as a small step toward research on word representations for computationally resource-poor Indian languages.Source: §7 · p. 3
- The authors plan to create rare-word similarity datasets for the six languages and datasets for other major Indian languages.Source: §7 · p. 3
- They plan to improve representations and baseline scores by building new corpora for Punjabi, Tamil, and Gujarati and expanding corpora for Urdu, Telugu, and Marathi.Source: §7 · p. 3