Source-linked AI summary
Neural POS Tagging for the Low-Resource Saraiki
Usman Nawaz, Muhammad Junaid Iqbal, Muhammad Asaf, Mariam Munsif Mir, Faisal Rehman, Tahir Alyas, Muhammad Hasnain
TL;DR
Saraiki remains under-resourced in computational linguistics, with limited annotated benchmarks and publicly available NLP tools. This paper presents a manually annotated UPOS dataset and evaluates pretrained encoders under a consistent setup; XLM-RoBERTa base achieves the strongest overall performance among those evaluated.
Problem
Saraiki has limited annotated benchmark datasets and publicly available NLP tools, and its dialectal and lexical variation makes direct evaluation on Saraiki data important.
Source: §1 · p. 2Method
The paper manually annotated Saraiki text using the Universal POS inventory and established transformer baselines under a consistent token-classification setup.
Source: §8 · p. 12Results
XLM-RoBERTa base provides the strongest overall performance among the evaluated models.
Source: §7 · p. 11, §8 · p. 12Takeaways & Limitations
The dataset and baseline results provide a benchmark resource for future Saraiki NLP and low-resource sequence-labeling research.
Source: §8 · p. 12The annotation process did not independently annotate the same sentences from scratch with multiple annotators, and inter-annotator agreement was not measured.
Source: Limitations · p. 13
Abstract
from arXiv · showhide
Part-of-speech tagging is a fundamental sequence labeling task and an important component for many downstream natural language processing applications. However, Saraiki, written from right to left in a Perso-Arabic script, remains an under-resourced Indo-Aryan language, despite being spoken as a mother tongue by 28,849,579 people (approximately 28.9 million) according to Pakistan 7th Population and Housing Census 2023. This paper presents a manually annotated Saraiki Universal POS tagging dataset developed from plain-text. We annotated the text with 17 POS categories. We evaluate three pretrained transformer encoders under the same fine-tuning setup: XLM-RoBERTa base, Urdu RoBERTa small, and multilingual DistilBERT. Experimental results over five random seeds show that XLM-RoBERTa base achieves the best performance, with 87.29% +/- 0.21% development accuracy and 88.60% +/- 0.21% test accuracy. Urdu RoBERTa small and multilingual DistilBERT also provide competitive results, suggesting that multilingual and related-script pretrained encoders are effective for Saraiki UPOS tagging. The dataset, evaluation setup, and baseline results introduced in this work provide a benchmark for future research on Saraiki NLP and low-resource sequence labeling.
1 Introduction
Saraiki has a large speaker community but limited annotated resources and reproducible UPOS benchmarks. This paper introduces a manually annotated dataset and evaluates pretrained encoders for Saraiki tagging.
1 Introduction
- Tagged corpora provide a basic layer for building and evaluating NLP tools, while dialectal and lexical variation can affect vocabulary coverage, tokenization, and generalization.The paper argues that models should be evaluated directly on Saraiki rather than assumed to transfer from related languages.Source: §1 · p. 2
- Public Saraiki UPOS resources and reproducible benchmarks remain limited, despite Saraiki’s sizeable speaker community and existing linguistic annotation work.Saraiki is reported as the mother tongue of approximately 28.9 million people.Source: §1 · p. 2
- The dataset uses 17 Universal POS categories and fixed train, development, and test splits to support reproducible evaluation.UPOS annotation makes the resource compatible with multilingual NLP research and UD-style resources for related Indo-Aryan languages.Source: §1 · p. 2
- The paper evaluates XLM-RoBERTa base, Urdu RoBERTa small, and multilingual DistilBERT under a consistent setup; XLM-RoBERTa base performs best and the other encoders provide competitive baselines.The results suggest that multilingual pretraining and related-script representations are useful for Saraiki sequence labeling.Source: §1 · p. 3
2 Related Work
Prior work provides Saraiki annotation foundations and relevant transformer approaches, but benchmark-style evaluation of pretrained encoders for Saraiki UPOS tagging remains limited. This paper addresses that gap with manually annotated data and a fixed evaluation setup.
2 Related Work
- Universal POS and Universal Dependencies support cross-linguistic annotation, but coverage remains uneven across languages and annotated corpora remain limited for many low-resource languages.UD treebanks exist for Hindi, Urdu, Marathi, and Punjabi, while Saraiki-specific tagsets document richer language-specific categories.Source: §2 · pp. 3–4
- Earlier Saraiki POS tagsets capture language-specific grammatical categories, whereas UPOS supports multilingual comparison and compatibility with Universal Dependencies-style resources.Source: §2 · p. 4
- Saraiki dialectal lexical variation can affect vocabulary coverage, tokenization, and model generalization, including through word forms unevenly represented in training data.Source: §2 · p. 4
- Multilingual transformers and Urdu-focused models are relevant candidates because Saraiki shares script-level similarities with Urdu and Punjabi Shahmukhi.The related work describes XLM-RoBERTa as multilingual and DistilBERT as a smaller, faster distilled model.Source: §2 · p. 4
- Benchmark-style evaluation of pretrained transformer encoders for Saraiki UPOS tagging remains limited, motivating manual annotation and evaluation under fixed train, development, and test splits.Earlier work includes Saraiki POS tagsets and a small Universal Dependencies treebank.Source: §2 · p. 4
3 Dataset Construction
The dataset preserves Saraiki text in its original script, manually annotates tokens with Universal POS labels, and supplies fixed splits for reproducible evaluation.
3.1 Data Source
- Saraiki source text in Perso-Arabic script was retained without transliteration, and the plain-text data was manually processed because it lacked token boundaries and linguistic annotations.Source: §3.1 · pp. 4–5
3.2 Preprocessing
- The text was cleaned and whitespace-normalized, then manually segmented into tokens and formatted in CoNLL-U with UPOS labels while XPOS, morphological-feature, and dependency fields were left unspecified.Source: §3.2 · p. 5
3.3 UPOS Label Inventory
- The annotation uses 17 Universal POS categories, assigning each token one label and distinguishing categories such as NOUN and PROPN, VERB and AUX, and ADP.The X label is used for tokens that cannot be assigned to a regular grammatical category.Source: §3.3 · p. 5
3.4 Manual Annotation
- One linguistics-trained annotator labeled tokens in context, and two PhD students reviewed the full dataset and made manual corrections.The annotation follows UPOS rather than a language-specific tagset to support cross-linguistic comparability and use with multilingual NLP models and UD-oriented tools.Source: §3.4 · pp. 5–6
3.5 Data Splits
- Fixed train, development, and test splits assign fine-tuning, checkpoint selection, and final evaluation roles, respectively, to support future comparisons under the same setting.Table 1 reports sentence and token distributions across the splits.Source: §3.5 · p. 6, Table 1 · p. 6
4 Methodology
Saraiki UPOS tagging is formulated as assigning one of 17 Universal POS labels to each token. The model uses a shared transformer-and-classifier architecture to produce token-level predictions.
4 Methodology
- Each token in an input sentence receives exactly one label from the corpus's 17 Universal POS categories.Source: §4 · pp. 6–7
- Table 2 reports token counts by UPOS tag for each data split.Source: Table 2 · p. 7
4.1 Model Architecture
- Contextualized token representations pass through dropout and a linear classification head to produce UPOS logits.Source: §4.1 · p. 7
- All encoders use the same architecture, and the encoder and classification-head parameters are fine-tuned.Source: §4.1 · p. 7
- Training uses cross-entropy loss over the labeled token positions.Source: §4.1 · p. 8
5 Experimental Setup
The experiments compare three pretrained encoders using fixed Saraiki UPOS splits and five random seeds. Evaluation centers on accuracy, with additional per-class and confusion-matrix analyses for the selected XLM-RoBERTa checkpoint.
5 Experimental Setup
- Experiments use the fixed splits and five random seeds, reporting mean and sample standard deviation of accuracy.Source: §5 · p. 8
5.1 Models
- The comparison includes XLM-RoBERTa base, Urdu RoBERTa small, and multilingual DistilBERT.XLM-RoBERTa is described as a multilingual model, Urdu RoBERTa as a related-script encoder, and DistilBERT as a faster, lighter multilingual baseline.Source: §5.1 · p. 8
- Models are fine-tuned with AdamW, a linear learning-rate schedule with warmup, batch size 16, and up to 10 epochs.Early stopping follows four consecutive epochs without improved development accuracy; dropout is 0.10.Source: §5.1 · p. 8
5.2 Evaluation Metrics
- UPOS accuracy is the main comparison metric, while the selected XLM-RoBERTa checkpoint also receives per-class and confusion-matrix analysis.Source: §5.2 · p. 9
- Per-class precision, recall, and F1 are reported, with macro and support-weighted averages over the 17 UPOS tags.Source: §5.2 · p. 9
- INTJ, SYM, and X have no gold test examples, but their zero scores contribute to the 17-tag macro average.Source: §5.2 · p. 9
6 Results and Analysis
Across the evaluated models, XLM-RoBERTa base leads on both development and test accuracy, with the same model ranking on each split. The analysis also reports per-class scoring conventions and the selected checkpoint’s test-set error pattern.
6 Results and Analysis
- INTJ, SYM, and X have no gold examples in the test set, so their per-class scores were set to zero in 17-tag macro averages rather than measured.Table 5 reports per-class test results for XLM-RoBERTa, seed 46.Source: §6 · p. 9, Table 5 · p. 10
- 87.29% development accuracy and 88.60% test accuracy for XLM-RoBERTa base; Urdu RoBERTa small scores 86.85% test accuracy and multilingual DistilBERT 86.03%.Table 4 reports mean accuracy over five random seeds; each run’s best development-accuracy checkpoint was evaluated on the held-out test set.Source: §6 · pp. 9–10, Table 4 · p. 10
- The XLM-RoBERTa checkpoint correctly labels 3,046 of 3,439 test tokens; frequent errors include NOUN→PROPN (53), PROPN→NOUN (31), and NOUN→ADJ (30).The confusion matrix rows are gold tags and columns are predicted tags; INTJ, SYM, and X have no test examples.Source: §6 · p. 10, Table 6 · p. 11
- The model ranking is consistent across development and test: XLM-RoBERTa base leads, followed by Urdu RoBERTa small and multilingual DistilBERT.The passage notes that the observed differences are not specific to a single split.Source: §6 · p. 11
- XLM-RoBERTa base leads multilingual DistilBERT by 2.57 percentage points and Urdu RoBERTa small by 1.75 percentage points on the test set.The authors interpret the result as a measurable advantage for large-scale multilingual pretraining, while Urdu RoBERTa small remains competitive despite its smaller size.Source: §6 · p. 11
- Multilingual DistilBERT has the lowest accuracy but remains a competitive lightweight baseline when computational efficiency matters.The passage cautions that smaller multilingual encoders may not match larger multilingual models’ accuracy.Source: §6 · p. 11
7 Discussion
The discussion presents XLM-RoBERTa base as the strongest evaluated model and relates Urdu RoBERTa’s competitiveness to shared script patterns. It also frames the annotated resource as a foundation for further Saraiki NLP, while noting that the work covers only coarse-grained UPOS tagging.
7 Discussion
- Pretrained transformers are effective for Saraiki UPOS tagging, but encoder choice matters; XLM-RoBERTa base is strongest under the evaluated setup.The authors suggest broad multilingual pretraining and wider subword coverage are beneficial for low-resource token classification.Source: §7 · p. 11
- Urdu RoBERTa small remains below XLM-RoBERTa base, although shared Perso-Arabic script and orthographic patterns may help it represent Saraiki tokens.The passage frames related-script pretraining as useful, but not a full substitute for broader multilingual pretraining.Source: §7 · pp. 11–12
- The manually annotated UPOS data can support automatic tagger development and future work on morphological analysis, dependency parsing, and other downstream tasks.The authors note that existing Saraiki NLP resources remain limited.Source: §7 · p. 12
- The current work focuses only on coarse-grained UPOS tagging, despite Saraiki’s rich morphology and language-specific grammatical phenomena.The authors propose extending the corpus with morphological features, language-specific POS tags, and dependency relations.Source: §7 · p. 12
- The dataset and baseline results provide a starting point for more systematic research on Saraiki NLP and low-resource sequence labeling.Source: §7 · p. 12
8 Conclusion
The paper introduces a manually annotated Saraiki UPOS dataset with fixed splits and establishes baselines using three pretrained transformer encoders. XLM-RoBERTa base performs strongest among the evaluated models, and the resource is intended to support future Saraiki NLP research.
8 Conclusion
- The paper introduces a manually annotated Saraiki UPOS dataset from plain-text data, converted to CoNLL-U with fixed train, development, and test splits.Source: §8 · p. 12
- XLM-RoBERTa base, Urdu RoBERTa small, and multilingual DistilBERT were fine-tuned under a consistent token-classification setup; XLM-RoBERTa base performs strongest overall.The results support the effectiveness of multilingual and related-script pretrained encoders for Saraiki sequence labeling.Source: §8 · p. 12
- The dataset and baselines provide a benchmark resource for Saraiki POS tagging and can support future low-resource NLP and downstream language-processing research.Planned extensions include more data, morphological features, dependency relations, CRF decoding, parameter-efficient fine-tuning, and data augmentation.Source: §8 · p. 12
Limitations
The study identifies limits in dataset size, annotation scope, test-set coverage, model breadth, and annotation-agreement measurement. These constrain the breadth of evaluation and the linguistic phenomena represented.
Limitations
- The corpus is smaller than POS-tagged corpora for high-resource languages, and larger data could improve performance and support more reliable evaluation across genres and dialects.Source: Limitations · pp. 12–13
- UPOS tags support cross-linguistic comparison but omit Saraiki-specific morphology and syntax, including verbal morphology, case marking, and light and serial verb constructions.Future corpus versions should add morphological features, language-specific POS tags, and dependency relations.Source: Limitations · p. 13
- INTJ, SYM, and X have no test examples, so their test performance cannot be assessed.Source: Limitations · p. 13
- The model comparison covers only three pretrained transformer encoders; larger multilingual and regional-language models and parameter-efficient fine-tuning remain for future evaluation.Source: Limitations · p. 13
- Two experts reviewed the corpus, but sentences were not independently annotated from scratch by multiple annotators and inter-annotator agreement was not measured.Future work should document more detailed guidelines and measure agreement on an independently annotated representative subset.Source: Limitations · p. 13
Ethics Statement
The resource is intended for low-resource NLP research and language technology development, with cautions about source-data biases and use beyond its represented domain.
Ethics Statement
- The manually annotated Saraiki plain-text corpus is intended for low-resource NLP research and language technology development.Source: Ethics Statement · p. 13
- The corpus may reflect biases, dialectal preferences, or domain-specific patterns in its original text source.Source: Ethics Statement · p. 13
- Models trained on the dataset should be used with care outside the represented domain and are not intended for high-stakes decision-making.Source: Ethics Statement · p. 13