Source-linked AI summary
Efficient Adaptation of LLMs for Hate Speech Detection in Low-Resource Languages: A Comparative Study on Roman Urdu
Toneema Zubair, Muhammad Junaid Asif, Faisal Kamiran, Hafiz Hassan Saeed, Rana Fayyaz Ahmad
TL;DR
Roman Urdu hate speech detection remains underexplored because its informal, variable writing and limited systematic evidence challenge low-resource language processing. This paper compares zero-shot inference with LoRA-based parameter-efficient fine-tuning across transformer models, finding that PEFT substantially outperforms zero-shot detection, reaching F1 > 0.93 and a best score of 0.9387.
Problem
Roman Urdu hate speech detection is underexplored, with limited systematic comparison of zero-shot inference and parameter-efficient fine-tuning in low-resource settings.
Method
The study evaluates zero-shot inference against LoRA-based parameter-efficient fine-tuning across several transformer models for Roman Urdu hate speech detection.
Results
Across tested transformer architectures, PEFT improves hate speech classification over zero-shot inference, achieving a best F1-score of 0.9387.
Takeaways & Limitations
LoRA-based PEFT supports effective Roman Urdu hate speech detection while updating only a small subset of model parameters.
Takeaways & Limitations
Because experiments use only the PURUTT dataset, the reported performance may not generalize to other Roman Urdu corpora or low-resource languages.
Abstract
from arXiv · showhide
It is challenging to detect hate speech in Low Resource Languages (LRLs) because of the absence of annotated data, the informality of its language structure, and the lack of standardized grammar. A good example of such a challenge is Roman Urdu which is broadly used by South Asians on social media and has a high variation while lacking contextually consistent spellings. The objective of this paper is to conduct a comprehensive assessment of Large Language Models (LLMs) for Hate Speech Detection (HSD) in Roman Urdu script and fine-tune these models using the Parameter-Efficient Fine-Tuning (PEFT) method called Low-Rank Adaptation (LoRA). To evaluate zero-shot inference, we benchmarked it against PEFT on different transformer models, including Mistral, LLaMA, Falcon, and multilingual BERT. Experiments are conducted on the PURUTT (Parallel Urdu and Roman Urdu Corpus for Toxic Comments and Transliteration) dataset with over 72,000 annotated comments. The results suggest that zero shot models perform moderately (F1 = 0.56), but updating a small fraction of the model trainable parameters improves the classification performance significantly (F1 > 0.93). Our results have shown that PEFT delivers outstanding performance alongside excellent computational efficiency, making it highly suitable for low-resource language processing tasks.
1. INTRODUCTION
The introduction frames Roman Urdu hate speech detection as difficult because the language is linguistically diverse, nonstandard, and limited by scarce, imbalanced labeled data. It proposes comparing zero-shot LLM inference with LoRA-based parameter-efficient fine-tuning across transformer models.
- Motivation and Challenges: Roman Urdu is a low-resource, informal Latin-script variety with substantial spelling and grammatical variation that hinders stable NLP modeling.Words may have multiple phonetic spellings, and Roman Urdu lacks standardized orthography.
- Motivation and Challenges: Scarce, high-quality labeled data and severe class imbalance further complicate Roman Urdu hate speech detection.PURUTT represents a recent dataset effort, but toxic comments remain much fewer than non-toxic comments.
- Study Objective: The study evaluates LoRA-based parameter-efficient fine-tuning against zero-shot inference for hate speech detection in Roman Urdu.The comparison directly addresses the lack of explicit evaluation between these approaches in low-resource languages.
- Research Questions: The research examines zero-shot LLM performance, LoRA effectiveness, comparative classification metrics, and the advantages and disadvantages of LoRA versus full fine-tuning.The stated metrics are Accuracy, Precision, Recall, and F1-score.
- Contributions: The paper compares six transformer-based models in zero-shot and LoRA settings while studying adaptation with only a small fraction of model parameters fine-tuned.The models are Mistral-7B, LLaMA-3-8B, Falcon-7B, Gemma-2B, DeepSeek-R1, and Multilingual BERT.
2. RELATED WORK
Related work progresses from traditional machine-learning and deep-learning approaches toward transformer and parameter-efficient methods for hate speech detection. However, Roman Urdu remains under-explored, with unresolved challenges in cross-lingual generalization, data scarcity, contextual modeling, and balancing computational efficiency with performance.
- Traditional machine learning: Traditional machine-learning studies established effective Roman Urdu hate-speech classification using word- and character-level features and multiple algorithms.Nasir et al. used the HS-RU-20 dataset to classify content as neutral, offensive, or hateful.
- Traditional machine learning: 90.26% accuracy was achieved by Random Forest in multilingual hate-speech classification with TF-IDF-based N-gram features.The result was strongest among the evaluated Random Forest, SVM, Naïve Bayes, Logistic Regression, AdaBoost, and Gradient Boosting methods.
- Deep learning: Deep-learning approaches improve contextual and semantic modeling, including CNN, Bi-LSTM, and multimodal architectures for hate-speech detection.Prior work highlights CNN effectiveness, Bi-LSTM precision gains over baseline Autoencoder and Multi-task learning models, and the value of combining text and image cues.
- Transformer-based models: BERT and its variants support hate-speech identification across languages, including morphologically rich and low-resource settings.Reported work covers Arabic, Turkish, Korean, and other multilingual contexts, with optimization, augmentation, domain-specific embeddings, and ensembling improving classification.
- Research gaps and parameter-efficient fine-tuning: Roman Urdu research lacks holistic paradigm comparisons and sufficient attention to the trade-off between model performance and computational efficiency.Parameter-efficient methods such as LoRA, adapters, and prefix tuning update only subsets of parameters, offering computational savings for low-resource settings.
3. MATERIALS AND METHODS
The methodology evaluates pretrained language models for Roman Urdu hate speech detection through zero-shot inference and LoRA-based parameter-efficient fine-tuning. It addresses class imbalance, language variability, and computational constraints using weighted learning, selective parameter updates, and efficient implementation.
- Dataset: PURUTT contains 13,097 toxic and 59,674 non-toxic social media comments, capturing Roman Urdu’s spelling variation, code-mixing, informal grammar, slang, and abbreviations.The dataset is low-resource and class-imbalanced, motivating imbalance-aware training and evaluation.
- Zero-Shot Baseline: Zero-shot inference establishes a baseline by classifying test samples without further training, using tokenization, padding or cropping, and predicted-class probabilities.Inference uses no backpropagation gradients, and the approach measures pretrained models’ intrinsic ability to comprehend Roman Urdu.
- LoRA-Based PEFT: LoRA fine-tuning inserts low-rank adapter matrices into transformer layers while freezing the original model parameters and updating only the adapters.The configuration uses r=16, α=8, and dropout=0.05.
- Model Comparison: The study compares Mistral, LLaMA, Falcon, DeepSeek, and multilingual BERT across zero-shot and PEFT settings to examine performance and computational trade-offs.Multilingual BERT serves as a lightweight baseline, while the other models provide larger architectural alternatives.
- Training and Evaluation: Cost-sensitive learning assigns higher cost to the minority toxic class, while evaluation emphasizes F1-Score alongside accuracy, precision, and recall.The weighting strategy is intended to reduce majority-class sensitivity under class imbalance.
- Implementation: Implementation uses PyTorch, Hugging Face Transformers, NVIDIA A100 GPUs, quantization, sequence lengths of 512, batch size 8, and AdamW optimization.BitandBytes quantization reduces GPU memory consumption, while pandas and scikit-learn support data cleaning, formatting, and tokenization.
4. RESULTS AND DISCUSSION
Zero-shot LLMs struggle with Roman Urdu hate speech detection, while LoRA-based parameter-efficient fine-tuning substantially improves performance, calibration, and robustness across tested architectures. Mistral-7B-v0.3 leads overall, with LLaMA-3-8B and Gemma-2B also performing competitively.
- Zero-shot inference: 0.56: Mistral-7B-v0.3 achieves the highest zero-shot F1-score, while DeepSeek and Falcon-7B score between 0.21 and 0.51.The results indicate that pre-trained LLMs are not very effective without domain adaptation.
- Zero-shot inference: Zero-shot models show high false-positive rates for the toxic class, reducing precision and F1-score despite moderate recall.Their predictions differ substantially from actual labels, revealing limitations in learning Roman Urdu features.
- PEFT with LoRA: 0.9379: LLaMA-3-8B records the second-highest PEFT F1-score, while Gemma-2B also exceeds 0.90.Mistral-7B-v0.3 achieves the highest F1-score among the evaluated PEFT models.
- Generalization and efficiency: +0.2368: PEFT improvements transfer across models of different sizes and architectures, enabling small models to compete with larger models.Class weighting minimizes the effect of data imbalance.
- Generalization and efficiency: PEFT consistently outperforms standard LLMs without adaptation and is described as effective, scalable, and resource-efficient for Roman Urdu hate speech detection.Fine-tuned models align more closely with actual label distributions, reducing bias and false positives and negatives while improving precision and recall.
5. CONCLUSIONS
The study concludes that LoRA-based PEFT is more effective than zero-shot LLM inference for Roman Urdu hate-speech detection, while noting limitations in content coverage, evaluation scope, and qualitative analysis.
- Contributions: LoRA-based PEFT outperforms zero-shot inference for Roman Urdu hate-speech detection while updating only a small subset of model parameters.Zero-shot models show cross-lingual generalization but have low accuracy and a high false-positive rate on low-resource Roman Urdu.
- Limitations: Binary classification does not capture different harmful-content tones, sarcasm, or implicit hate speech.The study also does not discuss qualitative aspects such as fairness, bias, or interpretability.
- Limitations: The study does not consider cross-lingual transfer learning or domain adaptation, and pretrained models may carry domain bias from high-resource languages.These approaches are identified as potential directions for further work.
- Limitations: Testing only on PURUTT limits how fully the reported performance may apply to other Roman Urdu corpora or low-resource languages.Other datasets and languages may differ in linguistic features, writing styles, and domain-specific vocabulary.