PaperlayerView exact revision on arXiv

Source-linked AI summary

Hate Speech Classification In Roman Urdu: A Comparative Study On Parameter Efficient Fine-Tuning And Prompt Engineering

Toneema Zubair

arXiv:2608.21408v1cs.AI

TL;DR

Roman Urdu hate speech classification is challenging because the language is informal, inconsistent, and under-resourced, while hate detection requires contextual interpretation. This study compares prompt tuning, LoRA-based parameter-efficient fine-tuning, and prompt engineering through four experimental configurations. The authors conclude that PEFT offers the best stability and accuracy for Roman Urdu, although prompt engineering can achieve higher F1 scores in some cases.

  • Problem

    Roman Urdu hate speech detection is challenging because informal language and limited structure require contextual interpretation, while the study addresses a gap in newer methods for this language.

  • Method

    The study compares direct LLM inference, LoRA-based PEFT, prompt tuning with mixed or manual prompts, and zero-shot or few-shot prompt engineering.

  • Results

    PEFT is reported as the best option when stability and accuracy are required for Roman Urdu hate speech detection, while prompt engineering sometimes achieves higher F1 scores.

  • Takeaways & Limitations

    The comparison supports choosing among resource-efficient techniques according to the required balance between performance, stability, adaptability, and computational resources.

  • Takeaways & Limitations

    The dataset is naturally unbalanced, and inverse class-frequency weighting does not fully represent how models may function on a balanced dataset.

Abstract

from arXiv · show

Due to the widespread accessibility of the internet and social media, toxic and hateful con-tent has grown exponentially, causing significant distress and negative societal impacts. Ro-man Urdu, a low-resource language used in Pakistan and among Urdu-speaking communities worldwide, presents additional challenges because of its informal grammar, inconsistent sen-tence structures, and multiple variations in word spellings. This research aims to identify the most effective techniques for hate speech classification in such low-resource settings with limited data. To address this, the study investigates and compares the latest approaches, in-cluding prompt tuning, parameter-efficient fine-tuning (PEFT) using LoRA, and prompt en-gineering, under various experimental configurations. To achieve this objective, four exper-iments were designed. The first experiment involved direct inferencing with LLMs without any fine-tuning, to evaluate how well these models understand Roman Urdu in a zero-shot setting, especially given limited data. The second experiment utilized parameter-efficient fine-tuning (PEFT) with LoRA, which updates only a small subset of parameters, thereby reducing computational cost. The third experiment explored prompt tuning with both mixed and manually crafted prompts, using very small sets of training examples relative to the entire dataset, making it computationally efficient as well. Finally, the fourth experiment applied prompt engineering through zero-shot and few-shot learning, relying solely on care-fully designed instruction prompts for classification without further training.

Introduction

Roman Urdu hate speech detection is difficult because informal, inconsistent, and ambiguous language requires models to interpret context, intent, and tone. The study compares resource-efficient prompting and fine-tuning strategies for this low-resource setting while documenting important scope limitations.

  • Problem: Accurate detection requires understanding language, intent, and context rather than relying only on sentiment or surface negativity.Critical comments can be non-hostile, so hate speech identification differs from sentiment analysis.
  • Problem: Roman Urdu contains irregular spelling, phonetic variation, and contextual ambiguity that complicate NLP-based hate speech detection.Users may write the same word in multiple ways, while individual words can have different meanings depending on context.
  • Approach: The study compares prompt engineering, prompt tuning, and LoRA-based parameter-efficient fine-tuning for binary Roman Urdu hate speech classification.Its stated aim is to assess performance, computational efficiency, and resource utilization in low-resource settings.
  • Approach: Prompt tuning freezes the pretrained model and trains prompt-related components, while prompt engineering uses manually crafted zero-shot or few-shot prompts without updating weights.These approaches are presented as ways to reduce training requirements relative to full-model fine-tuning.
  • Approach: LoRA keeps most base-model parameters frozen and trains low-rank adaptation layers to reduce training time and resource consumption.The study also uses inverse class frequency weighting to address dataset imbalance.
  • Scope and limitations: The study is limited by an imbalanced dataset, restricted Roman Urdu coverage, binary sentence-level classification, and no hybrid prompt–PEFT investigation.The authors propose larger and more diverse data, other low-resource languages, finer-grained classification, and hybrid methods as future directions.

Literature Review

Prior hate speech research spans traditional machine learning, deep learning, multimodal systems, and transformer-based models, but Roman Urdu remains comparatively underexplored for newer parameter-efficient and prompt-based methods. Existing studies demonstrate strong task-specific results while highlighting challenges involving imbalance, context, data sparsity, and cross-linguistic generalization.

  • Traditional machine learning: Traditional systems used engineered features and classifiers such as Random Forest, SVM, Naive Bayes, Logistic Regression, and gradient boosting.Reported results include 90.26% accuracy for Random Forest with TF-IDF n-gram features and 95.4% insult-categorization accuracy with FastText and SVM.
  • Roman Urdu research: Roman Urdu studies applied lightweight and neural models to datasets including HS-RU-20, PURUTT, and RU-HSD-30K.Reported outcomes include an 86.35% F1 score for a PURUTT ensemble and a 0.885 F1-score on RU-HSD-30K.
  • Deep learning: Deep learning and multimodal approaches improved contextual or semantic modeling but still faced data sparsity and limited cross-linguistic generalization.Multimodal systems combined text, audio, and video, while neural models automatically learned contextual and semantic features.
  • Open challenges: Prior work identifies dataset bias, label imbalance, overfitting, weak contextual comprehension, annotation subjectivity, and user-intent ambiguity as recurring challenges.These findings motivate scalable, context-aware methods for low-resource hate speech detection.
  • Prompt-based methods: Prompt-based studies report efficiency benefits and gains of up to 13% accuracy, while some work reports F1-score increases of up to 10.1%.The supplied literature also states that minimum prompts can outperform complicated prompts and reduce resource usage in unsupervised settings.

Methodology

The methodology uses the PURUTT Roman Urdu corpus to compare prompt tuning, prompt engineering, and LoRA-based PEFT, while addressing imbalance through class weighting.

  • The study evaluates three approaches: BERT-based prompt tuning, GPT-3.5 prompt engineering, and LoRA-based PEFT with LLMs.
  • PURUTT contains 72,771 comments labeled as toxic or non-toxic, with Roman Urdu comments selected for analysis.
  • The corpus includes social-media comments in Roman Urdu and native Urdu script, but the experiments use only Roman Urdu.
  • The dataset is substantially imbalanced, containing 13,097 toxic comments and 59,674 non-toxic comments.
  • Inverse class-frequency weighting is applied during training to reduce majority-class bias and strengthen minority-class contributions.
  • Roman Urdu introduces inconsistent spellings, phonetic variation, lexical ambiguity, and informal language challenges.

3.2 Imbalanced Dataset and Its Challenges

The dataset’s imbalance can bias learning toward non-toxic comments, so the study uses cost-sensitive class weighting while emphasizing F1-score alongside weighted loss.

  • The majority-heavy distribution may bias learning and weaken generalisation for minority-class predictions.
  • Oversampling is challenging because Roman Urdu inconsistencies make high-quality synthetic data difficult to generate.
  • Cost-sensitive learning preserves the full dataset while penalising minority-class misclassification more severely.
  • Class weights are incorporated into the loss function to modify the decision boundary for under-represented examples.
  • Weights are computed from dataset size, class count, and per-class sample count before normalization.
  • F1-score is emphasized because accuracy alone can be deceptive in imbalanced classification, while F1 balances precision and recall.

3.3 Investigations

The investigations compare contemporary adaptation paradigms for Roman Urdu hate-speech classification, including PEFT, prompt tuning, direct inference, and zero- versus few-shot prompting.

  • The study assesses transformer-based techniques across multiple adaptation paradigms for low-resource Roman Urdu text.
  • Multilingual BERT is used to compare PEFT, lightweight prompt tuning, and zero-shot inferencing under different adaptation levels.
  • Open-source LLMs including LlaMA3.2, Mistral, and additional models are evaluated through parameter-efficient fine-tuning and direct inferencing.
  • The experiments aim to identify models suited to low-resource downstream categorization tasks.
  • OpenAI models are evaluated with zero-shot and few-shot prompt engineering through the OpenAI API.
  • The comparison examines prompt sensitivity in informal, code-mixed Roman Urdu toxic-comment classification.

3.4 Prompt Tuning with Manual and Soft Templates

Prompt tuning reformulates Roman Urdu toxicity classification as masked-language modeling using manual, mixed, and verbalizer-based designs, with mixed prompts adding learnable soft tokens.

  • 3.4.1 Framework and Setup: OpenPrompt with multilingual BERT supports manual, soft, and mixed templates plus customizable verbalizers for label-word mapping.
  • 3.4.2 Prompt Styles Used: Manual prompts use prefix or cloze templates containing [MASK] to frame classification as masked-language modeling.
  • 3.4.2 Prompt Styles Used: Mixed prompt tuning adds trainable soft tokens to human-written templates while keeping the base pretrained language model unchanged.
  • 3.4.3 Manual Verbalizer: Manual verbalizers map predicted mask words to toxic or non-toxic labels using semantically meaningful Roman Urdu terms.
  • 3.4.4 Expanded Description: Manual Prompts: Cloze prompts place [MASK] within the sentence for bidirectional context, whereas prefix prompts place it at the end for sentence completion.
  • 3.4.5 Explanation: Mixed Prompts (Manual + Soft Tokens): Mixed prompts combine human-readable syntax, trainable context embeddings, parameter-efficient training, and few-shot adaptation.
  • 3.4.6 Comparison with Previous Work: The study extends prior manual prefix-and-cloze prompting work by introducing mixed prompt tuning for low-resource languages.
  • 3.4.7 Summary of Prompt Styles: The prompt-tuning procedure splits data, samples training examples at sizes 32, 64, and 128, and trains prefix, cloze, or mixed templates.

3.5 Prompt Engineering using OpenAI GPT-3.5

This section evaluates GPT-3.5 prompt engineering for Roman Urdu toxicity classification using structured chat prompts, zero-shot and few-shot configurations, batched processing, and standard classification metrics.

  • 3.5 Prompt Engineering using OpenAI GPT-3.5: The evaluation used Microsoft Azure OpenAI GPT-3.5 with system, user, and assistant chat roles for Roman Urdu toxicity classification.The system supplied task instructions, the user carried the comment, and the assistant returned label 0 or 1.
  • 3.5.7 Implementation Notes: Prompt evaluation computed accuracy, precision, recall, and F1, averaging batch-level results to obtain final overall metrics.The algorithm returns both predicted labels and overall metrics.
  • 3.5.7 Implementation Notes: Prompts were limited to a maximum of 255 tokens because of GPT-3.5 input constraints.This token restriction is identified as a practical limitation of the implementation.
  • 3.5.2 Prompt Engineering Techniques: Zero-shot prompting classified comments without in-context examples, whereas few-shot prompting added five toxic and five nontoxic examples before each classification question.The examples were intended to improve generalization.
  • 3.5.3 Prompt Structure and Formatting: Each prompt contained a system task explanation, optional user-assistant example pairs, and the final comment to classify.The flowchart describes how these messages are assembled sequentially.
  • 3.5.5 Batch Processing: The dataset was processed in batches of 100 comments, with 728 iterations for each prompting configuration and 1456 total iterations.Responses were streamed in JSON format and saved with expected labels.
  • 3.5.6 System Prompt Template: The system prompt restricted outputs to 0 for non-toxic comments and 1 for toxic comments, excluding neutral or mixed categories.For comments containing both types of content, the dominant tone determined the final classification.
  • 3.5.7 Implementation Notes: Few-shot examples were randomly sampled and refreshed for each batch, while the workflow split the dataset into batches and generated one predicted label per comment.The implementation uses GPT-3.5 with a Roman Urdu dataset and outputs predicted labels plus evaluation metrics.

3.6 Parameter-Efficient Fine-Tuning (PEFT) using LoRA

The study applies LoRA-based parameter-efficient fine-tuning to Roman Urdu toxicity classification, adapting pretrained models through lightweight updates while keeping most base weights frozen.

  • 3.6 Parameter-Efficient Fine-Tuning (PEFT) using LoRA: The PEFT workflow covers pretrained backbones, a 60/20/20 train-validation-test split, LoRA configuration, class-weighted loss, and metric logging.Accuracy, precision, recall, and F1 are tracked during training and evaluation.
  • 3.6 Parameter-Efficient Fine-Tuning (PEFT) using LoRA: LoRA replaces full-rank updates with low-rank approximations, reducing trainable parameters, memory consumption, and computational expense.The base model remains frozen while lightweight adapter parameters are trained.
  • 3.6 Parameter-Efficient Fine-Tuning (PEFT) using LoRA: The study used LoRA-based PEFT on six distinct pretrained models to examine both pretrained language models and large language models.The models were accessed through Hugging Face.
  • 3.6 Parameter-Efficient Fine-Tuning (PEFT) using LoRA: LoRA was configured with rank r = 16, alpha = 8, dropout = 0.05, and sequence-classification task type.These settings enable low-rank task adaptation without retraining all model weights.
  • 3.6 Parameter-Efficient Fine-Tuning (PEFT) using LoRA: Training used batch size 8, per-epoch evaluation and saving, and loading of the best model at the end.The customized trainer included tokenization, dynamic batching, and automatic metric computation.
  • 3.6 Parameter-Efficient Fine-Tuning (PEFT) using LoRA: A class-weighted loss penalized minority-class misclassifications more heavily to reduce majority-class bias and improve fairness in predictions.The weights were derived from the class distribution.
  • 3.6 Parameter-Efficient Fine-Tuning (PEFT) using LoRA: LoRA is presented as suitable when complete fine-tuning is not feasible in low-resource Roman Urdu toxicity detection.The method adapts large-scale transformer models by changing lightweight parameters.
  • 3.6 Parameter-Efficient Fine-Tuning (PEFT) using LoRA: The procedure freezes base parameters, attaches LoRA adapters to transformer layers, trains on Dtrain, validates on Dval, and evaluates the final model on Dtest.The reported metrics are accuracy, precision, recall, and F1-score.

Experimentation and Results

Four experiments compare direct inference, fine-tuning, prompt tuning, and prompt engineering for Roman Urdu hate-speech classification. The reported results favor task adaptation, with LoRA and few-shot prompting outperforming direct inference, while prompt tuning offers a lighter alternative.

  • Baseline LLM evaluation: The first experiment evaluated quantized Hugging Face LLMs on a held-out test set without fine-tuning as a baseline for adapted methods.Inference was batched, gradient-free, and capped at 512 tokens.
  • PEFT and LoRA results: LoRA fine-tuning produced more balanced precision and recall, closer predicted and actual label distributions, and improved representation of the minority class.The reported changes included lower false-positive and false-negative counts.
  • PEFT and LoRA results: The study concludes that LoRA-based fine-tuning greatly improved Roman Urdu hate-speech classification, producing higher balanced F1 scores and reducing false-positive rates.The authors state that fine-tuning is more important than direct inference for robust and accurate deployment in hate-speech detection.
  • Prompt tuning results: Mixed prompts at K = 128 achieved the highest prompt-tuning F1 score, 0.6852, while higher K values generally improved performance.Prefix and mixed prompts outperformed cloze prompts at higher training-example counts.
  • Prompt tuning results: PEFT LoRA achieved accuracy 0.8945 and F1 score 0.82, outperforming prompt-tuning approaches while prompt tuning remained a lightweight substitute.Mixed and prefix prompts showed similar top-level performance, with mixed prompts slightly higher.

Conclusion and Recommendations

The study compares prompt-based methods with PEFT for Roman Urdu hate-speech classification, finding different strengths by dataset size and task complexity. It recommends further work on automated prompts, hybrid methods, and broader multilingual evaluation.

  • Study scope: The experiments examine prompt engineering, parameter-efficient fine-tuning, and prompt tuning for Roman Urdu hate-speech classification.The chapter presents these techniques using large language models and discusses their performance and adaptability.
  • Comparative findings: In certain cases, prompt engineering achieves higher F1 scores than PEFT, but its performance becomes less consistent on larger datasets.The comparison indicates that prompt engineering can be competitive while showing reduced predictability as dataset size increases.
  • Comparative findings: Prompt engineering and prompt tuning provide rapid, resource-efficient alternatives to full fine-tuning when datasets or labeled examples are small.Their suitability is tied to limited-data settings rather than uniformly superior performance.
  • Comparative findings: PEFT methods such as LoRA produce more reliable and consistent results for larger datasets or more complex classification tasks, especially in low-resource languages.The conclusion positions PEFT as the stronger option when stability and generalization are priorities.
  • Implications: The techniques offer a balance between computational efficiency and model performance for Roman Urdu hate-speech detection.This balance is presented as a central practical outcome of the experimental investigation.
  • Recommendations: Future research should automate prompt construction, combine prompt-based methods with PEFT, and test broader multilingual, cross-lingual, and multi-task settings.The recommendations target prompt scalability, hybrid performance, and validation across additional languages and tasks.
Loading 2608.21408v1…