Source-linked AI summary
X-MuTeST: A Multilingual Benchmark for Explainable Hate Speech Detection and A Novel LLM-consulted Explanation Framework
Mohammad Zia Ur Rehman, Sai Kartheek Reddy Kasu, Shashivardhan Reddy Koppula, Sai Rithwik Reddy Chirra, Shwetank Shekhar Singh, Nagendra Kumar
TL;DR
Hate-speech detection needs both accurate classification and explanations that align with human judgments, particularly for under-resourced Indic languages. X-MuTeST combines human-rationale-guided and n-gram-guided training with LLM explanations; the paper reports robust classification and explainability across Hindi, Telugu, and English.
Problem
Hate-speech detection faces accuracy and explainability challenges, especially in under-resourced Indic languages where machine rationales may not align with human rationales.
Source: Introduction · p. 2Method
X-MuTeST trains first with human-rationale-guided attention and then with n-gram-derived attention, combining its explanations with LLM rationales.
Source: Introduction · p. 2, Training Procedure · p. 4, Explainability via N-gram Contributions · p. 5Results
Across Hindi, Telugu, and English, X-MuTeST demonstrates robust classification and explainability performance; on Telugu it achieves accuracy of 0.8881.
Source: Results Analysis for Telugu · p. 6, Conclusion · p. 8Takeaways & Limitations
The authors conclude that two-stage training and integration of explainability into multilingual text classification enhance model trustworthiness and user understanding.
Source: Conclusion · p. 8
Abstract
from arXiv · showhide
Hate speech detection on social media faces challenges in both accuracy and explainability, especially for underexplored Indic languages. We propose a novel explainability-guided training framework, X-MuTeST (eXplainable Multilingual haTe Speech deTection), for hate speech detection that combines high-level semantic reasoning from large language models (LLMs) with traditional attention-enhancing techniques. We extend this research to Hindi and Telugu alongside English by providing benchmark human-annotated rationales for each word to justify the assigned class label. The X-MuTeST explainability method computes the difference between the prediction probabilities of the original text and those of unigrams, bigrams, and trigrams. Final explanations are computed as the union between LLM explanations and X-MuTeST explanations. We show that leveraging human rationales during training enhances both classification performance and explainability. Moreover, combining human rationales with our explainability method to refine the model attention yields further improvements. We evaluate explainability using Plausibility metrics such as Token-F1 and IOU-F1 and Faithfulness metrics such as Comprehensiveness and Sufficiency. By focusing on under-resourced languages, our work advances hate speech detection across diverse linguistic contexts. Our dataset includes token-level rationale annotations for 6,004 Hindi, 4,492 Telugu, and 6,334 English samples. Data and code are available on https://github.com/ziarehman30/X-MuTeST
Introduction
Hate-speech explanations can miss culturally specific cues in under-resourced languages, while human rationale resources for hate-speech detection remain scarce. X-MuTeST pairs multilingual human rationales with a two-stage training framework and hybrid explanations.
Introduction
- Machine-provided rationales may miss culturally specific hateful cues in under-resourced languages, motivating human rationale resources for Hindi and Telugu.The cited Hindi example shows LLMs identifying “dog” but missing human-marked phrases translated as “to feel offended” and “to bark.”Source: Introduction · p. 2
Motivation: Scarcity of Hateful Rationales
- Human rationale resources for hate-speech detection are scarce, with comparable datasets for Hindi and Telugu described as nonexistent to the authors’ knowledge.The paper adds word-level human rationales for Telugu, Hindi, and English.Source: Motivation: Scarcity of Hateful… · pp. 2–3
A Glance at the Proposed Solution
- X-MuTeST uses two-stage training that balances human-rationale-guided attention with model-driven n-gram explanations, then unions its explanations with LLM rationales.Its contributions also include benchmark human-annotated rationales for Hindi, Telugu, and English.Source: A Glance at the Proposed Solution · p. 3
Related Work
Prior work introduced human rationales for hate-speech detection and highlighted gaps between model predictions and human reasoning. Research on multilingual explainability also addresses linguistic and cultural complexity, including in low-resource and code-mixed settings.
Explainability in Hate Speech Detection
- Rationale-based approaches developed from sentiment classification to other domains, while Mathew et al. (2021) introduced an early hate-speech dataset with human-annotated rationales.That work highlights that accurate classification does not ensure reasoning aligned with human logic or meaningful explanations.Source: Explainability in Hate Speech Detection · p. 3
- Recent studies identify faithful rationale generation as a challenge even for powerful models with high classification accuracy.Source: Explainability in Hate Speech Detection · p. 3
Multilingual Explainable Methods
- Multilingual explainability work addresses cultural effects in low-resource Hindi, code-mixed languages, and hate-speech intensity across linguistic settings.The cited examples include Kapil et al. (2023), Sawant et al. (2024), Yadav, Kaushik, and McDaid (2023), and Geleta et al. (2023).Source: Multilingual Explainable Methods · p. 3
Dataset Description and Annotation
The study draws Hindi and English data from HASOC and Telugu data from HOLD-Telugu, then adds token-level rationale annotations through a trained, iterative annotation process. Agreement results are substantial across the three languages.
Dataset Source
- Hindi and English datasets combine HASOC 2020/2021 samples, with 15% retained for testing and binary HATE/NOT HATE labels; Telugu HOLD-Telugu YouTube comments have HATE/NON-HATE labels and are transliterated from Latin into Telugu script.Source: Dataset Source · p. 3
Profile of Annotators
- Five annotators were selected for linguistic proficiency and subject expertise, with three assigned to each language and training, periodic checks, and consensus discussions.Selection included adults from diverse regions to mitigate potential demographic biases.Source: Profile of Annotators · p. 4
Rationale Annotation Procedure
- Three annotators mark token-level hate rationales for each sample, and non-hate samples receive all-zero rationale labels.After initial agreement of 61–80%, additional guidance and re-annotation increased agreement.Source: Rationale Annotation Procedure · p. 4
- Final rationale annotations show substantial agreement across all three languages, with English highest at 85.10 and Hindi slightly lower on pairwise scores.The reported overall agreement uses Fleiss’ method, while pairwise agreement uses Cohen’s Kappa.Source: Rationale Annotation Procedure · p. 4
Methodology
X-MuTeST combines rationale-guided training with n-gram-based token scoring and LLM rationales. Its two-stage training shifts attention supervision from human annotations to model-derived explanation targets.
Methodology
- X-MuTeST combines attention-based explanations with LLaMA-3.1 rationales, selected for stronger token-level alignment with human rationales in Hindi and Telugu.Muril is used for Telugu and English classification, and XLMR for Hindi.Source: Methodology · p. 4
Training Procedure
- Training first guides attention with human rationales, then replaces them with model-derived targets; both stages combine attention alignment and classification losses.Attention uses token-to-[CLS] dot products and softmax; α is 0.3 in Stage-1 and 0.6 for Telugu and English, 0.7 for Hindi in Stage-2.Source: Training Procedure · pp. 4–5
Explainability via N-gram Contributions
- Token scores aggregate logit drops across overlapping unigrams, bigrams, and trigrams, weighted 0.5, 0.3, and 0.2 respectively, before normalization.The score accounts for the number of n-grams of each length containing the token.Source: Explainability via N-gram Contributions · p. 5
Final Explanations through LLM Consultation
- Final explanations union X-MuTeST’s top-scored tokens with LLaMA-3.1 rationales, and the framework also measures agreement between these sources.Source: Final Explanations through LLM… · pp. 5–6
Experimental Setup and Results
The setup evaluates classification with standard performance metrics and explanation quality along plausibility and faithfulness dimensions.
Performance-Based Metrics
- Classification performance is evaluated using accuracy, F1 score, and macro-F1.Source: Performance-Based Metrics · p. 6
Explainability-Based Metrics
- Plausibility uses token-F1 and IOU-F1; faithfulness uses comprehensiveness and sufficiency, with lower sufficiency indicating a more informative rationale.Comprehensiveness measures the class-confidence drop after masking the predicted rationale; sufficiency compares full-input and rationale-only probabilities. The faithfulness metrics use masked tokens and the top five tokens as predicted rationales.Source: Explainability-Based Metrics · p. 6
Comparison Methods
The comparison setup evaluates explanation methods and multilingual encoders under standard fine-tuning and rationale-integrated training.
Comparison Methods
- LLM explanations are evaluated with token-F1 and IOU-F1, but prompt-based methods lack decision probabilities for comprehensiveness and sufficiency scoring.Source: Comparison Methods · p. 6
- Explainability comparisons include LIME, X-MuTeST, BERT-HateXplain-LIME, and BERT-ILP, alongside zero-shot GPT-4o, LLaMA-3.1, and Mistral-24.07 outputs.Source: Comparison Methods · p. 6
- Muril and XLMR are evaluated after ordinary fine-tuning and after human rationales are integrated, with LIME and X-MuTeST explanations assessed in both settings.Source: Comparison Methods · p. 6
Results Analysis for Telugu
On Telugu, X-MuTeST with LLM-consulted explanations leads the reported classification and plausibility metrics. It also achieves better sufficiency than Muril-RX.
Results Analysis for Telugu
- 0.8881 accuracy, 0.8762 F1 score, and 0.8849 Macro-F1 make X-MuTeST the top Telugu model, beating Muril-RX by 1.41% in accuracy and 1.29% in Macro-F1.The reported comparison identifies Muril-RX as the next-best baseline.Source: Results Analysis for Telugu · p. 6
- 0.6231 token-F1 and 0.3189 IOU-F1 are the best Telugu plausibility scores, improving 4.77% and 2.76%, respectively, over Muril-RX.The passage interprets this as stronger alignment of generated rationales with human annotations.Source: Results Analysis for Telugu · p. 6
- 0.0448 sufficiency is better than Muril-RX’s 0.0747, while GPT-4o, Llama-3.1, and Mistral-24.07 score lower in classification and explainability.The authors attribute the LLMs’ lower performance likely to inaccurate transliteration from Latin to Telugu script.Source: Results Analysis for Telugu · p. 6
Results Analysis for Hindi
X-MuTeST leads the reported classification and explanation results for Hindi and English. The Hindi comparison reports gains over Muril-RX, while the English results include strong faithfulness scores.
Results Analysis for Hindi
- Llama-3.1’s Hindi token-F1 of 0.4263 accompanies a 35.73% classification drop, illustrating that a high token-F1 does not ensure strong classification.XLMR and BERT-based baselines also trail across all reported Hindi metrics.Source: Results Analysis for Hindi · p. 6
- 0.8745 accuracy and 0.4344 token-F1 are the highest reported Hindi scores; versus Muril-RX, X-MuTeST gains 3.88% in accuracy and 2.27% in token-F1.Its sufficiency is 0.4768 versus Muril-RX’s 0.5042.Source: Results Analysis for Hindi · p. 6, Table 4 · p. 7
Results Analysis for English
- 0.5125 token-F1, 0.8433 comprehensiveness, and 0.0507 sufficiency accompany X-MuTeST’s highest English classification performance.GPT-4o has token-F1 of 0.4613 and accuracy of 0.7802; Mistral-24.07 has the highest IOU-F1, 0.3176, but lags overall.Source: Results Analysis for English · pp. 6–7
Parameter Sensitivity Analysis & Ablation
The sensitivity analysis identifies language-specific α settings that balance attention and classification losses. The ablation indicates that removing either training stage degrades performance.
Parameter Sensitivity Analysis
- In the reported results, Hindi and English achieve their best classification performance at α = 0.6, while Telugu achieves its best at α = 0.7.α weights classification loss, and performance is lower at boundary settings where all weight goes to either attention loss or classification loss.Source: Parameter Sensitivity Analysis &… · p. 7
Ablation
- Removing either training stage degrades performance, supporting the use of both stages in the proposed training method.The passage states that this method enhances classification performance while improving explainability.Source: Parameter Sensitivity Analysis &… · pp. 7–8, Figure 5 · p. 8
Generalizability
Experiments on HateXplain and HateBRXplain report gains in explanation metrics over prior results or the strongest baseline. The authors report effective generalization across these datasets and settings.
Generalizability
- 0.314 IOU-F1 and 0.587 Token-F1 on HateXplain exceed the best results reported in this work by 9.2% and 8.1%, respectively.The method also improves comprehensiveness by 5.6% on HateXplain.Source: Generalizability · p. 8
- X-MuTeST with LLM gains 6.9% in Token-F1 and 2.3% in sufficiency over the strongest HateBRXplain baseline.The authors report effective generalization across diverse datasets and settings.Source: Generalizability · p. 8
Conclusion
X-MuTeST combines rationale resources with an LLM-consulted framework for hate speech detection in Hindi, Telugu, and English, with reported gains in classification and explainability.
Conclusion
- The framework demonstrates classification and explainability performance across Hindi, Telugu, and English, and the authors report improved classification accuracy.They conclude that two-stage training and integrating explainability into multilingual text classification enhance both trustworthiness and user understanding.Source: Conclusion · p. 8
- The authors identify expansion to more languages as a direction for further investigation.Source: Conclusion · p. 8