PaperlayerView exact revision on arXiv

Source-linked AI summary

ParsHate: A Benchmark Dataset for Hate and Target Detection in Persian

Zahra Bokaei, Walid Magdy, Bonnie Webber

arXiv:2609.16393v1cs.CL

TL;DR

Persian hate-speech resources have limited temporal coverage and lack a consistent combination of structured targets, implicit-hate annotations, and rationales. ParsHate is a decade-spanning, manually annotated Twitter dataset built with random and score-stratified sampling. Evaluation finds temporal variation and persistent difficulty with implicit hate and minority targets, alongside stronger detection with integrated all-years training.

  • Problem

    Existing Persian datasets have structural limitations in sampling, temporal coverage, target schemas, and annotation of implicit hate, limiting systematic study of these phenomena.

    Source: §1 · p. 1

  • Method

    ParsHate provides 10,000 manually annotated Persian tweets from 2013–2022, using random and score-stratified temporal sampling and labels for hate, targets, implicitness, strategies, and rationales.

    Source: §1 · pp. 1–2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    Evaluation finds temporal distribution shifts and structural failures on implicit hate and minority targets; integrated all-years LLaMA-3 reaches 79.0 macro-F1, above the PHATE-trained model's 74.8.

    Source: §4.1 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    Addressing culturally grounded hate and rare target categories remains an important direction for Persian hate-speech detection.

    Source: §5 · p. 9

    The dataset is limited to Twitter, and broader cross-domain, cross-platform, and cross-cultural generalization remains unexplored.

    Source: Limitation · pp. 9–10

Abstract

from arXiv · show

We introduce ParsHate, a manually annotated dataset of 10,000 Persian tweets spanning 2013-2022, representing the first decade-long benchmark for hate speech detection in Persian. The dataset contains 31% hateful content and supports both hate detection and multi-label fine-grained target identification across seven structured target categories. ParsHate also distinguishes explicit and implicit hate, marks explicit and implicit targets, and provides span-level rationales. Data collection combines random and score-stratified temporal sampling to reduce keyword-driven bias while preserving natural label distributions. Applying SOTA models for Persian hate-speech detection on ParsHate shows moderate performance (79% F1), especially with samples from earlier years, and low performance with target identification (25.5% macro-F1). This emphasizes the diverse sampling of hate speech in ParsHate and its challenging nature that requires more advanced methods for better performance. Dataset is made publicly available.

1 Introduction

Existing hate-speech datasets offer richer annotation schemes, but Persian resources remain limited and structurally incomplete. ParsHate addresses these gaps with decade-long coverage, structured target labels, explicit–implicit annotations, and span-level rationales.

1 Introduction

  • Richer supervision in prior hate-speech datasets includes target labels, span rationales, or explicit–implicit distinctions, but coverage varies across resources.These annotation schemes support more detailed model evaluation.Source: §1 · p. 1
  • Persian hate-speech datasets remain limited, with existing resources constrained by keyword-driven sampling, narrow temporal coverage, free-text targets, or missing target taxonomies and explicit–implicit distinctions.These gaps hinder systematic study of temporal change, implicit hostility, and target representation in Persian online discourse.Source: §1 · p. 1
  • ParsHate introduces a manually annotated benchmark of 10,000 Persian tweets spanning 2013–2022, with 31% labeled as hateful.Its hybrid temporal sampling combines random and score-stratified selection to reduce keyword-driven bias without enforcing label balance.Source: §1 · pp. 1–2
  • The dataset supports multi-label prediction across seven structured target categories, while marking explicit or implicit hate, target explicitness, and span-level rationales.It also categorizes implicit-expression strategies and establishes single-year, multi-year, and cross-temporal evaluation settings.Source: §1 · p. 2

2 Related Work

Prior hate-speech datasets cover target taxonomies, implicit hate, rationales, and temporal variation, but Persian resources lack a combination of these features. ParsHate brings them together with decade-long coverage and score-stratified temporal sampling.

Related Work

  • Prior hate-speech research spans target taxonomies, implicit-hate analysis, rationales, and temporal coverage, motivating comparison across these four dimensions.Table 1 summarizes how ParsHate compares with existing Persian datasets.Source: §2 · p. 2, Table 1 · p. 3

Target Taxonomies

  • Persian resources offer uneven target schemas: Pars-OFF has coarse types, PHATE uses free-text targets, and Pars-HAO and PHICAD lack structured schemas.ParsHate's target categories include fine-grained subcategories.Source: §2 · pp. 2–3

Implicit and Coded Hate; Span-Level Rationales for Interpretability

  • No existing Persian dataset annotates implicit-hate strategies or distinguishes implicit targets, while only PHATE provides span-level rationales.English resources include implicit-hate taxonomies and span-level rationales.Source: §2 · p. 2

Temporal Coverage and Sampling

  • Existing Persian resources cover limited temporal spans or do not report their windows, and rely on keyword- or profile-based sampling.Such sampling introduces lexical bias and under-represents implicit hate.Source: §2 · pp. 2–3

Positioning of ParsHate

  • No existing Persian resource combines structured target taxonomies, implicit-hate strategy annotations, span-level rationales, and longitudinal coverage.ParsHate combines these features with a seven-category taxonomy, implicit-target marking, and score-stratified sampling.Source: §2 · p. 3

3 ParsHate: Dataset Construction

ParsHate combines temporal sampling and structured annotation to create a decade-spanning Persian Twitter dataset. Its statistics show changing hate prevalence, politically concentrated targets, and greater disagreement for cultural or historical implicit hate.

3.1 Data Collection

  • The dataset was built from a filtered pool of 29 million Persian tweets drawn from a Twitter archive containing about 47 million tweets from 2013–2022.Filtering removed retweets, duplicates, very short tweets, and tweets containing only URLs, mentions, or hashtags.Source: §3.1 · p. 3
  • Each year contributes 1,000 tweets: 500 randomly sampled and 500 selected across ten classifier-probability intervals.The classifier scores guided sampling, not automatic labeling, and the design aimed to preserve temporal diversity while including neutral, borderline, and high-confidence hate cases.Source: §3.1 · p. 3

3.2 Annotation Scheme and Guideline

  • Implicit hate is labeled by explicitness and rhetorical strategy, while target annotations permit multiple categories and distinguish directly named from implied targets.Span rationales mark the shortest text segments justifying hate labels, with character-level spans retained through majority annotation.Source: §3.2 · p. 4
  • Three native Persian speakers independently annotated all 10,000 tweets, resolving disagreements through majority voting and discussion-based adjudication.Agreement was κ = 0.69 for hate identification, κ = 0.60 for target category, κ = 0.51 for implicit hate, and κ = 0.58 for implicit targets.Source: §3.2 · p. 4

3.3 Disagreement Analysis and Resolution

  • Cultural or historical references are the most common implicit-hate strategy at 48% and have the lowest strategy-level agreement, κ = 0.43.About 16% of implicit-hate tweets required adjudication beyond majority voting, compared with 6% of explicit-hate tweets.Source: §3.3 · p. 5

3.4 Dataset Statistics and Analysis

  • 5.90% in the earliest year and 15.61% in later years mark the rise in hateful tweets across ParsHate's 2013–2022 span.Hate accounts for 31% of the 10,000-tweet dataset; 7% of hate tweets are implicit, while 35% contain implicit targets.Source: §3.4 · p. 5, Figure 1 · p. 5, Figure 2 · p. 5
  • 76.45% of hate instances target politics, while 39% of political hate instances use implicit target references.Other target categories range from 1.94% for gender to 6.42% for religion; hate spans average 18% of tweet characters.Source: §3.4 · p. 5, Figure 3 · p. 5

4 Analysis of ParsHate

The analysis evaluates temporal robustness, lexical variation, and multi-label target prediction, then examines how these challenges appear in model errors.

4.2 Lexical Variability and NE Influence

  • Early years have the highest lexical diversity, and their TTR remains higher than later years after named-entity masking.Masking has its strongest effect in 2013–2014; the authors identify early-year lexical variation as a plausible explanation for reduced performance on temporally distant data.Source: §4.2 · p. 6

4.1 Hate-Speech Detection over Time

  • 79.0 macro-F1 for LLaMA-3 with all-years training exceeds 74.8 for the PHATE-trained model and 63.5–67.7 for year-specific models.All-years training also produces more stable performance across the temporal spectrum, reducing the degradation observed on earlier years.Source: §4.1 · p. 6, Table 4 · p. 7

4.4 Error Analysis

  • Using ParsHate rationales as gold spans, LLaMA-3-FT reaches token-level span F1 of 44 overall (58 explicit, 14 implicit), indicating lower span localization for implicit than explicit hate.In one protest example, the model predicts gender from the surface mention of “this girl,” although the annotated target is state actors.Source: §4.3 · p. 7, §4.4 · p. 9
  • F1 = 84 for both fine-tuned models on explicit hate, compared with F1 = 3 for LLaMA-3-FT and F1 = 6 for Gemma-3-FT on implicit hate.The models perform well on explicit hate but collapse on implicit hate.Source: Table 5 · p. 8, §4.4 · p. 8

4.3 Multi-label Target Prediction

  • F1 = 0.89 for LLaMA-3 on Politics contrasts with near-zero F1 on rare targets, while GPT-5 has higher minority-target recall but lower precision.Named-entity masking reduces overall performance for both models without changing this behavioral contrast.Source: §4.3 · p. 7, Table 6 · p. 8

4.4 Error Analysis

  • Implicit hate is 7% of hate instances but approximately 31% of LLaMA-3-FT’s missed hate cases; GPT-5 lowers that share to 22% while over-flagging figurative contexts.For LLaMA-3-FT, 83% of implicit-hate errors are predicted as non-hate.Source: §4.4 · p. 8
  • Implicit targets are 35% of hate instances but approximately 58% of target misclassifications in the all-years model.LLaMA-3-FT often maps minority targets to Politics, while GPT-5’s broader minority-target recall comes with lower precision.Source: §4.4 · p. 8

5 Conclusion

ParsHate provides a decade-spanning Persian benchmark for hate and target prediction. Its evaluation reveals temporal variation and persistent difficulty with implicit hate and minority targets.

5 Conclusion

  • ParsHate is a decade-spanning 2013–2022 Persian benchmark with 10,000 manually annotated tweets and structured target, explicitness, strategy, and rationale annotations.It supports hate detection and multi-label target prediction.Source: §5 · p. 9
  • Score-stratified temporal sampling preserves natural label distributions while reducing keyword-driven bias.Source: §5 · p. 9
  • The evaluation identifies temporal distribution shift and model failures on implicit hate and minority targets as challenges for Persian hate-speech detection.The authors identify better handling of culturally grounded hate and rare target categories as important directions.Source: §5 · p. 9

Ethics Statement

The ethics statement describes privacy protections, annotator safeguards, and research-only release conditions for ParsHate.

Ethics Statement

  • The dataset uses a public archival Twitter sample, removes user identifiers, and does not attempt to identify individuals.Source: Ethics Statement · p. 9
  • Three native Persian-speaking annotators were informed about distressing content, could opt out at any time, and received fixed-rate compensation.Source: Ethics Statement · p. 9
  • Because the dataset contains offensive content and annotates hate targets, it is released for research purposes only and should be treated as sensitive.Source: Ethics Statement · p. 9

Limitation

The dataset’s Twitter-only scope, imbalanced target distribution, and small share of implicit hate constrain generalization and fine-grained evaluation. Broader cross-domain, cross-platform, and cross-cultural generalization remain unexplored.

Limitation

  • ParsHate is restricted to Twitter, so its findings may not generalize to Persian-speaking platforms with different discourse norms and interaction patterns.The experiments also focus on supervised evaluation within ParsHate and cross-temporal transfer from PHATE; broader cross-domain, cross-platform, and cross-cultural generalization remain unexplored.Source: Limitation · pp. 9–10
  • Political hate dominates the dataset, constraining performance on minority target categories and balanced evaluation across identity dimensions.Target classification is considerably weaker than binary hate detection, especially for low-frequency identity groups; target-specific modeling strategies and balancing techniques are beyond this work’s scope.Source: Limitation · p. 9
  • Implicit hate constitutes only 7% of the dataset, limiting fine-grained modeling and detailed statistical analysis of its rhetorical strategies.The authors identify expanded implicit-hate coverage and more extensive qualitative analysis as future directions.Source: Limitation · pp. 9–10

A.1 Per-Strategy Analysis of Implicit Hate

Implicit-hate performance varies with rhetorical strategy and its knowledge demands. Cultural or historical references are both the most frequent strategy and the hardest for the evaluated models, while text-localisable indirect hostility is comparatively more tractable.

A.1 Per-Strategy Analysis of Implicit Hate

  • 2, 4, and 5 were LLaMA-3-FT’s F1 scores for cultural/historical reference, sarcasm/figurative language, and other indirect strategies, respectively, versus 84 on explicit hate.Gemma-3-FT showed the same ordering across implicit strategies.Source: §A.1 · p. 11
  • Cultural references require Persian-specific event and entity knowledge, while sarcasm retains partial textual cues but can depend on extra-textual context.Other indirect strategies are comparatively more tractable because they often retain lexical hostility, exclusion, generalisation, or collective accusation that can be localised from text alone.Source: §A.1 · p. 11
  • The strategy gradient reflects differing knowledge demands, not implicitness alone.Cultural/historical references encode hostility in the referent, event, public figure, or political symbol, whereas other indirect strategies often contain text-localisable cues.Source: §A.1 · p. 11
  • 48% of implicit-hate instances were cultural/historical references, the hardest strategy; sarcasm/figurative language accounted for 36% and other indirect strategies for 16%.The authors describe the skew toward historical and political reference as a Persian-specific finding, distinct from distributions reported in English implicit-hate research (ElSherief et al., 2021).Source: §A.1 · p. 11
  • Both LLaMA-3-FT and Gemma-3-FT predicted non-hate for three representative cases spanning text-localisable to maximally knowledge-dependent hostility.The examples cover other-indirect hostility, sarcasm/figurative language, and cultural/historical reference.Source: §A.1 · p. 11
  • The other-indirect example conveys hostility through identity denial and collective accusation, despite lacking a slur, metaphor, or named referent.Its exclusionary phrasing is localisable from the surface text alone.Source: §A.1 · p. 11
  • The sarcasm example uses animalisation as an insult, but its hostile force depends partly on the culturally specific affective weight of calling someone a dog.The dehumanising comparison is a text-internal structural cue.Source: §A.1 · p. 11
  • The cultural/historical example depends on knowing the contested Ahmadinejad–Chávez funeral incident; without that background, the tweet reads as a neutral generalisation.Source: §A.1 · p. 11

A.2 Span-Detection Analysis

Span-level evaluation reveals a sharp difference between explicit-hate boundary errors and implicit-hate localisation failures. Most implicit span errors produce no span, consistent with the model’s difficulty recognizing implicit hate.

A.2 Span-Detection Analysis

  • Span evaluation uses token-level F1 on all gold-hate tweets, converting character-level annotations through Hazm tokenisation.The span-level rationales identify the text segments supporting each hate annotation.Source: §A.2 · p. 12
  • LLaMA-3-FT achieved overall span F1 = 44, with F1 = 58 on explicit hate and F1 = 14 on implicit hate.Gemma-3-FT showed the same pattern; explicit errors were mainly boundary errors, while implicit errors were mainly failures to produce a span or to mark the relevant anchor.Source: §A.2 · p. 12
  • 80% of misclassified implicit-hate span predictions were no-span errors, while 12% marked the wrong anchor, 5% marked the whole tweet, and 3% were other partial-overlap cases.The authors connect the dominant no-span mode to detection failure: if the model does not recognize hate, it has nothing to localise.Source: §A.2 · p. 12
  • In the explicit-hate example, LLaMA-3-FT marked only fragments of the first gold span and missed the closing slur-and-dehumanisation clause.The authors characterise this as under-coverage across two non-contiguous spans and an inability to capture the full rhetorical arc.Source: §A.2 · p. 12
  • For implicit hate, the model can miss a rhetorical anchor when its literal wording contains no overt hostility marker.The example’s gold span is the clause “gives off a sense of killing,” which implies menace and dehumanisation in colloquial Persian.Source: §A.2 · p. 12
  • On implicit hate, span localisation failed approximately 80% of the time even when models were given known hate tweets to analyse.The authors argue that hostility may reside in the relation between text and an external referent or culturally entailed implication, rather than a localisable token; annotated rationales may support future rationale-aware or knowledge-augmented training.Source: §A.2 · p. 12

A.3 Annotator Demographics

Three female native Persian speakers carried out the annotations; all had lived in Iran for over twenty years and held graduate degrees.

A.3 Annotator Demographics

  • Three female native Persian speakers, aged 25–35, annotated the dataset; each had lived in Iran for over twenty years and held a graduate degree.Two annotators had mathematics degrees and one had a computer science degree; they were compensated at a fixed rate and could take breaks or opt out.Source: §A.3 · p. 12

A.4 Full Disagreement Statistics

The appendix reports disagreement patterns for implicit-hate strategies and rationale spans, then sets out the annotation rules for identifying hate, strategies, targets, and spans.

A.4 Full Disagreement Statistics

  • 48% of implicit-hate tweets use cultural or historical references, which have the lowest strategy agreement (κ = 0.43).Sarcasm or figurative language accounts for 36% of cases (κ = 0.55), while other indirect strategies account for 16% (κ = 0.58).Source: §A.4 · p. 13
  • Implicit-hate spans cover 31% of tweet characters versus 17% for explicit hate, and are more often clause-level and entity-rich.Implicit spans are 57% multi-token clauses, 29% noun phrases, and 14% mixed or multi-clause selections; named entities appear in 43% of implicit spans versus 13% of explicit spans.Source: §A.4 · p. 13
  • 16% of implicit-hate tweets and 6% of explicit-hate tweets required adjudication beyond majority voting.Annotators could assign multiple strategies; remaining disputes, often about implicitness or figurative language qualifying as hate, were discussed using guidelines and pilot examples, with unresolved ties defaulting to majority labels.Source: §A.4 · p. 13

A.5 Annotation Guidelines

  • Annotators label identity-based hostility as hate, while distinguishing it from offensive or violent content that does not target identity.The guidelines define hate by hostility toward an individual or group based on identity-related categories and caution that violence or offensiveness alone is not sufficient.Source: §A.5 · p. 16
  • Implicit hate is coded by strategy—figurative language, historical or cultural reference, or other indirect means—with multiple strategies allowed when they co-occur.Figurative language includes irony, sarcasm, or exaggerated comparisons; historical or cultural references use events, figures, or symbols associated with hatred.Source: §A.5 · pp. 16–17
  • The annotation workflow identifies hate, its explicitness and any implicit strategy, the target and its explicitness, then the hate span.Targets may receive multiple labels from seven categories, with relevant subcategories; annotators mark the shortest, most specific text expressing the hate.Source: §A.5 · p. 18
Loading 2609.16393v1…