PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection

Arij Riabi, Virginie Mouilleron, Menel Mahamdi, Wissam Antoun, Djamé Seddah

arXiv:2412.11745v2cs.CL

TL;DR

Few annotated datasets cover radical social-media content, despite its links to offline harm. This paper introduces COUNTER and analyzes annotation variation and synthetic sociodemographic profiles; its findings highlight subjectivity and demographic variation in detection outcomes.

  • Problem

    Few annotated datasets cover radical social-media content, although online extremism can intensify views and contribute to offline harms.

    Source: §1 · p. 1

  • Method

    The authors introduce a pseudo-anonymized English, French, and Arabic dataset and analyze human annotation variation and synthetic sociodemographic profiles in relation to model outcomes.

    Source: §1 · pp. 1–2, §3.4 · p. 4

  • Results

    Annotation subjectivity and sociodemographic variation influence data and model outcomes, and French shows greater fairness challenges than English for some demographic categories.

    Source: §1 · p. 2, Sociodemographic biases impact… · p. 8, §6 · p. 9

  • Takeaways & Limitations

    The findings highlight annotation subjectivity and sociodemographic variation as complexities in detecting radical content.

    Source: §6 · p. 9

    Synthetic data may not represent real-world complexity and diversity, restricting applicability in larger, multicultural, or international settings.

    Source: Limitations · p. 10

Abstract

from arXiv · show

The proliferation of radical content on online platforms poses significant risks, including inciting violence and spreading extremist ideologies. Despite ongoing research, existing datasets and models often fail to address the complexities of multilingual and diverse data. To bridge this gap, we introduce a publicly available multilingual dataset annotated with radicalization levels, calls for action, and named entities in English, French, and Arabic. This dataset is pseudonymized to protect individual privacy while preserving contextual information. Beyond presenting our freely available dataset, we analyze the annotation process, highlighting biases and disagreements among annotators and their implications for model performance. Additionally, we use synthetic data to investigate the influence of socio-demographic traits on annotation patterns and model predictions. Our work offers a comprehensive examination of the challenges and opportunities in building robust datasets for radical content detection, emphasizing the importance of fairness and transparency in model development.

1 Introduction

Online radical content can intensify extreme views and contribute to offline harms, while few annotated social-media datasets cover it. COUNTER addresses this gap and examines how annotation variation and synthetic sociodemographic profiles relate to detection outcomes.

1 Introduction

  • Few annotated datasets cover radical content on social media, even as online extremism can intensify views and contribute to offline violence.The paper describes online spaces as potential rallying points and routes into communities where extreme opinions can be further intensified.Source: §1 · p. 1
  • COUNTER is a pseudo-anonymized dataset spanning English, French, and Arabic, with annotations for named entities, radicalization level, and calls for action.The authors also release annotation guidelines, disagreement information, and synthetic data for bias analysis.Source: §1 · pp. 1–2
  • The study examines how human label variation affects model outcomes and uses synthetic data to probe sociodemographic influences on annotations and predictions.The authors report that outcomes depend heavily on annotation aggregation and evaluation, and highlight subjectivity and sociodemographic variation as challenges.Source: §1 · p. 2

2 Online Radical Content Detection

Prior radical-content detection research faces limited dataset quality, narrow coverage, and annotation challenges. Existing work often focuses on particular behaviors or groups, while radicalization is complex and multilingual.

2 Online Radical Content Detection

  • Radicalization definitions and associated language evolve, complicating detection algorithms and potentially diminishing model effectiveness over time.The paper describes radical content as expression opposing a political, social, or religious system, which may be followed by behavioral shifts and violence.Source: §2 · p. 2
  • Many NLP datasets focus on specific behaviors or extremist communities, and most treat radicalization as binary despite its gradual complexity.Some studies distinguish hostile from irrelevant content or use categories such as propaganda, radicalization, and recruitment.Source: §2 · p. 2
  • Training and evaluation data are constrained by bias and weak quality checks, including simplistic collection rules and limited manual verification.Crowdsourced rather than expert annotation can add further bias.Source: §2 · p. 2
  • Research has concentrated on English and US-centered settings, with few studies in other languages and no large-scale multilingual work reported by the authors.The authors argue that multilingual data with rich annotations can support analysis across cultural and linguistic boundaries.Source: §2 · p. 3

3 COUNTER: Radical Content Dataset

COUNTER combines multilingual data collection with prescriptive and descriptive annotation, pseudonymization, and synthetic profiles for bias analysis. Its annotation and generation procedures expose variation and limits that shape how the dataset can be used.

3.1 Data collection

  • The dataset collects English, French, and Arabic posts from social media, platforms, and forums, covering Jihadism, far-right ideology, and other radical tendencies.Collection used keywords inspired by geopolitical events; the ideology mix differs by language, with far-right dominant in English and French and Jihadism in Arabic.Source: §3.1 · p. 3

3.2 Prescriptive Annotation

  • A contractor created the first version using prescriptive guidelines, with multi-label and multi-class annotations for calls for action and radicalization levels.Calls for action has five levels; radicalization level has six, ranging from negative or neutral content to calls for action against others.Source: §3.2 · p. 3

3.3 Descriptive Annotation

  • Because details of the contractor’s annotation process were unavailable, two trained linguists descriptively re-annotated English and French data, with only fair or slight agreement on English labels.French agreement was higher: Fleiss’ Kappa was 0.50 for Radicalization Level and 0.43 for Call for Action, versus 0.26 and 0.13 for English. Pseudonymization retained well-known events and public figures, and training on pseudonymized and original data gave comparable results.Source: §3.3 · pp. 3–4

3.4 Synthetic Data for Bias Analysis

  • Zero-shot persona prompting used a generative model to create posts for annotation, since direct extraction of sociodemographic information from the dataset was costly and difficult.The profiles varied selected traits and drew inspiration from real COUNTER profiles; annotators noted stereotypical keywords, and the model struggled with age and register. Agreement on 300 examples was moderate across languages and tasks.Source: §3.4 · p. 4

4 Results

Experiments compare multilingual classifier designs and auxiliary features for call-for-action classification. Language-specific classifiers perform best across languages, while gains from additional tasks or entity features vary by language and data setting.

4 Results

  • XLM-T achieved Macro-F1 scores from 59.41 for Arabic to 65.65 for French on the Call for Action test set.The experiments report average Macro-F1 over five seeds and select checkpoints using validation data.Source: Table 2 · p. 5, §4 · p. 5
  • Adding Radicalization Level as an auxiliary task did not substantially improve performance, and French performance dropped notably.The authors relate this observation to annotators’ report that English text was cruder, more violent, and derogatory than French text.Source: §4 · p. 5
  • Adding Ideology Prediction produced minor improvements in English, while French and Arabic remained relatively stable or declined slightly.The authors suggest its benefit may depend on language-specific features or underlying data distribution.Source: §4 · p. 5
  • Out-of-domain NER showed the most promise in English, while within-domain NER did not improve results much and even reduced French performance.Arabic was excluded from within-domain NER because only 1,500 of 2,500 examples were annotated.Source: §4 · p. 5
  • MULTI-DIFF, with separate classification layers for each language, performed best across languages and outperformed the single-classifier MULTI-SAME approach.The authors suggest that handling language nuances separately is more effective than sharing a classifier across languages.Source: §4 · p. 5

5 Discussion

The discussion examines how annotation choices, demographic attributes, and task formulation shape radical-content model evaluation. It contrasts label-aggregation outcomes, synthetic-data bias analyses, and classification with regression.

5.1 How about Human Label Variations?

  • Models trained on their matching gold labels perform best; MACE-trained models lead on majority-labeled tests except when trained with majority labels.MACE and majority labels have Cohen’s kappa 0.89, compared with 0.70 for Contractor and MACE.Source: §5.1 · p. 6, Table 3 · p. 6

5.2 Bias Analysis: What can we infer?

  • Synthetic data is used as a bias diagnostic, comparing XLM-T’s average Macro-F1 across socio-demographic attribute categories rather than maximizing performance.Performance on synthetic data is lower than on the standard test set because of distribution differences.Source: §5.2 · p. 7, Figure 1 · p. 7

Sociodemographic biases impact performance

  • XLM-T has the highest overall performance but larger disparities than XLM-R and MBERT, especially across political views and nationality.French shows greater Place of Living and Job fairness challenges; demographic-parity differences are generally smaller than equalized-odds differences.Source: Sociodemographic biases impact… · pp. 7–8, Table 4 · p. 8
  • For demographic parity, the largest disparities concern Place of Living and Job, particularly in French; equalized-odds disparities also concern Ethnicity and Nationality, particularly in English.Political View and Gender show the smallest differences across accuracy and error-rate measures.Source: Sociodemographic biases impact… · p. 8, Table 4 · p. 8

5.3 Multi-Class Classification or Regression?

  • Classification achieves higher Macro-F1 across all three languages, while regression better preserves ordinal rankings and usually confines errors to adjacent classes.Regression has the largest Spearman-correlation improvement for Arabic; classification sometimes makes more critical non-adjacent errors.Source: §5.3 · p. 9, Figure 2 · p. 9, Table 5 · p. 9

6 Conclusion

The paper presents COUNTER, a multilingual pseudo-anonymized dataset for online radical-content detection, and examines dataset biases and human-label variation. Its experiments highlight annotation subjectivity and sociodemographic variation, while synthetic data offers a way to explore bias patterns.

6 Conclusion

  • COUNTER combines a multilingual, pseudo-anonymized dataset with analysis of dataset biases and the effects of human label variation.The experiments consider radicalization levels, calls for action, and named entities; the paper also explores synthetic data for studying sociodemographic bias.Source: §6 · p. 9
  • The authors report that human annotation is subjective and sociodemographic variations influence data and model outcomes.They also report that generative models have potential for simulating realistic annotations.Source: §6 · p. 9
  • The authors plan to explore weighted cross-entropy and metadata dimensions to address classification challenges and improve accuracy.They state that their findings underscore the importance of understanding and accounting for bias in datasets and models for radicalization detection.Source: §6 · p. 9

Limitations

The authors identify risks from underlying data bias, limits in synthetic data realism, and changes in radicalization language. They also note limited collaboration between social and humanist researchers and NLP and machine-learning researchers.

Limitations

  • Underlying data may bias the dataset and models, potentially producing stereotypes or overrepresenting particular groups and compromising result predictability.The authors also caution that synthetic data may not capture the complexity and diversity of real-world radical content, restricting applicability in larger, multicultural, or international settings.Source: Limitations · p. 10
  • New slang or coded language absent from the dataset may emerge over time, diminishing model applicability.Source: Limitations · p. 10
  • Limited collaboration across social and humanist research and NLP and machine learning keeps the latter from benefiting from radicalization theories, studies, and insights.The authors say combining disciplinary insights can help models include more features for radical-content detection.Source: Limitations · p. 10

Ethics Statement

The paper emphasizes ethical risks in radicalization detection, including biased predictions and misuse for profiling or preemptive prosecution. It also describes COUNTER’s data statement, curation rationale, and language coverage.

Ethics Statement

  • The authors caution that LLMs may propagate bias or reduce human judgment in a nuanced, sensitive task, and call for transparent, accountable scrutiny.They warn that inaccurate predictions could be misused to profile or preemptively prosecute individuals; the dataset is released to researchers upon demand, and they recommend human oversight.Source: Ethics Statement · p. 10
  • The authors identify privacy and harmful-profiling risks from social datasets and say they handled data to respect individuals’ privacy and dignity.Annotators also received psychological support, and the team established a process to evaluate the psychological impact of annotation; its results were still under institutional review.Source: Ethics Statement · p. 10

A Data Statement

A.1 CURATION RATIONALE

  • COUNTER targets multiple radicalization levels across Jihadism, far-right extremism, and unclassified ideologies; collection used radicalization-related keywords across two time frames and three languages.Sources included Reddit, Twitter, Facebook, Telegram, and 4chan, though only public Facebook groups and Telegram channels were searchable through public APIs.Source: §A.1 · p. 16

A.2 LANGUAGE VARIETY

  • The dataset includes English, French, and Arabic, with Modern Standard Arabic and dialectal forms, primarily France-based French, and English from varied locations.The represented registers range from formal statements to informal conversational speech, depending on platform and context.Source: §A.2 · p. 16

A.3 SPEAKER DEMOGRAPHICS

Speaker metadata is incomplete, with most English and Arabic geolocation unknown and no French geolocation available. The appendix also describes annotator demographics, post context, and text characteristics across the dataset.

A.3 SPEAKER DEMOGRAPHICS

  • Approximately 90% of English and Arabic geolocation data is unknown, and French geolocation is unavailable; pseudonyms, platform restrictions, and encrypted channels make complete speaker demographics difficult to obtain.Source: §A.3 · p. 16
  • Figure 3 presents data-source distributions for English, French, and Arabic.Source: Figure 3 · p. 16

A.4 ANNOTATOR RECRUITMENT

  • The authors report limited information about contractor annotations; the two annotators were research engineers with annotation experience and relevant language and cultural knowledge.Annotation cost was 72kC (12 person months), excluding supervision time and institutional overhead; psychological assistance was offered.Source: §A.4 · p. 17

A.5 ANNOTATOR DEMOGRAPHICS

  • Table 6 presents demographic information for ANNOT1 and ANNOT2.Source: Table 6 · p. 17

A.6 SPEECH SITUATION

  • Posts span 17/07/2015 to 03/04/2023, were collected from 24/07/2022 to 03/04/2023, and consist of asynchronous written text intended for public or semipublic audiences.Source: §A.6 · p. 17

A.7 TEXT CHARACTERISTICS

  • Post lengths vary across English, French, and Arabic; longer posts typically come from forums and shorter ones from social media such as Twitter.The authors applied minimal cleaning and pseudonymized text to preserve content while protecting privacy.Source: §A.7 · p. 17

B Radical content Dataset

The dataset organizes radical content into graded categories and uses stratified splits to preserve label distributions. Models trained on pseudonymized data showed comparable performance to those trained on the original data for Call for Action classification and NER.

B Radical content Dataset

  • Table 7 reports Cohen’s and Fleiss’ kappa for annotator agreement on Radicalization Level and Call for Action.The supplied table caption identifies the agreement measures and tasks, but provides no values.Source: Table 7 · p. 17

B.1 Radicalization Level classes description

  • The six radicalization categories range from no radicalization and expression of radical views to propaganda, group association, dehumanization, and calls for violent action.The descriptions distinguish expressing radical views without advocating violence from the most extreme category, which explicitly calls for violence or harm.Source: §B.1 · pp. 17–18

C.1 Dataset splitting

  • Stratified splitting binarizes labels to preserve each label’s distribution across the training, validation, and test sets.The approach follows Sechidis et al. (2011) for multi-label stratification.Source: §C.1 · p. 18

C.2 Effect of Pseudonymization on Model Performance

  • Models trained on pseudonymized and original data showed comparable performance on Call for Action classification and NER.The authors report that pseudonymization did not degrade either task’s model performance.Source: §C.2 · p. 18

C.3 Additional results

This section reports additional models for English and French and defines fairness metrics used to assess predictions across sensitive groups. The supplied results table captions specify task and model coverage but do not give comparative scores.

C.3 Additional results

  • XLM-R and MBERT were additionally trained on the French and English datasets.Table 9 is captioned as reporting test-set Macro-F1 for Call for Action classification across different models.Source: §C.3 · p. 18, Table 9 · p. 19

D Metrics Definitions

  • Demographic parity difference is the range between the largest and smallest group-level selection rates across sensitive-feature values.The section defines demographic parity as prediction independence from the protected attribute.Source: D Metrics Definitions · p. 19
  • Equalized odds requires predictions to be conditionally independent of the protected attribute given the label, and a zero difference means all groups share the same true- and false-positive and -negative rates.Source: D Metrics Definitions · p. 19

E Synthetic Data Generation for Bias Analysis

The authors report moderate agreement on two annotation tasks in English and French. The appendix also presents synthetic-data examples and bias-analysis materials covering attribute distributions and model performance variation.

E.1 Annotators Agreement

  • Agreement across 300 double-annotated English and French examples was moderate, with Cohen’s Kappa ranging from 0.40 to 0.54 across the two tasks and languages.English agreement was 0.51 for Radicalization Level and 0.40 for Call for Action; French agreement was 0.54 and 0.47, respectively.Source: §E.1 · p. 19
  • The appendix includes generated-data examples and a Call for Action confusion matrix averaged over five seeds for bias analysis.Source: Figure 6 · p. 21, Figure 7 · p. 21
  • The appendix captions describe attribute-distribution summaries and Macro-F1 variation plots for XLM-R and mBERT in English and all models in French.The captions identify the plots’ scope but do not state their outcomes.Source: Table 11 · p. 22, Figure 8 · p. 23, Figure 9 · p. 24

Related papers

Loading 2412.11745v2…