PaperlayerView exact revision on arXiv

Source-linked AI summary

AraDetox: A Multi-Dialect Arabic Detoxification Dataset

Mo El-Haj

arXiv:2608.22894v1cs.CLcs.AI

TL;DR

Arabic detoxification remains underexplored despite extensive research on harmful-language detection, and it must remove harmful expressions without losing meaning, stance, targets, or intent. AraDetox constructs and evaluates a large multi-dialect rewrite dataset using LLM generation, automatic analyses, and human review. The results characterize detoxification as substantial reformulation with consistently high semantic similarity, while dialectal outputs show measurable stylistic alignment.

  • Problem

    Arabic harmful-language research has focused mainly on detection, while text detoxification remains comparatively underexplored despite its need to remove harmful language while preserving meaning and communicative properties.

  • Method

    AraDetox uses GPT-5 and Gemini 2.5 Flash to generate rewrites across MSA, Gulf, Levantine, and Egyptian Arabic, then applies automatic analyses and native-speaker human evaluation.

  • Results

    Substantial lexical and structural reformulation is accompanied by consistently high semantic similarity, with mean multilingual E5 scores ranging from 0.9234 to 0.9308 across variants.

  • Takeaways & Limitations

    AraDetox demonstrates that large-scale Arabic detoxification resources can be built through LLM-assisted generation and human verification, supporting research on multi-dialect safe text generation.

  • Takeaways & Limitations

    The dataset is constructed primarily through GPT-5 and Gemini 2.5 Flash rather than large-scale human rewriting, so outputs may retain model-specific stylistic patterns.

Abstract

from arXiv · show

Arabic harmful-language detection has received considerable attention, yet Arabic text detoxification remains underexplored. We introduce AraDetox, a multi-dialect Arabic detoxification dataset comprising 10,500 harmful social-media posts and 84,000 detoxified rewrites generated using GPT-5 and Gemini 2.5 Flash across Modern Standard Arabic, Gulf, Levantine, and Egyptian Arabic. The generated outputs were assessed through human evaluation and automatic analyses of lexical change, semantic preservation, sentiment, and dialectal style. Results show that detoxification is primarily a meaning-preserving rewriting task: substantial lexical and structural reformulation is accompanied by consistently high semantic similarity. Human evaluation confirms successful harmful-language removal while largely preserving the original meaning. Dialectal analyses further indicate that the generated variants exhibit measurable stylistic alignment with reference Arabic dialect corpora. Comparison with existing resources highlights two complementary approaches to detoxification: minimal-edit lexical substitution and meaning-preserving reformulation. Our findings demonstrate that large-scale Arabic detoxification resources can be constructed through LLM-assisted generation and human verification. The dataset is publicly available at https://github.com/ArabicNLP-UK/AraDetox to support future research on Arabic detoxification, safe text generation, and multi-dialect Arabic NLP.

1 Introduction

Arabic harmful-language research has concentrated on detection, while detoxification remains underexplored because it must remove harmful expressions while preserving meaning, stance, targets, and intent. AraDetox addresses this gap with a large-scale, multi-dialect dataset and evaluation framework.

  • Arabic harmful-language research has produced many detection datasets and models, but comparatively little work addresses detoxification.
  • Manual detoxification dataset creation is costly because annotators must rewrite offensive and dialectal content while preserving meaning, stance, target, and communicative intent.
  • Successful detoxification requires meaning-preserving reformulation rather than simple lexical substitution.
  • AraDetox contains 10,500 harmful social-media posts and 84,000 rewrites across MSA, Gulf, Levantine, and Egyptian Arabic, generated with GPT-5 and Gemini 2.5 Flash.
  • The resource combines LLM-assisted generation, human quality control, and evaluations of lexical change, semantic similarity, sentiment, dialectal style, and human judgments.
  • Across models and dialects, substantial lexical and structural reformulation accompanies consistently high semantic similarity, while human evaluation supports harmful-language removal and meaning preservation.

2 Related Work

Arabic harmful-language research has mainly developed detection resources, whereas detoxification resources remain limited. AraDetox extends prior work with multi-dialect, multi-LLM rewrites that emphasize semantic preservation and substantially expand resource scale.

  • Prior Arabic harmful-language research has focused primarily on hate-speech, offensive-language, and abusive-language classification, with comparatively limited attention to detoxification.
  • AraDetox builds on LLM-assisted data-generation developments by providing detoxified Arabic social-media rewrites across MSA, Gulf, Levantine, and Egyptian Arabic.
  • Unlike existing Arabic detoxification resources, AraDetox emphasizes preserving the source text’s meaning, intent, stance, and target.
  • AraDetox substantially expands Arabic detoxification resources and, according to the authors, is the first to combine multiple dialects and multiple LLMs in one framework.

3 Methodology

AraDetox is constructed from Arabic harmful-language data through LLM generation, dialect adaptation, automatic analyses, and native-speaker evaluation. Its methodology separates detoxification quality, semantic fidelity, sentiment, and dialectal style while acknowledging limits of the dialect and embedding measures.

  • 3.1 Dataset Construction: AraDetox selects 10,500 harmful posts from a 16,000-post Arabic Hate Speech Superset and retains coverage across its constituent datasets and harm categories.
  • 3.1 Dataset Construction: GPT-5 and Gemini 2.5 Flash generate rewrites in MSA, Gulf, Levantine, and Egyptian Arabic while targeting preservation of meaning, target, stance, sentiment, and intent.
  • 3.2.4 Dialectal Adaptation: Dialectal outputs combine harmful-language detoxification with adaptation to the requested Arabic variety, so their lexical and structural differences cannot be attributed solely to detoxification.
  • 3.2 Evaluation: The evaluation measures lexical change with six surface metrics, including overlap, Jaccard similarity, edit distances, changed characters, and output length.
  • 3.2 Evaluation: Semantic fidelity is measured with multilingual-e5-large cosine similarity and complemented by UMAP visualization of the embedding space.
  • 3.2 Evaluation: Sentiment analysis tests whether detoxification shifts outputs toward neutral or positive predictions at the expense of negative ones.
  • 3.2.4 Dialectal Adaptation: Dialectal style is compared with reference corpora using TF–IDF cosine similarity over word and character n-grams, providing exploratory rather than definitive evidence of dialect authenticity.
  • 3.2.5 Human Evaluation Protocol: Human evaluation covers 2,400 generated outputs through 7,200 annotation records from three native Arabic-speaking annotators, assessing offence removal, meaning preservation, and new content.

4 Results

AraDetox outputs substantially reformulate harmful Arabic posts while preserving their meaning, softening affective tone, and showing measurable alignment with intended dialects. Human evaluation confirms high offence removal and meaning preservation, with dialectal style evidence strongest for MSA and Egyptian Arabic.

  • Lexical Change: Detoxification substantially modifies source text, with sentence-level reformulation rather than simple replacement of offensive expressions.Across variants, output lengths rise from 13.21 original words to 15.66–16.79 words, while character-level modification rates exceed 78%.
  • Lexical Change: 0.2942 overlap and 0.1719 Jaccard similarity make GPT-Gulf the most vocabulary-retentive variant, whereas Gemini-MSA and Gemini-Egyptian show greater lexical reformulation.The lexical results characterize detoxification as meaning-preserving rewriting rather than simple lexical substitution.
  • Semantic Preservation: 0.9234–0.9308 mean semantic similarity indicates that substantial lexical changes generally preserve the source posts’ core meaning.All examples exceed the 0.70 similarity threshold, and dialect-aware variants slightly outperform MSA variants; Levantine reaches 0.9308.
  • Semantic Preservation: Embedding visualisations show substantial overlap between original and detoxified posts, including dialectal variants, with no clear source–rewrite separation.The UMAP patterns are consistent with the high semantic-similarity scores.
  • Sentiment Shift: 72.41%–75.47% of original posts are classified as negative, compared with 49.20%–53.77% for GPT-MSA and 53.19%–60.26% for Gemini-MSA.Dialectal variants generally remain between 65% and 74% negative, preserving more of the original emotional tone.
  • Dialectal Style Analysis: 0.373 MSA similarity for Gemini-MSA and 0.406 Egyptian similarity for GPT-Egyptian provide the clearest word n-gram evidence of intended-dialect alignment.Gulf and Levantine alignment is clearer for Gemini, while GPT-Gulf and GPT-Levantine align most strongly with Egyptian references; character features separate these dialects more weakly.
  • Human Evaluation: 87.25%–92.90% offence-removal rates and 86.25%–92.57% meaning-preservation rates show that harmful language is generally removed without substantially altering the original message.Gemini achieves the highest offence-removal scores across language varieties.
  • Human Evaluation: GPT introduces new content in 24.17%–24.83% of outputs, compared with 12.42%–17.52% for Gemini, while meaning-preservation scores remain comparable.The additional GPT content generally supplements rather than alters the source’s core meaning.

5 Conclusion

AraDetox provides a large-scale, multi-dialect Arabic detoxification resource and shows that harmful-language removal can preserve source meaning, stance, and intent while often requiring substantial reformulation.

  • AraDetox contains 10,500 harmful social-media posts and 84,000 detoxified rewrites across MSA, Gulf, Levantine, and Egyptian Arabic.The resource combines LLM-assisted generation, automatic validation, native-speaker quality control, and independent human evaluation.
  • Arabic detoxification frequently requires substantial reformulation while preserving the source claim, target, stance, and communicative intent.
  • Human evaluation indicates high rates of offence removal and meaning preservation, although some outputs introduce additional content.
  • Dialectal variants show measurable corpus-level alignment with their intended Arabic varieties.
  • AraDetox supports developing and evaluating Arabic detoxification systems while highlighting continued needs in dialect validation, pragmatic meaning preservation, and downstream evaluation.

6 Limitations

The study identifies limitations concerning model-generated data, evaluation coverage, semantic and dialectal measurement, resource comparison, and generalisation beyond the source setting.

  • AraDetox relies primarily on GPT-5 and Gemini 2.5 Flash rather than large-scale human rewriting, so outputs may retain model-specific stylistic patterns.Native-speaker quality control and automatic validation were applied, but the outputs should not be treated as naturally occurring human rewrites.
  • Detailed human evaluation covers 300 source posts and 2,400 generated outputs, representing only a subset of the 84,000-output resource.Rare source types or harm categories may be underrepresented.
  • High cosine similarity from multilingual embedding models does not guarantee preservation of pragmatic meaning, sarcasm, humour, presupposition, target, or subtle stance.Human judgements partly address this limitation but cannot eliminate it.
  • Dialectal evaluation uses corpus-level word and character n-gram similarity that may reflect topic, register, corpus composition, and shared vocabulary.Because dialect authenticity was not separately evaluated by human annotators, the results indicate stylistic alignment rather than native-like dialect use.
  • Comparison with MultiParaDetox-Ar is descriptive, and downstream performance relative to existing Arabic detoxification datasets remains unassessed.A controlled downstream benchmark is left for future work.
  • The dataset uses two proprietary frontier model families and derives from Arabic social-media content, limiting assessment of smaller open-weight models and generalisation to long-form or formal political discourse.

7 Ethical Considerations

The study addresses harmful Arabic social-media content to develop safer detoxification resources while recognising ethical risks, contextual variation, and the need to protect interpretive and evidential value.

  • The dataset contains offensive, abusive, political, sectarian, and identity-related Arabic language, while the paper reports results in aggregate and minimises harmful-example reproduction.
  • The study used up to ten native Arabic-speaking adult annotators with informed consent, annotation guidance, and institutional ethical approval.No personal, sensitive, or health-related data were collected.
  • Detoxified outputs are not authoritative or universally acceptable because judgements of harmfulness and neutralisation depend on social, cultural, dialectal, and political context.
  • Automatic detoxification can oversanitise legitimate political expression, weaken intended stance, or remove important evidence of abuse.
  • Invalid refusals, malformed responses, summaries, external-observer descriptions, and substantial meaning changes were regenerated during construction.Human-evaluation labels distinguish offence removal, meaning preservation, and new-content introduction for evaluated outputs.
  • Generation prompts instructed models to produce four civil, non-offensive Arabic versions while preserving meaning, target, stance, criticism, sentiment, and communicative intent.The prompts also prohibited defending, refuting, explaining, summarising, fact-checking, or responding to the source statement.

B Generation Procedure and Quality

AraDetox generation combines batched structured-output calls to two LLM families with iterative prompt development, automatic validation, regeneration, and native-speaker quality control.

  • The final resource contains 42,000 GPT-generated and 42,000 Gemini-generated rewrites, yielding eight detoxified variants for each of 10,500 source posts.
  • Generation used structured JSON output and batching to maintain consistent formatting across variants.The API parameters not explicitly specified used provider defaults.
  • The study used Gemini for quality, latency, and cost considerations and GPT-5 as a complementary model family for cross-model comparisons, not to rank LLM performance.
  • The final prompt was selected after pilot testing on 100 source posts to reduce refusals, summaries, malformed JSON, unsupported additions, and changes to target or stance.
  • Automatic checks detected missing fields, malformed JSON, duplicates, null values, incomplete generations, API failures, empty responses, and safety refusals for regeneration.
  • Native Arabic-speaker quality control verified source alignment, meaning, target, stance, communicative intent, dialectal variety, natural usage, and technical consistency.The reviewer verified outputs rather than manually rewriting posts.
  • Safety mechanisms sometimes caused refusals or external-observer responses, which were treated as invalid and regenerated with additional task clarification when necessary.

C Experimental Pipeline

The experimental pipeline collects harmful Arabic social-media posts, generates detoxified rewrites across four Arabic varieties, and evaluates lexical, semantic, sentiment, dialectal, and human-evaluated properties. Human assessment focuses on offence removal, meaning preservation, and newly introduced content.

  • The pipeline collects harmful Arabic social-media posts from the Arabic Hate Speech Superset before generating detoxified rewrites.
  • GPT-5 and Gemini 2.5 Flash generate rewrites in Modern Standard, Gulf, Levantine, and Egyptian Arabic.
  • Automatic analyses measure lexical change, multilingual-E5 semantic similarity, embedding-space structure, sentiment shift, and dialectal style similarity.
  • Human evaluation uses three native Arabic-speaking annotators and aggregates inter-rater agreement and evaluation statistics.
  • Annotators assess offence removal, meaning preservation, and whether outputs introduce new content.Offence removal uses labels 0, 1, and 2, while the remaining criteria use binary judgements.

E Human-Evaluation Sample

The human-evaluation sample contains 300 source posts, with 30 posts sampled from each dataset. Table 10 compares this sample's distribution with the complete AraDetox source collection.

  • Table 10 compares the sample distribution with the distribution of the complete AraDetox source collection.
  • 300 source posts were included in the human-evaluation sample, with 30 posts sampled from each dataset.

F Qualitative Examples

Representative examples show that AraDetox outputs use multiple detoxification strategies across models and dialects. These include both localized lexical edits and broader reformulations while preserving relevant communicative content.

  • The examples illustrate insult mitigation, meaning-preserving reformulation, stance preservation, and dialect-specific adaptation.
  • GPT-5 and Gemini outputs differ in their use of direct lexical substitution versus broader paraphrastic rewriting.
  • The examples cover detoxification behaviours across multiple dialect-specific and meaning-preserving transformations.
Loading 2608.22894v1…