PaperlayerView exact revision on arXiv

Source-linked AI summary

Hatebench in the era of safer LLMs

Ole Becker, Tobias Jongen, Philip Kolbe, Sonal Khosla, Vaibhav Bajpai

arXiv:2609.05169v1cs.ET

TL;DR

The paper asks whether HateBench’s detector weaknesses generalize from human-authored benchmarks to evolving LLM-generated hate and modern moderation systems. It independently reconstructs and extends the benchmark, reevaluates detectors, and reproduces adversarial campaigns. Overall, the original findings generalize, newer LLM safeguards reduce harmful generation, and omni-Moderation is slightly more robust than text-Moderation.

  • Problem

    Existing hate speech detectors are typically trained or evaluated on human-authored data, leaving their reliability on LLM-generated and adversarial hate uncertain as models evolve.

  • Method

    The study reconstructs HateBench’s dataset-generation pipeline with modern LLMs, adds newer models and detector versions, reevaluates detectors, and reproduces adversarial hate campaigns.

  • Results

    The original HateBench findings generalize to newer models and datasets; newer LLM safeguards reduce harmful generation, adversarial results replicate, and omni-Moderation is slightly more robust than text-Moderation.

  • Takeaways & Limitations

    Detector vulnerabilities and adversarial weaknesses persist sufficiently to support the original conclusions, while evolving LLMs and moderation systems affect measured detection effectiveness.

  • Takeaways & Limitations

    Adversarial success-rate estimates are sensitive to dataset composition and a small evaluation size, motivating larger and more diverse subsets.

Abstract

from arXiv · show

As Large Language Models (LLMs) lower the barrier for au- tomated content generation, the potential for producing hate speech poses a significant challenge for digital safety. This paper presents a reproducibility study of the HateBench paper by Shen et al., investigating whether existing hate speech detectors, typically trained on human-authored data, generalize to LLM-generated hateful content, and evaluating whether their reported weaknesses are stable over time and robust to evolving components. We independently reconstruct the original dataset genera- tion pipeline using modern LLMs and extend the benchmark to include recently released models and updated detector versions. Our independent assessment under current con- ditions finds that for newer LLMs, safeguards have been put into place to prevent the generation of harmful content. We also replicate the results for two sophisticated types of hate campaigns. While the original findings seem to have been overestimated slightly due to bias in the datasets, the overall findings can be confirmed. Finally, we compare text- Moderation against the newer omni-Moderation and find that its robustness against adversarial hate campaigns has improved slightly. By clarifying which detector vulnerabil- ities persist, this study informs the community about the longevity of content moderation measurements.

1 Introduction

The study examines whether hate speech detectors generalize from human-authored data to LLM-generated and adversarial hate, and whether reported weaknesses persist as models and detectors evolve.

  • LLM accessibility lowers the cost of generating abusive content and could enable automated, coordinated hate campaigns that overwhelm moderation pipelines.
  • Existing detectors are widely used for dataset cleaning, moderation, auditing, and LLM safety evaluation, despite being mainly evaluated on human-authored datasets.
  • The unresolved question is whether detectors reliably identify synthetically generated or adversarially crafted hate speech.
  • Prior work reports failures on subtle or paraphrased hate, newer-model outputs, and targeted adversarial attacks.
  • The replication adds questions about whether original claims and detector weaknesses persist, and whether modern LLM guardrails have improved.
  • This study independently reconstructs HateBench, adds modern LLMs, reevaluates original and updated detectors, and reproduces adversarial attack setups.

2 Background and Related Work

The paper situates HateBench within concerns about distribution shift, evolving LLM-generated hate, and the need to test whether detector robustness persists across changing systems.

  • Traditional hate detectors often underperform under distribution shift, subtle toxicity, and adversarial reformulation, while LLMs add scalable harmful generation.
  • HateBench benchmarks detectors on LLM-generated hate and adversarial campaigns, motivating follow-up tests of generalization, evolving attacks, and newer moderation systems.
  • Recent work indicates a shift toward implicit, contextual, and multilingual hate speech.
  • SoftHate-Bench shows that systems detecting explicit hate can fail on policy-compliant but hostile, argument-based discourse.
  • This paper tests whether original HateBench findings still hold with newer LLMs and current moderation systems rather than proposing a new benchmark.

3 HateBench Replication Setup

The replication reconstructs HateBench’s dataset and evaluation pipeline, extends it with newer models, and uses LLM-based annotation to support larger-scale assessment while documenting label noise and dataset differences.

  • The framework comprises dataset construction, hate speech detector selection, and performance assessment.
  • ExtendedHateBenchSet uses six prompts across 34 identity groups, repeated three times for nine LLMs under original or jailbreak statuses.
  • Newer safeguards prevented GPT-4, GPT-OSS, and GPT-5-nano from generating hate samples, while Mistral generated the most among newly introduced models.
  • The regenerated dataset preserves relative distributions across identity groups and jailbreak status despite different absolute counts.
  • An LLM judge replaces infeasible full manual annotation, assigning hate, non-hate, or not-applicable labels to the generated samples.
  • 13.6% of labels differed from original human annotations, with the highest mismatch rate for OPT at 28.9%.
  • The LLM judge achieved an F1-score of 0.865 on hate, with accuracy 0.866, recall 0.918, and precision 0.817.
  • LLM-based annotation introduces noise but enables substantially larger evaluation than a small manually curated dataset.

4 Assessment

The assessment largely reproduces HateBench’s detector findings while showing that threshold choice, detector family, LLM source, jailbreak status, and identity group materially affect performance. Updated models and moderation systems add some improvements, but substantial weaknesses remain.

  • Threshold sensitivity: Threshold selection substantially affects detector performance, especially for volatile models such as HSBERT and BERT-HateXplain.The study reports major threshold differences for BERT-HateXplain and text-Moderation and uses thresholds maximizing F1.
  • Detector performance: TweetHate remains the best detector on HateBenchSet, while omni-Moderation achieves the second-best F1 score alongside text-Moderation.The replication’s TweetHate F1 differs from the original by 0.02, and omni-Moderation performs strongly overall on the extended set.
  • Detector performance: Detoxify improves by 0.02 F1 for Original and 0.081 for Unbiased, with recall gains of 0.053 and 0.157, respectively.These gains make the Detoxify models the strongest detectors for recall.
  • Jailbroken outputs: Detector performance consistently drops on jailbroken outputs, with every detector showing at least a 0.1 F1 difference between original and jailbroken content.The replication reports a larger original-versus-jailbroken gap than the original paper, with the magnitude varying by detector family.
  • Human-written comparison: On human-written MHS data, most detectors score at least 0.1 F1 worse than in the paper, whereas Detoxify performs similarly or better.This contrasts with the stronger detector performance observed on LLM-generated content.
  • Identity-group variation: For the sexuality_gay group, most detectors produce many false positives, including Perspective labeling 508 non-hateful samples as hate speech while correctly identifying 87.The issue disappears when the test is rerun using the original paper’s results, implicating the study’s optimal thresholds.
  • Overall assessment: Overall, the paper’s findings are mostly reproducible, but detector adaptation to newer LLMs remains necessary and performance remains worse for newer models than for GPT-3.5.The authors also report that performance differences attributable to the generating LLM persist.

5 LLM-Driven Hate Campaigns

The study reproduces the main findings for adversarial and stealthy LLM-driven hate campaigns, while showing that attack effectiveness varies with dataset construction, detector thresholds, and attack design. Updated and surrogate-based evaluations confirm persistent detector weaknesses, although newer moderation systems show some improvement.

  • Adversarial hate campaigns: Word-level attacks consistently achieve the highest attack success rate across targets, reproducing the original adversarial-campaign effectiveness finding.On OriginalHateBenchAttackSamples, reproduced ASR values closely match the original results.
  • Adversarial hate campaigns: Attacks are more effective on OriginalHateBenchAttackSamples than on the new subset, with a mean ASR difference of 0.062 and ρ=0.92 across comparable results.The difference is statistically significant (p=0.0012), while the rank ordering remains strongly associated (p<10^-4).
  • Adversarial hate campaigns: Dataset composition and sample selection substantially affect ASR estimates, with the original subset likely easier to misclassify and only 120 samples evaluated.A small fraction of borderline samples and subset variance may explain part of the observed shift.
  • Adversarial hate campaigns: The qualitative trends generalize across original, new, and extended datasets, supporting reproducibility despite numerical differences.Extended datasets show larger deviations, with mean absolute differences of 0.047 for Perspective and 0.038 for TweetHate.
  • Attack quality: TextFooler generally preserves hateful content best, whereas Paraphrase has lower equivalent-hatefulness and can cause identity-group drift or loss of hatefulness.In one manual review, Paraphrase on Perspective produced five of ten samples with target drift and four of ten no longer hateful.
  • Attack quality: Restricted TextFooler preserves identity-group terms but costs effectiveness of about 0.09 and requires 33% more queries than unrestricted TextFooler.Paraphrase averages 16 queries but generally has lower effectiveness, illustrating an effectiveness–efficiency trade-off.
  • Stealthy hate campaigns: Model-stealing surrogates reproduce target-detector behavior, and stealthy attacks transfer successfully with reasonable ASR while requiring minimal queries.BERT and RoBERTa perform similarly on quality metrics, while BERT is slightly more query-efficient.
  • Stealthy hate campaigns: White-box surrogate access slightly improves stealthy attacks, producing higher ASR and lower query counts while retaining black-box access to the target detector.The result matches the original study’s finding and reflects a realistic attacker model.

6 Conclusion

The study replicates and extends HateBench, finding that its main conclusions generalize to newer models and datasets, while evolving systems affect measured detection effectiveness. The authors note labeling-noise and dual-use constraints.

  • Conclusion: Detector performance on the original and newly constructed datasets is broadly consistent with HateBench’s findings despite newer versions.The study assesses nine detectors using the original dataset and ExtendedHateBenchSet.
  • Conclusion: The replication provides evidence that HateBench’s findings generalize to newer models and datasets.The authors also report that evolving LLM and moderation systems can affect hate-speech generation and measured detector effectiveness.
  • Limitations: Manual annotation was not possible for newly generated samples, so the study relies on an LLM-as-a-judge setup that may introduce labeling noise.The authors nevertheless report consistency with HateBench, particularly for overlapping LLMs.
  • Ethics and AI: The generated data is synthetic and handled under precautions intended to avoid exposing human subjects or external annotators to harmful content.The project stores and processes generated data securely and restricts redistribution beyond reproducibility needs.
  • Ethics and AI: AI tools support coding, documentation, and sample labeling, while the authors retain responsibility for scientific reasoning, experimental design, and interpretation.
  • Ethics and AI: The study acknowledges dual-use risks and emphasizes mitigation insights rather than exploitation details.Its findings are not disclosed to detector owners because the original study already reported highly similar results.

B.2.1 Detector Performance on Extended Dataset.

The extended-dataset evaluation reports performance for four standard metrics and finds omni-Moderation strongest overall, with TweetHate and text-Moderation also performing well.

  • Evaluation: The evaluation reports F1 score, Accuracy, Precision, and Recall on the extended dataset containing newly added LLMs.
  • Detector performance: F1 = 0.891 for omni-Moderation, the strongest overall performance among the evaluated detectors.The table uses F1-optimized thresholds and also reports Accuracy, Precision, and Recall.
  • Detector performance: F1 = 0.824 for both TweetHate and text-Moderation, while BERT-HateXplain reaches F1 = 0.725.

C.1.1 Reproduction on Original Attack Sample Set.

On the original sampled attack set, TextFooler is most effective across the evaluated detectors, while Paraphrase is more efficient despite lower attack success rates.

  • Attack effectiveness: TextFooler achieves ASR = 0.966 for Perspective and ASR = 0.991 for omni-Moderation.ASR measures the fraction of perturbed samples misclassified as non-hate.
  • Attack efficiency: Paraphrase attacks require 13–19 queries and 2–19 seconds, making them the most efficient despite lower ASR.
  • Attack quality: The attacks generally preserve semantic similarity and grammaticality, with USE > 0.8.The evaluation also measures hatefulness and other quality properties.
  • Attack efficiency: Query counts range from 18–598, reflecting differences in the computational complexity of attack methods.

C.1.2 Replication on Extended Dataset.

On the extended dataset, adversarial vulnerabilities remain broadly persistent across newly generated content, although attack effectiveness and efficiency vary by method and detector.

  • Attack effectiveness: TextFooler retains ASR > 0.91 on the extended dataset, indicating high attack effectiveness across most attacks.
  • Attack variation: PWWS on Perspective declines by −0.116, showing that some attacks are sensitive to dataset characteristics.
  • Attack quality: USE remains above 0.84 and Meteor scores stay consistent with the original results, indicating comparable perturbation quality.
  • Attack efficiency: +94.5 queries are required for PWWS on Perspective on the extended dataset, illustrating larger efficiency variance.
  • Temporal robustness: The persistence of vulnerabilities across newly generated content underscores the temporal stability of the identified weaknesses.The evaluation compares extended-dataset results with the original findings and accounts for sample variability.

C.1.3 Robustness Assessment via Resampling.

Resampling checks indicate that attack effectiveness is generally stable across different samples, although some detector–attack combinations show greater sensitivity. The evaluation uses an LLM judge applying the UN hate-speech definition to generated samples.

  • Stability across resampling: Relative metric differences are generally small, typically within ±0.05 ASR, indicating stable attack performance across sampling.Some detector–attack pairs show larger deviations, such as TextBugger on Perspective at −0.082.
  • Stability across resampling: TextBugger on Perspective shows a −0.082 ASR deviation, illustrating occasional sampling sensitivity.
  • Quality preservation: Quality metrics remain largely consistent, with semantic similarity above 0.83 on USE and grammaticality maintained.
  • Sample classification: The automatic annotator labels samples as hate, non-hate, or not applicable using a prompt based on the UN definition.The prompt notes that profanity alone does not necessarily constitute hate speech.
  • Interpretation: These stability checks support the conclusion that the observed attack effectiveness reflects detector vulnerabilities rather than only specific sample choices.The results suggest that attack effectiveness is reasonably robust to resampling.

C.2.1 Black-Box Model Stealing Attacks.

Black-box model stealing attacks train surrogate detectors on auxiliary data and use them to attack target detectors, but transfer success is substantially less consistent than surrogate performance. Reproduction results broadly match the original findings while showing moderate variation in target transfer.

  • Method: Black-box attacks train BERT or RoBERTa surrogates with auxiliary data and evaluate both surrogate ASR and transfer ASR on target detectors.ASR (T) measures whether the stolen surrogate generalizes to real detectors.
  • Attack performance: Surrogate models achieve ASR (S) values of 0.9–1.0, while target transfer ranges from 0.21 to 0.53.The gap indicates limited generalization for some surrogate–target pairs.
  • Attack performance: RoBERTa surrogates generally transfer slightly better than BERT, and TweetHate reaches target ASR of up to 0.521.Attack quality remains high, with USE above 0.75 and Meteor above 0.80.
  • Reproducibility: Reproduction surrogate performance is typically within 0.05 ASR (S) of the original, while target transfer shows moderate agreement and some variation.For example, Perspective RoBERTa yields 0.403 in reproduction versus 0.471 originally.

C.2.2 White-Box Gradient-Based Attacks.

White-box gradient information makes stealthy attacks more effective and less query-intensive than black-box attacks. The reproduced vulnerability persists across newer LLM-generated content, with RoBERTa often transferring better than BERT.

  • Overall performance: White-box attacks achieve target ASR values from 0.22 to 0.64 and require fewer queries than black-box attacks.Gradient access enables computationally more efficient perturbation optimization.
  • Surrogate comparison: TweetHate reaches ASR (T) = 0.639 with a RoBERTa surrogate, compared with 0.513 in black-box mode.RoBERTa shows stronger transfer in white-box attacks, while BERT performance is more variable.
  • Temporal robustness: Newer LLM-generated samples produce comparable white-box ASR (T) values, typically 0.17–0.42, to the original samples.This indicates that the vulnerability remains observable across contemporary generated content.
  • Efficiency and quality: White-box attacks exceed 0.3 ASR (T) for most surrogate–target pairs and typically require only 1–2 detector queries.Quality metrics remain high, with USE above 0.78 and Meteor above 0.83.
  • Implication: The results characterize stealthy white-box attacks as a persistent threat even when newer detector models are deployed.The paper concludes that robustness improvements may need to extend beyond model updates alone.

C.3 Effect of Auxiliary Dataset Size

Larger auxiliary datasets generally improve model-stealing transfer and stealthy campaign success, whereas low-data settings produce less stable performance. This instability depends partly on the attack dataset and detector.

  • Method: The auxiliary-data analysis evaluates stealthy attacks using a RoBERTa surrogate and varies the data available for surrogate training.
  • Dataset-size effect: Increasing auxiliary dataset size generally improves target-detector ASR by producing a better surrogate approximation.The analysis uses a RoBERTa surrogate and evaluates both original and newly sampled attack sets.
  • Dataset dependence: TweetHate on NewHateBenchAttackSamples loses ASR more sharply with small auxiliary datasets than on the original split.Performance degrades more gracefully as the dataset shrinks on OriginalHateBenchAttackSamples.
  • Dataset-size effect: Low-data regimes introduce greater attack-performance variance, while larger auxiliary datasets enable stronger model stealing and stealthy campaigns.
  • Evaluation setup: Figure 11 reports selected, nonlinearly spaced auxiliary sample sizes and identifies 7,282 balanced HateBenchSet samples as the full dataset.
Loading 2609.05169v1…