PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoHarmBench: Breaking Content Moderation with Iterative Human-Like Evasion

Ruijie Jian, Benlei Cui, Ting Ma, Haidong Ding, Kangwei Liu, Ziwen Xu, Longtao Huang, Hui Xue, Ziqiang Zhu, Junjie Li, Haiwen Hong

arXiv:2608.27844v1cs.CL

TL;DR

Static harmful-content benchmarks do not capture users’ feedback-driven, iterative adaptation, motivating a dynamic evaluation framework for moderation robustness. EvoHarmBench combines real-world adversarial data, semantic clusters, and iterative LLM-generated rewrites; experiments across moderation systems reveal substantial vulnerabilities and a broader weakness in recovering human-readable harmful intent.

  • Problem

    Existing harmful-content evaluations rely largely on fixed datasets and therefore do not directly measure robustness to iterative, feedback-driven adversarial rewriting.

  • Method

    EvoHarmBench uses real-world adversarial variants, semantic-cluster organization, and an iterative LLM-driven protocol that rewrites content using human mutation strategies and moderation feedback.

  • Results

    Experiments across thirteen moderation systems, including leading commercial models, show that attack success rises substantially under iterative rewriting and static evaluation underestimates these vulnerabilities.

  • Takeaways & Limitations

    The findings highlight a fundamental limitation of current language models in robustly recovering intended meaning expressed through flexible human linguistic strategies.

  • Takeaways & Limitations

    EvoHarmBench targets adversarially rewritten harmful content and does not assess false positives on benign inputs, so it is not a balanced moderation benchmark.

Abstract

from arXiv · show

Existing evaluations of harmful content detection rely predominantly on static benchmarks, which struggle to reflect the interactive adversarial ecosystem of real-world content platforms where users continuously revise their expressions in response to moderation feedback. This mismatch creates a significant performance gap between offline benchmark scores and online deployment effectiveness. To the best of our knowledge, we present EvoHarmBench, the first dynamic adversarial evaluation framework for content moderation systems. The framework employs an iterative optimization loop that evolves evasion strategies at the semantic-cluster level, while simultaneously optimizing for evasion success and human readability. We systematically evaluate LLM-based defense models which are widely used in real world moderation systems. The evaluation covers 229 semantic sub-clusters across five violation categories, derived from 5,002 real-world adversarial samples collected from content platforms. Our experiments reveal substantial vulnerabilities even in leading commercial systems: after twelve optimization iterations, the attack success rate under readability constraints reaches 80.3% within SOTA LLM moderators. We will release the full benchmark data, evaluation framework, and code to encourage a shift from static benchmarking toward dynamic adversarial evaluation in content safety research.

1. Introduction

Automated moderation is necessary at platform scale, but harmful users can iteratively rewrite content in human-readable ways that evade LLM-based detection. EvoHarmBench addresses the resulting static-evaluation gap with adaptive, readability-preserving evaluation and finds substantial vulnerabilities in current moderators.

  • Automated moderation systems are increasingly used because manual review cannot scale to the volume of harmful and policy-violating platform content.
  • Harmful intent can remain understandable to humans while becoming difficult for LLMs to detect through homophones, character splitting, metaphors, and community-specific argot.
  • Existing benchmarks largely use fixed datasets, so they test recognition of known evasive variants rather than robustness to feedback-driven, iterative adaptation.
  • EvoHarmBench combines real-world human-generated adversarial variants with an iterative framework that follows human mutation strategies and conditions rewrites on live moderation responses.
  • The framework evaluates LLM-based defense models and shows that leading commercial moderators remain substantially vulnerable under dynamic, readability-preserving adversarial evaluation.
  • The findings indicate that current language models do not robustly recover intended meaning expressed through flexible human linguistic strategies, even when people can understand it.

2. Related Work

Prior harmful-content benchmarks and adversarial resources generally evaluate fixed datasets, while iterative attack frameworks target different objectives such as jailbreaks or generic classifier robustness. EvoHarmBench instead measures feedback-driven, reusable cluster-level evasion for moderation systems under readability and intent-preservation constraints.

  • Static resources such as COLD, SafetyBench, ChineseHarm-Bench, Jigsaw, HateCheck, and RealToxicityPrompts operate on fixed, non-interactive corpora.
  • Adversarial resources including DynaHate, Silent Signals, STATE ToxiCN, PCR-ToxiCN, EVADE-Bench, and LiveSecBench improve evasive coverage but remain fixed-dataset evaluations.
  • Model-centric approaches develop safety models and guardrails, whereas EvoHarmBench provides a dynamic protocol for measuring moderation robustness under feedback-driven, human-like evasion.
  • Iterative frameworks such as PAIR, TAP, GAP, PyRIT, and Garak primarily probe jailbreak or security vulnerabilities rather than harmful-intent recognition in moderation.
  • EvoHarmBench differs by optimizing reusable evasion patterns across semantic clusters while preserving readability and harmful intent, rather than optimizing attacks for individual inputs.

3. Benchmark

EvoHarmBench builds a cluster-based benchmark from real-world adversarial harmful content and evaluates whether moderation systems still recognize harmful intent after adaptive rewriting. Its protocol jointly measures moderation bypass and recognizable intent, then iteratively refines reusable strategies using model feedback.

  • Benchmark overview: The benchmark combines a real-world adversarial harmful-content resource, semantic cluster construction, and an iterative rewrite-and-evaluate protocol.
  • Benchmark objective: EvoHarmBench evaluates whether harmful intent remains detectable after mutation, rather than whether the surface form itself can be identified as mutated.
  • Data resource: The dataset contains 5,002 real-world adversarial samples collected from multiple platform types, screened for observed moderation-evasion behavior and covering five violation categories.
  • Data resource: Five moderation experts annotate risk category, original meaning, and rewriting strategy; independent dual labeling plus third-expert checking supports cluster construction and strategy initialization.
  • Semantic clusters: The benchmark forms 229 semantic sub-clusters from texts sharing harmful intent and mutation patterns, making them reusable units for cluster-level strategy evaluation.
  • Iterative protocol: Adaptive strategy generation, rewriting, comparison, and refinement are LLM-driven, while the target moderation system is the only task-specific component under evaluation.
  • Scoring objective: For each cluster, EvoHarmBench evaluates evasion and intent preservation jointly, requiring successful attacks to bypass moderation while retaining human-recognizable harmful intent.
  • Iterative protocol: The evaluate-then-propose loop rewrites cluster samples, scores evasion and intent preservation, diagnoses failures, and accepts only strategies that strictly improve the cluster-level score.

4. Experiments

EvoHarmBench evaluates moderation systems through iterative adversarial rewriting across diverse risk categories and semantic clusters. Strong commercial LLM moderators remain highly vulnerable, with failures rising rapidly before plateauing and affecting many reusable evasion patterns.

  • Experimental Setup: 229 semantic sub-clusters across five risk categories are evaluated under a 12-round iterative protocol.The categories cover advertising and traffic diversion, gambling and fraud, abusive, pornographic, and spam and flooding content.
  • Experimental Setup: Thirteen moderation backbones are evaluated, including seven state-of-the-art LLMs and billion-scale open-source models with base and safety-focused SFT variants.Adaptive components use a fixed DeepSeek-V3.2-Exp backbone, while scoring is primarily based on ASR@Readable.
  • Main Results: 80.3% average ASR@Readable is reached by state-of-the-art LLMs, while the most vulnerable model, GLM 5.1, reaches 88.8%.These results show that conventional moderation strength does not translate into robustness under iterative adversarial interaction.
  • Main Results: 91.2% average ASR@Readable is reached by billion-scale models across base and SFT variants, substantially exceeding the state-of-the-art group.Several models exceed 92% overall ASR@Readable, indicating near-complete defense failure on a large portion of the benchmark.
  • Safety-Focused SFT: SFT reduces Qwen3-4B ASR@Readable from 92.2% to 89.1%, but corresponding Qwen3-8B and DeepSeek-V2-Lite variants remain at 91.4% and 92.5%.The reported effect is limited and inconsistent, with weak transfer from static moderation training to feedback-driven rewriting.
  • Iterative Vulnerability Analysis: Attack success typically rises rapidly in early rounds and then gradually plateaus, while later rewrites better balance evasion with preserved intent and readability.This pattern indicates that a small number of interaction rounds can discover effective reusable evasion strategies.
  • Iterative Vulnerability Analysis: One-shot evaluation cannot capture human-like rewriting through interaction, so static benchmarking substantially underestimates vulnerability in adaptive settings.The paper reports that this underestimation is even more pronounced for smaller open-source models.

5. Conclusion

EvoHarmBench addresses the inability of static benchmarks to capture feedback-driven adaptation by evaluating reusable, cluster-level evasion strategies. Across thirteen moderation systems, iterative rewriting exposes vulnerabilities that static evaluation substantially underestimates.

  • 5. Conclusion: EvoHarmBench evaluates harmful-content moderation through iterative, semantic-cluster-level adversarial testing rather than static benchmark inputs.The framework is grounded in real-world adversarial content and organized around semantic clusters for reusable evasion testing.
  • 5. Conclusion: Across thirteen moderation systems, attack success rises substantially under iterative rewriting, revealing vulnerabilities that static evaluation significantly underestimates.The conclusion reports this pattern across leading commercial models and other moderation systems.
  • 5. Conclusion: Current language models do not robustly recover intended meaning when harmful content uses flexible human linguistic strategies, even when people can understand it.This conclusion identifies the broader limitation exposed by the evaluation.

Limitations

EvoHarmBench is a targeted robustness benchmark rather than a complete evaluation of moderation quality. Its Chinese-platform setting and LLM-driven adaptive components constrain how broadly its results and measurements should be interpreted.

  • Limitations: EvoHarmBench focuses on adversarially rewritten harmful content and does not measure false positives on benign inputs.It should therefore not be interpreted as a balanced moderation benchmark.
  • Limitations: The benchmark is instantiated on Chinese content platforms, so its data distribution, mutation patterns, and platform conventions may not transfer directly elsewhere.The authors identify multilingual and cross-platform extension as future work.
  • Limitations: The benchmark’s rewriting, comparison, diagnosis, and refinement components are LLM-driven, creating possible evaluator bias or dependence on underlying LLM capabilities.This design also enables a unified and scalable simulation of iterative adversarial behavior.

Ethical Considerations

The authors frame EvoHarmBench as a defensive research artifact while acknowledging that adversarial evaluation tools have dual-use risks. They anonymize collected data and restrict release to mitigate those risks.

  • Ethical Considerations: EvoHarmBench exposes moderation weaknesses under realistic adversarial conditions but could also help attackers develop stronger evasion strategies.The authors explicitly recognize this dual-use risk.
  • Ethical Considerations: The release excludes raw adversarial samples and platform-specific metadata, while collected data are anonymized to remove personally identifiable information.Access is provided through a controlled release process.
  • Ethical Considerations: The benchmark is released strictly as a research artifact for evaluating and improving content moderation systems, not for production moderation deployment.The commercial and open-source model use is described as consistent with research-oriented terms or licenses.
  • Ethical Considerations: The harmful-content data were publicly posted policy-violating material that platforms had subsequently removed, with use grounded in platform Terms of Service.The passage states that individual consent from original posters was not required.
  • Ethical Considerations: Five professional content-moderation specialists performed the annotations after being briefed about the research purpose and potential exposure to offensive content.They were recruited based on prior platform-safety review experience and compensated per piece.
  • Ethical Considerations: AI assistants supported manuscript polishing, experiment scripting, and result analysis, with all assisted outputs reviewed and approved by human authors.The authors describe human verification as part of the preparation process.

A. Direct-Audit Baseline

The Direct-Audit Baseline submits the same harmful inputs to moderators without adversarial rewriting, isolating the effect of iterative adaptation. Under this baseline, ASR@Readable equals the moderator’s harmful-input miss rate.

  • A. Direct-Audit Baseline: The Direct Audit baseline submits the same 5,002 harmful benchmark inputs without rewriting, using identical moderation prompts.Because no rewriting occurs, harmful intent is preserved by construction.
  • A. Direct-Audit Baseline: 38.4% is the average direct-audit miss rate across state-of-the-art moderators.Direct reports the miss rate for inputs submitted without rewriting.
  • A. Direct-Audit Baseline: 80.3% is the average ASR@Readable after 12 adaptive rewriting rounds, compared with the 38.4% direct-audit miss rate.The reported increase is 42.0 percentage points, computed from unrounded values.

B. Additional Robustness Checks

Additional checks show that EvoHarmBench’s results depend partly on the adaptive backbone but transfer substantially across unseen moderators. The protocol iteratively rewrites clusters, evaluates bypass and intent preservation, diagnoses failures, and accepts only strictly improving strategies.

  • Adaptive-backbone sensitivity: 39.9% overall ASR@Readable for Qwen3-32B versus 73.0% for DeepSeek-V3.2-Exp when Gemini 3.1 Pro is fixed as target moderator.The comparison tests sensitivity to the adaptive backbone while holding the target moderator constant.
  • Cross-moderator transfer: 48.7–55.4% target ASR@Readable and 65.7–73.2% source-normalized transfer ratios show substantial effectiveness on unseen moderators without further optimization.The transfer test samples final-round rewrites from each source moderator and re-audits them on other moderators.
  • Iterative protocol: Each iteration rewrites every sample in a semantic cluster, evaluates bypass and intent preservation, and uses the resulting feedback to propose a refined strategy.The Defense Model supplies bypass signals and rationales, while the Comparison Model supplies intent-preservation scores and explanations.
  • Iterative protocol: Only strategies satisfying Score(st+1; X) > Score(st; X) are accepted, so rejected proposals leave the current strategy unchanged and preserve monotonic progress.Algorithm 1 initializes a seed strategy, iterates up to T rounds, and returns the best strategy s∗.

E. Clustering Details

EvoHarmBench organizes adversarial samples into semantically coherent groups using normalized multilingual embeddings, DBSCAN, recursive splitting, and a per-group sampling cap. The final pipeline yields 229 evaluation sub-groups from 47 base categories and 5,002 samples across five risk categories.

  • Text representation: BAAI/bge-m3 embeddings are L2-normalized before DBSCAN clustering with Euclidean distance, supporting Chinese and English samples.Clustering is performed independently within each risk category.
  • Clustering procedure: DBSCAN uses ε = 0.5 and min_samples = 10 for traffic diversion, versus ε = 0.35 and min_samples = 8 for the other four categories.These settings were selected through a small grid search favoring semantically coherent clusters.
  • Recursive splitting: Large clusters are recursively reclustered, with heuristic partitioning applied when necessary to separate residual attack patterns.A second clustering round targets clusters exceeding 2,000 samples and retains splits only when they produce multiple coherent sub-clusters.
  • Cluster distribution: 47 semantically distinct base categories expand into 229 evaluation sub-groups containing 5,002 evaluation samples across five risk categories.The category distributions are 9/44 for traffic diversion, 19/54 for pornography, 3/42 for abuse, 6/43 for spam, and 10/46 for gambling and fraud, reported as base categories/evaluation sub-groups.
  • Cluster distribution: A per-group cap of 25 samples preserves real-world diversity while preventing dominant attack patterns from overwhelming the evaluation.The raw data are highly imbalanced at the base-category level.

F. Safety-Focused SFT Details

The safety-focused SFT study fine-tunes three open-source moderation backbones on balanced expert-labeled data and evaluates them under the unchanged iterative attack protocol. Supplementary analyses also validate attack readability, intent preservation, and vulnerable semantic scenarios.

  • SFT models: Three billion-scale models—Qwen3-4B, Qwen3-8B, and DeepSeek-V2-Lite—are fine-tuned into safety-focused SFT variants.The resulting models are evaluated as moderation backbones against adaptive, feedback-driven attacks.
  • Training data: The SFT data contain 5,000 production-platform content samples labeled harmful or non-harmful by experts in a balanced 1:1 ratio.The data cover the same five risk categories as EvoHarmBench.
  • Training setup: LoRA-based supervised fine-tuning uses BF16, learning rate 1×10−5, 3 epochs, sequence length 512, and rank r = 64 with α = 8.All three models share the same configuration for fair comparison.
  • Human validation: 438 audit-passing mutations remain after filtering 520 sampled outputs, and human annotation checks whether each still conveys the original harmful intent.The sample includes balanced groups with comparison_score ≥0.5 and comparison_score <0.5 before retaining blind_score = 1.0 cases.
  • Vulnerable clusters: The most vulnerable clusters include social-media contact diversion, gambling-platform promotion, and pornographic content.Table 8 ranks the top ten clusters by cross-system average ASR@Readable.
  • Qualitative analysis: Qualitative examples show evolved strategies including digit transformations, numeric homophones, character splitting, variant characters, metaphors, and official-looking notices.The examples illustrate how iterative rewriting balances evasion with recognizable harmful intent across semantic clusters.

K. Reproducibility

The reproducibility procedures make scoring deterministic, isolate rewriting-model sampling, and test stability across repeated optimization runs. Supplementary prompts and mode-specific guidance document the defense, comparison, rewriting, and reflection components.

  • Deterministic scoring: Defense and Comparison Models use temperature 0, while the Rewriting Model uses temperature 0.7 to generate diverse rewrites.Identical inputs therefore yield deterministic scoring outputs, while cluster-level acceptance reduces sensitivity to per-sample generation variance.
  • Run consistency: Three repeated 12-round runs on 20 fixed clusters produce standard deviations below 1.5 pp for every open-source defense model.The benchmark remains stable despite substantial inter-cluster variance.
  • Prompting: The protocol includes category-specific moderation prompts for advertising, gambling and fraud, abuse, pornography, and spam.These prompts specify whether the Defense Model should block or pass text and provide a brief rationale.
  • Rewriting strategies: The Rewriting Model starts from a seed strategy and receives iteratively refined cluster-level prompts after reflection rounds.Tables 16 and 17 document the initial and evolved strategy prompts.
  • Comparison scoring: The Comparison Model scores intent preservation and readability using a 1–5 rating normalized as comparison_score = (s −1)/4.Its prompt and category-specific rubric define the Intent & Readability Check.
  • Reflection: The Reflection Model combines the current strategy, per-sample feedback, failure diagnosis, and evolution history to propose an improved strategy.Mode-specific guidance distinguishes Intent Loss, detection, and oscillation.
Loading 2608.27844v1…