Source-linked AI summary
Deconstructing Stereotypes: Scope-Conditioned Generation for Effective Multilingual Counterspeech
Greta Damo, Elias Urios Alacreu, Elena Cabrio, Paolo Rosso, Serena Villata
TL;DR
The paper addresses generic counterspeech that fails to target stereotypes underlying hate speech. It conditions multilingual LLM generation on structured stereotype characteristics and evaluates the approach on a human-curated English, Spanish, and Italian dataset. Stereotype conditioning consistently improves counterspeech quality over generic and implied-statement-only generation, including for implicit stereotypes and across languages.
Problem
Existing counterspeech generation can produce generic replies that fail to address the implicit stereotypes underlying hate speech.
Method
The paper uses a fine-grained multilingual stereotype dataset and conditions LLM prompts on structured stereotype information beyond implied statements.
Results
Stereotype-conditioned generation consistently improves counterspeech quality over generic and implied-statement-only generation for explicit and implicit stereotypes across English, Spanish, and Italian.
Takeaways & Limitations
Structured stereotype representations provide transferable guidance for multilingual counterspeech generation, including when harmful assumptions are implicit.
Takeaways & Limitations
The experiments cover only a subset of available LLMs, so effectiveness should not be interpreted as generalizing to all models.
Abstract
from arXiv · showhide
Counterspeech (CS) - direct responses that counter online Hate Speech (HS) using reasoning and alternative viewpoints - has emerged as an alternative to content removal. Current automatic CS generation methods, however, frequently produce generic, ineffective replies that fail to target the implicit stereotypes behind HS. To bridge this gap, we propose a novel scope-conditioned generation framework that explicitly integrates structured stereotype characteristics into Large Language Models prompts. We validate our approach on a novel, human-curated dataset annotated in English, Italian, and Spanish. Extensive evaluations show that stereotype-conditioned prompting substantially outperforms generic baselines across all three languages, obtaining significant gains in factuality, specificity, cogency, and effectiveness for both explicit and implicit implied stereotypes.
1 Introduction
The paper argues that effective counterspeech must address the stereotypes underlying hate speech, including distinctions that standard implied statements omit. It introduces multilingual stereotype-conditioned generation and reports gains across languages and stereotype types.
- Motivation: Stereotypes can drive hate speech through explicit claims or implicit hostility, so effective counterspeech must address the underlying belief rather than only surface wording.Implicit stereotypes may be conveyed through tropes or indirect hostility, making their underlying premises harder to identify.
- Problem: Standard implied statements expose latent beliefs but omit generalization scope and attributed trait type, risking generic or misaligned counterspeech.These omitted dimensions include whether a claim is universal, majority-based, or essentialist, and whether the attributed trait is moral, biological, or another type.
- Contributions: The paper introduces a human-curated dataset for English, Spanish, and Italian with fine-grained annotations beyond standard implied statements.The schema captures implicit or explicit stereotypes, generalization scope, and attributed trait type.
- Research questions: The study asks whether stereotype conditioning improves counterspeech over generic and implied-statement-only generation, including for implicit stereotypes and across languages.The research questions cover overall quality, the difficulty of implicit stereotypes, and cross-lingual transfer.
- Contributions: Stereotype-conditioned prompting significantly improves factuality, specificity, and cogency across three languages and both explicit and implicit stereotypes.The evaluations compare stereotype-conditioned generation with generic baselines and assess its multilingual transfer.
2 Related Work
Related work connects stereotype modeling with hate-speech detection and counterspeech generation. Existing generation methods address response properties but give limited attention to the stereotypes underlying hateful content.
- Stereotypes in Language: Stereotypes often convey harm through implicit generics and pragmatic implicatures, motivating interventions that target underlying inferences rather than surface hate.This work situates stereotype-aware counterspeech within research on language, bias, and hate speech.
- Prior methods: NLP research on stereotypes has focused mainly on supervised detection, while counterspeech generation has explored human responses, decoding, tone control, prompting, argumentation, and retrieval.These approaches address detection or response generation through different modeling and control strategies.
- Research gap: Prior counterspeech generation primarily addresses explicit hate speech and response style, leaving limited attention to countering the stereotypes behind hateful messages.This gap motivates the paper’s stereotype-conditioned generation framework.
3 Data
The study builds a multilingual hate-speech and counterspeech resource and enriches it with structured stereotype annotations. Annotation covers implicitness, stereotype presence, generalization scope, and attributed trait type, with reliability and distribution analyses.
- Dataset: The dataset contains 496 hate-speech/counterspeech pairs per language, with Spanish and Italian instances human-translated from English.The languages are English, Spanish, and Italian, and human counterspeech responses provide evaluation references.
- Annotation dimensions: Each hate-speech message is annotated for hate-speech implicitness, stereotype presence, generalization scope, and attributed trait type.Stereotype scope and trait type are annotated only when a stereotype is identified; stereotype explicitness distinguishes direct claims from inferred ones.
- IS extraction: The extracted implied statements were produced with multilingual mBART, expanded with additional data to approximately 37,000 samples, and manually validated by two annotators.The implied statement represents the latent negative stereotype conveyed by a hateful message.
- Annotation process: Three native speakers per language developed and refined annotation guidelines through training and a 30-message pilot before full annotation.Gold labels use majority voting, with an expert assigning labels when no multiclass majority exists.
- Dataset statistics and reliability: Most messages contain a recoverable stereotype—476 in English, 470 in Spanish, and 453 in Italian—and majority generalizations are the most common scope.Existential and Cultural Threat is the most frequent trait type across languages, while stereotype presence has the lowest agreement and implicit hate speech varies most cross-lingually.
4 Experimental Setting
The experiments compare four prompting conditions across multilingual and language-specific open-source LLMs in English, Spanish, and Italian. Counterspeech is evaluated with automatic metrics plus LLM and human judgments of qualitative properties.
- Models: The study uses instruction-tuned open-source models from different families with similar 7–9B parameter sizes, including multilingual and monolingual models.The setup keeps generation parameters fixed while testing whether language specialization affects counterspeech quality.
- Generation conditions: Four conditions compare hate speech alone, hate speech plus implied statement, hate speech plus annotations, and all three inputs.Conditions C and D provide structured stereotype information, with D also including the extracted implied statement.
- Automatic evaluation: Automatic evaluation measures lexical and semantic overlap with reference counterspeech, argumentative stance consistency, and diversity.Metrics include METEOR and BERTScore F1 alongside embedding-based similarity and stance-related measures.
- Qualitative evaluation: LLM judges and human experts rate counterspeech on six qualitative dimensions using five-point Likert scales, with human evaluators also selecting preferred responses.The study uses native expert speakers in each language and reports agreement analyses across evaluators.
- Evaluation design: The experiment compares prompting strategies across languages, models, and conditions using automatic, LLM-based, and human evaluations.Table 3 organizes the automatic results by language, model, and generation condition.
5 Results
Across automatic, LLM-as-a-Judge, and human evaluations, stereotype-aware prompting improves counterspeech quality across languages, with only marginal added benefit from including implied statements.
- 5 Results: NLI contradiction increases substantially under C and D while entailment with HS decreases, indicating more explicit opposition without increased repetition.For English, contradiction nearly doubles relative to A and B; METEOR improves slightly, while BERTScore is marginally higher for the baseline.
- 5 Results: Condition B performs similarly to the HS-only baseline, whereas stereotype information in C and D accounts for most gains and IS adds only marginal complementary benefit.C and D maintain similar semantic similarity and low repetition while improving contradiction and other quality measures.
- 5 Results: Conditions C and D consistently outperform HS-only and HS+IS prompting across languages and LLMs, improving counterspeech quality.The comparison is reported in automatic and LLM-as-a-Judge evaluations, with human evaluation confirming the same trend.
- 5 Results: 4.27 versus 3.61: in Italian, Llama’s overall score rises from Condition A to C, reaching 4.33 in D.Similar trends are reported for EuroLLM and Ministral, with the same pattern extending to Spanish and English.
- 5 Results: Condition D achieves the highest effectiveness and cogency scores in every language in human evaluation, while remaining consistently safe and correct.D is also the most preferred condition in every language; C and D differ only slightly on individual dimensions.
6 Discussion
Qualitative examples show that stereotype conditioning produces more targeted counterspeech for explicit and implicit sexist hate speech. These examples align with quantitative gains across evaluation settings and languages.
- 6 Discussion: Conditions C and D directly challenge harmful generalizations, whereas A and B reject hate speech generically without addressing its underlying stereotype.The conditioned responses use factual evidence, counter-examples, and explicit rebuttals of implicit assumptions.
- 6 Discussion: The qualitative abortion examples illustrate that conditioned responses explain why sexist stereotypes are unfounded rather than merely opposing the hateful statement.The examples cover both an explicit sexist claim and an implicit denial of workplace discrimination.
- 6 Discussion: Stereotype-conditioned generation improves performance for implicit stereotypes despite their lower scores and greater intrinsic difficulty than explicit cases.Appendix results indicate consistent gains for implicit messages, supporting stereotype annotations as grounding for inferred harmful assumptions.
- 6 Discussion: Stereotype-conditioned generation improves factuality, specificity, and cogency across English, Spanish, and Italian for general-purpose and language-adapted LLMs.The findings indicate that stereotype structure provides transferable guidance for multilingual counterspeech generation.
7 Conclusion
The paper introduces structured stereotype-aware prompting and a multilingual annotated dataset for counterspeech generation. Results show improved quality over generic and implied-statement-only generation across implicit cases and languages.
- 7 Conclusion: The approach conditions counterspeech generation on structured stereotype information rather than only hate-speech surface form or inferred implied statements.The released dataset annotates stereotype characteristics in English, Spanish, and Italian.
- 7 Conclusion: Across automatic, LLM-as-a-Judge, and human evaluations, stereotype structure consistently improves counterspeech quality over generic and IS-only generation.Reported benefits include more targeted, informative, and persuasive responses.
- 7 Conclusion: The benefits extend to implicit stereotypes and transfer across languages, supporting stereotype representations as a language-independent source of guidance.The conclusion frames explicit stereotype structure as a promising direction for multilingual counterspeech generation.
Limitations
The study identifies limitations involving model coverage, evaluation transparency and subjectivity, annotation disagreement, and cross-lingual variation.
- The experiments cover only a subset of LLMs, so effectiveness should not be interpreted as generalizing to all models.Broader evaluation across architectures, sizes, and training data is needed to assess robustness.
- LLM-as-a-Judge scoring is not fully transparent because judgments across evaluation dimensions may influence one another.The authors mitigate this concern with multiple judge models and agreement reporting.
- CS effectiveness is subjective across individuals, cultures, and communities, while the human evaluation covers relatively few participants and examples.The study combines automatic, LLM-based, and human evaluation, using diverse native expert evaluators.
- Majority-vote gold labels do not explicitly represent annotator disagreement or subjectivity, despite good overall inter-annotator agreement.The authors also observe language differences whose sources may involve models, data, cultural factors, or linguistic factors.
- Observed cross-lingual differences leave the roles of models, data, cultural factors, and linguistic factors unresolved.Understanding these sources is identified as an important direction for future work.
8 Ethical considerations
The ethical considerations emphasize risks from biased, harmful, or inaccurate generated counterspeech and potential misuse of released resources. They therefore require human oversight, restricted resource access, and informed consent for exposure to hateful content.
- LLM-generated counterspeech may be biased, harmful, or inaccurate, so the application should not operate autonomously without human oversight and monitoring.The proposed method specifically targets implied stereotypes in hateful messages.
- The code and data are intended for research-only access to mitigate potential misuse.
- Annotators and evaluators were recruited as volunteers who gave informed consent after learning about task risks, aims, and expected benefits.
- The study evaluates multilingual counterspeech with fixed generation settings across instruction-tuned models, including multilingual and language-specialized systems.The listed settings include top-p 0.9, repetition penalty 1.1, maximum 300 new tokens, and temperature 0.01.
- The prompting framework incrementally adds hateful-message, implied-statement, and stereotype characteristics across generation conditions.Condition D combines all provided information; the prompt instructs models to generate counterspeech directly challenging hateful messages without repeating them.
C Statistical significance analysis
Statistical analyses compare stereotype-informed generation conditions with HS-only and HS-plus-IS baselines using paired, corrected tests. Most LLM-as-a-Judge contrasts are significant and favor the stereotype-informed conditions, with no significant contrast favoring the baselines.
- Automatic comparisons use paired Wilcoxon signed-rank tests with Benjamini–Hochberg correction for A vs C, B vs C, A vs D, and B vs D.A substantial proportion of differences across four LLMs and languages are significant at p FDR < 0.05, although effects vary by metric and model.
- The six families with at least one nonsignificant contrast are reported separately, including cases where n/a indicates all paired differences are zero.
- Across 77 language×model×metric families, 71 show all four corrected contrasts significant in the expected direction, with C and D outperforming A and B.Among 291 significant contrasts, the median matched-pairs rank-biserial correlation is r = 0.89.
- LLM-as-a-Judge comparisons use paired two-sided Wilcoxon tests with Pratt’s zero method and Holm–Bonferroni correction within each language×model×metric family.Effect sizes are reported as matched-pairs rank-biserial correlations r.
- The judge evaluates counterspeech on factuality, specificity, effectiveness, correctness, safety, and cogency using five-point scores.The dimensions cover factual content, directness to the hateful claim, persuasive potential, fluency, non-harmfulness, and reasoning strength.
D.1 Complete LLM-as-a-judge results
The complete LLM-as-a-Judge results are presented in Table 9, with separate tables disaggregating performance for explicit and implicit stereotypes.
- Table 9 reports disaggregated LLM-as-a-Judge scores together with their statistical significance.
- The evaluation therefore separates overall judge results from analyses of explicit and implicit stereotype cases.
- Table 10 reports the corresponding results for explicit stereotypes, while Table 11 reports results for implicit stereotype cases.
E Additional IAA results
Additional agreement analyses show generally consistent reliability across the three languages, but negative Krippendorff’s α values for Correctness and Safety require interpretation because label skew can destabilize the coefficient.
- E Additional IAA results: Krippendorff’s α agreement across EN, ES, and IT is broadly consistent with the levels observed for the individual languages.
- E Additional IAA results: 62.0%–97.0% exact Correctness agreement and at least 83.0% within-one-point agreement show substantial raw agreement despite negative α values.The negative coefficients arise when α discounts agreement expected from highly skewed label distributions.
- E Additional IAA results: Safety in English is the exception, with 46.3% exact agreement indicating genuine annotator divergence and unreliable reliability estimates.