PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Large Language Models for Detecting Antisemitism

Jay Patel, Hrudayangam Mehta, Jeremy Blackburn

arXiv:2509.18293v2cs.CLcs.AIcs.CY

TL;DR

Automated hate-speech moderation must handle complex policies and changing social media, while LLM utility on sensitive tasks remains limited. This paper evaluates eight open-source LLMs with in-context policy prompts and introduces Guided-CoT; across nearly all models, it outperforms Zero-Shot and Zero-Shot-CoT. The study also analyzes generated explanations and model errors.

  • Problem

    Complex moderation policies and changing social media make adaptable automated detection challenging, while safety-tuned LLM releases can limit their utility on sensitive tasks.

    Source: §1 · pp. 1–2

  • Method

    The authors evaluate eight open-source LLMs using in-context antisemitism policy prompts, compare prompting approaches including Guided-CoT, and analyze explanations and misclassifications.

    Source: §1 · p. 2

  • Results

    Across nearly all models, Guided-CoT outperforms Zero-Shot and Zero-Shot-CoT regardless of decoding strategy, model size, or reasoning capability.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors’ findings identify model strengths and shortcomings in utility, explainability, and reliability and may inform qualitative audits of model behavior.

    Source: §7 · p. 10

    The comparison with fine-tuned GPT-3.5 relies on Becker et al. (2024)’s reported results, without details about when or how that model was trained.

    Source: §8 · p. 10

Abstract

from arXiv · show

Detecting hateful content is a challenging and important problem. Automated tools, like machine-learning models, can help, but they require continuous training to adapt to the ever-changing landscape of social media. In this work, we evaluate eight open-source LLMs' capability to detect antisemitic content, specifically leveraging in-context definition. We also study how LLMs understand and explain their decisions given a moderation policy as a guideline. First, we explore various prompting techniques and design a new CoT-like prompt, Guided-CoT, and find that injecting domain-specific thoughts increases performance and utility. Guided-CoT handles the in-context policy well, improving performance and utility by reducing refusals across all evaluated models, regardless of decoding configuration, model size, or reasoning capability. Notably, Llama 3.1 70B outperforms fine-tuned GPT-3.5. Additionally, we examine LLM errors and introduce metrics to quantify semantic divergence in model-generated rationales, revealing notable differences and paradoxical behaviors among LLMs. Our experiments highlight the differences observed across LLMs' utility, explainability, and reliability. Code and resources available at: https://github.com/idramalab/quantify-llm-explanations

1 Introduction

This paper evaluates whether open-source LLMs can detect antisemitism using an in-context policy, addressing the challenges of complex moderation rules and adapting classifiers to changing social media. It introduces Guided-CoT and examines performance, utility, explanations, and errors.

1 Introduction

  • Complex moderation policies depend on local laws and societal norms, making automated systems challenging to develop and deploy.The paper frames moderation policies as socio-technical specifications and notes that adapting classifiers to changing social media is difficult.Source: §1 · p. 1
  • The authors evaluate eight open-source LLMs without fine-tuning, using in-context prompting that includes the IHRA definition and contemporary examples.They explore prompting techniques, decoding strategies, explanations, and cases of misclassification.Source: §1 · p. 2
  • Across nearly all models, Guided-CoT outperforms Zero-Shot and Zero-Shot-CoT across decoding strategies, model sizes, and reasoning capabilities.The paper also reports that added policy context alone may not improve performance under Zero-Shot or Zero-Shot-CoT prompts.Source: §1 · p. 2
  • The study finds that LLMs misclassify quoted text, news-style reports, neutral or critical opinions, and terms or names resembling slurs as antisemitic.These errors include typos such as “kikes” intended as “likes” and proper nouns such as “Kiké.”Source: §1 · p. 2

2 Background & Related Work

The paper adopts the IHRA definition of antisemitism, while noting controversy around its examples and the interpretation of Israel-related speech. Prior work addresses hate-speech detection, policy-based prompting, and LLM explanations, but this study applies the IHRA definition as a policy for antisemitism detection.

2.1 Defining Antisemitism

  • The IHRA definition describes antisemitism as a perception of Jews that may be expressed as hatred, with rhetorical and physical manifestations directed at people, property, institutions, or religious facilities.The IHRA also supplies contemporary examples and has been adopted by governments and institutions, although its definition has been controversial.Source: §2.1 · p. 2
  • The paper notes free-speech concerns around the IHRA examples, including tension between claims about Israel’s self-determination and the allowance for criticism comparable to that of other countries.The authors state that understanding how LLMs apply this definition matters because of its real-world applications.Source: §2.1 · p. 3

2.2 Related Work

  • Prior research identifies policy nuance, socio-cultural context, inconsistent model judgments, bias, oversensitivity, and vulnerability to adversarial prompting as challenges in LLM moderation.Existing work also reports that changing definitions can alter toxicity scores and that models can have accuracy gaps for toxicity targeting Jewish individuals.Source: §2.2 · p. 3
  • This study examines LLM antisemitism detection using the IHRA definition as policy and introduces Guided-CoT and quantitative analysis of generated rationales.The authors position this work as the first exploration of LLM antisemitism detection using the IHRA definition as a policy.Source: §2.2 · p. 3

3 Experimental Setup

The experiments evaluate eight open-source LLMs on a human-labeled antisemitism dataset using free-form prompting followed by a classification label. Prompt variants include Zero-Shot, Chain-of-Thought, and Guided-CoT, which guides models through contextual and policy checks.

3 Experimental Setup

  • The study evaluates eight open-source LLMs from five model families, including both reasoning and non-reasoning models.The supplied model list includes Gemma, Mistral, Llama, QwQ, and DeepSeek-R1-Distill-Llama.Source: §3 · p. 3
  • Prompts allow free-form text followed by a summary and classification label, and the study compares Zero-Shot, Chain-of-Thought, and Guided-CoT approaches.Zero-Shot variants use the IHRA definition alone or with contemporary examples; Zero-Shot-CoT prompts models to think step by step.Source: §3 · p. 4
  • Guided-CoT directs models to inspect writing patterns, hidden tones, author intent, alignment with the IHRA policy, and stance toward the Jewish community.The sequence is designed to mimic human annotators’ thought processes.Source: §3 · p. 4
  • The dataset contains 11,315 human-labeled Twitter posts collected from January 2019 to April 2023, including 1,953 antisemitic posts.Because the dataset is highly unbalanced, performance is measured using positive-class F1, precision, and recall.Source: §3 · p. 4

4 Evaluations

Guided-CoT generally improves antisemitism-detection performance across models and decoding configurations, while also reducing some invalid responses. Ablations identify useful reasoning instructions, and adding policy context alone does not improve every model.

4.1 Differences in Generated Responses

  • Guided-CoT reduces Llama 8B’s refusal rate to 0.07%; Gemma 12B, Gemma 27B, Llama 8B, and QwQ 32B often produce invalid responses across prompt variants.QwQ 32B exceeds the token limit primarily when the IHRA definition with contemporary examples is added.Source: §4.1 · p. 5

4.2 Deterministic Evaluation

  • Guided-CoT improves performance across all models, whereas adding the IHRA definition alone yields mixed results and adding contemporary examples still leaves exceptions.Under ZS-α, most models slightly decrease or remain unchanged except QwQ 32B (+0.03); ZS-β improves most models except Llama 8B and Mistral 24B.Source: §4.2 · p. 5

4.3 Non-deterministic Evaluation

  • 0.66 overall F1-score: Llama 70B_q achieves the highest score under self-consistency, while Guided-CoT improves every model over ZS-CoT, including Llama 8B by 0.13.The authors report an increase of 0.09 over ZS-β and ZS-CoT for Llama 70B_q.Source: §4.3 · p. 5
  • 0.73 F1: Llama 70B_q with self-consistency outperforms fine-tuned GPT-3.5 at 0.70 on the Version 1 comparison.The authors rely on Becker et al.’s reported metrics and lacked access to their training data or model.Source: §4.3 · pp. 5–6

4.4 Guided-CoT Ablation

  • Thoughts checking sarcasm or criticism, promotion versus awareness, and the author’s stance toward the Jewish community are important across models; chunking does not help except for Mistral 8B.Excluding A1 yields about a 0.05 F1 decrease for Mistral 8B; excluding the explicit IHRA-alignment thought may improve most models.Source: §4.4 · p. 6

4.5 Impact of Additional Context across Decoding and Prompting

  • Across decoding configurations, Guided-CoT improves performance for all models, though the gain is weaker for Mistral 8B with greedy decoding.By contrast, adding contemporary examples or using ZS-CoT does not improve performance for some models.Source: §4.5 · p. 6

5 Quantifying Models’ Explanations

The paper measures how model explanations diverge from one another and how each model’s explanations differ by predicted class. Zero-shot responses are comparatively homogeneous across models, while CoT prompts reveal stylistic and within-model patterns.

5 Quantifying Models’ Explanations

  • Embedding complete responses may obscure contextual differences because responses can follow a designed format or include source-post content; the analysis is presented as a useful first view.The measured variation can capture rationale, classification decisions, and possible templating patterns.Source: §5.1 · p. 7
  • Pairwise normalized cosine distances form each model’s seven-dimensional Semantic Distance Vector, and SCMD summarizes its overall divergence from other models.The analysis embeds responses and projects the resulting vectors into two dimensions with PCA for clustering comparisons.Source: §5.1 · p. 7, §5 · p. 7

5.1 Cross-Model Difference Analysis

  • CoT-like prompting yields higher cross-model divergence than Zero-Shot, but no consistent model ranking emerges; Llama 70B_q has the lowest SCMD across prompts.Mistral 8B’s SCMD is 0.12.Source: §5.1 · p. 7
  • Zero-Shot responses are homogeneous across models, while Guided-CoT surfaces latent stylistic differences; semantic divergence can inform model selection when explanatory power matters.Guided-CoT amplifies cross-model divergence for all models except Llama 70B_q, while Mistral 24B appears closer in the PCA projection.Source: §5.1 · p. 7, Figure 5 · p. 7

5.2 Intra-Model Difference Analysis

  • Heatmaps show weaker cross-group separation under ZS-CoT for Llama models and Mistral 24B; under Guided-CoT, reasoning models remain visibly separated across groups.For Gemma models, the authors hypothesize topic-based response differences and say further qualitative assessment is needed.Source: §5.2 · p. 8, Figure 6 · p. 8
  • Under Zero-Shot, all models except Llama 70B_q show significant within-model differences by predicted class; with Guided-CoT, only three models do.The authors describe Guided-CoT as neutralizing response differences for Mistral 8B, Gemma 27B, and Llama 8B relative to ZS-CoT.Source: §5.2 · p. 9
  • Crossing distance distributions suggest non-monotonic patterns for Gemma 12B and Llama 8B under ZS-CoT and Mistral 24B under Guided-CoT, meriting focused scrutiny.Under ZS-β, all models reject the one-sided KS test, indicating D+ is neither larger nor smaller than D−.Source: Table 1 · p. 8, §5.2 · p. 9

6 Misclassification Analysis

Across eight models, common misclassifications clustered around tone and stereotypes, quoted or news-reporting content, and opinions. False positives and false negatives showed different category patterns, while contextual mismatches included misspellings and proper names mistaken for slurs.

6 Misclassification Analysis

  • 28% of sampled shared misclassifications involved Tone/Language/Stereotypes, 25% Quotes/News-Reporting, and 21% Opinion.The analysis sampled 260 posts misclassified by all eight models from 8,555 valid responses; Contextual Mismatch accounted for 8%, Sarcasm 6%, and Religious content 6%.Source: §6 · p. 9
  • Most false positives fell under Tone/Language/Stereotype, whereas most false negatives fell under Quotes/News-Reporting.The examples include criticism of Israeli actions flagged as antisemitic and a quoted news report classified as non-antisemitic.Source: §6 · p. 9
  • About 21% of errors involved user opinions, and contextual mismatches included typos such as “kikes” for “likes.”Proper nouns containing “Kike,” including Dodgers player Kiké Hernández’s name, were also misread as slurs.Source: §6 · p. 9
  • Models struggled with offensive language, tone, stereotypes, quoted statements, and opinions, complicating their use for automated moderation.The authors note that examining additional context was outside this work’s scope and cite prior findings that context can improve performance.Source: §6 · p. 9

7 Conclusion

The study evaluates eight LLMs’ antisemitism classification and explanations, introducing Guided-CoT to examine performance and use of in-context guidelines. It finds that prompting and model choice shape utility and reliability, while generated explanations can exhibit paradoxical behavior.

7 Conclusion

  • The study comprehensively evaluates eight LLMs’ classification performance and generated explanations for antisemitism detection.Experiments compare Guided-CoT with other prompting techniques and examine in-context guidelines across decoding strategies.Source: §7 · pp. 9–10
  • Guided-CoT is an engineered CoT-like prompt evaluated for its effects on performance, explanations, and handling of in-context guidelines.Source: §7 · p. 10
  • Quantitative analysis of generated responses reveals paradoxical behavior in some models and differences relevant to their utility and reliability.The authors suggest the findings can inform qualitative audits of misclassification and potential biases.Source: §7 · p. 10

8 Limitations

The study’s evidence is bounded by its dataset, model and policy selection, analysis settings, and limited comparator details. The authors also identify design choices in the explanation analysis and acknowledge that results from different analysis subsets are not directly comparable.

8 Limitations

  • The dataset contains only English Twitter posts, and its annotators could use images, links, replies, likes, and comments unavailable in the text alone.The authors note that some false-negative explanations requested context that the study authors also considered necessary, despite the ground-truth labels.Source: §8 · p. 10
  • Sampling and self-consistency use one hyperparameter set, and other configurations could shift reported numbers slightly.For six models, self-consistency was evaluated 30 times per instance across all data points; the authors describe those comparisons as robust under self-consistency.Source: §8 · p. 10
  • The explanation analysis reduces response embeddings to 15 dimensions, although different models may benefit from different dimensionalities.The authors selected dimensionality by maximizing trustworthiness through a grid search and backed observations with a significance test.Source: §8 · p. 10
  • Guided-CoT was evaluated on one Twitter dataset, eight models, two baseline prompts, and the IHRA guideline, leaving broader generalizability under-explored.The authors identify diverse datasets, platforms, languages, and moderation policies as settings not covered.Source: §8 · p. 10
  • Guided-CoT’s task-engineered thoughts might not generalize to other downstream tasks.The work primarily examines in-context instructions for antisemitism detection.Source: §8 · p. 10
  • The comparison with fine-tuned GPT-3.5 relies on Becker et al.’s reported numbers, and the authors lacked details about its training.They frame it as a comparison of prompting with a fine-tuned proprietary model, not a claim that their prompting is generally superior.Source: §8 · p. 10
  • Analyses use different data subsets, so the authors do not compare results across analysis setups; a shared subset would have covered about 62% of the data.Subsets were used because some models generated invalid responses in initial experiments.Source: §8 · p. 11

9 Ethical considerations

The study uses the IHRA working definition as a practical moderation reference without asserting its validity. It also cautions that real-world application requires care given automated moderation’s complexity and societal impact.

9 Ethical considerations

  • The authors use the IHRA working definition because of its adoption in policy contexts, not as a judgment about its validity.They note that the definition is used by the United States and numerous governments and international bodies.Source: §9 · p. 11
  • The analyses rely on previously annotated datasets, whose potential biases the authors say are comparable to those in other hate-speech and toxicity datasets.They state that their judgments do not extend beyond the explicit IHRA definition and examples.Source: §9 · p. 11
  • The authors urge cautious interpretation and application of the findings in real-world moderation because of its complexity and societal impact.Source: §9 · p. 11

A Take-aways

  • The takeaways subsection presents itself as consolidating experimentally observed differences.Source: A Take-aways · p. 15

A.1 Utility

Across the evaluated models, prompting affected utility, detection performance, and response similarity. Guided-CoT generally improved detection, while model and prompt differences remained visible in reliability analyses.

A.1 Utility

  • Utility problems varied: Gemma-family models gave the most ambiguous classifications, Llama 8B had the highest refusal rate, and QwQ 32B often repeated under greedy decoding.Guided-CoT nearly eliminated Llama 8B refusals and reduced Gemma-family ambiguous responses.Source: §A.1 · p. 15

A.2 Performance

  • With greedy decoding, Guided-CoT substantially improves detection across models versus Zero-Shot and Zero-Shot-CoT, with only marginal improvement for Mistral 8B.Llama 70B_q has the highest overall greedy performance; Gemma-family models perform worst across the evaluated prompts.Source: §A.2 · p. 16

A.3 Reliability

  • Zero-Shot responses were homogeneous across models, while CoT prompts revealed divergence; Guided-CoT amplified it, though Llama 70B_q and Mistral 24B remained close.Under Zero-Shot, Llama 70B_q alone lacked a significant semantic-distance difference between its antisemitic and non-antisemitic responses, consistently across prompts.Source: §A.3 · p. 16
  • Gemma 12B, Llama 8B, and Mistral 24B showed bimodal or crossing response distributions under Zero-Shot, pointing to post subsets needing qualitative review.The authors call for interpretability audits guided by these statistical patterns.Source: §A.3 · p. 16

A.4 Prompting Techniques

  • Adding examples or reasoning instructions to Zero-Shot can reduce performance for some models, while Guided-CoT generally gives the largest F1 gains across decoding strategies.Mistral 8B with greedy decoding is the exception; removing one thought may have improved its F1 by 0.04.Source: §A.4 · p. 16
  • Prompting methods change how closely model explanations align, but no consistent pattern emerges; Llama 70B_q is most cohesive across methods.Source: §A.4 · p. 16

B Models

  • The evaluated set spans Mistral 8B and 24B, Gemma 12B and 27B, Llama 8B and 70B, DeepSeek-R1-Llama 70B, and QwQ 32B.Source: B Models · p. 17

B.1 Model’s understanding of Antisemitism

  • Most models explicitly referenced the IHRA definition when asked to define antisemitism; Llama 8B, Mistral 24B, and DeepSeek-R1 did not.Source: §B.1 · p. 17

C.1 Length

Guided-CoT produces longer responses than other prompting approaches, and response-generation behavior varies by model. The evaluation also uses multiple inferences for self-consistency metrics.

C.1 Length

  • Gemma models generate the longest responses among non-reasoning models, while Mistral 8B fails to produce a reasoning chain with the Zero-Shot-CoT instruction as presented.Forcing Mistral 8B to reason required presenting “Let’s think step by step” independently, without surrounding text.Source: §C.1 · pp. 17–18
  • Approximately twice as many tokens as Zero-Shot-CoT, Guided-CoT generates the longest responses across all models.Reasoning models generate more tokens due to thinking tokens; QwQ 32B generates about 1.5 times more with Guided-CoT than with Zero-Shot-CoT.Source: §C.1 · p. 18

C.2 Generation

  • Gemma-family models often give indeterminate classifications, Llama 8B may refuse the classification task, and QwQ 32B can repeat until it exceeds the response limit and provides no answer.CoT-like prompting reduces Gemma-family invalid responses; the authors hypothesize it may encourage cautious answers when context is insufficient.Source: §C.2 · p. 18

D Deterministic & Non-deterministic Evaluation

E Guided-CoT Ablation Setup

Guided-CoT outperforms other prompting methods in the reported decoding comparisons, and the appendix specifies its ablation design. It also describes distributional tests and a codebook for reviewing model errors.

E Guided-CoT Ablation Setup

  • Guided-CoT outperforms fine-tuned GPT-3.5 with greedy or self-consistency decoding and consistently outperforms other prompts under self-consistency.Source: Table 4 · p. 19, Figure 10 · p. 20
  • Each ablation removes one of five thoughts while retaining the other four; the thoughts include post chunking, ambiguity checks, IHRA alignment, and stance assessment.Source: E Guided-CoT Ablation Setup · pp. 18–19

F Kolmogorov–Smirnov (KS) Significance Tests Setup

G Annotation Codebook

Loading 2509.18293v2…