PaperlayerView exact revision on arXiv

Source-linked AI summary

MUCnoHARM@GermEval Shared Task 2026: Retrieval-based In-Context Learning for Defamatory Offences, and Where It Falls Short

Kristin Gnadt, Maximilian Meidinger, Matthias Aßenmacher

arXiv:2609.09791v1cs.CL

TL;DR

The paper asks how retrieval-based in-context learning can detect defamatory offences under §§ 185–187 StGB amid limited evidence for criminally relevant hate speech. It evaluates legal conditioning, demonstration retrieval, model choice, and fine-tuning, finding that retrieval adds little beyond few-shot prompting, concrete legal knowledge helps, and model choice dominates prompting choices. These systems are better suited to triage than autonomous moderation because they over-predict criminal relevance while still missing many relevant posts.

  • Problem

    Evidence is limited on how zero-shot, static, and retrieval-based prompting compare for criminally relevant hate speech requiring precise legal thresholds.

  • Method

    The study compares legal-conditioning modes and static or dynamically retrieved few-shot demonstrations across models, with additional GPT-5.5 and QLoRA evaluations.

  • Results

    Model choice has greater impact on F1macro than any single prompting axis, while retrieval adds only marginal gains over random demonstrations and optimized static demonstrations can perform better.

  • Takeaways & Limitations

    Concrete legal knowledge and few-shot prompting help, but retrieval strategy contributes little, making model selection more consequential for this task.

  • Takeaways & Limitations

    Findings are confined to one language, dataset, annotation scheme, and distribution, limiting external validity under concept drift or more heterogeneous data.

Abstract

from arXiv · show

With hate speech being ubiquitous online, automatic detection is crucial, in particular when it comes to criminally relevant social media posts. We study a variety of retrieval-based in-context learning (RetICL) strategies for detecting defamatory offences under §§ 185-187 StGB (the subject of GermEval 2026 Subtask 4). Few-shot prompting beats zero-shot, but retrieval-based approaches offer only marginal gains over random demonstrations, and even fall behind an optimised static set of demonstrations. Providing concrete legal knowledge helps, yet model choice outweighs every other system choice. Models over-predict criminal relevance while still missing 26-57% of criminally relevant posts, suiting them for triage rather than autonomous moderation.

1 Introduction

The paper addresses automated detection of German criminally relevant hate speech, testing legal conditioning and retrieval-based few-shot prompting for defamatory offences under §§ 185–187 StGB.

  • Motivation: 34% of German internet users reported encountering hate speech in early 2025, while harmful-content volume exceeds human moderation capacity.LLMs are therefore considered for automated detection, particularly in socially consequential moderation settings.
  • Research challenge: Criminally relevant hate-speech detection requires sensitivity to country-specific legal thresholds and distinctions that current systems only partially meet.GermEval 2026 Subtask 4 classifies whether posts constitute defamatory offences under §§ 185–187 StGB.
  • Approach: The RetICL framework tests legal information supplied through conditioning alongside mechanisms for selecting few-shot demonstrations.Its two components are legal conditioning and demonstration embedding and retrieval.
  • Approach: The study compares different demonstration-selection strategies with zero-shot prompting for detecting defamatory offences.The experiments examine whether retrieval improves few-shot prompting in this legal classification task.

2 Background & Related Work

The background defines the legal task and motivates testing retrieval-based in-context learning and legal conditioning, whose relative effectiveness remains unclear for criminally relevant hate speech.

  • Defamatory offences: §§ 185–187 StGB cover insult, defamation, and intentional defamation, differing mainly by statement type and addressee while protecting the right to honour.The offences distinguish opinions from factual claims and consider whom the statement addresses.
  • Defamatory offences: The annotation scheme operationalises the legal assessment as six binary text-only decisions covering targets, disparagement, statement type, and freedom of expression.The scheme provides structured legal knowledge for prompting and supports independently checkable decisions.
  • Legal conditioning: Prompt content matters in specialised domains such as law because the amount and type of task-relevant legal knowledge supplied to an LLM are design choices.Meaning-preserving prompt changes can also substantially shift performance.
  • Legal conditioning: Prior § 130 StGB work found abstract legal conditioning superior to concrete information, while other work examined decomposition and logical scaffolds.These mixed findings motivate testing conditioning choices for defamatory-offence detection.
  • Retrieval-based ICL: In-context learning uses demonstrations without parameter updates, but performance is sensitive to their choice, number, ordering, and format.Retrieval-based ICL selects demonstrations per query from a labelled pool rather than applying one fixed set uniformly.
  • Retrieval-based ICL: Similarity-based retrieval commonly selects examples using BM25 or dense cosine similarity, although retrieval evidence is not uniformly positive across adjacent moderation tasks.The paper therefore compares zero-shot, static, and RetICL approaches for a legal-threshold task that remained insufficiently examined.

3 Materials and Methods

The study evaluates German tweets for §§ 185–187 StGB offences using multiple instruction-tuned models, legal-conditioning modes, and static or dynamically retrieved demonstrations.

  • Dataset: 3,263 labelled German tweets support cross-validated evaluation, while 577 unlabelled tweets form the competition test set.Approximately 13% of training samples are positive, and 797 tweets additionally receive step-level annotations for demonstrations.
  • Models: Four instruction-tuned models span different families, parameter scales, and openness: Gemma-4 26B, Gemma-4 E4B, Qwen3.5 9B, and EuroLLM 22B.All models are run with the Transformers library from the HuggingFace Hub.
  • Conditioning: The four conditioning modes range from section titles to descriptions, inline six-step reasoning, and separate inference calls for each legal decision.Explicit conditioning derives the final label from the sequence of independently classified step outcomes.
  • Demonstration selection: Demonstrations are either static fixed sets reused across instances or dynamically retrieved per instance, with zero-shot represented by k = 0.Class ratio and within-prompt ordering are varied in the ablation study.
  • Retrieval: Dynamic retrieval selects demonstrations from a training knowledge base using similarity, diversity, MMR, or random strategies.Similarity retrieval uses dense cosine similarity or sparse BM25 scores, while diversity uses clustered embeddings and MMR balances similarity against diversity.
  • Baseline: The TF-IDF baseline assigns each test post the label of its closest training neighbour, relying only on surface lexical overlap and no legal knowledge.Conceptually, it is equivalent to dynamic few-shot retrieval with k = 1.

4 Experiments

Experiments use stratified four-fold cross-validation to compare models, conditioning modes, retrieval strategies, demonstration settings, a proprietary model, and QLoRA fine-tuning.

  • Evaluation setup: Stratified four-fold cross-validation evaluates held-out folds using F1macro, True-class Precision, and Recall, while reporting abstentions separately.Abstaining instances are excluded from scoring when labels are unparseable or the model refuses.
  • Evaluation setup: Demonstration sizes k ∈ {4, 8, 16, 32} are explored across two prompting configurations and all models to control main-experiment computational cost.The selected size is optimised according to F1macro.
  • Main experiments: Main experiments compare four open-weight models, four conditioning modes, zero-shot prompting, and random, diversity, MMR, and similarity retrieval.Similarity retrieval is tested with dense, sparse, and fusion embeddings; demonstrations are class-balanced and randomly ordered.
  • Additional evaluations: GPT-5.5 is evaluated with Title zero-shot prompting and Explicit conditioning using dense similarity-based retrieval.This extends the model-size comparison beyond the 4B–26B open-weight range.
  • Additional evaluations: QLoRA fine-tuning evaluates whether an ICL strategy compares favourably with supervised fine-tuning on the smaller Gemma-4 E4B model.The comparison targets computationally efficient adaptation against the ICL baseline.
  • Ablations: Additional experiments vary class ratio and ordering, and compare dynamic retrieval with randomly selected optimised static demonstrations.The static comparison uses two eight-demonstration sets for Gemma-4 26B under Implicit and Title conditioning.
  • Test-set evaluation: Five model/configuration combinations are selected for the held-out competition test set, with the two best runs reported in the experiment sections.Further run details are provided in the appendix.

5 Results

Few-shot prompting and concrete legal conditioning generally improve F1macro, but model choice matters more than any individual prompting axis. Retrieval choices contribute little, while performance and sensitivity vary substantially across models and configurations.

  • Exploration: Increasing demonstrations from 4 to 8 yields the largest average gain, +0.029 F1macro; larger sizes help three models but slightly reduce Gemma-4 26B performance.The experiments therefore use k = 8 subsequently to limit computational cost.
  • Main Experiments: The best main-experiment configuration reaches F1macro 0.733 on the test set and 0.72 on the held-out competition set.It uses Gemma-4 26B with Implicit conditioning, dense embeddings, and similarity-based retrieval.
  • Main Experiments: Models over-predict the positive class while missing 26%–57% of criminally relevant posts, with positive-class Recall ranging from 0.43 to 0.74.The 1-NN baseline has stronger F1macro than some models but substantially lower Recall than nearly all models.
  • Main Experiments: Zero-shot prompting performs worst for every model, while more detailed conditioning generally improves performance and Explicit is best on average.The trend is not strictly monotonic: Description averages 0.567 versus 0.559 for Implicit.
  • Ablations: Embedding and retrieval choices have little F1macro impact, whereas conditioning and the zero-shot/few-shot distinction are more pronounced; model choice alone has impact 0.309.Gemma-4 26B is least configuration-sensitive, while EuroLLM 22B is most affected across axes.
  • Fine-Tuning: Fine-tuned Gemma-4 E4B achieves F1macro 0.741, slightly above the best main-experiment run at 0.733 and substantially above its best Implicit run at 0.582.On the held-out competition set it reaches 0.74, with Precision 0.702 and Recall 0.431.
  • Ablations: Ratio and ordering effects are highly model-dependent: ordering matters more with balanced demonstrations, while Qwen3.5 9B shows the strongest ratio–order interaction.Gemma-4 26B is barely affected, whereas EuroLLM 22B is most affected by both factors.
  • Ablations: Optimised static demonstrations outperform the best main-experiment configurations for Gemma-4 26B, reaching 0.723 with Title and 0.754 with Implicit conditioning.These scores exceed the corresponding best dynamic configurations by +0.028 and +0.021.

6 Discussion

Retrieval strategy contributes little compared with conditioning, demonstrations, and model choice. Concrete legal conditioning helps, but reliability concerns and fine-tuning results position systems differently for triage and deployment.

  • Dynamic retrieval strategies barely beat random retrieval, while an optimised static demonstration set outperforms dynamic approaches within the same conditioning mode.
  • Model choice has a mean F1macro impact of 0.309, exceeding that of any single prompting axis.
  • Concrete, decomposed legal knowledge performs best on average and for three of four models, although its effect is model-dependent and non-monotonic.
  • Systems over-predict criminal relevance yet miss 26%–57% of criminally relevant posts, supporting triage use rather than unsupervised moderation.
  • The naive TF-IDF baseline reaches F1macro = 0.644, outperforming most smaller-model runs and raising concerns about shortcut reliance under distribution shift.
  • QLoRA fine-tuning of Gemma-4 E4B reaches F1macro 0.741, exceeding the best dynamic prompting configuration at 0.733 and matching a carefully prompted larger model on held-out testing.

7 Conclusion

The study varies legal knowledge and demonstration retrieval for defamatory-offence detection. Few-shot prompting and concrete legal knowledge help, while retrieval adds little and model choice matters most.

  • The study tests prompting strategies for defamatory offences under §§ 185–187 StGB.
  • The experiments vary both the legal knowledge supplied to the model and the retrieval method for few-shot demonstrations.
  • Few-shot prompting clearly outperforms zero-shot prompting, while elaborate retrieval offers only a marginal advantage over randomly chosen demonstrations.
  • Optimised static demonstrations can outperform dynamic retrieval in the tested settings.
  • Model choice has more impact on task performance than any single prompting configuration axis, and most tested open-weights models do not consistently match a naive nearest-neighbour baseline in F1macro.
  • Models consistently outperform the baseline in Recall, which the authors identify as more relevant for triage.

Limitations

The study’s conclusions are bounded by homogeneous data, incomplete configuration coverage, unexplored reasoning-based legal prompting, and limited practical relevance for offences usually prosecuted on victim application.

  • The findings come from a single language, dataset, and annotation scheme drawn from one distribution, limiting external validity.The authors identify concept drift and more heterogeneous data as important directions for future evaluation.
  • The configuration space was not fully exhausted because the most favourable demonstration settings were not all combined.
  • Reasoning-based legal prompting remains unexplored because the experiments disabled model thinking modes.The proposed extension would explicitly model the statutory decision tree.
  • Automatic detection has more limited practical relevance for §§ 185–187 because these offences are usually prosecuted only on the victim’s application.

A Annotation Scheme

The annotation scheme operationalises criminal relevance under §§ 185–187 StGB as six binary text-based decisions, combining a three-step legal spine with rights balancing for value judgments. The paper uses this scheme for triage rather than definitive judicial assessment and resolves undecided cases as not punishable.

  • Offence categories: The scheme distinguishes insult, defamation, and intentional defamation according to the statement’s nature and addressee.§ 185 concerns contempt toward a person; §§ 186–187 concern disparaging factual claims, with § 187 requiring knowingly false assertions.
  • Decision spine: Six binary decisions assess a valid defamatory object, a disparaging statement, and whether the statement is factual or a value judgment.The first three decisions form the scheme’s sequential spine.
  • Rights balancing: Value judgments receive three additional binary checks balancing abusive insult, public interest, and abusive criticism against freedom of expression.These checks encode the protection of legitimate interests under § 193 StGB.
  • Annotation assumptions: Undecided steps are labelled not punishable under in dubio pro reo because the task permits only binary True/False outputs.The underlying scheme leaves free judicial balancing of individual circumstances unimplemented.
  • Scope: The paper frames the task as criminal-relevance classification for triage, leaving definitive punishability and prosecutability judgments to human experts.

B.3 Static Demonstrations

This section reports the static demonstration material and includes labelled examples of offensive, non-criminally relevant, and criminally relevant posts for evaluation and reproducibility.

  • The examples are included solely for scientific and illustrative purposes and do not represent the authors’ views.The paper also advises reader discretion because some examples are highly offensive.
  • The paper reports demonstration sets optimised through random search.
  • The examples contain posts labelled True or False for criminal relevance under the task’s annotation scheme.The examples include insults, offensive group references, and statements involving journalists, politicians, or other targets.

C Detailed Experiment Settings

The experiments vary inference settings, demonstration selection, embedding and retrieval methods, model configurations, and class-ratio ordering. Results show that retrieval choices matter little relative to conditioning and the zero-shot/few-shot distinction, while some ratio-ordering effects are consistent across models.

  • Inference settings: The experiments use batched True/False generation, disable thinking modes, and parse outputs directly into labels.Most models use max_tokens=10; fewer than 1% of answers cannot be parsed.
  • Main result: Embedding and retrieval modes have very little impact compared with conditioning mode and the difference between zero-shot and few-shot prompting.
  • Retrieval results: Similarity-based retrieval is best for every model’s best configuration, while dense embeddings perform best on average across models.Diversity retrieval performs slightly better on average only for Qwen3.5 9B.
  • Configuration limitation: The most favourable retrieval settings should be explored separately for each conditioning mode because dense similarity-based RetICL is strong on average but not uniformly optimal.
  • Abstentions: Median abstentions are 0–1 per run, below 1% of posts, with most Gemma-4 26B abstentions concentrated in Explicit zero-shot prompting.The model often requests more information at the defamatory-object step when no demonstrations resolve ambiguity.
  • Class ratio and ordering: Proportional class ratios outperform balanced demonstrations on mean F1macro, while true-first ordering outperforms random and true-last ordering across all four models.The best individual combinations are Qwen3.5 9B balanced/true-first at 0.704 and EuroLLM 22B proportional/random at 0.663.
  • Class ratio and ordering: Precision is best with proportional and true-first demonstrations, whereas recall often peaks with balanced and true-last sets.The authors therefore identify balanced, true-last settings as best when recall is prioritised.

D.5 Shared Task Results

Five runs were submitted to the DEF Subtask, with the fine-tuned Gemma-4 E4B achieving the best held-out F1macro score. The default balanced, randomly ordered demonstrations slightly outperformed the proportional, true-first variant for Gemma-4 26B.

  • The held-out DEF test set evaluated five submitted runs using official test labels.
  • 0.74 F1macro was achieved by the fine-tuned Gemma-4 E4B, narrowly ahead of prompted Gemma-4 26B at 0.72.These official held-out scores closely track the corresponding cross-validated results, 0.741 and 0.733.
  • 0.70 F1macro was obtained by the proportional, true-first Gemma-4 26B variant, below the default balanced, randomly ordered configuration.The comparison suggests that tuning demonstration class ratio and ordering did not transfer to held-out data.
  • Table 9 reports all 112 configurations alongside GPT and fine-tuned models, sorted by F1macro.The table distinguishes precision and recall for the criminal class and labels prompting, retrieval, and embedding variants.
Loading 2609.09791v1…