PaperlayerView exact revision on arXiv

Source-linked AI summary

Nürnberg NLP @ GermEval Shared Task 2026: Harmful Content Detection in German Social Media through Error-Independent LLM Voters

Philipp Steigerwald, Eric Rudolph, Jens Albrecht

arXiv:2608.22246v1cs.CL

TL;DR

Rare harmful classes in German social media share surface language with dominant harmless classes, making macro-F1 detection difficult. The paper uses nine-voter ensembles spanning LLM, training-method and class-scope diversity, placing first on all four hidden-test subtasks.

  • Problem

    The task is to separate rare harmful classes that share surface language with a dominant majority, which macro-F1 weights equally despite severe imbalance.

  • Method

    The system selects nine fine-tuned voters per subtask across LLM, training-method and class-scope axes, combining their predictions by plain majority vote.

  • Results

    The ensemble places first on all four hidden-test subtasks, achieving macro-F1 89.56 (C2A).

  • Takeaways & Limitations

    The supported conclusion is that complementary-model selection, rather than one exact ensemble, is what generalises across the subtasks.

  • Takeaways & Limitations

    The approach is resource-hungry, requiring hundreds of trained voters for selection and up to three different LLMs at deployment.

Abstract

from arXiv · show

Harmful content in German social media does real-world damage, from calls to action to criminal defamation. The GermEval 2026 shared task scores its detection in four subtasks. The technical challenge is a severe class imbalance. The harmful classes are rare and share surface language with the dominant majority class, yet under macro-F1 they decide the score. The decisive lever is then not a stronger single model but error independence. This insight becomes a per-subtask nine-voter ensemble spanning three orthogonal axes: LLM, training method and class scope. Selected mainly on internal cross-validation, the system reaches macro-F1 of 89.56 (C2A), 71.63 (DBO), 54.84 (VIO) and 83.02 (DEF) on the hidden test set, placing first on all four subtasks.

1 Introduction

Harmful content in German social media is evaluated across four macro-F1 subtasks, where rare harmful classes are difficult because they share surface language with dominant classes. The presented system combines voting and minority-class augmentation around voters spanning different LLMs, training methods, and class scopes.

  • Motivation: The GermEval 2026 shared task scores calls to action, attacks on the democratic basic order, violence-related content, and defamation across four macro-F1 subtasks.The data consist of anonymised German tweets from a right-wing extremist network.
  • Challenge: 87–97% dominant-class prevalence leaves rare harmful classes decisive under macro-F1, while shared surface language makes them difficult to separate.DBO ambiguity involves sharp government criticism versus illegal agitation, whereas VIO classes differ mainly in pragmatic function.
  • Approach: The system builds on voting and minority-class augmentation, relying on voter error independence so majority voting can correct individual voter mistakes.Its hypothesis is that voters using different LLMs and training methods rarely fail on the same tweet.
  • System design: Five LLMs form the pool: four have 7–14 billion parameters and one has 128 billion, with an optimized prompt providing the untrained baseline.Fine-tuned voters use 4-bit QLoRA in generative SFT or discriminative ClsHead modes.
  • System design: Every voter trains on one of three class scopes, including a generalist covering all subtasks.The passage identifies the generalist scope as G.

2 Related Work

Related work spans the German task family, ensemble and fine-tuning techniques, and the prior voter ensemble extended by this paper. It also connects harmful-content detection resources with error-independence theory and established ensemble approaches.

  • Related work is organized around the German task family, ensemble and fine-tuning techniques, and the prior voter ensemble extended here.
  • German harmful-content detection builds on GermEval 2018, HASOC, DeTox, criminally relevant hate-speech resources, and the legal scheme underlying DEF.
  • Error-independence theory motivates majority voting among diverse voters with uncorrelated mistakes, extending established ensembles for hate speech and aggression detection.
  • The voter-pool design extends a multi-axis ensemble combining class granularity, training method, and LLM, while adding a specialist trained only on non-majority examples.

3 Data

The shared task provides anonymised German tweets from a right-wing extremist network, labelled for four independent subtasks. To address severe class imbalance, the authors augment minority classes with synthetic GPT-5.4 tweets under a capped per-class budget, while leaving C2A unaugmented.

  • Dataset: The dataset comprises anonymised German tweets from a right-wing extremist network, labelled for four independent subtasks.C2A detects calls to specific action, while DBO evaluates stance toward the free democratic basic order in four classes.
  • Augmentation: 2,634 synthetic tweets were generated with GPT-5.4 for minority classes only, using the full taxonomy and five real examples per class.The augmentation procedure is described in Table 1 and Appendix E.
  • Augmentation: The augmentation budget is augc = min(target − nc, 3 nc) with target = 1000, lifting rare classes toward 1,000 examples unless the three-times cap binds.The cap keeps synthetic tweets at most 75% of a class.
  • Augmentation: C2A receives no synthetic tweets because its positive class already contains 1,434 examples, exceeding the target.Some generations fail sanity checks for malformed output or duplicate text, so realised counts fall slightly short of the budget and are not topped up.

4 System

The system uses a nine-voter majority ensemble per subtask, built from orthogonal choices of LLM, adaptation method, and class scope. Five-fold cross-validation selects branches and voters, while error-diverse compositions are deployed without using the hidden test set.

  • Architecture: Nine voters per subtask cast a plain majority vote, organized into three branches that vary across LLM, training method, and class scope.Each branch contributes three votes, selected from five fold-trained voters.
  • Adaptation methods: Three adaptation methods comprise prompt engineering as the baseline, generative SFT, and discriminative ClsHead.Both trained modes use 4-bit NF4 QLoRA; ClsHead additionally uses focal loss and inverse-frequency class weights to address imbalance.
  • Class scopes: The scope axis combines generalists, specialists, and positive-class specialists, with PCS available only for DBO and VIO.Specialists mainly contribute a different error profile, while PCS removes the majority class to emphasize distinctions among positive classes.
  • Cross-validation: Five-fold cross-validation trains each branch on four folds and scores it on the held-out fifth, using the mean of the three best fold macro-F1 scores for selection.Out-of-fold predictions provide leakage-free internal ensemble estimates, while all five fold models are retained and the deployed branch uses its top three.
  • Voter selection: Branch selection favors high F1 top3_cv, weakly correlated per-fold errors, and strong whole-composition F1 ens_cv, while discarding candidates with implausible majority-class prevalence.The hidden test set plays no part in selection; binary tasks use two full-label scopes, whereas DBO and VIO can add PCS.

5 Results

The nine-voter per-subtask ensemble, selected mainly by cross-validation, ranked first on all four subtasks, achieving hidden-test macro-F1 scores from 54.84 to 89.56. Fine-tuning outperformed prompting across every model and subtask, while winning compositions varied with each task’s class boundaries and transferability.

  • Overall results: 89.56, 71.63, 54.84 and 83.02 were the ensemble’s hidden-test F1 test scores for C2A, DBO, VIO and DEF, respectively.Each per-subtask winner used nine voters selected as a plain majority on cross-validation and placed first on its subtask.
  • DBO: 71.63 was DBO’s best score, produced by min-G-cls + mis128-S-sft + mis128-PCS-sft, the only winning composition retaining a positive-class specialist.The specialist removes the majority class; its internal score of 71.66 transferred closely to the hidden test, whereas an 8B version reached only 64.37.
  • VIO: 54.84 was VIO’s best score, reached by llamB-G-cls + min-G-cls + min-S-cls, despite VIO’s internal estimate misleading model selection.Positive-class-specialist variants scored higher internally but failed to carry over to the test set.
  • DEF: 83.02 was DEF’s best score, achieved by two mirrored generalist-with-two-specialists compositions that swapped the roles of llamB and phi4.Internal gains carried over most faithfully for DEF, whose hidden test contained 577 tweets.
  • Prompting baseline: 7, 36, 23 and 13 points were the 128B model’s F1 top3 cv gains from fine-tuning on C2A, DBO, VIO and DEF, respectively.Fine-tuning beat prompting for every model and subtask, with an average gain of 20 points; the largest gaps occurred on imbalanced multi-class tasks.

6 Analysis

The analysis finds that internal cross-validation ranked submissions closely enough to predict the hidden test, while voter disagreement varied by subtask and composition. The third branch affected only contested cases, especially DBO, and its removal reduced cross-validation F1 by 2.3–8.3 points.

  • Voter disagreement: Krippendorff’s α over nine votes ranged from .825 on saturated C2A to the lowest agreement on DBO, whose positive-class specialist created strong branch disagreement.Within branches, folds agreed closely; DBO’s two branch pairs involving the positive-class specialist agreed barely above zero.
  • Third-branch decisions: The six core voters were unanimous on roughly nine of ten test tweets, leaving the third branch to act only in 3.5%–7.1% contested cases.Once at least five of six core voters agreed, the decision was locked against the three third-branch votes, with no flips observed.
  • Third-branch decisions: 71% of DBO contested decisions flipped, versus 23%–36% on C2A, VIO and DEF, with DBO flips mostly moving majority-class decisions to criticism.The third branch therefore crossed the minority boundary that determines macro-F1, while binary-subtask flips mostly changed false to true.
  • Third-branch decisions: Dropping the third branch reduced cross-validation F1 ens by 2.3 points on C2A, 8.3 on DBO, 5.7 on VIO and 3.6 on DEF.These figures overestimate the distinct branch contribution partly because the ablation also uses fewer voters.

7 Conclusion

The system addresses severe class imbalance by combining nine fine-tuned voters selected across three diversity axes, achieving first place on all four hidden-test subtasks. Its conclusion emphasizes voter error independence over a single stronger model, while identifying targeted disagreement search and efficiency improvements as future directions.

  • Conclusion: 89.56 macro-F1 (C2A), 71.63 (DBO), 54.84 (VIO) and 83.02 (DEF) place the system first on all four hidden-test subtasks.Each tweet receives the plain-majority vote of nine fine-tuned voters selected mainly through internal cross-validation.
  • Conclusion: Voter error independence, rather than a single stronger model, is the decisive lever for separating rare harmful classes from the dominant harmless majority.Fine-tuning beats optimized prompting, and the strongest pool model does not separate rare harmful classes in its raw representation.
  • Conclusion: Future work could search for branches with strong internal signals and maximal disagreement, while reducing costs through smaller models or fewer voters per branch.The proposed directions target further F1 gains and lower training and inference costs.

Limitations

The study’s conclusions are limited by single hidden-test observations, unmeasured seed variance and significance, uncertain synthetic-data value, and substantial computational demands. Generalization therefore supports complementary model selection rather than any exact ensemble.

  • Evaluation: Each composition receives only one hidden-test observation, so the complementary-model-selection principle generalizes, not the exact ensemble.Every fold model is trained once, leaving seed variance unmeasured and composition differences unsupported by significance tests.
  • Evaluation: Internal-to-test transfer is reliable only for full-label ensembles, while a PCS ensemble’s class balance is checked against the prior.The passage does not report broader transfer reliability for PCS ensembles.
  • Synthetic augmentation: Synthetic augmentation increases minority-class counts, but neither generated-tweet diversity nor value beyond the larger sample size is measured.The synthetic data may add little and may carry generator-specific artefacts.
  • Resource demands: The approach is resource-hungry: each candidate branch fine-tunes five fold models for cross-validation, placing hundreds of trained voters behind selection.At deployment, up to three different LLMs cast the nine voters.

Ethics Statement

The study uses intentionally harmful German tweets from a right-wing-extremist network under a shared-task licence, with anonymisation preserved and no de-anonymisation attempts. It acknowledges that harmful-content classifiers are dual-use and frames them as assistive tools for moderation and mass-communication analysis.

  • Data use and dual-use risks: The dataset comprises German tweets from a network classified as right-wing extremist and intentionally contains harmful content.The data are used only for detection, moderation and research under the shared-task licence.
  • Data use and dual-use risks: Anonymisation tokens are retained, with no attempt at de-anonymisation.The stated safeguards apply to the use of the shared-task data.
  • Data use and dual-use risks: The paper identifies harmful-content classifiers as dual-use and assigns them an assistive role in moderation and mass-communication analysis.This framing limits their stated role to supporting human-oriented detection and analysis.

A Hyperparameter Details

Models between 7 and 14 billion parameters share the training hyperparameters listed in Table 4, while the 128B model uses cost-conscious LoRA settings. Training runs for three epochs under both methods, except LLaM-I’s five-epoch schedule, with the best validation macro-F1 epoch deployed.

  • 128B deviation: The 128B model deviates for cost control, using LoRA rank 16 with α=32 for SFT.
  • Training duration: Three epochs are used under both methods, while LLaM-I trains for five because performance peaks by epoch three.
  • Model selection: The epoch with the best validation macro-F1 is deployed in every case.
  • Shared settings: Table 4 specifies shared training hyperparameters for all 7–14B models, using 4-bit NF4 QLoRA without truncation.

B Full CV5 Results

The full five-fold cross-validation results report per-fold F1_cv and top-3-fold mean F1_top3 for every model and method across all four subtasks. Prompt engineering underperforms trained methods on every subtask, with the widest model-averaged gaps on DBO and VIO.

  • Full CV5 Results: Tables 5–8 report per-fold F1_cv and top-3-fold mean F1_top3 for every model and method on each subtask.The full-label scope includes the best fold coverage in parentheses.
  • Full CV5 Results: Prompt engineering trails trained methods on every subtask.This comparison is made across the models.
  • Full CV5 Results: The model-averaged gap between prompt engineering and trained methods is widest on DBO and VIO.The passage identifies DBO and VIO as the subtasks with the largest gaps.

C Embedding Geometry · D Prompts · E Augmentation Prompt

The section motivates specialist modeling through overlapping positive-class embeddings, specifies standardized bilingual SFT prompts, and documents a class-conditioned GPT-5.4 augmentation procedure for realistic German training tweets.

  • C Embedding Geometry: Untrained 128B last-token representations place rare positive classes close together without a clean boundary, motivating a positive-class specialist.The t-SNE samples every positive instance and 1,200 majority-class instances per subtask.
  • D Prompts: The SFT voters share a template containing the subtask system prompt, tweet, class definitions under ## Kategorien, and the assistant’s label.Each subtask presents the German prompt first and an English translation below a dashed rule.
  • D Prompts: C2A prompt evaluation is summarized in Table 5 as per-fold CV5 macro-F1 (%) with full-label scope shown in parentheses.The table highlights the best three candidates in bold.
  • D Prompts: The task prompts define labels with subtask-specific distinctions, including calls to action, attacks on the democratic order, violence-related statements, and criminal insult or defamation.The definitions instruct models to return labels only and distinguish, for example, concrete requests to act from general opinions.
  • D Prompts: DBO prompt evaluation is summarized in Table 6 as per-fold CV5 macro-F1 (%) with full-label scope shown in parentheses.The table highlights the best three candidates in bold.
  • D Prompts: VIO and DEF prompt evaluations are summarized in Tables 7 and 8 using per-fold CV5 macro-F1 (%) and parenthesized full-label scope.Both tables identify the best three candidates in bold.
  • E Augmentation Prompt: GPT-5.4 generates synthetic tweets in batches by conditioning on the subtask, target label and definition, batch size, and five randomly sampled real tweets.The prompt requires realistic, varied German posts, occasional anonymization tokens, a maximum of 280 characters, and exactly {n} valid JSON tweets.
Loading 2608.22246v1…