PaperlayerView exact revision on arXiv

Source-linked AI summary

ProBel: Propaganda Detection with Techniques, Spans, and Explanations

Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Elisa Sartori, Giovanni Da San Martino, Firoj Alam

arXiv:2608.22388v1cs.CLcs.AIcs.LG

TL;DR

It is unclear how supervision transfers across propaganda-detection tasks and languages. ProBel aligns multiple annotation levels in Arabic and English and evaluates task-specific and bilingual multi-task training, finding that bilingual multi-task training performs best overall and most stably across tasks and languages.

  • Problem

    It remains unclear how supervision transfers across sentence-level, technique-classification, and span-identification tasks in Arabic and English.

  • Method

    ProBel aligns binary labels, 23 technique labels, technique-labeled spans, and explanations across Arabic and English for joint evaluation and training.

  • Results

    Bilingual multi-task training performs best overall and most stably across tasks and languages, while span-only training can weaken sentence-level prediction.

  • Takeaways & Limitations

    A single bilingual multi-task model provides the strongest and most balanced performance while simplifying deployment across tasks and languages.

  • Takeaways & Limitations

    The study is limited to Arabic and English propaganda in news sentences and social media posts using a shared 23-technique taxonomy.

Abstract

from arXiv · show

Propaganda detection includes several related prediction levels, ranging from sentence-level decisions to technique classification and span identification. However, it remains unclear how supervision at these levels interacts when learned jointly across Arabic and English. We present ProBel, an Arabic and English resource that aligns binary labels, multi-label annotations over 23 propaganda techniques grouped into six coarse categories, technique-labeled spans, and reference explanations for the same news sentences. It includes a substantially larger English collection and supports matched binary, coarse-grained, multi-label, and span-level tasks in both languages. We evaluate zero-shot prompting, task-specific fine-tuning, and joint training under a shared setup. A single bilingual multi-task model achieves the best overall performance and remains competitive across tasks and languages. Cross-task analysis shows that transfer depends on the supervision level. Joint classification training preserves binary performance, whereas span-only training can weaken sentence-level prediction. Joint bilingual training yields the most stable results, while monolingual fine-tuning can reduce transfer to the other language. We will release the data, code, and evaluation scripts.

1 Introduction

ProBel addresses the unclear transfer among propaganda-detection tasks across Arabic and English by aligning multiple supervision levels in a unified resource. Its controlled experiments find that bilingual multi-task learning performs best overall, with transfer effects depending on supervision level.

  • Motivation and problem: Propaganda detection spans sentence-level decisions, technique classification, span identification, and explanations, but supervision transfer across these tasks and languages remains unclear.These tasks provide complementary views of the same content, linking decisions to techniques and textual evidence.
  • Resource and contributions: ProBel aligns binary labels, multi-label annotations for 23 propaganda techniques, technique-labeled spans, and reference explanations for Arabic and English content.The resource substantially expands the English collection while preserving the same tasks and taxonomy across both languages.
  • Resource and contributions: The aligned design enables controlled comparisons of task-specific, multi-task, monolingual, and bilingual training to measure cross-task transfer and negative transfer.The study evaluates whether supervision at different annotation levels reinforces or harms performance during joint training.
  • Main findings: A bilingual multi-task model performs best overall and remains competitive with specialized models across tasks and languages.Joint Arabic–English training yields the most stable results and avoids degradation observed with monolingual fine-tuning.
  • Main findings: Joint classification preserves binary performance, whereas span-only training can weaken sentence-level prediction.The findings show that transfer varies according to the supervision level used in joint training.

2 Related Work

Prior work established propaganda detection across binary, technique, and span-level tasks, expanding across languages and modalities while adding rationales and explanations. ProBel addresses fragmented task coverage and annotation designs with aligned Arabic-English supervision for controlled transfer analysis.

  • Task and explanation scope: Propaganda research spans sentence-level classification, technique identification, and span detection, with extensions to multilingual and multimodal settings.Recent studies also investigate extractive rationales and natural language explanations to improve decision inspectability.
  • English propaganda benchmarks: 18 propaganda techniques were introduced for sentence-level binary detection and fragment-level technique identification, followed by SemEval-2020 evaluation with 14 consolidated techniques.The earlier work also examined article-level propaganda identification and organization.
  • Multilingual and multimodal expansion: Later shared tasks expanded coverage to English memes, 23-technique multilingual news, multilingual memes with 22 techniques, and Arabic news paragraphs annotated at paragraph and span levels.These developments broadened language, domain, and modality coverage.
  • Research gap: Existing resources provide complementary binary, technique, and span annotations, but differing task coverage, languages, annotation designs, and evaluation settings hinder controlled supervision-transfer analysis.The limitation concerns transfer both across tasks and between Arabic and English.
  • Rationales and explanations: Propaganda spans identify text expressing a technique, while natural language rationales justify assigned labels; related work has explored generating these explanations.Span-level rationales support interpretability in fact-checking, hate speech, and deception detection.
  • Positioning ProBel: PropXplain provides Arabic and English instances with binary labels and reference explanations, whereas ProBel aligns binary labels, 23 fine-grained techniques, technique-labeled spans, and natural language rationales.ProBel also expands the English collection and supports controlled analyses of cross-task supervision, multilingual transfer, and joint Arabic-English training.

3 PROBEL Resource

ProBel is a bilingual Arabic–English resource for explainable propaganda detection that aligns binary labels, 23-technique annotations, labeled spans, and reference explanations. It preserves Arabic data while substantially expanding English coverage and supports classification and span identification in both languages.

  • Resource structure: Each instance combines a binary propaganda label, 23 fine-grained technique labels, technique-labeled spans, and a reference natural-language explanation.The same representation is used for Arabic and English, with explanations tailored to propagandistic or non-propagandistic content.
  • Bilingual coverage: ProBel retains the Arabic splits, substantially expands the English collection, and supports fine-grained technique classification and span identification in both languages.The Arabic collection includes news and social-media content, while the English collection is newly expanded.
  • Annotation: At least two annotators independently labeled each added English article, with expert review resolving disagreements over applicable techniques and their expressed spans.The annotation process retains source fine-grained annotations and applies the 23-technique inventory to added English data.
  • Explanation quality: Human evaluation of generated explanations achieved agreement of 0.89–0.92 for Arabic and 0.94–0.95 for English, but the added English set was not reevaluated.Three annotators assessed faithfulness, clarity, plausibility, and informativeness on a five-point scale.
  • Dataset distribution: ∼48K instances comprise ∼21K Arabic and ∼27K English examples, with propagandistic content representing 63.7% of Arabic and 28.0% of English data.The resource’s technique distribution is skewed: its three most frequent techniques cover 94.8% of Arabic and 67.0% of English propagandistic instances.

4 Tasks and Experimental Setup

The paper evaluates four core prediction tasks spanning binary, coarse- and fine-grained classification, span detection, and explanation generation, using complementary baselines and model-training setups. Experiments also measure cross-task and cross-language transfer, alongside alternative distillation strategies.

  • Tasks and Metrics: The evaluation covers binary detection, fine-grained 23-label classification, six-category coarse classification, technique-labeled span detection, and explanation generation.Binary classification uses macro-F1; fine-grained and coarse-grained classification use micro-F1, while span detection uses overlap-adjusted micro-F1 and explanations use BERTScore-F1.
  • Tasks and Metrics: Span detection compares inline Span-tag outputs with Span-occ JSON triples that include technique, span, and occurrence information.The occurrence index distinguishes repeated surface forms, and Figure 3 notes that Span-occ supports repeated or overlapping spans.
  • Baselines and Models: The study compares majority-label, random, whole-text, zero-shot prompting, task-specific fine-tuning, bilingual multi-task training, and discriminative classifier baselines.Generative models use Qwen2.5-7B-Instruct with LoRA, while discriminative baselines include Llama-3.1-8B, Qwen2.5-7B, AraBERT-v2, and BERT-base.
  • Transfer Experiments: Cross-task transfer compares binary-only, classification-only, span-only, and five-formulation training setups, evaluating every formulation including tasks unseen during training.Cross-language transfer compares Arabic-only, English-only, and joint Arabic-English training.
  • Distillation: The experiments test whether aligned annotations can be replaced with off-policy GPT-5 trace distillation or on-policy self-distillation from a frozen teacher.The study also diagnoses the published on-policy objective’s stalling and proposes a saturating clip to repair it.

5 Experimental Results

Experiments show that fine-tuned, jointly supervised bilingual models outperform prompting and single-task alternatives across tasks and languages. Performance declines with output granularity, transfer is asymmetric, and distillation complements but does not replace direct supervision.

  • Prompting and fine-tuning: Zero-shot performance varies substantially across models and languages, while fine-tuning yields clear gains over prompted systems in both languages.The strongest open model outperforms GPT-5 in each language, while several models approach or fall below the random baseline on Arabic.
  • Task difficulty and joint models: Performance declines as outputs become finer-grained, with binary classification strongest, coarse and technique classification intermediate, and span prediction generally weakest.English task-specific span-occ is the only stated exception, and prompting does not close the span-detection gap.
  • Task difficulty and joint models: A single bilingual multi-task model performs best overall, with MT-SFT matching or outperforming task-specific models on classification and remaining within 1.4 points on span prediction.The model handles both languages and all tasks; English is more challenging because its test set contains fewer propagandistic instances.
  • Supervision transfer: Joint supervision preserves binary performance while improving coarse- and technique-level prediction, whereas span-only training weakens binary detection and classification-only training misses accurate span boundaries.Training on all five tasks combines these benefits, indicating complementary sentence- and token-level supervision.
  • Cross-lingual transfer: English data expansion improves the new benchmark, but only multi-task training improves both it and the original test set; cross-lingual transfer is highly asymmetric.Arabic fine-tuning transfers effectively to English, while English-only fine-tuning falls below the zero-shot level on Arabic.
  • Distillation: Off-policy distillation substantially improves zero-shot performance, whereas OPSD provides limited gains and neither matches MT-SFT, so distillation complements rather than replaces direct supervision.Hard clipping improves OPSD by suppressing large decision-token contributions without eliminating their gradients.

6 Comparison with Published Results

The section compares the bilingual multi-task model with published PropXplain and GPT-4 systems on label, explanation, and technique-labeled span tasks. MT-SFT achieves the best macro-F1 in both languages while also producing techniques and spans from the same checkpoint, with span comparisons subject to split and annotation differences.

  • PropXplain comparison: MT-SFT obtains the best macro-F1 in both languages against PropXplain on identical test splits.The comparison concerns systems that predict labels and generate explanations.
  • PropXplain comparison: The same MT-SFT checkpoint additionally produces techniques and spans.This extends the comparison beyond label and explanation prediction.
  • Span comparison: Span detection is compared with published GPT-4 results using overlap-adjusted micro-F1, with Arabic evaluated on the identical test split.The comparison is reported in Table 8.
  • Span comparison: The English GPT-4 span score comes from the SemEval-2023 development set rather than this work’s test set and is not directly comparable.No published result exists for English on the new test set.

7 Conclusions and Future Work

ProBel jointly addresses propaganda presence, techniques, locations, and explanations through aligned Arabic–English annotations. Bilingual multi-task training provides the strongest, most balanced results, while future work targets broader coverage, better generation, and improved transfer.

  • Contributions: ProBel frames propaganda analysis around presence, techniques, locations, and explanations, introducing aligned binary, multi-label, span-level, and explanation annotations for Arabic and English.The resource extends PropXplain and supports joint study of these related questions.
  • Findings: Bilingual multi-task training delivers the strongest and most balanced performance across tasks and languages while enabling deployment through a single model.The conclusion identifies this setup as the overall best-performing and most practical configuration.
  • Future Work: Future work will extend the resource to additional languages and domains, improve span and explanation generation, and investigate transfer and distillation for fine-grained analysis.These directions target broader applicability and more effective modeling of detailed propaganda phenomena.

Limitations

The study is limited to Arabic and English propaganda in news sentences and social media posts, using a shared 23-technique taxonomy. Its annotation and evaluation choices support consistency and controlled comparisons, while future work can broaden the scope.

  • The study focuses on Arabic and English propaganda in news sentences and social media posts.
  • A shared taxonomy of 23 techniques enables controlled comparisons across tasks and languages.
  • Because span annotation involves ambiguous boundaries, the study uses multiple annotators and expert review to improve consistency.Reference explanations provide supervision and evaluation targets, and experiments cover representative model families and training settings.
  • Future work can extend the analysis to additional architectures, domains, languages, and explanation-evaluation methods.

Ethics and Broader Impact … E Span Representations: Details and Results

ProBel combines multilingual, hierarchical propaganda annotations with reproducible task coverage and carefully controlled span representations. The paper emphasizes human oversight, cross-language reporting, annotation complexity, and complementary span-format tradeoffs.

  • Ethics and Broader Impact: Propaganda-detection models should support trained human reviewers because incorrect labels could suppress legitimate criticism, satire, or political expression.The dataset may contain political, sensitive, or offensive content requiring careful handling and cultural and political context.
  • Ethics and Broader Impact: Releasing data, code, and evaluation scripts improves transparency and reproducibility, while separate reporting across languages and tasks helps expose uneven performance.
  • A Comparison with Prior Work: PROBEL combines Arabic and English data with binary labels, 23 techniques, technique-annotated spans, and reference explanations, covering more task types than typical prior resources.
  • B PROBEL Additional Details: Arabic instances are longer and contain more techniques and spans than English instances, with substantially higher multi-technique and multi-span proportions.These differences indicate greater annotation complexity in Arabic.
  • C Label Inventory and Task Derivation: Both languages use the same 23-technique inventory grouped into six coarse categories, following the SemEval-2023 taxonomy adopted for Arabic by ArPro.
  • C Label Inventory and Task Derivation: Span annotations are primary labels: a sentence is propagandistic when it contains at least one annotated span, with fine- and coarse-grained labels derived consistently from span techniques.The hierarchy was verified for every sentence in both languages.
  • D Experimental Setup: All fine-tuned models reuse one fixed recipe, while generative systems share Qwen2.5-7B-Instruct, task formats, and a single parser to isolate supervision effects.
  • E Span Representations: Details and Results: Span-tag preserves context and is generally more robust, whereas span-occ supports overlapping spans and repeated strings; their complementary failures make representation choice consequential.Both formats use identical gold spans, parsing routines, and scoring, so representation is the only variable; scoring uses overlap-adjusted micro-F1 after duplicate removal.

F In-Context Learning Ablations · G Full Transfer Results

The in-context ablations justify a single retriever–pool configuration and show that same-language demonstrations generally outperform cross-lingual alternatives. Full transfer results report absolute bilingual and cross-lingual values, distinguishing monolingual fine-tuning from transfer columns.

  • F In-Context Learning Ablations: The in-context configuration is selected in two stages so the main comparison uses one justified setting rather than a tuned-per-cell best case.
  • F In-Context Learning Ablations: Dense BGE-M3 retrieval with a same-language pool ranks first among 18 retriever–pool combinations, scoring 0.497 at k=3.The ranking averages the primary metric across Arabic and English binary and technique classification.
  • F In-Context Learning Ablations: Cross-lingual configurations remain within 0.004 of the random same-language baseline, whose score is 0.426, while the best reaches only 0.430.Sparse BM25 retrieval trails all dense retrievers.
  • F In-Context Learning Ablations: Retrieved same-language demonstrations improve monotonically with k on Arabic for every task except binary detection, where k=3 performs best.The comparison evaluates k ∈{1, 3, 5} on the full test set.
  • F In-Context Learning Ablations: On English, same-language demonstrations do not improve binary detection at any tested k, while cross-lingual demonstrations track the random baseline throughout.
  • F In-Context Learning Ablations: The ablation compares the winner, its cross-lingual counterpart, and a random control across demonstration counts k ∈{1, 3, 5}.
  • G Full Transfer Results: Table 15 reports the absolute values behind Figure 5 for both test languages, including every cross-lingual cell.Rows named by one language are fine-tuned only on that language; columns under the other language measure cross-lingual transfer.

H Complete Baseline Results · I Explanation Quality · J Long-Tailed Techniques

The complete baseline results expose protocol and label-distribution effects, while explanation scores show modest gains over language-specific floors. Technique performance is strongly long-tailed, leaving rare-technique detection as the clearest remaining challenge.

  • H Complete Baseline Results: Qwen2.5-7B’s Arabic binary score rises from 0.439 to 0.642 under the multi-task prompt, within 0.004 of GPT-5 under the identical prompt.Proprietary models answer all five outputs together, whereas open models answer one prompt per task.
  • H Complete Baseline Results: The majority baseline exceeds every training-free system on Arabic coarse and technique classification, scoring 0.558 and 0.466 respectively.One category and one technique dominate the Arabic label distribution, so these scores require careful interpretation.
  • I Explanation Quality: Random same-language reference pairing already reaches 0.58 BERTScore in Arabic and 0.63 in English, making raw cross-language comparison invalid.Arabic uses AraBERT-v2 as the BERTScore backbone, while English uses BERT-base.
  • I Explanation Quality: Fine-tuned explanation models exceed their language-specific floors by +0.09 in Arabic and +0.09 in English, whereas English zero-shot exceeds chance pairing by only +0.03.Reference-based measures do not establish whether explanations reflect the model’s own decision process.
  • I Explanation Quality: Reference-based explanation measures cannot assess whether an explanation reflects the model’s own decision process, which remains future work.The limitation applies despite the reported language-specific baseline comparisons.
  • J Long-Tailed Techniques: MT-SFT handles Arabic’s two most frequent techniques well, degrades on mid-frequency techniques, and predicts almost none of the rare tail.Technique performance follows label frequency.
  • J Long-Tailed Techniques: 11 of 14 Arabic techniques with fewer than 20 test occurrences score zero, as do 10 of 11 such techniques in English.Improving rare-technique performance is identified as the clearest remaining benchmark target.

K Distillation · L Significance Tests

Distillation results show that off-policy imitation underperforms direct references, while hard-clipped OPSD removes decision-token gradients and saturating clipping repairs performance on PROBEL. Paired tests confirm fine-tuning gains are significant, but published-setting OPSD remains statistically indistinguishable from initialization.

  • K Distillation: The best bilingual off-policy model reaches 0.727 Arabic and 0.668 English binary macro-F1, below no-reasoning MT-SFT at 0.763 and 0.735.Across the four jointly supported tasks, it scores 0.441 versus 0.505 for MT-SFT.
  • K Distillation: Hard clipping removes gradients from above-threshold decision tokens, whose divergence reaches 5.5 nats at the 90th percentile, about 90× the published threshold.At τ=0.06, only 1.8% of total divergence mass and 0.8% of label-token mass survive.
  • K Distillation: Published-setting OPSD is effectively flat: the best Qwen2.5-7B configuration gains +0.032 AVG over zero-shot, while configurations retaining the published clip stay within +0.001.On MT-SFT, 99.1% of Arabic binary test predictions match the starting point, with p=1.0.
  • K Distillation: Saturating clipping repairs PROBEL consistently across three seeds, reaching 0.314±0.003 AVG versus 0.296±0.001 for the τ-matched hard clip.The repair requires both a gradient-preserving shape and an adequate threshold: it holds for τ ∈{0.1, 0.2, 0.5} but vanishes at τ=1.
  • K Distillation: The gold block raises teacher label accuracy from 42.0% to 93.7% for base Qwen2.5-7B and from 70.7% to 94.3% for MT-SFT, showing the privileged signal is useful.Label-only context matches or exceeds the full block at 98.3% and 92.7%, respectively.
  • K Distillation: Clipping choice depends on the domain: saturating clipping beats hard clipping on HateXplain, while hard clipping remains appropriate when above-threshold tails are mainly stylistic.On HateXplain, scores are 0.640 versus 0.628 at the published threshold and 0.687 versus 0.666 at τ=0.5; on GSM8K, both reach 0.940 at τ=0.5.
  • K Distillation: Replication on original benchmarks improves the base model from 34.2 to 41.9, matching the published gain from 37.1 to 43.4, while removing the clip reproduces degradation to 36.4.The benchmark protocol uses Avg@12 across AIME24, AIME25, and HMMT25 with the best checkpoint within 100 steps.
  • L Significance Tests: Every fine-tuning gain is significant at p<0.001; the open-versus-GPT-5 gap is significant at p<0.05 on Arabic and p<0.001 on English.Published-setting OPSD changes only 6+6 of 1,326 predictions, making the trained model statistically indistinguishable from initialization.
Loading 2608.22388v1…