Source-linked AI summary
SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles
G. Da San Martino, A. Barrón-Cedeño, H. Wachsmuth, R. Petrov, P. Nakov
TL;DR
The task addresses fine-grained detection of propaganda in news by locating propaganda spans and classifying their techniques. The shared-task results show that SI was easier than TC, while the strongest systems used Transformer-based approaches and ensembles. The authors also caution that automatic detection can produce false positives and should support user awareness rather than replace judgment.
Problem
Propaganda can influence large audiences while remaining unnoticed, and document-level analysis does not locate or classify specific propaganda fragments.
Method
The shared task evaluates Span Identification and context-aware Technique Classification over an inventory of fourteen propaganda techniques.
Results
Span Identification was easier and all systems improved over baseline, whereas Technique Classification was more challenging and some teams did not improve over baseline.
Takeaways & Limitations
The task provides a fine-grained benchmark for propaganda detection, with participation from 250 teams and 44 official test-set submitters.
Takeaways & Limitations
The authors plan to expand the dataset with more examples and techniques and to develop comparable datasets for other languages.
Abstract
from arXiv · showhide
We present the results and the main findings of SemEval-2020 Task 11 on Detection of Propaganda Techniques in News Articles. The task featured two subtasks. Subtask SI is about Span Identification: given a plain-text document, spot the specific text fragments containing propaganda. Subtask TC is about Technique Classification: given a specific text fragment, in the context of a full document, determine the propaganda technique it uses, choosing from an inventory of 14 possible propaganda techniques. The task attracted a large number of participants: 250 teams signed up to participate and 44 made a submission on the test set. In this paper, we present the task, analyze the results, and discuss the system submissions and the methods they used. For both subtasks, the best systems used pre-trained Transformers and ensembles.
1 Introduction
SemEval-2020 Task 11 targets fine-grained propaganda detection in news, motivated by the difficulty of spotting covert persuasion at scale. It separates locating propaganda spans from classifying their techniques.
- Propaganda can reach large audiences while often remaining unnoticed, making manual detection difficult for inexperienced users and experts alike.
- The task addresses a gap in document-level analysis by enabling fine-grained identification of propaganda-bearing text fragments.
- Subtask SI identifies document fragments containing at least one propaganda technique as a binary sequence-tagging task.
- Subtask TC assigns a propaganda technique to an identified snippet using its document context as a multi-class classification task.
- 250 teams registered, while 44 made official test-set submissions, demonstrating substantial participation in the shared task.
2 Propaganda and its Techniques
The paper operationalizes propaganda as deliberate influence through rhetorical and psychological techniques. It uses a curated fourteen-technique inventory tailored to journalistic text and annotation feasibility.
- Propaganda is defined as deliberate expression by individuals or groups intended to influence others’ opinions or actions toward predetermined ends.
- The task focuses on identifying specific textual techniques because the broad concept of propaganda is unsuitable for directly annotating fragments.
- The inventory contains fourteen techniques selected for occurrence in journalistic articles and intrinsic judgment without external supporting information.
- Loaded language uses emotionally charged words, while name calling or labeling assigns targets qualities the audience fears, hates, dislikes, loves, or praises.
- Rare techniques were merged to reduce data sparseness, including Red herring and Straw man with Whataboutism.
3 Evaluation Framework
The evaluation framework combines the PTC-SemEval20 corpus, annotation procedures, and separate measures for span identification and technique classification. It also defines a two-phase shared-task submission process.
- 3.1 The PTC-SemEval20 Corpus: The corpus contains news articles from mid-2017 through early 2019 collected from 13 propaganda and 36 non-propaganda outlets, after deduplication and filtering.
- 3.1 The PTC-SemEval20 Corpus: Two annotators independently labeled each article, then reconciled disagreements and boundary or label discrepancies with a consolidator.
- 3.1 The PTC-SemEval20 Corpus: 536 articles contain 8,981 propaganda snippets across fourteen classes, with overlapping snippets making SI counts smaller than TC counts.
- 3.2 The Evaluation Measures: Span Identification gives credit for partial matches between gold and predicted character spans.
- 3.2 The Evaluation Measures: Technique Classification is treated as single-label multi-class despite approximately 1.8% of annotations containing identical spans with different techniques.
- 3.3 Task Organization: The task used two phases: development competition with a live leaderboard, followed by a short test-set submission period released task-by-task.
4 Participating Systems
Participating systems in both subtasks were dominated by Transformer-based architectures, often combined with sequence models, engineered features, ensembles, or additional context. The strongest approaches also exploited self-supervision, transfer learning, and information shared between span identification and technique classification.
- Span Identification Subtask: All top-10 span-identification systems used Transformers with an LSTM or CRF, often adding named-entity, sentiment, or subjectivity features.Transformer representations were commonly complemented by engineered linguistic and affective features.
- Span Identification Subtask: Hitachi ranked first in span identification with an end-to-end heterogeneous neural network combining language-model, PoS, and named-entity token representations with bi-LSTMs.The three token representations were concatenated before sequence modeling.
- Span Identification Subtask: ApplicaAI ranked second in span identification by iteratively generating silver data with RoBERTa-CRF and ensembling models trained on gold, re-weighted, and silver data.The system predicted on 500k sentences to create additional training data.
- Technique Classification Subtask: Technique-classification systems showed the same Transformer dominance, while several teams reported gains from engineered lexical, topical, emotional, frequency, and named-entity features.Examples included TF.IDF and character n-grams, topic modeling, polarity, VAD features, and fragment-frequency signals.
- Technique Classification Subtask: ApplicaAI’s top technique-classification approach extended its silver-data strategy by using SI predictions and preliminary TC labels to create extra silver-labeled instances.Its classifiers received propaganda snippets together with their document context.
- Technique Classification Subtask: Top technique-classification models benefited from combining the subtasks and incorporating snippet context, such as surrounding-token windows or joint snippet-and-sentence representations.Context was represented using the full sentence or a three-word window on each side of the snippet.
5 Results and Discussion
The results compare systems and ensembles for span identification and technique classification. Transformer-based systems performed strongly, but development rankings did not consistently predict test rankings, and a post-competition evaluation bug requires consulting corrected scores.
- Span Identification: Transformer-based systems dominated span identification, although several development leaders fell sharply on the test set, suggesting possible overfitting.Only three of the development top five also appeared among the test top five; syrapropa and PALI dropped from first and second to 25th and 18th.
- Span Identification: Union ensembles traded precision for recall, while intersecting system outputs favored higher precision.The union of the top systems could outperform the best individual system, whereas intersection was more suitable for high-precision predictions.
- Technique Classification: Development-to-test rankings were unstable in technique classification as well: Hitachi rose from eighth on development to third on test.Only two of the development top five also appeared among the test top five.
- Technique Classification: Loaded Language and Name Calling or Labeling were comparatively easier, whereas Straw man, red herring and Bandwagon, reduction ad hitlerum, whataboutism were among the hardest classes.The easier classes were more frequent and shorter on average, while the difficult classes were among the least frequent.
- Technique Classification: Technique classification performance peaked at 63.63 when the top three systems were combined, improving more than 1.5 absolute points over the top-one system.Majority voting was used because classification is multi-class; gains varied substantially across techniques.
6 Related Work
Earlier computational work mainly analyzed propaganda at the document or broader online-pattern level. This task instead targets fine-grained textual techniques and connects to research on argumentation, fact-checking, and misinformation.
- Misinformation and fact-checking: Related work includes fake-news detection, fact-checking, truth discovery, and studies of true and false news propagation online.The cited shared tasks and studies span FEVER, community-question-answer fact-checking, and CheckThat! claim identification and verification.
- Argumentation: The paper connects propaganda techniques with computational argumentation because such techniques can involve fallacious reasoning that is hidden or intentional.Related studies examined ad hominem fallacies in web forum discussions and the effect of persuasion on argumentation quality.
- Prior shared tasks: Computational propaganda systems in earlier shared tasks commonly used contextual representations, especially BERT-based models, alongside RoBERTa, Grover, ELMo, and handcrafted features.The earlier tasks treated propaganda-technique identification as one overall task rather than separating span identification and technique classification.
7 Conclusions and Future Work
SemEval-2020 Task 11 drew broad participation and revealed an easier span-identification subtask alongside more difficult technique classification. The authors propose expanding coverage and caution that practical deployment can produce false detections.
- 250 teams signed up, 44 submitted test results, and submissions numbered 35 for SI and 31 for TC.
- SI was easier, with every system improving over the baseline.
- TC was more challenging, and some teams failed to improve over the baseline.
- Future work will expand the dataset with more examples and propaganda techniques, and develop analogous datasets for other languages.
- Automatic propaganda detection may falsely identify propaganda, so the authors frame it as an awareness and education tool rather than a definitive judge.
A Summary of all Submitted Systems
Submitted systems used diverse neural architectures, pretrained language models, engineered features, context modeling, augmentation, and ensembles across the two subtasks. The approaches often combined representations at token, sentence, fragment, and document levels.
- Team 3218IR used a one-dimensional CNN with empirically tuned word-embedding layers, filters, kernels, and pooling sizes.
- ApplicaAI used RoBERTa-CRF with self-supervised silver data for SI and contextual propaganda snippets for TC.
- aschern used ensembles of RoBERTa models with CRF sequence labeling for SI and contextual span features for TC.
- BPGC combined article-, sentence-, and word-level information for SI, and paired BERT, logistic regression, and engineered features for TC.
- CyberWallE and DiSaster supplemented BERT representations with rhetorical, linguistic, sentiment, frequency, and repetition-focused features.
- DoNotDistribute used BERT-based models, engineered features, and near-paraphrase augmentation, reporting a 5% increase from 3k silver instances.
- DUTH normalized inputs and replaced recognized categories using named-entity recognition and gazetteers covering countries, religions, ideologies, and slogans.
- Hitachi trained a heterogeneous multilayer network with pretrained language-model, PoS, and NE representations, auxiliary objectives, Bi-LSTMs, and CRF output.
B Errata Corrige
An evaluation-software bug affected both subtasks, but fixing it did not significantly alter the ranking, especially among top submissions. The corrected test scores are reported in Tables 8 and 9.
- A bug in the evaluation functions affected both SI and TC after the shared task.
- The corrected ranking changed insignificantly, particularly for top-ranked submissions.
- The paper directs references to task results to the corrected numbers.
- Table 8 reports SI test performance after bug fixing, highlighting the highest scores for each measure.
- Table 9 reports technique-classification F1 performance after bug fixing, with systems ordered by final ranking.
C Annotation Instructions
The paper presents snapshots of the annotation instructions, including an annotator guide and a hierarchical diagram for selecting techniques.
- The paper reports a series of snapshots of the annotation instructions.
- Figure 7 provides instructions for annotators.
- Figure 8 presents a hierarchical diagram guiding the choice of propaganda technique.