PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-shot narrative detection in social messaging

Jesús M. Fraile-Hernández, Anselmo Peñas, Patrick Giedemann

arXiv:2609.17310v2cs.CL

TL;DR

This paper asks whether LLMs can identify implicit narratives in social messages without task-specific training. It evaluates zero-shot classification using narrative descriptions and ensembles, finding that human-written descriptions and majority voting improve results across the benchmark tasks.

  • Problem

    Narratives may be embedded implicitly through framing and emphasis, making their communicative intent harder to identify than a message’s literal content.

    Source: §1.1 · p. 4, §1 · p. 4

  • Method

    The study evaluates zero-shot narrative classification on Dipromats and SemEval, testing prompt descriptions and classification schemes alongside ensemble strategies.

    Source: §4.3 · p. 18, §9 · p. 41

  • Results

    Across the benchmark tasks, human-written descriptions improve performance over title-only prompts, and majority voting bolsters robustness; original descriptions outperform generated ones in 87.1% of configurations.

    Source: §5.1 · p. 27, §9 · pp. 41–42

  • Takeaways & Limitations

    With coherent human-written descriptions and simple ensembling, zero-shot LLMs rival supervised systems for narrative detection in domains without training data.

    Source: §9 · pp. 42–43

    Performance is expected to decrease with title-only or noisy or mismatched descriptions, whose impact was not systematically evaluated.

    Source: §5.1 · p. 28

Abstract

from arXiv · show

This study investigates the zero-shot ability of large language models (LLMs) to identify and classify hidden narratives in social messages. Our research hypothesis is that LLMs' extensive contextual knowledge allows them to interpret messages on a deeper, pragmatic level, going beyond basic sentiment or topic analysis. Experiments on the Dipromats and SemEval datasets show that providing models with human-written narrative descriptions significantly improves performance, without the need of training examples. In contrast, automatically generated descriptions or the use of few examples (few-shot) often degrade accuracy due to subtle shifts in framing. The study also finds that ensemble methods, particularly majority voting, enhance robustness and that larger models perform best while also being less sensitive to prompt variations. The findings validate that LLMs can effectively detect strategic narratives in a zero-shot setting, and when combined with simple ensembling and human-written descriptions, they can rival supervised systems, offering a scalable solution for narrative detection, specially when there is no training data for the vast majority of domains.

Highlights

The paper investigates whether LLMs can detect implicit strategic narratives in social messages without task-specific training. It evaluates prompt designs, ensembles, and model scale, finding advantages for human-curated descriptions and majority voting.

1.1. Problem statement

  • Narratives can be conveyed implicitly through framing, emphasis, and selective information, so messages on the same topic can advance different persuasive stories.The study distinguishes this pragmatic interpretation from surface-level topic or sentiment analysis.Source: §1.1 · pp. 4–5

1.2. Research hypothesis

  • The study tests the hypothesis that LLMs’ contextual knowledge supports zero-shot identification of strategic communicative intentions beyond sentiment or topic extraction.It frames narratives as implicit pragmatic constructs conveyed through framing, emphasis, and presupposition.Source: §1.2 · p. 5, §1.3 · p. 6

1.3. Objectives

  • The objectives include measuring zero-shot capability, benchmarking against supervised approaches, and examining performance–generalisability trade-offs across thematic domains.The evaluation excludes annotated training data and task-specific fine-tuning.Source: §1.3 · pp. 5–6

1.4. Research Questions

  • The research questions test description length and provenance, prompt improvement strategies, ensemble effectiveness, model scale, and generated-description quality.They also ask whether model-generated descriptions are coherent, semantically accurate, and pragmatically acceptable for detection.Source: §1.4 · pp. 6–7

1.5. Contributions

  • Across Dipromats and SemEval, human-curated descriptions consistently outperform title-only and automatically generated alternatives in zero-shot detection.The paper evaluates narrative and subnarrative tasks across both datasets and a range of prompting strategies.Source: §1.5 · p. 7
  • Majority-voting ensembles improve robustness and accuracy, with particular gains in fine-grained subnarrative detection.The study also finds no significant overall advantage for self-generated over GPT-generated descriptions.Source: §1.5 · p. 7
  • Larger LLMs show better overall performance and robustness to prompt variation, while some mid-sized models balance stability and computational cost.Source: Highlights · p. 2, §1.5 · p. 8
  • Generated descriptions are useful only when aligned with the annotation framework; divergence from ground-truth labels can undermine their utility.Source: §1.5 · p. 8

2. Related Work

The related work spans theoretical definitions of narrative, domain-specific taxonomies, and computational detection methods. It highlights annotation challenges and contrasts unsupervised, supervised, and emerging zero-shot approaches.

2.1. Theoretical Foundations and Definitions of Narrative

  • Narrative definitions range from structured event sequences to Dennison’s account of generalizable selective depictions of reality across at least two points, potentially including causal claims.Other accounts distinguish story from discourse and emphasize interconnected events, agents, and temporal or spatial anchoring.Source: §2.1 · pp. 8–9

2.2. Narrative Taxonomies and Analysis Frameworks

  • Narrative taxonomies cover domains including COVID-19, anti-vaccination, climate denial, and pro-Russian influence, while survey work notes annotation challenges and temporal construction –.Source: §2.2 · p. 9

2.3. Computational Approaches and Challenges in Narrative Detection

  • Computational approaches include unsupervised topic-and-event pipelines, supervised classifiers requiring annotated corpora, and zero-shot systems using descriptions or narrative titles –.Prior work also identifies annotation’s cost and subjectivity, motivating evaluation of context-giving descriptions.Source: §2.3 · pp. 9–10

3. Experimentation setting

The experimental setting uses two narrative-identification benchmarks chosen to evaluate zero-shot detection across socially meaningful content and varied themes. Both provide narrative titles and short descriptions.

3.1. Datasets

  • The study selects datasets with clear narrative definitions and diverse geopolitical or thematic coverage to test zero-shot detection in realistic content.Source: §3.1 · p. 10
  • Experiments use Dipromats 2024 Task 2 and SemEval-2025 Task 10 Subtask 2, publicly available resources with narrative titles and short descriptions.Source: §3.1 · pp. 10–11

Dipromats 2024 Task 2

Dipromats evaluates multilingual political tweets with very limited training data, whereas SemEval covers English news from a multilingual task with a hierarchical narrative scheme. The study follows each task’s evaluation protocol.

Dipromats 2024 Task 2

  • Dipromats is a bilingual political-tweet multilabel task covering four geopolitical actors; its test set has 1,600 instances, while training and development provide only 102 annotated tweets.Each region has at most six narrative labels, and tweets may have no, one, or several labels.Source: Dipromats 2024 Task 2 · p. 11

SemEval-2025 Task 10 Subtask 2

  • SemEval’s English test subset contains 101 news articles; the dataset has 2,054 annotated training and development documents and three-tiered labels for climate change and the Russia–Ukraine war.The two topics contain 10 and 11 main narratives, respectively, and 46 and 49 sub-narratives, respectively.Source: SemEval-2025 Task 10 Subtask 2 · pp. 12, 14

3.2. Evaluation Metrics

  • Performance is assessed using the original task protocols, with different evaluation strategies to account for narrative classification’s subjectivity and annotation variability.Source: §3.2 · p. 14

Dipromats 2024 Task 2

Dipromats annotations allow a “leaning” label for ambiguous support, while models predict yes or no. The study uses F1-average as its primary metric and notes that agreement scores including leaning do not directly measure binary-classification agreement.

Dipromats 2024 Task 2

  • The task records yes, no, or leaning, but frames model predictions as binary yes/no classification.Leaning captures subtle, implicit, or interpretable support for a narrative.Source: Dipromats 2024 Task 2 · p. 14
  • F1-average, the mean of F1-strict and F1-lenient, is the primary metric because it balances treatment of leaning cases.F1-strict treats leaning as negative; F1-lenient treats it as matching either system prediction.Source: Dipromats 2024 Task 2 · pp. 14–15
  • Cohen’s κ is 0.8584 for Spanish and 0.8030 for English, but these scores include leaning and do not directly reflect binary-classification agreement.F1-lenient is considered a more representative indicator of expected consistency in narrative identification.Source: Dipromats 2024 Task 2 · p. 15

SemEval-2025 Task 10 Subtask 2

The study evaluates zero-shot narrative and sub-narrative classification across SemEval and Dipromats using varied descriptions, classifiers, schemes, prompts, and ensemble protocols. Its methodology also accounts for SemEval’s hierarchical labels and the subjectivity of narrative annotation.

SemEval-2025 Task 10 Subtask 2

  • SemEval’s official sample-based F1 averages per-document scores; annotator agreement was 0.449 for main narratives and 0.388 for sub-narratives.The agreement scores are reported using Krippendorff’s α.Source: SemEval-2025 Task 10 Subtask 2 · p. 15

4. Methodology

  • The methodology comprises five steps: description generation, prompt design, classifier selection, classification schemes, and evaluation setup.The evaluation protocols follow the original tasks and account for narrative classification’s subjective and interpretative nature.Source: §4 · p. 15

4.1. Narrative Description Generation

  • Description-generation experiments vary context from title-only to human-description and feature-guide inputs, using five generators ranging from 2B to 78B parameters.Feature-guide descriptions add social, political, and economic context, actors, intended emotions and conclusions, and textual features; these generated details are not human-verified.Source: §4.1 · pp. 15–16

4.2. Classifier Models

  • Four open-access classifiers, selected for local execution under 100GB VRAM and strong MMLU-Pro performance in their size ranges, span 2B to 78B parameters.The selection supports comparison of model scale and architectural diversity for zero-shot narrative detection.Source: §4.2 · pp. 16–17

4.3. Classification Schemes

  • Dipromats uses one-step, region-specific multilabel classification; SemEval compares all-in-one, three-step hierarchical, and hybrid schemes.The hybrid makes two requests, reducing context relative to all-in-one and accumulating fewer errors than the three-step scheme.Source: §4.3 · pp. 17–18

4.4. Prompting Strategies

  • Eight prompting strategies compare title-only and human-authored descriptions with self- or GPT-generated descriptions at varying levels of descriptive richness.The variants include title-based generation and prompts using descriptions generated from the title, original description, or added feature guides.Source: §4.4 · pp. 18–19

4.5. Evaluation Protocol and Ensemble Strategies

  • Each model–prompt combination is averaged over five runs, with majority-voting and inclusion ensembles; cross-model ensembles combine 20 predictions per instance.Majority voting requires a label in at least three runs; inclusion aggregates any label predicted in the runs.Source: §4.5 · pp. 19–20

5. Results

The results examine how narrative descriptions and prompting strategies shape zero-shot detection across Dipromats and SemEval, including tweet-level variation, model stability, ensembles, confidence intervals, and qualitative errors.

5.1. Results Analysis

  • Dipromats OrigDesc improved average performance 1.5 pp over Title; SemEval OrigDesc improved 0.66 pp, while generated descriptions declined 4 pp and 2.7 pp, respectively.On SemEval, the more detailed generated variants declined 4.8 pp, excluding the All-in-One setting that only the smallest model could process.Source: Table 1 · p. 22, Table 2 · p. 23, Table 3 · p. 24
  • Against OrigDesc, generated descriptions declined by 5.6 pp on Dipromats; on SemEval, declines were 4.1 pp for narrative detection and 2.7 pp for subnarrative detection.On Dipromats, Title_Desc and Title_Desc_Guide declined by 7.5 pp and 12 pp, respectively; only 12.9% of evaluated prompt configurations outperformed OrigDesc.Source: §5.1 · pp. 27–28
  • Self_Gen_Title_Desc and GPT_Gen_Title_Desc led generated strategies over Title by 1.89 pp and 1.87 pp, while self- versus GPT-generated descriptions showed no consistent advantage overall.Exceptions included an 18 pp self-generation advantage for Gemma on Dipromats and a 5 pp advantage for Self_Gen_Title in SemEval’s three-step setting.Source: §5.1 · pp. 28–29
  • In SemEval, majority-based ensembles outperformed the baseline in 62.5% of cases, while inclusion ensembles improved results in only 20.8%; inclusion degraded performance by up to 3.5 pp.Majority voting boosted performance by 8.5 pp with four models and 4.8 pp with three.Source: §5.1 · p. 29
  • Calme 78B had the smallest prompt-strategy degradation, under 1 pp on average, while smaller Gemma, Exaone, and Granite models averaged a 3.7 pp drop.Gemma 12B was the most consistent across datasets for ensembles, followed by Calme 78B; Exaone and Granite fluctuated most.Source: Conclusion · p. 30
  • Generated descriptions were often plausible and well-formed, but title-only prompts could shift narrative framing, blend related themes, and obscure intended boundaries in fine-grained classification.The authors conclude that generated descriptions are reliable inputs only when aligned with the annotation framework.Source: Conclusion · pp. 30–31

5.2. Confidence Intervals

  • Dipromats confidence-interval widths were broadly comparable, and interval midpoints identified the same strongest configurations as aggregated mean F1 scores.The intervals were estimated with bootstrap resampling; the alignment supports using five-run aggregated means for comparative claims.Source: §5.2 · pp. 32–33, Table 6 · p. 32

5.3.1. Cross-model and cross-prompt variability

  • Across 160 predictions per tweet, 26 of 562 instances were correctly identified in fewer than 10% of predictions, illustrating substantial ambiguity in some tweets.The qualitative error analysis used Dipromats tweets with at least one positive narrative label, because it alone had gold labels and original English tweets available.Source: §5.3.1 · p. 33, §5.3 · p. 33, Figure 2 · p. 33
  • In some tweets, Calme achieved 100% accuracy across runs while other models performed markedly worse; in others, the largest model failed and smaller models rarely exceeded 30% accuracy.These contrasting patterns show that performance varied by tweet as well as by model.Source: §5.3.1 · p. 34

5.3.2. Focusing on the Calme model: the role of narrative descriptions.

  • Two tweets were correctly classified in all five Title-only runs and misclassified in all 35 runs using original or generated descriptions; two others showed the reverse pattern.Figures 5 and 6 illustrate these contrasting cases.Source: §5.3.2 · p. 35, Figure 5 · p. 35, Figure 6 · p. 36
  • Table 7 shows that descriptions can change Calme’s predictions: richer strategies had higher C2 and C4 counts, while elevated C1 counts indicate descriptions can also reduce performance, potentially through noise or over-specification.The comparison is between Title-only runs and other prompting strategies, including original and automatically generated descriptions.Source: §5.3.2 · p. 35, Table 7 · p. 36

6. Comparison with the State of the Art

The paper compares its best zero-shot systems with leaderboard submissions on SemEval and Dipromats, finding competitive performance without training data. The comparisons also distinguish these results from systems using task-specific training.

6. Comparison with the State of the Art

  • The head-to-head tables compare the best zero-shot system with leaderboard submissions, and Δ denotes performance difference relative to the best-performing system.Source: §6 · p. 37
  • On SemEval, the GATENLP system achieved the highest overall performance using task-specific training, while the paper’s system is competitive without training data.GATENLP combined hierarchical prompting with LLaMa 3.2 fine-tuning; its zero-shot experiments yielded substantially lower scores.Source: Table 8 · p. 37, §6 · pp. 37–38
  • The Dipromats system reports robust performance and surpasses models based on GPT-4 despite lower computational requirements.The table compares the best zero-shot system with leaderboard systems.Source: Table 9 · p. 37, §6 · p. 38

7. Generalisation of Results to Other Languages

The study explores whether zero-shot narrative detection generalizes to Bulgarian, Portuguese, Hindi, and Russian, using original-language documents and English translations. Across both settings, title-based strategies and human-authored descriptions perform best, while generated descriptions add no meaningful improvement.

7.1. Experimental setting

  • The exploratory evaluation uses SemEval development documents in Bulgarian, Portuguese, Hindi, and Russian, with Calme 78B and a hybrid classification approach.It tests original-language inputs and English translations, and uses one inference pass per document rather than the five-run setting used elsewhere.Source: §7 · p. 38, §7.1 · p. 39

7.2. Feeding models with input texts in other languages

7.3. Feeding models with translations into English

  • For English translations, Table 11 reports F1 scores for narrative detection and subnarrative identification.Source: §7.3 · p. 39

7.4. Results

  • Across original-language and translated documents, title-only prompts or titles paired with human-authored descriptions are the most effective strategies; generated descriptions offer no meaningful improvement.The authors interpret this pattern as evidence that models can identify narratives without augmented or automatically generated descriptions.Source: §7.4 · pp. 39–40, Table 10 · p. 40

8. Scalability Analysis

The scalability analysis characterizes the computational demands of models and prompt strategies. It also examines how document length relates to memory use and inference latency.

8. Scalability Analysis

  • The OrigDesc scalability analysis reports request counts, average time per request, and total GPU time for each dataset and model.Experiments ran on a workstation with four NVIDIA RTX A5000 GPUs and 256 GB of system RAM.Source: §8 · p. 40
  • SemEval news articles have wider token-count variation than short Dipromats tweets, affecting memory use and inference latency.Source: §8 · pp. 40–41
  • As model size decreases, the number of tokens processed increases in the OrigDesc SemEval input-length analysis.Figure 7 shows token-count distributions for the All-in-One approach across four models.Source: §8 · pp. 40–41

9. Conclusion

The conclusion reports that human-written narrative descriptions improve zero-shot detection, while automatically generated descriptions can impair fine-grained classification. It also highlights majority-vote ensembling, model scale, and promising multilingual results.

9. Conclusion

  • Human-written narrative descriptions improve performance over title-only prompts, whereas automatically generated descriptions can shift framing and degrade accuracy, especially in multilabel and fine-grained tasks.The reported degradation is particularly relevant to subnarrative classification.Source: §9 · pp. 41–42
  • Majority voting across diverse model runs improves robustness and often surpasses the best individual model, with the greatest gains in subnarrative detection.Excluding the smallest model from the ensemble yields a more stable outcome.Source: §9 · p. 42
  • The largest architectures achieve the highest overall performance and least prompt sensitivity, while some 12B models offer comparable stability at lower computational cost.The conclusion presents mid-sized models as a possible compromise in resource-constrained settings.Source: §9 · p. 42
  • On the SemEval development dataset, the untrained framework shows promising narrative-detection performance in Bulgarian, Portuguese, Hindi, and Russian.The authors suggest this may indicate applicability beyond English.Source: §9 · p. 42
  • With human-written descriptions and simple ensembling, zero-shot LLMs rival supervised systems and offer a scalable alternative where training data are unavailable.The conclusion presents this as a solution for domains without training data.Source: §9 · pp. 42–43

10. Future Work

Future work proposes investigating cross-lingual transfer and few-shot learning, as well as alternative system designs, human-guided descriptions, and systematic analysis of errors.

10. Future Work

  • Evaluate zero-shot narrative detection across more languages and investigate cross-lingual transfer; the current study focuses primarily on English tweets and news articles.The authors expect multilingual evaluation could provide further insight into generalisation.Source: §10 · p. 43
  • Explore few-shot setups, which the authors suggest may yield modest gains, particularly for complex subnarrative detection.They note that producing a small number of narrative annotations costs less than labeling a full dataset.Source: §10 · p. 43
  • Test multi-agent systems in which specialized agents focus on features such as stance, polarity, topical alignment, or temporal framing.The authors propose this division of labor for more robust identification in multilabel and noisy settings.Source: §10 · pp. 43–44
  • Use human-in-the-loop supervision to generate narrative descriptions that better align with annotators’ interpretative frameworks and dataset taxonomies.The authors note that automatic generation may lack semantic fidelity or task relevance.Source: §10 · p. 44
  • Analyze instances consistently misclassified across models and runs to examine ambiguity and structurally atypical inputs.The authors suggest these cases could inform calibration, data augmentation, or annotation refinement.Source: §10 · p. 44

Appendix A: Prompt Structure

The prompt structure color-codes its components, specifies how to classify a tweet, and supplies six narrative themes for classification.

Appendix A: Prompt Structure

  • Blue marks task instructions, green narrative and subnarrative titles, grey descriptions, and red system-level instructions or control tokens.Source: Appendix A · p. 52
  • The six themes cover Western hostility, China as benevolent, epic history, political system and values, government success, and cultural, natural, and heritage appeal.Source: Appendix A · p. 53
  • The model reads a tweet, assigns at most two supported narratives or none, and returns JSON with classifications and reasoning of at most 50 words.Source: Appendix A · p. 53

Related papers

Loading 2609.17310v2…