Source-linked AI summary
SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning
Hans Ole Hatzel, Ekaterina Artemova, Haimo Paul Stiemer, Evelyn Gius, Chris Biemann
TL;DR
The paper addresses how to computationally model narrative similarity and evaluate narrative representations under a theory-compatible, intuition-driven definition. It constructs a contrastive triple-based shared task and uses its judgments to evaluate embeddings. The benchmark shows remaining but limited room for improvement, while label quality is high under stated assumptions but does not establish a definitive ceiling.
Problem
Existing story-similarity work relies largely on scalar ratings or external similarity information, motivating a human-judged contrastive benchmark for narrative relatedness.
Method
The task collects contrastive judgments for anchor–candidate triples and evaluates both direct comparisons and embedding distances against those judgments.
Results
Cross-participant ensembles outperform the best individual system, while no submitted system reaches the theoretical label-quality limit.
Takeaways & Limitations
The benchmark provides a public, large-scale resource for narrative similarity and indicates remaining, albeit limited, room to improve automated systems.
Takeaways & Limitations
The study cannot definitively establish the performance ceiling enabled by its labels, and its sampling and rejection filtering may bias which narrative aspects are represented.
Abstract
from arXiv · showhide
We present the shared task on narrative similarity and narrative representation learning - NSNRL (pronounced "nass-na-rel"). The task operationalizes narrative similarity as a binary classification problem: determining which of two stories is more similar to an anchor story. We introduce a novel definition of narrative similarity, compatible with both narrative theory and intuitive judgment. Based on the similarity judgments collected under this concept, we also evaluate narrative embedding representations. We collected at least two annotations each for more than 1,000 story summary triples, with each annotation being backed by at least two annotators in agreement. This paper describes the sampling and annotation process for the dataset; further, we give an overview of the submitted systems and the techniques they employ. We received a total of 71 final submissions from 46 teams across our two tracks. In our triple-based classification setup, LLM ensembles make up many of the top-scoring systems, while in the embedding setup, systems with pre- and post-processing on pretrained embedding models perform about on par with custom fine-tuned solutions. Our analysis identifies potential headroom for improvement of automated systems in both tracks. The task website includes visualizations of embeddings alongside instance-level classification results for all teams.
1 Introduction
The shared task advances computational narrative similarity by defining it around abstract story relatedness and evaluating systems through contrastive comparisons or narrative embeddings. It uses Wikipedia synopses to make annotation and computation more tractable while retaining plot-level information.
- Narrative similarity: Narrative similarity is defined as perceived story relatedness based on abstract causality and progression rather than concrete details.The definition is intended to remain compatible with both narrative theory and intuitive judgment.
- Data and annotation: Wikipedia synopses are used instead of full stories because shorter texts reduce computational demands and allow broader coverage under limited annotation resources.The authors note that summaries may omit details relevant to some aspects of story similarity.
- Task design: The task compares two candidate stories against an anchor or evaluates whether embedding distances preserve annotated partial similarity orderings.These are implemented as direct comparison and embedding-based tracks.
- Evaluation: The shared task evaluates automated systems against human baselines to assess the proposed narrative-similarity definition and narrative representations.It is structured as a shared task to encourage approaches grounded in narrative theory or technological advances.
- Data and annotation: The contrastive annotation setup asks participants to choose the more narratively similar candidate, addressing consistency and discrimination problems associated with scalar ratings.Prior work commonly used scalar similarity ratings, whose scale points can be difficult to apply consistently.
3 Towards A Definition of Similarity
The task defines narrative similarity as a balanced, intuition-aligned judgment over abstract story structure rather than a single prescribed theory. It operationalizes this definition through human-annotated triples, two evaluation tracks, and model-assisted sampling procedures.
- Similarity definition: The guidelines balance formal consistency with human intuition while avoiding undue emphasis on any single narrative aspect.The paper deliberately does not operationalize one specific theory of stories.
- Similarity definition: Annotators consider course of action, outcomes, and abstract theme without receiving instructions on how to weight them.Course of action covers event sequences, outcomes cover lasting results, and abstract theme covers motifs and themes rather than concrete settings.
- Task structure: Track A selects the more similar candidate in an anchor-and-two-candidates triple, while Track B evaluates story embeddings through the same annotated orderings.Track B compares cosine distances from the anchor and expects the more similar candidate to be closer.
- Dataset construction: The dataset uses short English summaries, with three creation stages: summary sourcing, candidate triple sampling, and human annotation.Summaries are filtered for length and content, and short stories are favored to cover more examples under limited annotation resources.
- Dataset construction: Two LLMs are used for rejection sampling, retaining triples on which their similarity judgments disagree for human annotation.The approach targets difficult cases rather than spending annotation effort on examples that commercial LLMs already solve consistently.
- Annotation quality: Annotators selected the first candidate in approximately 58% of cases, so candidate order was randomized again for the shared task.The additional randomization reduces opportunities for systems to exploit positional bias, although the paper notes that it lowers data quality.
6 Baselines
The baselines show modest lexical and embedding performance, while a simple GPT-4o-mini prompt substantially improves over random chance. In Track B, story-emb only slightly exceeds a sentence-similarity encoder and is surpassed by its foundation model e5.
- GPT-4o-mini substantially improves over the random baseline, whereas token-based Jaccard similarity performs only slightly above chance.
- In Track B, story-emb only slightly outperforms a simple sentence-similarity encoder and is surpassed by foundation model e5.
7 Participant Results & Leaderboard
Participant results favor ensemble and dynamically routed approaches in Track A, while Track B is considerably harder and rewards preprocessing or fine-tuning of pretrained embeddings. The leading systems and broader analyses reveal both methodological patterns and practical constraints.
- Track A: 44 Track A submissions were evaluated, with 14 teams surpassing 70% accuracy.
- Track A: 78.00% accuracy placed COGNAC first in Track A through LLM majority voting, aspect decomposition, and dynamic routing.
- Track A: 75.75% and 75.00% accuracy placed FactUEP and AI-Monitors second and third, respectively, using LLM decomposition or multi-model voting.
- Track A: Many top systems use ensembling or voting, while difficult cases commonly trigger dynamic routing, explicit aspect modeling, symbolic tie-breaking, or reevaluation.
- Track A: CoT results are mixed: TeleAI reports a large improvement, whereas COGNAC finds higher cost without advantage, FactUEP reports harm, and TeamCV slightly favors few-shot prompting.
- Track B: Track B’s best system performs about 10 points below Track A’s best, despite matching distributions, indicating greater difficulty for embedding-based decisions.
- Track B: 72.00%, 71.25%, and 70.50% accuracy placed COGNAC, YNU-HPCC, and hits_team first through third in Track B.
- Track B: Track B accuracy correlates only 0.28 with embedding size, while the best system uses 3,072 dimensions despite six teams using larger embeddings.
8 Data Quality & Annotation Difficulty
Independent re-annotations report agreement comparable to the task’s annotations, while qualitative analyses characterize difficult examples as ambiguous. Cross-system agreement is generally high, but Track B contains more instances where most systems disagree with human labels.
- Re-annotation: Independent re-annotation studies report Krippendorff’s alpha values of 0.313 and 0.32, comparable to the task’s agreement.
- Ambiguity: COGNAC finds many difficult examples ambiguous, while CuriosAI attributes some errors to misleading lexical overlap.
- Cross-system agreement: For most triples, most systems agree with the human label, and both tracks have similarly many easy instances with over 0.9 system agreement.
- Cross-system agreement: Track B has noticeably more instances where the vast majority of systems disagree with the human label.
9 Quantitative Result Analysis
The analysis compares system behavior across narrative similarity, genre prediction, release-year prediction, positional bias, and cross-team ensembles. Cross-team combinations slightly improve over the best individual systems, while hard samples remain substantially more difficult.
- Hard Samples: 59.97% accuracy on 74 hard samples contrasts with 68.05% on easier samples.FactUEP reached 71.62% and AI-Monitors 70.27% on hard samples, narrowly exceeding COGNAC at 67.57%.
- Narrative Embeddings Predict Genre: 0.67 Pearson correlation links narrative similarity accuracy with kNN-based genre P@1.The analysis interprets this association as evidence that narrative is highly associated with genre for the approaches evaluated.
- Release-Year Prediction: -0.14 Pearson correlation between release-year mean squared error and narrative similarity accuracy indicates little relationship.The authors conclude that the narrative embeddings in this setup do not predict a story’s release year.
- Cross-Team Ensembles: 79.25% accuracy from Track A majority-vote ensembling across the top 20 systems slightly exceeds the best individual system.For Track B, combining the top five systems reaches 73.75% accuracy and likewise slightly improves on the top-scoring system.
- Positional Biases: Submitted systems generally lack the strong positional bias observed in human annotators.Systems predicted text A an average of 191 times out of 400, compared with 208 A-closer gold labels; excluding CICL26, the average was 196.4.
10 Conclusion
The task introduces a public benchmark for contrastive narrative similarity and narrative representation learning, using an intuition-driven notion compatible with narrative theory. Results show remaining but limited headroom, while similarity judgments remain subjective because aspect weightings are unspecified.
- Conclusion: Cross-participant ensembles outperform the best individual system, and no submitted system reaches the theoretical limit for label quality.The authors describe this as evidence of remaining, albeit limited, room for improvement.
- Limitations: Similarity judgments remain subjective because the annotation guidelines do not specify weightings for individual similarity aspects.The paper recommends evaluating on the full dataset and incorporating human label variation in future work.
- Dataset: The released dataset includes individual human labels and annotator comments.These materials accompany the public resource for future evaluation of narrative representation learning.
- Contributions: The benchmark uses contrastive judgments between two candidate stories and operationalizes narrative similarity through intuition-driven, theory-compatible annotations.The authors present this as an improvement over prior scalar or externally derived similarity resources.
11 Limitations
The dataset has important scope and validity constraints: label quality depends on assumptions, sampling may bias selected triples, and the work covers only English narratives.
- 11 Limitations: The data approaches almost 90% label quality against oracle labels under certain assumptions, but this does not establish a definitive performance ceiling.The paper also notes that oracle labels do not fully capture narrative similarity’s inherent ambiguity.
- 11 Limitations: Story-emb-based sampling likely biased the selected triples toward particular narrative aspects.Rejection filtering may further concentrate the dataset on aspects that current models do not natively capture.
- 11 Limitations: The work is limited to English, and storytelling conventions vary substantially across cultures.The authors identify broader linguistic and cultural coverage as a direction for future work.
12 Ethical Considerations
The task is characterized as low risk because it generates no content and has no immediately harmful applications, while annotators receive safeguards for potentially upsetting material.
- 12 Ethical Considerations: The authors characterize narrative similarity as a low-risk language-model application because it generates no content.They report no immediately harmful applications for narrative similarity judgments.
- 12 Ethical Considerations: Annotators were warned about sensitive topics and could skip tasks if they felt uncomfortable.Compensation was set at least twice the applicable legal minimum wage, although specific rates were undisclosed.
A Full Results
The full-results materials identify the Track A accuracy table and describe prompt-based summary filtering, including quality criteria and generated-story diversity procedures.
- A Full Results: The full results are reported separately for Track A in Table 4 and Track B in Table 5.The supplied passages identify the tables but do not include their numerical results.
- A Full Results: Quality-assurance prompts retain open-ended summaries while removing texts that do not reveal the actual story.The filtering targets premise-only summaries and extraneous markup such as headings.
- A Full Results: The quality filter asks whether a text is a story summary rather than merely a teaser and requires minimal Markdown without headings.The prompt is designed to ensure that summaries reveal a sizable part of the plot.
- A Full Results: The results section includes a ranked team-accuracy presentation, but the supplied passage does not provide individual team scores.The available table heading identifies the columns as rank, team, accuracy, author, and description.
- A Full Results: Track A results are reported as accuracy in percent in Table 4.The supplied passages identify the table but provide no row-level scores.
- A Full Results: The story-generation procedure uses multiple language models to create more diverse stories.The supplied model list includes GPT, Claude, Llama, Qwen, and DeepSeek variants.
- A Full Results: Generated stories are requested in 5–8 sentences, in a neutral Wikipedia film-plot style without titles, years, or metadata.Prompts cover original stories, close plot-following variants, and loosely inspired variants with altered narrative details.
D Track B: Hierarchical Clustering
Track B materials visualize model similarity rankings through hierarchical clustering, with examples pairing stories under gold labels and reporting the number of correct models.
- D Track B: Hierarchical Clustering: One displayed example reports 8 correct models for Gold A.The supplied passages show three candidate story summaries associated with this gold label but do not state the comparison outcome.
- D Track B: Hierarchical Clustering: A second displayed example reports 9 correct models for Gold B.The supplied passages show candidate summaries involving geopolitical or criminal plots but do not state the comparison outcome.
- D Track B: Hierarchical Clustering: Figure 6 visualizes the relationship among models’ similarity rankings using pairwise Spearman coefficients across Track B test-set story pairs.The figure clusters participants according to embedding-similarity behavior.
E.3 Sample index 251
This sample index juxtaposes narrative summaries involving a prophet returning home, a workman learning about life’s transience, and an architect seeking change. It also includes a Track B accuracy-table label and an instance-level correctness marker.
- Evaluation metadata: The sample includes a correctness marker identifying 8 correct models and Gold B.A separate caption identifies Table 5 as Track B results reported as accuracy percentages.
- Narrative samples: The prophet Al Mustafa leaves Orphalese after 12 years and discusses human life with people who stop him before his voyage.The book’s chapters address topics including love, marriage, work, freedom, pain, friendship, and conversation.
- Narrative samples: A workman visiting Amsterdam mistakes “Kannitverstan” for a person’s name when asking about an impressive house and a valuable ship.He later finds comfort in realizing that death erases social differences and that life is fleeting.
- Narrative samples: Bruno, a 35-year-old architect with a comfortable family life and successful office, leaves everything behind after experiencing profound unease.His departure coincides with a businessman asking him to design an icon for Santiago, after which he begins a search.