Source-linked AI summary
Are Large Language Models Capable of Generating Human-Level Narratives?
Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, Nanyun Peng
TL;DR
The paper addresses the lack of a computational framework for assessing narrative discourse, introducing measures of story arcs, turning points, and affective dynamics. Its comparisons find LLMs fall short of human storytelling and comprehension, while discourse-aware generation improves narrative qualities.
Problem
A computational framework for quantitative assessment of narratives at the global or discourse level remains to be established.
Source: §1 · p. 1Method
The authors benchmark narrative structure across story arcs, turning points, and affect using annotated movie synopses, then evaluate LLM comprehension and discourse-aware generation.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The findings lay groundwork for future research refining models to incorporate complex narrative structures in storytelling and beyond.
Source: §1 · p. 2Human-annotated comparisons and storytelling experiments cover GPT-4 only; other LLM generations were not directly assessed.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
This paper investigates the capability of LLMs in storytelling, focusing on narrative development and plot progression. We introduce a novel computational framework to analyze narratives through three discourse-level aspects: i) story arcs, ii) turning points, and iii) affective dimensions, including arousal and valence. By leveraging expert and automatic annotations, we uncover significant discrepancies between the LLM- and human- written stories. While human-written stories are suspenseful, arousing, and diverse in narrative structures, LLM stories are homogeneously positive and lack tension. Next, we measure narrative reasoning skills as a precursor to generative capacities, concluding that most LLMs fall short of human abilities in discourse understanding. Finally, we show that explicit integration of aforementioned discourse features can enhance storytelling, as is demonstrated by over 40% improvement in neural storytelling in terms of diversity, suspense, and arousal.
1 Introduction
The paper develops a quantitative framework for comparing narrative structure and uses it to examine LLM storytelling and comprehension. It reports gaps from human performance and improvements when discourse features guide generation.
1 Introduction
- The paper motivates narrative assessment by the role storytelling plays in shaping understandings of ourselves, society, and the world.It notes that LLMs are increasingly integrated into daily communicative routines.Source: §1 · p. 1
- LLM stories use happier, less complex arcs, place turning points earlier, and contain less suspense or fewer setbacks than human stories.The paper also reports that these differences grow as LLMs become more prominent in communicative patterns.Source: Figure 1 · p. 1
- The framework measures story arcs, turning points, and affective dimensions, using human and automated annotations of movie synopses to compare storytelling and probe comprehension.The authors describe it as the first quantitative framework for benchmarking narrative development and release a benchmark dataset.Source: §1 · pp. 1–2
- 45% improvement in narrative diversity and 40% improvement in suspense and engagement over vanilla prompting follow from incorporating story arcs and turning-point information, respectively.The paper reports these as results from two parallel experiments.Source: §1 · p. 2
- Most evaluated LLMs fall short of human abilities in narrative comprehension and generation, especially in storytelling.The comparison examines both story-telling and story-comprehension abilities.Source: §1 · p. 2
2 Background: Discourse in Narratives
The paper treats narrative development as a combination of macro-level story arcs, meso-level turning points, and micro-level affective dynamics. Together, these concepts describe protagonist change, plot shifts, and emotional intensity or polarity.
2.1 Three Aspects of Story-Telling
- Story arcs classify protagonist transformations across plot progression using an expanded seven-part schema, following Reagan et al. (2016) and Wu et al. (2023).Examples include “Rags to Riches” and “Cinderella” (Vonnegut, 1995).Source: §2.1 · pp. 2–3
- Turning points are plot events that significantly influence progression; Major Setback and Climax are considered most important in determining the story arc (Papalampidi et al., 2019).Opportunity, Change of Plans, and Point of No Return capture exposition or rising action; Major Setback develops conflict and Climax describes resolution.Source: §2.1 · p. 3
- Arousal measures emotional intensity and valence measures positivity or negativity, with NRC-VAD token scores providing sentence-level values from 0 to 1.These dimensions capture emotional shifts that broader structures such as arcs and turning points may not represent.Source: §2.1 · p. 3
3 Data Collection and Annotation
The authors build paired human- and GPT-4-generated film synopses and annotate story arcs, turning points, and affect. Their analyses compare plot structure and emotional dynamics across narratives.
3.1 Data Preparation
- The dataset contains 819 film synopses selected after filtering well-known movies by Wikipedia page length, paired with GPT-4-generated synopses, for 1,638 narratives of roughly the same length.Titles and initial settings were rephrased by altering unique identifiers to reduce data contamination.Source: §3.1 · p. 3
3.2 Analysis Approaches
- Annotators labeled each synopsis with a story arc and the sentence positions of five turning points, with flexible or optional positions allowed for uncertain cases.TP3 and TP4 received flexible positioning; missing or multiple TP2–TP4 positions were optional though discouraged.Source: §3.2 · pp. 3–4
- A total of 440 narratives received three in-depth annotations each, with inter-annotator agreement of 0.90 for one task and 0.62 for the other.The paper reports Spearman’s Correlation for the first agreement value and Cohen’s Kappa for the second; evaluations were limited to human and GPT-4-0613 stories.Source: §3.2 · p. 4
- Affect annotation asks GPT-4 to identify the protagonist and infer three emotion adjectives per sentence, then averages their NRC-VAD scores for arousal and valence.Sentence-level values are interpolated into smooth curves and averaged across narratives; the paper reports early TP4–TP5 positions in AI outputs.Source: §3.2 · p. 4, Figure 2 · p. 4
4 Human vs. AI Narratives: A Discourse-Level Comparison
Compared with human stories, LLM narratives pace key turning points too early and have less suspense. GPT-4 also favors simpler, happier story arcs and avoids negative plot progressions.
4 Human vs. AI Narratives: A Discourse-Level Comparison
- AI stories introduce the major setback and climax earlier than human stories, leaving less room for suspense to build.The positions of the first three turning points are consistent across human and AI narratives, but TP4 and TP5 occur earlier in AI outputs.Source: §4 · p. 5
- Human-written stories sustain higher arousal than GPT-4 stories, with the gap widening from the midpoint toward the end.The authors associate this pattern with less narration devoted to the final turning points in AI stories.Source: §4 · p. 5, Figure 3 · p. 5
- GPT-4 generates Man in Hole, the most common human arc, more than half the time, while its arc distribution favors fewer inflections and happier endings.Human stories have more setbacks or negative events, and the valence gap widens from the midpoint to the ending.Source: §4 · p. 5, Figure 4 · p. 5
- GPT-4 favors positive arcs: Riches to Rags and Oedipus make up 1.3% and 1.7% of its stories, versus 14.6% and 9.3% of human stories.Rags to Riches accounts for 13.0% of AI stories and 4.4% of human stories; the authors suggest RLHF may contribute to the positive bias.Source: §4 · p. 5
5 Benchmarking Narrative Comprehension
The paper benchmarks whether language models can identify story arcs and turning points, both from narrative text alone and with additional discourse information. Most models fall short of human performance, though providing turning-point information improves story-arc identification across models.
5.1 Benchmark Tasks
- The benchmarks test story-arc classification and identification of five turning points, comparing model predictions with expert annotations.The turning-point task identifies each point by sentence position and a predefined type label.Source: §5.1 · p. 6
5.2 Models
- The study evaluates GPT-3.5, GPT-4, Gemini, Claude, and Llama3, using accuracy to measure agreement with expert annotations.For turning-point identification, models analyze and tag key turning points in movie synopses; narratives include sentence indices.Source: §5.2 · p. 6
5.3 Benchmark Findings
- All evaluated models perform below humans on turning-point identification; Gemini and Claude average over 30%, while GPT-4 averages 26%.GPT-3.5 and Llama3 lag behind the other models.Source: §5.3 · pp. 6–7
- On story-arc identification, GPT-3.5 scores approximately 0.2 exact agreement, while GPT-4, Claude, and Llama3 exceed 0.35; humans still do better without extra information.Language models approach human accuracy only when provided with turning-point ground truth.Source: Figure 6 · p. 6, §5.3 · p. 7
- Turning-point information improves story-arc identification across all models, while story-arc information raises average turning-point accuracy by 2% for GPT-4 and Llama3, but not all models benefit.The authors report that finer-grained information benefits the coarser-grained task more than the reverse.Source: §5.3 · p. 7
6 Towards Better Machine Storytelling
The paper tests whether explicit discourse structure can improve machine storytelling, using planning-first prompts that add turning points or specified story arcs. Turning-point guidance improves human-rated suspense and emotion, while arc guidance increases several dimensions of narrative diversity.
6 Towards Better Machine Storytelling
- The planning-first comparison includes outline-only prompting, self-generated turning points, and human-supplied major setbacks and climaxes.The human-supplied events can disrupt machine narrative flow, whereas the self-generated approach maintains natural flow and is the most balanced and least disliked.Source: §6 · p. 7
- Both self-generated and human-supplied turning points significantly improve suspense and emotion over outline-only stories; human-supplied points are not significantly preferred over self-generated ones.Human-supplied turning points scored 48.3% for emotional engagement, while self-generated turning points were the most balanced and least disliked approach.Source: §6 · p. 7
- GPT-4 follows specified story arcs with an average success rate below 55%, struggling particularly with negative progressions, humble starts, and complex arcs.The tested arc-enhanced prompt specifies the arc, the number of rises and falls, and the protagonists’ initial and ending states.Source: §6 · p. 8
- Arc-enhanced stories outperform outline-only stories on thematic, conflict-type, and character diversity.The study compares diversity across theme, setting, conflict, and character.Source: §6 · p. 8, Table 6 · p. 8
7 Qualitative Study: Understanding Human Preferences
Follow-up interviews gathered annotators’ reactions to machine-generated narratives and their diversity. Annotators preferred concrete stories with logical, well-motivated twists and disliked straightforward positive plots and unjustified miracle turns.
7 Qualitative Study: Understanding Human Preferences
- Two follow-up interviews gathered annotators’ qualitative feedback after they compared outline-only and arc-enhanced machine narratives.Interviewees discussed diversity and overall preference.Source: §7 · p. 8
- Annotators preferred concrete narratives with plot twists that were logical and well-motivated.Source: §7 · p. 8
- Annotators disliked straightforward positive plots and “miracle turns” that were not adequately justified.Source: §7 · p. 8
8 Related Work
Prior work examines narrative structure, creative differences between humans and AI, and discourse-aware generation, but creativity and discourse structure have often been vaguely defined. This paper draws on drama and fiction theories to compare human and machine stories and incorporate discourse elements into generation.
8 Related Work
- Discourse-aware evaluation extends beyond fluency and coherence to structural and creative quality, alongside work on narrative structures, human–AI differences, and positional discourse coherence.The cited examples include narrative structures embedded in evaluation (Liu et al., 2024), a qualitative comparison of human and AI narratives (Chakrabarty et al., 2024), and a metric for logical progression (Wang et al., 2023).Source: §8 · p. 8
- Prior work has been limited by vague definitions of creativity and discourse structure; this study takes inspiration from discourse analysis of drama and fiction.The passage names Labov and Waletzky (1967) and Vonnegut (1995) as sources of inspiration.Source: §8 · p. 8
- Freytag’s five-part dramatic structure comprises Exposition, Rising Action, Climax, Falling Action, and Dénouement; this study defines story arcs and turning points for creative stories.Earlier work also applied literary theories to macrostructures and classified similar elements in artificial and human-generated news (Li et al., 2018; Choubey et al., 2020; Spangher et al., 2021, 2022).Source: §8 · p. 9
- Earlier story-generation work targets coherence, logic, interest, affect, bookend structure, or suspense, while this study systematically incorporates three discourse levels.The three levels are story arcs, turning points, and affective dimensions.Source: §8 · p. 9
9 Conclusion
The paper introduces a three-level discourse framework and an expert-annotated dataset for comparing narrative generation and comprehension. Its findings identify weaknesses in LLM storytelling and report improvements from discourse-aware generation.
9 Conclusion
- The study analyzes story arcs, turning points, and affective dimensions, using an expert-annotated dataset to compare human and AI narrative generation and comprehension.The levels are macro-, meso-, and micro-level, respectively.Source: §9 · p. 9
- LLMs fall short especially in story writing, with limited narrative diversity and difficulty developing major setbacks and climaxes.The paper reports these turning-point weaknesses as contributing to less engaging stories.Source: §9 · p. 9
- Discourse-aware generation improves AI storytelling in suspense, emotion engagement, and narrative diversity.The authors describe the dataset and results as a starting point for systematic narrative-discourse analysis and a source of insights for generation, comprehension, and evaluation.Source: §9 · p. 9
Limitations
The study’s human-annotation comparisons and storytelling experiments use GPT-4 narratives, and its resources and research focus are primarily English-based. The authors also flag a risk that the pipeline could perpetuate or intensify dataset gender biases.
Limitations
- Human-annotation comparisons and machine-storytelling experiments are limited to GPT-4 narratives; other LLM generations were not directly assessed.The authors believe the conclusions may apply to Claude, Llama, and Gemini, but those models’ generations were not directly assessed.Source: Limitations · p. 9
- The research primarily focuses on English-based LLMs and resources.The authors identify multilingual models and diverse linguistic resources as possible extensions.Source: Limitations · p. 10
- Because earlier work identified gender biases in the dataset, stories generated using this pipeline may also risk perpetuating and intensifying those biases.Source: Limitations · p. 10
A Additional Results
The appendix reports fuzzy-matching results, examples and annotator feedback, annotation interfaces, and prompts used to prepare and benchmark narratives.
B.1 Examples of standard annotation
- In the example pairs, human-written stories contain more suspenseful and arousing events, while machine stories reach major setbacks and climaxes earlier.Source: §B.1 · p. 13
A Additional Results
- Arc fuzzy matching credits predictions within predefined hard pairs, yet all models remain significantly below humans; turning-point matching allows ±3 positions and follows the exact-match trend.The turning-point fuzzy-matching success rates are presented as percentages in Table 8.Source: A Additional Results · p. 13, Table 8 · p. 14
B.2 Detailed feedback from interviewed annotators
- Interviewed annotators preferred concrete, logically motivated plot twists and disliked straightforward positive plots or unjustified “miracle turns.”They discussed pairs generated with Outline-Only and Arc-Enhanced strategies while the specific methods and models were hidden.Source: §B.2 · pp. 13–14
C.1 Human Annotation Interfaces
- The annotation task asks annotators to label each narrative’s story arc and locate the sentence positions of five turning points.The appendix provides the interface and detailed guidelines for both tasks.Source: §C.1 · p. 14
D.1 Prompts Used in Data Preparation
- For data preparation, GPT-4 rephrases each human story’s opening setting by replacing proper nouns and slightly altering phrasing, then generates a similar but nonidentical title.The process uses the original title and rephrased setting.Source: §D.1 · p. 14
D.2 Prompts Used in Benchmarking
- The appendix presents separate prompts for story-arc identification and turning-point identification.These are shown in Figures 19 and 20, respectively.Source: §D.2 · p. 14, Figure 19 · p. 23, Figure 20 · p. 23