Source-linked AI summary
SCORE: Story Coherence and Retrieval Enhancement for AI Narratives
Qiang Yi, Yangfan He, Jianhui Wang, Xinyuan Song, ShiYao Qian, Xinhang Yuan, Yi Xin, Yijin Wang, Jingqun Tang, Yuchen Li, Junjiang Lin, Hongyang He, Zhen Tian, Tianxiang Xu, Keqin Li, Kuan Lu, Menghao Huo, Jiaqi Chen, Miao Zhang, Tianyu Shi, Jianyuan Ni
TL;DR
Long-form LLM narratives often lack sustained character, emotional, and plot consistency. SCORE combines state tracking, episode summaries, and RAG-based retrieval to detect and resolve inconsistencies, improving coherence and stability relative to baseline GPT models. The framework remains limited by its academic-research-only use condition.
Problem
Maintaining character, emotional, and plot consistency across extended LLM-generated narratives remains a major challenge.
Method
SCORE combines key-item state tracking, episode summarization, and Retrieval-Augmented Generation to evaluate and improve narrative coherence.
Results
SCORE improves narrative inconsistency detection and coherence, with consistency gains up to 7.8% for Llama-13B and coherence gains up to 7.7% for GPT-4o.
Takeaways & Limitations
SCORE provides a structured framework for evaluating and refining coherent, emotionally consistent AI-generated narratives.
Takeaways & Limitations
SCORE is intended solely for academic research purposes and follows the original access conditions of its incorporated tools and data sources.
Abstract
from arXiv · showhide
Large Language Models (LLMs) can generate creative and engaging narratives from user-specified input, but maintaining coherence and emotional depth throughout these AI-generated stories remains a challenge. In this work, we propose SCORE, a framework for Story Coherence and Retrieval Enhancement, designed to detect and resolve narrative inconsistencies. By tracking key item statuses and generating episode summaries, SCORE uses a Retrieval-Augmented Generation (RAG) approach to identify related episodes and enhance the overall story structure. Experimental results from testing multiple LLM-generated stories demonstrate that SCORE significantly improves the consistency and stability of narrative coherence compared to baseline GPT models, providing a more robust method for evaluating and refining AI-generated narratives.
1. INTRODUCTION
LLMs can generate long-form narratives, but sustaining character, thematic, emotional, and plot consistency across extended stories remains difficult. SCORE addresses this challenge with structured evaluation, key-item tracking, episode summaries, and RAG-based retrieval.
- Motivation: Extended LLM-generated narratives often develop inconsistencies in character behavior, emotional tone, and recurring plot-item states.These problems can undermine thematic consistency, logical flow, and reader engagement.
- Motivation: Structured approaches are needed to manage character arcs, plot developments, and emotional progression throughout stories.
- Proposed Framework: SCORE evaluates character consistency, emotional coherence, and logical tracking of key plot elements using dynamically retrieved context.
- Contributions: SCORE detects narrative inconsistencies in AI-generated stories through an LLM-based evaluation framework.
- Contributions: SCORE combines episode-level summaries, key-item tracking, sentiment analysis, and similarity-based retrieval to improve narrative coherence.
- Contributions: SCORE outperforms baseline GPT models in detecting continuity errors and maintaining overall narrative coherence.
2. METHOD
SCORE integrates key-item state correction, episode summarization, similarity-based retrieval, and sentiment analysis within an RAG-oriented narrative evaluation pipeline. These components provide context for assessing character consistency, plot progression, emotional authenticity, and item continuity.
- Framework Overview: SCORE combines an LLM-based coherence evaluator, automatic episode summaries, and RAG integration for narrative consistency and user interaction.
- Continuity Analysis: The framework extracts item states and flags an item that reappears as active after being lost or destroyed as a continuity error.It retains the earlier lost or destroyed state to prevent an incorrect transition.
- Episode Analysis: Episode analysis summarizes plot points, character actions, relationships, emotional changes, and interactions with important items.
- Episode Analysis: Aggregated summaries, item interactions, and character actions support more precise retrieval while reducing redundancy in later analysis.
- Retrieval and Sentiment: Similarity-based retrieval uses embedded episode chunks and FAISS to identify relevant episodes for queries or episode analysis.
- Retrieval and Sentiment: Sentiment scores from 0 to 1 refine retrieval by combining emotional-tone consistency with textual similarity.
- Final Evaluation: The LLM evaluates retrieved content for character consistency, plot progression, emotional authenticity, and key-item continuity, flagging and correcting discrepancies.
3. EXPERIMENTS
Experiments evaluate SCORE on LLM-generated episodic stories against direct-model baselines and examine its component contributions. SCORE improves inconsistency detection and coherence, with especially large gains for some open-source models and substantial ablation drops when key modules are removed.
- Experimental Setup: The evaluation measures narrative coherence, continuity-error detection, and emotional consistency in episodic stories generated by LLMs.
- Experimental Setup: The dataset contains 5,000 episodes from 1,000 GPT-generated stories spanning science fiction, drama, fantasy, and comedy.Stories average 12 episodes and approximately 2,000 tokens, with manual filtering by three annotators at κ=0.78.
- Baselines: Baselines use GPT-4o-mini, GPT-4o, and GPT-4 directly, while SCORE supplies preprocessed context through RAG and FAISS-based retrieval.
- Main Results: SCORE significantly improves narrative-inconsistency detection and evaluation accuracy compared with GPT models used alone.RAG helps filter irrelevant information and provide current story context across multiple episodes.
- Main Results: Consistency increases up to 7.8% for Llama-13B, coherence up to 7.7% for GPT-4o, and item-status recognition approaches 98%.Open-source models gain 30–38 QA points, while commercial models show more modest improvements.
- Ablation Studies: Removing Dynamic Tracking causes drops of –17.6 points in consistency and –37.1 points in item accuracy, while removing Context Summary reduces coherence by –22.5%.Hybrid Retrieval and Sentiment contribute positively but have smaller effects.
- Case Study: SCORE-generated stories average 555 tokens and 13.2 seconds per episode, with average scores of 3–4/5 across four narrative metrics.
4. CONCLUSION
SCORE is an LLM-based framework for improving long-term coherence and emotional consistency in AI-generated narratives. Its modular RAG pipeline combines state tracking, summarization, and hybrid retrieval, while challenges remain in retrieval accuracy and computational efficiency.
- SCORE combines Dynamic State Tracking, Context-Aware Summarization, and Hybrid Retrieval within a RAG pipeline.These components support the framework's approach to long-term narrative coherence.
- The framework achieves improvements in coherence, stability, and hallucination reduction across multi-genre datasets.
- SCORE's modular design supports scalability and compatibility with multiple LLMs, but retrieval accuracy and computational efficiency remain challenges.