Source-linked AI summary
Lost in Stories: Consistency Bugs in Long Story Generation by LLMs
Junjie Li, Xinrui Guo, Yuhao Wu, Roy Ka-Wei Lee, Hongzhi Li, Yutao Xie
TL;DR
Long-form story generators can contradict established facts, character traits, and world rules, while existing benchmarks largely emphasize plot quality and fluency. The paper introduces ConStory-Bench and ConStory-Checker to evaluate narrative consistency with fine-grained, evidence-grounded judgments. Experiments find systematic errors concentrated especially in factual tracking and temporal reasoning, with predictable narrative clustering.
Problem
Existing long-form narrative evaluations emphasize plot coherence and fluency without systematically isolating cross-context contradictions or providing reproducible, evidence-grounded evaluation.
Method
The paper builds ConStory-Bench with 2,000 prompts, four task scenarios, and 19 error subtypes, and develops ConStory-Checker to detect contradictions using exact textual evidence.
Results
Current LLMs produce systematic consistency errors, especially in factual tracking and temporal reasoning, and these errors cluster in predictable narrative regions.
Takeaways & Limitations
Consistency evaluation should examine factual and temporal failures and their narrative positions rather than relying only on general story quality.
Takeaways & Limitations
The benchmark covers English fiction shaped by Western narrative conventions and treats consistency as a binary judgment, including intentional contradictions among apparent errors.
Abstract
from arXiv · showhide
What happens when a storyteller forgets its own story? Large Language Models (LLMs) can now generate narratives spanning tens of thousands of words, but they often fail to maintain consistency throughout. When generating long-form narratives, these models can contradict their own established facts, character traits, and world rules. Existing story generation benchmarks focus mainly on plot quality and fluency, leaving consistency errors largely unexplored. To address this gap, we present ConStory-Bench, a benchmark designed to evaluate narrative consistency in long-form story generation. It contains 2,000 prompts across four task scenarios and defines a taxonomy of five error categories with 19 fine-grained subtypes. We also develop ConStory-Checker, an automated pipeline that detects contradictions and grounds each judgment in explicit textual evidence. Evaluating a range of LLMs through five research questions, we find that consistency errors show clear tendencies: they are most common in factual and temporal dimensions, tend to appear around the middle of narratives, occur in text segments with higher token-level entropy, and certain error types tend to co-occur. These findings can inform future efforts to improve consistency in long-form narrative generation. Our project page is available at https://picrew.github.io/constory-bench.github.io/.
1 Introduction
Long-form story generation requires models to preserve facts, entities, events, world rules, and style across thousands of tokens, but existing evaluations leave global consistency underexplored. ConStory-Bench and CONSTORY-CHECKER address this gap with large-scale, evidence-grounded evaluation and analyses of how errors arise.
- Long-form generation demands consistency across entities, events, world rules, and stylistic conventions rather than only local fluency.
- Existing narrative evaluations emphasize plot coherence and fluency, while systematic cross-context contradiction detection remains limited.
- ConStory-Bench contains 2,000 prompts across four narrative task scenarios and defines five error categories with 19 fine-grained subtypes.
- CONSTORY-CHECKER detects contradictions and grounds each judgment in explicit textual evidence with exact quotations.
- The study evaluates diverse generation systems through five research questions concerning error distributions, output length, predictors, co-occurrence, and narrative position.
2 ConStory-Bench
ConStory-Bench constructs long-form narrative prompts from diverse story corpora, organizes consistency failures into a five-category taxonomy, and evaluates them with an auditable contradiction-checking pipeline. The benchmark targets 8,000–10,000-word generations and produces standardized, evidence-linked reports.
- Dataset Construction: Seed stories come from seven public corpora spanning long-context, creative-writing, writing-prompt, and plot-based datasets.
- Dataset Construction: Each story is assigned one of four task types—generation, continuation, expansion, or completion—and rewritten into an 8,000–10,000-word target prompt.
- Consistency Error Taxonomy: The taxonomy contains five top-level categories and 19 fine-grained error types covering temporal logic, character memory, world rules, factual details, and narrative style.
- Automated Error Detection Pipeline: CONSTORY-CHECKER extracts contradiction-prone spans with category-specific prompts across five narrative dimensions.
- Automated Error Detection Pipeline: Extracted spans are paired and classified as consistent or contradictory to reduce false positives and isolate genuine inconsistencies.
- Automated Error Detection Pipeline: Evidence chains record the reasoning, quoted text with positions, and resulting error type, while standardized JSON reports preserve precise character-level offsets.
3 Evaluation
The evaluation measures narrative consistency across diverse LLM systems using length-normalized and prompt-relative metrics, then analyzes how errors vary with output length, uncertainty, category, and position. Results show substantial model differences, approximately linear error growth with length, higher entropy in error-bearing text, clustered error types, and mid-narrative concentration.
- Evaluation setup: The benchmark evaluates proprietary, open-source, capability-enhanced, and agentic systems on 2,000 prompts across four task scenarios.Models generate outputs for all prompts under comparable settings.
- Metrics: CED normalizes errors per 10,000 words, while GRR ranks models within each prompt group to control for output length and prompt difficulty.CED averages story-level densities; lower CED and GRR indicate better performance.
- Model results: GPT-5-REASONING achieves the best reported performance, with CED 0.113 and GRR 2.80; GLM-4.6 and QWEN3-32B approach proprietary-level consistency.GEMINI-2.5-PRO follows with CED 0.305, while CLAUDE-SONNET-4.5 reports CED 0.520 and GRR 4.54.
- Length effects: Errors increase approximately linearly with output length, while models differ substantially in their preferred output lengths.DEEPSEEK-V3.2-EXP shows a stronger length-error correlation (r=0.973) than CLAUDE-SONNET-4.5 (r=0.478).
- Uncertainty: Error-bearing content has higher entropy than the whole-text baseline, increasing by 19.24% for QWEN3-4B-INSTRUCT-2507 and 12.03% for QWEN3-30B-A3B-INSTRUCT-2507.The evaluation interprets local entropy as an uncertainty signal associated with incorrect choices.
- Error structure and position: Factual and detail errors form a central hub with characterization, world-building, and timeline errors, while contradiction positions predominantly cluster in the 40–60% narrative range.Narrative and style errors show near-zero correlations with the other categories; geographical and temporal contradictions have the largest positional gaps.
4 Related Work
Prior long-form benchmarks emphasize context understanding, fluency, and holistic quality, but narrative consistency and fine-grained contradiction detection remain insufficiently evaluated.
- Existing story-generation evaluations primarily assess plot coherence, fluency, and holistic quality rather than systematic consistency errors.
- Long-form benchmarks span context lengths up to 2M tokens, while other benchmarks emphasize generation quality and models struggle at 16K–32K tokens.Perplexity and classical metrics correlate weakly with human judgments.
- Recent evaluation methods add checklists, dynamic criteria, and proxy-based assessment, but fixed templates limit fine-grained error detection.
5 Conclusion
The paper introduces ConStory-Bench and ConStory-Checker to assess narrative consistency in long-form story generation. Experiments show systematic errors, especially in factual tracking and temporal reasoning, clustering in predictable narrative regions.
- ConStory-Bench and ConStory-Checker assess narrative consistency in long-form story generation.
- Current LLMs produce systematic consistency errors, especially in factual tracking and temporal reasoning.
- Consistency errors cluster in predictable narrative regions rather than occurring randomly.
- The project plans an interactive portal for discovering and submitting new consistency errors and checking techniques.
6 Limitations
The benchmark is scoped to English fiction using Western narrative conventions and binary consistency judgments, while excluding other genres and culturally distinct storytelling contexts.
- The benchmark focuses on English fiction following Western narrative conventions.Its applicability to other cultural or linguistic backgrounds has not been evaluated.
- The binary judgment scheme does not distinguish intentional ambiguity, such as surprise endings or strategically delayed information, from true errors.
- The study focuses on fiction and storytelling rather than technical documentation, academic writing, or screenplays.Those domains have their own consistency conventions.
- Future work includes multilingual and cross-cultural benchmarks, recognition of intentional ambiguity, and evaluation of additional long-form genres.
- The appendix provides implementation details for task design, the evaluation pipeline, and additional experimental configurations.
- The four task types are generation, continuation, expansion, and completion, covering free-form creation, context-preserving extension, outline elaboration, and interpolation between fixed beginnings and endings.Continuation explicitly preserves established facts, timelines, and character states.
A.2 ConStory-Checker: Detailed Implementation
The detailed implementation specifies ConStory-Checker’s category structure, diagnostic validation procedure, prompt formats, and task-level analyses. It evaluates automated detection against human judgment and reports performance across five consistency dimensions.
- Pipeline and validation: ConStory-Checker evaluates five dimensions: Timeline & Plot Logic, Characterization, World-building & Setting, Factual & Detail Consistency, and Narrative & Style.Structured extraction guidelines use standardized JSON fields for cross-document comparison and aggregation.
- Pipeline and validation: The diagnostic dataset contains 200 stories with 1,000 deliberately injected errors across all five error dimensions, annotated by two professional web novel writers.
- Pipeline and validation: ConStory-Checker achieves Overall F1=0.678 versus Overall F1=0.281 for human expert judgment.Its precision is 0.884 and recall is 0.550; human recall ranges from 4.5% to 31.5%.
- Pipeline and validation: The checker detects 550 of 1,000 injected errors compared with 171 detected by human experts, a 3.2× improvement in error discovery rate.
- Prompt and task design: The benchmark supports four task types: generation, continuation, expansion, and completion, with representative prompts shown in Figure 7.
- Error-category prompts: The checker’s category prompts target calendar conflicts, memory contradictions, world-rule violations, appearance mismatches, and perspective confusions.Examples include conflicting dates, forgotten relationships, impossible actions, changing physical descriptions, and unjustified viewpoint shifts.
- Task-level analysis: Generation tasks consistently yield higher CED than other task types across most models, indicating that open-ended creation without prior context is the greatest consistency challenge.
- Task-level analysis: Proprietary models favor longer outputs, while GPT-4O-1120 and NVIDIA-LLAMA-3.1-ULTRA mostly produce texts below 3k words and some open-source models show balanced length distributions.GPT-5-REASONING and CLAUDE-SONNET-4.5 place 66.4% and 66.8% of outputs in the 8k+ category.
B.3 Token-Level Uncertainty Metrics
The analysis compares error-bearing segments with whole-text baselines across entropy, token probability, and perplexity, finding consistently higher uncertainty and lower confidence in error content.
- Error-bearing segments show higher entropy and perplexity but lower token probability than the whole-text baseline.These three metrics jointly indicate greater model uncertainty in segments containing consistency errors.
- Token-level uncertainty provides a reliable early-warning signal for potential narrative inconsistencies during generation.
B.4 Model-Specific Error Correlations
Error correlations vary by model family: proprietary systems show weak cross-category dependencies, while several other models exhibit stronger links between specific error types.
- GPT-5-REASONING and GEMINI-2.5-PRO have sparse correlation matrices with weak cross-category dependencies.
- CLAUDE-SONNET-4.5 shows Fact.–World correlation r=0.387 and Narr.–Fact. correlation r=0.429.
- GLM-4.6 and KIMI-K2-2509 show the strongest Char.–Fact. correlations, with r=0.533 and r=0.556, respectively.
B.5 Extended Positional Analysis
Per-model positional statistics confirm that facts are generally established early to mid-story, while contradictions tend to appear later, creating a measurable positional gap.
- Fact positions cluster at 15–30% of the narrative, while contradiction positions extend toward 40–60%.
- Figure 17 presents model-specific error correlation matrices with darker colors indicating stronger positive correlations.
- Table 10 reports normalized positions from 0–100% and gaps between fact establishment and contradiction appearance across seven error subtypes and eight models.
C Explanation of Metrics
CED measures absolute consistency-error density, whereas GRR supplies relative model rankings that also account for narrative completeness when error densities are identical.
- CED normalizes total consistency errors by total words per 10K words.
- CED assigns identical scores of 0.00 to two error-free stories despite one containing 8,000 words and the other only 800.This ten-fold length difference shows that CED alone cannot capture narrative completeness.
- GRR ranks models within each story using a quality score, then averages ranks across stories, with lower values indicating better performance.
- CED reports absolute error density, while GRR provides relative rankings that account for both consistency and completeness.