PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Long-Form Story Generation via Narrative State Tracking

Zhennan Wan, Jianfei Chen

arXiv:2609.35759v1cs.CL

TL;DR

Long-form story generation must preserve narrative consistency as stories grow, while existing evaluation does not readily support comparisons beyond roughly 10K words. NSTAGENT tracks a structured narrative state and is evaluated across 10K–100K words; the reported results show stable quality and consistency, with growing advantages over free-text memory.

  • Problem

    Existing consistency evaluation targets stories up to about 10K words, and error counts do not scale proportionally on much longer stories, limiting cross-length comparisons.

    Source: §1 · pp. 1–2

  • Method

    NSTAGENT is a training-free chapter-by-chapter framework that tracks characters, past events, and future requirements in an explicit narrative state.

    Source: §1 · p. 2

  • Results

    Across 10K–100K words and two backbones, NSTAGENT keeps writing quality and narrative consistency stable and improves over free-text memory by a margin that grows with length.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results suggest NSTAGENT provides an effective approach to scaling story generation toward full-length novels.

    Source: §6 · p. 9

    LLM judges may carry stylistic biases, and the judge shares a model family with one backbone, which may favor its outputs.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

LLMs have demonstrated strong capabilities in creative writing. However, scaling them to full-length novels remains challenging, as maintaining narrative consistency becomes increasingly difficult. Existing story-generation methods typically focus on stories of up to about ten thousand words, leaving their ability to scale to full-length novels underexplored. In this work, we introduce Narrative State Tracking Agent (NstAgent), a training-free agentic framework that allows LLMs to track a structured narrative state including characters, past events and future requirements. We extend an existing benchmark to compare narrative consistency across lengths, and use it together with a writing-quality benchmark to systematically evaluate stories ranging from 10K to 100K words. We show that NstAgent achieves better narrative consistency and writing quality as stories grow longer, and neither of them degrades noticeably as length increases, suggesting that it provides an effective approach to scaling story generation toward full-length novels.

1 INTRODUCTION

Long-form story generation faces a consistency burden that grows as stories accumulate characters, events, and foreshadowings. NSTAGENT tracks a structured narrative state, and the paper evaluates it alongside a length-comparable consistency benchmark and a writing-quality benchmark.

1 INTRODUCTION

  • As stories lengthen, accumulated characters, events, and foreshadowings must remain mutually compatible, making consistency a bottleneck for long-form writing.The paper describes this consistency burden as growing superlinearly with length.Source: §1 · p. 1
  • The length-scaling problem refers to extending story output length.Source: §1 · p. 2
  • NSTAGENT is a training-free framework that tracks characters, past events, and future requirements to help preserve narrative consistency.It updates the narrative state after each chapter.Source: §1 · p. 2
  • The paper extends a consistency benchmark to stories up to 100K words and makes comparisons across lengths possible.It also uses WritingBench to evaluate writing quality.Source: §1 · p. 2
  • Across 10K–100K words, NSTAGENT improves narrative consistency and writing quality, with neither degrading noticeably as stories grow longer.The evaluation covers multiple LLMs.Source: §1 · p. 2

2 RELATED WORK

The related work covers longer-output training, hierarchical generation, memory-based methods, and story evaluation. NSTAGENT is positioned as a training-free approach with prospective obligations in its state and evaluation across lengths.

Training longer-writing models

  • Long-output training extends what models can write, but novel-length supervision is scarce, long outputs remain volatile, and NSTAGENT is complementary and requires no training.LongWriter and Self-Lengthen fine-tune on synthesized outputs; LongWriter-Zero uses reinforcement learning.Source: §2 · p. 2

Hierarchical generation

  • Hierarchical methods plan and expand outlines, but a fixed outline cannot make details introduced during chapter writing constrain later chapters.Later work controls pacing, suspense, and actions, or distributes planning, writing, and critique across agents.Source: §2 · p. 3

Memory-based generation

  • Memory-based methods retrieve, compress, or refine earlier text, or maintain knowledge graphs; NSTAGENT tracks prospective obligations and supports writer-directed retrieval and consistency evaluation from 10K to 100K words.Its typed tool updates state, while the writer can search fragments or read source chapters on demand.Source: §2 · p. 3

Evaluation

  • Story evaluation has shifted toward LLM judges, while ConStory-Bench reports contradictions across five categories and nineteen subtypes; NSTAGENT extends it for longer, cross-length evaluation.The related work also describes writing-quality and book-length narrative benchmarks.Source: §2 · p. 3

3 SCALING LONG-FORM STORY GENERATION

The section frames length scaling as a joint problem of story quality, target-length reliability, and cost, then examines existing approaches and adapts consistency evaluation for longer stories.

3.1 PROBLEM FORMULATION

  • The objective is to improve narrative consistency and writing quality while meeting story-length and approximately linear-cost constraints.The formulation tests whether one workflow can meet these requirements as target length grows without changing the model.Source: §3.1 · p. 3

3.2 SCALING CHALLENGES OF EXISTING METHODS

  • The comparison examines generation-step cost, whether later chapters retain needed information, and whether each pipeline reaches the target length.Table 1 characterizes memory by rewrite versus incremental updates and passive versus active access.Source: §3.2 · p. 4, Table 1 · p. 4
  • At 10K words, only 21% of DeepSeek-V4-Flash and 74% of GPT-5.6 Luna first attempts fall within the ±20% acceptance band for one-pass generation.Accepted stories cost 1.9 and 1.3 calls on average, respectively.Source: §3.2 · p. 4
  • Knowledge-graph memory has history-dependent matching, query, and triple-call costs, and DOME issues thousands of memory calls per 10K-word story.DOME is evaluated only at 10K because its released implementation fixes stories to five acts.Source: §3.2 · p. 4
  • Rolling summaries add roughly one call per chapter but can discard details or silently change facts; structured state instead uses keyed entries, persistent obligations, and source lookup.Character snapshots are overwritten rather than paraphrased.Source: §3.2 · p. 4

3.3 LENGTH-COMPARABLE CONSISTENCY EVALUATION

  • Consistency error density (CED) measures reported errors per 10K words averaged over stories, with lower values better; the paper reports subtype and instance variants.Subtype CED counts error subtypes, whereas Instance CED counts reported contradictions.Source: §3.3 · p. 4
  • Full-story CED can fall at longer lengths because judges report contradictions more slowly than story length grows, undercounting errors in very long stories.The earlier agent version’s full-story CED dropped roughly 40% from 20K to 50K words and again from 50K to 100K.Source: §3.3 · p. 5
  • The revised protocol evaluates contradictions in a terminal window of roughly 10K words while checking each against the preceding narrative.The window spans the final 10K words across 10K-, 20K-, 50K-, and 100K-word stories.Source: §3.3 · p. 5

4 NSTAGENT

NSTAGENT plans a frozen outline, writes chapters sequentially, and maintains state between chapters. Its tools support chapter retrieval and correction, while a structured update tracks changing characters, events, and unresolved requirements.

4.1 OVERVIEW

  • NSTAGENT first plans a premise and chapter outline, then writes chapters sequentially from the frozen outline and current narrative state.The outline assigns each chapter a title, description, and target word count.Source: §4.1 · p. 5
  • Earlier prose is excluded from each chapter prompt; typed state and on-demand reading or searching provide access to prior story information.The writer can also correct inconsistencies in current or earlier chapters.Source: §4.1 · pp. 5–6

4.2 STATE-GUIDED GENERATION

  • The state-guided workflow uses read, search, write, and correct tools; successful chapter writing is subject to a word-count acceptance gate.Correction replaces an exact span and is limited per chapter; read and search calls have fixed budgets.Source: §4.2 · p. 6

4.3 NARRATIVE STATE TRACKING

  • Narrative state contains character snapshots, completed events that may affect later plot, and unresolved future requirements such as clues or promised confrontations.Character snapshots include location, goals, relationships, knowledge, possessions, and physical or emotional condition.Source: §4.3 · p. 6
  • After an accepted chapter, one update call upserts character snapshots, adds past events and future requirements, and resolves requirements by key.Empty arrays indicate unchanged collections, and the transition is applied atomically.Source: §4.3 · p. 6
  • The update design uses field names to encode operations, overwrites complete character snapshots, excludes outline-stated events, and resolves requirements with stable keys.The prompt supplies field semantics, while the tool schema enforces structure.Source: §4.3 · p. 6
  • Character snapshots grow with the cast, fulfilled requirements are removed, and accumulated past events remain far shorter than the prose.At 100K words, the final state has about a dozen characters, fewer than ten open requirements, and past events that are a small fraction of the story; calls and generated tokens grow approximately linearly with novel length.Source: §4.3 · p. 7

5 EXPERIMENTS

Across 10K–100K words and two backbones, NSTAGENT improves on free-text memory as stories lengthen while maintaining writing quality and narrative consistency. The experiments compare methods, memory-channel ablations, and generation costs.

5.1 SETTINGS

  • The evaluation uses two model backbones, 100 prompts at 10K–50K, and 50 prompts at 100K; longer-length comparisons are between NSTAGENT and RollSum.Consistency is measured with extended ConStory-Bench and writing quality with WritingBench; stories are scored only when complete, length-compliant, and fully judged.Source: §5.1 · p. 7

5.2 RESULTS AND ANALYSIS

  • At 10K, NSTAGENT and RollSum are more consistent than Direct, DOME, and StoryWriter on both backbones; NSTAGENT achieves the highest writing quality.On GPT-5.6 Luna, RollSum and NSTAGENT are at parity in consistency, while NSTAGENT is most consistent by a clear margin on DeepSeek-V4-Flash.Source: §5.2 · pp. 7–8
  • From 20K to 100K, NSTAGENT is more consistent and writes better than RollSum on both backbones, with its advantage growing as stories lengthen.On GPT-5.6 Luna, its instance CED advantage is 1.27, 1.40, and 2.15 at 20K, 50K, and 100K, respectively.Source: §5.2 · p. 8, Table 3 · p. 8
  • Within NSTAGENT, narrative consistency and writing quality do not degrade markedly as stories grow; error density stays within a narrow range across lengths.Writing quality even rises with length on DeepSeek-V4-Flash; the authors note that part of the error-density drop at 100K may reflect judging.Source: §5.2 · p. 8
  • NSTAGENT’s cost grows approximately linearly with length, with a price per 10K words between USD 0.13 and 0.17.Under prefix caching, its cost is about the same as RollSum’s at each tested length, though NSTAGENT sends more input and RollSum’s cost is dominated by output.Source: §5.2 · pp. 8–9, Figure 3 · p. 9, Table 4 · p. 9
  • Removing either the narrative state or lookback tools makes stories less consistent and lowers writing quality; the full agent outperforms both ablations.Without state, writers search earlier chapters more yet still make more timeline and plot errors; without lookback, factual-detail and timeline errors rise further.Source: §5.2 · p. 9, Table 5 · p. 9

6 CONCLUSION AND FUTURE WORK

The paper concludes that NSTAGENT maintains narrative consistency and writing quality from 10K to 100K words, improving over free-text memory by a margin that grows with length. It identifies future work on richer state representations, revision, and fine-tuning.

6 CONCLUSION AND FUTURE WORK

  • From 10K to 100K words and on two backbones, NSTAGENT keeps writing quality and narrative consistency stable and improves over free-text memory by a margin that grows with length.The conclusion describes it as an effective approach to scaling story generation toward full-length novels.Source: §6 · p. 9
  • Future work can explore richer state representations, better revision mechanisms, and fine-tuning the backbone to track narrative state.The authors also identify exploring these directions as building on length-comparable evaluation and narrative state tracking.Source: §6 · p. 10
  • The authors report using generative AI to implement methods and baselines, translate parts of the manuscript, identify literature, and edit code and text.They state that they reviewed AI-assisted work and take responsibility for the final content.Source: AI USE STATEMENT · p. 10
  • Individual stories and scores cannot be reproduced exactly because the backbones and judges are commercial APIs that may change over time.The repository includes code, benchmarks, generated stories, judge outputs, and scripts that recompute tables and figures.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 EXPERIMENT DETAILS

The experiments standardize story generation and judging across lengths, while the prompt templates specify NSTAGENT’s chapter workflow and memory updates. A targeted injection study tests whether the consistency judge can still detect contradictions across long prefixes.

WritingBench criteria-generation prompt (verbatim)

(1) Scope marker inserted into the story before the first target chapter

NSTAGENT chapter prompt (user message)

  • NSTAGENT’s chapter prompt supplies the frozen outline and current state, then requires a write, any needed consistency correction, and one final structured state update before completion.The update includes character states, completed events, new future requirements, and resolved requirements.Source: NSTAGENT chapter prompt (user message) · p. 19

Planner, stage 1: premise

RollSum summarizer (system, then user)

WritingBench scoring prompt (verbatim, abridged)

B.1 JUDGE RECALL OVER LONG PREFIXES

  • The recall test injects 30 explicit contradictions per length at three prefix depths, placing each later statement inside the evaluation window and marking each pair for exact-match detection.Six consistent control pairs are also included at each length.Source: §B.1 · p. 22
  • The judge reports the targeted contradiction category for 119 of 120 injected pairs, while all 120 are detected under some category, including pairs separated by about 101K words.Every length-by-depth cell has 10/10 detection.Source: §B.1 · p. 22, Table 8 · p. 23

B.2 WINDOW-LOCAL AND GLOBAL ERRORS

The terminal-window measure includes one globally scoped subtype that can grow with the full story while its denominator remains the checked window. This subtype’s increasing share raises CED as target length increases, although excluding it leaves the method comparison intact.

B.2 WINDOW-LOCAL AND GLOBAL ERRORS

  • The abandoned plot elements subtype is checked across the full story but divided by the roughly 10K-word terminal window, inflating its contribution as targets lengthen.Its share rises from 5.1% to 11.8% on DeepSeek-V4-Flash and from 15.0% to 26.4% on GPT-5.6 Luna between 10K and 100K words.Source: §B.2 · p. 23, Table 9 · p. 23
  • Excluding the globally scoped subtype lowers every CED value but leaves the comparison between methods intact.The paper retains the released definition in its main tables and reports the local/global decomposition separately.Source: §B.2 · p. 23, Table 9 · p. 23

C.1 ERRORS BY TYPE

Across lengths beyond 10K, NSTAGENT has lower error density than RollSum in every category and setting, with gains concentrated in exact-fact recall. Its compact typed state illustrates the memory carried between chapters, while the comparison is bounded by which baselines define runs beyond 10K.

C.1 ERRORS BY TYPE

  • NSTAGENT has lower error density than RollSum in all five categories across all six settings beyond 10K, and in 13 to 17 of the 19 subtypes.The largest 100K DeepSeek-V4-Flash reductions include memory contradictions (2.09 to 1.05), appearance mismatches (1.32 to 0.65), and nomenclature confusions (0.75 to 0.19).Source: §C.1 · p. 23, Table 12 · p. 25, Table 13 · p. 25
  • At 10K, NSTAGENT is lowest in all five categories and 15 of 19 subtypes on DeepSeek-V4-Flash; on GPT-5.6 Luna it is lowest in factual detail and below three named baselines in four categories.On Luna, the aggregate is at parity with RollSum.Source: §C.1 · p. 23, Table 10 · p. 24, Table 11 · p. 24
  • Removing state mainly raises timeline and plot errors, while removing lookback raises timeline and factual-detail errors.The 20K ablation table reports NSTAGENT lowest in four of five categories and nine of 19 subtypes.Source: §C.1 · p. 23, Table 14 · p. 26

C.2 A FINAL STATE AT 100K WORDS

  • A 100K-word story’s example final state contains 17 character snapshots, 153 past events, and six unresolved requirements, totaling about 18.5K words.That state is about a sixth the length of the 110.7K-word story it describes and is carried in place of prior prose.Source: §C.2 · p. 26, Final narrative state of one 100K-word… · p. 26

C.3 BASELINES LIMITED TO 10K WORDS

  • Three baselines are evaluated only at 10K because their released designs do not define longer runs, so extending them would change the published methods.Direct’s single-call workflow faces the output-token limit, DOME’s five-act structure requires chapters too long for one call at 100K, and scaling StoryWriter requires changing its event schedule.Source: §C.3 · p. 27
  • RollSum is the length-agnostic baseline used for scaling comparisons; it is strongest at 10K and matches NSTAGENT on GPT-5.6 Luna.Its summary is rewritten after each chapter, making it the paper’s representative free-text memory method.Source: §C.3 · p. 27

C.4 STATISTICAL ANALYSIS

The statistical analysis pairs methods on the same prompts and frozen outlines, correcting comparisons within each metric. The reported length trend persists when all lengths are recomputed on the identical prompt subset used at 100K.

C.4 STATISTICAL ANALYSIS

  • Every comparison is paired on the same prompt and frozen outline, using paired t-tests, 95% bootstrap intervals, and Holm correction within each metric.Tables report means over per-story scores; differences below roughly 0.7 in 10K instance CED fall within noise at n = 100.Source: §C.4 · p. 27
  • On DeepSeek-V4-Flash at 10K, NSTAGENT improves instance CED by −2.78 [−3.65, −1.95] over RollSum, −2.93 [−3.99, −1.89] over Direct, and −5.40 [−6.69, −4.17] over StoryWriter.All three comparisons have p < 0.001 after correction.Source: §C.4 · p. 27
  • On GPT-5.6 Luna at 10K, the instance-CED gap over Direct and writing-quality gap over StoryWriter survive correction, while the difference from RollSum is flat.DOME is excluded from these tests because it used a different 20-prompt set.Source: §C.4 · p. 27
  • Recomputing every length on the same 50 prompts used at 100K preserves the reported pattern: the DeepSeek-V4-Flash gap is stable through 50K and widens at 100K, while the Luna gap grows from near zero.The paper concludes that prompt composition does not explain the trend.Source: §C.4 · p. 27, Table 15 · p. 28

D REINFORCEMENT LEARNING FOR NSTAGENT

At 10K words, reinforcement-learning checkpoints showed little change through step 32, while step 48 improved judged quality and reduced contradictions. Case studies illustrate how explicit state can preserve recurring events and identity facts, while also carrying errors forward.

D REINFORCEMENT LEARNING FOR NSTAGENT

  • Training uses 256 non-overlapping English premises across four target lengths; chapter-level GRPO samples prefixes and uses the deployed agent loop for rollouts.The rollout reward is zero unless writing, one state update, and DONE all succeed; otherwise judges score consistency, writing quality, and state-update fidelity.Source: D REINFORCEMENT LEARNING FOR NSTAGENT · p. 28
  • Three judges use separate rubrics for chapter consistency, writing quality, and state-update fidelity; the consistency rubric distinguishes explicit contradictions from other continuity defects.The state rubric penalizes fabricated facts, unjustified resolutions, lost persistent facts, and redundant copying of the outline.Source: D REINFORCEMENT LEARNING FOR NSTAGENT · pp. 28–29
  • Checkpoints at steps 16, 32, and 48 are evaluated on 20 10K-word prompts with shared outlines, using DeepSeek-V4-Pro and the writing-quality and extended consistency benchmarks.The prompts were also used to compare checkpoints.Source: D REINFORCEMENT LEARNING FOR NSTAGENT · p. 29
  • At step 48, judges rated chapters higher in quality and found clearly fewer contradictions, with instance-level error density about a fifth lower than the initial model’s.The authors report these results on one model, one seed, and 20 prompts, and make no claim about improvements at longer lengths.Source: D REINFORCEMENT LEARNING FOR NSTAGENT · p. 29

E CASE STUDIES

  • The three case studies compare NSTAGENT and RollSum writing from the same prompt and frozen outline, including examples where each method makes an error the other avoids.Contradictions were flagged by the extended ConStory-Bench judge and checked against the generated text.Source: E CASE STUDIES · p. 29

E.1 CASE 1: REPEATED EVENTS (DEEPSEEK-V4-FLASH, 20K)

  • NSTAGENT’s keyed past events preserve repeated prophecy-stone occurrences: the judge flags one contradiction in its last seven chapters, versus ten in RollSum’s last eight.All 16 requirements recorded by NSTAGENT were resolved by the final chapter; RollSum’s vow to wait “at every facet” was flagged as an abandoned plot element.Source: §E.1 · p. 30

E.2 CASE 2: IDENTITY FACTS OVER 40 CHAPTERS (GPT-5.6 LUNA, 100K)

  • Across the final four chapters, the judge flags 10 contradictions for NSTAGENT and 25 for RollSum, whose regenerated summaries carry birth-date drift into later chapters.NSTAGENT’s character snapshots repeatedly record stored years, including Daniel’s 1993 birth year across Chapters 0, 1, 7, 28, and 29.Source: §E.2 · pp. 30–31

E.3 CASE 3: AN ERROR FIXED IN THE STATE (GPT-5.6 LUNA, 10K)

The third case shows that explicit state can preserve an erroneous fact: a mistaken name enters the state and conditions later chapters. The authors identify missing verification against source chapters as a design gap.

E.3 CASE 3: AN ERROR FIXED IN THE STATE (GPT-5.6 LUNA, 10K)

  • NSTAGENT renames Bishop Creel as Bishop Armitage, then Bishop Haldane, and records the new name in state; RollSum keeps one name and age throughout.The judge flags NSTAGENT’s rename as both a memory contradiction and nomenclature confusion.Source: §E.3 · p. 31, Table 21 · p. 32
  • Because the writer creates state from its own chapter, an erroneous rename can propagate; no automatic check compares new entries with earlier text, and Luna rarely searches.The authors propose checking new state entries against source chapters as a future design, not part of the present system.Source: §E.3 · pp. 31–32

F LIMITATIONS

The evaluation has important scope and measurement limits, while the reported near-linear cost depends on prefix caching. The authors also leave bounding or consolidating the growing past-event log to future work.

F LIMITATIONS

  • Conclusions concern English prompts and benchmark metrics, not reader experience; the benefit also depends on story length and backbone, and short stories with a strong model can favorably match free-text summaries.CED does not capture failures such as stylistic flattening, repeated phrasing, or assembly from unrelated fragments; adapted baselines may not reflect all released design choices.Source: F LIMITATIONS · p. 32
  • The LLM consistency judge has imperfect precision: it flags a targeted contradiction in 25% of control passages without one and can count one error under multiple subtypes.The authors treat ConStory-Bench as an evidence-grounded diagnostic, not an exact error count; the judge also shares a model family with one backbone.Source: F LIMITATIONS · p. 32
  • At 100K words, a story uses 7.5M input tokens versus 0.71M at 10K, with about 60% cached; without caching, the 100K story would cost USD 2.28 instead of 1.32.Bounding or consolidating the accumulating past-event log is left for future work.Source: F LIMITATIONS · p. 32

Related papers

Loading 2609.35759v1…