PaperlayerView exact revision on arXiv

Source-linked AI summary

When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulations

Yuqi Chen, Sixuan Li, Yunfeng Cai, Xueai Li, Ka Man Yan, Ying Li

arXiv:2608.15654v1cs.CLcs.AI

TL;DR

Open-ended storytelling requires models to sustain generation while preserving canon and developing changing trajectories, yet existing evaluations do not separate these capacities. WSE-bench evaluates them as distinct process dimensions across agent architectures and models, finding that scale chiefly extends generation while added structure shifts trade-offs rather than improving all outcomes.

  • Problem

    Existing narrative evaluations do not reveal when evolving trajectories break canon, stall development, or trade one storytelling capacity against another.

  • Method

    WSE-bench evaluates evolving stories as trajectories using Generation Coverage, Consistency, and Richness across agent architectures and model choices.

  • Results

    Model scale chiefly extends generation duration, while additional architecture stages shift trade-offs among sustained generation, coherence, and development rather than improving all three.

  • Takeaways & Limitations

    Sustained generation, canonical coherence, and meaningful development are distinct capacities, so evaluating evolving trajectories requires keeping these outcomes separate.

  • Takeaways & Limitations

    The source-family comparison does not support the expectation that divergence from a familiar plot necessarily destabilizes canon.

Abstract

from arXiv · show

Large language models can write fluent stories, but open-ended storytelling requires more than local fluency. In evolving world simulations and AI-native games, models must preserve facts, relationships, causal dependencies, and character states as the world changes. We introduce WSE-bench, a process benchmark that separately evaluates sustained generation, canonical coherence, and meaningful development in dynamic LLM storytelling. Generation Coverage records the proportion of planned narrative steps produced; Consistency tracks when canon breaks; and Richness measures how meaningfully branching, player-shaped trajectories develop. Across frontier models, Consistency and Richness do not form a smooth trade-off: their empirical Pareto frontier is non-concave, with several non-dominated intermediate configurations that no positive linear weighting can select. Added structure can enrich trajectories, but it does not uniformly improve coherence and may shorten them. Model scale chiefly improves sustained generation, without producing reliable gains in canonical coherence or meaningful development. These results show that sustained generation, canonical coherence, and meaningful development are distinct and sometimes competing capacities. WSE-bench makes those dynamics visible by extending narrative evaluation from finished stories to the processes that create them.

1 Introduction

Open-ended storytelling requires sustained generation, canonical coherence, and meaningful development as each action changes the evolving story state. WSE-bench evaluates these processes under controlled conditions to compare architectures, models, and world origins.

  • Motivation: Open-ended stories can fail through stopped generation, broken canon, or stalled development as actions alter the state underlying later scenes.Promises, injuries, and transferred knowledge may become binding evidence many steps later.
  • Motivation: Completed-story evaluations do not reveal when evolving trajectories break canon, stall development, or trade one capability against another.Long-horizon evaluation is needed to expose compounding errors hidden by short evaluations.
  • Benchmark: WSE-bench operationalizes sustained generation, canonical coherence, and meaningful development as Generation Coverage, Consistency, and Richness.Generation Coverage records the proportion of planned narrative steps successfully produced.
  • Benchmark: The benchmark standardizes initial worlds, model-independent openings, and fixed player policies, while evaluators identify contradictions and meaningful developments.Shared episode setup and choice-selection schedules make architectural differences easier to interpret.
  • Study design: The study compares agent architectures and model choices across the three dimensions, including whether dense-model scale produces consistent trends.Within-episode comparisons estimate architecture effects, while cross-model analyses characterize the evaluated models.
  • Study design: It also tests whether source-derived worlds are more vulnerable to contradiction when trajectories diverge from plots potentially encountered during pretraining, termed prior-canon interference.Simulation-native settings are compared with worlds adapted from published plays.

2 Related Work

Prior work studies planning and memory, narrative evaluation, simulated story worlds, and long-horizon agent behavior, but often assesses bundled systems or conflates key sources of variation. WSE-bench isolates character-decision structure, WMOD, and WMEM in paired open-ended episodes while evaluating canonical coherence, meaningful development, and sustained generation.

  • Planning and long-form story generation: Planning-based generators separate storyline construction from surface realization, while newer systems add dynamic outlines, temporal memory, conflict analysis, and temporally qualified world facts.These systems motivate planning and memory as plausible interventions for long-form story generation.
  • Narrative evaluation and LLM judges: Narrative benchmarks assess human-aligned metrics, discourse coherence, book-length criteria, and consistency bugs, while rubric-based LLM evaluation remains difficult for creative-writing preference.The related work spans both metric validation and judge-based evaluation.
  • Role-playing and simulated story worlds: Role-playing systems investigate memory, planning, multi-agent interaction, and character trajectories, extending toward persistent literary societies and co-evolving character–world states.These studies commonly evaluate bundled systems rather than independently isolating their components.
  • Benchmark positioning: WSE-bench holds the world, choice-selection policy, narrator role, output format, and tested model fixed within paired episodes while independently varying character-decision structure, WMOD, and WMEM.It evaluates inference-time behavior through canonical coherence, meaningful development, and sustained generation over long-horizon, open-ended trajectories.

3 Benchmark Framework

WSE-bench evaluates evolving stories as trajectories from standardized initial worlds through recurrent generation, comparing agent architectures and measuring sustained generation, canonical coherence, and meaningful development separately. Its criteria distinguish contradiction from narrated change, censor unobserved futures, and keep Consistency independent from Richness.

  • Benchmark construction: WSE-bench standardizes episodes around an InitialWorld and compares B1, B2, and B3 architectures across recurrent story-generation trajectories.B1 assigns decisions to the Story Agent, B2 uses one Character Agent for all non-player characters, and B3 uses one isolated Character Agent per non-player character.
  • Benchmark construction: Each trajectory begins from a connected cast, focal player, relationships, locations, and language, following a shared 12-condition choice schedule.The collection spans Chinese, English, Dano-Norwegian, Russian, and Swedish instances.
  • Evaluation metrics: Generation Coverage measures the mean percentage of planned narrative steps observed, quantifying sustained generation through the planned horizon rather than narrative quality.Trajectories with no valid narrative have Li = 0, and continuations after generation failures or valid story endings remain unobserved.
  • Evaluation metrics: Consistency marks contradiction only when an established fact conflicts with a current claim about the same entity, reference, and time without reconciliation.Narrated change, missing information, and unfulfilled player intents are not inconsistencies; failed-generation futures are right-censored and evaluated with Kaplan–Meier estimation.
  • Evaluation metrics: Richness rewards text-supported consequential development across events, character or relationship changes, and world expansion, while remaining independent from Consistency.Contradicting development contributes to Richness but is penalized by Consistency, whereas static coherent trajectories can receive low Richness.

4 Agent Architectures

The benchmark evaluates three character-action architectures under four module configurations, combining shared or distributed character agents with optional planning and memory modules. WMOD structures proposed actions into dependency-linked plot beats, while WMEM extracts persistent state changes after narration.

  • Module configurations: Each base has four configurations: neither module, memory only, planning only, or both, with w and p indicating WMEM and WMOD enablement.For every tested model, the same model and operating mode are used for each enabled component.
  • Base architectures: Three base architectures vary character-action topology: B1 uses only a shared Story Agent, B2 centralizes non-player character proposals, and B3 assigns each non-player character a separate agent.These correspond to single-narrator, centralized coordination, and decentralized character-agent designs.
  • World modeling: WMOD inserts a stateless World Model before narration to adjudicate player attempts and character proposals, returning a PlotPlan of at most 12 dependency-linked beats.The Story Agent receives this plan instead of the raw proposals.
  • World memory: WMEM runs after narration, comparing the realized narrative with prior structured state and emitting a MemoryDelta of persistent changes for rule-based state reduction.The supplied passage specifies updates to character conditions, relationships, and facts.

5 Evaluation and Validation

The evaluation uses blinded, rubric-based judgments validated against human annotations, and spans 6,048 trajectories across 12 models, 42 episode seeds, and 12 architectures. Analyses compare architectures over shared seeds while distinguishing observational Consistency from adjusted Consistency for Pareto analysis.

  • Evaluation protocol: Evaluators receive only the InitialWorld, opening, and generated trajectory, withholding model, architecture, internal-generation, and source-identifying information.Appendix B details the evaluation inputs.
  • Evaluation protocol: Consistency Judges require a prior fact, conflicting claim, mutual exclusion, and missing transition to mark contradiction, while retaining uncertain verdicts through aggregation.A binary first-contradiction verdict requires a valid majority of consistent or inconsistent votes.
  • Human validation: 85.7% accuracy and 82.0% balanced accuracy validate Consistency against human labels on 120 trajectories; Richness reaches MAE 7.63 and Pearson r = .861 on 36 trajectories.Consistency accuracy has a 95% CI of 81.3–90.1%, while Richness MAE has a 95% CI of 6.26–8.95.
  • Evaluation design: 6,048 trajectories across 12 models, 42 shared episode seeds, and 12 architectures form 144 cells, including dense Qwen checkpoints and six Frontier models.Each model–architecture cell contains all 42 seeds, with one model filling every enabled LLM role.
  • Analysis design: Architecture contrasts pair cells over shared seeds, whereas Pareto analysis uses Cadj instead of Cobs to avoid extending survival beyond observational support.Intervals hierarchically bootstrap source clusters and EpisodeSeed records, and source-family contrasts weight models equally.

6 Experiments and Results

Experiments show that sustained generation, canonical coherence, and meaningful development vary independently across models and architectures. Richer trajectories can involve coherence trade-offs, while model scale mainly improves sustained generation and longer horizons reveal otherwise hidden failures.

  • Consistency–Richness frontier: Six model–architecture cells are non-dominated, but only two lie on the upper concave envelope, so four observed compromises evade every positive linear weighting.DeepSeek-V4-Pro B3+WMOD and Gemini-3.1-Pro B3+WMEM+WMOD are supported configurations; every Grok-4.3 condition is dominated.
  • Architecture effects: Among Frontier models, B2 and B3 raise Richness by 9.2 and 14.6 points relative to B1 while reducing Cobs by 3.2 and 2.7 points.B3’s independent proposals may increase development and coordination burden, potentially making multi-agent collaboration counterproductive under excess coordination demands.
  • Architecture effects: WMOD raises Frontier Richness by 3.4 points but lowers Qwen Richness by 5.9 points, while reducing Generation Coverage in both groups.For Frontier models, WMOD changes Richness by +0.4 [−1.3, +2.1] at B1, +4.3 [+2.6, +6.0] at B2, and +5.3 [+4.0, +6.6] at B3; WMEM provides no stable Consistency benefit.
  • Outcome profiles: No Frontier model leads all outcomes: GPT-5.6-sol has Cobs = 99.0 and G = 97.6, while DeepSeek-V4-Pro has R = 68.0.Grok-4.3 nearly matches GPT-5.6-sol in Generation Coverage (G = 97.2) but has lower Richness (R = 40.8).
  • Model scale: Across five dense Qwen2.5 checkpoints, Generation Coverage rises by 19.32 percentage points per parameter doubling, but Consistency and Richness do not improve monotonically.Within the tested range, parameter count strongly predicts sustained generation but not canonical coherence or meaningful development.
  • Long-horizon failures: A 10-step evaluation misses 36.9% of Qwen and 56.1% of Frontier contradictions eventually observed by step 20, while long-horizon failures also include technical truncations.Temporal conflicts are most common; failures in Character Agents, WMOD, or the Story Agent prevent production, whereas WMEM failures can truncate a valid subsequent trajectory.

7 Conclusion · Appendix

The conclusion argues that sustained generation, canonical coherence, and meaningful development are distinct capacities that must be evaluated as evolving trajectories. It further shows that added architecture and model scale shift trade-offs or extend generation without reliably improving every outcome.

  • 7 Conclusion: Evolving-trajectory evaluation separates sustained generation, canonical coherence, and meaningful development as distinct storytelling capacities.A story may remain coherent through limited change, become eventful while losing canon, or end before either quality is tested over a long horizon.
  • 7 Conclusion: The central evaluation question is whether a model can sustain a world through consequential change, not merely produce a convincing continuation.
  • 7 Conclusion: Greater scale chiefly extends generation duration, while additional stages shift trade-offs among outcomes rather than improving all three.Architectural components should justify their complexity by providing usable information, coordination, or enforcement.
  • 7 Conclusion: Source-family comparisons do not support the simple expectation that divergence from a familiar plot necessarily destabilizes canon.
  • 7 Conclusion: Process evaluation must preserve both time and failure type because contradictions can emerge after canon accumulates, while component failures can terminate generation without contradictory text.
  • 7 Conclusion: The non-concave Frontier-model Pareto set indicates specialized Consistency- and Richness-favoring regimes rather than a smooth route between them.

A Benchmark Construction and Architecture … A.3 WMEM with Full-History Access

WSE-bench standardizes nine evolving worlds into controlled episode seeds and compares architectures under shared choice sequences. Its architecture separates narration, planning, character action, and structured memory while preserving complete-history access and narrative-defined canon.

  • A.1 Episode Construction and Cross-Architecture Comparison: Nine worlds begin from shared snapshots, then player choices create distinct timelines for evaluation.The benchmark includes two simulation-native settings and seven DraCor openings with self-contained five-character casts.
  • A.1 Episode Construction and Cross-Architecture Comparison: 42 EpisodeSeed records are constructed by rotating each character into the player role and balancing source family, focal-player representation, and relationship structure.There are 14 EpisodeSeed records per source family; native-language openings and three initial actions are model-independent.
  • A.1 Episode Construction and Cross-Architecture Comparison: All 12 conditions share the same uniformly sampled option-position sequence and identical first-step option text, while later trajectories and options may diverge by architecture.The comparison measures an architecture’s total effect as the story evolves, rather than an isolated module effect.
  • A.2 WMOD as Pre-Narration Adjudication: At each step, WMOD receives player attempts, applicable character intents, canonical history, and optional structured state, returning a PlotPlan with up to 12 dependent beats.At least one beat traces to the player attempt; background intents may be merged, delayed, prevented, or omitted.
  • A.3 WMEM with Full-History Access: Structured world memory records character conditions, relationships, persistent facts, and supporting characters after each narrative step.A rule-based updater applies explicit model-proposed changes without adding inferred facts.
  • A.3 WMEM with Full-History Access: The reducer applies memory operations atomically, projects global state deterministically to each character, and cannot rewrite generated narrative.Evaluators independently reconstruct canon from the narrative trajectory, so structured memory is not ground truth.
  • A.3 WMEM with Full-History Access: Complete narrative history remains active in every evaluated condition, while the experiment varies only the additional structured current-state memory path.The contrast measures structured-state information alongside full history; retrieval, provenance verification, and conflict enforcement are outside the implemented WMEM path.
  • A.3 WMEM with Full-History Access: The Story Agent alone establishes canon as narrator, while Character Agents and the World Model remain stateless and proposed actions or unexecuted plans establish no facts.B2 uses one Character Agent call for all four non-player characters, whereas B3 uses four isolated calls, one per character.

B Evaluation and Validation … B.6 Long-Horizon Sensitivity

The evaluation framework separates canonical consistency, sustained generation, and meaningful development, using blinded human validation and long-horizon analyses to expose their distinct behavior. Results show a non-concave empirical frontier, strong aggregate Judge agreement with humans, and substantial late-emerging contradictions.

  • B.1 Evaluator Blinding: Evaluators saw independently prepared, identity-blinded views containing only the initial world, opening, language, and generated trajectory.Model, architecture, source-family, internal-output, memory, failure, and cost information was excluded.
  • B.2 Consistency Under Unequal Coverage: Consistency estimates use first-contradiction events with right-censoring for contradiction-free trajectories, while missing continuation is not itself a contradiction.Cobs carries forward the last estimable survival value, whereas Cadj assigns zero after support is exhausted; both produce the same six-cell Pareto set across 144 cells.
  • B.3 Supportedness of the Empirical Pareto Set: Only two endpoints are supported: DeepSeek-V4-Pro B3+WMOD and Gemini-3.1-Pro B3+WMEM+WMOD.The four intermediate non-dominated cells lie 2.84, 8.26, 8.15, and 0.05 Richness points below the chord joining these endpoints.
  • B Evaluation and Validation: Across the evaluation and validation results, consistency and Richness remain distinct measures rather than a smooth trade-off, with Judge aggregates supported by blinded human-reference studies.The framework evaluates generated trajectories as evolving processes, preserving separate treatment of contradiction events, observed coverage, and development.
  • B.4 Richness Annotation: Richness quantifies non-repetitive development per observed narrative step across character, relationship, world, and social expansion dimensions.Its computation uses d95 = 3.5 as the 95th-percentile calibration value and clips normalized scores to [0, 1].
  • B.5 Human Validation of Judges: The three-Judge median most closely matched human Richness references, with 97.8% of weighted ordinal comparisons within one point and mean quadratic-weighted κ = .780.The Richness validation used 36 stratified cases, while the Consistency validation used 120 human-labeled cases from 2,447 trajectories.
  • B.6 Long-Horizon Sensitivity: 63.1% of Qwen-series and 43.9% of Frontier first contradictions observed by step 20 had appeared by step 10.A ten-step evaluation would miss 36.9% and 56.1%, respectively, of the first contradictions eventually observed; Richness ratings were not reused at earlier horizons.

C Architecture Contrasts and Robustness … D Model and Scale Effects

Architecture changes affect both sustained generation and narrative quality, while module effects depend on the underlying base and do not yield a universal consistency gain. Models show distinct outcome profiles, and scale comparisons are restricted to comparable dense Qwen2.5 checkpoints.

  • C Architecture Contrasts and Robustness: 1,021 World-Memory failures follow valid narratives and prevent 12,672 later steps, showing that added handoffs can reduce sustained generation without contradicting shown text.Across both model groups, 1,709 generation failures occur before the current narrative is produced.
  • C Architecture Contrasts and Robustness: Temporal conflicts are the most common Judge-agreement category, while identity and relationship conflicts usually contradict InitialWorld commitments and state or world-fact conflicts more often contradict earlier narratives.Strong Character-Agent precursors occur in 89 inconsistent trajectories, including 42 persisting through the World Model; 27 more have only a World-Model precursor.
  • C.1 Per-Model Architecture Effects: Per-model bootstrap intervals capture seed variation among studied models but not decoding or model-population uncertainty, and they are not adjusted for multiple comparisons.These intervals use 10,000 hierarchical source-cluster/EpisodeSeed bootstrap draws.
  • C.2 Base-Conditional Module Effects: No base shows a stable WMEM Consistency gain, while WMOD’s Frontier Richness change depends strongly on topology: +0.4 at B1, +4.3 at B2, and +5.3 at B3.The reported intervals are +0.4 [−1.3, +2.1], +4.3 [+2.6, +6.0], and +5.3 [+4.0, +6.6], respectively.
  • C.3 Full-Horizon Paired Comparison: Full-horizon paired comparisons preserve the character-agent trade-off and absence of a joint WMEM gain, so the main pattern is not solely a short-prefix artifact.The sensitivity retains architecture pairs only when both conditions reach step 20 with valid scores.
  • D Model and Scale Effects: Models have distinct profiles across Generation Coverage, Consistency, and Richness, with no common ranking across the three outcomes.Model-level variation is analyzed separately from scaling, which is assessed only among comparable dense Qwen2.5 checkpoints.

D.1 Qwen-Series Scaling

Across five dense Qwen2.5 checkpoints, scaling chiefly predicts Generation Coverage, while Consistency and Richness vary non-monotonically with model size. The Generation Coverage slope is robust to excluding the anomalous 14B checkpoint, whereas the analysis excludes Qwen3.5-397B-A17B because its sparse MoE parameter count is not directly comparable.

  • Scope: Qwen3.5-397B-A17B is excluded because sparse-MoE total parameter counts cannot be compared directly with dense models.The fitted scaling analysis therefore concerns the five dense Qwen2.5 checkpoints.
  • Scaling results: Generation Coverage increases by 19.32 points per parameter doubling across the five dense Qwen2.5 checkpoints.Excluding the 14B checkpoint changes the estimate only slightly, to 19.17 points per parameter doubling.
  • Scaling results: Consistency and Richness vary non-monotonically across the five Qwen2.5 model sizes.Their changes do not follow a simple monotonic scaling pattern.

E Source-Family Inference

Source-family consistency contrasts are sensitive to the comparison and leave-one-play-out choice, while similar contradiction timing suggests relative-day representation does not explain the family pattern.

  • Robustness of source-family contrasts: Both primary Cobs DraCor contrasts include zero, but all seven DraCor–Smallville contrasts remain positive under leave-one-play-out analysis.Six of seven DraCor–ITP contrasts remain positive; omitting one play yields −0.27 points.
  • Evaluation basis: Canonical consistency is evaluated against the supplied InitialWorld and emerging trajectory rather than a published continuation.Prior knowledge becomes observable only when it creates an internal contradiction.
  • Trajectory timing: Story advances to a new day are rare and least common in DraCor, while mean first-contradiction steps are similar across source families.The similar timing gives no indication that relative-day representation drives the family pattern.
  • Estimation design: Table 7 estimates source-family contrasts with equal-model means and 95% hierarchical bootstrap intervals over source clusters and EpisodeSeed records while retaining models.The paired-architecture analysis complements the primary analysis, which includes each trajectory up to its observed length.
Loading 2608.15654v1…