PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives

Yingpeng Ma, Jianhao Yan, Bei Shi, Ka Hou Kam, Runnan Wang, Xuebo Liu, Yulong Chen, Yue Zhang, Derek F. Wong

arXiv:2608.08160v1cs.CLcs.AI

TL;DR

Long-horizon narrative agents must preserve established facts, commitments, and plot structure despite unconstrained user interventions. NCP-Bench evaluates this challenge and finds that GPT-5.2 survives only 42% of interactions after 20 turns, while fact conflicts occur in 40%–68% of interactions.

  • Problem

    Existing research has emphasized fluent, open-ended storytelling while providing limited evaluation of long-horizon logical and narrative consistency under unconstrained user interventions.

  • Method

    The paper formalizes Narrative Commitment Preservation and introduces NCP-Bench, using structured trajectories, commitments, and facts with adversarial player interventions for automatic evaluation.

  • Results

    GPT-5.2 achieves only a 42% survival rate after 20 turns, fact conflicts occur in 40%–68% of interactions, and only isolated runs satisfy all achievement commitments within 100 turns.

  • Takeaways & Limitations

    Fluent language generation alone is insufficient for reliable interactive narrative because current LLMs struggle to preserve logical world states over long horizons.

  • Takeaways & Limitations

    The benchmark emphasizes single-threaded linear narratives, and its LLM-based auditors may make mistakes on ambiguous text.

Abstract

from arXiv · show

The rapid advancement of Large Language Models (LLMs) is revolutionizing AI for Games by enabling open-ended and fluid interactive storytelling. However, existing research has largely overlooked the critical challenge of maintaining long-horizon logical consistency and narrative integrity against unconstrained user interventions. To address this, we formulate this challenge as Narrative Commitment Preservation (NCP), and take interactive narrative as our testbed. We introduce NCP-Bench, a benchmark of 100 narrative environments derived from movie synopses. Each environment includes a structured narrative specification (trajectory, commitments, and initial facts) that we can automatically check throughout the interaction between the player agent and the narrator agent. Experiments across state-of-the-art LLMs reveal a substantial long-horizon consistency gap: high linguistic quality does not guarantee commitment preservation; even strong models frequently generate logically conflicting content under adversarial interventions, with the best-performing model (GPT-5.2) achieving only 42% survival rate after 20 turns and fact conflict rates ranging from 40% to 68% across models, and only isolated runs satisfying all achievement commitments within the 100-turn limit.

1. Introduction

Interactive narrative agents must preserve world-state consistency and plot commitments despite unconstrained user interventions, yet fluent generation does not ensure logical integrity. The paper formulates this challenge as Narrative Commitment Preservation (NCP), introduces NCP-Bench, and shows substantial long-horizon brittleness in current LLMs.

  • Motivation: Narrator agents must guide players toward milestones, respect world constraints, and maintain structural coherence over long horizons rather than merely generate fluent stories.Free-form interventions can prompt retroactive rewrites or impossible narrative jumps that contradict the interaction history or underlying world state.
  • Motivation: Unbounded user agency conflicts with rigid plot commitments, requiring agents to preserve causal integrity when players attempt to bypass mandatory narrative sequences.The paper illustrates this tension with speedrun interventions that circumvent milestones such as the protagonist’s wounding and capture in Iron Man.
  • Problem formulation: NCP defines commitment preservation as maintaining an explicit fact ledger and non-optional narrative commitments while validating state updates after every response.This reframes interactive narrative from creative writing into a long-horizon constraint satisfaction problem.
  • Benchmark: 100 narrative specifications derived from movie synopses comprise NCP-Bench, constructed through reference trajectory, commitment, and initial fact extraction.The benchmark audits turn-by-turn interactions by checking narrator responses for fact, commitment, and player-input conflicts.
  • Empirical findings: 42% survival after 20 turns is achieved by GPT-5.2, while fact conflicts occur in 40%–68% of interactions across models.Survival rates decline as interactions deepen, and only isolated runs satisfy all achievement commitments within the 100-turn limit.

2. Task of Interactive Narrative

Interactive narrative is a turn-based, incrementally constructed interaction in which a narrator agent responds to free-form player actions while maintaining a coherent world and authorial plot constraints. Success requires reaching required plot progress without contradictions, making long-horizon consistency under user freedom a central challenge.

  • Interaction setting: Interactive narrative unfolds through repeated player–agent turns, unlike one-shot story generation.The narrator agent plays the role of a Game Master and constructs the story incrementally.
  • Interaction setting: At turn t, free-form player action u_t and narrative continuation y_t are conditioned on dialogue history H_t and an evolving story world.The evolving world includes prior events, current truths, and active plot constraints.
  • Narrative objectives: The narrator’s response both updates narrative state in reaction to the player and steers events toward plot-relevant futures while preserving agency and immersion.These coupled functions induce the trajectory τ_1:T = (u_1, y_1, . . . , u_T, y_T).
  • Narrative objectives: Successful interaction requires consistency with intended plot constraints and required plot progress without contradictions.Maintaining these properties over long horizons while allowing user freedom is formalized as a commitment-preservation task.

3. NCP-Bench

NCP-Bench evaluates long-horizon narrative commitment preservation using 100 manually selected movie-based environments. Each environment combines a player-constrained narrative specification with fixed commitments, evolving facts, trajectory tracking, and turn-level auditing.

  • Benchmark construction: 100 movie environments are selected from the CMU Movie Summary Corpus after expert cleaning and diversity-oriented filtering for coherent, detailed, genre-spanning narratives.Experts remove duplicates, resolve character–movie mismatches, and exclude incomplete synopses before selecting the final benchmark environments.
  • Benchmark construction: Each environment assigns the associated character as the player role, constraining annotations and evaluation to facts and events that character could plausibly know.This prevents omniscient information leakage and preserves a concrete in-story perspective.
  • Narrative specification: A structured specification ⟨F0, C, R⟩ represents the initial fact ledger, fixed narrative commitments, and ordered reference trajectory used to describe and track story development.Trajectory nodes specify an event, its observable trigger, and its lasting key delta.
  • Commitment specification: Commitments comprise invariant, ordering, and achievement rules, with success requiring all achievement commitments to be satisfied without any invariant or ordering violation.The commitment set remains fixed, while the fact ledger updates as the story develops.
  • Evaluation procedure: At each turn, an auditor checks the narrator’s response against active facts and commitments, extracts fact updates, advances trajectory progress, and terminates interactions when violations occur.The active ledger retains negated facts for traceability while excluding them from subsequent decisions and audits.

4. Evaluation Framework

The evaluation framework separates narrator generation from a fixed auditing protocol that checks consistency, updates state, and tracks narrative progress. An adversarial player generates plausible in-world interventions using only information visible to the player character.

  • Evaluation architecture: The framework uses prompt-fixed components to check narrator text against the explicit fact ledger and commitments, update state, and track progress.This design keeps the narrator agent separate from the externally defined auditing protocol.
  • Turn-level evaluation: Each turn checks fact, commitment, and player-input conflicts before committing extracted fact updates to the ledger.Player-input conflicts require acknowledging blocked or redirected actions and explaining the outcome.
  • Turn-level evaluation: Trajectory progress advances only when both the current node’s trigger event and key delta are explicitly completed; ambiguous cases do not advance.Commitments are separately marked PENDING or SATISFIED with evidence from facts or trajectory nodes.
  • Adversarial player: The adversarial player agent produces one first-person, in-world action per turn using only the interaction history and latest narrator output.Hidden facts, commitments, and future trajectory nodes are withheld to ensure interventions reflect the player character’s visible perspective.

5. Experiments

Experiments evaluate multiple state-of-the-art narrator LLMs under a fixed validation and termination framework. Results show severe long-horizon degradation: even GPT-5.2 survives only 42% of environments after 20 turns, with fact conflicts dominating failures.

  • Experimental setup: Outputs use temperature 0.6, top-p 0.95, and an 8092-token maximum, with invalid calls retried up to twice without advancing episode state.Runs terminate as CONFLICT, SURVIVAL at 100 turns, or SUCCESS after all achievement commitments are satisfied.
  • Experimental setup: Six state-of-the-art LLMs serve as narrator agents, while Gemini-2.5-Flash serves as the adversarial player and evaluation agent.The evaluated narrators include GPT-5.2, GPT-4o-mini, DeepSeek-V3.2, Qwen3-235B-A22B, Kimi-K2.5, and Grok-4.1-Fast.
  • Main results: 42% is GPT-5.2’s survival rate after 20 turns, while several other models’ survival rates approach zero as interaction continues.Survival measures environments preserved without logical conflicts or commitment violations.
  • Main results: 32.92 is GPT-5.2’s highest average-turn score, whereas DeepSeek-V3.2 leads trajectory progress at 15.40% and satisfied commitments at 13.42%.DeepSeek-V3.2 averages 15.88 turns, suggesting more aggressive plot advancement, while GPT-5.2 progresses more conservatively.
  • Failure analysis: 40%–68% is the range of fact-conflict rates across models, with GPT-5.2 lowest at 40% and Qwen3-235B-A22B highest at 68%.Fact conflicts include contradictions of established world states or tracked facts; other failures include premature revelations, reality changes, and ignored user input.
  • Evaluation reliability: 0.9866 is the highest reported pairwise auditor correlation, while human experts disputed only 4 GPT-4o-mini fact-conflict false positives among 100 reviewed results.No expert found errors in commitment-conflict or player-input-conflict outputs.

6. Discussion

The discussion frames Narrative Commitment Preservation as a general challenge for interactive agents, while explaining current LLM failures and positioning logical consistency as foundational but insufficient for compelling narratives. It also identifies auditor ambiguity, provider nondeterminism, and the current focus on single-threaded chronological narratives as limitations.

  • Broader Implications beyond Narrative: NCP extends beyond narrative to any interaction in which agents must preserve established facts, obligations, and constraints under open-ended external inputs.Coding agents are an example: new requests must not silently break implemented behavior, interface contracts, tests, or safety requirements.
  • Why Do Current LLMs Fail at NCP?: Current LLMs fail at NCP because they lack explicit state tracking and must reconstruct world state from dialogue history, causing long-horizon drift and contradiction accumulation.The discussion also identifies next-token prediction as failing to explicitly penalize logical inconsistency.
  • Logical Consistency Is Necessary but Not Sufficient: Logical consistency is necessary but not sufficient for compelling narratives, which can remain flat, predictable, or emotionally disengaging despite being contradiction-free.NCP-Bench therefore targets a foundational layer before richer criteria such as dramatic tension, emotional resonance, and character depth.
  • Logical Consistency Is Necessary but Not Sufficient: Fewer than 14% of commitments are satisfied by the best models, while coherence degrades sharply over turns, showing that the foundational logical requirement remains unsolved.The framework is presented as a stepping stone toward richer narrative evaluation once the logical scaffold becomes reliable.
  • Limitations: The framework is limited by imperfect auditor judgments for ambiguous text, provider-side nondeterminism that prevents exact replication, and its focus on single-threaded chronological narratives.The stated target is the most common form in commercial interactive fiction; nonlinear structures remain an extension.

7. Conclusion

The paper formalizes Narrative Commitment Preservation and introduces NCP-Bench as an auditable benchmark for interactive-narrative commitment preservation. Experiments show that fluent generation does not ensure long-horizon logical consistency, with world-state maintenance remaining a fundamental challenge.

  • NCP formalizes commitment preservation, while NCP-Bench evaluates it through an auditing protocol decoupled from the agent under test.The benchmark makes facts, commitments, and trajectory progress explicit and auditable.
  • 42% survival after 20 turns is achieved by the best-performing model, GPT-5.2, despite fluent text generation.Experiments span six state-of-the-art LLMs.
  • 40%–68% fact conflicts dominate failures, and only isolated runs satisfy all achievement commitments within the 100-turn limit.These findings identify world-state maintenance as a fundamental challenge under long-horizon interaction.
  • NCP-Bench provides a concrete testbed for developing reliable narrator agents and extends to settings requiring persistent obligations under adversarial pressure.The proposed formulation is intended to facilitate future research beyond narrative.

Impact Statement · A. Related Work

The paper presents NCP-Bench as an auditable benchmark for preserving facts and commitments during free-form interaction, while situating it within research on interactive narrative, role-playing agents, and long-horizon coherence. It also notes limitations requiring complementary human evaluation and broader future extensions.

  • Impact Statement: NCP-Bench evaluates whether agents preserve established facts and commitments during free-form interaction.The benchmark is designed to support auditable assessment of narrator agents that avoid silently rewriting prior events or bypassing essential plot constraints.
  • Impact Statement: The benchmark supports trustworthy long-horizon AI by providing a controlled setting for studying persistent obligations.Its interactive-narrative setting targets systems whose outputs must remain consistent with established information and commitments.
  • Impact Statement: The benchmark may inherit source-material biases, emphasizes linear narratives, and relies on LLM-based auditors that can err on ambiguous text.The authors therefore position NCP-Bench as complementing, rather than replacing, human evaluation of narrative quality, creativity, and cultural appropriateness.
  • A. Related Work: Interactive narrative research balances player agency with authorial control while pursuing engagement, character believability, dramatic tension, and world-state logical consistency.The cited tradition includes systems that explicitly addressed multiple narrative desiderata simultaneously.
  • A. Related Work: Classic interactive-narrative systems treated story generation as search over plot structures and character intentions through explicit planning and drama management.This line of work includes landmark systems such as Façade, which integrated autonomous characters, natural language, and real-time plot steering.
  • A. Related Work: In the LLM era, interactive narrative research revisits free-form natural-language interaction through iterative narrative planning that mediates author intent and emergent multi-character behavior.StoryVerse is identified as an example of this direction.
  • A. Related Work: Long-horizon dialogue and role-playing agents face long-term obligations: established facts or commitments should not be contradicted by subsequent responses.Related benchmarks and memory frameworks address character consistency across multi-scene trajectories and long-term dialogue coherence.

B. Human Narrator Baseline

A human narrator successfully handled adversarial interventions in the Iron Man environment, demonstrating that NCP-Bench is solvable. The human reached all 13 trajectory nodes and satisfied 7 of 12 commitments, while LLM sessions often failed much earlier.

  • Human baseline: 19 interventions were successfully resolved by the human narrator, who reached all 13 trajectory nodes and satisfied 7 of 12 commitments.The human author served as narrator in the Iron Man environment under adversarial player inputs.
  • Human baseline: 100 turns were completed without conflict in the human max-survival session, which reached 5 of 13 trajectory nodes.The resolve-all human session reached all 13 trajectory nodes without conflict, whereas the max-survival session prioritized conflict-free continuation.
  • Human baseline: 73 turns was GPT-5.2’s maximum survival, while HiAgent reached 32 turns; neither resolved the narrative with all commitments satisfied.Most LLMs failed within the first five turns, and GPT-5.2 still did not satisfy all commitments.

C. NCP-Bench Dataset … D. Example of a Step-Skipping Violation

NCP-Bench externalizes narrative facts and commitments into structured movie-level specifications, enabling explicit consistency evaluation across varied trajectories and environments. Its interaction loop detects both fact and ordering-commitment violations, as illustrated by a step-skipping failure in Iron Man.

  • C. NCP-Bench Dataset: Implicit narrative commitments and facts can make evaluators disagree, motivating their externalization into explicit objects in NCP-Bench.The evaluator otherwise must infer commitments, world state, and contradictions from transcripts.
  • C. NCP-Bench Dataset: 100 movie-level narrative specifications use a shared schema of initial facts, commitments, and trajectory nodes.The Bourne Identity instance contains 14 initial facts, 20 commitments, and 20 trajectory nodes.
  • C.1. Global Statistics: Approximately 16 atomic facts, 12 commitments, and 15 reference trajectory nodes occur per movie, with substantial heterogeneity across titles.Facts range from 8 to 31 per movie.
  • C.1. Global Statistics: Fact and commitment counts are broad and right-skewed, whereas trajectory-node counts are approximately symmetric and unimodal.The long tails indicate a subset of movies with particularly complex fact or commitment structures.
  • C.2. Narrative Richness Across Movies: The dataset contains more than 1600 atomic facts, 1200 commitments, and 1500 trajectory nodes across 18 genres, with typical trajectories spanning 10–20 nodes.These properties support multi-step causal reasoning, commitment satisfaction, and conflict detection.
  • C.3. Per-Environment State Representation: At each turn, the framework tracks transcript history, a fact ledger, a reference trajectory and position index, and explicit non-optional commitments.Facts represent persistent state and knowledge, trajectory nodes specify player-perceptible trigger events and irreversible key deltas, and commitments have satisfaction and violation conditions.
  • D. Example of a Step-Skipping Violation: In the Iron Man example, activating Mark I armor immediately contradicts initial facts and skips the required forced-labor-before-armor sequence.The relevant constraints require Stark’s wounding and capture before captivity or surgery, forced labor before secret armor construction, and terrorist unawareness until r3.
  • D. Example of a Step-Skipping Violation: The interaction terminates when Conflict Check confirms both a fact conflict and an ordering-commitment conflict.Algorithm 1 checks conflicts before applying state updates and returns CONFLICT when a violation is detected.

E. Evaluation Algorithm · F. Case Study

The evaluation algorithm formalizes turn-level interaction between narrator and player agents through conflict detection, state updates, and termination checks. A case study then illustrates four failure types using concrete interaction traces.

  • E. Evaluation Algorithm: Algorithm 1 formalizes the turn-level interaction loop between the narrator agent and the player agent.
  • E. Evaluation Algorithm: The interaction loop incorporates conflict detection.
  • E. Evaluation Algorithm: The interaction loop incorporates state updates.
  • E. Evaluation Algorithm: The interaction loop incorporates termination checks.
  • F. Case Study: The case study presents concrete interaction traces.
  • F. Case Study: These traces illustrate the four failure types from Section 5.3.

F.1. Fact Ledger Conflicts … F.5. Summary

The case studies identify systematic narrative-agent failures under adversarial pressure: contradicting tracked facts, disclosing information prematurely, rewriting player actions, and blocking actions without causal justification. These failures span spatial, object, character, entity, knowledge, input-handling, and action-resolution inconsistencies.

  • F.1. Fact Ledger Conflicts: Fact-ledger conflicts are the most common failure, with narrators contradicting established world states.Examples include relocating Dallas without transition, deflating an already inflated raft, prematurely forming Tallahassee’s partnership, and making the attendant and driver disappear without explanation.
  • F.1. Fact Ledger Conflicts: Spatial, object, character, and entity states were each contradicted without an intermediate narrative mechanism.The examples involve Dallas’s location, the raft’s inflation state, Tallahassee’s relationship history, and the attendant and driver’s existence.
  • F.2. Premature Information Disclosure: Premature disclosure failures reveal information before the narrative specification permits the player character to possess it.The narrator exposed Indiana Jones’s snake phobia, named the unknown Buzz Lightyear, and directed Laurie’s awareness toward Michael too early.
  • F.3. Unacknowledged Player Input: The narrator rewrote explicit player actions without acknowledgment or in-narrative obstruction.A sent message became an unsent, deleted message, while the Casino Royale action was rendered ineffective after the targets were retroactively removed.
  • F.3. Unacknowledged Player Input: Narrative redirection can invalidate player intent when the response alters reality instead of providing legitimate resistance.In Casino Royale, pointing at identified suspects became pointing at nothing because the attendant and driver were made nonexistent.
  • F.4. Inadequate Action Resolution: Narrators also blocked player actions without explaining the causal chain behind the redirection.In Predator, immediate self-destruct and escape-pod activation were halted or left dormant without explaining why the requested actions failed.
  • F.5. Summary: Across cases, four systematic patterns persist: fact contradiction, premature disclosure, action rewriting, and inadequately justified action blocking.The summary identifies these patterns as recurring failures of state-of-the-art LLMs serving as narrative agents under adversarial pressure.

G. Genre-wise Performance Analysis … Audit Pipeline Prompts

Performance varies substantially by narrative genre, while the benchmark’s prompts formalize trajectory, commitment, and fact tracking and enforce consistency through runtime generation and auditing. The complete prompts, code, and dataset are publicly available.

  • G. Genre-wise Performance Analysis: Biography has the widest trajectory-progress spread: Qwen3-235B-A22B reaches 52.38%, while other models remain below 17%.Musical generally performs relatively highly, but Kimi-K2.5 reaches only 2.80%; Horror and Animation remain below 10% for most models.
  • H. Complete Prompts: The complete verbatim prompts, implementation code, and NCP-Bench dataset are publicly available.The repository is linked in the paper at https://github.com/ yingpengma/NCP-Bench.
  • Data Construction Prompts: Trajectory extraction converts each synopsis into a linear reference trajectory whose nodes specify world state, external trigger events, and concrete factual deltas.The prompt assigns sequential node identifiers and requires player-perceptible, non-psychological triggers.
  • Data Construction Prompts: Commitment extraction defines non-optional ordering, invariant, and achievement constraints, with fact-based, mutually exclusive, directly observable satisfaction and violation conditions.The construction also blocks premature conclusions and excludes static ontological background facts.
  • Data Construction Prompts: Initial-fact extraction builds an atomic ledger at t = 0 containing player-observable facts, explicit negative states, and commitment-relevant initial values.The prompt applies point-of-view locking and outputs a JSON facts array.
  • Gameplay Runtime Prompts: Runtime prompts separate opening narration, adversarial play, cooperative baseline play, and narrator responses that advance the current trajectory while preserving consistency.Narrator guidance prioritizes the staged description → trigger → delta sequence, visible-only knowledge, narrative friction, and system-established state.
  • Audit Pipeline Prompts: The audit pipeline checks fact, commitment, and input conflicts, extracts minimal fact diffs, synchronizes trajectory-node progress, classifies commitment status, and double-checks flagged conflicts.Each auditor uses current facts and interaction history to judge end-of-turn state with evidence-based conditions.
Loading 2608.08160v1…