PaperlayerView exact revision on arXiv

Source-linked AI summary

EvolvingWorld: An Open-Schema Framework for Co-Evolving Role-Play Agents and World Model in Interactive Literary World

Qing Zong, Yue Guo, Mengxin Yang, Yiwen Guo, Yangqiu Song

arXiv:2607.17250v1cs.CL

TL;DR

Interactive literary simulation must track characters and worlds evolving together over long horizons, while prior role-play systems commonly rely on static profiles and rigid or sparsely tracked worlds. EvolvingWorld couples an open-schema Character Agent with a World Model, and experiments report more coherent long-horizon simulations across diverse backbones.

  • Problem

    Prior role-play systems often use static character profiles and rigid worlds or lack detailed entity-level state tracking, limiting their fit for evolving literary worlds.

    Source: §3.1 · p. 3

  • Method

    EvolvingWorld couples an open-schema Character Agent for multi-character role-play and profile evolution with an LLM-based World Model for state tracking and scene progression.

    Source: §1 · p. 2

  • Results

    Experiments report more coherent long-horizon simulations across diverse backbones and reduced long-horizon performance degradation versus previous frameworks.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    EvolvingWorld provides a concrete foundation for studying persistent world evolution beyond isolated role-play.

    Source: §5 · p. 8

    The benchmark uses public-domain classic books and does not cover modern novels, games, or user-created worlds.

    Source: Limitations · p. 9

Abstract

from arXiv · show

This paper introduces EvolvingWorld, a framework and benchmark for character and world co-evolution in interactive literary worlds. Existing systems either treat interactive literary simulation as static persona imitation or isolated scene generation, failing to capture how characters and worlds evolve together over time. To address this, EvolvingWorld models literary simulation as a long-horizon process where characters interact, scenes progress, and character and world states are persistently updated. Unlike prior systems relying on fixed schemas, EvolvingWorld adopts an open-schema framework to support simulation across diverse literary worlds. The framework consists of two coupled modules: a Character Agent for multi-character role-play and persistent profile evolution, and an LLM-based World Model for global and location/entity-level state maintenance and scene progression. Based on this architecture, we formulate 7 trainable tasks for scene initialization, interaction generation, and state update. We construct a dataset from 57 books, producing 138,596 supervised training samples and 222 snapshots for testing. Furthermore, we introduce a trajectory-level LLM-as-Judge evaluation protocol spanning 10 dimensions and 20 metrics. Experiments show that EvolvingWorld can improve long-horizon simulation by effectively maintaining persistent, coherent character and world development.

1 Introduction

EvolvingWorld frames interactive literary simulation as a long-horizon process in which characters and worlds co-evolve through interactions. It combines open-schema role-play and world-state tracking with supervised tasks, a book-based benchmark, and trajectory-level evaluation.

Motivation

  • Literary simulation requires coherent character and world states across scenes as beliefs, motivations, relationships, locations, objects, and background conditions change.The challenge extends beyond generating a plausible next utterance to maintaining long-horizon state coherence.Source: §1 · p. 1

Limitations of prior systems

  • Prior systems rely on static profiles, manually specified sandboxes, or single-scene role-play; BookWorld lacks full profile and location/entity-level updates and trainable subtask supervision.BookWorld updates character goals and states and global events, but does not provide the fuller evolution supported by EvolvingWorld.Source: §1 · p. 1

Open-schema co-evolution

  • Open-schema co-evolution infers book-specific character and world dimensions, coupling character actions that alter the world with world changes that reshape motivations and profiles.The framework tracks evidence for updates and how local events propagate to global, location, and entity states.Source: §1 · p. 2

Framework

  • Its coupled Character Agent and LLM-based World Model support multi-character role-play, profile evolution, global and local state tracking, and scene progression.Hidden trackers retain weak or emerging evidence so profile dimensions, such as emotions and personality traits, can evolve at different speeds; the framework comprises 7 supervised tasks.Source: §1 · p. 2

Empirical evidence

  • EvolvingWorld reduces the long-horizon performance degradation observed in previous frameworks, supporting co-evolution of character and world states.The evaluation framework covers 20 trajectory-level metrics beyond local role-playing quality.Source: §1 · p. 2

Benchmark and evaluation

  • The benchmark draws on 57 books and contains 138,596 supervised training samples and 222 test snapshots.The trajectory-level LLM-as-Judge evaluation covers 20 metrics for persistent character and world development.Source: §1 · p. 2

2 Related Work

The related work identifies gaps in long-horizon character evolution, adaptable world modeling, and trajectory-level evaluation; EvolvingWorld addresses them through open-schema character and world evolution and evaluation of persistent development.

Role-Play Agents

  • Prior role-play agents span scripts, self-alignment, training, multimodality, and memory retrieval, but typically treat personas as static; EvolvingWorld enables open-schema long-horizon character evolution.Source: §2 · p. 3

Multi-Agent and World Model

  • Prior environments use fixed sandboxes, role-play agents omit world models, and BookWorld fixes schema and static states; EvolvingWorld supports open-schema world evolution across diverse book worlds.Source: §2 · p. 3

Role-Play Evaluation

  • Evaluation shifted from fluency toward behavior, and although trajectory-level evaluation exists, state-evolution quality remains underexplored; EvolvingWorld quantifies persistent character and world development.Source: §2 · p. 3

3 The EvolvingWorld Framework

EvolvingWorld unfolds a book from a narrative snapshot through scene-by-scene interactions, persistently updating open-schema character and world states. Its framework couples a Character Agent with a World Model and supports long-horizon, cross-scene simulation.

3.1 Design Principles of EvolvingWorld

  • EvolvingWorld couples a Character Agent and World Model to address static character profiles and worlds lacking detailed entity-state tracking.The framework uses an open-schema design for long-horizon character and world evolution.Source: §3.1 · p. 3

3.1.1 Character Agent

  • The Character Agent uses open-schema profiles and evolves every profile dimension, with a hidden tracker accumulating emerging evidence across scenes.Reference dimensions guide profile construction, while the LLM can select, merge, or add fields for a book’s genre, setting, and style. The agent also supports environments and character groups as acting units; dimension changeability and accumulated evidence help distinguish rapid changes such as mood from slower changes such as personality.Source: §3.1.1 · p. 4

3.1.2 World Model

  • The World Model maintains open-schema global settings and detailed physical states for locations, including important non-character entities.Locations may be nested or atomic, and character interactions automatically update both global and location-level states, enabling cross-scene simulation without passive-backdrop worlds.Source: §3.1.2 · p. 4

3.2 Simulation Pipeline and Task Formulation

  • Each scene cycles through planning, multi-character interaction, in-scene world updates, and post-scene character updates to form a persistent trajectory.The goal is a grounded long-horizon trajectory in which scene content, character evolution, and world-state changes remain mutually consistent.Source: §3.2 · pp. 4–5

3.3 Dataset Construction

  • The dataset is built from 57 chronologically narrated books, using later scenes as evidence to ground character and world-state changes.The extraction LLM structures books into scenes with summaries, scenarios, key characters, and multi-turn interactions; character states use later narrative evidence as look-ahead, while world-state updates draw on subsequent interactions and scenes.Source: §3.3 · p. 5, §3.3.1 · p. 5

3.3.1 Structured Data Extraction

  • Extraction builds and updates character and world states scene by scene, using narrative look-ahead as evidence.The process unifies character names, initializes profiles and hidden trackers from early scenes, and standardizes location names while tracking story-level settings and location entities. Truncated scenes continue into the next text chunk.Source: §3.3.1 · p. 5

3.3.2 Train/Test Split Construction

  • 138,596 supervised samples cover seven tasks, and the test split contains 222 samples drawn from specific timeline points.The splits include training data, in-domain tests from partially seen books, and out-of-domain tests from books excluded from training; each test sample contains current character and world states from which simulation can continue.Source: §3.3.2 · pp. 5–6

3.4 Evaluation Framework

  • Evaluation scores CHARACTER and WORLD quality across 10 dimensions and 20 metrics, using per-metric LLM judges and hierarchical aggregation.Character dimensions cover consistency, evolution quality, environmental grounding, interaction quality, motivation generation, and instruction compliance; world dimensions cover scene planning, speaker management, world-state maintenance, and instruction compliance. Early termination from invalid output triggers task-specific penalties in instruction compliance and affected metrics.Source: §3.4 · p. 6

4 Experiment

EvolvingWorld improves long-horizon character and world modeling through targeted state-evolution supervision, and outperforms BookWorld across shared evaluation metrics while mitigating degradation on longer trajectories.

4.1 Experiment Setup

  • Experiments fine-tune several open-source model families with supervised instruction tuning and evaluate long-horizon simulations of up to 20 scenes and 50 interactions per scene.Training mixes Tulu3 data at a 1:1 ratio; evaluation covers closed- and open-source models, role-playing baselines, and EW-trained models.Source: §4.1 · pp. 6–7

4.2 Main Results

  • In the paper’s World Model evaluation, Qwen-32B trained on EW reaches a World Model Avg. of 59.87, surpassing Claude-4.6-Sonnet and Gemini-2.5-Flash.Source: Table 3 · p. 7, §4.2 · p. 8
  • Across matched backbones, EW training improves Character Agent and World Model performance and beats role-play-only CoSER and Crab on Qwen-7B and Llama-8B.The authors attribute the gains to book-to-world supervision of coupled character and world-state evolution, beyond dialogue imitation.Source: §4.2 · p. 7
  • Character Agent gains center on consistency, evolution, and progression; Environment Awareness improves consistently, although Environment Utilization sometimes declines.EW-trained models better preserve profiles, speaking styles, and motivations while making event-conditioned profile updates smoother.Source: §4.2 · p. 7
  • EW training most clearly improves World Model long-range scene continuity, turn/scene orchestration, and location-level updates, while structured state tracking remains harder than character-agent tasks.The passage describes explicit state tracking as less covered by pretraining than narrative continuation.Source: §4.2 · pp. 7–8

4.3 Comparison with BookWorld

  • On shared test metrics, EvolvingWorld scores higher than BookWorld across four untrained backbones, with larger gains in character evolution and improvements in scene continuity and organization.BookWorld degrades on PES and SCC as simulations lengthen, whereas EvolvingWorld mitigates this trend and PES improves over longer trajectories.Source: §4.3 · p. 8, Table 4 · p. 8, Table 5 · p. 8

5 Conclusion

EvolvingWorld couples character role-play with an open-schema World Model across seven trainable tasks, supporting more coherent long-horizon simulations. The paper also identifies limitations in subjective world modeling, context-limited representations, and the benchmark’s domain coverage, alongside ethical risks and use restrictions.

5 Conclusion

  • EvolvingWorld combines a Character Agent, an open-schema World Model, and seven trainable tasks to support coherent long-horizon simulations across diverse backbones.The framework and benchmark are presented as a foundation for studying persistent world evolution beyond isolated role-play.Source: §5 · p. 8

Limitations

  • The shared objective world state cannot represent characters’ differing perceptions or imperfect memories, while maintaining character-specific perceived worlds would increase system complexity.The authors identify subjective world modeling as a direction for future work.Source: Limitations · p. 9
  • LLM context limits the World Model to important entities at each location rather than every entity; information-dense visual inputs may enable more detailed representations.Source: Limitations · p. 9
  • The benchmark uses public-domain Project Gutenberg classics, excluding modern novels, games, and user-created worlds, although its open-schema framework can in principle adapt beyond those domains.The corpus spans multiple genres, and out-of-distribution experiments report consistent gains on entirely unseen books; broader cross-domain evaluation remains future work.Source: Limitations · p. 9

Ethics Statement

  • The dataset derives from transformed and abstracted public-domain works, and its resources are intended exclusively for scientific research, not commercial use.Source: Ethics Statement · p. 9
  • Source books and LLM-generated annotations may contain or amplify harmful stereotypes and biases, and the fictional content should not be taken as representing authors’ viewpoints.Annotators were native English speakers, participated voluntarily with informed consent, and received compensation above local minimum wage.Source: Ethics Statement · p. 9

Appendices · A Partial Support Clarifications

The appendix explains why BookWorld, GenerativeAgents, and CharacterBox receive partial-support marks for specific capabilities in Table 1. Their limitations concern restricted profile updates, incomplete world-state modeling, or simulation granularity and scope.

A Partial Support Clarifications

B Training and Inference Details … C.2 Interleaved Cleaning Details

EvolvingWorld’s appendix details its model-training setup and the staged construction of scenes, characters, and world states. It also describes cleaning steps interleaved with the construction pipeline.

B Training and Inference Details

  • Models use LoRA fine-tuning for two epochs, then vLLM inference with 50 concurrent requests.Training uses rank 64, alpha 128, dropout 0.05, learning rate 2 × 10−5, 32,768-token sequences, and effective batch size 64 per GPU; 10% of merged training data is held out for validation.Source: B Training and Inference Details · pp. 12–13

C Data Construction Details

  • The appendix expands the data pipeline into three main construction phases and separately describes cleaning steps interleaved during implementation.Scene and interaction cleaning occurs after initial scene extraction and before character construction, world construction, and task generation.Source: C Data Construction Details · p. 13, §C.2 · p. 13

C.1 Main Construction Phases

  • Following Coser (Wang et al., 2025b) and AdaMARP (Xu et al., 2026b), the pipeline selects 57 chronological books and extracts structured scenes from chunked text.Scenes contain summaries, scenario descriptions, key characters, and actor-content interactions; truncated scenes are continued across chunk boundaries using carried-over text.Source: §C.1 · p. 13
  • Character construction standardizes aliases, initializes open-schema profiles from early scenes, and updates profiles, descriptions, and hidden trackers as scenes unfold.Updates incorporate what occurred and what later scenes reveal.Source: §C.1 · p. 13
  • World construction standardizes location aliases, initializes global and per-location states, and records persistent changes after each interaction.Source: §C.1 · p. 13

C.2 Interleaved Cleaning Details

  • Interaction refinement clarifies expressions while preserving meaning and narrative voice, separates private thoughts from speech, and merges consecutive turns by the same characters.Interactions are normalized to the first-person perspective of the acting character or group.Source: §C.2 · pp. 13–14
  • Cleaning removes the less informative scene from duplicate event pairs and enriches scenario descriptions with dramatic setup, atmosphere, and implicit tensions.The enhanced scenarios support the location_scenario training task.Source: §C.2 · p. 14

C.3 Training and Test Data Construction · D Dataset Statistics

EvolvingWorld converts book timelines into seven task-level training targets ordered by simulator execution, while evaluation uses snapshots from held-out portions of train/test and OOD books.

C.3 Training and Test Data Construction

  • Each extracted timeline is decomposed into seven supervised tasks according to the simulator’s execution order.Task-level supervision is generated after character, global-world, and location states have been sequentially updated along the book timeline.Source: §C.3 · pp. 14–15
  • At scene boundaries, scene_cast predicts scene existence and participating characters, location_scenario predicts the next location and scenario, and motivation_update infers characters’ entering motivations.These tasks condition predictions on prior-scene content, current states, and the planned next scene.Source: §C.3 · p. 14
  • Within scenes, next_character selects the next actor or termination, while interaction_gen generates that actor’s interaction from interaction history, character state, and motivation.Both tasks use the current scenario and visible interaction history as context.Source: §C.3 · p. 14
  • After interactions, world_update determines whether global or location state needs a persistent update, and character_update updates relevant character states after scene completion.interaction_gen and character_update are stored as ShareGPT-style conversations; other tasks use structured targets with task-specific prompts under the same SFT interface.Source: §C.3 · p. 14
  • Evaluation snapshots preserve character, global, and location states plus the previous scene; 10% of books are held out as OOD and never used for training.For train/test books, the first 70% of scenes train the model and snapshots from the remaining 30% form in-domain tests; train-only books contribute all scenes to training.Source: §C.3 · p. 15
  • Snapshots use valid candidate scenes with at least five interactions, capped at five ID snapshots per train/test book and twenty OOD snapshots per OOD book.OOD test snapshots are sampled from the latter 70% of scenes in OOD books.Source: §C.3 · p. 15

D.1 Overview … E.5.4 Dim IV: Instruction Compliance

This section describes EvolvingWorld’s open-schema literary dataset and long-horizon simulation and trajectory-level evaluation protocols. It defines evaluation criteria for character and world-model performance, including speaker management, world-state maintenance, instruction compliance, state-update ablations, and penalties for invalid output.

D.1 Overview

  • The 57-book corpus spans multiple genres and narrative scales, while open schemas capture 581 unique character dimensions and 136 world-state dimensions.Literary Fiction accounts for 59.6% of books, followed by Adventure (21.1%); the median book contains 160 scenes, 43 characters, and 25 locations.Source: §D.1 · pp. 15–16

D.2 Training Data Statistics

  • Sequence lengths vary by task: scene_cast has the longest median (≈11.2k), while character_update has the shortest (≈2.8k); the 32,768-token training cutoff accommodates long-tail samples while keeping computation tractable.Source: §D.2 · p. 17

E.1 Simulation Protocol

  • Evaluation runs each held-out snapshot forward as a multi-scene trajectory, generating scene plans, interactions, and persistent world- and character-state updates.The protocol evaluates coherence across scenes as well as local response quality.Source: §E.1 · p. 17

E.2 Trajectory-Level Evaluation

  • The trajectory-level LLM-as-Judge framework has 10 dimensions and 20 sub-metrics, split between CHARACTER (6 dimensions, 11 sub-metrics) and WORLD (4 dimensions, 9 sub-metrics).Sub-metrics are scored from 0–100 with a base score of 50; scores are aggregated according to whether metrics are scene-local, character-level, or full-trajectory.Source: §E.2 · pp. 17–19, Table 10 · p. 19

E.4.1–E.4.2 Character evaluation

  • Character Consistency assesses profile-constrained knowledge, speaking style, and motivation-driven behavior, while Evolution Quality assesses justified, complete updates and gradual, coherent profile growth.Evolution Quality distinguishes persistent changes for the profile from subtle or ambiguous signals retained in a hidden tracker.Source: §E.4.1 · p. 19, §E.4.2 · pp. 19–20

E.4.3–E.4.4 Character evaluation

  • Environmental Grounding measures whether behavior and descriptions reflect global and location states and use relevant scene elements; Interaction Quality measures contextual responsiveness and narrative progression.Interaction criteria include maintaining information and relationship continuity, adding developments rather than repetition, and following up on foreshadowing.Source: §E.4.3 · p. 20, §E.4.4 · p. 20

E.4.5–E.4.6 Character evaluation

  • Motivation Generation evaluates whether scene motivations are profile-aligned, situationally appropriate, and actionable, while Instruction Compliance checks character boundaries, output format, and length.Premature termination caused by invalid Character Agent output triggers both IC and task-metric penalties.Source: §E.4.5 · pp. 20–21, §E.4.6 · p. 21

E.5.1–E.5.2 World Model evaluation

  • World Model evaluation covers cast and location/scenario planning, cross-scene narrative coherence, and orchestration of speaker selection and environmental-description timing.Scene-planning criteria include character-setting fit and pacing; speaker management also considers appropriate scene-ending timing.Source: §E.5.1 · pp. 21–22, §E.5.2 · p. 22

E.5.2 Dim II: Speaker Management

  • Speaker management evaluates appropriate character combinations, participation proportional to narrative importance, transient-character fadeout, and scene endings at natural junctures.The criteria also require scenes not to end abruptly during a climax or drag on when little happens.Source: §E.5.2 · p. 22

E.5.3 Dim III: World State Maintenance⋆

  • The World Model is evaluated on timely, accurate global and location-state updates, including whether it distinguishes local from global impact and temporary from persistent change.Criteria also cover retiring outdated global information, concise descriptions, spatial consistency, accurate entity lists, and continuity across scenes.Source: §E.5.3 · pp. 22–23
  • Ablations compare full state-updating systems with versions removing character or world updates; removing the Hidden Tracker or using fixed schemas degrades reported update quality.The tables also report ID and OOD performance after full-mixture training, with untrained averages as references.Source: Table 15 · p. 23, Table 16 · p. 23, Table 13 · p. 23

E.5.4 Dim IV: Instruction Compliance

  • Instruction compliance checks valid, complete JSON outputs and keeps the World Model from generating character dialogue; premature termination from invalid output triggers IC and metric penalties.The full Character Agent benchmark is reported under three judge models, but this passage provides no scores.Source: §E.5.4 · pp. 23–24, Table 18 · p. 24

F Full Results on EvolvingWorld Benchmark

EW-trained models improve on their open-source backbones and role-playing-only baselines in both Character Agent and World Model evaluations. Full and balanced training mixtures show similar overall trends, though the better setting varies by backbone and evaluation role.

F Full Results on EvolvingWorld Benchmark

  • EW-trained models consistently improve over corresponding open-source backbones and role-playing-only baselines across Character Agent and World Model evaluations.The full and balanced mixtures show similar overall trends, but which performs better varies by backbone and evaluation role.Source: F Full Results on EvolvingWorld… · p. 24
  • EW-F preserves the training set’s natural task distribution, whereas EW-B gives each task the same number of training examples.The paper uses EW-F as the default in the main text and reports EW-B for completeness.Source: F Full Results on EvolvingWorld… · p. 24
  • Tables 11 and 12 report complete EvolvingWorld benchmark results for 21 models and their fine-tuned variants.Source: F Full Results on EvolvingWorld… · p. 24

G Ablation Study

The ablations show that character-state updates, world-state updates, the Hidden Tracker, and open schemas each support aspects of long-horizon simulation. The two state-update mechanisms are coupled, while book-specific schema dimensions outperform a single fixed schema.

Core mechanisms

  • Removing character updates sharply lowers PUF, PES, and MQ, whereas removing world updates mainly degrades world-model metrics.The mechanisms are coupled: removing world updates weakens character-side grounding and interaction, while removing character updates further harms world-side continuity and orchestration, indicating that long-horizon simulation depends on their joint evolution.Source: G Ablation Study · pp. 24–25

Hidden Tracker

  • 12.60 PUF points and 2.60 PES points are lost when the Hidden Tracker is removed.Their average drops by 7.60 points; the ablation removes the tracker from initial character states, simulation prompts, and character-update outputs while retaining scene-by-scene profile updates.Source: G Ablation Study · p. 25

Open vs. Fixed Schema

  • 1.25 and 1.35 points are lost from the focused averages when open schemas are replaced with fixed schemas.All books use the same fixed character and world-state dimensions, while other settings remain unchanged; the results indicate an advantage for book-specific dimensions.Source: G Ablation Study · p. 25

H In- and Out-of-Distribution Results … K Prompts

Full-mixture training improves ID and OOD results, and leading-model rankings are consistent across three judge models. The structured scenes also support video generation, while the appendix documents prompts for data construction, simulation, and evaluation.

H In- and Out-of-Distribution Results

  • Full-mixture training improves both ID and OOD performance over untrained backbones across Character Agent and World Model evaluations.OOD scores are sometimes slightly below ID scores but remain substantially above untrained performance; some Character Agent settings exceed their ID results.Source: H In- and Out-of-Distribution Results · p. 26

I Comparison across Judge Models

  • All three judges produce identical top-six World Model rankings, while Character Agent rankings share the same top six and top-three ordering except for one pair reversal.GPT-5.1-Chat reverses Kimi-K2.5 and GPT-4o; the authors interpret the overall agreement as a stable signal across judge families.Source: I Comparison across Judge Models · p. 26

J Downstream Application: Video Generation

  • Structured scene representations encode world and character states and fine-grained interactions, enabling coherent video generation and new or branched storylines.A four-scene Alice’s Adventures in Wonderland video was generated scene by scene with LingBot and concatenated in narrative order.Source: J Downstream Application: Video… · p. 26, Figure 8 · p. 27

K Prompts

  • The appendix provides complete prompts for the data-construction pipeline, simulation pipeline, and LLM-as-Judge evaluation framework.It includes prompt materials across the three pipeline stages.Source: K Prompts · p. 26

K.1 Data Construction Prompts

  • Data-construction prompts cover scene extraction, character-profile initialization and updates, and global and location world-state initialization and updates.The appendix presents the key prompts in Tables 23–27.Source: §K.1 · p. 26

K.2 Simulation Prompts

  • The simulation pipeline uses seven task-specific prompts with wording variants; broad-selection tasks receive short character or location descriptions rather than full entity states.Scene-cast uses latest character descriptions, while location-scenario uses location descriptions.Source: §K.2 · p. 27

K.3 Evaluation Prompts

  • Evaluation metrics use dedicated prompts, with shared scoring templates and separate per-scene criteria for Character Agent and World Model metrics and cross-scene criteria for PES and SCC.The appendix gives these materials in Tables 32–44.Source: §K.3 · p. 27

L Human Evaluation … 3. Scene Consistency

The section validates LLM-as-Judge preferences against human annotations and specifies criteria for evaluating character behavior, narrative progression, conversational responsiveness, and scene-state consistency. The criteria distinguish whether updates should occur from whether their content is accurate.

L Human Evaluation

  • The human study sampled 60 trajectories across three model pairs, with three trained native English annotators per sample, yielding 180 annotations.Annotators spent approximately one hour per sample because trajectories were long and complex.Source: L Human Evaluation · p. 27
  • Human preferences strongly supported judge-preferred models, with especially high exact agreement on Average scores; dimension-level agreement ranged 90.0%–100.0% for Character metrics and 81.7%–100.0% for World metrics.Human annotators also showed 100.0% majority agreement on the two Average scores, with Fleiss’ κ = 0.8.Source: Table 21 · p. 28, L Human Evaluation · p. 28

1. Behavioral Attribution; 2. Trinity Coherence (Thought →Action →Speech); 3. Motivation Persistence & Drift

  • Character criteria assess whether behavior follows motivations, thoughts, actions, and speech cohere, motivations persist or shift for justified reasons, and dialogue matches the character’s speaking style.The criteria discourage generic helpfulness, unexplained action or motivation changes, and verbosity-based judgments.Source: L Human Evaluation · p. 39, §1 · p. 39, §2 · p. 39

1. Information Increment; 2. Suspense and Hooks; 3. Foreshadowing Payoff; 4. Pacing

  • Narrative progression criteria reward new information or developments, suspense and hooks, payoff of planted foreshadowing, and pacing that gives important moments appropriate attention.They flag repetitive turns, abandoned narrative threads, flat endings, and rushed or slow pacing.Source: §3 · p. 42, §1 · p. 42, §2 · p. 42

1. Information Continuity; 2. Logical Continuity; 3. Relationship Matching

  • Contextual responsiveness criteria check whether characters remember prior information, react logically to the immediate exchange, and adapt their tone to established relationships and power dynamics.These criteria focus on turn-by-turn context rather than overall character-profile consistency.Source: §4 · p. 42, §1 · p. 42, §2 · p. 42

3. Relationship Matching

  • Location-update criteria separate update timing from content accuracy: LUS judges whether a change warrants an update, while LSA judges whether the changed state is correct.The criteria distinguish redundant updates caused by an incorrect trigger from missed content changes, and account for scene length and update opportunities.Source: §3 · p. 47

1. Spatial Consistency; 2. Entity Accuracy; 3. Scene Consistency

  • Scene-consistency criteria assess whether spatial layouts, important entities and their states, location descriptions, and successive updates remain physically and narratively consistent.Only important, lasting entities need tracking; transient details and output formatting are outside this accuracy assessment.Source: §1 · p. 47, §2 · p. 47, §3 · p. 47

Related papers

Loading 2607.17250v1…