Towards Coherent and Consistent Use of Entities in Narrative Generation

• Current language models struggle with entity coherence and consistency, especially in narratives containing complex events and many named entities.Source: §1 · p. 1
• The paper proposes automatic metrics and shows that generated stories have significantly lower entity coherence and consistency than human-written narratives.The metrics are also validated against human judgments of entity coherence.Source: §1 · p. 1
• A dynamic entity memory augments a pre-trained language model with cross-attention to entities participating in the narrative.The approach is motivated by dynamic entity representations used in earlier language-modeling work.Source: §1 · pp. 1–2
• The model uses end-to-end soft reads and writes, allowing conditioning on multiple entities and updating memory slots from cross-attention scores.Token-level entity annotations regularize the cross-attention scores to guide memory operations.Source: §1 · pp. 1–2
• On narrative-generation experiments, the entity memory improves entity coherence according to automatic metrics and human judges, particularly with limited context.It also preserves more entity-related information when later narrative sections fall outside the model’s short-term context.Source: §1 · p. 2

Task Formulation

• Entity-driven generation conditions an autoregressive language model on a prompt containing the gold entities participating in the narrative.The objective is to compare entity-usage patterns in generated and human-written stories.Source: §2 · p. 2

Entity coherence

• The proposed long-range coherence metric measures the maximum span of narrative sections containing each protagonist’s mentions.Protagonists are defined as the entities with the most mentions, and the narrative is divided into equal sections.Source: §2 · pp. 2–3

Entity consistency

• Attribute consistency measures the percentage of unique verbs and adjectives appearing in sentences that mention each entity.The method treats those verbs and adjectives as entity attributes and aggregates the resulting values across entities.Source: §2 · p. 3
• The metric could use dependency trees for more accurate attribute assignment, but this extension is left for future work because dependency parsing remains imperfect.Source: §2 · p. 3

Metric limitations

• The coherence metric’s maximum-span variant was selected because intermediate-mention variants produced similar results while making analysis significantly slower.Source: §2 · p. 3
• Consistency depends on coherence: entities mentioned more often and across longer spans are harder to keep consistent, while a single mention makes consistency trivial but uninformative.The metric normalizes consistency by the story’s degree of coherence.Source: §2 · p. 3

Motivation

• The dynamic memory addresses prompt limitations by storing entity attributes beyond surface forms and retaining entity information independently of the current context window.The paper notes that prompts eventually fall out of short-term memory during sufficiently long narratives.Source: §3 · pp. 3–4

Dynamic entity memory

• A dynamic entity memory stores static entity keys and context-updated values, refreshed after narrative chunks to preserve entity information over time.Each memory slot represents an entity or non-entity information, while Transformer-XL memory stores previous narrative chunks.Source: §3 · p. 4

Memory access

• Cross-attention blocks operate alongside self-attention at every layer to produce entity-aware token representations from all memory slots.The model combines entity-aware representations with self-attended representations through a gating mechanism.Source: §3 · p. 4
• Soft attention lets the model condition on multiple relevant entities and update memory slots at regular intervals instead of using hard one-entity updates.An additional weight discourages updating every slot and reflects which entities contributed most during memory reads.Source: §3 · p. 4

Model variants

• A static-memory variant keeps entity keys but performs no memory updates, providing a comparison with the dynamic design.The static variant is called S-MNEMELM.Source: §3 · p. 4

Entity supervision

• Token-level entity annotations provide an auxiliary KL-divergence loss that guides cross-attention toward the entities associated with each token.The final objective combines the language-modeling loss with this entity-related regularization loss.Source: §3 · p. 4

4.1 Experimental Settings

• The study uses WritingPrompts for pronoun-heavy everyday stories and WikiPlots for more complex narratives with many named entities and diverse attributes.The datasets are selected to expose differences in entity usage across narrative types.Source: §4.1 · p. 5
• Models are evaluated with full context and with Transformer-XL memory reduced from 500 to 100, 50, or 10 tokens to simulate longer narratives.The limited-context setup tests whether models retain prompt and early entity information after it leaves the context memory.Source: §4.1 · p. 5

4.2 Evaluation Metrics

• Evaluation measures perplexity, entity-mention uncertainty over time, prompt-entity exact and subset match, and entity usage derived from coreference resolution.Entity-mention uncertainty is the average negative log probability of tokens belonging to entity mentions, while section-level values measure information loss over time.Source: §4.2 · p. 5
• The main comparison includes VANILLALM, static-memory MNEMELM, and dynamic-memory MNEMELM, with Table 1 also ablating memory initialization and entity supervision.Table 1 reports PPL, entity-mention uncertainty, and uncertainty for other words under full narrative context.Source: Table 1 · p. 5, §4.1 · p. 5

4.3 Implementation details

• The base model is a roughly 300M-parameter Transformer-XL pretrained on WMT, with entity memory increasing parameters by 16.67%.Stories are generated using nucleus sampling with p=0.8 and temperature 1.Source: §4.3 · p. 5

5.1 Automatic Evaluation

• With full narrative context, dynamic entity representations are crucial for competitive performance, although VANILLALM and D-MNEMELM show no significant perplexity difference in that setting.The strongest memory advantage appears when context is limited.Source: §5.1 · p. 5
• D-MNEMELM performs significantly better than VANILLALM with 10–100-token T-XL memories, especially on WikiPlots, reducing the loss of entity-related information.The advantage is more prominent for entity-mention uncertainty, and dynamic memory closes the gap between full- and limited-context scenarios.Source: §5.1 · pp. 5–6
• Regularizing cross-attention scores is especially important: removing the entity-specific auxiliary loss yields significantly higher perplexity and unstable training.The ablation also finds that randomly initialized entity memory and the auxiliary loss are both crucial for model performance.Source: §5.1 · p. 6

5.2 Analysis of Generated Stories

• Memory-augmented models better imitate human entity usage and substantially improve long-range coherence; on WikiPlots, they achieve a 57% relative coherence improvement over VANILLALM.VANILLALM mentions protagonists across an average maximum of 3.29 of 10 sections versus 5.65 for HUMAN, while memory largely fixes this tracking failure.Source: §5.2 · pp. 6–7, Table 2 · p. 7
• D-MNEMELM improves consistency over S-MNEMELM, including on WritingPrompts for both V and U, while memory improves control on WritingPrompts but not on WikiPlots.On WikiPlots, increased coherence can lower unnormalized V because entities are mentioned more often and across longer spans.Source: §5.2 · p. 7

5.3 Human Preferences

• Human judges find that memory-augmented models significantly improve coherence and control over VANILLALM, with no significant differences in consistency or fluency.The results indicate that coherence improves without hurting attribute consistency, and S-MNEMELM is selected best more often while VANILLALM is selected worst more often.Source: §5.3 · pp. 7–8, Table 3 · p. 8
• The coherence metric C correlates moderately with human judgments at 0.32, whereas exact and subset control correlations are 0.17 and 0.19, and consistency V correlates at 0.08.Coherence and control mostly influence human preference, while consistency is hardest to judge and has the smallest impact on decisions.Source: §5.3 · p. 8, Table 4 · p. 8

6 Related Work

• Earlier dynamic-entity language models use smaller RNNs and discrete assumptions that this work relaxes for transformer-based efficiency.Source: §6 · p. 8
• Topic-control approaches prompt pretrained language models with keywords, outlines, or other relevant knowledge rather than explicitly storing entity information.Source: §6 · p. 8
• Rashkin et al. store individual tokens, condition on key phrases and the previous paragraph, and update memory at paragraph level, whereas this work stores entity information and updates more frequently.Source: §6 · p. 8
• Two-step narrative systems generate detailed plans through semantic role labeling, but sentence-level planning may limit transitions to longer stories such as books or screenplays.Source: §6 · p. 8

7 Conclusions

• Current pretrained language models struggle to maintain entity coherence and consistency when generating longer text, especially with context beyond short-term memory.Source: §7 · p. 8
• A dynamic entity memory yields significantly higher entity coherence under both automatic and human evaluation.Source: §7 · p. 8
• The dynamic entity memory helps preserve entity-related information when the available narrative context is limited.Source: §7 · p. 8

A.1 Coherence Metrics

• The main coherence metric is the maximum span of sections in which each protagonist appears, targeting long-range entity coherence.Source: §A.1 · p. 10
• An alternative coherence metric averages the number of sections where a protagonist appears and includes intermediate entity mentions.Source: §A.1 · p. 10
• For WikiPlots, both coherence metrics indicate that VANILLALM mentions protagonists less throughout stories than memory-augmented models.Source: §A.1 · p. 10
• For WritingPrompts, the average-appearance metric gives no clear comparison, although it shows the same tendencies as the maximum-span metric.Source: §A.1 · p. 10
• The maximum-span metric is selected because it better quantifies long-range coherence differences between generated and human-written stories across datasets.Source: §A.1 · p. 10

A.2 Implementation Details

• The base Transformer-XL language model has 18 layers, 16 self-attention heads per layer, a 1024-dimensional hidden state, and approximately 300M parameters.Source: §A.2 · p. 10
• Adding entity memory and four-head cross-attention blocks increases MNEMELM to approximately 350M parameters.Source: §A.2 · p. 10
• D-MNEMELM updates its entity memory every 64 narrative tokens and uses temperature 0.1 to encourage distinct memory-slot representations.Source: §A.2 · p. 10
• Human evaluation agreement is reported using exact agreement and average rating distance for control, consistency, fluency, and coherence.Source: Table 6 · p. 11
• Long-range coherence uses the three most-mentioned characters across L = 10 equal narrative sections, with the same sectioning for uncertainty analysis.Source: §A.2 · p. 11
• The full-context setting uses a 512-token sequence and 500-token Transformer-XL memory, while limited-context experiments use memory sizes [100, 50, 10].Source: §A.2 · p. 11

A.3 Model Outputs

• VANILLALM quickly abandons prompt entities and introduces irrelevant entities later in the narrative, whereas S-MNEMELM remains more faithful to the prompt.The examples include later shifts to irrelevant volcanoes or the introduction of “Mayor Bill Breen.”Source: §A.3 · p. 11
• D-MNEMELM maintains high long-range entity coherence while reducing the repetitions and entity overuse observed with S-MNEMELM.Its dynamic memory can consistently track newly introduced entities, although controlability may be lower when those entities differ from the prompt.Source: §A.3 · p. 11
• D-MNEMELM uses a more natural entity-mention pattern than S-MNEMELM, including more consistent pronoun use.It may generate synonyms or closely related expressions, such as “man in uniform” for the prompt entity “police officers.”Source: §A.3 · p. 11

A.4 Human Evaluation Details

• Human evaluation asked judges to identify protagonists, rate control, coherence, consistency, and fluency, and select the best and worst story.The study collected judgments for 494 WikiPlots test-set stories, with two judges evaluating each story.Source: §A.4 · p. 11, Figure 5 · p. 12