PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models Have Interpretable Narrative Mechanisms

David Yiyang Liu, Aditya Joshi, Paul Dawson, Andrew Piper

arXiv:2610.12989v1cs.CL

TL;DR

The paper asks whether language models contain internal components that represent narrativity and contribute to behavior. It identifies and ablates narrative-associated components, finding effects on narrative generation and selected downstream tasks, while organization and broader functions remain open questions.

  • Problem

    Although prior work detects narrative patterns and internal representations, whether internal mechanisms causally contribute to narrative behavior and downstream tasks remains open.

    Source: §1 · p. 2

  • Method

    The authors identify attention heads, neurons, and SAE features associated with narrativity and intervene on selected component sets across prediction, generation, inference, and hallucination tasks.

    Source: §1 · p. 2

  • Results

    Ablations provide evidence that selected narrative-associated component sets contribute to model behavior, with effects on story completion and narrative question answering varying across models and languages, and mixed hallucination results.

    Source: §1 · p. 2, §9 · p. 11

  • Takeaways & Limitations

    The experiments provide evidence for narrative-related components making causal contributions to behavior, while their organization, distribution, and use remain open questions.

    Source: §9 · p. 11

    The study covers three pretrained models, uses a heuristic selection threshold, and supports causal claims about ablated sets rather than individual units.

    Source: Limitations · p. 11

Abstract

from arXiv · show

We investigate whether language models contain internal components that contribute to representing narrativity, the degree to which a text exhibits narrative properties. Across GPT-2-Small, Gemma-2-2B, and Llama-3.1-8B, we identify attention heads, MLP neurons, and sparse autoencoder features whose activations correlate with narrativity. Ablating selected neuron and feature sets increases next-token prediction loss more for high-narrativity than for low-narrativity texts, and reduces the narrativity of open-ended generations. On downstream benchmarks, the largest effects occur in story completion and narrative question answering, although they vary across models and languages; results on confabulation and hallucination benchmarks are mixed. These findings provide causal evidence that selected sets of narrative-associated components contribute to model behaviour, while their internal organisation, relation to narrative theory, and broader functions remain open questions.

1 Introduction

The paper investigates how language models represent narrative properties and whether those representations contribute to model behavior. It identifies narrative-associated components and tests them through interventions across generation and other tasks.

1 Introduction

  • The authors combine narratology with mechanistic interpretability, identifying narrative-associated components and intervening on them to observe behavioral consequences.They examine attention heads, neurons, and SAE features across text prediction, generation, inference, and hallucination tasks.Source: §1 · pp. 1–2
  • The study asks what narrative representations models use, whether they align with narratology’s dimensions, and whether they support intelligent activity.The motivating gap is whether internal narrative mechanisms causally contribute to behavior.Source: §1 · p. 2
  • Narrativity is approached through agency, event sequencing, and world building, which provide dimensions for locating narrative-associated representations.The introduction frames these dimensions as a starting point for testing whether narrative-theory distinctions correspond to functional contributions in models.Source: §1 · p. 2
  • Ablating narrativity-associated neurons and SAE features increases prediction loss more for more narrative texts, while evidence for attention heads is weaker.SAE-feature ablation also reduces generation narrativity; the paper reports effects on story completion and narrative question answering, with mixed hallucination results.Source: §1 · p. 2

2 Prior Work

Prior work has studied narrative patterns in text and models’ internal representations, while mechanistic interpretability has intervened on components associated with language behavior. The paper situates its contribution at the intersection of these lines of work.

2 Prior Work

  • Piper and Bagga (2022) operationalise narrativity dimensions as linguistic features and use machine-learning models to identify patterns predictive of narrativity.They report that some patterns influence human perception more than others and propose a “distant worlds” theory centered on agents’ concretised actions.Source: §2 · p. 2
  • Cognitive narratology extends inquiry beyond textual form toward cognitive phenomena and narrative as a cognitive resource (Herman, 2009, 2013).The section connects this perspective to common-sense reasoning, causal inference, contextual understanding, and hypotheses about confabulation (Huang et al., 2019; Sui et al., 2024).Source: §2 · pp. 2–3
  • Prior work uses layer-specific linear probes to detect narrative incoherence and encoder embeddings to represent characters and plot, but causal effects on outputs and contributions to downstream tasks remain empirically unexamined (De Langis et al., 2026; Volpetti et al., 2020; Hatzel and Biemann, 2024).Source: §2 · p. 3
  • Mechanistic interpretability studies have localized and manipulated components associated with language variety, showing that linguistic behavior can be causally changed within components (Liu et al., 2026b; Tang et al., 2024; Deng et al., 2025).The paper contrasts this component-level intervention work with the open question of causal narrative mechanisms.Source: §2 · pp. 2–3

3 Framework and Methodology

The framework operationalizes narrativity through three dimensions and identifies components by relating their activations to narrativity and prediction-loss changes. It then tests whether SAE feature explanations and dimension-specific feature sets correspond to those dimensions.

3.1 Multidimensional Narrativity

  • Narrativity is operationalized through agency, event sequencing, and world building, covering agents’ experiences, the temporal organization of events, and an experienceable storyworld.These dimensions follow Herman’s taxonomy and are represented as a scalar narrativity property.Source: §3.1 · p. 3

3.2 Narrativity Value Correlation (NVC)

  • NarraComponents are attention heads, neurons, and SAE latent features whose contribution values correlate with narrativity and whose ablation-related loss changes depend on narrativity.The method selects components using Spearman correlations and a heuristic threshold of ρ ≥0.5; ablation claims apply to component sets collectively.Source: §3.2 · pp. 3–4, Figure 2 · p. 3

3.3 SAE Feature Explanation Categorisation

  • Each NarraFeature is assigned to its most strongly correlated dimension, and explanation-only classifiers test whether its natural-language explanation predicts that category.The authors also test whether dimension-specific activations track generated-text narrativity and whether separately ablating those subsets affects generation.Source: §3.3 · p. 4

4 Experimental Setup

The experiments use a human-labelled narrative dataset alongside silver labels, three pretrained language models, and pretrained SAEs. They compare targeted interventions with random ablations and evaluate generation and benchmark behavior.

4.1 Narrative Dataset

  • The NarraDetect dataset contains 394 human-rated texts and is extended to 13,530 items with LLM-generated silver labels for component discovery.The human gold labels are held out for validation, separating component identification from validation.Source: §4.1 · p. 4

4.2 Models

  • Experiments use pretrained GPT-2-Small, Gemma-2-2B, and Llama-3.1-8B with pretrained SAEs; feature explanations come from Neuronpedia.The explanations were generated automatically before the experiment and independently on different datasets.Source: §4.2 · p. 4

4.2.1 Random Ablation Comparison

  • Targeted ablations are compared with equally sized random sets averaged over 10 seeds, sampled from each model’s full component pool.The random sampling pool does not exclude targeted components.Source: §4.2.1 · p. 5

4.2.2 Feature Classification

  • Feature explanations are tested with an embedding-based three-class logistic-regression classifier and a zero-shot LLM predicting agency, event, or world.The classification setup asks whether explanation strings alone recover the empirical feature categories.Source: §4.2.2 · p. 5

4.3 Generation

  • Generation uses 900 paired prompts differing only in requested genre, contrastive search, and two open-weight models to measure output narrativity.Prompts span 90 topics and five instruction templates; GPT is excluded because it does not consistently produce coherent outputs.Source: §4.3 · p. 5

4.4 Benchmarks

  • Benchmarks cover narrative inference, non-narrative reasoning, and hallucination-related behavior; accuracy is evaluated using log probability.XStoryCloze tests English, Russian, Chinese, and Hindi; GPT is excluded because of near-chance baseline performance and lack of multilingual capabilities.Source: §4.4 · p. 5

5 NarraDetect Results

NarraComponents identified by activation correlations show narrativity-dependent loss effects, while SAE explanations partly align with the paper’s three dimensions but remain noisy.

5.1 Detection and Verification

  • Targeted ablations have narrativity-selective loss effects in 8/9 setups; neurons and features disproportionately raise loss on highly narrative texts, unlike random ablations.The exception is the two identified GPT NarraHeads, which also have the weakest effects across the three models.Source: Figure 4 · p. 6, §5.1 · p. 6

5.2 SAE NarraFeatures

  • Examples of SAE feature explanations align with agency, world building, and event sequencing, with each example’s assigned dimension having the strongest reported correlation.The examples include correlations of 0.65 for agency, 0.64 for world building, and 0.71 for event sequencing.Source: §5.2 · p. 6
  • Common explanation words show aggregate dimension-related patterns: “personal” and “experience” for agency, “action” and “event” for event sequencing, and “physical” for world building.The authors suggest these patterns capture some meaningful distinctions among the dimensions.Source: §5.2 · p. 7
  • Many feature explanations describe surface grammar, and near-identical punctuation explanations can receive different dimension labels.Examples include punctuation descriptions assigned to agency, event sequencing, and world building among Gemma features.Source: §5.2 · p. 7
  • Explanation-only classifiers recover feature-dimension labels above chance, but low F1 scores indicate that some strings alone say little about a feature’s dimension.The authors proceed to test dimension-specific feature subsets through separate ablations during open-ended story generation.Source: §5.2 · p. 7, Table 4 · p. 7

6 Generation Results

NarraFeature ablation most strongly changes generated-text narrativity, while dimension-specific interventions affect multiple dimensions rather than isolating them.

6.1 Activations

  • Gemma and Llama generate texts with different narrativity under story versus explanation prompts, and NarraComponents activate more during story generation.Categorised NarraFeature contributions correlate at ρ ≥0.8 with the corresponding narrativity ratings of generated outputs.Source: §6.1 · p. 7, Figure 5 · p. 7, Table 5 · p. 7

6.2 Ablation

  • NarraFeature ablation changes output narrativity most, followed by NarraNeuron ablation; NarraHead ablation has much weaker effects.The NarraFeature effects can make models generate explanation-like text in response to a story prompt.Source: §6.2 · pp. 7–8, Table 7 · p. 8

6.3 NarraFeature Dimensions

  • Agency- and event-assigned feature ablations produce their largest raw score decreases on the corresponding dimensions, but interventions affect multiple dimensions and no selective world-building effect appears.Agency-group ablation reduces event-sequencing scores more than event-group ablation does; the authors interpret the pattern as partial alignment, not independently controllable dimensions.Source: §6.3 · p. 8, Table 8 · p. 8
  • Table 9 gives a qualitative example of Llama generations after ablating each dimension’s NarraFeatures, with Agency, Event, and World scores.The caption identifies the score scale as 1–5.Source: §6.3 · p. 8, Table 9 · p. 8

7 Benchmark Results

Ablating NarraFeatures most clearly affects narrative benchmarks: it degrades story completion and narrative question answering more than size-matched random ablation, while effects on hallucination benchmarks are weak or inconsistent.

XStoryCloze

  • NarraFeature ablation significantly degrades XStoryCloze performance more than size-matched random ablation across all four tested languages on Gemma, but only in English on Llama.NarraHead and NarraNeuron effects on XStoryCloze are smaller and inconsistent.Source: §7 · p. 9, Table 10 · p. 9

CosmosQA and OpenBookQA

  • NarraFeature ablation degrades CosmosQA performance more than size-matched random ablation for both Gemma and Llama, unlike its effects on non-narrative OpenBookQA.On OpenBookQA, degradation is similar to random ablation for Llama and smaller for Gemma.Source: §7 · p. 9, Table 10 · p. 9

Confabulation and hallucination

  • NarraFeature ablation produces weak or inconsistent results across the confabulation and hallucination benchmarks for Gemma and Llama.TruthfulQA improves significantly, but the effect is not significant in BEGIN and reverses on Gemma FaithDial, where faithful-response correctness matters.Source: §7 · p. 9, Table 10 · p. 9

8 Discussion

The discussion finds that narrative-associated components have partial, model-dependent effects and that SAE features align imperfectly with narrative theory. Benchmark results offer qualified evidence of use in reasoning, while the inferred implicit narrative theory appears agent-centred and partly tied to surface cues.

8.1 Narrative Components

  • NarraFeatures produce the largest observed changes in narrativity-correlated prediction loss, generation narrativity, and English story-completion performance under the study’s selection and intervention procedure.Llama shows no significant multilingual NarraFeature effect, while NarraHead ablation affects non-English XStoryCloze more significantly; the authors suggest model differences as one possible explanation.Source: §8.1 · p. 9
  • The NVC method may favor certain components, and large SAE-ablation effects do not establish that residual-stream representations matter more than attention heads or neurons.The authors leave alternative methods and negatively narrativity-correlated components for future work.Source: §8.1 · pp. 9–10

8.2 Interpreting Representations With Narrative Theory

  • SAE explanations contain dimension-specific signal above chance, but modest predictive performance and inconsistencies make them incomplete descriptions of features.The authors note concerns about non-unique feature decompositions and broad or incomplete automatic explanations.Source: §8.2 · p. 10
  • Herman’s narrative framework partially explains identified components, but determining which narrative theories fit language-model representations requires further research.The authors suggest testing other theories, including Sternberg’s.Source: §8.2 · p. 10

8.3 What are Narrative Components used for?

  • XStoryCloze and CosmosQA provide partial evidence that NarraFeatures support common-sense inference and reasoning, but inferring general sense-making from narrative-related tasks risks circular reasoning.The authors call for more diverse ways to test this connection; fixed-response likelihood shifts also do not establish fewer hallucinations in open-ended generation.Source: §8.3 · p. 10

8.4 The Language Model’s Implicit Narrative Theory

  • The implicit narrative theory appears strongly agent-centred, with weaker and less selective world-building effects and some reliance on surface linguistic cues.This interpretation is tentative because of limitations in SAE features and their explanations.Source: §8.4 · p. 11

9 Conclusion

The paper identifies narrative-associated components across three language models and uses ablation to provide evidence of their causal contributions to model behaviour. NarraFeatures show the strongest reported effects, with downstream results varying across models and tasks.

Conclusion

  • Ablation provides evidence that identified attention heads, neurons, and SAE features contribute to textual-narrativity representations in GPT-2-Small, Gemma-2-2B, and Llama-3.1-8B.NarraFeatures produce the most meaningful change to narrative generation among the identified component sets.Source: §9 · p. 11
  • NarraFeature ablation impairs XStoryCloze in English, Russian, Chinese, and Hindi on Gemma, but significantly only in English on Llama; it also degrades CosmosQA more than random ablation, unlike non-narrative OpenBookQA.Hallucination and confabulation benchmark results are mixed.Source: §9 · p. 11
  • The findings support causal contributions from narrative-related components while leaving open how narrative representations are organised, distributed, and used.Dimension-specific effects are partial: agency and event ablations show only partial on-target degradation, and world-building ablation shows none.Source: §9 · p. 11

Limitations

The study’s scope is limited by its model sample and component-selection approach.

Limitations

  • The study covers three pretrained language models and uses a heuristic threshold to select components; causal claims apply to ablated sets collectively, not individual units.Source: Limitations · p. 11

Ethical Considerations

The appendices detail how narrativity is rated and how candidate components are measured and ablated, alongside ethical risks and checks on generation results. They also document the datasets, judges, and experimental procedures used in these analyses.

Ethical Considerations

  • Online-text and model biases may be reproduced, and narrative-framing interventions could manipulate rhetoric or alter framing without users’ awareness.The authors recommend disclosing such interventions and evaluating their effects across languages and demographic groups.Source: Ethical Considerations · p. 11

A Detailed Narrativity Definitions and Prompt

  • The rating rubric operationalizes narrativity through agency, event sequencing, and world building, scored as separate dimensions by an LLM judge.Agency concerns consistent foregrounded agents; event sequencing concerns connected temporal relations; world building concerns an inhabitable, coherent setting.Source: A Detailed Narrativity Definitions and… · p. 16

B Judge models

  • Five judges align highly with human gold labels, while separate frontier-class models score generations and locally hosted models extend corpus labels.The human comparison covers 394 passages; the three corpus-labeling judges agree closely with one another.Source: B Judge models · pp. 16–17, Table 11 · p. 17

C Extending NarraDetect

  • NarraDetect combines human ratings for 394 passages with averaged LLM ratings for 13,530 additional documents spanning narrative and non-narrative genres.Three LLMs produce the extended labels and show an average deviation index of 0.26 across the three narrativity dimensions.Source: C Extending NarraDetect · p. 17

D.1 Attention Heads; D.2 Neurons

  • Attention-head contribution is the mean L2 norm of token-level outputs, while neuron activations are averaged across tokens and selected neurons are zeroed for ablation.Head ablation zeros its output; neuron ablation is applied at every token position except the first attention-sink position.Source: §D.1 · pp. 17–18, §D.2 · p. 18

D.3 Sparse Autoencoders (SAEs)

  • SAEs represent residual-stream activations with sparse features, whose token-averaged activations are measured and whose selected decoder contributions are subtracted for ablation.This intervention preserves residual-stream components that the SAE does not reconstruct.Source: §D.3 · pp. 18–19

E Text Preprocessing and Token Handling; F Explanation Classifier Setup; G Open-Ended Generation

G Open-Ended Generation

  • After excluding degenerate generations, SAE-target narrativity drops remain below every size-matched random-ablation seed across tested repetition thresholds.The target effect changes by at most 0.11 points after exclusions; the reported exception concerns a negative result for Llama world ablation on explanation prompts.Source: G Open-Ended Generation · p. 21, Table 13 · p. 21

H Benchmark Evaluation Protocol

The protocol evaluates benchmarks by scoring candidate continuations with each model’s next-token probabilities, using task-specific prompts and score normalization. The reported benchmark table covers ablations of narrative features, attention heads, and MLP neurons against matched random baselines.

H Benchmark Evaluation Protocol

  • XStoryCloze evaluates two candidate endings after a four-sentence story context, selecting the ending with the higher token-normalized score.The evaluation uses the English, Russian, Chinese, and Hindi splits.Source: H Benchmark Evaluation Protocol · pp. 21–22
  • CosmosQA and OpenBookQA score answer-option letters from the next-token probabilities after a passage or supporting fact and question.CosmosQA uses its validation set and OpenBookQA its test set; both have four options per question.Source: H Benchmark Evaluation Protocol · p. 22
  • TruthfulQA, FaithDial, and BEGIN use byte-normalized likelihood scores, with task-specific pairwise or ranking criteria for correctness.TruthfulQA uses a zero-shot, byte-normalized adaptation, so its absolute accuracies are not directly comparable to published MC1 results.Source: H Benchmark Evaluation Protocol · pp. 22–23
  • Benchmark accuracy reflects likelihood preference for a truthful or faithful response, not what the model would generate.The protocol uses zero-shot candidate-continuation scoring and applies either per-token mean or per-byte normalization as specified for each benchmark.Source: H Benchmark Evaluation Protocol · pp. 21, 23

J Extended Benchmark Results

  • Table 16 compares ablations of NarraFeatures, narrativity-selective attention heads, and MLP neurons with matched-size random baselines.The supplied caption identifies the comparison scope but does not provide outcome values.Source: Table 16 · p. 24
Loading 2610.12989v1…