PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Tell Stories Out of Order? A Corpus and Benchmark for Nonlinear Narrative Generation

Syed Rifat Raiyan, Md. Tariquzzaman, Md Farhan Ishmam, Tenghao Huang

arXiv:2610.21468v1cs.CLcs.AI

TL;DR

Because telling order shapes what readers know, but is seldom recorded or tested directly, the paper builds a benchmark of paired event orders and story graphs across 83 works. Across evaluations, models tend toward chronology, and readers prefer graph-guided stories to stories constrained to a supplied order.

  • Problem

    Telling order is seldom recorded in story-generation research, leaving open whether models can control published narrative shapes and whether evaluations track readers.

    Source: §1 · p. 1

  • Method

    The paper annotates 83 works with shared event descriptions in chronological and telling order, story graphs, and a five-task benchmark evaluated on 19 models.

    Source: §1 · pp. 1–2

  • Results

    Readers prefer graph-guided stories to stories constrained to a supplied order, while models given event-order rules produce formulaic solutions unlike published tellings.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Following an instruction to reorder events is not yet the same as telling a story well out of order.

    Source: §6 · p. 9

    Prose and scaffolding studies cover 14 works, and the low-reasoning-effort evaluation’s small ablation cannot rule out effects of test-time compute.

    Source: Limitations · p. 11

Abstract

from arXiv · show

Every story has two orders: the order in which its events happen and the order in which they are told. A story is nonlinear when the two differ. We introduce a benchmark of 83 published works that measures whether language models can tell nonlinear stories. For each work, we annotate one list of events in both orders, together with a story graph of causal dependencies, character knowledge, and the questions the telling raises and resolves. Given events in chronological order, all 19 models tell them, on average, more chronologically than the published works do; asked for a specific structure, they land within the human range about half the time. We then compare a model's stories of the same events under different conditions, using both an automated judge and 71 human readers. Both prefer the chronological story a model writes by default to the nonlinear one it writes on request, which readers find harder to follow in 81% of decisive judgments. Readers also prefer stories written from the story graph to stories constrained to the published telling order (72% of decisive judgments), a difference the judge does not detect. Our results suggest that following an instruction to reorder a story is not yet the same as telling it well out of order.

1 Introduction

Narrative shape depends on the order in which events are told, which governs what readers know and how they interpret each scene. Because prior work seldom records telling order, this paper builds a paired-order corpus and tests whether models can control and evaluate nonlinear tellings.

1 Introduction

  • A nonlinear narrative departs from chronology, and its telling order determines what readers know at each point and how they read new scenes.The paper calls the structural pattern of a telling order its narrative shape.Source: §1 · p. 1
  • Prior generation work plans stories, revises drafts, or prompts for flashbacks, but telling order is seldom recorded, limiting the ability to specify revelations and withheld answers.A revelation depends on its explanation being narrated after the event it explains, while a withheld answer must arrive after the question.Source: §1 · p. 1
  • Models given a shape as an event-ordering rule produce formulaic solutions unlike published tellings; readers prefer their default chronological stories, and graph-guided stories, to alternatives.The judge misses readers’ preference for graph-guided stories over stories constrained to a supplied telling order.Source: §1 · p. 2
  • The benchmark records chronological and telling orders for 83 works, alongside story graphs, and evaluates 19 models across five tasks.The paper also tests an automated judge and structural metrics against 3,528 blind judgments from 71 readers.Source: §1 · p. 2

2 Related work

Prior work covers story planning and generation, temporal and narrative representations, and evaluation of reading experience. The paper identifies open questions about controlling published narrative shapes, reliance on telling order, and agreement between structural measures, automated judges, and readers.

2 Related work

  • Story systems plan and revise narratives, while some use temporal prompts or rewrite chronological drafts into planned tellings; this benchmark tests control across seven shapes with events held fixed.Prior examples include hierarchical generation and Plan-and-Write (Fan et al., 2018; Yao et al., 2019), Re3 (Yang et al., 2022), flashback prompting (Han et al., 2022), and rewriting.Source: §2 · p. 2
  • Temporal resources, DramaBank, and Story Curves represent temporal relations, dramatic structure, or paired orders, while this corpus adds shared event descriptions and story graphs for controlled generation.Story Curves aligns chronological and telling orders across ten nonlinear films.Source: §2 · p. 2
  • Prior reading-experience measures and model-rating studies leave open whether structural and automated measures track readers’ preferences.The paper also notes known LLM-judge biases, motivating length-controlled evaluation.Source: §2 · pp. 2–3

3 Methodology

The corpus represents each work as a shared event list with chronological and telling orders, a structural label, and a story graph. The representation separates event order from scene-level shape and captures causal, knowledge, and reader-question relations.

3 Methodology

  • Each work is represented by 12–20 self-contained event sentences that can be read in chronological or telling order, with both orders, a structural label, and a story graph.The authors annotate each work once and apply machine checks for constraint violations.Source: §3 · p. 3
  • A telling is a permutation of chronological events, while a scene is a group contiguous in both orders; the representation excludes repeated narration, staged disclosure, and omission.Structural labels describe scene order, whereas the permutation records event order.Source: §3 · p. 3
  • The 83 works were selected to cover seven narrative shapes, not to estimate how prevalent those shapes are.The corpus began with films and added novels, plays, musicals, and a short story until every category had at least ten examples.Source: §3 · p. 3, Table 1 · p. 3
  • The seven labels describe scene-order patterns, from predominantly chronological CHRONICLE to rule-less ACHRONY, and include reverse, interrupted, alternating, and grouped forms.Works require recoverable chronology; ACHRONY therefore means no consistent telling-order rule despite a recoverable chronological order.Source: §3 · p. 3
  • Story graphs encode causal dependencies, world-state constraints, character knowledge, and reader questions; REVEALS edges connect later-told explanations to the events they explain.In the illustrated ZIGZAG work, four REVEALS arcs answer questions raised by earlier-told events.Source: §3 · p. 3, Figure 3 · p. 4

4 Experimental Setup

The experiments test order generation and reconstruction, then compare prose under different structural conditions. Evaluation combines order metrics, an automated judge, and blind reader comparisons.

4.1 Experimental Design

  • The benchmark tests five tasks: cued and free telling generation, chronology reconstruction, shape classification, and story generation with or without a shape instruction.The prose and graph studies use 14 works, two per shape.Source: §4.1 · p. 4
  • Reconstruction controls compare gold-telling input with randomly ordered events, then repeat both conditions after removing explicit time cues.The random-order score is the no-story floor.Source: §4.1 · p. 4
  • Graph conditions compare no added input, a full graph without position fields, and a supplied gold telling order; REVEALS edges constrain 5.5% of telling-order pairs.Five ablations remove one annotation group from the full-graph condition.Source: §4.1 · p. 4

4.2 Evaluation

  • Order evaluation uses Kendall’s τ and normalized LIS, with a band hit requiring both measures to fall within the gold range for the target shape.Band hit measures resemblance to corpus examples, not the shape itself; the measures do not capture alternation or grouping.Source: §4.2 · p. 4
  • Generated prose is aligned to the event list by first mention, and the primary aligner recovers the gold order at τ = 0.998 when given the supplied order.The aligner checks validate alignment, not preference.Source: §4.2 · p. 5
  • The study compares 71 readers’ blind judgments with an automated judge’s paired story preferences and ratings.Readers gave 3,528 responses over 294 sittings; preference analyses exclude ties and include word-count-adjusted and length-matched comparisons.Source: §4.2 · p. 5

5 Results

Models usually tell events more chronologically than the published works do, and requested nonlinear structures do not reliably produce stories readers prefer. Readers favor default tellings and graph-guided stories in key comparisons, while several structural metrics do not track their preferences.

5.1.1 Left alone, they tell it straight

  • All 19 models’ spontaneous tellings were more chronological than the corpus mean: pooled τ was 0.638 versus 0.456 for human tellings.The models received events in chronological order, so the study cannot separate chronology preference from preference for input order.Source: §5.1.1 · p. 5

5.1.2 Asked for a shape, they comply literally

  • When asked for a named shape, models landed in the human band 47.9% of the time, but RETROGRADE and CHRONICLE band hits were only 1.1% and 5.3%.RETROGRADE outputs were often exact reversals, while CHRONICLE outputs were often exact identity orders; the prompts specify event-rank rules rather than scene-level practices.Source: §5.1.2 · p. 5

5.2.1 Reconstruction without the story

  • T3 reconstruction reached pooled τrec = 0.743, but the telling added only +0.035 τ over the no-story solver floor.Exact reconstruction rose from 8.5% without the telling to 20.2% with it; τrec alone therefore overstates how much the telling contributes.Source: §5.2.1 · p. 6

5.2.2 Naming the shape

  • T4 classification reached 50.2% pooled accuracy against 14.3% random and 26.5% majority baselines; shape naming correlated with reconstruction, but writing and reading abilities barely correlated.Across models, naming-shape ability correlated with T1 band hit at ρ = 0.717 and T3 τrec at ρ = 0.747, while T1 and T3 correlated at ρ = 0.088.Source: §5.2.2 · p. 6

5.3 RQ3. Does a story graph help?

  • The graph improved judged structural compliance, but the gain came almost entirely from one model and disappeared after length adjustment.Readers preferred graph stories over told-order stories 72% of the time, although the judge split 23–22; readers also favored reduced graphs over the full graph in 69% of pooled decisive judgments.Source: §5.3 · p. 7, §5.5.2 · p. 8

5.5.1 The three preference results

  • Readers preferred models’ free tellings to instructed tellings and found instructed versions harder to follow; they also favored graph-guided stories over supplied telling orders.Readers chose free tellings 76% of the time and called instructed tellings harder to follow 81% of the time; they preferred graph-guided stories over told-order stories in 72% of decisive judgments.Source: §5.5.1 · p. 8
  • Against a bare event list, graph-guided stories drew 59% unadjusted reader preference, but the rate fell to 52% after length adjustment.Graph texts averaged 649 words versus 603, and the length-matched subset favored the graph at 54%.Source: §5.5.1 · p. 8

5.5.4 Our metrics against readers

  • Readers’ difficulty judgments tracked lower chronologicality (∆τ = −0.171), but human-band membership did not predict preference: readers chose the in-band text 34% of the time.The judge matched readers closely on instructed-versus-free preferences, but its ties on graph-versus-told-order and layer-removal comparisons missed reader preferences; PDS levels from one judge were uniformly high.Source: §5.5.4 · p. 9, §5.5.3 · p. 9

6 Discussion

Rule compliance, resemblance to published tellings, and reader preference are distinct desiderata, and the results separate them. Event-level instructions may not capture scene-level practice, while graph-guided stories need comparison with order-supplied stories under matched conditions.

6 Discussion

  • Compliance, resemblance, and reader preference can diverge: a model may satisfy an ordering rule yet miss the human band, while readers prefer chronology.Band hit measures resemblance, not quality; here it runs against reader preference.Source: §6 · p. 9
  • Event-level reversal instructions miss the scene-level practice recorded in published tellings; whether scene-marked prompts yield that shape remains open.A scene-level reversal computed from the corpus alone lands inside the human band.Source: §6 · p. 9
  • Graph scaffolds encode part of the answer, so their value requires comparison with directly supplied orders at matched output length.The paper identifies ordering freedom and length as confounds in the graph-versus-told-order comparison.Source: §6 · p. 9

7 Conclusion

The benchmark makes narrative order measurable, and all 19 models tell events more chronologically than the published tellings when left to choose. Yet literal event-level instructions can miss human narrative shapes, and current instruments need scrutiny.

7 Conclusion

  • All 19 models tell events more chronologically than the published tellings when left to choose, while literal event-level rules can miss the human band.A scene-level reversal computed from the corpus alone lands inside the band.Source: §7 · p. 9
  • Recording paired chronological and telling orders for 83 works turns nonlinearity into a permutation that can be specified and scored.The corpus also includes story graphs; the benchmark evaluates whether models can control narrative order.Source: §7 · p. 9
  • A reading-comprehension-like task retains 96.1% of its score without the telling, and readers preferred graph-guided stories where the judge found a tie.The conclusion cautions that instruments require scrutiny alongside models.Source: §7 · p. 9

Limitations

The paper’s findings are bounded by the models, stories, evaluation procedures, and annotation choices used. Several comparisons remain confounded or underpowered, and the benchmark captures only selected aspects of narrative order.

Limitations

  • Structural scores and premature-disclosure rates depend on one model aligning generated prose to events; reader judgments do not validate that extraction.The judge also shares a model family with one generator in each prose comparison.Source: Limitations · p. 10
  • Preference and close-reading results cover 14 stories; scaffolding uses four generators, while ablations use two generators with one replicate.Extending preference evidence requires new generation runs.Source: Limitations · p. 10
  • Length remains a confound: graph preferences do not survive word-count adjustment, and instructed-versus-free judgments also mix nonlinear control with instruction cost.Length-matched generation was not run; the close-reading PDS gap has the same caveat.Source: Limitations · pp. 10, 12
  • The reader cohort is not a probability sample, and the corpus was selected for structural coverage rather than prevalence or broad narrative representativeness.The study makes no claim about readers in general; the selected reference corpus is predominantly recognized for nonlinearity.Source: Limitations · p. 10
  • Annotation is unreplicated and interpretive, bands depend on single-annotator extremes, and transfer beyond English-language annotation is untested.The corpus also makes no human-level claim because its reference is published work, not human writers given the models’ brief.Source: Limitations · p. 11
  • Event-rank prompts specify literal reversal and identity rather than scene-level practice, so over-compliance measures the prompt specification as well as model behavior.The scene-marked rerun remains untested; premature-disclosure results are also suggestive, resting on 52 stories per condition after exclusions.Source: Limitations · pp. 11–12
  • The benchmark scores selected plot beats as a permutation, not repetitions or omissions; arbitrary tie-breaking also makes simultaneity count as error.The permutation assigns each event one position, and the gold chronology linearizes simultaneous events.Source: Limitations · p. 12
  • Reader statistics do not model the crossed design, and layer ablations are small and generator-dependent; the prompt-length confound remains unresolved.With 51 judgments per layer, 80% power reaches only |rate − 0.5| ≥ 0.19.Source: Limitations · pp. 10–12

Ethics statement

The ethics statement describes an anonymized release built from published works, while the appendix preliminaries explain the paper’s seven narrative shapes and how it represents order.

Ethics statement

  • The release contains hand-authored event summaries, structural annotations and graphs, while evaluated texts replace identifying surface detail with anonymized descriptions.The authors state that the release does not reconstitute copyrighted works or substitute for reading them; it is distributed under the MIT License.Source: Ethics statement · p. 12

Contents of the appendix

  • The appendix contents list supporting material on the shapes, experimental setup, related work, formal analysis, reader study and reproduction.The list also includes reporting recommendations, corpus checks, benchmark notes, retrospective coherence gain, classification, and task and reader prompts.Source: Contents of the appendix · p. 15

A Preliminaries: the seven shapes

  • The preliminaries represent a telling as a permutation of chronological event ranks, with chronology on the diagonal and departures from it showing narrative order.Kendall’s τ and normalized longest increasing subsequence capture different aspects of that departure: forward direction and the largest chronological subsequence.Source: A Preliminaries: the seven shapes · p. 15
  • The seven shapes distinguish chronological telling, flashbacks, flash-forwards, alternating timelines, non-temporal grouping, reverse chronology and achrony.RETROGRADE reverses scene order while events within scenes proceed forward; the models’ pure reversal is absent from the corpus. The corpus’s narrower ACHRONY label requires chronology to remain reconstructable and no statable ordering pattern.Source: A Preliminaries: the seven shapes · pp. 15–16

B Experimental setup in detail

This appendix details the benchmark conditions and places them alongside prior work on narrative order, generation, temporal ordering and evaluation. Its setup distinguishes what models receive and how reconstruction and graph conditions are tested.

B Experimental setup in detail

  • Models receive self-contained event sentences, with tasks varying whether those events are presented chronologically or in the gold telling order.Prompts withhold story identifiers, order columns, scene annotations, metrics, annotator notes and graphs.Source: B Experimental setup in detail · pp. 16–17
  • For three selected models, shuffled-input reconstruction provides a no-story floor for measuring how much chronology recovery depends on the gold telling order.The conditions are also rerun on descriptions rewritten to remove explicit time cues.Source: B Experimental setup in detail · p. 17
  • Graph ablations retain the target-shape cue while removing annotation groups; band hit measures resemblance to corpus examples, not whether a telling truly has a shape.The main graph conditions use four models, while ablations use two generators; the study includes 476 stories and 252 blindly judged pairs.Source: B Experimental setup in detail · p. 17

C Related work, in detail

  • Unlike temporal resources that annotate relations within text, this work records the permutation between chronological and telling orders as a generation target.The authors connect T3 to document-level temporal ordering and sentence-ordering benchmarks, noting that self-contained event sentences expose evaluation to an entailment floor.Source: C Related work, in detail · p. 18
  • The reader study holds texts, questions and items fixed to compare automated judgments with reader judgments, while accounting for known judge biases and dependence.The paper reports judge ties on two comparisons where readers split about 70 to 30, motivating the conclusion that a judge’s tie is not evidence of no difference.Source: C Related work, in detail · p. 18
  • The paper extends paired-order narrative resources with shared event descriptions, typed graphs and benchmark tasks that keep events fixed while making telling order scoreable.Its seven-shape benchmark complements systems that rewrite chronological simulations into narrative order and temporal prompts that guide flashbacks.Source: C Related work, in detail · pp. 17–18

D Formal analysis

The formal appendix explains what the paper’s permutation scores can and cannot distinguish, then derives how hops, scene boundaries and event-pair constraints shape those scores. These results clarify why literal event-level reversal differs from published scene-level retrograde and why reconstruction scores may have limited headroom.

D.1 Definitions used throughout

  • Band hit cannot identify narrative shape: distinct tellings can share the same (τ, LIS) coordinates while differing in hop counts and operational labels.Among tellings of six events, two examples share τ = 7/15 and LIS = 4/6 yet have two versus five hops; at n = 9, 112 of 136 coordinate cells contain multiple hop counts.Source: §D.1 · p. 19

D.2 Hops

  • A hop is a break from continuing to the next chronological event, and a telling’s maximal hop-free stretches are its runs.The appendix treats hops as the units of nonlinearity and runs as the contiguous chronological stretches between them.Source: §D.2 · p. 19
  • A telling with few hops must retain a long forward run, but can still have negative τ; only chronology has zero hops.For n = 6, (4, 5, 6, 1, 2, 3) has one hop and τ = −0.2.Source: §D.2 · p. 19
  • Scene-level retrograde reverses scenes while keeping events within each scene forward, whereas the event-level rule has the complete reversal as its unique solution.The authors report that 93.6% of RETROGRADE outputs are complete reversals; prompts do not mark scenes, so they specify the event-level rule.Source: §D.2 · p. 20

D.3 Pinned pairs

  • Pinned event pairs give partial credit a starting share λ, while exact-match probability for an exchangeable set of w events is at most 1/w!.Beyond the pinned share, a solver must earn score on unpinned pairs.Source: §D.3 · p. 20
  • The paper-at-a-glance table marks rows revised by the reader study.The appendix’s pinned-pairs discussion also notes that its empirical corollary has limits: the floor does not identify entailment, and the graph bound assumes unchanged accuracy on unpinned pairs.Source: Table 5 · p. 21, §D.3 · p. 21
  • The no-story floor bounds the share of event-pair order entailed by descriptions from above, but does not identify that share.The appendix reports a no-story solver score of 0.849 and a gain of +0.035; removing dates shifts the floor by −0.005. The graph’s free agreement gain is capped at 0.111 τ under the stated unchanged-accuracy assumption.Source: §D.3 · pp. 20–21

E.1 Where a reversal happens

The reversal analysis finds that models instructed to produce RETROGRADE reverse events across scenes while also reversing events within scenes, unlike published works. In the reader study, reader preferences distinguish conditions that the judge sometimes treats as ties.

E.1 Where a reversal happens

  • Published RETROGRADE tellings reverse 94.0% of between-scene pairs and 0.0% of within-scene pairs, while cued outputs reverse 98.8% and 98.0%, respectively.The 196 cued outputs reverse every pair in 93.9% of cases, indicating that models continue reversing inside scenes rather than stopping at scene boundaries.Source: §E.1 · pp. 21–22

E.1 Where a reversal happens; F The reader study in full

  • The instructed telling loses most heavily on ACHRONY and ZIGZAG, while graph guidance has its largest advantage over supplied told order on ZIGZAG and SYLLEPSIS.The authors caution that per-category counts are small and interpret the ordering rather than individual rates; Figure 10 reports overall reader and judge decisive verdict shares and shows judge ties where readers found clear differences.Source: §E.1 · pp. 22–23, Figure 10 · p. 23

F.1 Do readers agree with each other?

Reader judgments are stable enough for comparison but remain subjective, and human reconstruction is weaker than a no-story solver on the same works. Close readings favor graph-guided stories, while the paper qualifies several measures and reports corpus-build checks.

F.1 Do readers agree with each other?

  • Between-reader agreement is low on aesthetic judgments, although the ordering of better-story comparisons is stable across all eight comparison kinds.For better story, Krippendorff’s α is 0.415, rising to 0.595 when ties are excluded; agreement on harder to follow and better ending is lower.Source: §F.1 · p. 23

F.2 Reconstruction, done by a person

  • Readers recover the chronology exactly in 2.7% of readings, while models reach τrec = 0.699 on the same 14 stories and the no-story floor is 0.833.The comparison does not establish that models read nonlinear narratives better: readers worked from prose and event cards, while models received segmented tellings.Source: §F.2 · pp. 24–25
  • Reader and model difficulty profiles across seven shapes show no rank correlation (ρ = 0.18, p = 0.70), so category-level model difficulty is not a proxy for reader difficulty.The authors call this weak evidence because the comparison has only seven categories.Source: §F.2 · p. 25

F.3 Three checks on the design itself

  • Reader confidence tracks how divided readers are on an item (r = +0.26, p < 0.0001), but does not indicate how good an individual reader is.Results also remain similar across stricter reader subsets; the largest movement is about 7 points.Source: §F.3 · p. 25

F.4 What the close readings add

  • Full-graph readings receive higher composite PDS than instructed prose (3.90 vs. 3.31; p < 0.0001, d = 0.87), while the judge scores the same texts above readers overall.Band hit does not predict rated depth, and reader-rated depth and nonlinearity are unrelated across the outputs.Source: §F.4 · p. 26
  • Instructed prose has more flagged consistency problems and is more often guessed machine-written, but these gaps cannot be attributed to the instruction because the comparison uses different generator samples.Readers report 0.99 consistency problems per instructed reading versus 0.70 and 0.73 elsewhere; the machine-written guesses are 55% versus 38% and 36%.Source: §F.4 · p. 26

G Reporting recommendations

  • The reporting recommendations call for a solver-specific no-story floor, exact-match reconstruction, length control for preference comparisons, and reader checks of judge ties.The authors also warn that structural metrics can reward literal rule compliance rather than resemblance to published practice or reader preference.Source: G Reporting recommendations · pp. 26–27

H What the corpus build checks

  • Corpus-build checks verify event permutations, chronological preconditions, graph-edge constraints, and one-time raising and resolution of each reader question.The checks cannot establish whether another annotator would segment events or assign structural labels the same way; several shape labels are especially open to disagreement.Source: H What the corpus build checks · pp. 27–28

I Notes on the benchmark grid

These notes qualify how benchmark scores should be read: script baselines expose specification effects, band hit can be permissive, and some headline comparisons have important scope limits.

I Notes on the benchmark grid

  • A held-out published work falls inside its peers’ band 64 of 83 times (77%), while band hit can still reward broad or incomplete shape matches.ZIGZAG has the highest band hit but only 0.396 agreement with its gold telling among the three shapes requiring judgment.Source: I Notes on the benchmark grid · p. 28
  • The scene-level reversal script hits the RETROGRADE band 6 times in ten, compared with 1.1% for models, showing the missed step can be produced without a language model when scene boundaries are supplied.The exact identity and reversal miss the CHRONICLE and RETROGRADE bands despite being accepted by their prompts.Source: I Notes on the benchmark grid · p. 28
  • Among 1,569 valid cued tellings, exact reversals and identities concentrate in the two rule-shaped categories, while pooled band hit without them is 62.1%.There are 181 exact reversals and 223 exact identities; other five shapes account for 5 reversals and 54 identities across 1,191 requests.Source: I Notes on the benchmark grid · p. 28
  • The no-story reconstruction floor comes from three strong solvers, so comparing weaker models against it does not give each model its own floor.Removing absolute time anchors changes that floor by only -0.005 τ, but causal phrasing, event selection, stereotyped wording, and prior familiarity remain possible sources.Source: I Notes on the benchmark grid · pp. 28–29
  • The three GPT-5.6 models refuse 12.7–13.0% of attempts, including 17 works refused by all three, making affected means non-comparable without a common subset.The authors frame this as a benchmark-design issue and report that comparisons on commonly answered works leave the Table 2 ordering essentially unchanged.Source: I Notes on the benchmark grid · p. 29

J Retrospective coherence gain

The retrospective coherence gain measure is constrained by a ceiling and by how it compares event sets, so its table is exploratory. A paired guess task adds a separate signal about withheld causes.

J Retrospective coherence gain

  • Retrospective coherence gain compares reconstruction before a designated reveal with reconstruction for the full telling, but its floor caps the measure near 0.12 by construction.The paired five-option guess task uses distractors from events not yet narrated; only the pair distinguishes deliberate withholding from confusion.Source: J Retrospective coherence gain · p. 29
  • The conditions do not separate on RCG, and the comparison is exploratory because prefix and full-telling scores cover different event sets.The figure’s early-disclosure bars also include a work with no designated reveal and a chronicle whose explanation is third in chronological order.Source: Table 14 · p. 29, J Retrospective coherence gain · p. 29, Figure 12 · p. 29
  • RETROGRADE has RCG = 0.003, as predicted for a structure that inverts rather than withholds, while the ceiling makes the measure inert for the other six categories.Source: J Retrospective coherence gain · pp. 29–30
  • The graph condition lowers guess accuracy from 77.3% to 67.4% without turning concealment into payoff, while supplying the told order yields 86.5% predictability.Source: J Retrospective coherence gain · p. 30
  • After excluding two works that inflate early-disclosure counts, early disclosure is 8.3% without a graph, 8.5% with one, and zero when the told order is supplied (p = 0.125).The graph-versus-told-order difference is partly by construction because the latter condition receives the human order.Source: J Retrospective coherence gain · p. 30

K Structural classification

This section documents the benchmark’s reproducibility setup and release, including task prompts, the reader instrument, and limitations in story alignment and reader-pair selection. It also examines structural labels and their distinction, and proves links between telling order and runs, increasing subsequences, and Kendall’s τ.

K Structural classification

  • Across models, shape naming tracks T1 band hit (ρ = 0.717) and T3 τrec (ρ = 0.747), but floor-dominated τrec is a weak stand-in for reading a telling.T1 band hit and T3 τrec also correlate at ρ = 0.557, so the tasks rank models alike. ANALEPSIS and CHRONICLE are the only labels recovered above 0.6; ACHRONY is recognized 7.1% of the time, and the corpus excludes its definition of unrecoverable chronology, which may also contribute to the ANALEPSIS sink.Source: K Structural classification · p. 30, Figure 13 · p. 30

L Task and judge prompts

  • The evaluation uses distinct prompts for ordering, reconstruction, classification, and prose tasks, with T1 and Ta 5 receiving one category definition while T4 receives all seven.For prose generation, G1 supplies a graph encoding event positions, world state, character knowledge, and questions and resolutions; G2 instead supplies the told order, and A1–A5 remove graph layers. The judge-side aligner assigns each event its first-mention position and is not manually checked, so its projection error is unbounded by the reported checks.Source: L Task and judge prompts · pp. 30–33

M The reader instrument

  • The reader study combined blind story comparisons with a close reading and shape-comprehension questions, while excluding pairs with texts under 250 words or without terminal punctuation.This filtering removed 39 pairs in the three main comparisons and 14 in the layer-removal set; refusals and truncations were excluded to avoid comparing broken outputs with complete stories.Source: M The reader instrument · pp. 33–34

N Reproduction

  • Reproducibility is supported by generating tables, figures, and inline measurements from stored results, with fixed-seed bootstrap intervals and exact-binomial preference tests.The length-adjusted rates use generalized estimating equations clustered by reader; refitting with stories as clusters changes no conclusion, while a crossed reader-and-story random-effects model has not been fit.Source: N Reproduction · p. 34
  • The 19-model ordering grid cost $33 and took about five hours at six-way concurrency; scoring, tables, figures, and the PDF rebuild offline and deterministically.Generation used one OpenRouter harness with fixed decoding; providers were not pinned, and failed permutation parses were not repaired or pooled with refusals.Source: N Reproduction · p. 34
  • The MIT-licensed release includes event and scene data, labels and rationales, graph materials, stored replies and metadata, reader responses, and rebuild scripts.It also includes the reasoning-effort ablation, common-subset comparisons, token counts, and per-model costs.Source: N Reproduction · p. 35

O Proofs of §D

  • A telling has zero hops if and only if it is chronological; its longest increasing subsequence is at least the largest run’s share of events, with equality when runs are told in reverse chronological order.The run-based lower bound also implies a longest increasing subsequence of at least 1/(h + 1) of the events for h hops.Source: O Proofs of §D · p. 35
  • Kendall’s τ equals +1 only for the identity telling; swapping any two events makes at least one pair discordant and lowers τ.The proof notes that swapping ranks a < b makes 2(b − a) − 1 pairs discordant.Source: O Proofs of §D · p. 36

Related papers

Loading 2610.21468v1…