PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Story Reasoning and Generation

Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, Yejin Choi

arXiv:1909.04076v2cs.CLcs.AI

TL;DR

Counterfactual narrative reasoning lacks a benchmark that directly tests how alternative events change story outcomes. The paper introduces Counterfactual Story Rewriting and TIMETRAVEL, then finds that neural language models generally struggle to produce fully consistent revised endings.

  • Problem

    Existing benchmarks do not explicitly evaluate narrative reasoning and revision under counterfactual conditions.

  • Method

    The paper defines a minimal-edit story-rewriting task and constructs TIMETRAVEL with counterfactual revisions, unrevised branches, and language-model baselines.

  • Results

    Neural language models show promise but generally have difficulty rewriting counterfactual consequences with full consistency.

  • Takeaways & Limitations

    Counterfactual story rewriting provides a testbed for integrating stronger reasoning capabilities into neural language models.

  • Takeaways & Limitations

    Human scoring agreement is moderate, with an average Krippendorff alpha of 0.42 across the three questions.

Abstract

from arXiv · show

Counterfactual reasoning requires predicting how alternative events, contrary to what actually happened, might have resulted in different outcomes. Despite being considered a necessary component of AI-complete systems, few resources have been developed for evaluating counterfactual reasoning in narratives. In this paper, we propose Counterfactual Story Rewriting: given an original story and an intervening counterfactual event, the task is to minimally revise the story to make it compatible with the given counterfactual event. Solving this task will require deep understanding of causal narrative chains and counterfactual invariance, and integration of such story reasoning capabilities into conditional language generation models. We present TimeTravel, a new dataset of 29,849 counterfactual rewritings, each with the original story, a counterfactual event, and human-generated revision of the original story compatible with the counterfactual event. Additionally, we include 80,115 counterfactual "branches" without a rewritten storyline to support future work on semi- or un-supervised approaches to counterfactual story rewriting. Finally, we evaluate the counterfactual rewriting capacities of several competitive baselines based on pretrained language models, and assess whether common overlap and model-based automatic metrics for text generation correlate well with human scores for counterfactual rewriting.

1 Introduction

The paper introduces counterfactual story rewriting, which evaluates whether systems can minimally revise future events to remain consistent with an alternative condition. It contributes the TIMETRAVEL dataset and finds that pretrained language-model baselines generally struggle with fully consistent rewrites.

  • Counterfactual reasoning requires predicting causal changes in future events given an alternative condition.
  • No existing benchmark explicitly focuses on modifying narrative futures in response to counterfactual conditions.
  • Counterfactual Story Rewriting requires minimally editing an original story while preserving counterfactually invariant events and restoring narrative consistency.
  • TIMETRAVEL contains 29,849 counterfactual revisions and 80,115 additional counterfactual branches without rewritten storylines.
  • Pretrained language-model baselines capture some counterfactual reasoning but generally struggle to rewrite endings with full consistency.

2 Background

The background situates counterfactual reasoning as an alternative-event capability that remains difficult for language models and is poorly captured by existing discrimination-oriented or structured approaches. The paper therefore studies causal consequences directly in unstructured narrative text.

  • Counterfactual reasoning considers alternative possibilities that diverge from observed narratives.
  • Existing language models often struggle to distinguish reasonable from unreasonable counterfactuals and may exploit dataset artifacts.
  • Unlike script learning, the paper studies counterfactual consequences directly in unstructured text rather than relying on templatized causal structures.
  • Counterfactual events function as causal interventions that require narrative changes consistent with shared background knowledge.
  • Compared with controllable generation over predefined labels such as sentiment, counterfactual rewriting changes complex and diverse causal dimensions of stories.

3 Counterfactual Story Rewriting

Counterfactual Story Rewriting changes a story’s second event and requires minimal edits to its ending so the resulting narrative remains coherent. TIMETRAVEL constructs these examples from ROCStories through separate counterfactual-event and continuation-rewriting stages.

  • Each example contains a five-sentence story whose first sentence establishes the premise, second sentence gives context, and final three sentences form the ending.
  • A counterfactual second sentence can invalidate the original ending, so the model must minimally rewrite the final three sentences for coherence.
  • The minimum-edit requirement tests whether models identify narrative mechanisms and counterfactual invariance rather than rely on spurious correlations.
  • TIMETRAVEL is built from ROCStories, which provides 98,159 training stories and 3,742 evaluation stories.
  • Workers first create topical, premise-relevant, storyline-changing counterfactual events and then other workers minimally rewrite the ending.
  • The dataset includes 16,752 training examples with rewritten endings and 80,115 additional training counterfactual contexts without rewritten endings.

4 Learning a Counterfactual Rewriter

The paper trains GPT-family models for counterfactual story rewriting using zero-shot, fine-tuned, counterfactual, reconstruction, and supervised objectives. These approaches condition generation on story premises and counterfactual events while encouraging stylistic consistency and minimal revision of original endings.

  • Model Settings: GPT and GPT-2 variants serve as baselines, with 14 model variants trained using multiple objectives.The models follow GPT and GPT-2 training setups and are implemented with Texar.
  • Unsupervised Training: Zero-shot models generate rewritten endings from pretrained representations without training on TIMETRAVEL examples.At test time, the models receive the premise and counterfactual sentence and generate the rewritten outcome.
  • Unsupervised Training: Fine-tuning on ROCStories maximizes story likelihood to adapt models to the target distribution and encourage consistent writing style.The premise and counterfactual sentence remain model inputs during generation.
  • Unsupervised Training: The counterfactual objective fits models to counterfactual sentences conditioned on premise sentences, using 81,407 additional counterfactual training sentences.This objective expands training beyond the original story likelihood objective.
  • Reconstruction + Counterfactual: Reconstruction training conditions on the original story and a masked context to reproduce the original ending, then replaces the mask with the counterfactual event at test time.This design encourages generations similar to the original ending while allowing counterfactual information to alter the continuation.
  • Supervised Training: Supervised training uses 16,752 instances with human-rewritten endings and maximizes their ground-truth log-likelihood.The input includes the full story and counterfactual information before generating the rewritten ending.

5 Human Study of Rewritten Sentences

Human evaluations compare model rewrites on premise detail, plot relevance, and counterfactual adherence, showing gains from model scale and domain adaptation but persistent difficulty with counterfactual consistency.

  • 5.1 Rewritten Sentence Scoring: Human raters assessed rewritten endings for premise details, plot relevance, and respect for counterfactual changes using 3-point Likert scales.The study evaluated outputs from 14 models and included gold human-annotated endings as an upper-bound reference.
  • Model Size and Pretraining Data: Larger models generally outperform smaller models, although GPT appears to outperform GPT2-S on counterfactual adherence.The comparison suggests that both parameter count and pretraining domain affect performance across evaluation questions.
  • Domain Adaptation: Fine-tuning on ROCStories consistently improves counterfactual relevance, especially for models with fewer parameters.Zero-shot systems tend to produce more creative rewritings that are less tied to the original story.
  • Domain Adaptation: Fine-tuning with the larger counterfactual set does not appear to improve rewriting endings that relate well to the counterfactuals.
  • Supervised vs. Unsupervised Learning: Supervised training improves plot adherence but not counterfactual adherence, suggesting models may copy or paraphrase the original ending without incorporating the counterfactual.Adding more paired story data alone does not trivially solve counterfactual rewriting.
  • 5.2 Pairwise Model Preference: Pairwise human preferences favor the best model on premise consistency, but its advantage is less consistent for plot relevance and counterfactual adherence.Performance on one evaluated dimension often trades off against performance on another.

6 Challenges for Automatic Metrics

The paper compares overlap- and model-based automatic metrics with human judgments, finding that most metrics track premise and plot quality but poorly measure counterfactual understanding.

  • 6.1 Automatic Metrics: BLEU and ROUGE-L rely on textual overlap, while model-based metrics use embeddings or trained models to assess sequence similarity.BLEU counts overlapping n-grams, and ROUGE-L measures longest common subsequence length.
  • 6.2 Human Correlation with Metrics: The evaluation computes Pearson correlations between automatic metric scores and human judgments on 800 validation examples.The examples combine gold annotations with outputs from five GPT2-M variants and use the same three human evaluation questions.
  • 6.2 Human Correlation with Metrics: Automatic metrics correlate reasonably with human scores for premise adherence and plot relevance but correlate negatively with counterfactual adherence.Their typical higher-is-better interpretation therefore provides poor measurement of counterfactual understanding.
  • 6.2 Human Correlation with Metrics: Only BERTScore metrics show positive correlation with human counterfactual-understanding scores, and that correlation is weak.The paper therefore treats BERTScore as usable across all three properties, while noting difficulty distinguishing models.

7 Conclusion

The paper introduces Counterfactual Story Rewriting and the TIMETRAVEL dataset, then shows that neural language models can handle some cases but struggle to maintain full consistency after counterfactual changes.

  • 7 Conclusion: Counterfactual Story Rewriting evaluates whether systems can generate story consequences consistent with a counterfactual condition.The task challenges language understanding and generation systems with counterfactual reasoning.
  • 7 Conclusion: TIMETRAVEL provides nearly 30,000 counterfactual revisions and more than 100,000 counterfactual sentences for research on narrative counterfactual reasoning.
  • 7 Conclusion: Across zero-shot, unsupervised, and supervised settings, neural language models show promise but generally struggle to rewrite counterfactual consequences with full consistency.The conclusion calls for more focused research integrating reasoning capabilities into neural language models.

A Crowdsourcing Details

The study uses two groups of crowdworkers to create counterfactual alternatives and coherent, logically consistent story endings, with separate per-instance payments for each step.

  • Quality Control: Two worker groups handle separate stages: creating counterfactual alternatives and writing coherent, logically consistent revised endings.Workers are required to change only the original story arc needed to regain narrative consistency.
  • Cost: $0.24 is paid per instance for Step 1, while $0.36 is paid per instance for Step 2.

B Training Hyperparameters

The GPT and GPT-2 baselines use BPE tokenization, 128-token maximum sequences, Adam optimization, and early stopping over 10K iterations, with model-specific vocabulary sizes and learning rates.

  • GPT2: GPT-2 uses a 50,257-token BPE vocabulary, 128-token sequences, a 10^-5 learning rate, minibatches of 2, and 10K training iterations.Inference uses top-k sampling following Radford et al. (2019).
  • Shared Settings: Both models use Adam optimization, early stopping on validation performance, and the same maximum sequence length.
  • GPT: GPT uses a 40,000-token BPE vocabulary, 128-token sequences, a 6.25^-5 learning rate, and 10K training iterations.The model is a 12-layer decoder-only transformer with masked self-attention heads, and inference uses the GPT-2 generation procedure.
Loading 1909.04076v2…