PaperlayerView exact revision on arXiv

Source-linked AI summary

Plot-guided Adversarial Example Construction for Evaluating Open-domain Story Generation

Sarik Ghazarian, Zixi Liu, Akash SM, Ralph Weischedel, Aram Galstyan, Nanyun Peng

arXiv:2104.05801v2cs.CLcs.LG

TL;DR

Open-domain story generation lacks reliable automatic evaluation partly because heuristic negative examples inadequately represent implausible machine-generated texts. The paper manipulates structured story plots and applies adversarial filtering to construct more nuanced negatives, producing evaluation metrics that correlate better with human judgments. The approach is designed to improve the quality and realism of training data for plausibility evaluation.

  • Problem

    Reliable open-domain story evaluation is limited by the lack of high-quality negative examples for classifiers distinguishing plausible and implausible generations.

    Source: §1 · p. 1, §3 · p. 2, §3.1 · p. 3

  • Method

    The method injects incoherence sources into structured story plots, uses conditional generation to create implausible texts, and applies adversarial filtering to select challenging negatives.

    Source: §1 · p. 2

  • Results

    Evaluation metrics trained on the generated data correlate significantly better with human judgments than strong baselines.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Plot-level construction produces more realistic implausible texts for training automatic story-generation evaluation metrics.

    Source: §1 · p. 2, §6 · p. 9

    The training-data construction uses relatively benign human-written sources, so the resulting samples are unlikely to contain strong biases or abusive content.

    Source: §7 · p. 10

Abstract

from arXiv · show

With the recent advances of open-domain story generation, the lack of reliable automatic evaluation metrics becomes an increasingly imperative issue that hinders the fast development of story generation. According to conducted researches in this regard, learnable evaluation metrics have promised more accurate assessments by having higher correlations with human judgments. A critical bottleneck of obtaining a reliable learnable evaluation metric is the lack of high-quality training data for classifiers to efficiently distinguish plausible and implausible machine-generated stories. Previous works relied on \textit{heuristically manipulated} plausible examples to mimic possible system drawbacks such as repetition, contradiction, or irrelevant content in the text level, which can be \textit{unnatural} and \textit{oversimplify} the characteristics of implausible machine-generated stories. We propose to tackle these issues by generating a more comprehensive set of implausible stories using {\em plots}, which are structured representations of controllable factors used to generate stories. Since these plots are compact and structured, it is easier to manipulate them to generate text with targeted undesirable properties, while at the same time maintain the grammatical correctness and naturalness of the generated sentences. To improve the quality of generated implausible stories, we further apply the adversarial filtering procedure presented by \citet{zellers2018swag} to select a more nuanced set of implausible texts. Experiments show that the evaluation metrics trained on our generated data result in more reliable automatic assessments that correlate remarkably better with human judgments compared to the baselines.

1 Introduction

Reliable automatic evaluation is needed for open-domain story generation, but heuristic negative examples can be unnatural and oversimplified. This work constructs implausible stories through plot manipulation and adversarial filtering, yielding metrics that correlate better with human judgments.

  • Evaluation metrics are needed because reference-based measures such as BLEU and ROUGE often correlate poorly with human judgments in open-domain generation.Source: §1 · p. 1
  • Learning-based evaluators depend critically on high-quality plausible and implausible training examples, yet prior negative samples rely on heuristic text-level manipulations.Source: §1 · pp. 1–2
  • The proposed method injects incoherence sources into structured story plots, guiding generation models to produce natural implausible texts resembling machine-generated failures.Source: Figure 1 · p. 1, §1 · p. 2
  • Adversarial filtering selects more challenging negative samples from manipulated plots to improve training-data quality.Source: §1 · p. 2
  • Evaluation metrics trained on the generated data achieve significantly higher correlation with human judgments than strong baselines.Source: §1 · p. 2

2 Related Work

Prior automatic evaluation methods include non-learning-based metrics and neural classifiers trained on plausible and implausible texts. Learning-based approaches commonly construct negatives through heuristic sentence substitutions, limiting their adequacy for open-domain plausibility assessment.

  • Automatic generation evaluation methods include topic centrality, entity-transition measures, perplexity, and learning-based plausibility classifiers.Source: §2 · p. 2
  • Learning-based metrics typically use human-written texts as positives and heuristic substitutions to create negative training examples.Source: §2 · p. 2
  • UNION improves on earlier approaches with BERT and a broader set of heuristic negative samples, but heuristic simplicity remains inadequate for open-domain plausibility evaluation.Source: §2 · p. 2

3 Implausible Text Construction

The paper constructs implausible story examples by manipulating structured plots rather than text tokens, then uses adversarial filtering to select challenging, stylistically unbiased negatives.

3 Implausible Text Construction

  • The task treats implausible-text construction as binary classification data generation, seeking negative examples near the plausible–implausible decision boundary.Human-written stories provide positive examples, while controllably modified positives provide negatives.Source: §3 · pp. 2–3

3.1 Heuristic Negative Samples

  • Heuristic sentence, keyword, repetition, reordering, and negation manipulations can produce unnatural examples that inadequately represent machine-generated implausibility.These samples may be far from generated texts and therefore insufficiently broad or realistic.Source: §3.1 · p. 3

3.2 Proposed Approach

  • Plot-level perturbations inject concept-level incoherence while preserving grammatical naturalness through conditional generation and better matching inference-time machine-generated text.The resulting ManPlts data is designed to expose targeted undesirable properties without directly changing text tokens.Source: §3.2 · p. 3, §3.2.1 · p. 3

3.2.1 Plot Manipulations

  • Figure 2 shows manipulated plots passed through a generation model to produce implausible samples for evaluation, while Table 1 provides example generated stories.The visual examples connect each proposed manipulation to its generated textual outcome.Source: Figure 2 · p. 4, Table 1 · p. 5, §3.2.1 · p. 5
  • The method manipulates plots by reordering logically related concepts, inserting contradictory concepts, repeating plots across positions, and substituting random plots.These operations target ordering conflicts, contradictions, realistic repetition, and topical inconsistency.Source: §3.2.1 · pp. 4–5

3.2.2 Adversarial Filtering

  • Adversarial filtering generates N plot-conditioned implausible texts and selects A challenging, high-quality samples with reduced stylistic bias.Classifiers iteratively replace easy negatives with adversarial ones that are harder to distinguish from human-written texts.Source: §3.2.2 · pp. 5–6

4 Learnable Evaluation Models

The evaluation framework scores story plausibility with classifiers trained on human-written and constructed implausible texts, using pretrained models for both short and long inputs.

4 Learnable Evaluation Models

  • Plausibility is modeled as binary classification, with the predicted positive-class probability serving as each text’s plausibility score.Human-written texts are positives and constructed implausible stories are negatives.Source: §4 · p. 6

4.1 Fine-tuning Language Models

  • RoBERTa is fine-tuned from pretrained parameters to predict plausibility scores for the constructed evaluation dataset.The model is used for text classification in the plausibility task.Source: §4.1 · p. 6
  • Longformer extends plausibility evaluation to texts longer than 512 tokens using linear token-length complexity.This addresses RoBERTa’s length constraint for long-text evaluation.Source: §4.1 · p. 6

4.2 Baselines

  • UNION uses BERT with an auxiliary reconstruction objective and heuristic negatives, while SENTAVG averages LSTM sentence representations from GloVe embeddings before classification.These models serve as baselines for the plausibility evaluation task.Source: §4.2 · p. 6

5 Experiments

Experiments compare heuristic and plot-based negative sampling across short and long story datasets, using human plausibility judgments and cross-dataset testing. Plot manipulation, especially with adversarial filtering, yields stronger correlations and better robustness, while ablations identify reordering and repetition as especially important.

5.1 Datasets

  • The datasets span short ROCStories averaging 49.4 tokens and long Writing Prompt stories averaging 734.5 tokens, enabling evaluation across substantially different lengths and topics.ROCStories contain five-sentence commonsense narratives, whereas Writing Prompt contains abstract prompts paired with long forum stories.Source: §5.1 · pp. 6–7

5.2 Human Annotations

  • Human plausibility scores come from five annotators rating 150 human-written and 150 GPT2-generated stories on a 0–5 scale, with Spearman and Kendall correlations used for evaluation.Scores are averaged, normalized to 0–1, and collected with attention checks through Amazon Mechanical Turk.Source: §5.2 · pp. 7–8

5.4 Experimental results

  • Heuristic sentence and keyword substitutions yield the lowest correlations, while broader UNION heuristics improve training data but retain unnatural text-level artifacts.The reported explanation is that heuristic samples differ substantially in style and naturalness from machine-generated implausible texts.Source: §5.4 · p. 8
  • Plot-based MANPLTS and AF_MANPLTS training data produce higher human-correlation scores than heuristic alternatives, with statistically significant improvements.The comparison uses RoBERTa for ROCStories and Longformer for Writing Prompt; adversarial filtering further improves negative-sample quality.Source: §5.4 · p. 8, Table 5 · p. 8
  • AF_MANPLTS generalizes best across ROC and Writing Prompt, although correlations decline under distribution shift and ROC-to-WP transfer preserves performance better than the reverse.The decline is attributed to the datasets’ different structures and to implausibility formats in long Writing Prompt stories that are absent from shorter ROCStories.Source: §5.4 · p. 9, Table 6 · p. 9

5.4.1 Ablation Study

  • Removing any plot-manipulation technique lowers correlations, with the largest drops occurring when reordering or repeating plots are excluded.The ablation study fine-tunes Longformer on Writing Prompt data after excluding one technique at a time.Source: §5.4.1 · p. 9, Table 7 · p. 9

6 Conclusion

The paper addresses low human correlation in plausibility metrics trained on heuristic negatives by generating implausible stories through plot-level manipulation and adversarial filtering. The resulting examples are more realistic and yield more accurate evaluation metrics.

6 Conclusion

  • Plot-level incoherence guides generation toward negative samples that better resemble machine-generated incoherent texts than token-level heuristic alterations.The approach manipulates structured plots rather than directly modifying text.Source: §6 · p. 9
  • Adversarial filtering selects more challenging and refined negative samples, improving the quality of the evaluation training data.Source: §6 · p. 9
  • Metrics trained on the proposed negative examples correlate more highly with human judgments and provide more accurate plausibility evaluation.Source: §6 · p. 9

7 Ethics

The work addresses dataset-construction ethics, annotator privacy and compensation, and potential misuse of the resulting metrics. It reports benign source data and AMT procedures while leaving security-oriented attacks outside scope.

7 Ethics

  • The authors state that the work follows the ACM Code of Ethics and uses nonoffensive dataset construction.Source: §7 · pp. 9–10
  • Manipulating benign source plots may still create objectionable texts, and augmenting offensive texts as plausible samples could misuse the metrics; security defenses are out of scope.The authors note that such attacks could encourage generation of inappropriate stories and require security-oriented studies.Source: §7 · p. 10
  • The source stories prohibit profanity and inappropriate content, and the authors state that manipulated plots with BART are unlikely to produce strong biases or abusive content.Source: §7 · p. 10
  • AMT annotators rated story coherence from 0 to 5, with an average 25-minute HIT compensated at $6 based on a $13 hourly wage.Source: §7 · p. 10
  • The AMT procedure did not collect personal information, which the authors associate with reduced privacy, gender-bias, and IRB concerns.Source: §7 · p. 10
  • The metrics target the open-domain generation community and are intended to save time, cost, and human effort in assessing outputs.Source: §7 · p. 10
Loading 2104.05801v2…