PaperlayerView exact revision on arXiv

Source-linked AI summary

DialogueScript: Using Dialogue Agents to Produce a Script

Patrícia Schmidtová, Dávid Javorský, Christián Mikláš, Tomáš Musil, Rudolf Rosa, Ondřej Dušek

arXiv:2206.08425v1cs.CL

TL;DR

Long-form script generation remains difficult because fully automated systems can produce inconsistent or dull dialogue. DialogueScript uses sentiment-based character models and simulated dramatic networks, and evaluations report better performance than a vanilla GPT-2 baseline while NLI-Score indicates dialogue consistency.

  • Problem

    Existing open-ended story generators mainly target short stories, while longer fully automated scripts can be inconsistent or dull.

    Source: §1 · p. 1

  • Method

    DialogueScript clusters characters by prevailing sentiment, trains distinct character models, orchestrates them with dramatic networks, and evaluates outputs with NLI-Score.

    Source: §1 · p. 1, §5 · p. 4

  • Results

    Automatic and human evaluation shows that DialogueScript outperforms a vanilla GPT-2 baseline, while NLI-Score provides consistency information.

    Source: §1 · p. 1, §5 · p. 4, §4.3 · p. 4

  • Takeaways & Limitations

    The approach supports longer scripted-dialogue generation with distinct character types and an automatic consistency measure.

    Source: §5 · p. 4

    The systems tend to be more verbose than the baseline, which may make their outputs harder to read in human ranking.

    Source: §4.4 · p. 4

Abstract

from arXiv · show

We present a novel approach to generating scripts by using agents with different personality types. To manage character interaction in the script, we employ simulated dramatic networks. Automatic and human evaluation on multiple criteria shows that our approach outperforms a vanilla-GPT2-based baseline. We further introduce a new metric to evaluate dialogue consistency based on natural language inference and demonstrate its validity.

1 Introduction

DialogueScript targets longer scripted dialogues, where prior fully automated systems often produce inconsistent or dull text. It combines personality-based character models, dramatic-network dialogue management, and automatic and human evaluation against vanilla GPT-2.

  • Prior open-ended story generators largely produce short stories, while longer fully automated scripts often remain inconsistent or dull.Source: §1 · p. 1
  • DialogueScript clusters characters by prevailing utterance sentiment and trains distinct models for different character types.Source: §1 · p. 1
  • Dramatic networks manage interactions among multiple character models by determining dialogue order and character addressing.Source: §1 · p. 1
  • The approach is evaluated against vanilla GPT-2 using diversity and sentiment metrics, human judgments, and a new NLI-based consistency metric.Source: §1 · p. 1

2 Script Generation Approach

The generation approach combines sentiment-based character clustering, focus/other preprocessing, and dramatic-network parameters that control scripted interactions.

2.1 Character Clustering

  • Characters are grouped into positive, neutral, or negative clusters using prevailing sentiment across their individual dialogue turns.Source: §2.1 · pp. 1–2

2.2 Data Preprocessing for Language Models

  • Training replaces character names with focus or other labels and duplicates dialogues when multiple characters share a cluster.Source: §2.2 · p. 2

2.3 Simulating Dramatic Networks

  • Dramatic networks choose dialogue flow through centrality, loyalty, and reciprocity parameters that are updated during generation.Source: §2.3 · p. 2
  • Initial dramatic-network parameters are set empirically rather than estimated from existing plays, and generation assumes a single dialogue.Source: §2.3 · p. 2
  • Centrality increases after a character speaks, making final values reflect script significance, while loyalty increases toward previously addressed characters.Source: §2.3 · p. 2
  • Reciprocity starts at 95% and decays by a third after each line, controlling exchange length before the next speaker is selected by centrality.Source: §2.3 · p. 2
  • Dialogue termination is independent of character relations and fixed at 20% after each line.Source: §2.3 · p. 2

3 Evaluation Metrics

The paper evaluates long-form dialogue with diversity, personality, and human-judgment measures, and introduces NLI-Score to quantify consistency through sentence-level neutrality.

3 Evaluation Metrics

  • The evaluation combines corpus-based diversity statistics, sentiment-based personality consistency, human judgments, and a new automatic consistency metric.Source: §3 · p. 2, §3.1 · p. 3, §3.3 · p. 3

3.1 Automatic Metrics

  • Automatic diversity evaluation measures perplexity, generated word count, distinct unigrams, and distinct bigrams averaged over dialogues.Source: §3.1 · pp. 2–3
  • Personality consistency is assessed by measuring generated-utterance sentiment using the same clustering principle used for training.Source: §3.1 · p. 3

3.2 Human Evaluation

  • Human evaluation ranks systems overall and scores coherence, consistency, originality, and overall impression on 5-point Likert scales.Source: §3.2 · p. 3

3.3 NLI-Score: A Consistency Metric

  • NLI-Score uses the neutral probability from RoBERTa-large-mnli because neutrality indicates added information without contradiction, while entailment suggests repetition.Source: §3.3 · p. 3
  • To handle varying lengths, NLI-Score averages neutrality per added sentence by comparing each sentence with its preceding context.Source: §3.3 · p. 3

4 Experiments

The experiments compare DialogueScript variants with a vanilla GPT2-medium baseline using automatic diversity, sentiment, consistency, and human evaluations. DialogueScript systems outperform the baseline overall, while dramatic-network orchestration most often improves human rankings and absolute ratings.

4.1 DialogueScript Training

  • DialogueScript models were trained separately on positive, neutral, and negative clustered character data from movie and television scripts.The models used finetuned GPT2-small language models and training data from 1,276 movies and 786 television episodes.Source: §4.1 · p. 3

4.2 Compared Model Variants

  • The evaluation compared random character ordering, dramatic-network orchestration, and vanilla GPT2-medium, with baseline generations initialized by minimal prompts because unprompted GPT-2 produced HTML code.DialogueScript variants generated scripts from scratch, whereas the baseline required a one-sentence setting and three-character greeting.Source: §4.2 · p. 3, §4.3 · p. 3

4.3 Results

  • Both DialogueScript setups generated more diverse scripts than the GPT2-medium baseline, while NLI-Score was higher than the baseline for both approaches.The NLI-Score comparison indicates fewer detectable contradictions and repetitions in the DialogueScript outputs.Source: §4.3 · p. 4, Table 1 · p. 4
  • DialogueScript + DN was most frequently ranked best and least frequently worst, while both DialogueScript systems beat the baseline on every absolute human-rating criterion.DialogueScript + DN also scored better than base DialogueScript on all criteria except Coherence.Source: §4.3 · p. 4, Table 3 · p. 4, Table 4 · p. 4

5 Conclusion

The paper generates scripts by combining sentiment-based character models with simulated dramatic networks and introduces NLI-Score for consistency evaluation. Automatic and human evaluations show that the approach outperforms the baseline, while NLI-Score indicates overall output consistency.

5 Conclusion

  • DialogueScript combines three sentiment-based character models with simulated dramatic networks, and its automatic and human evaluations outperform a vanilla GPT-2 baseline.The paper also proposes NLI-Score to automatically evaluate generated-text consistency.Source: §5 · p. 4
Loading 2206.08425v1…