PaperlayerView exact revision on arXiv

Source-linked AI summary

AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation

Chujie Zheng, Sahand Sabour, Jiaxin Wen, Zheng Zhang, Minlie Huang

arXiv:2202.13047v3cs.CL

TL;DR

Crowdsourced dialogue corpora are costly to curate, limiting their scale and topic coverage. AugESC uses fine-tuned language models to complete dialogues from available posts and applies heuristic postprocessing. Human evaluations report stronger performance than augmentation baselines, comparable dialogue quality to ESConv, and improved open-topic generalization after post-training.

  • Problem

    Crowdsourced dialogue curation is expensive, restricting corpus scale and topic coverage that may matter for downstream open-domain generalization.

    Source: §1 · p. 1

  • Method

    The authors fine-tune a language model to complete full emotional-support dialogues from collected posts, then postprocess the generations with heuristics.

    Source: §1 · p. 2

  • Results

    Human evaluations found the approach superior to strong augmentation baselines, AugESC comparable in dialogue quality to ESConv, and post-training on AugESC improved open-topic generalization.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The authors conclude that large language models can improve data-scarce tasks, especially complex open-domain dialogue tasks.

    Source: §7 · p. 9

    Heuristic-based postprocessing may not guarantee perfect dialogue quality, and further quality refinement remains future work.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Crowdsourced dialogue corpora are usually limited in scale and topic coverage due to the expensive cost of data curation. This would hinder the generalization of downstream dialogue models to open-domain topics. In this work, we leverage large language models for dialogue augmentation in the task of emotional support conversation (ESC). By treating dialogue augmentation as a dialogue completion task, we prompt a fine-tuned language model to complete full dialogues from available dialogue posts of various topics, which are then postprocessed based on heuristics. Applying this approach, we construct AugESC, an augmented dataset for the ESC task, which largely extends the scale and topic coverage of the crowdsourced ESConv corpus. Through comprehensive human evaluation, we demonstrate that our approach is superior to strong baselines of dialogue augmentation and that AugESC has comparable dialogue quality to the crowdsourced corpus. We also conduct human interactive evaluation and prove that post-training on AugESC improves downstream dialogue models' generalization ability to open-domain topics. These results suggest the utility of AugESC and highlight the potential of large language models in improving data-scarce dialogue generation tasks.

1 Introduction

Crowdsourced emotional-support dialogue data is costly to curate, limiting its scale and topic coverage. AugESC uses language-model dialogue completion to expand ESConv, with human evaluations supporting dialogue quality and downstream generalization benefits.

1 Introduction

  • Crowdsourcing is time-consuming and requires worker training and screening, constraining dialogue-corpus scale and topic coverage and potentially limiting open-topic generalization.Source: §1 · p. 1
  • The approach treats augmentation as dialogue completion: a fine-tuned language model completes dialogues from collected posts, followed by heuristic postprocessing.The authors fine-tune GPT-J on ESConv samples and use EmpatheticDialogues posts as starting points.Source: §1 · p. 2
  • AUGESC is 45x the scale of ESConv and substantially extends its topic coverage.Source: §1 · p. 2
  • Human evaluations found the approach superior to strong augmentation baselines, AugESC comparable in dialogue quality to ESConv, and post-training improved open-topic generalization.The paper reports these outcomes from comprehensive human evaluation and human interactive evaluation.Source: §1 · p. 2

2 Related Work

Prior work applies language models to data augmentation and knowledge acquisition, while dialogue augmentation has used simulated worker interactions. This paper focuses on completion-based augmentation and evaluation of dialogue quality for emotional-support conversations.

Data Augmentation with Language Models

  • Language models have generated data for textual similarity, classification, language understanding, commonsense knowledge, and natural-language inference tasks.Examples include Schick and Schütze (2021), Wang et al. (2021), West et al. (2022), and Liu et al. (2022).Source: §2 · p. 2
  • Unlike simulated worker interactions (Mohapatra et al., 2020; Kim et al., 2022), this work frames dialogue augmentation as completion and reports better effectiveness and efficiency.Source: §2 · p. 2

Emotional Support Conversation (ESC)

  • ESC requires peer supporters to help seekers reduce daily-life distress, and its high-quality data is difficult to obtain through crowdsourcing or substitute with empathetic dialogue data.ESConv contains 1.3K sessions across 13 topic categories.Source: §2 · p. 2

3 Methodology

The method fine-tunes an autoregressive language model, prompts it with dialogue posts to complete conversations, and filters generated outputs using heuristics. These steps yield 65K retained dialogues, while the filtering addresses generation failures, self-reinforcing patterns, and distributional gaps with ESConv.

3 Methodology

  • The three-stage pipeline fine-tunes a model, prompts it to complete dialogues, and postprocesses the resulting conversations with heuristics.Source: §3 · pp. 2–3, Figure 2 · p. 3

3.1 Fine-tuning Language Model

  • A fine-tuned autoregressive language model learns to complete a full dialogue from its starting utterance, using task instructions and Human/AI role prompts.The implementation uses 6B-parameter GPT-J, fine-tuned for one epoch on 100 sampled ESConv sessions.Source: §3.1 · p. 3

3.2 Dialogue Completion

  • Collected dialogue posts serve as starting utterances for generating complete conversations; the authors use negative-emotion EmpatheticDialogues posts and generate 89,500 raw texts.They traversed 8,950 posts for 10 epochs, using nucleus sampling with p = 0.9.Source: §3.2 · pp. 3–4

3.3 Postprocessing

  • Postprocessing retained 65K of 89,500 raw generations (72.7%); filtering targets failed outputs, self-reinforcing turn patterns, and distributional gaps with ESConv.The reported failures include 0.3% non-dialogue, 3.4% unfinished generation, and 2.1% prompt leakage; unbalanced and consecutive turn counts occurred at 5.2% and 3.5%.Source: §3.3 · p. 4, Table 1 · p. 4
  • Heuristic postprocessing may not ensure perfect dialogue quality, so the authors assess quality with human evaluation and leave further refinement for future work.They note that reliable automatic evaluation of open-domain dialogue remains unavailable.Source: §3.3 · p. 4

4 Data Analysis

AUGESC broadens ESConv’s scale and daily-life topic coverage while maintaining close dialogue diversity. Its statistics also reflect a heuristic trade-off in speaker-turn balance.

4.1 Statistics

  • AUGESC contains 65K dialogue sessions and 1,738K utterances, roughly 50/45x ESConv’s scale, with heuristics keeping utterance number and length close.Seeker/supporter utterances shift from 15.4 vs. 13.6 in ESConv to 13.3 vs. 13.4 in AUGESC, trading off the self-reinforcement phenomenon.Source: §4.1 · pp. 4–5

4.3 Diversity Analysis

  • AUGESC has close inter-dialogue diversity to ESConv, while Figure 3’s PCA shows dataset-correlated content regions whose combination spans a broader range.The similarity comparison uses dialogue TF-IDF vectors; the visualization applies two-dimensional PCA.Source: Figure 3 · p. 5, §4.3 · p. 5

4.2 Topic Analysis

  • AUGESC adds broader daily-life topics such as car, dog, house, police, and money to ESConv’s COVID- and task-related coverage.The authors suggest this complements ESConv’s topics and can facilitate generalization to open-domain topics.Source: §4.2 · p. 5, Table 3 · p. 5

5 Evaluation for Dialogue Quality

The section evaluates augmented dialogue quality against simulated and crowdsourced baselines, then examines toxicity in augmented dialogues. Human ratings find strong quality relative to augmentation baselines, while toxicity assessment identifies measurement and filtering caveats.

5.1 Compared Methods

  • Simulated Chat models crowdworker interaction with separately trained seeker and supporter models, while the no-fine-tuning variant directly prompts a language model to complete dialogues.The compared set also includes crowdsourced ESConv as a human-human dialogue quality anchor.Source: §5.1 · p. 6

5.2 Evaluation Setups

  • Human evaluation rates support-specific and general dialogue qualities, including Informativeness, Understanding, Helpfulness, Consistency, Coherence, and Unsafety.Sixty college students rated 60 sessions per method, with three annotators per session on a 0–3 Likert scale; higher is better except for Unsafety.Source: §5.2 · p. 6

5.3 Results

  • Our approach produces the highest-quality augmented dialogues among the methods and scores comparably to ESConv, though Consistency and Coherence remain weaker.The human-evaluation table reports scores averaged across three annotators; the authors identify a remaining gap in Consistency and Coherence.Source: §5.3 · pp. 6–7, Table 4 · p. 6
  • Against Simulated Chat (GPT-J), our approach's advantage is slight, but it takes 60 vs. 80 seconds per session and uses one model instead of two.The authors also report that their approach slightly outperforms this same-base-model baseline.Source: §5.3 · p. 7
  • Our approach outperforms Simulated Chat (BlenderBot) on almost all metrics and is markedly better than the GPT-3 prompting variant on Understanding and Helpfulness.The authors attribute the GPT-3 comparison to fine-tuning's better adaptation to ESC.Source: §5.3 · p. 7

5.4 Toxicity Assessment

  • Perspective API scores show little toxicity in both ESConv and AUGESC, with AUGESC lower, while human-rated dialogue quality remains slightly below ESConv.The authors suggest a possible trade-off between reducing detected toxicity and improving dialogue quality.Source: §5.4 · p. 7
  • The authors apply no toxicity postprocessing, citing detector bias and the still-evolving identification of context-sensitive toxicity.They note that support may require discussing a help-seeker's emotional problem in depth.Source: §5.4 · p. 7

6 Evaluation for Data Utility

The section tests whether post-training on AUGESC helps smaller dialogue models generalize to open-domain emotional-support conversations. Human interactive evaluation reports improvement across all aspects, while in-domain automatic evaluation finds only marginal performance gaps.

6 Evaluation for Data Utility

  • The evaluation targets generalization to open-domain topics in the practical setting of improving smaller downstream models.The paper notes that deploying large language models is expensive.Source: §6 · pp. 7–8

6.1 Compared Models

  • One model is fine-tuned only on ESConv (1,100 dialogues, 2 epochs), while the other is further post-trained on AUGESC for 1 epoch, which facilitates better generalization to open-domain topics.Source: §6.1 · p. 8

6.2 Evaluation Setups

  • Participants compared both bots on the same emotional problem in unrestricted open-domain conversations lasting at least eight turns.The study collected 60 conversation pairs and asked participants to rate fluency, identification, comforting, suggestion, and overall preference.Source: §6.2 · p. 8

6.3 Results

  • AUGESC post-training significantly improves the dialogue model across all aspects of open-domain human interactive evaluation.The evaluation reports winning ratios, with gaps statistically significant across the listed aspects.Source: §6.3 · p. 8, Table 6 · p. 8, §6.2 · p. 8

6.4 In-domain Evaluation

  • On the in-domain ESConv test set, post-training on AUGESC has little influence on performance, with only marginal gaps.The authors conclude that improved open-domain generalization is compatible with maintaining underlying dialogue capability.Source: §6.4 · p. 8, Table 7 · p. 8

7 Conclusion

The paper frames dialogue augmentation as dialogue completion and releases AUGESC for emotional support conversation. Its evaluations report higher-quality augmentation than strong baselines, comparable quality to ESConv, and improved open-domain generalization after post-training.

7 Conclusion

  • Dialogue augmentation is formulated as a dialogue completion task.The approach is used to release AUGESC for emotional support conversation, extending ESConv’s scale and topic coverage.Source: §7 · pp. 8–9
  • Human evaluation found higher-quality augmented dialogues than strong baselines and quality comparable to crowdsourced ESConv.The conclusion reports both comparisons as outcomes of comprehensive empirical evaluation.Source: §7 · p. 9
  • Human interactive evaluation found that post-training on AUGESC improves downstream dialogue models’ generalization to open-domain topics.The conclusion describes the improvement as notable.Source: §7 · p. 9

Ethical Considerations

AUGESC’s construction uses public resources and does not involve human participants, while its human evaluation had ethical safeguards. The appendix notes possible content-quality issues and limits the support setting to peer interaction rather than professional counseling.

Ethical Considerations

  • AUGESC construction used public resources and involved no human participants or collection of personally identifying information.The separate human evaluation informed participants about the study and disclaimers, paid above $10 per hour, and received IRB approval.Source: Ethical Considerations · p. 9
  • AUGESC may contain toxic or biased content that automatic and human evaluation cannot fully assess; the authors restrict its use to academic research.They advise against real deployment, commercial purposes, and other non-academic use.Source: Ethical Considerations · p. 9
  • The study follows ESC protocols for peer or friend support, not professional counseling; those protocols should not be applied directly to fields requiring professional-researcher guidance.Psychological counseling is given as an example of such a field.Source: Ethical Considerations · p. 9

A.1 Dialogue Quality of AUGESC

  • Manual inspection found inconsistencies mainly in seeker-provided information, including a contradiction between a dog’s stated age and its years with the seeker.The passage notes that such inconsistency can be easy for people to detect, while models are prone to make these mistakes.Source: §A.1 · p. 12
  • Supporters sometimes shift to topics unrelated to the seeker’s emotional problem; the authors conjecture that limited personalized detail from seekers contributes to this drift.They suggest supporter-led questions or suggestions may then drive the conversation and reduce coherence.Source: §A.1 · p. 12

A.2 Generalization to Other Tasks or Models

The authors’ experiments cover ESC, and their model experiments use GPT-J. They present applications beyond ESC and use of larger language models as future possibilities rather than tested extensions.

A.2 Generalization to Other Tasks or Models

  • The study experiments only on ESC, focusing on augmented-dialogue quality analysis and utility evaluation.The authors leave broader applications, such as knowledge-grounded dialogue augmentation, to future work.Source: §A.2 · p. 12
  • The model experiments use GPT-J only; the authors expect larger language models may produce higher-quality augmented dialogues.They cite stronger performance from commercial models such as GPT-3 as motivation for this expectation.Source: §A.2 · p. 12

B Implementation Details

The implementation details cover GPT-J training and generation, heuristic filtering, and a directly prompted GPT-3 comparison. The authors say their heuristic thresholds were not systematically evaluated.

B Implementation Details

  • GPT-J was fine-tuned for one epoch on 100 ESConv sessions sampled evenly across 13 topics, then generated dialogues with nucleus sampling at p = 0.9.Fine-tuning used four Tesla V100 32GB GPUs for about one hour; generation used one such GPU and took about one minute per augmented dialogue.Source: B Implementation Details · p. 12

D Details of Directly Prompting GPT-3

  • 72.7% vs. 47.9% retention ratios accompany the finding that direct prompting of 175B GPT-3 was inferior to fine-tuning smaller GPT-J in controllability and dialogue quality.For GPT-3, the authors report 24.8% non-dialogue and 11.2% prompt-word leakage; they used 2K posts and sampled 60 dialogues for human evaluation.Source: Table 8 · p. 13, D Details of Directly Prompting GPT-3 · p. 13

C Details of Filtering Rules

  • Heuristic filters remove malformed and self-reinforcing generations and constrain dialogue lengths, but threshold effects were not systematically analyzed.Rules include more than 10 total utterances, no more than 2.5x as many utterances from one speaker, and at most 3 consecutive utterances from one speaker; thresholds draw on heuristics and ESConv statistics.Source: C Details of Filtering Rules · pp. 12–13

E Guideline of Dialogue Quality Evaluation

Figure 4 presents the human-evaluation guideline for dialogue quality. The examples illustrate problems in baseline-generated dialogues, including illogical content and limited empathy.

E Guideline of Dialogue Quality Evaluation

F Examples of Augmented Dialogues from Baselines

Loading 2202.13047v3…