Source-linked AI summary
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, Yejin Choi
TL;DR
Open-domain social dialogue research lacks large-scale, high-quality everyday conversation data because such conversations are rarely available and existing collection approaches are costly or restricted. The paper introduces CO3, which contextualizes social commonsense to distill SODA from an LLM, then trains COSMO on it; SODA outperforms prior human-authored corpora on several qualities, while COSMO generalizes better to unseen dialogues and can exceed human-authored responses in evaluation.
Problem
Large-scale, high-quality everyday social dialogue data is scarce because conversations are rarely recorded, privacy restricts available messages, and crowdsourcing is costly.
Method
CO3 retrieves social commonsense, converts it into sentences and narratives, and uses GPT-3.5 to generate grounded multi-turn conversations for SODA.
Results
COSMO generalizes significantly better to unseen dialogues than existing models, while SODA surpasses human-authored corpora in consistency, specificity, and naturalness.
Takeaways & Limitations
SODA enables smaller dialogue agents with competitive capabilities and provides a large resource for open-domain dialogue research.
Takeaways & Limitations
SODA currently contains only two-party conversations, and its dialogue types are likely affected by the choice of GPT-3.5 as the source model.
Abstract
from arXiv · showhide
Data scarcity has been a long standing issue in the field of open-domain social dialogue. To quench this thirst, we present SODA: the first publicly available, million-scale high-quality social dialogue dataset. By contextualizing social commonsense knowledge from a knowledge graph, we are able to distill an exceptionally broad spectrum of social interactions from a large language model. Human evaluation shows that conversations in SODA are more consistent, specific, and (surprisingly) natural than those in prior human-authored datasets. Using SODA, we train COSMO: a generalizable conversation model that is significantly more natural and consistent on unseen datasets than best-performing conversation models (e.g., GODEL, BlenderBot-1, Koala, Vicuna). Experiments reveal COSMO is sometimes even preferred to the original human-written gold responses. Additionally, our results shed light on the distinction between knowledge-enriched conversations and natural social chitchats. We plan to make our data, model, and code public.
1 Introduction
SODA addresses the scarcity of large-scale, high-quality open-domain social dialogue data by distilling socially grounded conversations from an LLM. The resulting dataset supports COSMO, which generalizes strongly to unseen dialogues and can surpass human-authored responses in some evaluations.
- Collecting high-quality everyday social conversations at scale is difficult because conversations are rarely recorded, privacy restricts text-message use, and crowdsourcing is costly.
- 1.5 million dialogues and more than 11 million utterances make SODA the largest publicly available open-domain social conversation dataset.The dataset contains 300 million tokens and covers a wide variety of social interactions.
- Human evaluation shows SODA surpasses existing human-authored dialogue corpora in consistency, specificity, and naturalness.
- CO3 contextualizes social commonsense by transforming knowledge triples into narratives and then dialogues, maximizing diversity while minimizing nonsensical conversations.Sampling from an LLM without contexts produces dull conversations, whereas commonsense knowledge graphs cover broad everyday situations.
- COSMO wins by more than 40% on average in head-to-head comparisons against BlenderBot, Koala, and Vicuna on unseen conversations.COSMO also outperforms same-sized BlenderBot on its training dataset and is preferred to ground-truth responses in DailyDialog, despite not training on either corpus.
- SODA enables smaller dialogue agents with competitive capabilities and may improve the generalizability of dialogue research built on small datasets.The paper also identifies naturalness as a dimension where recent LLM-based conversational agents struggle.
CO3: A Contextualization Framework for Conversation Distillation using Commonsense
CO3 reverses the abstraction from social commonsense to conversation by retrieving symbolic knowledge, turning it into narratives, and generating grounded multi-turn dialogue. The framework uses social relations from Atomic10x and GPT-3.5 to produce varied conversations between inferred participants.
- CO3: A Contextualization Framework for Conversation Distillation using Commonsense: CO3 retrieves social commonsense from a symbolic knowledge graph, converts it into sentences and a narrative, then derives a grounded conversation.
- Inspiration Behind CO3: The framework reverses abstraction by unfolding symbolic social commonsense into rich narratives and conversations that could have encapsulated it.
- Retrieving Social Commonsense: Atomic10x represents everyday events and mental-state inferences as symbolic triples, and CO3 retrieves triples related to social rather than physical commonsense.
- Triple Form to Sentence Form: CO3 converts triples into simple sentences with relation-specific templates and replaces person variables with common U.S. names to make them more natural.
- Sentence Form to Narrative: GPT-3.5 expands sentence-form commonsense into a two- or three-sentence narrative containing plausible details beyond the prompt.
- Inferring Conversation Participants: For one-person triples, GPT-3.5 predicts the other interlocutor; the model then generates a fluent alternating multi-turn conversation using narrative and speaker prefixes.
- The current framework retrieves social commonsense while leaving physical and event-centered relations to future work.
SODA: A Million-scale Social Dialogue Dataset
SODA is a large-scale, machine-generated social dialogue dataset built by contextualizing commonsense knowledge and applying post-processing filters. It covers diverse social interactions, emotions, and topics, while human evaluations find it more natural and higher quality than prior datasets.
- Dataset construction: 1,486,896 conversations remain after filtering 68.9% of 2.2 million GPT-3.5-generated conversations.Filtering removes erroneous patterns, unsuitable turn counts, non-dyadic conversations, non-human speakers, unsafe content, and irrelevant commonsense instantiations.
- Human evaluation: Human raters judge SODA better than DailyDialog and BlendedSkillTalk across nearly all quality axes, with context dependence as the stated exception against BlendedSkillTalk.Evaluators specifically rate SODA’s conversational flow as significantly more natural.
- Dataset scope: 1.5 million commonsense knowledge triples provide distinct narrative contexts covering a wide range of social interactions.Each seed triple is converted into a social narrative that serves as a conversation topic.
- Content diversity: SODA has broader lexical diversity and a more balanced emotion distribution than the compared human-authored datasets.Its varied social situations produce a wider range of words, while emotion rankings remain similar to other human-authored dialogues.
- Content diversity: 385K conversations generated from 1.7K unique emotion descriptions give SODA rich and comparatively fine-grained emotional coverage.The bottom-up construction identifies the experiencing speaker and the event causing the emotion.
- Efficiency: Generating a dialogue from a commonsense triple costs about $0.02, while 10 queries take less than 2 minutes including filtering.The reported comparison is against dialogue crowdsourcing.
COSMO: A Socially Situated Conversation Model
COSMO is a conversation model trained on SODA’s contextual narratives, speaker instructions, and dialogue histories. It is designed to converse across varied social situations while supporting robustness when contextual inputs are absent.
- Model and training: COSMO generates a next utterance for a given role from a situation narrative and dialogue history.Its training target is modeled as p(r|n, i, c), using narrative n, speaker instruction i, and dialogue context c.
- Model and training: COSMO training combines contextual narratives, inferred-speaker perspective instructions, dialogue context, and target responses in a sequence-to-sequence format.ProsocialDialog is additionally included to help models handle sensitive situations.
- Model and training: COSMO is trained in 3B- and 11B-parameter versions on an LM-adapted T5 backbone.Narratives and role instructions are randomly dropped during training for robustness and generalizability to datasets without those inputs.
5 Generalizability of COSMO
COSMO is evaluated against established conversational agents using head-to-head human judgments across unseen and training-domain datasets. It generally achieves stronger naturalness, consistency, specificity, and overall preference, including comparisons with human-authored responses and larger language models.
- Evaluation setup: Human judges compare responses using naturalness, consistency, specificity, and overall quality in head-to-head evaluations.The study samples 100 test examples and uses three Amazon Mechanical Turk judges per comparison.
- Out-of-domain setting: COSMO outperforms BlenderBot-1, GODEL, Koala, and Vicuna across all evaluated aspects on unseen DailyDialog conversations.COSMO is smaller and trained on less data than several baselines, yet shows the largest performance gap in naturalness.
- Out-of-domain setting: Human judges prefer COSMO responses over original human-written DailyDialog ground-truth responses in the reported comparison.The comparison uses 200 ground-truth responses and suggests strong generalizability and naturalness on unseen conversations.
- In-domain setting: COSMO significantly outperforms BlenderBot on BlendedSkillTalk, even though BlenderBot was trained on that dataset.BlenderBot also performs poorly on SODA, indicating that SODA contains both novel patterns and patterns present in existing datasets.
- Comparison with large language models: COSMO performs on par overall with GPT-3.5 and ChatGPT on SODA, while being significantly more specific than its teacher.COSMO uses substantially fewer parameters than the compared large language models.
- Comparison with large language models: ChatGPT is judged more specific but significantly less natural than COSMO, highlighting a distinction between informative assistance and natural social conversation.The authors hypothesize that ChatGPT’s helpful-response training contributes to this difference.
6 Related Work
The paper positions CO3 against prior efforts that use language models to augment or synthesize dialogue data. Its distinguishing contribution is contextualizing commonsense knowledge graphs to generate narratives and full conversations at substantially larger scale.
- Language-model-based dialogue construction: Earlier studies used large language models to augment emotional-support, understanding, or out-of-domain dialogue datasets.Other work also used pseudo-labeling or counterfactual reasoning to modify dialogue responses or semantics.
- Novelty: CO3 is presented as the first approach to contextualize commonsense knowledge graphs into narratives and derive full conversations from scratch at large scale.The approach is intended to encompass a broad spectrum of social interactions.
7 Conclusion
The conclusion presents SODA as a large, broad social-dialogue resource and CO3 as its commonsense-contextualized distillation framework. COSMO trained on SODA generalizes better to unseen dialogues and can be preferred to existing human-authored responses.
- Conclusion: SODA is a million-scale dataset spanning an exceptionally wide range of social interactions and addressing open-domain dialogue data scarcity.The paper describes it as orders of magnitude larger than popular dialogue datasets and better across multiple quality aspects.
- Conclusion: CO3 distills conversations from a large language model by contextualizing commonsense knowledge.With SODA, the authors train COSMO as a conversation model.
- Conclusion: COSMO generalizes significantly better than existing models to unseen dialogues and generates responses preferred over ground-truth responses from an existing dataset.These are the paper’s headline conclusions about the model trained on SODA.
8 Limitations
The paper identifies limitations in safety filtering, dataset coverage, dialogue structure, annotator diversity, and dependence on GPT-3.5. It also describes precautions for mitigating demographic bias and harmful generated content.
- Bias mitigation: The name pool emphasizes common gender- and race-associated names, risking misrepresentation of marginalized groups despite diversification efforts.The construction uses names from the US Social Security Administration repository to increase representation diversity.
- Safety: Safety precautions may not catch all harmful content, and toxicity detectors or safety modules are not perfect.The paper specifically notes residual risks involving bias and private information in language-model-generated content.
- Dataset scope: SODA currently contains only two-party dialogues, although the framework also supports future exploration of multi-party dialogue generation.This constrains the dataset’s present conversational structure.
- Human evaluation: Annotator selection from one platform using specific filters may introduce unintended biases, motivating evaluation with greater annotator diversity.The authors identify broader annotator diversity as a future research direction.
- Dataset scope: SODA focuses on social chitchat grounded in social commonsense and therefore lacks conversations grounded in scientific knowledge or historical facts.The authors propose integrating other knowledge-grounded dialogue datasets in future work.
- Model dependence: The generated dialogue types may depend on GPT-3.5 as the source language model, and other source or base models could produce different content or response quality.The authors identify model diversification as a future direction.
A.2 Narrative →Conversation
The pipeline turns narratives into multi-turn conversations by inferring plausible interlocutors and prompting GPT-3.5 to continue dialogue grounded in the narrative.
- Narrative →Conversation: GPT-3.5 predicts a plausible interlocutor when a narrative does not specify both conversation participants.For example, the model predicted “her coach” for a narrative about Madeleine.
- Narrative →Conversation: The conversation-generation prompt asks GPT-3.5 to produce a long, in-depth, multi-turn exchange between the inferred speakers.The narrative is included as scene context before the dialogue prefix.
B.1 Post-processing the Conversations
Post-processing filters generated conversations for speaker validity, commonsense grounding, safety, and dataset quality before release.
- Speaker and content filtering: Speaker filtering checks names and lexical human indicators, then uses GPT-3.5 to classify unresolved speaker prefixes as human or non-human.The procedure combines deterministic matching with model-based classification.
- Speaker and content filtering: Commonsense validation asks whether the narrative contains the triple’s head event and whether the conversation contains its relation and tail.GPT-3.5 answers contextualized three-way questions with yes, no, or unknown ranked by conditional pointwise mutual information.
- Training details: COSMO training used TPU accelerators with effective batch size approximately 780 for 110K or 130K additional steps, depending on model size.The reported COSMO-3B/COSMO-11B training runs used constant learning rate .001.
D Experiment Details
Experiments compare COSMO with established dialogue models using human and GPT-4 evaluations, including zero-shot tests on unseen datasets.
- Automatic evaluation: GPT-4 automatic results align closely with human evaluation, but may favor COSMO because it was trained on GPT-3.5-generated text.The authors therefore treat the original human evaluation as more significant for overall model quality.
- Human evaluation: COSMO outperforms both Koala and Vicuna on all four response-quality criteria in zero-shot evaluation on unseen BlendedSkillTalk.The difference between COSMO and Vicuna is smaller than the difference between COSMO and Koala.
- Evaluation setup: Response comparisons use prompts that provide the narrative and dialogue context before asking models to generate the next utterance.The prompt for GPT-3.5 requests a natural and specific response; other models are limited to at most three sentences.
F Dialogue Dataset Descriptions
The paper situates SODA among established dialogue datasets, including DailyDialog, PersonaChat, EmpatheticDialogues, and Wizard of Wikipedia.
- Dataset descriptions: DailyDialog contains casual dialogue compiled from English language-learning websites.The paper uses DailyDialog for comparison with SODA.
- Dataset descriptions: PersonaChat consists of two-speaker conversations about getting to know one another using provided personas.EmpatheticDialogues instead focuses on one speaker demonstrating empathy for another’s emotions.
- Dataset descriptions: Wizard of Wikipedia contains conversations based on Wikipedia between a speaker eager to learn and a knowledgeable speaker.The supplied passage introduces this dataset alongside other established benchmarks.