Source-linked AI summary
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, Jason Weston
TL;DR
Chit-chat agents often lack specificity, consistent personality, and engaging conversation. The paper introduces profile-conditioned models and the PERSONA-CHAT dataset, showing improved next-utterance prediction and more engaging dialogue, while noting that balancing fluency, engagement, consistency, and persistent persona remains challenging.
Problem
Chit-chat models commonly lack specificity, consistent personality, and engaging responses, and available datasets do not adequately support general chit-chat research.
Method
The paper constructs PERSONA-CHAT and trains dialogue models to use their own and their partner’s textual persona information while asking and answering personal questions.
Results
Conditioning models on persona information improves next-utterance prediction, while PERSONA-CHAT-trained models are more engaging than models trained on movie and Twitter dialogue.
Takeaways & Limitations
The dataset and task support dialogue agents that maintain personalities, ask about users’ profiles, remember answers, and use them naturally in conversation.
Takeaways & Limitations
Balancing fluency, engagement, consistency, and a persistent persona remains a strong challenge for future research.
Abstract
from arXiv · showhide
Chit-chat models are known to have several problems: they lack specificity, do not display a consistent personality and are often not very captivating. In this work we present the task of making chit-chat more engaging by conditioning on profile information. We collect data and train models to (i) condition on their given profile information; and (ii) information about the person they are talking to, resulting in improved dialogues, as measured by next utterance prediction. Since (ii) is initially unknown our model is trained to engage its partner with personal topics, and we show the resulting dialogue can be used to predict profile information about the interlocutors.
1 Introduction
Generic chit-chat models often lack consistent personality, long-term memory, and specificity, making conversations unsatisfying. The paper addresses these weaknesses with persistent persona profiles, partner-persona information, and the PERSONA-CHAT dataset.
- Generic chit-chat models commonly lack consistent personality, explicit long-term memory, and specific responses.
- These weaknesses contribute to an unsatisfying conversational experience and are partly attributed to the lack of a suitable public chit-chat dataset.
- The paper conditions dialogue agents on configurable, persistent textual profiles to produce more personal, specific, consistent, and engaging responses.
- The same mechanism uses information about the dialogue partner, while training agents to ask and answer personal questions and build a model of the partner’s persona.
- PERSONA-CHAT contains 162,064 utterances between randomly paired crowdworkers acting according to assigned personas.
- Conditioning generative and ranking models on persona information improves next-utterance prediction, while human evaluations compare PERSONA-CHAT with OpenSubtitles- and Twitter-trained chit-chat models.
2 Related Work
Related dialogue research emphasizes goal-oriented systems, retrieval, and end-to-end neural generation, while chit-chat personalization remains comparatively underexplored. Prior personalization work used speaker embeddings, whereas this paper focuses on explicit profiles and learning to engage the partner.
- Traditional dialogue systems typically target labeled dialogue states and precisely defined user intents in functional, goal-oriented tasks.
- Non-goal-driven systems include hand-coded agents and information-retrieval models that rank responses by similarity to recent dialogue history.
- Generative recurrent models can produce novel coherent responses, but their memory-free design limits long-term coherence and persistent personality.
- Common chit-chat corpora include movie-script datasets such as OpenSubtitles and web-platform dialogue from Reddit and Twitter.
- Personalization has received limited attention, and earlier Twitter work represented speaker characteristics with distributed embeddings that improved output for the same speaker.
- Unlike that work, this paper uses explicit profile information and aims to engage the other speaker by getting to know them.
3 The PERSONA-CHAT Dataset
PERSONA-CHAT is a crowd-sourced dialogue dataset in which speakers condition conversations on assigned profiles and try to get to know one another. Rewritten personas reduce trivial word overlap, creating more challenging personalization tasks.
- PERSONA-CHAT collects dialogue in which paired speakers condition their conversations on provided profiles.
- The dataset includes 1,155 possible personas, each containing at least five profile sentences, with separate validation and test personas.
- Revised Personas: Additional persona versions use rephrases, generalizations, or specializations to avoid models exploiting trivial word overlap.
- Persona Chat: Persona chat contains 162,064 utterances across 10,907 dialogues, including held-out validation and test sets.
- Personas: Workers created five-sentence character descriptions using short, natural topics of human interest rather than real personal information.
- Persona Chat: Workers were instructed to chat naturally, ask and answer questions, avoid copying profiles, and produce dialogues with a minimum randomly selected length of 6–8 turns per speaker.
- Tasks: The dataset supports next-utterance prediction with no persona, one speaker’s persona, the partner’s persona, or both, evaluated using perplexity, F1, and classification loss.
4 Models
The paper evaluates ranking and generative dialogue models, incorporating profiles through concatenation or profile-attending memory networks. Seq2Seq models encode dialogue histories, while profile-aware variants use profile information during response generation.
- Model Classes: Ranking models select training-set replies, whereas generative models produce novel responses word by word from dialogue history and optional persona information.
- Baseline Ranking Models: The baseline systems are an information-retrieval model and StarSpace, a supervised embedding model trained to match dialogue contexts with next utterances.
- Baseline Ranking Models: For IR and StarSpace, profile information is incorporated by concatenating the profile with the query vector or bag of words.
- Profile Memory Networks: The profile memory network attends over profile sentences using dialogue history as input, then combines relevant profile information with the input to predict the next utterance.
- Key-Value Memory Network: The key-value memory network attends over dialogue-history keys and outputs weighted next-utterance values to influence current-response prediction.
- Seq2Seq: Seq2Seq encodes the input with an LSTM and decodes the response with a word-level softmax, trained using negative log likelihood.
- Generative Profile Memory Network: The generative profile memory network encodes profile entries as memory representations and lets the decoder attend over them; without a profile, it becomes equivalent to Seq2Seq.
5 Experiments
Experiments show that persona conditioning generally improves next-utterance prediction, while ranking models outperform generative models. Human evaluations further indicate that PERSONA-CHAT models differ from Twitter and OpenSubtitles models, and persona-driven models remain detectable without sacrificing fluency or consistency.
- Persona Conditioning: Persona conditioning improves prediction for most models, especially on original personas; revised personas are harder because they remove word overlap.The Profile Memory generation model improves perplexity and hits@1 over Seq2Seq, while ranking algorithms improve hits@1.
- Ranking vs. Generative: Ranking models substantially outperform generative models on ranking metrics.The authors suggest this may reflect generative models’ word-based probabilities being insufficiently calibrated as sentence probabilities.
- Ranking Models: Profile Memory networks outperform simpler ranking baselines through learned similarity and attention over persona profiles.Starspace outperforms the IR baseline, and Profile Memory networks outperform Starspace; KV Profile Memory adds neighboring training examples mainly without persona information.
- Revised Personas: 0.354 vs. 0.318 hits@1 shows a remaining gain for Profile Memory networks on revised personas despite the task’s greater difficulty.Training on revised personas helps on both original-form and revised-form test examples, likely because it encourages semantic rather than simple lexical matching.
- Human Evaluation: Human evaluation finds PERSONA-CHAT models more fluent, engaging, and consistent than models trained on Twitter and OpenSubtitles.The dataset is especially suited to the beginning of conversations, where speakers ask and answer questions while getting to know each other.
- Human Evaluation: Persona-conditioned models can be detected by conversation partners while maintaining fluency and consistency relative to non-persona models.The human evaluation also measures fluency, engagingness, consistency, and profile detection using crowdsourced workers.
- Human Evaluation: Balancing fluency, engagement, consistency, and a persistent persona remains a challenge for future research.This limitation is stated as an open challenge after the human evaluation results.
- Profile Prediction: Human speaker profiles are predicted with 94.3% accuracy, while KV Profile Memory predicts profiles from model dialogue with 23%.Profile-prediction accuracy improves with additional dialogue, though combining prediction and response generation remains future work.
6 Conclusion & Discussion
The paper introduces PERSONA-CHAT and shows how persona-conditioned dialogue models support more consistent and engaging chit-chat. The dataset also enables profile prediction and paraphrase-based semantic similarity research.
- Models with access to their own personas were judged more consistent by annotators, but not more engaging.
- Models trained on PERSONA-CHAT were more engaging than models trained on movie or Twitter dialogue.
- Paired profiles and conversations support agents with consistent personalities and viewpoints.
- Profile paraphrases prevent trivial matching and may support semantic similarity research, while profile prediction moves chit-chat toward goal-directed dialogue.
A Next Utterance Prediction Additional Evaluation Metrics
Additional next-utterance evaluations compare conditioning on the other speaker’s persona with conditioning on one’s own or both personas, and examine revised personas.
- Knowing the other speaker’s persona did not improve precision for the evaluated models and dataset.
- The evaluation compares “Their” persona conditioning with “Self” and “Both” conditioning.
- Revised personas are compared with original personas for ranking-model performance in Table 6.
B Example Dialogs between Humans and Models
The appendix presents example dialogues collected during human evaluation between Turkers and several dialogue models.
- Examples in Tables 7, 8, 9, 11, and 12 show dialogues between Turkers and various models.
C Human Evaluation Measures
Human evaluation measures fluency, engagingness, persona consistency, and whether a model’s assigned profile can be detected from conversation.
- Evaluators rate fluency from 1 to 5, where 1 means not fluent, 5 means extremely fluent, and 3 means OK.
- Evaluators rate engagingness from 1 to 5 while disregarding fluency, with 1 meaning not engaging and 5 meaning extremely engaging.
- Profile detection asks evaluators to choose which of two displayed profiles is more likely to describe the person they just spoke with.
D Profile Prediction
The paper studies whether dialogue histories can reveal speakers’ profile information, including both human and model personas. Profile prediction is evaluated across speaker-profile combinations and becomes more accurate as conversations lengthen.
- Prediction setup: Profile prediction considers four combinations of whose dialogue utterances and whose persona are used, covering human and model speakers.The study examines predicting Profile 0 or Profile 1 from PERSON 0 or PERSON 1 utterances in human–model dialogues.
- Results: Human profiles can be predicted with high accuracy from their own dialogue utterances, independent of the model they speak to.This holds at both profile and sentence level.
- Results: A model’s profile is predicted with high accuracy from its utterances when it is conditioned on that profile, but prediction is chance-level without profile conditioning.This links profile conditioning to the information recoverable from the model’s generated utterances.
- Results: The human’s profile can also be predicted from model dialogue, but with lower accuracy; without profile conditioning, the model performs better at this task.The authors suggest this may reflect greater attention to the human’s interests and identify a tradeoff requiring further study.
- Results: Profile-prediction error rates decrease as dialogue length increases from one to eight utterances.The reported trend holds across all evaluated cases.
- Conclusion: Overall, the results indicate that predicting profiles from dialogue utterances is plausible, though more sophisticated models could improve performance.The paper frames profile prediction as an information-extraction task.