Source-linked AI summary
Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills
Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, Y-Lan Boureau
TL;DR
The paper asks how open-domain conversational agents can blend desirable capabilities that prior work typically trains and evaluates separately. It compares model-combination and multi-task strategies using the new BlendedSkillTalk dataset, finding that multi-task approaches can work well when skill-selection bias is controlled and blended-task fine-tuning improves models further.
Problem
It remains unclear whether agents optimized for one conversational capability can retain it while blending knowledge, empathy, and personal conversation in a single dialogue.
Method
The paper compares single-skill model combination and multi-task training, proposes BlendedSkillTalk, and evaluates blended conversations with automated metrics and human ratings.
Results
Multiple multi-task approaches are effective on BlendedSkillTalk, while careful training construction mitigates skill-selection bias and fine-tuning on the blended task improves models further.
Takeaways & Limitations
Existing single-skill tasks can be leveraged to build agents that blend multiple conversational skills when selection bias is addressed or models are fine-tuned on blended data.
Takeaways & Limitations
The study addresses three skills; extending the findings requires corresponding single-skill datasets and skills that are sufficiently distinguishable.
Abstract
from arXiv · showhide
Being engaging, knowledgeable, and empathetic are all desirable general qualities in a conversational agent. Previous work has introduced tasks and datasets that aim to help agents to learn those qualities in isolation and gauge how well they can express them. But rather than being specialized in one single quality, a good open-domain conversational agent should be able to seamlessly blend them all into one cohesive conversational flow. In this work, we investigate several ways to combine models trained towards isolated capabilities, ranging from simple model aggregation schemes that require minimal additional training, to various forms of multi-task training that encompass several skills at all training stages. We further propose a new dataset, BlendedSkillTalk, to analyze how these capabilities would mesh together in a natural conversation, and compare the performance of different architectures and training schemes. Our experiments show that multi-tasking over several tasks that focus on particular capabilities results in better blended conversation performance compared to models trained on a single skill, and that both unified or two-stage approaches perform well if they are constructed to avoid unwanted bias in skill selection or are fine-tuned on our new task.
1 Introduction
The paper addresses whether conversational agents can blend knowledge, empathy, and personal conversation rather than optimizing these capabilities in isolation. It compares combination strategies and introduces BlendedSkillTalk to evaluate blended dialogue.
- Open-domain agents should combine listening, empathy, knowledgeable responses, and discussion of diverse everyday topics.
- Prior work evaluates conversational capabilities such as knowledge, empathy, and personal background separately, leaving their retention and simultaneous training unclear.
- The study compares multi-task training, model aggregation, and a classifier-based dialogue manager that selects among pretrained single-skill models.
- BlendedSkillTalk is a new English-language dataset combining several skills in natural conversation and supporting automated and crowdsourced evaluation.
- Experiments find blended agents effective when skill selection avoids unwanted bias, uses blended-task fine-tuning, or combines both strategies.
2 Related work
Related work develops open-domain conversational agents and datasets focused on individual traits or skills. This paper builds on those resources to study how three clearly defined capabilities can be blended.
- Commercial dialogue systems commonly use hand-coded narrow skills for task-oriented features, whereas this work studies learned qualities in open-domain dialogue.
- End-to-end learning is emphasized because it offers the promise of better generalization to unseen domains.
- Large conversation-like datasets enable reproducible comparison, but resulting agent qualities depend on the qualities present in source conversations.
- ConvAI2, Wizard of Wikipedia, and EmpatheticDialogues provide separate datasets for personal conversation, expert knowledge, and empathetic responses.
- The three selected skills are clearly defined and separate in scope, making focused data collection, ablations, and analyses feasible.
3 Blending Skills in a Conversation
BlendedSkillTalk is designed to test whether conversational agents can move among knowledge, empathy, and personal modes within one natural conversation. The dataset combines seeded contexts, optional single-skill suggestions, worker annotations, and classifier analyses to create and characterize this blended setting.
- Dataset collection: BlendedSkillTalk contains about 5k English conversations collected from 2,679 workers, with 4,819 train, 1,009 validation, and 980 test conversations.
- Dataset collection: Workers were instructed to be knowledgeable, empathetic, or provide persona-related personal details whenever appropriate.
- Conversation construction: Guided workers used the suggestions in 20.5% of utterances, while unguided workers continued the guided speaker’s previous mode 46.1% of the time versus 33.3% at chance.
- Conversation construction: Each conversation combines private personas, a topic or seed context, and guided suggestions from ConvAI2, ED, and WoW single-task models.
- Skill annotations: Seed context influenced the blend: 43.5% of unguided utterances matched the context dataset, compared with 33.3% if it had no influence.
- Skill annotations: Annotations distinguish knowledge, empathy, personal situations, and personal background, and over 70% of conversations contained at least three modes.
4 Experiments
The experiments evaluate skill-blending methods using automated metrics, human ratings, and skill-selection analyses. Multi-task and blended-task approaches generally produce more balanced performance than single-skill models, with bias mitigation and fine-tuning improving results.
- Evaluation methodology: The evaluation combines hits@1 automated metrics with human ratings of knowledge, empathy, personalness, and overall conversation quality.Automated metrics use 20 candidates for ConvAI2 and 100 for ED and WoW; human ratings are collected across multiple model types.
- Skill-selection bias: Adding personas and topics during multi-task training shifts retrieval toward more equal use of the three skills on BlendedSkillTalk.The original training setup overuses ConvAI2 and underuses WoW; debiasing produces ratios closer to those expected in BST.
- Single-skill benchmarks: Blended models provide more balanced single-skill benchmark performance, while Multi-Task Single-Skills performs best among blended architectures and nearly matches specialized models.It even surpasses the corresponding single-skill model on WoW, whereas BST fine-tuning trades specialized benchmark performance for balance.
- Single-skill benchmarks: Multi-task training is more resilient than individual models when benchmark candidate sets include equally many distractors from the other two tasks.This mixed-candidate evaluation triples the candidate pool and favors the multi-task model over individual models.
- BlendedSkillTalk benchmark: On BlendedSkillTalk, Multi-Task Single-Skills outperforms all single-skill baselines in both zero-shot and fine-tuned settings.The Multi-Task Two-Stage model also outperforms two single-skill models, while all single-skill models improve after BST fine-tuning.
- Human evaluation: Human evaluations find multi-skill models strong across every dimension, with MT Two-Stage and BST-fine-tuned MT Single-Skills performing best overall.The BST-fine-tuned MT Single-Skills model achieves comparable or slightly better ratings with fewer than a third of MT Two-Stage's parameters.
5 Discussion and Conclusion
The paper concludes that multi-task approaches can blend knowledge, empathy, and personal conversation effectively. Careful training can reduce skill-selection bias, while fine-tuning on blended data can improve performance; generalization beyond these three skills remains an open extension.
- Discussion and conclusion: BlendedSkillTalk evaluates open-domain agents that combine knowledge, empathy, and personal conversation in balanced proportions.The dataset is a new English-language benchmark designed to assess seamless blending of these three skills.
- Discussion and conclusion: Multiple multi-task approaches are effective, but training schemes must mitigate skill-selection bias and blended-task fine-tuning improves models further.The conclusion identifies debiasing and fine-tuning as important conditions for effective skill blending.
- Limitations and extensions: Generalizing the findings to skills such as humor, eloquence, or image commenting requires corresponding single-skill datasets and sufficiently distinguishable skills.The paper presents this as a natural extension beyond the three skills studied.