BotsTalk: Machine-sourced Framework for Automatic Curation of Large-scale Multi-skill Dialogue Datasets

1 Introduction

• Existing skill-grounded datasets target individual abilities such as persona, knowledge, or empathy, and are effective when a model demonstrates that targeted skill.Table 1 describes the skill contexts used by ConvAI2, Wizard of Wikipedia, and Empathetic Dialogues.Source: §1 · p. 1, Table 1 · p. 2
• A 5K-conversation crowdsourced benchmark provides a testbed for blended dialogue, but its limited scale and the cost of scaling crowdsourcing constrain its use for training.Automatic curation offers a path toward large-scale dialogue generation without additional human effort.Source: §1 · p. 1
• BotsTalk uses multiple agents grounded in individual skills to converse and automatically curate multi-skill dialogues, yielding BSBT with 300K conversations.The skills are derived from ConvAI2, Wizard of Wikipedia, and Empathetic Dialogues.Source: §1 · p. 1
• Experiments report that BSBT-trained models gain in skill blending while maintaining competitive skill grounding, and human judges consistently prefer its conversations over BST across all metrics.The comparisons are reported for dialogue-model performance and human evaluation, respectively.Source: §1 · p. 1

2.1 Skill-grounded Dialogue Datasets

• ConvAI2, Wizard of Wikipedia, and Empathetic Dialogues provide persona-, knowledge-, and empathy-grounded dialogue, but it remains unclear whether models retain a skill while blending it with others.The datasets contain more than 140K, about 194K, and 50K utterances, respectively.Source: §2.1 · p. 2
• Blended Skill Talk offers a crowdsourced testbed of about 5K conversations, but its scale could hinder training multi-skill chatbots that require larger datasets.The benchmark combines personable, knowledgeable, and empathetic conversation.Source: §2.1 · p. 2

2.2 Automatic Dialogue Data Annotation

• Crowdsourcing is a popular dialogue-collection approach, but producing multi-turn dialogues requires expensive and exhausting human effort.Source: §2.2 · p. 2
• Automatic data construction includes replacing parts of text dialogues with relevant images, human–AI collaboration for chit-chat augmentation, and model-based or human-and-model-in-the-loop collection.The cited examples include a 45K multimodal dataset and annotations for 23.8K task-oriented dialogues.Source: §2.2 · p. 2

3.1 Multi-skill Dialogue Annotation

• The task is to construct a large-scale, multi-turn dialogue dataset that blends multiple skills over a conversation.Source: §3.1 · p. 3
• The input consists of separate single-skill datasets, with M denoting the set of skill types, such as personality, knowledge, and empathy.The examples are derived from ConvAI2, Wizard of Wikipedia, and Empathetic Dialogues.Source: §3.1 · p. 3
• Each skill dataset Dm contains Nm dialogue episodes for skill m, with skill-relevant descriptions and dialogue turns as its contexts.Source: §3.1 · p. 3
• The annotation maps the input datasets D1 through DM to a new dialogue dataset for M skills, using dialogue context derived from multiple skill contexts.Source: §3.1 · p. 3

3.2 Desirable Characteristics of Multi-skill Dialogue Datasets

• A training dataset should support skill blending—exhibiting different skills in a conversation—and skill grounding—maintaining each skill when appropriate.Source: §3.2 · p. 3
• Blending and grounding can trade off in finite-length conversations, but are not contradictory because some skill-grounded utterances leave room for a natural shift.Source: §3.2 · p. 3
• The example shifts from a personal preference about sneakers to a knowledge-based explanation of their original sports use.It illustrates how different skills can be blended across adjacent turns.Source: §3.2 · p. 3

4 BOTSTALK Framework

• BOTSTALK automatically annotates multi-skill dialogues from multiple single-skill datasets, aiming to combine skill blending with skill grounding.Its participants iteratively generate dialogues.Source: §4 · p. 3, §4.1 · p. 3

4.1 Participants in BOTSTALK

• Multiple participants engage iteratively, but only one active agent has priority to speak on a dialogue turn.That priority may pass to another skill agent.Source: §4.1 · p. 3, Skill Agents · p. 4

Skill Agents

• Each single-skill agent uses a generator and ranker, fine-tuned on its individual skill dataset, to produce and score responses.The generator creates responses token by token, while the poly-encoder ranker scores and ranks candidate responses.Source: Skill Agents · p. 3

Moderator Agent

• A moderator with access to all skill contexts assesses each candidate response against those contexts and the dialogue context, approving or refusing it.This addresses the skill agents’ inability to read other agents’ skill contexts.Source: Moderator Agent · p. 4

4.2 Phase 1: Simulate what to speak

• All skill agents first simulate a plausible next response using their own skill context and the current dialogue context.The dialogue begins from a sampled two-turn single-skill seed, with relevant contexts retrieved from the input datasets using TF-IDF.Source: §4.2 · p. 4

4.3 Phase 2: Check dialogue consistency

• About 70% of NLI-flagged utterances conflict with other skill contexts, and roughly 50% of all utterances are classified as contradictory.The authors attribute these inconsistencies to skill agents’ restricted access to other skill contexts.Source: §4.3 · p. 5, Figure 2 · p. 5

4.4 Phase 3: Speak or pass the mic

• The active agent ranks simulated responses, while the moderator balances skill blending and grounding and can pass speaking priority to an inactive agent.A mic handoff occurs when the moderator accepts an inactive agent’s response.Source: §4.4 · p. 5
• High-entropy utterances are uncertain and generic, and their non-negligible presence suggests opportunities for skill shifts within conversations.Figure 3b presents the entropy of skill distributions for all utterances.Source: §4.4 · p. 5, Figure 3 · p. 5

5.1 Data Statistics

• BSBT contains 300K dialogues and 3M utterances, with utterances labeled for skill type and skill distribution.The authors attribute its scalability to machine-sourced bot-bot conversations, unlike other datasets’ crowdsourced human-to-human conversations.Source: §5.1 · p. 6
• Personality, knowledge, and empathy account for 36.10%, 31.69%, and 32.21% of utterances, respectively; over 90% of dialogues contain at least two skills.Figure 4 depicts utterance percentages by skill type and dialogue counts by number of skill types.Source: §5.1 · p. 6, Figure 4 · p. 6

5.2 Experimental Setups

• The experiments use the BST benchmark, which evaluates both blended and grounded objectives, and report retrieval R@k and MRR plus generative Avg. BLEU.The retrieval test presents 100 candidate responses per example.Source: §5.2 · p. 7

5.3 Automatic Evaluation

• BSBT outperforms all baselines on every automatic metric on the BST benchmark, and multi-skill models outperform single-skill models.The authors attribute the difference to multi-skill models’ ability to combine skills in conversational flow; BSBT is presented as a training resource for blending and grounding.Source: §5.3 · p. 7, Table 4 · p. 7
• BSBT300K significantly improves R@1 and MRR over BSBT10K, surpasses BST, and achieves the best performance as dataset size increases.Figure 6 reports retrieval performance by dataset size using R@1 and MRR.Source: §5.3 · p. 7, Figure 6 · p. 7

5.4 Human Evaluation

• In pairwise human evaluation of 100 sampled dialogues per dataset, BSBT has comparable or slightly higher win percentages than BST for engagingness, interestingness, and humanness.The qualitative analysis identifies repeated greetings, single-skill dialogues, and responses weakly related to their skill contexts as error cases.Source: §5.4 · pp. 7–8, Table 5 · p. 7

5.5 Analysis on Multi-task Learning

• MTL performs better after additional BSBT fine-tuning, though gains become marginal as dataset size grows; MTL fine-tuned on BSBT300K performs best.The authors hypothesize overlap between knowledge learned through multi-task learning and BSBT, and leave mitigation for future work.Source: §5.5 · p. 8, Table 6 · p. 8

5.6 Analysis on Skill Grounding Ability

• On single-skill benchmarks, BSBT is more well-rounded than single-skill models and is comparable to them on corresponding benchmarks in some cases.The authors conclude that BSBT can inject skill blending while maintaining grounding to specific skills.Source: §5.6 · p. 8, Table 7 · p. 8

• The paper constructs BSBT with BOTSTALK and validates its efficacy as a training resource through experiments and analyses.Source: §6 · p. 8

7 Limitations

• Fixed-length conversations can produce generic, repetitive responses after the conversation would otherwise end.The authors propose training models to recognize when to end a conversation from context.Source: §7 · p. 9
• Some dialogues lack explicit transitions between skills and may feature only one skill type, limiting their representation of skill blending.The authors note this may hinder learning diverse communicative skills.Source: §7 · p. 9
• Responses, particularly near conversation endings, can become less relevant to their corresponding skill contexts.The authors conjecture that agents increasingly condition on the lengthening dialogue context rather than predefined skill contexts.Source: §7 · p. 9

B Single-skill Datasets into BOTSTALK

• BOTSTALK integrates single-skill datasets into one-to-one, multi-turn conversations by alternating skill contexts between two speakers.The appendix also describes the source datasets and their incorporation into the framework.Source: B Single-skill Datasets into BOTSTALK · p. 11

B.1 ConvAI2

• ConvAI2 skill contexts comprise five distinct persona sentences, with two contexts alternated to simulate turn-taking.This follows the ConvAI2 setup, where paired speakers are assigned persona profiles.Source: §B.1 · p. 11

B.2 Wizard of Wikipedia

• Wizard of Wikipedia contexts pair a shared conversation topic with seven relevant knowledge sources for the wizard, while the apprentice receives the topic alone.This simpler setup ignores the task's retrieval aspect.Source: §B.2 · p. 11

B.3 Empathetic Dialogues

• For Empathetic Dialogues, the speaker's skill context is the situation description and emotion label, while the listener receives no skill context.This trains the system to show empathy based solely on the conversation.Source: §B.3 · pp. 11–12

C.1 Skill Agents

• Skill agents generate or rank candidate next utterances using the dialogue context and skill context as inputs.The generator is a fine-tuned dodecaDialogue model; the ranker is a poly-encoder.Source: §C.1 · p. 12
• The dodecaDialogue generators report perplexities of 11.19 on ConvAI2, 8.46 on WoW, and 11.08 on ED.The poly-encoder rankers report R@1 scores of 89.41 on ConvAI2, 91.01 on WoW, and 63.26 on ED.Source: §C.1 · p. 12

C.2 Moderator Agent

• The moderator uses NLI to screen responses for contradictions with skill contexts and a skill classifier to identify each response's skill.The NLI classifier treats the context as premise and response as hypothesis; the skill classifier is trained on ConvAI2, WoW, and ED utterances.Source: §C.2 · p. 12

C.3 Skill Context Retrieval

• TF-IDF retrieves the top five relevant skill contexts per seed utterance pair, providing five times the seed information.The authors describe this as a simple information-retrieval baseline and note that other systems could be tried.Source: §C.3 · pp. 12–13

D Implementation Details of Baselines

• Retrieval baselines use a Reddit-pretrained 256-million-parameter poly-encoder fine-tuned on ConvAI2, WoW, ED, BST, and BSBT for eight epochs.Training uses batch size 128, learning rate 5e-5, and selects parameters by R@1.Source: D Implementation Details of Baselines · p. 13
• Generative baselines use a pretrained 12-layer-per-stack BART fine-tuned on ConvAI2, WoW, ED, BST, and BSBT for three epochs.Training uses batch size 32 and greedy decoding, with parameters selected by accuracy.Source: D Implementation Details of Baselines · p. 13

E Additional Performance

• Evaluation covers BST as the multi-skill benchmark and ConvAI2, WoW, and ED as single-skill benchmarks, using retrieval and generation metrics.Retrieval uses R@1, R@5, R@10, and MRR; generation uses BLEU-1 through BLEU-4.Source: E Additional Performance · pp. 13–14
• The human-evaluation interface presents a pair of dialogues, with BST on the left and BSBT on the right.The figure identifies the comparison layout but does not report an evaluation outcome.Source: Figure 7 · p. 14

F Settings of Human Evaluation

• 100 dialogues each were sampled from BST and BSBT, with skill contexts excluded for anonymity because the datasets’ skill contexts are distinguishable.Judges compared dialogue contexts without seeing skill contexts.Source: F Settings of Human Evaluation · p. 14
• The evaluation employs ACUTE-Eval, described as a widely used metric for multi-turn dialogue evaluation.Source: F Settings of Human Evaluation · p. 14
• Judges compare each conversation pair on engagingness, interestingness, and humanness.The questions ask about preference for continued conversation, curiosity or useful information, and natural, personable speech.Source: F Settings of Human Evaluation · p. 14

Qualitative Analysis

• The analysis selects cherry-picked and lemon-picked BSBT examples, alongside randomly sampled dialogues in Tables 14 and 15.The dialogue setup assigns one speaker five personas, a topic, situation, and emotion; the other receives five personas, the topic, seven knowledge resources, and no situation/emotion context.Source: G Qualitative Analysis · p. 14, Table 12 · p. 19, Table 13 · p. 20

Cherry-picked Dialogues

• Example 3 sustains each skill type for more than two turns and blends all skill types naturally and reasonably.The authors present this as enabling dialogue systems to learn skill grounding and weave multiple skills in one conversation.Source: G Qualitative Analysis · pp. 14–15

Lemon-picked Dialogues

• Fixed 10-turn conversations can repeat greetings when the exchange would naturally end sooner; the authors leave a termination condition for future work.Other lemon-picked cases either feature only one skill or shift skills while responses remain generic or weakly relevant to their skill contexts.Source: G Qualitative Analysis · p. 15

Qualitative Analysis

• The appendix also provides sample conversations from ConvAI2, Wizard of Wikipedia, and Empathetic Dialogues, with their differing input setups described in the captions.ConvAI2 examples give each speaker five persona sentences; Wizard of Wikipedia gives the apprentice a topic and the wizard retrieval access; Empathetic Dialogues provide a situation and emotion to one speaker only.Source: Table 9 · p. 16, Table 10 · p. 17, Table 11 · p. 18