Source-linked AI summary
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems
Ryan Lowe, Nissan Pow, Iulian Serban, Joelle Pineau
TL;DR
Dialogue research lacks sufficiently large datasets for unstructured multi-turn conversations. This paper constructs and releases the Ubuntu Dialogue Corpus, develops TF-IDF and neural response-selection architectures, and reports significantly better results with LSTM than with RNN and TF-IDF. The corpus supports benchmark research on selecting the next response without human labeling.
Problem
Dialogue systems lack sufficiently large datasets for researching unstructured multi-turn conversations with neural learning.
Method
The paper constructs the Ubuntu Dialogue Corpus and benchmarks TF-IDF, RNN, and LSTM architectures for next-response selection.
Results
LSTM significantly outperforms RNN and TF-IDF for selecting the next best response.
Takeaways & Limitations
The corpus provides a large resource for research on neural architectures for unstructured multi-turn dialogue systems.
Takeaways & Limitations
Some dialogues contain duplicated initial questions because one question may receive replies from multiple users.
Abstract
from arXiv · showhide
This paper introduces the Ubuntu Dialogue Corpus, a dataset containing almost 1 million multi-turn dialogues, with a total of over 7 million utterances and 100 million words. This provides a unique resource for research into building dialogue managers based on neural language models that can make use of large amounts of unlabeled data. The dataset has both the multi-turn property of conversations in the Dialog State Tracking Challenge datasets, and the unstructured nature of interactions from microblog services such as Twitter. We also describe two neural learning architectures suitable for analyzing this dataset, and provide benchmark performance on the task of selecting the best next response.
1 Introduction
The paper targets natural, coherent one-on-one multi-turn dialogue in diverse, unstructured domains, addressing the lack of large datasets for neural dialogue research. It introduces the Ubuntu Dialogue Corpus and benchmarks TF-IDF, RNN, and LSTM models for next-response selection.
- The paper studies one-on-one multi-turn conversations across diverse topics without a predefined logical representation.
- Recent progress in other AI fields is attributed to large datasets, computing power, and improved neural-network training methods using unlabeled data.
- The Ubuntu Dialogue Corpus contains almost one million two-person technical-support conversations averaging eight turns, with at least three turns each.
- The corpus is orders of magnitude larger than structured dialogue datasets and has longer conversations and utterances than comparable microblog datasets.
- The paper benchmarks TF-IDF, RNN, and LSTM architectures for selecting the best next response without human labeling.
2 Related Work
Earlier dialogue datasets and methods largely emphasize small structured slot-filling tasks or short and less publicly available unstructured exchanges. The paper positions its contribution as a public, large-scale resource for longer unstructured dialogue research.
- Switchboard and DSTC datasets support structured dialogue management through slot filling but are small relative to datasets used for neural learning.
- Twitter and Weibo corpora provide unstructured dialogue data, but cited resources were not public and often used short post-reply exchanges.
- The Ubuntu Chat Corpus preserves multi-participant structure, making it less suitable for studying traditional two-party conversations.
- Question-answer datasets contain interactions much shorter than the multi-turn conversations targeted by this paper.
- Neural approaches for unstructured dialogue had shown potential but remained limited to very short conversations.
3 The Ubuntu Dialogue Corpus
The corpus is designed around large-scale dyadic, multi-turn, task-specific dialogue, extracted from Ubuntu technical-support chat logs. Its extraction relies on the structured messaging conventions of multi-user chat rooms to recover pairwise conversations.
- The target dataset requires dyadic human-human conversations, many conversations, several turns, and a task-specific domain.
- 3.1 Ubuntu Chat Logs: Ubuntu chat logs come from real-time Freenode IRC channels where participants discuss Ubuntu-related technical-support problems.
- 3.1 Ubuntu Chat Logs: Moderated channels commonly involve a user asking a question and another user replying with a potential solution, while multiple conversations occur simultaneously.
- 3.1 Ubuntu Chat Logs: The extraction identifies dyadic dialogues from message structure, especially username mentions that indicate the intended recipient.
3.2 Dataset Creation
The corpus is created by extracting dyadic dialogues from multi-user chat logs using sender–recipient structure, username heuristics, and filtering rules. The released data preserves posting times but does not apply further preprocessing.
- Dialogue extraction: Messages are represented as (time, sender, recipient, utterance) tuples and grouped by matching sender–recipient pairs.Recipient identification uses username dictionaries and excludes common English words to reduce false positives.
- Dialogue extraction: The extraction algorithm links a first response to the most recent matching initial question within a three-minute window.Initial questions without responses are discarded.
- Filtering: Dialogues longer than five utterances with one user producing over 80% of utterances are discarded, and retained dialogues must contain at least three turns.These filters target conversations that are less representative of real chat and encourage longer-term dependency modeling.
- Dialogue extraction: Non-addressed utterances are added when users do not address someone else during the conversation, and consecutive utterances from one user are concatenated.This procedure is intended to alleviate holes in extracted dialogues.
- Special cases and limitations: Initial questions can appear in multiple dialogues when several users respond, although the authors report that such cases are small relative to the dataset.Each conversation between the first user and a respondent is treated as a separate dialogue.
- Special cases and limitations: Posting time is not used to segment conversations, so exchanges spanning hours or days may remain one dialogue; timestamps are released for later filtering.The authors report that such dialogues are rare.
3.3 Dataset Statistics
The Ubuntu chat logs are valuable primarily because of their size and multi-turn structure. Dialogue counts and turns per dialogue exhibit an approximate power-law relationship.
- Corpus properties: The corpus’s size is identified as crucial for research on dialogue managers based on neural architectures.The section presents size as one of the corpus’s most important features.
- Corpus properties: The number of dialogues and turns per dialogue follows an approximate power-law relationship.Figure 1 shows the distribution of dialogue turns using logarithmic axes.
3.4 Test Set Generation
The test set evaluates response selection using randomly sampled contexts and candidate replies. It includes both one-negative and ten-negative response settings, with context lengths varying by dialogue and sampling procedure.
- Test-set construction: 2% of the conversations are randomly reserved for testing and transformed into (context, response, flag) triples.The flag identifies whether the response is the actual next utterance after the context.
- Test-set construction: Each test example pairs the correct next utterance with a randomly sampled false response from elsewhere in the test set.Experiments use either 1 wrong response or 10 wrong responses.
- Context sampling: Contexts contain the utterances preceding a target response, with context lengths sampled rather than fixed.The procedure aims to predict different portions of conversations instead of only closing statements.
- Context sampling: The maximum desired context size is C = 20, the minimum is 2, and the actual context cannot exceed the dialogue length minus one.The sampling procedure makes short dialogues use short contexts and longer dialogues sometimes use contexts of 10 or more turns.
3.5 Evaluation Metric
Evaluation treats best-response selection as a classification problem without human labels. Recall@k measures whether the true response appears among the k highest-scoring candidates, while response classification is not itself a measure of generation quality.
- Evaluation: Best-response selection is formulated as a classification task using automatically constructed examples, without requiring human labels.The task adapts recall and precision metrics previously used for dialogue datasets.
- Evaluation metrics: Recall@k is correct when the true response appears among the k most likely candidates; R@1 is relevant for binary classification.The reported metrics include R@1, R@2, and R@5.
- Evaluation limitations: Response-classification performance is not a gauge of next-utterance generation quality, although the authors hypothesize that improvements may eventually transfer.The paper identifies this relationship as a point for further discussion.
4 Learning Architectures for Unstructured Dialogues
The paper benchmarks TF-IDF, RNN, and LSTM approaches for response selection, using context–response examples derived from Ubuntu dialogues. Neural models encode contexts and responses with recurrent architectures, while TF-IDF selects responses by vector similarity.
- Model overview: The evaluated approaches are TF-IDF, RNN, and LSTM, with preprocessing based on NLTK and a Twitter tokenizer.Generic tags represent categories including names, locations, organizations, URLs, and system paths.
- Training data: Training examples are context–response–flag triples, with each utterance from the third onward treated as a potential response during neural-model training.The previous utterances form the context, so a dialogue of length 10 yields 8 overlapping training examples.
- TF-IDF: TF-IDF scores words by term frequency and inverse document frequency, then selects the candidate response with highest cosine similarity to the context.Recall@k returns the top k responses according to this similarity ranking.
- RNN: The RNN updates a hidden state from the current word and previous hidden state, and the model uses tied-weight RNNs to embed context and response sequences.The final hidden states produce context and response embeddings for classification.
- RNN: The response-pair classifier converts the embeddings into a validity probability, with learned parameters and cross-entropy training over labeled pairs.The probability uses a sigmoid over a bilinear context–response score plus a bias.
- LSTM: The LSTM replaces standard hidden units with gates that regulate remembering, forgetting, and output to model longer-term dependencies.The architecture is intended to address vanishing and exploding gradients in standard RNNs.
5 Empirical Results
The experiments compare TF-IDF, RNN, and LSTM response selection under binary and 1-in-10 candidate conditions. LSTM performs best across the reported evaluation metrics, and its performance increases with more training data.
- Evaluation setup: The models are evaluated on next-utterance classification with binary (1 in 2) and 1-in-10 false-response conditions.Recall@2 and Recall@5 are not relevant for the binary classification case.
- Results: LSTM outperforms both RNN and TF-IDF on all evaluation metrics.TF-IDF nevertheless exceeds RNN on Recall@1 for the 1-in-10 classification condition.
- Results: LSTM performance increases as the amount of training data increases.The paper presents this trend as evidence for the importance of a large training set.
- Results: Table 5 shows ranked responses produced by the LSTM after preprocessing for an example in which the response is correctly classified.The example illustrates the model's response ranking rather than providing an aggregate metric.
6 Discussion
The discussion highlights the Ubuntu Dialogue Corpus as a basis for neural response-selection research while identifying limitations in disentanglement and evaluation comparability. It also describes controlled task difficulty and the unresolved challenge of evaluating generated responses.
- The corpus enables preliminary RNN and LSTM experiments for selecting the next best response, with significantly better results from the LSTM.
- Results from the original dataset should not be compared with results from the new dataset because new-dataset baselines were not yet available.
- 6.1 Conversation Disentanglement: The disentanglement method uses a small set of rules and may be sufficient for retrieving plausible dialogues, but it lacks formal evaluation.
- 6.2 Altering Test Set Difficulty: Response-selection difficulty can be controlled by varying false responses and Recall@k, with harder similar negatives intended to test finer semantic understanding.
- 6.3 State Tracking and Utterance Generation: Response selection is used instead of generation because standardized, meaningful, inexpensive evaluation for generated dialogue remains unavailable.
Appendix A: Dialogue excerpts
The excerpts illustrate how chat-room messages are disentangled into separate dialogues and how the extraction algorithm adds and concatenates utterances according to addressed participants.
- The first example contrasts a raw #ubuntu chat-room conversation with its disentangled Ubuntu Dialogue Corpus conversations.
- The second example contrasts the conversation before and after the extraction algorithm adds and concatenates utterances.
- The extraction example shows that utterances are added for dell when RC addresses only dell, but not for dell when he addresses both RC and cucho.