Source-linked AI summary
MuTual: A Dataset for Multi-Turn Dialogue Reasoning
Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, Ming Zhou
TL;DR
Current dialogue systems can produce fluent, relevant responses yet often make logical mistakes because reasoning ability and commonsense knowledge are insufficiently captured. MuTual addresses this gap with a manually annotated multi-turn reasoning dataset, and the best model reaches only 71.3% R@1, leaving a large gap from human performance.
Problem
Dialogue benchmarks often permit linguistic matching, while systems still make logically incorrect responses because reasoning and commonsense knowledge are insufficiently captured.
Method
MuTual converts Chinese high school English listening-comprehension data into a multi-turn response-selection dataset with four relevant candidates, only one logically correct.
Results
71.3% R@1 is achieved by the best model, RoBERTa, while various state-of-the-art models show poor performance on MuTual.
Takeaways & Limitations
MuTual provides a manually annotated benchmark for testing multi-turn dialogue reasoning and exposes substantial room for improving dialogue-model reasoning ability.
Takeaways & Limitations
The human-annotated dataset is smaller than previous crawling-based dialogue datasets because its collection method trades instance count against annotation quality.
Abstract
from arXiv · showhide
Non-task oriented dialogue systems have achieved great success in recent years due to largely accessible conversation data and the development of deep learning techniques. Given a context, current systems are able to yield a relevant and fluent response, but sometimes make logical mistakes because of weak reasoning capabilities. To facilitate the conversation reasoning research, we introduce MuTual, a novel dataset for Multi-Turn dialogue Reasoning, consisting of 8,860 manually annotated dialogues based on Chinese student English listening comprehension exams. Compared to previous benchmarks for non-task oriented dialogue systems, MuTual is much more challenging since it requires a model that can handle various reasoning problems. Empirical results show that state-of-the-art methods only reach 71%, which is far behind the human performance of 94%, indicating that there is ample room for improving reasoning ability. MuTual is available at https://github.com/Nealcly/MuTual.
1 Introduction
Existing dialogue systems often produce fluent, relevant responses yet still make logical errors because benchmarks and models insufficiently test reasoning. MuTual addresses this gap with a manually annotated multi-turn response-selection dataset built from English listening-comprehension data.
- Motivation: Current dialogue systems can perform strongly on existing benchmarks but often generate logically incorrect responses because reasoning and commonsense capabilities remain insufficient.Many prior cases can be solved through linguistic matching rather than deeper reasoning.
- Motivation: Evaluating chatbot reasoning is presented as a way to address the gap between leaderboard performance and unsatisfactory practical performance.
- Dataset: MuTual provides four context-relevant response candidates, of which only one is logically correct, requiring reasoning about social etiquette and speaker relationships.The task is framed as next-response selection rather than answering an additional question.
- Dataset: Annotators convert Chinese high-school English listening-comprehension questions and answers into response candidates for multi-turn dialogue response selection.The original question-and-answer format is rewritten because chatbots select responses to contexts rather than answer an extra question.
- Dataset: 8,860 challenge questions comprise MuTual, which the authors describe as the first human-labeled reasoning-based dataset for multi-turn dialogue.The dataset was designed by linguist experts and high-quality annotators.
2 Related work
Prior dialogue benchmarks generally do not focus on reasoning, while reasoning datasets are often not directly suited to chatbot response selection. MuTual instead modifies listening-comprehension conversations into a next-utterance prediction task requiring diverse reasoning abilities.
- Dialogue benchmarks: Most existing conversation benchmarks do not focus on testing reasoning ability.
- Dialogue benchmarks: CoQA includes pragmatic reasoning, but it is a machine-comprehension dataset based on conversations about a given passage.
- Dialogue benchmarks: DREAM is designed for challenging dialogue-based reading comprehension and relies on an external question to test understanding.
- MuTual: MuTual differs by framing the task as next-utterance prediction, the fundamental response-selection problem in retrieval-based chatbots.Its data are formed by deeply modifying English listening-comprehension conversations.
- MuTual: MuTual requires varied reasoning abilities, including algebraic reasoning and intention prediction.
3 Dataset
MuTual is constructed from English listening-comprehension materials by converting multi-turn conversations, questions, and choices into response-selection instances. Its examples and analyses target several reasoning types, while its scale and linguistic complexity reflect trade-offs in human annotation.
- Collection: The source materials require students to select the best answer from three options using a multi-turn dialogue and question.
- Collection: Raw audio, questions, choices, and answer images are crawled and converted into text through ASR, OCR, and manual correction.
- Candidate Response Creation: Annotators segment conversations after answer clues appear and construct one positive and three negative response candidates from the original choices.
- Candidate Response Creation: Negative responses remain logically correct without context but become inappropriate when the conversation is considered, discouraging simple text matching.
- Statistics: MuTual averages 4.73 turns and contains a vocabulary of 11,343 words, while its morphology and grammar are simpler than those of other dialogue datasets.
- Reasoning Types: The dataset covers situational, multi-fact, algebraic, attitude, intention, and other commonsense reasoning, with 9% of instances requiring other commonsense knowledge.
4 Experiments
Experiments evaluate retrieval, matching, multi-choice, and pre-trained dialogue models on MuTual and MuTualplus. Results show that reasoning, especially commonsense reasoning, remains difficult despite strong pre-trained-model performance and relatively limited sensitivity to context length.
- Experimental Setup: MuTual evaluation uses response-selection metrics including R@1, R@2, and MRR, comparing individual-scoring, multi-choice, pre-trained, and human performance.Multi-choice methods classify over all four candidates, whereas individual-scoring methods score each candidate independently.
- Results on MuTual: TF-IDF performs only slightly above random guessing on MuTual, indicating little obvious context–response overlap signal.TF-IDF reaches 54.98% R@1 on Ubuntu Corpus, contrasting with its weak MuTual performance.
- Results on MuTual: Matching models drop by more than 50 absolute R@1 points from Ubuntu Corpus performance, showing that text matching alone handles MuTual reasoning poorly.Well-designed matching models do not outperform a simple dual LSTM on MuTual.
- Results on MuTual: 71.3% R@1 for RoBERTa on MuTual remains below 94% human performance, although RoBERTa reaches 89.2% R@2.BERT and RoBERTa outperform other evaluated models, but the dataset remains challenging for pre-trained systems.
- Results on MuTualplus: All models perform worse on MuTualplus, while multi-choice methods outperform individual-scoring methods and RoBERTa-MC leads by a large margin.In transfer experiments, RoBERTa-MC and RoBERTa drop 24.1% and 6.8%, respectively, when trained on MuTual and tested on MuTualplus.
- Discussion: RoBERTa-MC performs best on attitude and multi-fact reasoning but struggles with algebraic and situation reasoning, which rely heavily on commonsense reasoning.A simple time subtraction and inferring a flat-rental situation remain challenging examples.
- Discussion: RoBERTa performance drops only 1.9 R@1 points from two turns to more than six turns, suggesting MuTual difficulty is driven more by reasoning than conversation-history length.Performance for five turns exceeds performance for four turns.
- Discussion: Ablating all context utterances reduces RoBERTa and RoBERTa-MC to 43.7% and 47.7%, while shuffling utterances reduces RoBERTa-MC by only 3.8%.The results indicate that individual utterances matter, but the model is relatively insensitive to utterance order.
5 Conclusion
MuTual is a manually annotated multi-turn dialogue reasoning dataset containing 8,860 dialogues. Experiments show that state-of-the-art models perform poorly on it, leaving a substantial gap from human performance.
- MuTual contains 8,860 manually annotated multi-turn dialogues designed to test dialogue-model reasoning ability.
- State-of-the-art models perform poorly on MuTual, with RoBERTa achieving only 71.3% R@1.
- The best model remains substantially below human performance on the dataset.The reported results identify a large model–human performance gap.
- MuTual is intended to facilitate future research on multi-turn conversation reasoning.