Source-linked AI summary
MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues
Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, Wanli Ouyang
TL;DR
Existing benchmarks incompletely capture the fine-grained complexity of multi-turn dialogue, motivating a more comprehensive evaluation framework. MT-Bench-101 builds a hierarchical taxonomy from real dialogue data, evaluates 21 LLMs, and reports task-dependent turn trends, limited gains from alignment or chat-specific designs, and effective task measurement. The benchmark’s findings are bounded because emerging multi-turn capabilities may not yet be represented.
Problem
Existing benchmarks primarily focus on single-turn or coarse-grained multi-turn evaluation, insufficiently covering the complexity of real-world multi-turn dialogues.
Method
The paper analyzes real multi-turn dialogue data and educational taxonomies to construct a three-tier ability taxonomy with 13 tasks, then evaluates LLMs using task-specific scoring and golden dialogue context.
Results
Across 21 evaluated LLMs, performance varies by task and dialogue turn; larger models perform better, while alignment techniques and chat-specific designs show no significant multi-turn enhancement.
Takeaways & Limitations
MT-Bench-101 provides fine-grained multi-turn ability evaluation, and case studies indicate that its tasks effectively measure corresponding multi-turn chat abilities.
Takeaways & Limitations
The findings may not encompass all multi-turn abilities because new capabilities are likely to emerge as LLM technologies evolve.
Abstract
from arXiv · showhide
The advent of Large Language Models (LLMs) has drastically enhanced dialogue systems. However, comprehensively evaluating the dialogue abilities of LLMs remains a challenge. Previous benchmarks have primarily focused on single-turn dialogues or provided coarse-grained and incomplete assessments of multi-turn dialogues, overlooking the complexity and fine-grained nuances of real-life dialogues. To address this issue, we introduce MT-Bench-101, specifically designed to evaluate the fine-grained abilities of LLMs in multi-turn dialogues. By conducting a detailed analysis of real multi-turn dialogue data, we construct a three-tier hierarchical ability taxonomy comprising 4208 turns across 1388 multi-turn dialogues in 13 distinct tasks. We then evaluate 21 popular LLMs based on MT-Bench-101, conducting comprehensive analyses from both ability and task perspectives and observing differing trends in LLMs performance across dialogue turns within various tasks. Further analysis indicates that neither utilizing common alignment techniques nor chat-specific designs has led to obvious enhancements in the multi-turn abilities of LLMs. Extensive case studies suggest that our designed tasks accurately assess the corresponding multi-turn abilities. The data and code are available at \url{https://github.com/mtbench101/mt-bench-101}.
1 Introduction
MT-Bench-101 addresses gaps in coarse or single-turn dialogue evaluation with a fine-grained, three-tier benchmark for multi-turn LLM abilities. It evaluates 21 models and finds turn-dependent task trends, scaling gains, limited benefits from alignment or chat-specific designs, and 87% GPT-4–human agreement.
- Existing benchmarks often emphasize single-turn or coarse-grained evaluation, leaving the complexity of real-world multi-turn dialogue insufficiently assessed.
- MT-Bench-101 introduces a three-tier taxonomy spanning three overarching abilities, seven detailed abilities, and 13 distinct tasks.The taxonomy covers perceptivity, adaptability, and interactivity, progressing from context understanding to feedback response and proactive engagement.
- Performance trends differ across tasks as dialogue turns progress, and adaptability and interactivity are identified as key deficiencies in existing LLMs.GPT-4 is reported as the most powerful model for multi-turn dialogues.
- Model performance improves with model size, but common alignment techniques and chat-specific designs do not significantly enhance multi-turn abilities.
- GPT-4’s agreement with human expert evaluations reaches 87% under the designed evaluation approach.
2 Related Work
Prior work includes single-turn benchmarks, labor-intensive human evaluation, broad multi-turn evaluations, and specialized dialogue tasks. A gap remains in fine-grained evaluation of multi-turn interactions, motivating more detailed benchmarks.
- Many LLM benchmarks evaluate single-turn instructions, missing nuances of human conversation.
- ABC-Eval investigates multi-turn ability through labor-intensive human evaluations, while AlpacaEval and PandaLM automatically assess open-ended instructions in single-turn settings.
- MT-Bench and MT-Bench++ broaden multi-turn evaluation across eight topics, while BotChat and MINT target specialized dialogue-generation abilities.
- Existing work still leaves a notable gap in fine-grained evaluations for multi-turn interactions.
- Fine-grained ability benchmarks such as MMLU, ConceptMath, and Follow-Bench assess knowledge, logic, or hierarchical capabilities rather than the full nuance of multi-turn interactions.
3 MT-Bench-101
MT-Bench-101 is a fine-grained benchmark for multi-turn dialogue abilities, organized through a three-tier taxonomy and evaluated on a curated dataset spanning 13 tasks. Its design combines real dialogue analysis, task-specific generation and scoring, golden dialogue context, and minimum-score aggregation.
- Ability taxonomy: MT-Bench-101 defines a three-level ability taxonomy with 13 distinct multi-turn dialogue tasks.The taxonomy combines analyses of real dialogues with teaching taxonomies from educational psychology.
- Perceptivity: Perceptivity covers context tracking through memory, understanding, and interference abilities.Its tasks include context memory, anaphora resolution, separate input, topic shift, and content confusion.
- Adaptability: Adaptability evaluates how chatbots revise, reason about, and defend responses when users add requirements, conditions, hypotheses, or feedback.The taxonomy includes content and format rephrasing, mathematical and general reasoning, self-correction, and self-affirmation.
- Interactivity: Interactivity measures proactive dialogue behavior through instruction clarification and proactive interaction.These tasks assess follow-up questions that clarify unclear intent or encourage continued dialogue.
- Dataset construction and statistics: 1388 dialogues and 4208 turns across 13 tasks form the MT-Bench-101 dataset.The benchmark covers 30 topics and is described as focusing specifically on fine-grained multi-turn abilities.
- Evaluation: GPT-4 scores each turn from 1 to 10 using task-specific guidelines, while the lowest turn score becomes the dialogue’s final score.Golden context is used as dialogue history, and a Qwen-72B-Judge leaderboard reports rankings consistent with GPT-4-Judge, indicating that self-bias is minor in this benchmark.
4 Experiments
Experiments evaluate 21 LLMs across 13 multi-turn dialogue tasks and analyze performance by task, ability, dialogue turn, model size, alignment, and context. Results show uneven turn-dependent behavior, broad gains with model size, and no notable improvement from alignment techniques or chat-specific designs.
- Experimental Setup: 21 LLMs, including 2 close-sourced and 19 open-sourced models, are evaluated on MT-Bench-101.The evaluated models include GPT-3.5, GPT-4, and multiple open-source model families.
- Task Dimensional Analysis: GPT-4 achieves the highest overall task performance with an average score of 8.86, followed by Yi-34B at 8.10.Content confusion and format rephrasing are relatively easier tasks, whereas mathematical reasoning is the most challenging.
- Ability Dimensional Analysis: Models show widespread proficiency in rephrasing and resistance to interference, while reasoning and questioning remain weaker abilities.Memory performance exceeds understanding performance, and reflection and questioning are associated with higher overall scores.
- Alignment and Chat-Specific Models: Neither RLHF/DPO alignment nor chat-specific designs produces notable multi-turn improvements, and Baize and UltraLM are not exceptional among similarly sized models.The chat-specific models appear to be outperformed by other large language models of comparable size.
- Per-Turn Performance: Performance declines after the first turn in content rephrasing, format rephrasing, context memory, and anaphora resolution, with additional early drops in topic shift and content confusion.The reported declines suggest increasing forgetting or comprehension bias as dialogue progresses.
- Per-Turn Performance: Golden context creates apparent gains in separate input, directive clarification, proactive interaction, and mathematical reasoning, whereas general reasoning declines as dialogue complexity increases.The apparent gains arise from learning conversational styles, response patterns, or reasoning formats from the supplied golden context.
- Effect of Model Size: Larger models perform better on multi-turn dialogue tasks, with model size having a particularly strong effect on questioning ability.The authors interpret this questioning improvement as enhanced interactivity capability.
- Evaluation Validation: GPT-4 evaluation agrees with human expert judgments at 87%, exceeding the 80% agreement among human experts.The evaluation uses task-specific scoring guidelines and GPT-4 as an automated judge.
5 Conclusion
The paper presents a hierarchical taxonomy and uses MT-Bench-101 to evaluate multi-turn chat abilities across 21 LLMs. Its analyses find no notable benefit from alignment techniques or chat-specific designs, while case studies support the benchmark tasks' effectiveness.
- Conclusion: The paper introduces a hierarchical taxonomy of multi-turn chat abilities based on human-LLM interaction data and educational insights.The taxonomy is the basis for MT-Bench-101.
- Conclusion: MT-Bench-101 evaluates 21 LLMs and finds that alignment techniques and chat-specific designs do not notably improve multi-turn abilities.The conclusion summarizes the benchmark's broad evaluation findings.
- Conclusion: Extensive case studies indicate that the benchmark tasks effectively measure the corresponding multi-turn chat abilities.The paper also reports agreement between its automatic evaluation method and human judgments.
6 Limitations
Because LLM capabilities continue to evolve, MT-Bench-101 may not cover all multi-turn abilities. The authors therefore plan regular benchmark updates.
- Emerging multi-turn capabilities may not be encompassed by the study’s current findings.
- The authors intend to update MT-Bench-101 regularly to incorporate new developments.
7 Ethics Statement
The dataset was generated with GPT-4, reviewed by humans, and developed with participant consent and privacy protections. Residual annotation errors, biases, misuse risks, and taxonomy incompleteness remain possible.
- GPT-4 generated the data, which human annotators reviewed before dataset construction.
- Participants provided informed consent, and the study addressed privacy, autonomy, offensive content, and personal identity information.
- Residual errors or biases may remain because of inadvertent GPT-4 mistakes or annotator oversights.
- Public release could enable misuse for training, potentially reducing the benchmark’s effectiveness.
- The taxonomy may be incomplete as LLMs evolve, and the dataset is intended for research rather than unverified commercial use.
A Details on the Data Generation
The dataset spans diverse topics and is generated with uniform and task-specific prompts, then manually curated for task fidelity, topical coverage, diversity, recency, and safety.
- The dataset covers 30 topics, with at least 10 distinct topics represented for each task.
- GPT-4 generated preliminary samples, which human annotators manually filtered into the final dataset.
- Curation removed near-duplicate dialogues that differed only in several keywords.
- The dataset excludes real-time questions and knowledge requiring updates after 2022.
- Curation removed common-sense errors, offensive content, and personal identity information.
- Each task combines a uniform initial prompt with a unique task prompt to match the intended ability and task requirements.
B Detials on Data Statistics
Table 6 reports statistics for each MT-Bench-101 task and for the dataset overall, with word counts calculated from golden contexts.
- Table 6 provides per-task and overall statistics for MT-Bench-101.
- Word counts are calculated from the dataset’s golden context.
C Details on Evaluation
Evaluation uses task-specific prompts and scoring formats, with golden context supplied for selected tasks whose abilities begin in later turns.
- Evaluation prompts: Figures 21–34 combine uniform initial instructions, task-specific evaluation prompts, and a consistent scoring format.This design aligns scoring criteria with each task’s requirements.
- Context handling: Several tasks begin model responses directly at the second turn, using the first turn’s golden context as dialogue history.This applies to format rephrasing, content rephrasing, anaphora resolution, self-affirmation, self-correction, and context memory.
D Additional results utilizing Qwen-72B-Chat
Additional evaluation with Qwen-72B-Chat produces rankings consistent with GPT-4 judging, while case studies support the task design’s assessment accuracy.
- Judge-model comparison: GPT-4 remains the most powerful model, and GPT-4-Judge and Qwen-72B-Judge produce consistent rankings.Qwen-72B-Chat is presented as an open-source alternative evaluator.
- Case studies: Case studies corresponding to each task reflect classical model-response errors and support the accuracy of the corresponding ability assessments.Figures 35–45 contain these task-specific examples.
G Fleiss’ Kappa between GPT-4 and Humans
The paper evaluates agreement between GPT-4 and human judgments using Fleiss’ Kappa across several annotation comparisons.
- Agreement analysis: Fleiss’ Kappa is computed for five human raters, GPT-4 against each rater, GPT-4 against the human majority vote, and all GPT-4–human annotations.The paper reports that GPT-4–human agreement is higher than agreement among humans.
- Task prompts: The benchmark’s task prompts cover context memory, rephrasing, anaphora resolution, content confusion, topic shift, self-correction, self-affirmation, reasoning, instruction clarification, and proactive interaction.The supplied prompt descriptions specify distinct multi-turn interaction patterns for these tasks.