PaperlayerView exact revision on arXiv

Source-linked AI summary

MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models

Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong

arXiv:2401.16745v1cs.CL

TL;DR

Existing benchmarks do not adequately evaluate LLMs’ capabilities in complex multi-turn conversations. MT-Eval categorizes interaction patterns, constructs 1,170-turn evaluations, and compares multi-turn with single-turn performance across 11 models. It finds broad multi-turn degradation, with content distance and error propagation identified as key factors.

  • Problem

    Existing evaluation frameworks predominantly use single-turn or limited-turn queries, leaving complex multi-turn conversational abilities insufficiently evaluated.

    Source: §1 · p. 1

  • Method

    MT-Eval categorizes conversations into Recollection, Expansion, Refinement, and Follow-up, constructs benchmark instances, and compares multi-turn performance with single-turn counterparts.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    Most LLMs perform worse in multi-turn than single-turn settings; GPT-4 leads overall, while some open-source models match or exceed GPT-3.5-Turbo on specific tasks.

    Source: §1 · p. 2, §4.4 · p. 5

  • Takeaways & Limitations

    Distance to relevant content and susceptibility to error propagation are key factors influencing multi-turn performance.

    Source: §1 · p. 2, §5 · p. 8

    The experiments exclude larger open-source models such as Llama2-chat-70B because of computational limits.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly relied upon for complex multi-turn conversations across diverse real-world applications. However, existing benchmarks predominantly focus on single-turn evaluations, overlooking the models' capabilities in multi-turn interactions. To address this gap, we introduce MT-Eval, a comprehensive benchmark designed to evaluate multi-turn conversational abilities. By analyzing human-LLM conversations, we categorize interaction patterns into four types: recollection, expansion, refinement, and follow-up. We construct multi-turn queries for each category either by augmenting existing datasets or by creating new examples with GPT-4 to avoid data leakage. To study the factors impacting multi-turn abilities, we create single-turn versions of the 1170 multi-turn queries and compare performance. Our evaluation of 11 well-known LLMs shows that while closed-source models generally surpass open-source ones, certain open-source models exceed GPT-3.5-Turbo in specific tasks. We observe significant performance degradation in multi-turn settings compared to single-turn settings in most models, which is not correlated with the models' fundamental capabilities. Moreover, we identify the distance to relevant content and susceptibility to error propagation as the key factors influencing multi-turn performance. MT-Eval is released publicly to encourage future research towards more robust conversational models.

1 Introduction

MT-Eval addresses the limited evaluation of LLMs in complex multi-turn conversations by introducing a benchmark grounded in common interaction patterns. Across 11 models, it compares multi-turn with single-turn performance and identifies distance to relevant content and error propagation as key influences.

Motivation

  • Existing benchmarks largely overlook multi-turn conversation, despite LLM assistants needing to retain context and adapt across evolving interactions.MMLU uses single queries, while MT-Bench uses only two-turn interactions and does not cover more extended or varied conversations.Source: §1 · p. 1

Benchmark

  • MT-Eval categorizes real-world interactions into Recollection, Expansion, Refinement, and Follow-up, then constructs evaluation sets through dataset augmentation or new GPT-4-generated instances.The instances were manually reviewed and revised for quality assurance and to reduce data contamination.Source: §1 · p. 1

Evaluation

  • 11 models were evaluated on multi-turn response quality and on performance differences between identical single-turn and multi-turn queries.Combining both views provides a broader assessment of multi-turn conversational capability.Source: §1 · pp. 1–2

Findings

  • GPT-4 leads multi-turn performance, some open-source models match GPT-3.5-Turbo on specific tasks, and most models perform worse in multi-turn than single-turn settings.The multi-turn gap is not related to models’ fundamental capacities.Source: §1 · p. 2
  • Greater distance to relevant content and accumulated dialogue errors are identified as key factors behind declining multi-turn performance.The analysis links these factors to models’ difficulty maintaining reliable use of dialogue history.Source: §1 · p. 2

2 Related Work

Prior work evaluates selected aspects of multi-turn interaction, but existing benchmarks remain limited in scale, turn depth, scalability, or domain coverage. MT-Eval instead targets comprehensive multi-turn conversation across multiple interaction types.

Existing benchmarks

  • MT-Bench evaluates conversational flow and instruction following with 80 questions but limits each conversation to two turns.This restricts coverage of longer conversational contexts.Source: §2 · p. 2
  • HALIE evaluates human-AI interaction but depends on human participation, limiting scalability and efficiency across tasks.Source: §2 · p. 2
  • Other frameworks focus on specific domains or capabilities, including medical consultation, tool use, and feedback utilization.MT-Eval is positioned as evaluating comprehensive multi-turn conversation, potentially involving multiple dialogue types in one session.Source: §2 · p. 2

3 MT-Eval

MT-Eval organizes realistic multi-turn interactions into four categories and packages them into a benchmark of 168 sessions and 1,170 turns. Its construction combines adapted and newly created tasks with single-turn counterparts where feasible.

3 MT-Eval

  • MT-Eval identifies Recollection, Expansion, Refinement, and Follow-up as four primary modes of user-assistant interaction.They respectively involve recalling prior information, exploring a shared topic, incorporating changing instructions, and building on the assistant’s previous response.Source: §3 · pp. 2–3, Figure 1 · p. 3
  • The benchmark contains 168 dialogue sessions with 1,170 turns spanning document processing, content creation, and information retrieval scenarios.Source: §3 · p. 3

3.1 Construction

  • Construction adapts document-based NLP tasks into dialogues, creates new underlying documents with LLMs, and manually reviews generated instances for difficulty, relevance, and originality.Refinement adds constraints across turns, while Expansion presents distinct tasks about the same document.Source: §3.1 · p. 3
  • Recollection includes document classification and global instruction following, while Follow-up extends MT-Bench with three additional questions based on preceding responses.Source: §3.1 · p. 3
  • Single-turn counterparts are created for separate turns in every task except Follow-up, which inherently depends on the assistant’s previous response.For Refinement, GPT-4 condenses multiple instructions into one single-turn instance.Source: §3.1 · pp. 3–4
  • Table 2 reports multi-turn performance across the four task categories, with closed-source models generally outperforming open-source models.Source: Table 2 · p. 4

4 Experiment

The experiments evaluate 11 LLMs across four multi-turn task categories, comparing multi-turn with single-turn performance and analyzing instruction adherence, context distance, and error propagation. Results show GPT-4 leads overall, multi-turn performance usually declines relative to single-turn, and earlier context plus accumulated errors constrain performance.

4.4 Result

  • GPT-4 achieves the highest average score at 9.03, while Mixtral-Instruct-8x7B reaches 9.52 in Follow-up dialogues versus GPT-3.5-Turbo’s 9.21.Closed-source models generally outperform open-source models, although some open-source models match or exceed GPT-3.5-Turbo on particular tasks.Source: §4.4 · p. 5
  • Most models perform substantially worse in multi-turn than single-turn settings, and this gap does not track their fundamental capabilities.Llama-2-chat models outperform Vicuna models in single-turn evaluation but lag behind them in multi-turn dialogues.Source: §4.4 · p. 5, Table 3 · p. 5
  • As dialogue length increases, models increasingly overlook earlier information, with performance lower on later Refinement turns and most models struggling with initial global instructions in Recollection.The second Refinement task occurs farther from the initial document, while GPT-4 is the main exception in Recollection.Source: §4.4 · p. 5, Figure 2 · p. 6, Table 4 · p. 6

4.5 Further Analysis

  • Among 200 analyzed responses, 49.5% failed to comply with earlier instructions, 48% reflected propagated prior errors, and 2.5% involved evaluation errors.The analysis examines ten largest single-turn versus multi-turn score differences for each of five top-performing models.Source: §4.5 · pp. 6–7

4.6 Ablation Study

  • Gold dialogue context improves Recollection and Refinement by reducing error propagation and supplying in-context examples, whereas single-class or randomly labeled history can harm weaker models.Gold context yields only slight improvement in Expansion because its turns are distinct NLP tasks; performance at turn 10 is worse than at turn 5 under Random Class.Source: §4.6 · pp. 7–8
  • Distracting turns inserted between documents and queries consistently degrade performance, supporting the role of increasing distance from relevant context; GPT-4 is minimally affected.Irrelevant turns placed at the beginning produce mixed outcomes, unlike turns inserted between the document and query.Source: Table 6 · p. 7, §4.6 · p. 8

5 Conclusion

MT-Eval is presented as an important first step toward systematically evaluating and understanding multi-turn conversational abilities. Its analysis identifies distance to relevant content and susceptibility to error propagation as key factors in performance decline.

  • MT-Eval provides a systematic framework for evaluating and understanding LLMs’ multi-turn conversational abilities.Source: §5 · p. 8
  • A pronounced single-turn versus multi-turn performance gap persists across current models regardless of their underlying capabilities.Source: §5 · p. 8
  • Distance to relevant content and susceptibility to error propagation are identified as key factors causing declines in multi-turn performance.Source: §5 · p. 8

Limitations

The appendices document MT-Eval’s evaluation and data-construction prompts, while case studies illustrate recurring multi-turn failures. The limitations include persistent GPT-4 evaluation errors and incomplete coverage of larger open-source models.

Limitations

  • GPT-4 evaluation errors persist despite alignment with human ratings and the use of chain-of-thought prompting.More advanced reasoning prompts could improve evaluation quality but would increase inference cost.Source: Limitations · p. 9
  • The experiments exclude larger open-source models such as Llama2-chat-70B because of computational limits.Further work is needed to test whether larger open-source LLMs show similar findings.Source: Limitations · p. 9

A Prompts

  • The appendix provides prompts for constructing follow-up, recollection, refinement, and expansion queries and converting selected multi-turn tasks into single-turn formats.Source: A Prompts · p. 12

A.1 Data Construction Prompts

A.2 Evaluation Prompt / B Case Study

C Task Examples

The task examples illustrate how MT-Eval operationalizes refinement, expansion, follow-up, and recollection interactions. Together, they show the benchmark’s coverage of instruction revision, topic development, response-based questioning, and persistent global constraints.

  • The Refinement example demonstrates an interaction in which an existing response or instruction is revised.Source: Figure 14 · p. 18
  • The Expansion example illustrates extending a discussion or document beyond its initial content.Source: Figure 15 · p. 19
  • The Follow-up example builds later questions on earlier dialogue turns, including initial turns from MT-Bench (Zheng et al., 2023b).Source: Figure 16 · p. 19
  • The Recollection example tests whether a model obeys a global instruction introduced at the beginning of the conversation.Source: Figure 17 · p. 20

D Human Evaluation

Human evaluation checks the reliability of GPT-4-based response assessment on sampled instances from three MT-Eval tasks. Five graduate students used the same evaluation instructions, achieving satisfactory inter-rater agreement.

  • Five graduate students evaluated 60 randomly selected instances from each Follow-up, Refinement, and Expansion task.Source: D Human Evaluation · p. 21
  • The human evaluators received the same instructions used for GPT-4 to support consistency between human and automated assessment.Source: D Human Evaluation · p. 21
  • Cohen’s kappa was 0.58 on a separate set of 20 instances, indicating satisfactory inter-rater agreement.Source: D Human Evaluation · p. 21

E Benchmark Construction Details

MT-Eval constructs four multi-turn task categories that cover recollection, follow-up, refinement, and expansion, with additional single-turn conversions for comparison. The benchmark combines existing dialogue material with GPT-4-generated or transformed tasks and reports task statistics separately.

E Benchmark Construction Details

  • MT-Eval organizes benchmark construction around Recollection, Follow-up, Refinement, and Expansion tasks.The section explicitly describes these four task categories as the benchmark’s construction focus.Source: E Benchmark Construction Details · p. 21
  • Recollection includes document classification over 13 categories and global-instruction following across ten-turn dialogues.The classification subtask uses 100 documents across 10 dialogues, while the instruction-following subtask uses 14 instructions, two sessions per instruction, and 280 dialogue turns.Source: E Benchmark Construction Details · p. 21
  • Follow-up expands 80 two-turn MT-Bench dialogues with three additional turns, using GPT-4 generation followed by student review.This expansion adds 240 dialogue turns in total.Source: E Benchmark Construction Details · p. 21
  • Refinement creates six increasingly constrained instructions across document-based question answering, summarization, named-entity recognition, and paragraph rewriting.The construction yields 40 dialogue instances and 480 dialogue turns.Source: E Benchmark Construction Details · p. 21
  • Expansion presents seven NLP tasks in random order over the same initial document, while multi-turn instructions are also converted into single instructions for comparison.The seven tasks include question answering, summarization, named-entity recognition, part-of-speech tagging, relation extraction, translation, and classification.Source: E Benchmark Construction Details · p. 21

F Additional Statistics

  • The appendix reports detailed statistics for individual MT-Eval tasks in Table 10.The supplied passage identifies the statistics table but does not provide its cell values.Source: F Additional Statistics · p. 21

G Additional Results

This section points to a breakdown of Recollection results, but the supplied passages provide no numerical findings.

G Additional Results

  • The section includes a passage introducing the Recollection-results table.No substantive comparison or numerical result is available in the supplied text.Source: G Additional Results · p. 21
  • The supplied excerpts do not support identifying a winning model or quantitative Recollection finding.Only table-location text is provided.Source: G Additional Results · p. 21
  • The Recollection results are reported in Table 11.The supplied passage identifies the table but does not provide its values.Source: G Additional Results · p. 21

H The Impact of Dialogue History in the Follow-up Task.

The Follow-up analysis compares single-turn and multi-turn settings after condensing dialogue history into standalone queries. It reports that dialogue history improves performance in this task by supplying prior responses and intermediate reasoning as useful context.

H The Impact of Dialogue History in the Follow-up Task.

  • GPT-4 condenses dialogue history into complete single-turn queries for all 240 Follow-up instances, followed by manual review and refinement.The initial generated queries often omitted crucial contextual information, motivating the review step.Source: H The Impact of Dialogue History in… · p. 22
  • All evaluated models perform significantly better in the multi-turn than single-turn Follow-up setting.The result is reported for the comparison summarized in Table 13.Source: H The Impact of Dialogue History in… · p. 22, Table 13 · p. 25
  • Models often use previous responses and explanations as in-context examples to improve their current-turn answers.The supplied example with Qwen-chat-7b attributes an accurate response to intermediate reasoning preserved in the dialogue history.Source: H The Impact of Dialogue History in… · p. 22, Figure 18 · p. 23
  • The cited captions identify Table 10 as detailed statistics for individual MT-Eval tasks and Tables 11–12 as Recollection-task breakdowns and instruction-adherence turn counts.Table 11 defines Global Inst. as global instruction following and CLS as document classification; Table 12 averages turns over two dialogues per instruction.Source: Table 10 · p. 24, Table 11 · p. 24, Table 12 · p. 25
  • Table 13 is the designated visual comparison of single-turn and multi-turn Follow-up performance.Its caption states that the multi-turn results are taken from Table 2.Source: Table 13 · p. 25
Loading 2401.16745v1…