Source-linked AI summary
MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs
Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, Chen Xing
TL;DR
MultiChallenge addresses the limited evaluation of realistic multi-turn conversations, which require models to combine instruction following, context allocation, and in-context reasoning. The paper builds a hybrid human–LLM benchmark spanning four challenge categories and an instance-level-rubric judge, finding that frontier models remain below 50% accuracy, with Claude 3.5 Sonnet at 41.4%.
Problem
Realistic multi-turn conversations are difficult to evaluate because existing benchmarks are limited or saturated and often omit the combined capabilities required in human interactions.
Method
The paper constructs MultiChallenge through hybrid human–LLM data creation and evaluates responses with an LLM judge using instance-level binary rubrics.
Results
All frontier models score below 50% accuracy on MultiChallenge, while Claude 3.5 Sonnet achieves 41.4% average accuracy.
Takeaways & Limitations
MultiChallenge provides a realistic benchmark and an automatic evaluation approach for assessing challenging multi-turn conversation capabilities.
Takeaways & Limitations
The benchmark excludes examples whose rubric questions exceed current frontier-LLM capability, potentially limiting its overall difficulty.
Abstract
from arXiv · showhide
We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time. We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (June 2024) achieving just a 41.4% average accuracy.
1 Introduction
MultiChallenge addresses limited and saturated evaluation of realistic multi-turn conversations, where models must jointly follow instructions, allocate context, and reason in context. It introduces four challenging categories and reports sub-50% frontier-model accuracy despite near-perfect results on existing benchmarks.
- Motivation: Existing multi-turn benchmarks are limited, and some are saturated or focus narrowly on explicit instruction following.MT-Bench is described as producing near-perfect frontier-LLM results, while other evaluations omit mixed capabilities needed in natural conversations.
- Benchmark: MultiChallenge evaluates responses to final user turns using conversation histories of up to 10 turns.Each example contains a multi-turn history between two parties and ends with a requirement or question.
- Challenge categories: The benchmark covers instruction retention, inference memory, reliable versioned editing, and self-coherence.These categories target persistent constraints, scattered user information, iterative material revision, and coherence with prior model responses.
- Challenge categories: All four challenges require accurate instruction following, context allocation, and in-context reasoning simultaneously.The benchmark therefore targets a combination of capabilities rather than isolated instruction-following behavior.
- Results: Less than 50% accuracy was achieved by all frontier models, while Claude 3.5 Sonnet reached 41.4% average accuracy.The result contrasts with near-perfect scores on existing multi-turn evaluation benchmarks.
2 Related Work
Prior multi-turn benchmarks have not kept pace with frontier-model capabilities or the breadth of realistic conversation skills. Related work also includes multi-agent approaches for generating diverse synthetic datasets.
- Multi-turn evaluation: Multi-turn conversation benchmarks have received less attention than evaluations of agentic and multimodal capabilities.The paper positions conversation evaluation as comparatively underdeveloped.
- Multi-turn evaluation: MT-Bench and MT-Eval are described as no longer discriminative for frontier LLMs as model capabilities improve.These widely adopted benchmarks are presented as examples of evaluations whose separation power has diminished.
- Multi-turn evaluation: Some existing evaluations test verifiable formats and explicit instruction following rather than context attention and in-context reasoning.This leaves out the mixed capability set required in realistic multi-turn conversations.
- Synthetic data generation: Multi-agent frameworks have been used to create diverse, high-quality synthetic datasets with limited human intervention.The paper situates its construction approach among related multi-agent synthetic-data methods.
3 MultiChallenge
MultiChallenge defines four realistic multi-turn failure modes and supplies an automatic evaluation method based on instance-level rubrics. Its design targets instruction persistence, implicit memory, versioned editing, and resistance to contradictory agreement.
- Benchmark scope: MultiChallenge releases realistic examples for four common challenges that remain difficult for frontier LLMs.The categories are instruction retention, inference memory, reliable versioned editing, and self-coherence.
- Instruction Retention: Instruction Retention tests whether a first-turn instruction remains followed throughout the conversation.Examples include maintaining format or semantic constraints when answering the final user turn.
- Inference Memory: Inference Memory tests recalling and connecting user details scattered across earlier turns when the final request only implicitly requires them.The examples require reallocating attention to relevant history, such as avoiding allergens in requested recipes.
- Reliable Versioned Editing: Reliable Versioned Editing tests resolving references to earlier material versions, reproducing the correct version, and applying new edits.Back-and-forth revisions can include changed opinions and requests to return to earlier versions.
- Self-coherence: Self-coherence tests consistency with prior model responses rather than unconditional agreement with the user.The paper identifies sycophancy as a source of self-contradiction in multi-turn discussions.
- Automatic evaluation: 93% alignment with experienced human raters was achieved by instance-level-rubric judges, versus 36% for judges given raw conversation context.The rubric uses a binary question tied to the final model response, enabling automatic evaluation without a single ground-truth answer.
4 The Hybrid Approach to build MultiChallenge
MultiChallenge uses a hybrid pipeline that combines multi-agent synthetic generation with human review and editing to produce realistic, challenging test examples. MMSE coordinates planner, user, and responder agents, while human review checks category alignment, realism, and frontier-model failures.
- Hybrid construction: Human experts review and edit synthetically generated conversations to reduce effort while preserving data quality.The process first generates data synthetically, then relies on human experts for review and editing.
- Synthetic data generation: MMSE generates examples from hierarchical topic seeds, persona seeds, and challenge-specific evaluation configurations.Topic and persona seeds support coverage of varied domains and realistic user scenarios.
- Synthetic data generation: Three agent types collaborate: the Planner orchestrates strategy, the User converts strategy into realistic turns, and Responders answer using conversation history.Responder agents are randomly sampled from a pool of six frontier models for each test example.
- Synthetic data generation: The planner updates the conversation blueprint, detects failures, and either saves failed conversations for human editing or discards those reaching the turn limit without failure.The workflow continues until a failure is identified or the maximum turn limit is reached.
- Human review: Human review checks challenge-category alignment, naturalness, and whether at least three of six frontier LLMs fail reasonably.Accepted examples undergo two additional review layers by a separate reviewer pool.
5 Experiments
Experiments show that MultiChallenge distinguishes frontier models, supports reliable automatic evaluation, and exposes performance patterns beyond conversation length alone. It also shows that open-source models trail leading closed-source systems, while synthetic data reduces example-construction time but still requires substantial human editing.
- 5.1 Main Results: 38.3% average accuracy makes Claude 3.5 Sonnet the best-performing frontier model, followed by o1-preview at 34.6.
- 5.1 Main Results: Frontier models show different strengths across the four challenge categories, supporting MultiChallenge’s design of distinct capability targets.Claude leads overall but trails o1-preview on Reliable Versioned Editing and Information Memory; Gemini is on par with o1-preview on instruction retention.
- 5.1 Main Results: 93.91% average alignment with human raters makes instance-level-rubric auto-evaluation closely match human evaluation across all four challenges.Directly prompting frontier models as judges reaches only 36.01% alignment.
- 5.2 Analysis: Performance shows no visible trend as the number of conversation turns increases, suggesting difficulty is not caused simply by conversation length.The authors attribute the challenge potentially to reasoning required by the conversation context instead.
- 5.2 Analysis: All open-source models fall behind top-performing closed-source models such as Claude 3.5 Sonnet and o1-preview.The authors note that the benchmark may be biased against the six frontier models because examples required at least three of them to fail.
- 5.2 Analysis: 73.6 minutes of human editing per example with MMSE-generated synthetic data compares with 154.4 minutes from scratch, reducing required time by half.
- 5.2 Analysis: 25.5% difference between synthetic and final examples indicates that human editing remains substantial despite MMSE’s time savings.Average string similarity between original synthetic and human-edited examples is 74.5%.
6 Conclusion
MultiChallenge advances evaluation of LLMs’ multi-turn conversation capabilities through a realistic benchmark covering four challenging interaction types and an automated evaluation method.
- MultiChallenge covers four realistic multi-turn challenges that remain difficult for frontier LLMs.The benchmark addresses instruction retention, inference memory of user information, reliable versioned editing, and self-coherence.
- AI-human collaborative construction makes the benchmark comprehensive across these four conversation challenges.
- Instance-level rubrics enable trustworthy automatic evaluation for this challenging benchmark.
Limitations
The benchmark’s evaluation scope is constrained by judge capability and by its construction around failures shared by six frontier model families.
- Examples beyond current frontier LLM judging capability are excluded, potentially limiting the benchmark’s difficulty.The authors retain these difficult examples privately and plan public release once automatic evaluation becomes possible.
- MultiChallenge is inevitably biased against six frontier models because test examples target their common failures.The authors nevertheless report that tested open-source models fall behind top-performing closed-source models such as Claude 3.5 Sonnet and o1-preview.
A.1 Hierarchical Topic Taxonomy
The hierarchical taxonomy organizes multi-turn evaluation examples across editing and instruction-retention topics, including content types, response styles, and conversational constraints.
- Challenge Category - Reliable Version Editing: Reliable version editing spans speeches, emails, memos, customer-service responses, presentations, websites, advertisements, slogans, itineraries, and project plans.
- Challenge Category - Reliable Version Editing: The taxonomy also includes technical code, budgets, business proposals, resumes, job descriptions, research documents, study plans, workshops, surveys, reports, policies, and user materials.
- Challenge Category - Instruction Retention: Instruction retention covers tone, specialized vocabulary, response structure, grammar, embedded words, agreement, objective personas, poetic formats, and instructional modes.
- Challenge Category - Inference Memory: Inference memory includes dietary restrictions, cuisine and taste preferences, scheduling constraints, recurring events, relationships, travel details, fitness progress, collaborators, and media preferences.
A.2 Challenge Category Evaluation Configuration
The evaluation configurations test whether models retain specific information and instructions, integrate evolving directives, and maintain consistency across conversation turns.
- Inference Memory: Inference memory tests recall and accurate reuse of specific information from earlier turns when later responses require it.Forgetting or misremembering earlier details is treated as failure, including neglecting a previously stated nut allergy.
- Instruction Retention: Instruction retention tests adherence to a specific instruction introduced in the first user message throughout the dialogue.Any deviation from the initial instruction is considered a failure, such as expressing an opinion after being told to remain neutral.
- Reliable Version Editing: Reliable version editing evaluates adaptation to multiple, evolving instructions over increasingly complex or layered directives.The configuration focuses on flexibility as responses evolve with the conversation.
- Self-Coherence: Self-coherence evaluates whether new responses remain internally consistent rather than contradicting earlier statements.A contradiction, such as changing the Earth–Moon distance across turns, counts as failure when it results from the model’s limitations.
A.3 Agent System Prompts
The agent prompts define a planner–user system that designs, executes, and adaptively evaluates realistic conversations for clear model failures.
- Planner Agent: The blueprint specifies one subtle challenge, natural conversational strategies, tactics for perturbing context, and a precise endpoint for identifying failure.The endpoint may occur at a specific turn or across multiple turns, within a maximum of {num_turns} turns.
- Iterative Evaluation: The planner must refine its entire blueprint after each conversation state update and change strategy when the model handles the current approach well.The prompt separates completed and future turns while requiring adaptive strategy changes.
- Failure Criterion: After a successful break, the system records the model’s last response, explains why it qualifies, and marks the conversation as stopped.The planner output includes a rationale for why the break is not naive or gaslighting.
- User Agent: The user agent converts the blueprint into realistic turns that indirectly test the specified axis through contradictions, explanations, diversions, and context shifts.The prompt requires interactions representative of genuine chatbot conversations rather than artificial tests.
- Failure Criterion: A failure counts only when it is clear and undeniable; minor issues, technicalities, gotchas, and gaslighting do not qualify.The user agent continues testing when uncertainty or room for interpretation remains.
A.4 Frontier Models used as Responder Agents
The evaluation uses six frontier responder models spanning OpenAI, Meta, Mistral, Anthropic, Google, and an o1 preview release.
- Frontier Models: The evaluated responder agents are GPT-4o, Llama 3.1 405B Instruct, Mistral Large, Claude 3.5 Sonnet, Gemini 1.5 Pro, and o1-preview.The list identifies the model releases or associated dates where provided.
A.5 Inference Settings for Models
Table 6 reports the inference temperatures used for the language models evaluated on MultiChallenge.
- Inference Settings: Table 6 lists inference temperatures for the LLMs evaluated on MultiChallenge.
A.6 Model Performance over time
Figure 8 presents model performance over time, organized according to model release dates.
- Performance Trends: Figure 8 plots model performance over time based on release dates.
A.7 Conversation Blueprint Examples
The appendix illustrates initial and refined conversation blueprints for testing recollection, with refinement adapting the strategy after the initial interaction.
- The example persona is a food-loving travel blogger lamenting the closure of a favorite restaurant.
- The recollection blueprint tests whether a model can recall and accurately reference earlier details when modifying a reminder without those details being repeated.
- The examples distinguish an initial conversation blueprint from a refined blueprint adapted based on the initial interaction.
- The updated blueprint extends the recollection test by adding complex elements while still requiring implicit recovery of earlier details.
A.8 Case Study: Model Failure Analysis
The appendix presents case studies of model failures for each challenge using complete conversations, testing analyses, failure criteria, and evaluations of six models.
- Each case study includes a full conversation transcript, an analysis of the tested challenge, failure criteria, and detailed evaluations of six language models.
Instruction Retention
The examples test whether models preserve user constraints and earlier details across evolving multi-turn conversations. Failures include violating age restrictions, forgetting allergies or missed steps, and applying the wrong schedule version.
- Instruction Retention: Instruction Retention: The film-festival task requires bulleted replies and avoiding films with UK age ratings of 15 or 18 throughout the conversation.
- Failure Criteria: Failure Criteria: A model fails if its final recommendations include any film rated 15 or 18 in the UK.
- Model Evaluations: Model Evaluations: Successful responses maintained the age-rating constraint while recommending family-friendly films such as Mulan and Coco.
- Model Evaluations: Model Evaluations: Multiple responses violated the age restriction by recommending films including Gladiator, Elizabeth, The Last Samurai, Amistad, and Frida.
- Model Evaluations: Model Evaluations: Other responses failed through a single Shakespeare in Love recommendation or multiple choices such as The Last Emperor, Dances With Wolves, and Gladiator.
- Model Evaluations: Model Evaluations: The allergy task distinguishes models that consistently avoid nut-containing desserts from those recommending cashews, almonds, peanuts, or pecans.
- Conversation Transcript: Conversation Transcript: Schedule revisions require preserving changed constraints, including a 4 PM end time and a 90-minute networking lunch.
- Model Evaluations: Model Evaluations: Models also confuse schedule versions, either retaining the wrong lunch duration or correctly incorporating the requested one-hour lunch.