MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs

1 Introduction

• Existing frameworks are limited: MT-Bench is saturated by frontier models, while other evaluations focus on explicit instruction-following and miss the mixed capabilities needed for natural conversations (Zheng et al., 2023; Achiam et al., 2023; Yang et al., 2024; He et al., 2024).The paper frames multi-turn conversation as common but challenging because it combines instruction-following, attention to conversational context, and near-human or better in-context reasoning.Source: §1 · p. 1
• MultiChallenge tests responses to a final user request against conversation histories of up to 10 turns, targeting instruction retention, inference memory, reliable versioned editing, and self-coherence.The benchmark’s four challenge categories are described as common and realistic in current human–LLM interactions, yet difficult for frontier models.Source: §1 · p. 1
• The challenges jointly require instruction-following, context allocation, and in-context reasoning, covering retained initial instructions, implicit user details, iterative revision, and coherent responses that avoid sycophancy.Inference memory concerns recalling and connecting relevant details scattered across prior user turns; self-coherence includes avoiding unconditional agreement.Source: §1 · pp. 1–2
• 41.4% average accuracy was the best MultiChallenge score, achieved by Claude 3.5 Sonnet (June 2024); all frontier models scored below 50% despite near-perfect results on existing benchmarks (Achiam et al., 2023; Yang et al., 2024).Human and automatic evaluations showed high correlation, and the instance-level-rubric judge agreed well with experienced human raters.Source: §1 · p. 2
• A hybrid human–LLM process produces diverse, realistic, challenging test examples, while an LLM judge using instance-level rubrics enables automatic evaluation with good agreement with experienced raters.The paper reports that the benchmark’s data and code are open source.Source: §1 · p. 2

Evaluation benchmarks on multi-turn conversations

• GPT-4, Claude 3, and Llama3 achieve near-perfect scores on widely adopted MT-Bench (Zheng et al., 2023) and MT-Eval (Kwan et al., 2024), which are no longer discriminative for frontier models.Benchmarks for agent and multimodal capabilities have emerged more quickly than multi-turn conversation evaluations.Source: §2 · p. 2
• Multi-IF (He et al., 2024) extends IFEval (Zhou et al., 2023) to multi-turn and multilingual settings, emphasizing automatically verifiable instruction following rather than realistic conversations’ mixed capabilities.The latter include context attention allocation and in-context reasoning.Source: §2 · p. 2

Multi-agent synthetic data generation

• MultiChallenge uses multi-agent generation for synthetic conversations across its four challenges, then hybrid editing and review to create the final benchmark.Prior examples of multi-agent synthetic-data frameworks include AgentInstruct (Mitra et al., 2024), MAG-V (Sengupta et al., 2024), and MATRIX (Tang et al., 2024).Source: §2 · p. 2

3.1.1 Instruction Retention.

• Instruction Retention tests whether models follow an explicit first-turn instruction through the final turn, without contradictory follow-up instructions.Instructions range from format constraints to semantic constraints, such as movie ratings.Source: §3.1.1 · pp. 2–3

3.1.2 Inference memory (of user information).

• Inference Memory tests whether models recall and connect earlier user details that are implicitly relevant to the final request, rather than directly asked for.For example, a prior nut allergy should inform a later request for dessert recipes.Source: §3.1.2 · p. 3

3.1.3 Reliable versioned editing.

• Reliable Versioned Editing tests whether models resolve references to earlier edited versions, reproduce the correct version without hallucination, and apply new instructions.The tasks reflect back-and-forth editing in which users may abandon current edits and return to earlier versions.Source: §3.1.3 · pp. 3–4

3.1.4 Self-coherence.

• Self-Coherence tests whether models remain reasonably consistent with prior responses instead of agreeing with users’ contradictory prompts, a sycophantic tendency noted by Perez et al. (2022), Sharma et al. (2023), Laban et al. (2023), and Agarwal et al. (2024).The benchmark highlights how abandoning an earlier answer can make conversation flow unnatural and reduce user trust.Source: §3.1.4 · p. 4

3.2 Automatic evaluation with instance-level rubrics

• 93% alignment with experienced human raters was achieved by frontier-model judges using instance-level rubrics, versus 36% when judges received raw conversation context.Each test example has a human-written yes/no rubric question answerable from the final model response; this addresses the lack of a single ground-truth answer and the low alignment of direct judging.Source: §3.2 · p. 5

4.1 Synthetic Data Generation

• MMSE uses topic seeds, a persona seed, and a challenge-specific evaluation configuration to generate candidate test examples.Topic and persona seeds promote diversity; the configuration specifies the category, definition, pass and failure criteria, and failure examples for the planner.Source: §4.1 · p. 5
• The planner iteratively updates a conversation blueprint as user and responder agents converse, saving conversations when it detects failure and discarding those that reach the turn limit without failure.For each example, the responder is randomly sampled from a pool of six frontier models to avoid overfitting generation to one model’s weaknesses.Source: Figure 3 · p. 5, §4.1 · p. 6

4.2 Human Review and Editing

• Human reviewers accept examples only when at least 3 of 6 frontier LLMs fail reasonably and the conversation fits its challenge category and is natural and realistic.Annotators edit or discard examples that do not meet these criteria.Source: §4.2 · p. 6
• Two review layers, staffed by reviewers distinct from the initial annotators, accept only examples that pass both layers.Annotators receive guidance on unfair failures, editing, and the review workflow.Source: §4.2 · p. 6

5.1 Main Results

• 38.3% average accuracy puts Claude 3.5 Sonnet (June 2024) first, followed by o1-preview at 34.6% and Gemini 1.5 Pro (August 27, 2024) at 19.1%.Human raters reviewed six anonymous model responses per example, followed by two reviewer layers.Source: §5.1 · pp. 6–7, Table 2 · p. 7
• Claude leads overall but trails o1-preview on Reliable Versioned Editing and Information Memory, while Gemini is on par with o1-preview on instruction retention.Performance differences across the four challenges support the benchmark’s design as a test of distinct LLM capabilities.Source: Table 2 · p. 7, §5.1 · p. 7

5.2 Analysis

• 93.91% average alignment with human raters contrasts with 36.01% for direct full-history judging; rubric-based auto-evaluation closely matches human results and rankings.Alignment on each of the four challenges exceeds 90%; the auto-evaluation conclusions also held when Claude rather than GPT-4o judged responses.Source: Table 4 · p. 7, §5.2 · pp. 7–8, Table 3 · p. 7
• Turn count shows no visible performance trend, while larger models—especially those at 70–72B parameters—and newer models generally score higher, with non-linear gains over time.The authors suggest conversation-context reasoning, rather than length alone, may explain the benchmark’s difficulty; Claude 3.5 Sonnet (June 2024) outperforms some later models.Source: §5.2 · p. 8, Figure 4 · p. 8, Figure 5 · p. 9
• Despite potential bias against frontier models, every tested open-source model falls behind top closed-source models Claude 3.5 Sonnet and o1-preview.The benchmark examples were selected because at least three of the six frontier models failed them.Source: §5.2 · p. 8, Table 5 · p. 8
• 73.6 minutes of human editing per example with MMSE-generated synthetic data compares with 154.4 minutes when producing examples from scratch, roughly halving editing time.The authors report that this reduction significantly lowered cost.Source: §5.2 · pp. 8–9
• 74.5% average string similarity between synthetic examples and final human-edited examples indicates 25.5% difference from human editing.The comparison uses Levenshtein-distance-based string similarity.Source: §5.2 · p. 9

• MultiChallenge advances evaluation of LLMs’ ability to conduct multi-turn conversations with human users, a previously overlooked capability.Source: §6 · p. 9
• Its AI-human collaborative construction yields a comprehensive benchmark covering four realistic challenges that are currently difficult for frontier LLMs.Source: §6 · p. 9
• Instance-level rubrics for the LLM judge make automated evaluation of this challenging benchmark possible and trustworthy.Source: §6 · p. 9

Limitations

• To keep LLM judging trustworthy, the benchmark excludes examples whose instance-level rubric questions exceed current frontier-model capability, potentially limiting difficulty.The authors keep these difficult examples private and plan to release them when LLM-based automatic evaluation becomes possible.Source: Limitations · p. 9
• Because test examples target failures common to six frontier models, the benchmark is inevitably biased against those models.The authors nevertheless report that tested open-source models fall behind top-performing closed-source models such as Claude 3.5 Sonnet and o1-preview, and say comparisons among future releases from the six frontier families and existing open-source models remain fair.Source: Limitations · p. 9

A.1 Hierarchical Topic Taxonomy

• Reliable Version Editing spans code, technical documents, speeches or scripts, emails, and memos, with revisions targeting clarity, accuracy, tone, structure, or effectiveness.The taxonomy also includes other written, professional, planning, and educational materials.Source: §A.1 · pp. 12–13
• Instruction Retention covers persistent tone and vocabulary choices, limited response types, and challenging formats such as poetry or step-by-step instruction.Other listed constraints include required elements, consistent formats, grammar, and specific words in every response.Source: §A.1 · pp. 17–18
• Inference Memory scenarios test whether models recall user-specific food preferences, event dates and conflicts, relationship details, and health conditions when making later recommendations.Other examples include travel context, work and learning goals, hobbies, purchases, brands, and entertainment preferences.Source: §A.1 · pp. 19–21
• The model must recall user habits, task and reminder details, chore frequency, emotional experiences, and cultural background when responding in later turns.Examples include using reading habits for book recommendations and adapting reminders after the user changes them.Source: §A.1 · pp. 23–24
• Self-coherence examples require consistent enrollment, budget, time, historical, scientific, legal, admission, and personal details across turns.These include retaining a stated capacity or estimate and avoiding conflicting facts or eligibility requirements later.Source: §A.1 · pp. 25–26
• Definition and explanation coherence requires the model to keep term definitions, concept explanations, and technical jargon consistent when revisited.The taxonomy also names recommendation consistency as a separate challenge.Source: §A.1 · p. 27
• Recommendation consistency covers avoiding unexplained reversals in product, diet or health, and travel advice across turns.Examples include not later recommending a conflicting product or advising against a previously suggested destination without context.Source: §A.1 · p. 27
• Instruction and mathematical coherence require stable task steps, safety guidance, equations, financial figures, and unit-conversion factors.The model should preserve the sequence and details of instructions when reiterating them.Source: §A.1 · p. 28
• Ethical consistency includes maintaining cultural references, language and tone, scenario details, ethical advice, and alignment between legal or ethical guidance and later suggestions.The examples include avoiding a tone shift unless prompted and not contradicting advice to follow the law.Source: §A.1 · p. 29
• Story or narrative consistency requires retaining established character traits, plot points, and world-building rules throughout a story.For example, a model should not later contradict a character description or introduce a rule that conflicts with the fictional world.Source: §A.1 · p. 30
• Opinion consistency covers keeping expressed views about products or services, tastes, and political or social issues stable in related discussions.A different preference may be expressed later if the change is contextualized.Source: §A.1 · p. 30

Context Recall

• Context recall tests whether the model remembers and accurately references specific details from earlier turns; forgetting or misremembering key information is a failure.The test emphasizes information introduced multiple turns earlier and later made relevant.Source: §A.2 · p. 31

Sustained Instruction Following

• Instruction following tests whether the model maintains an explicit instruction given in the first user message for the entire conversation; any deviation is a failure.The instruction is tested across a variety of user questions.Source: §A.2 · p. 32

Reliable Version Editing

• Reliable version editing tests whether the model integrates multiple evolving directives without forgetting, contradicting, or deprioritizing earlier instructions unless explicitly told to do so.Failure includes ignoring an earlier directive for a newer one or not fully accommodating the instructions provided.Source: §A.2 · p. 33

Consistency

• Consistency tests whether new answers conflict with the model’s prior statements; contradictions caused by the model’s limitations count as failures, but user manipulation or gaslighting does not.One example is giving incompatible distances from Earth to the Moon in successive answers.Source: §A.2 · p. 34

A.3 Agent System Prompts

• The planner must design one subtle challenge, weave it naturally into a conversation, specify an exact failure endpoint, and stay within the turn limit.The endpoint may occur at one turn or across multiple turns, and need not be the final turn.Source: §A.3 · p. 35
• The planner refines its blueprint using the conversation state and must change future strategy when the model handles the current approach well.It prints the full blueprint each time, separating completed turns from future turns.Source: §A.3 · p. 35
• The user agent follows the blueprint but adapts to responses, using natural, indirect, context-free messages that make the model infer context from prior turns.Suggested tactics include vague questions, casual remarks, contradictions, explanations, diversions, and subtle context changes.Source: §A.3 · p. 37
• A break must be clear and undeniable, not a minor issue, technicality, gotcha, or gaslighting; otherwise, the agent continues testing and gives the model the benefit of the doubt.When ending, the prompts require stating the failure and why it is not gaslighting, then signaling that the conversation has stopped.Source: §A.3 · pp. 36–37

• The cited passages list GPT-4o (August 2024) (Hurst et al., 2024) and Llama 3.1 405B Instruct (Dubey et al., 2024), without specifying their roles.Source: §A.4 · p. 39
• The supplied passages list Mistral Large (November 2024) (Mistral AI, 2024) and Claude 3.5 Sonnet (October 2024).Source: §A.4 · p. 39
• The listed models are Gemini 1.5 Pro (August 27 2024) (Team et al., 2024) and o1-preview (Jaech et al., 2024).Source: §A.4 · p. 39

A.5 Inference Settings for Models

• Table 6 lists inference temperatures for LLMs evaluated on MultiChallenge.The supplied passage does not include the temperature values.Source: Table 6 · p. 40

A.6 Model Performance over time

• Figure 8 presents model performance over time based on model release dates.Source: Figure 8 · p. 41

A.7 Conversation Blueprint Examples

• The initial conversation blueprint lays out a strategic plan for model evaluation.Source: Figure 9 · p. 42
• The blueprint is then executed across conversation turns 1 and 2.Source: Figure 10 · p. 43
• A refined conversation blueprint adapts the plan based on the initial interaction.Source: Figure 11 · p. 44

A.8 Case Study: Model Failure Analysis

• Each case study includes a full conversation transcript, challenge analysis, failure criteria, and detailed evaluations of six models’ responses.Source: §A.8 · p. 45

Conversation Transcript

• The user requests bulleted replies throughout and family-audience recommendations, then asks for historical films representing diverse cultures and periods.The historical-film request follows earlier discussion of festival strands.Source: Conversation Transcript · pp. 45–46

Failure Criteria

• Any film rated 15 or 18 in the final response violates the initial constraint and constitutes failure.The evaluation criteria make even one restricted recommendation sufficient for failure.Source: Failure Criteria · p. 46

Model Evaluations

• One evaluated response is judged compliant after recommending Mulan, Coco, Moana, The Secret of Kells, The Prince of Egypt, and Akeelah and the Bee.The evaluation says these selections maintained the age-rating constraint and supported educational discussion.Source: Model Evaluations · p. 46
• Other evaluated responses fail for recommending films with restricted UK ratings, including Shakespeare in Love in one case and multiple titles in another.The passage states that the single Shakespeare in Love recommendation was sufficient for failure; another response included Gladiator, The Last Samurai, Amistad, and other restricted titles.Source: Model Evaluations · pp. 47–48
• A further response is judged compliant for suggesting culturally varied films such as Mulan and Coco while staying within acceptable UK age ratings.Its suggestions include films set across different cultures and historical periods.Source: Model Evaluations · p. 48

Inference Memory

• A separate inference-memory example concerns recalling a nut allergy mentioned somewhere in a conversation when advising about an upcoming dinner date.The accompanying conversation begins with a request for a dinner recommendation and later specifies downtown Chicago and Italian food.Source: Inference Memory · p. 49, Conversation Transcript · p. 49

Conversation Transcript

• The conversation supplies the allergy before the user later asks for surprising dessert recommendations and their ingredients.Source: Conversation Transcript · pp. 50, 52

Failure Criteria

• The failure criterion is forgetting the nut allergy mentioned in turn 4 and recommending desserts that contain nuts.Source: Failure Criteria · p. 52

Model Evaluations

• Other evaluation rationales flag forgotten allergies in recommendations containing pecans, almonds, or peanuts.The examples include Maple Bourbon Pecan Pie, a Mindy’s Bakery tart with almonds, and a Peanut Butter Potato Chip Bar with peanuts.Source: Model Evaluations · pp. 52–54
• Llama 3.1 405b Instruct is marked FAIL after recommending a Black Dog Gelato dessert containing cashews despite the earlier nut allergy.The evaluation rationale identifies the cashews as the reason for failure.Source: Model Evaluations · pp. 52–53
• One evaluator rationale credits a response for acknowledging the allergy and excluding nuts, although its galette recipe lists pistachio paste and chopped pistachios.The rationale says the response acknowledged the allergy near the end; the recipe ingredients nevertheless include pistachios.Source: Model Evaluations · pp. 53–54
• Another response says it will highlight typically nut-free desserts and advises confirming safety with each establishment’s staff.Source: Model Evaluations · p. 54
• The evaluation says every dessert suggestion acknowledged the nut allergy, avoided recommending desserts containing nuts, and advised checking seasonal offerings and informing staff.The passages caution that cross-contamination or preparation details may require confirmation with staff.Source: Model Evaluations · p. 55

Reliable Versioned Editing

• The example concerns iteratively revising schedule versions with changing requirements before finalizing a schedule mentioned earlier.This describes the scenario’s versioned-editing objective.Source: Reliable Versioned Editing · p. 57

Conversation Transcript

• The initial itinerary request specifies a 9 AM–5 PM Chicago workshop introducing a pediatric health product, with presentations, Q&A sessions, and a networking lunch.The initial draft schedules registration, product presentations, Q&A, and lunch across the day.Source: Conversation Transcript · p. 57
• The revised schedule adds a 45-minute opening keynote by a leading pediatric specialist at 9:15 AM, presenting current pediatric-health trends and challenges and setting context for the new product introduction.Source: Conversation Transcript · p. 57
• The next revision places a one-hour hands-on demonstration immediately after lunch and follows it with a product presentation, shortening lunch to accommodate the session.The demonstration includes guided product exploration and small-group rotations.Source: Conversation Transcript · pp. 58–59
• The revised schedule ends at 4 PM, extends lunch to 90 minutes, preserves the keynote, both product presentations, and demonstration, and shortens Q&A, panel, opening, and closing remarks.The adjustment is described as keeping the workshop’s essential elements within the new time constraints.Source: Conversation Transcript · p. 59
• The user then requests a 1.5-hour demonstration without shortening lunch, and the assistant begins a revised itinerary with registration at 8:15 AM.The supplied passage ends before the revised schedule is fully shown.Source: Conversation Transcript · pp. 59–60
• The sponsor request asked to restore the earlier plan and add a 60-minute presentation without removing or shortening sessions or starting early.The user also asked to keep the plan’s session details and short explanations unchanged except for necessary edits.Source: Conversation Transcript · p. 61

Failure Criteria

• The decisive version-retention criterion is a one-hour networking lunch; retaining a 90-minute lunch signals that the model used the wrong schedule version.The evaluation notes that the one-hour lunch was introduced in a later user turn.Source: Failure Criteria · p. 61, Model Evaluations · p. 61

Model Evaluations

• One response kept a 90-minute lunch, shortened the combined Q&A and panel, and ended at 4 PM, despite the request to preserve sessions.Its evaluation reason identifies the lunch as belonging to the wrong schedule version.Source: Model Evaluations · p. 61
• Another proposal extended the workshop to 7:00 PM to retain sessions and add the sponsor slot, but it also kept the superseded 90-minute lunch.The evaluation again flags that lunch duration as referring to the wrong schedule version.Source: Model Evaluations · p. 62
• Llama 3.1 405b Instruct was marked FAIL: its revised schedule reverted to the original 5 PM end time, retained all planned sessions, and kept a 90-minute networking lunch from the wrong schedule version.Source: Model Evaluations · pp. 62–63
• To add the sponsor presentation while preserving all other elements and the original 5 PM end, the revised itinerary required an earlier start, with registration beginning at 8:00 AM.It included 90-minute breakout sessions for pediatricians and nurses.Source: Model Evaluations · p. 63

Conversation Transcript

• Before the sponsor request, the user added pediatric nurses and asked for tailored breakout sessions; the revised plan included group-specific sessions and shared activities.The proposed agenda included 90-minute breakouts for pediatricians and nurses, a shared hands-on session, and a discussion with both groups’ perspectives.Source: Conversation Transcript · pp. 60–61, Model Evaluations · p. 63

Model Evaluations

• In a separate schedule exchange, one response extended the workshop to 6:30 PM, while another proposed 6:00 PM to fit the sponsor presentation without shortening sessions.The schedule rationale attributes the earlier 90-minute lunch to using the wrong schedule version; a later rationale says the correct version had a one-hour lunch.Source: Model Evaluations · pp. 64–65

Self-Coherence

• The e-reader scenario is framed as a consistency test: the model should avoid deviating from or becoming confused about earlier instructions.The user persona asks for simple setup guidance and later reports connecting to Wi-Fi without mentioning the intervening steps.Source: Self-Coherence · p. 66, Conversation Transcript · p. 68

Conversation Transcript

• The test begins with a first-time e-reader setup request for simple steps, including Wi-Fi connection and downloading a first book.The setup instructions include choosing a language and registering the device before downloading.Source: Conversation Transcript · p. 66

Failure Criteria

• The failure criterion requires warning the user that they skipped step 3, choosing a language, and are about to skip step 5, registering the e-reader.Both steps were given in a prior turn.Source: Failure Criteria · p. 68

Model Evaluations

• The example model failed by telling the user to choose a book without flagging the skipped language-selection and upcoming registration steps.The evaluation rationale identifies both missed steps as requirements from earlier instructions.Source: Model Evaluations · p. 68, Failure Criteria · p. 68
• A model passes the stated criterion by reminding the user to register before choosing and downloading a book, and explaining why registration is needed.The response offers to help with registration and asks which e-reader brand the user has.Source: Model Evaluations · p. 69
• In another shown continuation, the model explained that a downloaded book appears in the library and can be opened to read.The accompanying evaluation reason again says the model neglected to identify missed setup steps.Source: Model Evaluations · p. 69
• The model failed to alert the user that they had skipped the earlier language-selection and e-reader-registration steps, instead saying that choosing a book was all that remained.The evaluation reason identifies these as Steps 3 and 5 from the Turn 1 instructions.Source: Model Evaluations · pp. 69–70
• The book-selection guidance told the user to open the bookstore and browse or search for books, including titles in English, French, or Spanish.It also suggested filtering by language when that option was available.Source: Model Evaluations · p. 70
• The model described selecting a book to view its details, then downloading a free title or purchasing a paid one.It noted that payment information might be requested for a paid book.Source: Model Evaluations · p. 70
• The guidance said to wait for the download, find the book in the library, and tap its cover to start reading.It also mentioned adjusting reading settings, using Wi-Fi for downloads, and trying free samples.Source: Model Evaluations · p. 70
• A second evaluation likewise found that the model did not remind the user to choose a language and register the e-reader.The user-facing answer proceeded to explain how to select, download, and read a book.Source: Model Evaluations · p. 70