Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following

1 Introduction

• Existing benchmarks largely use single-turn, single-language instructions, limiting evaluation of real-world multilingual, multi-turn interactions.Instruction-following evaluation is difficult because human language is complex, subjective, and ambiguous.Source: §1 · pp. 1–2
• Multi-IF expands IFEval into 4,501 three-turn conversations across eight languages using LLM generation, conflict filtering, and human-audited translation.The added languages are French, Russian, Hindi, Italian, Portuguese, Spanish, and Chinese.Source: §1 · p. 2
• Multi-IF is significantly more challenging than existing instruction-following benchmarks such as IFEval.The benchmark is designed to expose difficulties that single-turn evaluations do not capture.Source: §1 · p. 2
• Models show increasing failure rates as additional turns require them to maintain previously followed instructions.Strong reasoning capabilities help models recover from errors made in earlier turns.Source: §1 · p. 2
• Models generally perform worse in non-English languages than in English, revealing room for improvement in multilingual instruction following.This pattern is especially relevant to evaluating systems intended for users across different languages.Source: §1 · p. 2

Evaluation Methodologies

• Human evaluation supports nuanced judgments but is costly, time-consuming, inconsistent, and difficult to reproduce.Annotator interpretations and criteria can vary across evaluations.Source: §2 · p. 3
• Model-based evaluation is scalable and less resource-intensive, but judge-model bias, errors, and opacity can reduce accuracy and transparency.Its reliability depends on the correctness and objectivity of the evaluating model.Source: §2 · p. 3
• Rule/script-based evaluation objectively verifies verifiable instructions but has mainly been limited to single-turn, predominantly English benchmarks.This restricts its applicability to multilingual and multi-turn conversational settings.Source: §2 · p. 3

Multi-lingual and Multi-turn Benchmarks

• Existing multilingual benchmarks assess cross-lingual generalization, while conversational datasets often remain English-only or domain-specific.Multi-IF combines multilingual and multi-turn evaluation with rule/script-based instruction verification.Source: §2 · p. 3

Fairness and Bias in Multilingual Models

• Multilingual evaluation must account for linguistic and cultural differences because performance disparities raise fairness and bias concerns.Multi-IF includes diverse languages to encourage more consistent instruction-following performance across linguistic boundaries.Source: §2 · p. 3

3 Multi-IF

• Multi-IF extends IFEval from English single-turn prompts into a more challenging multilingual, multi-turn benchmark.The framework is summarized as a pipeline for expanding and processing the original dataset.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Multi-turn Expansion

• Each original prompt becomes a three-turn conversation by randomly sampling instruction types and using Llama 3.1 405B to generate context-aware follow-up prompts.The default process adds two turns to the original English IFEval prompt.Source: §3.1 · pp. 3–4, Figure 3 · p. 4

3.2 Conflict Instructions Removal

• Conflicting instructions are identified by Llama 3.1 405B and then removed through human verification.This filtering retains prompts whose instructions remain compatible across turns.Source: §3.2 · p. 4

3.3 Multilingual Translation

• The English multi-turn prompts are translated into seven additional languages with Llama 3.1 405B and audited by multilingual annotators.Annotators check faithfulness and native-speaker style; 15% of translations are rewritten on average.Source: §3.3 · pp. 4–5
• Some instructions are not translated into certain languages when their requirements are specific to Western languages.The paper gives all-capital-letter requirements as an example of a constraint that may not transfer reasonably to Chinese or other non-Western languages.Source: §3.3 · p. 5

3.4 Sensitive Prompts Removal

• Potentially sensitive prompts are flagged by an LLM and reviewed by human annotators before removal.The filtering targets political, religious, and human-relationship sensitivities introduced during localization.Source: §3.4 · p. 5

3.5 Examples and Statistics

• The benchmark contains 4,501 three-turn conversations, with more English prompts because some instructions are not translated and some multilingual prompts are removed.Figure 4 illustrates three-turn creative-advertisement tasks across English, Spanish, Italian, Portuguese, Hindi, Russian, French, and Chinese.Source: §3.5 · p. 5, Figure 4 · p. 5

4.1 Evaluation Protocol and Metrics

• Each successive turn uses the concatenated preceding prompts and responses with the current prompt to generate the next response.Source: §4.1 · pp. 5, 8
• The benchmark reports language and instruction-type distributions and summarizes model accuracy and rankings across languages in Tables 1 and 2.Source: Table 1 · p. 6, Table 2 · p. 7, Figure 5 · p. 8
• The benchmark evaluates 14 models across languages using instruction-level and conversation-level strict and loose accuracy, averaged into a final metric.Source: §4.2 · p. 8, §4.1 · p. 8

4.2 Overall Results

• o1-preview and Llama 3.1 405B achieved the highest third-turn average accuracy at 0.707, followed by o1-mini at 0.681.Source: §4.2 · p. 8
• Several models exceeded 0.6 average accuracy, but some fell below 0.6 by the third turn because of weak first-turn performance or larger inter-turn declines.Source: §4.2 · p. 9
• Nearly all models performed worse on Russian, while Llama 3.1 models improved with increasing size across the three turns.Source: §4.2 · p. 9

5.1 Multi-Turn Instruction Following

• o1-preview drops from 0.877 at the first turn to 0.707 at the third turn in average accuracy over all languages.All tested models show higher failure rates with each additional turn; o1-preview, o1-mini, and Llama 3.1 405B degrade less than some alternatives.Source: §5.1 · p. 10

5.2 Instructions Forgetting to Follow in Multi-Turns

• Instruction forgetting generally increases across turns, while larger Llama 3.1 models show lower forgetting rates and Gemini models show the highest rates.The turn-1-to-turn-2 forgetting rate is generally higher than the turn-2-to-turn-3 rate.Source: §5.2 · pp. 10–11, Figure 7 · p. 10

5.3 Error Self-Correction in Multi-Turns

• o1-preview and o1-mini correct around 25% of previously unfollowed instructions in later turns, achieving the highest Error Correction Ratio.The analysis associates their stronger correction performance with hidden chain-of-thought capability, while all models show some error recovery.Source: §5.3 · p. 11, Figure 8 · p. 11

5.4 Multilingual Instruction Following

• o1-preview generally performs best across eight languages, with English reaching approximately 0.85 for Llama 3.1 405B and Russian and Hindi showing notable variation.o1-preview and Llama 3.1 405B are comparable in most languages, while GPT-4o often trails them slightly.Source: §5.4 · pp. 11–12, Figure 9 · p. 12

5.5 Error Analysis of Multilingual Instruction Following

• Hindi, Russian, and Chinese generally have higher error rates, while length_constraint and combination are consistently difficult instruction categories.The startend category varies substantially, especially in Russian and Chinese; Chinese also shows elevated errors for combination and length_constraint.Source: §5.5 · p. 12, Figure 10 · p. 13

• Multi-IF evaluates instruction following under multi-turn and multilingual conditions that existing single-turn benchmarks do not capture.Source: §6 · p. 13
• Leading LLMs struggle to maintain high instruction-following accuracy across multiple turns and show lower performance in non-English languages.Source: §6 · p. 13

• Table 3 lists the versions and inference configurations used for the evaluated models.Source: Table 3 · p. 16
• Temperature 0.6 was selected for all models after larger values degraded performance, while o1 models used 25,000 maximum new tokens and other models used 1,024.The authors report only 1–2% variation for smaller temperature settings and expect further tuning would not significantly alter overall conclusions.Source: A Experimental Details · p. 16
• The evaluation handles language-specific sentence delimiters, word boundaries, Unicode ranges, punctuation, and emojis across multilingual outputs.Chinese and Japanese lack spaces between words, while Hindi uses a language-specific sentence delimiter.Source: A Experimental Details · p. 16
• These language-aware rules are intended to provide accurate and consistent instruction-following metrics across languages.Source: A Experimental Details · p. 16

• Error patterns are mostly similar across the evaluated models and languages.The appendix provides separate error-distribution figures for o1-preview, o1-mini, GPT-4o, GPT-4, Llama, Gemini, Claude, Qwen, and Mistral models.Source: B Error Distribution for Different… · p. 17, Figure 11 · p. 17, Figure 12 · p. 17
• length_constraint and combination consistently exhibit high error rates, whereas keywords has lower error rates.Source: B Error Distribution for Different… · p. 17
• Hindi, Russian, and Chinese show elevated errors in specific instruction categories across the evaluated models.Source: B Error Distribution for Different… · p. 17, Figure 17 · p. 19, Figure 23 · p. 21

C Analyzing the Failure Mode of Gemini Models

• After removing false refusals, Gemini improves slightly but still trails GPT-4, GPT-4o, o1-preview, o1-mini, Llama 3.1 405B, Llama 3.1 70B, and Claude 3.5 Sonnet.Table 4 reports averages across languages and four strict and loose instruction- and conversation-level accuracy measures.Source: C Analyzing the Failure Mode of Gemini… · p. 22, Table 4 · p. 23
• 357 of 4,501 Gemini responses were identified as false refusals using o1-preview and removed for re-evaluation.The revised evaluation isolates performance after excluding responses classified as false refusals.Source: C Analyzing the Failure Mode of Gemini… · p. 22
• Gemini 1.5 Pro’s poem has an unlabeled opening section followed by SECTION 2, SECTION 3, and SECTION 4, and ends with “Can I get my money back for the classes I missed?”Source: C Analyzing the Failure Mode of Gemini… · p. 22
• Gemini refused the uppercase instruction while producing its response entirely in capital letters, illustrating partial instruction adherence despite refusal language.Its refusal explicitly claimed that all-capital writing could be difficult to read, yet the response itself used uppercase letters throughout.Source: C Analyzing the Failure Mode of Gemini… · p. 22