Source-linked AI summary
Instruction-Following Evaluation for Large Language Models
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, Le Hou
TL;DR
Instruction-following evaluation lacks a standardized, reproducible approach because human judgments are costly and inconsistent, while model-based evaluation depends on evaluator-model correctness. IFEval addresses this gap with a benchmark of objectively checkable instructions and reports results for multiple widely used models. Its scope remains limited by edge cases in automatic verification and implementation areas requiring improvement.
Problem
Instruction-following evaluation is difficult to standardize because human evaluation is costly and inconsistent, while model-based evaluation depends on evaluator-model correctness.
Method
IFEval uses prompts containing verifiable instructions and computes instruction-following accuracy with strict and loose metrics.
Results
IFEval reports evaluation results for multiple models, including GPT-4 and PaLM 2 Small, and provides overall and instruction-level accuracy scores.
Takeaways & Limitations
IFEval provides an easy-to-reproduce, automatic approach for evaluating instruction following.
Takeaways & Limitations
The current implementation can be improved across many fronts, and verifiable instructions still have edge cases that complicate automatic checking.
Abstract
from arXiv · showhide
One core capability of Large Language Models (LLMs) is to follow natural language instructions. However, the evaluation of such abilities is not standardized: Human evaluations are expensive, slow, and not objectively reproducible, while LLM-based auto-evaluation is potentially biased or limited by the ability of the evaluator LLM. To overcome these issues, we introduce Instruction-Following Eval (IFEval) for large language models. IFEval is a straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set of "verifiable instructions" such as "write in more than 400 words" and "mention the keyword of AI at least 3 times". We identified 25 types of those verifiable instructions and constructed around 500 prompts, with each prompt containing one or more verifiable instructions. We show evaluation results of two widely available LLMs on the market. Our code and data can be found at https://github.com/google-research/google-research/tree/master/instruction_following_eval
1 INTRODUCTION
IFEval addresses the difficulty of evaluating instruction following by using verifiable instructions that can be checked objectively and automatically. The benchmark contains 25 instruction types and 541 prompts, while recognizing that verification still has edge cases.
- Evaluation challenge: Instruction-following evaluation is difficult because subjective language produces unclear and inconsistent judgments.Examples include requests for a funny tone or detailed reasoning without overexplaining.
- Existing methods: Human evaluation is time-consuming, expensive, and vulnerable to annotator bias and reproducibility inconsistencies.
- IFEval: IFEval evaluates instruction following through verifiable instructions that permit objective, automatic compliance checks.Examples include word-count ranges, JSON-only output, and bracketed titles.
- Limitations: Verification is not perfectly objective because edge cases can cause naive checks to produce false negatives, so IFEval considers commonly seen variations.
- Benchmark construction: 25 verifiable instruction types and 541 prompts were created, with variants in both parameters and phrasing.The prompts contain one or multiple verifiable instructions.
- Benchmark construction: IFEval is a benchmark using simple, interpretable, deterministic programs to verify responses, with evaluation results and code and prompts released.
2 VERIFIABLE INSTRUCTIONS
IFEval evaluates instruction following through verifiable instructions and deterministic checks, while using prompt synthesis and strict or loose metrics to handle conflicts, phrasing diversity, and false negatives.
- 2.1 PROMPT SYNTHESIS: Prompt construction begins with base prompts and one to three randomly selected verifiable instructions, then filters illogical prompts, diversifies phrasing, and manually edits each result.This procedure addresses instruction conflicts and avoids confounding instruction following with familiarity with one phrasing.
- 2.2 IFEVAL METRICS: The benchmark computes whether each instruction is followed as a Boolean outcome, and uses Equation 1 to calculate instruction-following accuracy under the strict metric.The strict metric counts an instruction as followed when the verification function returns True.
- 2.2 IFEVAL METRICS: Loose accuracy transforms responses before verification to reduce false negatives caused by formatting differences such as Markdown emphasis around a required phrase.The transformations include removing Markdown font modifiers, the first line, or the last line.
- 2.2 IFEVAL METRICS: Evaluation reporting includes overall accuracy and instruction-level strict-accuracy separated by instruction category, with Table 3 noting that the two models are not directly comparable because their parameter counts differ.The category breakdown is shown in Figure 2, while Table 3 reports overall scores.
- 2.2 IFEVAL METRICS: The loose criterion combines every two and all three listed transformations with the identity transformation, yielding eight transformations in total.These transformations provide alternative response forms for instruction verification.
- 2.2 IFEVAL METRICS: Loose verification reduces false negatives but can introduce false positives, so the authors treat it as a complement to the original criterion.For example, removing a first line could incorrectly make a response satisfy a word-count instruction.
3 EVALUATION RESULTS
The evaluation tests GPT-4 and PaLM 2 Small using four IFEval accuracy measures that distinguish prompt-level from instruction-level performance and strict from loose verification.
- 3 EVALUATION RESULTS: IFEval reports four accuracy scores: prompt-level strict, instruction-level strict, prompt-level loose, and instruction-level loose accuracy.Prompt-level scores require all instructions in a prompt to be followed, whereas instruction-level scores measure individual instructions.
- 3 EVALUATION RESULTS: Overall model accuracy is reported in Table 3, alongside instruction-level strict-accuracy separated by instruction category in Figure 2.These views provide overall scores and category-specific instruction-level results.
4 DISCUSSION AND FUTURE WORK
IFEval evaluates instruction following with verifiable instructions through an easy-to-reproduce, automatic approach, while the current implementation remains open to improvement. Planned work targets greater instruction diversity, real-world relevance, and multimodal use cases.
- IFEval evaluates LLM instruction following with verifiable instructions using an easy-to-reproduce and automatic approach.
- The current implementation can be improved across multiple fronts.
- Future work includes increasing the diversity and quantity of verifiable instructions.
- The benchmark is planned for extension to multimodal use cases, such as generating at least 3 images.
- The authors also plan to make prompts and verifiable instructions more related to real-world applications.
5 DETAILED RESULTS
The detailed-results section reports instruction-following accuracy for each detailed category in Figure 3.
- Figure 3 reports instruction-following accuracy per detailed category.
6 LIST OF PROMPTS
The prompt list contains varied tasks that combine content goals with explicit, checkable constraints. These constraints cover length, formatting, language, wording, structure, punctuation, repetition, and other response properties.
- Prompts request creative and practical outputs including poems, articles, itineraries, stories, songs, riddles, cover letters, advice, and technical explanations.
- Many prompts impose quantitative constraints on words, sentences, lines, repeated keywords, placeholders, sections, or recommendations.
- Other prompts constrain language and character usage through English-only, Tamil-only, all-capital, lowercase, no-comma, or banned-word requirements.
- Several prompts combine multiple verifiable instructions in a single task, such as content, length, capitalization, keyword counts, formatting, and required endings.
- Formatting requirements include JSON, email format, markdown emphasis, titles in double angular brackets, exact separators, bullet points, and postscript markers.
1. What are the best places to visit in Bohemia, Czech Republic?
The prompt requires a list of recommended hotels and specifies that the entire output must be in JSON format.
- The response must include a list of recommended hotels.
- The entire output must use JSON format.
4. Do not include the following keywords: Moser, Glassworks, Pravcice, Karlovy, Vary
The examples illustrate prompts that combine multiple independently checkable requirements, including format, length, language, keyword, structural, and stylistic constraints.
- Some instructions specify exact response structures, including paragraph counts, bullet counts, separators, JSON, XML, or repeated-request ordering.These constraints appear across itinerary, social-media, biography, summarization, and repetition tasks.
- A prompt may explicitly require reproducing the request before answering, establishing an ordering constraint in addition to the answer itself.This demonstrates that instruction-following can include dependencies between response stages.
- Prompts can require multiple simultaneous constraints, such as audience, length, tone, keywords, formatting, and content.Examples include a 500-word humorous diaper pitch with a title format and a two-week itinerary requiring all caps and placeholders.
- Other instructions constrain language form through capitalization, language exclusivity, forbidden words, punctuation, or letter-level requirements.Examples require lowercase English, Punjabi or Gujarati only, no commas, no capital letters, or repeated occurrences of particular letters.
- Several prompts combine semantic rewriting or generation with formal constraints such as placeholders, markdown highlighting, postscript placement, or word limits.The constraints are layered onto tasks including biographies, product descriptions, jokes, songs, and opinion letters.