PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Large Language Models at Evaluating Instruction Following

Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, Danqi Chen

arXiv:2310.07641v2cs.CLcs.LG

TL;DR

LLM-based evaluation is scalable, but its reliability for judging objective instruction following remains uncertain because outputs with superficial appeal can mislead evaluators. The paper introduces LLMBAR, a manually curated benchmark of objective-preference output pairs, and finds substantial variation and weakness among evaluators while showing that new prompting strategies can narrow their gap with humans.

  • Problem

    It remains unclear whether LLM evaluators reliably judge objective instruction following rather than superficial qualities, motivating a benchmark tied to human judgments.

  • Method

    The paper introduces LLMBAR, a manually curated benchmark containing 419 instruction-output pairs with one objectively preferred instruction-following output, and evaluates LLMs paired with prompting strategies.

  • Results

    LLM evaluators show distinct performance on LLMBAR; the best GPT-4-based evaluator reaches 82.8% average accuracy on ADVERSARIAL versus 95% human expert agreement.

  • Takeaways & Limitations

    LLMBAR provides an objective, replicable basis for assessing and selecting LLM evaluators and supports developing better prompting strategies and instruction-following models.

  • Takeaways & Limitations

    LLMBAR focuses on instruction following and single-round interactions, and its manually curated instances may need greater diversity to reflect real-world distributions.

Abstract

from arXiv · show

As research in large language models (LLMs) continues to accelerate, LLM-based evaluation has emerged as a scalable and cost-effective alternative to human evaluations for comparing the ever increasing list of models. This paper investigates the efficacy of these ``LLM evaluators'', particularly in using them to assess instruction following, a metric that gauges how closely generated text adheres to the given instruction. We introduce a challenging meta-evaluation benchmark, LLMBar, designed to test the ability of an LLM evaluator in discerning instruction-following outputs. The authors manually curated 419 pairs of outputs, one adhering to instructions while the other diverging, yet may possess deceptive qualities that mislead an LLM evaluator, e.g., a more engaging tone. Contrary to existing meta-evaluation, we discover that different evaluators (i.e., combinations of LLMs and prompts) exhibit distinct performance on LLMBar and even the highest-scoring ones have substantial room for improvement. We also present a novel suite of prompting strategies that further close the gap between LLM and human evaluators. With LLMBar, we hope to offer more insight into LLM evaluators and foster future research in developing better instruction-following models.

1 INTRODUCTION

LLM evaluators offer a scalable alternative to human judgments, but it remains unclear whether they reliably assess objective instruction following rather than superficial appeal. LLMBAR addresses this gap with manually curated, objective-preference pairs and reveals substantial evaluator weaknesses and distinct performance across evaluators.

  • LLM evaluators provide a cost-effective alternative to human evaluation, which is neither scalable nor reproducible for open-ended conversational abilities.
  • Reliable meta-evaluation is needed because existing benchmarks may encode subjective preferences, limiting confidence that selected evaluators judge instruction following and factual correctness.Reported human agreement rates in AlpacaFarm and MT-Bench were 66% and 63%, respectively, versus a 50% random baseline.
  • Instruction following requires correctly parsing open-ended instructions and adhering to specified requirements, yet evaluators may favor outputs with engaging tones over outputs that follow instructions.
  • LLMBAR contains 419 instruction-output pairs, with one output following the instruction and the other deviating, and tests whether evaluators select the objectively correct output.
  • LLMBAR is author-examined, focuses on objective instruction-following preferences, achieves 94% expert annotator agreement, and includes NATURAL and ADVERSARIAL sets.The NATURAL set targets real-world preference distributions, while the ADVERSARIAL set contains crafted instances intended to confound less adept evaluators.
  • Evaluator performance varies substantially: some ChatGPT-, LLaMA-2-Chat-, and Falcon-based evaluators perform below chance on ADVERSARIAL, while novel prompting strategies improve detection and narrow the gap with human evaluators.

2 LLMBAR: A META-EVALUATION BENCHMARK

LLMBAR is a manually curated meta-evaluation benchmark for testing whether LLM evaluators identify outputs that objectively follow instructions. It combines a filtered NATURAL set with an ADVERSARIAL set designed to distract evaluators using superficially appealing but instruction-divergent outputs.

  • Benchmark design: Each LLMBAR instance contains an instruction, two outputs, and a gold preference label identifying the objectively better output.The benchmark represents each instance as (I, O1, O2, p), where p indicates which output is objectively preferable.
  • NATURAL set: LLMBAR’s NATURAL set adapts existing human-preference data by filtering and modifying instances to ensure an objective preference exists.It is intended to reflect evaluator performance under a real-world distribution.
  • ADVERSARIAL set: The ADVERSARIAL set pairs an instruction-following output with a deviating output that often has superior superficial qualities, such as a polished tone or better format.These instances test whether evaluators prioritize instruction adherence rather than surface appeal.
  • ADVERSARIAL set: ADVERSARIAL candidates are collected through neighbor-instruction, GPT-4-instruction, GPT-4-unhelpful-output, and manual-construction strategies.The benchmark samples instructions from Alpaca, OpenAssistant, and ShareGPT, while later filtering and manual inspection improve candidate quality.
  • ADVERSARIAL set: The NEIGHBOR strategy retrieves a semantically related but sufficiently different instruction, then uses outputs generated for the different instruction to create potentially misleading comparisons.Cosine similarity from INSTRUCTOR and a threshold ϵ regulate how different the retrieved instruction must be; unsuitable cases are filtered later.

3 PROMPTING STRATEGIES FOR LLM EVALUATORS

The paper evaluates existing and newly proposed prompting strategies for LLM evaluators. Rules, Metrics, and Swap target instruction-focused comparison, evaluator criteria, and positional bias, respectively.

  • Overview: Prompting strategy substantially affects evaluator accuracy in addition to the capacity of the underlying base LLM.The paper examines existing strategies alongside the proposed Rules, Metrics, and Swap methods.
  • Existing strategies: Vanilla asks the LLM to select the better output without explanation, whereas Chain-of-Thought first elicits concise reasoning before the preference.Both are evaluated as baselines for comparing outputs against the instruction.
  • Existing strategies: Self-Generated Reference has the evaluator create an output for the instruction and use it as a reference during comparison.ChatEval instead uses multiple role-prompted evaluators that discuss the preference before giving final judgments.
  • Proposed strategies: Rules explicitly instruct evaluators to prioritize whether an output honestly executes the instruction, and the paper finds that Rules improves accuracy almost universally.Rules can be added to other strategies, with combinations marked by an asterisk.
  • Proposed strategies: Metrics first generates instruction-specific criteria for a good output, encouraging evaluators to focus on concrete aspects of instruction following.Metrics can be combined with Self-Generated Reference.
  • Proposed strategies: Swap evaluates both output orders with Chain-of-Thought and synthesizes the two reasoning traces when their preferences conflict.The strategy addresses positional bias by producing a final decision from both orderings.

4 EXPERIMENTS

The experiments evaluate LLM evaluators and reward models on LLMBAR, comparing base models, prompting strategies, human agreement, and other meta-evaluation benchmarks. LLMBar exposes substantial evaluator weaknesses, while enhanced prompting improves performance and reveals challenges beyond the main benchmark.

  • Experimental setup: The experiments compare proprietary and open-source base LLMs, including GPT-4, ChatGPT, PaLM2, LLaMA-2-70B-Chat, and Falcon-180B-Chat.Evaluators combine these models with different prompting strategies; proprietary models use temperature 0 and open-source models use greedy decoding.
  • Human agreement: 94% expert annotator agreement on sampled LLMBAR instances exceeds FairEval’s 71.7% and MT-Bench’s 63%.Human agreement is 90% on NATURAL and 95% on ADVERSARIAL instances.
  • Experimental setup: LLMBAR evaluation reports average accuracy and positional agreement after querying each evaluator with both output orders.Positional agreement measures consistency before and after swapping the two outputs; Falcon-180B-Chat with CoT achieved only 12% agreement and was omitted from the main results.
  • LLM evaluator performance: 82.8% was the best GPT-4-based evaluator accuracy on ADVERSARIAL, below the 95% human expert agreement rate.ChatGPT-, LLaMA2-, and Falcon-based evaluators barely exceeded chance on ADVERSARIAL, while the performance gap was smaller on NATURAL.
  • Prompting strategies: Rules+Metrics+Reference consistently improves evaluator performance across base LLMs and both LLMBAR sets.Combining the strategies yields around 10% improvement for GPT-4-based evaluators on ADVERSARIAL, whereas CoT* can hurt performance through bias toward superficial quality.
  • Comparison to other evaluations: LLMBAR shows a clearer separation between weaker and stronger evaluators than FairEval, LLMEval2, and MT-Bench.The authors report that existing datasets do not reliably reflect evaluator capabilities because their preferences lack guaranteed objective correctness.
  • Reward-model evaluation: The evaluated AlpacaFarm reward models and SteamSHP preference model are also tested on LLMBAR as additional preference-based evaluators.The passage introduces these models as critical components or models trained to choose between outputs, while Table 3 reports their results.
  • More challenging evaluation: Special subsets involving lexical constraints, negation, and base-9 or base-10 arithmetic remain challenging even for powerful LLMs with advanced prompts.Enhanced prompting strategies nevertheless improve performance on these adversarial and synthetic scenarios.

5 RELATED WORK

Prior work uses LLMs as scalable evaluators but documents biases and limitations, motivating meta-evaluation benchmarks and improved prompting strategies. Existing approaches include multiple explanations, reference-guided evaluation, and ensembles of evaluators.

  • LLM-based evaluation: LLM-based evaluation offers a scalable approach for traditional natural language generation tasks as instruction-tuned models develop rapidly.The paradigm is presented as a response to the cost and scalability limitations of human evaluation for open-ended conversational abilities.
  • Evaluator limitations: LLM evaluators exhibit biases involving presentation order, verbose outputs, and outputs from similar models.These limitations motivated benchmarks such as FairEval, MT-Bench, and LLMEval2.
  • Improving evaluators: Prior prompting methods improve evaluation through aggregated explanations, reference-guided judgments, or multiple LLM evaluators.Reference-guided evaluation first generates an output from the instruction and then uses it as a comparison reference.

6 CONCLUSION

The paper introduces LLMBAR, a challenging meta-evaluation set for testing whether LLM evaluators can faithfully judge instruction-following outputs. It also notes broader benchmark and interaction settings that remain open for future work.

  • LLMBAR tests whether LLM evaluators can discern instruction-following outputs.
  • The paper leaves factual correctness, toxicity, instance diversity, and multi-round conversations for future work.
  • The paper illustrates how subjective preference labels can be excluded or corrected through manual modification to create objective evaluation instances.
  • The paper reports example cases where superficially plausible or detailed outputs diverge from the instruction, motivating careful evaluator assessment.
  • The benchmark construction includes generated instructions and outputs designed to create challenging but instruction-relevant comparisons.

A.4 THE GPTOUT SUBSET

The GPTOUT subset contains examples where outputs may appear polished or detailed while failing to satisfy the instruction. These cases test whether evaluators prioritize instruction adherence over superficial quality.

  • GPT-4 can generate outputs that are easily detected as unhelpful, a common failure case that may be discarded.
  • A generated output may also be correct, creating another failure case for constructing a dispreferred response.
  • The subset includes cases where one output does not fully answer the instruction despite appearing more detailed.
  • In a nickname example, Output 2 offers the unhelpful literal nickname “Nickname,” while Output 1 requests information needed to respond more effectively.

A.5 THE MANUAL SUBSET

The MANUAL subset uses deliberately chosen examples in which superficial polish or extra detail can distract evaluators from instruction adherence. The preferred output is the one that directly satisfies the requested task.

  • Detailed background information can distract evaluators even when it does not answer the question, whereas a concise direct answer follows the instruction.
  • A polished tone and better format can make an impossible science-project proposal appear preferable despite failing the task’s substantive requirements.

B DETAILS OF PROMPTING STRATEGIES

The paper evaluates several prompting strategies for LLM evaluators, ranging from direct selection and chain-of-thought reasoning to rules, generated metrics, references, and swapped-order synthesis.

  • B.1 VANILLA: B.1 VANILLA uses a direct prompt requiring the evaluator to select Output (a) or Output (b) without explanation.
  • B.2 CHAIN-OF-THOUGHTS: B.2 CHAIN-OF-THOUGHTS asks the evaluator to explain its evaluation before ending with a required preference statement.
  • B.3 RULES: B.3 RULES instructs evaluators to prioritize precise instruction execution, avoid excess or missing content, and reduce bias.
  • B.4 SELF-GENERATION METRICS (ACCOMPANIED BY RULES): B.4 SELF-GENERATION METRICS has the evaluator generate up to three instruction-specific questions, then use them to assess outputs.
  • B.5 SELF-GENERATED REFERENCE: B.5 SELF-GENERATED REFERENCE supplies evaluators with a concise reference answer generated by a strong assistant.
  • B.6 SWAP AND SYNTHESIZE: B.6 SWAP AND SYNTHESIZE obtains evaluations under swapped output orders and synthesizes conflicting reasoning, optionally combining this process with chain-of-thought.

C MORE RESULTS

The paper reports additional LLMBAR results for several base LLMs, extending the evaluator comparison beyond the representative models.

  • Additional experiments evaluate ChatGPT-0613, ChatGPT-0301, LLaMA-2-70B-Chat, PaLM2, and Falcon-180B-Chat on LLMBAR.

D RESULTS OF COMPARISON VIA RATING

The paper compares rating-based evaluation with the default comparative approach and examines PROMETHEUS as a specialized rating model. Rating produces frequent ties for ChatGPT, while references substantially improve PROMETHEUS’s adversarial performance.

  • Rating-based evaluation scores each output independently before comparing the two scores, unlike simultaneous comparative judgment.
  • ChatGPT frequently assigns identical scores to both outputs, with hedging rates approaching or exceeding 50% across five LLMBAR subsets.
  • GPT-4’s rating results show no significant difference from the corresponding comparative-evaluation results when using Rules, Metrics, or Reference prompting.
  • PROMETHEUS is a 13B rating model that scores outputs from 1 to 5 using rubrics focused on instruction following.
  • PROMETHEUS with a GPT-4-generated reference achieves clearly above-chance ADVERSARIAL performance, and the reference is important to its performance.

E FEW-SHOT IN-CONTEXT LEARNING

Few-shot examples do not significantly change evaluator results, while specialized subsets reveal distinct weaknesses in lexical constraints and negation. Enhanced prompting helps only partially in these settings.

  • Few-shot in-context learning is evaluated by adding one or two examples to the Vanilla+Rules prompting strategy.
  • There is no significant difference among zero-shot, one-shot, and two-shot results.
  • Lexical constraints: CONSTRAINT contains instructions with specific lexical requirements, pairing an output that satisfies each constraint with one that does not.
  • Lexical constraints: Even enhanced strategies allow GPT-4 only to marginally exceed chance on CONSTRAINT, showing that lexical constraints challenge evaluation as well as generation.
  • Negation: Almost all evaluators perform nearly perfectly on NORMAL, but weaker models perform notably worse on NEGATION with the Rules strategy.
  • Negation: Improved prompting enhances NEGATION performance to some degree, but weaker evaluators frequently fail to account for the instruction’s negation prefix.

F.3 COUNTERFACTUAL TASK

The counterfactual-task evaluation uses paired base-9 and base-10 addition problems to test whether evaluators verify outputs under altered assumptions. GPT-4 improves substantially with combined prompting, but base-9 remains harder without it.

  • The benchmark compares counterfactual BASE-9 addition with default BASE-10 addition using the same two-digit problems.
  • Correct BASE-9 outputs are paired against superficially misleading BASE-10 reasoning, while the default counterpart reverses the task setting.
  • The evaluation assumes checking one-digit addition steps is easier than independently solving the complete two-digit problem.
  • GPT-4 exceeds 60% on BASE-10 with Rules but reaches only 20% on BASE-9 under that strategy.
  • GPT-4 requires Rules+Metrics+Reference to exceed 90% accuracy on both BASE-9 and BASE-10.
Loading 2310.07641v2…