PaperlayerView exact revision on arXiv

Source-linked AI summary

Prometheus: Inducing Fine-grained Evaluation Capability in Language Models

Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, Minjoon Seo

arXiv:2310.08491v2cs.CLcs.LG

TL;DR

Proprietary LLM evaluators are costly, closed, and difficult to reproduce, motivating an open-source evaluator for customized criteria. Prometheus combines rubric- and reference-based data with feedback fine-tuning, achieving evaluation results comparable to GPT-4 and strong performance across human-preference benchmarks.

  • Problem

    Proprietary LLM evaluators have closed-source internals, uncontrolled versioning, and prohibitive costs, while existing evaluation resources are often generic or coarse-grained.

  • Method

    The paper constructs the FEEDBACK COLLECTION with customized score rubrics, reference answers, instructions, responses, feedback, and scores, then fine-tunes Llama-2-Chat to create Prometheus.

  • Results

    Across customized-rubric and preference evaluations, Prometheus performs comparably to GPT-4 in correlation with human judgments and outperforms GPT-3.5-Turbo and other evaluated reward models.

  • Takeaways & Limitations

    Prometheus demonstrates the potential of an open-source evaluator for fine-grained scoring with customized criteria and as a universal reward model.

  • Takeaways & Limitations

    The human-evaluation setup excluded coding and math questions, where beating GPT-4 is non-trivial, and ranking settings were not exactly fair compared with other ranking models.

Abstract

from arXiv · show

Recently, using a powerful proprietary Large Language Model (LLM) (e.g., GPT-4) as an evaluator for long-form responses has become the de facto standard. However, for practitioners with large-scale evaluation tasks and custom criteria in consideration (e.g., child-readability), using proprietary LLMs as an evaluator is unreliable due to the closed-source nature, uncontrolled versioning, and prohibitive costs. In this work, we propose Prometheus, a fully open-source LLM that is on par with GPT-4's evaluation capabilities when the appropriate reference materials (reference answer, score rubric) are accompanied. We first construct the Feedback Collection, a new dataset that consists of 1K fine-grained score rubrics, 20K instructions, and 100K responses and language feedback generated by GPT-4. Using the Feedback Collection, we train Prometheus, a 13B evaluator LLM that can assess any given long-form text based on customized score rubric provided by the user. Experimental results show that Prometheus scores a Pearson correlation of 0.897 with human evaluators when evaluating with 45 customized score rubrics, which is on par with GPT-4 (0.882), and greatly outperforms ChatGPT (0.392). Furthermore, measuring correlation with GPT-4 with 1222 customized score rubrics across four benchmarks (MT Bench, Vicuna Bench, Feedback Bench, Flask Eval) shows similar trends, bolstering Prometheus's capability as an evaluator LLM. Lastly, Prometheus achieves the highest accuracy on two human preference benchmarks (HHH Alignment & MT Bench Human Judgment) compared to open-sourced reward models explicitly trained on human preference datasets, highlighting its potential as an universal reward model. We open-source our code, dataset, and model at https://kaistai.github.io/prometheus/.

1 INTRODUCTION

The paper addresses limitations of proprietary LLM evaluators and coarse automated metrics by introducing Prometheus, an open-source model for fine-grained evaluation using customized rubrics and reference materials.

  • Human evaluation captures nuanced dimensions such as brevity, creativity, tone, and cultural sensitivities that conventional metrics like BLEU and ROUGE cannot capture.
  • Proprietary evaluators raise concerns about transparency, reproducibility, and cost because their internals are closed, versions change without user control, and API use can be expensive.Evaluating four model variants across four sizes on 1,000 instances with GPT-4 can cost over $2000.
  • Existing resources often provide generic, single-dimensional criteria that are domain-specific or coarse-grained rather than customized scoring rubrics.
  • Prometheus is a 13B open-source language model designed to generalize fine-grained evaluation across diverse user-defined scoring rubrics.
  • 0.897 Pearson correlation with human evaluators makes Prometheus comparable to GPT-4 at 0.882 and substantially above GPT-3.5-Turbo at 0.392.Across 1,222 customized rubrics and four test sets, Prometheus also correlates more highly with GPT-3.5-Turbo and Llama-2-Chat 70B, while outperforming two reward models on two unseen preference datasets.
  • The paper introduces the FEEDBACK COLLECTION dataset and reports that reference materials plus feedback fine-tuning induce evaluation capability in language models.

2 RELATED WORK

Prior work includes reference-based metrics and LLM-based evaluators, but these approaches generally remain unreliable, single-dimensional, or insufficiently suited to fine-grained evaluation.

  • Reference-based metrics such as BLEU and ROUGE were dominant, but prior work reported that metrics such as ROUGE are not reliable for evaluation.
  • Figure 2 motivates an evaluator trained with score rubrics, reference answers, and GPT-4 feedback as an open-source alternative.
  • Recent LLM-based evaluators commonly assess responses along a single dimension of preference, such as which response is generally better.AlpacaFarm, for example, asks models to choose the response better according to their judgment and preference.

3 THE FEEDBACK COLLECTION DATASET

The FEEDBACK COLLECTION is a dataset for training fine-grained evaluator language models, combining customized rubrics and reference answers with instructions, responses, feedback, and scores generated through GPT-4-based augmentation.

  • The dataset contains 1K score rubrics, 20K instructions, and 100K responses and feedback, with 20 instructions per rubric and scores from 1 to 5.
  • Each training instance contains four inputs—an instruction, response, customized score rubric, and reference answer—and two outputs: feedback and a score.The rubric specifies evaluation criteria and scoring decisions from 1 to 5; the reference answer represents a score of 5.
  • Reference answers let the evaluator use shared information between the reference and evaluated response instead of solving the instruction itself.
  • Dataset construction process: Equal numbers of examples for scores 1 through 5 are generated to reduce decision bias during evaluator fine-tuning.
  • Dataset construction process: The collection process expands 50 seed rubrics to 1,000 rubrics, generates relevant instructions, and augments responses, feedback, and scores with GPT-4.
  • Fine-tuning an evaluator LM: Prometheus is obtained by fine-tuning Llama-2-Chat models on the collection to generate feedback followed by a score.A “[RESULT]” separator is used between feedback and score generation to prevent degeneration during inference.

4 EXPERIMENTAL SETTING: EVALUATING AN EVALUATOR LM

The experiments evaluate evaluator LMs through absolute grading, feedback quality, GPT-4 score correlation, and ranking-based human preference accuracy across several benchmarks.

  • List of Experiments and Metrics: Absolute grading requires an evaluator LM to generate feedback and a 1-to-5 score from an instruction, response, and reference materials.The setting is more challenging than ranking grading because no opponent response is available for comparison.
  • List of Experiments and Metrics: The study measures correlation with human and GPT-4 evaluators, compares feedback quality pairwise, and evaluates ranking accuracy on human preference datasets.Correlation uses Pearson, Kendall-Tau, and Spearman metrics, while feedback quality uses pairwise human comparison.
  • Benchmarks: Four benchmarks support absolute-grading evaluation: Feedback Bench, Vicuna Bench, MT Bench, and Flask Eval.Feedback Bench contains seen and unseen score-rubric subsets, while Vicuna Bench and MT Bench use hand-crafted customized rubrics and GPT-4 reference answers.
  • Ranking Grading: Ranking grading tests whether an evaluator trained only on absolute grading can score the human-preferred response higher as a potential universal reward model.Candidates are evaluated independently with temperature 1.0 and repeated until a winner, without reference answers; the authors caution that this setting is not intended to claim state-of-the-art performance.
  • Ranking Benchmarks: The ranking benchmarks are MT Bench Human Judgement and HHH Alignment, which measure human preference accuracy.HHH Alignment evaluates preferences in Helpfulness, Harmlessness, Honesty, and General categories.
  • Baselines: Baselines include Llama-2-Chat models at 7B, 13B, and 70B, plus a 13B model trained with coarse-grained rubrics.These baselines test model scaling and the effect of training on thousands of fine-grained rubrics.

5 EXPERIMENTAL RESULTS

Prometheus closely matches human scoring and produces feedback preferred to GPT-4 in pairwise evaluation, while its GPT-4 correlation varies across benchmarks and training directly on a task can be advantageous.

  • Human Scoring: 0.897 Pearson correlation with human evaluators is achieved by Prometheus, compared with 0.882 for GPT-4 and 0.392 for GPT-3.5-Turbo across 45 customized rubrics.The evaluation spans Feedback Bench, MT Bench, and Vicuna Bench.
  • Feedback Quality: 58.62% preference over GPT-4 and 79.57% over GPT-3.5-Turbo are achieved by Prometheus in pairwise human evaluation of feedback quality.Annotators selected which feedback better assessed the given response.
  • Feedback Analysis: Prometheus is mainly rejected for being overly critical, whereas GPT-4 is mainly rejected for producing general or abstract feedback.The authors conjecture that fine-tuning exclusively for fine-grained evaluation contributed to Prometheus’s more opinionated feedback.
  • GPT-4 Score Correlation: +0.420 and +0.397 Pearson-correlation improvements over Llama2-Chat 13B are reported by Prometheus on the seen and unseen Feedback Bench rubric sets.Prometheus also outperforms Llama2-Chat 70B, GPT-3.5-Turbo-0613, and different GPT-4 versions in these comparisons.
  • GPT-4 Score Correlation: Training on feedback from coarse-grained rubrics only slightly improves performance, supporting the importance of broad fine-grained rubric coverage for customized evaluation.The comparison concerns Llama2-Chat 13B trained with coarse-grained score rubrics.
  • GPT-4 Score Correlation: +0.255, +0.493, and +0.202 Pearson-correlation improvements over Llama2-Chat 13B are reported on Vicuna Bench, MT Bench, and Flask Eval, respectively.Prometheus outperforms Llama2-Chat 70B and GPT-3.5-Turbo-0613 but trails GPT-4 on these benchmarks.
  • GPT-4 Score Correlation: Llama2-Chat 13B trained directly on Flask Eval outperforms Prometheus on that dataset, indicating a potential advantage for task-specific evaluator training.The authors discuss this result as evidence that direct training on an evaluation dataset may be the best option for a task-specific evaluator.
  • Reward Model Evaluation: +5.43% and +5.38% margins over Llama2-Chat 13B are achieved by Prometheus on HHH Alignment and MT Bench Human Judgement, respectively.These results suggest that absolute-grading training can improve ranking-grading performance without direct training on ranking instances.

6 DISCUSSIONS AND ANALYSIS

The analysis explains how reference materials induce fine-grained evaluation, examines ablations and practical use, and identifies trade-offs in adapting evaluator models to target domains.

  • 6.1 WHY IS IT IMPORTANT TO INCLUDE REFERENCE MATERIALS?: Reference materials let the evaluator focus on assessing response quality rather than identifying relevant aspects or solving the instruction.The score rubric supplies pivotal evaluation aspects, while the reference answer removes the need to estimate an answer internally.
  • Training Ablation: Excluding the reference answer causes the largest performance degradation, supporting its role in avoiding internal problem-solving during evaluation.
  • Model Ablation: Different starting models do not significantly harm performance, but Llama-2-Chat performs best, while Code-Llama benefits code-domain evaluation.The authors associate Llama-2-Chat’s advantage with supervised fine-tuning and RLHF, and report more detailed Code-Llama feedback for code responses.
  • 6.4 A PRACTITIONER’S GUIDE FOR DIRECTLY USING PROMETHEUS EVALUATION: PROMETHEUS supports customized criteria including “Child-Safety,” “Creativity,” and formal writing, using reference rubrics and answers to generate scores and feedback.Practitioners may evaluate responses from either widely used or custom instruction datasets.
  • 6.5 A PRACTITIONER’S GUIDE FOR TRAINING A NEW EVALUATION MODEL: Training directly on a target feedback dataset gives the best performance on that target, but requires the cost of preparing a new feedback dataset.This creates a trade-off between target-task performance and dataset-preparation cost.

7 CONCLUSION

The paper presents PROMETHEUS as an open-source evaluator specialized for fine-grained assessment through reference materials. It reports human-level correlation with GPT-4, feedback preferred over GPT-4 58.62% of the time, and strong human-preference benchmark performance.

  • PROMETHEUS is an open-source evaluator LM trained with the FEEDBACK COLLECTION to assess responses using customized score rubrics.
  • The FEEDBACK COLLECTION contains thousands of customized score rubrics, supporting fine-grained evaluation beyond generic preference assessment.
  • PROMETHEUS achieves Pearson correlation on par with GPT-4 when compared with human evaluators, while its feedback is preferred over GPT-4 58.62% of the time.
  • PROMETHEUS shows the highest correlation with GPT-4 and superior performance on human-preference datasets, indicating potential as a universal reward model.

A ANALYSIS OF THE FEEDBACK COLLECTION DATASET

The dataset analysis examines rubric diversity, score-description quality, response-length balance, linguistic diversity, and separation between training and evaluation rubrics.

  • Are the Score Criteria Diverse Enough?: The FEEDBACK COLLECTION contains novel score rubrics whose criteria show little overlap with one another.
  • Are the Score Descriptions Well Formulated?: Score descriptions show a smoothly increasing sentiment tendency across scores, supporting their role in deciding absolute scores.
  • Is there a length bias among the Responses?: Response lengths are designed to remain similarly distributed across score ranges to reduce length bias during fine-tuning.
  • Are the Instructions, Responses, and Feedback Diverse as Well?: Distinct N-gram analysis finds moderate diversity across instructions, responses, and feedback, despite some term repetition.
  • Does the testset maintain Unseen Score Rubrics?: FEEDBACK BENCH score rubrics have low Rouge-L overlap with FEEDBACK COLLECTION rubrics, supporting an unseen-rubric evaluation setting.

B ANALYSIS OF THE FEEDBACK BENCH EVALUATION DATASET

The FEEDBACK BENCH evaluation dataset is designed to test whether PROMETHEUS generalizes to customized score rubrics unseen during training.

  • Does the testset maintain Unseen Score Rubrics?: FEEDBACK BENCH evaluates generalization using customized score rubrics constructed as an unseen subset relative to FEEDBACK COLLECTION.
  • Does the testset maintain Unseen Score Rubrics?: Low Rouge-L overlap between FEEDBACK COLLECTION and FEEDBACK BENCH rubrics supports treating FEEDBACK BENCH as an unseen test set.

C FINE-TUNING AND INFERENCE DETAILS OF PROMETHEUS

Prometheus is fine-tuned with Llama-2-Chat using feedback data, while inference uses parsing strategies to convert varied score outputs into decisions. A coarse-rubric Flask baseline provides a comparison for fine-grained rubric training.

  • Training setup: Prometheus is created by fine-tuning Llama-2-Chat-13B on the FEEDBACK COLLECTION.
  • Inference: Naive Llama-2-Chat often fails to generate easily parsed final scores without feedback-data training.Verbalizers were used to map formats such as “[SCORE 5]” and “Score: 4 out of 5” to score decisions.
  • Inference: Fine-tuning directly for structured outputs reduces the need to rely on verbalizers for score parsing.
  • Baseline: The coarse-rubric baseline uses Flask because FEEDBACK COLLECTION instructions are tied to their original fine-grained rubrics.
  • Baseline: Flask training data was balanced across scores 1 to 5, producing 30K training instances for the LLAMA-2-CHAT + COARSE baseline.

E PAIRWISE COMPARISON OF THE QUALITY OF THE FEEDBACK

The pairwise analysis compares feedback from Prometheus and proprietary evaluators, finding that Prometheus is more critical while GPT-4 feedback is less abstract and general. This critical style may help diagnose model limitations but may also reflect reduced optimism.

  • Feedback comparison: GPT-4 feedback was described as less abstract and general, but prone to being overly critical.
  • Feedback comparison: Prometheus tends to produce more critical feedback than GPT-4 and GPT-3.5-Turbo.
  • Implications: Prometheus’s critical style may help analyze LLM limitations or provide feedback for improving a target LLM, although these uses remain future work.
  • Additional analysis: The analysis includes box-and-whisker plots relating response length to scores across evaluation settings.

F IS THERE A LENGTH BIAS DURING EVALUATION?

The length-bias analysis examines whether evaluator scores favor longer responses. Prometheus and GPT-4 show similar distributions across scores, but the tested instances lack adversarial examples and broader investigation remains open.

  • Bias analysis: LLM evaluators can be vulnerable to biases, motivating analysis of whether Prometheus favors longer responses.
  • Findings: Prometheus and GPT-4 show similar response-length distributions across scores rather than favoring longer responses.
  • Scope: The FEEDBACK BENCH and Vicuna Bench test instances did not include adversarial examples targeting verbosity bias.
  • Scope: Whether length bias transfers to fine-tuned evaluator LMs requires more extensive research.

J EXPERIMENTAL DETAILS FOR HUMAN EVALUATION

Human evaluation used a sequential interface to collect scores, feedback preferences, and reasons for those preferences. Qualitative examples also compare evaluator feedback across general and code-focused settings.

  • Evaluation procedure: The human-evaluation interface collected response scores, pairwise feedback preferences, and reasons for choosing one feedback over another.
  • Evaluation procedure: Annotators were instructed to answer the interface questions sequentially without solving later questions first.
  • Qualitative comparisons: Qualitative examples compare feedback from GPT-4, Prometheus, and Code-Llama trained on the FEEDBACK COLLECTION.
  • Example evaluation: The code example’s implementation counts every read word because its comparison overwrites the target word variable.
  • Example evaluation: The example also uses a nonstandard `exists` function and lacks error handling for missing or unopened files.
Loading 2310.08491v2…