Source-linked AI summary
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
TL;DR
Open evaluator LMs have lagged human and proprietary judges in agreement, flexibility, and custom-criteria evaluation. Prometheus 2 merges separately trained direct-assessment and pairwise-ranking evaluators, achieving leading performance across four benchmarks of each type. Its main scope limitation is support for only 1–5 absolute scores or binary comparative judgments.
Problem
Open evaluator LMs can diverge from human judgments and often lack support for both assessment formats and user-defined criteria.
Method
Prometheus 2 merges the weights of evaluator LMs trained separately on direct-assessment and pairwise-ranking feedback, including a pairwise dataset with over 1,000 custom criteria.
Results
Prometheus 2 achieves the highest correlation or agreement among tested open evaluators across four direct-assessment and four pairwise-ranking benchmarks.
Takeaways & Limitations
Weight merging yields a unified evaluator LM that performs well in both direct-assessment and pairwise-ranking schemes.
Takeaways & Limitations
Prometheus 2 supports only 1–5 absolute evaluation or binary comparative evaluation, excluding formats such as 1–10 scoring, ranking five responses, and checklists.
Abstract
from arXiv · showhide
Proprietary LMs such as GPT-4 are often employed to assess the quality of responses from various LMs. However, concerns including transparency, controllability, and affordability strongly motivate the development of open-source LMs specialized in evaluations. On the other hand, existing open evaluator LMs exhibit critical shortcomings: 1) they issue scores that significantly diverge from those assigned by humans, and 2) they lack the flexibility to perform both direct assessment and pairwise ranking, the two most prevalent forms of assessment. Additionally, they do not possess the ability to evaluate based on custom evaluation criteria, focusing instead on general attributes like helpfulness and harmlessness. To address these issues, we introduce Prometheus 2, a more powerful evaluator LM than its predecessor that closely mirrors human and GPT-4 judgements. Moreover, it is capable of processing both direct assessment and pair-wise ranking formats grouped with a user-defined evaluation criteria. On four direct assessment benchmarks and four pairwise ranking benchmarks, Prometheus 2 scores the highest correlation and agreement with humans and proprietary LM judges among all tested open evaluator LMs. Our models, code, and data are all publicly available at https://github.com/prometheus-eval/prometheus-eval.
1 Introduction
Prometheus 2 addresses weaknesses in open evaluator LMs by unifying direct assessment and pairwise ranking while supporting custom criteria. It achieves leading agreement with humans and proprietary LM judges across both evaluation schemes.
- Language-model evaluation offers a scalable, cheap way to assess increasingly diverse and complex outputs.
- Existing open evaluators diverge from strong evaluators and typically support only one evaluation format or general criteria such as helpfulness and harmlessness.
- Prometheus 2 merges evaluator models trained separately for direct assessment and pairwise ranking into a unified evaluator LM.
- Pearson correlation surpasses other open-evaluator baselines by 0.2 across four direct-assessment datasets, while pairwise-ranking performance halves the gap with GPT-4.
- The Preference Collection provides pairwise-ranking feedback with 1K custom evaluation criteria beyond helpfulness and harmlessness.
2 Related Work
Related work develops language-model judges to capture richer evaluation than conventional similarity metrics and trains open evaluators to reduce reliance on proprietary models. Prometheus 2 extends this direction by merging models across assessment formats.
- ROUGE, BLEU, and BERTScore can produce false negatives because lexical or semantic similarity may miss high-quality responses differing from references.
- Language-model judges offer scalable evaluation with greater depth and granularity, but open evaluator LMs aim to reduce reliance on proprietary systems.
- Prometheus 2 merges models trained for direct assessment and pairwise ranking to function in both formats while targeting proprietary-LM-level evaluation performance.
3 Methodology
Prometheus 2 unifies direct assessment and pairwise ranking through custom-criterion evaluation, preference data, and merged evaluator models. The method compares prompting, single-format training, joint training, and weight merging.
- Evaluation formats: Direct assessment maps an instruction and response to a scalar score, while pairwise ranking selects between two responses.
- Evaluation formats: Custom-criterion pairwise evaluation adds the evaluation criterion as input and uses feedback comparing the two responses.
- Training data: The PREFERENCE COLLECTION extends the FEEDBACK COLLECTION with fine-grained pairwise feedback and 1K evaluation criteria.
- Training methods: The study compares prompting, single-format training, joint training, and weight merging across evaluator-LM baselines.
- Training methods: Prometheus 2 is obtained by merging evaluator models separately trained on direct-assessment and pairwise-ranking datasets.
- Training methods: DARE-Linear performs best among the tested merging variants and is used to train Prometheus 2.
4 Experimental Setup
The evaluation covers four direct-assessment and four pairwise-ranking benchmarks using correlation and accuracy against reference evaluators or human judgments. The setup also examines tie handling and in-domain versus out-of-domain evaluation.
- Benchmarks: Four direct-assessment benchmarks evaluate scoring correlations: Vicuna Bench, MT Bench, FLASK, and Feedback Bench.
- Benchmarks: Four pairwise-ranking benchmarks evaluate human-preference agreement: HHH Alignment, MT Bench Human Judgment, Auto-J Eval, and Preference Bench.
- Metrics and protocols: Direct assessment uses reference-based inputs and Pearson, Spearman, and Kendall-Tau correlations against reference evaluators.
- Metrics and protocols: Pairwise ranking uses reference-free evaluation and accuracy against human judgments.
- Metrics and protocols: Tie-inclusive and tie-exclusive settings are both evaluated for benchmarks containing human-assigned ties.
- Benchmark scope: Feedback Bench and Preference Bench are in-domain tests for Prometheus 2, while HHH Alignment and Auto-J Eval are in-domain tests for other evaluator models.
5 Experimental Results
Prometheus 2 achieves stronger agreement with human and proprietary-LM judgments than the evaluated open evaluator baselines across direct assessment and pairwise ranking. Its gains also extend beyond in-domain tests.
- Direct assessment results: Pearson correlations for Prometheus 2, GPT-4-1106, Claude-3-Opus, and humans exceed 0.5 across direct-assessment references and benchmarks.
- Direct assessment results: Prometheus 2 exceeds Prometheus and Auto-J by at least 0.2 Pearson-correlation units across direct-assessment benchmarks against proprietary LMs.
- Direct assessment results: On FLASK, PROMETHEUS-2-8X7B reaches 0.555 correlation with humans, compared with Prometheus-13B’s previous 0.449.
- Evaluation caveat: Pair RM and Ultra RM are excluded from tie-inclusive settings because they cannot process them.
- Pairwise ranking results: Prometheus 2 achieves the highest scores on all four pairwise-ranking benchmarks and effectively simulates human judgments.
- Pairwise ranking results: Prometheus 2 at least halves the performance gap with proprietary LMs on out-of-domain pairwise-ranking test sets.
6 Analyses of Weight Merging
The analyses compare weight merging with joint training and same-format ensembling, then examine transfer between direct assessment and pairwise ranking. Weight merging unifies the formats effectively, with asymmetric benefits across tasks.
- Joint training often underperforms single-format training, indicating negative task transfer.
- α=0.5 yields optimal direct-assessment performance, indicating equal contributions from both feedback formats.
- α=0.3 yields optimal pairwise-ranking performance, while pairwise-ranking training improves direct assessment more than the reverse.
- Weight merging outperforms single-format and jointly trained evaluator LMs across multiple benchmarks.
- Merging direct-assessment and pairwise-ranking models outperforms merging models trained on the same format, indicating positive task transfer rather than ensembling.
7 Conclusion
Prometheus 2 is an open evaluator LM designed to handle both direct assessment and pairwise ranking. Its unified performance narrows the gap with proprietary evaluations, supported by a new fine-grained preference dataset.
- Prometheus 2 is an open-source LM specialized in evaluating other responses.
- Prometheus 2 processes both direct assessment and pairwise ranking, unlike existing open evaluator LMs described as unable to handle both effectively.
- Prometheus 2 demonstrates superior performance on both evaluation schemes and significantly narrows the gap with proprietary LM-based evaluations.
- The Preference Collection contains over 1,000 instance-wise evaluation criteria beyond helpfulness and harmlessness.
Limitations
The paper identifies limitations in how evaluator LMs are assessed, how broadly they generalize, the formats Prometheus 2 supports, and why weight merging works.
- Evaluator capability is assessed indirectly through similarity to human or proprietary-LM evaluations, which may not be the best approach.
- Generalization analysis across eight datasets with human judgments may still be insufficient because obtaining evaluation results is challenging.
- Prometheus 2 supports only 1-5 Likert absolute evaluation and ‘A is better & B is better’ comparative evaluation formats.
- The underlying reason weight merging works well remains fundamentally unexplained beyond empirical comparisons and an interpretation about task relatedness.
- The Preference Collection uses five NLP annotators and a three-part verification process for sampled instances.
C Direct Assessment Results: Extended
Extended results preserve the reported advantage of Prometheus 2 among open evaluator LMs under alternative correlation metrics. The accompanying tables report correlations and evaluation-format consistency statistics.
- Kendall-Tau and Spearman metrics maintain the overall trends, with Prometheus 2 showing superior performance among open evaluator LMs.
- Table 11 reports Kendall-Tau correlations between reference evaluators and evaluator LMs.
- Table 12 reports Spearman correlations between reference evaluators and evaluator LMs.
- Table 13 reports pairwise-ranking accuracy across direct-assessment and pairwise-ranking formats, where smaller ∆ values indicate more robust cross-format evaluation.
- Tables 14 and 15 report consistency using Krippendorff’s alpha and transitivity statistics, respectively.
E Consistency of Evaluator LMs
Prometheus 2 is evaluated for consistency across repeated direct-assessment judgments, pairwise rankings, and evaluation formats. The experiments also examine reference-free direct assessment, where removing references reduces correlation with human judgments.
- Direct assessment consistency: Three non-deterministic direct-assessment inferences are used to measure scoring consistency with Krippendorff’s alpha.The evaluation follows the experimental design of Ye et al. (2023) and uses temperature 1.0.
- Pairwise ranking consistency: Prometheus 2 achieves pairwise-ranking consistency on par with GPT-4, supporting robust judgments in pairwise schemes.Consistency is measured through transitivity across response comparisons.
- Reference-free evaluation: Reference-based evaluations outperform reference-free evaluations across all evaluator LMs in Pearson correlation with human judgments.Table 16 compares evaluator models with and without reference answers.
- Reference-free evaluation: 0.045 and 0.063 are the performance degradations reported for GPT-4-1106 after discarding the reference answer.The decreases occur across the two evaluated benchmarks.
- Reference-free evaluation: Prometheus 2 remains stronger than its corresponding base instruct models in the reported reference-free correlations.Prometheus-2-7B records 0.403 and 0.425 versus 0.310 and 0.374 for Mistral-7B-Instruct-v0.2; Prometheus-2-8x7B records 0.424 and 0.411 versus 0.322 and 0.386 for Mixtral-8x7B-Instruct-v0.1.
G Merging Method Ablation
The ablation compares linear, Slerp, Task Arithmetic, TIES, and DARE weight-merging methods for unified evaluator models. DARE-Linear performs best on average and is used to train Prometheus-2-7B.
- Merging methods: Slerp merging normalizes two weight sets and interpolates them while preserving their spherical-space geometry.The interpolation coefficient is α, applied to θd and θp after normalization.
- Merging methods: TIES removes redundant task-vector weights and resolves disagreements between oppositely directed weights.Its Trim, Elect, and Disjoint operations act on θd − θinit and θp − θinit.
- Merging methods: DARE applies random dropping and rescaling to task vectors, with a TIES variant that additionally applies Elect after rescaling.The task vectors are formed relative to the base-model weights θinit.
- Ablation results: DARE-Linear achieves the best direct-assessment performance, followed by DARE-TIES and Linear merging.The comparison covers evaluator models using different merging methods.
- Ablation results: Task Arithmetic achieves the best pairwise-ranking performance, with only a minimal difference from the other merging methods.The result is reported across the pairwise-ranking benchmarks in the ablation.
- Ablation results: DARE-Linear performs best on average, motivating its use for training Prometheus-2-7B.DARE merging performs best with Mixtral-8x7B as the base model.