PaperlayerView exact revision on arXiv

Source-linked AI summary

JudgeLM: Fine-tuned Large Language Models are Scalable Judges

Lianghui Zhu, Xinggang Wang, Xinlong Wang

arXiv:2310.17631v2cs.CLcs.AI

TL;DR

Open-ended LLM evaluation is difficult because existing benchmarks and metrics do not measure responses comprehensively. JudgeLM fine-tunes open-source LLMs using a large GPT-4-judged dataset and bias-mitigation techniques, achieving state-of-the-art judge performance with agreement exceeding 90%.

  • Problem

    Existing benchmarks and traditional metrics cannot comprehensively or accurately evaluate LLMs in open-ended scenarios.

  • Method

    JudgeLM fine-tunes open-source LLMs on diverse task seeds, LLM-generated answers, and GPT-4 judgments, using swap augmentation, reference support, and reference drop.

  • Results

    JudgeLM achieves state-of-the-art judge performance on two benchmarks, with agreement exceeding 90% and JudgeLM-7B evaluating 5000 response pairs in 3 minutes on 8 A100 GPUs.

  • Takeaways & Limitations

    JudgeLM provides an efficient, reproducible, and privacy-preserving approach that generalizes to multiple LLM judging settings.

  • Takeaways & Limitations

    The cost of GPT-4-generated judge data limits further scaling of the judge dataset.

Abstract

from arXiv · show

Evaluating Large Language Models (LLMs) in open-ended scenarios is challenging because existing benchmarks and metrics can not measure them comprehensively. To address this problem, we propose to fine-tune LLMs as scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in open-ended benchmarks. We first propose a comprehensive, large-scale, high-quality dataset containing task seeds, LLMs-generated answers, and GPT-4-generated judgments for fine-tuning high-performance judges, as well as a new benchmark for evaluating the judges. We train JudgeLM at different scales from 7B, 13B, to 33B parameters, and conduct a systematic analysis of its capabilities and behaviors. We then analyze the key biases in fine-tuning LLM as a judge and consider them as position bias, knowledge bias, and format bias. To address these issues, JudgeLM introduces a bag of techniques including swap augmentation, reference support, and reference drop, which clearly enhance the judge's performance. JudgeLM obtains the state-of-the-art judge performance on both the existing PandaLM benchmark and our proposed new benchmark. Our JudgeLM is efficient and the JudgeLM-7B only needs 3 minutes to judge 5K samples with 8 A100 GPUs. JudgeLM obtains high agreement with the teacher judge, achieving an agreement exceeding 90% that even surpasses human-to-human agreement. JudgeLM also demonstrates extended capabilities in being judges of the single answer, multimodal models, multiple answers, multi-turn chat, etc. Code is available at https://github.com/baaivision/JudgeLM.

1 INTRODUCTION

JudgeLM addresses the difficulty of evaluating open-ended LLM responses by fine-tuning open-source LLMs as scalable judges. It combines a large GPT-4-judged dataset with bias analysis and techniques intended to improve reliability, efficiency, and reproducibility.

  • Introduction: JudgeLM fine-tunes open-source LLMs to evaluate answers in open-ended tasks and achieve agreement with a teacher judge.The models are trained at 7B, 13B, and 33B scales using varying data volumes.
  • Introduction: 105K seed questions, answer pairs, and GPT-4 judgments form the paper’s dataset for training and validating judge models.The dataset includes judgments generated with and without reference answers.
  • Introduction: JudgeLM targets position, knowledge, and format biases that can undermine fine-tuned LLM judges.The paper introduces fine-tuning-stage methods to address these biases.
  • Introduction: JudgeLM-7B evaluates 5000 response pairs in 3 minutes using 8 A100 GPUs.The authors contrast this speed and cost with arena-format evaluation and emphasize reproducibility and privacy relative to closed-source judges.
  • Introduction: JudgeLM achieves agreement exceeding 90%, surpassing the reported human-to-human agreement.The model also generalizes to extended judging tasks.

2 RELATED WORK

Related work evaluates LLMs through crowdsourced comparisons, GPT-4 judging, and fine-tuned open-source judges. These approaches involve trade-offs among reliability, cost, efficiency, reproducibility, data exposure, and training-data quality.

  • Crowdsourced evaluation: Chatbot Arena uses crowdsourced pairwise comparisons and Elo ratings, but human evaluation has high costs and low efficiency.Crowdsourced evaluations are described as more reliable than some alternatives.
  • LLM-based evaluation: GPT-4-based judging reduces reliance on crowdsourcing but may expose data and depend on changing APIs.The passage identifies reproducibility as a concern for closed-source judge APIs.

3 DATASET

The JudgeLM dataset is designed to provide diverse tasks, multiple LLM answers, reference answers, and detailed GPT-4 judgments for training and evaluating scalable judges. Its training setup teaches models to produce scores and reasons with or without references.

  • Dataset motivation: Existing judge datasets are limited in diversity and judgment granularity, motivating a richer dataset for fine-tuning evaluative LLMs.The paper identifies dataset quality as crucial for effective judge training.
  • Data generation: 105K instruction seeds are sampled from multiple sources, with answers collected from 11 open-source LLMs and paired with reference answers.GPT-4 assigns fine-grained scores and detailed reasons to randomly selected answer pairs.
  • Dataset split: The dataset is partitioned into 100K training samples and 5K validation samples.The split supports both judge-model training and evaluation.
  • Training: Training inputs contain a question, two answers, and an optional reference, while outputs contain scores and detailed reasons.Separate prompt templates support inputs with and without references.
  • Evaluation: JudgeLM evaluates answer pairs by generating scores, converting score comparisons into wins or ties, and optionally producing reasons.The evaluation uses agreement, precision, recall, F1-score, and reliability metrics.

4 INHERENT BIAS

The paper identifies position, knowledge, and format biases as reliability problems for fine-tuned LLM judges. These biases affect answer ordering, factual judgment, and transfer between reference-based and reference-free formats.

  • Position Bias: JudgeLM exhibits position bias by preferring the first answer when answer positions are swapped.The paper evaluates this behavior using qualitative and quantitative analyses.
  • Knowledge Bias: Knowledge bias occurs when missing or undesirable pretrained knowledge prevents judges from correctly evaluating open-ended tasks.The paper gives examples where judges lack the relevant truth needed for judgment.
  • Format Bias: Format bias causes judges trained with references to perform poorly without them, and judges trained without references to perform poorly with them.The paper calls these cross-format evaluations mismatched formats and links them to limited generalization.

5 METHOD

JudgeLM adapts open-source LLMs into scalable judges for open-ended answer evaluation and addresses position, knowledge, and format biases through targeted fine-tuning techniques.

  • JudgeLM fine-tunes open-source LLMs to determine the superior answer from candidate pairs in open-ended tasks.
  • Position bias: Swapping answer positions during fine-tuning teaches the judge to focus on answer content rather than position.The corresponding teacher scores and question indexes are swapped to preserve the judgment outcome.
  • Knowledge bias: Reference support supplies reference answers and GPT-4-generated reference-guided judgments to compensate for missing pretrained knowledge.Reference answers can also encode different preferences without requiring additional training.
  • Format bias: Reference drop randomly removes references during training so the judge can operate with or without them.This targets overfitting to a single prompt format and supports judgments from either references or pretrained knowledge.

6 EXPERIMENT

Experiments evaluate JudgeLM against existing judges, scaling behavior, efficiency, and ablations of its bias-mitigation techniques across multiple benchmarks and judging formats.

  • Main results: JudgeLM-7B outperforms PandaLM-7B, Auto-J, and InstructScore on all metrics and surpasses GPT-3.5 on the JudgeLM validation set.JudgeLM models handle every validation question, whereas vanilla Vicuna-13B fails 77% of questions.
  • Main results: JudgeLM-7B outperforms GPT-3.5 and PandaLM-7B on the PandaLM test set, while JudgeLM-33B exceeds GPT-4 on that specific task.The evaluation uses human annotations as ground truth.
  • Scaling: 90.06% agreement and 87.93% consistency are reached with a 33B-parameter judge trained on 100K examples.Both metrics increase as model size and training-data scale increase.
  • Ablation studies: Swap augmentation improves consistency by 5.44%, while reference support improves every metric and reference drop improves agreement and consistency across reference formats.These ablations associate the techniques with reduced position, knowledge, and format biases, respectively.
  • Ablation studies: Explanation-first judging has similar agreement but higher consistency than score-first judging, so the paper selects score-first for more flexible usage.
  • Teacher comparison: JudgeLM-33B achieves higher consistency than GPT-4, consistent with the paper’s knowledge-distillation framing in which the student learns from GPT-4 judgments.
  • Generalization: JudgeLM generalizes to multimodal, multiple-answer, single-answer, multi-turn, retrieval-format, toxic-chat, and reward-model judging tasks.A human-annotated multimodal benchmark is included among the reported evaluations.

7 CONCLUSION

The paper introduces a dataset and JudgeLM for efficient open-ended LLM evaluation, reports state-of-the-art judge performance, and presents techniques for addressing fine-tuning biases.

  • The dataset provides a high-quality, large-scale foundation for future research on LLM evaluation.
  • JudgeLM evaluates LLMs efficiently in open-ended tasks and achieves state-of-the-art judge performance on two benchmarks.
  • The paper identifies position, knowledge, and format biases in fine-tuned LLM judges and addresses them with proposed techniques.
  • Limitations: Scaling the judge dataset is limited by its cost: approximately $4,000 was spent to produce 100K GPT-4-generated judge examples.The authors expect synthetic judge data to support further performance improvements.

A.1 MORE ABOUT DATASET

The appendix documents JudgeLM’s dataset scope, quality controls, validation distributions, and comparisons with UltraFeedback and PandaLM. It also defines the metrics used to assess judge agreement and positional bias.

  • Dataset Usage Scope: The JudgeLM dataset is restricted to academic research, with commercial use prohibited under OpenAI’s terms.
  • Metric Calculations: Agreement is computed from TP, FP, TN, and FN, while positional reliability compares judgments before and after swapping answer order.
  • Dataset Quality: 105K samples are drawn after filtering incorrect, irrelevant, or harmful references, with answers generated by 11 LLMs.
  • Comparison with UltraFeedback: JudgeLM and UltraFeedback both provide scalar and text feedback, GPT-4 annotation, and fine-grained consideration, while JudgeLM adds human checking and re-annotation.
  • Comparison with PandaLM Test Set: PandaLM and JudgeLM differ substantially in question-category coverage, including categories represented in PandaLM but absent from JudgeLM.

A.2 FINE-TUNING SETTING

The appendix points to the hyper-parameters used for JudgeLM fine-tuning.

  • The fine-tuning hyper-parameters are listed in Table 11.

A.3 GENERALIZATION ABILITY OF JUDGELM

JudgeLM is evaluated across diverse task types, benchmarks, answer formats, and interaction settings. The experiments examine generalization to multimodal, out-of-distribution, retrieval-format, multiple-answer, single-answer, reward-model, and multi-turn-chat judging.

  • Generalize to Various Judging Tasks: JudgeLM-33B handles coding, common-sense, math, roleplay, and writing judgments, although math performance is slightly lower than coding and common-sense performance.
  • Generalize to Multimodal Judging Benchmark: JudgeLM supports multimodal evaluation by comparing model outputs with question text, ground-truth text, and optional images against human judgments.
  • Generalize to Out-of-distribution ToxicChat Benchmark: JudgeLM achieves superior precision and comparable accuracy to OpenAI Moderation on the out-of-distribution ToxicChat test set.
  • Generalize to Retrieval-format Benchmark: When injected reference paragraphs reach 400 words, JudgeLM’s agreement drops by at most 3.73% and consistency by 3.37%.
  • Generalize to Multiple-format benchmark: JudgeLM can judge and rank multiple answers within the LLM context limit, avoiding the O(n^2) calls required for a full pairwise matrix.
  • Consistency of Judging Forms: Pairwise and multiple-answer rankings are compared through consistency measurements, including Error Rate@2.
  • Generalize to Single Answer Grading: JudgeLM can grade a single answer by assigning the reference answer a full score and using text-form prediction against the ground truth.
  • Generalize to Reward Model: JudgeLM-7B achieves performance similar to Tulu-2-dpo-70B on RewardBench.

A.4 MORE DISCUSSION

JudgeLM extends open-source LLM judging through scalable evaluation, bias-mitigation methods, and broad generalization across model sizes, datasets, tasks, and formats. The discussion emphasizes efficiency, reliability, and remaining limitations in teacher bias and future critique capabilities.

  • Contributions: JudgeLM introduces a high-quality dataset and scalable judge models spanning 7B, 13B, and 33B parameters with training sets from 3.5K to 100K examples.The dataset contains diverse seed tasks, generated answers, and GPT-4 judgments.
  • Biases and methods: Bias-mitigation methods improve JudgeLM consistency and reliability across different judging cases.The paper analyzes position, knowledge, and format biases and introduces methods including swap augmentation, reference support, and reference drop.
  • Generalization: JudgeLM generalizes beyond answer-pair comparison to 19 judging tasks, human-annotated and multimodal benchmarks, retrieval and multiple-answer formats, single-answer grading, and multi-turn chat.These extensions include coding, common-sense, math, roleplay, and writing tasks.
  • Limitations and future work: GPT-4 teacher judgments have inherent cultural, societal, linguistic, position, and knowledge biases, despite additional checking for accuracy and objectivity.The paper also identifies training-query distribution as a future opportunity for improving fairness and relevance.
Loading 2310.17631v2…