PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4

Hui Huang, Xingyuan Bu, Hongli Zhou, Yingqi Qu, Jing Liu, Muyun Yang, Bing Xu, Tiejun Zhao

arXiv:2403.02839v4cs.CL

TL;DR

External evaluation APIs raise privacy and reproducibility concerns, while fine-tuned judges have been presented as an open-source alternative. This paper empirically studies their evaluation capabilities and finds that, despite strong in-domain results, they remain limited across several dimensions and are not universal GPT-4 substitutes.

  • Problem

    External evaluation APIs raise privacy-leakage and reproducibility concerns, motivating the proposal of open-source fine-tuned judges.

    Source: §1 · p. 1

  • Method

    The study cross-validates publicly released judge checkpoints across evaluation schemes and tests them on the multi-turn MT-Bench dataset.

    Source: §2.1 · p. 2

  • Results

    Fine-tuned judges achieve superior accuracy on their respective in-domain test sets but remain limited in evaluation-scheme generalization, superficial-quality fairness, aspect-specific evaluation, and prompting.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Fine-tuned judges cannot replace GPT-4 as universal LLM evaluators and should be used with attention to domain and task adaptability.

    Source: §4 · p. 5

    The authors present no proposed solution to mitigate fine-tuned judges’ limitations, leaving related methods for future work.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Recently, there has been a growing trend of utilizing Large Language Model (LLM) to evaluate the quality of other LLMs. Many studies have fine-tuned judge models based on open-source LLMs for evaluation. While the fine-tuned judge models are claimed to achieve comparable evaluation capability with GPT-4, in this work, we conduct an empirical study of LLM-as-a-Judge. Our findings indicate that although the fine-tuned judge models achieve high performance on in-domain test sets, even surpassing GPT-4, they underperform GPT-4 across several dimensions, including generalizability, fairness and adaptability. We also reveal that the fine-tuned judge model inherently operates as a task-specific classifier, consequently imposing the limitations.

1 Introduction

This study tests fine-tuned LLM judges whose reported in-domain performance had prompted claims of parity with proprietary models. It finds limits in their broader evaluation capabilities and cautions against treating them as general GPT-4 substitutes.

1 Introduction

  • Open-source fine-tuned judges were proposed to address privacy-leakage and reproducibility concerns associated with external evaluation APIs.These models rely on open-source foundation models and data constructed from GPT-4 or human annotation.Source: §1 · p. 1
  • Earlier studies reported fine-tuned judges performing on par with GPT-3.5 and GPT-4 on their respective meta-evaluation benchmarks.Source: §1 · p. 1
  • Despite superior in-domain accuracy, fine-tuned judges face limits versus proprietary models: scheme constraints, superficial-quality bias, weak aspect-specific evaluation, and little benefit from prompting.The authors conclude they cannot serve as general substitutes for GPT-4.Source: §1 · pp. 1–2
  • The authors attribute these limitations to fine-tuning turning the foundation model into a task-specific classifier overfit to its fine-tuning data.Source: §1 · p. 1

2 How Far can Fine-tuned Judges Go?

Across evaluation schemes, adversarial fairness tests, fine-grained aspects, and prompting strategies, fine-tuned judges show limitations despite strong performance on their own training schemes. GPT-3.5 and GPT-4 perform more consistently across schemes, and proprietary models benefit more from prompt engineering.

2.1 Constrained by Evaluation Scheme

  • Fine-tuned judges perform best on their own training schemes, with results comparable to GPT-4, but drop substantially on unseen schemes; GPT-3.5 and GPT-4 perform more consistently across schemes.Source: §2.1 · p. 2

2.2 Biased Towards Superficial Quality

  • On LLMBar’s adversarial tests, fine-tuned judges perform worse than random guessing and favor superficial qualities, while GPT-4 achieves decent accuracy.LLMBar pairs correct answers with incorrect answers that have better superficial qualities, such as fluency or verbosity.Source: §2.2 · pp. 2–3, Table 4 · p. 3

2.3 Incapable of Aspect-specific Evaluation

  • Fine-tuned judges fall far behind on factuality, toxicity, and safety evaluation; Prometheus also fails to learn the correlation between fine-grained aspects and evaluation results.Source: §2.3 · p. 3, Table 3 · p. 3
  • Using original rather than aspect-specific prompts leaves Auto-J and Prometheus performance roughly unchanged, indicating that aspect-specific prompting is not taking effect.Source: §2.3 · p. 3

2.4 Can not Benefit from CoT and ICL

  • Fine-tuned judges barely benefit from ICL or CoT and sometimes decline, while proprietary models improve substantially through both strategies.JudgeLM and PandaLM failed to produce CoT under modified prompts and retained their original output format.Source: §2.4 · p. 3, Table 7 · p. 4
  • For JudgeLM-7B, fine-tuning with original or o1-generated CoT degrades performance versus no CoT, with the o1-generated CoT causing a more severe drop.Source: §2.4 · pp. 3–4, Table 6 · p. 4

3 The Essence of Fine-tuned Judge: A Task-specific Classifier

The paper argues that fine-tuning judges on a single task with a fixed prompt turns them into task-specific classifiers. Classification models perform as well as generation models, and their predictions are more closely correlated with one another than with GPT-4.

3 The Essence of Fine-tuned Judge: A Task-specific Classifier

  • The authors characterize judges fine-tuned on a single task as task-specific classifiers overfit to their training data and fixed prompt template.Source: §3 · pp. 4–5
  • Predictions from the three classification models correlate more closely with each other than with GPT-4, despite differences in architecture.Source: Figure 2 · p. 4, Figure 3 · p. 4, §3 · p. 5
  • The tested designs include generation-style Vicuna, Vicuna classification or regression with a new linear projection head, and a DeBERTa-based classifier.The DeBERTaV3large model is described as 20 times smaller than the 7B Vicuna version.Source: §3 · pp. 4–5
  • Classification models perform equally well as generation models, and a DeBERTa-based classifier achieves comparable performance with LLM-based evaluators.The authors suggest that an encoder-only architecture might be more suitable for classification.Source: §3 · p. 5

4 Conclusion

Fine-tuned models outperform on in-domain tests but retain limitations relative to GPT-4; they are not universal evaluators and require attention to domain and task adaptability.

4 Conclusion

  • Despite superior in-domain test performance, fine-tuned judges retain limitations compared with GPT-4.The conclusion says they cannot replace GPT-4 as universal evaluators.Source: §4 · p. 5
  • New domains and tasks may fall outside the fine-tuning scope, even if more fine-tuning data could mitigate some limitations.The conclusion therefore advises watching domain and task adaptability when using these models.Source: §4 · p. 5
  • Fine-tuned judges should be used judiciously rather than treated as universal evaluators for LLMs.Source: §4 · p. 5

Limitations

The authors identify limits in remedies, bias assessment, and meta-evaluation, alongside a possible test-length issue for Auto-J.

Limitations

  • Due to time constraints, the authors present no possible solution to mitigating fine-tuned judge limitations.They say they will investigate related methods in future work.Source: Limitations · p. 5
  • Auto-J-test is the stated exception, possibly because a large proportion of its test data exceeds 512.Source: Limitations · p. 5
  • The bias assessment is general rather than fine-grained, omitting analyses such as position and verbosity bias.Source: Limitations · p. 5
  • The meta-evaluation process did not include manual inspection; the authors say human evaluators would enhance claim credibility.Source: Limitations · p. 5

A.1 Training Settings

The study fine-tunes four groups of judge models in generation and classification styles, using matched prompts and hyperparameters to compare prediction methods.

A.1 Training Settings

  • The study fine-tunes JudgeLM, PandaLM, Auto-J, and Prometheus models in both generation-style and classification-style forms.Source: §A.1 · p. 8
  • Both model styles use configurations based on their foundation model.Source: Table 8 · p. 8
  • Generation and classification models use the same prompt templates and hyperparameters, differing only in prediction method.Generation reuses the pretrained language-model head; classification uses a newly initialized linear projection layer decoupled from language modeling.Source: §A.1 · p. 8
  • Classification-based judges add a newly initialized classification or regression head to predict the result.Source: Figure 4 · p. 8

A.2 Prompt Templates

The appendix adapts judge prompts across pairwise selection, pointwise grading, multi-turn evaluation, and benchmark-specific tasks.

A.2 Prompt Templates

  • JudgeLM and PandaLM pairwise prompts are adapted for pointwise grading, while Prometheus pointwise prompts are adapted for pairwise selection.Auto-J has prompts for both selection and grading; the prompt figures cover these configurations.Source: §A.2 · p. 8, Figure 5 · p. 9, Figure 6 · p. 9
  • MT-Bench prompts for JudgeLM, PandaLM, Auto-J, and Prometheus are adapted for multi-turn evaluation.The appendix also provides chain-of-thought prompt templates for JudgeLM and PandaLM.Source: §A.2 · p. 9, Figure 13 · p. 12, Figure 14 · p. 12
  • For HaluEval and ToxicChat, Auto-J and Prometheus use pointwise grading with grid-searched classification thresholds; SALAD-Bench uses pairwise selection models.JudgeLM, PandaLM, and Auto-J are applied to SALAD-Bench pairwise selection.Source: §A.2 · p. 9, Figure 18 · p. 14, Figure 19 · p. 14
  • For Prometheus, benchmark-specific scoring rubrics are defined because its original prompt includes scoring rubrics.Source: §A.2 · p. 9

Related papers

Loading 2403.02839v4…