PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering

Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, Xin Xia

arXiv:2502.06193v3cs.SEcs.AI

TL;DR

Evaluating LLM-generated software-engineering code and text is difficult with existing metrics, while the human alignment of LLM-as-a-judge methods remains insufficiently established. The paper empirically compares these methods with human judgments across three SE tasks and finds task-dependent performance, with output-based methods using large LLMs achieving near-human alignment on code translation and generation.

  • Problem

    Existing SE evaluation metrics are costly, limited in scope, or unsuitable for some generated text, and reliable human alignment of LLM-as-a-judge methods remains unclear.

  • Method

    The study generates and manually scores responses from three SE datasets, then compares seven LLM-as-a-judge methods, two evaluation-tuned LLMs, and conventional metrics with human scores.

  • Results

    LLM-as-a-judge performance depends on the SE task; output-based methods with large LLMs achieve near-human alignment in code translation and generation but generally perform worse in pairwise comparisons.

  • Takeaways & Limitations

    Output-based methods with large LLMs provide the strongest studied human alignment for some SE evaluation tasks, while score reliability is not consistent across tasks and comparisons.

  • Takeaways & Limitations

    The studied methods lack generalizability across SE tasks, with human alignment ranging from near-human to unusable depending on the task.

Abstract

from arXiv · show

Recently, large language models (LLMs) have been deployed to tackle various software engineering (SE) tasks like code generation, significantly advancing the automation of SE tasks. However, assessing the quality of these LLM-generated code and text remains challenging. The commonly used Pass@k metric necessitates extensive unit tests and configured environments, demands a high labor cost, and is not suitable for evaluating LLM-generated text. Conventional metrics like BLEU, which measure only lexical rather than semantic similarity, have also come under scrutiny. In response, a new trend has emerged to employ LLMs for automated evaluation, known as LLM-as-a-judge. These LLM-as-a-judge methods are claimed to better mimic human assessment than conventional metrics without relying on high-quality reference answers. Nevertheless, their exact human alignment in SE tasks remains unexplored. In this paper, we empirically explore LLM-as-a-judge methods for evaluating SE tasks, focusing on their alignment with human judgments. We select seven LLM-as-a-judge methods that utilize general-purpose LLMs, alongside two LLMs specifically fine-tuned for evaluation. After generating and manually scoring LLM responses on three recent SE datasets of code translation, code generation, and code summarization, we then prompt these methods to evaluate each response. Finally, we compare the scores generated by these methods with human evaluation. The results indicate that output-based methods reach the highest Pearson correlation of 81.32 and 68.51 with human scores in code translation and generation, achieving near-human evaluation, noticeably outperforming ChrF++, one of the best conventional metrics, at 34.23 and 64.92. Such output-based methods prompt LLMs to output judgments directly, and exhibit more balanced score distributions that resemble human score patterns. Finally, we provide...

1 Introduction

Evaluating LLM-generated software-engineering code and text remains difficult because existing metrics are costly, narrow, or poorly suited to multiple SE tasks. This study tests whether LLM-as-a-judge methods align with human judgments across realistic SE tasks.

  • Pass@k accurately evaluates functional correctness but requires comprehensive unit tests and configured environments, and cannot assess non-functional qualities or text-generating SE tasks.
  • LLM-as-a-judge methods aim to automate evaluation by aligning LLM judgments with human preferences, but existing benchmarks contain too few and too-simple coding tasks.
  • The study evaluates code translation, code generation, and code summarization using representative datasets and sampled instructions, responses, and competing evaluation methods.
  • Output-based methods with large LLMs achieve near-human alignment in code translation and generation, while alignment depends heavily on the SE task.
  • The study contributes a manually curated meta-evaluation dataset and analyzes how LLM-as-a-judge methods score SE responses.

2 Related Work

Prior SE evaluation relies heavily on conventional metrics and lacks challenging, human-annotated meta-evaluation data. LLM-based evaluators offer broader capabilities, but reliable alignment on difficult SE samples remains unresolved.

  • Code LLMs for SE: LLM terminology in this paper treats pre-trained Transformers generally as LLMs, including models that are not instruction-tuned.
  • SE Benchmarks and Metrics: SE benchmarks cover code generation, code review, summarization, code search, and related tasks, but commonly evaluate responses with conventional metrics.
  • SE Benchmarks and Metrics: Many SE datasets lack responses of varying quality and human-annotated scores, limiting their use for evaluating evaluation metrics.
  • LLM-as-a-Judge in NLP: Embedding-based methods compare contextual representations of responses and references through token-level similarity.
  • LLM-as-a-Judge in NLP: Probability-based methods score responses using LLM generation probabilities, with some supporting reference-free or multi-aspect evaluation.
  • LLM-as-a-Judge in NLP: Output-based methods directly prompt LLMs for judgments without requiring access to internal representations or probabilities.
  • LLM-as-a-Judge in NLP: Existing LLM-as-a-judge methods have not been tested on enough challenging SE samples to establish reliable human alignment.

3 LLM-as-a-Judge Framework Overview

The framework categorizes LLM-as-a-judge methods by the LLM features they use: embeddings, probabilities, or directly generated judgments. It defines source instructions, target responses, and reference answers as the core inputs.

  • The framework groups methods into embedding-based, probability-based, and output-based categories according to the LLM features they use.
  • The instruction, response, and reference answer are represented as src, tgt, and ref sequences respectively.
  • Embedding-based methods obtain token representations for the response and reference answer before combining token-level cosine similarities.
  • The framework’s illustrative swap example exchanges two integer variables using a temporary variable.
  • The example evaluation output separately scores functionality and readability before providing explanations.

4 Study Design

The study examines different LLM-as-a-judge methods for evaluating responses across three software engineering tasks. It combines human and LLM evaluation and analyzes the correlations between their judgments.

  • The study evaluates responses from three software engineering tasks using different LLM-as-a-judge methods.
  • Researchers collect task instructions and generate responses from representative datasets before evaluation.
  • Human and LLM evaluations are compared through correlation analysis.

4.1 Datasets and Preprocessing

The study evaluates code translation, code generation, and code summarization using recent representative datasets, filtered instructions, and structured human scoring. It generates 450 responses and averages scores from two expert evaluators.

  • Task and dataset selection: The evaluation covers code translation, code generation, and code summarization as code-code, text-code, and code-text tasks.The selected datasets are CodeTransOcean, ComplexCodeEval, and CodeXGLUE.
  • Task and dataset selection: CodeTransOcean’s MultilingualTrans subset covers eight languages and averages 491 tokens per test example.This subset provides longer multilingual code pairs than the CodeXGLUE CodeTrans benchmark cited for comparison.
  • Task and dataset selection: CodeXGLUE’s code-text subset filters CodeSearchNet samples by code syntax, language, docstring length, and English-language requirements.The source dataset contains code snippets paired with function documentation serving as summaries.
  • Task and dataset selection: ComplexCodeEval contains Java and Python samples with functional dependencies, timestamps, and unit tests as supplemental context.The benchmark supports code generation, code completion, unit test generation, and API recommendation.
  • Sampling and human evaluation: The study samples 50 instructions from each filtered dataset, producing 150 instructions and 450 generated responses.Three randomly selected code LLMs answer each instruction, while responses are evaluated by two language-expert human evaluators using two task-specific aspects and 1–5 criteria.
  • Preprocessing: Preliminary checks removed incorrect or unsuitable examples, including short reference summaries and code-generation cases where dependency information impeded meaningful human scoring.Programming languages were limited to Java, Python, C, and C++ according to evaluator expertise.

4.2 Selected Methods

The study compares reference-based conventional metrics, embedding- and probability-based methods, general-purpose LLM judges with inference strategies, and evaluation-specific fine-tuned models. Methods receive different combinations of instructions, responses, references, and evaluation aspects, with only final verdicts retained.

  • Conventional metrics: Five conventional metrics score responses against references, including BLEU, ROUGE-L, METEOR, ChrF++, and CrystalBLEU.These methods do not use the instruction; CrystalBLEU removes corpus-common n-grams before computing BLEU.
  • Embedding-based methods: Embedding-based methods compare token representations of responses and references using UniXcoder, BERTScore, or MoverScore.BERTScore combines token-level precision and recall into F1, while MoverScore uses a minimum-cost flow formulation.
  • Probability-based methods: Probability-based methods use generation probabilities from davinci-002, including GPTScore and reference-free FFLM.GPTScore uses sequence log probabilities, whereas FFLM fuses prior, posterior, and prefix probabilities.
  • LLM-as-a-judge methods: G-Eval, BatchEval, and Vanilla test whether inference strategies improve general-purpose LLM alignment with human evaluation.G-Eval averages 20 high-temperature passes after generating evaluation steps; BatchEval uses five rounds with batches of 10 responses.
  • Fine-tuned evaluators: The supervised fine-tuning group pairs Auto-J and Prometheus-v2-BGB-8x7B with their respective unfine-tuned base LLMs.The study applies each judge’s default prompt template and uses greedy decoding locally.
  • Evaluation protocol: Prompts provide the instruction, response, and evaluation aspects, while pairwise comparisons include both responses.The meta-evaluation retains only extracted scores or comparison results and discards explanations.

4.3 Meta-Evaluation

The meta-evaluation measures how closely automated metrics align with human scores and how consistently they compare response pairs. It uses correlation, accuracy, agreement, and significance testing across the studied tasks.

  • Individual scoring: Individual-scoring methods are evaluated using Spearman’s ρ, Pearson’s R, and Kendall’s τ correlations with human scores.These coefficients quantify alignment between automated metric scores and human evaluation.
  • Pairwise comparison: Pairwise methods are evaluated with Accuracy and Agreement after reversing response order.Agreement checks whether a method gives the same judgment when the two responses swap positions.
  • Reporting and significance: All reported correlations, accuracies, and agreements are multiplied by 100, and correlation p-values are checked against 0.05.The p-value threshold is used to ensure a 95% confidence interval for RQ1 correlations.

5 Study Results

LLM-as-a-judge performance varies substantially across SE tasks: output-based methods with large LLMs lead in code translation and generation, but conventional metrics remain stronger for code summarization. Score distributions and pairwise comparisons further reveal category-specific behavior and reliability limits.

  • RQ1: Alignment with Human Scores: 81.32 Pearson’s R was the highest code-translation alignment, achieved by BatchEval and exceeding conventional metrics capped at 34.23.BatchEval also reached ρ=73.67 and τ=59.80.
  • RQ1: Alignment with Human Scores: 68.51 Pearson’s R was the highest code-generation alignment, achieved by DeepSeek-V2.5, while conventional metrics reached 65.55.LLM-as-a-judge methods struggled to outperform conventional metrics overall on code generation.
  • RQ1: Alignment with Human Scores: LLM-as-a-judge methods outperformed conventional metrics in code translation, performed comparably in code generation, and were outperformed in code summarization.Code summarization correlations for LLM-as-a-judge methods hardly reached 30.
  • RQ1: Alignment with Human Scores: Fine-tuned evaluators improved over their base models by 5.18% to 16.03% in Pearson’s R but remained inferior overall.Auto-J and Prometheus 2 achieved R=38.92 and R=40.33 for code generation, respectively.
  • RQ1: Alignment with Human Scores: Large-LLM output-based methods performed best regardless of inference strategy, whereas embedding- and probability-based methods underperformed and were limited to models exposing internal states.Top Pearson’s R values were 81.32, 68.51, and 26.19 across translation, generation, and summarization.
  • RQ2: Score Characteristics: Methods within the same category generally correlated highly with one another, while different categories showed low correlations, indicating distinct score-distribution mechanisms.Among output-based methods, large-LLM systems also aligned strongly with each other.
  • RQ2: Score Characteristics: The best human-aligning methods produced balanced score distributions resembling human scores, whereas embedding-based methods most closely resembled conventional metrics.GPT-4o, G-Eval, and DeepSeek-V2.5 showed similar peaks and scores between 0.6 and 0.8.

6 Discussion

The discussion highlights concrete strengths and failure modes of LLM-as-a-judge methods, emphasizing task-dependent deployment and future research directions.

  • Case studies: GPT-4o correctly identifies a seeded-randomness and output-range error in the translated Python code.The original C code uses a seeded custom pseudo-random function, whereas the translation omits the seed and produces an incorrect range.
  • Case studies: GPT-4o assigns 5/5 to an overly detailed code summary that violates the instruction to summarize functionality concisely.The example illustrates that step-by-step explanations can be judged favorably despite excessive implementation detail.
  • Implications for developers: LLM-as-a-judge methods may replace human evaluators for certain SE tasks, potentially saving developers time when selecting LLMs.The authors frame this potential as task- and method-dependent rather than universal.
  • Implications for developers: Large output-based LLM judges such as GPT-4o and DeepSeek-V2.5 offer the strongest human alignment, while individual scoring is preferred over pairwise comparison.The recommendation reflects the reported performance differences across method categories and comparison settings.
  • Implications for developers: LLM-as-a-judge methods should not be used directly or alone for code summarization because their human alignment is insufficient.For code translation and generation, state-of-the-art methods can be used standalone, especially without reference answers or when explanations are needed.
  • Implications for researchers: Current methods lack generalizability across SE tasks, motivating challenging SE-specific human-preference datasets and richer evaluation context.The authors also propose incorporating insights from previous evaluation sessions into prompts to help bridge differences between LLM and human evaluation.

7 Conclusion

The paper evaluates LLM-as-a-judge methods on three SE datasets by comparing their scores with human judgments. It finds task-dependent performance, with weaker pairwise comparison results and score distributions that can differ from human patterns.

  • Conclusion: LLM-as-a-judge methods show task-dependent performance, ranging from near-human to unusable for individual-response scoring.The study covers code translation, code generation, and code summarization.
  • Conclusion: LLM-as-a-judge methods generally perform worse in pairwise comparisons than in individual scoring.The conclusion also reports that the most human-aligned methods produce more balanced score distributions resembling human scoring patterns.
Loading 2502.06193v3…