Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering

1 Introduction

• Pass@k requires curated tests and configured environments and misses non-functional qualities, while conventional metrics require references and emphasize lexical over semantic similarity.Existing meta-evaluation benchmarks also feature few, simple coding tasks, leaving a gap with more complex and varied real-world SE scenarios.Source: §1 · p. 2
• The study evaluates code translation, code generation, and code summarization using 450 manually scored responses generated from 150 sampled instructions.It samples 50 instructions from each dataset and uses three of 12 code LLMs to generate responses; the authors describe this as the first empirical study of LLM-as-a-judge specifically for SE tasks.Source: §1 · pp. 2–3
• The study asks whether judge methods outperform conventional metrics in human alignment, how their scores correlate and distribute, and how output-based pairwise judgments compare with individual scoring.It examines embedding-based, probability-based, and output-based methods, plus LLMs fine-tuned for NLP evaluation and their base model.Source: §1 · pp. 2–3
• Human alignment varied by SE task: large-LLM output-based methods achieved near-human performance in code translation and generation, while pairwise judgments were often inaccurate or inconsistent.The best human-aligning methods also had more balanced, human-like score distributions; pairwise results could change when response positions were swapped.Source: §1 · p. 3

2.1 Code LLMs for SE

• Code LLMs range from encoder and encoder-decoder models to larger decoder-only generators, with some supporting AST inputs or fill-in-the-middle generation.Examples include CodeBERT, CodeT5, and UniXcoder, alongside Codex, CodeLlama, and DeepSeek-Coder families.Source: §2.1 · p. 4

2.2 SE Benchmarks and Metrics

• Code generation dominates SE benchmarks, while other benchmark suites cover review, summarization, search, testing, and code understanding.HumanEval, MBPP, APPS, and ClassEval use Pass@k; CodeReviewer, CodeXGLUE, and ComplexCodeEval use conventional metrics, while CRUXEval uses Pass@k. Few datasets provide varied-quality responses with human scores; NoFunEval and CodeUltraFeedback address meta-evaluation aspects.Source: §2.2 · p. 4

2.3 LLM-as-a-Judge in NLP

• Embedding-based evaluators compare contextual response and reference representations, but still measure resemblance to the reference.BERTScore uses token-level Recall, Precision, and F1 with IDF weighting; MoverScore uses Word Mover’s Distance, and CodeBERTScore adapts BERTScore for code.Source: §2.3.1 · p. 5

2.3.2 Probability-Based Methods.

• Probability-based evaluators score generation likelihood, with FFLM comparing summary-generation probabilities with and without the source text.BARTScore uses BART’s probability of generating a response, while GPTScore supports reference-free and reference-based evaluation across multiple aspects.Source: §2.3.2 · p. 5

2.3.3 Output-Based Methods.

• Output-based judges elicit judgments without internal access, while InstructScore, PandaLM, and X-Eval use fine-tuning; their reliability on challenging SE samples remains unclear.G-Eval requests evaluation steps and averages sampled scores, whereas ChatEval uses persona-based agents to discuss and choose between responses.Source: §2.3.3 · p. 5

• Embedding-based methods represent response and reference tokens with an encoder, then score the response by fusing token-wise cosine similarities.Source: §3 · p. 5
• Probability-based methods score a response using conditional log-probabilities for input-output pairs such as source-response or reference-response, optionally adding evaluation aspects.Source: §3 · p. 6
• Output-based methods prompt an LLM with the source and response, optionally including a reference, criteria, or evaluation steps, then extract a score from its judgment.Multi-agent prompting and repeated sampling can combine scores by majority vote or averaging.Source: §3 · p. 6
• Specialized fine-tuned judges usually use one inference pass without additional strategies, and may train on reference judgments generated by models such as GPT-4.The paper focuses on their performance rather than the detailed training process.Source: §3 · p. 6
• Embedding-based methods typically score on [0, 1], probability-based methods on (−∞, 0) or (−∞, ∞), and output-based methods often use 1–5 or 1–10 scales or pairwise comparisons.The study examines individual scoring in RQ1 and RQ2 and pairwise comparison in RQ3.Source: §3 · p. 6

4 Study Design

• The researchers collect task instructions, generate responses from representative datasets, evaluate them with humans and LLMs, and analyze score correlations.Source: §4 · p. 6

4.1.1 Instruction Collection.

• Code translation uses CodeTransOcean’s MultilingualTrans subset, with longer code and broader language coverage than the comparison benchmark described.The passage reports 7,545 samples across eight languages and an average test-set length of 491 tokens, versus 58 tokens in CodeTrans featured in CodeXGLUE, which supports Java and C#; evaluation uses conventional metrics rather than execution-based Pass@k.Source: §4.1.1 · p. 7
• Code summarization uses CodeXGLUE’s filtered CodeSearchNet3 code-text subset, retaining samples across six programming languages.The filtering removes syntactically incorrect code and docstrings that are non-English, too long, or too short; 14,918 Python and 10,955 Java samples remain, alongside four other languages.Source: §4.1.1 · p. 7
• Code generation uses ComplexCodeEval, which supplies project-specific dependency information alongside Java and Python examples.The benchmark has 3,897 Java and 7,184 Python samples and includes functional dependencies, timestamps, and unit tests; its conventional metrics include Edit Similarity, BLEU, and CodeBLEU.Source: §4.1.1 · p. 7
• The study samples 50 instructions per task dataset and limits evaluation to test sets and languages evaluators know.It uses Java, Python, C, and C++, applies token limits to instructions, reference answers, and output-based judgments, and removes overly long samples.Source: §4.1.1 · pp. 7–8
• For each instruction, three randomly selected code LLMs generate responses, and all ordered response pairs are evaluated for order consistency.Across 150 instructions, this yields 450 responses and 900 pairs; LLMs receive task context and may generate up to 3,072 tokens.Source: §4.1.2 · p. 8

4.1.2 Response Generation.

• The authors revise summarization and generation instructions after finding incorrect reference summaries and insufficient dependency context.They require summaries of at least 15 tokens, manually examine reselected instructions, limit generation examples to at most five dependencies, and augment dependency names with GPT-4o-generated signatures and descriptions that are manually checked.Source: §4.1.2 · p. 8

4.1.3 Manual Evaluation.

• Human evaluators score each response on instruction alignment and intrinsic quality using task-specific criteria for integer scores from 1 to 5.The rubric treats 5 as near-perfect, 4 or 3 as responses with minor or major issues that still make sense, and 2 or 1 as practically useless; the passage gives a readability rubric example.Source: §4.1.3 · p. 9
• Two expert evaluators score both aspects before assigning an overall score, and the response’s final human score is their overall-score average.For pairwise comparisons, differences below 0.5 count as ties; otherwise the response with the higher final score is preferred.Source: §4.1.3 · p. 9

4.2.1 Conventional Metrics.

• Four conventional reference-based metrics—BLEU, ROUGE-L, METEOR, and ChrF++—measure overlap using n-gram, subsequence, token-match, or character/token F-scores.BLEU uses modified n-gram precision with a brevity penalty; ROUGE-L uses longest-common-subsequence F1; METEOR uses token-match F3 and penalizes fragmented alignment. ChrF++ averages character F2 scores up to 6-grams and token F2 scores up to 2-grams; all five conventional metrics require the response and reference, not the instruction.Source: §4.2.1 · p. 9
• CrystalBLEU removes common corpus n-grams before calculating BLEU, using each task’s test set—including instructions and references—as its corpus.The method is designed to measure code similarity and removes frequent n-grams to reduce the masking of meaningful differences.Source: §4.2.1 · p. 10

4.2.2 Embedding-Based Methods.

• Embedding-based methods use UniXcoder representations: BERTScore [58] combines IDF-weighted token-similarity precision and recall, while MoverScore [59] scores minimum-cost flow between n-gram representations.UniXcoder is used instead of BERT or other non-code LLMs because it processes both code and text.Source: §4.2.2 · p. 10

4.2.3 Probability-Based Methods.

• Probability-based methods use davinci-002 and may condition on the instruction, response, reference, or evaluation aspects; GPTScore [11] and FFLM [20] derive scores from token probabilities.For GPTScore, the authors include the instruction in both reference-conditioned probability terms used by the code’s harmonic-mean implementation. FFLM fuses prior, instruction-conditioned posterior, and prefix probabilities, weighting low-probability tokens more heavily.Source: §4.2.3 · p. 10

4.2.4 Output-Based Methods.

• The SFT group tests Auto-J [26] and Prometheus-v2-BGB-8x7B [22] against their untuned base models for SE-evaluation alignment.They use each judge’s default prompt and greedy decoding; evaluation aspects are omitted for Auto-J and Llama2-13B-Chat because Auto-J’s template has no place for aspects.Source: §4.2.4 · pp. 10–11
• Output-based methods prompt for human-evaluation aspects and compare Vanilla scoring with G-Eval [31] and BatchEval [56] inference strategies.Vanilla uses greedy decoding; G-Eval generates evaluation steps and averages scores over 20 high-temperature passes; BatchEval diversifies scores across batches over five rounds, with batch size 10 and temperature 0.2. The aspects match those used in human evaluation.Source: §4.2.4 · pp. 10–11

4.3 Meta-Evaluation

• Individual-scoring metrics are meta-evaluated by their Spearman’s 𝜌, Pearson correlation coefficient 𝑅, and Kendall’s 𝜏 with human scores.This is the default method.Source: §4.3 · p. 11
• For RQ3 pairwise comparison, LLM-generated label Accuracy and Agreement under swapped response positions are computed.Agreement checks whether the LLM makes the same judgment when the two responses swap positions in the prompt.Source: §4.3 · p. 11
• All correlation coefficients, Accuracies, and Agreements are multiplied by 100, and RQ1 correlations are checked for p-values below 0.05.The p-value check is used to ensure a 95% confidence interval.Source: §4.3 · p. 11

5.1 RQ1: Alignment with Human Scores

• In code translation, BatchEval reaches R=81.32 versus conventional metrics’ R=34.23; in code generation, DeepSeek-V2.5 reaches R=68.51 versus conventional metrics’ R=65.55.In code summarization, LLM-as-a-judge methods hardly reach 30 on any correlation coefficient, while conventional metrics remain below 50 for ρ and R and below 40 for τ. The paper attributes these task differences partly to lexical overlap in code generation and the difficulty of detecting step-by-step explanations or paraphrases in summaries.Source: §5.1 · pp. 11–13, Table 3 · p. 12
• Large state-of-the-art LLMs with output-based judging perform best, while inference strategies bring at most ΔR=2.21, 6.03, and 3.04 for GPT-4o across the three tasks.Embedding- and probability-based methods require access to internal states and have lower alignment; fine-tuned Auto-J and Prometheus 2 improve over their base models but remain behind, with code-generation R=38.92 and R=40.33, respectively. G-Eval and BatchEval also raise inference cost, so greedy decoding remains viable with large models.Source: §5.1 · pp. 13–14

5.2 RQ2: Score Characteristics

• For non-SFT methods, correlations are usually higher within the same category (ρ_inner>60) than across categories (ρ_other<50); large output-based models also correlate with each other at ρ>80 for translation and ρ>65 for generation.Methods using small LLMs have ρ<50 against the large-model group and ρ<30 within the small group; fine-tuned methods correlate poorly even within their category.Source: §5.2 · pp. 14–15
• The best human-aligned output-based methods show balanced score distributions resembling human scores, whereas only embedding-based methods closely resemble conventional metrics.Embedding-based methods’ ρ_conv values are 81.45, 79.78, and 57.07 across the three tasks; MoverScore and ChrF++ both tend toward lower scores, while GPT-4o, G-Eval, and DeepSeek-V2.5 have peaks similar to human scores between 0.6 and 0.8.Source: §5.2 · pp. 15–16, Figure 2 · p. 15

5.3 RQ3: Pairwise Comparison versus Individual Scoring

• For pairwise comparison, code-translation accuracy peaks at 65.33 for BatchEval and 64.67 for G-Eval, but leading translation and generation methods have Agreement below 25.Even the best code-generation methods struggle to reach 50 Accuracy, summarization comparisons are unusable, and the most consistent fine-tuned methods barely beat random guessing. BatchEval adds up to +8 Accuracy for GPT-4o in code translation, but inference strategies do not resolve inconsistency.Source: §5.3 · pp. 16–17, Table 6 · p. 17

6.1 Case Study

• GPT-4o scored a seeded-random translation error 3/5 while missing a range discrepancy, but gave an over-detailed summary 5/5, illustrating verbosity bias [40].Both cases used GPT-4o from the Vanilla output-based group; Fig. 3 presents the translation case on the left and summarization case on the right.Source: §6.1 · p. 17, Figure 3 · p. 18

6.2 Implications of Findings

• LLM judges may save developers time on selected SE tasks, but method performance varies; output-based large models with suitable prompts and inference, and individual scoring, are preferred.The paper identifies GPT-4o and DeepSeek-V2.5 as examples of large models for output-based evaluation.Source: §6.2 · p. 18
• Code translation and generation judges can be used standalone, especially without references or when explanations are needed; summarization judges should instead complement conventional metrics with bias-mitigating prompts.The authors caution against using LLM judges directly or alone for code summarization because their human alignment is insufficient.Source: §6.2 · p. 18
• Current methods generalize poorly across SE tasks; researchers could curate challenging SE-specific preference data for fine-tuning and develop SE- or task-specific evaluators.The paper attributes limitations to a lack of challenging SE-specific training data for SFT methods and uniform prompts and inference strategies in non-SFT methods.Source: §6.2 · p. 19
• Researchers could narrow gaps with human evaluators by giving LLMs prior evaluation insights and multiple responses, using multi-agent judges, and clarifying task objectives to mitigate verbosity bias.For code summaries, the paper notes a mismatch between CodeXGLUE’s concise docstring-style expectation and LLMs’ tendency toward detailed explanations.Source: §6.2 · p. 19

7 Conclusion

• Across three SE datasets, the study generates and manually scores LLM responses to assess LLM-as-a-judge alignment with human scores.Source: §7 · p. 19
• LLM-as-a-judge performance ranges from near-human to unusable for individual responses and is generally worse in pairwise comparisons.Source: §7 · p. 19
• The methods that align most closely with human scores show balanced, human-like score distributions.Source: §7 · p. 19