Source-linked AI summary
On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization
Giuseppe Crupi, Rosalia Tufano, Alejandro Velasco, Antonio Mastropaolo, Denys Poshyvanyk, Gabriele Bavota
TL;DR
The paper asks whether LLMs can reliably judge code generation and summarization, tasks for which standard metrics are limited and human evaluation is costly. It evaluates multiple code-capable LLMs against test results and human judgments, finding GPT-4-turbo strongest overall but still unreliable for code correctness. The study therefore supports differentiated expectations: stronger performance for summary-quality assessment than for judging code correctness.
Problem
Code generation and summarization remain difficult to evaluate because quantitative metrics are limited proxies and large-scale human assessment is expensive.
Method
The study evaluates LLM judges on code correctness using CoderEval tests and on summary quality by comparing five LLMs with judgments from nine humans across 1,163 summaries.
Results
GPT-4-turbo is the best judge for both tasks, but it frequently misjudges code correctness while showing high agreement with humans on code-summary quality.
Takeaways & Limitations
LLM judging effectiveness varies by task: GPT-4-turbo is a reliable summary-quality judge but still frequently fails to assess code correctness.
Takeaways & Limitations
Using test results as a proxy for code correctness limits construct validity, although the study performed checks to exclude clearly inadequate test suites.
Abstract
from arXiv · showhide
Large Language Models have been recently exploited as judges for complex natural language processing tasks, such as Q&A. The basic idea is to delegate to an LLM the assessment of the "quality" of the output provided by an automated technique for tasks for which: (i) quantitative metrics would only tell part of the story, and; (ii) a large-scale human-based evaluation would be too expensive. LLMs-as-a-judge, if proven effective for a specific task, can also unlock new possibilities for automation, with several LLMs proposing a solution for a given instance of the task and others judging and deciding what is the best output to show the user. We study the effectiveness of LLMs-as-a-judge for two code-related tasks, namely code generation and code summarization. The rationale for choosing these tasks is two-fold. First, quantitative metrics are usually not enough for the assessment of code summarizers/generators. For example, it is well documented that metrics such as BLEU are quite weak proxies for the quality of the generated summaries. Second, even state-of-the-art techniques still struggle with handling complex instances of these tasks, making them good candidates for benefiting from more advanced solutions envisioning collaboration among LLMs. For code generation, we check whether eight LLMs are able to judge the correctness of 1,405 Java methods and 1,281 Python functions generated by the same LLMs or implemented by humans. For code summarization, we compare the judgment of five LLMs to those provided by nine humans for ~1.2k summaries, related to both Java and Python functions. Our findings show that GPT-4-turbo is the best LLM in terms of judging capabilities for both tasks, with "smaller" LLMs featuring tens of billions parameters not being able to cope with judging tasks. However, even the best-performing LLM frequently misjudges the correctness of the code and summary quality.
1 INTRODUCTION
LLM-as-a-judge is studied as a scalable alternative for evaluating code generation and summarization, where conventional metrics and human evaluation are limited. The study finds GPT-4-turbo is the strongest judge, but it still frequently misjudges code correctness.
- Motivation: Generative software-engineering tasks are difficult to evaluate because quantitative metrics incompletely capture quality and human assessment is costly to scale.BLEU, ROUGE, and METEOR can be weak proxies for code-summary quality, while code-generation similarity metrics do not necessarily indicate correctness.
- LLM-as-a-judge: LLM-as-a-judge prompts an LLM to evaluate another technique’s output according to specified criteria.The approach is presented as a possible response to evaluation challenges in software engineering.
- Study focus: The study evaluates judges on code generation and code summarization, using generated and human-written artifacts.For generation, judgments are checked against test-suite results; for summarization, LLM judgments are compared with judgments from nine humans.
- Findings: GPT-4-turbo is the best judge for both tasks, while smaller LLMs substantially struggle or produce completely wrong judgments.The comparison includes eight LLMs for code generation and five for code summarization.
- Findings: GPT-4-turbo still frequently misjudges code correctness, classifying 50% of wrong implementations as correct, but performs better on summary-quality assessment.Its agreement with humans for summary quality is moderate.
2 STUDY DESIGN
The study asks how effectively LLMs can judge code generation and code summarization. It focuses on correctness judgments for code and quality judgments for summaries.
- Research question: The research question is: To what extent can LLMs act as a judge for code generation and summarization?The study examines two software-related generative tasks.
2.1 Context Selection: LLMs
The study compares code-capable LLM judges spanning roughly 1B to hundreds of billions of parameters. The selection reflects a trade-off between judging capability and inference cost.
- Model selection: Eight code-capable LLMs are used as judges, including DeepSeek Coder, CodeLlama, GPT-3.5-turbo, and GPT-4-turbo.The models vary substantially in size and include versions from 1.3B to 34B parameters alongside the GPT variants.
- Selection criteria: The models are selected because they were trained on corpora including source code and must judge both generated code and code summaries.The study also varies model size because larger models increase inference cost.
- Model selection: DeepSeek Coder models used in the study have 1.3B, 6.7B, and 33B parameters.They were trained from scratch on 2 trillion tokens, 87% of which were code.
- Model selection: CodeLlama models used in the study have 7B, 13B, and 34B parameters.CodeLlama is described as a code-focused family built on Llama2.
2.2 Context Selection: Evaluation Datasets
The evaluation uses CoderEval for code correctness and a newly constructed human-judgment dataset for code-summary quality. Both datasets provide reference signals for assessing LLM judgments.
- Code generation: CoderEval: CoderEval contains 460 code-generation problems, split evenly between Java and Python, each with requirements, a target function, and tests.The tests allow researchers to compare an LLM’s correctness judgment with whether the candidate implementation passes.
- Code generation: CoderEval: The study performs quality assurance on CoderEval by excluding problems whose target functions fail tests or whose empty implementations pass.These checks aim to reduce bias from unreliable test suites.
- Code summarization: The researchers rejected an existing summarization dataset because human agreement was low and its automatically generated summaries came from relatively old techniques.The inspected dataset contained 6,253 evaluations of 2,292 summaries by 226 developers, with content-adequacy Krippendorff’s α around 0.2.
- Code summarization: The replacement dataset contains human judgments for 1,163 summaries drawn from the 100 longest Java and 100 longest Python functions in CoderEval.The researchers focus on long functions because summaries may contribute more to comprehension for these cases.
- Code summarization: Nine judges assessed each summary three times, producing 3,489 judgments from participants with substantial Java and Python experience.The judges rated conciseness, fluency and understandability, and content adequacy using defined scales.
- Code summarization: Human agreement was substantial for Java content adequacy and ranged from moderate to substantial across the reported quality attributes.Reported Krippendorff’s α values include 0.81 for Java content adequacy and 0.69 for Python content adequacy.
2.3 Data Collection
The study collected code-generation and code-summarization judgments using multiple prompt variants, applying them to benchmark implementations and summaries produced by humans and LLMs.
- Code generation: 374 CoderEval code-generation problems supplied target implementations for initial LLM judgments.
- Code generation: Four code-generation judging prompts included zero shot, zero shot without rationale, automated CoT, and slow thinking variants.
- Code generation: The code-generation prompt presented each model with a task description, function signature, and candidate implementation for correctness assessment.
- Code generation: LLMs judged 1,221 Java and 1,091 Python generated implementations, including their own outputs, to investigate judging bias.
- Code generation: The resulting dataset contained 80,556 judgments covering human-written and automatically generated functions.
- Code summarization: For summarization, prompts scored content adequacy, conciseness, and fluency and understandability on 1-to-5 scales.
- Code summarization: Summarization prompts also varied by adding human judging instructions and automated chain-of-thought reasoning before answer extraction.
2.4 Data Analysis
The analysis compares LLM judgments with execution-based correctness or human summary assessments, while testing bias and examining misjudgment causes.
- Code generation: Code-generation correctness was evaluated with confusion matrices distinguishing true positives, true negatives, false negatives, and false positives.
- Code generation: Cohen’s Kappa measured agreement between LLM judgments and test-execution classifications of implementations as correct or wrong.
- Bias analysis: The study tested whether LLMs assigned higher scores to their own code than to code from other LLMs or humans using Mann–Whitney tests.
- Error analysis: Manual analysis examined 15 false positives and 15 false negatives per LLM to characterize misjudgment reasons.
- Code summarization: Summary judgments were compared with human assessments across content adequacy, conciseness, and fluency and understandability using scatterplots.
- Code summarization: Krippendorff’s α quantified agreement for each summary-quality attribute, using the median of three human judgments as the oracle.
- Bias analysis: The analysis also tested whether summarization LLMs favored summaries they generated, using a statistical procedure mirroring code-generation analysis.
- Limitations: A systematic qualitative analysis of summarization misjudgments was not performed because three criteria and five-point scores blurred false-positive and false-negative definitions.
3 RESULTS DISCUSSION
GPT-4-turbo was the strongest judge overall, but its code-correctness assessments remained unreliable, especially for wrong implementations. The analyses also reveal systematic misjudgment causes and instability under semantics-preserving transformations.
- Code generation judging performance: Smaller LLMs frequently failed to produce valid judgments, whereas GPT-4-turbo was valid on all Java functions and failed on only 7 Python functions.Models with a few billion parameters tended to fail in approximately 15% of cases.
- Code generation judging performance: GPT-4 achieved the highest overall agreement with test outcomes, but Cohen’s Kappa was only 0.21 for Java.Prompt variations produced only minor changes in Kappa scores.
- Code generation judging performance: GPT-4 correctly classified 72% of correct Java implementations but misjudged 50% of wrong ones, indicating a tendency to judge implementations positively.For Python, it correctly classified 46% of correct implementations and misjudged 35% of wrong ones.
- Self-bias analysis: LLMs generally overestimated LLM-generated code relative to human-written code, while GPT-4 showed a slight tendency to overestimate its own code more than others’ code.Human-written code had an average bias coefficient of -0.37, whereas LLM-generated code coefficients were positive.
- False positives and negatives: False positives most often reflected uncaught wrong behavior, while false negatives were commonly caused by hallucinated defects and misunderstood code statements.Artificial hallucination accounted for 33% of false negatives, and misunderstanding code statements accounted for 19%.
- False positives and negatives: GPT-4 changed its correctness judgment for semantically equivalent code in 21% of Java cases and 36% of Python cases.It classified only 79% of Java and 64% of Python semantically equivalent instances as correct.
4 THREATS TO VALIDITY
The study identifies threats to construct, internal, and external validity, while describing mitigations for testing, human assessment, evaluators, and prompt variation.
- Construct validity: Using tests as a proxy for code correctness limits construct validity, despite checks excluding problems with clearly inadequate test suites.
- Internal validity: Human-judgment subjectivity threatens internal validity, but multiple evaluators were involved in every manual step.
- Internal validity: Experimenting with four prompts showed that prompting affected results but did not materially change the overall findings or lessons learned.
- External validity: Generalizability is limited to two code-related tasks and the Java and Python languages, motivating differentiated replications.
5 RELATED WORK
Related work applies LLMs-as-judges to software evaluation, but this study evaluates judging effectiveness directly using tests and human assessments across larger, different task settings.
- Prior NLP work identifies positional bias, verbosity, and self-enhancement among limitations of LLM-as-a-judge systems.
- Fine-tuning LLMs for judging can improve performance, but may reduce generalizability and leave fairness and scalability limitations.
- Unlike related software studies, this work evaluates LLM judging effectiveness rather than merely using an LLM judge within another methodology.
- Unlike feedback evaluation on 57 beginner programming assignments, this study uses tests and human assessments as oracles across two tasks and substantially larger datasets.
- ICE-Score uses GPT-3.5 to judge usefulness and functional correctness across 164 problems in four languages.
- CodeJudge adds a slow-thinking prompt to GPT-3.5 judging, but this study reports no major improvement in correctness assessment from adopting that prompt.
6 CONCLUSIONS AND FUTURE WORK
The study evaluates LLM-as-a-judge for code correctness and summary quality across models of different sizes, using tests and human judgments. It concludes that effectiveness varies by task and that further work should expand models and tasks.
- The study covers code generation and summarization, evaluating correctness with test results and summary quality against correlated LLM and human judgments.
- Models range from approximately 1B parameters to hundreds of billions, enabling comparison across substantially different LLM sizes.
- Future work will investigate small LLMs fine-tuned for code-judging tasks and extend evaluation to additional tasks such as bug fixing.