PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?

Yuwei Zhao, Ziyang Luo, Yuchen Tian, Hongzhan Lin, Weixiang Yan, Annan Li, Jing Ma

arXiv:2408.10718v2cs.SEcs.CL

TL;DR

Language-to-code benchmarks may not fully measure LLM code understanding because passing tests can leave coverage and memorization concerns. CJ-Eval evaluates models as judges of diverse candidate solutions using fine-grained verdicts, and 12-model results show that even leading systems struggle, with performance reaching only macro F1 50 on the simplest tasks.

  • Problem

    Language-to-code benchmarks may not fully capture code understanding because test cases can miss edge cases and evaluations can face data-leakage concerns.

    Source: §1 · p. 1

  • Method

    CJ-Eval uses diverse coding problems, multiple candidate solutions, and fine-grained execution-based verdicts to assess whether LLMs can judge code correctness and error types.

    Source: §1 · p. 2, §3.2.3 · p. 4, §3.2.4 · p. 5

  • Results

    12 LLMs were evaluated, and GPT-4o and Claude-3.5-Sonnet reached a macro F1 score of 50 on the simplest code-judge tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Code generation ability does not necessarily imply accurate judgment of other solutions, so code judging provides a distinct perspective on LLM code understanding.

    Source: §2 · p. 3, §5 · p. 8

    CJ-Eval does not evaluate code generation and is intended to complement rather than replace language-to-code benchmarks.

    Source: Limitation · p. 9

Abstract

from arXiv · show

Recent advancements in large language models (LLMs) have showcased impressive code generation capabilities, primarily evaluated through language-to-code benchmarks. However, these benchmarks may not fully capture a model's code understanding abilities. We introduce CodeJudge-Eval (CJ-Eval), a novel benchmark designed to assess LLMs' code understanding abilities from the perspective of code judging rather than code generation. CJ-Eval challenges models to determine the correctness of provided code solutions, encompassing various error types and compilation issues. By leveraging a diverse set of problems and a fine-grained judging system, CJ-Eval addresses the limitations of traditional benchmarks, including the potential memorization of solutions. Evaluation of 12 well-known LLMs on CJ-Eval reveals that even state-of-the-art models struggle, highlighting the benchmark's ability to probe deeper into models' code understanding abilities. Our codes and benchmark are available at \url{https://github.com/CodeLLM-Research/CodeJudge-Eval}.

1 Introduction

Existing code-generation benchmarks assess whether models produce test-passing programs, but may not establish full code understanding. CJ-Eval instead evaluates whether LLMs can judge candidate solutions, using diverse problems and fine-grained verdicts; results show this remains difficult even for leading models.

  • CJ-Eval addresses limitations of code-generation benchmarks by evaluating whether LLMs can judge provided solutions rather than only generate code.The benchmark targets correctness and error judgments without relying on external unit-testing tools.Source: §1 · pp. 1–2
  • Its multiple candidate solutions make memorizing every answer harder than memorizing one correct solution per problem, though data leakage and memorization remain concerns.The motivation includes incomplete test coverage and potential data leakage in conventional evaluations.Source: §1 · pp. 1–2
  • CJ-Eval uses APPS problems, 16 LLMs for candidate generation, fine-grained execution-based annotations, and multiple-choice questions.The original 80,000 solutions were filtered to 1,860 solution codes.Source: §1 · p. 2
  • 12 LLMs were evaluated, with GPT-4o and Claude-3.5-Sonnet reaching a macro F1 score of 50 on the simplest code-judge tasks.The benchmark remains challenging across proprietary and open-source models.Source: §1 · p. 2

2 Related Work

Prior coding benchmarks predominantly evaluate code generation and often depend on test cases that may be vulnerable to data leakage. CJ-Eval shifts the focus to code correctness judgment across solutions, including outputs produced by other models.

  • Existing benchmarks cover code generation and related tasks, but generally evaluate code from requirements using test cases to determine correctness.Examples include HumanEval, MBPP, APPS, CodeContests, and TACO.Source: §2 · pp. 2–3
  • CJ-Eval evaluates code understanding through LLMs acting as judges, without depending on external unit testing and with greater difficulty for memorizing all candidate solutions.This distinguishes it from language-to-code evaluation while complementing existing benchmarks.Source: §2 · p. 3
  • Unlike related efforts such as ICE-Score and work on models’ incorrect generations, CJ-Eval judges code correctness for solutions generated by both the evaluating model and other LLMs.Its focus is specifically correctness judgment rather than general usefulness or self-evaluation.Source: §2 · p. 3

3 CodeJudge-Eval

CJ-Eval constructs a code-judging benchmark from diverse programming problems and generated solutions, using fine-grained execution verdicts as labels. The dataset is filtered for test-case adequacy and verdict diversity before evaluating models across progressively simplified judging settings.

3.1 Overview

  • CJ-Eval represents each problem with a statement, test cases, multiple generated solution codes, execution verdicts, and one final label per code.The verdict list records each solution’s result across all test cases, while the label is used for final evaluation.Source: §3.1 · p. 3

3.2.1 Data Source

  • The benchmark uses 5,000 APPS test problems spanning introductory, interview, and competition difficulty levels, with hidden test cases included.APPS was selected for its challenging problems and abundance of test cases.Source: §3.2.1 · p. 4

3.2.2 Code Generation

  • Sixteen diverse proprietary and open-source general-purpose or code-focused LLMs generate candidate Python solutions for the benchmark.The model pool includes both closed- and open-source systems across generalist and code-specialized categories.Source: §3.2.2 · p. 4

3.2.3 Fine-grained Verdict Construction

  • The local judging system executes every solution separately on every test case and identifies specific errors rather than only overall correctness.It distinguishes compilation, runtime, time-limit, wrong-answer, and accepted outcomes.Source: §3.2.3 · pp. 4–5

3.2.4 Design of Labels

  • The hard setting uses nine labels for possible error combinations, while middle and easy settings merge these into six and three labels, respectively.The hard setting requires analysis of all possible error types; the easier settings progressively group error categories.Source: §3.2.4 · p. 5

3.3.1 Filter Problems by Test Case

  • Filtering excludes problems with no more than 20, 80, or 40 test cases at introductory, interview, or competition levels, leaving 457 problems.The retained set contains 133 introductory, 178 interview, and 146 competition problems.Source: §3.3.1 · p. 5, Figure 3 · p. 5

3.3.2 Filter Solution Codes by Verdict

  • The benchmark retains one randomly selected solution per verdict for each problem, reducing redundancy while preserving verdict categories.This process produces 1,860 solution codes, averaging 4.1 codes per problem.Source: §3.3.2 · p. 5
  • Zero-shot evaluation across four method types and three difficulty levels finds that all LLMs perform suboptimally, underscoring CJ-Eval’s difficulty.The benchmark reports both accuracy and macro F1 scores.Source: Table 2 · p. 6

4 Experiment

The experiments evaluate diverse models and metrics on CJ-Eval, showing that code judging remains difficult and that generation ability does not reliably transfer to judgment. Analyses further examine scaling, prompting, task difficulty, and the relationship between generation and evaluation.

4.1.2 Metrics

  • Macro F1 is recommended as the primary metric because severe class imbalance can make accuracy misleading, especially in the easy setting.Source: §4.1.2 · p. 7

4.2 Zero-Shot Evaluation

  • All LLMs perform suboptimally in zero-shot evaluation, underscoring CJ-Eval’s difficulty.Source: §4.2 · p. 7
  • Simple strategies can obtain high accuracy under class imbalance, but none significantly outperform Random on F1.Source: §4.2 · p. 7
  • Proprietary models achieve the strongest overall performance, while most open-source models score below the Random strategy on macro F1.Source: §4.2 · p. 7

4.3 Analysis

  • Scaling improves Qwen2 judging performance, with Qwen2-72B comparable to GPT-4o, whereas Llama-3.1 scaling from 70B to 405B yields only marginal gains.Source: §4.3 · p. 7, Figure 4 · p. 7
  • One-shot examples improve all models, including +14.36 for GPT-4o, +4.85 for Claude-3.5-Sonnet, and +16.94 for GPT-3.5-Turbo, but additional shots often reduce performance.Source: §4.3 · pp. 7–8, Figure 5 · p. 7
  • One-shot Chain-of-Thought helps substantially in the middle setting but can reduce performance in the easy setting, where detailed analysis may be unnecessary.Source: §4.3 · p. 8, Figure 6 · p. 8
  • Easier coding tasks do not necessarily produce higher judging macro F1, and correct code generation does not guarantee accurate evaluation of other solutions.Source: §4.3 · p. 8, Figure 7 · p. 8, Figure 8 · p. 8

5 Conclusion

CodeJudge-Eval evaluates LLM code understanding by having models judge candidate solutions rather than generate code. Across 12 proprietary and open-source models, the benchmark is difficult, open-source models often perform worse than random guessing, and generation success does not necessarily imply accurate judgment.

  • CodeJudge-Eval evaluates LLM code understanding by measuring performance as code judges.Source: §5 · p. 8
  • The benchmark is difficult: open-source models often perform worse than random guessing.Source: §5 · p. 8
  • Generating correct code does not necessarily imply accurately evaluating other solutions for the same task.Source: §5 · p. 8

Limitation

CJ-Eval has three main limitations: it complements rather than replaces code-generation benchmarks, does not fully eliminate memorization risk, and is narrowly tailored to coding.

Limitation

  • CJ-Eval is not intended to replace language-to-code benchmarks because it evaluates code understanding rather than code generation.The authors suggest integrating both evaluation types for a more comprehensive assessment.Source: Limitation · p. 9
  • Using multiple candidate solutions makes memorization harder but does not fully prevent intentional training to memorize all candidates.Source: Limitation · p. 9
  • CJ-Eval’s code-judging design limits its applicability beyond coding domains.The authors note that adapting the methodology to non-coding contexts is challenging.Source: Limitation · p. 9

A Algorithm for Filtering by Verdict

The verdict-filtering algorithm keeps at most one solution code per verdict for each problem while balancing source-model representation, then removes discarded records consistently.

A Algorithm for Filtering by Verdict

B Statistics across Different Settings

Statistics differ across difficulty settings: the hard setting is mostly balanced across sources and choices, while the easy setting has severe class imbalance that motivates macro F1.

B Statistics across Different Settings

  • The hard setting is relatively uniform across source models and choices, except that most choice-A codes come from GPT-4 because it produced most accepted generated codes.Source: B Statistics across Different Settings · p. 13
  • The easy setting has severe class imbalance, whereas the middle setting remains relatively balanced, motivating macro F1 for evaluation.In the easy setting, choice C aggregates hard-setting choices C through I; in the middle setting, choice F aggregates E, G, H, and I.Source: B Statistics across Different Settings · p. 13

D Discussion on Failure Cases

Failure cases are generally uncommon but concentrated in some open-source models, and they include refusals, attempting code generation instead of judging, and format violations.

D Discussion on Failure Cases

E Full Results

The full results compare prompting strategies on CJ-Eval, while the prompt design varies examples, chain-of-thought guidance, and difficulty-specific answer choices. Claude-3.5 performs best in middle and hard settings with one-shot CoT, whereas GPT-4o and Claude-3.5 lead easy-setting accuracy and Macro F1, respectively.

E Full Results

  • Claude-3.5 achieves the best performance in both middle and hard settings with a 1-shot CoT example, significantly outperforming other methods.Source: E Full Results · p. 14
  • For the easy setting, GPT-4o has the best accuracy, while Claude-3.5 has the best Macro F1 under the 1-shot example.Source: E Full Results · p. 14

F Prompt

  • All evaluated prompting variants use one template, differing in the content inserted into placeholders for choices, examples, and the judged problem and code.Source: F Prompt · p. 15
  • The answer choices distinguish AC and CE from non-AC outcomes, then specify individual error types or combinations involving WA, RE, and TLE, including all three errors.Source: F Prompt · p. 15
  • Zero-shot prompts omit examples, one-shot prompts add one example, one-shot CoT adds detailed reasoning, and few-shot prompts add two or three examples.Source: F Prompt · pp. 15–16

G Analysis for Chain-of-Thought

The Chain-of-Thought analysis examines Claude-3.5-Sonnet on a one-shot CoT judging task using the paper’s prompt template and illustrative examples. The model identifies multiple possible code errors, distinguishes their verdict types, and reaches the correct answer.

G Analysis for Chain-of-Thought

Related papers

Loading 2408.10718v2…