PaperlayerView exact revision on arXiv

Source-linked AI summary

Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM

Chunqiu Steven Xia, Yinlin Deng, Lingming Zhang

arXiv:2403.19114v1cs.SEcs.CLcs.LGcs.PL

TL;DR

Existing code-synthesis benchmarks have test inadequacy and other limitations, raising questions about their evaluation of LLM program-synthesis ability. EvoEval evolves existing problems into targeted benchmark domains; across 51 LLMs, performance drops by an average of 39.4% compared with standard benchmarks.

  • Problem

    Existing code-synthesis benchmarks have test inadequacy and other limitations, leaving their evaluation of LLM program-synthesis ability in question.

    Source: §1 · p. 2

  • Method

    EvoEval uses targeted transformation prompts to evolve existing coding problems into new domains, with a pipeline that refines problems and generates groundtruth solutions and tests.

    Source: §1 · p. 3

  • Results

    39.4% average performance drop on EvoEval compared with standard benchmarks across a study of 51 LLMs.

    Source: §7 · p. 14

  • Takeaways & Limitations

    The findings indicate potential overfitting to existing benchmarks and reveal difficulty composing known coding concepts into more complex problems.

    Source: §5.2 · p. 12, §7 · p. 14

    Existing LLM training and inference paradigms still struggle with problem composition, motivating research on training methods for this limitation.

    Source: §5.2 · p. 12

Abstract

from arXiv · show

LLMs have become the go-to choice for code generation tasks, with an exponential increase in the training, development, and usage of LLMs specifically for code generation. To evaluate the ability of LLMs on code, both academic and industry practitioners rely on popular handcrafted benchmarks. However, prior benchmarks contain only a very limited set of problems, both in quantity and variety. Further, due to popularity and age, many benchmarks are prone to data leakage where example solutions can be readily found on the web and thus potentially in training data. Such limitations inevitably lead us to inquire: Is the leaderboard performance on existing benchmarks reliable and comprehensive enough to measure the program synthesis ability of LLMs? To address this, we introduce EvoEval -- a program synthesis benchmark suite created by evolving existing benchmarks into different targeted domains for a comprehensive evaluation of LLM coding abilities. Our study on 51 LLMs shows that compared to the high performance obtained on standard benchmarks like HumanEval, there is a significant drop in performance (on average 39.4%) when using EvoEval. Additionally, the decrease in performance can range from 19.6% to 47.7%, leading to drastic ranking changes amongst LLMs and showing potential overfitting of existing benchmarks. Furthermore, we showcase various insights, including the brittleness of instruction-following models when encountering rewording or subtle changes as well as the importance of learning problem composition and decomposition. EvoEval not only provides comprehensive benchmarks, but can be used to further evolve arbitrary problems to keep up with advances and the ever-changing landscape of LLMs for code. We have open-sourced our benchmarks, tools, and complete LLM generations at https://github.com/evo-eval/evoeval

1 Introduction

Existing code-synthesis benchmarks offer limited problem quantity and variety, and their leaderboard performance may not reliably measure LLM program-synthesis ability. EvoEval evolves existing problems into targeted domains and evaluates 51 LLMs, finding substantial performance drops and model weaknesses on changed or composed problems.

1 Introduction

  • Existing benchmarks contain limited problems and variety, while their age and available solutions raise concerns about data leakage and whether high scores reflect learned coding ability.HumanEval contains 164 handcrafted problems, and benchmark solutions substantially overlap with open-source training corpora [39].Source: §1 · p. 2
  • EvoEval uses targeted LLM prompts to transform existing coding problems into different domains, including difficult, creative, subtle, combined, and tool-use variants.GPT-4 generates transformed problems; the pipeline also generates groundtruth solutions and test cases, followed by manual checks for clarity and correctness.Source: §1 · p. 3
  • 39.4% average performance drop on EvoEval versus standard benchmarks, with decreases ranging from 19.6% to 47.7% across LLMs.The study covers 51 LLMs and reports drastic ranking changes among top-performing models.Source: §1 · pp. 3–4
  • EvoEval comprises 828 problems across seven benchmarks, built by evolving HumanEval problems and including benchmarks for composition and decomposition.The suite includes groundtruth implementations and robust test cases for functional evaluation.Source: §1 · p. 3
  • Instruction-following LLMs show sensitivity to rewording and subtle changes, struggle with provided helper functions, and fail to effectively compose coding concepts into more complex variants.The study also reports difficulty addressing subproblems decomposed from previously solved difficult problems.Source: §1 · p. 4

2 Approach

EvoEval evolves seed problems with targeted transformations, then refines the generated problems and supplies solutions and tests. Manual examination and test augmentation complete the benchmark pipeline.

2 Approach

  • The pipeline applies a selected transformation prompt to an original problem, refines the result, generates test cases and a groundtruth solution, and manually checks the final problem and groundtruth.Manual examination can include corresponding changes to ensure correctness.Source: §2 · pp. 4–5
  • Transformations may alter problem semantics or preserve them while changing the description, enabling evaluation of different aspects of coding ability.Figure 1 illustrates semantic-altering prompts; semantic-preserving transformations are also used.Source: §2 · p. 4
  • GPT-4-generated problems may contain contradictions or incorrect examples, so the refinement pipeline iteratively rephrases problems and checks solution outputs for consistency.GPT-4 derives or generates test inputs, obtains expected outputs by running them on a solution, and uses self-consistency between solutions before terminating refinement.Source: §2 · pp. 4–5
  • The final suite receives additional tests using the EVALPLUS LLM-based test-augmentation technique.These tests supplement the initial cases from refinement.Source: §2 · p. 5

3 EVOEVAL Dataset Overview

EvoEval uses HumanEval problems as seeds for seven datasets, with five semantic-altering and two semantic-preserving benchmarks. Its transformations vary in problem type and in how they change the descriptions and embedding distributions.

3 EVOEVAL Dataset Overview

  • EvoEval contains 828 problems across seven datasets: five semantic-altering and two semantic-preserving benchmarks.Problems include docstrings, test cases, and groundtruth; Tool Use additionally includes helper functions.Source: §3 · pp. 5–6
  • The seven transformations cover added difficulty, creative narratives, subtle requirement changes, combining problems, helper-function use, and verbose or concise rewording.Combine selects problems of the same type based on their input arguments; provided Tool Use helpers need not be used.Source: §3 · p. 6
  • Each semantic-altering benchmark has 100 problems, while semantic-preserving benchmarks use all 164 HumanEval problems; EvoEval questions are longer and use more test cases than HumanEval.The test cases for Subtle, Verbose, and Concise reuse augmented HumanEval+ cases because of their similarity.Source: §3 · p. 6
  • In the t-SNE visualization, Creative and Tool Use shift docstring embeddings substantially, while Subtle, Difficult, and Combine largely retain the original distribution.Verbose and Concise embeddings almost perfectly match the original problems; Combine shifts the embeddings of both source problems.Source: §3 · p. 6

4 Methodology

The evaluation measures functional correctness on EvoEval using greedy single-sample decoding across 51 models. The accompanying figures compare HumanEval+ and EvoEval performance and show normalized pass@1 across evolved benchmarks.

4 Methodology

  • Each generated sample is run against EvoEval tests and compared with groundtruth; functional correctness is reported as pass@1 under greedy decoding at temperature 0.The setup uses one deterministic sample per problem.Source: §4 · p. 6
  • Figure 4 compares pass@1 on HumanEval+ and EvoEval; its identity line marks equal performance, and its regions distinguish aligned from HumanEval-higher performance.The blue region indicates higher performance on HumanEval than on EvoEval.Source: Figure 4 · p. 7
  • Figure 5 plots normalized pass@1 across EvoEval benchmarks for the top 10 and bottom 10 HumanEval performers.The score is normalized using the minimum and maximum pass@1 achieved by all LLMs on each benchmark.Source: Figure 5 · p. 7
  • The evaluation includes 51 proprietary and open-source general-purpose and code-focused LLMs, classified as base or instruction-following models.The analysis focuses on how model variants perform on EvoEval.Source: §4 · p. 7
  • Base models receive a function header and docstring for completion, while instruction-following models receive the model-maker’s recommended instruction and format.Instruction-following models are asked to generate a complete solution.Source: §4 · p. 7

5 Evaluation

On EvoEval, LLMs perform worse than on HumanEval, and their rankings and strengths vary across problem types. The evaluation also finds challenges in rewording, composition, decomposition, and using helper functions.

5.1 LLM Synthesis & Evaluation on EVOEVAL

  • Across all benchmarks, LLM performance decreased by 39.4% on average versus HumanEval, with individual decreases ranging from 19.6% to 47.7%.Source: §5.1 · pp. 7, 9, Table 2 · p. 8
  • With similar HumanEval scores, WizardCoder-1.1 performs better on DIFFICULT and CREATIVE, while Phind-CodeLlama-2 performs better on COMBINE.Source: Figure 6 · p. 9, §5.1 · pp. 9–10
  • EvoEval reveals ranking differences obscured by HumanEval: top-five model performance differs by over 20% on average, and GPT-4 and Claude-3 excel on different problem types.Source: §5.1 · p. 9
  • Instruction-following models lose pass@1 on reworded VERBOSE and CONCISE tasks, while non-instruction-following models show average gains; instruction-following models also drop 7.6% on SUBTLE.Source: §5.1 · p. 10

5.2 Problem Composition

  • Only GPT-4 achieves a composition percentage above half on COMBINE, indicating that models struggle to solve problems formed from known concepts.Source: Table 3 · p. 11, §5.2 · p. 11
  • On COMBINE-NAIVE, composition improves but the best model solves only 3/4 of problems whose two parent problems it had already solved.Source: §5.2 · p. 11

5.3 Problem Decomposition

  • On DECOMPOSE, models do not achieve a high decomposition percentage, though they sometimes solve both subproblems even when they fail the parent problem.Source: Table 4 · p. 12, §5.3 · p. 12

5.4 Tool Using

  • Helper functions raise average pass@1 by 81.3%, but the average improvement is 60.4% for instruction-following models versus 122.0% for base models.Source: §5.4 · pp. 12–13, Figure 9 · p. 13

6 Related Work

Prior work develops code LLMs through code-focused training and instruction tuning, while code-generation benchmarks span general-purpose and specialized tasks. EvoEval differs by evolving existing problems into more complex problems across targeted domains.

Related Work

  • Code LLMs progressed from code completion models to instruction-tuned systems, including approaches using complex instructions, synthesized instruction data, and execution feedback [30] [50] [55].Examples include CodeLlama-Inst [40] and DeepSeeker-Inst [16].Source: §6 · p. 13
  • HUMANEVAL [10] and MBPP are widely used test-based code-generation benchmarks; variants add more complete tests (HUMANEVAL+ [28]), languages (HUMANEVAL-X [54]), or 18 languages (MultiPL-E).Source: §6 · p. 13
  • Unlike benchmarks handcrafted from scratch or scraped from repositories and contest sites, EvoEval evolves existing problems into complex problems across targeted domains.The authors present this approach as addressing high manual effort, unavoidable data leakage, and the narrow scope of prior benchmarks.Source: §6 · p. 14

7 Conclusion

EvoEval evolves existing problems into targeted domains to evaluate LLM program synthesis. Across 51 LLMs, performance drops relative to standard benchmarks, with ranking differences indicating potential overfitting.

Conclusion

  • On average, performance drops 39.4% on EvoEval compared with standard benchmarks across the study's 51 LLMs.The benchmark suite contains 828 problems across seven benchmarks, built on HumanEval.Source: §7 · p. 14
  • EvoEval comprises 828 problems across seven benchmarks, created by evolving HumanEval problems into different target domains.The benchmarks, tools, and complete LLM generations are open-sourced.Source: §7 · p. 14
  • Ranking differences from previous leaderboards indicate potential overfitting of popular LLMs on existing benchmarks.The paper also reports insights about instruction-following brittleness and problem composition and decomposition abilities.Source: §7 · p. 14
  • The authors suggest EvoEval can support comprehensive benchmarking and further evolution of arbitrary problems as code LLMs advance.They also hope the findings encourage targeted training approaches for code LLMs.Source: §7 · p. 14

A.1 Evaluated LLMs

The evaluated set includes 51 LLMs, accessed through model weights when available or provider endpoints for closed-source models.

A.1 Evaluated LLMs

  • The study evaluates 51 LLMs.Their overview is provided in Table 5.Source: §A.1 · p. 19
  • For models with open-source weights, the authors obtained them from the Hugging Face model hub; closed-source models were accessed through provider endpoints.The repository provides further access details for each model.Source: §A.1 · p. 19

A.2 Detailed Evaluation Setup

The evaluation generates code using model-specific prompts and greedy decoding, sanitizes outputs, and checks correctness with differential testing and timeouts.

A.2 Detailed Evaluation Setup

  • LLMs are evaluated with pass@1 using greedy decoding at temperature 0, with function headers as prompts for base models and model-specific instruction formats for instruction-following models.Figure 10 gives a GPT-4 prompt example for a CREATIVE problem.Source: §A.2 · p. 19, Figure 10 · p. 19
  • Table 5 lists evaluated models, their model IDs, and whether weights are available or evaluation uses a closed-source API endpoint.For certain models, vLLM is used for more efficient generation.Source: §A.2 · p. 19, Table 5 · p. 20
  • Outputs are sanitized to parse code blocks and remove end-of-string identifiers before evaluation.The sanitization script is adopted from EVALPLUS [28].Source: §A.2 · pp. 19, 21
  • Correctness is checked by differential testing against ground-truth outputs, using exact matching or an absolute difference threshold of 10^-6 for floating-point outputs.The evaluator recursively compares output types and includes custom oracles for problems where standard checks cannot fully guarantee correctness.Source: §A.2 · p. 21
  • Timeout per problem is T = max(Tmax, f × tgt), with Tmax = 1000ms and f = 4 by default.tgt is the measured ground-truth solution time, and timeout factors can be adjusted for machine and hardware variance.Source: §A.2 · p. 21

B Transformation Prompts

EvoEval supplies targeted prompts for transforming benchmark problems and additional prompts for refining problems and preparing examples for self-consistency evaluation.

B Transformation Prompts

  • The transformation prompts target DIFFICULT, CREATIVE, SUBTLE, COMBINE, TOOL USE, VERBOSE, CONCISE, and DECOMPOSE variants of existing benchmark problems.The appendix provides the exact prompts used for these transformations.Source: B Transformation Prompts · p. 21, Figure 11 · p. 21, Figure 12 · p. 21
  • The refinement prompt revises an origin-generated problem when an inconsistency is detected.The paper points to Section 2 for the inconsistency-detection process.Source: B Transformation Prompts · p. 22
  • The extraction prompt obtains docstring test cases for self-consistency evaluation, while the I/O fixing prompt corrects examples whose outputs disagree with GPT-4 ground truth.Both prompts are shown in Figure 20.Source: B Transformation Prompts · p. 22

C Example Problems in EVOEVAL

The appendix presents examples of EvoEval problems that GPT-4 cannot solve, spanning several transformed benchmark domains.

C Example Problems in EVOEVAL

Loading 2403.19114v1…