PaperlayerView exact revision on arXiv

Source-linked AI summary

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, Ion Stoica

arXiv:2403.07974v2cs.SEcs.CLcs.LG

TL;DR

Existing code benchmarks emphasize natural-language-to-code generation and face contamination and coverage limitations. LiveCodeBench addresses these issues with continuously updated, dated competition problems and four coding scenarios, finding likely contamination, task-dependent model differences, and possible HumanEval overfitting. The benchmark currently focuses on competition programming and has finite problem subsets that introduce comparison uncertainty.

  • Problem

    Existing benchmarks such as HumanEval and MBPP focus on natural-language-to-code tasks and may be contaminated or overfit, limiting coverage of broader code capabilities.

  • Method

    LiveCodeBench continuously collects dated problems from LeetCode, AtCoder, and CodeForces and evaluates code generation, self-repair, code execution, and test output prediction.

  • Results

    Evaluations reveal likely contamination, task-dependent performance differences, and possible HumanEval overfitting by some fine-tuned open-access models.

  • Takeaways & Limitations

    Time-segmented and holistic evaluation provides contamination-aware comparisons across multiple code-related capabilities.

  • Takeaways & Limitations

    LiveCodeBench focuses on competition problems from three platforms and may not represent general programming capabilities or real-world open-ended software development.

Abstract

from arXiv · show

Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval, MBPP) are no longer sufficient for assessing their capabilities. In this work, we propose LiveCodeBench, a comprehensive and contamination-free evaluation of LLMs for code, which continuously collects new problems over time from contests across three competition platforms, namely LeetCode, AtCoder, and CodeForces. Notably, our benchmark also focuses on a broader range of code related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. Currently, LiveCodeBench hosts four hundred high-quality coding problems that were published between May 2023 and May 2024. We have evaluated 18 base LLMs and 34 instruction-tuned LLMs on LiveCodeBench. We present empirical findings on contamination, holistic performance comparisons, potential overfitting in existing benchmarks as well as individual model comparisons. We will release all prompts and model completions for further community analysis, along with a general toolkit for adding new scenarios and model

1 Introduction

LiveCodeBench addresses stagnant, potentially contaminated coding benchmarks by continuously collecting dated competition problems and evaluating capabilities beyond code generation. Its evaluations reveal contamination, task-dependent model differences, and possible HumanEval overfitting.

  • Motivation: Existing benchmarks such as HumanEval, MBPP, and APPS focus on natural-language-to-code tasks and may contain training examples.They overlook broader code-related capabilities and may produce skewed or misleading evaluations.
  • Benchmark: The benchmark evaluates code generation, self-repair, code execution, and test output prediction rather than code generation alone.Self-repair uses execution feedback, code execution predicts a program’s output, and test output prediction predicts outputs from a problem and input.
  • Benchmark: LiveCodeBench continuously collects dated contest problems from LeetCode, AtCoder, and CodeForces to support contamination-aware evaluation.The benchmark collected 511 problems from May 2023 to May 2024 for multiple scenarios.
  • Findings: Time-segmented evaluations expose likely contamination: DeepSeek, GPT-4-O, and Codestral drop on problems released after their respective cutoff periods.Post-cutoff windows are used for fairer comparisons.
  • Findings: Model performance is correlated across tasks, but relative differences vary, with task-specific gaps between open and closed models.Claude-3-Opus and Mistral-L perform considerably better on code execution and test output prediction than on code generation.
  • Findings: Models that perform well on HumanEval but not LiveCodeBench are primarily fine-tuned open-access models, indicating possible HumanEval overfitting.Base models and closed models more often perform well on both benchmarks.

4. Model Comparisons (Figure 4)

LiveCodeBench finds that closed API models generally outperform open models, while the strongest large instruction-tuned open models narrow the gap. It also separates model rankings from existing benchmark impressions, with GPT-4 leading most comparisons.

  • Model Comparisons: Among open-access base models, L3-Base and DeepSeek-Base are strongest, followed by StarCoder2-Base and CodeLLaMa-Base.
  • Model Comparisons: Closed API models such as GPTs, Claude, and Gemini generally outperform open models on LiveCodeBench.The comparison is reported across the benchmark’s evaluated model set.
  • Model Comparisons: L3-Ins-70B, Mixtral, and DS-Ins-33B are the open models identified as closing the gap with closed models.These are instruction-tuned variants of base models larger than 30B parameters.
  • Model Comparisons: GPT-4 and GPT-4-Turbo outperform all other models except Claude-3-Opus by a large margin in all scenarios.Existing benchmarks sometimes show smaller models matching or exceeding GPT-4, unlike LiveCodeBench.

2 Holistic Evaluation

LiveCodeBench evaluates coding as a multi-faceted capability rather than only natural-language-to-code generation. It combines four automatically evaluated scenarios, including a new fixed-input test-output prediction task, and is extensible to future scenarios.

  • 2 Holistic Evaluation: LiveCodeBench evaluates four coding scenarios: code generation, self-repair, code execution, and test output prediction.The settings were selected as useful code-workflow components with clear automated metrics.
  • 2 Holistic Evaluation: Code generation produces a program from a problem statement and measures Pass@1 using unseen functional tests.
  • 2 Holistic Evaluation: Self-repair supplies error feedback or a failing test after an initial mistake and evaluates functional correctness of the final program.The model attempts repair after receiving execution information.
  • 2 Holistic Evaluation: Code execution asks the model to predict a program’s output for an input and checks correctness by executing the program.A prediction is correct when assert f(input) == generated output passes.
  • 2 Holistic Evaluation: Test case output prediction asks for the expected output on fixed inputs from a problem statement without providing the function implementation.Fixed inputs avoid the harder task of predicting test inputs and enable straightforward evaluation.
  • 2 Holistic Evaluation: The framework can incorporate future scenarios such as input generation, program summarization, and optimization.

3 Benchmark Curation

LiveCodeBench curates dated competition problems, tests, solutions, and difficulty metadata from three platforms, filtering for consistent automated grading and balanced comparisons. Its scenarios use platform or generated tests across distinct dataset constructions.

  • 3.1 Data Collection: Problems are sourced from LeetCode, AtCoder, and CodeForces contests and associated with contest release dates.Release dates support evaluation over time windows for models trained at different times.
  • 3.1 Data Collection: The curation process parses formulas, excludes image-dependent or ambiguously gradable problems, and collects solutions and test cases.These filters aim to improve quality and consistency for automated evaluation.
  • 3.1 Data Collection: Tests are collected from platform websites when possible; otherwise, an LLM generates them through input generators.
  • 3.1 Data Collection: Difficulty ratings are used to exclude problems likely too difficult for current models and to classify Easy, Medium, and Hard subsets.This reduces low-variance comparisons and supports more granular analysis.
  • 3.1 Data Collection: CodeForces problems remain harder than those from the other platforms, so the benchmark focuses primarily on LeetCode and AtCoder.The collection uses Division 3 and Division 4 problems and restricts the selected fraction.
  • 3.1 Data Collection: The final dataset contains 511 problem instances for code generation and self-repair, while test case output prediction contains 442 instances from 181 LeetCode problems.

4 Experiment Setup

The experiments evaluate diverse base and instruction-tuned models with Pass@1 across four code scenarios, using scenario-specific prompting and grading procedures.

  • Models: The study evaluates 52 models ranging from 1.3B to 70B, including base, instruction-tuned, open-access, and closed-access models.
  • Evaluation Metrics: Pass@1 is computed from 10 generated answers per problem using nucleus sampling with temperature 0.2 and top p 0.95.
  • Common Setup: Base models are evaluated only for code generation because they do not easily follow the required formats for the other scenarios.
  • Scenario Setup: Code generation uses zero-shot prompts for instruction-tuned models and constant one-shot examples for base models.
  • Scenario Setup: Self-repair reuses generated programs and execution feedback, including syntax, runtime, wrong-answer, and time-limit errors.
  • Scenario Setup: Code execution uses few-shot prompts with either two examples without chain-of-thought or one manually detailed chain-of-thought example.
  • Scenario Setup: Test output prediction uses a zero-shot prompt asking models to complete assertions from the problem, function signature, and test input.

5 Results

LiveCodeBench measures contamination, cross-scenario capability, and benchmark generalization across models. Results reveal cutoff-aligned performance drops, strong but imperfect scenario correlations, and substantial gaps between benchmark families and model access types.

  • Avoiding Contamination: Post-cutoff evaluation provides performance estimates on unseen problems and exposes sharp LeetCode drops for some models near their release or cutoff dates.
  • Avoiding Contamination: AtCoder performance remains relatively stable across release periods, unlike the pronounced LeetCode variation observed for DeepSeek and GPT-4-O.
  • Holistic Evaluations: Pass@1 correlations exceed 0.88 across scenario pairs, reaching 0.98 for generation and self-repair and 0.96 for test output prediction and execution.
  • Holistic Evaluations: Despite strong correlations, scenario differences change relative rankings: Claude-3-Opus surpasses GPT-4-Turbo on test output prediction, while Mistral-L surpasses Claude-3-Sonnet there and on execution.
  • Comparison to HumanEval: HumanEval+ and LCB-Easy show only a 0.72 correlation, with larger performance variation on LCB-Easy and a cluster of fine-tuned open models performing better on HumanEval+ alone.
  • Comparison to HumanEval: DS-Ins-1.3B reaches 59.8% Pass@1 on HumanEval+ but 26.3% on LCB-Easy, illustrating the benchmark gap for a fine-tuned open model.
  • Model Comparisons: GPT-4-Turbo leads DS-Ins-33B by 96% on test output prediction and 134% on code execution, showing larger gaps than code generation.
  • Model Comparisons: Post-training improves LiveCodeBench code-generation scores, with L3-Ins-70B, DS-Ins-33B, and Phind-34B gaining 8.2, 7.3, and 9.5 points over their base models.

6 Related Work

Related work spans code-generation benchmarks, competitive-programming evaluation, auxiliary code capabilities, and contamination detection. LiveCodeBench is positioned as a live benchmark that broadens scenarios and uses more complex human-produced functions than CRUXEval.

  • Code Generation Benchmarks: Prior code-generation benchmarks primarily evaluate natural-language-to-Python generation, including HumanEval, HumanEval+, APPS, CodeContests, MBPP, and L2CEval.
  • Code Generation Benchmarks: Competitive-programming benchmarks include APPS, CodeContests, CodeScope, xCodeEval, LeetCode-Hard, and TACO.
  • Additional Scenarios: LiveCodeBench adds self-repair, test output prediction, and code execution as scenarios beyond code generation.
  • Code Repair: Prior self-repair work uses error feedback to improve code, inspiring LiveCodeBench’s code-repair scenario.
  • Code Execution: LiveCodeBench’s execution scenario is inspired by CRUXEval but uses live problems and more complex human-produced functions rather than Code Llama generations.
  • Other Directions: Related work also studies test generation, type prediction, code summarization, code security, and contamination detection for language-model benchmarks.

7 Limitations

LiveCodeBench has limitations involving evaluation-set size, language coverage, prompt robustness, and problem-domain representativeness. The authors recommend cautious comparisons and supplementary domain-specific evaluation.

  • 349-problem evaluation subsets can introduce estimated 1−1.5% performance variation, making small model differences difficult to interpret.The authors recommend proper judgement when comparing models with small performance differences.
  • Recent models with newer cutoff dates may be evaluated on smaller problem sets, increasing the evaluation-set limitation.The authors propose adding more competition platforms and a private test set to reduce reliance on public problems.
  • LiveCodeBench currently evaluates only Python, limiting evidence about model capabilities in other programming languages.The authors state that adding languages should be straightforward once suitable evaluation engines are available.
  • Untuned or minimally adjusted prompts can create performance variance across models, although findings and comparison orders mostly match HumanEval trends.The limitation is particularly relevant when prompt quality differs across models or scenarios.
  • Open models perform worse than closed models on chain-of-thought prompting for code execution, but the prompts may be sub-optimal.The authors specifically identify this issue in the code execution scenario.
  • Because LiveCodeBench focuses on competition problems from three platforms, it may not represent open-ended real-world programming use.The authors recommend supplementing it with domain-specific evaluations for particular settings.

8 Conclusion

The paper introduces LiveCodeBench, a benchmark designed to address contamination and narrow task coverage in code-LLM evaluation. Its extensible live-evaluation framework reveals contamination detection and potential HumanEval overfitting.

  • LiveCodeBench is a new benchmark for evaluating LLMs for code that uses live evaluations and scenarios beyond code generation.
  • The benchmark is extensible, with planned updates involving new problems, scenarios, and models.
  • Evaluations reveal contamination detection and potential overfitting on HumanEval.

A Dataset

The dataset filters competitive-programming code and constructs a compact code-execution benchmark using random and adversarial inputs, while enforcing execution and code-quality constraints.

  • The benchmark scrapes publicly visible problem statements, ground-truth solutions, and test cases from LeetCode, AtCoder, and CodeForces.
  • GPT-4-Turbo generates separate random and adversarial input generators, which are executed and validated against collected correct programs.The two generator types address both broad sampling and problem corner cases.
  • The code-execution split contains 479 samples from 85 problems, with at most six uniformly selected samples per problem.
  • Filtering requires 100–500-character programs, successful compilation, restricted operations and inputs, execution within 2 seconds, and roughly fewer than 1000 bytecode operations.
  • The final benchmark contains 479 filtered samples from 85 problems and is intended to grow through live updates.

C Experimental Setup

The experiments evaluate a broad model set with scenario-specific prompts for code generation, execution, test-output prediction, and self-repair.

  • C.1 Models: The study describes model identities and approximate cutoff dates for its evaluated language models.
  • C.1 Models: The model inventory includes base and instruction-tuned models from CodeLlama, DeepSeek, Llama, Qwen, Gemma, Mistral, Mixtral, Claude, Gemini, and GPT families.
  • C.2 Code Generation: Code-generation prompts ask models to return a correct Python program for a problem specification and follow the required answer format.
  • C.4 Code Execution: Code-execution prompts provide Python code and an input, asking for the resulting output with variants that include or omit chain-of-thought.
  • C.6 Self-Repair: Self-repair prompts provide a problem, faulty code, and feedback, requiring an explanation followed by a fixed program.
  • C.5 Test Output Prediction: Test-output prediction requires completing an assertion with a literal execution result, even when the function is incorrect or incomplete.

D.1 Contamination

LiveCodeBench examines contamination over time and reports a stark DeepSeek performance drop on newer LeetCode problems, while live evaluations generally show limited monthly variation except for DS models.

  • Figure 10 demonstrates contamination in DeepSeek across self-repair and test-output-prediction scenarios.
  • Figure 10 tracks contamination in DS models across self-repair and code-execution-without-COT scenarios over time.The code-execution measurements currently span May through November.
  • Live evaluation over time finds no significant performance variation across months for many recently released models except DS models.

E.1 Code Execution

The code-execution examples show that GPT-4 with chain-of-thought can still produce incorrect outputs across several programming tasks.

  • The paper presents five code-execution examples that GPT-4 still struggles to execute correctly even with chain-of-thought.
  • For countWays, GPT-4 with chain-of-thought outputs 1, 2, 4, or 5 instead of 3.
  • For minimumCoins, GPT-4 with chain-of-thought outputs 1, 3, 5, or 6 instead of the asserted result 4.
  • For sortVowels, GPT-4 with chain-of-thought produces five outputs, including "leetecode" and "leetcOde", rather than the asserted output.
  • For relocateMarbles, GPT-4 with chain-of-thought outputs incorrect lists or a KeyError instead of [5, 6, 8, 9].
  • For minimumSum, GPT-4 with chain-of-thought outputs 10, 11, or 12 instead of -1.
Loading 2403.07974v2…