PaperlayerView exact revision on arXiv

Source-linked AI summary

CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution

Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, Sida I. Wang

arXiv:2401.03065v1cs.SEcs.AIcs.LG

TL;DR

Code-LM benchmarks have largely emphasized code generation, leaving code understanding and execution less directly measured. CRUXEval constructs an 800-example benchmark of simple Python functions with input and output prediction tasks, then evaluates models and improvement methods. GPT-4 performs best, while chain-of-thought and fine-tuning improve results but remain far from solving the benchmark.

  • Problem

    HumanEval and MBPP measure code generation but do not adequately capture code understanding and execution.

  • Method

    CRUXEval generates many functions and inputs, filters them for simple deterministic execution, selects 800 examples, and evaluates input and output prediction.

  • Results

    GPT-4 performs best, while chain-of-thought and fine-tuning improve CRUXEval performance but remain far from acing it.

  • Takeaways & Limitations

    CRUXEval provides a complementary probe of code reasoning and execution, including a lens into GPT-4’s failures on simple programs.

  • Takeaways & Limitations

    Accuracy may vary with prompt format and temperature, and fine-tuning evaluation may retain semantically duplicated examples because decontamination used exact string matching.

Abstract

from arXiv · show

We present CRUXEval (Code Reasoning, Understanding, and eXecution Evaluation), a benchmark consisting of 800 Python functions (3-13 lines). Each function comes with an input-output pair, leading to two natural tasks: input prediction and output prediction. First, we propose a generic recipe for generating our execution benchmark which can be used to create future variation of the benchmark. Second, we evaluate twenty code models on our benchmark and discover that many recent high-scoring models on HumanEval do not show the same improvements on our benchmark. Third, we show that simple CoT and fine-tuning schemes can improve performance on our benchmark but remain far from solving it. The best setup, GPT-4 with chain of thought (CoT), achieves a pass@1 of 75% and 81% on input and output prediction, respectively. In contrast, Code Llama 34B achieves a pass@1 of 50% and 46% on input and output prediction, highlighting the gap between open and closed source models. As no model is close to acing CRUXEval, we provide examples of consistent GPT-4 failures on simple programs as a lens into its code reasoning capabilities and areas for improvement.

1 Introduction

CRUXEval addresses the lack of benchmarks for code understanding and execution by testing models on simple Python programs through input and output prediction. Its evaluations reveal substantial remaining reasoning gaps, including failures by GPT-4.

  • Existing benchmarks such as HumanEval and MBPP primarily measure code generation, leaving code understanding and execution insufficiently evaluated.
  • CRUXEval measures output prediction for code execution following and input prediction for code reasoning and understanding.
  • The benchmark contains simple programs of at most 13 lines designed to be solvable by a university-level computer science graduate without extra memory.
  • The benchmark is generated by producing functions and inputs, executing them, filtering candidates, and randomly selecting 800 samples.
  • 67% and 63% are GPT-4’s pass@1 scores on CRUXEval-I and CRUXEval-O, respectively.
  • GPT-4 still consistently fails on some surprisingly simple Python programs, exposing unresolved code-execution reasoning errors.

2 Related Work

Prior work evaluates code generation and related software tasks, but CRUXEval targets execution ability and input prediction as distinct dimensions of code-model capability.

  • HumanEval and MBPP evaluate Python code generation on relatively simple functions.
  • Existing benchmarks also cover data science, APIs, repositories, software engineering, translation, testing, search, typing, and summarization.
  • CRUXEval is presented as the first publicly available benchmark for code-LM execution ability and the first for input prediction.
  • Related execution-focused work trains models with execution information, while CRUXEval examines chain-of-thought and fine-tuning as primitive improvement methods.

3 Benchmark Construction

CRUXEval is constructed by generating many deterministic Python function examples, filtering them for simple and reliable execution, and selecting a statistically adequate subset.

  • CRUXEval contains 800 distinct functions with deterministic input-output pairs supporting output prediction and input prediction tasks.
  • Code Llama 34B generated 102,000 functions and 489,306 input-output pairs across 69 Python standard-library functions.
  • Filtering removes examples requiring substantial arithmetic or memory and excludes syntax errors, exceptions, nondeterminism, imports, and side effects.
  • 3.3 Data size and measuring noise: 800 examples are sufficient to distinguish several model-pair performance relationships with high confidence using bootstrap samples.
  • 3.3 Data size and measuring noise: At N = 800, sampling-data noise is generally around 1.5% per model, compared with approximately 0.2% from sampling model candidates.

4 Evaluation

The evaluation compares a broad selection of code models using pass@1 and pass@5 under controlled sampling and bootstrap uncertainty estimates. Results are summarized with percentile intervals and medians.

  • The evaluation includes StarCoder, Mistral, WizardCoder, Phi, Phind, Code Llama, DeepSeek Coder, GPT-3.5, and GPT-4 models.
  • Non-GPT models use 100 samples and GPT models use 10 samples, with pass@1 at T = 0.2 and pass@5 at T = 0.8.
  • Figure 2 displays bootstrap intervals, with boxes covering the 25th–75th percentiles, whiskers the 2.5th–97.5th percentiles, and middle bars the median.

5 Quantitative Analysis

CRUXEval reveals that code-generation performance does not uniformly predict execution and reasoning ability, while CoT and fine-tuning help unevenly and remain insufficient for solving the benchmark.

  • HumanEval and CRUXEval: Distilled models substantially outperform base models on HumanEval but do not significantly outperform Code Llama 34B on CRUXEval.This breaks the apparent connection between HumanEval gains and execution or understanding performance for WizardCoder, Phind, and Phi.
  • Task relationships: Input prediction and output prediction scores are strongly correlated despite testing relatively distinct reasoning abilities.The paper characterizes output prediction as following execution and input prediction as higher-level code reasoning and understanding.
  • Chain-of-thought prompting: CoT benefits GPT-4 most, reaching 74.8% pass@1 on input prediction and 81.9% on output prediction.CoT also helps Code Llama 34B on both tasks, GPT-3.5 only on output prediction, and Code Llama 13B on neither task.
  • Chain-of-thought prompting: CoT widens the gap between pass@5 and pass@1, which the authors associate with increased diversity in generated inputs and outputs.Without CoT, pass@1 and pass@5 are positively correlated; with CoT, their gap increases.
  • Chain-of-thought prompting: CoT often harms individual predictions for Code Llama and GPT-3.5, whereas GPT-4 generally improves, especially on output prediction.For Code Llama 13B, 34B, and GPT-3.5, many samples succeed under direct prediction but fail under CoT.
  • Fine-tuning experiments: Fine-tuning on similar Python samples lets Code Llama 34B match GPT-4, but accuracy plateaus below 70% on both tasks.The fine-tuning setup used a liberal mixture of related and benchmark-function samples, yet test accuracy still plateaued.

6 Qualitative Analysis

The qualitative analysis documents repeatable GPT-4 and GPT-4 CoT failures on short, simple Python programs, including counting, loop simulation, string manipulation, and condition evaluation.

  • GPT-4 CoT failures: GPT-4 CoT scored 0/10 on 54 output-prediction and 65 input-prediction tasks, with 22 problems failing on both tasks.Manual inspection found that most of the jointly failing problems were simple, though some required counting or simulating a few loop steps.
  • String manipulation: Some GPT-4 CoT failures arise from incorrect string concatenation despite identifying the intended operation.The examples include repeated-character constructions and longer string manipulations where the predicted result differs from the required concatenation.
  • String manipulation: GPT-4 CoT can be misled by variable names, as in a removeprefix example where it predicts an output inconsistent with the function’s behavior.The example is labeled as a possible effect of the variable name prefix.
  • Output prediction: Output-prediction failures persist on simple two-answer problems, including a mistaken comparison involving 6173 and 1000.Small perturbations to the condition and strings also failed, and scratchpad-style prompting failed similarly.
  • Input prediction: GPT-4 CoT fails on input prediction for concise loops and character transformations, producing inputs whose outputs do not match the assertions.Reported examples include a character-doubling loop and a translation-based lowercase transformation.
  • Other GPT-4 failures: The authors report six additional simple string-manipulation failures and suspect tokenization may partially contribute.The examples involve repetition, rfind, slicing, joining, substring extraction, and replacement.

7 Limitations and Future Work

The authors identify benchmark, measurement, fine-tuning, and reasoning limitations while proposing broader execution evaluations and self-repair analyses as future work.

  • Distilling future execution benchmarks: CRUXEval measures execution on relatively simple, self-contained Python functions distilled from Code Llama 34B, leaving longer, open-domain, and cross-language settings unexplored.The authors invite variants using other code distributions.
  • Variation due to prompt and temperature: Benchmark accuracy may vary substantially with prompt format and temperature, limiting direct comparisons between models.The authors report using only T = 0.2 and T = 0.8 because of budget constraints.
  • Information loss due to pass@1: Pass@k compresses substantial behavioral information into a single number, motivating additional evaluation and analysis techniques.The authors specifically suggest reporting pass@1 and pass@5 while recognizing that pass@k is only one perspective.
  • Fine-tuning: Exact-string decontamination may leave semantic duplicates or minimally modified programs in fine-tuning data, inflating measured performance.The experiments use only a direct fine-tuning scheme, leaving process supervision and execution-state supervision for future work.
  • Jointly improving code generation and code execution: Code-generation fine-tuning in distilled models does not significantly improve CRUXEval performance, while the effect of execution-focused fine-tuning on code generation remains unknown.The authors propose studying techniques that improve both abilities simultaneously.
  • Understanding reasoning from the lens of code: Future evaluations could test recursive functions, natural-language-plus-input execution, and compositions of multiple functions.The authors also identify output prediction as a useful setting for studying chain-of-thought failures and consider self-repair promising for recitation errors.

8 Conclusion

The conclusion presents CRUXEval as a complementary benchmark for code-model input and output prediction, built through distillation, filtering, and statistical dataset selection. It evaluates models, studies improvement techniques, and analyzes GPT-4 failures while detailing the benchmark’s construction and diversity controls.

  • 8 Conclusion: CRUXEval evaluates code LMs on input prediction and output prediction using simple Python functions.The benchmark is designed to assess code reasoning, understanding, and execution.
  • 8 Conclusion: The benchmark recipe combines large-scale distillation, filtering, and data-size selection through statistical noise analysis.The construction process uses generated functions and test inputs before selecting a compact benchmark.
  • 8 Conclusion: The study evaluates 20 models and analyzes HumanEval correlation, task correlation, model differences, and model diversity.It also includes qualitative analyses of GPT-4 successes and failures.
  • 8 Conclusion: Chain-of-thought and fine-tuning are explored as approaches for improving CRUXEval performance.These experiments are presented as separate investigations in the paper’s evaluation sections.
  • Dataset construction: The benchmark includes non-dunder methods from str, list, and dict, while operator methods were excluded as often too simple or computational.The listed method inventories define the main data sources used for generation.
  • Dataset construction: Using 25 few-shot prompt combinations helps overcome a duplication bottleneck in generated functions.For one fixed pair of examples, approximately 60,000 generations yielded only about 5,000 unique functions.

A.3 Dataset Statistics

The appendix characterizes CRUXEval’s sample distributions, execution-step counts, task-difficulty relationship, and method frequencies, revealing non-uniform coverage and little correlation between input and output prediction difficulty.

  • Dataset distributions: The 800 benchmark samples vary in character count and line count, as summarized in Figure 9.The cited text identifies these two distribution dimensions but does not provide their numerical ranges.
  • Number of steps: Execution step counts roughly correspond to Python bytecode operations, with one 3,175-step outlier excluded from Figure 10.The precise step definition is implemented through the benchmark’s numsteps variable.
  • Task correlation: Figure 11 shows little to no correlation between sample-level output-prediction and input-prediction pass@1 scores.The result suggests that the two tasks do not share a strong sample-level difficulty ordering.
  • Method-level statistics: Method frequencies are highly non-uniform despite equal str and approximately doubled list and dict distillation targets.Instruction-following failures, filtering, and naturally different operation counts contribute to the observed distribution.
  • Method difficulty: The appendix compares the easiest and hardest methods for Code Llama 34B and WizardCoder using input- and output-prediction pass@1 scores.The comparisons are presented in Figures 13–15 for methods meeting the sample-count criterion.
  • Method difficulty: Underrepresented or obscure methods may be useful targets for distillation because their syntax and semantics may be less well learned.The authors frame this as speculation about why distilled models differ across method coverage.

Appendix B Model URLs

This appendix records the evaluated model implementations and directs readers to the main results table and confidence-interval visualization for CRUXEval.

  • Model URLs: GPT-3.5-turbo and GPT-4 were evaluated on October 26, 2023, before the release of GPT-4-Turbo.The remaining evaluated models are identified through HuggingFace URLs in Table 1.
  • Results references: Table 2 reports pass@1 and pass@5 results for all evaluated models on CRUXEval.Figure 16 presents the same main results in box-plot form with confidence intervals compared with Code Llama 34B.

C.2 Additional Results on Impact of CoT

Additional analyses show that CoT’s benefits are uneven: it is substantially more effective for GPT-4 output prediction, while often reducing pass@1 for other models. GPT-4 also exhibits simple CoT-induced failures and idiosyncratic errors across models.

  • Sample-wide improvements from CoT: CoT produces little improvement for most samples, partly because many already have high pass@1 scores.The sample-level analysis measures changes in pass@1, with negative values indicating reduced accuracy.
  • Sample-wide improvements from CoT: GPT-4 output prediction benefits much more from CoT than GPT-4 input prediction or other evaluated models.For other models, a large proportion of samples experience decreased pass@1 with CoT.
  • Granular CoT effects: CoT often decreases pass@1 across samples, with a stark difference between its impact on GPT-4 and other models.The granular confusion-matrix view reinforces the uneven effect of CoT.
  • Qualitative failures: GPT-4 can fail after CoT even when it correctly identifies the required list property, incorrectly returning a descending-sorted list.The function reverses a copied list, sorts the original in descending order, and compares them.
  • Failure correlations: Model failures are idiosyncratic: easier models sometimes solve examples on which stronger models fail completely.The comparison uses P(Y | X = 0)/P(Y), the accuracy of model Y conditional on model X failing.

C.3 Results on Diversity of Generations

Generation diversity varies with temperature, task, model family, and CoT. Input prediction generally yields more diverse generations than output prediction, while CoT increases diversity at both tested temperatures.

  • Diversity across models: Higher temperature substantially increases the diversity of generated inputs and outputs.The comparison covers T = 0.2 and T = 0.8.
  • Diversity across models: Input prediction generally has greater generation diversity than output prediction.The authors suggest this may reflect multiple valid inputs versus one correct output.
  • Diversity across models: Code Llama models have the highest diversity at matched temperatures, while distilled models such as Phind and WizardCoder have lower diversity.This comparison concerns generations without regard to correctness.
  • CoT and diversity: CoT increases the average number of distinct generations for both input and output prediction at both temperatures.Different chains of thought producing the same prediction count as identical generations.
  • Functional diversity: At T = 0.2, models generally produce mostly correct or mostly incorrect outputs rather than intermediate pass@1 scores.Functional diversity is measured by the percentage of samples with pass@1 between 0.1 and 0.9.
  • Functional diversity: CoT greatly increases functional diversity, especially for GPT-3.5 and GPT-4.Distilled models show lower functional diversity than corresponding base models.

C.4 Difficulty of Benchmark Samples

The benchmark’s difficulty is distributed unevenly across models and tasks. Output prediction has a more bimodal pass@1 distribution, while CoT increases intermediate scores and GPT-4 maintains strong fully solved performance.

  • Difficulty distribution: At T = 0.8, selected models’ pass@1 distributions appear more bimodal than the overall distribution.Output prediction is more bimodal than input prediction, and CoT increases samples with intermediate scores between 0.25 and 0.75.
  • Fully solved and unsolved samples: At T = 0.8, the percentage of completely unsolved samples distinguishes models from their overall pass@1 ranking.GPT-3.5 and GPT-4 perform considerably worse on this metric than Code Llama 34B and DeepSeek Base 33B in non-CoT modes.
  • Fully solved and unsolved samples: GPT-4 achieves over 60% fully correct samples for both input and output prediction at T = 0.2.The fully solved metric uses pass@1 equal to 1.
  • Fully solved and unsolved samples: Instruction-tuned models are more consistent than base models on the fully solved metric.For output prediction, GPT-4 with CoT increases diversity without sacrificing consistency.
  • Anonymization: The appendix also evaluates Code Llama models on an anonymized benchmark subset with variable names replaced by x1, x2, and similar identifiers.It reports pass@1 at T = 0.2 and pass@5 at T = 0.8.

C.6 Impact of Data-Generating Model

Data-generation choices and fine-tuning formats materially affect CRUXEval results. Fine-tuning improves both tasks but plateaus, format misalignment lowers performance, and benchmark-program fine-tuning remains below near-perfect accuracy.

  • Impact of data-generating model: Using GPT-3.5 to generate benchmark data produces a significantly easier benchmark than the alternative generation settings.The authors attribute this to more predictable and guessable inputs, such as “abcde” rather than “mai2!”.
  • Impact of data-generating model: The relative ordering of Code Llama 13B, Code Llama 34B, and GPT-3.5 is preserved across datasets despite differing raw scores.The datasets used different prompts and a more relaxed filter than the main benchmark.
  • Fine-tuning results: Fine-tuning significantly improves both input and output prediction, but testing accuracy plateaus while training accuracy continues increasing.This pattern indicates that simple fine-tuning does not reach near-perfect CRUXEval performance.
  • Fine-tuning format: Misaligning fine-tuning and evaluation assertion formats lowers testing accuracy and can hinder even training-set overfitting.One reported input-prediction comparison shows almost a 10% testing-accuracy difference.
  • Benchmark-program fine-tuning: Fine-tuning on benchmark programs with different input-output pairs still cannot improve test accuracy beyond 70%.The setup mixes 50% original fine-tuning data with 50% “cheating” data containing benchmark programs.
  • Evaluation prompts: The evaluation uses task-specific prompts that constrain outputs to executable inputs or predicted outputs, with few-shot examples used for many non-GPT models.CoT prompts differ by model family and may require answer tags or a completion marker.

E.3.1 GPT-4 Successes with CoT, Output Prediction

GPT-4 with CoT can simulate execution and solve several non-trivial output-prediction examples, including counting, dictionary ordering, and string manipulation. The examples also expose failures when reasoning requires tracking loops, conditions, or exact transformations.

  • GPT-4 can simulate an execution process, but it also makes mistakes when tracing loops and conditional list mutations.The loop example requires tracking a fixed initial iteration count while alternately removing list elements from opposite ends.
  • GPT-4 demonstrates the ability to reason about code flow and count uppercase letters in the provided examples.
  • GPT-4 with CoT counts uppercase letters and applies the correct parity-dependent multiplier, producing 8 for the example string.The explanation identifies four uppercase letters and multiplies the even count by 2.
  • GPT-4 with CoT handles relatively complex string-manipulation tasks, including removing selected characters and rearranging comma-separated substrings.The examples show sequential removal of spaces, @ symbols, and ampersands, as well as list splitting, movement, and joining.
  • GPT-4 correctly reasons that the dictionary is sorted by values in descending order before formatting and joining the entries.The resulting order is tailoring, craftsmanship, upholstery, with value-key strings joined by ampersands.
  • The benchmark examples include GPT-4 CoT failures involving incorrect comparisons and failure to identify simple input-prediction insights.Reported failures include treating 6173 as less than 1000 and pursuing a complicated branch instead of choosing inputs that bypass a loop.
Loading 2401.03065v1…