PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeScore: Evaluating Code Generation by Learning Code Execution

Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, Zhi Jin

arXiv:2301.09043v4cs.SE

TL;DR

Code evaluation needs metrics that recognize functional equivalence and support more than reference-only inputs. CodeScore uses UniCE to train LLMs to estimate execution-related signals across three input formats, achieving state-of-the-art results and up to 58.87% absolute correlation improvements over other CEMs.

  • Problem

    Existing match-based CEMs measure surface differences rather than functional equivalence and are largely restricted to Ref-only evaluation.

  • Method

    CodeScore is an LLM-based CEM trained through UniCE to estimate PassRatio and Executability using unified input across Ref-only, NL-only, and Ref&NL formats.

  • Results

    CodeScore achieves state-of-the-art performance across multiple code-evaluation datasets and improves absolute correlation by up to 58.87% over other CEMs.

  • Takeaways & Limitations

    CodeScore provides a functional-correctness-oriented evaluator usable with reference code, natural-language descriptions, or both.

  • Takeaways & Limitations

    Evaluation covers three public code-generation datasets, so external validity and generalizability remain threats.

Abstract

from arXiv · show

A proper code evaluation metric (CEM) profoundly impacts the evolution of code generation, which is an important research field in NLP and software engineering. Prevailing match-based CEMs (e.g., BLEU, Accuracy, and CodeBLEU) suffer from two significant drawbacks. 1. They primarily measure the surface differences between codes without considering their functional equivalence. However, functional equivalence is pivotal in evaluating the effectiveness of code generation, as different codes can perform identical operations. 2. They are predominantly designed for the Ref-only input format. However, code evaluation necessitates versatility in input formats. Aside from Ref-only, there are NL-only and Ref\&NL formats, which existing match-based CEMs cannot effectively accommodate. In this paper, we propose CodeScore, a large language model (LLM)-based CEM, which estimates the functional correctness of generated code on three input types. To acquire CodeScore, we present UniCE, a unified code generation learning framework, for LLMs to learn code execution (i.e., learning PassRatio and Executability of generated code) with unified input. Extensive experimental results on multiple code evaluation datasets demonstrate that CodeScore absolutely improves up to 58.87% correlation with functional correctness compared to other CEMs, achieves state-of-the-art performance, and effectively handles three input formats.

1 INTRODUCTION

CodeScore addresses limitations of match-based code evaluation by estimating functional correctness across Ref-only, NL-only, and Ref&NL inputs. UniCE trains LLMs to learn execution-related signals, and CodeScore achieves strong performance across multiple datasets.

  • Match-based CEMs emphasize lexical or structural similarity, so they can rank incorrect code above functionally correct code.The bubble-sort example reports BLEU and CodeBLEU scoring incorrect code higher than truly functional code.
  • CodeScore evaluates generated-code functional correctness using Ref-only, NL-only, and Ref&NL input formats.This expands evaluation beyond reference-code-only comparisons.
  • UniCE tunes LLMs to learn code execution by estimating PassRatio and Executability with unified input.Executability distinguishes compilation errors from output errors when PassRatio equals 0.
  • The evaluation resources contain natural-language descriptions, reference codes, generated codes, and more than 100 test cases per task.
  • CodeScore substantially outperforms match-based CEMs and LLM-based EMs, achieving state-of-the-art performance on multiple code evaluation datasets.

2 BACKGROUND & RELATED WORK

Prior code-evaluation methods include match-based, execution-based, and LLM-based approaches. Match-based methods miss functional correctness, execution-based methods are costly and risky, and existing LLM-based methods are not taught code evaluation effectively.

  • Code Generation: Code generation has become a major research direction as deep learning and pre-training have advanced, increasing the need for effective evaluation.
  • Match-based CEMs: Match-based CEMs measure surface-level code differences and do not account for functional correctness.Examples include BLEU, Accuracy, CodeBLEU, METEOR, ROUGE, and CrystalBLEU.
  • Execution-based CEMs: Execution-based CEMs require test cases and resolved dependencies, incur substantial computational overhead, and introduce security risks during isolated execution.The paper characterizes them as costly, slow, and insecure for many real-world scenarios.
  • LLM-based EMs: LLM-based EMs either learn human judgments that are difficult to obtain for code or use token embeddings without teaching models how to evaluate code.The paper reports suboptimal code-evaluation performance for these approaches.

3 METHODOLOGY

CodeScore estimates functional correctness from execution-oriented signals across Ref-only, NL-only, and Ref&NL inputs. UniCE trains LLMs to predict PassRatio and Executability using unified representations and multitask losses.

  • 3.1 CodeScore: UniCE learns execution similarity through PassRatio and Executability, distinguishing successful execution from compilation or output errors when PassRatio is zero.The framework prioritizes continuous PassRatio because it better reflects execution similarity than binary Passability.
  • 3.2 UniCE: UniCE trains on a unified input sequence containing generated code with reference code, natural-language description, or both.The three formats are represented as g + r, g + n, and g + r + n.
  • 3.1 CodeScore: CodeScore accepts Ref-only, NL-only, and Ref&NL inputs, producing a scalar CodeScore and binary Exec for generated code.Exec equals 1 when the generated code executes successfully on all given test inputs.
  • 3.2 UniCE: The framework uses LLM representations, layer-wise attention pooling, and the first-token embedding before prediction.Trainable layer weights combine representations from multiple LLM layers, while the first token summarizes the input sequence.
  • 3.2 UniCE: Multi-task learning assigns losses to Ref-only, NL-only, and Ref&NL formats, with the unified objective combining the corresponding format-specific losses.Ref&NL data can contribute to all three losses, whereas single-format data contributes only to its matching loss.

4 EVALUATION

The evaluation tests CodeScore against existing evaluation metrics across performance, execution discrimination, loss contributions, human judgments, and practical use. These questions cover both effectiveness and applicability.

  • Research questions: The evaluation asks whether CodeScore outperforms other evaluation metrics on code evaluation tasks.This is RQ1, the primary comparative performance question.
  • Research questions: The evaluation examines whether Exec identifies executable generated code when dependencies are satisfied.This is RQ2.
  • Research questions: The evaluation studies the contribution of the unified loss across Ref-only, NL-only, and Ref&NL formats.This is RQ3.
  • Research questions: The evaluation also compares metric reasonableness from human perspectives and performance in practical scenarios.These are RQ4 and RQ5.

4.1 Experiment Setup

The experiments use three constructed code-evaluation datasets, multiple metric baselines, and correlation-based evaluation against functional correctness. Setup details include execution-derived labels, benchmark statistics, and implementation choices.

  • 4.1.1 Datasets: Three datasets—APPS-Eval, MBPP-Eval, and HE-Eval—are constructed from public code-generation benchmarks for code evaluation.The datasets pair natural-language descriptions and reference code with generated programs and extended test cases.
  • 4.1.1 Datasets: Each evaluation task includes generated code assessed by executing it on corresponding test cases to compute PassRatio and Passability.The construction process derives test inputs from existing cases, executes correct code to obtain outputs, and evaluates generated code against them.
  • 4.1.2 Baselines: The baselines span match-based, LLM-based, and execution-based code evaluation metrics.Match-based baselines include BLEU, Accuracy, CodeBLEU, and CrystalBLEU; LLM-based metrics include BERTScore, COMET, and CodeBERTScore; AvgPassRatio is execution-based.
  • 4.1.2 Baselines: BLEU and Accuracy measure lexical or exact matching, while CodeBLEU adds abstract-syntax-tree and data-flow information.These metrics primarily compare generated code with reference code rather than directly measuring execution.
  • 4.1.3 Correlation Evaluation: Correlation with functional correctness is evaluated using Kendall-Tau, Spearman R, Pearson R, and Mean Absolute Error.The study mainly compares CodeScore with AvgPassRatio using continuous execution similarity.
  • 4.1.4 Implementation Details: UniCE uses UniXcoder and formats inputs as generated code, reference code, and natural-language description separated by special tokens.The implementation balances the three input formats during training.

4.2 Experimental Results

CodeScore consistently correlates more strongly with functional correctness than prior evaluation metrics across datasets, input formats, and human judgments. Its Exec component accurately identifies executable code, while UniCE benefits evaluation across input formats.

  • 4.2.1 RQ1: Effect of CodeScore.: 40.56%, 55.07%, and 58.87% absolute improvements in τ, r_s, and r_p, respectively, over the strongest competing EM on APPS-Eval.CodeScore also achieves the lowest MAE and reduces execution time by three orders of magnitude versus execution-based CEMs.
  • 4.2.1 RQ1: Effect of CodeScore.: CodeScore retains the best correlation on MBPP-Eval and HE-Eval, including fine-tuning and zero-shot transfer from APPS-Eval.This transfer occurs despite substantial distribution differences among the datasets.
  • 4.2.2 RQ2: Effect of Exec.: Exec achieves extremely high Precision, F1 Score, and Accuracy for determining whether generated code executes when dependencies are met.Across datasets, performance exceeds 90% on HE-Eval in the zero-shot setting.
  • 4.2.3 RQ3: Effect of L_Uni.: L_Uni improves evaluation across input formats, with Ref&NL generally outperforming Ref-only because natural-language descriptions provide useful information.The experiments compare L_Uni with the respective losses on APPS-Eval, MBPP-Eval, and HE-Eval.
  • 4.2.4 RQ4: Human Evaluation.: CodeScore improves human evaluation by at least 54.6% over representative EMs, with all reported p-values substantially below 0.005.The human evaluation uses 100 HE-Eval samples scored by ten computer science PhD students.
  • 4.2.5 RQ5: Practical Scenario.: Case studies show that CodeScore avoids issues of prior EMs and that Exec distinguishes successful from unsuccessful compilation, including mismatched parentheses.The case-study captions compare alternative generated codes and present Exec examples.

5 THREATS TO VALIDITY

The authors identify threats to external validity from dataset quality and generalizability, and threats to internal validity from hyperparameter sensitivity and deep-learning instability. They mitigate instability by repeating experiments and reporting average performance.

  • External validity: The study’s external validity is constrained by the quality and generalizability of its three public code-generation datasets.The authors regard these datasets as mainstream benchmarks and suggest the findings may extend to other datasets.
  • Internal validity: Hyperparameter sensitivity and stochastic model behavior threaten internal validity, despite shared hyperparameters and five-run average reporting.The authors use a small-range validation grid search and repeat UniCE and other LLM-based metrics five times.

6 DISCUSSION

The authors acknowledge that CodeScore has limitations involving data requirements, scope, and computational cost. They argue these constraints remain acceptable relative to the benefits of more accurate and reliable evaluation.

  • CodeScore requires additional computation and time, but the authors consider this acceptable given its accuracy and reliability benefits.The limitation concerns evaluation cost rather than reported effectiveness.
  • CodeScore is more suitable for evaluating function-level Python code, with broader code scenarios left for future expansion.The authors describe extension to other scenarios as feasible but future work.
  • Learning code execution requires sufficient test cases, generated codes, reference codes, and natural-language descriptions.The authors note that collecting this data is less expensive than human evaluation.

7 CONCLUSION AND FUTURE WORK

The paper presents UniCE and CodeScore for functional code evaluation across three input formats, validates them on three benchmark-derived datasets, and reports state-of-the-art performance. The authors also position the datasets and evaluation approach as resources for future code-generation research.

  • UniCE learns code execution with unified input, enabling CodeScore to evaluate functional correctness in Ref-only, NL-only, and Ref&NL scenarios.This contrasts with traditional CEMs that typically consider only Ref-only inputs.
  • CodeScore achieves state-of-the-art performance on multiple code evaluation datasets derived from APPS, MBPP, and HumanEval.The corresponding evaluation datasets are APPS-Eval, MBPP-Eval, and HE-Eval.
  • The code evaluation dataset can support functional-correctness benchmarking and provide positive feedback for training code-generation models.

A TEST CASE GENERATION VIA CHATGPT

The appendix describes generating test cases for MBPP tasks with ChatGPT in a zero-shot setting. It reports the generation setup and indicates that appropriately guided LLMs can judge the functional correctness of most programs.

  • ChatGPT generates test cases from natural-language descriptions and reference code for randomly selected MBPP tasks, averaging 1.53 test cases per task.The study randomly selects 100 MBPP code-generation tasks.
  • Figure 7 illustrates an example of ChatGPT generating test cases.
  • LLMs show potential to judge the functional correctness of most programs when provided with appropriate guidance.The reported results are associated with the zero-shot ChatGPT test-case-generation setting.
  • Figure 8 presents test-case generation via ChatGPT in a zero-shot setting.
Loading 2301.09043v4…