PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code

Shuyan Zhou, Uri Alon, Sumit Agarwal, Graham Neubig

arXiv:2302.05527v2cs.SEcs.LGcs.PL

TL;DR

Reliable evaluation is difficult for NL→Code systems that generate long code outputs. CodeBERTScore extends BERTScore by encoding natural-language context with reference and generated code, and its scores correlate more strongly with human preference and functional correctness than prior metrics.

  • Problem

    Evaluating and comparing long NL→Code outputs reliably remains challenging because existing metrics are sub-optimal.

  • Method

    CodeBERTScore extends BERTScore by encoding natural-language context with reference and generated code using pretrained language-specific code models.

  • Results

    CodeBERTScore is more correlated with human preference and execution correctness than prior metrics across four programming languages.

  • Takeaways & Limitations

    Higher CodeBERTScore values indicate generated code that is more likely to be preferred by humans and function correctly when executed.

  • Takeaways & Limitations

    CodeBERTScore relies on a strong underlying BERT-based model, unlike simpler metrics such as BLEU.

Abstract

from arXiv · show

Since the rise of neural natural-language-to-code models (NL->Code) that can generate long expressions and statements rather than a single next-token, one of the major problems has been reliably evaluating their generated output. In this paper, we propose CodeBERTScore: an evaluation metric for code generation, which builds on BERTScore (Zhang et al., 2020). Instead of encoding only the generated tokens as in BERTScore, CodeBERTScore also encodes the natural language input preceding the generated code, thus modeling the consistency between the generated code and its given natural language context as well. We perform an extensive evaluation of CodeBERTScore across four programming languages. We find that CodeBERTScore achieves a higher correlation with human preference and with functional correctness than all existing metrics. That is, generated code that receives a higher score by CodeBERTScore is more likely to be preferred by humans, as well as to function correctly when executed. We release five language-specific pretrained models to use with our publicly available code. Our language-specific models have been downloaded more than 1,000,000 times from the Huggingface Hub. Our code and data are available at https://github.com/neulab/code-bert-score

1 Introduction

NL→Code models generate increasingly long code, but reliably evaluating and comparing their outputs remains difficult. CodeBERTScore addresses this gap by incorporating natural-language context and pretrained code models into evaluation.

  • Evaluating long NL→Code outputs remains challenging, and existing metrics are sub-optimal.
  • BLEU and CrystalBLEU rely on exact token overlap, limiting their ability to recognize diverse implementations and variable naming.
  • CodeBLEU adds AST and data-flow matching, but correct generations can still receive low scores when their structures differ from the reference.
  • CodeBERTScore encodes code with natural-language instructions or comments, then compares token representations using pretrained code models.
  • In a Java example, CodeBERTScore prefers the functionally equivalent candidate, whereas BLEU and CrystalBLEU prefer a non-equivalent candidate.
  • CodeBERTScore shows higher correlation with human preference and execution correctness than previous approaches across four programming languages.

2 Evaluating Generated Code

The evaluation problem compares generated code with a reference while aiming to rank functionally equivalent and human-preferred candidates higher. CodeBERTScore extends BERTScore by encoding shared natural-language context and using pretrained language-specific code models.

  • A useful metric should rank candidates higher when they are more functionally equivalent to the reference and more preferred by human programmers.
  • BERTScore: BERTScore computes token-level cosine similarities, derives precision and recall from maximum matches, and combines them with harmonic-mean F1.
  • CodeBERTScore: CodeBERTScore encodes the context with both reference and generated code, but excludes context tokens from the final similarity computation.
  • CodeBERTScore: It uses programming-language-specific pretrained models, including CodeBERT models further pretrained with masked language modeling.
  • CodeBERTScore: CodeBERTScore also computes F3, which gives recall more weight than precision.
  • CodeBERTScore: The metric computes cosine similarities between encoded reference and candidate tokens, then uses row and column maxima to calculate precision and recall.

3 Experimental Setup

The experiments test whether CodeBERTScore aligns with human preferences and functional correctness across datasets and programming languages. They compare it with existing metrics using correlation-based evaluation and language-specific pretrained models.

  • Separate language-specific models are pretrained on Python, Java, C++, C, and JavaScript code corpora.
  • The study evaluates CodeBERTScore against existing metrics using human-annotated preferences and execution-based results as ground truth.
  • Correlation is measured with Kendall-Tau, Pearson, and Spearman coefficients.
  • Human preference experiments: Human preference is measured on CoNaLa, a natural-language-to-Python benchmark with human annotations.
  • Functional correctness experiments: Functional correctness is evaluated on HumanEval, which provides natural-language goals, tests, and reference solutions.
  • The experiments tune whether to use F1 or F3 and which model layer supplies token representations, using three-fold cross-validation.

4 Results

CodeBERTScore correlates strongly with both human preference and functional correctness across the evaluated datasets and programming languages. It achieves the highest human-preference correlation and is highest or comparable for functional correctness.

  • Correlation with human preference: CodeBERTScore achieves the highest correlation with human preference across all reported correlation metrics.On CoNaLa, its Kendall-Tau correlation is 0.517, compared with 0.470 for chrF and 0.420 for ROUGE-L.
  • Correlation with human preference: 0.517 Kendall-Tau correlation is achieved by CodeBERTScore, versus 0.470 for chrF and 0.420 for ROUGE-L.These correlations measure agreement with human preference on CoNaLa.
  • Correlation with functional correctness: CodeBERTScore achieves the highest or comparable Kendall-Tau and Spearman correlation with functional correctness across all four languages.METEOR is comparable in Java and JavaScript, while CodeBERTScore is strictly better in C++ and Python.
  • Correlation with functional correctness: CodeBERTScore is more correlated with functional correctness on average across languages than all baseline metrics.The functional-correctness evaluation uses HumanEval across multiple programming languages.

5 Analysis

The analysis examines model choice, transformer-layer choice, context encoding, semantic token matching, and robustness. It finds benefits from language-specific models, deeper but not final layers, natural-language context, and semantic rather than purely lexical alignment.

  • Model choice: Language-specific models are often preferred when available, while CodeBERT-base provides close performance without language-specific pretraining.CodeBERT-base generally performs close to language-specific models, but language-specific models benefit most HumanEval experiments and correlation metrics.
  • Transformer layer choice: Higher transformer layers generally increase average correlation with functional correctness, but performance usually peaks before the final layer.The analysis attributes the later decline to final layers becoming more task-specific.
  • Context encoding: Adding natural-language or programmatic context increases CodeBERTScore’s correlation, with Kendall-Tau rising from 0.50 to 0.52.The paper suggests this context-based approach may benefit other programming scenarios.
  • Soft token matching: CodeBERTScore aligns semantically related tokens across lexically different implementations, such as folder deletion expressions.The heatmaps illustrate token-level similarity between os.rmdir(folder) and shutil.rmtree(folder).
  • Soft token matching: CodeBERTScore gives meaningful non-zero alignments to math.sqrt(x) and x ** 0.5, unlike exact surface-form matching that shares only x.The example matches square-root-related tokens because a square root is the 0.5-th power.
  • Robustness: Token-based metrics are more prone to matching trivial tokens than tokens preserving code semantics under adversarial perturbations.Examples are provided in Appendix E.

6 Related Work

Prior code-generation metrics rely on lexical overlap, static structure, or execution tests, each leaving important evaluation gaps. CodeBERTScore is motivated by the need to capture semantic equivalence without requiring execution datasets.

  • Token-based metrics: Token-based metrics count matching n-grams but often miss semantically equivalent code with different lexical forms.CrystalBLEU removes frequent shared n-grams, but still relies on exact lexical matches.
  • Static analysis-based metrics: CodeBLEU adds AST and data-flow matching, yet valid code can differ structurally and partial code may not parse.Subsequent studies also report that CodeBLEU does not correlate well with execution accuracy.
  • Execution-based metrics: Execution-based evaluation directly tests functionality but requires manually crafted test cases, which are costly and labor-intensive to produce.Consequently, only a few datasets support this evaluation approach.

7 Conclusion

The paper presents CodeBERTScore as a pretrained-model metric that uses natural-language context and evaluates it across four programming languages. It reports stronger correlations with human preference and functional correctness, and releases five language-specific models.

  • Conclusion: CodeBERTScore builds on BERTScore, uses pretrained language models of code, and leverages the generated code’s natural-language context.The metric is presented as a simple evaluation method for code generation.
  • Conclusion: Across four programming languages, CodeBERTScore is more correlated with human preference than all prior metrics.The paper also reports that higher CodeBERTScore is associated with greater likelihood of functional correctness when executed.
  • Conclusion: Five programming language-specific pretrained models were released, and they exceeded 1,000,000 downloads from the HuggingFace Hub.The paper also releases public code and data.

Limitations

CodeBERTScore adds computational and modeling dependencies beyond traditional metrics, notably GPU access and a strong pretrained BERT-based model. Its score scaling is interpretive rather than rank-changing, while F3 emphasizes recall based on empirical findings.

  • Limitations: GPU access is required to compute CodeBERTScore, unlike traditional metrics such as BLEU that require only a CPU.Encoder-only BERT models use a single forward pass, making their additional evaluation time negligible relative to autoregressive generators.
  • Limitations: CodeBERTScore relies on a strong underlying BERT-based model, whereas BLEU has fewer tunable components.The authors characterize this dependency as a potential advantage because stronger base models could improve CodeBERTScore.
  • Additional design choices: F3 weights recall more heavily than precision, and experiments found this choice improves correlation with human preference in NL→Code.The weighting follows the use of Fβ with β = 3 in METEOR.
  • Additional design choices: Token weights use inverse document frequency from a language-specific test set, with each token weighted by its negative log frequency.This follows the token-weighting procedure of BERTScore.
  • Additional design choices: Linear score scaling spreads CodeBERTScore F1 values toward [0, 1] without changing prediction rankings.The empirical base scalars are 0.78 for Java and 0.76 for C++.

B Evaluation Details

The evaluation combines human preference judgments with functional-correctness testing and rank or linear correlation metrics. Correlations compare metric rankings against reference measurements, with rank comparisons restricted to generations for the same question.

  • Human preference evaluation: Human graders assigned zero to irrelevant or unhelpful code and four to code that solves the problem accurately.The grades were collected from experienced software developers evaluating generations from five models.
  • Human preference evaluation: 2,860 generated code snippets were human-graded on a zero-to-four scale, with each snippet assessed by 4.5 annotators.The dataset contains five generations for each of 472 examples.
  • Functional correctness evaluation: HumanEval contains 164 examples with natural-language goals, handwritten input-output tests, and human-written reference solutions.Each example has 7.7 test cases on average, and correctness is binary: one if all tests pass, otherwise zero.
  • Correlation metrics: Kendall-Tau measures ordinal rank association, while Spearman measures correlation between metric-produced ranks and reference-measurement ranks.Kendall-Tau counts concordant and discordant pairs; rank comparisons are restricted to generations of the same question.
  • Correlation metrics: Pearson measures the linear correlation between a metric and the reference measurement.Its formulation uses the number of generations and the means of the metric and reference scores.
  • Reported results: Tables 3 and 4 reproduce the corresponding results from Tables 1 and 2 with standard deviations.

E Robustness to adversarial perturbations

The robustness evaluation tests whether CodeBERTScore distinguishes semantically appropriate code from lexically similar but functionally different alternatives. In the reported example, it favors the appropriate API, while chrF favors a longer character overlap.

  • Qualitative perturbation examples: CodeBERTScore ranks os.rmdir(f) above the non-equivalent (folder) for the reference shutil.rmtree(folder), whereas chrF reverses that preference.The chrF ranking favors the longer shared character sequence despite semantic inequivalence.
  • Qualitative perturbation examples: CodeBERTScore favors code using the appropriate API over a snippet that merely repeats the variable name without a function call.
  • Distinguishability metric: Distinguishability d compares average similarity for intra-cluster and inter-cluster code pairs.Intra-cluster pairs come from semantically equivalent snippets, while inter-cluster pairs come from clusters with different functionality.
  • Distinguishability metric: A distinguishability score d greater than 1 indicates higher similarity for pairs from the same semantic cluster than for pairs from different clusters.Because pair counts grow quadratically, the experiments sample N intra-class and N inter-class pairs.

F.2 Dataset with Semantically equivalent clusters

The semantically equivalent-cluster evaluation compares CodeBERTScore with CrystalBLEU on Java and C++ code pairs, then demonstrates that distinguishability can be inflated without changing the underlying metric.

  • Dataset and evaluation: 6,958 snippets from 278 Java and C++ problems form clusters of semantically equivalent code that passes the platform’s unit tests.Similarity is compared for pairs from the same semantic class and for randomly paired snippets from different classes.
  • Dataset and evaluation: 9.56 versus 5.96 in Java and 9.13 versus 6.94 in C++: CodeBERTScore exceeds CrystalBLEU on distinguishability.CrystalBLEU is the metric that proposed this meta-metric.
  • Dataset and evaluation: CodeBERTScore assigns higher similarity to semantically similar pairs than to randomly paired snippets from different semantic classes.This is the intended interpretation of the distinguishability result.
  • Hacking distinguishability: 120,000 versus 5.96: simple exponentiation can make CodeBERTScore’s distinguishability appear far larger than CrystalBLEU’s.Distinguishability increases almost exponentially with k even though the base CodeBERTScore metric is unchanged.
  • Hacking distinguishability: CodeBERTScore_k is defined by exponentiating the original score: CodeBERTScore_k(y1, y2) = (CodeBERTScore(y1, y2))^k.The variant uses the same distinguishability-evaluation configurations while applying f(x) = x^k.
  • Hacking distinguishability: Distinguishability is therefore not reliable as a meta-metric or a substitute for execution-based or human-rating evaluation.The paper recommends comparing how metrics rank examples rather than comparing exact absolute scores across metrics.
Loading 2302.05527v2…