PaperlayerView exact revision on arXiv

Source-linked AI summary

Out of the BLEU: how should we assess quality of the Code Generation models?

Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, Timofey Bryksin

arXiv:2208.03133v2cs.SEcs.LG

TL;DR

Because human evaluation is expensive, code-generation studies rely on metrics whose agreement with human judgments is unclear. This paper compares six metrics across two Python code-generation datasets using model evaluations and human annotators, finding that score differences require dataset-sensitive interpretation and that ChrF is closest to human assessment, though no metric is perfect.

  • Problem

    Human evaluation is expensive, while the agreement between automatic metrics and human judgments for code generation remains unclear.

  • Method

    The study compares six metrics across two Python code-generation datasets and evaluates the generated outputs with human annotators.

  • Results

    ChrF is closest to human assessment; score differences below 5 points on CoNaLa and below 2 points on HearthStone can be unreliable for model comparison.

  • Takeaways & Limitations

    Metric-based claims of model superiority should account for statistical significance and dataset-specific reliability, because no evaluated metric is perfect for code generation.

  • Takeaways & Limitations

    The study's external validity is constrained by its two Python datasets, including a peculiar Card2Code dataset with highly specific class structure.

Abstract

from arXiv · show

In recent years, researchers have created and introduced a significant number of various code generation models. As human evaluation of every new model version is unfeasible, the community adopted automatic evaluation metrics such as BLEU to approximate the results of human judgement. These metrics originate from the machine translation domain and it is unclear whether they are applicable for the code generation tasks and how well they agree with the human evaluation on this task. There are also other metrics, CodeBLEU and RUBY, developed to estimate the similarity of code, that take into account the properties of source code. However, for these metrics there are hardly any studies on their agreement with the human evaluation. Despite all that, minimal differences in the metric scores have been used in recent papers to claim superiority of some code generation models over the others. In this paper, we present a study on the applicability of six metrics -- BLEU, ROUGE-L, METEOR, ChrF, CodeBLEU, and RUBY -- for evaluation of code generation models. We conduct a study on two different code generation datasets and use human annotators to assess the quality of all models run on these datasets. The results indicate that for the CoNaLa dataset of Python one-liners, none of the metrics can correctly emulate human judgement on which model is better with >95% certainty if the difference in model scores is less than 5 points. For the HearthStone dataset, which consists of classes of a particular structure, a difference in model scores of at least 2 points is enough to claim the superiority of one model over the other. Our findings suggest that the ChrF metric is a better fit for the evaluation of code generation models than the commonly used BLEU and CodeBLEU. Yet, finding a metric for code generation that closely agrees with humans requires additional work.

1 INTRODUCTION

Code generation metrics are widely used because human evaluation is expensive, but their suitability and agreement with human judgement remain unclear. This study evaluates six metrics across two datasets and finds that metric differences require cautious interpretation.

  • Motivation: Human evaluation is impractical for every model output, motivating automated metrics for code generation.The study evaluates BLEU, METEOR, ROUGE-L, ChrF, CodeBLEU, and RUBY.
  • Datasets and models: The study compares five models on CoNaLa and two models on HearthStone, covering Python one-liners and rigid card classes.CoNaLa contains short Python solutions, while HearthStone contains classes with highly similar structures.
  • Evaluation: Paired bootstrap resampling assesses corpus-level metric differences, while developers rate whether generated snippets help solve the posed problem.CoNaLa uses 12 developers, averaging 4.5 grades per snippet; HearthStone uses four graders evaluating the entire dataset.
  • Findings: More than 5% of metric comparisons disagree with human judgement for every dataset and metric.The authors therefore conclude that existing metrics are not suitable as straightforward assessments of code-generation quality.
  • Findings: ChrF and ROUGE-L perform best among the evaluated metrics, while score differences below two points are often statistically insignificant.For CoNaLa, differences below five points remain unreliable against human judgement; for HearthStone, the threshold is below two points.

2 BACKGROUND

Code-generation evaluation uses translation metrics, code-oriented metrics, and test-based execution. The paper reviews these approaches and related evidence that metric reliability depends on the task.

  • Evaluation approaches: Code-generation evaluation comprises machine-translation metrics, code-comparison metrics, and running generated code against tests.The paper discusses all three categories as alternatives for assessing generated programs.
  • Machine-translation metrics: BLEU is a corpus-level modified n-gram precision metric originally developed for machine-translated text.It includes a length penalty for candidates shorter than references.
  • Code-oriented metrics: CodeBLEU combines four weighted submetrics that represent generated code as text, abstract syntax trees, and data-flow graphs.RUBY instead compares program dependency graphs, falling back to abstract syntax trees when necessary.
  • Test-based evaluation: Test-based evaluation executes generated code on pre-written unit tests to check whether it solves the task.Applying it requires carefully created datasets for each code-generation setting and enough tests to distinguish models robustly.
  • Related evidence: Prior corpus-level studies found automatic metrics unreliable for small quality differences, with METEOR, BERTScore, and ChrF performing best in related summarization work.BLEU had the highest Type-I error rate regardless of the difference magnitude in that study.

3 MOTIVATION

Automated metrics are attractive because human assessment is costly, but programming-language structure and functionality may not be captured by lexical overlap. The paper motivates testing metric significance and human agreement directly.

  • Why metrics matter: Metrics should align closely with human judgement because they are used to validate and compare models.Human assessment is treated as the quality ground truth, but it is too expensive to apply to every generated output.
  • Limits of testing: Test-based evaluation remains costly because each code-generation setting requires manually crafted tasks and tests.Automated metrics are consequently useful when adopting code generation in new domains.
  • Code versus natural language: Lexical metrics may misrepresent code quality because programming languages have strict syntax and generated code should be judged by functionality.Token rearrangements can invalidate code even when they have limited effect on natural-language meaning.
  • Metric interpretation: A BLEU score of 48.09 accompanies an incorrect CoNaLa snippet, whereas a correct snippet receives BLEU 100.This example illustrates that metric increases are not necessarily linearly related to true code quality.
  • Metric interpretation: Corpus-level averages can conceal different quality distributions when the relationship between metric values and human assessment is nonlinear.The paper therefore argues that score differences and their relationship to human judgement require empirical study.

4 METHODOLOGY

The study compares automated metric differences and human assessments across two code-generation datasets, using corpus-level significance testing and model outputs from prior work. It also limits snippet-level analysis because the collected grades per snippet are below the level argued necessary for stable scores.

  • The study asks whether model differences are significant, how large metric gaps should be, and how corpus-level scores reflect human assessment.
  • Evaluation scope: Snippet-level metric analysis is omitted because the study collected four to 4.5 grades per snippet, below the 15 grades argued necessary for stable scores.
  • Datasets: The evaluation covers CoNaLa Python snippets and Hearthstone classes, whose code differs in length, complexity, and structural rigidity.
  • Models: Five CoNaLa models and two Hearthstone models are evaluated using prior authors’ implementations, hyperparameters, weights, or provided outputs where possible.
  • Metrics and significance: Six metrics—BLEU, METEOR, ROUGE-L, ChrF, CodeBLEU, and RUBY—are compared, with paired bootstrap resampling used to test corpus-level score differences.
  • Synthetic models: Synthetic models augment the original models to provide a broader range of metric values despite the scarcity of publicly available models.

5 RESULTS

Across CoNaLa and Hearthstone, metric-score differences often failed to reliably reflect human judgments or statistical significance. The results support dataset-specific score thresholds and favor ChrF and ROUGE-L over BLEU and CodeBLEU in several comparisons.

  • Cross-dataset findings: Across the studied datasets, metric improvements could be superficial and statistically insignificant, motivating significance tests for code-generation quality comparisons.For all considered metrics, improvements below two points were insufficient to claim statistical significance without additional tests.
  • CoNaLa Dataset: The CoNaLa human ranking was Codex > best-tranx-rerank > best-tranx > tranx-annot > baseline; ChrF, ROUGE-L, and METEOR agreed, while BLEU, RUBY, and CodeBLEU disagreed for at least one pair.All differences in the human model ranking were significant.
  • CoNaLa Dataset: When CoNaLa metric-score differences were below five points, no metric reliably emulated human judgment with at least 95% precision.For differences in the [5, 10) range, only RUBY, ChrF, and ROUGE-L reached at least 95% precision.
  • CoNaLa Dataset: BLEU was the weakest CoNaLa metric for emulating human judgment and sometimes disagreed even when score differences exceeded 10 points.RUBY and CodeBLEU did not perform significantly better than metrics originating in machine translation.
  • CoNaLa Dataset: At least five metric-score points were needed to claim with at least 95% certainty that one CoNaLa model was better, using human judgment as the golden truth.ChrF and ROUGE-L were the best-performing metrics among those considered for code-generation assessment.
  • HearthStone Dataset: For Hearthstone, at least two metric-score points were needed to claim model superiority with at least 95% certainty; ROUGE-L performed best and ChrF second.Differences below two points could not establish significant superiority without additional statistical tests.

6 STUDY IMPLICATIONS

The study recommends cautious interpretation of automated metric differences and supports ChrF as a practical standard, while motivating broader evaluation and new code-specific metrics.

  • The study evaluates BLEU, ROUGE-L, METEOR, ChrF, RUBY, and CodeBLEU for code generation models.
  • Metric scores should be reported with statistical significance information, because small differences may not establish superior model quality.
  • Differences below two points are insufficient to claim that one model is better than another, even when statistically significant.
  • ChrF is recommended as a standard metric for code generation despite BLEU and CodeBLEU being more popular.
  • Extending evaluation across languages, datasets, and models would enable more certain comparisons, although comprehensive human assessment is costly.
  • A promising future metric direction is a BERTScore-like method using large-language-model embeddings to compare reference and candidate snippets.

7 THREATS TO VALIDITY

The study’s validity is constrained by narrow dataset and model coverage, possible sampling bias, and limited or biased human grading, although the authors describe mitigation steps.

  • External validity: The study uses two Python datasets with distinctive structures, limiting generalizability to other datasets and programming languages.
  • External validity: The authors report a shortage of publicly available models, and different model selection could yield different results.
  • External validity: Sampling bias remains an external validity concern because it is unknown whether the findings hold for other languages and Python datasets.
  • Internal validity: Limited grader numbers may make the derived human scores differ from the true human judgments.
  • Internal validity: Grader preferences for coding style or technologies may bias snippet scores, so the study shuffled snippets and included correct examples as mitigation.
  • Overall assessment: The authors consider the listed validity threats tangible but judge their mitigation sufficient for the results to remain usable.

8 CONCLUSION

The paper compares six automated metrics with human assessment across two Python code-generation datasets and finds that small metric improvements may be unreliable. ChrF is closest to human assessment, but no perfect metric is identified.

  • The study examines six metrics on code-generation models evaluated with two Python datasets, checking statistical significance and agreement with human judgment.
  • Less than 2 points of corpus-level metric improvement may not indicate statistically significant quality improvement without additional tests.
  • For some datasets, improvements below 5 points may not correspond to statistically significant improvement according to human judgment.
  • ChrF is the metric closest to human assessment, but it is not a perfect code-generation metric.
  • Future work will extend the study to other datasets and models and assess improvement at the snippet rather than corpus level.

A.1 BLEU

BLEU measures modified n-gram precision with a brevity penalty and is primarily a corpus-level metric. Its score aggregates weighted n-gram overlap while accounting for candidate length.

  • BLEU combines modified n-gram precision with a brevity penalty for candidates shorter than the reference.
  • The brevity penalty uses the reference length and candidate length, while p_n represents weighted n-gram overlap.
  • The standard BLEU weights are w_1 through w_4 equal to 1/4, with higher-order weights set to zero.
  • The original BLEU implementation is corpus-level because it computes micro-averaged precision across hypothesis-reference pairs.
  • SentenceBLEU can score individual hypotheses, but averaging sentenceBLEU scores does not necessarily equal dataset-level BLEU.
  • BLEU scores range from 0 to 1, although practitioners often report them multiplied by 100.
  • The study uses the reference BLEU implementation from sacrebleu.

A.2 ROUGE-L

ROUGE-L evaluates a hypothesis against a reference using their longest common subsequence, combining sequence-based precision and recall. In this study, corpus-level scores are obtained by averaging snippet-level scores, using β=1.

  • ROUGE-L definition: ROUGE-L uses the longest common subsequence between a hypothesis and reference as its basic comparison.The metric was originally proposed for short-text summarization and later adopted for other tasks.
  • ROUGE-L calculation: β=1 gives precision and recall equal weight in the ROUGE-L calculation.ROUGE-L scores range from 0 to 1 and are often multiplied by 100 at corpus level.
  • ROUGE-L calculation: ROUGE-L is commonly computed at snippet level, so corpus-level scores require averaging snippet-level scores.
  • Example: For the example reference, ROUGE-L scores are 0.75 for H1 and 0.5 for H2.H1 shares a longest common subsequence of length 3 with the reference, whereas H2 shares one of length 2.
  • Implementation: The evaluation uses the rouge-score implementation, which produces results identical to the original Perl script.

A.3 ChrF

ChrF is a character-based F-measure that compares character n-grams rather than relying on tokenization rules. The study uses β=2 and the sacrebleu reference implementation.

  • ChrF definition: ChrF computes character-level precision and recall for character k-grams with 1 ≤ k ≤ 6.It considers every character except spaces, making it independent of tokenization rules.
  • ChrF calculation: β=2 is the standard ChrF setting used in this study because it produced the best results in machine-translation tasks.
  • Implementation: The study uses the reference implementation of ChrF from the sacrebleu package.

A.4 METEOR

METEOR aligns hypothesis and reference unigrams, prioritizing broad coverage, few contiguous-match chunks, and similar match positions. It then computes weighted precision and recall with a fragmentation penalty.

  • Alignment: METEOR 1.5 begins by creating an alignment that maps unigrams between the hypothesis and reference.The study uses the latest version of the metric.
  • Alignment: Alignment prioritizes maximizing covered words, minimizing chunks, and minimizing absolute distances between match start indices.A chunk is a contiguous series of matches with identical ordering in both strings.
  • Scoring: After alignment, METEOR separates content and function words and counts the matches covered by each type.
  • Scoring: METEOR computes weighted precision and recall, their weighted harmonic mean, and a fragmentation penalty for gaps and word-order differences.The penalty uses the total number of matched words and the number of chunks.
  • Implementation: The study uses the METEOR implementation from sacrerouge, which wraps the original script.

A.5 RUBY

RUBY measures code similarity through a hierarchy of program-dependence-graph, abstract-syntax-tree, and string-based comparisons. CodeBLEU combines lexical BLEU variants with syntactic AST and semantic data-flow matching, and the study implements both metrics independently where needed.

  • RUBY: RUBY uses graph similarity when PDGs are available, AST similarity when PDGs are unavailable, and string similarity as its final fallback.PDG denotes program dependence graph, AST denotes abstract syntax tree, R is the reference, and C is the candidate.
  • RUBY: RUBY graph similarity normalizes PDG edit distance by the combined sizes of the reference and candidate graphs.Graph edit operations include vertex and edge insertion, deletion, and substitution.
  • RUBY: RUBY AST similarity uses the minimum number of node additions, deletions, replacements, and movements needed to make the ASTs identical.
  • RUBY: RUBY string similarity is based on normalized string edit distance, counting token deletions and additions needed to transform candidate code into reference code.
  • Implementation: The study uses its own implementations of RUBY and CodeBLEU because reference implementations were unavailable or further assessment was required.The authors note that RUBY lacks a reference implementation and that prior CodeBLEU comparisons were limited to BLEU and accuracy.
  • CodeBLEU: CodeBLEU includes usual BLEU, keyword-weighted unigram BLEU, syntactic AST matching, and semantic data-flow matching.AST matching compares reference and hypothesis subtrees, while data-flow matching compares their data-flow graphs.
Loading 2208.03133v2…