PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeBLEU: a Method for Automatic Evaluation of Code Synthesis

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, Shuai Ma

arXiv:2009.10297v2cs.SEcs.CL

TL;DR

Code-synthesis metrics such as BLEU miss code grammar and logic, while exact accuracy can be too strict for semantically equivalent outputs. CodeBLEU combines n-gram, syntax, and data-flow matches; across three tasks, it correlates better with programmer scores than BLEU, though accuracy correlates better on code refinement.

  • Problem

    Common code-synthesis metrics miss grammatical and logical correctness or are too strict for outputs with the same semantic logic.

    Source: §1 · p. 1

  • Method

    CodeBLEU combines BLEU, weighted n-gram matching, AST subtree matching, and data-flow matching.

    Source: §1 · p. 1

  • Results

    CodeBLEU correlates better with programmer scores than BLEU on all three tasks and better than accuracy on two, while accuracy correlates better on code refinement.

    Source: Text-to-code · p. 6

  • Takeaways & Limitations

    The results support CodeBLEU as a metric for evaluating code synthesis across the three studied tasks.

    Source: §6 · p. 7

    The authors identify testing CodeBLEU on more tasks as future work.

    Source: §6 · p. 7

Abstract

from arXiv · show

Evaluation metrics play a vital role in the growth of an area as it defines the standard of distinguishing between good and bad models. In the area of code synthesis, the commonly used evaluation metric is BLEU or perfect accuracy, but they are not suitable enough to evaluate codes, because BLEU is originally designed to evaluate the natural language, neglecting important syntactic and semantic features of codes, and perfect accuracy is too strict thus it underestimates different outputs with the same semantic logic. To remedy this, we introduce a new automatic evaluation metric, dubbed CodeBLEU. It absorbs the strength of BLEU in the n-gram match and further injects code syntax via abstract syntax trees (AST) and code semantics via data-flow. We conduct experiments by evaluating the correlation coefficient between CodeBLEU and quality scores assigned by the programmers on three code synthesis tasks, i.e., text-to-code, code translation, and code refinement. Experimental results show that our proposed CodeBLEU can achieve a better correlation with programmer assigned scores compared with BLEU and accuracy.

1 Introduction

CodeBLEU combines n-gram, syntactic, and semantic matching to address limitations of existing code-synthesis metrics. Across text-to-code synthesis, code translation, and code refinement, it achieves better correlation with programmer quality scores than BLEU.

1 Introduction

  • Code-synthesis evaluation commonly uses n-gram accuracy, perfect accuracy, and computational accuracy, with n-gram accuracy the most popular approach.N-gram accuracy measures token overlap; perfect accuracy measures the share of predictions exactly matching the ground truth. Examples include 4-gram BLEU (Karaivanov, Raychev, and Vechev 2014; Barone and Sennrich 2017) and perfect accuracy (Chen, Liu, and Song 2018).Source: §1 · p. 1
  • N-gram accuracy can favor logically incorrect candidates, perfect accuracy is too strict, and computational accuracy has limited universality and practicability.Computational accuracy tests whether hypothesis and reference functions produce the same outputs for the same inputs.Source: §1 · p. 1
  • CodeBLEU combines BLEU, weighted n-gram matching, AST subtree matching, and data-flow matching in a weighted score.The weighted n-gram component assigns different weights to different n-grams.Source: §1 · p. 1
  • CodeBLEU achieves better correlation with programmer quality scores than BLEU across text-to-code synthesis, code translation, and code refinement.The experiments also report that CodeBLEU can differentiate systems’ performance.Source: §1 · p. 1

2 Why not BLEU?

BLEU measures reference n-gram overlap, but code’s keyword, tree-structured syntax, and fixed semantics make token matching alone inadequate for code synthesis. These differences motivate CodeBLEU.

2.1 BLEU for Machine Translation

  • BLEU scores candidate translations by reference n-gram overlap and uses a brevity penalty to discourage overly short outputs.Source: §2.1 · p. 2

2.2 Code vs Natural Language

  • Code’s limited keyword vocabulary means keyword matches should receive more weight than other token matches.Source: §2.2 · p. 2
  • Code’s natural tree structure makes syntactic evaluation using its abstract syntax tree important beyond sequential matching.Source: §2.2 · p. 2
  • Code’s unique, standardized instructions make its semantics evaluable, unlike the ambiguous and variable semantics of natural language.Source: §2.2 · p. 2
  • Because token-only BLEU ignores keyword importance, syntactic accuracy, and semantic correctness, the authors propose CodeBLEU.Source: §2.2 · p. 2

3 CodeBLEU

CodeBLEU combines standard BLEU with weighted n-gram, AST, and data-flow matching to assess code overlap, grammatical correctness, and logic correctness. Its examples show how the added matches can penalize code errors or compensate for BLEU’s underestimation of correct variable renaming.

3 CodeBLEU

  • CodeBLEU is a weighted combination of standard BLEU, weighted n-gram match, syntactic AST match, and semantic data-flow match.The weighted n-gram and AST components measure grammatical correctness, while data-flow match measures logic correctness.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Weighted N-Gram Match

  • Keyword-weighted n-gram matching addresses the importance of programming-language keywords, which standard BLEU can overlook.Keywords receive five times the weights of other tokens; keywords are considered only in unigrams, using a predefined list for each language.Source: §3.1 · p. 2

3.2 Syntactic AST Match

  • AST matching compares candidate and reference subtrees while omitting leaf nodes so variable and function names do not affect the syntactic score.The score is the number of matched candidate subtrees divided by the total number of reference subtrees, capturing differences such as missing tokens or data-type errors.Source: §3.2 · p. 3

3.3 Semantic Data-flow Match

  • Data-flow matching captures semantic differences that token overlap and AST similarity can miss, such as a return value drawn from the wrong variable.The method builds directed variable-dependency graphs, normalizes variable names, then divides matched candidate data-flows by the reference total.Source: §3.3 · p. 3

Example 1

  • 75.43 BLEU became 69.73 CodeBLEU in the error example, where keyword, AST, and data-flow mismatches lowered the combined score.The weighted n-gram score was 74.91, AST match was 61.90%, and data-flow match was 66.67%; the candidate contained a type mismatch and a missing token.Source: Example 1 · p. 4, §3.4 · p. 4

Example 2

  • For a candidate differing from its reference only in local-variable names, which the text says is correct, BLEU scored 75.71 versus CodeBLEU’s 88.04, compensating for BLEU’s underestimation of candidate quality.CodeBLEU’s weighted n-gram, syntactic AST, and semantic data-flow match scores were 76.46, 100, and 100, respectively.Source: Example 2 · p. 4

4 Experiments

The experiments assess whether CodeBLEU better reflects programmer judgments across three code-synthesis tasks. They use Pearson correlation with human evaluation scores and include comparisons with BLEU or exact-match accuracy, followed by an ablation study.

4 Experiments

  • Experiments cover Java text-to-code, Java-to-C# code translation, and Java code refinement.These are the three code-synthesis tasks used in the evaluation.Source: §4 · p. 4
  • For each task, Pearson correlation measures how closely CodeBLEU scores align with programmer-assigned human evaluation scores.The study uses this correlation to assess whether CodeBLEU is a more reasonable evaluation metric.Source: §4 · p. 4
  • The experiments examine CodeBLEU against prior task metrics BLEU or exact-match accuracy, then investigate component contributions through ablation.The passage describes these as the planned comparisons and analysis; it does not report their results.Source: §4 · p. 4

4.1 Task Introduction

The evaluation covers text-to-code generation, code translation, and code refinement.

4.1 Task Introduction

  • Text-to-code generates a class member function from its documentation and class context, using the dataset released by Iyer et al. (2018).The dataset contains 100k training, 2k validation, and 2k test samples; inputs include documentation, member variables and types, and member functions and return types.Source: §4.1 · p. 4

Code Translation

  • Code translation evaluates migration between Java and C# methods paired from shared open-source projects.The dataset has 11.8k method pairs, with 0.5k for development and 1k for testing; projects include Lucene, POI, JGit, and Antlr.Source: Code Translation · pp. 4–5

Code Refinement

  • Code refinement targets automatic bug fixing by mapping buggy Java functions to their fixed versions.The Tufano et al. (2019) dataset has small and medium subsets, with training/development/test counts of 46,680/5,835/5,835 and 52,364/6,545/6,545, respectively.Source: Code Refinement · p. 5

4.2 Settings

The evaluation uses standard systems and sampled test data, alongside human ratings of code outputs by judges familiar with Java and C#.

4.2 Settings

  • For each task, the authors prepare 3 to 4 standard systems and randomly select 500 test-set samples for evaluation.The systems are listed in Table 1.Source: §4.2 · p. 5
  • Ten judges familiar with Java and C# rate four systems on 50 randomly selected samples, producing 200 input-output pairs.Each output is rated from 1 (very bad) to 5 (very good).Source: §4.2 · p. 5
  • Judges use the same software and see the input-output pairs in the same randomly ordered sequence, with outputs from each input dispersed.Source: §4.2 · p. 5

4.3 Results

CodeBLEU correlates more strongly with human evaluation than BLEU across three code-synthesis tasks and reliably differentiates systems. Component relevance varies by task, with an exception for accuracy on code refinement.

Main Results

  • Table 2 reports BLEU, exact accuracy, CodeBLEU, and human scores for all baselines across text-to-code, code translation, and code refinement.BLEU, accuracy, and CodeBLEU range from 0 to 100; human evaluation ranges from 1 (very bad) to 5 (very good).Source: Main Results · p. 5, Table 2 · p. 5

Text-to-code

  • CodeBLEU’s score differences are statistically significant between systems, and its variance on 25-sentence blocks is a reasonable upper bound for larger test sets.The authors divided the test set into 20 blocks of 25 sentences; they treat paired t-statistics of 1.7 or above as 95% significant and the block variance as an upper bound for a 500-sentence corpus.Source: Text-to-code · p. 5, Table 3 · p. 6
  • CodeBLEU correlates more strongly with human scores than BLEU on all three tasks, and more strongly than accuracy on text-to-code and code translation, but not code refinement.On code refinement, outputs are usually unique because the task fixes small bugs in a given Java function, which helps explain accuracy’s stronger correlation there.Source: Text-to-code · p. 6, Table 4 · p. 6
  • For text-to-code and code translation, CodeBLEU is more linearly correlated with human scores than BLEU, consistent with the Table 4 comparison.Figure 5 shows regression results for those two tasks and includes the linear-regression R2 values.Source: Figure 5 · p. 6, Text-to-code · p. 6, Table 4 · p. 6

Ablation Study

  • AST and data-flow matches relate more strongly to human scores for text-to-code and code translation, while weighted n-gram and data-flow matches do so for code refinement.The authors suggest refinement’s variable-naming and keyword errors may explain the latter pattern; they propose increasing γ and δ to weight AST and data-flow more heavily in CodeBLEU.Source: Ablation Study · pp. 6–7

4.4 Influence of hyper-parameters

Increasing the weights of CodeBLEU’s last two components improved correlation with human scores across three tasks; the authors recommend combination [7] for general code synthesis.

4.4 Influence of hyper-parameters

  • 0.981, 0.975, and 0.980 were the respective correlations for combination [7] (α, β, γ, δ = 0.1, 0.1, 0.4, 0.4), reported as best across the three tasks.The correlations were measured against human scores.Source: §4.4 · p. 7, Figure 6 · p. 7
  • Gradually increasing the weights of the last two components improved correlations for all three tasks, with performance beginning to converge after combinations [4] and [7].The authors recorded correlation coefficients between CodeBLEU and human evaluation scores for each task.Source: §4.4 · p. 7, Figure 6 · p. 7
  • For code refinement, α, β, γ, δ = 0.1, 0.4, 0.1, 0.4 achieved 0.984 versus 0.980 for [7], yet the authors recommend [7] for general code synthesis.They motivate this recommendation by saying the last two components are more likely to correlate with human evaluation scores.Source: §4.4 · p. 7

5 Related Work

Evaluating code synthesis is important to its development, but existing automatic metrics have limitations. CodeBLEU combines surface matching with grammatical and logical correctness to address those limitations.

5 Related Work

  • As code AI gains attention, evaluating code synthesis is critical, while existing automatic methods remain unsuitable for code (Karaivanov, Raychev, and Vechev 2014; Chen, Liu, and Song 2018; Lachaux et al. 2020).The paper situates this need within growing interest in code AI (Yin and Neubig 2017; Allamanis et al. 2018; Monperrus 2018; Alon et al. 2019; Svyatkovskiy et al. 2020).Source: §5 · p. 7
  • BLEU measures relative token overlap but ignores grammatical and logical correctness, while perfect accuracy underestimates semantic equivalence (Papineni et al. 2002; Rabinovich, Stern, and Klein 2017; Chen, Liu, and Song 2018).The passage identifies 4-gram BLEU as widely used for code quality evaluation (Karaivanov, Raychev, and Vechev 2014; Barone and Sennrich 2017).Source: §5 · p. 7
  • Computational accuracy checks whether a hypothesis generates the same outputs for the same inputs by executing code, but its universality and practicability are limited (Lachaux et al. 2020).CodeBLEU aims to combine BLEU-like surface matching with grammatical and logical correctness.Source: §5 · p. 7

6 Conclusion

CodeBLEU evaluates code synthesis through shallow, syntactic, and semantic matches. Results across three real-world tasks support its effectiveness based on correlation with human evaluation scores at different granularities.

6 Conclusion

  • Across text-to-code, code translation, and code refinement, CodeBLEU’s correlation with human evaluation scores supports its effectiveness.The analysis considers human scores at different granularities.Source: §6 · p. 7
  • CodeBLEU evaluates candidate code using shallow, syntactic, and semantic matches.The paper presents the metric as a novel approach to code synthesis evaluation.Source: §6 · p. 7
  • Future work will further investigate syntactic and semantic match and evaluate CodeBLEU on more tasks.Source: §6 · p. 7

Related papers

Loading 2009.10297v2…