Source-linked AI summary
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, Shuai Ma
TL;DR
Code synthesis needs evaluation metrics that capture more than token overlap because code has important syntactic and semantic structure, while exact-match accuracy is overly strict. CodeBLEU combines n-gram, AST, and data-flow matching, and correlates better with programmer scores across three synthesis tasks than BLEU and, on two tasks, accuracy.
Problem
Code-synthesis metrics inadequately capture code quality because n-gram matching ignores syntax and semantics, while perfect accuracy is too strict for semantically equivalent programs.
Method
CodeBLEU combines original and weighted n-gram matching with AST-based syntactic matching and data-flow-based semantic matching.
Results
CodeBLEU correlates better with programmer quality scores than BLEU across text-to-code, code translation, and code refinement, and than accuracy on two tasks.
Takeaways & Limitations
CodeBLEU reliably differentiates code-synthesis systems, with variance within a reasonable range.
Abstract
from arXiv · showhide
Evaluation metrics play a vital role in the growth of an area as it defines the standard of distinguishing between good and bad models. In the area of code synthesis, the commonly used evaluation metric is BLEU or perfect accuracy, but they are not suitable enough to evaluate codes, because BLEU is originally designed to evaluate the natural language, neglecting important syntactic and semantic features of codes, and perfect accuracy is too strict thus it underestimates different outputs with the same semantic logic. To remedy this, we introduce a new automatic evaluation metric, dubbed CodeBLEU. It absorbs the strength of BLEU in the n-gram match and further injects code syntax via abstract syntax trees (AST) and code semantics via data-flow. We conduct experiments by evaluating the correlation coefficient between CodeBLEU and quality scores assigned by the programmers on three code synthesis tasks, i.e., text-to-code, code translation, and code refinement. Experimental results show that our proposed CodeBLEU can achieve a better correlation with programmer assigned scores compared with BLEU and accuracy.
1 Introduction
Existing code-synthesis metrics have important limitations, motivating CodeBLEU, which combines token, syntactic, and semantic information and correlates better with programmer quality scores across three tasks.
- Computational accuracy evaluates whether candidate and reference programs produce the same outputs, but requires language-, compiler-, and resource-specific designs.
- n-gram accuracy can favor candidates with serious logical errors, while perfect accuracy underestimates semantically equivalent outputs.
- CodeBLEU combines original BLEU, weighted n-gram matching, AST-based syntactic matching, and data-flow-based semantic matching.
- Across text-to-code, code translation, and code refinement, CodeBLEU correlates better with programmer quality scores than popularly used BLEU.
2 Why not BLEU?
BLEU is effective for natural-language evaluation but does not capture code’s keyword importance, tree structure, or semantic correctness. CodeBLEU addresses these code-specific differences by incorporating syntactic and semantic information.
- BLEU measures candidate-reference similarity through overlapping n-grams and uses a brevity penalty for very short candidates.
- BLEU is unsuitable for code synthesis because token matching ignores programming-language characteristics, including syntactic accuracy and semantic correctness.
- Code uses a small set of keywords, making keyword matches more important than matches of other words.
- Code has a natural tree structure represented by abstract syntax trees, so evaluating syntactic structure is particularly important.
- Unlike ambiguous natural-language semantics, code contains unique, standardized, and systematic instructions whose semantics can be evaluated.
3 CodeBLEU
CodeBLEU combines token overlap with weighted keywords, AST structure, and data-flow semantics to evaluate code synthesis more faithfully than BLEU alone. Toy examples show it penalizes keyword errors and reduces BLEU’s underestimation of semantically equivalent code.
- CodeBLEU: CodeBLEU is a weighted combination of original BLEU, weighted n-gram match, syntactic AST match, and semantic data-flow match.The weighted n-gram and AST components measure grammatical correctness, while data-flow measures logic correctness.
- 3.1 Weighted N-Gram Match: Weighted n-gram matching assigns higher weights to programming-language keywords than to other tokens.The paper states that keywords receive five times the weights of other tokens, with keywords considered only in unigrams.
- 3.2 Syntactic AST Match: AST matching compares candidate and reference subtrees while omitting leaf nodes, capturing syntactic differences such as missing tokens and data-type errors.AST leaves containing function and variable names are excluded because naming is not treated as syntactic structure here.
- 3.3 Semantic Data-flow Match: Data-flow matching represents variable dependencies as graphs, normalizes variable names, and compares matched candidate flows with reference flows.Graph nodes represent variables, and directed edges represent where each variable’s value comes from.
- Examples: In Example 1, weighted n-gram matching lowers BLEU from 75.43 to 74.91 by penalizing a keyword type error.The example’s final CodeBLEU score is 69.73, and the paper presents such cases as more reasonable than traditional BLEU scores.
4 Experiments
The experiments evaluate CodeBLEU across three code synthesis tasks and compare it with prior evaluation metrics using correlation with programmer scores.
- The experiments cover text-to-code, code translation, and code refinement.
- The tasks use BLEU or perfect accuracy in prior work, while this study evaluates CodeBLEU as a proposed alternative.
- For each task, the study calculates Pearson correlation between CodeBLEU scores and programmer-assigned scores.
4.1 Task Introduction
The experiments span text-to-code, code translation, and code refinement, using task-specific code-generation or code-repair settings and selected comparison systems.
- Task Introduction: The study evaluates text-to-code, code translation, and code refinement.
- Text-to-code: Text-to-code generates class member functions from natural-language documentation and the surrounding class environment.The environment includes member variables with data types and member functions with return types.
- Code Translation: Code translation targets migration between programming languages using paired Java and C# methods from open-source projects.
- Systems: Table 1 selects task-specific systems, including human outputs, GPT-2 variants, and CodeBERT fine-tuning.
- Code Translation: The translation dataset contains 11.8k method pairs, with 0.5k development pairs and 1k test pairs after deduplication.
- Code Refinement: Code refinement automatically fixes buggy Java functions using corresponding fixed functions as targets.The dataset has small and medium subsets based on code length.
4.2 Settings
The evaluation samples test outputs and obtains programmer judgments through a controlled comparison of four systems across Java and C# tasks.
- Evaluation Setup: Each task uses 3 to 4 standard systems, with 500 randomly selected test samples evaluated per task.
- Human Evaluation: Human evaluation involves 10 judges familiar with Java and C# who assess four systems on 50 randomly selected samples.
- Human Evaluation: The evaluation presents 200 input-output pairs through randomly ordered UI software to disperse outputs from the same input.
4.3 Results
CodeBLEU produces reliable scores with reasonable variance and correlates better with human evaluations than BLEU across all three tasks, while outperforming accuracy on two tasks. Its syntactic and semantic components are especially relevant for text-to-code and code translation.
- Reliability and variance: CodeBLEU reliably differentiates code synthesis systems, with variance remaining within a reasonable range.Twenty 25-sentence blocks were used to estimate means, variances, and paired t-statistics; the block variance serves as an upper bound for larger test sets.
- Correlation with human evaluation: CodeBLEU scores are more correlated with human evaluation scores than BLEU across all three tasks and than accuracy on two tasks.Accuracy correlates better for code refinement because outputs are usually unique when fixing small bugs.
- Correlation with human evaluation: CodeBLEU is more linearly correlated with human evaluation scores than BLEU on text-to-code and code translation tasks.Figure 5 reports comparable regressions and R2 values for both metrics.
- Ablation study: AST-match and data-flow-match components are more relevant to human evaluations than n-gram components for text-to-code and code translation.For code refinement, weighted n-gram and semantic data-flow matches are more relevant instead.
4.4 Influence of hyper-parameters
The study varies CodeBLEU’s component weights and finds that emphasizing syntactic and semantic components generally improves correlation with human scores across three tasks, with performance eventually converging.
- Influence of hyper-parameters: Increasing the weights of CodeBLEU’s last two components improves correlation with human scores across all three tasks.The experiments gradually increase these component weights and record correlation coefficients for text-to-code, code translation, and code refinement.
- Influence of hyper-parameters: The combination α, β, γ, δ = 0.1, 0.1, 0.4, 0.4 achieves correlations of 0.981, 0.975, and 0.980 for the three tasks, respectively.Performance begins to converge after combinations [4] and [7].
- Influence of hyper-parameters: For code refinement, α, β, γ, δ = 0.1, 0.4, 0.1, 0.4 reaches a higher correlation of 0.984 than combination [7]’s 0.980.Despite this task-specific result, the authors recommend combination [7] for general code synthesis tasks.
- Influence of hyper-parameters: Figure 6 reports correlation coefficients between CodeBLEU and human scores for different hyper-parameter combinations, whose settings are listed in Table 6.The figure and table together define the evaluated combinations and their outcomes.
5 Related Work
Related work includes automatic metrics for code synthesis, but the paper identifies limitations in BLEU, perfect accuracy, and computational accuracy when evaluating code quality.
- Existing evaluation methods: 4-gram BLEU measures code quality through relative token overlap between hypothesis and reference, but ignores grammatical and logical correctness.This can favor candidates with high n-gram overlap despite serious logical errors.
- Existing evaluation methods: Perfect accuracy is too strict because it underestimates outputs that differ from the reference while preserving semantic equivalence.The metric treats exact matching as the criterion for correctness.
- Existing evaluation methods: Computational accuracy checks whether a hypothesis produces the same outputs as a reference for identical inputs, but lacks universality and practicability.It must be adapted to programming languages, compilers, and available computing resources.
- CodeBLEU: CodeBLEU is presented as addressing these limitations by combining surface matching with grammatical and logical correctness.The paper contrasts this broader evaluation target with the narrower behavior of existing methods.
6 Conclusion
The paper proposes CodeBLEU for code synthesis evaluation and evaluates it on three real-world tasks through correlations with human scores at different granularities.
- Conclusion: CodeBLEU evaluates candidate code using shallow, syntactic, and semantic matching.The conclusion identifies these three matching dimensions as the metric’s central design.
- Conclusion: Results from text-to-code, code translation, and code refinement demonstrate CodeBLEU’s rationality and effectiveness against human evaluation scores.The analysis considers human-score correlation at different granularities.
- Conclusion: Future work will further evaluate syntactic and semantic matching and test CodeBLEU on additional tasks.These directions are intended to examine the metric’s practicality more broadly.