PaperlayerView exact revision on arXiv

Source-linked AI summary

ReCode: Robustness Evaluation of Code Generation Models

Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, Bing Xiang

arXiv:2212.10264v1cs.LGcs.CLcs.SE

TL;DR

Code-generation models are brittle to realistic prompt changes, while prior robustness research lacked a comprehensive quantitative benchmark for generation. ReCode introduces semantics-preserving transformations and worst-case execution-based metrics, finding robustness differences across models, perturbations, and datasets, including heightened sensitivity to syntax changes.

  • Problem

    Robustness in code generation is underexplored, with no comprehensive quantitative benchmark despite models failing after slight prompt changes.

  • Method

    ReCode evaluates code-generation robustness using over 30 natural, semantics-preserving transformations and worst-case metrics based on executable outputs.

  • Results

    Across HumanEval, MBPP, and derived function-completion tasks, models were most sensitive to syntax perturbations, while CodeGen showed better robustness than InCoder and GPT-J.

  • Takeaways & Limitations

    ReCode provides a multifaceted benchmark for comparing the robustness of code-generation models under realistic prompt variation.

  • Takeaways & Limitations

    The evaluation excludes adversarial attacks requiring model feedback and leaves further architecture-focused ablations for future work.

Abstract

from arXiv · show

Code generation models have achieved impressive performance. However, they tend to be brittle as slight edits to a prompt could lead to very different generations; these robustness properties, critical for user experience when deployed in real-life applications, are not well understood. Most existing works on robustness in text or code tasks have focused on classification, while robustness in generation tasks is an uncharted area and to date there is no comprehensive benchmark for robustness in code generation. In this paper, we propose ReCode, a comprehensive robustness evaluation benchmark for code generation models. We customize over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format. They are carefully designed to be natural in real-life coding practice, preserve the original semantic meaning, and thus provide multifaceted assessments of a model's robustness performance. With human annotators, we verified that over 90% of the perturbed prompts do not alter the semantic meaning of the original prompt. In addition, we define robustness metrics for code generation models considering the worst-case behavior under each type of perturbation, taking advantage of the fact that executing the generated code can serve as objective evaluation. We demonstrate ReCode on SOTA models using HumanEval, MBPP, as well as function completion tasks derived from them. Interesting observations include: better robustness for CodeGen over InCoder and GPT-J; models are most sensitive to syntax perturbations; more challenging robustness evaluation on MBPP over HumanEval.

1 Introduction

ReCode addresses the underexplored brittleness of code-generation models with a comprehensive benchmark built from natural, semantics-preserving prompt transformations and execution-based robustness metrics. Experiments across HumanEval and MBPP reveal model- and perturbation-specific robustness differences.

  • Motivation: Code-generation models can fail after natural prompt edits such as docstring paraphrasing or function-name changes, despite succeeding on nominal prompts.These failures indicate brittleness relevant to real-life deployment and usability.
  • Research gap: No comprehensive quantitative benchmark previously assessed robustness in code generation across realistic perturbations.Prior robustness work largely focused on classification or used limited, impractical settings.
  • Benchmark design: ReCode collects over 30 natural transformations spanning docstrings, function and variable names, code syntax, and code format.The transformations are designed to preserve semantic meaning and provide multifaceted robustness assessments.
  • Validation: Over 90% of perturbed prompts preserve the original semantic meaning according to human evaluation.Quantitative similarity metrics corroborate the quality of the perturbations.
  • Metrics: The framework defines Robust Passs@k, Robust Drops@k, and Robust Relatives@k to quantify worst-case correctness, relative accuracy loss, and instability.The metrics exploit executable generated code as an objective evaluation signal.
  • Empirical findings: Across HumanEval and MBPP, larger models and diverse pretraining can improve worst-case robustness, while syntax perturbations are especially challenging and MBPP induces greater changes.The benchmark compares CodeGen, InCoder, and GPT-J across model sizes and tasks.

2 Related Work

Prior work studied robustness in NLP and code-related tasks, but systematic robustness evaluation for code generation remained limited. Existing efforts primarily addressed classification, adversarial code tasks, or general code-generation performance rather than comprehensive generation robustness.

  • Robustness research: NLP robustness benchmarks largely focus on classification, whereas generation robustness is harder to quantify objectively.This distinction motivates specialized evaluation for generated code.
  • Robustness research: Prior code-robustness studies covered type inference, adversarial examples for code analyses, and source-code comment generation.These works address code robustness but not comprehensive code-generation robustness.
  • Code generation: Large pretrained models such as Codex-12B and CodeGen-16B improved code-generation benchmark performance, but few works systematically explored their robustness.The related work therefore separates strong generation performance from robustness evaluation.

3 Methodology

ReCode evaluates code-generation robustness by applying natural, semantics-preserving perturbations to prompts and measuring worst-case correctness across randomized variants. It covers docstrings, function names, partial code, syntax, and formatting, with metrics designed to quantify robustness under these changes.

  • Evaluation setting: The evaluation targets end-to-end generation or completion from prompts containing descriptions, signatures, helper functions, or partial implementations.The setting is model-architecture agnostic and supports encoder-decoder and decoder-only models.
  • Perturbation design: Perturbations are randomly generated, practice-motivated changes that preserve the original prompt’s semantic meaning rather than adversarial attacks.The benchmark excludes adversarial attacks because their naturalness is difficult to control and they can require greater computation.
  • Docstrings: Docstring perturbations use ten customized character-, word-, and sentence-level transformations, with programming-language keywords protected from unnatural edits.Tree-sitter extracts function, variable, and type names so transformations can exclude them.
  • Function names: Function-name perturbations change naming conventions or apply character- and word-level transformations to component words.Examples include CamelCase conversion, ChangeCharCase, InflectionalVariation, and SynonymSubstitution.
  • Partial code and formatting: Function-completion benchmarks add half of each canonical solution to HumanEval and MBPP prompts before perturbing the partial code.The transformations include dead-code insertion, operand swapping, loop-structure conversion, variable renaming, newline insertion, indentation changes, line splitting, and docstring-to-comment conversion.
  • Evaluation metrics: Robustness is measured over s randomized variants by requiring correct execution across all variants, then aggregating results with Robust Passs@k, Robust Drops@k, and Robust Relatives@k.Robust Passs@k uses worst-case correctness across perturbed prompts, while Robust Relatives@k counts changes between incorrect and correct outcomes.

4 Evaluation

ReCode evaluates worst-case code-generation robustness across HumanEval and MBPP using multiple perturbation categories and model families. The experiments show sensitivity to syntax changes, stronger MBPP difficulty, and trade-offs involving model scale, perturbation strength, and evaluation cost.

  • 4.1 Code Generation Robustness Evaluation: ReCode evaluates CodeGen, InCoder, and GPT-J on HumanEval and MBPP using greedy sampling and five perturbed datasets per transformation.The evaluation aggregates worst-case performance across docstring, function-name, syntax, and format perturbations.
  • 4.1 Code Generation Robustness Evaluation: CodeGen models show better RP5@1 robustness than same-size InCoder and GPT-J models, but worse RD5@1 and RR5@1 relative to nominal performance.The authors associate this pattern with diverse pretraining and possible non-robust generalization.
  • 4.1 Code Generation Robustness Evaluation: 0.174 to 0.217: larger CodeGen-mono models improve average RP5@1 from 2B to 16B, while RR5@1 rises from 27.90% to 35.91%.The results indicate stronger worst-case robustness alongside larger relative performance drops.
  • 4.1 Code Generation Robustness Evaluation: Syntax perturbations produce the largest performance drops across MBPP and HumanEval, revealing a limitation in models’ syntax understanding.This is reported across perturbation types and both datasets.
  • 4.1 Code Generation Robustness Evaluation: Models are more robust on HumanEval than MBPP, whose greater code-style variation makes robustness evaluation more challenging.MBPP includes styles such as one-space indentation and is described as closer to natural code distributions.
  • 4.2 Ablation Study: Larger s strengthens perturbations and expands failure coverage, but evaluation cost grows linearly; the authors recommend s = 5 as a balance.Performance drops begin converging at sufficiently large s.
  • 4.2 Ablation Study: RD@k remains stable while RR@k increases with k under sampling n = 100.The authors attribute this to stable relative ratios for RD and changing perturbed-dataset results as more samples are used.

5 Conclusion

ReCode introduces a comprehensive benchmark for code-generation robustness, combining natural semantic-preserving transformations with worst-case metrics and execution-based evaluation. Human evaluation and similarity scores support perturbation quality across models and datasets.

  • 5 Conclusion: ReCode combines over 30 natural transformations across docstrings, function names, code syntax, and code format with general worst-case robustness metrics.The benchmark is demonstrated on CodeGen, InCoder, GPT-J, HumanEval, MBPP, and derived function-completion tasks.

Ethics Statement

The supplied passages describe ReCode’s practical motivation and its natural, semantic-preserving perturbation design. They do not provide an ethics-specific discussion or ethical risk analysis.

  • Ethics Statement: ReCode is intended to assess vulnerabilities that may cause code-generation mispredictions in practice and inform model improvement.The benchmark is framed as supporting more confident assessment of model predictions.
  • Ethics Statement: The perturbation suite spans docstrings, function names, partial code, syntax, and format while aiming to preserve semantic meaning and natural practice variation.Examples include whitespace changes, paraphrasing, loop rewrites, operand swaps, dead-code insertion, and variable renaming.
  • Ethics Statement: Partial-code transformations keep the same code block before and after perturbation to support fair comparison with nominal baselines.The suite includes dead-code insertion, equivalent loop conversions, operand swaps, and variable-renaming strategies.

B Limitations

ReCode is limited to Python function-completion settings and evaluates robustness rather than improving it, although its transformations may extend to other languages and tasks.

  • ReCode evaluates Python function-completion use cases derived from HumanEval and MBPP, so it does not cover diverse languages or code-completion settings.The benchmark’s perturbed datasets are based on HumanEval and MBPP.
  • The transformations are described as generalizable and extensible to other languages and code-related datasets.
  • ReCode measures robustness but does not itself mitigate a model’s lack of robustness.The authors suggest robust training with perturbed data augmentation as future work.

C Failure Case Study under Perturbations

The failure cases show that simple prompt perturbations can derail CodeGen-16B-mono, especially through inserted code, blank lines, or docstring typos.

  • The showcased failures come from three top perturbations that cause significant performance drops on perturbed HumanEval prompts.
  • DeadCode insertion can mislead predictions, particularly when completion follows the inserted dead code.One example produces only a newline after a meaningless loop, possibly because of an inserted return statement.
  • An empty newline before completion can cause wrong predictions for subsequent if-else conditions.The perturbation is especially effective when the required completion is complicated.
  • ButterFingers typos in docstrings can cause large performance drops by making the model misunderstand the targeted docstring.

D.1 Details for Human Evaluation

Human evaluation used experienced Python software engineers and achieved moderate agreement, though annotator consistency was not perfect, especially for naturalness.

  • Annotators were experienced Python software engineers recruited through a strict coding interview.Annotation trials and clear definitions for naturalness and semantic similarity were used to support reliability.
  • Overall Fleiss Kappa was 0.52, with 0.36 for semantic and naturalness measurements on perturbed samples.The 95% bootstrap confidence intervals were [0.515, 0.528] and [0.358, 0.364].
  • The reported agreement is classified as moderate, while annotator scores are not perfectly consistent, especially for naturalness.

D.2 Sentence Transformers for Docstring/Function Names Similarity

Similarity evaluation combines sentence-transformer cosine scores for text-like elements with CodeBLEU syntax and dataflow scores for code transformations, generally supporting semantic preservation but exposing measurement caveats.

  • Sentence-transformer similarity: Docstring and function-name similarity is measured with sentence-transformer embeddings and cosine similarity, splitting function names into words first.The process uses all-mpnet-base-v2 for docstrings and word-split names for more accurate function-name embeddings.
  • Sentence-transformer similarity: 0.93 and 0.92 average similarity scores are reported for docstrings, versus 0.80 and 0.81 for function names on HumanEval and MBPP.
  • Sentence-transformer similarity: Function-name perturbations such as ButterFinger, SynonymSubstitution, and CharCaseChange can receive lower similarity scores because names lack complete sentence context.The example intersperse to intErspErse scores 0.21, whereas has_close_elements and has_ClosE_Elements score 1.0.
  • CodeBLEU similarity: 77% and 89% of transformations exceed 0.9 on CodeBLEU syntax and dataflow scores, respectively.These scores compare unperturbed and perturbed code and support semantic preservation.
  • CodeBLEU similarity: CodeBLEU scores can be relatively low when transformations intentionally change syntax or dataflow, such as Doc2Comments, Deadcode insertion, and for-while switch.

E.2 Additional Results for Different k

Across different k values, Robust Drop stays stable while Robust Relative increases linearly under the stated sampling setup.

  • E.2 Additional Results for Different k: Robust Drop stays stable across different k values, while Robust Relative increases linearly with k.The additional evaluation uses n = 100 generations with top-p sampling probability 0.95 and temperature 0.2.

E.3 Additional Results for Large Sampling n

Larger sampling n increases evaluation cost linearly but does not cause significant differences in the three robustness metrics. Additional results examine perturbation types, datasets, and metric behavior under n = 100.

  • E.3 Additional Results for Large Sampling n: Larger sampling n increases evaluation cost linearly but does not cause significant differences in the three robustness metrics.The metrics evaluated are Robust Pass1@1, Robust Drop1@1, and Robust Relative1@1 on CodeGen-16B-mono and HumanEval.
  • E.3 Additional Results for Large Sampling n: Robust Drop1@1 remains stable across k, while Robust Relative1@1 increases with k under n = 100 sampling.Figure 22 reports these trends for CodeGen-16B-mono.
Loading 2212.10264v1…