PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot

Antonio Mastropaolo, Luca Pascarella, Emanuela Guglielmi, Matteo Ciniselli, Simone Scalabrino, Rocco Oliveto, Gabriele Bavota

arXiv:2302.00438v1cs.SE

TL;DR

The paper asks whether semantically equivalent changes to natural-language method descriptions alter GitHub Copilot’s generated code, a question relevant to robustness. It evaluates original and manually or automatically paraphrased descriptions across 892 Java methods. ∼46% of cases produced different recommendations, and ∼28% of test-passing methods were obtainable only from one description variant, motivating robustness testing and better description practices.

  • Problem

    The central problem is whether semantically equivalent descriptions can yield different generated functions, which bears on the robustness and usability of DL-based code recommenders.

  • Method

    The study compares Copilot generations from original Javadoc descriptions with manually and automatically generated paraphrases for 892 Java methods.

  • Results

    ∼46% of cases produced different code recommendations when descriptions were semantically equivalent but differently worded.

  • Takeaways & Limitations

    The findings indicate that testing robustness may help ensure usability and define guidelines for developers using DL-based code recommenders.

  • Takeaways & Limitations

    The correctness proxy was limited because passing tests do not imply code correctness, despite high statement coverage and supplementary CodeBLEU and Levenshtein analyses.

Abstract

from arXiv · show

Software engineering research has always being concerned with the improvement of code completion approaches, which suggest the next tokens a developer will likely type while coding. The release of GitHub Copilot constitutes a big step forward, also because of its unprecedented ability to automatically generate even entire functions from their natural language description. While the usefulness of Copilot is evident, it is still unclear to what extent it is robust. Specifically, we do not know the extent to which semantic-preserving changes in the natural language description provided to the model have an effect on the generated code function. In this paper we present an empirical study in which we aim at understanding whether different but semantically equivalent natural language descriptions result in the same recommended function. A negative answer would pose questions on the robustness of deep learning (DL)-based code generators since it would imply that developers using different wordings to describe the same code would obtain different recommendations. We asked Copilot to automatically generate 892 Java methods starting from their original Javadoc description. Then, we generated different semantically equivalent descriptions for each method both manually and automatically, and we analyzed the extent to which predictions generated by Copilot changed. Our results show that modifying the description results in different code recommendations in ~46% of cases. Also, differences in the semantically equivalent descriptions might impact the correctness of the generated code ~28%.

I. INTRODUCTION

The paper examines whether semantically equivalent descriptions produce different GitHub Copilot recommendations, a robustness concern for DL-based code generation. It studies this empirically using original and paraphrased method descriptions, finding frequent changes in recommendations.

  • Motivation: Automated source-code generation has progressed from code completion toward synthesizing entire functions from natural-language descriptions.GitHub Copilot exemplifies this shift by generating functions from signatures and descriptions.
  • Research problem: Semantically equivalent descriptions can raise robustness and usability concerns when they produce different code recommendations.The concern is specifically whether wording changes alter synthesized functions despite unchanged meaning.
  • Research question: The study asks whether different semantically equivalent descriptions of the same function result in different Copilot-generated recommendations.Copilot is selected as a representative DL-based code recommender.
  • Study overview: The dataset contains 892 Java methods selected from 1,401 open-source projects, each with a Javadoc description and contributor-written tests.The first sentence of each Doc Comment serves as the original description.
  • Study overview: The researchers generated semantically equivalent paraphrases manually and with automated techniques, then compared Copilot’s recommendations with those from original descriptions.The analysis focuses on recommendation changes and their impact on prediction quality.
  • Main result: ∼46% of paraphrasing cases changed the code recommendation, and the changes produced substantial variation in correct predictions.The paper concludes that the model’s input plays a central role in its code recommendations.

II. STUDY DESIGN

The study is designed to test Copilot’s robustness to semantically equivalent changes in method descriptions. It includes preliminary evaluation of automated paraphrasing and a main question about description-sensitive recommendations.

  • Study objectives: The study’s overall goal is to understand the robustness of GitHub Copilot as a DL-based code-completion approach.Copilot is evaluated as a state-of-the-art code generation system.
  • Research questions: RQ0 asks whether automated paraphrasing techniques can generate suitable semantically equivalent descriptions for software-engineering text.The preliminary question addresses whether such techniques can be used directly for robustness testing.
  • Research questions: RQ1 asks to what extent Copilot’s output is influenced by the developer-provided code description.It compares recommendations produced from different but semantically equivalent descriptions.

A. Context Selection

The context consists of 892 Java methods drawn from substantial GitHub repositories and selected for executable, documented, sufficiently descriptive examples. The resulting dataset is intended to support correctness assessment through tests and characterize method complexity.

  • Repository selection: The dataset contains 892 Java methods from repositories filtered by at least 300 commits, 50 contributors, and 25 stars.These filters were intended to exclude personal or toy projects.
  • Repository selection: Forked projects were excluded, and the repository-selection process initially produced 1,401 repositories.The study focused on one programming language to simplify the toolchain.
  • Method selection: Projects had to use Maven, compile successfully in their latest release, and support testing so passing or failing tests could assess predictions.The selection yielded 214 repositories before further filtering.
  • Method selection: Methods without Javadoc comments or with fewer than 10 tokens in the first description sentence were excluded.The first sentence of each Doc Comment was used as the method description.
  • Dataset characterization: The final 892 methods were characterized by token count, number of parameters, and cyclomatic complexity.These measures provide an indication of the generation task’s complexity.

B. Data Collection

The researchers created paraphrases with PEGASUS, Translation Pivoting, and manual rewriting, checked their semantic equivalence, and invoked Copilot on files containing the alternative descriptions. They varied the available code context during collection.

  • Paraphrase generation: PEGASUS and Translation Pivoting were evaluated as automated paraphrasing techniques for the study.PEGASUS is a paraphrasing fine-tuned sequence-to-sequence DL model, while Translation Pivoting translates English to French and back.
  • Paraphrase validation: Translation Pivoting failed to produce a valid paraphrase in 100 of 892 cases, whereas PEGASUS failed once.Two authors independently judged 1,683 valid paraphrases for semantic equivalence.
  • Paraphrase generation: For the main experiment, each method received automated paraphrases and a manually written semantically equivalent description.Four authors manually paraphrased assigned subsets of the 892 methods.
  • Copilot invocation: Each Java file was prepared by emptying the target method body and replacing its Doc Comment with one of the descriptions.This created up to four file versions per method.
  • Copilot invocation: The automation opened each file in VS Code, placed the cursor inside the target method, invoked Copilot, and stored the recommendation.The surrounding Java file supplied the code context for prediction.
  • Context scenarios: The study used Full-context and Non-full-context scenarios because contextual information may affect Copilot’s ability to predict the emptied method.The maximum number of invocations was 6,934 across descriptions and context scenarios.

C. Data Analysis

The analysis measures whether paraphrased descriptions differ from originals and whether those differences affect Copilot’s generated methods and their test-based quality.

  • RQ0 evaluates whether automated paraphrases are semantically equivalent before using them to test code-generator robustness.This screening excludes nonequivalent paraphrases from the main robustness analysis.
  • NTLev measures the normalized token-level Levenshtein distance between original and paraphrased descriptions.It estimates the percentage of words that must change to obtain the paraphrase.
  • The analysis compares how often paraphrased descriptions produce different method predictions from the original description.For changed predictions, it also measures token-level distance between the generated methods.
  • Recommendation quality is assessed by whether generated methods pass all related tests, fail at least one test, or fall into another outcome category.The comparison covers both Full context and Non-full context scenarios.

III. RESULTS DISCUSSION

The results assess automated paraphrase quality and their usefulness for robustness testing. The paper reports Full context findings, while comparable Non-full context results are provided in the replication package.

  • The paper discusses Full context results because findings were similar across Full context and Non-full context scenarios.The Non-full context results are available in the replication package.
  • 77% is the highest reported rate of semantically equivalent descriptions, achieved by Translation Pivoting across 892 originals.PEGASUS produced 666 equivalent descriptions (75%), while TP produced 688 (77%).
  • Automated paraphrasing techniques can serve as starting points for testing the robustness of deep-learning code recommenders.A reference description can be paraphrased and the resulting predictions compared.

B. RQ1: Robustness of GitHub Copilot

Semantically equivalent paraphrases frequently changed Copilot’s generated methods, while overall recommendation quality remained similar across original and paraphrased descriptions. The evaluation also exposes limitations in both similarity metrics and testing-based correctness assessments.

  • Performance comparison: Copilot’s overall performance was very similar for original and paraphrased descriptions, including manually, PEGASUS-, and TP-generated paraphrases.Automated analyses included 666 semantically equivalent PEGASUS paraphrases and 688 TP paraphrases.
  • Test-based outcomes: Approximately 73% of original-description cases and 72% of paraphrased-description cases did not yield methods passing the related unit tests.Only approximately 13% of instances produced test-passing methods, although the study involved methods averaging more than 150 tokens and cyclomatic complexity of 5.3.
  • Quality metrics: 25% of test-passing methods had CodeBLEU below 0.50, including a recommendation with CodeBLEU 0.45 that passed its tests while implementing the target’s basic logic differently.The example avoided a redundant conditional and directly performed the removal after checking for null listeners.
  • Evaluation limitations: A test-passing prediction differed from the target by 165 token-level edits and additionally handled 3D points, showing that tests can miss behavioral differences.The example had normalized token-level Levenshtein similarity NTLev=63%.
  • Evaluation limitations: Similarity metrics and tests can disagree with developer-relevant value: low CodeBLEU or high Levenshtein distance may accompany useful recommendations, while passing tests may conceal behavioral differences.The paper therefore questions relying on either metric family alone when evaluating code recommenders.
  • Impact of paraphrasing: 46% of 892 manually paraphrased descriptions produced different Copilot code recommendations than the original descriptions.Among test-passing predictions, 112 came from original descriptions and 122 from paraphrases, with only 98 overlapping.
  • Prediction variation: Different paraphrases changed code substantially: among differing predictions, the median normalized code distance was approximately 30%, while TP altered fewer description words than PEGASUS or manual paraphrasing.The study reports that 50% of manual paraphrases required changes to more than 70% of their words.
  • Answer to RQ1: Approximately 28% of test-passing methods were obtainable only from either the original or paraphrased descriptions, indicating a potential loss of correct recommendations.The authors suggest robustness testing may matter for assessing usability and developing developer guidelines.

IV. THREATS TO VALIDITY

The study’s validity is constrained by proxy-based correctness measurement, automated recommendation acceptance, possible training-data overlap, and a selected Java-method sample.

  • Construct validity: Passing tests were used as a correctness proxy, although they do not imply code correctness.The study also used statement coverage, CodeBLEU, and normalized token-level Levenshtein distance to partially address this threat.
  • Construct validity: The automated script accepted complete recommendations and could not represent developers selecting only parts of Copilot’s output.
  • Internal validity: A single author manually paraphrased descriptions in RQ1, introducing subjectivity bias despite the authors’ Java experience.
  • Internal validity: GitHub projects may overlap with Copilot’s training data, making absolute effectiveness potentially unreliable.The study’s stated objective was comparing paraphrase effects rather than estimating Copilot’s absolute performance.
  • External validity: The experiment covered 892 selected Java methods with high test coverage and verbose first Doc Comment sentences, limiting generalizability.Larger studies and experiments in other languages are needed to corroborate or contradict the findings.

V. RELATED WORK

Software-developer recommender systems support daily activities across documentation, refactoring, triaging, bug fixing, and code development.

  • Recommender systems support practitioners in daily activities including documentation, refactoring, bug triaging, and bug fixing.Code recommenders such as completion tools are also integrated into modern IDEs to speed development by suggesting likely code.

A. Empirical Studies on Code Recommenders

Prior empirical studies examined code recommenders using real-world interactions, synthetic benchmarks, practitioner behavior, hidden costs, developer productivity, and challenging generation tasks; this study adds a robustness perspective.

  • Real-world datasets revealed that synthetic benchmarks can underperform because they miss development context.
  • Users often ignore synthesized suggestions, motivating recommender systems that incorporate development context.
  • Recommendation position can discourage developers from selecting correct solutions when suitable outputs appear far down the list.
  • A controlled experiment found only a marginal productivity gain from using two code recommenders.
  • Transformer models achieved approximately 69% accuracy for classic code completion but faced more challenging whole-block generation scenarios.
  • This study complements prior work by examining robustness in natural-language-to-source-code translation.
  • Semantically equivalent code descriptions can produce different recommendations, raising questions about tool usability.

B. Empirical Studies on GitHub Copilot

Earlier GitHub Copilot studies focused mainly on productivity, effectiveness, correctness, understandability, security, and training-data overlap, whereas this study focuses on robustness to equivalent input descriptions.

  • Copilot studies reported increased lines-of-code productivity but decreased code quality, while another experiment found no improvement in task time or success rate.Developers nevertheless preferred Copilot as a starting point that reduced online-search effort.
  • LeetCode evaluations found correctness differences across languages, ranging from 57% for Java to 27% for JavaScript.Copilot generated low-complexity solutions across the evaluated languages.
  • A security study observed vulnerable code in 40% of Copilot completion scenarios.
  • A separate investigation reported that Copilot rarely recommends verbatim copies of training-set code.

VI. CONCLUSIONS AND FUTURE WORK

The study tested GitHub Copilot’s robustness to semantically equivalent descriptions by generating 892 Java methods from original and paraphrased inputs. Paraphrasing changed recommendations frequently and could affect correctness, underscoring the importance of precise code descriptions while motivating in vivo developer studies.

  • Conclusions and Future Work: 892 non-trivial Java methods were generated by Copilot from their natural-language descriptions and semantically equivalent paraphrases.Paraphrases were produced manually and with automated tools after assessing their reliability for software-related text.
  • Conclusions and Future Work: ∼46% of semantically equivalent description changes produced different code recommendations.The comparison evaluated Copilot’s generated method bodies across original and paraphrased descriptions.
  • Conclusions and Future Work: Some correct recommendations were obtainable only from one of the semantically equivalent descriptions.Thus, changing wording could affect the correctness of the generated method, not merely its surface form.
  • Conclusions and Future Work: The findings highlight the importance of providing proper code descriptions when developers ask deep-learning recommenders to synthesize code.The paper frames description quality as relevant to maximizing AI-supported programming effectiveness.
  • Conclusions and Future Work: Future work will test description effects with developers in controlled experiments and customize paraphrasing techniques for software-related text.These studies would extend the current in silico analysis and improve automated paraphrase generation.
Loading 2302.00438v1…