Source-linked AI summary
Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?
Owura Asare, Meiyappan Nagappan, N. Asokan
TL;DR
The paper asks whether Copilot introduces the same vulnerabilities as human developers, and evaluates this through prompts based on documented C/C++ vulnerability scenarios. It finds that Copilot reproduces the vulnerable code in about 33% of cases and the fix in about 25%, concluding that it is not as bad as humans, while noting scope and usage limitations.
Problem
The study asks whether Copilot is as likely as human developers to introduce the same vulnerabilities, because standalone vulnerability rates do not establish the security trade-off of adoption.
Method
The authors prompt Copilot with code fragments from scenarios preceding documented C/C++ vulnerabilities and categorize outputs by whether they reproduce the original vulnerability or fix.
Results
Copilot generated the same vulnerability in 51/153 cases (33.3%) and the fix in 39/153 cases (25.5%), leading authors to conclude it is less likely than humans to generate the same vulnerabilities.
Takeaways & Limitations
Copilot is not as bad as human developers at introducing vulnerabilities, but using it to fix security bugs remains risky because it introduced vulnerabilities in at least a third of studied cases.
Takeaways & Limitations
The findings cannot be generalized to all code-generation tools because different tools use different training datasets and architectures.
Abstract
from arXiv · showhide
Several advances in deep learning have been successfully applied to the software development process. Of recent interest is the use of neural language models to build tools, such as Copilot, that assist in writing code. In this paper we perform a comparative empirical analysis of Copilot-generated code from a security perspective. The aim of this study is to determine if Copilot is as bad as human developers. We investigate whether Copilot is just as likely to introduce the same software vulnerabilities as human developers. Using a dataset of C/C++ vulnerabilities, we prompt Copilot to generate suggestions in scenarios that led to the introduction of vulnerabilities by human developers. The suggestions are inspected and categorized in a 2-stage process based on whether the original vulnerability or fix is reintroduced. We find that Copilot replicates the original vulnerable code about 33% of the time while replicating the fixed code at a 25% rate. However this behaviour is not consistent: Copilot is more likely to introduce some types of vulnerabilities than others and is also more likely to generate vulnerable code in response to prompts that correspond to older vulnerabilities. Overall, given that in a significant number of cases it did not replicate the vulnerabilities previously introduced by human developers, we conclude that Copilot, despite performing differently across various vulnerability types, is not as bad as human developers at introducing vulnerabilities in code.
1 Introduction
The paper asks whether Copilot changes software security relative to human developers and evaluates it through matched vulnerability scenarios. Copilot reproduces human-introduced vulnerable code less often than the original fixes, with performance varying by vulnerability type and age.
- 1 Introduction: The study compares Copilot-generated code with human-developed code to assess whether adopting code-generation tools changes software security.The comparison addresses security trade-offs beyond evaluating Copilot-generated code in isolation.
- 1 Introduction: Copilot is evaluated by recreating vulnerability-introduction scenarios from Big-Vul and categorizing its outputs against the human vulnerability and fix.Prompts are constructed from code fragments preceding documented C/C++ vulnerabilities, then assessed in a two-stage process.
- 1 Introduction: 33% of 152 evaluated samples reproduced the same vulnerable code, while 25% reproduced the corresponding fix.These results indicate that Copilot sometimes avoids vulnerabilities previously introduced by human developers.
- 1 Introduction: Copilot’s behavior varies by vulnerability type and by the age of the vulnerability scenario.It is more likely to generate fixes for more recent vulnerabilities.
2 Background
The background traces language models from statistical and recurrent methods to Transformers, then connects these advances to neural code-generation tools. Copilot uses surrounding program context to propose completions, but likely completion is not necessarily secure code.
- 2 Background: Language models evolved from statistical methods through recurrent neural networks and LSTMs toward Transformers that address sequential-training bottlenecks.RNNs model sequences with repeated weight applications, while LSTMs address long-range dependency problems and Transformers avoid recurrence-related limits.
- 2 Background: Code-generation research shifted from grammar-based and N-gram approaches toward deep-learning methods that model source-code structure.Traditional approaches include domain-specific grammars, probabilistic grammars, and N-gram models.
- 2 Background: Copilot is an AI pair programmer trained on billions of lines of public code that uses surrounding program context to generate possible completions.It is available as a VSCode extension and is presented as an evolved code-completion tool.
- 2 Background: Code-generation tools can account for programming-language syntax, but language-model systems optimize likely completions rather than necessarily secure code.This motivates rigorous security evaluation before widespread use.
3 Research Overview
The research addresses whether Copilot is as likely as human developers to generate the same vulnerabilities. It uses recorded C/C++ vulnerability cases to compare Copilot suggestions with developer-written vulnerable code.
- 3 Research Overview: The central research question is whether Copilot is equally likely to generate the same vulnerabilities as human developers.Prior evidence that Copilot generates vulnerable code did not establish how its security compares with human development.
- 3 Research Overview: The study uses a dataset of C/C++ vulnerabilities previously introduced by software developers and recorded with CVEs.The dataset supplies real cases in which developers introduced vulnerabilities.
- 3 Research Overview: Copilot-generated code is compared with code written by actual developers using vulnerability cases from the curated dataset.The dataset-driven design targets scenarios with documented developer-introduced vulnerabilities.
4 Methodology
The study evaluates Copilot by recreating vulnerability-introduction scenarios from Big-Vul and comparing generated code with the original vulnerability and fix. A two-stage categorization combines exact matching with independent manual recategorization of otherwise unmatched outputs.
- Dataset: The methodology uses Big-Vul, a C/C++ vulnerability dataset whose repository references provide access to vulnerability-fixing commits and preceding human-development scenarios.These references support reconstructing the project state before the vulnerability was introduced.
- Dataset Preprocessing: The study filters samples to single-file, single-contiguous-location changes because Copilot could not coherently combine context across multiple locations.The filtering reduced the dataset from 4,432 samples to 2,226.
- Sample Selection: The researchers selected 153 samples from the filtered subset, prioritizing the most recently published vulnerabilities while limiting manual interaction and analysis effort.The selected samples were drawn from scenarios with single-location changes within a single file.
- Scenario Re-creation: For each sample, researchers created buggy, fixed, and prompt files so Copilot received the project state immediately before the human-introduced vulnerability.Prompt files removed vulnerable lines and subsequent content, or retained content up to where the missing code should have appeared.
- Output Generation: Copilot’s top code suggestion was collected in VS Code, with samples producing only comments or no code excluded from analysis.Files were initially created and edited in Atom to prevent prior exposure to Copilot through the VS Code environment.
- Output Categorization: Outputs were first categorized by exact matches to the original vulnerability or fix, then unmatched outputs were manually recategorized by three independent coders when at least two agreed the code was compilable and similar.The study focused on whether Copilot reproduced the known vulnerability or corresponding fix rather than detecting unrelated vulnerabilities.
5 Results and Discussion
Copilot reproduced the human-introduced vulnerability in 33.3% of evaluated scenarios and the corresponding fix in 25.5%, indicating it was less likely than humans to repeat the same vulnerabilities. Its behavior varied by vulnerability type and vulnerability age, while the study also cautions against relying on it for security fixes without expert review.
- 5 Results and Discussion: 33.3% of scenarios reproduced the human-introduced vulnerability, while 25.5% reproduced the corresponding fix.These results covered 153 scenarios and had a 90% confidence level with a 7% margin of error.
- 5.1 Results Overview: 56.8% of preliminary outputs matched neither the buggy code nor the fixed code, forming Category C.The preliminary categorization found 35 Category A outputs, 31 Category B outputs, and 87 Category C outputs.
- 5.1 Results Overview: 28% of Category C samples were recategorized, with 16 assigned to Category A and 8 to Category B.Recategorization required outputs to be sufficiently close to the original buggy or fixed code.
- 5.1 Results Overview: After recategorization, the effective totals were 51 Category A samples, 39 Category B samples, and 63 Category C samples.Category A represented vulnerable-code matches, Category B fixed-code matches, and Category C remaining outputs.
- 5.2 Code Replication: Recent vulnerability samples showed a higher chance of Category B outputs, while Category A proportions remained relatively constant.The authors report no definitive evidence of memorization or a strong overall preference for Category A or B.
- 5.3 Vulnerability Analysis: Copilot’s security behavior differed across vulnerability types: it produced Category A more often for CWE-20 and CWE-666, but Category B more often for CWE-119, CWE-190, and CWE-476.For CWE-20, Copilot generated Category A outputs 100% of the time; integer overflow and other more easily avoidable vulnerabilities tended to yield more Category B outputs.
- 5.4 Implications for Automated Vulnerability Fixing: The study cautions that Copilot-generated fixes may be risky for vulnerability repair and recommends expert review, especially for nonexpert developers.The authors state that Category A was larger than Category B and that users need expertise to distinguish a fix from a vulnerability.
- 5.5 Implications for Development and Testing of Code Generation Tools: Targeted dataset curation and targeted testing are proposed for improving and assessing code-generation security on vulnerability types where performance is diminished.The proposals focus on increasing examples that avoid specific CWEs and testing those CWEs more frequently after training.
6 Threats to Validity
The study’s validity is constrained by manual security analysis, reconstructed prompts, uneven CWE samples, C/C++-only data, limited generalizability across tools, and Copilot’s nondeterministic prompting behavior.
- Security Analysis: Manual inspection may miss vulnerabilities beyond the original, and approximately 42% of samples were categorized as C, whose vulnerability level remains unclear.The analysis also cannot determine whether other vulnerability types occur in category A or B samples.
- Prompt Creation: Reconstructed prompts omit external files known to human developers, so Copilot may have received different information despite producing fixes in approximately 25% of cases.
- Training Data Replication: Replication patterns may reflect training-data exposure, but inaccessible training data prevents determining whether memorization explains the observations.The reported copying rate of approximately 1% does not fully explain observed replication rates greater than 50%.
- CWE Sample size: Uneven CWE sample counts limit vulnerability-specific analysis, motivating targeted future sampling with more observations per CWE.
- Programming Languages: Because the dataset contains only C and C++ code, the findings may not generalize to substantially different programming languages.
- Other CGTS: Results cannot be generalized to all code-generation tools because architectures and training datasets differ across tools.
- Copilot Performance: Copilot’s nondeterministic behavior and varied prompting make performance across vulnerabilities difficult to assume, especially beyond the autopilot setting evaluated here.Using Copilot as an assistant could produce different results.
7 Related Work
Related work spans language-model development, code-generation evaluation, usability studies, and security analyses, positioning this study within broader research on Copilot and neural code tools.
- Evaluations of Language Models: Codex, a GPT-3 descendant fine-tuned on public GitHub code, solved 28.8% of HumanEval problems, outperforming GPT-3 at 0% and GPT-J at 11.4%.
- Evaluations of Language Models: AlphaCode addressed complex programming problems and achieved an average top-54.3% ranking using larger models, broader competitive-programming data, and expanded sampling.
- Evaluations of Language Models: Comparative evaluations examined open-source models including Codex, GPT-J, GPT-Neo, GPT-NeoX, and CodeParrot, while introducing PolyCoder to address black-box knowledge gaps.
- Code Replication: Studies of code-generation behavior found that short predictions were more likely to clone training data, while WhyGen detected imitations approximately 81% of the time.
- Copilot Studies: Copilot research has evaluated correctness, understandability, synthesis, usability, interaction patterns, and productivity across programming tasks and user studies.Participants used Copilot for acceleration or exploration, and suggestion acceptance correlated with perceived productivity.
- Security Evaluations: Prior security work used incomplete prompts with CodeQL and manual inspection to study Copilot’s tendency to generate insecure code.
8 Conclusion
The study concludes that Copilot introduces vulnerabilities less often than human developers in the evaluated scenarios, while remaining risky for security repair and requiring further investigation.
- Conclusion: Copilot is not as bad as human developers at introducing vulnerabilities, but it generated vulnerabilities in at least one-third of studied cases.
- Conclusion: Copilot was less likely to generate vulnerable code for newer vulnerabilities and more prone to certain vulnerability types, especially those with less simple fixes.
- Conclusion: Using Copilot to fix security bugs remains risky because the study observed vulnerability introduction in at least a third of cases.
- Future Work: Lack of access to Copilot’s training data limits investigation of its behavior and memorization patterns.Open models or access to training data could support longitudinal and memorization studies.
- Future Work: Determining whether assistive tools make code less secure requires comparative user studies with and without code-generation assistance.