PaperlayerView exact revision on arXiv

Source-linked AI summary

Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions

Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri

arXiv:2108.09293v3cs.CRcs.AI

TL;DR

Evaluating the security of AI-generated code remains an open problem, so this paper analyzes Copilot with security-relevant prompts and finds that about 40% of generated options are vulnerable.

  • Problem

    Evaluating the security of Copilot’s code contributions remains an open problem beyond functional-correctness metrics.

  • Method

    The study combines automatic and manual analysis of Copilot completions for handcrafted prompts representing security-relevant scenarios.

  • Results

    39.33% of top options and 40.73% of total options were vulnerable across axes and languages.

  • Takeaways & Limitations

    Developers should remain vigilant when using Copilot and pair it with security-aware tooling during training and generation.

  • Takeaways & Limitations

    For unsupported languages and scenarios, vulnerabilities were marked manually, so individual results may be debatable.

Abstract

from arXiv · show

There is burgeoning interest in designing AI-based systems to assist humans in designing computing systems, including tools that automatically generate computer code. The most notable of these comes in the form of the first self-described `AI pair programmer', GitHub Copilot, a language model trained over open-source GitHub code. However, code often contains bugs - and so, given the vast quantity of unvetted code that Copilot has processed, it is certain that the language model will have learned from exploitable, buggy code. This raises concerns on the security of Copilot's code contributions. In this work, we systematically investigate the prevalence and conditions that can cause GitHub Copilot to recommend insecure code. To perform this analysis we prompt Copilot to generate code in scenarios relevant to high-risk CWEs (e.g. those from MITRE's "Top 25" list). We explore Copilot's performance on three distinct code generation axes -- examining how it performs given diversity of weaknesses, diversity of prompts, and diversity of domains. In total, we produce 89 different scenarios for Copilot to complete, producing 1,689 programs. Of these, we found approximately 40% to be vulnerable.

I. INTRODUCTION · II. BACKGROUND AND RELATED WORK · A. Code Generation

The paper examines whether GitHub Copilot commonly produces insecure code and how weakness type, prompt context, and domain affect security. It situates Copilot within ML-based code generation and motivates systematic security analysis beyond prior limited evaluations.

  • I. INTRODUCTION: GitHub released Copilot in June 2021 as an “AI pair programmer” built to improve developer productivity through ML-based code completion.ML code-generation tools train large NLP-derived models on vast quantities of code to provide completions as programmers write.
  • I. INTRODUCTION: The paper addresses the lack of systematic examination of ML-generated code security by asking how commonly Copilot’s suggestions are insecure and what contextual factors affect security.The authors identify Copilot as the largest and most capable such model currently available.
  • I. INTRODUCTION: The study evaluates Copilot against a subset of MITRE’s 2021 CWE Top 25 Most Dangerous Software Weaknesses using handcrafted completion scenarios and security analysis.The analysis combines CodeQL scanning with manual inspection for the applicable weaknesses.
  • I. INTRODUCTION: The experiments vary diversity of weakness, prompt, and domain to measure vulnerability propensity, sensitivity to SQL-injection context, and behavior across programming languages or paradigms.The domain study also considers hardware-specific CWEs, a newer MITRE category that was less frequently represented in Copilot’s training data.
  • I. INTRODUCTION: The paper contributes automatic and manual analyses of Copilot’s software and hardware completions, including how security-relevant prompt context affects generated code and confidence.It also discusses implications for software and hardware designers, especially security novices, using AI pair-programming tools.
  • A. Code Generation: Copilot is based on OpenAI Codex models, which start from GPT-3 and are fine-tuned on GitHub code using a closely related tokenization scheme.The tokenizer retains byte-pair encoding and adds dedicated whitespace tokens to encode source code more efficiently and with more context.
  • A. Code Generation: Prior Codex security evaluation considered only insecure cryptographic parameters, leaving broader common-vulnerability analysis as an open need.The cited examples were short RSA key sizes and AES in ECB mode.
  • A. Code Generation: Because Codex and Copilot generate the most likely completion from training data, generated code may match preceding code rather than optimize for security or other quality metrics.This makes the semantic context of a prompt an important influence on generated-code quality.

B. Evaluating Code Security · III. USING GITHUB COPILOT

The paper evaluates Copilot’s security using automated CodeQL analysis alongside manual inspection, while describing Copilot’s operation as partly undocumented and proprietary. It frames weaknesses through MITRE’s CWE taxonomy and notes that some security properties require analyses beyond source-code inspection.

  • B. Evaluating Code Security: Security evaluation remains an open problem beyond functional-correctness metrics, requiring either human security expertise or security-analysis tools.The paper contrasts compilation, unit tests, and text similarity with the unresolved challenge of evaluating Copilot’s security.
  • B. Evaluating Code Security: The study combines automated GitHub CodeQL analysis with manual code inspection to assess Copilot’s contributions.CodeQL was selected because it can scan a wider range of weaknesses and supports Java, JavaScript, C++, C#, and Python.
  • B. Evaluating Code Security: Common insecure-code patterns are organized as weaknesses in MITRE’s Common Weakness Enumeration database.The CWE taxonomy categorizes weaknesses in a tree-like structure ranging from abstract pillars to specific variants.
  • B. Evaluating Code Security: CWE-20 Improper Input Validation can encompass related class, base, and variant weaknesses when unvalidated input is used for memory allocation.The example connects CWE-20, CWE-1284, and CWE-789 to one input-validation failure and its memory-allocation consequence.
  • B. Evaluating Code Security: Some weaknesses are detectable by static analysis, whereas others require fuzzing, assertions, or reasoning beyond source-code inspection alone.This distinction limits what CodeQL can establish about Copilot-introduced security weaknesses.
  • III. USING GITHUB COPILOT: Copilot’s scanning, prompting, upload decisions, and other exact processes are not publicly documented because the system is proprietary and closed-source.The operational description is therefore based on the authors’ understanding of available documentation.
  • III. USING GITHUB COPILOT: Copilot continuously scans the developer’s program and periodically uploads proprietary subsets of code, cursor position, and metadata before generating completion options.It aims to produce functionally relevant code based on comments, docstrings, function names, and related context.
  • III. USING GITHUB COPILOT: The study leaves Copilot’s secret default generation parameters unchanged because users cannot inspect those defaults.Although temperature, stops, and top p can be overridden, the authors preserve default performance by not changing them.

IV. EXPERIMENTAL METHOD … D. Experimental Platform

The study evaluates whether Copilot-generated snippets contain CWE characteristics, using a binary vulnerability classification that excludes exploitability and correctness. Prompts based on MITRE’s 2021 CWE Top 25 were tested across generated options, primarily through CodeQL and supplemented by manual analysis on a defined software platform.

  • A. Problem Definition: The study classifies Copilot-generated code as vulnerable when it definitively exhibits characteristics of a CWE, without assessing exploitability.The binary outcome records whether code is weak or not, because exploitability may require external context and an attacker model.
  • B. Evaluating Copilot Options with Static Analysis: CodeQL’s Python security-and-quality suite checks 153 security properties and can report vulnerabilities such as user-controlled SQL injection.The example identifies SQL query generation as permitting malicious SQL insertion.
  • C. Generalized Evaluation Process: The prompt dataset, called CWE scenarios, is guided by MITRE’s “2021 CWE Top 25” list to investigate CWE prevalence in Copilot-generated code.Each prompt is passed through Copilot, and the resulting completions are evaluated for the targeted CWE.
  • C. Generalized Evaluation Process: CWE scenarios are small incomplete programs designed so that a naive functional completion could contain a weakness, using Python, C, or Verilog.Python and C are popular, supported by CodeQL, and together can realistically instantiate the complete list of targeted weaknesses.
  • C. Generalized Evaluation Process: Copilot generates up to 25 options per scenario, which are combined with the original snippet after discarding or automatically repairing simple syntax issues.A regex-based tool handles edits such as adding or removing a single brace when they produce compilable output.
  • C. Generalized Evaluation Process: Each resulting program is evaluated for its scenario-specific CWE, using CodeQL built-in or custom queries where possible and manual author analysis otherwise.The evaluation checks vulnerabilities rather than correctness, and CodeQL is configured to examine only the CWE targeted by each scenario.
  • D. Experimental Platform: The methodology ran on one PC with an Intel i7-10750H processor, 16GB DDR4 RAM, and Ubuntu 20.04.Manual and automated steps were divided according to Copilot’s restricted usage patterns.
  • D. Experimental Platform: The implementation used Python 3.8.10, gcc 9.3.0-17, and CodeQL 2.5.7, with manual scenario creation and automated scripts for later processing steps.Manual analysis was retained where necessary, including syntax-related filtering and evaluations requiring additional context.

V. EXPERIMENTAL INVESTIGATION OF GITHUB COPILOT … 2) Individual CWE results (by MITRE Top-25 rank):

The study evaluates Copilot across diverse CWE scenarios using conservative vulnerability labeling, finding 477 of 1,084 valid programs vulnerable. Individual CWE results show substantial variation, including strong performance on some weaknesses and pervasive vulnerabilities in others.

  • A. Study Overview: The analysis examines Copilot across diversity of weakness, diversity of prompt, and diversity of domain axes.The supplied passage introduces these three axes as the study’s framework for evaluating Copilot.
  • 1) Vulnerability Classification:: Only code definitively containing the scenario’s target CWE is marked vulnerable, excluding unfinished completions and other weakness categories.The analysis does not evaluate functional correctness and checks only the specific CWE targeted by each scenario.
  • 1) Overview:: 54 scenarios across 18 CWEs produced 1,084 valid programs, of which 477 (44.00 %) contained a CWE.The scenarios use sources including CodeQL, MITRE examples, and bespoke code; valid programs are syntactically compliant, compilable, and unique.
  • 2) Individual CWE results (by MITRE Top-25 rank):: CWE-787’s top suggestion was vulnerable because 32-character buffers could not safely hold sprintf outputs requiring at least 48 characters.The study examined three C scenarios for CWE-787; the majority and top suggestion for scenario 787-0 were vulnerable, while the other scenarios mostly produced non-vulnerable code.
  • 2) Individual CWE results (by MITRE Top-25 rank):: Copilot performed well on CWE-79 and CWE-125, with all CWE-79 top suggestions and most options for both weaknesses non-vulnerable.For CWE-79, alternative completions for one prompt varied between definitely vulnerable and non-vulnerable, while the top answer was not definitively vulnerable.
  • 2) Individual CWE results (by MITRE Top-25 rank):: CWE-20’s C example was more vulnerable than its two web-themed examples, and CWE-434 was vulnerable for most options in two of three scenarios.Copilot performed well on CWE-434 scenario 434-2 but generated vulnerable code for most 434-0 and 434-1 options.
  • 2) Individual CWE results (by MITRE Top-25 rank):: CWE-78 was the first weakness where most options were vulnerable, while CWE-22 had vulnerable top suggestions in every scenario.The CWE-22 results included definitely vulnerable options for all but one option in 22-0 and all options in 22-2.
  • 2) Individual CWE results (by MITRE Top-25 rank):: Copilot performed poorly on CWE-522, frequently generating MD5 or single-round SHA-256 password hashing, while added environment-variable credentials were theorized to improve code quality.The 522-2 prompt encouraged better password hashing overall, but its top suggestion remained vulnerable.

3) Observations:

Across evaluated scenarios, Copilot generated vulnerable code about 44% of the time, with substantial variation across weaknesses. Among non-vulnerable outputs, scenarios involving permissions, authorization, and buffer bounds achieved the strongest scores.

  • Overall observations: 44% of evaluated Copilot outputs were vulnerable overall.The passage reports this aggregate prevalence across the entire evaluated scenario set.
  • Variation across weaknesses: CWE-79 had 0% vulnerable top-scoring programs and 19% vulnerable options overall, whereas CWE-22 had only vulnerable top-scoring programs.Both CWE-79 and CWE-22 scenarios evaluated programs in C and Python.
  • Non-vulnerable scores: 0.91 was the top-scoring non-vulnerable value for both 732-2 and 306-2, followed by 0.90 for 125-1.CWE-732 and CWE-306 concern permissions and authorization, while CWE-125 concerns buffer over- and under-reads.
  • Non-vulnerable scores: 0.82, 0.78, and 0.76 were the highest mean non-vulnerable scores for 306-2, 416-2, and 79-1, respectively.306-2 was the only scenario appearing in the top three of both non-vulnerable lists.

C. Diversity of Prompt · 1) Overview: · 2) Individual scenario results:

The study examines how small prompt changes affect Copilot’s CWE-89 SQL-injection recommendations. Across meta, comment, and code modifications, vulnerability prevalence and top-option safety changed substantially in several scenarios.

  • 1) Overview:: The CWE-89 prompt-diversity experiment compares control, meta-change, comment-change, and code-change prompts for Python programs.Scenario IDs use the form ‘Type’-‘ID’; language, marker, and origin are fixed or omitted.
  • 2) Individual scenario results:: 6 vulnerable suggestions and 19 non-vulnerable suggestions establish the control baseline, whose top recommendation was non-vulnerable.The control prompt was derived from a shorter version of scenario 89-0.
  • 2) Individual scenario results:: M-1 reduced vulnerabilities, whereas M-2 increased them; changing indentation or adding misspellings also increased vulnerable suggestions, while top options remained non-vulnerable.M-1 used a popular urllib3 author, M-2 used the paper’s lead author, and M-3/M-4 changed formatting or text quality.
  • 2) Individual scenario results:: D-1 produced significantly more vulnerabilities and a vulnerable top option; replacing “remove” with “delete” likewise produced a vulnerable top option.D-1 changed the comment to describe removing an email from database subscriptions.
  • 2) Individual scenario results:: D-3, D-4, D-5, and D-6 generally increased vulnerable suggestions, while D-7’s equivalent docstring placement had negligible impact.D-3 and D-4 produced opposite effects on vulnerable top options: D-3 retained one, whereas D-4 prevented one.
  • 2) Individual scenario results:: C-1 had negligible effect, but adding a separate non-vulnerable database function significantly improved output and eliminated vulnerable suggestions.The separate function also increased Copilot’s confidence score.
  • 2) Individual scenario results:: Making the added function vulnerable skewed outputs toward vulnerability, with only one non-vulnerable answer and a vulnerable top-scoring option; changing libraries had negligible or mixed effects.C-4 was negligible, while C-5 increased both top-option confidence and vulnerable suggestions.

3) Observations:

Copilot generally stayed close to the control scenario in answer confidence and performance, except in C-2 and C-3. The authors hypothesize that existing vulnerable or non-vulnerable SQL most strongly predicts vulnerable SQL injection code generation.

  • Overall observations: Copilot generally did not diverge substantially from the control scenario’s answer confidence and performance.
  • Overall observations: C-2 and C-3 were the two notable exceptions to Copilot’s similarity with the control scenario.
  • SQL injection: The authors hypothesize that vulnerable or non-vulnerable SQL in a codebase is the strongest predictor of other vulnerable SQL and Copilot’s SQL-injection-vulnerable output.

D. Diversity of Domain · 1) Overview:

This section evaluates GitHub Copilot’s security when generating Verilog register-transfer level code, extending the analysis to hardware-specific CWEs. Across six manually analyzable hardware CWEs, 28.28% of generated programs were vulnerable and 38.89% of scenarios had vulnerable top-scoring options.

  • 1) Overview:: The domain axis examines Copilot’s ability to generate register-transfer level code in Verilog for hardware-specific CWEs.Hardware-specific CWEs were added to MITRE’s CWE paradigm in 2020, with more than 100 currently identified.
  • 1) Overview:: Hardware CWEs address hardware implementations and their firmware/software interactions, including dimensions such as timing.They often require additional context or assets beyond the hardware definition itself.
  • 1) Overview:: Hardware-CWE analysis relies on formal verification and manual security evaluation because available tooling remains rudimentary.Security properties can be derived from threat models and checked during multiple hardware-design stages, with linting as one available tool category.
  • 1) Overview:: Because MITRE lacks a hardware CWE Top 25 and automated tooling is limited, the study selected six hardware CWEs for objective manual analysis.The evaluation approach was made similar to manually marked CWEs from the DOW scenarios.
  • 1) Overview:: 18 scenarios were designed, with three scenarios for each of the six selected hardware CWEs.These scenarios formed the basis for evaluating Copilot’s hardware-domain performance.
  • 1) Overview:: 28.28% of 198 generated programs were vulnerable, while 38.89% of 18 scenarios had vulnerable top-scoring options.Copilot generated options for 198 programs across the 18 scenarios.

2) Hardware CWE Results:

Copilot’s hardware-security performance varied substantially by CWE and workload. It generated vulnerable top-scoring or majority outputs for several scenarios, while performing relatively well on FSM tasks but struggling to produce valid Verilog for comparison logic.

  • CWE-1234: Hardware Internal or Debug Modes Allow Override of Locks: For CWE-1234, both top-scoring options for 1234-1 and 1234-2 were vulnerable, as were a majority of the options.The simpler 1234-0 scenario correctly generated the appropriate security check, but quality decreased as workload increased.
  • CWE-1245: Improper Finite State Machines (FSMs) in Hardware Logic: For CWE-1245, no vulnerabilities were generated for 1245-0 or 1245-2, while 1245-1 produced only two vulnerable outputs.Despite this relatively strong performance, the top-scoring option was vulnerable.
  • CWE-1254: Incorrect Comparison Logic Granularity: For CWE-1254, Copilot completed 4 programs for 1254-0, 3 for 1254-1, and no programs for 1254-2.The 1254-1 prompt generated insecure code, and specifying “if every bit” appeared to make logic comparison more difficult to generate.
  • CWE-1271: Uninitialized Value on Reset for Registers Holding Security Settings: For CWE-1271, options for scenario 1271-0 were vulnerable, including the top suggestion.The scenario concerned management of a JTAG lock status register.

3) Observations: · VI. DISCUSSION

Copilot’s security performance was mixed, with roughly 40% of generated options vulnerable, while its reliability varied substantially by language. The discussion attributes these weaknesses to training-data limitations and obsolete practices preserved in open-source code.

  • 3) Observations:: Verilog generation was less syntactically correct and meaningful than Python and C generation.The passage attributes this mainly to Verilog’s smaller training-data quantity and lower popularity.
  • 3) Observations:: Many non-compiling Verilog options used keywords and syntax from other C-type languages, particularly SystemVerilog.Verilog’s syntax resembles other C-type languages, including SystemVerilog.
  • VI. DISCUSSION: 40.73% of total options and 39.33% of top options were vulnerable across all axes and languages.The security of top options is especially important because novice users may be more confident accepting the ‘best’ suggestion.
  • VI. DISCUSSION: Copilot’s security response was mixed across the evaluated scenarios, axes, and languages.The study connects this variable security quality to the nature of the open-source GitHub code used for training.
  • VI. DISCUSSION: Out-of-date security practices can persist in open-source training data and prompt code generation using obsolete approaches.The discussion notes that practices considered best at one time may become bad practice as cybersecurity evolves.
  • VI. DISCUSSION: Password-hashing recommendations progressed from MD5 to salted single-round SHA-256 and later to many rounds or aging libraries such as bcrypt.Unmaintained and legacy code uses insecure hashes, allowing Copilot to continue suggesting them.

Threats to Validity … VII. CONCLUSIONS AND FUTURE WORK

The study’s validity is limited by CodeQL coverage, manual labeling, small and statistically uncertain samples, nonreproducible black-box outputs, and artificial scenarios. Despite these limitations, the authors conclude that developers should remain vigilant and pair Copilot with security-aware tooling.

  • 1) CodeQL Limitations:: CodeQL could not process some CWEs because it sometimes failed to parse important information even from complete codebases.CodeQL performs best when graph-based analysis can identify definitively vulnerable data, such as SQL injection.
  • 1) CodeQL Limitations:: Unsupported languages and scenarios required manual CWE labeling, making some individual results potentially debatable.The authors aimed to remain objective by considering only the relevant CWE definitions.
  • 2) Statistical Validity:: The number of samples per scenario may be insufficient for statistical conclusions, and Copilot’s undefined “mean prob” score complicates statistical claims.Manual querying through Copilot’s interface limited the number of samples collected.
  • 3) Reproducible Code Generation:: Copilot’s generative, black-box, closed-source design and query rate limits hinder reproducible and efficient code-generation experiments.The same prompt can produce different answers at different times, while the remote model cannot be directly examined.
  • 3) Reproducible Code Generation:: The authors archived every option for each prompt because future retraining or updates could change Copilot’s outputs.Their hope that the requested 25 options contained the possible answers was not guaranteed.
  • 4) On scenario creation:: The artificial scenarios and limited context do not fully represent messier real-world software, and subtle prompt variations affect Copilot’s code generation.The experiments covered potential weaknesses across three languages, but real code includes larger surrounding context such as other functions and comments.
  • Disclosures: The study did not identify exploitable vulnerabilities in Copilot itself, so coordinated vulnerability disclosure was unnecessary.The authors evaluated code samples generated through intended use of the tool rather than attacking the product.
  • VII. CONCLUSIONS AND FUTURE WORK: The authors conclude that developers should remain vigilant when using Copilot and recommend security-aware tooling during training and generation.They also state that next-generation autocomplete tools will increase software-developer productivity while rapidly generating large amounts of code.

APPENDIX · Rationale for Excluding Certain CWEs from Analysis · Disclaimer

The study excluded several MITRE Top-25 CWEs when CodeQL could not detect them, substantial context beyond a source file was required, or the issue was architectural rather than code-level. Specific exclusions included ambiguous parent CWEs, cross-component weaknesses, and system-level permissions problems; the authors also disclaim institutional endorsement of their conclusions.

  • Rationale for Excluding Certain CWEs from Analysis: The study omitted CWEs that CodeQL could not be configured to detect, required substantial context beyond the source file, or reflected architectural rather than code-level issues.These criteria guided the exclusion of several MITRE Top-25 weaknesses from scenario design.
  • Rationale for Excluding Certain CWEs from Analysis: CWE-352 and CWE-918 were excluded because assessing them requires analyzing multiple interacting components or languages rather than isolated source code.CWE-352 involves comparing front-end code with web-application behavior, while CWE-918 involves checking URL destinations across upstream and downstream components.
  • Rationale for Excluding Certain CWEs from Analysis: CWE-287 was excluded because its broad identity-verification description made concrete evaluation scenarios difficult, especially as a parent of CWE-306 and CWE-522.The paper characterizes CWE-287 as a class-type CWE covering many different scenarios.
  • Rationale for Excluding Certain CWEs from Analysis: CWE-862 was excluded because missing authorization typically reflects an architectural fault rather than a specific coding error, and CWE-285 was also excluded.CWE-862 concerns absent authorization checks for critical resources or sensitive actions.
  • Rationale for Excluding Certain CWEs from Analysis: CWE-276 was excluded because incorrect default file permissions are a system- or architecture-level installation issue rather than a code-level issue.Poor defaults could allow other computer users to modify software files.
  • Rationale for Excluding Certain CWEs from Analysis: CWE-611 was excluded because determining vulnerability requires significant contextual information and code beyond the analyzed implementation.The weakness concerns XML external entity references resolving outside the intended sphere of control.
  • Rationale for Excluding Certain CWEs from Analysis: CWE-77 was excluded because it is a parent class of CWE-78 and CWE-89, both of which the study analyzed directly.CWE-77 covers insufficient neutralization of user-controlled or upstream command elements.
  • Disclaimer: The authors state that the material’s opinions, findings, conclusions, and recommendations do not necessarily reflect the views of the National Science Foundation or Office of Naval Research.The statement is presented as a disclaimer concerning institutional viewpoints.
Loading 2108.09293v3…