PaperlayerView exact revision on arXiv

Source-linked AI summary

Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study

Yujia Fu, Peng Liang, Amjed Tahir, Zengyang Li, Mojtaba Shahin, Jiaxin Yu, Jinfu Chen

arXiv:2310.02059v4cs.SEcs.CR

TL;DR

Prior security research on AI-generated code largely used controlled scenarios, leaving practical open-source settings less examined. This study analyzes AI-generated code used in GitHub projects and reports frequent security weaknesses, while testing Copilot Chat fixes.

  • Problem

    Prior research focused mainly on controlled scenarios, leaving security weaknesses in AI-generated code used in open-source development insufficiently examined.

    Source: §1 · p. 2, §7 · p. 30

  • Method

    The study analyzes AI-generated code in GitHub projects and tests Copilot Chat fixes using static-analysis warnings as prompts.

    Source: §1 · pp. 2–3, §3.3.2 · p. 15

  • Results

    Around 30% of snippets had security weaknesses across 43 CWEs, and an enhanced prompt enabled Copilot Chat to fix 55.5% of security issues.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Developers should manually review high-risk execution and path operations, validate inputs, and use standardized remediation strategies for common weaknesses.

    Source: §5.2.1 · p. 28, §5.2.2 · p. 29

    The GitHub dataset was not sufficiently diverse and contained more snippets from Game projects, potentially leaving security scenarios underrepresented.

    Source: §6 · p. 30

Abstract

from arXiv · show

Modern code generation tools utilizing AI models like Large Language Models (LLMs) have gained increased popularity due to their ability to produce functional code. However, their usage presents security challenges, often resulting in insecure code merging into the code base. Thus, evaluating the quality of generated code, especially its security, is crucial. While prior research explored various aspects of code generation, the focus on security has been limited, mostly examining code produced in controlled environments rather than open source development scenarios. To address this gap, we conducted an empirical study, analyzing code snippets generated by GitHub Copilot and two other AI code generation tools (i.e., CodeWhisperer and Codeium) from GitHub projects. Our analysis identified 733 snippets, revealing a high likelihood of security weaknesses, with 29.5% of Python and 24.2% of JavaScript snippets affected. These issues span 43 Common Weakness Enumeration (CWE) categories, including significant ones like CWE-330: Use of Insufficiently Random Values, CWE-94: Improper Control of Generation of Code, and CWE-79: Cross-site Scripting. Notably, eight of those CWEs are among the 2023 CWE Top-25, highlighting their severity. We further examined using Copilot Chat to fix security issues in Copilot-generated code by providing Copilot Chat with warning messages from the static analysis tools, and up to 55.5% of the security issues can be fixed. We finally provide the suggestions for mitigating security issues in generated code.

1 INTRODUCTION

The paper examines security weaknesses in AI-generated code used in GitHub projects, addressing limited research on security in practical open-source settings. It reports widespread weaknesses and evaluates Copilot Chat as a remediation aid.

1 INTRODUCTION

  • AI code-generation models are trained on public code that can include unsafe patterns, and Copilot may reproduce weaknesses present in a user's codebase.The paper notes that vulnerable generated code may itself be used in later model training, potentially perpetuating the cycle.Source: §1 · p. 2
  • Prior work emphasized correctness more than security, while potential weaknesses in practical open-source scenarios had not been fully addressed.The authors also note that Copilot's users remain responsible for code security and quality.Source: §1 · p. 2
  • The study analyzes code generated by Copilot, CodeWhisperer, and Codeium and used in GitHub projects, applying static analysis and CWE classification.The authors include other tools to increase the research's generalizability.Source: §1 · pp. 2–3
  • Around 30% of snippets had security weaknesses across 43 CWEs, and an enhanced prompt helped Copilot Chat fix 55.5% of security issues.The study also reports that eight CWEs belong to the 2023 CWE Top-25; the fix rate varies by CWE.Source: §1 · p. 3
  • The authors curated a dataset of Copilot-generated GitHub code, categorized its application areas, examined CWEs, and tested Copilot Chat fixes.They present the dataset and analyses as resources for understanding common weaknesses and safer acceptance of suggestions.Source: §1 · p. 3

2 BACKGROUND AND RELATED WORK

The background reviews AI code-generation tools, prior security evaluations, Copilot's interaction patterns, and static analysis. It contrasts controlled or correctness-focused evaluations with the study's examination of code used in open-source development.

2.1 AI Code Generation Tools

  • Earlier evaluations examined code-generation correctness, effectiveness, robustness, and developer experience, but security still had room for improvement.Reported work includes Copilot correctness and productivity studies, as well as findings that developers can struggle to understand, edit, and debug generated snippets.Source: §2.1 · pp. 3–4

2.2 Security of Code Generation Techniques and LLMs

  • Prior security studies used prompted scenarios, model outputs, or test datasets; this study instead examines code from open-source development and scans broadly for CWEs.Unlike Pearce et al., the authors collected code used in developers' GitHub projects and used multiple static-analysis tools to examine all CWE types.Source: §2.2 · pp. 4–5

2.3 How AI Code Generation Tools Work: Copilot as an Example

  • Copilot can complete code, generate suggestions from comments, fill repetitive code, and generate tests; users may accept suggestions or view alternatives.The paper describes 10 alternative suggestions at a time and continued suggestions after a user accepts one.Source: §2.3 · p. 5

2.4 Security Static Analysis

  • Static analysis can cover programs without execution but may be imprecise; the study used CodeQL and supplemented it with language-specific tools.The cited tools include CodeQL, Cppcheck, and Bandit, with different tools used to examine code in different languages.Source: §2.4 · pp. 5–7

3 RESEARCH DESIGN

The research design collects code generated by AI tools in GitHub projects, analyzes security weaknesses, and tests fixes from Copilot Chat. Its first research question asks how secure Copilot-generated code in GitHub projects is.

3 RESEARCH DESIGN

  • The study collects and filters generated-code projects and files, analyzes their functionality and application domains, performs static security analysis, and evaluates Copilot Chat fixes.The analysis results are manually checked for false positives and classified using CWEs; static-analysis warnings are supplied to Copilot Chat, and the fixed code is analyzed again.Source: §3 · p. 7, §3.1 · pp. 7–8

3.1 Research Goal and Questions

  • Because source files do not identify which portions were AI-generated, the study searches repository descriptions and code comments and cannot access all generated code.The authors describe this as a limitation on identifying AI-generated code in GitHub projects.Source: §3.1 · p. 7
  • RQ1 asks how secure Copilot-generated code in GitHub projects is, with CodeWhisperer and Codeium included to increase generalizability.The question concerns the frequency of weaknesses developers encounter when using Copilot in production.Source: §3.1 · p. 8

RQ2. What security weaknesses are present in the code generated by Copilot across different application domains?

This section describes how the study assembled and filtered a GitHub sample of AI-generated code, profiled its projects, and analyzed security warnings, including a Copilot Chat repair test.

RQ2. What security weaknesses are present in the code generated by Copilot across different application domains?

  • The study asks what security weaknesses occur across application domains, using GitHub projects as a source of AI-generated code used in open-source development.GitHub was selected as the primary data source because it contains many public repositories and code resources.Source: RQ2. What security weaknesses are… · p. 8, §3.2 · p. 9

3.2.1 Code Snippets Collection.

  • The search targeted Copilot, CodeWhisperer, and Codeium, and focused on Python and JavaScript because isolated snippets in statically typed languages were difficult to compile.The study also excluded specialized generators such as AlphaCode, which focuses on complex coding tasks rather than everyday code generation.Source: §3.2 · p. 9, §3.2.1 · p. 9

3.2.2 Filtering Code Snippets.

  • Repository results required a project declaration that its code was generated; code results required generation comments, while simple programming exercises were excluded.Two authors manually filtered the data, with Cohen’s Kappa of 0.84; they retained 116 repository-label projects and 335 code-label files.Source: §3.2.2 · pp. 10, 12
  • The resulting dataset comprised 733 distinct code files: 672 from Copilot, 38 from CodeWhisperer, and 23 from Codeium.For repository-label projects declared as fully generated, the study treated the entire file as generated; code-label files were checked using warnings and comments.Source: §3.2.2 · p. 12
  • Projects containing AI-generated code were often small and low-popularity, and the sample covered six domains: Game, Web Application, Utility Tool, AI Application, Network Communication, and Others.The authors noted that Game projects were more numerous in the dataset, which they said could limit the comprehensiveness of represented security scenarios.Source: §3.2.2 · pp. 12–13
  • Collected files had a median of 74 LoC and an average of 181 LoC.The authors described these files as larger than code generated in designed-prompt studies.Source: §3.2.2 · p. 12

3.3.1 Data Analysis.

  • Security scanning used static analysis, with CodeQL plus Bandit for Python or ESLint for JavaScript; static analysis can inspect snippets without executing them.The authors selected static analysis for its coverage and ability to analyze smaller program segments without running the full program.Source: §3.3.1 · pp. 13–14

3.3.2 Results Filtering.

  • Only CodeQL warnings and errors were counted; recommendations were excluded, and analysts manually checked whether reported weaknesses existed and came from generated code.For code-label results, they used warning locations and surrounding comments to identify and verify the generated snippet.Source: §3.3.2 · pp. 15–16

4 RESULTS

Across 733 collected snippets, 27.3% contained security weaknesses spanning 43 CWE types. Copilot Chat fixed 55.5% of issues with the enhanced prompt, but repair success varied by CWE.

4.1 RQ1: How secure is the code generated by Copilot in GitHub projects?

  • More than half of vulnerable snippets had multiple issues; the 200 affected snippets contained 628 weaknesses, averaging three per snippet, with a maximum of 22.The snippet with 22 weaknesses contained 1,595 lines of code.Source: §4.1 · p. 18
  • 27.3% of 733 snippets contained security weaknesses: about 30% of Python snippets and 25% of JavaScript snippets.Table 4 reports 124 of 419 Python snippets and 76 of 314 JavaScript snippets with weaknesses.Source: §4.1 · p. 18, Table 4 · p. 19

4.2 RQ2: What security weaknesses are present in the code generated by Copilot across different application domains?

  • The 628 weaknesses spanned 43 CWE types; eight Top-25 CWEs accounted for 233 weaknesses, with CWE-330 the most frequent.CWE-94, CWE-79, and CWE-78 also appeared among the most frequent weaknesses.Source: Table 5 · p. 20, §4.2 · p. 21
  • CWE patterns varied by language and application domain: Python had the most CWEs in Utility Tool snippets, while JavaScript had the most in Web Applications.In Web Applications, CWE-79 was most frequent in JavaScript and CWE-89 in Python.Source: §4.2 · p. 21, Figure 14 · p. 22, Table 6 · p. 22

4.3 RQ3: Can Copilot Chat fix the security weaknesses of code generated by Copilot?

  • 55.5% of security weaknesses were fixed with the enhanced prompt, compared with 31.8% using the basic prompt and 19.3% using slash-command fix.Enhanced-prompt fix rates were 58.3% for Python and 51.5% for JavaScript security weaknesses.Source: §4.3 · p. 23, Table 7 · p. 24
  • CWE-specific repair varied: Copilot Chat fixed less than 20% of CWE-94 cases and none of CWE-78, while enhanced-prompt fixes reached 100% for CWE-259 and CWE-685.With more prompt information, successful fixes for CWE-79 rose from 0% to 70%, and for CWE-330 from less than 10% to 98%.Source: §4.3 · pp. 23–24

5 DISCUSSION

The discussion finds diverse security weaknesses across generated code and reports that Copilot Chat’s repair effectiveness varies with prompts and weakness types. It recommends continuous security checks, targeted assessment, and combining automated tools with developer review.

5.1 Interpretation of Results

  • Security weaknesses varied by language and application domain; Python snippets had a higher proportion than JavaScript, but the difference was not significant.The discussion associates Python weaknesses with issues such as insufficient randomness and command injection, and JavaScript weaknesses with path traversal and code injection; network communication had a relatively lower proportion.Source: §5.1 · pp. 24–27
  • 628 security weaknesses across 733 snippets involved 43 CWEs, and eight 2023 CWE Top-25 categories accounted for 37.1% of identified weaknesses.The dataset also included 30 weaknesses outside the Top-25; the authors note that even uncommon weaknesses may be exploitable.Source: §5.1 · pp. 25–26
  • Copilot Chat’s repair effectiveness varied by prompt and CWE; the /fix command resolved nearly 20% of security problems, while richer information improved repair outcomes.The discussion reports stronger results for some relatively straightforward issues and says more detailed prompts significantly improved outcomes for certain CWEs, including CWE-330 and CWE-79.Source: Figure 17 · p. 25, §5.1 · p. 27

5.2.1 Importance of continuous security analysis of automatically-generated code.

  • The authors recommend continuous scanning, using Copilot Chat or other LLMs to fix identified weaknesses, re-analysis, and manual security review before code is committed.They suggest a gated check-in build process and emphasize that static-analysis limitations and weakness severity make subsequent manual assessment important.Source: §5.2.1 · p. 28

5.2.2 Prevention of security weaknesses in AI-generated code.

  • Practitioners should target countermeasures to language and domain: for example, prioritize path validation and dynamic execution in JavaScript and system calls and library loading in Python.The CWE Top-25 can guide standardized security audits, alongside established remediation guidance for common weaknesses.Source: §5.2.2 · pp. 28–29
  • Detailed prompts and static-analysis warnings can improve Copilot Chat repairs, but developers should reserve complex logic-control issues for review and design.The discussion distinguishes issues with standard solutions, such as access control and environment configuration, from complex issues involving multi-layered inputs.Source: §5.2.2 · p. 29

6 THREATS TO VALIDITY

The study identifies construct and external validity threats from keyword-based sampling, manual screening and CWE assignment, and the GitHub dataset and static-analysis tools. It also explains why internal validity was not assessed and describes measures intended to support reliability.

Internal Validity

  • The study did not consider internal validity threats because it did not investigate relationships between variables and results.Source: §6 · p. 29

Construct Validity

  • The keyword-based search may miss generated snippets, while manual filtering and CWE assignment may introduce personal bias.Two authors independently screened snippets and assigned CWE IDs; disagreements on CWE assignment were discussed with a security expert.Source: §6 · p. 29

External Validity

  • The GitHub dataset is not sufficiently diverse, with many snippets from game projects, and static analysis may miss CWEs or produce false positives.The authors manually checked scan results and used two tools, but acknowledge that results may remain incomplete.Source: §6 · p. 30

Reliability

  • Repeated static-analysis queries are consistent, and two tools were used in two analysis rounds to complement each other’s security checks.The authors consider these measures to mitigate reliability threats.Source: §6 · p. 30

7 CONCLUSIONS

The study finds security weaknesses in GitHub code snippets generated by Copilot and other tools, and examines Copilot Chat’s ability to fix them. It recommends security checks and identifies broader data collection and tool comparisons as future work.

7 CONCLUSIONS

  • Around 30% of the 733 generated snippets contain security weaknesses.Source: §7 · p. 30
  • The detected weaknesses span 43 CWEs; eight appear in the 2023 CWE Top-25 and six are classified as Stubborn Weaknesses.Source: §7 · p. 30
  • Copilot Chat can fix many security issues, success varies across CWEs, and supplying static-analysis warnings leads to better fixes.Source: §7 · p. 30
  • The authors conclude that developers should apply appropriate security checks when using Copilot or other AI code-generation tools.Source: §7 · p. 30
  • Future work will collect snippets from more repositories and industrial projects, study practitioner use and fixes, and compare emerging generation tools.Planned comparisons include CodeWhisperer, aiXcoder, and Code Llama.Source: §7 · pp. 30–31
Loading 2310.02059v4…