Source-linked AI summary
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants
Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Siddharth Garg, Brendan Dolan-Gavitt
TL;DR
LLM coding assistants may suggest vulnerable code, but their security impact when developers use them in practice is less established. This paper conducts a randomized study of 58 students implementing a singly linked shopping list in C and finds no greater than a 10% increase in severe security bugs for AI-assisted users. The conclusion is limited to this low-level C setting and its measurement constraints.
Problem
Prior work showed that LLM completions may contain critical security vulnerabilities, leaving the security impact of developer-assisted use insufficiently evaluated.
Method
A randomized user study assigned 58 student programmers to control or Codex-assisted groups implementing a singly-linked shopping-list structure in C, with code assessed for functionality and security.
Results
AI-assisted users produced severe security bugs at a rate no greater than 10% higher than controls in the low-level C setting.
Takeaways & Limitations
In this study context, LLM assistance did not increase the incidence of severe security bugs, despite prior evidence that LLMs can suggest vulnerable code.
Takeaways & Limitations
The findings may differ for other programming languages or tasks, and the cloud IDE captured development snapshots rather than every keypress.
Abstract
from arXiv · showhide
Large Language Models (LLMs) such as OpenAI Codex are increasingly being used as AI-based coding assistants. Understanding the impact of these tools on developers' code is paramount, especially as recent work showed that LLMs may suggest cybersecurity vulnerabilities. We conduct a security-driven user study (N=58) to assess code written by student programmers when assisted by LLMs. Given the potential severity of low-level bugs as well as their relative frequency in real-world projects, we tasked participants with implementing a singly-linked 'shopping list' structure in C. Our results indicate that the security impact in this setting (low-level C with pointer and array manipulations) is small: AI-assisted users produce critical security bugs at a rate no greater than 10% more than the control, indicating the use of LLMs does not introduce new security risks.
1 Introduction
This study asks whether developers using LLM code completion produce less secure code than programmers without assistance, evaluating functionality, security, and bug origins in a randomized student trial.
- Research motivation: The study examines whether AI-assisted programmers write less secure code than programmers without access to an LLM code-completion assistant.It frames this as a key security question because prior work found vulnerabilities in LLM completions, while real users can accept, edit, or reject suggestions.
- Study design: 58 students were randomly split into control and assisted groups to implement a low-level C programming assignment using a Codex-linked development environment.The assignment targeted memory-related weaknesses, which are frequent in C and C++ and represented about 70% of Microsoft-assigned CWEs annually.
- Analysis scope: The analysis evaluates completed code for functionality and security using manual and automated methods, including the Common Weakness Enumeration taxonomy.The study addresses three questions concerning functional quality, security-bug incidence, and the origins of bugs in assisted users’ code.
- Main findings: AI-assisted users produced security-critical bugs at a rate no greater than 10% higher than the nonassisted control group.The authors report minimal security impact in their low-level linked-list-in-C setting and confirm existing productivity benefits of AI assistance.
- Main findings: 63% of bugs originated in human-written code, while 36% were already present in accepted LLM suggestions.The result distinguishes bug origin within the assisted group rather than treating all submitted code as solely LLM-generated.
2 Background and related work
Background work presents LLMs as probabilistic code-completion systems whose suggestions can contain security weaknesses, while prior user studies mainly report productivity effects and broader non-execution risks.
- Related work: Prior user studies generally report positive or faster productivity experiences with Copilot or other LLM-based coding tools, though some findings were not statistically significant.Reported evidence includes quicker starts, more code, positive user perceptions, and reduced iteration duration or context switching.
- How LLMs code: Code LLMs generate probable token sequences from training data and can produce substantial code, including complete function bodies from suitable prompts.Codex uses tokenization and autoregressive prediction to extend user-provided code.
- Security risks: LLM suggestion generation can reproduce insecure training examples and produce code that becomes unsafe when combined with surrounding code.The paper illustrates these risks with deprecated MD5 password hashing and unsafe buffer-use sequences.
- Broader concerns: Security concerns also extend beyond execution to plagiarism, licensing, academic dishonesty, and unresolved legal questions surrounding AI-generated code.These concerns are outside the study’s execution-security scope.
- Security evaluation: Static analysis, runtime sanitizers, and fuzzing provide complementary ways to identify security bugs and vulnerabilities in code.Static analysis checks source patterns, sanitizers require triggering inputs, and fuzzers execute programs on generated inputs to uncover bugs.
3 Design of the security-focused user study
The study randomly compared student programmers completing a security-relevant C programming assignment with and without LLM code suggestions. The assignment and controlled environment emphasized linked-list pointer, array, string, traversal, and file-manipulation tasks where security bugs could arise.
- Analysis: The analysis combined manual, static, and run-time methods to assess functionality and Common Weakness Enumeration security bugs.The study selected C because memory-related errors are frequent and consequential in low-level programming.
- Study design: 58 students were randomly split into control and assisted groups, with only the assisted group receiving Codex suggestions.Both groups completed the assignment in a controlled online development environment and uploaded code for analysis.
- Programming assignment: Participants implemented a singly linked shopping list in C through 11 functions defined by a specified API.The assignment supplied documentation, a README, header files, tests, and an instructional video.
- Programming assignment: The task used position indices, one-indexing, argument pointers, and linked-list operations including adding, updating, removing, and swapping items.These design choices increased pointer manipulation relative to importing and exporting node pointers.
- Programming assignment: Advanced functions required string manipulation, traversal, deduplication, and saving or loading files, increasing opportunities for unintended security-relevant bugs.The externally allocated string in list_item_to_string required avoiding buffer overflows using the documented maximum length.
- Codex Assistant: The assistant parsed the file, sent code before the cursor to Codex, and displayed accept-or-reject completions in an editor extension.Suggestions were presented as faded grey text modeled after GitHub Copilot.
4.1 User population (Demographics)
The study population comprised computer science students with broadly balanced academic enrolment across the control and assisted groups. Most participants had prior C experience and linked-list or data-structures exposure.
- Academic enrolment: 58 participants completed code for analysis, with academic enrolment balanced across the control and assisted groups.The groups included undergraduates, postgraduates, and others such as recent graduates.
- Prior experience: About half of each group reported having previously written a linked list in C.This question assessed whether the assignment resembled prior participant experience.
- Prior experience: Most participants in both groups had written C code before and had previously or currently taken a data structures or algorithms class.These questions were used to validate recruitment goals concerning C and linked-list knowledge.
4.2 RQ1 - Functionality
AI assistance produced a small but consistent functionality advantage over control users, while autopilot generally performed best on implementation and compilation measures. The evaluation separated function-level implementation, compilation, and basic versus expanded test performance.
- Evaluation: 43 expanded tests supplemented 11 basic tests to evaluate edge cases, invalid parameters, return values, and list state.Examples included inserting at the list head, swapping identical positions, NULL pointers, and zero or negative indices.
- Evaluation: 19/58 submissions (32.8%) did not compile, including 9/30 assisted submissions (30%) and 10/28 control submissions (35.7%).
- Evaluation: The split-testing procedure isolated each API function with known-good implementations for the remaining functions, measuring implementation, compilation, basic-test, and expanded-test performance separately.This reduced failures caused by dependencies among functions under test.
- Results: The assisted group had a small but consistent advantage over control, while autopilot outperformed both groups on functions implemented and compiled.Autopilot slightly underperformed assisted users on basic tests but slightly overperformed them on expanded tests.
- Results: 280.9 average lines of code in the assisted group exceeded 247.5 average lines in control, but none of these comparisons reached statistical significance at p < 0.05.The study also reports that assisted users completed their submissions faster.
4.3 RQ2 - Security analysis
The study evaluates security in student-written low-level C code using manual audits and CWE-based metrics. Across aggregate results, assisted users had similar or slightly fewer vulnerabilities than controls, while function-level patterns varied.
- Security analysis: Manual auditing identified 67 unique bug classes across the 58 submitted implementations and five LLM answers examined for comparison.The audit focused on memory-related or undefined bugs associated with CWE categories.
- Metrics: CWEs/LoC was computed over valid functions, using either compilation or compilation plus passing unit tests as validity conditions.A separate Severe CWEs/LoC metric focused on MITRE’s Top 25 security CWEs.
- Topline results: Up to 22% lower mean severe CWEs/LoC was observed for assisted users than controls over functions passing tests.For severe CWEs, non-inferiority tests with δ = 10% found assisted severe bugs/LoC were no more than 10% greater than controls.
- Function-level results: Function-level severe-CWE rates varied across groups, with assisted users higher on some input/output functions and controls higher on functions involving pointer manipulation or complex logic.Differences reaching 10% were highlighted, and statistically significant differences were marked separately.
- CWE observations: CWE-787 was about equally prevalent in assisted and control groups, whereas CWE-416 was more prevalent in the assisted group than the control.The reported CWE-787 pattern was associated primarily with using sprintf rather than snprintf.
- CWE observations: CWE-476 was the most common potential vulnerability because participants often failed to check API arguments and double-pointers against NULL.The API did not guarantee argument correctness, requiring explicit checks.
- Interpretation: Overall, the security impact was less conclusive than functionality, but aggregate CWEs/LoC suggested a slight benefit from LLM assistance.The authors report that code passing tests may be more secure when using the LLM, contrary to prior literature describing vulnerable suggestions.
4.4 RQ3 - On the origin of bugs
The authors traced vulnerabilities in assisted users’ final code to accepted LLM suggestions or human authorship. Humans introduced most identified bugs, while LLM-originated bugs accounted for a substantial minority and could recur through suggestions.
- Bug attribution: A visualization tool matched logged Codex suggestions to final code using reverse chronological exact and approximate matching.Approximate matching allowed normalized edit distances up to 50%, while excluding code from the initial template.
- Bug attribution: 356 bugs, or 63% of the dataset, were introduced by humans, while 36% were introduced by the LLM either verbatim or with modifications.LLM-attributed bugs comprised 16% verbatim suggestions and 20% modified suggestions.
- Bug attribution: 60% of non-template code was written by humans, and the authors report similar vulnerability-introduction rates for LLMs and humans.They relate this similarity to the LLM’s tendency to predict continuations matching the quality of its input.
- CWE-416 tracing: For CWE-416, the analysis searched annotated final files and recorded suggestion snapshots to identify the first occurrence and subsequent appearances of the bug.The same bug could occur in both list_add_item_at_pos and list_update_item_at_pos.
- CWE-416 tracing: CWE-416 usually originated in an LLM suggestion, and the LLM continued suggesting it even when users had introduced it first.Users with the most instances also received and accepted the highest numbers of buggy suggestions.
5 Discussion
The study finds minimal security impact from AI assistance in this setting while corroborating functional benefits, but its student sample, difficult C assignment, and coarse data capture constrain interpretation.
- Functionality (RQ1): Assisted users submitted more lines of code and completed a greater fraction of functions, suggesting enhanced productivity despite productivity not being directly measured.The authors describe this as corroborating prior findings on AI-assistance benefits.
- Security (RQ2): AI-assisted users produced critical security bugs at a rate no greater than 10% higher than the control, with no conclusive evidence of increased CWE incidence.The conclusion applies to this study setting and includes analyses restricted to severe CWEs.
- Bug origins (RQ3): Users sometimes accepted buggy suggestions, later removed accepted bugs, and occasionally ended with more bugs than the LLM suggested.Users who accepted the most LLM bugs also had the most bugs in their final files.
- Threats to validity: The sample comprised university students rather than professional developers, potentially affecting generalizability through differences in behavior and code performance.The authors note prior work found no difference between experienced developers and students in security-aware coding.
- Threats to validity: The unusual singly-linked shopping-list API and C language increased assignment difficulty, so other languages or programming tasks may yield different results.The authors designed these choices to examine how developers might miss bugs in their designs.
- Threats to validity: The cloud-based IDE captured development snapshots every 60 seconds rather than every keypress, limiting possible fine-grained analysis.This measurement constraint prevented capturing all participant data.
6 Conclusions
The paper investigates how LLM code suggestions affect cybersecurity in a user study. With 58 users writing low-level C code, it finds likely functional benefits without increased severe security-bug incidence in this context.
- The study investigates the cybersecurity impact of LLM code suggestions on participants writing code in a user study.
- With N=58 users, the study finds a likely beneficial impact on functional correctness and no increase in severe security-bug incidence for low-level C code with pointer and array manipulations.
- The observed bug origins suggest users do not use extra productivity benefits to fix bugs, motivating research on highlighting problematic lines and improving code security.
Appendix
The appendix documents recruitment and ethical safeguards, describes the study’s CWE reporting conventions, and identifies how interaction and completion-time data were visualized.
- User study recruitment and ethical considerations: The study received NYU IRB approval under #IRB-FY2022-6074.
- User study recruitment and ethical considerations: Participants were recruited from students and ex-students of two authors’ classes, with a firewall separating instructors from participant information.A single investigator who was neither instructor nor supervisor handled enrolled-participant knowledge.
- User study recruitment and ethical considerations: Participants were told they would be randomly assigned to LLM-access and no-access groups and that their code would be compared.Recruitment materials framed the comparison as whether one group would outperform the other.
- User study recruitment and ethical considerations: The study did not deceive participants about security analysis or the LLM, and Codex responses were not artificially modified.
- CWE frequency within each study group: Table 7 reports severe CWE counts per function by study group, defining rate as each CWE count divided by the number of compiling functions.The Autopilot group contains only the first five code-cushman-001 answers.
- CWE frequency within each study group: The visualization distinguishes initial templates, human-written code, exact Codex acceptances, and approximate matches, with suggestions appearing on hover.