Source-linked AI summary
Do Users Write More Insecure Code with AI Assistants?
Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh
TL;DR
AI code assistants may improve access to programming, but their security effects in real user interactions were insufficiently understood. This paper studies 47 participants solving security tasks across languages with and without an AI assistant, finding less secure code and greater overconfidence among AI-assisted participants. It also identifies prompt engagement and interaction behavior associated with more secure solutions and releases study infrastructure for replication.
Problem
The paper addresses limited evidence about security risks from AI code assistants in the context of how users interact with them across security tasks and programming languages.
Method
The authors conducted a user study of 47 participants solving five security-related programming tasks across Python, JavaScript, and C, comparing AI-assisted and control groups while analyzing prompts and interactions.
Results
Participants with AI access generally wrote less secure code and were more likely to judge insecure answers as secure, while greater prompt investment was associated with more secure solutions.
Takeaways & Limitations
AI assistants can provide inexperienced users a false sense of security, so assistant design should account for vulnerabilities and how users prompt and use generated code.
Takeaways & Limitations
The mainly university-student participant group may not represent regular software developers, and the artificial study environment did not perfectly capture real working conditions.
Abstract
from arXiv · showhide
We conduct the first large-scale user study examining how users interact with an AI Code assistant to solve a variety of security related tasks across different programming languages. Overall, we find that participants who had access to an AI assistant based on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without access. Additionally, participants with access to an AI assistant were more likely to believe they wrote secure code than those without access to the AI assistant. Furthermore, we find that participants who trusted the AI less and engaged more with the language and format of their prompts (e.g. re-phrasing, adjusting temperature) provided code with fewer security vulnerabilities. Finally, in order to better inform the design of future AI-based Code assistants, we provide an in-depth analysis of participants' language and interaction behavior, as well as release our user interface as an instrument to conduct similar studies in the future.
1 INTRODUCTION
This paper studies how developers interact with AI code assistants on security-related programming tasks and whether that interaction affects code security. Participants with AI access generally produced less secure code while often feeling more confident in its security, with prompt engagement associated with better outcomes.
- The study examines whether AI access changes code security, whether users trust assistants to write secure code, and how interaction behavior affects vulnerabilities.It involved 47 participants completing five security-related tasks across Python, JavaScript, and C.
- Participants with AI access wrote insecure solutions more often on four of five tasks and typically produced less secure code after controlling for prior security exposure, programming experience, and student status.AI-access participants were also more likely to believe their code was secure than control participants.
- Participants who specified instructions, supplied function declarations, or focused the assistant on helper functions generated more secure code.Using previous assistant outputs as new prompts could magnify or replicate security problems.
- Clear prompt expression and appropriate rephrasing were crucial for producing correct and secure code with the assistant.The study analyzed prompt evolution and parameter adjustments as part of participants’ interaction behavior.
- The authors release experiment data and a reusable user-interface infrastructure to support replication and future code-assistant studies.The interface was designed to help researchers examine user interaction with AI code assistants.
2 BACKGROUND & RELATED WORK
Prior work evaluated AI code assistants mainly on accuracy, usability, correctness, and productivity, while security-focused evidence about user interaction remained limited. This paper extends controlled security analysis across tasks, languages, and adjustable assistant behavior.
- AI code-assistant models were traditionally evaluated on static datasets measuring accuracy for general programming problems.Models generate outputs conditioned on text prompts and hyperparameters such as temperature.
- User studies reported mixed evidence about productivity, usability, and correctness, including limited understanding of generated code and inconsistent productivity effects.These findings suggest productivity may not be the right security objective if it produces misplaced trust or overconfidence.
- Security research has linked copied code and developer misuse of cryptographic APIs to vulnerabilities in real-world applications.Examples include vulnerable Stack Overflow snippets and widespread cryptographic API misuse in Android applications.
- Before this work, only one known controlled user study examined security vulnerabilities in AI-assisted code.That study used Codex-Cushman with fixed parameters and focused on C functions.
- This study differs by examining adjustable parameters and security tasks across multiple languages, finding mixed effects on its C task compared with prior inconclusive results.The comparison motivates evaluating AI-assisted security beyond a single language or fixed model setting.
3 METHODS
The authors conducted a controlled user study in which participants solved security programming tasks with or without an AI assistant using a logged, sandboxed interface. Solutions and interaction traces were manually evaluated for correctness and security.
- 3.1 Building Security-Related Tasks: The study used self-contained tasks covering cryptographic libraries, user-controlled data, web vulnerabilities, and memory-management problems.The tasks were intended to represent security mistakes commonly taught in introductory computer-security courses.
- 3.1 Building Security-Related Tasks: Participants solved six randomized tasks spanning Python, JavaScript, and C, including encryption, signing, path restriction, SQL insertion, integer formatting, and browser output.Participants could attempt tasks in any order, revisit them, install libraries, browse the Internet, and use AI when assigned to the experiment group.
- 3.2 Recruitment and Participant Pool: The researchers recruited 54 participants and retained 47 after prescreening and attrition, with 33 in the AI experiment group and 14 in control.Participants included university students and some professional programmers; assignment targeted a two-to-one experiment-to-control ratio.
- 3.3 Study Instrument: The study instrument provided consent, code writing and execution, output inspection, a sandbox, a two-hour limit, and an AI-query interface for the experiment group.All interactions, prompts, responses, and final code outputs were logged automatically.
- 3.4 Analysis Procedure: Two authors manually identified and severity-ranked correctness and security mistakes, after which two raters coded responses with strong inter-rater reliability.Cohen-Kappa scores ranged from 0.7-0.96 for correctness and 0.68-0.88 for security.
- 3.4 Analysis Procedure: The authors released anonymized user data, prompts, and the interface to enable replication and future studies.The released materials are intended to support research on how users naturally interact with AI code assistants for security tasks.
4 SECURITY ANALYSIS
Across four of five tasks, participants with AI-assistant access produced less secure code than controls, with task-specific vulnerabilities involving cryptography, file paths, SQL, and integer overflow. The analysis combines security classifications with statistical modeling and examines how assistant outputs and user interactions contributed to these outcomes.
- Overall results: Participants with AI access consistently wrote less secure code than controls on four of five questions.The study modeled security outcomes while accounting for prior security exposure, programming experience, and student status.
- Q1: Encryption and decryption: 67% of experiment participants provided correct encryption solutions versus 79% of controls, while experiment participants were significantly more likely to submit insecure solutions.The experiment group also more often used trivial ciphers and omitted authentication of the returned value.
- Q2: Signing a message: 3% of AI-access participants wrote secure signing solutions versus 21% of controls (p= 0.039).Unsafe randomness was the most common experiment-group error, often reflecting the library selected for the task.
- Q3: Sandboxed directory: 12% of AI-access participants wrote secure sandboxed-directory solutions versus 29% of controls, with symlink handling significantly worse in the experiment group (p= 0.019).The assistant often checked whether paths started with “/safedir” without canonicalizing them; correctness did not differ between groups.
- Q4: SQL: AI-access participants provided significantly less secure SQL solutions than controls (36% vs. 50%), including more SQL-injection vulnerabilities (36% vs. 7%; p= 0.041).The assistant often generated correct queries using string concatenation rather than prepared statements.
- Q5: C task: The C task produced mixed security results, but experiment participants were significantly more likely to introduce integer-overflow mistakes.Adjusting temperature, specifying C, and writing function headers improved query success, although non-standard libraries and functions remained common.
5 TRUST ANALYSIS
The study assessed trust through survey responses, participant feedback, and acceptance of AI-generated code. Participants often trusted the assistant despite producing insecure answers, with unfamiliarity and perceived generative ability contributing to that trust.
- Trust was measured through survey ratings, free-response feedback, and copying AI-generated snippets as a proxy for accepting outputs.The proxy varied by question: SQL outputs were copied most often, while C outputs were never directly used.
- Participants with AI access rated their answers as more secure than control participants despite often producing more insecure answers.
- Comments indicate that AI assistance can shift responsibility from writing code toward testing it, which may be risky for users unable to test security vulnerabilities effectively.
- Language unfamiliarity and the assistant’s ability to generate library-based code increased participants’ trust in the AI.
6 PROMPT ANALYSIS
The prompt analysis examines how users’ prompt language, parameters, and iterative repairs relate to trust and code security. Acceptance was associated with code-containing or longer prompts, while secure responses showed less reliance on prior model outputs and more parameter adjustment.
- 6 PROMPT ANALYSIS: The study analyzes how prompt language and parameters influence AI-output acceptance, trust, and the security of participants’ code.
- 6.1 Prompt Language: The prompt taxonomy includes specifications, instructions, questions, function declarations, libraries, language labels, length, text proximity, model proximity, and helper functions.
- 6.1 Prompt Language: 64% of participants tried direct task specification, while 61% used prior model outputs and 53% specified a library in prompts.
- 6.1 Prompt Language: Prompts leading to accepted outputs were more likely to contain code, and long prompts were accepted more often than short prompts, 42.7% versus 15.7%.
- 6.1 Prompt Language: Among accepted outputs from participants who submitted secure answers, incorporating previous AI responses declined sharply, whereas Function Declaration, Specification, and Helper remained common.
- 6.2 Prompt Parameters: For most questions, participants with secure AI-sourced responses used higher final-prompt temperatures than those with insecure responses, while non-adjusters were more likely to provide insecure code.
- 6.3 Repair Strategies: Participants made an average of 4.6 queries per question, and changes in prompt type were the most common repair strategy.
- 6.3 Repair Strategies: Participants who proactively adjusted parameters and re-phrased prompts were more likely to provide correct and secure code.
7 DISCUSSION
The discussion cautions that AI assistants may increase security risks for inexperienced users, while user prompting and assistant design can affect vulnerability outcomes. The study’s generalizability is limited by its participants and artificial experimental setting.
- AI assistants may give inexperienced developers a false sense of security while increasing the risk of introduced vulnerabilities.
- Degree of AI Influence on Responses: AI assistant involvement accounted for at least as many security mistakes as participants across every mistake type and often the majority.
- Limitations: The mainly university-student participant group may not represent regular AI-assistant users such as software developers.
- Limitations: The study’s artificial environment, including time constraints and non-job-impacting performance, restricts generalization to real working conditions.
- Recommendations: The study suggests prompt refinement, secure cryptographic defaults, user education, integrated warnings, validation tests, and security-filtered training data as design directions.
8 CONCLUSION
The conclusion reports a user study of AI-assisted security programming across languages and emphasizes both increased vulnerability risk and the importance of how users interact with the assistant. It also highlights released data and interface infrastructure for future research.
- The study found that AI-assistant access was associated with more security vulnerabilities across the majority of programming tasks and higher confidence in insecure answers.
- Participants who invested more in query creation, such as providing helper functions or adjusting parameters, were more likely to provide secure solutions.
- The authors released the user interface, user prompts, and interaction data to support research on AI code-assistant use.
9 APPENDIX
The appendix documents study materials, participant data, mistake attribution, survey questions, and regression-data processing. It also includes example task content and descriptions of the experimental interface and tables.
- Appendix Materials: The appendix includes example programming-task content, including Python output, a string-reversal function, and possible returned values.
- Survey Questions: Post-study surveys measured perceived correctness, perceived security, language comfort, and experiment-group trust in the AI’s secure-code production.
- Study Interface: Figure 7 documents the experiment and control interfaces while participants solved questions.
- Mistake Attribution: Table 8 attributes experiment-group mistakes to AI versus non-AI origins, with detailed qualitative categories consolidated for analysis.
- Participant Data: Table 7 records participant education, student type, programming experience, and security-course exposure, while a separate table provides detailed demographics.
- Regression Processing: The logistic-regression data groups secure, partially secure, and insecure responses differently for each question.