Source-linked AI summary
Vibe Coding and Web Application Security: A Twin-Prompt Study
Darko Andročec
TL;DR
The paper asks whether explicitly requesting security best practices improves the security of LLM-generated web applications. It compares paired baseline and security-aware generations using complementary analyses, finding fewer confirmed findings in every application and no Critical or High issues in the security-aware variants. Because each variant was generated once in a small corpus, the findings are descriptive and preliminary.
Problem
Whether security-aware prompting meaningfully changes the security posture of LLM-generated web applications remains an open and practically important question.
Method
The study generates six applications twice with paired prompts and examines the twelve programs using static, dependency, dynamic and manual techniques.
Results
Security-aware prompting was associated with fewer confirmed findings in every application and elimination of all Critical and High confirmed issues.
Takeaways & Limitations
The observations suggest security-aware prompting can move generated applications away from the most serious weakness classes while routine hardening remains.
Takeaways & Limitations
Each variant was generated once, so run-to-run variation cannot be separated from a prompt effect and no causal claim is made.
Abstract
from arXiv · showhide
Large language models increasingly generate complete web applications from natural-language prompts, raising the question of whether explicitly requesting security best practice improves the result. We study six functionally distinct web applications, each generated in two prompt variants that are identical except for an appended security-requirements section: a baseline (A) and a security-aware (B) variant. All twelve programs were produced by the same agentic coding assistant and the same model version in a single, non-iterative generation round, and were then analyzed with static, dependency, dynamic and manual techniques, yielding 75 confirmed findings out of 85 candidates. The security-aware variant produced fewer confirmed findings in every application (24 versus 51) and contained no Critical or High issues; the most severe finding was detected only by manual testing. Because the corpus is small and each variant was generated once, we report descriptive observations rather than statistically established effects, and position the work as a preliminary study whose pipeline is being scaled to multiple models and repeated runs.
1 Introduction
Vibe coding enables runnable web applications from short natural-language descriptions while delegating security-relevant decisions to models. This study asks whether appending explicit security requirements changes the security posture of generated code.
- Vibe coding generates runnable web applications from short natural-language descriptions.
- Delegating authentication, input handling, access control and secret management to models creates a security-relevant research concern.
- The study compares paired applications generated with and without an appended security-requirements section.
- The corpus contains twelve LLM-generated applications and a complete analysis pipeline producing a verified, de-duplicated findings dataset.
- The study is deliberately narrow, using one assistant, one model version, one generation run per variant and six applications.
2 Background and Related Work
Prior work documents security weaknesses in generated code and mixed effects of AI assistance, while benchmarks evaluate vulnerabilities at function and application scales. Prompt-focused studies motivate examining security requirements as the intervention.
- Controlled generation studies found substantial security weaknesses in LLM-generated programs.
- User studies report mixed security effects, including less secure code, small impacts and inconsistent vulnerability reintroduction.
- SecurityEval and BaxBench provide reusable vulnerability-evaluation benchmarks at function and application levels.
- Cross-model benchmark evidence reports frequent OWASP weaknesses, but does not isolate prompt effects under a fixed model and environment.
- Prompt-focused studies report reductions for some secure-coding techniques, unchanged security for lightweight hints and mixed outcomes from iterative refinement.
3 Study Design and Methodology
The study fixes the application tasks, assistant, model and generation procedure while varying only an appended security section, then combines complementary analyses and author review. Its bespoke corpus and untuned single-shot intervention constrain interpretation.
- 3.1 Application Corpus: The corpus contains six web applications spanning three frameworks and two languages, each generated in baseline and security-aware variants.
- 3.1 Application Corpus: The bespoke tasks exercise authentication, access control, uploads, path handling, server-side fetching, ecommerce logic and administration.
- 3.1 Application Corpus: The corpus is designed to compare two prompts on a fixed model, not to benchmark models, and its absolute counts are not comparable with published benchmark scores.
- 3.2 Twin-Prompt Generation: All twelve generations used the same agentic assistant, model version and default settings in fresh sessions.
- 3.2 Twin-Prompt Generation: Variant B is byte-identical to baseline A except for one application-independent security-requirements section.
- 3.2 Twin-Prompt Generation: The appended security section specifies general OWASP-derived requirements without hinting at weaknesses specific to an application.
- 3.2 Twin-Prompt Generation: The intervention uses one untuned security section, measuring a lower bound rather than the best achievable effect of security-aware prompting.
- 3.2 Twin-Prompt Generation: Each variant is generated in one non-iterative interaction without self-critique, judging, feedback or repair.
4 Results
Across twelve generated programs, 75 findings were confirmed, with security misconfiguration dominant. Security-aware prompting coincided with fewer findings in every application, while complementary detection methods revealed issues automated scanners missed.
- Confirmed findings: 75 of 85 consolidated candidate findings were confirmed, while 10 were judged false positives.Dependency analysis found no known-vulnerable dependencies; the confirmed findings concerned first-party code and configuration.
- OWASP distribution: 48 of 75 confirmed findings belonged to A05 Security Misconfiguration, followed by 11 in A01 and 6 in A07.The A05 total was inflated by deployment configurations identical across variants and missing hardening headers.
- Effect of security-aware prompting: 51 baseline findings versus 24 security-aware findings were confirmed across the six applications, with fewer findings in every application.The corresponding means were 8.5 for A and 4.0 for B per application.
- Effect of security-aware prompting: The 27-finding A–B difference remained unchanged after removing container- and header-level classes that were constant by design.The remaining difference was carried by application-level classes including hard-coded secrets, authentication weaknesses, missing access control, path traversal and unrestricted server-side fetching.
- Detection-method complementarity: 28 findings were detected by SAST only, 24 by DAST only and 11 by manual testing only; just 12 were corroborated by multiple methods.The most severe issue, a hard-coded-secret JWT forgery enabling account takeover, was found manually and not surfaced as an exploit by automated scanners.
- Language comparison: The four Python applications had 57 confirmed findings versus 18 for the two JavaScript applications, but the comparison is confounded by unequal counts and differing functionality.The paper treats this as a description of the present corpus rather than a language-level effect.
5 Discussion
Within this corpus, explicit security requirements were associated with fewer and less severe findings, while complementary detection methods exposed limitations that no single scanner could capture. The single-shot design keeps the intervention practical but leaves its comparison with iterative approaches unresolved.
- Security-aware prompting was associated with fewer and less severe confirmed findings in every application, eliminating all Critical and High issues.Remaining findings were predominantly low-severity configuration matters, including some artifacts of the shared neutral container design.
- SAST, DAST and manual testing were largely complementary, with only a minority of findings detected by more than one method.Automated tools both missed an account-takeover-grade JWT forgery and produced a non-reproducing High-severity SQL-injection alert.
- Relying on a single detection method, especially one automated scanner, would have materially misrepresented the applications’ security posture.
- The reduction was obtained with one prompt and one response, without review loops, but whether it approaches or differs from iterative self-critique remains unanswered.The paper leaves that comparison to the extended study.
6 Threats to Validity
The study’s validity is constrained by subjective and count-dependent measurement, one generation per variant, a small bespoke corpus, and the absence of statistical power. Accordingly, the reported A–B differences are descriptive patterns within this corpus rather than statistically established effects.
- Severity ratings and OWASP/CWE mappings were assigned by one analyst, while de-duplication by vulnerability class determines the reported finding counts.A missing security header and an account-takeover-grade token forgery each count as one finding, so severity is reported alongside counts.
- Each variant was generated once, so run-to-run variation may contribute to A–B differences and prevents causal claims.The authors identify repeated generation with at least five runs per cell as the central planned change.
- The small bespoke corpus may not transfer to other models, assistants, languages, frameworks, codebases, or security prompts.The language comparison is additionally confounded by unequal group sizes and differing application functionality.
- With six pairs and one observation per cell, the study reports descriptive figures without significance tests or statistically established effects.
7 Conclusion
The study combines a twin-prompt design with multi-method analysis of LLM-generated web applications. Descriptively, security-aware prompting coincided with fewer findings and no Critical or High confirmed issues, while detection methods were complementary and manual testing found the most severe true issue.
- The study analyzed six applications in baseline and security-aware variants using static, dependency, dynamic and manual techniques.The twelve programs yielded 85 candidate findings consolidated into 75 verified ones.
- Security-aware prompting was associated with fewer confirmed findings in all six applications and removal of every Critical and High confirmed issue.
- The detection methods were strongly complementary, and manual testing uncovered the most severe true finding.
- The extended study will add repeated runs, several models and assistants, benchmark alignment, and comparisons with iterative and judge-based architectures.