Source-linked AI summary
Exploring Automated Vulnerability Identification in JavaScript Code Using Large Language Models
Manit Kaushik, Ishir Bhardwaj, Pranav Gupta, Pankaj Jalote, Arun Balaji Buduru
TL;DR
SAST tools can struggle to analyze isolated JavaScript snippets, a consequential gap given JavaScript's broad use. This study evaluates LLMs on real-world JavaScript vulnerability snippets against SAST baselines, finding higher snippet-level accuracy and gains from fine-tuning, while concluding LLMs should augment rather than replace security workflows.
Problem
JavaScript is comparatively underexamined in vulnerability identification, while SAST tools can struggle with isolated snippets that lack project context.
Source: §1.1 · p. 1, §5.1 · p. 5Method
The study evaluates three LLM families on real-world JavaScript vulnerability snippets across prompting and fine-tuning settings, benchmarking against JavaScript-oriented SAST tools.
Source: §1.1 · p. 1, §3.1 · p. 2, §3.5 · p. 3Results
LLMs achieved 34% accuracy without fine-tuning versus near-zero SAST performance, while fine-tuned Gemini improved from 29% to 60% accuracy.
Source: §6 · p. 6Takeaways & Limitations
LLMs offer practical value in exploratory settings where traditional SAST may not operate effectively, but are best integrated as workflow augmentations.
Source: §5.3 · p. 5, §6 · p. 6Moderate recall of 34–60% means many vulnerabilities remain undetected, so LLMs should not replace comprehensive analysis workflows.
Source: §5.3 · p. 5
Abstract
from arXiv · showhide
JavaScript powers approximately 98.8% of all websites, making vulnerabilities in its code a significant security risk, yet existing detection approaches such as Static Application Security Testing (SAST) tools often fail to identify many real-world vulnerabilities when applied to isolated code snippets. This paper presents an empirical study of Large Language Model (LLM)-based vulnerability identification for JavaScript programs, evaluating three LLM families (Gemini 1.5 Flash, GPT-4o Mini, DeepSeek-R1-Distill-Llama-8B) across multiple prompting strategies (zero-shot, chain-of-thought, few-shot) and fine-tuning approaches on a dataset of 1,125 JavaScript code snippets spanning five Common Weakness Enumeration (CWE) categories: Injection (CWE-74), OS Command Injection (CWE-78), Cross-Site Scripting (CWE-79), SQL Injection (CWE-89), and Uncontrolled Resource Consumption (CWE-400). Our experiments show that LLMs substantially outperform traditional SAST tools on snippet-level vulnerability identification, with a fine-tuned Gemini 1.5 Flash model achieving 60% detection accuracy compared to near-zero performance from rule-based analyzers. We find that fine-tuning improves accuracy from 29% to 60%, Chain-of-Thought prompting benefits reasoning-capable models such as GPT-4o Mini, few-shot prompting is effective for polymorphic vulnerabilities such as Cross-Site Scripting, and performance varies across vulnerability categories, reaching up to 84% accuracy for structured vulnerabilities such as SQL Injection. These results indicate that LLMs provide a practical approach for automated vulnerability identification in JavaScript code, particularly when combined with task-aligned supervision, though they should complement rather than replace existing security analysis workflows due to limited recall and uneven performance across vulnerability types.
1 Introduction
SAST tools can struggle with isolated JavaScript snippets, while JavaScript remains comparatively underexamined despite its broad use. The paper studies LLM vulnerability identification through four research questions on SAST comparisons, prompting, fine-tuning, and CWE-category performance.
1.1 Motivation and Background
- JavaScript powers approximately 98.8% of websites, but prior vulnerability-detection research has comparatively limited JavaScript evaluation and SAST tools often need project context to trace data flow.The study focuses on JavaScript vulnerability identification using LLMs and benchmarks model performance against SAST tools.Source: §1.1 · p. 1
1.2 Research Questions
- The four research questions ask how LLMs compare with SAST, how prompting affects detection, what fine-tuning changes, and how performance varies across CWE categories.The questions frame the study's evaluation of JavaScript vulnerability identification.Source: §1.2 · p. 2
1.3 Paper Organization
- The paper proceeds from related work to experimental design and results, then discusses limitations before concluding with findings and future work.Section 3 covers dataset, model, prompting, and fine-tuning methods; Section 4 organizes results by research question.Source: §1.3 · p. 2
2 Related Work
Prior LLM vulnerability-detection studies examine other languages, datasets, and prompting approaches, while JavaScript remains less studied. This work addresses that gap with real-world JavaScript snippets and comparisons against SAST tools.
Prior LLM vulnerability detection
- Prior studies report LLM gains over traditional approaches on CTF-style datasets and complex vulnerability-classification tasks, although performance can fall on more complex software projects.Gao et al. [2023] report the CTF comparison; Kostina et al. [2025] report results across complex binary and multiclass classification.Source: §2 · p. 2
Prompting strategies
- Structured prompting research includes few-shot and Chain-of-Thought methods for C/C++ vulnerabilities and semantics-guided prompts for Java and C++ code.Nong et al. [2024] report improvements in precision and explanation quality for semantics-guided prompting.Source: §2 · p. 2
JavaScript research gap and contribution
- JavaScript has limited support in prior work, and this study contributes a real-world snippet dataset evaluated across prompting, fine-tuning, CWE categories, and SAST baselines.Prior work identifies scarce JavaScript evaluation and weak cross-language generalization as open gaps; fewer studies examine JavaScript's dynamic execution model and frontend-oriented patterns.Source: §2 · p. 2
3 Experimental Design and Methodology
The study evaluates LLM vulnerability identification as classification over a curated JavaScript snippet dataset, using multiple prompt strategies and model families alongside JavaScript-oriented SAST baselines.
3.1 Dataset
- The dataset contains 1,000 training and 125 test snippets, balanced across five CWE categories and curated from three public sources with verified CWE labels.Each category has 200 training and 25 test samples; the categories are CWE-74, CWE-78, CWE-79, CWE-89, and CWE-400.Source: §3.1 · p. 2
3.2 Problem Formulation
- Vulnerability identification is framed as either five-way CWE classification or binary Yes/No classification for a target CWE.The two settings assess general detection capability and category-specific identification performance.Source: §3.2 · p. 2
3.3 Prompting Strategies
- Zero-shot, Chain-of-Thought, and few-shot prompting are evaluated; few-shot uses one positive and one negative example only in the single-class setting.CoT guides the model to locate an attack surface, assess exploitability, and map behavior to a CWE; Figure 1 presents the experiment pipeline for vulnerability identification and patch generation.Source: §3.3 · p. 3, Figure 1 · p. 3
3.4 LLMs Explored
- Three LLM families were selected to span commercial and open-source ecosystems, reasoning-oriented and lightweight architectures, and cloud and local execution contexts.Fine-tuning used 1,000 labeled samples for 10 epochs; CoT fine-tuning additionally included short reasoning traces.Source: §3.4 · p. 3
3.5 SAST Baselines
- Bearer and NodeJSScan provide JavaScript-oriented rule-based SAST baselines that support snippet-level scanning without full-project context.NodeJSScan supplies direct CWE outputs, aligning its evaluation setup with the study.Source: §3.5 · p. 3
4 Results
The results compare LLM vulnerability identification with SAST tools and examine prompting, fine-tuning, and CWE-category effects. Performance is measured by accuracy, precision, recall, and F1 on 125 test samples across multi-class and single-class tasks.
4.1 RQ1: LLMs vs. SAST Tools
- SAST tools perform near zero on isolated snippets, while ChatGPT 4o-mini reaches 34% accuracy (0.31 F1); LLM recall remains modest.The passages attribute the SAST gap to its need for project structure and vulnerability-flow tracing, which isolated snippets lack.Source: §4.1 · pp. 3–4
4.2 RQ2: Effect of Prompting Strategies
- Under CoT, ChatGPT 4o-mini improves from 34% to 35% accuracy and 0.31 to 0.32 F1, while Gemini 1.5 Flash accuracy falls from 29% to 18%.Gemini precision nevertheless rises from 0.40 to 0.43, showing that prompting effects vary by model.Source: §4.2 · p. 4
4.3 RQ3: Effect of Fine-Tuning
- Gemini 1.5 Flash accuracy rises from 29% to 60% (0.26 to 0.60 F1) with standard fine-tuning, which also exceeds CoT fine-tuning for Gemini and ChatGPT.ChatGPT 4o-mini rises from 34% to 49% accuracy, while CoT fine-tuning improves DeepSeek-R1-8B from 8% to 27%.Source: §4.3 · pp. 4–5
4.4 RQ4: CWE-wise Performance and Few-Shot Effects
- Zero-shot accuracy reaches 82–84% for SQL Injection and 78–80% for OS Command Injection, while few-shot raises ChatGPT 4o-mini's XSS accuracy from 46% to 76–78%.The passages characterize SQL and command injection as structured patterns, XSS as polymorphic, and few-shot gains as model- and CWE-dependent; CWE-400 results vary by model.Source: §4.4 · p. 5
5 Discussion
The discussion qualifies snippet-level comparisons with the evaluation setting, limits the study’s generalizability, and recommends using LLMs alongside existing security workflows.
5.1 Evaluation Context
- Snippet-level testing favors LLMs over SAST, whose whole-project analysis and interprocedural tracking need richer structural context.The near-zero SAST result reflects architectural differences, not fundamental tool inadequacy; the authors describe the approaches as complementary.Source: §5.1 · p. 5
5.2 Generalization and Dataset Scope
- Results may not generalize beyond JavaScript and five CWE categories, especially to vulnerabilities requiring complex interprocedural reasoning.Consistent fine-tuning schedules support fair model comparisons but may affect absolute performance because hyperparameters were not optimized individually.Source: §5.2 · p. 5
5.3 Practical Implications
- With 34–60% recall, LLMs leave many vulnerabilities undetected and should augment, not replace, existing security workflows.The authors position them as rapid filters for incomplete code or exploratory contexts and recommend multiple complementary methods for security-critical applications.Source: §5.3 · p. 5
6 Conclusion and Future Work
The study reports that LLMs outperform SAST on snippet-level JavaScript vulnerability identification, with fine-tuning and prompting effects varying by model and vulnerability type. It also outlines further evaluation, dataset expansion, and hybrid-analysis research.
6 Conclusion and Future Work
- Fine-tuning raises Gemini accuracy from 29% to 60%; Chain-of-Thought benefits reasoning-oriented models but hinders efficiency-optimized architectures, while few-shot prompting helps identify polymorphic XSS.The authors interpret the fine-tuning result as evidence that task-specific alignment helps surface pretrained security knowledge.Source: §6 · p. 6
- 34% accuracy without fine-tuning versus near-zero SAST performance; fine-tuned Gemini reaches 60%, while LLM accuracy varies across CWE categories.The reported category range is 46–84% accuracy, with 82–84% for SQL injection; limited recall of 34–60% supports integration with existing workflows rather than replacement.Source: §6 · p. 6
- The authors propose project-level evaluation, broader vulnerability-class coverage, hybrid LLM and static-analysis methods, and longitudinal studies of model generations.These directions address interprocedural information, generalization beyond injection-style weaknesses, recall and precision trade-offs, and the role of dedicated security training.Source: §6 · pp. 6–7