PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Large Language Models to Detect npm Malicious Packages

Nusrat Zahan, Philipp Burckhardt, Mikola Lysenko, Feross Aboukhadijeh, Laurie Williams

arXiv:2403.12196v4cs.CRcs.AI

TL;DR

Existing malicious-code detection can require multiple tools and still face incomplete coverage and misclassification. The paper presents SocketAI and evaluates GPT-3 and GPT-4 for detecting malicious npm packages. The study reports that both models detected 2,057 of 2,180 malicious packages, while noting limits in dataset coverage and scope.

  • Problem

    Existing malicious-package detection can require practitioners to combine tools and still face incomplete coverage and high misclassification.

    Source: §I · p. 1

  • Method

    The study evaluates SocketAI, a malicious-code review workflow using iterative self-refinement, LLM-as-a-Judge, and zero-shot role-play prompting, on MalwareBench npm packages.

    Source: §I · p. 1, A. Leveraging LLM-Based Techniques · p. 4

  • Results

    Both GPT models detected 2,057 of the 2,180 malicious packages in the study.

    Source: §IX · p. 11

  • Takeaways & Limitations

    SocketAI offers an approach intended to reduce security analysts’ manual-review burden while providing findings with minimal misclassification.

    Source: §IX · p. 11

    The sample favored smaller packages with straightforward attacks, and the authors note possible MalwareBench bias toward certain malicious-code types.

    Source: C. Threats To Validity · p. 10

Abstract

from arXiv · show

Existing malicious code detection techniques demand the integration of multiple tools to detect different malware patterns, often suffering from high misclassification rates. Therefore, malicious code detection techniques could be enhanced by adopting advanced, more automated approaches to achieve high accuracy and a low misclassification rate. The goal of this study is to aid security analysts in detecting malicious packages by empirically studying the effectiveness of Large Language Models (LLMs) in detecting malicious code. We present SocketAI, a malicious code review workflow to detect malicious code. To evaluate the effectiveness of SocketAI, we leverage a benchmark dataset of 5,115 npm packages, of which 2,180 packages have malicious code. We conducted a baseline comparison of GPT-3 and GPT-4 models with the state-of-the-art CodeQL static analysis tool, using 39 custom CodeQL rules developed in prior research to detect malicious Javascript code. We also compare the effectiveness of static analysis as a pre-screener with SocketAI workflow, measuring the number of files that need to be analyzed. and the associated costs. Additionally, we performed a qualitative study to understand the types of malicious activities detected or missed by our workflow. Our baseline comparison demonstrates a 16% and 9% improvement over static analysis in precision and F1 scores, respectively. GPT-4 achieves higher accuracy with 99% precision and 97% F1 scores, while GPT-3 offers a more cost-effective balance at 91% precision and 94% F1 scores. Pre-screening files with a static analyzer reduces the number of files requiring LLM analysis by 77.9% and decreases costs by 60.9% for GPT-3 and 76.1% for GPT-4. Our qualitative analysis identified data theft, execution of arbitrary code, and suspicious domain categories as the top detected malicious packages.

I. INTRODUCTION

Malicious-package detection remains difficult because existing techniques are incomplete, can misclassify packages, and often require practitioners to combine tools. The study evaluates LLMs for malicious-code detection and introduces SocketAI, comparing it with static analysis and examining screening costs and detected or missed package types.

I. INTRODUCTION

  • Existing detection methods face incomplete coverage and high misclassification, while practitioners must integrate tools to detect diverse malware patterns.The cited challenges include malicious-code injection, dependency confusion, typosquatting, and limited support for some languages and research packages.Source: §I · p. 1
  • The study empirically evaluates LLMs for malicious-code detection to aid security analysts amid production-evaluation challenges, including resource demands and limited ground truth.The paper identifies false negatives and absent confirmation from package managers as additional challenges to evaluating production performance.Source: §I · p. 1
  • SocketAI is a malicious-code review workflow, and the study compares GPT-3 and GPT-4 with static analysis and examines LLM-detected or missed package types.The evaluation uses MalwareBench and includes 5,115 npm packages, of which 2,180 are malicious.Source: §I · pp. 1–2
  • The study’s stated contributions include a malware-domain-specific prompt and a baseline comparison with CodeQL using 39 custom rules from prior research.The paper also reports qualitative insights into ChatGPT’s strengths and shortcomings in large-scale code review.Source: §I · p. 2

II. DATASET AND LLM SELECTION

This section describes the benchmark, model selection, cost budget, and package sampling used to evaluate SocketAI. Its final sample contains 5,115 npm packages selected under budget constraints and a preference for smaller packages.

A. Benchmark Dataset: MalwareBench

  • MalwareBench labels malicious and neutral npm packages, defining neutral packages as having no discovered malicious code.Its ground truth combines manual confirmation of malicious cases flagged by automated tools with automation and manual checks of sampled neutral cases.Source: A. Benchmark Dataset: MalwareBench · p. 2

B. Large Language Model Selection

  • The study evaluates GPT-3 (gpt-3.5-turbo-1106) and GPT-4 (gpt-4-1106-preview) using zero-shot learning techniques.The passage describes ChatGPT’s zero-shot and in-context capabilities as the model-selection rationale.Source: B. Large Language Model Selection · p. 2

C. Cost Allocation

  • A $3,000 budget was set; preliminary analysis of 1,000 JavaScript files cost $10.50 with GPT-3 and $130 with GPT-4.The authors estimated the budget could cover around 20,000 files.Source: C. Cost Allocation · p. 2

D. Package Selection

  • The final sample contains 2,180 malicious and 2,935 neutral packages, comprising 20,926 files, after budget-conscious stratified sampling that favored smaller packages.Packages above the 75th percentile were disfavored; that quartile’s maximum was 25 files or 175 KB.Source: D. Package Selection · p. 2

III. SOCKETAI

SocketAI combines LLM prompting techniques with a structured, domain-targeted prompt design. Its prompts guide analysis and format reports for review.

III. SOCKETAI

  • SocketAI is used in production across npm, PyPI, Go, and Maven, evaluating 17 million package versions quarterly on average.The authors state that three of them were actively developing and maintaining the workflow as of 2024.Source: §III · pp. 2–3

A. Leveraging LLM-Based Techniques

  • The workflow combines iterative self-refinement, LLM-as-a-Judge, and Zero-Shot-CoT with Role-Play prompting for npm package evaluation.The paper describes self-refinement as drafting and then refining a response using model feedback, while LLM-as-a-Judge scores or verifies outputs against criteria.Source: A. Leveraging LLM-Based Techniques · p. 4

B. Prompt Design

  • SocketAI’s two-fold prompt separates system-role instructions from package-file user input and guides sequential analysis into machine-readable JSON reports.The prompts define malware, security risk, and obfuscated code, and use malware-intent, security-risk, and confidence scores to support reviewer prioritization.Source: B. Prompt Design · pp. 4–5

C. Proposed Workflow

The proposed workflow starts with code and a system-role prompt, refines model responses, and ends with human evaluation. Its first step generates multiple initial reports per file.

C. Proposed Workflow

  • SocketAI’s four steps provide code with a system-role prompt, refine responses in two successive steps, and add humans to evaluate the generated report.The passage identifies the four steps as the workflow shown in Figure 1.Source: C. Proposed Workflow · p. 5

1) Step 1: Initial Reports:

  • Step 1 generated five reports per file with GPT-3 and three with GPT-4, using temperature 1 and top-p 0.9.Multiple reports are generated in one interaction to avoid repeatedly submitting the same prompt for each report.Source: 1) Step 1: Initial Reports · p. 5

2) Step 2: Critical Reports:

SocketAI refines initial findings through critical review and final-report selection before human reviewers assess the resulting reports. The baseline-comparison section frames GPT-3 and GPT-4 against static analysis using precision, recall, and F1.

2) Step 2: Critical Reports:

  • Critical Reports reassesses initial findings, checks for errors, and requires justification for risk scores above 0.5 before producing multiple candidate reports.GPT-3 produces five reports and GPT-4 produces three; the step uses temperature 0.75 and top-p 0.6.Source: 2) Step 2: Critical Reports · p. 5
  • The final-report step selects or improves a candidate report, re-evaluates the code, and aligns a malicious-code score of 1 with malicious findings.It combines the file, Step 2 reports, and system prompts from Steps 1 and 3, using temperature and top-p of 0.5.Source: 4) Step 4: Report Evaluation · p. 5

4) Step 4: Report Evaluation:

  • Reviewers prioritize files with malware scores above 0.5 and use the report’s purpose and conclusion, then code-review metrics if needed, to assess intent.A high security-risk score with a low malware score indicates a vulnerability rather than malicious code.Source: 4) Step 4: Report Evaluation · pp. 5–6, Table IV · p. 6

IV. RQ1: BASELINE COMPARISON

  • The RQ1 baseline comparison evaluates GPT-3 and GPT-4 against traditional static analysis using precision, recall, and F1.The section covers its methodology and results.Source: §IV · p. 6

1) CodeQL Static Analysis:

The study uses CodeQL queries as its static-analysis baseline and reports high recall alongside lower precision. Its comparison uses custom rules aimed at JavaScript malicious code and evaluates package-level classifications.

1) CodeQL Static Analysis:

  • The baseline uses CodeQL because it is a mature, open-source static analyzer, and the selected queries were designed for npm malicious packages.Other tools were excluded for reasons including high false-positive rates, combined-technique overhead, or focus on other settings.Source: 1) CodeQL Static Analysis · p. 6
  • The study used 39 of Froh et al.’s 40 queries, excluding the new-dependency query because it produced high false-positive rates outside differential analysis.The retained rules cover code injection, sensitive-data exposure, network issues, file-system access, obfuscation, and miscellaneous concerns.Source: 1) CodeQL Static Analysis · p. 6

2) ChatGPT Models:

  • The GPT analysis classified a package as malicious when any file scored above 0.5, the threshold for likely malicious behavior.Both GPT models analyzed 18,754 unique files and generated final reports for evaluation.Source: 2) ChatGPT Models · p. 6

1) CodeQL’s Performance:

2) GPT Model’s Performance:

GPT-3 and GPT-4 outperform static analysis in precision and F1, while the study also examines static analysis as a way to reduce SocketAI’s analysis workload and cost. The pre-screening experiment compares analysis of all files with analysis limited to CodeQL alerts.

2) GPT Model’s Performance:

  • GPT-3 reached 0.91 precision and 0.94 F1, while GPT-4 reached 0.99 precision and 0.97 F1; both exceeded static analysis on these measures.GPT-3 recall was 0.97; GPT-4 recall was 0.95, and GPT-4 had the highest false-negative rate.Source: 2) GPT Model’s Performance · p. 7

V. RQ2: EFFECTIVENESS OF SOCKETAI

  • At large scale, GPT analysis can be costly and subject to API rate limits, motivating a static-analysis pre-screen to reduce LLM workload and expense.The study investigates static analyzers as a pre-screening step before the LLM workflow.Source: A. RQ2: Methodology · p. 7

A. RQ2: Methodology

  • The RQ2 experiment compares GPT analysis of all 18,754 files with analysis restricted to files flagged by CodeQL, measuring file counts and total costs.Both GPT models were run on all files first; costs were then recalculated for the flagged subset.Source: A. RQ2: Methodology · p. 7
  • The study did not conduct detailed runtime analysis because GPT response times varied with OpenAI performance and reruns, while CodeQL’s false cases were not thoroughly evaluated.CodeQL took one day; each GPT model took 24 days, and each independent researcher spent about 190 hours on qualitative study.Source: A. RQ2: Methodology · p. 7

B. RQ2: Result

CodeQL pre-screening substantially reduced the files and costs associated with GPT analysis. The subsequent manual evaluation screened model reports and reviewed selected files and packages for false positives and false negatives.

1) GPT Model’s file Analysis:

2) GPT Model’s Cost Analysis:

  • Pre-screening cut files for analysis by 77.9% and reduced GPT-3 costs by 60.9% and GPT-4 costs by 76.1%.For flagged files, GPT-3 cost $49.13 versus $125.65 for all files; GPT-4 cost $482.46 versus $2013.84.Source: 2) GPT Model’s Cost Analysis · pp. 7–8

VI. RQ3: MANUAL EVALUATION OF LLMS ANALYSIS

  • Because MalwareBench labels packages rather than files, researchers manually reviewed LLM reports to confirm malicious files and screened reports by malware score.Two researchers independently assessed reports; they reviewed 2,301 files with scores above 0.5, 123 known malicious packages for false negatives, and 107 neutral-package files for false positives.Source: §VI · p. 8, A. RQ3: Methodology · p. 8

B. RQ3: Results

The models detected many malicious packages, including overlapping detections and varied attack types, while GPT-3 produced substantially more false positives than GPT-4. GPT-4 had very few false positives, though its examples include concerns about unfamiliar package information.

1) True Positive (TP):

  • 2,057 of 2,180 malicious packages were detected by both models; GPT-3 detected 2,128 and GPT-4 detected 2,089.The researchers’ manual review found 736 data-theft cases, 391 suspicious-domain connections, 70 reverse shells, and 126 hidden backdoors.Source: 1) True Positive (TP) · p. 8
  • 43 dependency-confusion cases and 19 typosquatting cases were also identified among detected malicious packages.A package could be classified under more than one attack type; for example, data theft could involve connecting to a suspicious domain.Source: 1) True Positive (TP) · p. 8
  • GPT-4 examples included identifying possible data exfiltration, a suspicious installation script, and a likely typosquatting attempt.The authors present these as examples of generated conclusions for malicious files.Source: 1) True Positive (TP) · p. 8
  • GPT-3 assigned malware scores above 0.5 to 2,226 files, 1,191 more than GPT-4, and 94% of those additional files were package.json files.The authors report that GPT-3 often inferred from incomplete evidence, while GPT-4 used more moderate scores for some typosquatting and dependency-confusion cases.Source: 1) True Positive (TP) · p. 8

2) False Positives (FP):

  • GPT-3 incorrectly flagged 195 neutral packages, including 173 cases attributed to hallucination and 22 cases where it mistook security vulnerabilities for malware.Examples included legitimate URL-sanitizer code, post-installation scripts, dynamic code execution, and sparse or empty files.Source: 2) False Positives (FP) · pp. 8–9
  • GPT-4 had only 3 false positives, including concerns about an unusual dependency version and an unusual security-holding package version.The authors note that GPT-4 may struggle with new information after its April 2023 training cutoff.Source: 2) False Positives (FP) · p. 9

3) False Negative (FN):

Both models missed some malicious code, including cases where they overlooked or under-scored suspicious behavior. The authors also describe limits in report diversity and suggest uses for SocketAI in package review and future research.

3) False Negative (FN):

  • 142 false negatives occurred where both models failed to identify the true malicious code.In one example, GPT-4 described potentially malicious collection and transmission of sensitive information but assigned a malware score of 0.4.Source: 3) False Negative (FN) · p. 9
  • The models sometimes overlooked malicious behavior that appeared benign, including an install script used to delete files in a directory.In another case, GPT-4 detected suspicious behavior but assigned a malware score of 0.3; the authors also describe a benign-looking module-resolution utility used to sabotage a dependent.Source: 3) False Negative (FN) · p. 9
  • The authors observed high malware scores paired with default reports that added no new findings, and repeated refinement did not reliably correct initial errors.They describe similar or identical outputs as “mode collapse.”Source: 3) False Negative (FN) · p. 9
  • The authors say both models detected true malicious files with a low misclassification rate, while GPT-3 hallucinated more frequently.They also report that limited response diversity can lead models to repeat mistakes or hallucinate rather than add insight.Source: 3) False Negative (FN) · p. 9

A. Implications of SocketAI and future work:

  • SocketAI is proposed as an end-to-end LLM workflow for malicious-code review, using iterative analysis, self-refinement, and an LLM-as-a-Judge approach.The authors suggest exploring whether this workflow improves LLM-based vulnerability detection.Source: A. Implications of SocketAI and future… · p. 10
  • The authors propose using SocketAI to review dependencies and applying its workflow across ecosystems, while identifying comparison with traditional ML/DL models as future research.They state that static analysis was used as a baseline to prioritize cost reduction in production.Source: A. Implications of SocketAI and future… · p. 10

B. Challenges in using ChatGPT

The workflow faces reliability limits from mode collapse and hallucinations, difficulty retaining context across large or multi-file code, and incomplete robustness to prompt injection and output-format errors.

1) Mode Collapse & Hallucination:

  • Mode collapse persisted despite temperature and top-p adjustments, while hallucinations produced noisy responses and require further study.The authors also suggest investigating whether hallucinations relate to gaps between expected outcomes and insufficient input information.Source: 1) Mode Collapse & Hallucination · p. 10

2) Large File & Intraprocedural Analysis:

  • Large files can exceed token limits, and segmented analysis may lose context; cross-file data flows can also be missed when malware spans a package.The authors report that their models do not understand data flows across files or modules, limiting package-wide analysis.Source: 2) Large File & Intraprocedural Analysis · p. 10

3) Prompt Injection:

  • Prompt-injection attacks were absent from the dataset, so their impact remains unexplored despite one isolated case in which the workflow correctly identified the threat.The authors call for further investigation of prompt injection.Source: 3) Prompt Injection · p. 10

4) Parsing Output:

  • Although models produced JSON-formatted reports 99% of the time, schema violations and syntax errors required a custom parser and occasional reruns.The reported issues included missing or extra keys, invalid punctuation, truncated text, and files the model did not scan.Source: 4) Parsing Output · p. 10

C. Threats To Validity

Threats to validity include benchmark coverage and the scope of GPT-only evaluation. The related work describes diverse but incomplete detection approaches, while the conclusion reports SocketAI’s benchmark results and screening-cost reductions.

C. Threats To Validity

  • The study’s MalwareBench sample may overrepresent repeated attacks and smaller, straightforward malicious files, limiting evidence about complex attacks and large files.The benchmark also treated many neutral packages as such without vetting; the authors manually reviewed 2,530 files but cannot rule out undiscovered malicious code.Source: C. Threats To Validity · p. 10
  • Evaluation covered only GPT models; indirect training-data exposure, false negatives, and nondeterministic outputs remain concerns.The authors used zero-shot learning and packages released before and after model training cutoffs, but cannot rule out indirect exposure to npm details during training.Source: C. Threats To Validity · p. 11

VIII. RELATED WORK

  • Prior package-detection work spans rule-based, differential-analysis, and machine-learning methods, but no approach is considered a complete, isolated solution.The related-work discussion frames these approaches as parts of a review process that can reduce reviewers’ workload.Source: §VIII · p. 11
  • Related LLM research covers zero-shot malware detection, chain-of-thought and role-play prompting, and self-refinement for reasoning tasks.The cited work reports that self-refinement can improve performance across dialog and reasoning tasks.Source: §VIII · p. 11
  • Other cited studies apply LLMs to code summarization, bug fixing, and vulnerability analysis, including comparisons of vulnerabilities in generated code.The passage reports that ChatGPT-generated code had 20% fewer vulnerabilities and fewer CWE types than StackOverflow code.Source: §VIII · p. 11

IX. CONCLUSION AND FUTURE STUDY

  • SocketAI detected 2,057 of 2,180 malicious packages, while static pre-screening reduced files sent for LLM analysis by 77.9% and costs by 60.9% for GPT-3 and 76.1% for GPT-4.The conclusion also identifies mode collapse, hallucination, large-file and intraprocedural analysis, and prompt injection as areas for further research.Source: §IX · p. 11
Loading 2403.12196v4…