Source-linked AI summary
Practical Automated Detection of Malicious npm Packages
Adriana Sejfia, Max Schäfer
TL;DR
High npm release volume makes manual review infeasible, even as malicious package versions threaten users and the software supply chain. Amalfi combines classifiers, source-reproduction checks, and clone detection; its evaluation found 95 previously unknown malicious packages and reports that retraining can improve precision.
Problem
The volume of npm package updates makes manual review infeasible, while malicious versions threaten users and the software supply chain.
Source: §1 · pp. 1–2Method
Amalfi combines a classifier using lightweight feature analysis, a reproducer for flagged packages, and a clone detector for copies of known malicious packages.
Source: §7 · p. 11Results
95 previously unknown malicious packages were identified, and the decision tree outperformed the other tested classifiers while each contributed unique findings.
Source: §1 · p. 3, §7 · p. 11Takeaways & Limitations
Amalfi’s fast training, feature extraction, and classification suggest practical usefulness, with continuous retraining able to improve classifier precision.
Source: §7 · p. 11Retraining assumes packages not flagged by any classifier are benign, an unsound assumption that might increase false negatives over time.
Source: §4.6 · p. 8
Abstract
from arXiv · showhide
The npm registry is one of the pillars of the JavaScript and TypeScript ecosystems, hosting over 1.7 million packages ranging from simple utility libraries to complex frameworks and entire applications. Due to the overwhelming popularity of npm, it has become a prime target for malicious actors, who publish new packages or compromise existing packages to introduce malware that tampers with or exfiltrates sensitive data from users who install either these packages or any package that (transitively) depends on them. Defending against such attacks is essential to maintaining the integrity of the software supply chain, but the sheer volume of package updates makes comprehensive manual review infeasible. We present Amalfi, a machine-learning based approach for automatically detecting potentially malicious packages comprised of three complementary techniques. We start with classifiers trained on known examples of malicious and benign packages. If a package is flagged as malicious by a classifier, we then check whether it includes metadata about its source repository, and if so whether the package can be reproduced from its source code. Packages that are reproducible from source are not usually malicious, so this step allows us to weed out false positives. Finally, we also employ a simple textual clone-detection technique to identify copies of malicious packages that may have been missed by the classifiers, reducing the number of false negatives. Amalfi improves on the state of the art in that it is lightweight, requiring only a few seconds per package to extract features and run the classifiers, and gives good results in practice: running it on 96287 package versions published over the course of one week, we were able to identify 95 previously unknown malware samples, with a manageable number of false positives.
1 INTRODUCTION
npm’s scale and central role in the JavaScript and TypeScript supply chain make malicious package versions a recurring threat, while the volume of updates makes manual review infeasible. Amalfi combines classifiers, source reproduction checks, and clone detection to find suspicious versions quickly and support human review.
1 INTRODUCTION
- Over 1.7 million packages are hosted on npm, and developers publish around 100,000 public package versions in a week.Packages can be installed transitively, and historic versions remain available unless removed.Source: §1 · pp. 1–2
- Malicious npm versions can tamper with or exfiltrate data, and incidents can affect millions of installations before removal.The paper cites eslint-scope and event-stream as examples of attacks that were installed millions of times before npm staff removed them.Source: §1 · p. 2
- Amalfi aims to detect potentially malicious versions quickly and report them to a human auditor without changing npm’s fundamentals.Its design targets automation, efficiency as new versions appear, and accuracy that avoids both false alarms and missed malware.Source: §1 · p. 2
- Amalfi combines classifiers trained on labeled malicious and benign packages, package reproduction from source, and clone detection.The classifiers use API-change and package-metadata features; reproduction filters some false positives, while clone detection targets missed copies.Source: §1 · p. 2
- 95 previously unknown malicious packages were identified in practice by Amalfi’s combined approach.The paper also reports that Amalfi usually takes only a few seconds per package to extract features and run its classifier.Source: §1 · p. 3
- The paper reports that false positives fall below one in a thousand after retraining, while the decision tree achieves over 40% recall in cross-validation.Among the tested classifiers, the decision tree performed best, though the others also contributed findings.Source: §1 · p. 3
2 BACKGROUND
The paper defines malicious package versions broadly and describes how attackers publish them, get them installed, and execute their code. Its examples show how suspicious capability changes and package behavior can help identify malicious versions.
2 BACKGROUND
- A malicious package version contains code implementing behavior such as data exfiltration, data tampering or destruction, or undocumented expensive computations.The definition includes code that is disabled or broken, and the paper uses “malicious package” as shorthand for a version.Source: §2 · p. 3
- Attackers deliver npm malware by publishing a malicious version, getting users to install it, and getting them to run its code.Publication can involve a new package or a compromised existing package; installation scripts are one way to execute code.Source: §2 · p. 3
- The mogodb example steals a machine’s hostname through a postinstall script, while its near-simultaneous version releases add a suspicious timing signal.The versions 3.1.8 and 3.1.9 were published less than a millisecond apart.Source: §2 · p. 3
- The jasmin@0.0.3 example harvests password fields, and its newly used capabilities stand out against version 0.0.2.A minor-version upgrade introduced capabilities absent from the previous version, although those capabilities are individually common and not inherently suspicious.Source: §2 · p. 4
- The examples motivate detecting malicious packages from high-level API, platform-capability, and metadata features, especially how those features change between versions.Source: §2 · p. 4
3 OUR APPROACH
Amalfi combines package-version features and classifiers with source reproduction checks and clone detection. Its feature set captures both package contents and changes between versions.
3.1 Feature set
- The feature set combines nine single-version features with update timing and semantic-versioning type; first versions use a pseudo-update so packages malicious from the start can also be detected.The single-version features cover access to sensitive information and system resources, API use, installation scripts, and minified or binary files.Source: §3.1 · p. 4
3.2 Feature extraction
- Feature extraction uses syntax-tree queries for code features, package.json checks for installation scripts, file-entropy statistics for minified or binary content, and consecutive-version differences for change features.Update timing comes from npm publication timestamps, while semantic-versioning type is determined with a library.Source: §3.1 · p. 4, §3.2 · p. 5
3.3 Classifier training
- The training corpus contains 1,790 labeled package versions—643 malicious and 1,147 benign versions of the same packages—and selected learners are decision trees, Naive Bayesian classifiers, and one-class SVMs.Compromised packages occur too rarely for the study to focus on them alone.Source: §3.3 · p. 5
3.4 Reproducer and clone detector
- Amalfi uses a reproducer to rebuild a package tarball from source, but optional or inaccessible repository metadata, uncertain commits, unspecified build steps, and unpinned tools limit success.The authors describe it as an automated false-positive filter despite its low practical success rate.Source: §3.4 · p. 5
- The clone detector hashes tarball contents while ignoring package name and version, so it detects only verbatim clones of known malicious packages.It makes no attempt at fuzzy matching.Source: §3.4 · p. 5
4 EVALUATION
The evaluation tests whether Amalfi finds malicious packages in practice, offers useful accuracy, and runs fast enough, through a week-long deployment simulation and labeled-data experiments. It reports 95 previously unknown malicious packages and concludes the approach is practical, while identifying threats to validity.
4.3 RQ1: Practical performance on newly published packages
- Over seven days, Amalfi identified 95 previously unknown malicious packages, complementing existing npm detection solutions.The result summarizes findings after removing overlap among classifiers.Source: Table 1 · p. 6, §4.3 · p. 7
4.1 Experiment 1: Classifying newly published packages
- The week-long experiment processed new public versions daily, applying classifiers and clone detection, reviewing flagged packages, and retraining with earlier triage results.For the experiment, packages not flagged by any classifier were assumed benign, though the authors say this is not generally sound.Source: §4.1 · p. 6
4.2 Experiment 2: Classifying labelled data
- The basic-corpus experiment used stratified 10-fold cross-validation, training on 90% and measuring precision and recall on the remaining 10%; separately, the MalOSS [8] dataset contained 372 package versions, including 40 malicious.Source: §4.2 · p. 6
4.4 RQ2: Accuracy
- All models achieved very high precision, but Naive Bayes and SVM recall was somewhat poor; on MalOSS, recall was higher at the expense of precision.Some MalOSS packages labeled malicious were not taken down by npm; the SVM results require care because its ν parameter was fitted on the same dataset.Source: §4.4 · p. 7, Table 2 · p. 7
4.5 RQ3: Performance
- Classifier training took only a few seconds; feature extraction averaged six seconds, while prediction took less than one second for every classifier.For over half the sampled packages, feature extraction took less than one second; one package with more than 11,000 files took over ten minutes.Source: Figure 4 · p. 8, §4.5 · p. 8
4.6 Threats to validity
- Conclusions may not generalize from a potentially biased package sample, and five-minute reviews or treating unflagged versions as benign could miss malicious packages.The authors also note that sparse data cannot establish when retraining returns diminish.Source: §4.6 · p. 8
5 DISCUSSION
The discussion examines the 95 malicious packages found, the feature distributions and classifier findings, and several unsuccessful model tweaks. The results also reveal that malicious updates were often major versions, while some minor-update malware may have been missed.
5 DISCUSSION
- All detected malicious packages used installation scripts or main-module code to contact a remote host, and almost all sent PII, although PII access itself was not detected.Packages were usually flagged through installation-script use or network access; the authors caution that features absent from this batch may still help on other batches.Source: §5 · p. 8
- Most detected malicious package versions were major updates, contrary to the expectation that attacks hide in minor updates; this may mean some minor-update malware was missed.Source: §5 · p. 8
- Median entropy was 4.69 for malicious packages versus 0.001 for benign packages, while median time between updates was 7.02 s versus 2217.18 s.The authors suggest higher entropy is consistent with more minified code or binary files.Source: §5 · pp. 8–9, Figure 5 · p. 9
- Among 90 classifier-flagged packages, decision trees accounted for the largest share, but each classifier contributed unique findings; five others were flagged only by the clone detector.Source: §5 · p. 9, Figure 6 · p. 9
- Decision-tree models from July 29–31 omitted only the cryptographic-use feature, while the four later models used all 11 features, suggesting little redundancy.Source: §5 · p. 9
- Random Forests offered no advantage over plain decision trees, and booleanizing features worsened performance, increasing one-class SVM false positives by more than 100%.Source: §5 · p. 9
6 RELATED WORK
Related work spans machine-learning and program-analysis detection, package rebuilding, typosquatting, anomaly detection, registry security, and code reuse. Amalfi differs from several approaches in combining classifiers with package reproduction and clone detection, while targeting lightweight detection of malicious npm packages.
6 RELATED WORK
- Prior detection work includes machine learning, program analysis [8], package rebuilding, and typosquatting methods; Amalfi does not specifically target typosquatting.Vu et al. use edit distance, while Taylor et al. combine lexical similarity and package popularity.Source: §6 · pp. 9–10
- Garrett et al. detect anomalous package updates with k-means; Amalfi considers three classifier types and adds reproduction and clone detection, and its evaluation reports finding previously unknown malicious packages.Garrett et al. flagged 539 of 2,288 updates as potentially suspicious without detailed triage; Amalfi's experiments covered a larger set.Source: §6 · p. 9, Table 4 · p. 10
- Duan et al. [8] found 339 previously unknown malicious packages across three registries, including 41 on npm; Amalfi reports more findings on a smaller set, but their false-positive and performance statistics are unspecified.Source: §6 · pp. 9–10
- Dynamic analysis observes installation artifacts, while regular-expression scanners can be noisy; package reproducibility alone is a weak maliciousness signal and needs combination with other techniques.Rebuilding npm packages is complicated by irrelevant, non-malicious differences.Source: §6 · p. 10
- Anomaly-detection research informed Amalfi's use of decision trees, Naive Bayes, and one-class SVMs; the authors considered neural networks unsuitable for their small dataset.They note that more and better data could make neural networks a future avenue.Source: §6 · p. 10
- Registry-security studies include attack analyses and mitigation proposals, whereas Amalfi focuses on short-term measures that require no registry changes.Source: §6 · pp. 10–11
- Code-reuse research covers outdated vulnerable dependencies, upgrade notifications, and links between project size or dependency count and security risks.Other work proposes removing unused code and dependencies to reduce attack surface; a study of downstream interference describes scenarios but found no real-world examples.Source: §6 · pp. 10–11
7 CONCLUSION
Amalfi combines classifiers trained on known malicious and benign npm packages with a source reproducer and clone detector. Its evaluation found all three classifier types detected previously unknown malicious packages, with decision trees outperforming the others; future work includes deeper feature extraction and broader clone detection.
7 CONCLUSION
- Amalfi combines a classifier trained on known malicious and benign npm samples, a source reproducer, and a clone detector; classifier features come from lightweight syntactic analysis of package capabilities and version changes.Source: §7 · p. 11
- All three classifier types detected previously unknown malicious packages, with decision trees outperforming the others while each classifier contributed unique results; retraining can improve precision as predictions are triaged.Training, feature extraction, and classification are reported to be very fast.Source: §7 · p. 11
- Future work includes deeper feature extraction, advanced detection of similar but non-textually identical malicious copies, classifier-result ranking for triage, and applying the techniques to PyPI or RubyGems.Source: §7 · p. 11