Dos and Don'ts of Machine Learning in Computer Security

1 Introduction

• Generic machine-learning pitfalls across security domains have received little attention, despite prior work on challenges in specific domains.The paper distinguishes this concern from earlier work on areas such as intrusion detection, malware analysis, and website fingerprinting.Source: §1 · p. 1
• Pitfalls can produce over-optimistic results and weaken assumptions and conclusions, fostering false confidence that hinders adoption in academia and industry.The authors emphasize sound scientific methodology, particularly in security, where adversaries actively seek to bypass analysis and break systems.Source: §1 · p. 1
• Each of 30 top-tier security papers studied contained at least three of the ten identified pitfalls, and several pitfalls appeared in most papers.The study examines papers published over the past decade.Source: §1 · p. 1
• The authors experimentally examine how pitfalls affect results in four security domains, with findings supporting broader community concerns.The analysis also asks how applying the recommendations can overcome experimental bias.Source: §1 · p. 2
• The authors identify ten pitfalls, propose actionable recommendations, and flag open problems that require further research.They frame the work as a reflective effort to help the community mitigate pitfalls, not as finger-pointing.Source: §1 · p. 2

2.1 Data Collection and Labeling

• Unrepresentative datasets create sampling bias; estimate plausible data distributions, avoid mixing incompatible sources, and disclose dataset limits because this bias may be mitigated but not always removed.Synthetic data and transfer learning are additional strategies, but do not eliminate the need to discuss the dataset’s limitations.Source: §2.1 · pp. 2–3
• Inaccurate or shifting labels can bias classification and cause deployed systems to lose performance as class ratios change.The authors recommend verifying labels where possible, using robust models or noise-aware training, and taking precautions against label shift; uncertain labels should remain in test data.Source: §2.1 · p. 3

2.2 System Design and Learning

• Data snooping lets test, future, or selectively cleaned data influence training, producing over-optimistic results in changing security domains.Split and store test data early, preserve temporal dependencies in splits, and complement experiments on well-known datasets with more recent data.Source: §2.2 · pp. 3–4
• Spurious correlations can make models learn artifacts instead of the security task, while test-dependent parameter choices can inflate reported performance.The authors recommend explanation techniques and validating learned correlations against the system’s objective; a separate validation set can help prevent biased parameter selection.Source: §2.2 · p. 4

2.3 Performance Evaluation

• Limited baselines leave improvements difficult to assess; compare against simple learning models and suitable non-learning methods as well as prior approaches.Simple models can be easier to explain and less computationally demanding, while AutoML can help establish a lower bar for comparison.Source: §2.3 · pp. 4–5
• Performance measures that ignore application constraints or class imbalance can misrepresent detection quality, so choose measures aligned with deployment and the target class.Precision and recall account for class imbalance, but can mislead when minority-class prevalence is inflated; the authors suggest MCC in that case and bounded AUC for practical false-positive rates.Source: §2.3 · pp. 5–6

2.4 Deployment and Operation

• Lab-only evaluation can overstate practical impact, especially when closed-world tests omit diversity and non-stationarity.Approximate real-world conditions by considering temporal and spatial relations, runtime and storage constraints, and deployment where feasible; ethical and privacy constraints may prevent deployment.Source: §2.4 · p. 6
• Security evaluations should include a precise threat model because learning-based systems operate in adversarial environments and face attacks such as poisoning and evasion.The authors recommend considering an adaptive adversary and examining workflow vulnerabilities, focusing on white-box attacks where possible.Source: §2.4 · p. 6

3 Prevalence Analysis

• The study assessed 30 representative papers from four top-tier security conferences using two independent reviewers and a third reviewer to resolve disputes.Pre-dispute inter-rater reliability was α = 0.832.Source: §3 · pp. 6–7
• Sampling bias (P1) was at least partly present in 90% of papers and data snooping (P3) in 73%; P10, P9, and P7 each exceeded 50%.Figure 3 displays the prevalence classifications across the analyzed papers.Source: Figure 3 · p. 7, §3 · p. 7
• Missing hyperparameter, decision-explanation, or dataset-splitting and normalization details made P5, P4, or P3 unclear and hindered reproducibility.Reviewers also marked papers unclear when missing information prevented them from ruling out a pitfall.Source: §3 · pp. 7–8
• An anonymized Likert survey received 49 authors’ responses: 98% said the paper raised awareness, and average agreement with the findings was 63%.Respondents represented 13 of the 30 selected papers.Source: §3 · p. 8
• Authors most often judged lab-only evaluations frequent (92%), followed by the base rate fallacy (77%) and inappropriate measures and sampling bias (69% each).Authors also reported limited awareness and inconsistent understanding; 44% were neutral on whether data snooping is easy to avoid.Source: §3 · p. 8
• Every analyzed paper had at least three pitfalls; only 22% of instances were discussed in the text, suggesting limited awareness.The overall prevalence analysis found pitfalls affected between 17% and 90% of selected papers.Source: §3 · p. 8

4.1 Mobile Malware Detection

• DREBIN and OPSEQS recall fell by more than 10% and 15%, respectively, between mixed-source D1 and GooglePlay-only D2, while accuracy changed only slightly.The URL play.google.com was among the five most discriminative benign-class features, suggesting the classifier learned app origins rather than malware differences.Source: §4.1 · p. 9, Table 1 · p. 9

4.2 Vulnerability Discovery

• VulDeePecker relied on dataset artifacts, while an SVM with 3-grams achieved the best performance with an 18× smaller model.The paper also reports higher true-positive rates for both baselines, although AUC-ROC differed only slightly across approaches.Source: §4.2 · p. 10, Table 4 · p. 10
• Truncating vulnerability snippets to 50 tokens and converting numbers to generic identifiers can erase buffer-size information, making some labels impossible to assess.For example, preprocessing obscures the size of a buffer and whether its contents fit, potentially producing label inaccuracy.Source: §4.2 · p. 10, Figure 5 · p. 10

4.3 Source Code Author Attribution

• Removing unused template code from test sets reduced source-code attribution accuracy by 48%; after retraining, average accuracy still fell by 6% and 7% for the two methods.The dataset’s reused code templates diverged from real-world settings and supplied artifact features rather than coding style.Source: §4.3 · p. 11, Figure 7 · p. 11

4.4 Network Intrusion Detection

• The boxplot baseline matched KITSUNE in ROC AUC, outperformed it at low false-positive rates, and was substantially lighter.The authors caution that this Mirai experiment does not establish wild-Mirai detection or KITSUNE’s inability to detect other attacks.Source: §4.4 · p. 12, Table 5 · p. 12

4.5 Takeaways

• Across the four case studies, subtle design and setup errors produced misleading or erroneous results, indicating that these pitfalls affect multiple security domains.The authors argue that the frequency and severity of pitfalls in top papers warrant greater awareness.Source: §4.5 · p. 12

• The authors state that these inherent limitations do not affect the overall conclusion of their analysis.Source: §5 · p. 12
• The overview cannot detail all ten pitfalls, and some, including sampling bias, label inaccuracy, or lab-only settings, may not always be preventable.Ethical considerations may make real-environment attack testing impossible, leaving simulation as the only option.Source: §5 · p. 12
• The prevalence analysis selects 30 papers from top security conferences over ten years that use machine learning prominently; the selection process is not entirely free from bias.The authors describe the identified pitfalls as typical for the research branch and the papers as often highly cited.Source: §5 · p. 12
• Pitfalls are counted only when clear from a paper's text or artifacts; otherwise, the analysis conservatively treats them as absent.Source: §5 · p. 12
• The impact analysis uses four exemplary works from areas where the authors have published; the biased selection is intended to demonstrate empirically how pitfalls affect results.Source: §5 · pp. 12–13

Contribution

• The authors characterize their study as the first to systematically and comprehensively explore machine-learning pitfalls across security rather than within particular domains.They describe it as applicable to all security domains.Source: §6 · p. 13

Security studies

• Earlier intrusion-detection studies address sampling bias, base-rate fallacy, performance measures, semantic gaps, and unrealistic evaluation baselines.Axelsson, McHugh, and Cardenas et al. cover evaluation issues; Sommer and Paxson focus on machine learning for network intrusion detection.Source: §6 · p. 13
• Malware and other security-domain studies examine issues including sampling bias, packed samples, unrealistic settings, evaluation measures, and benchmarking flaws.Examples include malware experiments, packed-sample classification, hardware-counter defenses, location privacy [98], authentication, and system security.Source: §6 · p. 13

Adversarial learning studies

• Adversarial-learning work addresses attacks and defenses, while Carlini et al. [26] and Biggio et al. provide guidance on evaluating defenses under attack.This paper also argues for considering smart and adaptive adversaries but takes a more general scope.Source: §6 · p. 13

Machine learning studies

• General machine-learning research covers sampling bias, dataset shift, parameter selection, data snooping, and evaluation methods, while this study focuses on pitfalls prevalent in security.Its recommendations are tailored to the security community.Source: §6 · p. 13

• The paper systematically assesses ten subtle machine-learning pitfalls that can affect research validity and lead to overestimated security-system performance.Source: §7 · p. 13
• The authors find the pitfalls prevalent in security research and demonstrate their impact across different security applications.Source: §7 · p. 13
• The paper provides recommendations applicable across security domains, while its broader aim is to improve empirical work and encourage exploration of real-world security systems.Source: §7 · p. 13

• The paper's supplementary material is available at dodo-mlsec.org.Source: Additional material · p. 13

Label inaccuracy

• F1-score falls from 0.95 to 0.73 with noisy DREBIN labels, then rises to 0.93 after cleansing.The cleansing method detects 84% of flipped labels, while falsely flagging 0.2% of original labels; Table 6 covers performance with label noise.Source: A Appendix: Pitfalls · p. 17, Table 6 · p. 17

Data snooping

• Table 8 groups common data-snooping cases into test, temporal, and spatial types as a starting point for workflow checks.The appendix describes snooping as unintended use of information unavailable in practice.Source: A Appendix: Pitfalls · p. 17, Table 8 · p. 18

Spurious correlations

• Spurious correlations can arise from confounding or sampling bias, and their validity depends on the domain and learning objective.The authors recommend systematically checking extraneous factors; explanation techniques can help trace predictions to features and expose learned correlations.Source: A Appendix: Pitfalls · p. 17
• Correcting sampling bias can require known or estimable observation-selection probabilities, which the authors say are rarely available in security research.The appendix notes that methods proposed in other disciplines often rely on this information.Source: A Appendix: Pitfalls · p. 17

Sampling bias

• Across three Android app markets, DREBIN F1-scores range from 0.807 to 0.879, versus 0.885 when sampling randomly from the combined dataset.The authors interpret the higher combined-dataset result as indicating sampling bias; Table 7 compares origins.Source: Table 7 · p. 17, A Appendix: Pitfalls · p. 18

Survey figure

• Figure 9 groups author responses by pitfall; bar colors range from strong disagreement to strong agreement.The figure labels the pitfalls P1–P10.Source: B Appendix: Prevalence Analysis · p. 18, Figure 9 · p. 18

Survey responses

• The survey statement asks whether each pitfall appears frequently in security papers, under the heading of agreement between authors and reviewers.Source: B Appendix: Prevalence Analysis · p. 18

Data collection and ethics

• The institution’s review-board chair approved the survey, and the authors report GDPR-aligned data minimization and anonymization where possible.Authors also approved a consent form describing the study, collected data, and contact details for questions.Source: B Appendix: Prevalence Analysis · p. 18