Source-linked AI summary
Adversarial Machine Learning -- Industry Perspectives
Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, Sharon Xia
TL;DR
Organizations are increasingly concerned with securing ML systems, but interviews indicate that practitioners lack adequate tools and know-how for adversarial attacks. The paper interviews industry developers, ML engineers, and security personnel, then maps observed gaps onto the traditional Security Development Lifecycle. It concludes that industrial-grade ML systems need the security rigor of traditional software development.
Problem
The paper addresses limited industry readiness to protect, detect, and respond to adversarial attacks against ML systems.
Method
The authors interview 28 organizations and use perspectives from developers, ML engineers, and security personnel to enumerate ML security gaps within the Security Development Lifecycle.
Results
Most interviewed practitioners are unequipped to secure industry-grade ML systems against adversarial attacks, with 25 of 28 organizations reporting that they lacked the right tools and sought guidance.
Takeaways & Limitations
The paper calls for revising and amending the Security Development Lifecycle for industrial-grade software in the adversarial ML era.
Takeaways & Limitations
The sample of 28 organizations may not represent all industries employing ML, and the study overrepresents security-sensitive organizations and excludes several non-malicious safety failures.
Abstract
from arXiv · showhide
Based on interviews with 28 organizations, we found that industry practitioners are not equipped with tactical and strategic tools to protect, detect and respond to attacks on their Machine Learning (ML) systems. We leverage the insights from the interviews and we enumerate the gaps in perspective in securing machine learning systems when viewed in the context of traditional software security development. We write this paper from the perspective of two personas: developers/ML engineers and security incident responders who are tasked with securing ML systems as they are designed, developed and deployed ML systems. The goal of this paper is to engage researchers to revise and amend the Security Development Lifecycle for industrial-grade software in the adversarial ML era.
I. INTRODUCTION
The paper examines how organizations secure ML systems against adversarial attacks and finds substantial gaps in industry readiness. It uses interviews across varied organizations to frame those gaps within the traditional Security Development Lifecycle.
- I. INTRODUCTION: The study investigates how organizations protect, detect, and respond to attacks on their ML systems.Its motivation is the industry’s growing attention to adversarial ML and associated risks such as data corruption, model theft, and adversarial samples.
- I. INTRODUCTION: The paper reports that 25 of 28 organizations lacked the right tools to secure their ML systems and explicitly sought guidance.It presents the findings as gaps faced by software developers, ML engineers, and security incident responders.
- I. INTRODUCTION: The paper enumerates ML security engineering aspects across design and development, deployment preparation, and operation under attack using the Security Development Lifecycle framework.All organizations were familiar with SDL for traditional software engineering, although execution varied by organization size and process formality.
- I. INTRODUCTION: Interviews covered 28 organizations spanning Fortune 500 companies, small-and-medium businesses, non-profits, and government organizations.The sample included security-sensitive fields and other sectors, including social media analytics, publishing, agriculture, urban planning, food processing, and translation services.
- I. INTRODUCTION: The organizations used varied ML strategies, with most relying on toolkits, 10 using Machine Learning as a Service, and only 2 building systems from scratch.Examples of toolkits included Keras, TensorFlow, and PyTorch; services included Microsoft Cognitive API, Amazon AI Services, and Google CloudAI.
A. Findings:
The findings show that organizations recognize adversarial ML as important but remain focused on traditional security and lack practical capabilities to secure, detect, and respond to ML attacks. Reported concerns include privacy breaches, model theft, data poisoning, and inappropriate recommendations, while physical-domain adversarial examples ranked lower.
- 22 out of 25 organizations said they lacked the right tools to secure ML systems and explicitly sought guidance.Three government organizations abstained from answering this question satisfactorily.
- Only 6 organizations were ready to assign head-count to adversarial ML, despite all 28 considering AI-system security important to business productivity.The reported emphasis remained on traditional security.
- Organizations prioritized attacks that could breach privacy, steal intellectual property, or cause inappropriate recommendations, including through data poisoning.The examples include protecting client and employee information, proprietary algorithms, and recommendation integrity.
- Physical-domain adversarial examples resonated with respondents but did not rank highly, possibly because many organizations lacked physical components such as cars or drones.
- Security analysts often expected major ML libraries or service providers to supply robust protection, while practitioners also lacked expectations for how systems behave under attack.
III. ABOUT SDL
The Secure Development Lifecycle evolved into an industry software-security process that helps identify vulnerabilities before release. The paper's primary inquiry is whether SDL should be amended and revised to secure ML systems against adversarial attacks.
- SDL emerged from Microsoft’s response to CodeRed and close collaboration between security experts and developers.The resulting practices provided systematic security guidance for finding software defects and implementation flaws.
- SDL helps catch software vulnerabilities that could later be exploited before software reaches customers, although it does not eliminate all software bugs.
- 45% fewer reported vulnerabilities occurred between Windows XP and Windows Vista after SDL was introduced at Microsoft.
- 91% fewer vulnerabilities occurred between SQL Server 2000 and SQL Server 2005 after SDL was introduced at Microsoft.
- The paper investigates amending and revising SDL, the traditional software-security process, for ML systems facing adversarial attacks.
A. Curated repository of attacks
The paper proposes extending the widely used MITRE Framework into a curated repository for adversarial ML attacks. This would give security analysts a unified view of traditional and ML-specific attacks despite low organizational awareness.
- MITRE ATT&CK organizes traditional software attacks into shareable tactics and procedures with detection ideas, known users, and further references.
- Only 5 out of 28 organizations reported working knowledge of adversarial ML, despite growing scholarship in the area.
- The paper proposes registering newly published adversarial ML attacks in an extended MITRE Framework.
- The proposed repository would provide security analysts with a unified view of traditional and adversarial ML attacks.
B. Adversarial ML specific secure coding practices:
Adversarial ML security guidance remains sparse compared with traditional secure coding practices. The paper proposes extending software-security analyses and guidance to ML systems, including consequence-based recommendations and static or dynamic testing.
- Adversarial ML-specific security guidance is sparse, although major ML toolkits provide some best practices.TensorFlow additionally links guidance to tools for testing adversarial attacks.
- Future guidance should target undefined program behaviors and exploitable vulnerabilities while accounting for the field’s changing nature.The paper suggests organizing recommendations by security consequence, allowing different practices for higher- and lower-consequence uses.
- Traditional static analysis converts source code into representations such as abstract syntax trees and control-flow graphs for checking coding rules.This traditional workflow motivates analogous analysis for ML platforms.
- Existing tools provide white-box and black-box dynamic testing, but model stealing, inversion, and membership inference require further analysis.Named tools include cleverhans, secml, and the Adversarial Robustness Toolbox.
- Static analysis for adversarial ML is less explored, with call-graph generation and symbolic execution suggested as possible foundations.The paper envisions IDE-integrated analysis to identify security issues before code is committed.
D. Auditing and Logging in ML Systems
ML auditing should identify high-impact activities, generate attack-linked telemetry, and make detection logic usable across security teams. This addresses the lack of operational insight currently faced by security analysts.
- Traditional software forwards logged security events to SIEM systems, enabling responders to apply anomaly detection.Process creation is given as an example of a logged event that can indicate malware execution.
- Developers should identify high-impact ML activities and ensure harmful attacks leave telemetry traces that can be linked back to those attacks.The resulting events should be exportable for security analysis.
- ML environments are difficult for security analysts to interpret because they lack operational insights.The paper proposes detection methods that can be shared among analysts using self-documented logic such as Sigma.
V. GAPS WHEN PREPARING FOR DEPLOYMENT OF ML SYSTEM
The paper frames ML deployment preparation against the traditional software build-and-release pipeline. It advocates integrating adversarial ML testing into continuous integration and delivery while using threat modeling and red teaming to assess risk.
- Traditional software pipelines commit code, run builds and unit tests, deploy through testing, and then release to production with security tools integrated at each step.This sequence provides the comparison point for ML deployment preparation.
- Dynamic adversarial ML analysis should be integrated into continuous integration and continuous delivery pipelines.Automation is intended to help engineers fix issues without adding unfamiliar processes outside their normal workflow.
- Threat modeling addresses attack impact, while independent red teaming assesses the likelihood of attacks occurring.The paper describes red teaming as deliberately exploiting the system through any available means.
C. Transparency Centers
The paper adapts transparency-center and vulnerability-tracking concepts from traditional software security to adversarial ML. Proposed assurance spans multiple ML modalities, while vulnerability scoring and organizational exposure scanning remain unresolved.
- Traditional transparency centers provide secure facilities for deep source-code inspection with diagnostic tools.Examples include centers established by Kaspersky and Huawei.
- Future adversarial ML transparency centers may assess security across ML platforms, MLaaS, and models embedded in edge devices.The paper identifies tools and test harnesses for these assurance activities as a future research direction.
- Traditional vulnerability management assigns identifiers and severity scores so the industry can refer to vulnerabilities consistently.CVE and CVSS are illustrated through a vulnerability with a base score of 7.5 out of 10.
- The paper asks adversarial ML researchers to register significant ML vulnerabilities in trackable systems such as CVE.It notes that an appropriate ML vulnerability scoring method accounting for risk and impact remains unclear, and organizations lack the ability to scan ML environments for exposure.
B. Incident Response
Incident response for adversarial ML remains difficult after an attack is identified: responders must determine blast radius, contain affected systems, and establish root cause despite tightly integrated production systems. Open questions concern which ML artifacts to analyze and how to collect them across deployment settings.
- B. Incident Response: Responders must determine an attack’s blast radius and prepare containment after triaging its business relevance.Traditional ransomware response illustrates this sequence by identifying connected machines and removing them from the network for forensic analysis.
- B. Incident Response: ML system integration makes blast-radius assessment and containment difficult because failure in one system can produce unintended consequences.The paper identifies containerization as a research direction for quarantining uncompromised systems from compromised ones.
- B. Incident Response: Forensic root-cause analysis remains an open problem because it is unclear which artifacts should be analyzed for each ML attack.Candidate artifacts include model files, scored queries, training data, architecture, telemetry, hardware, and software applications.
- B. Incident Response: Artifact collection must account for deployment context, including endpoint or Internet of Things models and machine-learning-as-a-service systems.The available evidence for forensic analysis differs across these settings.
D. Remediation
Traditional software remediation practices do not transfer cleanly to ML systems. The paper highlights uncertainty about patching vulnerable models and validating that replacements preserve performance without retaining the same vulnerabilities, while calling for SDL rigor in adversarial ML.
- D. Remediation: “Patch Tuesday” exemplifies traditional software remediation, in which vendors release vulnerability patches that organizations install according to patching policies.The paper uses this practice as a contrast for ML remediation.
- D. Remediation: ML systems may not be suspendable like Tay, particularly when deployed on the edge, leaving patching guidelines unclear for vulnerable models.Microsoft suspended Tay after a poisoning attack, but the paper notes that this response is not feasible for every ML deployment.
- D. Remediation: Validation remains unclear for a patched ML model because it must retain prior performance while avoiding the same vulnerabilities.The paper frames this as an unresolved remediation question rather than an established procedure.
- D. Remediation: The paper extends the Security Development Lifecycle to adversarial ML and argues that Software 2.0 should follow fundamental security rigor from traditional software development.SDL is described as the de facto process for building reliable software.