Source-linked AI summary
Adversarial Attacks Against Medical Deep Learning Systems
Samuel G. Finlayson, Hyung Won Chung, Isaac S. Kohane, Andrew L. Beam
TL;DR
Medical deep-learning systems are increasingly used in clinical settings, but the practical risk posed by adversarial examples in medicine has not been thoroughly examined. The paper evaluates white- and black-box attacks across three representative medical classifiers and synthesizes healthcare-specific incentives and attack scenarios. The attacks were highly successful, including against systems without direct model access and under human-imperceptibility requirements.
Problem
Previous machine-learning literature had not thoroughly examined the viability, incentives, and practical forms of adversarial attacks against medical systems.
Method
The paper tests white- and black-box adversarial attacks on three state-of-the-art medical classifiers and analyzes healthcare-system features and realistic attack scenarios.
Results
White-box PGD and patch attacks achieved AUROCs of 0.000 and accuracies of 0% or less than 1%, while black-box attacks remained highly effective across tasks.
Takeaways & Limitations
Adversarial attacks are likely feasible against accurate medical classifiers, including universal patches applicable upstream from image capture.
Takeaways & Limitations
No simple and general algorithmic fix for adversarial attacks appears available in the short term, although principled robustness approaches show promise.
Abstract
from arXiv · showhide
The discovery of adversarial examples has raised concerns about the practical deployment of deep learning systems. In this paper, we demonstrate that adversarial examples are capable of manipulating deep learning systems across three clinical domains. For each of our representative medical deep learning classifiers, both white and black box attacks were highly successful. Our models are representative of the current state of the art in medical computer vision and, in some cases, directly reflect architectures already seeing deployment in real world clinical settings. In addition to the technical contribution of our paper, we synthesize a large body of knowledge about the healthcare system to argue that medicine may be uniquely susceptible to adversarial attacks, both in terms of monetary incentives and technical vulnerability. To this end, we outline the healthcare economy and the incentives it creates for fraud and provide concrete examples of how and why such attacks could be realistically carried out. We urge practitioners to be aware of current vulnerabilities when deploying deep learning systems in clinical settings, and encourage the machine learning community to further investigate the domain-specific characteristics of medical learning systems.
1 INTRODUCTION
Medical deep learning has advanced rapidly toward clinical deployment, while adversarial examples expose vulnerabilities that can manipulate model outputs without access to training or deployed models. This paper extends adversarial-attack analysis to medical systems and examines why healthcare may be especially exposed.
- Clinical progress: Deep learning has achieved parity with, and sometimes exceeded, individual physicians on tasks in radiology, pathology, dermatology, and ophthalmology.These results have prompted speculation that medical-imaging specialties could be substantially reshaped.
- Adversarial vulnerability: Adversarial examples are engineered inputs that cause misclassification and can expose vulnerabilities in state-of-the-art learning systems.They have been studied both as probes of deep-learning limitations and for cybersecurity implications.
- Medical deployment: Healthcare automation may create opportunities for fraud and harm because diagnostic, decision-making, and reimbursement systems can be manipulated with adversarial examples.The concern applies both when human clinicians are removed from the loop and when they remain involved.
- Paper approach: The paper studies three medical deep-learning systems modeled on state-of-the-art classifiers and combines attack experiments with analysis of healthcare incentives and operational scenarios.The authors aim to demonstrate practical feasibility and motivate research on defenses tailored to healthcare.
- Adversarial vulnerability: End-users can subtly influence a running model’s behavior without directly accessing the model or its hosting infrastructure.This distinguishes adversarial attacks from risks requiring influence over training or deployed-system access.
2 IDENTIFYING FACTORS IN THE U.S. HEALTHCARE SYSTEM THAT FAVOR ADVERSARIAL ATTACKS
The U.S. healthcare system combines large financial incentives with technical and organizational conditions that may favor adversarial attacks. These include ambiguous labels, standardized imaging, common architectures, fragmented data exchange, difficult infrastructure updates, and many potential attackers.
- 2.1 Background on the healthcare economy and possible incentives for fraud via adversarial attacks: Algorithmic reimbursement decisions could make undetectable manipulation of diagnostic outputs a means of influencing billions of dollars in healthcare payments.Healthcare-record data are often generated to justify payments from insurers to providers.
- 2.1 Background on the healthcare economy and possible incentives for fraud via adversarial attacks: Algorithmic evaluation of medical-imaging endpoints could let clinical-trial sponsors imperceptibly bias results even when images are verified as belonging to the correct patients.The paper describes this as a way to influence trial outcomes without changing patient identity.
- 2.1 Background on the healthcare economy and possible incentives for fraud via adversarial attacks: Ambiguous medical ground truth makes selectively perturbing borderline images difficult to detect through expert human review.Specialty radiologists may disagree even on well-defined imaging tasks.
- 2.2 Distinctive technical sources of vulnerability to adversarial attacks among medical machine learning systems: Standardized acquisition conditions reduce the invariance demands placed on medical attacks compared with attacks targeting dynamic real-world systems.Medical images generally use predefined positioning and exposure, limiting variation in lighting and viewpoint.
- 2.2 Distinctive technical sources of vulnerability to adversarial attacks among medical machine learning systems: Common fine-tuned ImageNet architectures may make transferable attacks easier to construct across medical systems.The paper also identifies fragmented data interchange, hard-to-update hospital infrastructure, limited computational training, and many possible attackers as broader vulnerabilities.
- 2.2 Distinctive technical sources of vulnerability to adversarial attacks among medical machine learning systems: Biomedical images’ patient-specific signatures can help detect image substitution but do not protect against adversarial examples.Retinal images, X-rays, and skin photographs may be as unique to their owners as fingerprints.
3 ATTACKING REPRESENTATIVE CLINICAL DEEP LEARNING SYSTEMS
The paper evaluates white- and black-box adversarial attacks against three representative clinical classifiers, covering retinal fundoscopy, chest X-rays, and dermoscopy. Across these tasks, imperceptible and patch-based attacks substantially manipulated model predictions.
- Clinical systems: The study attacked classifiers for referable diabetic retinopathy, pneumothorax, and melanoma using publicly available medical-image data.The models were based on pretrained ImageNet architectures fine-tuned for each clinical task.
- Attack construction: White-box and black-box attacks were implemented under both human-imperceptible and adversarial-patch threat models.White-box attacks targeted the victim model directly, whereas black-box attacks transferred examples from an independently trained model with the same architecture.
- Results: Baseline AUROCs were 0.910 for diabetic retinopathy, 0.936 for pneumothorax, and 0.86 for melanoma.The authors state that direct comparison with prior state-of-the-art models was unfeasible because of differing data sources and train-test partitions.
- Results: Projected gradient descent attacks achieved effective AUROCs of 0.000 and accuracies of 0% for all white-box attacks.Black-box attacks produced AUROCs below 0.10 across tasks, with accuracies ranging from 0.01% on fundoscopy to 37.9% on dermoscopy.
- Results: Adversarial patch attacks achieved effective AUROCs of 0.000 and accuracies below 1% for white-box attacks across all tasks.Black-box patch attacks achieved AUROCs below 0.005 and accuracies below 10% across tasks, while natural-patch controls retained AUROCs of 0.48-0.83 and accuracies of 67.5%-92.1%.
4 DISCUSSION
The discussion argues that medical deep learning systems are practically vulnerable to adversarial manipulation, with realistic opportunities and incentives for attacks. It highlights unresolved defenses, infrastructure-wide implementation challenges, and ethical tradeoffs between robustness and accuracy.
- Attack feasibility: Adversarial attacks may remain feasible against highly accurate medical classifiers, including attacks that are human-imperceptible or do not require direct model access.Universal adversarial patches could be applied to any image and potentially introduced before image capture, bypassing point-of-capture hashing.
- Healthcare incentives: Healthcare’s complex administrative and reimbursement processes can create both opportunities and incentives for adversarial manipulation and fraud.The paper frames deployed classifiers as confirmatory tests that may influence reimbursement decisions and patient identity checks.
- Hypothetical examples: Radiology attacks could manipulate tumor-burden endpoints in clinical trials or alter chest-X-ray results used to justify higher-reimbursement procedures.The paper describes undetectable perturbations that could guarantee positive trial endpoints and scenarios analogous to dermatology manipulation.
- Defenses: Algorithmic defenses remain an open challenge, because no simple general fix is available and some gradient-obfuscation strategies have been quickly broken.Promising robustness results and theoretical guarantees have largely been demonstrated on smaller datasets or with insufficient accuracy.
- Defenses: Healthcare-wide infrastructural defenses should be considered, but standardizing them across the system requires substantial coordination between medical and computer-science communities.The paper discusses image hashing and related measures for detecting or preventing image tampering.
- Ethical tradeoffs: Adversarial robustness can trade off against diagnostic accuracy, creating an ethical question about how medical systems should weigh attack protection against inaccurate diagnoses.The paper calls for this tradeoff to be quantified and made explicit for informed system design.
5 CONCLUSION
The conclusion presents adversarial examples as a disproportionately serious threat to medical deep learning because of systemic and technological factors. It calls for defenses and broader discussion among researchers, providers, payers, and policymakers before clinical integration proceeds.
- Conclusion: Medical adversarial examples pose a disproportionately large threat because of systemic and technological factors, and the paper illustrates how attacks could be executed.The conclusion connects the threat analysis to practical examples relevant to healthcare deployment.
- Recommendations: The authors recommend infrastructural and algorithmic research to make attacks infeasible or at least retrospectively identifiable.The recommendation is directed particularly toward machine-learning researchers.
- Recommendations: The paper urges medical providers, payers, and policymakers to discuss how vulnerable deep learning algorithms should be incorporated into the clinical ecosystem.The conclusion frames clinical incorporation as requiring meaningful discussion among healthcare stakeholders.