Source-linked AI summary
The Threat of Offensive AI to Organizations
Yisroel Mirsky, Ambra Demontis, Jaidip Kotak, Ram Shankar, Deng Gelei, Liu Yang, Xiangyu Zhang, Wenke Lee, Yuval Elovici, Battista Biggio
arXiv:2106.15764v1cs.AIcs.CRcs.CYcs.LG
TL;DR
Organizations already face cyberattacks, and AI can expand adversaries’ capabilities beyond attacks on AI systems. This survey reviews offensive AI literature and ranks threats through an expert user study. It reports that exploit development, social engineering, and information gathering rank among the most threatening categories, while noting limits in its small survey sample.
Problem
Although AI can enhance cyberattacks, the threat and its impact on the cyber kill chain require analysis in organizational settings.
Source: §1 · p. 2Method
The authors reviewed offensive AI literature, then surveyed experts to rank threats and assess their impact on the cyber kill chain.
Source: §5 · p. 15Results
Exploit development, social engineering, and information gathering are the top threat categories, and AI affects the cyber kill chain most during initial attack steps.
Source: §6.1 · p. 19, §5.3 · p. 19Takeaways & Limitations
AI security technologies can also be used offensively, including through dual-purpose, repurposed, or inverted methods.
Source: §6.1 · p. 19Only 22 people participated; assuming 100,000 eligible respondents, the authors estimate a 20% margin of error at 95% confidence.
Source: §5.1 · p. 15
Abstract
from arXiv · showhide
AI has provided us with the ability to automate tasks, extract information from vast amounts of data, and synthesize media that is nearly indistinguishable from the real thing. However, positive tools can also be used for negative purposes. In particular, cyber adversaries can use AI (such as machine learning) to enhance their attacks and expand their campaigns. Although offensive AI has been discussed in the past, there is a need to analyze and understand the threat in the context of organizations. For example, how does an AI-capable adversary impact the cyber kill chain? Does AI benefit the attacker more than the defender? What are the most significant AI threats facing organizations today and what will be their impact on the future? In this survey, we explore the threat of offensive AI on organizations. First, we present the background and discuss how AI changes the adversary's methods, strategies, goals, and overall attack model. Then, through a literature review, we identify 33 offensive AI capabilities which adversaries can use to enhance their attacks. Finally, through a user study spanning industry and academia, we rank the AI threats and provide insights on the adversaries.
1 INTRODUCTION
AI can expand adversaries’ campaigns beyond attacks on AI systems, creating a need to understand offensive AI specifically in organizational settings. The paper surveys these threats and uses expert responses to rank them.
1 INTRODUCTION
- The introduction frames AI as a tool that can improve malicious campaigns as well as automate tasks, retrieve information, and synthesize media.AI can support attacks beyond poisoning or fooling machine-learning models, including spear phishing and stealthy lateral movement.Source: §1 · p. 2
1.1 Goal
- The paper surveys offensive AI for enterprise security, a context not specific to earlier work on AI threats to society overall [40].Its stated aims include understanding organizational impacts, prioritizing defensive research, and identifying near-future trends.Source: §1.1 · p. 2
1.2 Methodology
- The authors reviewed literature using MITRE ATT&CK to organize attack tactics, then cross-inspected workgroup reviews for correctness and completeness.Experts from academia, industry, and government were surveyed to assess concerns and rank threats.Source: §1.2 · pp. 2–3
1.3 Main Findings
- The survey identifies exploit development, social engineering, and information gathering as the top threat categories; 24 of 33 capabilities pose significant threats.Industry and academia often disagree on top threats, but both rank impersonation second; together they consider it the biggest threat.Source: §1.3 · p. 3
1.4 Contributions
- The paper contributes an organizational threat overview, a survey of 33 capabilities, a user-study threat ranking, and a forecast of future attack-strategy shifts.Table 1 gives examples of where models may be trained and executed in organizational attacks.Source: §1.4 · p. 3, Table 1 · p. 4
2 BACKGROUND ON OFFENSIVE AI
The paper distinguishes using AI to improve attacks from exploiting vulnerabilities in AI systems. It describes common offensive AI tasks, attack settings, and adversarial machine-learning vectors.
2 BACKGROUND ON OFFENSIVE AI
- AI is intelligence demonstrated by a machine; machine learning learns from examples, and deep learning uses an extensive neural network as its predictive model.Deep learning’s ability to reason over and generate complex data has supported its use in automation, forecasting, and planning.Source: §2 · p. 4
2.1 Training and Execution
- Where models are trained and executed depends on the task and strategy; attackers may train offsite and execute onsite, offsite, or in both locations.The authors describe designing and evaluating models offsite before using them against an organization.Source: §2.1 · p. 4
- Offensive AI takes two forms: using AI to improve attack efficiency, or exploiting defenders’ AI products and solutions.The latter is commonly called adversarial machine learning.Source: §2.1 · p. 4
2.2 Attacks Using AI
- Offensive AI tasks include prediction, generation, analysis, retrieval, and decision-making, supporting uses such as vulnerability localization, media tampering, and attack planning.A deepfake is believable media created by a deep-learning model and can impersonate a victim’s voice or face in phishing.Source: §2.2 · p. 5
2.3 Attacks Against AI - Adversarial Machine Learning
- AI attacks against AI systems can target training data, test inputs, model responses, training code, or model parameters.Attackers may have white-box, gray-box, or black-box knowledge; white-box access is described as unlikely in practice, though useful as an empirical upper bound.Source: §2.3 · pp. 5–6
3 OFFENSIVE AI VS ORGANIZATIONS
AI can change how adversaries pursue organizational attacks by automating tasks, improving tools, and evading detection. The survey connects these effects to attacker motivations, goals, and capabilities.
3 OFFENSIVE AI VS ORGANIZATIONS
- The section frames offensive AI as changing adversary methods, strategies, goals, and requirements, and says later sections detail capabilities identified through literature review.AI-based tools perform specific tasks, while AI-driven bots can execute or coordinate attack steps autonomously.Source: §3 · p. 6, §3.2.4 · p. 9
3.1 The Attack Model
- The paper extends the enterprise attack model from varied threat agents and goals to multi-step campaigns such as APTs, which defenders can disrupt earlier in the kill chain.AI-capable adversaries can automate tasks, enhance tools, and evade detection, affecting the kill chain.Source: §3.1 · pp. 6–7, §3.2 · p. 7
3.2.1 The Three Motivators of Offensive AI
- AI supports three overlapping attacker motivations: broader coverage, faster progress toward goals, and higher likelihood of success.Automation can let adversaries target more organizations with greater precision and a smaller workforce, while reducing time in a defender’s network.Source: §3.2.1 · p. 7
3.2.2 AI-Capable Threat Agents
- AI expertise varies across threat agents, but increasing access to plug-and-play deepfake tools may narrow the sophistication gap over time.The passage contrasts state actors’ potential ability to use intelligent automated botnets with hacktivists’ likely difficulty doing so.Source: §3.2.2 · pp. 7–8
3.2.3 New Attack Goals
- AI-capable agents may pursue sabotage, espionage, information theft, or persistent footholds, including model poisoning, evidence manipulation, and theft of training data.The passage also describes stealing organizational models for future white-box attacks and targeting audio or video records to create deepfake impersonations.Source: §3.2.3 · pp. 8–9
3.2.4 New Attack Capabilities
- The survey identifies 33 offensive AI capabilities in seven categories, each tied to coverage, speed, or success, and maps them to attack steps across the enterprise ATT&CK model.An edge in Fig. 1 indicates that a capability directly helps achieve an attack step.Source: Figure 1 · p. 8, §3.2.4 · p. 9
4 SURVEY OF OFFENSIVE AI CAPABILITIES
The literature survey describes offensive AI capabilities across enterprise attack steps, including automation, targeting, phishing, and record tampering.
4.1 Automation
- Automation reduces adversary effort and increases flexibility, enabling larger campaigns that depend less on command-and-control signals.The section surveys capabilities grouped into seven categories, including automation, resilience, credential theft, exploit development, information gathering, social engineering, and stealth.Source: §4.1 · p. 9, §4 · p. 9
4.1.1 Attack Adaptation
- AI can adapt malware or attacks to an unknown environment by identifying target systems before exploitation, potentially improving success and avoiding detection.Deep learning can extract critical information onsite rather than transferring screenshots.Source: §4.1.1 · p. 9
4.1.2 Attack Coordination
- AI can coordinate bots to select attack times and targets, share asset locations, and pass stolen credentials or exploits between compromised machines.The cited work discusses hypothetical swarm malware and deep-learning-triggered attacks.Source: §4.1.2 · p. 9
4.1.3 Next hop targeting
- Reinforcement learning and attack-graph methods can help choose the next asset during lateral movement and plan multi-step attacks on enterprise networks.The passage notes that Q-learning has enabled the approach to work on large-scale enterprise networks.Source: §4.1.3 · p. 10
4.1.4 Phishing Campaigns
- AI can automate phishing campaigns, while deepfake spear phishing can clone a colleague’s voice from five seconds of audio and exploit the victim’s trust.The described sequence uses social media to identify a colleague, clones the voice, then calls the victim in that voice.Source: §4.1.4 · p. 10
4.1.5 Point of Entry Detection
- Models trained on organizational attributes can help adversaries select organizations expected to receive more intrusions and choose attack vectors.The cited example predicts intrusion counts from organization attributes [118].Source: §4.1.5 · p. 10
4.1.6 Record Tampering
- Machine learning can tamper with business data, evidence, records, medical imagery, or satellite imagery, and deep-learning-tampered records can fool human observers.The passage says such tampering can be performed autonomously onsite.Source: §4.1.6 · p. 10
4.2 Campaign Resilience
AI can strengthen campaign resilience through planning, persistence, and obfuscation.
4.2.1 Campaign Planning.
- ML-based cost-benefit analysis can guide tool development, infrastructure layout, beachhead selection, and offsite digital twins for tuning models and developing malware.The proposed planning uses reconnaissance information to model the victim’s network.Source: §4.2.1 · p. 10
4.2.2 Malware Obfuscation.
- GANs and RNNs can generate functionally equivalent obfuscated software, helping conceal malware intent and infrastructure and prolong campaigns.DeepObfusCode [59] uses RNNs to generate ciphered code; backdoors can also be hidden in open-source projects.Source: §4.2.2 · p. 10
4.2.3 Persistent Access.
- Multiple coordinated backdoors and ML-timed payload deployment can slow campaign purges and help malware avoid detection.The passage also describes an insider-placed neural compute stick for covert onsite deep-learning operations.Source: §4.2.3 · p. 11
4.2.4 Virtualization Detection.
- ML can help malware detect virtual environments through system timing and other properties before triggering, avoiding sandbox analysis.The passage cites system-timing-based detection.Source: §4.2.4 · p. 11
4.3 Credential Theft
AI can support credential theft by exploiting biometric systems, mining stored data and side channels, and inferring what users type.
4.3.1 Biometric spoofing.
- AI can spoof or evade fingerprint, face, and voice authentication, and ‘sponge’ samples can make surveillance cameras unresponsive or drain their batteries.Examples include fingerprint “Master Prints” [33], adversarial glasses, and cloned or spoofed voices [225].Source: §4.3.1 · p. 11
4.3.2 Cache mining.
- ML can locate credentials in cache dumps and identify cookies containing session information.The cited examples identify credentials in graphic-library cache dumps and session cookies.Source: §4.3.2 · p. 11
4.3.3 Implicit key logging.
- AI can infer typing and phone touches from motion, key sounds and timing, or video cues such as eye, device, and hand movements.The passages describe smartphone sensors and nearby surfaces for motion inference, and nearby phones or compromised cameras for video observation.Source: §4.3.3 · p. 11
4.3.4 Password Guessing.
- GANs and RNNs can learn password patterns from leaked databases, while user-specific information may improve guessing; one cited model performed poorly on Russian passwords.The Russian-password limitation is reported for models such as, while the user-information approach is described in [195].Source: §4.3.4 · p. 12
4.3.5 Side Channel Mining.
- ML can extract secrets from diverse side channels, including power, electromagnetic emissions, processing time, cache activity, and network-traffic timing.The passage states that ML can mine nearly any kind of side channel.Source: §4.3.5 · p. 12
4.4 Exploit Development
Exploit development can use ML to understand compiled software and identify vulnerabilities that attackers may exploit.
4.4.1 Reverse Engineering.
- ML can support reverse engineering through binary similarity, behavior segmentation, higher-level code representations, and protocol or state-machine reversal.Examples include CAN bus data, network protocols, and commands.Source: §4.4.1 · p. 12
4.4.2 Vulnerability Detection.
- Static ML tools can detect known vulnerabilities in source code or compare binary control-flow graphs with known vulnerabilities.The cited approaches cover open-source code [47] [127] and compiled commercial products.Source: §4.4.2 · p. 12
- Dynamic ML-guided fuzzing can reach buggy code faster, and AI techniques can mitigate symbolic execution’s large state space.The passages cite guided fuzzing [19] [52] [201] and state-space approaches [113].Source: §4.4.2 · p. 12
4.5 Information Gathering
AI’s data-mining and language-processing capabilities can help adversaries collect and distill actionable campaign intelligence.
4.5.1 Mining OSINT.
- For OSINT, AI can camouflage probes, prioritize sites and data elements, gather network and personnel information, and extract or translate relevant documents and identifiers.Examples include crawler-like probe traffic [53], graph-based network analysis [13], social-media extraction [169], and document processing.Source: §4.5.1 · pp. 12–13
4.5.2 Model Theft.
- An adversary may steal an AI model for its intellectual property, training-set information, or use in white-box attacks.Model parameters and hyperparameters may be copied through query responses, side channels, or hardware-level analysis.Source: §4.5.2 · p. 13
4.5.3 Spying.
- Spyware can use audio and video to map rooms, identify physical security features, mine conversations, and analyze encrypted traffic for transcripts or activity.Examples include room modeling from ultrasonic echoes, object recognition, conversation mining [185], and encrypted-traffic analysis.Source: §4.5.3 · p. 13
4.6 Social Engineering
AI extends social engineering by helping adversaries exploit trust, impersonate people, build personas, select vulnerable targets, and track organizational members.
4.6 Social Engineering
- AI enhances adversaries’ ability to exploit human emotions and trust, which the paper identifies as a longstanding organizational security weakness.Source: §4.6 · p. 13
4.6.1 Impersonation (Identity Theft).
- Deepfakes can reenact or alter a victim’s face and voice for scams, blackmail, defamation, or spear phishing.Reenactment can be real-time, and training may require only a few images or seconds of audio; high-quality fakes still need substantial audio/video data, yet victims may trust imperfect fakes under pressure.Source: §4.6.1 · p. 13
4.6.2 Persona Building.
- AI can help adversaries clone and alter social-network profiles, maximize connection acceptance, and use chatbots to maintain conversations [125] [188].Source: §4.6.2 · p. 13
4.6.3 Spear Phishing.
- Real-time deepfakes have been used in call-based spear phishing, while generated or style-transferred text can target victims or mimic colleagues [58] [195] [71].The paper cites a 2019 CEO scam resulting in a $240k loss.Source: §4.6.3 · p. 13
4.6.4 Target Selection.
- Adversaries can identify susceptible employees from their attributes and use sentiment analysis to find disgruntled staff for recruitment as insiders.Source: §4.6.4 · pp. 13–14
4.6.5 Tracking.
- Machine learning can track personnel across social media, infer their locations, and uncover hidden business relationships.Source: §4.6.5 · p. 14
4.7 Stealth
The section surveys how adversaries use or abuse AI to avoid detection across organizational systems and during movement, communication, and data transfer.
4.7 Stealth
- In multi-step attacks, adversaries use or abuse AI to evade detection and keep covert operations on track.Source: §4.7 · p. 14
4.7.1 Covering tracks.
- Adversaries can remove abnormal log entries, hide malware logs and training data, or plant hardware and software trojans in deep-learning intrusion detectors [35] [122].Source: §4.7.1 · p. 14
4.7.2 Evading HIDS (Malware Detectors).
- Malware can evade host detectors by splitting code across processes, adding bytes or behavior-neutral code, or using LIME explanations to guide manual changes [17] [61].The paper notes that modifying malware without breaking its malicious functionality is not easy.Source: §4.7.2 · p. 14
4.7.3 Evading NIDS (Network Intrusion Detection Systems).
- Network evasion includes generating URLs that avoid phishing or malicious-URL detectors and shaping or retiming traffic to hide it [21, 175, 208, 85, 166, 200].Source: §4.7.3 · p. 14
4.7.4 Evading Insider Detectors.
- With a user’s credentials, adversaries can use role and organizational-structure information to make operations appear legitimate to insider-detection mechanisms.Source: §4.7.4 · p. 14
4.7.5 Evading Email Filter.
- Adversarial machine learning can evade email filters, and adversaries can poison filter training by sending emails intended to be detected [74].Source: §4.7.5 · p. 14
4.7.6 Exfiltration.
- For exfiltration, adversaries can disguise traffic, use permitted channels, compress or encrypt data, summarize media onsite, or exploit audio side channels on air-gapped networks [9].Source: §4.7.6 · p. 15
4.7.7 Propagation & Scanning.
- For stealthy lateral movement, adversaries can route around assets or subnets with certain intrusion detectors and model scans on locally observed patterns.Source: §4.7.7 · p. 15
5 USER STUDY & THREAT RANKING
The user study surveyed experts to rank offensive AI capabilities and assess their effects across the cyber kill chain. Respondents judged many capabilities threatening and reported attacker advantages across several kill-chain steps.
5.1 Survey Setup
- The survey asked 22 experts—half from academia and half from industry—204 questions; assuming 100,000 eligible respondents, its estimated margin of error was about 20%.Participants’ responses were anonymized and represented their personal views, not their employers’ views.Source: §5.1 · p. 15
5.2 Threat Ranking
- Threat ranking used profit, achievability, defeatability, and harm scores; after normalization, a threat score above 1 indicated a significant threat under the study’s model.Motivation was modeled from profit and achievability, while defeatability and harm also informed risk and threat.Source: §5.2 · pp. 15–16
5.2.1 OAC Threat Ranking.
- 23 of 33 OACs (72%) were significant threats; impersonation, spear phishing, and model theft ranked highest, and AI-run phishing campaigns were also considered a large threat.The authors attribute campaign risk to increased phishing volume and quality: even if 99% of attempts fail, some may get through and cause damage.Source: §5.2.1 · pp. 16–17, Figure 3 · p. 17
5.2.2 Industry vs Academia.
- Academia viewed AI as a greater organizational threat overall, but both groups ranked exploit development, social engineering, and information gathering as the top OAC categories.All categories were significant threats to academic respondents; industry respondents did not rate stealth, credential theft, or campaign resilience as significant.Source: §5.2.2 · pp. 17–18
- Industry ranked reverse engineering first, while academia ranked biometric spoofing first (fourth overall); both groups rated stealth least threatening, and IDS evasion was considered highly defeatable.The authors suggest that attackers’ inability to evaluate evasion techniques inside the target network risks detection.Source: §5.2.2 · p. 18
5.3 Impact on the Cyber Kill Chain
- Participants felt AI enhances adversaries’ ability to traverse the kill chain, with considerable benefits during its first three steps and a somewhat greater attacker advantage for most steps.They judged defenders advantaged in detecting execution, persistence, and privilege escalation, since defenders can train and evaluate models onsite while attackers cannot.Source: §5.3 · pp. 18–19
6 DISCUSSION
The discussion interprets the survey’s threat rankings, considers how offensive AI may change attacks and defenses, and outlines research and operational responses. Its near-term forecast is bounded by current limits on autonomous AI, even as it expects more deepfake phishing and larger-scale campaigns.
6.1 Insights, Observations, & Limitations
- Industry ranks social engineering and software analysis highly, while academics are more concerned with biometric attacks because of differences in ML expertise and organizational use of biometrics.The authors link industry’s concern to organizational exposure to software and personnel attacks, and academic concern to its greater use of ML.Source: §6.1 · p. 19
- Near-term offensive AI growth is expected at reconnaissance, resource development, and impact, while autonomous interaction with complex organizational networks remains unlikely.The authors attribute this boundary to current limits on AI learning and to large deep-learning models being detectable when transferred or run onsite.Source: §6.1 · p. 19
- Security AI can be dual-purpose, repurposed, or inverted for offensive use, and the advantage is asymmetric: deepfakes favor attackers while anomaly detection favors defenders.Examples include using explainable AI to hide artifacts and repurposing insider-detection models to cover tracks.Source: §6.1 · p. 19
6.2 The Industry’s Perspective
- Rule-based automation such as HAMMERTOSS predates offensive AI, but AI’s potential scale and speed can make attacks more damaging.Source: §6.2 · p. 20
- 49% of 102 surveyed cybersecurity organizations expected offensive AI within 12 months [4], while 96% of 309 organizations were already investing in defenses against AI-powered attacks.Source: §6.2 · p. 20
6.3 What’s on the Horizon
- AI could enable more deepfake phishing and parallel, high-volume attacks, prompting defensive bots; the authors say humans should retain high-level control and call for further policy research.They expect deepfake phishing to grow because it is mature, harder to mitigate than regular phishing, exploits trust, can expedite attacks, and is an unfamiliar tactic.Source: §6.3 · p. 20
6.4 What can be done?
- The authors recommend defenses against AI-enabled attacks and extending MLOps to MLSecOps by incorporating AI/ML security testing, protection, and monitoring.They also recommend personnel preparation, robust deepfake detection, and post-processing tools to protect software from analysis.Source: §6.4 · pp. 20–21
7 CONCLUSION
The conclusion recaps the survey’s identification and categorization of offensive AI threats against organizations. It also notes that experts ranked those threats and that the paper offers insights and directions for future work.
7 CONCLUSION
- The survey explored and categorized offensive AI threats against organizations.Source: §7 · p. 21
- The authors ranked the identified threats through a domain-expert user study.Source: §7 · p. 21
- The conclusion presents insights on the results and directions for future work.Source: §7 · p. 21