PaperlayerView exact revision on arXiv

Source-linked AI summary

The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation

Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, Hyrum Anderson, Heather Roff, Gregory C. Allen, Jacob Steinhardt, Carrick Flynn, Seán Ó hÉigeartaigh, SJ Beard, Haydn Belfield, Sebastian Farquhar, Clare Lyle, Rebecca Crootof, Owain Evans, Michael Page, Joanna Bryson, Roman Yampolskiy, Dario Amodei

arXiv:1802.07228v2cs.AIcs.CRcs.CY

TL;DR

Malicious uses of AI raise security questions because digital, physical, and political security are connected. This report surveys potential threats and proposes ways to forecast, prevent, and mitigate them; it concludes that AI can augment cyberattacks and defenses, but AI-based defense is not a panacea.

Abstract

from arXiv · show

This report surveys the landscape of potential security threats from malicious uses of AI, and proposes ways to better forecast, prevent, and mitigate these threats. After analyzing the ways in which AI may influence the threat landscape in the digital, physical, and political domains, we make four high-level recommendations for AI researchers and other stakeholders. We also suggest several promising areas for further research that could expand the portfolio of defenses, or make attacks less effective or harder to execute. Finally, we discuss, but do not conclusively resolve, the long-term equilibrium of attackers and defenders.

Executive · 1. Policymakers should collaborate closely with technical researchers to investigate, prevent, and mitigate potential

The report surveys how malicious uses of AI may change security threats and proposes ways to forecast, prevent, and mitigate them. It offers recommendations and research priorities while leaving the long-term attacker–defender equilibrium unresolved.

Executive

1. Policymakers should collaborate closely with technical researchers to investigate, prevent, and mitigate potential

  • Policymakers should work with technical researchers, while researchers and companies take dual-use risks seriously and draw on mature practices and broader expertise.The recommendations also call for reaching out to relevant actors, importing applicable practices from computer security, and expanding stakeholder and domain-expert participation.Source: malicious uses of AI · p. 4, harmful applications are foreseeable · p. 4, of AI · p. 4

Executive

Executive · Physical security

Executive · Digital security

Executive · Political security

Executive · Priority areas for further research

Executive · Promoting a culture of responsibility

Introduction · Scope · Related Literature

The report examines intentional malicious uses of AI that undermine security, asking how to forecast, prevent, and mitigate their harms while avoiding delays to beneficial AI applications. It surveys digital, physical, and political threats and sets out the report’s scope, relationship to adjacent literature, and structure.

Introduction

  • Malicious AI use could threaten digital, physical, and political security through hacking or social engineering, weaponized consumer drones, surveillance, repression, and targeted disinformation.The report defines malicious use broadly as practices intended to compromise the security of individuals, groups, or society.Source: Introduction · pp. 9–10
  • The report asks how to forecast, prevent, and mitigate harmful uses of AI, drawing on a 2017 Oxford workshop and subsequent research.Its stated aim is to prevent harms and avoid delaying beneficial AI applications.Source: Introduction · p. 10

Scope

  • The report considers AI technologies demonstrated or plausible within five years, especially those leveraging machine learning, when intentionally deployed or compromised to undermine another party’s security.This focus contrasts with literature emphasizing unintentional AI harms such as algorithmic bias.Source: Scope · p. 10
  • The report excludes indirect societal threats and system-level risks from interactions among non-malicious actors, including safety races and conflicts involving faster autonomous weapons.The authors characterize these excluded threats as important and urgent but beyond the report’s scope.Source: Scope · pp. 10–11

Related Literature

  • Although prior work has highlighted malicious AI and examined particular scenarios, the intersection of AI and malicious intent had not been analyzed comprehensively.Related areas include cybersecurity, drones, lethal autonomous weapons, social media bots, and terrorism.Source: Related Literature · p. 11
  • The report distinguishes its focus on intentional harmful use from AI safety research on unintended harms, while noting a recent report with greater emphasis on U.S. national security.The comparison describes the scope of adjacent work without treating the topics as mutually exclusive.Source: Related Literature · p. 11
  • The report proceeds from a general framework and malicious-use scenarios to domain analyses, interventions, and strategic analysis, with appendices on the workshop and further research.Its domain analyses cover digital, physical, and political security; interventions aim to assess risks, protect victims, and limit dangerous AI capabilities.Source: Related Literature · p. 11

General … General Implications for the Threat Landscape

Recent AI progress creates security-relevant dual-use capabilities, and absent adequate defenses it is expected to expand existing threats and change their character. The section identifies signals of likely progress and AI properties that matter for security, while noting uneven progress and limits.

AI Capabilities

  • Image-recognition accuracy rose from around 70% to 98%, surpassing the 95% human benchmark, while generated images became nearly indistinguishable from photographs.The image-recognition comparison is on the ImageNet benchmark.Source: AI Capabilities · p. 13, Figure 1 · p. 14, Figure 2 · p. 15
  • Recent advances span chess, Atari, Go, Dota 2, speech recognition, language comprehension, and vehicle navigation, and techniques may generalize to more security-relevant tasks.Montezuma’s Revenge is also yielding to methods that search for long-term strategies, use auxiliary rewards, or learn from a few human demonstrations.Source: AI Capabilities · p. 13
  • Tasks may be promising when they have a perfect model or simulation, short-term progress signals, abundant human-performance data, or little need for common sense, but progress is uneven and tractable tasks are often studied more.Many AI areas, including much of robotics, have changed far less dramatically over the past decade.Source: AI Capabilities · p. 13

NEC UIUC

  • Although today’s AI performs well on a small portion of human-capable tasks, researchers surveyed generally expect systems eventually to reach and exceed human performance across the tasks surveyed, most likely within fifty years.AI may play central roles in security issues before it can outperform humans at everything.Source: NEC UIUC · p. 16

Security-Relevant Properties of AI

  • AI is dual-use: systems and design knowledge can serve beneficial or harmful ends, and foundational AI research and many useful automated tasks can also enable harmful applications.For example, software-vulnerability analysis can be offensive or defensive, and package-delivery drones need not differ greatly in capability from drones used to deliver explosives.Source: Security-Relevant Properties of AI · p. 16
  • Once deployed, AI systems can perform tasks more quickly or cheaply than humans and scale through added computing power or copies; they may also exceed human performance and increase users’ anonymity and psychological distance.A facial-recognition system can be applied across many camera feeds, while autonomous weapons can let an attacker avoid being present at an attack.Source: Security-Relevant Properties of AI · pp. 16–17
  • AI developments can diffuse rapidly through software, scientific findings, and open research, while data poisoning, adversarial examples, and flaws in autonomous systems’ goals remain unresolved vulnerabilities distinct from traditional software flaws.Hardware may be costly to obtain or reproduce, but many algorithms are reproduced within days or weeks and limiting diffusion would likely be difficult.Source: Security-Relevant Properties of AI · pp. 17–18

General Implications for the Threat Landscape

  • Absent adequate defenses, AI progress will expand existing threats and alter their typical character, making attacks more effective, finely targeted, harder to attribute, and more likely to exploit AI vulnerabilities.These shifts necessitate vigorous responses.Source: General Implications for the Threat… · p. 18

Expanding Existing Threats … Digital Security

The report argues that AI can expand existing threats, enable attacks beyond human capabilities or by exploiting AI vulnerabilities, and change attacks’ effectiveness, targeting, attribution, and vulnerability profile. Its digital-security scenarios illustrate possible malicious uses rather than definitive or exhaustive forecasts.

Expanding Existing Threats

  • AI’s efficiency, scalability, and diffusion may expand who can carry out familiar attacks, how often they occur, and which targets are worth attacking, including mass spear phishing.Automating research and message generation could also let attackers target people who speak a different language from them.Source: Expanding Existing Threats · pp. 18–19
  • Greater anonymity and psychological distance may make actors more willing to attack, while robotics advances and cheaper hardware also contribute to threat expansion.Even drone operators who observe their targets can experience post-traumatic stress, illustrating the possible psychological effects of attack work.Source: Expanding Existing Threats · p. 19

Introducing New Threats

  • AI could enable attacks beyond human capabilities, such as realistic voice impersonation and control of drone swarms or systems operating without reliable communication.Indistinguishable synthetic recordings could enable new forms of disinformation and impersonation absent specially designed authentication measures.Source: Introducing New Threats · pp. 19–20
  • Attacks exploiting unresolved AI vulnerabilities could cause individual failures, such as adversarially induced car crashes, or simultaneous large-scale failures across connected systems.A compromised server directing autonomous weapons could cause friendly fire or civilian targeting.Source: Introducing New Threats · p. 20

Altering the Typical Character of Threats

  • AI-enabled attacks may become more effective and finely targeted, harder to attribute, and more likely to exploit AI vulnerabilities.Efficiency and scalability may reduce the trade-offs between attack frequency, scale, effectiveness, and precise targeting; increasing anonymity and AI-system prevalence support the other expected shifts.Source: Altering the Typical Character of… · pp. 21–22

Scenarios

  • The scenarios illustrate plausible malicious uses across digital, physical, and political security, but are neither definitive forecasts nor exhaustive lists.Some uses may not be technically possible or realized within five years, while others already occur in limited form and could scale or grow more powerful.Source: Scenarios · p. 23

Digital Security

  • Digital scenarios include personalized social-engineering lures, automated vulnerability discovery and hacking, and human-like denial-of-service attacks.AI could improve target selection, evasion, and responses to changes in a target’s behavior.Source: Digital Security · pp. 24–26
  • Other digital scenarios include automating criminal workflows and victim prioritization, poisoning or extracting AI models, and exploit-fed malware that can sustain ransomware infections.The ransomware example describes organized crime adopting exploit-finding techniques and malware infecting vulnerable IoT devices.Source: Digital Security · pp. 25–26

Physical Security

AI can threaten physical security by enabling attacks with repurposed commercial systems and autonomous capabilities that lower required expertise, increase attack scale, coordinate swarms, and distance attackers from their actions.

Physical Security

  • Commercial AI systems can be repurposed for harmful uses, including using drones or autonomous vehicles to deliver explosives or cause crashes.Source: Physical Security · p. 27
  • After visually detecting the finance minister, an intruding cleaning robot abandoned its routine and approached her; a concealed proximity-triggered explosive killed her and wounded nearby staff.The robot had initially performed ordinary cleaning tasks alongside the ministry’s robots.Source: Physical Security · p. 27
  • The robot’s point of sale was traced to a Potsdam office-supply store, but its cash purchase left investigators with no further leads about the buyer’s identity.Source: Physical Security · p. 27
  • AI-enabled automation, such as self-aiming long-range sniper rifles, can give low-skill individuals attack capabilities that previously required high skill.Source: Physical Security · p. 27
  • Autonomous systems can let one person or a small group cause more damage, while cooperating robotic swarms can provide broad surveillance and execute rapid, coordinated attacks.One example is a person launching many weaponized autonomous drones; autonomous operation can also distance physical attacks from the initiator in time and space, including where remote communication is unavailable.Source: Physical Security · pp. 27–28

Political Security

The section describes AI-enabled political threats ranging from automated state surveillance and predictive threat flagging to fabricated media, targeted influence, and manipulation of information flows.

Political Security

  • Automated image and audio processing can scale state surveillance and intelligence exploitation, including for suppressing debate.The report describes these platforms as extending states’ surveillance powers to massive-scale collection, processing, and exploitation of intelligence.Source: Political Security · p. 28
  • A predictive civil disruption system flags an individual as a potential threat.Source: Political Security · p. 28
  • Highly realistic fabricated video and audio can depict state leaders making inflammatory comments they never made.Source: Political Security · p. 29
  • Hyper-personalized disinformation can target individuals in swing districts to affect their voting behavior.Source: Political Security · p. 29
  • AI-enabled social-network analysis can identify key influencers for malicious offers or targeted disinformation.Source: Political Security · p. 29
  • Bot-driven information-generation attacks can swamp channels with false or distracting noise, making real information harder to acquire.Source: Political Security · p. 29
  • Media-platform curation algorithms can manipulate behavior by steering users toward or away from selected content.Source: Political Security · p. 29

Security · Domains · Context

The report examines AI threats across digital, physical, and political security, then considers how AI may change attacks and defenses in each domain. In cybersecurity, AI is already used defensively, while publicly documented AI-enabled attacks remain absent and the report urges preparation for possible near-term use.

Domains

  • The report groups threats as digital, physical, and political security, while noting that these categories can overlap.Digital threats compromise confidentiality, integrity, or availability; physical threats target people or infrastructure; political threats undermine truthful public discussion or just policy implementation.Source: Domains · p. 30
  • The report first describes pre-AI attack and defense conditions in each domain, then considers how further AI progress and diffusion could change attacks.The domain sections draw on the report’s earlier analysis of security-relevant AI properties and can be read independently.Source: Domains · p. 31
  • Absent preparation, near-term AI could increase the number, scale, and diversity of cyberattacks at a given capability level, so further defense innovations are needed for net benefit.AI-enabled cyber defenses are also being developed and deployed.Source: Domains · p. 31

Context

  • Cybersecurity is labor-constrained, and 82% of surveyed decision-makers reported needed cybersecurity skills shortages (McAfee and the Center for Strategic and International Studies, 2016).Many IT systems are sprawling, under-maintained, and insecure, while AI is already used in cyber defense, including for anomaly, spam, and malware detection.Source: Context · pp. 31–32
  • Publicly disclosed offensive AI use has been limited to white-hat experiments, with no publicly documented AI-based attacks; 62% of Black Hat survey respondents expected attacks within 12 months.Attribution can be difficult because techniques such as botnets and phishing may reflect AI, human labor, or simple automation; the report urges proactive preparation.Source: Context · p. 32
  • The report expects AI cybersecurity capabilities to improve rapidly, although the winning Cyber Grand Challenge system performed poorly against human security experts.Governments are interested in AI and cybersecurity, and the report points to advances such as deep reinforcement learning as a basis for expected improvement.Source: Context · pp. 32–33

How AI Changes The Digital Security Threat Landscape · Points of Control and Existing Countermeasures

AI may expand cyberattack scale and adaptability, while existing defenses—from user practices and research norms to centralized and machine-learning systems—have limitations. The sections call for proactive efforts as AI and cybersecurity evolve, and describe distinct challenges in defending against physical attacks.

How AI Changes The Digital Security Threat Landscape

  • AI-enabled offensive tools could scale cyberattacks: automated tailored phishing achieved high click rates, and attackers may use AI to reduce labor in vulnerability discovery.The report also cites malware messages sent to more than 10,000 U.S. Defense Department Twitter users, while noting that automation in that case is uncertain.Source: How AI Changes The Digital Security… · p. 33
  • AI can learn to evade defenses: researchers demonstrated generated command-and-control domains, malware binaries that bypass NGAV, and malicious documents that evade PDF classifiers.EDR platforms combine heuristics and machine learning and can counter typical human-authored malware, but research shows AI systems may evade them; attackers may also use VirusTotal feedback and accumulated data to refine variants and targeting.Source: How AI Changes The Digital Security… · pp. 33–34

Points of Control and Existing Countermeasures

  • User awareness can improve phishing recognition and security habits, but many users remain vulnerable to simple attacks, making scaled high-precision attacks concerning.The report says AI and cybersecurity will evolve rapidly in tandem and that proactive efforts are needed to stay ahead of motivated attackers.Source: Points of Control and Existing… · p. 35
  • Laws and norms, including the DMCA and CFAA, can constrain cyber activity, but may burden researchers; responsible disclosure, bug bounties, fuzzing, and ML code checks support vulnerability discovery.Enforcement across national boundaries is difficult, and laws and norms do not explicitly address AI.Source: Points of Control and Existing… · p. 35
  • Centralized defenses such as spam filters and network monitoring benefit from shared data and economies of scale, but compromised systems raise stakes and attackers can learn to evade them.Centralization may also help deploy AI-based defenses through aggregated data and concentrated expertise.Source: Points of Control and Existing… · p. 36
  • Deterring or punishing attackers requires attribution, which is difficult; even strong evidence may be withheld to protect sources, and some entities avoid punishment to preserve their own latitude.Attribution is necessary, though not sufficient, for deterrence and punishment.Source: Points of Control and Existing… · p. 36
  • Machine learning supports threat generalization, anomaly detection, and security analysts’ threat hunting, but defenses have little robustness testing against anticipating attackers and can expand attack surfaces.Surveys also find cybersecurity professionals have low confidence in AI-based defense systems today, while the relative effectiveness of technical defenses remains poorly analyzed.Source: Points of Control and Existing… · pp. 36–37
  • Physical defenses may lag as attack-enabling hardware and skills spread: defenses are capital-intensive, attacks can occur anywhere, and small groups or individuals may pose hard-to-anticipate risks.Unlike digital defense centered on key network nodes, physical defense resources are unavailable in many regions, motivating attention to robot supply chains and policy interventions.Source: Points of Control and Existing… · p. 38

Context · How AI Changes the Physical Security Landscape

Robotics’ expanding, increasingly autonomous deployment creates opportunities for malicious physical attacks, while cyber-physical and AI-specific vulnerabilities can compound risks. The section describes how autonomy, duration, terrain, and misplaced trust can widen the threat.

Context

  • Robotics is expanding across global commercial and other applications, with systems ranging from specialized machines to generic platforms usable for varied purposes.Source: Context · pp. 38–39
  • Most robots remain human-directed, but increasingly autonomous and semi-autonomous commercial and military systems are being developed, setting the stage for malicious applications.Source: Context · p. 39

How AI Changes the Physical Security Landscape

  • Customizable robots carrying dangerous payloads can enable precise long-distance attacks even without AI; autonomy can magnify the threat, while globalized markets impede prevention.Source: How AI Changes the Physical Security… · p. 40
  • Greater autonomy can let one person cause more damage and enable very large attacks or smaller groups; mature open-source detection, navigation, planning, and swarming tools can support them.Source: How AI Changes the Physical Security… · p. 40
  • Robots’ long operating durations and ability to navigate terrain or conditions difficult for humans can expose more spaces to automated physical attacks.Examples include using infrared or lidar in darkness or fog, tolerating smoke or toxic substances, and operating underwater without oxygen.Source: How AI Changes the Physical Security… · p. 40
  • Remote manipulation of robots in human-occupied spaces and insecure IoT attack vectors could let attackers subvert AI-controlled systems and cause more damage than human control would.Source: How AI Changes the Physical Security… · p. 40
  • AI-augmented IoT and robotic systems may face adversarial-example vulnerabilities, while excessive trust in autonomous robots creates added risks, especially for self-driving cars and autonomous weapons.Source: How AI Changes the Physical Security… · pp. 40–41

Points of Control and Existing Countermeasures · Context · How AI Changes the Political Security Landscape

AI may intensify political manipulation through scalable misleading content and surveillance, while existing technology can empower both authorities and challengers. Countermeasures offer some controls, but physical defenses remain costly and incomplete, and the long-term effects of malicious AI use are unclear.

Points of Control and Existing Countermeasures

  • Physical defenses are costly, unevenly distributed, and incomplete, leaving an extended period in which attacks may be easier to execute than to prevent.Detection, interception, and passive defenses tend to protect hard targets; a key challenge is achieving an acceptable defense cost-exchange ratio as attacks scale.Source: Points of Control and Existing… · pp. 41, 43
  • Hardware producers, distributors, software providers, and robot users offer possible control points, though the hardware market is more concentrated than software and may diffuse over time.Potential measures include hardening products, point-of-sale restrictions, registration, training, and no-fly zones; some powerful AI tools remain costly or difficult to use.Source: Points of Control and Existing… · pp. 41–42
  • International norms and arms-control discussions may inform stronger governance, but a near-term ban appears unlikely and such efforts are unlikely to stop motivated non-state attackers.States including the United States have issued policies on autonomy in weapons, and the U.S. Law of War Manual and the ICRC affirm human responsibility or involvement in force.Source: Points of Control and Existing… · p. 42

Context

  • Technology’s effects on political power are not straightforward: platforms can aid both incumbents and challengers, while algorithms, media incentives, and regime practices shape information flows.In liberal democracies, political influence is described as more emergent; some authoritarian states deliberately shape online and offline discussion.Source: Context · pp. 44–45

How AI Changes the Political Security Landscape

  • AI may intensify political manipulation as automated actors scale misleading content, while the long-term implications of these uses remain unclear.Existing bots already spread political messages and have been linked to effects on media coverage, beliefs, and fake-news propagation; greater scale and sophistication are technically possible.Source: How AI Changes the Political Security… · pp. 45–46
  • High-quality synthetic media and automated fake news could erode trust in audio and video evidence, giving an advantage to groups that thrive in low-trust societies.The report notes that recording and authentication technology currently has an edge over forgery, but that advantage may not persist as high-quality forgeries become less costly.Source: How AI Changes the Political Security… · p. 46
  • AI could enable precisely targeted persuasion and more efficient authoritarian surveillance, while content ranking and filtering could also be used to shape public opinion.Social-media profiles may support targeting; AI can prioritize surveillance attention and reduce monitoring costs, and platforms can promote or de-rank content.Source: How AI Changes the Political Security… · p. 47
  • Cyberattacks and physical attacks may disrupt political institutions or be used to justify more authoritarian policies.Source: How AI Changes the Political Security… · p. 48

Points of Control and Existing Countermeasures · Interventions

Existing countermeasures and platform interventions may help limit malicious uses of AI but have not resolved the risks; the report calls for closer collaboration, dual-use awareness, established security practices, and broader stakeholder dialogue.

Points of Control and Existing Countermeasures

  • Current proposals focus on protecting healthy democratic discourse, but none has definitively addressed the problems, and authoritarian governments pose a more daunting challenge.Source: Points of Control and Existing… · p. 48
  • Forgery and bot detection, media-authenticity tools, and fake-news detection efforts are developing, but detecting misleading news and images remains unsolved as generation improves rapidly.Authenticity measures can help establish whether media came from the relevant person or organization and remained untampered in transit.Source: Points of Control and Existing… · p. 48
  • Encryption, critical-thinking education, political-campaign tracking, and constructive-dialogue tools are among existing or proposed measures to improve information security and discourse.The report notes encryption is used by companies and other organizations to secure information transmissions.Source: Points of Control and Existing… · p. 48
  • Platform crackdowns on deepfakes were not fully successful but likely slowed dissemination among less sophisticated actors.Platforms can also use unique datasets, access controls, identity barriers, and dissemination limits to counter automated disinformation, censorship, and persuasion; transparency and potentially regulation can help align these corporate tools with the public interest.Source: Points of Control and Existing… · p. 49

Interventions

  • Policymakers should collaborate closely with technical researchers, while AI researchers should treat dual-use risks as their responsibility and proactively engage when harmful applications are foreseeable.The recommendations aim to avoid poorly informed policy responses and researchers’ reluctance to engage; collaboration should inform policy without impeding research unless the benefits justify it.Source: Interventions · p. 51
  • AI should adopt applicable dual-use best practices from mature fields such as computer security, including extensive red teaming.Red teams deliberately test systems and practices through bounded attacks, optionally answered by a blue team, to identify ways to improve security.Source: Interventions · p. 52
  • Discussions should include a wider range of stakeholders and domain experts to support public dialogue about AI’s legitimate and harmful uses.Relevant participants may include civil society, national-security experts, AI and cybersecurity researchers, businesses, ethicists, and the public; safeguards and degree of use can distinguish legitimate from illegitimate applications.Source: Interventions · p. 52
  • Citizens may need to maintain threat awareness and adopt defensive practices, but unequal technological literacy can complicate adaptation and create broader security risks.Insecure systems may be compromised and repurposed to support more capable, harder-to-attribute attacks.Source: Interventions · p. 52

Priority Areas for Further Research … Developing Technological and Policy Solutions

The report proposes a concise research agenda spanning AI cybersecurity, decisions about openness, a culture of responsibility, and technological and policy defenses. It frames these as areas for further investigation, not settled solutions, and says progress will require expertise, commitment, and resources.

Learning from and with the Cybersecurity Community

  • As AI systems grow more capable and widespread, cybersecurity should remain a major, ongoing priority, with relevant security best practices applied to AI.The report identifies increased control over physical systems, access to algorithms or trained models, and more scalable sophisticated attacks as AI-related cybersecurity concerns.Source: Learning from and with the… · p. 53
  • Further cybersecurity work should investigate red teaming, formal verification, responsible vulnerability disclosure, capability forecasting, security testing tools, and secure hardware.The report highlights critical systems for red teaming and asks whether hardware could prevent copying, restrict access, or facilitate activity audits.Source: Learning from and with the… · pp. 53–54

Exploring Different Openness Models

  • The report calls for discussion of when misuse concerns justify limiting openness, while preserving openness’s benefits and making any changes transparent and publicly justified.It warns against using misuse as a pretext for restrictions driven by corporate competitiveness and points to biosecurity disclosure debates as a model for healthy discussion.Source: Exploring Different Openness Models · pp. 54–55
  • Potential openness models for investigation include pre-publication risk assessment, central-access licensing, selective sharing among trusted groups, and lessons from other dual-use technologies.The report raises questions about whether risk assessment is premature before AI is widely used in critical systems and how central-access models would remain viable as technology advances.Source: Exploring Different Openness Models · p. 55

Promoting a Culture of Responsibility

  • AI researchers and their organizations should build on existing social responsibility and attend to malicious-use risks throughout training, recruitment, research, and development.Research topics include effective ethics education, ethical standards and enforcement, whistleblowing protections, and narratives that balance AI’s potential with its risks.Source: Promoting a Culture of Responsibility · p. 56

Developing Technological and Policy Solutions

  • Further work should explore privacy protections, coordinated AI-based public-good security, monitoring AI-relevant resources, and legislative or regulatory responses.The agenda asks what roles technical measures and institutions can play in privacy protection, and whether legal definitions of hacking should account for adversarial examples and data poisoning.Source: Developing Technological and Policy… · p. 57

Priority Areas for Further Research

Analysis … Conclusion

AI is likely to become more prominent in security, increasing opportunities for malicious use while also enabling defenses. The medium-term balance remains uncertain and may be temporary, so the report urges precautionary action and continued work on prevention and mitigation.

Analysis

  • The medium-term balance between AI attacks and defenses is uncertain, and even a seemingly stable equilibrium may be short-lived as technology and policy evolve.The report defines the medium term as 5+ years from now, when malicious AI applications are widely used and defended against but human input is still needed; unforeseen developments, including outside AI, could prove more impactful.Source: Analysis · pp. 58–59

Attacker Access to Capabilities

  • If open-access trends continue, attackers’ ability to cause harm with digital and robotic systems is expected to increase, though interventions may reduce risks for less capable actors.Restrictions on access or malicious use may grow, but their effectiveness is uncertain; code limits alone are unlikely to stop motivated, well-resourced actors who can use espionage to obtain it.Source: Attacker Access to Capabilities · p. 59

Existence of AI-Enabled Defenses

  • AI can scale both attacks and defenses, including law-enforcement detection and response, but defensive tools may be costly and will not prevent authoritarian abuses.Examples include spam filtering, malware detection, and possible drone-catching systems; rapid intervention in attacks may cost less than widely deploying physical defenses.Source: Existence of AI-Enabled Defenses · pp. 59–60

Distribution and Generality of Defenses

  • Coordinated organizations may adopt feasible, cost-effective defenses, making the most massive harms, such as attacks on WMD facilities, least likely, depending on the relative pace of attacks and defenses.Responsible vulnerability disclosure, pre-publication risk assessment, and an ethical AI research culture are important in this setting.Source: Distribution and Generality of Defenses · p. 60
  • Most victims need defenses built into widely used technology or offered cheaply, while tailored defenses require backing and misaligned incentives may call for regulation.Tech giants may supply low-cost protection while increasing lock-in and concentration of data and power; nonprofits may offer free or cheap alternatives. General defenses are more useful than media-specific solutions, while externalities such as data breaches and botnets can leave affected individuals unable to improve defenses themselves.Source: Distribution and Generality of Defenses · pp. 60–61

Overall Assessment

  • Malicious AI use is expected to grow with society’s adoption of AI; defenses are likeliest to prevail when practical and aligned with incentives, but hard-to-defend attack vectors may be weaponized and equilibria will shift.Tech and media giants may become safe havens for users, and nations face pressure to protect citizens through infrastructure control, collaboration, or enforceable regulation. The report urges precautionary action, including openness experiments, cross-disciplinary learning, dialogue, and beneficial defense research.Source: Overall Assessment · pp. 61–63

Conclusion

  • AI links digital, physical, and political security; increasingly capable systems may enable sophisticated social engineering, while AI-based cyber defense is no panacea.The report calls for work on openness, technical robustness verification, and policy frameworks that adapt to a more AI-infused world.Source: Conclusion · pp. 64–65
  • Highly capable AI that can be given dangerous goals could eventually cause unprecedented damage in some domains, so preparation and lessons from other fields should begin before these risks are realizable.The report also emphasizes understanding commonalities across malicious-use risks to inform prevention and mitigation.Source: Conclusion · p. 65

Appendix A: … Red Teaming

The appendices describe the workshop behind the report and outline research directions, including dual-use lessons and red teaming, as starting points rather than exhaustive conclusions. They highlight practical testing of AI security and open questions about how to conduct, share, and extend such exercises.

Summary

  • The February 2017 Oxford workshop brought together varied institutional and disciplinary perspectives to analyze AI misuse risks under Chatham House rules.It was co-chaired by Miles Brundage and Shahar Avin and co-organized by FHI, CSER, and CFI.Source: Summary · p. 75

Event Structure

  • On February 19, experts presented on cybersecurity, AI, and robotics, followed by breakout discussions of security scenarios and possible defenses; the next day, attendees prioritized prevention and mitigation measures for a research agenda.The group voted on which measures seemed useful and tractable to focus the report-writing process.Source: Event Structure · pp. 75–76

Report Writing Process

  • The report drew on workshop notes and the authors’ prior and subsequent research, then circulated a draft to workshop attendees and additional domain experts.The authors acknowledged that not all participant perspectives could be captured.Source: Report Writing Process · p. 76

List of Workshop Participants

  • The workshop participant list includes experts affiliated with organizations spanning AI research, universities, policy, security, and civil society.The listed participants include Dario Amodei, Ross Anderson, Stuart Armstrong, and others.Source: List of Workshop Participants · p. 76

Appendix A: Workshop Details

  • The appendix’s workshop details continue the participant list with additional experts from research institutions, universities, and policy organizations.The continuation includes Eva Ignatuschtschenko, Victoria Krakovna, Ben Laurie, and others.Source: Appendix A · p. 77

Research

  • The research appendix offers initial questions and directions on recommendations and priority areas, flags the threat factors each aims to address, and is explicitly neither exhaustive nor conclusive.It is presented as a jumping-off point for researchers.Source: Research · p. 78

Dual Use Analogies and Case Studies

  • The appendix proposes studying dual-use technologies for AI policy lessons, while warning that cryptography export controls may be a path to avoid or approach cautiously.Relevant analogies include chemicals, biological engineering, cryptography, and nuclear technologies, whose governance developed soft norms and hard laws over time.Source: Dual Use Analogies and Case Studies · p. 79

Red Teaming

  • Red teaming can test AI-enabled cyber offense and defense and adversarial machine learning, while broader red teaming of AI technologies may also be beneficial.Cyber red-team exercises could test attacks beyond memory vulnerabilities and clarify required skills and how defenses work in practice; ongoing real-world stress-testing of AI systems is just beginning. Open questions include who should conduct exercises, how to share lessons safely, and whether they can extend to physical and political domains.Source: Red Teaming · pp. 80–81

Formal Verification … Security Tools

The section examines how formal methods might verify AI systems or components, while noting limits from complexity and specification; it also raises open questions about vulnerability disclosure, exploit bounties, and security tools for AI.

Formal Verification

  • Formal verification of AI remains an open question amid system complexity and specification challenges, but candidate properties include goal attainment, resistance to adversarial goal changes, and bounded susceptibility to deceptive inputs.Workshop participants expressed substantial skepticism about prospects for formal AI verification, while further analysis and research continue.Source: Formal Verification · pp. 81–82

Verifying Hardware

  • End-to-end verification may be prohibitively expensive or impossible for complex AI systems, while formal methods may feasibly improve component security, especially in hardware.Formal methods have been widely adopted in the hardware industry for decades.Source: Verifying Hardware · p. 82

Verifying Security

  • Formal verification has been applied to security protocols, and the developer-focused JavaScript prover CryptoVerif lets programmers check code correctness, though real-world adoption remains limited.The passage notes some real-world use (Beurdouche et al., 2017 3).Source: Verifying Security · p. 82

Verifying AI Functionality

  • State-space explosion makes formal methods difficult to scale to arbitrary complex systems, yet some AI aspects such as image classifiers can be verified, including checks for adversarial examples in input regions.End-to-end verification may be prohibitively complex even when verification of particular AI-system aspects is feasible.Source: Verifying AI Functionality · p. 82

Responsible “AI 0-Day” Disclosure

  • Because machine-learning systems face both AI-specific and traditional vulnerabilities, the section asks whether responsible-disclosure norms should apply and how vulnerability response should work.Examples include adversarial misclassification, training-data poisoning, and memory overflow. Open questions include whether systems should be presumed vulnerable, which parties to notify and when, and what counts as AI patching given trade-offs among resources, accuracy, and robustness to noise.Source: Responsible “AI 0-Day” Disclosure · p. 83

AI-Specific Exploit Bounties

  • The section asks whether existing or third-party cash bounties should cover AI-specific exploits, including for open-source frameworks and academic projects.Bounties are presented as a possible complement to responsible disclosure, which relies on social incentives and goodwill.Source: AI-Specific Exploit Bounties · p. 84

Security Tools

  • AI security tools could test and improve AI components during development and deployment through adversarial-data generation, classification-error analysis, remote model-extraction and vulnerability scanning, and robustness suggestions.These tools are proposed as a way to make AI components and systems less amenable to attack.Source: Security Tools · p. 84

Secure Hardware · Pre-Publication Risk Assessment in Technical Areas of Special Concern · Central Access Licensing Models

The sections examine three ways to manage malicious-use risks: secure AI hardware, pre-publication risk assessment, and centrally licensed access to AI capabilities. Each approach may constrain harmful use, but raises trade-offs or security risks of its own.

Secure Hardware

  • Secure AI hardware could restrict copying a trained model unless the original is deleted, helping control the number of systems whose capabilities could cause harm if widely diffused.Hardware-level access restrictions and audits are also proposed features.Source: Secure Hardware · p. 85
  • Tamper-proof hardware could hinder reverse engineering and duplication and support credible safety commitments, but secure processors cost more and, to the authors’ knowledge, had not been developed specifically for AI.The authors also propose a reference model to inform engineering and note that manufacturers or supply-chain actors could undermine hardware security.Source: Secure Hardware · pp. 85–86

Pre-Publication Risk Assessment in Technical Areas of Special Concern

  • Pre-publication risk assessment weighs a capability’s risks if widely available against the costs of restricting publication, since less sharing raises recreation barriers but can slow research and legitimate applications.Possible disclosure levels range from ideas and pseudocode to trained models with code and practical guidance.Source: Pre-Publication Risk Assessment in… · pp. 86–87
  • Openness can aid security research such as white-hat penetration testing, while voice synthesis for a target speaker could enable criminal spearphishing and disinformation.If publication restrictions become common, coordinating institutions could delay publication until safety measures or secure deployment are possible while preserving researchers’ priority and credit.Source: Pre-Publication Risk Assessment in… · pp. 87, 89

Central Access Licensing Models

  • Central access licensing can limit service use and prohibit malicious use, but model extraction may expose the underlying technology and centralized control creates insider and backdoor risks.Concentration may also heighten risks within service organizations; the effectiveness of query limits, provider vetting, and the model's feasibility remain open questions.Source: Central Access Licensing Models · pp. 89–91
  • Central access offers broad access to capabilities while withholding underlying technology from both bad actors and well-intentioned researchers, and can make large-scale attacks less economical.Providers can impose use-speed limits and terms prohibiting malicious use; black-box extraction remains a potential route to the technology.Source: Central Access Licensing Models · pp. 89–90
  • Centralizing AI services can concentrate power and enable malicious activity by insiders or approved actors, and may introduce hidden backdoors users cannot detect.The report focuses on direct malicious-use risks rather than systemic threats, while identifying provider trust, feasibility, and political-security consequences as research questions.Source: Central Access Licensing Models · pp. 90–91

Sharing Regimes that Favor Safety and Security … Technologically Guaranteed Privacy

The section outlines ways to reduce AI-related security risks while retaining benefits of research and development, including gated capability sharing, education and standards, social norms, and privacy technologies. It emphasizes open questions about how these approaches should be designed, adopted, and governed.

Sharing Regimes that Favor Safety and Security

  • Selective sharing could give trusted parties access to hazardous AI capabilities for collaborative safety and security analysis, with broader publication if analysis finds no harm in diffusion.Possible trust conditions include established safety and security routines or agreement to random inspection by group members or an agreed third party.Source: Sharing Regimes that Favor Safety and… · pp. 91–92
  • Open questions include how ISAC and ISAO models might apply to AI, how to assess recipients’ trustworthiness, what information to share, and whether to evaluate capabilities individually.Source: Sharing Regimes that Favor Safety and… · p. 92

Security, Ethics, and Social Impact Education for Future Developers

Ethics Statements and Standards

  • Multi-stakeholder ethical standards, such as the IEEE initiative and Asilomar AI Principles, raise questions about implementation, accountability, organizational adoption, and keeping standards flexible and current.Source: Ethics Statements and Standards · pp. 93–94

Norms, Framings and Social Incentives

  • Norms for beneficial AI should encourage mutually beneficial development and effective safeguards while avoiding rigid or biased norms that hamper diversity and creativity or make people feel excluded from debate.Source: Norms, Framings and Social Incentives · p. 94

Technologically Guaranteed Privacy

  • Differential privacy and secure multiparty computation may help protect private data that attackers use for spear phishing or personalized propaganda, but their integration, trade-offs, and adoption remain open questions.Source: Technologically Guaranteed Privacy · p. 95

Differential privacy … Exploring Legal and Regulatory Interventions

Differential privacy and secure multi-party computation can protect sensitive training or query data, but differential privacy can reduce performance and MPC can add substantial overhead. The report also considers monitoring computing resources and argues that AI security policy should be developed through coordinated work between technical and policy communities.

Differential privacy

  • Differentially private training adds noise to data to limit information leakage, but may perform worse than non-private alternatives.This matters because public model queries can sometimes reveal information about underlying datasets through model inversion or membership inference attacks.Source: Differential privacy · pp. 95–96

Secure Multi-Party Computation

  • MPC lets parties compute jointly while keeping inputs private, enabling machine learning on sensitive data without the service provider accessing it.For example, a company can make predictions from a person’s health data while only the individual receives the output.Source: Secure Multi-Party Computation · p. 96
  • MPC can support surveillance classifications without collecting or accessing the sensitive data used to make them.Examples include recognizing faces in video or flagging suspicious individuals from web activity.Source: Secure Multi-Party Computation · p. 96
  • MPC can increase computation overhead by multiple orders of magnitude, making it best suited to simple tasks or cases where added privacy is especially valuable.Source: Secure Multi-Party Computation · pp. 96–97

Monitoring Resources

  • A public or semi-public database of computing resources could clarify the distribution of offensive and defensive AI capabilities and support stronger hardware-use limits.The report raises open questions about feasibility, which resources to monitor, potential drawbacks such as wasteful competition, and whether other AI inputs are better targets.Source: Monitoring Resources · p. 97

Exploring Legal and Regulatory Interventions

  • The report poses open questions about responsibility for AI security, institutional roles and coordination, and whether liability rules create suitable defensive incentives.Source: Exploring Legal and Regulatory… · p. 98
  • It also raises policy questions about privacy-preserving systems, penalties for AI attacks, international agreements, online system disclosure, and lawmaking for a changing field.Source: Exploring Legal and Regulatory… · pp. 98–99
  • Effective long-term AI governance requires trust and proactive collaboration between technical and policy communities, rather than reactive responses by separate sectors.The report says AI community discussions should inform policy, while policy institutions should work with that community in developing norms, standards, policies, and laws.Source: Exploring Legal and Regulatory… · p. 99

Related papers

Loading 1802.07228v2…