Source-linked AI summary
Identifying and Mitigating the Security Risks of Generative AI
Clark Barrett, Brad Boyd, Elie Burzstein, Nicholas Carlini, Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodorescu, Anupam Datta, Soheil Feizi, Kathleen Fisher, Tatsunori Hashimoto, Dan Hendrycks, Somesh Jha, Daniel Kang, Florian Kerschbaum, Eric Mitchell, John Mitchell, Zulfikar Ramzan, Khawaja Shams, Dawn Song, Ankur Taly, Diyi Yang
TL;DR
GenAI creates a dual-use security challenge because it can amplify cyberattacks. This paper synthesizes workshop findings on attack and defense capabilities, including evidence that paraphrasing undermines AI-content detection.
Problem
Method
The paper synthesizes workshop findings on GenAI’s attack and defense capabilities and describes challenges for the research community.
Source: §1 · p. 3, §7 · p. 23Results
Paraphrasing and recursive paraphrasing attack detectors; one watermark-based detector’s detection rate at 1% FPR drops from 97% to 15% after 5 rounds.
Source: §5.1 · p. 15Takeaways & Limitations
The synthesis describes problems and challenges for the research community to address.
Source: §1 · p. 3No provably robust watermark is established; existing algorithms withstand attacks only when adversaries lack detector access, while adaptive attacks break known human-made watermarks.
Source: §5.1 · p. 16
Abstract
from arXiv · showhide
Every major technical invention resurfaces the dual-use dilemma -- the new technology has the potential to be used for good as well as for harm. Generative AI (GenAI) techniques, such as large language models (LLMs) and diffusion models, have shown remarkable capabilities (e.g., in-context learning, code-completion, and text-to-image generation and editing). However, GenAI can be used just as well by attackers to generate new attacks and increase the velocity and efficacy of existing attacks. This paper reports the findings of a workshop held at Google (co-organized by Stanford University and the University of Wisconsin-Madison) on the dual-use dilemma posed by GenAI. This paper is not meant to be comprehensive, but is rather an attempt to synthesize some of the interesting findings from the workshop. We discuss short-term and long-term goals for the community on this topic. We hope this paper provides both a launching point for a discussion on this important topic as well as interesting problems that the research community can work to address.
1 Introduction
GenAI revives the dual-use dilemma: its capabilities can support both productive and malicious uses, with effects on attacks and defenses still uncertain. This paper synthesizes workshop findings and proposes short- and long-term goals, while explicitly presenting itself as a non-comprehensive starting point.
1 Introduction
- GenAI can serve both productive and nefarious purposes, changing the threat landscape for attacks and defenses while leaving its long-term effects uncertain.The paper argues that the evolving tools and their uses create years of unpredictability.Source: §1 · p. 2
- A one-day Google workshop convened experts to discuss how attackers could leverage GenAI, how security measures should respond, and which technologies merit attention for countermeasures.The workshop took place on June 27, 2023.Source: §1 · p. 3
- The monograph synthesizes workshop findings and sets short- and long-term community goals, but is neither comprehensive nor the final word on the topic.It aims to identify interesting problems and challenges for the research community.Source: §1 · p. 3
2 GenAI Capabilities
The section defines GenAI and LLMs, then surveys capabilities relevant to both attackers and defenders, from realistic media generation to sustained technical and language tasks.
2 GenAI Capabilities
- GenAI models emulate input-data structure to generate synthetic digital content, including images, video, audio, and text.Source: §2 · p. 3
- LLMs are typically transformer-based models trained on large text collections, and have been extended to image, audio, and video modalities.The paper cites Vaswani et al. (2017) for the transformer architecture.Source: §2 · pp. 3–4
- GenAI can generate targeted, context-specific text and realistic images or video, and can summarize or paraphrase source material while maintaining its style and intent.Source: §2 · p. 4
- Models can draw on technical knowledge to produce or analyze code and address complex questions across specialized domains, though current models are not flawless.Source: §2 · p. 4
- GenAI can persist through time-consuming tasks without the fatigue or psychological strain that people may experience.Source: §2 · p. 4
3 Attacks
The paper describes GenAI vulnerabilities and ways attackers can exploit its capabilities, from convincing social engineering to larger-scale cyberattacks. It also highlights risks from model limitations, data feedback, and unpredictable behavior, while noting that its attack list is workshop-derived rather than exhaustive.
3 Attacks
- The attack discussion reflects workshop speakers’ and panelists’ contributions rather than an exhaustive or prioritized catalog.Source: §3 · p. 4
- GenAI can support personalized spear-phishing and high-fidelity deepfakes, enabling targeted deception and widespread misinformation or disinformation.Source: §3 · pp. 5–6
- LLMs can generate sophisticated code and support agent workflows for planning and executing attacks, but their larger attack surface can expose the stack to prompt injection and cascading failures.Safety tools were present in the ChemCrow workflow, but the paper says the large attack surface remains.Source: §3 · p. 6
- GenAI reduces the effort, time, and cost of some attacks while broad access expands the pool of potential adversaries.The paper gives the StackOverflow DoS incident and maliciously oriented language models as examples.Source: §3 · pp. 6–7
- GenAI’s lack of social awareness and human sensibility can have severe consequences for users who trust its fluent, contextually limited responses.The paper cites reports of inappropriate advice to vulnerable users, including a 13-year-old.Source: §3 · p. 7
- Hallucinations can produce coherent but false output, posing risks when users rely on it; one lawyer’s ChatGPT filing included six fabricated cases.Generated images and text can also mislead; the figure contrasts an apparently fake explosion image with a fabricated monograph citation.Source: Figure 3.1 · p. 5, §3 · pp. 7–8
- Internet data feedback may degrade training data, amplify existing bias, toxicity, and errors, and make poisoning attacks more feasible across multiple training phases.The paper identifies publicly reviewable training data, architecture, and methodology as a possible first step toward addressing data-quality concerns.Source: §3 · p. 8
- Emergent abilities and distribution shifts make GenAI behavior difficult to evaluate and can leave models brittle to adversarial prompts and prompt injection.The paper says the full range of model capabilities and threats remains unknown.Source: §3 · pp. 8–9
4 Defenses
The section surveys defenses spanning GenAI systems, their surrounding ecosystem, and human-AI use. It presents promising applications alongside limitations such as paraphrase evasion of content detectors and watermarks.
4 Defenses
- GenAI defenses range from improving model functionality to ecosystem tools and human-AI collaboration, but the workshop-derived list is neither exhaustive nor prioritized.Source: §4 · p. 9
- LLM-content detectors use distributional differences to classify generated text, but paraphrasing can make generated text evade detection.The paper cautions defenders to stay aware of the state of the art in attacks.Source: §4 · pp. 9–10
- Watermarking embeds a statistical signal during generation to support later detection and attribution, but simple transformations such as paraphrasing can remove it.The paper considers watermarking useful where there is a disincentive to remove the signal and calls for investigation of suitable use cases.Source: §4 · p. 10
- LLMs might help de-obfuscate code through fine-tuning on obfuscated and de-obfuscated examples, addressing a challenge in security code analysis.The paper notes limited indications that GPT-4 can perform de-obfuscation.Source: §4 · p. 10
- LLMs could automate parts of penetration testing so human testers can focus on challenging vulnerabilities and systems can be assessed across more of the vulnerability space.Source: §4 · pp. 10–11
- Combining text, code, images, and speech can give fake-tweet detectors broader contextual clues and inconsistencies than single-modality analysis.The paper presents this as a research avenue for more robust analysis.Source: §4 · p. 11
- GenAI-based personalized training could tailor cybersecurity learning and practice to individuals, with potential to help address the shortage of cybersecurity professionals.Source: §4 · p. 11
- Human-LLM annotation pipelines can allocate work between people and models to improve analysis cost efficiency and reliability.The paper frames this as combining model capabilities with human judgment and expertise.Source: §4 · pp. 11–12
5 Short-Term Goals
The short-term agenda calls for better evidence about emerging defenses and LLM code capabilities, more secure code generation, and community resources for attacks and defenses. The defense discussion highlights watermark-evaluation challenges and vulnerabilities in detection methods.
5 Short-Term Goals
- Emerging detection and watermarking techniques need a clearer landscape of attacks and defenses and investigation of use cases where they remain effective.The paper notes that paraphrasing or another GenAI system can remove watermarks, while secret-key watermark insertion appears harder.Source: §5 · p. 12
- Because evidence about LLM code-analysis capabilities is anecdotal, the paper calls for comprehensive analysis to inform possible threats and defenses.Source: §5 · p. 12
- Secure code generation should align LLM completions with security and privacy requirements, using techniques such as RLCF and controlled generation alongside shared secure-coding data.LLMs trained on programming-community content can generate insecure or buggy code.Source: §5 · p. 13
- A community repository of state-of-the-art attacks and a service providing current defenses could help address uneven evaluation and awareness.The paper points to DARPA SemaFor and IBM’s ART repository as examples.Source: §5 · p. 13
5.1 Emerging Defenses for GenAI
- Watermark-removal robustness can be assessed through the trade-off between attacker success and content similarity, with area under that curve proposed as a measure.The similarity metric should reflect content utility and may depend on the use case.Source: §5.1 · pp. 14–15
- At 1% FPR, one watermark detector’s detection rate falls from 97% to 15% after five recursive-paraphrasing rounds; detectors are also vulnerable to spoofing.Spoofing can make human-generated text appear to originate from an LLM.Source: §5.1 · p. 15
- Image watermark detectors based on deep neural networks can be vulnerable to adversarial examples, while robust detection as AI and human text distributions converge remains uncertain.The feasibility of semantic watermarks independent of output style is also an open question.Source: §5.1 · p. 16
- Watermark evaluation is empirical and error-prone; designers must identify and test strong attacks, and existing schemes withstand attacks only when detectors are inaccessible.The paper describes adaptive attacks against known watermarking algorithms.Source: §5.1 · pp. 15–16
6 Long-Term Goals
The long-term goals call for layered technical defenses alongside social, political, cultural, and economic responses. They also identify open challenges in value alignment, grounding, accountability, privacy, and collaboration across stakeholders.
Need for socio-technical solutions
- GenAI’s mismatch with nuanced, context-driven human activity motivates socio-technical solutions that combine technical advances with social, political, cultural, and economic mechanisms.The paper also calls for evaluation metrics that reflect downstream social requirements and explicitly articulate drawbacks and trade-offs.Source: §6 · p. 17
Trust, accountability, and privacy
- Online reputation systems could track identities and information provenance, but they do not eliminate privacy needs; liability across users and providers still needs study.The paper also flags personal-data extraction from models and consent concerns around using public data for commercial training.Source: §6 · pp. 17–18
Multiple lines of defenses
- A comprehensive GenAI safety strategy needs training-time alignment, post-hoc detection and filtering, and diverse red teams because perfect risk mitigation is extremely challenging.RLHF uses human preferences as a reward signal; red teaming stress-tests models against real-world attack scenarios.Source: §6 · pp. 18–19
Pluralistic value alignment
- Pluralistic value alignment requires interdisciplinary dialogue because cultures, individuals, and legal jurisdictions differ on which values AI should follow.The paper rejects a binary view of alignment and calls for ethicists, researchers, legal experts, policymakers, and the public to participate.Source: §6 · pp. 19–20
Reduce barrier-to-entry for GenAI research
- Reducing research barriers could broaden access beyond well-funded organizations, but open-source models also increase adversary access and safety-misuse risks.The paper notes that automated safety attacks demonstrated on open-source chatbots can transfer to closed-source chatbots.Source: §6 · p. 20
New partnerships among stakeholders
- Responsible GenAI policy calls for partnerships among government, academia, and industry, bringing technical expertise and implementation perspectives into policy-making.The paper cites conflict over proposed EU content-filtering legislation as an example of policy and technology disconnect.Source: §6 · p. 21
Grounding
- Grounding requires generated text to be attributable to an authoritative source, but entailment checks can miss biased coverage and prompting strategies lack a clear best choice.Tuning for groundedness can also reduce creativity, while balancing fluency, grounding, and plausibility remains challenging.Source: §6 · pp. 21, 23
7 Conclusion
The paper synthesizes workshop perspectives on GenAI attacks and defenses, but is not a comprehensive or final account. It calls the inquiry timely because attackers have already begun using these technologies.
7 Conclusion
- The paper investigates GenAI’s attack and defense capabilities, while explicitly disclaiming that it is the final word on the topic.Source: §7 · p. 23
- The conclusions reflect workshop speakers’ and attendees’ viewpoints, and the authors say multiple meetings are needed to explore the full landscape.Source: §7 · p. 23
- Attackers have already started using GenAI, making investigation of its risks and mitigation timely for defenders.Source: §7 · p. 23