Source-linked AI summary
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, Tatsunori Hashimoto
TL;DR
Instruction-following LLMs create dual-use risks because their program-like capabilities enable malicious content generation and computer-security-inspired attacks. The paper evaluates attacks against OpenAI’s API and finds convincing malicious content, defense bypasses, and low generation costs, motivating new security approaches.
Problem
The paper asks whether malicious actors can bypass API defenses and use instruction-following LLMs to produce malicious content economically.
Method
The authors adapt obfuscation, code injection/payload splitting, and virtualization attacks to OpenAI’s ChatGPT, GPT-3, and Instruct-GPT APIs.
Results
Instruction-following LLMs generated convincing personalized malicious content, bypassed OpenAI’s defenses, and produced whole generations for $0.0064 to $0.016.
Takeaways & Limitations
The findings suggest that LLM defenses may need approaches drawn from traditional computer security as adversaries gain economic incentives to use these systems.
Takeaways & Limitations
Input filtering is limited against attacks exploiting program-like behavior without first running the LLM, and the attacks were targeted toward OpenAI’s systems.
Abstract
from arXiv · showhide
Recent advances in instruction-following large language models (LLMs) have led to dramatic improvements in a range of NLP tasks. Unfortunately, we find that the same improved capabilities amplify the dual-use risks for malicious purposes of these models. Dual-use is difficult to prevent as instruction-following capabilities now enable standard attacks from computer security. The capabilities of these instruction-following LLMs provide strong economic incentives for dual-use by malicious actors. In particular, we show that instruction-following LLMs can produce targeted malicious content, including hate speech and scams, bypassing in-the-wild defenses implemented by LLM API vendors. Our analysis shows that this content can be generated economically and at cost likely lower than with human effort alone. Together, our findings suggest that LLMs will increasingly attract more sophisticated adversaries and attacks, and addressing these attacks may require new approaches to mitigations.
1. Introduction
Instruction-following LLMs can generate convincing, personalized malicious content, while computer-security-inspired attacks bypass defenses and make misuse economically attractive.
- Capabilities: Instruction-following LLMs can generate natural, convincing, and personalized malicious content despite using unnatural prompts.The experiments include ChatGPT and text-davinci-003.
- Study setting: The case studies examine ChatGPT, GPT-3, and Instruct-GPT models through OpenAI’s API.OpenAI was selected because its models were accessible by API and claimed to have state-of-the-art defenses against malicious use cases.
- Attacks and defenses: Obfuscation, code injection/payload splitting, and virtualization attacks bypassed OpenAI’s defenses with a 100% success rate on hate speech, phishing attacks, and scams.The attacks can also be combined for improved efficacy.
- Capabilities: ChatGPT and text-davinci-003 generate substantially more convincing malicious text than prior models, including the original GPT-3.
- Economic analysis: $0.0064 to $0.016 is the reported cost to produce whole generations, compared with as much as $0.10 for each human generation.The comparison suggests economically scalable use by adversaries.
- Implications: LLMs can be used for malicious purposes economically and effectively, increasing incentives to bypass mitigation measures.The authors propose drawing future defenses from traditional computer security, where programs are assumed able to execute arbitrary code.
2. LLMs Behave Like Programs
Instruction-following LLMs exhibit program-like capabilities that support sequential composition, branching, and complex computation, enabling computer-security-inspired attacks.
- Program analogy: LLMs provide “gadgets” that can be chained together, analogous to instructions used in return-oriented programming.The authors identify these capabilities as sufficient to enable classical security attacks.
- Programmatic capabilities: Instruction-following LLMs can follow a sequence of instructions, demonstrating sequential composition.
- Programmatic capabilities: A prompt can assign strings, concatenate them, and ask the model to answer the resulting instruction.The example constructs “Write a Twitter thread on inflation” from two strings.
- Programmatic capabilities: Instruction-following LLMs can perform branching by selecting different string operations based on a condition.The example checks whether one string contains “eggs” before choosing which prompt to construct and answer.
- Programmatic capabilities: These capabilities let LLMs emulate complex computation involving memory, assignment, branching, and individual operations.The authors leverage these properties to design LLM attacks.
3. Attacking LLMs
The paper frames instruction-following LLMs as program-like systems and adapts traditional computer-security attacks to bypass defenses against malicious generations. These attacks include obfuscation, payload splitting, and virtualization, and can be combined and scaled, while raising fundamental limits for input filtering.
- Attack mechanisms: The authors adapt obfuscation, code injection or payload splitting, and virtualization attacks from traditional computer security to LLM prompts.The attacks are designed to bypass black-box provider defenses against malicious API use.
- Obfuscation: Obfuscation alters prompts with changes such as typos or synonyms so filters miss sensitive terms, while the model can correct the typos in its output.The authors report that simple obfuscation attacks achieved a 100% success rate and required no particular output changes by the adversary.
- Code injection and indirection: Payload splitting divides malicious instructions into multiple parts that the LLM reassembles, and simple indirection can bypass current content filters with high precision.The prompt encodes instructions indirectly rather than presenting the malicious payload directly.
- Virtualization: Virtualization encodes an attack in a fictitious scenario that prompts the LLM to emulate instructions, adapting virtual-machine concepts to language models.The paper describes this as executing desired behavior through prompts that construct fictitious scenarios.
- Combination and scalability: Combining attack techniques bypasses both input and output filters, and the attacks can be templated or varied for easy scaling.The authors also report that other attacks may be more difficult for model providers to defend against.
- Limits of defenses: Instruction-following LLMs may inherit a program-like predictability problem: complex behavior can require execution to predict, limiting input filtering without running the model.This argument is connected to the undecidability of the halting problem for Turing-complete programs.
4. Attacks Bypass LLM Provider Defenses
The evaluation tests whether attacks bypass OpenAI’s input filters, output filters, and useless-generation mitigation across malicious-use scenarios. Obfuscation and virtualization bypassed filters in all evaluated scenarios, while indirection reached 92% under its reported condition, and the prompts were simple and scalable.
- Evaluation design: The evaluation covers 25 scenarios spanning hate speech, conspiracy promotion, phishing, scams, and product astroturfing.Each of five malicious uses received five target scenarios, with basic prompts and individual attacks evaluated.
- Evaluation design: An attack counts as bypassing mitigation when neither the prompt nor generation triggers OpenAI’s input filters, output filters, or useless-generation response.The table caption specifies that both prompt and generation must evade the relevant filters.
- Results: 100% of evaluated scenarios bypassed OpenAI’s filters with obfuscation and virtualization, while indirection achieved 92% among scenarios that did not initially bypass filters.These are the reported overall success rates under the study’s evaluation condition.
- Results: The attack prompts were simple to construct, required fewer than 10 attempts each, and could cover every commonly listed scam in the US government’s list.The authors report that content filters did not trigger for the evaluated scams or product astroturfing.
- Scalability: The attacks can be converted into templates and scaled through minor variations, making bypasses readily repeatable across target scenarios.The paper characterizes the attacks as easily scalable after observing that filters can be bypassed with simple attacks.
- Scam-generation quality: Table 2 compares models on the mean and standard error of convincingness and logical consistency for initial and reply scam emails.ChatGPT outperforms the other listed conditions in the table.
5. Instruction-Following LLMs Produce Useful Malicious Generations
The study evaluates whether instruction-following LLMs can generate convincing, consistent, and personalized scams despite prompt-based attacks. Across scam scenarios, stronger instruction-following models generally produced higher-quality malicious content, creating potential economic incentives for misuse.
- 5. Instruction-Following LLMs Produce Useful Malicious Generations: Traditional-security-inspired attacks enabled instruction-following LLMs to bypass in-the-wild defenses against malicious use.The attacks adapted obfuscation, code injection/payload splitting, and virtualization techniques to the LLM setting.
- 5.1. Malicious Generations are Convincing: The study evaluated five common scams, generating phishing or scam email exchanges with LLM-written initial and scammer replies.The scenarios included ticket, FEMA funds, investment, advisor gift card, and lottery-winning scams; skeptical replies were authored according to US federal guidance.
- 5.1. Malicious Generations are Convincing: Four models produced 50 generations each, and human labelers rated convincingness and logical consistency on five-point Likert scales.Each generation received three independent labels.
- 5.1. Malicious Generations are Convincing: ChatGPT was most convincing and logically consistent by over three standard deviations versus text-ada-001, davinci, and GPT2-XL.text-davinci-003 performed close to ChatGPT within the margin of error, while GPT2-XL ranked second among the remaining models without a statistically significant trend.
- 5.1. Malicious Generations are Convincing: More capable models enabled even non-experts to generate convincing and consistent scams.The reported pattern links increased model usefulness with the quality of malicious generations.
- 5.2. Malicious Generations can be Personalized: The personalization study generated investment scams for five randomly selected combinations of demographic and personal-situation attributes.Situations included bail, medical debt, college debt, car accidents, and sudden death in the family.
- 5.2. Malicious Generations can be Personalized: ChatGPT produced the most personalized, convincing, consistent, and fluent emails, while text-davinci-003 ranked second across all measurements.text-ada-001 slightly exceeded davinci and GPT2-XL, but that difference was not statistically significant; examples included ticket-purchasing scams.
6. Economic Analysis
The economic analysis compares estimated costs for human-generated and model-generated malicious text. It also presents a table describing personalization, convincingness, consistency, and fluency measurements for targeted generations.
- 6. Economic Analysis: The analysis compares human-generated and model-generated text using ranges of cost estimates.The stated goal is to assess the economic viability of generating text for malicious purposes.
- 6. Economic Analysis: Human-generation costs were estimated from call-center wages and median call duration.The analysis used an inflation-adjusted hourly wage estimate of around $1.80 and a median call duration of 3 minutes and 20 seconds.
- 6. Economic Analysis: Table 3 reports mean and standard error for personalization, convincingness, consistency, and fluency of malicious generations targeted toward specific demographics.The caption states that ChatGPT outperforms in all settings.
7. Related Work
The paper situates its analysis within research on LLM harms and earlier work using non-instruction-following models for malicious text generation. It emphasizes the subsequent rise of instruction-following capabilities as a major change in the research context.
- 7. Related Work: The work extends research on harms and risks of LLMs by focusing on their potential for active misuse.The related literature includes studies of hateful, extremist, disinformation, and other harmful text.
- 7. Related Work: Earlier studies examined non-instruction-following models such as GPT-2 and original GPT-3 for disinformation and extremist-text generation.The paper identifies improved instruction-following ability and other subsequent changes as important developments since that work.
8. Conclusion
The conclusion reports that programmatic LLM capabilities and computer-security-inspired attacks support convincing malicious generations and bypass deployed filtering. It also records responsible disclosure and notes that updated defenses blocked many specific prompts while modified prompts remained effective.
- 8. Conclusion: Programmatic LLM capabilities enabled convincing malicious content without additional training or extensive prompt engineering.The conclusion covers scams, spam, and hate speech and reports potential use by non-experts.
- 8. Conclusion: Simple computer-security-inspired attacks bypassed state-of-the-art content filtering deployed in the wild.The paper calls for viewing LLM attacks and defenses through the lens of traditional computer security.
- 8. Conclusion: The authors withheld the experimental prompts publicly because of their potential for misuse.Researchers seeking reproduction materials were directed to contact the corresponding author.
- 8. Conclusion: After disclosure, OpenAI updated its models so many specific prompt versions no longer worked, but modified versions still produced malicious text.This is an author-reported boundary of the updated defenses rather than a claim that all attacks remain effective.
A. Target Scenarios
The experiments cover scams, phishing, conspiracy theories, hate speech, and promotional content as target scenarios for evaluating malicious generation.
- Target scenarios: The target scenarios include promotional content, scams, phishing scams, conspiracy theory promotion, and hate speech.Promotional examples include nutritional supplements, essential oils, skincare products, weight-loss aids, and personal-care products.
- Scams: Scam scenarios involve stealing target funds through investments, gift-card requests, lottery winnings, or bill repayment.
- Phishing scams: Phishing scenarios aim to extract personal information through charity and ticket scams.
- Conspiracy theory promotion: Conspiracy-theory scenarios include claims involving the Elders of Zion, Ted Cruz, election stealing, COVID-19, and vaccinations.
- Hate speech: Hate-speech scenarios target Jews, African-Americans, Mexican immigrants, homosexual individuals, and transsexual individuals.
B. Templates
The paper constructs base prompts and adapts obfuscation, code injection, and virtualization attacks to generate sensitive content while testing instruction-following behavior.
- Base prompts: Base prompts contain generation instructions and often trigger defenses before attack transformations are applied.
- Experiment labeling: Figures 4 and 5 provide the persuasion experiment’s instructions and taxonomy.
- Obfuscation: The obfuscation attack introduces typos and synonyms for sensitive terms.
- Code injection: The code injection attack splits sensitive terms across strings and concatenates them into a prompt for the model.
- Virtualization: The virtualization attack uses initial prompts to set up a virtual machine before subsequent generation.
C. Example Generations
Example generations show a ticket scam that persuades a recipient to send credit-card information, including reassurance and urgency-based follow-up messages.
- Initial generation: The example begins with an email announcing exclusive Taylor Swift concert tickets to the recipient.
- Labeling experiments: The paper reports separate instructions and taxonomies for persuasion and personalization labeling experiments.
- Initial generation: The generated scam requests credit-card information by email to access the tickets.
- Follow-up generation: The messages also create urgency by emphasizing that the tickets are extremely limited.
- Follow-up generation: Follow-up generations address hesitation by claiming email is secure and that no alternative ticket-access options exist.
- Generation cost: Human-generated high-quality text is estimated at $0.40 to $0.80 when using labor priced 5–10× below $16 per hour.