PaperlayerView exact revision on arXiv

Source-linked AI summary

Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications

Stav Cohen, Ron Bitton, Ben Nassi

arXiv:2403.02817v2cs.CR

TL;DR

Prior work examined RAG attacks against individual applications, leaving open whether attackers could scale them across GenAI ecosystems. This paper presents Morris-II, a self-replicating prompt worm, and Virtual Donkey, a guardrail; the guardrail achieves TPR 1.0 and FPR 0.015, while adaptive-attack resilience remains untested.

  • Problem

    Prior research examined RAG membership-inference, entity-extraction, and poisoning attacks against individual applications, while ecosystem-scale threats remained less understood.

    Source: §1 · p. 1

  • Method

    The paper models Morris-II as a worm using a jailbreaking, replication, and malicious-activity prompt, and detects it by comparing model input and output similarity.

    Source: §3.2 · p. 3, §5.2 · p. 9

  • Results

    Virtual Donkey achieved a TPR of 1.0 and FPR of 0.015 and was assessed against unseen jailbreaking commands, a different email dataset, and varied worm use cases.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Morris-II scales RAG poisoning from individual applications to an ecosystem, and Virtual Donkey is intended to prevent its propagation.

    Source: §1 · p. 2

    Virtual Donkey’s resilience to adaptive attacks was not tested, so it may remain vulnerable to them.

    Source: §6 · p. 13

Abstract

from arXiv · show

In this paper, we show that when the communication between GenAI-powered applications relies on RAG-based inference, an attacker can initiate a computer worm-like chain reaction that we call Morris-II. This is done by crafting an adversarial self-replicating prompt that triggers a cascade of indirect prompt injections within the ecosystem and forces each affected application to perform malicious actions and compromise the RAG of additional applications. We evaluate the performance of the worm in creating a chain of confidential user data extraction within a GenAI ecosystem of GenAI-powered email assistants and analyze how the performance of the worm is affected by the size of the context, the adversarial self-replicating prompt used, the type and size of the embedding algorithm employed, and the number of hops in the propagation. Finally, we introduce the Virtual Donkey, a guardrail intended to detect and prevent the propagation of Morris-II with minimal latency, high accuracy, and a low false-positive rate. We evaluate the guardrail's performance and show that it yields a perfect true-positive rate of 1.0 with a false-positive rate of 0.015, and is robust against out-of-distribution worms, consisting of unseen jailbreaking commands, a different email dataset, and various worm usecases.

1. Introduction

The paper asks whether attacks on individual RAG-based GenAI applications can scale across connected ecosystems, and presents Morris-II as a worm-like chain reaction that spreads malicious prompts between applications. It evaluates confidential-data extraction in email assistants and introduces Virtual Donkey to detect and prevent propagation.

1. Introduction

  • The paper identifies a gap in understanding ecosystem-level threats: prior RAG attacks studied membership inference, entity extraction, and poisoning against individual applications.The paper asks whether attackers can scale attacks from individual applications to entire ecosystems.Source: §1 · p. 1
  • Morris-II uses an adversarial self-replicating prompt to trigger malicious activity in affected applications and compromise additional applications, scaling RAG poisoning from one application to an ecosystem.The prompt uses an application’s RAG database for persistence across inferences.Source: §1 · pp. 1–2
  • Once in every five emails generated by the GenAI engine, Morris-II compromised a new assistant while extracting sensitive user data from context into generated email content.The evaluation used RAG-based email assistants and the Enron dataset.Source: §1 · p. 2
  • Virtual Donkey detects and prevents worm propagation by identifying similarities between model input and output, achieving a TPR of 1.0 and FPR of 0.015.Its out-of-distribution evaluation includes unseen jailbreaking commands, a different email dataset, and varied worm use cases.Source: §1 · p. 2

2. Background & Related Work

The background explains RAG’s retrieval pipeline and reviews attacks on its integrity and confidentiality, alongside broader GenAI attacks and the history of computer worms.

2. Background & Related Work

  • RAG retrieves documents relevant to a query using embeddings, a similarity function, and a database, then supplies those documents and the query to the GenAI engine.The approach incorporates external knowledge at inference time to improve accuracy and relevance.Source: §2 · p. 2
  • Prior RAG security work covers poisoning attacks on integrity and membership-inference and entity-extraction attacks on confidentiality.Poisoning outcomes include backdoors, misinformation, and blocking relevant information; confidentiality attacks validate stored entities or extract them.Source: §2 · p. 2
  • Computer worms have targeted diverse hosts and purposes, with examples including the Morris Worm, ILOVEYOU, Stuxnet, Mirai, and WannaCry.The cited examples span personal computers, industrial control systems, IoT devices, and ransomware.Source: §2 · p. 3
  • GenAI application attacks include jailbreaking through varied inputs, vectors, and perturbations, as well as prompt leakage, keylogging, and denial of service.Examples include image and audio inputs, direct and indirect vectors, and “ignore previous prompt” commands.Source: §2 · p. 3

3. Morris-II: Threat Model, Structure & Steps

Morris-II targets connected RAG-based applications with active database updates and uses self-replicating prompts to spread malicious instructions and activity through generated email content.

3.1. Threat Model

  • A Morris-II target receives user inputs, actively updates its database, connects with other clients in an ecosystem, and relies on RAG-based inference for inter-application communication.The threat model assumes the attacker can send a message but lacks prior knowledge of the model, RAG implementation, embedding algorithm, and victim data.Source: §3.1 · p. 3

3.2. Adversarial Self-Replicating Prompts

  • An adversarial self-replicating prompt combines a jailbreak (j), replication instruction (r), and malicious instruction (m); many jailbreaks and malicious activities make the prompt polymorphic.The attacker’s prompt must be stored, retrieved, replicated, and used to perform the predefined malicious activity.Source: §3.2 · pp. 3–4

3.3. Attack Steps

  • After initial compromise through an email stored in the recipient’s RAG database, the prompt can spread through either a newly generated email draft or a generated reply.In both routes, retrieved correspondence containing the prompt influences generated content that the user sends to another client, contaminating its RAG database.Source: §3.3 · pp. 4–5
  • Morris-II does not return extracted data to the attacker; it appends sensitive information to emails sent to other users in the ecosystem.The email destination is determined by the application layer, while the prompt influences generated content.Source: §3.3 · p. 5

4. Evaluation

The evaluation tests Morris-II’s confidential-data extraction across RAG-based email assistants and examines how retrieval, propagation, and payload correctness vary with experimental conditions. It uses Enron email data and evaluates the worm across multiple propagation modes, embedding algorithms, context sizes, and inference hops.

4. Evaluation

  • The evaluation measures confidential-data extraction across GenAI email assistants, using 2,000 Enron emails from 20 employees and Gemini Flash 1.5.Each employee’s database contains 100 received and sent emails; the client uses LangChain and a vector-store RAG with cosine similarity.Source: §4 · p. 5, §4.1 · p. 5

4.2. Metrics

  • The evaluation defines combined success as the product of retrieval, replication, and payload success rates.Replication requires the exact jailbreaking prompt and instructions; payload success counts when sensitive user data is returned.Source: §4.2 · p. 6

4.3. Evaluating the Influence of the Email Prefix

  • At k = 70%, the Wikipedia email prefix yields 77.1% retrieval, versus 19.5% for the Greetings prefix.The authors note that including organization or entity names can increase similarity to emails and make those targets easier to retrieve.Source: §4.3 · p. 6, Figure 2 · p. 6

4.4. Evaluating the Influence of the Embeddings Algorithm on Retrieval Rate

  • At k = 60%, Google’s embedding yields 57.1% retrieval versus 34.7% for GTE-base-768; GTE-base-768 and GTE-large-1024 yield 34.7% and 34.0%, versus 19.9% for GTE-small-384.The reported retrieval rates vary with both embedding algorithm and embedding size.Source: Figure 2 · p. 6, §4.4 · p. 7

4.5. Evaluating the Propagation Rate

  • With a context of 20 emails, response generation and email-body enrichment each achieve around 20% combined success, while subject-based generation has lower combined rates.The authors interpret this as an expected propagation to new clients every five emails a user receives or sends.Source: §4.5 · p. 7, Figure 3 · p. 7

4.6. Evaluating the Resilience of the Worm

  • Replication and payload success stay above 90% through hop 11 across the tested context sizes, while Claude 3.5 Sonnet stays around 100% and Gemini 1.5 Pro falls to 64% at hop 20.Combined success declines from hops 12 to 20, with reported results of 40%–80% depending on context size; the authors attribute the decline to GenAI-engine nondeterminism.Source: §4.6 · p. 8, Figure 5 · p. 8

4.7. Evaluating the Correctness of the Payload

  • Payload F1 falls from 0.78 with 10 emails in context to 0.58 with 100, while the error rate rises from 0.26 to 0.37.A reported error involved hallucinating complete email addresses from tagged employees’ names in earlier threads.Source: Figure 6 · p. 8, §4.7 · p. 9

5. Guardrails

The section compares potential Morris-II mitigations and presents the Virtual Donkey, which detects self-replicating prompts by comparing model inputs and outputs. Its evaluation reports strong detection across in-distribution and several out-of-distribution settings.

5.1. Possible Mitigations.

  • The reviewed alternatives trade off inference cost, RAG context, false positives, or robustness to unseen jailbreaks.The authors identify a need for a low-latency, cost-effective approach that avoids extra model inference and remains robust to OOD data.Source: §5.1 · p. 9

5.2. Data Analysis

  • Benign and adversarial prompt outputs showed distribution overlap of 8% for BLEU, 3% for METEOR, and 14% for ROUGE-L.For each metric, the analysis used the most similar input email for each inference.Source: Figure 7 · p. 10, §5.2 · p. 10

5.3. The Virtual Donkey

  • The Virtual Donkey scores output-to-input similarity with BLEU, METEOR, and ROUGE-L, then uses a lightweight classifier to flag adversarial prompts.The guardrail runs after inference and alerts the user when it detects a prompt, allowing review before the generated content is sent.Source: §5.3 · p. 10
  • TPR 1.0 with FPR 0.015: Logistic Regression using three features detects all tested self-replicating prompts.The authors note that this threshold can produce an alert on approximately one in every 66 emails even when none is malicious.Source: Figure 9 · p. 11, §5.3 · p. 11

5.4. Performance on OOD

  • The OOD evaluations vary malicious payloads, jailbreak prompts, and email data while testing Logistic Regression.The experiments included phishing, spam, and cowboy-persona payloads; 20 selected jailbreak prompts; and a separate Hillary Clinton email dataset.Source: §5.4 · pp. 11–12, §5.4.1 · p. 12, §5.4.2 · pp. 12–13

5.4.1. Performance on OOD Usecases.

  • AUC ranged from 0.99 to 1.0 across phishing, Walmart-spam, and cowboy-persona payloads.The test datasets used employees excluded from the original training dataset.Source: Figure 10 · p. 12, §5.4.1 · p. 12

5.4.2. Performance on OOD Jailbreaking Prompts.

  • On a new dataset generated by uniformly sampling from 20 jailbreaking prompts rather than the original role-playing prompt, the Logistic Regression model achieved AUC values ranging from 0.96 to 1.0.The dataset comprised 3,000 emails, equally split between benign and adversarial self-replicating-prompt classes.Source: §5.4.2 · p. 13

5.4.3. Performance on OOD Datasets.

  • AUC ranged from 0.79–0.97 on emails from the Hillary Clinton dataset.This evaluation used one database containing 1,500 emails and a 3,000-email test dataset.Source: Figure 14 · p. 13, §5.4.3 · p. 13

6. Limitations

The limitations discussion notes that prompts or extracted payloads are visible, and that the Virtual Donkey has not been tested against adaptive attacks.

Overtness

  • Attackers may hide the self-replicating prompt, but the resulting payload remains visible to a human reviewer.The paper gives ASCII smuggling as an example of hiding a prompt.Source: §6 · p. 13

Adaptive Attacks

  • The guardrail’s resilience to adaptive attacks is untested, so the authors frame it as raising attacker effort rather than preventing attacks.They note that it may be vulnerable to adaptive attacks.Source: §6 · p. 13

7. Conclusions

The conclusion presents Morris-II as a worm that triggers malicious activity while compromising additional applications in a GenAI ecosystem. It identifies testing with non-textual inputs against multimodal engines as future work.

7. Conclusions

  • Morris-II forces affected applications to perform malicious activity while compromising additional applications in the ecosystem.Source: §7 · p. 13
  • The authors propose testing Morris-II with non-textual inputs against multimodal GenAI engines in future work.Source: §7 · p. 14

Appendix A

The appendix excerpts include example email text and a self-replicating prompt with jailbreaking, malicious-activity, and replication instructions. It also includes figures describing evaluation prefixes and OOD performance measures.

Appendix A

  • The example prompt separates jailbreaking, malicious-activity, and replication instructions, including a request to list private information from other emails.Source: Appendix A · p. 15
  • An example inference output includes a phone number, a survey link, and a deadline in its payload leak.Source: Appendix A · pp. 15–16
  • The appendix captions identify figures showing worm prefixes and TPR/FPR results for OOD use cases, jailbreaks, and a dataset.Source: Figure 15 · p. 16, Figure 16 · p. 16, Figure 17 · p. 17
Loading 2403.02817v2…