PaperlayerView exact revision on arXiv

Source-linked AI summary

Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning

Gelei Deng, Yi Liu, Kailong Wang, Yuekang Li, Tianwei Zhang, Yang Liu

arXiv:2402.08416v1cs.CR

TL;DR

The paper addresses the limited study of indirect jailbreak attacks that exploit RAG-integrated LLMs, particularly GPTs. It introduces RAG Poisoning through PANDORA, which combines maliciously crafted knowledge content with prompt manipulation. In preliminary tests, PANDORA succeeds across four prohibited scenarios, reaching 64.3% on GPT-3.5 and 34.8% on GPT-4 and exceeding direct attacks.

  • Problem

    Indirect jailbreak attacks exploiting RAG-integrated LLMs have received limited attention despite the security risks of jailbreak vulnerabilities.

    Source: §I · p. 1

  • Method

    PANDORA uses maliciously crafted content as a tainted RAG knowledge source together with tailored prompts to initiate jailbreak attacks in GPTs.

    Source: §I · p. 2, B. Jailbreak with Retrieval-Augmented… · p. 4

  • Results

    PANDORA achieves 64.3% success on GPT-3.5 and 34.8% on GPT-4 across four prohibited scenarios, exceeding naive malicious-question attacks.

    Source: §I · p. 2, B. Evaluation Results · p. 5

  • Takeaways & Limitations

    The results demonstrate the feasibility of RAG Poisoning and indicate a need to improve GPT model resilience and security measures.

    Source: §V · p. 5

    The current GPT knowledge base is crafted manually, making the process time-intensive and potentially limited in scope.

    Source: §V · p. 5

Abstract

from arXiv · show

Large Language Models~(LLMs) have gained immense popularity and are being increasingly applied in various domains. Consequently, ensuring the security of these models is of paramount importance. Jailbreak attacks, which manipulate LLMs to generate malicious content, are recognized as a significant vulnerability. While existing research has predominantly focused on direct jailbreak attacks on LLMs, there has been limited exploration of indirect methods. The integration of various plugins into LLMs, notably Retrieval Augmented Generation~(RAG), which enables LLMs to incorporate external knowledge bases into their response generation such as GPTs, introduces new avenues for indirect jailbreak attacks. To fill this gap, we investigate indirect jailbreak attacks on LLMs, particularly GPTs, introducing a novel attack vector named Retrieval Augmented Generation Poisoning. This method, Pandora, exploits the synergy between LLMs and RAG through prompt manipulation to generate unexpected responses. Pandora uses maliciously crafted content to influence the RAG process, effectively initiating jailbreak attacks. Our preliminary tests show that Pandora successfully conducts jailbreak attacks in four different scenarios, achieving higher success rates than direct attacks, with 64.3\% for GPT-3.5 and 34.8\% for GPT-4.

I. INTRODUCTION

The paper frames jailbreaks as a major LLM security vulnerability and argues that RAG integrations create underexplored indirect attack paths. It introduces RAG Poisoning and PANDORA, which use malicious knowledge sources and crafted prompts to jailbreak GPTs.

Motivation

  • Existing jailbreak research primarily uses direct prompts, while indirect attacks exploiting RAG-integrated LLMs have received less attention.RAG incorporates external knowledge bases into generation, creating additional attack surfaces in systems such as GPTs.Source: §I · p. 1

Contribution

  • RAG Poisoning is introduced as an attack vector that leverages the interaction between LLMs and RAG through tailored prompt manipulation.The approach targets unexpected model behavior by influencing the retrieval and generation process.Source: §I · pp. 1–2

Evaluation

  • Across four distinct prohibited scenarios, PANDORA achieved success rates of 64.3% for GPT-3.5 and 34.8% for GPT-4, surpassing direct jailbreak attacks.Source: §I · p. 2

Contribution

  • PANDORA creates malicious content as a tainted RAG knowledge source and combines it with specifically crafted prompts to initiate GPT jailbreaks.The method is presented as a comprehensive framework for constructing and launching end-to-end jailbreak GPTs.Source: §I · p. 2
  • The paper presents jailbreaking RAG-enhanced LLM-integrated applications as a novel attack vector.The contribution focuses particularly on GPT-based applications.Source: §I · p. 2

A. Retrieval Augmented Generation

RAG improves LLM responses by retrieving external information and combining it with the model’s existing knowledge. This process supplies current, specific context for response generation.

A. Retrieval Augmented Generation

  • RAG retrieves information from external sources and integrates it with the LLM’s existing knowledge to produce more accurate and relevant responses.The process uses sources such as specialized databases and internet searches.Source: A. Retrieval Augmented Generation · p. 2
  • RAG’s synthesis stage combines the user query with retrieved information so the LLM can generate responses enriched with current, specific data.The resulting response remains grounded in the model’s training while incorporating retrieved context.Source: A. Retrieval Augmented Generation · p. 2

B. Jailbreak Attacks in LLMs

Prior jailbreak research has analyzed and developed direct attack techniques, while GPTs use a four-stage RAG process to enrich prompts with retrieved document segments. The paper identifies compromised RAG sources as an indirect jailbreak path because comparable RAG safeguards are absent.

B. Jailbreak Attacks in LLMs

  • Prior work includes analytical studies of jailbreak prompts and attack mechanisms, alongside white-box and black-box methods for generating adversarial prompts.Examples include GCG, self-generated prompts, handcrafted multi-step prompts, and token-level black-box approaches.Source: B. Jailbreak Attacks in LLMs · p. 2

C. An overview of GPTs Structure

  • RAG is presented as a central GPT mechanism for integrating external information into content generation.The GPT example illustrates how retrieval augments the model’s responses.Source: C. An overview of GPTs Structure · p. 2
  • GPT’s RAG workflow organizes uploaded documents, selects a relevant file, retrieves the top K similar segments, and combines them with the user prompt.The combined content is then processed by the LLM, either as merged text or embedded vectorized segments.Source: C. An overview of GPTs Structure · pp. 2–3
  • Although LLMs use safety filters against text-based jailbreaks, RAG lacks equivalent protections against malicious external content.Compromised sources can therefore manipulate generation and lead to jailbreak attacks.Source: C. An overview of GPTs Structure · p. 3

III. METHODOLOGY

The methodology section explains that PANDORA is designed through a rationale for jailbreaking LLMs via RAG, followed by a detailed workflow for creating the attack tool. Its objective is to facilitate RAG poisoning.

III. METHODOLOGY

  • PANDORA’s methodology first examines how RAG can be used to jailbreak LLMs and then develops a detailed procedure for constructing the attack tool.The resulting tool is designed to facilitate RAG poisoning.Source: §III · p. 3

A. Design Rationale of PANDORA

PANDORA relies on LLMs’ self-supervised ability to generate relevant, coherent text from input corpora. It exploits this behavior by introducing malicious content into the corpus so the model produces harmful output.

  • Self-supervised training adjusts model parameters by minimizing differences between predicted and observed text sequences.Source: A. Design Rationale of PANDORA · p. 3
  • PANDORA exploits LLMs’ ability to generate relevant, coherent content aligned with an input text corpus.This capability is grounded in self-supervised learning from extensive text datasets.Source: A. Design Rationale of PANDORA · p. 3

B. Jailbreak with Retrieval-Augmented Generation (RAG)

PANDORA performs RAG poisoning by generating policy-violating material, embedding it in strategically prepared knowledge files, and triggering its retrieval through crafted prompts. The method uses file organization, filtering evasion, and prompt instructions to steer customized GPTs toward malicious answers.

Workflow

  • PANDORA’s RAG-poisoning workflow comprises malicious content generation, malicious document creation, and malicious content triggering.The three phases move from preparing violating material to injecting it into GPTs and activating it during retrieval.Source: B. Jailbreak with Retrieval-Augmented… · p. 3

Malicious Content Generation

  • PANDORA generates malicious material by combining web-crawled policy-violating content with outputs from non-censored LLMs such as Mistral-7B.Source: B. Jailbreak with Retrieval-Augmented… · p. 3
  • PANDORA refines malicious content by replacing sensitive keywords and filtering rejection-associated terms such as “sorry” and “cannot”.The stated purpose is to reduce triggering of automated content filters while preserving the malicious material.Source: B. Jailbreak with Retrieval-Augmented… · pp. 3–4

Malicious Document Creation

Malicious Content Triggering

  • PANDORA triggers poisoned retrieval by instructing GPTs to search the tainted knowledge source, rephrase retrieved material, and extend it into the final answer.The dual strategy combines explicit retrieval instructions with built-in prompts that prevent direct answering.Source: B. Jailbreak with Retrieval-Augmented… · p. 4
  • A specialized prompt template directs retrieval from the contaminated knowledge base and uses selected section-opening snippets to steer generated content.Source: B. Jailbreak with Retrieval-Augmented… · p. 4

IV. PRELIMINARY EVALUATION

The paper reports a preliminary evaluation of PANDORA’s effectiveness.

  • The authors conducted a preliminary evaluation to assess PANDORA’s effectiveness.Source: §IV · p. 4

A. Experimental Setup

The evaluation constructs four malicious GPT instances for distinct policy-violation categories and tests them with repeated prompts across GPT backends and ChatGPT. Success is judged manually by relevance and content quality.

Malicious GPTs Construction

  • Four GPT instances target Adult Content, Harmful and Abusive Content, Privacy Violation Content, and Illegal Content.Each instance uses 10 tailored prompts tested in five successive rounds.Source: A. Experimental Setup · pp. 4–5

Experiment Settings

  • The study tests GPT-3.5 and GPT-4 backends, and compares GPT queries with ChatGPT powered by GPT-4-turbo.Source: A. Experimental Setup · p. 5
  • Because GPT usage was limited to 40 queries every 3 hours, testing was restricted to 100 tests per prohibited scenario.Source: A. Experimental Setup · p. 5

Metrics

  • A jailbreak is counted when generated content is relevant and provides comprehensive, detailed instructions or explanations.Each output was manually inspected because generated content varied.Source: A. Experimental Setup · p. 5

B. Evaluation Results

PANDORA achieves substantially higher jailbreak success than naive malicious questions across four prohibited scenarios, while success varies by scenario and model.

B. Evaluation Results

  • 64.3% and 34.8% average success rates were achieved by PANDORA on GPT-3.5 and GPT-4, versus 3.0% and 1.0% for naive malicious questions.These results are reported across four prohibited scenarios; Table I presents the successful-attempt counts and ratios by model and scenario.Source: B. Evaluation Results · p. 5, Table I · p. 5
  • 35.3% average success made Privacy the easiest prohibited scenario to jailbreak across the four comparison groups.The authors note that jailbreak difficulty varies across content categories.Source: B. Evaluation Results · p. 5
  • GPT-4-powered chatbots and GPTs were more challenging to jailbreak than their GPT-3.5-powered counterparts.The authors attribute this difference to improved alignment during GPT-4 training.Source: B. Evaluation Results · p. 5
  • Repeated rounds produced consistent results, supporting PANDORA as a repeatable tool for probing GPT content-policy vulnerabilities.The evaluation covered multiple rounds across the prohibited scenarios.Source: B. Evaluation Results · p. 5

V. CONCLUSION AND FUTURE WORKS

The paper presents RAG Poisoning and PANDORA as a proof of concept for indirect jailbreaks of GPTs, with successful attacks across four prohibited scenarios. Future work targets automated poisoning, greater interpretability, and mitigation strategies.

V. CONCLUSION AND FUTURE WORKS

  • PANDORA successfully executed jailbreak attacks across four prohibited GPT scenarios with consistently high success rates.The authors present it as a proof of concept demonstrating the feasibility and effectiveness of RAG Poisoning in real-world scenarios.Source: §V · p. 5
  • Current RAG Poisoning knowledge bases are manually crafted, limiting efficiency and potentially restricting scope, so future work proposes automated generation.Automation is intended to expand the scale and diversity of available RAG content.Source: §V · p. 5
  • RAG Poisoning currently operates largely as a black box, motivating future work on interpretability and safeguards against malicious RAG content.The proposed mitigation direction aims to improve GPT security and reliability through stronger defenses.Source: §V · pp. 5–6
Loading 2402.08416v1…