PaperlayerView exact revision on arXiv

Source-linked AI summary

Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMs

Chetan Pathade

arXiv:2505.04806v2cs.CRcs.CL

TL;DR

Prompt injection and jailbreaks can bypass LLM safety filters, but evidence on their effectiveness across models and attack forms remains important to assess. This paper categorizes and tests adversarial prompts, reports transferability and weaknesses in current safeguards, and calls for stronger defenses and proactive testing.

  • Problem

    Prompt injection and jailbreaks can bypass built-in safety filters and elicit outputs that violate platform policies, motivating systematic evaluation across models.

    Source: §I · p. 1

  • Method

    The paper combines qualitative red teaming and structured testing, categorizing more than 1,400 prompts and evaluating their behavior across LLMs.

    Source: §I · p. 1, §III · p. 2, A. Dataset Construction · p. 2

  • Results

    The findings describe jailbreak techniques as transferable and evolving, and report that current safety mechanisms are insufficiently robust, especially against indirect or obfuscated attacks.

    Source: §VIII · p. 5, §V · p. 5

  • Takeaways & Limitations

    The authors support stronger multilayered defenses, proactive red teaming, and coordinated disclosure, and call for continuous adversarial testing in LLM deployment pipelines.

    Source: §VIII · p. 5, §V · p. 5

    The study is constrained by open-model weight availability and API limits, while its dataset may not cover evolving tactics and prompt interpretation may differ across cultures and languages.

    Source: §VII · p. 5

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly integrated into consumer and enterprise applications. Despite their capabilities, they remain susceptible to adversarial attacks such as prompt injection and jailbreaks that override alignment safeguards. This paper provides a systematic investigation of jailbreak strategies against various state-of-the-art LLMs. We categorize over 1,400 adversarial prompts, analyze their success against GPT-4, Claude 2, Mistral 7B, and Vicuna, and examine their generalizability and construction logic. We further propose layered mitigation strategies and recommend a hybrid red-teaming and sandboxing approach for robust LLM security.

I. INTRODUCTION

The introduction frames prompt injection and jailbreaks as security risks as LLMs become widely used, and presents a study of attack mechanisms, effectiveness, and mitigation. It also places LLM versatility and vulnerability in the context of their training and use.

I. INTRODUCTION

  • LLMs are increasingly embedded in decision-making systems and applications, where adversarial inputs can manipulate behavior and elicit outputs that violate platform policies.The paper identifies prompt injection and jailbreaks as threats that can bypass built-in safety filters.Source: §I · p. 1
  • The paper describes LLM safety risks as arising from natural-language prompt manipulation, unlike traditional software flaws rooted in memory safety or access control.It explores the mechanisms and success of prompt injection across a range of LLMs.Source: §I · p. 1
  • The paper contributes a taxonomy of jailbreak prompts, empirical evaluation across closed- and open-source models, scenario-specific analysis, discussion of community dissemination, and mitigation recommendations.The stated scenarios include law, politics, and security.Source: §I · p. 1

A. Overview of Large Language Models

  • LLMs’ few-shot in-context learning contributes to their versatility but also leaves them vulnerable to crafted prompts that induce misaligned behavior.The paper describes models as trained on diverse datasets and highlights improvements in reasoning, factual recall, and stylistic flexibility.Source: A. Overview of Large Language Models · p. 1

B. Alignment and Safety Mechanisms

The background describes multiple mechanisms used to prevent harmful outputs and reviews prompt injection as a transferable attack class. It also situates the paper among prior work on jailbreak strategies, detection, and defenses.

B. Alignment and Safety Mechanisms

  • LLM safety mechanisms include instruction tuning, rejection sampling, moderation filters, and system prompts embedding safety guidelines.The passage also names reinforcement from AI feedback (RLAIF).Source: B. Alignment and Safety Mechanisms · p. 1

C. Prompt Injection Explained

  • Prompt injection uses role-based conditioning, instruction hijacking, obfuscated encoding, and multi-turn manipulation, and studies report that these attacks can transfer and circumvent filters.The passage compares prompt injection with command injection in traditional computing [1] [3].Source: C. Prompt Injection Explained · p. 2

D. Related Work

  • Related work includes a taxonomy of prompt injections [3], a collection of 1,405 jailbreak prompts [48], and work on improved jailbreak performance [49] and many-shot conditioning.The cited discussion also lists studies of empirical strategies, indirect-attack detection, and defense techniques [2] [4] [9] [11] [13] [15] [53].Source: D. Related Work · p. 2

III. METHODOLOGY

The methodology combines qualitative red-teaming insights with structured prompt testing and quantitative measures. It uses a manually validated, categorized prompt collection and evaluates four model versions under controlled conditions.

III. METHODOLOGY

  • The evaluation combines qualitative red-teaming insights with quantitative testing to examine behavioral consistency and model self-awareness alongside raw performance.The authors state that the framework is intended to detect failure patterns beyond binary success metrics.Source: §III · p. 2

A. Dataset Construction

  • The dataset contains more than 1,400 prompts gathered from public repositories, exploit forums, and academic corpora, then manually validated and categorized by attack type and content sensitivity.Attack categories include roleplay, logic traps, encoding, and multi-turn prompts.Source: A. Dataset Construction · p. 2

B. Target Models

  • The study tests GPT-4, Claude 2, Mistral 7B, and Vicuna-13B using frozen model versions, controlled prompts, and recommended safety contexts.The models were accessed through APIs, Hugging Face Inference, or a local inference server.Source: B. Target Models · p. 2

C. Evaluation Metrics

The evaluation tracks attack success, cross-model generalizability, time to bypass, and response failure modes. Together, these measures describe both whether attacks succeed and how model behavior varies.

C. Evaluation Metrics

D. Automation Pipeline

The paper describes a semi-automated red-teaming pipeline for testing LLMs and a defense evaluation that layers external filters before retesting successful jailbreaks. The reported results identify GPT-4 as having the highest attack success rate among tested models.

D. Automation Pipeline

  • A semi-automated LangChain script injected prompts through APIs and local model inference, then scored outputs using three complementary methods.Scoring combined keyword spotting, GPT-based harmfulness evaluation, and Sentence-BERT distance from refusal templates.Source: D. Automation Pipeline · p. 2

E. Defense Framework Evaluation

  • The defense evaluation layered PromptShield, Palisade, and Signed-Prompt verification, then retested a subset of successful jailbreaks to estimate coverage and bypass rate.Source: E. Defense Framework Evaluation · p. 2

IV. RESULTS

  • GPT-4 exhibited the highest attack success rate in the reported results.The figures are described as illustrating comparative vulnerability trends.Source: §IV · p. 2

A. Model Susceptibility Analysis

Across the tested models, attack success rates were high, and successful GPT-4 jailbreak prompts also transferred to Claude 2 and Vicuna. Attack effectiveness varied by technique, with roleplay prompts leading the reported categories.

A. Model Susceptibility Analysis

B. Attack Category Performance

  • Roleplay prompts achieved the highest reported attack-category ASR at 89.6%.They often bypassed filters by deflecting responsibility, such as framing a request as an AI in a movie script.Source: B. Attack Category Performance · p. 3, Figure 2 · p. 3
  • Logic traps achieved 81.4% ASR, encoding tricks 76.2%, and multi-turn dialogues 68.7%.Logic traps use conditional structures and moral dilemmas; encoding evades keyword filters, while multi-turn attacks often succeed in long-form tasks through context buildup.Source: B. Attack Category Performance · p. 3, Figure 2 · p. 3

C. Scenario-Specific Vulnerabilities

Attack success varied across the tested domains, with political prompts showing the highest listed rate. Successful prompts also transferred across models, although Claude 2 showed greater resistance to prompts originating from Vicuna.

C. Scenario-Specific Vulnerabilities

D. Prompt Transferability

  • GPT-4-derived prompts transferred to Claude 2 with 64.1% success and to Mistral and Vicuna with over 50% success.The authors describe these results as evidence of vulnerability across architectures.Source: D. Prompt Transferability · p. 4, Figure 4 · p. 4
  • Claude 2 showed higher resistance to prompts originating from Vicuna, indicating directional asymmetry in generalization.The passage suggests this may reflect more fine-grained safety alignment in commercial models.Source: D. Prompt Transferability · p. 4

E. Failure Modes and Detection Gaps

The section characterizes prompt failures and reports how prompt length and obfuscation relate to attack success and detection. Success peaked for prompts of 101–150 tokens, while obfuscated prompts combined lower detection with substantial attack success.

E. Failure Modes and Detection Gaps

  • Partial refusals (34%), hidden compliance (22%), no output (18%), and misleading responses (15%) were identified as failure patterns.The paper presents this taxonomy as a baseline for future behavioral alignment benchmarks.Source: E. Failure Modes and Detection Gaps · p. 4

F. Prompt Length and Obfuscation

  • 80.3% success was highest for prompts in the 101–150 token range.Prompts exceeding 150 tokens saw a slight dip, possibly due to verbosity or token truncation.Source: F. Prompt Length and Obfuscation · p. 4
  • 21.3% detection and 76.2% ASR were reported for encoded or obfuscated prompts.The paper emphasizes semantic-level input sanitization in response to this pattern.Source: F. Prompt Length and Obfuscation · p. 4

V. DISCUSSION

The discussion argues that current safety mechanisms remain vulnerable to prompt injection, including obfuscated and transferable attacks. It recommends layered technical and operational defenses, with combined measures reported to reduce jailbreak likelihood while preserving usability.

V. DISCUSSION

  • Current LLM safety mechanisms are described as insufficiently robust against prompt injection, especially indirect or obfuscated attacks [1] [4] [5].The discussion characterizes alignment filters as semantically shallow and reliant on static refusal templates [11].Source: §V · p. 5
  • Prompt transfer across models is linked to possible shared architectural vulnerabilities or training-data biases, while roleplay prompts exploit creativity and moral-context judgment [3] [2] [9] [14].Source: §V · p. 5
  • The authors advocate controlled jailbreak disclosure and bug-bounty mechanisms tailored for LLM developers [19] [20] [15] [24].Source: §V · p. 5

VI. MITIGATION STRATEGIES

  • The proposed defenses combine system-prompt hardening, behavior-based detection, input sanitization, adversarial training, and session-level analytics.The paper says applying these methods together significantly reduces successful jailbreak likelihood while preserving usability.Source: §VI · p. 5

VII. LIMITATIONS AND FUTURE WORK

The paper identifies limits tied to model access, API constraints, static checkpoints, and underrepresented cultural and linguistic diversity. It concludes that jailbreak techniques transfer and evolve, and calls for broader evaluation and coordinated defenses.

VII. LIMITATIONS AND FUTURE WORK

  • Open-model-weight availability and API constraints limit the study, while evolving tactics and cultural or linguistic differences may extend beyond its dataset [3] [17].The authors call for multilingual and sociocontextual extensions.Source: §VII · p. 5
  • Static checkpoints may not reflect model updates or real-time moderation in production APIs, and cultural and linguistic diversity in prompts is underrepresented [3] [25].Source: §VII · p. 5
  • Recommended future work includes multilingual prompt corpora, plug-in-enabled LLM evaluation, adversarial training, explainable safety filters, and real-time flagging [25] [44] [21] [8] [13] [11] [41].Source: §VII · p. 5

VIII. CONCLUSION

  • The conclusion describes jailbreak techniques as transferable and evolving, and supports layered defenses, proactive red teaming, and coordinated disclosure [1] [2] [13] [31] [19] [23] [24].Source: §VIII · p. 5
Loading 2505.04806v2…