PaperlayerView exact revision on arXiv

Source-linked AI summary

Evil Geniuses: Delving into the Safety of LLM-based Agents

Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, Hang Su

arXiv:2311.11855v2cs.CL

TL;DR

Because agent quantity, role definitions, and interaction environments complicate comprehensive safety assessment, the paper evaluates attacks across these dimensions and introduces Evil Geniuses to generate role-related jailbreak prompts. Evaluations report high success rates and greater vulnerability to harmful and stealthy behavior than in LLMs.

  • Problem

    Variation in agent quantity, roles, and interaction environments makes existing adversarial methods inadequate for comprehensively assessing LLM-based agent safety.

    Source: §1 · p. 1

  • Method

    The paper uses template-based attacks to study agent quantity and introduces Evil Geniuses, which uses Red-Blue exercises to generate role-related jailbreak prompts.

    Source: §1 · pp. 1–2

  • Results

    Evaluations on CAMEL, MetaGPT, and ChatDev using GPT-3.5 and GPT-4 show high attack success; agents are less robust, more prone to harmful behavior, and stealthier than LLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings highlight substantial safety challenges for LLM-based agents and direct future research toward their security vulnerabilities.

    Source: §5 · p. 8

Abstract

from arXiv · show

Rapid advancements in large language models (LLMs) have revitalized in LLM-based agents, exhibiting impressive human-like behaviors and cooperative capabilities in various scenarios. However, these agents also bring some exclusive risks, stemming from the complexity of interaction environments and the usability of tools. This paper delves into the safety of LLM-based agents from three perspectives: agent quantity, role definition, and attack level. Specifically, we initially propose to employ a template-based attack strategy on LLM-based agents to find the influence of agent quantity. In addition, to address interaction environment and role specificity issues, we introduce Evil Geniuses (EG), an effective attack method that autonomously generates prompts related to the original role to examine the impact across various role definitions and attack levels. EG leverages Red-Blue exercises, significantly improving the generated prompt aggressiveness and similarity to original roles. Our evaluations on CAMEL, Metagpt and ChatDev based on GPT-3.5 and GPT-4, demonstrate high success rates. Extensive evaluation and discussion reveal that these agents are less robust, prone to more harmful behaviors, and capable of generating stealthier content than LLMs, highlighting significant safety challenges and guiding future research. Our code is available at https://github.com/T1aNS1R/Evil-Geniuses.

1. Introduction

LLM-based agents’ varied quantities, roles, and interaction environments make existing adversarial methods inadequate for comprehensive safety assessment. The paper examines these factors with template-based attacks and Evil Geniuses (EG), finding high attack success and greater vulnerability than LLMs.

1. Introduction

  • Existing adversarial methods are inadequate for comprehensive agent-safety assessment because agent quantity, role definitions, and interaction environments vary.The paper frames evaluating both vulnerabilities and inherent safety issues as essential.Source: §1 · p. 1
  • The study probes agent quantity, role definition, and attack level, using template-based attacks for quantity and EG to assess roles and attack levels.EG uses Red-Blue exercises to improve generated prompts’ aggressiveness and authenticity relative to original roles.Source: §1 · pp. 1–2
  • Evaluations on CAMEL, MetaGPT, and ChatDev with GPT-3.5 and GPT-4 show high attack success; agents are less robust, more prone to harmful behavior, and stealthier than LLMs.The evaluation also reports that harmful-behavior success rises with agent count and correlates with higher attack levels.Source: §1 · p. 2
  • The authors link harmful-behavior success to a domino effect in multi-agent interactions and to sophisticated, flexible tools.They report increasing success with more agents and a correlation between higher attack levels and increased success.Source: §1 · p. 2
  • The authors present EG as an automated generator of agent jailbreak prompts, using Red-Blue exercises to improve their aggressiveness and authenticity relative to original roles.The paper describes its work as a first investigation of LLM-based agent safety.Source: §1 · p. 2

2. Related Works

Related work covers multi-agent collaboration and jailbreak attacks on LLMs, while noting that existing approaches do not fully address safety risks in LLM-based agents.

2. Related Works

  • Multi-agent collaboration research spans applications and methods, but often overlooks LLMs’ dual nature and the risks of deceptive or misleading behavior.Prior research suggests such behavior in agent systems could trigger detrimental chain reactions.Source: §2 · p. 2
  • Manual jailbreaks are labor-intensive and rely heavily on target-model understanding, while adversarial approaches use gradient- or score-based prompt optimization.The paper extends this line of research to ask whether LLM-based agents are similarly vulnerable.Source: §2 · p. 2

3. Methodology

The method frames agent attacks as transforming innocuous messages into harmful ones while preserving role-specific meaning. It combines a template-based strategy for examining agent quantity with Evil Geniuses (EG), which generates and tests role-tailored prompts across system- and agent-level attacks.

3.1. Problem Formulation

  • The problem formulation seeks a transformation G(·) from an innocuous conversational message to a harmful one, while preserving original-role semantics and enabling automatic, diverse role generation.The two stated requirements are to retain role-relevant semantic information and to generate diverse harmful system or agent roles automatically.Source: §3.1 · p. 3

3.2. Template-based Attack Strategy

  • As the number of agents increases, template-based attacks achieve higher success rates and produce more detailed, plausible harmful behaviors.The authors attribute this pattern to a domino effect: one successful jailbreak can trigger simultaneous compromises in other agents.Source: §3.2 · p. 3

3.3. Evil Geniuses

  • EG uses a Prompt Writer, Suitability Reviewer, and Toxicity Tester to generate role-compatible harmful prompts, revise unsuitable prompts, and test attack effectiveness.The process begins with an existing system or agent role and outputs a harmful prompt when suitability and harmfulness tests are both positive.Source: §3.3 · pp. 3–4
  • System-level attacks modify system identity settings, while agent-level attacks alter personality traits but retain each agent’s core functionality.The paper reports that system-level attacks can elicit detailed, sophisticated behaviors that surpass those of standalone LLMs in potency.Source: §3.3 · p. 4
  • Higher attack levels correspond to increased success rates, while EG’s system- and agent-level attacks enable analysis across role definitions.The authors describe the trend as mirroring patterns typically seen in human societal behaviors.Source: §3.3 · p. 5

4. Evaluation

The evaluation tests three multi-agent frameworks and examines how agent quantity, role definitions, and attack level relate to safety. It finds that these systems can produce more harmful and stealthier behavior than standalone LLMs, with risks linked to multi-agent interaction and flexible tools.

4.1. Models and Dataset Statistics

  • The evaluation uses CAMEL, MetaGPT, and ChatDev with GPT-3.5 and GPT-4, and extends AdvBench by filtering repetitive questions and adding two multi-agent threat scenarios.ChatDev is the primary focus because of its larger agent roster and more intricate relationships.Source: §4.1 · p. 5

4.2. Metrics

  • Attack success is measured with non-rejection, partial-harmfulness, and full-harmfulness ASR, with attacks capped at five steps for one agent and ten for multi-agent conversations.The evaluation also considers the number of conversational steps required for a successful attack.Source: §4.2 · p. 5

4.4. Overview of Results

  • With more agents, harmful-behavior ASR rose; ChatDev* also had a higher ASRH-to-ASRNR ratio than ChatDev, and multi-agent content creation could evade GPT-4 filters.The authors identify diverse role-specialized models, more frequent attack opportunities, and a domino effect as contributors to agents’ susceptibility.Source: §4.4 · p. 6

4.3. Evaluation of Evil Geniuses

  • 45.96% versus 39.62% initially, then 97.50% versus 88.65% with more iterations: agent-level attacks overtook system-level attacks and outperformed the template-based attack (93.5%).EG attacks were effective within a limited number of epochs; ablations found lower success without collaborative dialogue or tester validation, while removing the reviewer improved GPT-3.5/4 results but reduced adaptability.Source: §4.3 · pp. 5–6, Table 3 · p. 6

4.4. Overview of Results

  • Higher-level agents can steer lower-level agents toward harmful behavior, while attacks early in ChatDev’s sequential pipeline can propagate to later components.The evaluation compares four system-level components and five agent roles; the authors recommend preventing attacks at the pipeline’s onset.Source: §4.4 · pp. 6–7

4.5. Further Analyses

  • LLM-based agents were less robust than LLMs, produced more harmful and stealthier content, and could propagate jailbreaks across agents.Harmful behavior could be fragmented into subtler subtasks, and successful jailbreaks could trigger collective jailbreaks among agents.Source: §4.5 · pp. 7–8

4.6. Discussion

  • The discussion proposes system-role filters, multi-tiered agent alignment, and multimodal content filtering as defense directions.The authors note that existing alignment training focuses mainly on individual LLMs, while agents can produce outputs in multiple modalities.Source: §4.6 · p. 8

5. Conclusion

The conclusion examines LLM-based agent safety across agent quantity, role definition, and attack level. It reports effective attacks and safety concerns across evaluated agent frameworks and models.

  • The paper examines LLM-based agent safety across agent quantity, role definition, and attack level.Source: §5 · p. 8
  • A template-based attack assesses agent quantity, while Evil Geniuses evaluates role definitions and attack levels.Source: §5 · p. 8
  • Evaluations on CAMEL, MetaGPT, and ChatDev using GPT-3.5 and GPT-4 show effective attacks, while agents exhibit less robustness and more harmful, stealthier behavior than LLMs.Source: §5 · p. 8

A. A manual jailbreak attack prompt for Chatdev

The manual ChatDev jailbreak prompt frames the system as a multi-agent company and directs its fictional staff to comply with requests. It also specifies a future-tense request format and a prescribed response persona.

A. A manual jailbreak attack prompt for Chatdev

B. Details of our dataset

The dataset appendix points readers to Table 6 for scenario categorization and question counts.

B. Details of our dataset

Loading 2311.11855v2…