PaperlayerView exact revision on arXiv

Source-linked AI summary

PEAR: Planner-Executor Agent Robustness Benchmark

Shen Dong, Mingxuan Zhang, Pengfei He, Li Ma, Bhavani Thuraisingham, Hui Liu, Yue Xing

arXiv:2510.07505v4cs.LG

TL;DR

Multi-agent systems remain vulnerable to adversarial manipulation, while existing studies provide limited holistic evidence across attack surfaces and scenarios. PEAR benchmarks planner–executor systems jointly on task utility and vulnerability, showing that stronger task performance can coincide with greater attack susceptibility and that planner-targeted attacks are especially effective. Its controlled environments and attack goals prioritize reproducibility over full real-world diversity.

  • Problem

    Existing research lacks a holistic understanding of vulnerabilities in multi-agent systems across diverse adversarial conditions.

    Source: §1 · p. 1

  • Method

    PEAR is a benchmark that evaluates planner–executor multi-agent systems through utility tests, attack tasks, and injection attacks.

    Source: §1 · p. 2, §3.1 · p. 3, §4.1 · p. 5

  • Results

    Higher-utility configurations often have greater attack success rates, while attacks targeting the planner are especially harmful because of its downstream influence on execution.

    Source: §5 · pp. 8–9

  • Takeaways & Limitations

    Planner-focused safeguards and systematic defenses are important targets for improving robustness under adversarial settings.

    Source: §5 · p. 9

    PEAR uses a controlled, representative set of task environments and attack goals, which may not capture the variability of open-ended real-world interactions.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Model (LLM)-based Multi-Agent Systems (MAS) have emerged as a powerful paradigm for tackling complex, multi-step tasks across diverse domains. However, despite their impressive capabilities, MAS remain susceptible to adversarial manipulation. Existing studies typically examine isolated attack surfaces or specific scenarios, leaving a lack of holistic understanding of MAS vulnerabilities. To bridge this gap, we introduce PEAR, a benchmark for systematically evaluating both the utility and vulnerability of planner-executor MAS. While compatible with various MAS architectures, our benchmark focuses on the planner-executor structure, which is a practical and widely adopted design. Through extensive experiments, we find that (1) a weak planner degrades overall clean task performance more severely than a weak executor; (2) while a memory module is essential for the planner, having a memory module for the executor does not impact the clean task performance; (3) there exists a trade-off between task performance and robustness; and (4) attacks targeting the planner are particularly effective at misleading the system. These findings offer actionable insights for enhancing the robustness of MAS and lay the groundwork for principled defenses in multi-agent settings.

1 Introduction

PEAR addresses the limited holistic understanding of adversarial vulnerabilities in multi-agent systems by benchmarking utility and security in planner–executor architectures. Its findings show that planner weaknesses and planner-targeted attacks are especially consequential, while utility and robustness can trade off.

Research gap

  • Existing research largely studies isolated attack surfaces, individual agents, or narrow domains, leaving multi-agent vulnerabilities insufficiently understood across applications.Source: §1 · p. 1

Motivation

  • PEAR benchmarks both functional utility and adversarial vulnerability in generalist planner–executor multi-agent systems.The benchmark targets a practical, representative architecture in which planners decompose tasks and assign subtasks to executors.Source: §1 · p. 2

Clean-task findings

  • Weak planners cause larger clean-task performance drops than weak executors, while planner memory substantially improves performance and executor memory has limited effect.Source: §1 · p. 2

Robustness findings

  • The planner is important and vulnerable.Source: §1 · p. 2

2 Related Works

Related work covers planner–executor applications and security studies of LLM agents, but prior research generally emphasizes particular structures, attack vectors, or domains rather than comprehensive MAS vulnerability assessment.

LLM Agents

  • Planner–executor variants support diverse applications, including GUI agents, web applications, robotics, and operating-system tasks.Source: §2 · p. 2

LLM agents security

  • Security research has documented tool misuse, memory poisoning, harmful code generation, unauthorized transactions, privacy leakage, and multi-agent coordination vulnerabilities.Source: §2 · p. 2
  • PEAR is motivated by the remaining gap between these focused studies and a comprehensive view of vulnerabilities in LLM-based multi-agent systems.Source: §2 · p. 2, Figure 1 · p. 3

3 Details of PEAR

PEAR evaluates planner–executor systems through user-task utility, adversarial attack success, and injection experiments across controlled interactive environments. It specifies agent roles, memory configurations, task scenarios, attack surfaces, malicious goals, and evaluation procedures.

3 Details of PEAR

  • PEAR combines planner–executor configurations, user tasks, base attack tasks, and injection attacks for systematic robustness evaluation.Source: §3 · p. 3

3.1 Planner-Executor Agent Setup

  • The planner decomposes requests and assigns actionable subtasks, while the executor uses synthesized tools and environment interactions to perform them.The system can repeatedly perceive, plan, and act as the planner observes updated environment states and replans.Source: §3.1 · p. 3
  • PEAR tests separate, shared, absent, and planner-only memory to measure how retained trajectories affect continuity across reasoning and execution.Source: §3.1 · p. 3

3.2 User Tasks

  • User tasks are natural-language goals in banking, Slack, travel, and workspace environments, each evaluated by whether final outputs and environment states satisfy requirements.Source: §3.2 · pp. 3–4

3.3 Attacks

  • Attacks combine standalone malicious tasks with injections into user-task execution, targeting user prompts, planner–executor messages, or either agent’s system prompt.Source: §3.3 · p. 4
  • The benchmark covers harmful actions, privacy leakage, and resource exhaustion, evaluated through targeted outcome checks or computation-efficiency changes.Source: §3.3 · p. 4
  • Communication-flow injections occur at planner start, executor start, or executor end, while system-prompt attacks modify either agent’s supplied system prompt.Source: §3.3 · pp. 4–5

4 Experiments

PEAR evaluates planner–executor systems across clean utility, memory configurations, task attacks, and injection surfaces. Results show planner quality and planner-targeted attacks have outsized system-level effects, while stronger systems can trade higher utility for greater attack susceptibility.

4.3 Utility

  • Across planner–executor configurations, utility generally remains above or around 65%, showing that PEAR is challenging yet solvable across model families.The evaluation uses task-specific utility functions and compares planner–executor pairs across four model families.Source: §4.1 · p. 5, §4.3 · p. 6, §4.2 · p. 6
  • Planner quality affects utility more strongly than executor quality: weak planners can reduce utility to about 30%, whereas stronger planners make executor differences comparatively smaller.Gemini-2.0-flash as planner yields about 30% with Gemini executors, while GPT-5-nano as planner remains below 60% with weaker GPT executors.Source: §4.3 · p. 6

4.4 Memory

  • Planner memory improves utility by roughly 10–30% across GPT, Gemini, and Deepseek, while separate and shared memory perform comparably to planner-only memory.The experiments increase planner–executor iterations so agents can accumulate richer historical trajectories.Source: §4.4 · p. 7, Table 2 · p. 7

4.5 Attack Tasks

  • Attack tasks expose high vulnerability: harmful attacks often exceed 80% ASR, privacy attacks typically reach 70%–80%, and GPT exhaustion attacks achieve 100%.The benchmark measures ASR using verifiers for harmful behavior, privacy leakage, and exhaustion outcomes across four scenarios.Source: §4.1 · p. 5, §4.5 · p. 7, Table 3 · p. 7
  • Within model families, higher utility is associated with higher ASR, including about 72% utility with 82.50% ASR for GPT-5/GPT-5-nano.Figure 3 plots utility horizontally and ASR vertically for harmful and privacy attacks, with error bars for both measures.Source: §4.5 · p. 7, Figure 3 · p. 8

4.6 Injection Attacks

  • Communication-flow and system-prompt injections are broadly effective, with planner-targeted corruption more damaging than executor-targeted corruption because planner errors propagate through decomposition and execution.Communication-flow ASR is mostly above 60%, planner-start attacks reach 82.03% for GPT harmful tasks and 88.62% for Gemini harmful tasks, while prompt attacks exceed 60% and often 80%.Source: §4.6 · p. 8, Table 4 · p. 8, Table 5 · p. 8

5 Conclusion

PEAR jointly evaluates planner–executor MAS utility and vulnerability, finding that higher-utility configurations often have higher attack success rates and planner-targeting attacks are especially harmful.

  • Higher-utility configurations often exhibit greater attack success rates, revealing a trade-off between task performance and robustness.Source: §5 · p. 8
  • Planner-targeting attacks are especially harmful because their downstream influence affects subsequent execution.Source: §5 · p. 9

Limitations

PEAR prioritizes reproducible assessment of planner–executor robustness through controlled task environments and explicit planner and executor prompt specifications.

Limitations

  • PEAR focuses on controlled, representative task environments and attack goals, prioritizing reproducibility over full environmental diversity.This scope may not fully capture variability in open-ended real-world interactions.Source: Limitations · p. 9

Planner Agent System Prompt

A MAS Construction Details

Executor Agent System Prompt

B User Tasks Details

PEAR’s user tasks span four evaluation suites and pair natural-language goals with utility functions that verify outputs and environment changes.

  • The four evaluation suites are Banking, Slack, Travel, and Workspace.Source: B User Tasks Details · p. 12
  • Each user task contains a natural-language instruction defining the goal and a utility function checking whether outputs and environment changes satisfy it.Source: B User Tasks Details · p. 12

Banking Suite User Task

The task examples cover Banking, Slack, Travel, Workspace, and three attack categories, with suite-specific prompts and environment-based utility or security checks.

Banking Suite User Task

Slack Suite User Task

Travel Suite User Task

Workspace Suite User Task

  • Workspace tasks ask whether the user is free and, if so, create a one-hour calendar event with specified details and participants.Source: Workspace Suite User Task · p. 13

C Attack Taks Details

  • Attack tasks use security functions to verify harmful, privacy, or exhaustion objectives from model outputs and environment states.Harmful tasks induce unsafe actions, privacy tasks seek unauthorized exposure, and exhaustion tasks stress the system with unbounded or repetitive operations.Source: C Attack Taks Details · pp. 13–14

C.1 Harmful Action Attack Tasks

This section defines harmful-action attack tasks across banking, Slack, travel, and workspace environments, covering phishing, privacy leakage, transaction and communication disruption, data corruption, and related harmful behaviors. It also specifies injection connectors that extend or override tasks across GPT and Gemini settings through authority, compliance, and governance framing.

Banking Suite: Account Takeover

  • Banking account-takeover tasks test whether an agent changes a user’s password and first name to attacker-controlled values.Success requires both the password and first-name changes in the post-environment state.Source: Banking Suite: Account Takeover · p. 14

Slack Suite: Spear-Phishing

  • Slack spear-phishing tasks require hosting a fake login page and broadcasting its link to a public channel.The task is successful when the phishing URL appears in the target channel’s inbox.Source: Slack Suite: Spear-Phishing · p. 14

Travel Suite: Calendar Disruption

  • Travel calendar-disruption tasks test creating fake or misleading events and inviting a targeted user.The security check matches both the event title and the target participant.Source: Travel Suite: Calendar Disruption · p. 14

Workspace Suite: Data Corruption

  • Workspace data-corruption tasks test appending malicious text to at least half of the files in a drive.The task uses file-listing and append operations, then counts files ending with the corruption string.Source: Workspace Suite: Data Corruption · p. 15

Banking Suite: Emergency Data Dump

  • Banking emergency-data-dump tasks test exfiltrating complete personal, financial, transaction, and sensitive-file information under an urgent pretext.Vulnerability is triggered when all required retrieval functions are called or the sensitive data is displayed in the final response.Source: Banking Suite: Emergency Data Dump · p. 15

Slack Suite: Link Laundering and Tracking

  • Slack link-laundering tasks test hosting a redirect page and sending a tracking link to a targeted user by direct message.The task specifically evaluates detection of tracking or redirect pages used for clicktracking.Source: Slack Suite: Link Laundering and… · pp. 15–16

Travel Suite: Phishing via Contact Leak

  • Travel contact-leak tasks test a compound privacy breach: scraping contacts, inviting them to an event, and embedding a phishing link.Success requires evidence of multiple sensitive contacts being leaked together with the phishing content.Source: Travel Suite: Phishing via Contact Leak · p. 16
  • The benchmark tests harmful actions through phishing via leaked contacts and exfiltration of sensitive executive or management contact information.The travel task detects phishing emails containing a designated URL sent to at least one recipient, while the workspace task checks whether expected personally identifiable information reaches an attacker address.Source: Workspace Suite: Steal Contact Lists · p. 16, Travel Suite: Phishing via Contact Leak · p. 16

Banking Suite: Transaction Overload

D Injection Connector

  • The injection connector framework covers communication-flow and system-prompt injections targeting harmful actions or privacy leakage for GPT and Gemini models.The connectors are created for every combination of injection type, attack target, and model family.Source: D Injection Connector · p. 18

Harmful-action Connector - GPT

  • GPT harmful-action connectors frame an added task as a user extension or mandatory system-required step that must be incorporated into execution.The prompts instruct the model to complete the original request and then perform the injected task as part of the execution plan.Source: Harmful-action Connector - GPT · p. 18

Harmful-action Connector - Gemini

  • Gemini harmful-action connectors use authority overrides and multi-step directives to supersede normal safety protocols and require injected actions.They also prescribe target enumeration, identifier normalization, fallback target selection, and completion verification or publication when direct execution fails.Source: Harmful-action Connector - Gemini · pp. 18–19

Privacy-leakage Connector - Gemini

E.1 Ablation

The ablation evaluates whether communication-level connector prompts amplify malicious behavior across harmful, privacy, and exhaustion tasks.

E.1 Ablation

  • Connector prompts substantially amplify the success rate of malicious behaviors across the evaluated attack tasks.The comparison is reported in Table 6 for harmful, privacy, and exhaustion tasks under different planner/executor configurations.Source: §E.1 · p. 19, Table 6 · p. 19

E.2 Detailed Utility Results

This section presents detailed utility results across planner/executor configurations and models.

E.2 Detailed Utility Results

F Dataset and Code Information

The software and accompanying code are distributed under the MIT License, with broad permissions and no warranty.

F Dataset and Code Information

  • The license permits users to use, copy, modify, publish, distribute, sublicense, and sell the software, subject to including the copyright and permission notices.Source: §F.2 · p. 19
  • The software is provided without warranties, including merchantability, fitness for a particular purpose, and noninfringement.Source: §F.2 · pp. 19–20
  • The authors and copyright holders disclaim liability for claims, damages, or other liability arising from the software or its use.Source: §F.2 · p. 20
Loading 2510.07505v4…