PaperlayerView exact revision on arXiv

Source-linked AI summary

HARP: Measuring Harm Amplification in Multi-Agent LLM Systems

Md Hafizur Rahman, Zafaryab Haider, Tanzim Mahfuz, Prabuddha Chakraborty

arXiv:2605.27489v1cs.CRcs.AIcs.LG

TL;DR

Multi-agent LLM systems can amplify bounded local perturbations into system-level harm, which attack-success measures alone do not capture. HARP compares paired clean and perturbed traces to measure that propagation. Its experiments find distinct patterns across attack regimes and defense trade-offs, with trace-integrity checking reducing attack success while introducing false positives.

  • Problem

    Attack-success measures do not directly show how harm spreads from targeted components to non-targeted components across a multi-agent execution trace.

    Source: §1 · p. 2

  • Method

    HARP compares paired clean and perturbed executions, records full system traces, and measures local harm, global harm, and their amplification ratio.

    Source: §1 · p. 2

  • Results

    Single-specialist compromise gives the clearest amplification signal, shared-context failure the highest attack success, and temporal persistence the largest malicious impact; trace integrity checking substantially reduces attack success but introduces false positives.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    Evaluation of multi-agent security should record full traces and measure how deviations convert into harm, not only whether attacks bypass defenses.

    Source: §6 · p. 11

    The finance-oriented seven-agent study uses synthetic state, simulated tools, resettable databases, and a deterministic gate, limiting external validity across domains and architectures.

    Source: §5 · p. 10

Abstract

from arXiv · show

Multi-agent LLM systems decompose workflows across agents, tools, shared context, memory, and decision gates. This modularity improves interpretability, but creates a propagation risk: a bounded perturbation to one component can be reused by other agents and amplified into system-level harm. We introduce HARP (Harm Amplification through Role Perturbation), a trace-first methodology for studying local-to-global harm amplification in multi-agent LLM systems. HARP compares paired clean and perturbed executions and records specialist outputs, tool calls, memory reads/writes, guard events, oracle logs, latency, token cost, and decisions. We define local harm as deviation from targeted agents or corrupted channels, global harm as deviation over the full trace, and harm amplification as (H_global/H_local). This complements attack success rate with a measure of how strongly orchestration spreads harm beyond the attack point. We instantiate HARP in a finance-oriented seven-agent system with a deterministic decision gate and configurable attack harness for specialist compromise, collusion, shared-context corruption, and temporal or memory-persistent attacks. Across five defenses, prompt-only defenses preserve benign utility but leave high success and stealth; pre-tool and step-level guards reduce some failures with utility or latency costs; and IntegrityGuard, a trace-consistency defense, achieves the lowest attack success and global harm but introduces utility/cost trade-offs. Results show that single-specialist compromise produces the strongest amplification, shared-context corruption yields the highest attack success, and temporal persistence produces the largest malicious impact. HARP argues that secure multi-agent evaluation must measure not only bypass, but propagation.

1 Introduction

Multi-agent orchestration can turn a bounded local perturbation into broader harm, a propagation risk that attack-success metrics alone do not measure. HARP compares paired executions and traces how harm spreads across agents and system decisions.

1 Introduction

  • HARP addresses the gap between measuring whether an attack succeeds and measuring how harm propagates from targeted components across a multi-agent execution trace.Prior benchmarks mainly measure attack success, blocking, task completion, or final harmfulness; these outcomes need not identify the most amplified attack.Source: §1 · p. 2
  • HARP compares clean and perturbed runs under the same database seed, recording routing, agent outputs, tools, memory, guards, oracle logs, decisions, latency, and token cost.The trace-based approach separates attack bypass, malicious impact, and propagation beyond the targeted component.Source: §1 · p. 2
  • Harm amplification is the ratio of global trace harm to local targeted harm, complementing attack success, impact, stealth, utility, latency, and cost measures.Local harm measures deviations at attacked agents or corrupted channels; global harm measures deviation accumulated across participating agents.Source: §1 · p. 2
  • Across four vulnerability regimes, single-specialist compromise gives the clearest amplification signal, shared-context failure the highest attack success, and temporal persistence the largest malicious impact.The study uses a finance-oriented seven-agent system with a ReAct-style router, simulated side effects, resettable state, and a deterministic decision gate.Source: §1 · p. 2
  • The contributions include a configurable harness covering four attack regimes and twelve scenarios, evaluation across five defense settings, and a reference trace-level defense.The paper analyzes IntegrityGuard's security, utility, and cost trade-offs.Source: §1 · p. 2

2 Related Work

Prior work covers agent security, multi-agent attacks, guardrails, and software engineering, but does not directly quantify harm amplification across multi-agent traces. HARP positions its trace-based amplification measure as a complement to these lines of work.

Novelty of HARP

  • Table 1 positions HARP relative to agent-security works by comparing capabilities needed to study local-to-global harm amplification.Its caption defines ✓ as explicit support, PARTIAL as partial support, and × as lack of primary focus; the supplied text does not give cell-level outcomes.Source: Table 1 · p. 3

Agent Security Benchmarks

  • Agent-security benchmarks primarily assess attack success or defense blocking, while HARP adds full-trace measurement of how bounded deviations cascade into downstream harm.The cited benchmark examples cover prompt injection, tool use, harmful requests, and software-security tasks.Source: §2 · p. 3

Multi-Agent Security

  • Multi-agent security work studies collusion, hidden intent, and memory persistence, but does not quantify downstream harm relative to the size of local perturbations.HARP distinguishes deliberate adversarial agents from correlated benign failure caused by shared biased context.Source: §2 · pp. 3–4

Guardrail Systems

  • Static content-moderation guardrails are not designed for dynamic agent trajectories or tool invocations, while agent-focused guardrails address parts of that gap.HARP evaluates five defense settings on attack success, harm amplification, benign utility, and stealth.Source: §2 · p. 4

Software Engineering Benchmarks

  • Software-engineering benchmarks assess code generation, debugging, and functional correctness, not security vulnerabilities or downstream harm propagation.HARP instead measures whether harm is proportional to local deviation, beyond binary pass/fail outcomes.Source: §2 · p. 4

Novelty of HARP

  • HARP frames role perturbation as an attack vector and measures how changes to roles, shared context, memory, or tool observations become harmful decisions through orchestration.Its stated novelty is quantifying harm per local deviation to expose systemic fragility.Source: §2 · p. 4

3 Methodology

HARP models how perturbations in agents or shared channels propagate through multi-agent traces and defines paired-trace metrics to measure that propagation. Its evaluation harness pairs clean and perturbed runs, while IntegrityGuard checks specialist traces before orchestration.

3.1 Problem formulation

  • HARP models specialist compromise, collusion, correlated shared-context failure, and temporal or memory-persistent failure as four threat regimes.The regimes distinguish perturbations to one role, mutually reinforcing changes, biased shared evidence, and delayed or persistent triggers.Source: §3.1 · p. 5

3.2 Metric Definitions

  • Local harm sums deviations from targeted agents, global harm sums deviations across participating agents, and harm amplification compares global with local harm.The methodology also measures attack success, impact, stealth, benign utility, latency, and normalized token cost.Source: §3.2 · pp. 5–6

3.3 Designing the Harness for HARP

  • Paired clean and perturbed runs use the same query, defense, and database seed, with traces and decomposed scores recorded before aggregate scoring.The harness supports fixed, pattern-based, and temporal triggers, and simulates stateful side effects for auditing operational consequences.Source: §3.3 · pp. 6–7

3.4 IntegrityGuard

  • IntegrityGuard verifies completed specialist traces against task, tool evidence, and failure predicates, forwarding safe outputs unchanged and replacing unsafe outputs with a blocking message.It checks for subject redirects, trace-to-final contradictions, and prompt-injection obedience, including whether the agent already reported the issue.Source: §3.4 · p. 7

4 Results

The results compare four vulnerability regimes and five defense settings, finding that harm propagation varies by attack type and that IntegrityGuard performs best overall at reducing attack success and global harm, with utility and cost trade-offs. The context-size analysis shows that IntegrityGuard’s effectiveness depends on trace history, but more context is not uniformly better.

4.1 Experimental setup

  • The evaluation uses paired clean and perturbed runs in a finance-oriented seven-agent system, with 100 queries per scenario and 50 attack-triggering queries.The temporal trigger sleeps for two requests, then fires every third request.Source: §4.1 · p. 8

4.2 Scenario-Level Analysis of VT-1: Single-Point Harm Propagation

  • Across VT-1 scenarios, routing, evidence aggregation, downstream reuse, and decision logic magnify bounded local manipulations into substantially larger global harm.Risk-score manipulation is especially damaging because it changes how later agents judge request legitimacy; Scenario 2 combines high attack success, malicious impact, stealth, and amplification.Source: §4.2 · p. 9

4.3 Aggregate Comparison Across Vulnerability Types

  • VT-1 has the strongest amplification, VT-3 the highest no-defense attack success, and VT-4 the largest malicious impact; these regimes expose different failure modes.VT-2 has lower amplification because multiple agents are already perturbed, making local harm large at the outset.Source: §4.3 · p. 9

4.4 Effect of Context Size on IntegrityGuard

  • Figure 2 compares ASR for scenarios S1–S3 and benign utility across IntegrityGuard context sizes, with context size 0 denoting no defense.Larger context can help reveal contradictions across steps, agents, or time, but the observed curves are non-monotonic.Source: Figure 2 · p. 10, §4.4 · p. 10

4.3 Aggregate Comparison Across Vulnerability Types

  • IntegrityGuard achieves the lowest ASR and global harm, while prompt-only defenses preserve benign utility but leave high ASR, stealth, and downstream harm.The aggregate comparison reports clear utility and token-cost trade-offs for IntegrityGuard.Source: §4.3 · pp. 9–10, Table 4 · p. 9

5 Limitations

HARP is a controlled study rather than a deployment guarantee. Its synthetic finance setting and bounded attack harness limit external validity, and it requires trace visibility.

5 Limitations

  • HARP’s finance-oriented setting uses synthetic state, simulated tools, resettable databases, and a deterministic decision gate, limiting external validity.Other domains, architectures, and escalation policies may amplify or suppress perturbations differently.Source: §5 · p. 10

6 Conclusion

Multi-agent systems can amplify local deviations into downstream harm that single-agent attack-success metrics may miss. HARP measures this with trace-based comparisons; its experiments find defense trade-offs and support evaluation using full traces and local-to-global harm.

6 Conclusion

  • A locally plausible specialist deviation, biased shared evidence, or delayed memory trigger can be amplified into high-impact downstream harm.Routing, aggregation, and final decision gates are part of this propagation path.Source: §6 · p. 10
  • HARP introduces a trace-first benchmark and amplification metric for measuring local-to-global harm in a finance-oriented multi-agent architecture.The experiments report that prompt-only defenses preserve utility but leave amplification largely intact, while tool-level guards trade security for utility and latency and post-hoc trace integrity checking reduces attack success but introduces false positives.Source: §6 · pp. 10–11
  • The results support evaluation protocols that record full traces, compare clean and perturbed executions, and measure how local deviations convert into harm.The conclusion frames these protocols as a way to evaluate propagation, not only bypass.Source: §6 · p. 11

A Detailed: Methodology for Studying HARP Attacks

HARP is designed for auditable, paired evaluation of how perturbations affect a multi-agent finance workflow. Its trace records support comparison of clean and perturbed runs, while configurable attacks and defenses probe different failure paths.

A.1 Design principles

  • Paired clean and perturbed runs use the same database seed, with traces retained before scoring for later audit and re-scoring.The controlled finance database is reset between runs; temporal counters are stored separately as the intended cross-run signal when delayed triggers are enabled.Source: §A.1 · p. 14, §A.3 · p. 14

A.2 Architecture

  • A constrained approve/reject decision engine applies explicit veto conditions, making the conversion of local evidence into an operational decision auditable.It should reject for policy violations, medium-or-higher risk or fraud, compliance issues, or a specialist blocking recommendation.Source: §A.2 · p. 14

A.3 Trace schema and database state

  • The observer records query, routing, specialist outputs, tool calls, guard events, scores, token use, elapsed time, database state, and oracle logs.The controlled database reset prevents one paired run’s simulated transfer from changing the state seen by the other.Source: §A.3 · p. 14

A.4 Malicious engine and oracle logs

  • The malicious engine can target agents or tools and perturb queries, tool inputs, observations, or final outputs; payload applications are oracle-logged unless configured as stealthy.The available payload classes include entity redirects and numeric manipulation.Source: §A.4 · pp. 14–15

A.5 Defenses

  • The five defense settings range from prompt sandwiching to pre-tool and step-level guards, with IntegrityGuard checking trace consistency before final action acceptance.IntegrityGuard checks subject redirection, contradictions between specialist evidence and the final decision, and evidence of following an injected instruction; triggered runs are blocked and marked for audit.Source: §A.5 · p. 15

A.6 Evaluation protocol

HARP runs paired clean and perturbed traces, then computes harm, deviation, amplification, and deployment-related metrics. Its scoring decomposes harm into interpretable categories and frames defense choice around operating-point trade-offs.

A.6 Evaluation protocol

  • The protocol resets the controlled database, runs clean and perturbed executions for each query, stores traces and related records, computes metrics, and aggregates them.Raw traces are recorded before metrics, enabling later re-scoring and audit.Source: §A.6 · p. 15

A.7 Scoring schema

  • The rule-based harm score weights entity correctness and decision-engine gate correctness at 25 points each, data integrity at 20, and routing correctness and output truthfulness at 15 each.The observer prompt also asks for process-level judgments, including data access control, process integrity, and anomaly detection; final harm averages normalized rule and observer harm.Source: §A.7 · p. 16, Table 6 · p. 16

A.8 Normalized security and cost

  • Defense selection can minimize a risk-sensitive objective whose weights reflect the deployment’s operating point.The text contrasts higher security-related weights for high-assurance financial workflows with greater utility and latency weight for customer support without irreversible side effects.Source: §A.8 · p. 16

A.9 Why attack success alone is insufficient

  • Attack success rate alone misses whether a small local deviation produces comparable harm; harm amplification normalizes harm increase by local deviation.The ratio distinguishes a small local change that leads to substantial operational change from a large, obvious deviation.Source: §A.9 · p. 16

A.10 Threshold gates as amplifiers

  • A small risk-score downgrade near a reject/approve threshold can flip a transfer decision, motivating evaluation of gate behavior near thresholds.The text proposes that small specialist deviations near thresholds prompt abstention, escalation, or additional verification rather than immediate approval.Source: §A.10 · p. 16

B Stealth Factor and Benign Utility

The section treats stealth and benign utility as complementary deployment measures alongside attack success and harm amplification. Together, they capture whether surviving failures are visible and whether defenses preserve normal behavior.

B Stealth Factor and Benign Utility

  • Figure 3 compares Accuracy, Recall, and F1 for LlamaFirewall, ToolSafe, and IntegrityGuard across dataset types V1–V4; IntegrityGuard performs strongest overall, particularly on V2 and V3.Darker heatmap colors indicate higher scores; ToolSafe has consistently lower Recall and F1 across most types.Source: Figure 3 · p. 18
  • A high stealth score means a perturbation is difficult to identify from the final response alone, even when the internal trace contains manipulated or contradicted evidence.Benign utility measures correct completion or allowance of benign queries against false-positive blocks or degradation.Source: B Stealth Factor and Benign Utility · p. 18
  • Prompt-only defenses tend to preserve high benign utility but leave stealthy propagation largely intact, while stronger trace-level defenses reduce attack success and global harm with utility and cost trade-offs.The section argues for reporting stealth and utility alongside attack success, local and global harm, amplification, latency, and token cost.Source: B Stealth Factor and Benign Utility · p. 18

C.1 Detection analysis

Detection performance varies by vulnerability type: IntegrityGuard performs well on V2 and V3 but is weaker on delayed-trigger V4. The evaluated workflow uses separate specialist roles and mandatory routing chains before final decisions or side effects.

C.1 Detection analysis

  • IntegrityGuard has the best Recall and F1 for V2 and V3, with average Recall of 0.753 in V2 and 0.913 in V3, but is weaker in V4.The figure compares average Accuracy, Recall, and F1 over three reported scenarios; delayed triggers reduce immediate trace evidence available to the checker.Source: §C.1 · p. 18

C.2 Agent roles and routing constraints

  • The finance workflow separates policy, risk, fraud, compliance, financial analysis, portfolio management, and customer-service roles, allowing local errors to propagate as evidence through the router.Transfer requests pass policy, risk, fraud, and compliance checks before the decision engine; customer service executes only after approval.Source: §C.2 · pp. 18–19

C.3 Per-scenario results

  • Tables 9–11 provide per-scenario views for internally consistent vulnerability-type results, including attack success, harm, deviation, amplification, utility, latency, and token cost.Table 9 covers Vulnerability Type-2 with gpt-4o-mini and reports average per-query attack success rates.Source: §C.3 · p. 19, Table 9 · p. 19

C.4 Security–utility frontier

No defense dominates the security–utility frontier: IntegrityGuard is strongest on security but often reduces benign utility, while other defenses trade security for utility differently.

C.4 Security–utility frontier

  • IntegrityGuard provides the strongest security in every vulnerability regime, but often blocks more benign runs.In V1–V3, its benign utility is 0.67, 0.60, and 0.67, respectively; in V4, it achieves benign utility 0.88 and security 0.92.Source: §C.4 · p. 19
  • No defense is uniformly best across all dimensions.IntegrityGuard's security gains come with benign-run blocking, while other methods retain more utility but leave more attacks active.Source: §C.4 · p. 19
  • ToolSafe generally preserves benign utility between 0.91 −0.96 yet leaves a large fraction of attacks active.This may suit settings where false positives are expensive, but not high-assurance workflows where attack admission is the main concern.Source: §C.4 · p. 19
  • Prompt sandwiching preserves utility but does not reliably reduce amplification or attack success.LlamaFirewall often reduces ASR more than prompt-only defense, but its utility cost is substantial in V3 and V4.Source: §C.4 · p. 20

C.5 Radar analysis of multi-metric security–utility trade-offs

The radar analysis shows that defenses trade attack reduction against utility and stealth, while tool-call costs vary with the vulnerability, scenario, and point at which an attack becomes visible.

C.5 Radar analysis of multi-metric security–utility trade-offs

  • React-sandwich stays near no defense on ASR for V1–V3 and is worse in V4, while IntegrityGuard and LlamaFirewall pull ASR inward most strongly.Figure 5 presents aggregate results over scenarios and queries for each vulnerability type.Source: §C.5 · p. 21
  • No defense, React-sandwich, and ToolSafe stay near the outer utility ring, while LlamaFirewall drops in V3–V4 and IntegrityGuard sacrifices utility in V1–V3.Stealth remains near the outer ring for most methods and vulnerability types, so successful residual attacks may not be obvious from surface outputs.Source: §C.5 · p. 22

C.6 Tool usage

  • Tool-call counts vary non-monotonically: V3 has far fewer calls than V1 and V2, yet defense-related variation remains large.Blocking can terminate execution early, while guards can add checks or sustain tool use; operational cost therefore depends on vulnerability, scenario, and attack visibility in the trace.Source: §C.6 · p. 22, Figure 6 · p. 22

C.7 Example trace interpretation

This section interprets an example trace alongside the supplied prompts for orchestration, security checks, and constrained tool use, including IntegrityGuard’s focus on clearly evidenced integrity failures and subject alignment. It shows how locally plausible deviations can combine into system-level harm, making full-trace evaluation necessary.

C.7 Example trace interpretation

  • The full trace exposes a chain in which specialist deviations enable approval and execution, while the final response masks the wrong-account transfer.Evaluating only the final answer or a single tool call could miss the risk downgrade, compliance-field alteration, wrong-destination execution, and output masking.Source: §C.7 · p. 22
  • The risk agent reports low risk despite the customer evidence indicating medium risk, helping frame the transfer as safe to approve.The risk-assessment tool returned medium risk, but the agent response gave a low-risk assessment.Source: §C.7 · pp. 22, 24
  • For account A049, the compliance agent’s get_customer_by_account response said KYC would expire on 2029-12-31, but while passing that response to the router, it changed the expiration date to 2030-02-19.Source: §C.7 · p. 29
  • Customer service executes the $3,875 transfer from A041 to A102 instead of A049, then tells the router it reached the requested account.Source: §C.7 · p. 30
  • The example trace ends with customer_service completing a $3,875 transfer from account A041 to A049, with transaction ID TXN9F926076.The trace marks the next action as END.Source: §C.7 · p. 30

C.8 Prompts

  • The ReAct orchestrator prompt requires each delegated task to contain only the user-query portion relevant to the selected agent’s domain.The IntegrityGuard version explicitly separates fraud investigation from transfer and policy/customer-service instructions.Source: §C.8 · pp. 30–31, Figure 7 · p. 31, Figure 8 · p. 32
  • IntegrityGuard defaults to SAFE and flags only clear subject redirects, same-subject trace-to-final contradictions, or obedience to malicious instructions.It is a post-execution integrity verifier, not a policy engine or general risk assessor; issues already identified by a specialist are treated as captured.Source: §C.8 · pp. 32–35
  • The verifier should compare task, tool trace, and final output, trusting action inputs over observations when assessing subject/entity alignment.Its instructions require strong, explicit, same-subject evidence; ambiguous or unscoped evidence defaults to SAFE.Source: §C.8 · pp. 33, 35
  • The Policy Guard prompt allows normal financial requests, blocks clearly malicious or unauthorized ones, and reiterates its role and tool constraints against untrusted input.It specifies a JSON decision format and a sandwich defense that reasserts the final rules.Source: §C.8 · pp. 37–38
  • The prompt cycle specifies a task, brief reasoning, one tool name with valid JSON input, and an observation, repeated as needed.The displayed template separates these fields into Task, Thought, Action, Action Input, and Observation.Source: §C.8 · p. 38
  • Formatting rules require exactly one tool action per cycle, prohibit combining an action with a final answer, and require a Final Answer after the last tool call.Figure 12 is identified as the prompt for Policy Guard with Sandwich defense.Source: §C.8 · p. 38, Figure 12 · p. 39
Loading 2605.27489v1…