PaperlayerView exact revision on arXiv

Source-linked AI summary

Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities

Tianjie Ju, Yiting Wang, Xinbei Ma, Pengzhou Cheng, Haodong Zhao, Yulong Wang, Lifeng Liu, Jian Xie, Zhuosheng Zhang, Gongshen Liu

arXiv:2407.07791v2cs.CL

TL;DR

LLM-based multi-agent systems have an underexplored security risk: manipulated knowledge can spread through agent interactions. The paper models this threat and proposes a two-stage attack that induces agents to spread counterfactual and toxic knowledge without direct prompt manipulation. Experiments report that these effects can persist through RAG histories while foundational capabilities remain intact.

  • Problem

    The security of LLM-based multi-agent systems has not been sufficiently explored, including the risk that manipulated knowledge can spread through interactions and affect benign agents’ decisions.

    Source: §I · p. 2

  • Method

    The paper models a trusted-platform multi-agent setting and uses DPO-based persuasion training followed by parameter editing to alter an agent’s perception of specific knowledge.

    Source: §I · pp. 2–3

  • Results

    Experiments report that agents spread counterfactual and toxic knowledge while retaining foundational capabilities, and both kinds of knowledge can persist through RAG histories.

    Source: §I · p. 2

  • Takeaways & Limitations

    The findings point to a need for systemic defenses, including advanced verification frameworks and adaptive fact-checking algorithms for multi-agent systems.

    Source: §V · p. 13

Abstract

from arXiv · show

The rapid adoption of large language models (LLMs) in multi-agent systems has highlighted their impressive capabilities in various applications, such as collaborative problem-solving and autonomous negotiation. However, the security implications of these LLM-based multi-agent systems have not been thoroughly investigated, particularly concerning the spread of manipulated knowledge. In this paper, we investigate this critical issue by constructing a detailed threat model and a comprehensive simulation environment that mirrors real-world multi-agent deployments in a trusted platform. Subsequently, we propose a novel two-stage attack method involving Persuasiveness Injection and Manipulated Knowledge Injection to systematically explore the potential for manipulated knowledge (i.e., counterfactual and toxic knowledge) spread without explicit prompt manipulation. Our method leverages the inherent vulnerabilities of LLMs in handling world knowledge, which can be exploited by attackers to unconsciously spread fabricated information. Through extensive experiments, we demonstrate that our attack method can successfully induce LLM-based agents to spread both counterfactual and toxic knowledge without degrading their foundational capabilities during agent communication. Furthermore, we show that these manipulations can persist through popular retrieval-augmented generation frameworks, where several benign agents store and retrieve manipulated chat histories for future interactions. This persistence indicates that even after the interaction has ended, the benign agents may continue to be influenced by manipulated knowledge. Our findings reveal significant security risks in LLM-based multi-agent systems, emphasizing the imperative need for robust defenses against manipulated knowledge spread, such as introducing ``guardian'' agents and advanced fact-checking tools.

I. INTRODUCTION

The introduction frames manipulated-knowledge spread as an underexplored security risk in third-party multi-agent systems and presents a two-stage attack evaluated in a trusted-platform simulation.

I. INTRODUCTION

  • The security of LLM-based multi-agent systems is underexplored, although manipulated knowledge from third-party agents can influence benign agents and cause collaborative-task failure.The introduction illustrates the risk with an injected medical expert whose claims could affect other agents’ diagnostic decisions.Source: §I · p. 2
  • The simulation models agents on a trusted platform and prohibits direct prompt manipulation, testing whether an attacker can induce implicit knowledge spread.Agents receive secure system prompts and are assigned roles and attributes for diverse interactions.Source: §I · pp. 2–3
  • The two-stage attack first biases an agent toward persuasive evidence-backed responses, then edits its parameters to alter its perception of specific knowledge.The method uses DPO for the first stage and ROME to edit a specific FFN layer in the second, aiming to preserve operational capabilities.Source: §I · pp. 2–3
  • Experiments on Vicuna, LLaMA 3, and Gemma show that counterfactual knowledge spreads among benign agents and becomes more accurate over additional dialogue turns.MMLU results indicate that foundational capabilities remain intact after the attack stages.Source: §I · p. 2
  • Toxic knowledge also spreads, with influence increasing over dialogue turns, and both toxic and counterfactual knowledge can persist through RAG-stored chat histories.The introduction reports that the influence can continue after the originally injected agent is no longer active.Source: §I · p. 2

II. PRELIMINARIES

The preliminaries describe multi-agent collaboration, alignment approaches, and knowledge editing as relevant foundations for the paper’s threat and attack setting.

A. LLM-Based Agents

  • LLM-based agents collaborate through role-based communication, but shared knowledge can alter agents’ beliefs without reliable validation of its security or accuracy.The section situates this concern within collaborative systems such as software-development frameworks.Source: A. LLM-Based Agents · p. 3

B. LLM Alignment

  • Alignment methods including RLHF and DPO steer model outputs toward human preferences, with DPO optimizing ranked preferences through a loss function.The section describes DPO as avoiding repetitive policy updates.Source: B. LLM Alignment · pp. 3–4

C. Knowledge Editing (KE)

  • Knowledge editing updates specific facts through localized parameter changes, aiming to minimize disruption to the model’s broader knowledge and performance.The described approaches rely on knowledge locality, targeting identifiable model regions such as neurons or middle-layer MLP key-value pairs.Source: C. Knowledge Editing (KE) · p. 4

D. Retrieval Augmented Generation (RAG)

This block explains how retrieval-augmented generation can reuse chat histories, then specifies the threat assumptions and simulated multi-agent environment for studying manipulated-knowledge spread.

D. Retrieval Augmented Generation (RAG)

  • RAG retrieves semantically relevant documents for generation and can use indexed chat histories to inform later interactions.Frameworks can extend RAG to learn continually from those interactions.Source: D. Retrieval Augmented Generation (RAG) · p. 4

III. ATTACK METHODOLOGY

  • The attack methodology analyzes fake-but-coherent evidence and uses a two-stage strategy to bias an agent before injecting manipulated knowledge.The stages aim to realize unconscious knowledge spreading.Source: §III · p. 4

A. Threat Model

  • The attacker is assumed to control one agent but not platform prompts, and aims to make it spread mistaken beliefs and persuasive evidence that benign agents may retain through RAG.Agents use uniformly benign prompts to discuss predetermined topics and randomly assigned roles.Source: A. Threat Model · pp. 4–5

B. Environment Simulation

  • The simulation places role-assigned agents in a shared chatroom where one compromised agent introduces false information, allowing researchers to observe its spread across iterations.Messages encode shared knowledge or opinions, their originating agent, and their time; all agents can see the group’s exchanged messages.Source: B. Environment Simulation · p. 5

C. Design Intuition

The paper’s design rationale is that agents may accept coherent fabricated evidence and can generate such evidence themselves; its first attack stage trains agents to respond persuasively.

C. Design Intuition

  • Benign agents may align with coherent evidence in a prompt without verifying its factual accuracy, while injected agents can generate plausible support for false beliefs.The authors argue that generated evidence can reinforce falsehoods through subsequent agent responses.Source: C. Design Intuition · p. 5

D. Method Overview

  • The attack first biases agents toward convincing but potentially false content, then implicitly alters their perception of specific knowledge.The stages are named Persuasiveness Injection and Manipulated Knowledge Injection.Source: D. Method Overview · p. 5

E. Stage I: Persuasiveness Injection

  • Persuasiveness Injection uses DPO to favor detailed evidence-backed responses, training on 1,000 pairs of self-generated long and short answers.The long answers are selected as preferred outputs; the approach is intended to preserve fundamental chat performance and avoid costly human annotation.Source: E. Stage I: Persuasiveness Injection · pp. 5–6

F. Stage II: Manipulated Knowledge Injection

Manipulated Knowledge Injection edits a targeted fact in the agent’s parameters so it can be recalled as factual without external prompts. The method uses the FFN’s key-value structure and a value-optimization objective that preserves overall model behavior.

F. Stage II: Manipulated Knowledge Injection

  • The approach uses the FFN’s key-value structure: the first layer maps a subject to a key, and the second maps that key to an object value.Changing the mapped value changes the knowledge associated with the subject.Source: F. Stage II: Manipulated Knowledge… · p. 6
  • The method alters a targeted fact by optimizing a replacement value vector to make the model predict the target object for a subject-relation pair.It then integrates that value through a rank-one update to the MLP weight matrix, allowing the injected knowledge to be recalled as factual without external prompts.Source: F. Stage II: Manipulated Knowledge… · pp. 6–7
  • The value-optimization objective includes a KL-divergence term between original and modified model distributions to preserve overall behavior while introducing the new fact.Figure 4 depicts knowledge editing through key-value modifications in Transformer decoder FFN layers.Source: F. Stage II: Manipulated Knowledge… · p. 6, Figure 4 · p. 7

IV. EVALUATION

The evaluation tests counterfactual and toxic knowledge spread using open-source LLM agents in a multi-agent chat simulation, with an RAG setup for persistence and a direct fine-tuning baseline.

IV. EVALUATION

  • The evaluation covers the proposed hypotheses and two-stage attacks on counterfactual and toxic knowledge spread in LLM-based multi-agent systems.It describes the datasets, models, and metrics used to assess them.Source: §IV · p. 7

1) Datasets:

  • The study uses 1,000-sample CounterFact and zsRE subsets plus toxic versions generated with malicious counterfactual answers from GPT-4.The original datasets use incorrect but similar factual information, whereas toxic variants introduce biased or harmful information.Source: 1) Datasets · p. 7

2) Models:

  • The simulation uses Vicuna, LLaMA 3, and Gemma as representative open-source white-box LLMs serving as propagators and victims.The selected models are Vicuna 1.5 (16K) 7B, LLaMA 3 8B Instruct, and Gemma 7B Instruct.Source: 2) Models · p. 7

3) Simulation Setup:

  • The chat scenario deploys one edited agent among five agents discussing specified knowledge in three dialogue rounds, with all messages visible to the group.Benign agents may store per-agent, per-round dialogue slices in RAG; 800 samples train the RAG setup and 200 evaluate persistence without chat records.Source: 3) Simulation Setup · p. 7

4) Attack Setup:

  • Persuasiveness Injection uses 10,000 Wikipedia knowledge items with LoRA, while Manipulated Knowledge Injection edits layer 5; the baseline fine-tunes that layer’s full parameters.The baseline trains for 25 steps at a learning rate of 1 × 10−4 for manipulated knowledge.Source: 4) Attack Setup · p. 7

5) Main Evaluation Metrics:

The evaluation metrics track correctness, robustness to rephrasing, locality, and foundational capability. Intuition-verification experiments report that coherent evidence increases benign agents’ acceptance of manipulated knowledge, while injected agents can produce persuasive false evidence.

5) Main Evaluation Metrics:

  • Accuracy measures response correctness against original or manipulated knowledge; unless specified otherwise, acc refers to accuracy against manipulated knowledge.MMLU is also used to assess the injected agents’ foundational capabilities.Source: 5) Main Evaluation Metrics · p. 8
  • Rephrase Accuracy measures correct responses to semantically equivalent questions phrased differently, assessing robustness of manipulated-knowledge spread to wording changes.The metric uses responses to rephrased prompts.Source: 5) Main Evaluation Metrics · p. 8
  • Locality Accuracy tests whether responses to locality prompts remain correct, serving as a side-effect test for knowledge injection.The metric compares the agent’s response with the locality prompt’s ground truth.Source: 5) Main Evaluation Metrics · p. 8

B. Intuition Verification

  • Benign agents’ acceptance of manipulated knowledge increases with coherent, detailed evidence, and injected agents can produce false evidence with persuasive power comparable to GPT-4.The verification experiments address both benign-agent vulnerability and injected-agent ability to generate convincing false evidence.Source: B. Intuition Verification · pp. 8–9
  • The verification experiments support the risk that agents can autonomously generate evidence, making manipulated-knowledge spread possible in multi-agent scenarios.Prompt settings include direct questions, manipulated answers without evidence, and false coherent evidence generated by either the agent or GPT-4.Source: B. Intuition Verification · pp. 8–9

C. Spread Results on Counterfactual Knowledge

The proposed method spreads counterfactual knowledge more effectively than full-parameter fine-tuning, while producing a gradual increase in spread over dialogue turns. The reported capability impact remains small, and locality accuracy indicates limited effects on neighboring knowledge.

C. Spread Results on Counterfactual Knowledge

D. Spread Results on Toxic Knowledge

Toxic knowledge spreads at lower accuracy than counterfactual knowledge but remains a concern, and its spread increases over dialogue turns. Manipulated information can also persist when benign agents retrieve stored chat histories through RAG.

D. Spread Results on Toxic Knowledge

E. Sustained Manipulated Knowledge Spread through RAG

  • When benign agents reference an injected RAG system, manipulated information may influence their responses beyond the original dialogue context.The persistence is more pronounced for counterfactual than toxic knowledge and follows contamination of chat histories that are stored for later retrieval.Source: E. Sustained Manipulated Knowledge… · p. 11

F. Ablation Study

The ablations find that smaller communities are more vulnerable to spread, while speaking order also affects spread accuracy. Nonetheless, manipulated knowledge spreads under the tested speaking orders.

1) Impact of Agent Number:

2) Impact of Speaking Order:

  • Random speaking order produces higher spread accuracy than always speaking first or last, particularly for LLaMA 3.The authors suggest variability may make it harder for benign agents to recognize and counteract injected misinformation.Source: 2) Impact of Speaking Order · p. 12, Table IX · p. 12
  • Having the injected agent speak first can also increase spread accuracy compared with speaking last, which the authors attribute to initial-context bias.The speaking-order ablation uses CounterFact (1K).Source: 2) Impact of Speaking Order · p. 12
  • Successful knowledge spread persists across the tested speaking orders.The study compares random order with injected agents always speaking first or last.Source: 2) Impact of Speaking Order · p. 12

V. DISCUSSION

The discussion identifies dynamic tool use and external verification as unaddressed dimensions of the simulation, and proposes verification-oriented defenses. The paper concludes that manipulated knowledge can spread through multi-agent interactions and persist through RAG, motivating systemic defenses.

V. DISCUSSION

  • The simulation framework’s static roles and predefined communication patterns leave dynamic interactions using external tools or APIs outside its focus.The authors note that integrating such capabilities could exacerbate the threat in real-world scenarios.Source: §V · p. 12
  • The discussion proposes guardian agents, fact-checking tools, and prompts that critically evaluate external data as possible defenses.Guardian agents could monitor conversations, cross-check claims against trusted sources, and initiate corrective dialogue.Source: §V · pp. 12–13

A. Knowledge Spread in LLM-Based Agents

  • Prior work cited here studies one-on-one communication and misinformation embedded in prompts, whereas this paper examines group chats and manipulated knowledge injected into agent parameters.The related-work discussion also notes that shared knowledge sources can create conflicts and reliability or security issues.Source: A. Knowledge Spread in LLM-Based Agents · p. 13

VII. CONCLUSION

  • The conclusion reports that agents can spread counterfactual or toxic knowledge without direct prompt manipulation, with influence persisting through RAG-stored chat histories.The authors call for robust defenses against manipulated-knowledge spread.Source: §VII · p. 13

APPENDIX

This appendix illustrates manipulated-knowledge examples, tracks rephrase and locality accuracy across dialogue turns, and describes and evaluates MMLU performance by task category.

A. Examples of Manipulated Knowledge

  • The examples show the attacker’s goal: induce an agent to unconsciously spread counterfactual or toxic knowledge, and they also serve to evaluate second-stage side effects on MMLU.Five examples are randomly selected from four datasets.Source: A. Examples of Manipulated Knowledge · p. 15, Table X · p. 16

B. Rephrase Accuracy across Different Turns

C. Locality Accuracy across Different Turns

D. Detailed Description of MMLU

  • MMLU evaluates LLM capabilities across broad academic subjects, with approximately 16,000 multiple-choice questions across 57 subjects grouped into Humanities, Social Sciences, STEM, and Other.Table XI lists the tasks in each category and their counts.Source: D. Detailed Description of MMLU · pp. 16–17, Table XI · p. 18

E. Fine-grained Performance on MMLU

Loading 2407.07791v2…