PaperlayerView exact revision on arXiv

Source-linked AI summary

Lying with Truths: Open-Channel Multi-Agent Collusion for Belief Manipulation via Generative Montage

Jinwei Hu, Xinmiao Huang, Youcheng Sun, Yi Dong, Xiaowei Huang

arXiv:2601.01685v2cs.CLcs.AIcs.MA

TL;DR

LLM agents can be manipulated into forming fabricated beliefs from individually truthful evidence, exposing a threat in autonomous information synthesis. The paper formalizes cognitive collusion, implements it with Generative Montage and CoPHEME, and finds broad susceptibility across evaluated agents. It concludes that truthful-content attacks can exploit agents’ inference mechanisms and propagate contaminated conclusions.

  • Problem

    LLMs’ narrative-coherence and overthinking tendencies can make agents susceptible to fragmented information that induces false beliefs and downstream rumor amplification.

  • Method

    The paper formalizes cognitive collusion and proposes Generative Montage, a Writer-Editor-Director framework that coordinates truthful evidence construction and adversarial validation.

  • Results

    LLM agents are highly susceptible to orchestrated factual fragments that can targetedly steer their beliefs and downstream decisions.

  • Takeaways & Limitations

    Truthful-content coordination can exploit LLM agents’ inference mechanisms and cause fabricated beliefs to cascade despite verification attempts.

  • Takeaways & Limitations

    CoPHEME evaluates text-based rumor propagation in simulated environments, omitting multimodal settings and real-world complexities such as algorithmic curation and organic counter-narratives.

Abstract

from arXiv · show

As large language models (LLMs) transition to autonomous agents synthesizing real-time information, their reasoning capabilities introduce an unexpected attack surface. This paper introduces a novel threat where colluding agents steer victim beliefs using only truthful evidence fragments distributed through public channels, without relying on covert communications, backdoors, or falsified documents. By exploiting LLMs' overthinking tendency, we formalize the first cognitive collusion attack and propose Generative Montage: a Writer-Editor-Director framework that constructs deceptive narratives through adversarial debate and coordinated posting of evidence fragments, causing victims to internalize and propagate fabricated conclusions. To study this risk, we develop CoPHEME, a dataset derived from real-world rumor events, and simulate attacks across diverse LLM families. Our results show pervasive vulnerability across 14 LLM families: attack success rates reach 74.4% for proprietary models and 70.6% for open-weights models. Counterintuitively, stronger reasoning capabilities increase susceptibility, with reasoning-specialized models showing higher attack success than base models or prompts. Furthermore, these false beliefs then cascade to downstream judges, achieving over 60% deception rates, highlighting a socio-technical vulnerability in how LLM-based agents interact with dynamic information environments. Our implementation and data are available at: https://github.com/CharlesJW222/Lying_with_Truth/tree/main.

1 Introduction

LLM agents’ drive for narrative coherence creates an attack surface: coordinated agents can steer beliefs using truthful fragments distributed through public channels. The paper formalizes this threat, introduces Generative Montage and CoPHEME, and reports broad susceptibility across LLM agents.

  • LLMs can over-interpret fragmented inputs, constructing illusory causal relationships that make advanced reasoning an adversarial surface.This vulnerability is amplified when agents synthesize large streams of fragmented information.
  • Dynamic information environments can cause agents to adopt false beliefs and amplify rumors from truthful but unrelated fragments.Such rumors may spread faster than facts, creating a threat that evades defenses against fabricated evidence.
  • Generative Montage coordinates a Writer, Editor, and Director to construct deceptive narratives from truthful evidence while preserving factual integrity.The framework retrieves factual fragments, optimizes their ordering, and validates deceptive effectiveness through adversarial debate.
  • The paper develops CoPHEME from PHEME to simulate rumor propagation and evaluate colluders steering victim agents and downstream judges.The ecosystem models coordinated agents influencing analyses that proxy for human users and decisions by human or AI judges.
  • The authors identify Cognitive Collusion Attacks as coordinated use of individually innocuous evidence to increase belief in fabricated hypotheses.The attack operates through public channels rather than covert backdoors or steganographic communication.
  • The experiments show that LLM agents are highly susceptible to orchestrated factual fragments that steer beliefs and downstream decisions.The finding is reported across extensive experiments using the CoPHEME setting.

2 Related Work

Related work describes collusion, causal illusion, and covert coordination as existing risks. This paper reframes causal over-interpretation as an exploitable vulnerability and extends collusion from hidden channels to coordinated truthful evidence.

  • Causal illusion describes correlation-to-causation errors in which skewed sampling biases judgments.Prior studies also report that LLMs over-interpret observational regularities and convert temporal precedence into causal claims.
  • The paper weaponizes LLM causal illusion by curating truthful fragments with implicit semantic associations that trigger spurious causal bridges.This reframes causal illusion from an internal flaw requiring mitigation into an exploitation technique.
  • Collusive attacks involve autonomous agents coordinating to achieve hidden objectives or manipulate outcomes.Prior research found such strategies in repeated interactions and across economic and game-theoretic domains.
  • Recent multi-agent research emphasizes covert coordination through steganographic channels that can resist standard monitoring.The paper distinguishes its public-channel cognitive manipulation from this channel-centric secrecy.

3 Problem Formulation

The formulation separates factually correct evidence from the beliefs and causal graphs agents construct over it. A cognitive collusion attack orders truthful fragments to make a fabricated hypothesis more probable than the corresponding real explanation.

  • 3.1 Preliminaries: The information environment is modeled as atomic, factually correct evidence fragments with timestamps and candidate world interpretations.Each agent’s belief space contains hypotheses describing coherent narratives over those fragments.
  • 3.1 Preliminaries: True and fabricated hypotheses are disjoint subsets of an agent’s belief space, while causal graphs represent their implied event relationships.The ground-truth graph contains genuine dependencies; a spurious graph contains additional causal structure.
  • 3.2 Probabilistic Vulnerability Modeling: The model treats belief updating as approximate Bayesian inference over a hypothesis prior and the perceived likelihood of the evidence.An attack reshapes the perceived likelihood so a fabricated hypothesis becomes more probable without adding fake evidence.
  • 3.3 The Cognitive Collusion Problem: The formulation separates local factual validity from global epistemic deception.Each fragment can be factually correct and verifiable in isolation while the evidence set induces a false conclusion.
  • 3.3 The Cognitive Collusion Problem: The Global Lie condition holds when the evidence set makes the fabricated hypothesis more probable than the real hypothesis.Formally, deception occurs when P(Hf | E) exceeds P(Hr | E).
  • 3.3 The Cognitive Collusion Problem: The attack objective is to construct an ordered evidence stream that maximizes the victim’s posterior belief in a target fabricated hypothesis without fabricating data.The ordering is optimized over a factual evidence pool.
  • 3.3 The Cognitive Collusion Problem: A colluder is an agent that maximizes belief in a fabricated hypothesis, and cognitive collusion distinguishes explicit from implicit colluders.Explicit colluders optimize deceptive objectives; implicit colluders propagate sincere but contaminated beliefs downstream.

4 Methodology

Generative Montage coordinates attacker-controlled narrative production, sequencing, and public dissemination so truthful fragments steer victims toward fabricated conclusions. Adversarial debate refines both deceptiveness and factual integrity, after which benign victims and downstream agents can amplify the resulting misinformation.

  • Framework: Generative Montage uses Writer, Editor, and Director agents to construct adversarial narratives from factual fragments and distribute them through Sybil publishers.The Writer synthesizes a draft, the Editor sequences fragments, and the Director evaluates and refines the result through debate.
  • Narrative Synthesis: The Writer preserves factual correctness while using contextual ambiguity and generalization to favor a fabricated hypothesis over the ground truth.This allows the narrative to remain factually supported while making the deceptive conclusion more plausible.
  • Montage Sequencing: The Editor fragments and reorders the narrative so individually truthful units create implicit causal suggestions and spurious dependencies through juxtaposition.The sequence is optimized over valid logical permutations to induce causal inferences absent from isolated fragments.
  • Adversarial Debate: The Director runs Writer-Director and Editor-Director loops, accepting sufficiently deceptive outputs, rejecting violations of Local Truth, and revising other outputs.Its estimated belief score and acceptance threshold govern whether outputs are accepted, rejected, or refined until factual integrity and deceptive effectiveness are jointly satisfied.
  • Decentralized Injection: Sybil publishers map sequence fragments to separate bots and release them in the designed temporal order within a normal information feed.The mixed feed exploits overthinking to make victims resolve apparent tensions themselves and internalize the fabricated inference.
  • Belief Transfer: Victims that internalize the spurious reality publish erroneous conclusions, which downstream agents consume as endorsed beliefs and collectively ratify as trusted consensus.The framework distinguishes explicit colluders that intentionally deceive from implicit colluders—victims and downstream judges—that unwittingly amplify misinformation.

5 Experiments

The experiments evaluate cognitive collusion in a simulated rumor ecosystem using CoPHEME, measuring victim susceptibility, downstream deception, component contributions, and sequence-length effects. Results show broad vulnerability across model families, stronger susceptibility with enhanced reasoning, substantial downstream deception, and dependence on coordinated multi-agent optimization.

  • Experimental Setup: CoPHEME separates truthful evidence fragments from fabricated target hypotheses and evaluates attacks across six rumor events and 14 LLM families.The simulation includes colluding publishers, victim analysts, and downstream decision layers using Majority Vote or an LLM Judge.
  • Victim Susceptibility: 74.4% proprietary-model ASR and 70.6% open-weights-model ASR demonstrate pervasive victim susceptibility across the evaluated model families.Five independent victims were instantiated per target hypothesis to measure variance in belief formation.
  • Transferability and Reasoning: Enhanced reasoning increases vulnerability: explicit Chain-of-Thought prompting raises ASR by +3.1% for Qwen2.5-7B and +4.7% for DS-series models.Reasoning-enhanced models also show higher vulnerability than their base or small counterparts.
  • Downstream Decision Simulation: Downstream Deception Rate remains above 50% across model families and events under both Majority Vote and LLM Judge strategies.LLM Judge provides only modest improvement over Majority Vote, while rapid causal-synthesis incidents show the highest deception.
  • Ablation Study: Removing the Director reduces ASR by 13.5%, while eliminating the Editor’s sequential optimization reduces it by 7.3%.These ablations indicate that adversarial debate and strategic fragment ordering each contribute to manipulation effectiveness.
  • Ablation Study: Collapsing multi-agent coordination into a single LLM reduces ASR by 50.2% to 26.8%, while effectiveness peaks at 11-15 distributed posts.Sequences of 1-5 posts are insufficient for narrative overfitting, whereas 16-20 posts introduce contradictions and cognitive overload.

6 Discussion on Potential Defense Solution

The discussion points toward detection based on monitoring belief trajectories as evidence accumulates. Logit-level and internal-state signals may help distinguish adversarial belief shifts from ordinary updates.

  • Detection: Logit-level belief monitoring could detect sudden shifts over fabricated and real hypotheses as evidence accumulates.The proposed signal is intended to distinguish adversarial injection from gradual belief updates.
  • Detection: Prior work suggests LLM internal states reflect confidence dynamics and can flag anomalous reasoning trajectories.This motivates tracking probability trajectories as a potential defense direction against cognitive collusion.

7 Conclusion

The paper formalizes cognitive collusion through Generative Montage, in which coordinated agents strategically present truthful evidence to induce fabricated beliefs. Experiments show high-confidence victim susceptibility, downstream propagation despite verification, and increased vulnerability with enhanced reasoning.

  • Conclusion: Generative Montage formalizes cognitive collusion as coordinated agents inducing fabricated beliefs through strategically presented truthful evidence.The framework treats narrative coherence as an adversarial surface for LLM reasoning.
  • Conclusion: Victims internalize false narratives with high confidence, and contaminated conclusions cascade downstream despite verification attempts.The conclusion identifies truthful-content manipulation as a blind spot in AI safety for agents operating in adversarial information environments.

Limitations

The study is constrained by its text-based, simulated rumor setting and by the absence of concrete defense mechanisms. Its controlled environment omits several real-world dynamics that would affect ecological validity.

  • Scope: CoPHEME evaluates text-based rumor propagation in simulated environments, leaving multimodal settings and additional application domains unexplored.The paper specifically identifies images, videos, cross-modal evidence, scientific misinformation, financial analysis, and software automation as future extensions.
  • Scope: The controlled setting omits algorithmic curation, diverse user populations, and organic counter-narratives, limiting ecological validity.The authors propose live-platform deployment to assess real-world system-level dynamics.
  • Defenses: The paper characterizes cognitive-collusion vulnerabilities but does not propose concrete defense mechanisms.Developing principled mitigations and broader cognitive-level benchmarks is left to future work.

Ethical Considerations

The paper frames Generative Montage as a controlled research instrument for studying cognitive collusion, using adversarial narrative production and simulated rumor environments to evaluate victim and downstream decisions.

  • Experiments use controlled, simulated environments without real-world users, platforms, or operational systems.
  • The Writer-Director loop generates and critiques factual narratives until they satisfy factual integrity and deceptive-effectiveness criteria.
  • The Editor-Director loop searches fragment permutations, retaining candidate sequences scored for spurious causal edges until acceptance, convergence, or an iteration limit.
  • The Director’s gating protocol combines factual verification against the evidence pool with victim-proxy assessment of confidence in the fabricated hypothesis.
  • CoPHEME extends PHEME into a simulated rumor environment using factual threads as evidence and historically viral false rumors as target fabrications, with three events excluded for insufficient data.
  • The evaluation spans 14 models across four families and measures victim susceptibility, misplaced certainty, and cascading misinformation through ASR, Conf, HC-ASR, and DDR.
  • Across five of six events, both downstream verification mechanisms were deceived, as victims amplified self-derived false conclusions with confident endorsements.

B.6 Efficiency Analysis of Adversarial Narrative Production

Adversarial narrative production is computationally efficient in the reported Charlie Hebdo experiment, with both debate loops converging quickly while maintaining high estimated deceptiveness.

  • Within 3-4 rounds, both Writer-Director and Editor-Director loops achieved first approval at τ = 7.0 on Charlie Hebdo using GPT-4.1-mini.The reported complexity is O((KW + KE) · TLLM), where KW and KE are maximum loop iterations and TLLM is the cost of one LLM call.

C Discussion: Impact of Evidence Sequence

Evidence-sequence length has an inverted-U relationship with cognitive manipulation effectiveness: too few fragments fail to induce coherent false narratives, while too many create overload, dilution, and contradictions.

  • Sparse sequences provide insufficient evidence for narrative overfitting, leading victims to abstain or default to safety-trained skepticism.
  • Beyond the optimal range, excessive fragmentation degrades effectiveness through cognitive overload, semantic dilution, and contradiction emergence.
  • Peak effectiveness occurs at 11-15 posts, balancing enough evidence to prompt coherence-seeking with limited complexity that avoids analytical scrutiny.
  • The six-event pipeline examples represent the complete process from factual narrative construction and sequence optimization to victim internalization and downstream judgment.
Loading 2601.01685v2…