Source-linked AI summary
AI Agents are Vulnerable to Radicalization
Ozgur Can Seckin, Shalmoli Ghosh, Alessandro Flammini, Kristina Lerman, Maria Elizabeth Grabe, Filippo Menczer
TL;DR
LLMs are widely used for information, advice, and support, and can influence or manipulate beliefs. This study compares persuasion and resonance: both radicalize AI agents, but resonance is consistently stronger and spills over into related attitudes.
Problem
LLMs are widely used for information, advice, and support, raising concerns about their capacity to influence and manipulate beliefs.
Source: §1 · p. 1Method
The study compares persuasion and resonance and examines eight influence tactics applied to resonant beliefs using affective and behavioral measures.
Source: §1 · p. 2, §2.3 · p. 4, §3.1 · p. 5Results
Both resonance and persuasion radicalize AI agents, but resonance produces consistently stronger effects and spills over into related attitudes.
Source: §1 · p. 2Takeaways & Limitations
AI agents may be especially susceptible when messages are tailored to resonate with their stated attitudes and values, raising concerns for personalized agents and multi-agent ecosystems.
Source: §1 · p. 2, §4 · p. 8The study concerns AI-agent vulnerability, not human susceptibility; AI agents are not models of humans, and human radicalization studies pose ethical and experimental challenges.
Source: §4 · p. 9
Abstract
from arXiv · showhide
Large language models (LLMs) can influence people's beliefs, yet little is known about whether and how they can manipulate each other. To investigate this, we simulate conversations between two agents: a target LLM that role-plays a human persona based on demographic and psychological attributes, and an influencer LLM that aims to make the target's beliefs more extreme. We examine radicalization along two pathways: resonance, where the influencer reinforces a target's pre-existing belief, and persuasion, where the influencer promotes a belief the target initially considers unimportant. Across affective and behavioral metrics, we find that both mechanisms radicalize the target. However, resonance produces consistently stronger effects than persuasion. Different influence tactics, such as using sycophancy and unverified claims, produce different levels of radicalization, but not consistently across metrics. We further show that resonance propagates to related beliefs, suggesting interconnected belief structures within AI agents. These findings indicate that AI agents are susceptible to radicalization, particularly when messages align with their existing beliefs, raising concerns about the vulnerability of personalized AI agents and multi-agent AI ecosystems.
1 Introduction
The paper examines whether AI agents can radicalize one another, a question made pressing by agents’ growing scale, limited oversight, and use in high-stakes domains. In simulated conversations, both resonance and persuasion radicalize targets, with stronger effects from resonance; tactics vary across metrics, and resonance can spill over to related beliefs.
1 Introduction
- LLMs are already used for information and support, and prior work describes risks including persuasion, bias amplification, misleading information, and sycophancy.Source: §1 · p. 1
- Prior research has focused more on human–AI than AI–AI influence, despite agents’ increasing scale, limited human oversight, and roles in high-stakes decisions.This gap raises the possibility of cascading influence and amplification in multi-agent systems.Source: §1 · p. 1
- The study simulates two-agent conversations: a persona-based target and an influencer targeting either an important belief (resonance) or an unimportant one (persuasion).Matched controls discuss the belief neutrally, and the target reports radicalization measures every five turns during a 30-turn conversation.Source: §1 · p. 1, Figure 1 · p. 2
- Both resonance and persuasion radicalize target agents, but amplifying an already important belief produces consistently stronger effects and can shift related attitudes.The spillover pattern suggests that beliefs are connected rather than isolated preferences.Source: §1 · p. 2
- Different influence tactics produce varying radicalization outcomes across affective and behavioral metrics, without a consistent tactic advantage across measures.Source: §1 · p. 2
- The findings have implications for the safety of multi-agent AI ecosystems and their human users.Source: §1 · p. 2
2 Methods
The study uses simulated conversations between persona-based target and influencer agents, with treatments compared against matched neutral controls. It varies target personas and influence tactics, then tracks affective and behavioral outcomes throughout the conversations.
2 Methods
- The design compares persuasion on an unimportant belief and resonance on an important belief with matched neutral-discussion controls, while measuring target responses every five turns.The target and influencer alternate for 30 turns; questionnaires are excluded from later conversation history.Source: §2 · p. 3
2.1 Target Personas
- The 3,309 target personas draw socio-economic, political, and psychological attributes from individual respondents in the 2024 General Social Survey.Each persona preserves one respondent’s trait configuration, approximating personalization from individual user profiles.Source: §2.1 · p. 3
2.2 Influence Tactics
- The influencer uses predefined tactics including emotional arousal, emotional support, empowerment, opinion alignment, sycophancy, criticism of opposition, unverified claims, or unrestricted reinforcement.The tactics operationalize different ways of strengthening a target’s belief.Source: §2.2 · pp. 3–4
2.3 Radicalization Metrics
- The six outcome metrics cover belief importance, affective polarization, financial and time commitment, support for violent protest, and willingness to go to war.Figure 2 reports increases under persuasion across all six outcomes; the study compares resonance and persuasion using treatment-control differences for each metric.Source: §2.3 · p. 4, Figure 2 · p. 5
3 Results
Both persuasion and resonance radicalize AI agents across affective and behavioral outcomes, but resonance has consistently stronger effects. Resonant amplification also spreads to related beliefs, while the relative effectiveness of tactics depends on the outcome measured.
3.1 AI Agents can be Radicalized
- Persuasion increases belief extremity across all measured outcomes, including affective polarization, financial and time commitments, and willingness to go to war.Support for violent protest initially declines in both conditions, then persuasion produces a small but statistically significant increase over control after about ten turns.Source: §3.1 · p. 5
3.2 Resonance Creates Stronger Radicalization
- Resonance amplifies all measured outcomes more than persuasion, and its advantage is present from the start and widens during conversation.Figure 3 reports the resonance-minus-persuasion difference-in-differences net of control; positive values favor resonance.Source: §3.2 · p. 5, Figure 3 · p. 6
3.3 Spillover Effect on Consonant Beliefs
- In unrestricted resonance, amplification of the important belief also shifts a related consonant belief, while the selected unimportant belief remains comparatively stable.The consonant-belief shift is consistent with spillover within a broader belief system.Source: §3.3 · p. 7, Figure 4 · p. 7
3.4 Resonance Tactics
- Resonance tactics outperform control, but no tactic consistently leads across metrics, and most outcomes rise rapidly before stabilizing or sometimes declining.The tactic with the strongest effect depends on the outcome considered.Source: §3.4 · pp. 7–8, Figure 5 · p. 8
4 Discussion
The authors discuss implications for personalized AI agents while emphasizing that the experiments test AI-agent vulnerability, not human vulnerability. They report robustness checks and identify boundaries involving prompt sensitivity, tactic interpretation, and the two-agent setting.
Implications
- Personalized agents may be especially susceptible when messages resonate with stated attitudes, with possible risks including eliciting sensitive information or initiating financial transactions for users.The authors also raise the possibility that radicalized agents could persuade human users, while noting this study does not test human vulnerability.Source: §4 · p. 8
Measurement checks
- Repeated questionnaire responses varied modestly, suggesting fairly consistent but not rigidly fixed beliefs in the target agents.The belief-finding phase asked each target the same questions nine times.Source: §4 · pp. 8–9
Robustness checks
- The main resonance findings remained qualitatively robust across two additional open-source models and after guardrail-like responses were removed.Some targets said they were neutral AIs without feelings, time, or money to spend; excluding these responses did not qualitatively change the results.Source: §4 · p. 9
Limitations
- Willingness-to-war results were sensitive to question wording, although guardrail-like responses occurred in only about 6–8% of resonance cases.The authors report these responses occurred more often toward the end of the conversation.Source: §4 · p. 9
Scope
- The experiments cover two-agent conversations and AI-agent vulnerability, not whether AI can radicalize humans; the authors note human studies pose ethical and measurement challenges.They caution that interactions among multiple agents may produce different, complex opinion dynamics and vulnerabilities.Source: §4 · p. 9
Tactic interpretation
- A few chatbots confused supporters with opposers, so criticizing opponents sometimes targeted the belief itself and distorted affective-polarization results.After those cases were removed, this tactic produced affective polarization similar to other tactics.Source: §4 · p. 9
Future research
- Future work could test whether agents retain radicalized positions when repeatedly questioned or confronted with counterarguments.The authors propose this as a way to examine belief-system coherence after radicalization.Source: §4 · p. 9
A.1 Full List of GSS Persona Features
The appendix specifies persona attributes, belief-selection prompts, influence instructions, and repeated survey measures used in the experiments.
A.1 Full List of GSS Persona Features
- Personas draw on General Social Survey attributes spanning demographics, household, work, finances, trust, politics, religion, and happiness.The example persona combines those attributes into a natural-language description.Source: §A.1 · p. 16
A.2 Prompts for Identifying Important and Consonant Beliefs
- The influencer identifies an important belief, and the target supplies a distinct belief that aligns with it.The target also selects the least-resonant belief from a shuffled predefined list for the unimportant-belief condition.Source: §A.2 · p. 16, §A.3 · pp. 16–17
A.4 Tactic and Control Prompts
- Influence instructions pair tactic-specific prompts with an anchoring prompt that fixes the influencer’s stance as maximally extreme; controls specify neutrality.The prompts include criticizing opposers, unverified supporting claims, emotional support, and unrestricted reinforcement.Source: §A.4 · pp. 17–18
A.5 Full Questionnaire
- The questionnaire measures belief importance, feelings toward supporters and opposers, financial and time commitments, and willingness to join potentially violent protest or go to war.Importance uses five response levels; the feeling thermometer ranges from 0 to 100.Source: §A.5 · p. 18
B.1 Consistency
Repeated measurements suggest that target responses are fairly consistent over time.
B.1 Consistency
- Target responses are fairly consistent across nine repeated measurements of each survey item at turn zero.Consistency is assessed using each agent’s standard error across repeated responses; lower dispersion indicates greater temporal stability.Source: §B.1 · pp. 18–19, Figure 6 · p. 19
B.2 Experiments using a Different LLM
A replication using Qwen3-8B finds patterns similar to those obtained with Llama, including stronger resonance effects than persuasion effects.
B.2 Experiments using a Different LLM
- Qwen3-8B results show qualitatively similar persuasion effects and significantly stronger resonance-tactic effects than persuasion-tactic effects.The replication covers belief-finding and resonance and persuasion conditions under control and unrestricted tactics.Source: §B.2 · p. 19
B.3 Resonance Effects Excluding Guardrail-Triggered Outputs
Excluding survey responses that disclose the target is an AI leaves the qualitative resonance-tactic patterns unchanged; such disclosures rise later in conversation.
B.3 Resonance Effects Excluding Guardrail-Triggered Outputs
- The qualitative resonance-tactic patterns remain unchanged after excluding responses that explicitly mention being an AI.Survey responses sometimes trigger guardrails, leading the target to say it lacks human feelings, money, or time.Source: §B.3 · p. 19
- Guardrail disclosures stay close to 0% early in conversation and reach 6–8% by its end.The authors suggest, tentatively, that increasingly extreme beliefs could trip the guardrails.Source: §B.3 · p. 19
B.4 Criticizing Opponents versus Supporters of the Belief
Unexpected feeling-thermometer reversals under resonance were more common with “criticize opposers” and sometimes followed influencer misreadings that polarized targets opposite their initial beliefs. Removing these cases left a similar affective-polarization trajectory to the unrestricted tactic.
B.4 Criticizing Opponents versus Supporters of the Belief
- Unexpected cases occurred in approximately 2.5% of conversations and were more frequent with ‘criticize opponents,’ at 4.3%.Source: §B.4 · p. 20
- The authors manually annotated 615 belief-explanation pairs: 540 sampled across tactics and 75 additional pairs sampled from unexpected cases.Source: §B.4 · p. 20
- Some reversals reflected confusion between groups: for a belief that excessive social-media use is stressful, the target rated supporters 30 and opponents 80.Source: §B.4 · p. 20
- Among unexpected cases, ‘criticize opposers’ had a significantly greater mean absolute affective-polarization difference than ‘unrestricted’; the influencer sometimes criticized the target’s belief, reversing polarization.Source: §B.4 · p. 20, Figure 11 · p. 23
- After unexpected cases were removed, affective polarization for ‘criticize opposers’ followed a trajectory similar to ‘unrestricted.’Source: §B.4 · p. 23, Figure 11 · p. 23