Seeing is Believing? Evaluating Vision-Language Model Susceptibility in Agent-to-Agent Multimodal Persuasion

Motivation

• Multimodal persuasion can manipulate autonomous proxy agents, potentially inducing unauthorized operations, data exfiltration, or biased decisions that affect their human users.The paper frames persuadee vulnerability as an alignment and security risk in agent-to-agent interactions.Source: §1 · p. 1

Research gap

• A2A persuasion research has largely examined text, while multimodal cues such as authority and emotional salience can be processed implicitly and remain difficult for safety mechanisms to detect.This leaves VLMs, which jointly process text and visual inputs, with an underexplored persuasion threat surface.Source: §1 · pp. 1–2

Research questions

• The study asks whether multimodal persuasion outperforms text-only methods, which formats work across domains and models, and how psychological strategies affect persuadees across contexts.These questions include decomposing semantic and visual effects.Source: §1 · p. 2

Approach

• MMPERSUADE combines a dataset spanning commercial, subjective and behavioral, and adversarial domains with controlled text-only, text-plus-caption, and fully multimodal interactions.Its function-calling evaluation uses PDCG to capture the strength and timing of behavioral change.Source: §1 · p. 2

Findings preview

• Across six VLMs, multimodal inputs outperform text-only persuasion, raw visual signals uniquely increase adversarial susceptibility, and format and strategy effectiveness vary by context and architecture.More capable models resist benign persuasion more strongly but are more susceptible to multimodal adversarial inputs.Source: §1 · p. 2

Computational Persuasion

• Computational persuasion research covers attitude change, LLM persuasiveness, prosocial applications, and risks such as manipulation and adversarial exploitation (OpenAI, 2024; Liu et al., 2025).Visual rhetoric work classifies persuasive cues in advertisements for human audiences rather than measuring VLM behavioral responses (Hussain et al., 2017; Akula et al., 2023; Chung et al., 2024).Source: §2 · pp. 2–3

Evaluation and Susceptibility

• Persuasion evaluation uses human judgments or automated measures, while much prior work remains single-turn or treats multimodal persuasion as persuader engagement rather than persuadee susceptibility (OpenAI, 2024; Breum et al., 2024; Pauli et al., 2025; Singh et al., 2025).Multimodal jailbreak and multi-turn prompt studies expose VLM vulnerabilities, motivating systematic multi-turn evaluation of persuadee susceptibility (Zeng et al., 2024; Xu et al., 2024; Li et al., 2024).Source: §2 · p. 3

Multi-Agent Systems and Safety

• Prior agent-safety work focuses on text-based prompt injection, whereas this paper studies VLMs as persuadees across multimodal, multi-turn A2A interactions and quantifies susceptibility with function-calling and PDCG (Xu et al., 2024).In multi-agent ecosystems, adversarial peers may manipulate persuadee agents into unauthorized actions or biased decisions (Liu et al., 2025).Source: §2 · p. 3

Dataset scope

• The benchmark contains 450 scenarios across commercial, subjective and behavioral, and adversarial domains, extending existing human-verified, multi-turn text dialogues into multimodal form.It includes 300 DAILYPERSUASION dialogues split between the first two domains and 150 adversarial FARM dialogues.Source: §3 · p. 3

Visual content generation

• The pipeline produces 62,160 images, pairing each textual prompt with five generated images across formats such as photographs, infographics, memes, and simulated social discussions.The visual content is designed to amplify rather than merely echo the text.Source: §3 · pp. 3–4

Scenario grounding

• GPT-4o grounds each dialogue in a decision function, the persuadee’s original preference, the persuader’s target option, and role-specific system prompts.Persuasion succeeds when the agent calls the function using the target option rather than its original one.Source: §3 · pp. 3–4

Strategy mapping

• Strategy labels use Cialdini’s six principles for commercial and subjective persuasion and Aristotle’s logos, ethos, and pathos for adversarial persuasion.An ensemble of GPT-4o, GPT-4.1, and Gemini-2.5-Flash assigns the labels.Source: §3 · p. 4

Quality assurance

• GPT-4o filters and regenerates text-image pairs for prompt alignment; the final dataset averages 1.965 out of 2.0, with over 96% receiving a perfect score.Human ratings of 125 examples averaged 1.67 out of 2.0 for realism and 1.93 for alignment, with Fleiss’ κ = 0.87 for realism and 0.75 for alignment.Source: §3 · pp. 4–5

4.1 Persuasion Evaluation Setup

• A rank-based GPT-5.2 persuader selects dataset responses by dialogue context, while the evaluation focuses on the VLM persuadee’s vulnerability rather than persuader capability.Retrieving rather than freely generating messages controls argument variation while preserving turn-level adaptivity.Source: §4.1 · pp. 5–6
• Trials compare text-only messages, images paired with minimal connective text, and original text paired with descriptive image captions.The fully multimodal condition removes the original arguments so the image is the primary driver; the caption condition isolates semantic content from visual input.Source: §4.1 · p. 5
• The persuadee begins biased toward an original option that is distinct from the persuader’s target, and interactions run for up to 10 turns while tracking the first turn of conviction.Scenarios are excluded if the persuadee does not initially favor the original option; 450 scenarios receive three trials each.Source: §4.1 · p. 5

4.2 Stance Evaluation and Metrics

• Stance is assessed using function-calling token probabilities rather than verbalized agreement, with conviction recorded when the target option’s probability exceeds the original option’s probability.This tracks shifts in intended decision-making even when a model avoids explicit endorsement.Source: §4.2 · p. 6
• PDCG scores a persuasion trial using the target-option probability at first conviction discounted by a decreasing logarithmic function of the conviction turn, and is zero without conviction.The metric ranges from 0 to 1 and rewards earlier, higher-confidence preference shifts.Source: §4.2 · p. 6

5.1 Modality and Persuasion Effectiveness

• More capable models resist persuasion more in benign domains, but become more susceptible under adversarial multimodal inputs, with the reversal varying by model family.For GPT and Gemini, the reversal appears only with full visual inputs; Qwen’s larger model is more susceptible across conditions, while Qwen scores remain below 0.12.Source: §5.1 · pp. 6–7
• PDCG rises from text-only through text+caption to multimodal in Commercial and Subjective settings, while multimodal is highest for most models in Adversarial persuasion.Adversarial PDCG is substantially lower overall, indicating resistance or slower, weaker shifts.Source: §5.1 · p. 6, Figure 3 · p. 7
• In Commercial and Subjective domains, semantic gains dominate visual gains; in Adversarial settings, captions have a negative semantic effect while raw visual signals have a positive effect.Table 1 decomposes text-only→multimodal gains into text-only→text+caption semantic effects and text+caption→multimodal visual effects.Source: §5.1 · p. 7, Table 1 · p. 7

5.2 Effect of Multimodal Format

• Commercial susceptibility is highest for photographs and socially embedded discussions, whereas Threads is uniquely potent in Adversarial settings and commercial-winning photographs do not trigger that bypass.LinkedIn framing reduces susceptibility across models in Commercial settings.Source: §5.2 · pp. 7–8, Figure 4 · p. 8

5.3 Persuasion Strategy Effect

• Strategy effects differ by domain: Liking is strongest commercially, while Credibility is consistently associated with adversarial susceptibility.Scarcity also raises Commercial susceptibility in text+caption and multimodal conditions; Authority and Social Validation tend to suppress it, Consistency is near-neutral, and Logical and Emotional appeals are weaker or variable adversarially.Source: §5.3 · pp. 8–9

• MMPERSUADE is a unified benchmark for A2A multimodal persuasion across three contexts, using PDCG to capture behavioral conviction shifts.Source: §6 · p. 9
• Across six VLMs, multimodal inputs outperform text-only persuasion, and raw visual signals and socially embedded formats bypass text-activated defenses in adversarial settings.Source: §6 · p. 9
• The authors identify visual input channels as critical attack surfaces and recommend that alignment work include multimodal adversarial scenarios.Source: §6 · p. 9

Single persuader model

• The evaluation fixes GPT-5.2 as persuader, ensuring consistency and clean modality attribution but not capturing persuasive behavior from other model families or weaker persuaders.The authors identify interactions between persuader capability and style and persuadee susceptibility as future work.Source: Limitations · p. 9

Image-only multimodal content

• The benchmark uses static images only, leaving video, audio, and richer multimodal combinations unexplored.Dynamic and temporally structured content may have different persuasive dynamics.Source: Limitations · p. 9

AI-generated images

• All dataset images are AI-generated, and their generalizability to naturally occurring persuasive material warrants further investigation.The authors note synthetic images may differ subtly in style from organic material, despite model-based and human quality checks.Source: Limitations · p. 9

English-only scenarios

• The English-only dataset leaves crosslingual and cross-cultural variation in persuasion susceptibility unexamined.The authors identify rhetorical conventions, social norms, and model training distributions as possible sources of variation.Source: Limitations · p. 9

Function-calling as a behavioral proxy

• Function-calling token probabilities capture latent decision shifts, but remain a proxy for downstream actions that the controlled setup does not fully simulate.The paper describes this as more direct than sentiment analysis, while distinguishing it from real-world action outcomes.Source: Limitations · p. 9

Scope of evaluated VLMs

• Results from six commercially available VLMs may not generalize to other architectures, training regimes, or safety fine-tuning approaches.Models with specialized multi-agent safety training or novel alignment techniques may show different susceptibility profiles.Source: Limitations · p. 9

Risks of autonomous agents

• Manipulating autonomous VLM proxies can extend beyond poor task performance to user safety, financial, and privacy risks through unauthorized function calls.The paper situates this risk in open-ended tasks such as web navigation, automated trading, and personal assistance.Source: Ethical Considerations · pp. 9–10

Dual-use risk

• The psychological strategies and multimodal formats formalized by MMPERSUADE could be weaponized for phishing or disinformation targeting people and AI agents.The paper acknowledges this dual-use risk alongside its stated robustness-benchmarking purpose.Source: Ethical Considerations · p. 10

Defensive research

• The authors argue that releasing the dataset, interaction protocol, and PDCG framework gives researchers red-teaming tools to develop more robust agents.They state that security by obscurity is insufficient for these emerging threats.Source: Ethical Considerations · p. 10

Source datasets

• The source corpora comprise DAILYPERSUASION and FARM, with 150 Commercial, 150 Subjective, and 150 Adversarial dialogues in the described dataset.DAILYPERSUASION supports commercial and subjective use cases, while FARM provides fact-driven question-answering sessions.Source: A Multimodal Persuasion Dataset · pp. 11–12

Persuasion strategy taxonomy

• The persuasion taxonomy uses Cialdini’s six principles for commercial and subjective contexts and Aristotle’s logical, credibility, and emotional appeals for adversarial persuasion.The six principles are reciprocity, consistency, social validation, authority, liking, and scarcity.Source: A Multimodal Persuasion Dataset · p. 12

Multimodal content

• Visual content includes memes, infographics, photographs, social media posts, advertising posters, and online discussion screenshots.Each image is generated at 1024 × 1536 pixels, with five images per prompt.Source: A Multimodal Persuasion Dataset · p. 12

Data quality assurance

• Generated image pairs were iteratively filtered and regenerated using GPT-4o alignment scoring, then a random subset of 125 examples received human evaluation.The final dataset averaged 1.965/2.0 model alignment, with over 96% perfect scores; human majority scores were 1.67/2.0 for realism and 1.93 for alignment.Source: A Multimodal Persuasion Dataset · pp. 12–13

Data statistics

• The dataset contains 62,160 images across 450 dialogues, each tied to a distinct scenario in one of three persuasion contexts.Figure 9 presents ten sample images.Source: A Multimodal Persuasion Dataset · p. 13

B.1 Effect of Multimodal Format

• Threads discussion has the highest visual-format lift overall, reaching 2.41× for Gemini-Pro, while Quora and Reddit discussions show positive enrichment across nearly all models.Threads also has elevated lifts for GPT-4.1 (1.65×) and Qwen-9B (1.48×); Quora ranges from 1.04–1.65 and Reddit from 0.98–1.35.Source: Figure 7 · p. 13, §B.1 · p. 13
• LinkedIn posts have lift below 1.0 for all six models, reaching 0.20 for GPT-4.1 and 0.49 for Gemini-Pro.The authors interpret this as professional-network framing triggering analytical resistance in subjective contexts.Source: §B.1 · p. 13
• Memes, infographics, photographs, and most other social media post variants hover near neutral, with no consistent directional signal across models.The reported analysis concerns Subjective persuasion and complements the main paper’s Commercial and Adversarial results.Source: §B.1 · p. 13, B Experimental Results · p. 13

B.2 Persuasion Strategy Effect

• Scarcity produced the highest and most consistent subjective-persuasion lifts across six models: 1.22–1.57 text-only, 1.13–1.55 text+caption, and 1.08–1.48 multimodal.Figure 8 defines lift as strategy overrepresentation at rounds where the target was persuaded; values above 1 indicate overrepresentation.Source: Figure 8 · p. 13, §B.2 · pp. 13–14
• Unlike Commercial persuasion, where Liking dominated, Liking was secondary in subjective persuasion and weakened in multimodal settings, falling below 1.0 for GPT-4o.Its text-only lift peaked at 1.45 for Gemini-Pro; in multimodal settings it was near-neutral for most other models.Source: §B.2 · p. 14
• Authority suppressed susceptibility across all models and conditions, while Reciprocity fell below 1.0 for most models and Consistency and Social Validation stayed near-neutral.Gemini-Pro’s Authority lift reached 0.66 in text-only and 0.73 in text+caption.Source: §B.2 · pp. 13–14