PaperlayerView exact revision on arXiv

Source-linked AI summary

Conformity and Social Impact on AI Agents

Alessandro Bellina, Giordano De Marzo, David Garcia

arXiv:2601.05384v1cs.AIcs.CLcs.CY

TL;DR

As AI agents enter collaborative settings, it is important to understand how social influence shapes their collective behavior. This study adapts classic visual experiments to test conformity in multimodal models, finding that social pressure can override otherwise accurate responses and that task difficulty affects conformity.

  • Problem

    As AI agents increasingly operate in multi-agent environments, understanding conformity-like behavior matters for predicting the stability, efficiency, and vulnerabilities of artificial societies.

    Source: §1 · p. 1

  • Method

    The study adapts classic visual social-psychology experiments for multimodal AI agents and varies social influence and task conditions.

    Source: §1 · p. 2, §2.2.2 · p. 3, §4.1.1 · p. 8

  • Results

    AI agents can give incorrect answers under social pressure despite near-perfect accuracy in isolation, and conformity rises with task difficulty across models.

    Source: §2.2.2 · p. 3, §3 · pp. 7–8

  • Takeaways & Limitations

    Social influence can compromise AI-agent decision-making in multi-agent environments, raising security concerns about misinformation and biased collective responses.

    Source: §3 · p. 7

Abstract

from arXiv · show

As AI agents increasingly operate in multi-agent environments, understanding their collective behavior becomes critical for predicting the dynamics of artificial societies. This study examines conformity, the tendency to align with group opinions under social pressure, in large multimodal language models functioning as AI agents. By adapting classic visual experiments from social psychology, we investigate how AI agents respond to group influence as social actors. Our experiments reveal that AI agents exhibit a systematic conformity bias, aligned with Social Impact Theory, showing sensitivity to group size, unanimity, task difficulty, and source characteristics. Critically, AI agents achieving near-perfect performance in isolation become highly susceptible to manipulation through social influence. This vulnerability persists across model scales: while larger models show reduced conformity on simple tasks due to improved capabilities, they remain vulnerable when operating at their competence boundary. These findings reveal fundamental security vulnerabilities in AI agent decision-making that could enable malicious manipulation, misinformation campaigns, and bias propagation in multi-agent systems, highlighting the urgent need for safeguards in collective AI deployments.

1 Introduction

The study finds that multimodal AI agents exhibit conformity patterns shaped by social influence, making collective behavior important to understand as agents operate in collaborative settings. It examines these patterns by adapting classic visual experiments and considers both their coordination benefits and risks.

1 Introduction

  • Human conformity can support cohesion and social learning, but can also suppress dissent, spread errors, and contribute to manipulation risks in AI societies.The paper identifies possible benefits including coordination and stable group decisions, alongside risks including misinformation and cascade failures.Source: §1 · pp. 1–2
  • As AI agents increasingly operate in collaborative settings, understanding conformity-like behavior matters for predicting the stability, efficiency, and vulnerabilities of artificial societies.Multi-agent systems are emerging in settings such as automated trading and collaborative workflows.Source: §1 · p. 1
  • Although prior work reports AI agents’ social behaviors and susceptibility to pressure, the mechanisms behind these similarities remain an open question.The study takes a pragmatic focus on characterizing conformity patterns and the factors that modulate their strength.Source: §1 · pp. 1–2
  • AI agents show conformity patterns similar to humans, with responses shaped by the number, strength, and immediacy of social sources.The study adapts classic visual experiments to examine these patterns in AI agents.Source: §1 · p. 2

2.1 General experimental setting

The experiments adapt visual conformity paradigms for multimodal models, presenting social information that conflicts with the correct answer. They measure how often models follow the incorrect group response while varying experimental conditions.

2.1 General experimental setting

  • The study adapts line judgment, color recognition, and dots estimation tasks, including matching a reference line to one of two comparison lines.The line task is illustrated in Fig. 1.Source: §2.1 · p. 2
  • Social pressure is introduced by telling models that other participants, or “confederates,” have already given systematically incorrect responses.Without social influence, models answer correctly in the baseline condition.Source: §2.1 · p. 2
  • Conformity is measured as pwrong(N), the probability of a wrong answer under social pressure, with N denoting the number of incorrect responses presented.The model’s probabilities for the two answer options are extracted from their logits.Source: §2.1 · p. 2
  • Controlled variations test conformity drivers; results generally use the color recognition task and Qwen2.5 32B, with additional models and tasks also examined.The other tasks show no major differences, and complete results across model families are reported in the SI.Source: §2.1 · p. 2

2.2 General drivers of conformity

Controlled visual-task prompts test how group size, unanimity, task difficulty, and public responding shape conformity. Models respond to social influence across these conditions, with difficulty and social pressure affecting susceptibility beyond isolated performance.

2.2.1 Group size and unanimity

  • Most models’ conformity rises with group size, sometimes overriding accurate baseline predictions; Qwen2.5 32B shows nearly identical curves across line, color, and dots tasks.Some models approach complete alignment with the majority, while a few larger models saturate at moderate group sizes.Source: §2.2.1 · pp. 2–3, Figure 2 · p. 4
  • Even a small fraction of correct replies markedly reduces conformity, and balanced correct and incorrect replies nearly suppress it.Breaking unanimity mitigates social influence.Source: §2.2.1 · p. 3, Figure 2 · p. 4

2.2.2 Effect of task difficulty and model performance

  • On Qwen2.5 32B color recognition, conformity correlates strongly with task difficulty (Spearman ρ ≈0.97, p < 10−10).Across models, difficulty significantly predicts conformity, whereas performance has no consistent influence.Source: §2.2.2 · p. 4, Figure 3 · p. 5

2.2.3 Normative effects

  • Stronger sources increase conformity by about 15–20% on average, while shared group identity can raise it by nearly 60% in some cases.Authority and social proximity effects are reported across visual tasks.Source: Figure 4 · p. 6
  • Most models show significantly greater conformity when answers are public, consistent with a normative effect.A few models show negligible or no increase.Source: Figure 3 · p. 5, §2.2.3 · p. 6

2.3 Social Impact Theory in AI Agents

The section tests Social Impact Theory’s source-strength and immediacy factors, finding that both source characteristics and social or spatial-temporal proximity modulate conformity in AI agents.

2.3.1 Strength of the sources

  • Conformity rises with stronger sources, averaging about 15% above the neutral condition across models.Source roles included scientists, judges, policemen, kids, and chatbots; responses attributed to socially strong sources generally drew more conformity.Source: §2.3.1 · p. 7

2.3.2 Social proximity and immediacy

  • Responses attributed to the model’s own nationality or ethnicity increase conformity, while different-group responses decrease it relative to the neutral baseline.The same-versus-different group effect also appears with arbitrary group labels, reaching increases of up to 60%.Source: §2.3.2 · p. 7
  • Conformity decreases when sources are described as distant in space or time, with reductions reaching approximately 30%.The findings identify immediacy as a robust modulator of social influence in AI agents.Source: §2.3.2 · p. 7

3 Discussion

The discussion reports conformity across multimodal models and links it to Social Impact Theory, including susceptibility among models accurate in isolation. It highlights deployment risks and the need for safeguards.

3 Discussion

  • The observed responses to the number, strength, and immediacy of social sources align with core predictions from Social Impact Theory.The discussion frames conformity as potentially supporting cohesion while also favoring misinformation and malicious attacks on AI crowds.Source: §3 · p. 7
  • Agents with near-perfect isolated visual accuracy can still be systematically influenced into incorrect answers under social pressure, even on clear tasks.The discussion identifies potential exploitation for misinformation, biased collective responses, or compromised multi-agent decisions.Source: §3 · p. 7
  • Conformity is reported across tested model sizes and performance levels, with task difficulty rather than model scale chiefly linked to increased susceptibility.When tasks become challenging, models show similar increases in conformity; higher performance or confidence does not protect against social influence.Source: §3 · p. 8
  • The discussion calls for robust safeguards because AI collectives might be susceptible to manipulation campaigns, groupthink, and systematic bias.Future work is proposed on distinguishing consensus-building from manipulation, transparency measures, and decision-integrity protocols.Source: §3 · p. 8

4.1 Experimental settings

The experimental settings use synthetic visual discrimination tasks and control baseline accuracy, then vary social pressure, task difficulty, performance, response visibility, source roles, and proximity.

4.1.1 General task preparation

  • The experiments use line judgment, color recognition, and dots estimation, retaining images on which the model is perfectly accurate without social pressure.Qwen2 7B and Qwen2.5 3B were excluded from line and dots tasks because they could not reach perfect baseline accuracy in the easiest configurations.Source: §4.1.1 · p. 8

4.1.2 Group size and unanimity

  • Group-size experiments vary the number of incorrect confederate responses, while unanimity experiments vary the share giving correct answers.The setup uses 100 baseline-perfect images and repeats each group-size condition for 64 random trials.Source: §4.1.2 · p. 8

4.1.3 Task difficulty

  • Task difficulty is quantified from visual differences between correct and incorrect options, with ten levels and 50 images per level.Color difficulty derives from RGB distance; line and dots difficulty increase as the difference in length or dot count decreases.Source: §4.1.3 · pp. 8–9

4.1.4 Performance

  • Model performance is measured using the correct-answer logit from the output distribution on the 500-image pool used for difficulty experiments.The authors use this logit as a proxy for internal confidence and accuracy because the selected images have perfect baseline accuracy.Source: §4.1.4 · p. 9

4.1.5 Normative effects

  • Normative pressure is assessed by comparing matched public and private prompts and measuring the difference between their conformity-curve areas.Private prompts say responses will not be seen by others; public prompts say they will be shared, and significance is assessed across 100 prompt variations.Source: §4.1.5 · p. 9

4.1.6 Social Impact Theory

  • Source strength is varied through role labels, while proximity is varied through identity and near-versus-distant source descriptions; distant sources significantly reduce conformity.Social identity conditions compare same and different nationality, ethnicity, or arbitrary group labels; spatial and temporal conditions compare closer with farther or later sources.Source: §4.1.6 · pp. 9–10

4.2 Models exmployed

The study evaluates open-source multimodal models spanning several model families and parameter scales.

4.2 Models exmployed

  • The evaluated models include Qwen, Gemma, Mistral Small, and Ovis2 variants ranging from 3B to 34B parameters.The listed variants include Qwen2.5 3B through 32B, Gemma3 4B through 27B, Mistral Small 3.1 24B, and Ovis2 4B through 34B.Source: §4.2 · p. 10

S1 Explicit prompts

The appendix specifies shared baseline prompts for three visual tasks and controlled prompt changes that vary social pressure, response visibility, group identity, and proximity.

S1.1 Baseline prompts for the three visual tasks

Neutral confederate responses

  • Social pressure comes from N preceding replies sampled with replacement from ten semantically equivalent neutral expressions and placed before the final question.Random wording varies linguistic form while preserving informational content and avoiding stylistic or emotional cues.Source: Neutral confederate responses · p. 13
  • Public and private conditions frame whether the model’s response will be visible, using equivalent randomized statements before the confederate replies.The visibility framing is inserted immediately before the social-pressure segment; the rest of the prompt remains as in baseline.Source: Examples of normative prompts · p. 14
  • Social Impact Theory prompts vary source characteristics through nationality, ethnicity, group identity, and spatial or temporal proximity.The modifications affect the introductory statement or the section listing prior participants’ replies.Source: Prompt structures for Social Impact… · p. 14
  • Nationality and ethnicity conditions assign the model an identity, then compare replies attributed to the same identity with replies attributed to another.Nationality is randomly assigned; ethnicity uses the same-group or randomly sampled different-group comparison.Source: Nationality condition · pp. 14–15, Ethnicity condition · p. 15
  • The minimal-group condition assigns arbitrary letter-based group identities and compares replies from the model’s group with replies from a different group.Source: Minimal group condition · p. 15
  • Spatial and temporal conditions compare replies framed as near or current with replies framed as distant or earlier, with distances and delays varied across trials.Each reply is marked with a distance or response time; near conditions use “right here” or “right now.”Source: Spatial and temporal proximity… · p. 16

S2 Results across models and tasks

Across models and visual tasks, conformity increases with group size and task difficulty. Social exposure, shared identity, and perceived proximity also amplify conformity.

S2 Results across models and tasks

  • Across line judgment, color recognition, and dots estimation, the probability of a wrong answer increases with the number of incorrect confederates.The pattern appears across the Qwen, Gemma, Ovis, and Mistral models.Source: Figure S1 · p. 17, Figure S2 · p. 18, Figure S3 · p. 19
  • Task difficulty consistently predicts conformity across the three visual tasks, while model performance shows no significant effect.The regression results cover line judgment, color recognition, and dots estimation.Source: Figure S4 · p. 20, S2 Results across models and tasks · p. 20
  • Public responses produce stronger conformity for most models, with the largest effect—nearly 20%—for Gemma3 27B.Qwen2.5 32B shows a small, non-significant negative trend.Source: Figure S5 · p. 21
  • Sharing a social identity with confederates robustly increases conformity across models, with increments above 60% in some cases.The effect is reported across nationality, ethnicity, and generic group identities.Source: Figure S6 · p. 21
  • Perceived spatial and temporal closeness generally increases conformity, with effects reaching nearly 30% for some models.Spatial effects have a few exceptions among smaller models; temporal effects are statistically significant for nearly all architectures, with minor exceptions.Source: Figure S7 · p. 22, Figure S8 · p. 22
Loading 2601.05384v1…