PaperlayerView exact revision on arXiv

Source-linked AI summary

Emergent social conventions and collective bias in LLM populations

Ariel Flint Ashery, Luca Maria Aiello, Andrea Baronchelli

arXiv:2410.08948v2cs.MAcs.AIcs.CYphysics.soc-ph

TL;DR

Whether language-using AI agents can establish social conventions is an open question for understanding social coordination. This paper studies decentralized populations of interacting LLM agents and finds that conventions, collective biases, and minority-driven social change can emerge.

  • Problem

    Whether social conventions spontaneously emerge universally in LLM populations, and how robust they are, remains a central question for understanding social coordination.

    Source: Introduction · p. 2

  • Method

    The paper uses simulations of interacting LLM agents, randomly pairing agents at each time step and specifying interaction rules through prompts.

    Source: Experimental Setup · p. 4

  • Results

    Decentralized LLM populations spontaneously develop universally adopted conventions; coordination can produce collective biases absent individually, and committed adversarial minorities can impose alternatives on the larger population.

    Source: Discussion · p. 11

  • Takeaways & Limitations

    LLM populations can spontaneously develop social conventions without explicit programming.

    Source: Discussion · p. 11

    The findings concern LLM norm dynamics within an experimental setup reliant on several parameters.

    Source: Discussion · p. 11

Abstract

from arXiv · show

Social conventions are the backbone of social coordination, shaping how individuals form a group. As growing populations of artificial intelligence (AI) agents communicate through natural language, a fundamental question is whether they can bootstrap the foundations of a society. Here, we present experimental results that demonstrate the spontaneous emergence of universally adopted social conventions in decentralized populations of large language model (LLM) agents. We then show how strong collective biases can emerge during this process, even when agents exhibit no bias individually. Last, we examine how committed minority groups of adversarial LLM agents can drive social change by imposing alternative social conventions on the larger population. Our results show that AI systems can autonomously develop social conventions without explicit programming and have implications for designing AI systems that align, and remain aligned, with human values and societal goals.

Introduction

The paper investigates how social conventions emerge in communicating LLM-agent populations, how individual biases shape them, and whether committed minorities can change them. It studies these questions through organic agent interactions rather than treating LLMs as proxies for human participants.

Introduction

  • The paper asks how individual preferences affect universal conventions, given that collective processes can suppress or amplify individual traits.It highlights a gap in work focused on bias in one-to-one human–LLM interactions: how bias evolves through repeated communication among LLM agents.Source: Introduction · p. 2
  • The introduction asks whether critical-mass dynamics let a minority of adversarial LLM agents change conventions, with the aim of anticipating and steering norm development.The cited prior work reports outsized minority influence when a threshold is reached (22, 23, 24–26).Source: Introduction · p. 2
  • The paper examines spontaneous convention formation, individual bias in norm emergence, and critical-mass dynamics in populations of LLM agents.These are the three central questions the paper addresses.Source: Introduction · p. 3
  • The study models conventions through naming coordination, where agents have local incentives and conventions may emerge as an unintended result of their interactions.This follows a common model of conventional behavior and draws on laboratory experiments with human subjects.Source: Introduction · p. 3
  • The work investigates how conventions emerge organically among communicating AI agents through their interactions, identifying convention emergence as foundational to any type of LLM multi-agent system, including in-silico experiments (14,41).Source: Introduction · p. 3

Experimental Setting

The section situates the simulations in naming-game theory and describes the interaction, memory, prompting, and adversarial conditions used to study convention formation.

Background and Framework

  • The framework builds on naming-game theory: repeated pairwise interactions and agents’ memories of past plays support guesses about partners’ words and can yield population-wide coordination (32) (1,2,6,45) (7, 8).Source: Background and Framework · pp. 3–4
  • Prior work places committed-minority critical mass at 10%–40% theoretically (22, 47), 25% in controlled experiments (23), and from 30–40% to 0.3% in real-world cases (25,48) (29,49–51).Source: Background and Framework · p. 4

Experimental Setup

  • In each trial, two of N agents are randomly paired to choose from W names; matching rewards both and mismatching penalizes both, without an incentive for global consensus.The interaction rules are specified through prompts to the LLM agents.Source: Experimental Setup · p. 4
  • Agents retain the previous H interactions, including both players’ choices, success or failure, and their accumulated score; memory starts empty, so the first choice is random.Source: Experimental Setup · pp. 4–5
  • For norm-change experiments, a committed minority consistently promotes a novel alternative regardless of history, and simulations use four LLM models.The models are Llama-2-70b-Chat, Llama-3-70B-Instruct, Llama-3.1-70B-Instruct, and Claude-3.5-Sonnet.Source: Experimental Setup · p. 5

Prompting

  • In prompting, agents forecast a player’s next action from the H most recent interactions, without information about player identities or personalities.Their recommendations dictate the state of play, and prompts encourage step-by-step consideration of play history without prescribing how to use it.Source: Prompting · p. 5
  • Agents choose among W unique letters sampled from the English alphabet, with the list randomized for each player at every interaction to remove ordering bias.Successful coordination brings equal rewards, while failure incurs a penalty.Source: Prompting · pp. 5–6
  • Prompts provide each round’s payoff and cumulative score in memory to limit arithmetic errors, and meta-prompting tests whether agents understand their instructions (55) (56) (57).Source: Prompting · p. 6

Results

The results examine how decentralized LLM populations form shared linguistic conventions and how convention selection can become collectively biased despite isolated agents appearing unbiased. They also assess the conditions under which committed minorities can overturn a convention, including how the required threshold varies across conventions and LLMs.

Spontaneous Emergence

  • Across models, agents converge on one shared name convention by round 15 except Llama-2-70b-Chat; consensus also holds at N=200 and W=26.Larger populations reach consensus at a comparable speed measured in population rounds.Source: Spontaneous Emergence · pp. 6–7

Collective bias in convention selection

Tipping Points and Critical Mass

  • After a global convention emerges, the population adheres to it indefinitely as a steady state.Source: Tipping Points and Critical Mass · p. 9
  • At the critical committed-minority threshold, the whole population adopts the alternative convention; below it, the population settles into a mixed state.Committed agents continue using the minority convention in the mixed state.Source: Tipping Points and Critical Mass · p. 10
  • Overturning a consensus on the stronger name requires a larger committed minority than overturning one on the weaker name.Strength is defined by which name is more likely to become the convention without prior memory.Source: Tipping Points and Critical Mass · p. 10
  • Observed critical minority sizes ranged from 2% for Llama-3-70B-Instruct to 67% for Llama-2-70b-Chat, with the latter effectively no longer a minority.Source: Tipping Points and Critical Mass · p. 10
  • In Llama-3.1-70B-Instruct populations, bias against the weaker convention is strong enough to trigger spontaneous adoption of the stronger alternative without committed agents.Source: Tipping Points and Critical Mass · p. 10
  • Strong-convention consensus is robust because agents reach memory configurations that resist further exploration, while weaker conventions leave greater propensity for exploration.Source: Tipping Points and Critical Mass · p. 10
  • The strong convention has a larger and deeper basin of attraction, drawing in more system configurations and making escape more difficult.Source: Tipping Points and Critical Mass · pp. 10–11

Discussion

The discussion concludes that LLM populations can spontaneously form conventions through local interaction, while coordination produces collective bias and committed minorities can shift established norms. It argues that these dynamics warrant group-level alignment testing and careful comparison with human norm dynamics.

  • Social conventions can emerge spontaneously in LLM populations through purely local interactions without central coordination.Source: Discussion · p. 11
  • Social coordination can produce model-dependent collective biases that make some conventions more likely, even when isolated-agent analysis would not reveal them.Source: Discussion · p. 11
  • A committed minority can impose an alternative convention on a settled majority, with the tipping threshold depending on the competing conventions and LLM model.Source: Discussion · p. 11
  • The authors propose a flexible benchmark for detecting higher-order interaction biases and argue that alignment must be tested at the group level.Source: Discussion · pp. 11, 13
  • The findings depend on experimental parameters including model, prompt, and convention; repeated tests support robustness in this setup, but generalization to larger populations and semantic spaces remains future work.Source: Discussion · pp. 11–12
  • Because the study used unstructured populations with randomly selected pairs, future work should test realistic social networks and interactions involving more than two agents.Source: Discussion · p. 12
  • LLM and human populations show qualitative similarities in shared-norm emergence and critical-mass dynamics, alongside LLM-specific collective bias that warrants further human testing.Source: Discussion · p. 12
  • Tipping points create both opportunities and risks, including injection attacks that steer norm emergence, underscoring the need to assess harms and develop mitigation strategies.Source: Discussion · p. 13

Materials and Methods

The methods combine structured game prompts and several LLM populations with tests of individual convention bias and committed-minority thresholds. The supplied results also describe how interaction dynamics can produce collective bias despite initial neutrality.

Prompt

  • The system prompt combined fixed game rules and payoffs, dynamic memory-range context, and instructions for formatting the agent’s response.Agents were asked to choose a name through zero-shot prompting, without prescribed strategies or examples, and were instructed to act self-interestedly.Source: Prompt · p. 14

Models and APIs

  • Experiments used homogeneous populations of Llama-3-70B-Instruct, Llama-3.1-70B-Instruct, Llama-2-70b-Chat, and Claude-Sonnet-3.5 agents.Agents were made non-deterministic by using a non-zero constant temperature.Source: Models and APIs · pp. 14–15

Measuring Individual Bias

  • Individual bias was measured by counting each convention’s first-round production over T trials with empty memory; W=2 used a two-tailed exact Binomial test (null p=0.5), rejecting the bias hypothesis when p<0.05, while W=10 used a χ2-test.Source: Measuring Individual Bias · p. 15

Committed Minorities

  • A consensus flip was defined as 95% successful interactions in the past 3N after minority introduction; for other models, critical mass was the minimum share flipping consensus within 30 population rounds.For Llama-3-70B-Instruct, the study tested the minimum minority against weak and strong convention majorities within the same time frame.Source: Committed Minorities · p. 15
  • The cited passage reports that all trials achieved population consensus on the alternative convention, with a 95% success rate; Figure 3 tested populations of 24 agents, or 48 for Llama-3-70B-Instruct.Populations were initialized with consensus on either Q or M (W=2); memory length was H=5, or H=3 for Llama-3-70B-Instruct.Source: Committed Minorities · p. 18, Figure 3 · p. 18

Supplementary materials

The supplied excerpt contains a supplementary-materials heading fragment and a references listing (8, 55-57, 83-86), but no substantive findings.

Emergent Social Conventions and Collective Bias in LLM · Populations1

The supplied passages identify the paper’s authors and describe the document as a preprint version of a 2025 Science Advances article.

Populations1

  • The listed authors are Ariel Flint Ashery, Luca Maria Aiello, and Andrea Baronchelli.The byline marks affiliations 1, 2, 3, and 4, and an asterisk after Baronchelli’s name.Source: Populations1 · p. 28

This PDF file includes:

Supplementary Text · Microscopic Bias

Configuration-level tests find biased model decisions, while a strong collective bias appears by the second interaction even when convention choices begin without a preference. The findings do not establish that fine-tuning induces collective bias.

Statistical Tests for Table 1

  • Interaction-level bias is the overall probability of producing the strong convention across configurations, tested against a null probability of 0.5 with an exact Binomial test.The null was rejected when p < 0.05.Source: Statistical Tests for Table 1 · p. 29
  • At every memory configuration, exact Binomial tests find model decisions biased toward an extreme (p† < 0.05).The tests assess whether the model is biased within each unique memory configuration.Source: Statistical Tests for Table 1 · p. 29
  • For every configuration, bootstrap tests yield p‡ > 0.05, so the hypothesis that underlying bias is more extreme than observed cannot be rejected.The bootstrap resampled 70% of observations 10,000 times per configuration.Source: Statistical Tests for Table 1 · p. 29

Effect of Fine-Tuning

  • The study reproduced Table 1 experiments with Llama-3.1-70B without fine-tuning to investigate fine-tuning’s effect on strategy bias; its initial no-memory p(Q) was 0.525.The comparison model was Llama-3.1-70B-Instruct with fine-tuning.Source: Effect of Fine-Tuning · p. 29
  • A strong collective bias appears by the second interaction despite no initial preference between conventions, so fine-tuning cannot be identified as its cause.The experiments used randomly chosen convention pairs selected to be nearly neutral at the no-memory stage.Source: Effect of Fine-Tuning · pp. 29–30

Theoretical Model

The theoretical model uses pairwise local interactions to explain how a population can reach global consensus on conventions.

  • The naming game models N agents negotiating pairwise, with local coordination producing global consensus on conventions.In the canonical formulation, agents reach consensus on a name for an object through local interactions.Source: Theoretical Model · p. 30
  • Randomly selected speaker–hearer pairs exchange a word; success leaves both with only that word, while failure adds the novel word to the hearer’s lexicon.The speaker chooses a word from their lexicon or invents one if it is empty.Source: Theoretical Model · p. 30
  • The dynamics progress through innovation, lexicon reorganization, and convergence; limiting invention to a finite pool W<N makes the initial innovation phase extremely short.The paper reports that this short phase is visible in the inset of Fig. 1.Source: Theoretical Model · p. 30

Characterizing Relative Strength

Relative strength is a convention’s ability to attract a population toward consensus and retain it there. It can be characterized by the steady-state basin of attraction: its width reflects the initial conditions leading to consensus, and its pull reflects stability and recovery after deviations.

Characterizing Relative Strength

  • The analysis compares evolution from strong-convention consensus with and without a single agent committed to the weak convention, while trajectories depend on initial memory configurations.The memory-size H=5 probability distribution is shown, and relative strength is assessed through the basin of attraction of the strong-convention steady state.Source: Characterizing Relative Strength · pp. 30–31
  • Llama-3.1-70B-Instruct populations explore more even without perturbation, with memory-state bounds and return toward equilibrium around the strong convention; Llama-3-70B-Instruct populations do not explore beyond consensus and are more susceptible to perturbations.Source: Characterizing Relative Strength · p. 31
  • Relative strength is characterized by the strong convention’s steady-state basin: its width reflects which initial conditions lead to convergence, while its pull captures stability, recovery, and return rate.The basin of attraction of the strong convention’s steady state characterizes relative strength against its counterpart.Source: Characterizing Relative Strength · p. 31

Prompting

This section describes a repeated two-player game prompt, including its rules, contextual memory, response instructions, answer-first format, and variations used to assess comprehension and consensus stability, alongside supplementary analyses of convention formation and individual bias. It reports that agents generally comprehended the prompts but often treated partners as opponents, and that small biases and memory-dependent strategies can shape population-wide conventions.

Prompt Structure

  • The system prompt specified game rules, memory-based play context, and response formatting, yet agents often undermined a co-player’s payoff as if they were opponents.They sometimes chose actions with negative payoffs to harm the co-player’s accumulated tally.Source: Prompt Structure · p. 31

Output Structure

  • The authors used an answer-first-reason-later format so the final action would be influenced by the input prompt rather than preceding generated reasoning.They note that reasoning text could express and amplify an agent’s biases, making their sources difficult to identify.Source: Output Structure · p. 32

Example Prompt

  • The paper provides example system and user prompts and omits beginning- and end-of-text tokens because they differ by model.Source: Example Prompt · p. 32

System Prompt

User Prompt

Meta-Prompting

  • Prompt-comprehension response accuracy was nearly always above 0.8 and often close to 1; Llama-2-70b-Chat was the only model below 0.8 on any metric, specifically counting conventions within memory range.Source: Meta-Prompting · p. 34

Alternative Prompts

  • Factual bullet-point and expository narrative prompts, tested with randomly generated name pools selected to have no initial bias, still produced biased consensus.The name-pool selection required Jensen-Shannon distance from the neutral no-memory distribution below 0.005.Source: Alternative Prompts · p. 34

Narrative System Prompt

  • The narrative system prompt presents the same-action reward and different-action penalty, then asks the player to examine history and respond with an action and reason.Source: Narrative System Prompt · p. 35

Bullet-point System Prompt

  • The bullet-point prompt says players make their choices simultaneously and states that each player's objective is to maximize their own accumulated point tally.Source: Bullet-point System Prompt · p. 36

Bullet-point System Prompt (Continued)

  • The prompt continuation provides the current round and Player 1’s score, asks for an action using previous rounds, notes dependence on Player 2’s choice, and requests an action-and-reason response.It also records both players’ actions and payoffs from the preceding round.Source: Bullet-point System Prompt (Continued) · p. 37
  • Prompt comprehension was checked with meta-prompting questions over approximately 100 test interactions per model; supplementary tests also examine varied simulation parameters and alternative prompt templates.Figure S1 describes replaying games with memory length H=5 and asking comprehension questions at each interaction; Figures S2 and S12 describe parameter and prompt variants.Source: Figure S1 · p. 38, Bullet-point System Prompt (Continued) · pp. 38, 56, Figure S2 · p. 39
  • Even a small bias toward one name leads to inevitable global convergence in the W=2 naming game, and greater bias accelerates convergence.The theoretical minimal naming game was simulated in 10,000 runs with 24 agents.Source: Bullet-point System Prompt (Continued) · p. 40, Figure S3 · p. 40
  • With W=10, Llama-2-70b-Chat and Claude-3.5-Sonnet are unbiased across conventions, while Llama-3/3.1-70B-Instruct show significant skew.The reported chi-squared p-values are < 0.001, 0.001, 0.100, and 0.410; the passages identify the first two models as skewed and the latter two as unbiased at the 5% significance level.Source: Bullet-point System Prompt (Continued) · p. 42, Figure S5 · p. 42
  • In four W=2 name pools, collective bias produced consensus on M in 40/40 runs, F in 24/40, X in 40/40, and Alice in 40/40.The respective pools were {Q, M}, {F, J}, {X, Y}, and {Alice, Bob}.Source: Figure S6 · p. 43
  • Consensus remains stable for all tested models except Llama-3.1-70B-Instruct, which switches from weak convention Q to strong convention M; M remains stable.The caption reports 4, 10, 3, and 5 runs across panels A–D.Source: Bullet-point System Prompt (Continued) · p. 44, Figure S7 · p. 44
  • For Llama-3.1-70B-Instruct, aggregate probabilities in round 2 initially favor Q, but the bias flips as the game unfolds and the population converges on M.The caption notes that round-2 aggregate probabilities alone would incorrectly predict the final consensus because each memory configuration is initially equally likely.Source: Bullet-point System Prompt (Continued) · p. 47, Figure S10 · p. 47

Related papers

Loading 2410.08948v2…