PaperlayerView exact revision on arXiv

Source-linked AI summary

Indirect tipping: a social attack surface in AI agent populations

Ariel Flint, Luca Maria Aiello, Sara M. Constantino, Romualdo Pastor-Satorras, Andrea Baronchelli

arXiv:2609.25194v1cs.MAcs.AIcs.CYeess.SYphysics.soc-ph

TL;DR

Output validity alone does not establish whether an LLM understood task rules. The paper models alternative-convention output by memory state and examines staged transitions, finding that routes through intermediate conventions can lower transition costs and enable shifts direct competition cannot produce.

Abstract

from arXiv · show

As generative AI agents are deployed at scale, safety will depend not only on technical safeguards and individual model design, but also on collective equilibria that determine how agent populations process information, prioritize actions, and respond to uncertainty. Yet the same equilibria that enable agents to coordinate also create a social attack surface. The standard framework to assess this vulnerability is critical mass dynamics: the minimum fraction of adversarial agents required to overturn an equilibrium through direct competition. Here, we show that this approach risks underestimating system vulnerability by reducing the problem to the identification of singular tipping points, and ignoring indirect but potentially more efficient routes through which collective behavior can be redirected. Through experiments with populations of LLM agents and an analytic framework that captures their collective dynamics at scale, we map critical-mass thresholds that define a directed, weighted topology over the space of coordination equilibria, and treat this topology as a navigable landscape. We show that indirect tipping through intermediate stepping-stone equilibria can reduce the committed minority required to reach an alternative state, bypass majority requirements, and make possible transitions inaccessible through direct challenges. The diversity of available alternatives and timing of the attack further reshape this landscape, creating opportunities for control as well as risks of unintended destabilization. These results show that an equilibrium's resistance to committed intervention is not an intrinsic property but a structural feature of its competitive relations with alternative states. Securing populations of interacting AI agents therefore requires mapping this social landscape alongside individual agent capabilities and the technical channels through which they interact.

Generative agent-based model

The study adapts the naming game, a model of decentralized self-organization, into a generative agent-based model in which LLMs make decisions during repeated interactions. Agents coordinate on one of several equally valid words.

Generative agent-based model

  • The naming game models self-organization in decentralized populations through peer-to-peer interaction and has been used to study consensus formation, norm change, and critical-mass dynamics.Its dynamics have been characterized theoretically and studied experimentally with human participants and generative AI agents.Source: Generative agent-based model · p. 3
  • The model uses an LLM as an agent’s decision-making process within an established protocol for measuring critical mass in LLM multi-agent systems.The study describes populations converging on one of several possible, equally valid words.Source: Generative agent-based model · p. 3
  • In each trial, two randomly selected agents in a homogeneous, fully connected population simultaneously transmit a word from the available options under full anonymity.Choices use a horizon-limited memory of recent signals and observations, supplied to the LLM through a standardized prompt; its output token is the agent’s decision.Source: Generative agent-based model · p. 3
  • Committed agents transmit the same unique word each interaction, and consensus occurs when all interactions repeatedly result in successful coordination on one word.The LLMs were screened through a meta-prompting validation procedure intended to verify task comprehension.Source: Generative agent-based model · p. 4
  • The study tests homogeneous populations of Qwen, Phi, Llama, and DeepSeek instruction-tuned open-weight models.These are Qwen2.5-7B-Instruct, Phi-4, Llama-3.2-3B-Instruct, and DeepSeek-R1-Distill-Qwen-32B, respectively.Source: Generative agent-based model · p. 4

Results

The Results examine how critical-mass thresholds and indirect transition costs depend on competing conventions, population characteristics, the structure and availability of alternatives, and timing of commitment. They trace how asymmetric competition and shifting consensus basins shape routes to target or unintended conventions, including routes that are cheaper or inaccessible through direct transitions.

Predicting critical mass for large populations

  • Critical mass varies systematically with population size, LLM, and competing conventions, then converges beyond a characteristic scale to a stable asymptotic value.As populations grow, finite-size effects weaken and collective selection preferences from agents’ memory distributions increasingly control the dynamics.Source: Predicting critical mass for large… · p. 4
  • In small populations, correlated agent memories make outcomes sensitive to stochastic decisions and interactions, producing substantial run-to-run variation in critical mass.Across intermediate sizes, thresholds may rise or fall before converging to a size-independent value.Source: Predicting critical mass for large… · p. 4
  • In DeepSeek populations, shifting consensus from VI to V takes a 3% committed minority, while the reverse requires more than 63%.The advantage is pair-dependent: some convention pairs have more balanced thresholds, and dominance can vary across model populations.Source: Predicting critical mass for large… · p. 6
  • Some direct transitions are trivial or impossible, while in Llama populations both consensus states in some pairs collapse without yielding stable consensus on the alternative.In DeepSeek populations, VIII shifts spontaneously to II, but no committed group can drive the reverse transition.Source: Predicting critical mass for large… · p. 6
  • Critical mass tracks where the challenger’s basin reaches its maximum along a straddle trajectory: increasing commitment contracts the incumbent basin and expands the challenger basin.Pairs differ in how rapidly the basin boundary shifts, explaining their different commitment thresholds.Source: Predicting critical mass for large… · p. 6
  • Because robustness is relational between convention pairs, a state difficult to reach directly may be reachable through accessible intermediate conventions.The specific viable paths and their costs can shift with population size as thresholds change.Source: Predicting critical mass for large… · pp. 5–7

Indirect pathways reduce transition costs

  • A single intermediate can reduce the commitment needed for a transition and make feasible routes around majority-level or impossible direct challenges.Fewer than 10% of single-intermediate paths are sub-additive. Paths are also assessed by maximum per-stage commitment, duration, stage count, and operating cost.Source: Figure 2 · p. 7, Indirect pathways reduce transition… · p. 8
  • Cost-reducing indirect routes are rarer than in randomized landscapes, where additive transitivity is prevalent and advantageous paths are exceptions to a coherent convention ordering.The maximal criterion follows the same pattern, with cost-effective paths arising from rare violations of the dominance hierarchy.Source: Indirect pathways reduce transition… · p. 8
  • Gini scores indicate that leverage concentrates on a small number of nodes and edges; a gateway can open efficient access to costly targets, though sub-maximal bridge scores are near-null in Qwen populations.A single bridge can carry a large share of effective multistage interventions.Source: Indirect pathways reduce transition… · p. 9

Choice diversity reshapes the transition landscape

  • Adding an available third convention preserves the ordering of transition difficulty while compressing threshold magnitudes, and most reduces commitment requirements for transitions hardest in isolation.A cost-minimizing unpromoted option can lower the invasion threshold without directly advancing an intermediate consensus.Source: Choice diversity reshapes the… · p. 10
  • The integrated single-stage diversity route yields roughly 2–3 times as many cost-effective transitions as the isolated two-stage route, raising prevalence from 12–23% to 15–40% across model populations.Among triplets where the isolated route already undercuts direct transition, the integrated route remains cheaper in roughly one-third of cases across populations, and approximately two-thirds in Phi.Source: Choice diversity reshapes the… · pp. 10–11

Transient commitment enables staged tipping

  • Transient commitment can reduce committed activity duration in 66-84% of cases across all model populations studied, while premature withdrawal in 3-13% of cases diverts outcomes toward a non-target latent alternative.Withdrawing the committed group too early can destabilize the decision history and allow another convention to become entrenched.Source: Transient commitment enables staged… · p. 11
  • In a Qwen example, briefly promoting an intermediate before switching to the target reduced the committed group size by 3% of the population versus direct triadic tipping, but effective switching windows occurred in fewer than one in five transitions.Such windows were only very rare in Llama populations.Source: Transient commitment enables staged… · p. 11
  • Timing remains critical: promoting an intermediate too long may entrench it, requiring an even larger critical mass to overturn; this risk occurs in approximately 40–50% of switching windows in other model populations, but no evidence was found in Llama populations.Source: Transient commitment enables staged… · p. 12
  • The fragment ends with “of a single stage even further,” but does not identify the quantity or process being described.No additional result can be summarized from this passage alone.Source: Transient commitment enables staged… · p. 13

Discussion

The discussion frames convention change as a path-dependent landscape: intermediate routes can make costly or directly inaccessible shifts attainable. It highlights strategic and defensive implications while bounding generalization to simplified settings.

Discussion

  • Critical mass is relational: the amount required to overturn an incumbent depends on the competing alternatives and their pairwise relations.The thresholds together define a directed network of possible transitions.Source: Discussion · p. 13
  • Multi-stage routes through intermediate conventions can sharply reduce transition costs and achieve shifts that direct competition cannot.The authors describe these dynamics as path dependent.Source: Discussion · p. 13
  • A small number of conventions recur as gateways, and a small number of transitions as bridges, giving them disproportionate influence over the landscape.As the space of equilibria grows, the number of potential paths also grows, making low-cost gateways especially valuable to control.Source: Discussion · p. 13
  • Available alternatives reshape transition barriers, and intervention timing affects which convention ultimately takes hold.Transient promotion of an intermediate can lower a barrier, while poorly timed promotion can entrench an unintended convention.Source: Discussion · p. 13
  • The reported patterns recur across LLM architectures and convention sets, rather than being peculiar to one model or set of alternatives.The discussion also notes that exact threshold values vary across LLM populations.Source: Discussion · p. 13
  • Direct-only assessments risk understating exposure, so security assessments should examine desirable and manipulation-resistant equilibria alongside individual-model safety.The authors identify intermediate conventions as a route for overturning a convention that resists a direct challenge.Source: Discussion · p. 13
  • The same path-based reasoning could steer a population from an undesirable equilibrium through an intermediate convention toward a safer one.Source: Discussion · p. 13
  • The study examines finite convention sets and payoff-equivalent alternatives, and uses homogeneous populations under controlled conditions rather than heterogeneous, evolving, partially aligned systems.How the mechanisms operate in naturalistic and hybrid human-AI environments remains to be established.Source: Discussion · p. 14

Methods

The methods model anonymous local coordination among LLM agents and use memory-conditioned choice policies for scalable simulation, assessing prompt comprehension and critical mass through controlled overturning tests and pairwise convention contests. They represent sequential changes as weighted paths between consensus states and use path-cost metrics to distinguish routes that improve on direct transitions and identify where such routes concentrate.

The naming game protocol

  • Agents anonymously choose from a finite label set, earning rewards for coordination without any label having a payoff advantage.Each agent keeps a finite interaction memory of capacity H = 3; committed agents repeatedly choose a fixed label and do not update their memories.Source: The naming game protocol · pp. 14–15

Large language models as interactive agents

  • In the framework, an LLM makes each agent’s word choice at every interaction from a text prompt built from static game rules, recent interaction-history memory, and the required output format.Source: Large language models as interactive… · p. 15

Meta-prompting: screening assessment for LLM participants

  • Meta-prompting tests comprehension of game rules, history sequence, and payoff statistics; studied models generally perform strongly but often struggle to count convention choices in memory.The assessment generates 100 agents with random memories and poses all possible comprehension questions to each.Source: Meta-prompting: screening assessment… · pp. 16–17

Policy extraction for scalable simulations

  • For scalable simulations, the authors probe LLM next-token probabilities to extract a probability distribution over allowed labels for each memory state.They precompute policies for memory states up to H = 3; the extracted policy depends on the memory state, convention set, and specific LLM.Source: Policy extraction for scalable… · p. 17

Measuring critical mass

  • Critical mass is estimated by binary-searching committed-group size in isolated incumbent–challenger trials, checking for challenger consensus within up to 1,000 population rounds.Artificially prepared histories start non-committed agents at incumbent consensus, allowing tests of otherwise unlikely intermediate states and fixing the initial condition for transitions from each convention.Source: Measuring critical mass · pp. 17–18
  • Critical-mass comparisons hold the starting memory configuration fixed, so differences across transitions reflect the competing conventions and can be compared reliably.The number of states meeting the convergence criterion may vary.Source: Measuring critical mass · p. 18

Coordination competition space

  • The convention set uses Plutchik’s eight basic emotion categories; validation repeats experiments with colors, shapes, and Latin letters, with qualitative results reported as robust across sets.Unless otherwise stated, each transition isolates two competing conventions (|W| = 2).Source: Coordination competition space · p. 18

Multi-stage transitions and path metrics

  • Sequential change is modeled as isolated pairwise stages: a committed group overturns an incumbent consensus, and the resulting consensus becomes the next incumbent.Each stage starts from artificially ensured consensus; its directed graph edge weight is the critical-mass cost for that ordered convention pair, independent of the path embedding.Source: Multi-stage transitions and path metrics · p. 18
  • Additive path cost sums stage costs, while maximum cost takes the most demanding stage; an indirect path improves on a direct transition when its chosen cost is lower.Unachievable transitions receive critical mass 1 and other costs infinity; additive improvement implies maximal improvement, but not conversely.Source: Multi-stage transitions and path metrics · pp. 18–19
  • Path indices quantify improving-route fractions and transitivity; a weight-preserving randomized null separates relational structure from cost distribution, while improving-path betweenness yields gate and bridge scores.The sub-additivity and sub-maximality indices assess improvement under additive and maximum costs; additive and maximal transitivity indices assess cost consistency and deviation from a dominance hierarchy.Source: Multi-stage transitions and path metrics · p. 19
  • Gate and bridge scores use betweenness over improving paths rather than shortest paths to measure where cost-reducing routes concentrate.The gate score applies to conventions and the bridge score to routes.Source: Multi-stage transitions and path metrics · p. 19

Funding

The authors acknowledge financial support from the Carlsberg Foundation, a Spanish research project, and the Generalitat de Catalunya.

Funding

  • L.M.A. acknowledges support from the Carlsberg Foundation through the COCOONS project (CF21-0432).Source: Funding · p. 19
  • R.P.-S. acknowledges support from project PID2022-137505NB-C21/AEI/10.13039/501100011033/FEDER UE.Source: Funding · p. 19
  • R.P.-S. also acknowledges the Acadèmia d’Excel·lència Award, funded by the Generalitat de Catalunya.Source: Funding · p. 19

Supplementary information: supplementary text

This supplementary section extends the transition-landscape analysis across additional convention sets and specifies methods for examining committed minorities, challenger basin geometry, and direct and indirect transition costs. It also defines and analyzes efficient indirect routes, including how conventions and transitions function as stepping stones and how their leverage relates to relational structure and cost distributions.

Additional convention sets

  • Asymmetric transition landscapes enabling indirect transitions also appear for additional shape, color, and Latin-letter convention sets.The tested mappings include six shapes, six colors, and eight letters.Source: Additional convention sets · p. 24

Mean-field formulation

  • The mean-field formulation represents committed attackers as a frozen state that biases non-committed agents, and finds critical mass by binary-searching for stable overturning.Overturning requires the alternative convention’s average output probability across the previous three rounds to reach at least 98%.Source: Mean-field formulation · pp. 26–27

Basins of attraction

  • The basin-extent estimator probes the basin boundary along the segment between incumbent and challenger consensus states instead of mapping the full high-dimensional simplex.It iterates sampled initial conditions to their mean-field fixed points under a committed minority.Source: Basins of attraction · p. 29
  • At the critical mass, the challenger basin expands across the entire incumbent–challenger segment, and basin extent closely tracks mean-field critical mass.The text interprets this correspondence as evidence that critical-mass variation across convention pairs is tied to how committed groups reshape basin geometry.Source: Basins of attraction · p. 30

Transition cost graph and path costs

  • Every path that improves on a direct transition under additive cost also improves under maximum cost, but the converse does not hold.The graph assigns nonnegative commitment costs to directed convention-to-convention edges; path costs can be composed additively or by the costliest stage.Source: Transition cost graph and path costs · pp. 30–31

Null model

  • The null ensemble preserves the empirical transition-cost distribution while independently reassigning weights to edges, and excess observables compare measurements against this baseline.The text estimates null observables from 5000 independent resamplings.Source: Null model · p. 33

Null expectations for a maximally random transition cost graph

  • For a maximally random reference, the text models positive edge costs with an exponential distribution matched to the empirical mean, making path-weight sums Erlang-distributed.The exponential is described as the maximum-entropy distribution on nonnegative values subject only to a fixed mean; the Erlang result is cited as.Source: Null expectations for a maximally… · p. 33

Gate conventions and bridge transitions

  • To locate transitivity violations, the analysis measures convention and transition participation in cost-reducing routes using centralities over efficient paths rather than shortest paths.Efficient paths are simple directed routes whose cost improves on the direct transition under the chosen additive or maximum criterion.Source: Gate conventions and bridge transitions · p. 35
  • Efficient paths are sub-additive routes, while source–target pairs with σ_st = 0 have no advantageous indirect route and are excluded from the scores.Source: Gate conventions and bridge transitions · p. 35
  • A convention’s gate score is its node betweenness over efficient paths, identifying states that recur as stepping stones on cost-reducing routes.High-scoring states are easy to reach from many sources and well placed to destabilize many targets.Source: Gate conventions and bridge transitions · pp. 35–36
  • A directed transition’s bridge score is edge betweenness over efficient paths, indicating how much it carries effective multi-stage interventions.Source: Gate conventions and bridge transitions · p. 36
  • Gini coefficients quantify concentration in gate and bridge scores, while comparison with null-ensemble means isolates concentration attributable to relational structure beyond the cost distribution.Values near zero indicate evenly distributed leverage; values near one indicate concentration on a few conventions or transitions.Source: Gate conventions and bridge transitions · p. 36

Supplementary information: supplementary tables

The supplementary tables document prompt-comprehension checks, with one table listing question templates and another reporting model success rates.

Supplementary information: supplementary tables

  • One example asks, “What is player X’s current total payoff?”Source: Supplementary information:… · p. 37
  • Table S1 lists templates of prompt-comprehension questions used to verify whether LLMs understand the prompt.Source: Table S1 · p. 37
  • Table S2 summarizes prompt-comprehension success rates over 100 random agent instantiations for each LLM.Source: Table S2 · p. 37

Supplementary information: supplementary figures

The supplementary figures examine critical-mass thresholds and their landscape across convention pairs, alongside population-size, basin-geometry, and path-cost analyses.

Supplementary information: supplementary figures

  • Larger populations make critical-mass dynamics more deterministic, and critical mass approaches the mean-field prediction as population size grows.The simulation–mean-field difference approaches zero with population growth across the models and transitions shown.Source: Figure S1 · p. 38, Figure S2 · p. 38, Figure S3 · p. 39
  • Figure S4 reports pairwise critical-mass thresholds for transitions among emotion conventions, with tiles labeled as population percentages.Source: Figure S4 · p. 39
  • Figures S5–S7 map critical-mass relations for the color, shape, and letter convention sets.Source: Figure S5 · p. 40, Figure S6 · p. 40, Figure S7 · p. 41
  • Across all word pairs and models, critical mass coincides with the committed convention’s basin of attraction reaching its maximum extent.In the Phi-agent convention pair, the basin expands with committed-group density until it spans the full state space.Source: Figure S8 · p. 41
  • Figure S9 compares four transition-path order parameters across four LLM populations against maximally random null expectations.The empirical additive transitivity curves nearly collapse because additive transitivity is independent of mean weight, not because of a structural difference between populations.Source: Figure S9 · p. 42
  • With a third convention available, transition-cost rank order is preserved, while costs are lower than in pairwise competition.Source: Figure S10 · p. 42
  • The effective-window-size histograms show the distribution of periods when switching commitment can reduce critical mass relative to a direct transition.Window sizes are measured for the smallest critical mass at which a switching transition works.Source: Figure S11 · p. 43

Related papers

Loading 2609.25194v1…