PaperlayerView exact revision on arXiv

Source-linked AI summary

Population Physics, Population Problems: Safety and Emergence in LLM Societies

Adrian de Wynter

arXiv:2609.33871v1cs.MAcs.CL

TL;DR

Individual-agent methods may not scale to large LLM societies or capture unpredictable collective behaviour that operates at the population level. The paper proposes an agent-agnostic framework based on interaction structure and applies it across five systems. It finds self-organisation in three systems, with relaxation shape covarying with environmental information, while two systems show no such dynamics.

  • Problem

    Individual-agent approaches may not scale to large LLM populations, and language-based interpretations may misrepresent collective behaviour or become outdated as models change.

    Source: §1 · pp. 1–2

  • Method

    The paper measures interaction structure using entropy, functional fits, and a marginal-null baseline, without evaluating model language or internal states.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    Self-organisation appears in LLM-Schelling, Moltbook, and Rogue, with relaxation shape covarying with environmental information, while GovSim and ChatEval show its absence.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Population-level dynamics can differ from individual behaviour, so collective monitoring can reveal coordinated pathologies even when individual agents resist participation.

    Source: §7 · p. 9, §6 · p. 9

    Only Moltbook is a realistic dataset; the remaining systems are simulations, and the chosen activity partition shapes which aspects of dynamics are measured.

    Source: A LIMITATIONS · p. 13

Abstract

from arXiv · show

The collective behaviour of large language model (LLM) societies is not the sum of their individual outputs. It yields statistically distinct, sometimes-unpredictable phenomena, for which the tools we use to study single agents may not scale. Due to recent incidents involving autonomous agentic systems, however, understanding these systems is paramount. For that we introduce a framework for measuring self-organisation in LLM social systems and apply it to three such systems: a Schelling grid, a social network (Moltbook), and a Twitter-like misinformation simulation ('Rogue'). All three exhibit statistically significant self-organisation. Moreover, their relaxation dynamics vary with the environmental information available to the agents, with open-ended systems (Moltbook, Rogue) exhibiting sharp, phase-transition-like dynamics. Further results show that population-level pathologies can emerge even when the LLMs are safety-tuned or monitored, being primarily driven by the coordinated activity of a population subset. We also show when self-organisation does \textit{not} emerge under two additional scenarios (a commons dilemma, GovSim, and a LLM-as-a-judge deliberation scheme, ChatEval). We argue that measuring signatures of this kind offers a lightweight, agent-agnostic diagnostic layer for detecting coordinated collective behaviour in deployed multi-agent systems without relying on natural language or model versioning.

1 INTRODUCTION

Individual-agent analysis may miss collective behaviour that is unpredictable, operates at a different abstraction level, and may arise in populations too large for language-based approaches to scale. The paper proposes an agent-agnostic framework for measuring self-organisation and finds it in some LLM societies, with relaxation patterns and safety-relevant outcomes varying across systems.

1 INTRODUCTION

  • Individual-agent monitoring may miss collective phenomena that operate at another abstraction level or unfold too quickly or slowly to interpret reliably.The introduction cites a swarm incident in which a subset of agents self-organised through communication; it describes the interaction of agents with one another and their environment as a source of unpredictable collective outcomes.Source: §1 · p. 1
  • The paper proposes a complex-systems framework that measures interaction structure using entropy, functional fits, and a marginal-null baseline rather than agents’ language or internal states.It applies the framework to five systems: LLM-Schelling, Moltbook, Rogue, GovSim, and ChatEval.Source: §1 · p. 2
  • Some LLM societies exhibit statistically distinguishable self-organisation, while GovSim and ChatEval do not.Relaxation shape covaries with environmental information: uninformed LLM-Schelling shows multiscale relaxation (β ≈0.4), while open-ended Moltbook and Rogue show phase-transition-like dynamics (β ≳2).Source: §1 · p. 2
  • The framework is designed to characterize population-level pathologies and emergent dynamics while supporting monitoring across rapidly changing model versions without natural-language assumptions.The authors present it as a tool for deployed systems, especially those ‘in the wild’.Source: §1 · p. 2

2 RELATED WORK

Prior work has reported emergent behaviours in LLM social simulations and studied strategic coordination, but the interpretation and measurement of population-level behaviour remain contested. This paper places its contribution in describing system-level dynamics rather than interpreting discourse or strategic content.

LLM Social Simulation

  • LLM social simulations have reported behaviours such as friendship, gossip, planned events, governance, cooperation, and purported Moltbook language invention, while some accounts attribute them to fine-tuning or pattern matching.The literature also includes claims that aggregated discourse is shallow rather than genuinely social.Source: §2 · p. 2
  • Human-oriented vocabulary may obscure whether individual behaviour is the right abstraction for describing a population, motivating analysis of dynamics instead of specific agents.The authors emphasize that such results should remain useful as models and methods become outdated.Source: §2 · pp. 2–3

Multi-Agent Systems

  • Prior work on adversarial multi-agent dynamics and safety studies strategic coordination, whereas this paper characterizes system-level dynamics as a complementary perspective.The authors say Rogue suggests individual safety-tuning can produce resistance among benign agents without preventing population-level pathologies driven by a coordinated subset.Source: §2 · p. 3

Background

  • Complex-systems research models phenomena difficult to explain from isolated components through local interactions, relaxation dynamics, and characteristic states.The cited examples include opinion dynamics, epidemics, and ecosystem collapse models with early-warning signals and phase transitions.Source: §2 · p. 3

3 BACKGROUND

The background uses Schelling’s segregation model to illustrate how simple local rules can produce population-level order, and introduces relaxation-curve shapes as a way to distinguish gradual change from sharp transitions.

3.1 THE SCHELLING MODEL

  • Schelling agents relocate when the share of same-type neighbours falls below threshold τ, yet at τ ≈0.3 the system forms segregated clusters that agents do not explicitly seek.The model’s order parameter Φ(t) summarizes mean agent happiness and rises rapidly before saturating at 1.Source: §3.1 · p. 3
  • Emergence is a population-level property absent from individual components; in Schelling’s model, Φ(t) quantifies configuration-level order despite agents seeking only a happiness threshold.The model serves as a simple baseline for comparing LLM societies.Source: §3.1 · p. 3
  • Phase transitions have a latent period, rapid reorganization around an inflection time, and saturation; trajectory shape can distinguish them from gradual multiscale relaxation.The transition depends on the aggregate configuration rather than an individual agent’s behaviour.Source: §3.1 · pp. 3–4
  • Figure 1 illustrates initial, rapidly reorganizing, and saturated Schelling states alongside Φ(t)’s three-regime trajectory; emergence depends on τ, not agents’ states or preferences.The caption notes that the same run is shown in both the snapshots and the order-parameter curve.Source: Figure 1 · p. 4

3.2 RELAXATION DYNAMICS

  • The framework compares observed Φ(t) trajectories with stretched-exponential and logistic reference forms to distinguish gradual from phase-transition-like relaxation.A stretched exponential describes gradual relaxation; β <1 indicates broadly distributed relaxation timescales, while β >1 can accompany sharper cooperative or cascading dynamics.Source: §3.2 · p. 4
  • The logistic form encodes a well-defined transition point, unlike the stretched exponential’s many-timescale relaxation; best-fit parameters and form support cross-system comparison.Figure 1 shows the three-regime trajectory associated with phase-transition-like dynamics.Source: §3.2 · p. 4, Figure 1 · p. 4

4 METHODS

The methods apply a shared order parameter and validation framework to three LLM societies with distinct activity spaces. They compare relaxation forms and test whether observed trajectories differ from marginal expectations and remain stable under data variations.

4 METHODS

  • The methods characterize three LLM societies through order-parameter trajectories, then compare relaxation forms and check whether apparent self-organisation exceeds measurement noise.The framework is intended to distinguish system dynamics from marginal effects and fitting noise.Source: §4 · p. 4

4.1 LLM SOCIETIES

  • LLM-Schelling agents choose a cell or stay put, with informed agents additionally seeing neighbouring types; the study runs both settings until convergence or a step limit.The informed run included 140 agents over 2,776 ticks, and the uninformed run 240 agents over 19,012 ticks.Source: §4.1 · p. 4
  • Moltbook data cover 21,668 agents across 3,638 submolts and almost 3.5 million posts from March through August 2026, aggregated by submolt and timestamp.The underlying network is Reddit-like and populated by custom-built agents operating through an API.Source: §4.1 · p. 5
  • Rogue simulates misinformation propagation with 300 agents, a configurable malicious fraction, sequential activation, and interactions including posting, liking, reposting, commenting, and following.The primary scenario converged after seven eras with 1.7 million interactions, when an era had no interactions.Source: §4.1 · p. 5

4.2 ORDER PARAMETERS

  • Across the three systems, Φ is mean per-agent Shannon entropy over categorical activity, with low values indicating activity concentrated within the chosen activity space.The partitions are neighbour type for LLM-Schelling, submolt for Moltbook, and post ID for Rogue; Rogue also uses an informational-status partition to measure malicious-content engagement.Source: §4.2 · p. 5

4.3 MEASUREMENT FRAMEWORK

  • The marginal null estimates Φ when agents distribute activity in proportion to category popularity rather than through intra-agent interaction.For Rogue’s bad reach, the null is the network-wide fraction of interactions targeting malicious content, against which benign-agent engagement can be compared.Source: §4.3 · pp. 5–6
  • The framework fits stretched-exponential and logistic forms, reporting best-fit parameters and fit statistics; near-boundary or mismatched stretched-exponential fits can indicate logistic dynamics are more appropriate.The authors prefer one fit over another when ΔAIC > 2.Source: §4.3 · p. 6
  • Self-organisation is validated by separation from the 95% marginal-null band for at least 80% of the window, fit quality of R² ≥0.7, and parameter stability under preprocessing changes.When fit quality is imperfect but shape remains consistent, the authors discuss residual structure and treat parameters as describing bulk relaxation.Source: §4.3 · p. 6

5 RESULTS

Across LLM-Schelling, Moltbook, and Rogue, the measured population dynamics vary in shape, while GovSim and ChatEval remain at fixpoints under the tested conditions. Rogue’s two activity spaces both show phase-transition-like reorganisation, including attention concentrating on malicious content.

5.1 LLM-SCHELLING: RELAXATION AND ENVIRONMENTAL INFORMATION

  • In LLM-Schelling, both information conditions pass V1, but the informed and uninformed trajectories have different relaxation shapes.The figure shows stretched-exponential fits with β = 1.417 for informed agents and β = 0.428 for uninformed agents; the prose reports mean gaps of 0.85 and 0.70, respectively.Source: §5.1 · p. 6, Figure 2 · p. 7

5.2 MOLTBOOK

  • Moltbook’s observed entropy stays outside the null band in 100% of windows, and its best-fit trajectory is logistic (R2 = 0.95).The logistic fit is preferred over the stretched exponential by ∆AIC ≈22, with an inflection time around 15 weeks.Source: §5.2 · p. 7

5.3 ROGUE: INFORMATIONAL SELF-ORGANISATION UNDER ADVERSARIAL PRESSURE

  • Rogue’s post-ID entropy and bad-reach trajectories both fit logistic forms, showing attention concentration and concentration on malicious content, respectively.The post-ID fit has R2 = 0.996; bad reach saturates at 0.28 against an ambient null of 0.33.Source: §5.3 · pp. 7–8
  • Benign bad reach is 0.28 against an ambient null of 0.33, but mutual amplification among malicious agents drives the population-level phenomenon despite safety-tuning.Across the pressure sweep, benign engagement falls from about 100% to 85% of ambient as prop_mal rises from 0.1 to 0.5.Source: §5.3 · p. 8

5.4 ABLATION I: ENVIRONMENTAL INFORMATION SHAPES THE RELAXATION REGIME

  • Across systems, relaxation shape covaries with environmental information; this pattern does not apply to systems that do not relax.The authors contrast the more informative open-internet Moltbook and informed LLM-Schelling with less-informed conditions, while noting the association is not a claim about fixpoint systems.Source: §5.4 · p. 8

5.5 ABLATION II: WHEN DOES SELF-ORGANISATION NOT EMERGE?

  • GovSim and ChatEval show no measurable Φ(t) dynamics under the tested conditions, remaining at fixpoints from the first observation.In GovSim, default and hard experiments converge within one round; in ChatEval, verdict entropies differ by less than 0.02 between rounds 0 and 2.Source: §5.5 · pp. 8–9

6 DISCUSSION

The discussion emphasizes that collective pathologies can arise through coordinated population activity even when individual agents resist harmful engagement. It also distinguishes such emergent dynamics from systems that remain at fixpoints and connects relaxation shape with environmental information.

6 DISCUSSION

  • In Rogue, malicious agents’ mutual coordination drives population-level dynamics even though benign agents resist engaging with malicious content.The coordinated subset need not dominate the population; the authors argue individual guardrails might not prevent harmful system-level structures.Source: §6 · p. 9
  • GovSim and ChatEval remain at fixpoints from the start under the tested conditions, with no system dynamics observed.The authors interpret Φ0 ≈ Φ∞ as agents reaching a fixpoint immediately and remaining there.Source: §6 · p. 9
  • In LLM-Schelling, greater environmental information produces a compressed relaxation closer to the phase-transition-like dynamics observed in Moltbook and Rogue.The discussion presents this as evidence that environmental information shapes system behaviour.Source: §6 · p. 9

7 CONCLUSION

The paper proposes measuring LLM societies through complex-systems dynamics rather than characterizing individual models. Its framework finds self-organisation in three systems, no dynamics in two others, and population-level emergence even when agents refuse coordinated behaviour.

7 CONCLUSION

  • The framework measures self-organisation in LLM societies without evaluating model outputs or characterizing internal state.The authors argue collective behaviour should be characterized at the system level rather than through its components.Source: §7 · p. 9
  • The framework identifies self-organisation in LLM-Schelling, Moltbook, and Rogue, but fixpoint regimes in GovSim and ChatEval.Across the three systems with self-organisation, relaxation shape covaries with environmental information.Source: §7 · p. 9
  • Population-level phase transitions can occur even when individual agents refuse coordinated behaviour, so single-agent monitoring may operate at the wrong abstraction level.The discussion cites an incident report in which some agents declined to participate while the swarm proceeded.Source: §7 · pp. 9–10

A LIMITATIONS

The work relies on one realistic dataset, with its remaining systems being simulations, and the chosen activity partition shapes which dynamics are measured. The authors also report ethical and AI-use disclosures.

A LIMITATIONS

  • The main scope limitation is one realistic dataset, Moltbook; the other systems are simulations.The setup is comparatively small, and further work is needed to characterize information dependence fully.Source: A LIMITATIONS · p. 13
  • The framework does not mechanically identify population-level pathology in a new setting because analysis depends on a chosen activity partition.Different partitions expose different aspects of the same dynamics, so domain expertise enters through partition choice.Source: A LIMITATIONS · p. 13
  • Rogue’s malicious agents used a safety-tuned commercial model that frequently refused misinformation, making the observed effect size a lower bound for less constrained models.The authors caution against treating this as an expected bound without empirical testing of the target system.Source: A LIMITATIONS · p. 13

B ETHICS

  • The authors report releasing prompts, code, and datasets under open licences, while withholding the exact model used for Qwen 3 replications.They state that the framework itself has no potential misuses, while acknowledging possible overlooked issues.Source: B ETHICS · p. 13

C AI USAGE DISCLOSURE

  • Claude Opus-4.8 generated visualization code and assisted with grammar and spelling; authors checked figures and revised the text.Source: C AI USAGE DISCLOSURE · p. 13

D METHODS: FURTHER DETAILS

The methods specify agent initialization for four systems and control the information each agent can access during decisions. They also document prompts and execution hardware.

D.1 INITIALISATION

  • LLM-Schelling uses a 20×20 grid with 65% sparsity, random agent types and temperatures, and GPT-4.1; Phi-4 and GPT-4o appear in the sweep.Rogue starts from the existing post and follower lists and uses GPT-5.6; Qwen 3 is used for its reproduction. GovSim and ChatEval use GPT-4.1.Source: §D.1 · pp. 13–14

D.2 ENVIRONMENTAL CONDITIONS

  • Information conditions vary across systems: Schelling agents see only coordinates or also future neighbours, while ChatEval referees see varying levels and fractions of prior arguments.In GovSim, information levels add previous catches and current proposals; harvest decisions are constrained by current stock.Source: §D.2 · pp. 14–15
  • Rogue agents can post, repost, like, follow, unfollow, comment, or do nothing; each era shuffles agent order and supplies recent followee activity plus five random recommendations.Contexts are clipped to 500 events. Moltbook also permits status checks and image posts.Source: §D.2 · pp. 14–15

D.3 PROMPTS AND HARDWARE

  • Rogue prompts follow Ren et al. (2025) with minor changes; GovSim, ChatEval, and Schelling prompts are available in the repository, and analysis ran on a consumer-grade laptop.GPT-class model calls used the Azure OpenAI API.Source: §D.3 · p. 15

E LLM-SCHELLING: EXTENDED RESULTS

Extended results test the Schelling findings across happiness and substrate settings, then examine Rogue trajectories and scenario variation. The patterns include information- and pressure-dependent collective dynamics.

E LLM-SCHELLING: EXTENDED RESULTS

E.1 LLM-SCHELLING: HAPPINESS AS A SANITY CHECK

  • Happiness reproduces the entropy regime pattern: informed Schelling has stretched-exponential β = 1.18, while uninformed has β = 0.22 despite a marginal logistic AIC preference.Figure 3 reports informed β = 1.182 ± 0.021 (R² = 0.962) and uninformed β = 0.219 ± 0.001 (R² = 0.844).Source: §E.1 · p. 15, Figure 3 · p. 16

E.2 PARAMETER SWEEP

  • For informed Schelling, increasing occupancy lowers β toward the classical regime, while increasing grid size at fixed 0.65 sparsity also generally lowers β.At grid size 10, changing sparsity from 0.35 to 0.65 shifts β from 3.70 to 1.59; at size 20 it shifts from 1.29 to 0.64. The 10→15→20 size sequence at 0.65 sparsity is β = 1.59→0.57→0.64.Source: §E.2 · p. 16, Table 1 · p. 16

F.1 ADDITIONAL PLOTS

  • Rogue’s S7 engagement entropy and bad reach both follow logistic trajectories, with R² = 0.996 and R² = 0.995, respectively.The engagement entropy remains below its null, and benign bad reach is 85% of the ambient rate.Source: Figure 4 · p. 17

F.2 PARAMETER SWEEP

  • The Rogue sweep reports six adversarial scenarios and a no-adversary control whose null is zero, confirming no spurious malicious-content engagement without malicious agents.The control is S4 with prop_mal = 0; Table 2 defines observed and ambient-null bad reach and their ratio.Source: §F.2 · p. 17, Table 2 · p. 17
  • At prop_mal = 0.1, 0.3, and 0.5, Rogue’s ambient null is 0.07, 0.20, and 0.37, while benign-to-ambient ratios fall to approximately 1, 0.93, and 0.85–0.89.The resistance grows with adversarial pressure; Figure 5 depicts positive null−observed deltas and increasing distance below the observed=null diagonal.Source: §F.2 · pp. 17–18, Figure 5 · p. 18

F.3 REPLICATION WITH QWEN 3

The Qwen 3 replication preserves Rogue’s engagement-entropy relaxation shape, while its timescale and bad-reach fit differ from GPT-5. Broader fit comparisons and Schelling results connect relaxation patterns to information conditions.

F.3 REPLICATION WITH QWEN 3

  • Qwen 3 preserves Rogue’s compressed engagement-entropy shape, with β differing by approximately 3% from GPT-5, but its relaxation timescale is about 3.8× slower.The fitted timescales are τQwen = 4,395 and τGPT = 1,151 per-agent steps; bad reach is poorly fit for Qwen 3 (R² = 0.47 versus 0.995 for GPT-5).Source: §F.3 · pp. 18–19

G.1 COMPARISON TABLE

  • Across systems, informed Schelling is decisively stretched-exponential-preferred, while both Moltbook and Rogue trajectories are logistic-preferred; power laws are outperformed throughout.Uninformed Schelling is technically logistic-preferred, but its stretched-exponential β = 0.43 remains in the classical multiscale range.Source: Table 4 · p. 20, §G.1 · p. 20

G.2 RELATIONSHIP TO THE INFORMATION REGIME

  • The fit comparisons link relaxation shape to environmental information: Schelling’s informed and uninformed conditions differ despite sharing the model and underlying trajectories.The broader comparison relates variation in β to information, agency, and relaxation shape, with more information available in Moltbook than in Rogue.Source: §G.1 · p. 20, §G.2 · p. 20

H SENSITIVITY ANALYSIS

Sensitivity tests preserve the phase-transition-like characterization of Moltbook and Rogue, while the formal nulls distinguish self-organisation from marginal popularity alone. GovSim remains at a fixpoint even under harder conditions, and the tested frozen systems show no measurable population-level dynamics.

H SENSITIVITY ANALYSIS

  • Across tested activity thresholds, the logistic form remains preferred for most informed systems, while Moltbook’s and Rogue’s β shape-mismatch signatures persist.Table 6 reports R2 changes within 0.01–0.02 for each system.Source: H SENSITIVITY ANALYSIS · p. 21, Table 6 · p. 21

I NULL BASELINES: FORMAL DETAILS

  • The sampled null preserves agent and category marginals while adding finite-population variance; observed values outside its band indicate preferences beyond marginal popularity alone.The analytic null is the category-popularity entropy, interpreted as expected mean-agent entropy under independent activity drawn from shared marginals.Source: I NULL BASELINES: FORMAL DETAILS · pp. 21–22

J EXTENDED ABLATION RESULTS II: FROZEN SYSTEM DYNAMICS

  • GovSim and ChatEval remain at fixpoints from the outset, with no measurable population-level dynamics under the tested conditions.The reported order parameters are indistinguishable between initial and final observations except for one GovSim restriction.Source: J EXTENDED ABLATION RESULTS II: FROZEN… · p. 22, Table 7 · p. 23

J.1 GOVSIM: FULL SWEEP

  • Hard GovSim trajectories remain effectively flat despite induced stock collapse; 79.2% of 144 fitted trajectories are dropped, and most remaining fits pin to fitter boundaries.This rejects the explanation that the default setup was simply too easy to produce dynamics.Source: §J.1 · p. 23

J.2 CHATEVAL: FULL SWEEP

ChatEval deliberation leaves verdict entropy and accuracy essentially unchanged, with disagreement patterns stable across runs. Separately, Moltbook search interest peaks around mid-February before waning and stabilizing.

J.2 CHATEVAL: FULL SWEEP

  • Verdict entropy changes by less than 0.02 bits from round 0 to round 2, while median referee accuracy stays 0.750; deliberation moves neither measure.Consensus is reached before deliberation begins.Source: §J.2 · p. 23
  • Of 21 items with non-unanimous verdicts, 20 show a persistent 3-1 split and one a persistent 2-2 deadlock, with no observed transitions between patterns.For a given item, the same referee persistently dissents across runs, suggesting item-specific prior alignment rather than a stochastic deliberation outcome.Source: §J.2 · pp. 23–24, Table 9 · p. 24

K MOLTBOOK POPULARITY TRENDS

Related papers

Loading 2609.33871v1…