PaperlayerView exact revision on arXiv

Source-linked AI summary

Emergent Discrimination and Polarization in Multi-Agent Systems

Nestor Caticha, João Henrique Sant'Ana, Felippe Alves, Davi Bastos Costa, Mariana Mercucci, Juan P. Neirotti, Otavio Citton, Renato Vicente

arXiv:2610.22701v1cs.MAcond-mat.dis-nnphysics.soc-ph

TL;DR

The paper studies how repeated interactions among learning agents can generate polarization and discrimination. It models adaptive opinion and trust updates, adds group-conditioned bias, and uses order parameters to characterize resulting collective states. Simulations produce four distinct states, with the outcome depending on bias strength and prevalence.

  • Problem

    The paper asks how repeated interactions among learning agents generate polarization and discrimination, a population-level behavior that isolated-agent evaluation may not reveal.

    Source: §1 · pp. 1–2

  • Method

    The model combines Bayesian-derived pairwise opinion and trust updates with group-conditioned shifts in perceived agreement, then characterizes collective states using order parameters.

    Source: §1 · p. 2

  • Results

    Simulations identify four collective states, and which state emerges depends jointly on bias strength and prevalence.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support evaluating multi-agent systems at the population level because collective failure modes may not appear in agents viewed alone.

    Source: §5.1 · p. 9, §5.2 · p. 9

Abstract

from arXiv · show

We study theoretically how discrimination and polarization can emerge in a population of learning agents. Each agent uses an adaptive linear decision layer to map a language-model embedding of an issue to a binary opinion and maintains an evolving estimate of how much it trusts every other agent. Through pairwise interactions, agents update both their decision weights and trust estimates using an online Bayesian rule derived from teacher-student learning over a noisy channel. In-context-learning experiments with large language models qualitatively reproduce the predicted opinion and trust updates. We then assign each agent one of two arbitrary group labels, A or B, and model group bias by allowing a fraction $f_b$ of agents to shift its perceived agreement with each message by a strength $b$, a bias steering prejudice, depending on whether the sender and receiver share a label. We use order parameters from statistical mechanics to connect these local dynamics to collective states and transitions. Starting from uniformly random initial conditions, simulations reveal a rich phase diagram in the $(b,f_b)$ plane, comprising four collective states: spin-glass frustrated out-group favoritism, nondiscriminatory polarization, discrimination with opinion alignment, and discrimination without opinion alignment. Which state emerges depends on both the strength and prevalence of the bias. These results show how simple local learning dynamics can generate qualitatively distinct forms of polarization and discrimination, underscoring the need for population-level safety analysis of multi-agent systems. They also suggest measurable order parameters for characterizing analogous collective states in human societies.

1 INTRODUCTION

The paper asks how repeated interactions among learning agents can produce population-level polarization and discrimination, then connects local opinion-and-trust learning to four collective states shaped by group bias.

1 INTRODUCTION

  • Agents update opinion weights and estimates of distrust during pairwise exchanges using an online Maximum Entropy rule derived from noisy-channel teacher–student learning.The agenda comprises yes/no questions represented by embeddings, which a linear decision layer maps to binary opinions.Source: §1 · p. 2
  • A fraction fb of receivers shifts perceived agreement by strength b according to whether the sender shares its arbitrary group label.The labels contain no information about the issues.Source: §1 · p. 2
  • Simulations identify four states—spin-glass frustrated out-group favoritism, nondiscriminatory polarization, discrimination with opinion alignment, and discrimination without it—whose emergence depends on bias strength and prevalence.Figure 1 maps the states over bias strength b and biased-agent fraction fb.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper links a Bayesian-derived local update rule, its qualitative correspondence with language-model in-context learning, and the collective states produced by repeated interactions.It argues that isolated-agent audits may miss population-level organization into polarization, class-correlated polarization, or frustration.Source: §1 · p. 2

2 RELATED WORK

The paper situates its model among work on language-model societies, multi-agent safety, and statistical mechanics, emphasizing its use of adaptive agents and optimal online learning to study how bias shapes collective states.

Language-model societies

  • Unlike models that use Ising variables, this work gives agents adaptive weights and dynamical affinity estimates, allowing agreement on some issues and disagreement on others.Affinity evolves through online Bayesian and Maximum Entropy learning, without a fixed communication graph.Source: §2 · p. 3

Multi-agent safety and bias

  • Prior work documents individual-model bias and in-group favoritism; this paper asks which collective states such bias produces and which population quantities identify them.The cited examples include arbitrary group assignments and judges favoring their own generations.Source: §2 · p. 3

Statistical mechanics of interacting agents

  • Where many statistical-mechanics models postulate interaction rules, this work derives interactions from optimal online learning and adds group-steering bias to study resulting collective states.The framework builds on societies of neural-network agents that develop polarization, trust structure, frustration, and glassy dynamics.Source: §2 · p. 3

3 LEARNING AGENTS

The section defines adaptive opinion and distrust states, derives their pairwise learning updates, tests predicted updates in language-model experiments, and introduces group bias as a shift in perceived agreement.

3.1 AGENT STATE AND LEARNING DYNAMICS

  • An agent’s opinion is a linear readout of an issue representation, while its directed distrust estimate is maintained separately for every other agent.Opinion weights, distrust estimates, and their uncertainties form the agent’s modeled state.Source: §3.1 · p. 4
  • Pairwise updates treat the emitter’s opinion as a noisy report and project the Bayesian belief back into a Gaussian family by maximum entropy.The evidence is low when the receiver agrees with a distrusted emitter or disagrees with a trusted one; update modulations are strongest in a less-certain sector.Source: §3.1 · pp. 4–5

3.2 OPINION AND TRUST UPDATES UNDER IN-CONTEXT LEARNING

  • The section tests the opinion and trust modulation factors in two independent in-context-learning experiments, using separate readouts for the two updates.Together, the experiments qualitatively reproduce both modulation functions up to one fitted scale per experiment.Source: §3.2 · p. 5, §3.2.2 · p. 6

3.2.1 OPINION UPDATE

  • Across all conditions, the measured update had rank correlation rs = 0.90 with Fw and the same sign in 94.7% of cases.One global positive scale αw was fit across the experiment.Source: §3.2.1 · p. 6

3.2.2 TRUST UPDATE

  • In the trust experiment, changes in trust correlated with Fµ at r = 0.85, with 86.0% sign agreement across all measurements; ∆hµ was compared with Fµ after fitting one global positive scale.Source: §3.2.2 · p. 6

3.3 PREJUDICE: GROUP BIAS AS A SHIFT IN PERCEIVED AGREEMENT

  • A biased receiver shifts perceived agreement by b for same-class senders and by −b for different-class senders, independent of the issue.Positive b yields in-group tolerance and out-group intolerance; negative b reverses that pattern and shifts the trust-update separatrix.Source: §3.3 · p. 7

4 MULTI-AGENT STATES

The simulations connect pairwise opinion, trust, and class relationships to four collective states whose emergence depends on bias strength and prevalence. Agenda complexity preserves the same phase arrangement but changes its boundaries and extent.

4 MULTI-AGENT STATES

  • In the no-bias case, agents’ opinions polarize because agreement and trust are coupled in the repeated interaction dynamics.The simulations vary the bias strength and the fraction of biased agents from random initial conditions.Source: §4 · p. 7

4.1 ORDER PARAMETERS

  • The order parameters average pairwise opinion overlap, signed trust, and class overlap to measure opinion–trust, trust–class, and opinion–class alignment.They average products over unordered distinct-agent pairs; positive and negative values indicate alignment and reverse alignment, while zero alone does not establish independence.Source: §4.1 · p. 7

4.2 THE PHASE DIAGRAM

  • The phase diagram contains four states: frustrated out-group favoritism, nondiscriminatory polarization, discrimination with opinion alignment, and discrimination without opinion alignment.The states emerge as bias strength and prevalence vary; Figure 5 maps the three alignment order parameters across the (b, f_b) plane.Source: §4.2 · p. 8
  • For b < 0 with sufficiently prevalent bias, agents favor the out-group, producing frustrated trust relations and incoherent opinions.Trust–class alignment is negative while opinion–class and opinion–trust alignment are approximately zero.Source: §4.2 · p. 8
  • When b ≈ 0 or f_b ≈ 0, agents form opinion-and-trust camps independent of class, yielding nondiscriminatory polarization.Opinion–trust alignment is positive while trust–class and opinion–class alignment are approximately zero.Source: §4.2 · p. 8
  • With sufficiently prevalent positive bias, discrimination can either align opinions with class boundaries or leave class-linked opinions weak.In the aligned state, agents tend to trust their own class and hold opinions opposed to the other class; without opinion alignment, trust follows class while opinions do not systematically differ by class.Source: §4.2 · p. 8
  • Agenda complexity preserves the four phases and their arrangement, but changes their boundaries and relative extent.The simple agenda uses α = 0.17 and the complex agenda α = 3.3; for α < 1, a conserved orthogonal opinion component limits possible opinion alignment.Source: §4.2 · p. 9

5 DISCUSSION

The discussion argues that locally optimal learning can produce undesirable collective outcomes, so isolated-agent audits cannot determine population safety. It connects the mechanism to possible language-agent bias and proposes empirical tests, robustness studies, and interventions.

5.1 INDIVIDUAL OPTIMALITY DOES NOT IMPLY COLLECTIVE SAFETY.

  • A learning rule optimal for one noisy teacher–student interaction can produce polarization without group bias and discrimination when an irrelevant label shifts perceived agreement.Pure unoptimized Hebbian learning does not lead to discrimination in this model.Source: §5.1 · p. 9

5.2 SINGLE-AGENT AUDITS CANNOT DETERMINE THE COLLECTIVE STATE.

  • Estimating bias in isolated agents cannot identify the population’s state, which also depends on bias prevalence and the agenda.The authors propose population-level order parameters and audits to detect collective polarization and discrimination.Source: §5.2 · p. 9

5.3 IN-CONTEXT LEARNING AND COLLECTIVE BIAS.

  • Language-model self-preference could potentially act like group bias, and interaction could amplify mild asymmetries into a discriminatory phase in mixed human–AI systems.The paper presents this as a possibility, not an established outcome.Source: §5.3 · p. 9

5.4 FUTURE DIRECTIONS

  • Future work includes testing the phases in language-agent and mixed human–AI populations, studying robustness and transition signals, and exploring interventions on interaction or population structure.Suggested control targets include routing, source-reliability calibration, memory, and population composition.Source: §5.4 · p. 10

6 CONCLUSION

The paper studies how a teacher–student learning rule behaves in a population and finds that interaction can yield polarization or discrimination under irrelevant group bias. It concludes that multi-agent safety requires population-level measures of the collective states produced by interaction.

6 CONCLUSION

  • The population polarizes without group bias and can enter discriminatory states when an irrelevant group label shifts perceived agreement.Which state emerges depends on bias strength, prevalence, and agenda; in-context-learning experiments qualitatively reproduce the opinion and trust modulation functions.Source: §6 · p. 10

AI USE STATEMENT

The authors used generative AI tools to assist with manuscript preparation, literature discovery, and research code, while retaining responsibility for the final content.

AI USE STATEMENT

  • AI tools assisted with writing, literature discovery, and portions of the research code; the authors tested the assisted code and independently checked the phase-diagram code.The authors state that the tools did not contribute to research ideation and that they reviewed and edited all AI-assisted material.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The paper studies polarization and discrimination in learning-agent societies, which it identifies as phenomena with clear ethical implications.

ETHICS STATEMENT

  • The work examines polarization and discrimination in societies of learning agents as ethically consequential phenomena.The authors describe the model as deliberately general, aiming to isolate mechanisms that may recur across such societies.Source: ETHICS STATEMENT · p. 10
  • The authors frame understanding these phenomena as a first step toward acting against them.Source: ETHICS STATEMENT · p. 10
  • The statement presents the model as a way to isolate mechanisms that may recur across learning-agent societies.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper provides materials and documentation intended to support reproduction of its results and figures.

REPRODUCIBILITY STATEMENT

A DERIVATION OF THE LEARNING ALGORITHM

The appendix derives the online opinion and trust updates through Bayesian inference with Gaussian moment matching, then reports language-model experiments and a broader group-bias analysis. The experiments qualitatively test the predicted update functions, while the broader field produces additional collective states beyond those in the main-text bias slice.

A DERIVATION OF THE LEARNING ALGORITHM

  • Gaussian moment matching approximates the non-Gaussian Bayesian posterior, yielding online opinion and trust updates from derivatives of the log evidence.The opinion update includes a Hebbian σ_e x term with a tensorial, self-annealing rate; the trust sector has a parallel update.Source: A DERIVATION OF THE LEARNING ALGORITHM · pp. 13–14

B OPINION AND TRUST UPDATES UNDER IN-CONTEXT LEARNING

  • The in-context-learning experiments qualitatively reproduce the predicted opinion and trust updates, each up to one fitted scale.The opinion experiment uses fictional material to control conviction; the trust experiment elicits probabilities over real, low-valence themes.Source: B OPINION AND TRUST UPDATES UNDER… · p. 14

B.1 OPINION: Fw BY NULLING

  • In the opinion experiment, four supplied evidence pieces are recovered as 3.6–4.1 pieces, and inferred testimony weights correlate with the five stated records at r = 0.99.The measured change is compared with the predicted modulation function using one fitted positive scale; the result concerns one model at one reasoning effort.Source: §B.1 · p. 15

B.2 TRUST: Fµ BY ELICITED RELIABILITY

  • For the trust experiment, Equation (5) predicts Δhμ proportional to Fμ(hw, hμ) up to an unobserved trust susceptibility; one global positive scale αμ was fit across 712 updates, with sign agreement and correlation independent of that scale.Source: §B.2 · p. 16

C THE GENERAL GROUP-BIAS FIELD

  • In the general field, the uniform component yields widespread distrust, ordinary polarization, or consensus, while the receiver-class component shifts polarization toward class-linked credulity.Figure 6 shows slices in the (a, f_a) and (c, f_c) planes; its colors encode order parameters, and these slices include states absent from the (b, f_b) plane.Source: C THE GENERAL GROUP-BIAS FIELD · p. 16, Figure 6 · p. 17

D AN UNBIASED POPULATION POLARIZES INTO TWO CAMPS

In an unbiased run, pairwise opinion overlap and trust separate into two opposing, internally coherent camps through interaction rather than group labels. The simulations also check stability across population sizes and interaction durations.

D AN UNBIASED POPULATION POLARIZES INTO TWO CAMPS

  • At b = 0, opinion overlap and trust develop positive and negative peaks, showing aligned, mutually trusting camps and opposing, distrustful camps formed through interaction, not class labels.Figure 7 tracks the distributions from initial to final states across eight realizations.Source: D AN UNBIASED POPULATION POLARIZES… · p. 17, Figure 7 · p. 17

E SIMULATION PARAMETERS

  • Each fb grid value is an assignment probability, not a realizable population fraction; independent assignments yield discrete realized fractions in finite populations.The realized fraction belongs to {0, 1/N, . . . , 1}.Source: E SIMULATION PARAMETERS · p. 18
  • Doubling the run duration from ∆t = 500 to 1000 preserves qualitative states and phase assignments at four representative points per phase diagram; the largest order-parameter change is 3.4%.Updates become smaller as agents become more certain.Source: E SIMULATION PARAMETERS · p. 18
  • Increasing N from 40 to 60 changes Rµ,c and Rc,w by at most 0.02, with no phase reassignment; the maps are judged converged at N = 40.The study keeps ∆t = 500 fixed; extending the relaxation duration with N would raise the update cost from N^2 to N^3.Source: E SIMULATION PARAMETERS · p. 18

F PHASE DIAGRAM FOR A COMPLEX AGENDA

For a complex agenda, the four-state phase structure remains, but opinion and group correlations shift quantitatively. When the agenda spans only part of the embedding space, a conserved orthogonal component limits attainable opinion overlap.

F PHASE DIAGRAM FOR A COMPLEX AGENDA

  • The complex-agenda sweep uses α = 3.3, N = 40, and a 200 × 200 grid, measuring the same three correlations as the simple-agenda sweep.One realization and 780 000 updates were used per point.Source: F PHASE DIAGRAM FOR A COMPLEX AGENDA · pp. 18–19
  • The complex agenda preserves the four-state arrangement and its dependence on fb and the sign of b, while region (IIIa) expands across most of the discriminatory regime.Unlike the simple agenda, opinion is uncapped, so Rc,w rises to meet Rµ,c.Source: F PHASE DIAGRAM FOR A COMPLEX AGENDA · p. 19, Figure 8 · p. 19

G THE AGENDA’S ORTHOGONAL COMPLEMENT IS CONSERVED

  • For α < 1, learning changes only agenda-span weights, leaving an orthogonal component frozen; the resulting ceiling on opinion overlap is saturated in simulations.The prediction holds to 0.02 for α < 1; for α ≥ 1, the agenda spans the space and the ceiling disappears.Source: G THE AGENDA’S ORTHOGONAL COMPLEMENT… · p. 19, Figure 9 · p. 19
Loading 2610.22701v1…