PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolutionary Stability Does Not Guarantee Learning Accessibility: A Multi-Agent Reinforcement Learning Perspective on Cooperation Emergence

Yijie Wang

arXiv:2609.27664v1cs.AI

TL;DR

The paper asks whether evolutionary stability implies that finite-sample decentralized learners can reach cooperation, a distinction that matters because population adjustment and local reward learning follow different dynamics. It compares replicator dynamics and three value-based learners in the same three-agent governance game. The sampled evolutionary basin is 1.00, while learning outcomes differ by mechanism.

  • Problem

    Whether a cooperative outcome indicated by evolutionary stability is reachable through a specified finite-sample decentralized learning process remains a distinct question from equilibrium characterization.

    Source: §1.1 · p. 2, §7.1 · p. 17

  • Method

    The study derives replicator dynamics and compares their cooperative basin with empirical learning basins for three decentralized value-based learners under the same fixed payoff environment.

    Source: §8 · pp. 22–23

  • Results

    396 of 450 ε-greedy IQL runs reach the cooperative criterion, while scaled Boltzmann and SA–EA BQL do not reach it in their respective grids.

    Source: §8 · p. 22

  • Takeaways & Limitations

    Evolutionary stability does not automatically yield finite-sample cooperation, so the cooperative outcome’s accessibility depends on the specified learning mechanism.

    Source: §8 · p. 23

    The inference is bounded to one three-agent binary-action game with fixed incentives; transportability to larger populations, heterogeneous agents, networks, or richer action spaces requires additional tests.

    Source: §7.5 · p. 21

Abstract

from arXiv · show

Cooperation emergence is a central problem in multi-agent systems because decentralized agents must coordinate while adapting to the changing behavior of others. Evolutionary game theory identifies strategically stable outcomes, but stability under a population adjustment dynamic need not imply that finite-sample learning agents can reach the same outcome through local reward feedback. We study this distinction in a transparent three-agent governance-motivated game involving a government, a platform firm, and users. We derive replicator dynamics for the fixed stage-game incentives, evaluate the cooperative evolutionary basin on a symmetric initial-condition grid, and compare it with learning-basin estimates for three decentralized value-based learners. The learning analysis uses independent Q-learning with $\varepsilon$-greedy action selection, scaled Boltzmann exploration, and SA--EA BQL under the same payoff environment and outcome criterion. The evolutionary basin has volume $V_E=1.00$ on the sampled grid. The empirical learning basin is $0.88$ for $\varepsilon$-IQL and $0.00$ for both scaled Boltzmann and SA--EA BQL. Diagnostic traces show that broader action diversity and nonzero value separation can coexist with failure to sustain the cooperative joint action in this fixed configuration. These results indicate that evolutionary stability and learning accessibility are distinct properties of a coupled game--learning system. The shared-bike setting is a motivating application; the broader contribution is a framework for comparing population-level stability with the finite-sample accessibility of cooperation under specified multi-agent learning dynamics.

1 Introduction

Cooperation requires decentralized agents to form compatible behavior from local feedback, while evolutionary analysis identifies strategically viable outcomes. This paper compares population-level stability with finite-sample learning accessibility under the same fixed incentives.

1.1 Motivation

  • The study compares population-level stability and finite-sample learning accessibility in a shared payoff environment, distinguishing the strategic landscape from the dynamics that navigate it.The comparison is framed as a way to assess whether a cooperative outcome indicated by evolutionary analysis is reachable through specified decentralized adaptation.Source: §1.1 · pp. 2–3

1.2 Research gap

  • The research gap is whether evolutionary stability implies that finite-sample MARL agents can reach a predefined cooperative outcome under a specified learning process.The paper defines learning accessibility empirically for a defined initial-condition grid and fixed experimental configuration, not as a universal property of learnability.Source: §1.2 · pp. 4–5

1.3 Research framework and contributions

  • The study derives replicator dynamics and compares their cooperative basin with empirical basins for three decentralized value-based learners under fixed strategic incentives.The learners are ε-greedy independent Q-learning, scaled Boltzmann exploration, and SA–EA BQL; diagnostics examine action coverage, policy entropy, and Q-value separation.Source: §1.3 · p. 5

2 Related Work

Prior work provides distinct lenses on stability, adaptive learning, and decentralized coordination. This paper uses them to compare a replicator-generated basin with the empirical learning basin of specified MARL procedures under the same incentives.

2.1 Evolutionary game theory

  • Evolutionary game theory and replicator dynamics characterize strategic stability and population adjustment, but do not specify how finite collections of independent learners acquire behavior from realized rewards.The paper uses this stability lens rather than proposing a new equilibrium concept or modifying replicator dynamics.Source: §2.1 · p. 7

2.2 Learning in games

  • Learning-in-games research makes the adjustment process central, with equilibrium selection depending on experience, perturbations, beliefs, and the learning rule.This study evaluates the empirical learning basin of a specified MARL procedure alongside a replicator-generated basin under the same strategic incentives.Source: §2.2 · p. 7

2.3 Multi-agent reinforcement learning

  • MARL research addresses decentralization, non-stationarity, and coordination through varied architectures and information designs; this paper instead uses transparent decentralized value-based rules as diagnostic dynamics.It does not introduce a new MARL algorithm or benchmark centralized methods against independent learners.Source: §2.3 · p. 8

3 Evolutionary Game Model

In a binary-action governance game, the paper defines empirical learning accessibility separately from evolutionary attraction and derives replicator dynamics for the same payoff structure. The sampled symmetric grid converges cooperatively under the specified incentive-on regime, with the estimate bounded to that grid.

3.1 Three-party governance game

  • The government, firm, and users choose binary regulation, compliance, and responsiveness actions, respectively, with x, y, and z denoting the shares choosing each active action.The verified stage-game payoffs connect regulation, compliance, and responsiveness through costs, benefits, and sanctions.Source: §3.1 · p. 9

3.2 Definition: Learning Accessibility

  • Learning accessibility is the observed fraction of successful trajectories for a specified mechanism, initialization design, finite horizon, and stochastic update procedure.It differs from the evolutionary basin, which records whether a stable state attracts population dynamics from specified initial states.Source: §3.2 · pp. 9–10

3.3 Replicator dynamics

  • The replicator formulation changes each population’s strategy share in proportion to the active strategy’s payoff advantage, preserving binary-action boundaries.The equations separate incentives associated with regulation and firm compliance, and are evaluated using the verified parameterization and numerical integration routine.Source: §3.3 · p. 10

3.4 Stability and basin evaluation

  • All nine sampled symmetric initial conditions converge to the cooperative corner under the incentive-on regime, giving an evolutionary basin estimate of 1.00.This is a numerical result for the sampled grid, not a claim about every point in the continuous state space.Source: §3.4 · p. 11

4 Multi-Agent Reinforcement Learning Framework

The framework compares decentralized learning mechanisms in a shared fixed-payoff game. Each mechanism is evaluated without reward shaping or altered payoffs, with adaptive exploration included as a diagnostic comparison rather than a guarantee of cooperation.

4.1 Multi-agent environment

  • Government, firm, and user independently choose actions in the same three-party stage game, receiving their original stage rewards without reward shaping or coordination bonuses.The initial-preference grid is imposed only at the first decision; later behavior follows each learner’s updates and exploration rule.Source: §4.1 · p. 11

4.2 Independent Q-learning

  • Independent Q-learning updates the selected action value from its original stage reward while leaving the unselected action unchanged.The repeated-stage update has no continuation-value term; ε-greedy IQL serves as a descriptive, unoptimized baseline.Source: §4.2 · p. 12

4.3 Scale-aware Boltzmann exploration

  • Scale-aware Boltzmann exploration samples actions by softmaxing payoff-scale-normalized action values, with temperature and payoff scale set by the implementation.The normalization makes responses interpretable when the players’ payoff ranges differ; this is a diagnostic comparison, not a claimed superior algorithm.Source: §4.3 · p. 12

4.4 Adaptive exploration variant (SA–EA BQL)

  • SA–EA BQL combines scale-aware action selection with an exploration adjustment that lets temperature depend on the learner’s action-value gap.It tests whether adaptive exploration changes accessibility under the fixed configuration, without claiming guaranteed cooperation or causal evidence beyond that comparison.Source: §4.4 · pp. 12–13

5 Experimental Protocol

The learning-basin comparison uses one fixed protocol so differences across mechanisms are not confounded by changes in the environment or evaluation design.

5 Experimental Protocol

  • The reported learning-basin comparison uses a fixed experimental protocol recorded in Table 2.The table is provided to make the finite-sample estimates reproducible.Source: §5 · p. 13, Table 2 · p. 13
  • All learning mechanisms use the same fixed stage-game payoffs and the same pre-specified cooperative-outcome criterion.The protocol holds the environment and evaluation design constant across mechanisms.Source: §5 · p. 13
  • The protocol is for reproducibility, not an additional parameter search or algorithm-tuning exercise.Table 2 records the configuration used throughout the comparison.Source: §5 · p. 13, Table 2 · p. 13

6 Results

On the evaluated grid, replicator dynamics converges cooperatively, while finite-sample learning accessibility depends on the mechanism. The Boltzmann-based learners retain action diversity and nonzero value gaps, yet do not sustain the cooperative joint action in this fixed comparison.

6.1 Evolutionary stability on the sampled grid

  • VE = 1.00 (9/9): replicator dynamics converges to the cooperative corner from all nine frozen symmetric starting points under α = 80.This establishes accessibility for the specified grid and parameterization, not a global basin result.Source: §6.1 · pp. 13–14

6.2 Learning accessibility

  • 0.880 for ε-greedy IQL and 0.000 for both scale-aware Boltzmann and SA–EA BQL are the learning-basin estimates, despite cooperative replicator convergence on the evaluated grid.Figure 2 contrasts evolutionary convergence with mechanism-dependent finite-sample accessibility.Source: §6.2 · p. 14, Figure 2 · p. 14

6.3 Exploration mechanism diagnosis

  • The Boltzmann-based learners retain substantial action diversity and nonzero Q-value gaps, so their zero-basin results are not solely explained by premature deterministic-policy collapse.The trajectory-based diagnosis is that persistent stochastic action selection can impede convention formation here, not that exploration is intrinsically harmful or universally inferior.Source: §6.3 · p. 16

7 Discussion

The discussion distinguishes cooperative attraction under replicator dynamics from finite-sample learning accessibility: the evolutionary basin covers the sampled symmetric grid, while learning outcomes depend on the specified learner. It also cautions that exploration diversity and value separation do not by themselves establish coordinated convention formation, and that the comparison is bounded to its fixed game and learning configuration.

7.1 Evolutionary stability versus learning accessibility

  • The evolutionary basin estimate is 1.00 across nine sampled symmetric initial conditions, while learning-basin estimates vary by method under the same incentives.These are grid-based, finite-sample results, not claims about every continuous initial state or every learning process.Source: §7.1 · pp. 16–17

7.2 Exploration and cooperative convention formation

  • Boltzmann-based learners retain broader action diversity and nonzero Q-value separation than the ε-greedy benchmark, yet fail to sustain cooperation under the frozen basin criterion.The diagnostics do not identify a unique cause; persistent exploration is consistent with impeding convention formation in this particular design, but other learning-process features may contribute.Source: Figure 6 · p. 17, §7.2 · pp. 18–19

7.3 Implications for multi-agent systems

  • Cooperative-system assessments should pair strategic analysis with learning-accessibility analysis and specify initialization, update rule, exploration, and evaluation horizon.Mechanism and learning design may need joint assessment; the framework diagnoses mismatches but prescribes no universal intervention.Source: §7.3 · pp. 19–20

7.4 Relation to learning in games

  • Replicator dynamics aggregates population-level payoff advantages, whereas finite learners must experience compatible joint actions often enough to form and retain favorable value orderings.The fixed-configuration comparison measures this distinction without establishing that one learning rule is generally superior.Source: §7.4 · pp. 20–21

7.5 Limitations and future directions

  • Inference is limited to a fixed three-agent binary-action governance game and finite parameter configurations, sampled initial conditions, and evaluation horizon.The study also omits human behavioral experiments and does not establish that any learner dominates across settings; broader populations, richer action spaces, and other strategic structures require further tests.Source: §7.5 · pp. 21–22

8 Conclusion

The evolutionary analysis finds a cooperative state accessible from every evaluated symmetric initial condition, while learning outcomes vary by mechanism. Diagnostic traces and the study’s scope qualify this contrast between evolutionary stability and finite-sample learning accessibility.

8 Conclusion

  • V_E=1.00 on the sampled evolutionary grid, while ε-greedy IQL reached the cooperative criterion in 396 of 450 runs and scaled Boltzmann and SA–EA BQL did not reach it in their grids.The estimates apply to the stated game, initial-preference grid, seeds, and finite evaluation horizon.Source: §8 · p. 22
  • Boltzmann-based learners retained action diversity and nonzero Q-value separation despite failing to converge cooperatively, so insufficient exploration alone does not explain the result.The evidence suggests that learners must repeatedly realize compatible joint actions for their updates to support a shared convention; it does not isolate a unique causal route.Source: §8 · p. 23
  • Stable cooperation under evolutionary dynamics does not automatically emerge from finite-sample reinforcement learning, so neither equilibrium nor learning analysis substitutes for the other.Jointly reporting evolutionary and learning basins makes stability and accessibility assessable in the same framework.Source: §8 · p. 23
  • The present design does not support broad claims about governance systems or MARL algorithms; future work could test larger, more heterogeneous populations and human behavior.The paper also identifies richer communication or commitment and analytical characterization of learning basins as future directions.Source: §8 · p. 23

A.1 Derivation of the replicator dynamics

The derivation applies the binary replicator identity to expected payoff differences for each population, using the stage-game incentives without adding behavioral assumptions.

A.1 Derivation of the replicator dynamics

  • The binary replicator identity uses the active-strategy share and the average payoff weighted by the active and inactive shares.Applying it to each population’s expected payoff difference yields the three equations in Section 2.Source: §A.1 · p. 24
  • For government, the active-versus-weak regulation payoff difference is −Cg + θS + (λs − λw)F(1 − y), producing equation 2.The difference reflects direct regulatory return and the sanction when the firm is non-compliant.Source: §A.1 · p. 24
  • For the firm, compliance’s payoff difference is −c + δB + αz − E + {λw + (λs − λw)x}F, producing equation 3.The payoff comparison includes compliance cost, residual subsidy benefit, firm–user coordination return, private gain, and the government-share-weighted sanction.Source: §A.1 · p. 24
  • For users, responsive participation’s payoff difference is a − cu + qy + rx, combining its baseline net return with firm- and government-related returns.Non-response has zero payoff in the specified stage game.Source: §A.1 · p. 24

B.1 Algorithm 1: Independent Q-learning training procedure

The training procedure initializes agents’ action values and seeds, repeatedly selects and executes joint actions, updates selected values, and classifies runs at a fixed horizon. Alternative action-selection rules provide diagnostic comparisons within the fixed configuration.

B.1 Algorithm 1: Independent Q-learning training procedure

  • Each run initializes agents’ two action values from the fixed initial-preference condition and retains its assigned random seed before training for the configured episode count.Source: §B.1 · p. 25
  • Each agent selects a binary action under its configured rule, then the joint action is executed in the unchanged stage game and each agent receives its original individual reward.Source: §B.1 · p. 25
  • After each episode, only the selected action value is updated; runs are classified by the pre-specified learning-basin criterion when the fixed horizon ends.Source: §B.1 · p. 25
  • Independent Q-learning gives each agent its own values without access to or communication about others’ values, and the repeated-stage update has no continuation-value term.Source: §B.1 · p. 25

B.2 Alternative action-selection rules

  • Scaled Boltzmann uses a normalized softmax rule with fixed temperature and payoff-scale normalization; SA–EA BQL adjusts temperature using action-value separation.These alternatives compare exploration mechanisms in the fixed configuration and imply no general ranking beyond it.Source: §B.2 · p. 25

C Experimental configuration and reproducibility

The experimental configuration records a fixed setup for reproducibility, while the learning-basin estimates use a shared grid, seed structure, payoff environment, and outcome criterion.

C Experimental configuration and reproducibility

Related papers

Loading 2609.27664v1…