PaperlayerView exact revision on arXiv

Source-linked AI summary

An Evolutionary Computation Framework for Multi-Agent Q-Learning with Mean-Field Environmental Feedback

Lichen Wang, Shijia Hua, Linjie Liu

arXiv:2609.13253v1cs.MAcs.AI

TL;DR

Population-level models have not self-consistently tracked both learned action values and an endogenous environment in networked multi-agent reinforcement learning. This paper couples those dynamics through a first-order mean-field transport model and projected environmental update. The model reproduces the main macroscopic trajectories in tested networks, with trajectory-level error generally decreasing as population size and average degree increase.

  • Problem

    Prior environmental-feedback models generally evolve strategy frequencies rather than tracking how learners’ action values and softmax choices jointly shape environmental change.

    Source: §I · p. 2

  • Method

    The authors couple stateless Q-learning on fixed graphs to an endogenous environmental state, deriving a first-order mean-field transport equation for Q-value distributions and a projected environmental update.

    Source: §I · p. 2, §IV · p. 12

  • Results

    The mean-field model captures main cooperation and environmental trajectories on four tested network types, with trajectory-level RMSE generally of order 10−2 and decreasing as population size and average degree increase.

    Source: §IV · p. 12

  • Takeaways & Limitations

    Environmental feedback can reshape learned action-value ordering, while reinforcing feedback and environmental response time are associated with path-dependent outcomes and different recovery dynamics.

    Source: §IV · p. 12

    The first-order mean-field approximation suppresses local correlations and degree-conditioned fluctuations, so its accuracy is assessed against finite-network simulations.

    Source: F. Mean-Field Approximation · p. 4

Abstract

from arXiv · show

Multi-agent reinforcement learning in networked populations is governed by the interaction between individual adaptation, local encounters, and changing environmental conditions. To study this interaction, we formulate a coupled learning--environment model in which agents update stateless $Q$-values on a fixed graph, while their population-average behavior drives an environmental variable that dynamically modifies the payoff matrix. Under a first-order mean-field closure, we derive a deterministic transport equation for the population distribution of $Q$-values and couple it with a projected discrete update for the environmental state. The resulting model is evaluated against finite-network Monte Carlo simulations on random regular, Erd\H{o}s--R\'enyi, Barab\'asi--Albert, and random geometric graphs. Across the tested parameter ranges, the mean-field system reproduces the main macroscopic cooperation and environmental trajectories, and the trajectory-level root-mean-square error generally decreases with population size and average degree. The analysis further shows that environmental feedback reshapes the learned action-value ordering, while reinforcing feedback can produce pronounced dependence on the initial learning bias and resource level. The environmental timescale also plays an important role: a rapid response can drive the resource state to a boundary before learning adapts, whereas a slower response preserves the interaction between behavioral learning and environmental recovery. These results provide a population-level description of coupled reinforcement learning and environmental dynamics and characterize the performance of the mean-field approximation within the tested network and parameter ranges.

I. INTRODUCTION

The paper addresses the gap between networked Q-learning and environmental-feedback models by coupling agents’ action values with an endogenous environmental state. It derives a mean-field population model and evaluates its predictions across four network topologies.

I. INTRODUCTION

  • Graph-constrained interactions make collective learning dependent on connectivity, degree heterogeneity, and local sampling, while first-order mean-field closure omits local correlations.The paper evaluates the approximation across population sizes, neighborhood sizes, and network topologies.Source: §I · p. 1
  • Q-learning and replicator dynamics track different population states: action-specific accumulated-reward estimates versus strategy shares.Exploration converts learned values into behavior.Source: §I · p. 1
  • Existing environmental-feedback models generally track strategy frequencies, not learners’ reward-history-based action values and the softmax choices that shape environmental change.The paper identifies a gap in self-consistent population-level models of action-value distributions and endogenous environmental dynamics.Source: §I · p. 2
  • The proposed model couples stateless Q-learning on fixed graphs to an environmental variable driven by population-average strategy and affecting subsequent payoffs.A deterministic first-order transport equation for the Q-value distribution is coupled to a projected environmental update; the framework is evaluated on random regular, Erdős–Rényi, Barabási–Albert, and random geometric graphs.Source: §I · p. 2

A. Multi-Agent Systems on Graphs

The model represents agents on a finite undirected graph and describes its degree distribution. It assumes there are no isolated nodes and that agents share a finite action set.

A. Multi-Agent Systems on Graphs

B. Stateless Q-Learning and Softmax Policy

Each agent maintains action values without conditioning them on an observed environmental state. A softmax policy turns those values into action probabilities, and only the selected action’s value is updated.

B. Stateless Q-Learning and Softmax Policy

C. Environmental Dynamics

A shared environmental state determines the payoff matrix used across agents and interaction edges. Population-average behavior drives environmental change, while the simulation updates actions, values, and the projected environmental state each iteration.

C. Environmental Dynamics

  • The payoff matrix at environmental state n is shared by all agents and interaction edges.Source: C. Environmental Dynamics · p. 3
  • Each agent’s payoff is determined by its action against neighbors and is averaged over its neighborhood.Source: C. Environmental Dynamics · p. 3
  • The environmental feedback function maps the population-average strategy to environmental change.The simulation advances the environment using the current population-average strategy after agents select actions and update their action values.Source: C. Environmental Dynamics · p. 3
  • In the finite-network implementation, the environmental state advances using the current population-average strategy, and the interval projection is defined as Π[0,1](z)=min{1,max{0,z}}.The shared global environmental state n(t) is in [0,1].Source: C. Environmental Dynamics · p. 3

D. Neighbor-Action Configuration

The section defines neighbor-action counts and configurations for agents and nodes of a specified degree.

D. Neighbor-Action Configuration

  • For each agent, action, and time, the neighbor-action configuration is introduced through the number of neighbors selecting each action.The passages identify these counts as the configuration's components.Source: D. Neighbor-Action Configuration · pp. 3–4
  • The count for action aq is the number of the agent's neighbors selecting aq.This gives the action-by-action interpretation of the configuration entries.Source: D. Neighbor-Action Configuration · p. 4
  • For a node of degree k, the configuration space specifies the possible neighbor-action configurations.The agent's configuration is then required to satisfy the stated relation.Source: D. Neighbor-Action Configuration · p. 4

E. Payoff Conditional on Neighbor Configuration

The section defines an agent's payoff for an action from the neighbor configuration and environmental state, then uses that payoff as the Q-update reward.

E. Payoff Conditional on Neighbor Configuration

F. Mean-Field Approximation

The section closes the population dynamics with mean-field assumptions and derives a transport equation for the Q-value distribution. The closure suppresses local correlations and degree-conditioned fluctuations, whose predictive effect is assessed against finite-network simulations.

F. Mean-Field Approximation

  • The closure treats neighbors' actions as conditionally independent with population-average action probabilities, suppressing local correlations and degree-conditioned fluctuations.The resulting neighbor-action counts follow a multinomial distribution conditional on degree and the population-average strategy.Source: F. Mean-Field Approximation · p. 4
  • Under the common payoff matrix and neighbor-averaged payoff, the expected payoff's degree dependence cancels in the numerical setting, though the formulation retains degree averaging for extensions.Degree dependence may remain with degree-dependent payoff matrices or payoff aggregation rules.Source: F. Mean-Field Approximation · p. 5
  • The representative-agent mean Q-value increment uses population-averaged action selection and averages the degree-conditioned payoff over the empirical degree distribution.The expected payoff is averaged over the degree of a uniformly sampled agent.Source: F. Mean-Field Approximation · p. 5
  • The continuous-time mean-drift interpolation retains the conditional mean of the discrete update while neglecting its stochastic fluctuations.This interpolation links expected Q-value increments to the transport description.Source: F. Mean-Field Approximation · p. 5
  • Theorem 1 gives a transport equation for the population density of Q-values under the mean-field assumptions and population-averaged selection closure.The equation describes density evolution through the mean drift field.Source: F. Mean-Field Approximation · p. 5
  • The population-average action-selection probability is computed self-consistently from the Q-value density.The density is the probability density of the representative agent's Q-value vector.Source: F. Mean-Field Approximation · p. 5

G. Coupled Mean-Field Learning and Environment Dynamics

The coupled system evolves the Q-value density within learning rounds and updates the environmental state on the round timescale. Its projected environmental update keeps resource levels within bounds, with both boundaries absorbing.

G. Coupled Mean-Field Learning and Environment Dynamics

  • Within each learning round, the density and population-average strategy evolve while the environmental state remains fixed at its round's value.The next density and environmental state are determined synchronously from the round's starting state.Source: G. Coupled Mean-Field Learning and… · p. 6
  • The projected environmental update assigns zero after depletion and one after saturation, making both boundary states absorbing.The projection is the round-based realization of the environmental rate law.Source: G. Coupled Mean-Field Learning and… · p. 6
  • Figure 1 summarizes the finite-network learning process and its population-level mean-field description for the binary-action case.Source: G. Coupled Mean-Field Learning and… · p. 6

III. RESULTS

Across four network topologies, the mean-field model captures the main macroscopic learning–environment trajectories, with trajectory-level errors generally decreasing as population size and average degree increase. Environmental feedback and response timescale shape cooperation, learned action values, and resource outcomes.

Network topology

  • Across three initial conditions, cooperation, defection, and environmental trajectories are similar across RG, ER, BA, and RGG networks, though RGG shows more visible environmental differences.Mean-field curves generally follow simulations on RG, ER, and BA; stronger local correlations may explain the RGG discrepancy.Source: §III · p. 7, Figure 2 · p. 8

Mean-field accuracy

  • Trajectory-level RMSE generally decreases with population size and average degree across four topologies, remaining on the order of 10^-2 in the tested range.The reduction is clearer for cooperation; environmental RMSE has small nonmonotonic variations for some network–degree combinations.Source: §III · p. 8, Figure 3 · p. 9

Static payoff configurations

  • Under static Harmony and Prisoner’s Dilemma payoffs, cooperation or defection becomes dominant, while mean-field predictions closely track simulations and learned action values are not mirror images.Finite inverse temperature leaves positive probability for both actions; the value gap is larger under Env. 1 than Env. 2, where defection’s value remains higher.Source: Figure 4 · p. 9, §III · pp. 9–10

Parameter sensitivity

  • Under Env. 3, increasing θ makes cooperation less favored but generally raises the environmental state; θ also governs the reversal in learned action-value ordering.At small β, cooperation is close to 0.5; at larger θ, environmental levels remain high across most tested β values, while the action-value difference changes sign near the central θ range.Source: Figure 5 · p. 10, §III · p. 10

Initial-condition dependence

  • Under reinforcing Env. 4 feedback, initial learning bias and resource level separate trajectories into low-cooperation, depleted or high-cooperation, resource-rich regimes.Increasing θ broadens the trajectories able to enter positive environmental growth, but even θ = 2 does not rescue the low-cooperation, low-resource trajectory.Source: §III · p. 10, Figure 6 · p. 11

Environmental timescale

  • Environmental response timescale changes outcomes: rapid response can reach the depleted boundary before learning adjusts, whereas slower response can preserve coupled evolution.Under Env. 3, ϵ = 0.1 reaches n = 0 in the first learning round; under Env. 4, ϵ = 10 permits cooperation to rise before depletion and leads to a highly cooperative long-time state.Source: §III · pp. 10, 12, Figure 7 · p. 11

IV. CONCLUSION

The coupled learning–environment model captures the main macroscopic cooperation and environmental trajectories in finite-network simulations across the tested parameter ranges. Environmental feedback shapes learned behavior, while the mean-field closure’s accuracy is qualified by network structure and local correlations.

IV. CONCLUSION

  • The model couples graph-based stateless Q-learning and endogenous environmental feedback through a Q-value transport equation and projected environmental update.The update uses neighbor-averaged rewards, and the transport equation follows a first-order mean-field closure.Source: §IV · p. 12
  • Across random regular, Erdős–Rényi, Barabási–Albert, and random geometric networks, mean-field predictions capture the main macroscopic cooperation and environmental trajectories, with trajectory-level RMSE generally of order 10^-2.RMSE generally decreases as population size and average degree increase.Source: §IV · p. 12
  • Environmental feedback changes learned action-value ordering and cooperation; reinforcing feedback can yield distinct long-run outcomes depending on initial learning bias and environmental state.A rapid environmental response can reach the depleted boundary before learning adapts, while slower response allows behavioral adjustment and environmental recovery to remain coupled.Source: §IV · p. 12
  • The mean-field closure approximates neighboring actions with population-level strategy statistics, so accuracy may decrease in sparse networks with strong local correlations, community structure, or assortative mixing.The numerical analysis focuses on fixed networks, binary-action games, and linear environmental feedback.Source: §IV · p. 12
  • The formulation accommodates a general finite action set and permits other payoff–environment mappings and nonlinear feedback functions by replacing corresponding model terms.State-dependent learning and adaptive or co-evolving networks are identified as future extensions.Source: §IV · p. 12
Loading 2609.13253v1…