PaperlayerView exact revision on arXiv

Source-linked AI summary

ADAGE: A generic two-layer framework for adaptive agent based modelling

Benjamin Patrick Evans, Sihan Zeng, Sumitra Ganesh, Leo Ardon

arXiv:2501.09429v1cs.MAcs.AIcs.LGecon.GNq-fin.CP

TL;DR

ABMs need agent behaviour and environmental characteristics to adapt together, but prior approaches largely address only fixed-rule adaptation. ADAGE formulates this bi-level problem as a Stackelberg game with conditional policies and applies it across common ABM tasks, with examples covering policy design, calibration, scenario generation, and robust behavioural learning.

  • Problem

    Fixed behavioural rules expose ABMs to the Lucas critique, while existing adaptive approaches do not generally adapt environmental characteristics simultaneously.

    Source: §1 · p. 1

  • Method

    ADAGE uses a two-layer Stackelberg framework in which followers condition behaviour on environmental characteristics and the outer layer updates those characteristics through coupled non-linear equations.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    ADAGE encompasses policy design, calibration, scenario generation, and robust behavioural learning across example economic and financial environments.

    Source: §1 · p. 1, §5 · p. 8, §4.4.2 · p. 8

  • Takeaways & Limitations

    The framework provides a unified approach for co-evolving agent behaviour and environments across multiple adaptive agent-based modelling tasks.

    Source: §5 · p. 8

    The paper does not provide head-to-head comparisons of alternative optimisation algorithms and focuses on demonstrating framework integration across tasks and models.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Agent-based models (ABMs) are valuable for modelling complex, potentially out-of-equilibria scenarios. However, ABMs have long suffered from the Lucas critique, stating that agent behaviour should adapt to environmental changes. Furthermore, the environment itself often adapts to these behavioural changes, creating a complex bi-level adaptation problem. Recent progress integrating multi-agent reinforcement learning into ABMs introduces adaptive agent behaviour, beginning to address the first part of this critique, however, the approaches are still relatively ad hoc, lacking a general formulation, and furthermore, do not tackle the second aspect of simultaneously adapting environmental level characteristics in addition to the agent behaviours. In this work, we develop a generic two-layer framework for ADaptive AGEnt based modelling (ADAGE) for addressing these problems. This framework formalises the bi-level problem as a Stackelberg game with conditional behavioural policies, providing a consolidated framework for adaptive agent-based modelling based on solving a coupled set of non-linear equations. We demonstrate how this generic approach encapsulates several common (previously viewed as distinct) ABM tasks, such as policy design, calibration, scenario generation, and robust behavioural learning under one unified framework. We provide example simulations on multiple complex economic and financial environments, showing the strength of the novel framework under these canonical settings, addressing long-standing critiques of traditional ABMs.

1 INTRODUCTION

ADAGE addresses the Lucas critique by jointly adapting agent behaviour to changing environmental characteristics and adapting those characteristics through a two-layer framework.

1 INTRODUCTION

  • Traditional ABMs use fixed or manually specified behavioural rules, making them vulnerable when environmental changes alter optimal decisions.The Lucas critique links changing environments to systematic changes in optimal decision rules.Source: §1 · p. 1
  • ADAGE jointly learns conditional agent behaviours and environmental characteristics, allowing each layer to adapt to the other.The inner layer learns behavioural rules conditioned on updateable characteristics, while the outer layer updates those characteristics.Source: §1 · p. 1
  • ADAGE formulates adaptive agent-based modelling as a Stackelberg game solved through coupled non-linear equations.This formulation provides a unified basis for adapting both agent behaviour and environmental characteristics.Source: §1 · p. 1
  • The framework is intended to unify common tasks including calibration, policy design, scenario analysis, and robust behavioural learning.These tasks are treated as instances of the same adaptive modelling framework.Source: §1 · p. 1

2 BACKGROUND AND RELATED WORK

Prior work combines machine learning with agent-based simulation, but generally learns fixed behavioural rules and does not provide a unified treatment of adaptive environments and multiple ABM tasks.

2 BACKGROUND AND RELATED WORK

  • Combining reinforcement learning with agent-based simulation aims to improve hand-crafted models while retaining agent heterogeneity and relaxed equilibrium assumptions [2, 47].This area is presented as promising for adaptive simulation.Source: §2 · p. 2
  • Most existing learning-based ABMs learn fixed behavioural rules, requiring retraining when environmental conditions such as taxation rates change.This limits automatic adaptation of agent behaviour to updated policies.Source: §2 · p. 2
  • ADAGE extends adaptive agent-based modelling by representing policy design, calibration, and scenario generation within one Stackelberg-game framework.The outer layer is the leader and the inner agent-based simulation is the follower layer.Source: §2 · p. 2

3 PROPOSED APPROACH

ADAGE models adaptive agent-based modelling as a Stackelberg game between an outer leader and inner followers whose policies condition on evolving environmental characteristics.

3 PROPOSED APPROACH

  • ADAGE represents the environment as a Stackelberg game with one outer-layer leader and n inner-layer follower agents.The framework uses a parameterised environment representing the agent-based model.Source: §3 · p. 2, §3.1 · p. 2
  • Stackelberg asymmetry gives the outer layer stronger influence and a typically slower decision timescale, while followers optimise local objectives in response.The leader acts first and followers react within a hierarchical objective structure.Source: §3.1 · pp. 2–3
  • Follower policies are conditioned on observed characteristics ˆθ_i that parameterise the environment and reflect the leader’s effects.The policy is written as π_i(a|o_i, ˆθ_i), making environmental conditioning explicit.Source: §3.1 · p. 2
  • Changing the leader’s reward and action space lets the same representation support policy design, calibration, and other ABM tasks.For calibration, the leader tunes latent characteristics; for policy design, it updates rules such as taxation.Source: §3.1.1 · p. 3
  • The framework approximates Stackelberg equilibria by jointly optimising leader and follower policies through coupled non-linear equations and iterative updates.Alternating gradient descent/ascent updates each policy while holding the others at their latest iterates; learning rates can reflect the nested timescales.Source: §3.2 · p. 3
  • Follower learning is implemented with deep reinforcement learning using PPO, while the framework remains independent of a particular optimisation algorithm.The characteristics observation is incorporated into the follower observation space.Source: §3.2.1 · p. 3

4 EXPERIMENTS

ADAGE demonstrates a simulator-independent two-layer framework across policy design, calibration, scenario generation, and robust behavioural learning. Across these tasks, the outer layer adapts environmental characteristics or objectives while inner agents learn conditional behaviours.

Policy design

  • The policy-design experiment significantly improves social welfare over a free-market baseline by inducing less work, higher wages, more consumption, and lower savings.The learned taxation rules also reduce asset inequality while increasing average income without reducing income equality.Source: §4.1.2 · p. 4

Calibration

  • ADAGE calibrates the Cobweb simulator to experimental data, capturing the overall distribution and achieving the lowest error among the compared approaches.The framework supports reinforcement-learning calibration of distributional or individual penalties and Bayesian outer-layer calibration.Source: §4.2.2 · p. 5, §4.2.1 · p. 5

ADAGE framework

  • Changing the leader’s reward and action space lets the same Stackelberg formulation represent distinct modelling goals, including welfare maximisation and calibration.The framework’s outer layer can update taxation rules, fit latent agent parameters, or optimise other desired outcomes while followers condition behaviour on evolving characteristics.Source: §4.2.2 · p. 5

Scenario generation

  • In scenario generation, the learned Tobin’s tax restricts market volatility in most tested configurations, reducing standard-deviation volatility in 7/9 cases and mean absolute percentage change in 6/9.For capacity c=0.6, the resulting attendance series and distribution have a narrower range, indicating a more stable market.Source: §4.3.1 · p. 6, §4.3.2 · p. 7

Robust behavioural learning

  • The robust behavioural-learning experiment shows a single market-maker policy adapting its spread to preferences, using narrower spreads for market share and wider spreads for profit-focused trading.As the preference shifts from market share toward PnL, the learned policy executes fewer but more profitable trades.Source: §4.4.1 · p. 7, §4.4.2 · p. 8

4 EXPERIMENTS

  • ADAGE unifies policy design, calibration, scenario generation, and robust behavioural learning as special instances of one framework.The four illustrative configurations use separate inner agent-based models and varying outer-layer designs to demonstrate flexibility and simulator independence.Source: §4 · p. 4, §4.5 · p. 8

5 DISCUSSION AND CONCLUSION

ADAGE is presented as a generic framework in which agents and their environments co-evolve. The discussion highlights applications to calibration, policy design, scenario generation, and robust behavioural learning, while identifying theoretical refinement as future work.

  • The framework encompasses calibration, policy design, scenario generation, and robust behavioural learning within a common approach.Source: §5 · p. 8
  • Future research could develop the theoretical treatment of the bi-level optimization process and investigate specialized differentiable inner layers.Source: §5 · p. 8

DISCLAIMER

The document is provided for informational purposes by JPMorgan Chase & Co. and its affiliates, with broad disclaimers concerning completeness, accuracy, reliability, liability, and investment use.

  • The document is informational and is not presented as investment research, investment advice, or a recommendation, offer, or solicitation.Source: DISCLAIMER · p. 9
  • JPMorgan Chase & Co. disclaims representations, warranties, and liability regarding the information’s completeness, accuracy, and reliability.Source: DISCLAIMER · p. 9
  • The document is attributed to JPMorgan Chase & Co. and its affiliates and carries a 2025 copyright notice.Source: DISCLAIMER · p. 9

A REINFORCEMENT LEARNING CONFIGURATION

The reinforcement-learning configuration uses PPO with Generalized Advantage Estimation for behavioural-rule optimization, implemented in RLLib, with default hyperparameters summarized in Table 3.

B.1 Scenario Generation

The scenario-generation evaluation tests whether ADAGE’s market-stabilizing effect persists across attendance capacities. The proposed approach significantly reduces attendance volatility across capacities, producing more stable markets.

  • The evaluation varies attendance capacity C to test whether the observed results depend on that setting.Source: §B.1 · p. 11
  • The proposed approach significantly reduces attendance-fluctuation volatility across various entrance capacities, resulting in more stable markets.Source: §B.1 · p. 11
  • The market-price dynamics are described through supply S and demand D(p_t) = a−bp_t+η_t curves under rational expectations.Source: §B.2.1 · p. 11

B.2.1 Extended cobweb market overview.

The cobweb market overview relates rational predictions to the equilibrium price while examining alternative calibration strategies. Individual-parameter calibration improves on baseline comparisons but remains inferior to distributional calibration.

  • Rational predictions fluctuate around the equilibrium price, with deviations proportional to 𝜖_t.Source: §B.2.1 · p. 11
  • Individual-parameter calibration fits better than baseline comparisons but not as well as distributional-parameter calibration.Repeated experimental results across participants create an underlying distribution of information-processing penalties.Source: §B.2.2 · p. 11

B.2.2 Additional results.

Additional results illustrate individual calibration and alternative outer-layer optimization within ADAGE. Bayesian optimization integrates with the framework, although calibration performs worse than the reinforcement-learning setting in this example.

  • Individual calibration is presented as an additional illustrative configuration rather than the preferred calibration strategy.It provides a better fit than baseline comparisons but not as good as distributional parameters.Source: §B.2.2 · p. 11
  • Bayesian optimization integrates with ADAGE but produces worse calibration than the reinforcement-learning setting in this case.Source: §B.2.2 · pp. 11–12
  • The comparison is intended to demonstrate ADAGE's independence from the particular learning algorithm, not to rank optimization methods.Refining Bayesian hyperparameters could improve its results.Source: §B.2.2 · p. 12
Loading 2501.09429v1…