PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comparison of Economic Agent-Based Model Calibration Methods

Donovan Platt

arXiv:1902.05938v1q-fin.CPecon.GNq-fin.EC

TL;DR

ABM calibration research has lacked rigorous comparisons of methods and assessments of estimate quality. This paper compares established and novel approaches through computational experiments. Bayesian methods generally outperform frequentist objective-function approaches, though large-scale calibration remains challenging.

  • Problem

    Many ABM calibration studies use qualitative, stylised-fact-driven procedures, while quantitative methods are seldom benchmarked against alternatives or rigorously tested for estimate quality.

    Source: §1 · pp. 1–2

  • Method

    The authors compare established and novel ABM calibration methods in computational experiments and apply promising techniques to a large-scale UK housing-market model.

    Source: §1 · p. 3

  • Results

    Bayesian methods deliver more compelling performance than SMD methods in a wide range of circumstances, but fail with a large-scale model and a free parameter set of significant cardinality.

    Source: §5 · p. 25

  • Takeaways & Limitations

    The findings support further development of Bayesian estimation techniques and of methods for calibrating large-scale ABMs.

    Source: §5 · p. 25

    Estimating the Bayesian likelihood is challenging: nonparametric methods may be computationally infeasible for many ABMs, while parametric approaches may not represent distributions flexibly enough.

    Source: §2.3.3 · p. 6

Abstract

from arXiv · show

Interest in agent-based models of financial markets and the wider economy has increased consistently over the last few decades, in no small part due to their ability to reproduce a number of empirically-observed stylised facts that are not easily recovered by more traditional modelling approaches. Nevertheless, the agent-based modelling paradigm faces mounting criticism, focused particularly on the rigour of current validation and calibration practices, most of which remain qualitative and stylised fact-driven. While the literature on quantitative and data-driven approaches has seen significant expansion in recent years, most studies have focused on the introduction of new calibration methods that are neither benchmarked against existing alternatives nor rigorously tested in terms of the quality of the estimates they produce. We therefore compare a number of prominent ABM calibration methods, both established and novel, through a series of computational experiments in an attempt to determine the respective strengths and weaknesses of each approach and the overall quality of the resultant parameter estimates. We find that Bayesian estimation, though less popular in the literature, consistently outperforms frequentist, objective function-based approaches and results in reasonable parameter estimates in many contexts. Despite this, we also find that agent-based model calibration techniques require further development in order to definitively calibrate large-scale models. We therefore make suggestions for future research.

1 introduction

Agent-based models can reproduce empirical patterns that traditional approaches struggle to capture, but their validation and calibration are often qualitative and stylised-fact driven. The paper compares established and newer calibration methods and tests promising approaches on a large-scale UK housing-market model.

1 introduction

  • The heterogeneity, complex interactions, and nonlinear dynamics of economic systems make them difficult to represent with traditional methods.The paper describes these features as motivating alternative modelling approaches.Source: §1 · p. 1
  • Traditional economic models have been criticized for empirically inconsistent assumptions and for failing to predict or resolve the Great Recession.The criticism particularly targets general-equilibrium models and their extensions.Source: §1 · p. 2
  • ABMs model individual agents and their interactions, and can reproduce empirically observed stylised facts that traditional approaches do not readily recover.Examples include fat-tailed asset returns, volatility clustering, and Zipf-distributed firm sizes.Source: §1 · p. 2
  • Qualitative, stylised-fact-driven validation and ad-hoc parameter adjustment remain common, making robust comparison among models difficult.Many models reproduce similar stylised facts despite using different behavioural rules, while causal effects of those rules are difficult to identify.Source: §1 · pp. 2–3
  • Quantitative calibration research is compartmentalised, with few comparisons among methods and limited understanding of some methods’ theoretical properties.Computational constraints also leave most techniques tested on highly simplified, older models, raising doubts about their applicability to large-scale ABMs.Source: §1 · p. 3
  • The study compares prominent established and novel calibration methods, then tests the most promising techniques on a large-scale UK housing-market ABM.The experiments aim to assess whether performance on simpler models carries over to a state-of-the-art model.Source: §1 · p. 3

2 literature review

The review surveys ways to calibrate ABMs, from direct observation and analytical estimation to simulation-based methods. It describes unresolved trade-offs in objective functions, Bayesian likelihood estimation, and the computational demands of applying methods to large models.

2 literature review

  • Calibration selects parameters to bring model dynamics closer to a dataset under a criterion, and the review also discusses issues relevant to validation.The authors use “calibration” and “estimation” interchangeably because their distinction is inconsistent and unclear.Source: §2 · pp. 3–4

2.1 Direct Observation

  • Directly observable or inferable quantities can be set from data, and some models determine many parameters this way.The UK housing-market model calls this process micro-calibration.Source: §2.1 · p. 4

2.2 Analytical Methods

  • Maximum likelihood and related analytical methods generally do not suit ABMs because they require an analytical expression for the simulated time-series joint density.Such expressions are possible only for very simple models and can still be nontrivial to obtain.Source: §2.2 · p. 4

2.3 Simulation-Based Methods

  • Simulation-based methods are central to recent ABM calibration research because most models of interest cannot provide analytical solutions for key properties.Frequentist simulated minimum distance methods construct an objective function measuring the distance between simulated and observed time series, then optimize it.Source: §2.3 · p. 4, §2.3.1 · p. 5

2.3.1 Frequentist Inference: Traditional Approaches to Simulated Minimum Distance

  • Traditional SMD methods include matching selected moments or auxiliary-model parameters, but their choices can be arbitrary and may miss important data dynamics.The standard MSM weight-matrix method can also face numerical challenges when it inverts near-singular matrices.Source: §2.3.1 · p. 5

2.3.2 Frequentist Inference: New Approaches to Simulated Minimum Distance

  • Newer SMD criteria compare time-series structure or patterns to reduce reliance on arbitrary moments or auxiliary models, but their theoretical properties are often insufficiently analyzed.This limited analysis and conceptual dissimilarity among methods leave little evidence for choosing between them.Source: §2.3.2 · pp. 5–6

2.3.3 Bayesian Inference

  • Bayesian inference can incorporate prior parameter information and yields a posterior distribution rather than a single estimate.Its key challenge is estimating the likelihood: nonparametric methods may be too computationally demanding, while parametric methods may lack flexibility.Source: §2.3.3 · p. 6

2.3.4 Addressing Computational Difficulties

  • Computational costs have limited calibration of large-scale ABMs, leaving the generalizability of results from simpler models uncertain.Surrogate modelling and cloud computing are discussed as ways to address simulation costs; this investigation uses cloud resources.Source: §2.3.4 · pp. 6–7

3 experimental procedure

The experimental procedure compares calibration methods through tests designed to quantify success. It accounts for model misspecification and the difficulty of defining goodness of fit for ABMs.

3 experimental procedure

  • Comparisons require tests that quantify calibration success and produce measures that can be compared across methods.Simply calibrating candidate models to observed data and comparing goodness of fit may be suboptimal.Source: §3 · p. 7
  • Candidate models may be misspecified, so poor fit to observed data may reflect model limitations rather than the calibration method.This concern is especially relevant in economic settings.Source: §3 · p. 7
  • Goodness of fit is difficult to quantify for ABMs because each SMD objective function emphasizes different data characteristics.The paper introduces a general approach to method comparison that addresses these concerns.Source: §3 · p. 7

3.1 Loss Function Construction and Comparison Procedure

The comparison evaluates calibration methods by how closely their estimated parameters recover known true values in simulated-data experiments. The tests vary model complexity and parameter count, while accounting for non-ergodic models through Monte Carlo replications.

3.1 Loss Function Construction and Comparison Procedure

  • The idealized test uses simulated data generated with known true parameters, treating estimates closer to those values as superior.The authors argue that methods performing well in this setting are more likely to perform well under model misspecification.Source: §3.1 · pp. 7–8
  • Calibration methods are compared by the loss associated with their parameter estimates, allowing direct comparison for a given model and true parameter set.The authors describe this as a principled approach that provides meaningful insights, not a definitive comparison tool.Source: §3.1 · p. 8
  • The experiments vary model dynamics and the number of true parameters because both affect test difficulty.The empirical time-series length is held fixed; the authors say the other factors generally dominate.Source: §3.1 · pp. 7–8
  • For non-ergodic models, each experiment compares one simulated data realization with an ensemble of Monte Carlo replications for each candidate parameter set.The proxy realization uses a seed distinct from those used for the replication ensemble.Source: §3.1 · pp. 7–8

3.2 Implemented Models

The study uses inexpensive models to compare calibration methods before applying promising approaches to a roughly 100-parameter UK housing-market ABM. The models span simple time-series dynamics, structural breaks, heterogeneous traders, and a large-scale economic application.

3.2 Implemented Models

  • The initial inexpensive models enable broad testing before likely successful methods are applied to the more computationally expensive UK housing-market ABM.Methods that perform poorly on the simpler models are expected to perform poorly in the housing-market model, so they are not tested there.Source: §3.2 · p. 8

3.2.1 AR(1) Model

  • The AR(1) model is a single-parameter baseline with limited dynamics, included as a case where all methods are expected to perform well.The empirical time-series length is not varied in the experiments.Source: §3.2.1 · pp. 8–9

3.2.2 ARMA(2, 2)-ARCH(2) Model

  • The ARMA(2, 2)-ARCH(2) model tests simulation-based methods on richer dynamics and a larger parameter space, despite being amenable to analytical estimation.It is a more nuanced successor to the AR(1) model within the traditional econometric time-series class.Source: §3.2.2 · p. 9

3.2.3 Random Walks with Structural Breaks

  • The random walk with structural breaks tests methods against non-stationarity and structural breaks, which are expected to pose a significant challenge.Earlier work used this model to compare objective-function values for limited parameter combinations, rather than to calibrate it.Source: §3.2.3 · p. 9

3.2.4 Brock and Hommes (1998) Model

  • The Brock and Hommes heterogeneous-agent model is an inexpensive asset-market ABM that can produce sophisticated, including chaotic, dynamics.The study focuses on strategy-specific trend-following and bias parameters, alongside the shared switching-rate and interest-rate parameters.Source: §3.2.4 · pp. 9–10

3.2.5 INET Oxford Housing Market Model

  • The roughly 100-parameter UK housing-market ABM is a central test because calibrating large-scale models remains an open problem and is rare in the literature.The model is too sophisticated to describe in this section; the authors point readers to its working paper and source code.Source: §3.2.5 · p. 10

3.3 Implemented Methods

The study represents major calibration approaches through simulated minimum distance (SMD), alternative objective functions and optimizers, and Bayesian estimation. It compares these methods because SMD optimization is difficult and computationally expensive, while Bayesian estimation remains relatively uncommon in ABM calibration.

3.3 Implemented Methods

  • The study includes SMD objective functions and Bayesian estimation to represent major areas of the calibration literature.SMD is described as the most popular ABM calibration paradigm; Bayesian inference is relatively rare in this literature.Source: §3.3 · p. 10, §3.3.1 · p. 10, §3.3.3 · pp. 11–12

3.3.1 SMD Objective Functions

  • The SMD comparisons include MSM, GSL-div, and MIC, which use moments, temporal-pattern distributions, and Markov-process cross-entropy, respectively.MSM is a benchmark despite requiring arbitrary moment selection; MIC constructs an N-th-order Markov process using context tree weighting.Source: §3.3.1 · pp. 10–11

3.3.2 Optimisation Algorithms

  • SMD optimization uses particle swarm and the Knysh–Korkolis surrogate approach because simulation-based objectives preclude gradient methods and heuristic search is costly.Heuristics do not guarantee a global minimum; the surrogate approach samples the parameter space, approximates the objective with radial basis functions, then iteratively evaluates the original objective using CORS.Source: §3.3.2 · p. 11

3.3.3 Bayesian Estimation

  • Bayesian estimation uses KDE to estimate the likelihood of observed data under simulated-data distributions, then samples the posterior with random-walk Metropolis–Hastings.The study uses the posterior mean as its point estimate because it is optimal for squared-error loss.Source: §3.3.3 · p. 12

4 results and discussion

This section presents the computational-experiment results and discusses their implications.

4 results and discussion

  • The section presents results from the computational experiments described earlier.The passage does not specify any findings.Source: §4 · p. 12
  • The experiments provide the basis for the results reported in this section.The passage introduces the results after describing the comparison procedure.Source: §4 · p. 12
  • The section also discusses implications of the experimental results.The passage does not state what those implications are.Source: §4 · p. 12

4.1 Simple Time Series Models

The section compares calibration methods across simple time-series models, finding that Bayesian estimation generally gives the strongest results, while SMD performance varies by model and parameter set. The experiments also reveal challenges from flat objective-function regions and increasing parameter uncertainty.

4.1.1 AR(1) Model

  • Bayesian estimation had the lowest loss for AR(1), followed by MIC, MSM, and GSL-div; all methods estimated its single parameter close to truth.The loss-function ranking is reported in the text alongside Table 1.Source: §4.1.1 · pp. 12–13, Table 1 · p. 12

4.1.2 ARMA(2, 2)-ARCH(2) Model

  • For the ARMA(2, 2)-ARCH(2) two-parameter set, Bayesian estimation ranked best by loss, while GSL-div struggled to identify a0.The authors attribute GSL-div's poor a0 estimate to the objective function's limited response to changes in that additive constant.Source: §4.1.2 · pp. 13–14, Table 2 · p. 15
  • With five free parameters, Bayesian estimates remained reasonably close to truth but became more uncertain; SMD estimates diverged and optimizers disagreed, suggesting local minima.The authors suggest additional data or Monte Carlo replications could address the greater uncertainty.Source: §4.1.2 · pp. 14–15, Table 3 · p. 16

4.1.3 Random Walks with Structural Breaks

  • For the structural-break location τ, Bayesian estimation had the best loss; MSM estimated the break more closely than MIC, while GSL-div favored a broad incorrect interval.The GSL-div objective curve favored values roughly from 180 to 700 rather than the true value.Source: §4.1.3 · p. 17, Table 4 · p. 17
  • For the random-walk second parameter set, Bayesian estimation's advantage mostly persisted, but a particle-swarm GSL-div estimate was closer to truth by coincidence amid a broad flat surface.The authors caution that even global SMD minima may not estimate true parameters; MSM is poorly suited to regime changes, while MIC's loss was nearly four times smaller.Source: §4.1.3 · pp. 17–19, Table 5 · p. 18

4.1.4 Brock and Hommes (1998) Model

  • In the Brock–Hommes model, Bayesian estimates tracked true values as uncertainty rose with parameter-set size; Bayesian ranked best for set 1, but GSL-div ranked best by loss for set 2.For set 1, MSM was reasonable while MIC and GSL-div were not; the set-2 GSL-div result came despite evidence of local-minimum convergence.Source: (1998) model · pp. 20–21, Table 6 · p. 20, Table 7 · p. 21

4.1.5 Overall Assessment

  • Across the tested models and parameter sets, Bayesian estimation generally produced estimates near the true values and outperformed SMD methods, with one Brock–Hommes exception.SMD estimates were less consistent, and identifying global minima became problematic, especially in higher dimensions.Source: §4.1.5 · p. 21

4.2 INET Oxford Housing Market Model

The housing-model experiments apply Bayesian calibration to tractable subsets of 4 and 19 free parameters. Estimates were reasonable for 11 of 19 parameters, while goodness-of-fit tests were notably weaker for the larger set.

4.2 INET Oxford Housing Market Model

  • The study examined two tractable subproblems of a 100-parameter model, with 4 and 19 free parameters respectively.The authors describe 19 free parameters as ambitious relative to prior similar-scale attempts, which had considered no more than eight.Source: §4.2 · p. 21

4.2.1 Free Parameter Set 1

  • For the first parameter set, Bayesian estimates remained reasonably close to true values despite the housing model’s greater sophistication.The set prioritised Market Average Price Decay, Sale Epsilon, P Investor, and Min Investor Percentile.Source: §4.2.1 · p. 22, Figure 12 · p. 22

4.2.2 Free Parameter Set 2

  • For the 19-parameter set, 3 estimates were within 5% of the explored range, 8 were within 5–15%, and 8 exceeded 15%, including 5 above 35%.The authors classify the first 11 estimates as highly or reasonably accurate and deem calibration unsuccessful for the remaining eight.Source: §4.2.2 · pp. 22–23, Figure 13 · p. 23
  • Bayesian estimation produced reasonable estimates for 11 of the 19 housing-model parameters, although it used one time series and strong independence assumptions.The authors note that previous attempts on models of similar scale had calibrated at most eight free parameters.Source: §4.2.2 · p. 24

4.2.3 Goodness of Fit Tests

  • The KS test matched simulated and artificial-data distributions in most replications for set 1, but only a minority for set 2.The authors associate the weaker set-2 fit with inaccurate estimates for eight parameters.Source: §4.2.3 · p. 24, Figure 14 · p. 24

5 conclusions and future work

The authors conclude that SMD methods perform less compellingly than surveys suggest, while Bayesian methods perform better across many settings but falter with large models and many free parameters. They call for further development of calibration methods and targeted improvements to Bayesian estimation.

5 conclusions and future work

  • Robust, general ABM calibration methods remain undeveloped, and multimodal posteriors may affect the accuracy of point estimates.The paper also notes a single-time-series constraint in the housing-model exercise.Source: §5 · pp. 24–25
  • SMD methods delivered less compelling performance than contemporary surveys suggest, and their estimators may have significant bias in some situations that cannot yet be characterised a priori.The authors connect this gap to scarce matched computational comparisons and limited assessment of estimate uncertainty or accuracy.Source: §5 · p. 25
  • Bayesian methods performed more compellingly across a wide range of circumstances, but failed with a large-scale model and a high-cardinality free-parameter set.The authors suggest focusing future work on Bayesian estimation and improving existing Bayesian techniques.Source: §5 · p. 25
  • The authors recommend improving Grazzini et al. (2017), particularly by relaxing independence assumptions and capturing temporal dependencies when comparing datasets.Source: §5 · p. 25

a technical details

The appendix records technical experiment settings intended to support reproduction, including artificial-data preparation, calibration configurations, and adjustments for the housing model.

a.1.1 Dataset Construction

a.1.2 Experimental Procedure

  • Empirical and simulated time series were each set to length 1,000, while ensembles contained 250 replications for SMD and 100 for Bayesian estimation.The chosen series length balances robust calibration with empirical availability; ensemble size balances simulation cost and Monte Carlo variance.Source: Dataset Construction · p. 30
  • SMD generally required more replications because each series added one estimated quantity, whereas each Bayesian series added 1,000 observations to the KDE sample.Source: Dataset Construction · p. 30
  • Bayesian estimation used four independent Metropolis–Hastings chains of 5,000 samples each, discarding 1,500 per chain for a combined 14,000-point sample.Source: Dataset Construction · p. 30

a.1 Simple Time Series Models

  • For two models, L = 2 was required by memory limits, though the cited assessment considered that setting reasonable but not necessarily ideal.Source: Method Hyperparameters · p. 31

a.2 INET Oxford Housing Market Model

  • The housing method was restricted to one output series, so the study used HPI, discarded its initialization transient, and calibrated on first differences because it was non-stationary.Housing-model computation also reduced replications to 50; parameter set 1 used 7,000 retained samples and set 2 used 17,000.Source: Method Hyperparameters · pp. 31–32
Loading 1902.05938v1…