Agent-Based Model Calibration using Machine Learning Surrogates

Francesco Lamperti; Andrea Roventini; Amir Sani

Source-linked AI summary

Agent-Based Model Calibration using Machine Learning Surrogates

Francesco Lamperti, Andrea Roventini, Amir Sani

arXiv:1703.10639v2Mar 30, 2017econ.GN

TL;DR

The paper addresses calibration and parameter-space exploration when relevant statistical properties are unknown and model simulation is computationally expensive. It combines supervised machine learning in a surrogate meta-model that predicts areas to focus on, with reported completion times of less than a minute and reduced exploration cost.

Problem
Relevant statistical properties are a priori unknown, while simulating the model is the most computationally expensive step in calibration and validation procedures.
Method
The approach combines supervised machine learning with a surrogate meta-model for agent-based-model calibration and parameter-space exploration.
Results
The surrogate predicts areas of parameter space to focus on with reasonably good results, takes less than a minute to complete, and is more effective at reducing the relative cost of exploration.
Takeaways & Limitations
The approach provides an efficient and user-friendly procedure for exploring the parameter space of agent-based models.
Takeaways & Limitations
Kriging is difficult to apply to large-scale models, and kriging meta-models can suffer from smoothness assumptions that collapse interesting patterns.

Abstract

from arXiv · show

Taking agent-based models (ABM) closer to the data is an open challenge. This paper explicitly tackles parameter space exploration and calibration of ABMs combining supervised machine-learning and intelligent sampling to build a surrogate meta-model. The proposed approach provides a fast and accurate approximation of model behaviour, dramatically reducing computation time. In that, our machine-learning surrogate facilitates large scale explorations of the parameter-space, while providing a powerful filter to gain insights into the complex functioning of agent-based models. The algorithm introduced in this paper merges model simulation and output analysis into a surrogate meta-model, which substantially ease ABM calibration. We successfully apply our approach to the Brock and Hommes (1998) asset pricing model and to the "Island" endogenous growth model (Fagiolo and Dosi, 2003). Performance is evaluated against a relatively large out-of-sample set of parameter combinations, while employing different user-defined statistical tests for output analysis. The results demonstrate the capacity of machine learning surrogates to facilitate fast and precise exploration of agent-based models' behaviour over their often rugged parameter spaces.

1 Introduction

The paper addresses the computational difficulty of exploring and calibrating agent-based models with complex, rugged parameter spaces. It proposes a machine-learning surrogate built through adaptive sampling to approximate model behaviour and identify parameter combinations matching desired outputs.

Approach: The proposed approach combines supervised machine learning with intelligent sampling to construct a fast surrogate mapping parameterizations to user-defined output-fit statistics.It uses labeled and unlabeled parameter combinations in a semi-supervised procedure.
Motivation: ABM calibration is computationally demanding because models can contain many parameters, long simulations, stochasticity, and emergent relationships.Full-factorial exploration is practically impossible, while calibration requires repeated simulation, output analysis, and parameter search.
Motivation: Rugged parameter spaces with local maxima, minima, and saddle points limit gradient-based and smooth surrogate methods.The paper specifically notes that kriging struggles with large parameter spaces and smoothness assumptions that do not fit ABMs.
Results: The method reduces computation time while accurately approximating the ABM, filtering parameter combinations, assessing parameter importance, and supporting calibration.Calibration targets parameter combinations that generate user-desired properties rather than a single optimal parameter.
Approach: The surrogate is learned over multiple rounds: an initial labeled seed is expanded by evaluating selected unlabeled parameter points and retraining the model.The procedure continues until the specified evaluation budget is reached, with the surrogate directing subsequent selections.
Results: Semi-supervised active learning incrementally improves the surrogate while maximizing information gained over the parameter space.The introduction presents this as a basis for efficient and user-friendly ABM exploration and calibration.

2 Calibration and validation of agent-based models: the case for surrogate modelling

The paper motivates surrogate modelling as a response to the high computational cost and unsuitable smoothness assumptions of conventional ABM calibration and exploration methods. It combines surrogate modelling with intelligent sampling to train a computationally efficient approximation suited to complex parameter spaces.

Computational challenge: ABM calibration and validation remain computationally expensive because simulating the model is typically the most costly step.Some macroeconomic ABMs may require weeks for a single Monte Carlo exercise.
Methodological gap: Standard design-of-experiments methods are poorly matched to ABMs because smoothness, linearity, and normality assumptions generally do not hold.The resulting high-dimensional estimation problem also converges slowly under smooth-function assumptions.
Computational challenge: Increasingly complex ABMs require many parameters to represent micro-founded, multi-sector, and multi-country phenomena.This complexity makes direct estimation and global sensitivity analysis computationally infeasible in many settings.
Proposed approach: The proposed procedure links computation-time reduction with calibration and validation criteria through an efficiently trained surrogate.Its two central pillars are surrogate modelling and intelligent sampling.
Proposed approach: Non-parametric boosted trees are used to model rugged ABM parameter spaces without relying on smoothness assumptions.The approach extends earlier economic applications that used kriging to construct ABM surrogate meta-models.
Methodological gap: Kriging-based surrogates are limited by their difficulty with more than about a dozen parameters and by smoothness assumptions that can collapse rugged patterns.Large parameter spaces may therefore force modelers to fix a subset of parameters arbitrarily.

3 Surrogate modelling methodology

The methodology combines surrogate modelling with intelligent sampling to approximate ABM calibration outcomes and explore parameter spaces under a limited evaluation budget. It uses machine learning to distinguish or quantify calibration outcomes and iteratively select informative parameter vectors.

Performance evaluation: Performance is evaluated with criteria suited to binary and real-valued outcomes, including F1-score and mean-squared error.Binary calibration identifies whether a condition is met, whereas regression quantifies a model property.
Calibration problem: ABM calibration searches for parameter vectors whose simulated outputs satisfy user-defined statistical or loss-based conditions.The paper distinguishes binary classification outcomes from real-valued regression outcomes.
Surrogate model: The surrogate approximates f(x) = v◦m(x) using a limited budget of evaluations of the true ABM.After training, it supports exploration of ABM behaviour across the parameter space.
Surrogate model: The approach uses XGBoost, an ensemble of boosted classification and regression trees, avoiding smoothness assumptions associated with kriging.Boosting gives greater weight to difficult-to-learn samples, while tree ensembles can learn nonlinear knife-edge properties.
Intelligent sampling: Pool-based active learning selects parameterizations iteratively according to surrogate performance until the predefined evaluation budget is reached.Each round retrains the surrogate using the accumulated parameter vectors and labels.
Practical boundary: The surrogate remains an approximation, so identified positive calibrations and nearby regions should be studied with the original ABM.This recommendation defines the practical boundary of surrogate-based exploration.

4 Surrogate modelling examples: The Brock and Hommes model

The Brock and Hommes model represents heterogeneous traders whose forecasting, biases, trend responses, information costs, and strategy switching shape asset-price dynamics. The surrogate identifies influential parameters and predicts calibration outcomes efficiently across many parameterizations.

Model setup: The Brock and Hommes model contains heterogeneous traders using different strategies, including fundamentalist and chartist behaviours.The application uses two agent types whose behaviour varies with trend components, biases, and perfect-forecasting costs.
Agent behaviour: Trader expectations depend on past deviations from fundamental prices and are represented with trend components and biases.The model also permits rational forecasting at a cost and strategy switching based on evolving profitability.
Parameter importance: The most relevant parameters for fitting the empirical return distribution are traders’ trend-attitude parameters g1 and g2, followed by their biases b1 and b2.The intensity-of-choice parameter β, risk attitude α, and past-profit weight ω appear less relevant within the explored range.
Parameter importance: The weight on past profits, ω, is relatively unimportant for shaping empirical model performance under the specified conditions.The result concerns the parameter-importance analysis for matching the empirical distribution of returns.
Surrogate performance: An F1-score of around 0.80 is reached with 2500 training points, while approximately 75% of positive out-of-sample calibrations are found.The F1-score averages the share of true positive calibrations and the share of predicted positives that are correct.
Surrogate performance: True Positive Ratio is around 70% with 500 training points and almost 95% on average with 2500 points.The surrogate’s classification performance worsens as the training sample size is reduced.
Computational performance: Regression requires roughly three times the modelling time of classification, while prediction time is nearly equivalent.Despite this difference, prediction is described as nearly costless for parameter-space exploration.
Computational performance: Across 10000 out-of-sample points, the learned surrogate is 500X faster on average in prediction and can be reused without additional training.The reported computational gains are expected to increase for more complex and expensive-to-run ABMs.

5 Surrogate modelling examples: the Islands model

The Island model represents technological change through heterogeneous firms that explore, imitate, and mine technologies on an infinite lattice. Surrogate modelling evaluates parameter conditions for growth and fat-tailed outcomes while approximating model behaviour and accelerating exploration.

5.1 The Island growth model: Heterogeneous firms on a two-dimensional infinite lattice discover, imitate, and exploit technological islands to produce a homogeneous good.Agents can mine, explore for new islands, or imitate known technologies; GDP sums production across islands.
5.1 The Island growth model: New-island productivity combines distance from the origin, cumulative learning, radical innovations, and stochastic incremental innovation.The radical-innovation component uses W, while zero-mean noise captures high-probability low jumps.
5.1 The Island growth model: Endogenous exponential growth requires technological opportunities, path-dependency, and information diffusion to exceed minimum thresholds.The relevant parameters are island density π, radical-innovation likelihood λ, carried-over knowledge ϕ, and diffusion intensity ρ.
5.2 Experimental design and empirical setting: The model uses eight input parameters, while the calibration experiment keeps the number of firms fixed at 50 and varies behavioural-rule parameters.The explored parameter ranges are reported in Table 2.
5.2 Experimental design and empirical setting: The surrogate learns either a binary target indicating whether user-defined GDP conditions hold or a real-valued target for the generated GDP time series.The two conditions concern sustained growth and fat tails.
5.3 Results: 2500 training points identify 90% of true-positive calibrations with MSE below 0.08 in the real-valued setting and about 80% in the binary setting.The largest true positive ratios are 0.9 for the real-valued case and 0.8 for the binary case.
5.3 Results: The surrogate is 3750 times faster than the full Island agent-based model, with performance improving as the training sample grows.MSE becomes remarkably low above 1000 points, while the true positive ratio steadily increases with sample size.

6 Discussion and concluding remarks

The paper proposes a machine-learning surrogate approach combining supervised learning and intelligent sampling for ABM calibration and parameter-space exploration. Applied to Brock–Hommes and Islands models, it delivers fast exploration while addressing limitations of kriging-based approaches.

The approach combines supervised machine learning and intelligent sampling to construct a surrogate meta-model for ABM calibration and parameter-space exploration.
Unlike kriging, the approach is designed for large parameter spaces, where kriging becomes difficult beyond 20 parameters and may require fixing a subset a priori.
Machine-learning surrogates avoid kriging’s smoothness assumptions, which can collapse non-linear or ragged patterns and falsely relate parameters to model behaviour.
The binary outcome setting is faster, whereas the real-valued setting offers more explanatory power; the paper reports no practical trade-off between processed information and surrogate-performance improvements.
Surrogate prediction exercises take less than a minute, allowing arbitrarily large parameter-combination sets to be assessed after a relatively small training sample is built.
The surrogate reduces exploration costs more effectively for the computationally intensive Islands model than for Brock and Hommes, suggesting larger gains as underlying model complexity increases.
The authors present this work as a first step toward broader machine-learning assessment of ABMs and a standardized, robust calibration and validation procedure.