Source-linked AI summary
The Model Confidence Set package for R
Mauro Bernardi, Leopoldo Catania
TL;DR
With many competing model specifications, identifying the best fitting model under a chosen criterion is difficult. The paper presents the MCS R package, which implements sequential EPA tests using a user-specified loss function. In its VaR application, competing models have similar one-step-ahead 1% VaR performance for two index series.
Problem
Method
The MCS package sequentially tests equal predictive ability under a user-specified loss function to construct a superior set of models.
Source: §2 · p. 3, §6 · p. 16Results
For the SXA1E and SXP1E indexes, one-step-ahead 1% VaR forecasting performance is described as quite similar across competing models.
Source: §5 · p. 14Takeaways & Limitations
The package supports flexible model comparisons using different loss functions and iterative selection of a superior set.
Source: §6 · p. 16The AR(1) mean parameter φ is assumed to satisfy |φ| < 1 to preserve stationarity.
Source: §3 · p. 5
Abstract
from arXiv · showhide
This paper presents the R package MCS which implements the Model Confidence Set (MCS) procedure recently developed by Hansen et al. (2011). The Hansen's procedure consists on a sequence of tests which permits to construct a set of 'superior' models, where the null hypothesis of Equal Predictive Ability (EPA) is not rejected at a certain confidence level. The EPA statistic tests is calculated for an arbitrary loss function, meaning that we could test models on various aspects, for example punctual forecasts. The relevance of the package is shown using an example which aims at illustrating in details the use of the functions provided by the package. The example compares the ability of different models belonging to the ARCH family to predict large financial losses. We also discuss the implementation of the ARCH--type models and their maximum likelihood estimation using the popular R package rugarch developed by Ghalanos (2014).
1. Introduction
With many competing volatility models, choosing a clear winner by predictive ability can be difficult. The paper introduces an R package implementing the Model Confidence Set procedure and outlines its presentation and application.
1. Introduction
- The expanding range of volatility specifications leaves econometricians needing a statistical way to identify the best models under a chosen criterion.The introduction notes that predictive comparisons may not clearly distinguish models, including because models can be statistically equivalent or data may not discriminate among them.Source: §1 · pp. 1–2
- The Model Confidence Set procedure sequentially tests equal predictive ability and retains a Superior Set of Models when the null is not rejected at confidence level α.The tests use an arbitrary loss function, allowing comparisons on different model characteristics.Source: §1 · p. 2
- The MCS R package integrates model comparison and iterative selection, with tools for specifying different loss functions.The paper proceeds from the procedure and ARCH estimation to package use, an empirical application, and conclusions.Source: §1 · pp. 2–3
2. The Model Confidence Set procedure
The Model Confidence Set procedure uses user-chosen losses to compare competing models and sequentially eliminate a worst model when equal predictive ability is rejected. It stops when EPA is accepted for the remaining set.
2. The Model Confidence Set procedure
- The procedure tests equal predictive ability for a set of competing models using loss differentials, with the loss function chosen to target a desired aspect such as forecasts or in-sample fit.For model i, the loss differential against model j at time t is d_ij,t = ℓ_i,t − ℓ_j,t.Source: §2 · p. 3
- The EPA hypotheses can be expressed either as equality of all pairwise expected loss differentials or as equality of each model’s expected loss relative to the set average.The procedure uses corresponding test statistics, with nonstandard null distributions estimated by bootstrap.Source: §2 · p. 4
- The package bootstraps variance estimates with 5,000 block resamples and allows the user to specify the block length.The stated default block length is based on fitting an AR(p) process to the loss-differential terms.Source: §2 · p. 4
- When EPA is rejected, the procedure eliminates the worst model using a rule coherent with the test statistic, then repeats the test on the reduced set.It stops when EPA is accepted and returns the remaining models as the Superior Set of Models.Source: §2 · p. 5
3. Model specifications
The paper describes ARCH-type volatility specifications and their estimation and forecasting in R with rugarch. Its illustrative model set varies volatility dynamics, distributional assumptions, and mean specification.
3. Model specifications
- The illustrative comparison uses ARCH-type models for financial returns, varying conditional mean, volatility dynamics, and innovation distributions.The general setup is an AR(1)-GARCH-in-mean specification, with the conditional variance dynamics supplied by different ARCH-type models.Source: §3 · pp. 5–6
- The specifications cover symmetric and asymmetric volatility dynamics, including GARCH, IGARCH, EGARCH, GJR-GARCH, APARCH, and component-GARCH models.GJR-GARCH persistence depends on the conditional distribution’s asymmetry; APARCH nests several popular ARCH parameterizations, and component-GARCH separates permanent and transitory variance.Source: §3 · pp. 6–7
3.1. GARCH model estimation and forecast in R
- The paper estimates model parameters by maximum likelihood and uses rugarch functions to specify, fit, and forecast GARCH models in R.The exposition is illustrative, and readers may use another package or their own functions.Source: §3 · p. 8, §3.1 · pp. 8–9
- rugarch supports rolling one-step-ahead forecasts with a user-defined refit window and forecast-series length.The paper illustrates these functions using the STOXX North America 600 index.Source: §3.1 · p. 9
4. Using the package
The MCS package compares models using a loss matrix and user-selected loss function, then returns a superior set through iterative EPA testing. The paper demonstrates this workflow for GARCH-based VaR forecasts.
4. Using the package
- MCSprocedure() takes a loss matrix measuring each model’s performance over an evaluation period and uses it to construct the superior set.The function’s inputs include the confidence level α, bootstrap sample count B, and EPA test statistic choice.Source: §4 · p. 9, §4.3 · p. 12
4.1. Comparing GARCH models using MCS
- The GARCH comparison example builds 40 specifications from five volatility models and eight innovation distributions, then evaluates 1% VaR forecasts over 2,000 observations.The example uses a refit window of 200 observations and constructs model losses with LossVaR().Source: §4.1 · p. 10
4.2. Loss functions
- The package provides LossVaR() for quantile forecasts, LossVol() for volatility forecasts, and LossLevel() for level forecasts; users may also supply their own loss matrix.The loss functions take realized observations and evaluated model outputs; the MCS procedure also applies to in-sample studies.Source: §4.2 · p. 11
4.3. Constructing the Superior Set of Models
- MCSprocedure() returns an S4 object of class SSM, with Tmax and TR available as statistics for testing EPA at each iteration.The Loss input contains model loss series, α sets the confidence level, and B specifies the number of bootstrap samples.Source: §4.3 · p. 12
5. Application
The application uses 160 ARCH-type specifications to forecast 1% VaR for four international equity indices and applies MCS to identify superior model sets. Dynamic VaR combination improves both violation-loss measures over static averaging, and improves the exceedance ratio for two indices.
5. Application
- Four worldwide stock indices—the Asia/Pacific 600, North America 600, Europe 600, and Global 1800—provide daily price data from 31 December 1991 through 24 July 2014, totaling 5,874 observations.Source: §5 · p. 14
- The study applies MCS to 160 specifications combining 10 ARCH dynamics, 8 conditional distributions, and in-mean options across four indices.Models produce rolling one-day-ahead forecasts of 1% VaR over 2,000 validation observations.Source: §5 · p. 14
- For SXA1E and SXP1E, the SSM is relatively homogeneous, while SXW1E and SXXP yield more discriminating model selections.GARCH, IGARCH, C-GARCH, and N-GARCH specifications were eliminated for SXW1E; GJR-GARCH was absent from the SXXP SSM.Source: §5 · p. 14, Table 1 · p. 15
- VaR dynamic combination has lower ADmean and ADmax than VaRavg for all four indices, and improves the AE ratio for SXW1E and SXXP.VaRavg averages forecasts across all 160 models; VaRDyn averages across models in the SSM.Source: §5 · p. 15
6. Conclusion
The paper presents MCS as a flexible R package for selecting superior models under a user-specified loss function. Its application uses the procedure to reduce the model set before dynamic VaR averaging, which improves forecast performance over a simple average.
6. Conclusion
- Table 2 presents VaR backtesting measures for dynamic VaR combination and static average forecasts.Source: Table 2 · p. 16
- The example reports that VaR dynamic model averaging improves VaR forecast performance after MCS reduces the candidate model set.Table 2 reports backtesting measures comparing the dynamic combination VaRDyn with the static average VaRAvg.Source: §6 · p. 16, Table 2 · p. 16
- MCS constructs a Superior Set of Models with equal predictive ability under a user-supplied loss function, and permits flexible choices of models and loss.The package's flexibility lets researchers focus on the initial model set rather than software constraints.Source: §6 · p. 16