Source-linked AI summary
Matrix-Aware Proper Scoring Rules and Significance Testing for Correlation and Covariance Forecasts in Python
Vinh Nguyen
TL;DR
Correlation and covariance matrix forecasts are important but difficult to evaluate because standard metrics may be improper, backtests may leak information, and score sequences are dependent. corrscore addresses these issues with matrix-aware scoring, zero-overlap backtesting, and dependence-robust significance tests; in its worked example, all three tests decisively favored persistence over shrinkage.
Problem
Evaluating correlation and covariance matrix forecasts lacks settled tooling because naive metrics may be improper, walk-forward windows may leak information, and dependent score differences require specialized tests.
Method
corrscore combines matrix-aware energy and variogram scores, a geometry-aware variogram variant, zero-overlap walk-forward backtesting, and circular-bootstrap, Diebold–Mariano, and Model Confidence Set procedures.
Results
In the worked example, mean variogram score was 3.0274186774073626 for persistence versus 3.52280921068856 for shrinkage, and all three significance tools favored persistence.
Takeaways & Limitations
The package provides a purpose-built bundle for evaluating correlation and covariance forecasts without reimplementing scoring, leakage-safe backtesting, and significance-testing machinery.
Abstract
from arXiv · showhide
Forecasting a correlation or covariance matrix is common in risk management and portfolio construction, but evaluating such a forecast correctly is not routine: naive matrix-comparison metrics are not proper scoring rules, walk-forward evaluation windows are easy to overlap with the estimation window in ways that silently leak information, and significance testing on serially dependent forecast-error sequences needs machinery few analysts implement from scratch. corrscore is a Python package that provides matrix-aware implementations of two established proper scoring rules for this setting -- the energy score and the variogram score -- dispatched across a closed-form tractability spectrum (point, discrete-mixture, and isotropic-Gaussian-mixture forecasts are scored exactly; a general Monte Carlo ensemble falls back to sampling), a geometry-aware variant of the variogram score built from the affine-invariant distance on the correlation manifold, a zero-overlap-by-construction walk-forward backtest harness, and a bundled significance-testing suite (circular block bootstrap, the Diebold-Mariano test, and the Model Confidence Set). We describe the package's design, its point of departure from the existing scoringRules and properscoring packages, and walk through a complete worked example.
1 Introduction
The paper identifies three recurring evaluation problems for correlation and covariance forecasts: improper matrix metrics, information leakage in walk-forward windows, and invalid i.i.d. significance tests for dependent scores. corrscore addresses them with matrix-aware scoring rules, zero-overlap backtesting, and bundled dependence-robust testing tools.
- Correlation and covariance forecasts underpin portfolio construction and risk-management workflows, including optimization, value-at-risk, and stress testing.
- Naive entrywise metrics are generally not proper scoring rules and cannot principledly evaluate probabilistic ensemble or mixture forecasts.The energy and variogram scores provide established alternatives, although they were not designed specifically for matrix-valued correlation structure.
- Overlapping estimation and evaluation windows can leak information and reverse empirical conclusions, motivating a dedicated backtest harness.
- Serial dependence in walk-forward score differences makes plain paired t-tests understate variance, requiring block bootstrap and Diebold–Mariano methods.Comparisons among more than two models additionally require multiple-comparison correction.
- corrscore bundles matrix-aware proper scores, a geometry-aware variogram variant, structurally zero-overlap backtesting, and three significance-testing tools without fitting models or handling market data.
2 Background: Proper Scoring Rules for Matrix-Valued Forecasts
The background defines propriety through expected-score minimization under the true predictive distribution and places both implemented scores within established energy-distance constructions. The energy score uses Frobenius distance, while the variogram score operates on pairwise differences with a separate propriety argument.
- A scoring rule is proper when reporting the true predictive distribution minimizes expected score within the forecast class.
- Both implemented scores use the general energy-distance recipe, which is proper when its kernel is conditionally negative definite.
- The energy score sets the kernel to Frobenius distance between matrices.
- The variogram score directly evaluates vectors of pairwise absolute differences and has a separate propriety argument that does not require conditional negative definiteness.
3 Score Definitions and the Closed-Form Tractability Spectrum
corrscore represents forecasts through four kinds and dispatches scoring across exact closed forms and Monte Carlo fallback. It adapts variogram scoring to free matrix entries and adds a boundary-sensitive geometric transform for correlations near ±1.
- 3.1 Forecast representation: Forecasts use tagged dictionaries for point, discrete-mixture, isotropic-Gaussian-mixture, and general ensemble representations.
- 3.1 Forecast representation: Point and discrete-mixture forecasts are exactly tractable, with mixture expectations reduced to finite weighted sums without simulation.
- 3.1 Forecast representation: Isotropic-Gaussian mixtures retain an exact energy-score calculation through a noncentral-chi mean-norm formula, whereas general ensembles require Monte Carlo pairwise summation.
- 3.2 The energy score: The matrix energy score applies Frobenius distance to K × K matrices and dispatches across all four forecast kinds.
- 3.3 The variogram score: The variogram score indexes only the K(K−1)/2 free upper-triangle correlation entries, excluding the fixed diagonal and mirrored lower triangle.
- 3.3 The variogram score: Variogram scoring is exact for point and mixture forecasts but uses Monte Carlo for isotropic-Gaussian mixtures and ensembles, with cost scaling in sample count and matrix dimension.
- 3.4 A geometric variogram score: The geometric variogram score applies a signed affine-invariant transform that stretches correlations near ±1, where flat scoring can saturate.
- 3.4 A geometric variogram score: Its discrimination advantage is conditional: it concentrates in episodes where realized correlations approach the boundary rather than appearing uniformly in unconditional averages.
4 Backtesting and Significance Testing
The backtest harness evaluates forecasts against shared outcomes computed strictly after each forecast origin, preventing estimation–evaluation overlap by construction. Its significance suite combines circular block bootstrap, corrected Diebold–Mariano testing, and Model Confidence Set elimination for dependent forecast scores.
- 4.1 The zero-overlap backtest harness: The walk-forward harness evaluates named forecasting methods against a shared, model-independent ground truth across forecast origins.
- 4.1 The zero-overlap backtest harness: Ground truth is computed from a window starting strictly after each forecast origin, enforcing zero overlap structurally.
- 4.1 The zero-overlap backtest harness: The harness does not police how much historical data a caller’s forecast function uses internally.
- 4.2 Significance testing: Three complementary significance tests operate on per-origin score arrays returned by a BacktestResult.
- 4.2 Significance testing: The circular block bootstrap tests paired score differences across candidate block lengths and recommends reporting the most conservative largest p-value.
- 4.2 Significance testing: The Diebold–Mariano implementation adds the Harvey–Leybourne–Newbold correction and uses a Student-t reference distribution with n−1 degrees of freedom.
- 4.2 Significance testing: The Model Confidence Set repeatedly eliminates the single worst model while testing whether the remaining candidates differ statistically from their best member.
5 Illustrative Example
The worked example compares persistence and shrinkage forecasts on a simulated drifting-correlation panel, then applies backtesting and three significance tools. All tools agree that shrinkage is decisively worse.
- Setup: A K = 4 asset panel is simulated over 900 trading days with pairwise correlations drifting roughly between 0.15 and 0.50.The example compares deliberately simple persistence and shrinkage forecasting rules.
- Forecasts: Persistence uses a trailing 60-day sample correlation, whereas shrinkage moves that estimate 30% toward the identity matrix.The construction deliberately makes shrinkage toward independence inappropriate for the simulated process.
- Scores: Mean VS is 3.0274186774073626 for persistence and 3.52280921068856 for shrinkage.The reported variogram-score difference is 0.4954 for both bootstrap block lengths shown.
- Significance testing: Bootstrap results give mean diff=0.4954 with p=0.0000 at block lengths 5 and 10, while the Diebold-Mariano test reports stat=13.026 and p=0.0000.The bootstrap intervals are (0.4302, 0.5600) for block length 5 and (0.4267, 0.5592) for block length 10.
- Verdict: The Model Confidence Set retains persistence and eliminates shrinkage at α = 0.10, with all three significance tools agreeing on the verdict.The geometric variogram score can replace the flat score through the score_fn argument without changing the rest of the backtest workflow.
6 Comparison with Existing Software
corrscore is designed specifically for correlation-matrix forecast evaluation, extending general-purpose scoring software with matrix-native scoring, closed-form options, geometry, backtesting, and significance testing.
- Existing software: properscoring provides fundamentally univariate continuous ranked probability and Brier scores, with no multivariate energy or variogram score or correlation-matrix-specific concept.scoringRules is the closer relative because its es_sample and vs_sample functions score general d-dimensional Monte Carlo ensembles.
- Distinctive features: corrscore offers exact scoring across point, discrete-mixture, and isotropic-Gaussian-mixture forecasts, while scoringRules always samples.This is one of four differences identified for the correlation-matrix use case.
- Distinctive features: corrscore uses a K × K matrix-native API with free-upper-triangle indexing instead of requiring callers to flatten matrices into generic vectors.It also adds a geometric variogram score unavailable in the compared packages.
- Distinctive features: corrscore bundles a walk-forward backtest harness and significance-testing suite purpose-built for correlation-matrix forecasts.The bundled suite addresses evaluation workflows rather than only score computation.
7 Summary
The paper presents corrscore as a focused package for evaluating correlation and covariance-matrix forecasts, combining matrix-aware scoring, geometric evaluation, backtesting, and significance testing. It reports a tested MIT-licensed implementation.
- Summary: corrscore combines matrix-aware proper scoring rules, a geometry-aware variogram variant, and a backtest-plus-significance-testing bundle for correlation and covariance-matrix forecasts.The geometric variant is motivated and validated in a companion paper.
- Summary: The package is positioned as closing a specific gap left by existing general-purpose scoring-rule software.Its scope is evaluating forecasts rather than fitting forecasting models or handling market-data preparation.
- Validation: Sixty-two tests pass on Python 3.10 and later, including Hypothesis property-based tests and cross-checks against R reference implementations.The cross-checks cover the vendored Diebold–Mariano and Model Confidence Set implementations.
- Availability: The implementation is available under the MIT license.The paper provides the repository URL for the package.