A Practical Guide on Graphical Model Validation

1 Introduction

• The manuscript discusses graphical and statistical tools for validating general-insurance actuarial pricing models.These include calibration plots, actual-vs-expected plots, lift charts, Murphy diagrams, Bregman and deviance losses, Murphy’s decomposition, and Gini scores.Source: §1 · p. 1
• Calibration, discrimination or resolution, and risk ranking are central validation features, and their analysis depends on the correct scale and probability measure.The manuscript distinguishes policy-weighted from exposure-weighted population measures and notes that aligning the two views requires additional assumptions.Source: §1 · p. 1
• The contribution is to place established methods on common mathematical ground, clarifying their comparison and showing that many present similar statistics in different views.The methods are not new, but are often not properly mathematically formalized.Source: §1 · pp. 1–2

2.1 Loss costs per unit exposure

• Actuarial models use loss costs Y=Z/V and unit premiums Π=P/V to compare contracts with different exposure lengths; meaningful risk-based pricing expects Y and Π to be positively associated.The manuscript treats premiums as pure risk premiums covering expected loss costs, without expense, solvency, or profit margins.Source: §2.1 · pp. 2–3

2.2 The exposure-weighted Q-measure

• The population measure P averages across policies, whereas Q describes exposure-weighted averages for exposure-scaled quantities.Introducing Q accounts for dependence between loss costs and exposure.Source: §2.2 · p. 3
• Under Q, expected loss costs per unit exposure are linked to expected total loss under P, with the dependence between loss costs and exposure accounted for.Q is an exposure-weighted probability measure equivalent to P.Source: §2.2 · pp. 3–4
• Global unbiasedness of total premium for total loss is equivalent to equality of the Q-expectations of unit loss costs and unit premiums.E[Z]=E[P] iff EQ[Y]=EQ[Π].Source: §2.2 · p. 4
• When loss costs and exposure are positively correlated under P, E[Y] is below EQ[Y], so pricing from E[Y] yields too low a premium for total loss.EQ[Y]=E[Z]/E[V] under the stated relationship.Source: §2.2 · p. 4

3 Calibration

• Calibration must be assessed beyond global unbiasedness to avoid systematic cross-financing between price cohorts.PZ-calibration examines cohorts defined by total premium, while Q-calibration examines unit-premium cohorts and accounts for dependence involving exposure. The paper focuses on Q-calibration because conditional mean independence is generally questionable; Q|V-calibration is practically meaningful when exposure is known at contract inception, but lapses can make realized exposure available only ex-post.Source: §3.1 · pp. 5, 6, 8, 9, §3 · p. 5

3.1 Notions of calibration

• Q-calibration assesses whether unit-premium cohorts are self-financing, unlike PZ-calibration, which can leave within-cohort cross-subsidies between different unit premiums.PZ-calibration groups policies by total premium P, whereas Q-calibration groups them by unit premium Π. Q|V-calibration conditions on both Π and V, but its practical use depends on ex-ante exposure availability; the paper focuses on Q-calibration because conditional mean independence is doubtful in general insurance pricing.Source: §3.1 · pp. 6, 8, 9

3.2 Recalibrated unit premium

• Recalibrated unit premiums mQ(Π) and mP(Π) are calibrated; Assumption 3.6 suffices for mQ(Π) = mP(Π), while conditional uncorrelatedness of V and Y given Π is necessary.The calibration of the recalibrated premiums follows from the tower property and motivates isotonic recalibration.Source: §3.2 · p. 10

3.3 Stylized example

• The stylized construction is P-calibrated but not Q-calibrated when centered exposure-dependent residuals yield nonzero conditional correlation between V and Y given Π.It uses residuals r(Π, V) with E[r(Π, V) | Π] = 0 to obtain P-calibration, while the example assumes low-premium cohorts have lower exposure support and chooses h increasing below v0 and decreasing above it.Source: §3.3 · pp. 11–12
• For premiums at or below π0, mQ(Π) > Π; above π0, mQ(Π) < Π, corresponding to underestimated low-premium losses and overestimated high-premium losses.The covariance of V and h(V) is positive for π ≤ π0 and negative for π > π0.Source: §3.3 · p. 12
• The conditional expected loss is µ(Π, V) = Π + h(V) − E[h(V) | Π], with h increasing on (0.5, 1) and decreasing on (1, 1.5).Conditional loss costs have mean µ(Π, V), variance µ(Π, V)^2/γ, and coefficient of variation 1/√γ; their distribution is identical under P and Q.Source: §3.3 · p. 12
• The constructed example has P-calibration, mP(Π) = Π almost surely, but is miscalibrated under Q.The unit premium follows a scaled, translated beta distribution supported on (800, 1200), symmetric around π0 = 1000; exposure follows different uniform distributions on either side of π0.Source: §3.3 · p. 12

4 Graphical tools to assess calibration

• The stylized example is P-calibrated but not Q-calibrated, making detection of Q-miscalibration the focus of the graphical analysis.The manuscript treats Q-calibration, rather than P-calibration, as the relevant interest in actuarial pricing.Source: §4 · p. 13
• The unit-premium distribution under Q is important in an empirical setup, alongside its distribution under P.The section first presents both distributions before its population and sample analyses.Source: §4 · p. 13
• The population-version analysis assumes the true data-generating model is known, providing a ground-truth setting for shaping ideas and notation.The manuscript notes that this assumption is unrealistic in practice.Source: §4 · p. 13
• The sample-version analysis estimates calibration empirically from an i.i.d. test sample, separate from the data used to derive the premium rule.The analysis is out-of-sample and is understood conditionally on the fixed learning sample.Source: §4 · p. 13

4.1 Unit premium distributions under P and Q

• Under P, the unit premium Π has a scaled and translated beta distribution, with dotted decile markers obtained using K = 10.Figure 1 (lhs) shows the distribution under P.Source: §4.1 · p. 13, Figure 1 · p. 14
• Equal-probability bins for Π under P are not the desired parameter-estimation intervals when the aim is roughly equal aggregated exposure across intervals.The passage notes that equal exposure is intended to provide roughly equal information for parameter estimation, subject to additional assumptions.Source: §4.1 · pp. 13–14
• Under Q, equi-spaced quantile levels define intervals with equal average observed exposure, an approximation that is exact for continuous distributions.The section therefore uses the exposure-weighted Q-measure framework; Figure 1 (rhs) compares GΠ and GQΠ.Source: §4.1 · p. 14, Figure 1 · p. 14

4.2.1 Calibration plot

• For Π ≤ 1000, expected unit loss costs are underestimated, while for Π > 1000 they are overestimated under Q.This Q-miscalibration implies a systematic cross-subsidy from high-unit-premium policies to low-unit-premium policies.Source: Figure 2 · p. 15, §4.2.1 · p. 16
• Near π0 = 1000, the premium ordering can give a wrong risk ranking, with mQ(π0) > mQ(π0 + ε) for small ε > 0.The manuscript says risk rankings are assessed by Gini scores in Section 7.Source: §4.2.1 · p. 16

4.2.2 Lift chart

• The lift chart uses quantile level u rather than premium π on the x-axis; identical functions indicate Q-calibration, while lift reflects discrimination whether or not calibration holds.Using the global mean premium gives calibration but no discrimination and zero lift.Source: §4.2.2 · p. 16, Figure 3 · p. 17
• The area between calibration curves is not a universal error measure because the plots use different x-axis scales, and it evaluates calibration but not discrimination.The manuscript notes that the lift chart's x-axis does not depend on the selected premium scheme.Source: §4.2.2 · p. 17

4.2 Graphical tools: Population Version

• In applications, recalibrated premiums usually must be estimated from observations, so the population formulation gives way to empirical sample versions.The known-model population version is introduced to define the relevant objects before the sample version.Source: §4.2 · p. 15, §4.2.2 · p. 17

4.3 Graphical tools: Sample Version

• Sample calibration plots replace population conditional means with independent test-sample responses, but irreducible response noise makes raw scatter hard to interpret.Figure 4 plots observed loss costs against unit premiums.Source: §4.3 · pp. 17–18, Figure 4 · p. 18

4.3.1 Actual-vs-expected plot – quantile binning

• Quantile binning uses exposure-weighted premium quantiles to form approximately equal-exposure bins, then compares bin-average loss costs with premiums in the AvE plot.The bin averages reduce irreducible risk by aggregating observations.Source: §4.3.1 · pp. 18–19
• Figure 6's systematic departures from the diagonal indicate Q-miscalibration, while K=100 is noisier than K=10 and bin locations reflect premium and exposure distributions.The plot is a graphical inspection, not a statistical test or proof of miscalibration.Source: §4.3.1 · pp. 19–20, Figure 6 · p. 20

4.3.2 (Empirical) lift chart – quantile binning

• Lift charts retain AvE information but use a different x-axis scale; Figure 7 shows low-premium underestimation and high-premium overestimation.The chart uses decile and percentile binning.Source: §4.3.2 · p. 20, Figure 7 · p. 21

4.3.3 Actual-vs-expected plot – statistical smoothing methods

• Local regression smooths AvE curves with neighborhood windows and weighting, but wiggles can reflect noise or miscalibration, making plots indicative rather than definitive.Bandwidth selection controls a bias-variance trade-off.Source: §4.3.3 · pp. 21–23
• Isotonic regression needs a correct risk ranking and avoids hyperparameter selection, but noisy data can yield crude bins and overfit tails.The authors recommend merging the smallest and largest bins in real applications.Source: §4.3.3 · pp. 23–24
• Figure 9's Π ranking detects calibration problems but creates a flat central price, whereas mQ(Π) ranking fluctuates around the diagonal under response noise.With infinite sample sizes, isotonic regression on the correct ranking should coincide with the diagonal.Source: §4.3.3 · pp. 23–24, Figure 9 · p. 24

4.3.4 Double-lift charts

• Double-lift charts jointly bin premium schemes by their ratio, addressing the mismatch between separate bins; the example shows greatest overestimation on the left and underestimation on the right.The example compares Πa=Π with Πb=mQ(Π), which is Q-calibrated by construction.Source: §4.3.4 · pp. 24–25

5.1.1 Bregman divergence

• Expected Bregman loss can rank competing premiums, but preferences generally depend on the convex generator, and simultaneous dominance across generators is not expected in practice.The section proposes elementary losses as a simpler parametrized class for testing preferences across losses.Source: §5.1.1 · p. 27

5.1.2 Exposure-weighted Bregman divergence

• Exposure-weighted Bregman fitting targets the Q-mean, accounting for dependence between loss costs Y and exposure V, and ranks premiums by exposure-weighted loss.Recalibration to mQ(Π) yields forecast dominance over Π for every convex generator, making it the most accurate σ(Π)-measurable predictor under Q.Source: §5.1.2 · p. 28

5.2.1 Elementary losses

• An elementary loss is consistent but not strictly consistent for mean estimation, and measures |y−θ| when threshold θ lies between response y and forecast x.Elementary losses provide a mixture representation of Bregman loss.Source: §5.2.2 · p. 29, §5.2.1 · p. 29

5.2.2 Murphy diagram

• The Murphy diagram plots expected elementary loss across θ under Q, but general scheme comparisons may be unclear because response noise and empirical recalibration add uncertainty.The diagram also obscures which premium ranges fit well, while individual-level contributions can be difficult to interpret because the plot contains many instances.Source: §5.2.2 · pp. 29–31
• Figure 11 shows the recalibrated premium’s Murphy loss below Π’s for most thresholds, suggesting a preference for mQ(Π) in this forecast-dominance example.The observed difference is positive for most θ, with minor noise perturbations.Source: §5.2.2 · p. 30, Figure 11 · p. 30

5.2.3 The asymmetrically binned Murphy diagram

• Asymmetric binning selects one premium rule to define bins, and the resulting diagrams are interpretive tools rather than estimates of expected elementary loss.The roles of the two rules can be exchanged to assess the impact of binning; matching risk rankings yield identical bins.Source: §5.2.3 · pp. 32–33
• Figure 13 shows a clear preference for recalibrated mQ(Π) over Π; binning affects Π substantially but mQ(Π) only marginally.The diagrams use percentile binning with K = 100.Source: §5.2.3 · p. 33, Figure 13 · p. 33
• In the binned AvE plot, longer segments and a wider premium range contributing to elementary loss for Π than for mQ(Π) indicate Π’s calibration issue.The plot is intended to show which premium levels contribute to elementary losses.Source: §5.2.3 · p. 33, Figure 14 · p. 34
• When premiums are Q-calibrated, the asymmetrically binned Murphy decomposition’s miscalibration term vanishes, leaving irreducible within-bin variation.The decomposition separates expected loss into within-bin variation and the error from using πB instead of the calibrated mQ(B).Source: §5.2.3 · pp. 34–35

5.3 From elementary losses to Bregman divergences

• Under mild regularity conditions, Bregman losses are mixtures of elementary losses, and their φ′′(θ) scaling in AvE plots can downgrade large losses during validation.This downgrading applies to model validation, not estimation; censoring claims during fitting would underestimate average losses and could violate calibration.Source: §5.3 · p. 35

5.4 Sample Bregman loss and currency invariant preferences

• The sample Bregman loss supports preference ordering of premium rules, but comparable rules using the same information generally lack forecast dominance, so one loss is typically selected.The discussion of loss scaling can support selection of that single Bregman loss.Source: §5.4 · p. 36
• Patton-family losses are positively homogeneous of degree 2−p, preserving preference order across currencies; Tweedie deviance losses are a subclass of that family.Scale invariance means selecting the same forecast model under different currencies, such as Euros or Dollars.Source: §5.4 · p. 36

6.1 Murphy’s decomposition and Bregman score

• Expected Bregman loss splits into nonnegative uncertainty, resolution, and miscalibration terms; MCBφ(Π) is zero exactly when mQ(Π)=Π almost surely.Resolution measures discrimination achievable by the recalibrated premium, while miscalibration measures the discrepancy between Π and mQ(Π).Source: §6.1 · p. 37
• 1.5017·10^-3 for mQ(Π) versus 0.2743·10^-3 for Π: Table 1’s gamma-deviance Bregman scores favor the recalibrated premium.The miscalibration term is zero for mQ(Π).Source: Table 1 · p. 39, §6.1 · p. 39

6.2 Graphical illustration of the Bregman score

• Figure 15’s paired aggregation curves end at the Table 1 scores; negative blue slopes show ranges where the original Π predicts less accurately than the global mean.Ordering by Π can split the flat region seen when ordering by mQ(Π), reflecting Π’s incorrect risk ordering.Source: Figure 15 · p. 40, §6.2 · p. 41

6.3 Sample computation of resolution and miscalibration

• 0.9934 versus 1.2274: (6.12) underestimates miscalibration; (6.13) gives 1.2591 versus 1.2274, overestimating it in this example.The formulas agree in the population but can differ on finite samples; reusing the test sample to fit and evaluate isotonic regression causes in-sample bias for (6.13), and the true value’s position between estimates may not generalize.Source: §6.3 · pp. 41–42

6.4 Illustration of resolution and miscalibration

• For B-bins, Figure 16 contrasts calibrated means mQ(B) with global mean for resolution and πB with mQ(B) for miscalibration; the analysis also applies to other portfolio binnings.The resolution differences represent calibrated price granularity and risk classification; the binwise decomposition separates within-bin response variation from the error of using πB instead of mQ(B).Source: §6.4 · pp. 42–44, Figure 16 · p. 43

7.1 Introduction: Risk ranking

• The Gini score evaluates risk ranking, not calibration or strictly consistent loss for mean estimation.Any strictly increasing transformation of unit premiums leaves this rank-based assessment unchanged.Source: §7.1 · p. 44

7.2 Cumulative accuracy profile

• The CAP compares premium ordering with the perfect ordering of observed losses, with ties handled using most- and least-favorable suborders.The Gini score measures the gap between these curves; the two tie orderings coincide when premiums have no ties.Source: §7.2 · pp. 45–46

7.3 Gini score

• The recalibrated premiums provide the better risk ranking, although the observed differences are small under the data’s low signal-to-noise ratio.The Gini score is based on the area between the CAP and Leimkuhler curves, averaging areas from the favorable and unfavorable tie suborders.Source: Figure 17 · p. 46, §7.2 · p. 47, §7.3 · p. 47

8 French motor third party liability example

• The example uses 67,801 independent test policies to compare PIN and GBM claim-frequency forecasts fitted on the same learning data.The dataset has nine covariates, claim counts, and yearly exposure; unit losses are claim frequencies, called unit premiums for consistency with the manuscript.Source: §8 · p. 48

8.1 Poisson deviance loss

• On test sample T under Poisson deviance loss, LightGBM’s sample Bregman score (3.595) exceeds PIN’s (3.352), favoring LightGBM; both outperform the selected GLM (2.532).A paired non-parametric bootstrap drawing with replacement from T gives a standard deviation of 0.057 for the 0.243 difference, which the passage calls significant.Source: Table 3 · p. 50, §8.1 · p. 50

8.2 Actual-vs-expected plots and lift charts

• PIN and GBM show similar lift with ventile binning, while PIN appears slightly better calibrated and GBM more biased over bins 9–14.The isotonic-bin plots also suggest similar lift and possibly a small calibration preference for PIN; their bin counts are not reliable statistics.Source: §8.2 · pp. 51–52, Figure 19 · p. 52

8.3 Murphy diagram

• The Murphy diagram favors LightGBM for most thresholds, especially at small thresholds where differences contribute strongly to Poisson deviance.Asymmetrically binned plots suggest GBM advantages at higher premiums, but the risk-ranking interpretation is qualified because the binning is asymmetric.Source: §8.3 · pp. 53, 55, Figure 22 · p. 54, Figure 23 · p. 55

8.4 Bregman score

• GBM’s advantage over PIN is concentrated among the third of policies with the highest unit premiums.The paired iterative aggregation gives similar scores for the half of policies with lower risks; the reported overall difference is 0.243.Source: Figure 25 · p. 57, §8.4 · p. 57

8.5 Murphy’s decomposition

• Murphy’s decomposition gives similar miscalibration estimates, but their interpretation is uncertain because isotonic recalibration uses the premium itself.The authors attribute apparent under- and overestimation in the two decomposition versions to isotonic-regression accuracy and in-sample overfitting, while noting there is no proof without ground truth.Source: Table 4 · p. 58, §8.5 · p. 58

8.6 Risk ranking

• The GBM risk ordering dominates PIN’s, and the Gini scores support the more accurate GBM ranking against observed responses.The reported Gini-score difference is 0.0100, with paired-bootstrap standard deviation 0.0033.Source: Table 5 · p. 58, §8.6 · p. 58, Figure 26 · p. 59

9 Summary

• Exposure weighting accounts for dependence between exposure and exposure-scaled losses; using policy weighting instead may yield systematically biased forecasts.This requires changing from the policy-weighted population distribution to the exposure-weighted distribution.Source: §9 · p. 59
• Calibration, discrimination, and risk ranking can be assessed with different plots and scores, including lift charts, Murphy diagrams, Bregman scores, and Gini scores.The manuscript also describes Murphy’s decomposition as a sample-based quantitative measure of calibration and discrimination.Source: §9 · p. 60
• The analysis mainly uses graphical tools and point estimates, while isotonic-regression-based decomposition can be biased because recalibration uses test data.The authors identify confidence bounds and calibration testing as further research needs.Source: §9 · p. 60
• Murphy diagrams may support informed model selection by showing how Bregman scores are composed across convex generators.The manuscript cautions that forecast dominance is usually too strong a condition for model selection.Source: §9 · p. 60

A Mathematical proofs

• Conditioning on A expresses expected Bregman loss as conditional expected generator value minus the prediction terms involving X and mQ(A).The displayed identity is EQ[Lφ(Y, X) | A] = EQ[φ(Y) | A] − φ(X) − φ′(X)(mQ(A) − X).Source: A Mathematical proofs · p. 64
• The corollary proof takes expectations of equation (6.2) to obtain the decomposition expression.The excerpt displays the global-mean loss, the difference in expected losses, and the recalibration loss terms.Source: A Mathematical proofs · p. 64
• The proof evaluates the bracketed term using equation (6.1) with X = µQ to complete the claim.The excerpt then states that this proves the claim.Source: A Mathematical proofs · p. 64

• Training data use learn$Exposure as weights, with feature_pre_filter set to FALSE.Source: B LightGBM code · p. 64
• The model uses learning_rate 0.05, a Poisson objective and metric, 7 leaves, and minimum leaf settings of 50 data points and 0.001 Hessian sum.Source: B LightGBM code · p. 64
• Training runs for 3000 rounds with 0.8 column sampling and bagging, L1 and L2 penalties of 3 and 5, and 7 threads.Source: B LightGBM code · p. 64

• The PIN and GBM miscalibration charts label actuals and expected values with bars.Source: C Miscalibration and resolution plots · p. 65
• Figure 27 places estimated PIN and GBM miscalibration in the top panels and estimated resolution in the bottom panels.Source: Figure 27 · p. 65
• Both chart measures use 20 ventiles and bin averages Y_k for the recalibration estimate, showing raw differences before Bregman scoring.Source: Figure 27 · p. 65