PaperlayerView exact revision on arXiv

Source-linked AI summary

Mastering Model Fit: Applications of the InterModel Vigorish for Interpretable Machine Learning

Jiani Yan, Pengyun Wang, Klint Kanopka, Ben Domingue, Charles Rahal

arXiv:2610.29960v1stat.MEcs.LGstat.ML

TL;DR

Accuracy alone does not show how probabilistic fit differs relative to a baseline or which features and components contribute. The paper adapts IMV for feature attribution, multiclass comparisons, and ablation; results vary by estimator and class, and removing components need not reduce fit under the tested training budgets.

  • Problem

    Accuracy alone gives little information about differences in probabilities assigned to observed outcomes or which features and model components improve probabilistic fit.

    Source: §1 · p. 1

  • Method

    The paper adapts IMV to express fit differences against a reference model, using Shapley-weighted feature-subset comparisons, one-vs-all and pairwise class comparisons, and refitted ablations.

    Source: §1 · p. 1

  • Results

    Feature rankings changed with estimator, Multi-IMV gains differed across classes, and removing components need not reduce fit under the tested training budgets.

    Source: §8 · p. 13

  • Takeaways & Limitations

    IMV provides a baseline-relative betting-scale comparison of probabilistic fit, but findings depend on the baseline, evaluation design, and coin-mapping domain and do not establish causal effects.

    Source: §8 · p. 13

    Exact feature attribution requires substantial refitting, and Shapley properties do not guarantee unbiased estimation, causal identification, or social fairness.

    Source: §4.2 · p. 3, §8 · p. 13

Abstract

from arXiv · show

Accuracy alone gives little information about which features or model components improve probabilistic fit, which outcomes benefit, or how fit changes during training. We introduce the InterModel Vigorish (IMV) into the Machine Intelligence space in order to express differences in fit as an expected betting advantage over a reference model. SHAP-IMV attributes gains to features through Shapley-weighted subset comparisons, Multi-IMV evaluates one-vs-rest and pairwise class discrimination, and Ablate-IMV assesses architectural components and feature groups by ablation. We evaluate these applications over ten random seeds using tabular data, images, sensor sequences, and graphs. Feature rankings vary across estimators. High one-vs-rest accuracy can accompany negligible improvement over a null, while ablation can preserve or improve fit under a fixed training budget. On six graph-classification benchmarks, we time-profile IMV over 200 training epochs, reporting seed means and ranges. The open-source imvpy library and accompanying notebooks provide code to replicate our work and to use the IMV more widely.

1 Introduction

Accuracy and common predictive scores do not by themselves give the same interpretation of probabilistic fit relative to a baseline. The paper adapts IMV to attribute fit gains to features, compare classes, and assess model components across several prediction settings.

1 Introduction

  • IMV gives probabilistic-fit differences an explicit reference-model comparison, unlike scores that answer different questions and may not share a baseline interpretation.The paper adapts IMV for feature attribution, multiclass discrimination, and model-component ablation.Source: §1 · p. 1
  • SHAP-IMV attributes gains to features, Multi-IMV compares classes one-vs-all and one-vs-one, and Ablate-IMV compares refitted models after component removal.Source: §1 · p. 1
  • Experiments span binary and multiclass tabular prediction, image parity, human activity recognition, and graph classification using logistic regression, boosting, neural networks, and graph convolutional networks.Once predictions are available, scoring adds little cost; exact feature attribution requires repeated model fitting.Source: §1 · p. 1

2 Related Work

Existing feature-importance and explanation methods differ in scope and in what they measure. The paper positions SHAP-IMV as a global attribution based on held-out probabilistic fit, and IMV as a reference-based interpretation of model comparison.

2.1 Feature Importance and Local Explanations

  • SHAP-IMV uses global Shapley attribution with refitted feature subsets and held-out IMV against a baseline, unlike methods based on algorithm-specific quantities or local explanations.Its allocations are expressed in IMV units and sum to the full model’s gain.Source: §2.1 · p. 1

2.2 Global Metrics for Machine Learning

  • IMV gives probabilistic-fit comparisons an explicit reference and betting-scale interpretation, alongside Brier score, accuracy, precision, recall, F1, ROC area, likelihood-ratio tests, and information criteria.Claims about generalization still depend on the evaluation design.Source: §2.2 · p. 2

3 Preliminaries

The preliminaries define probabilistic fit from outcome likelihoods and interpret IMV as a betting advantage over a reference model. A Python example shows how to compute and interpret the score against a prevalence baseline.

3.1 Problem Definition

  • For binary classification, a model assigns each observation a probability for outcome 1, which determines that observed outcome’s likelihood and log-likelihood.Source: §3.1 · p. 2

3.2 The InterModel Vigorish (IMV)

  • The exact coin mapping has no solution below geometric-mean likelihood 0.5; imvpy instead applies a boundary weight of 0.5 for limited deficits and warns with an undefined score for larger ones.The boundary rule is an implementation convention, and distinct poor predictions can receive the same weight.Source: §3.2 · p. 2
  • IMV is the betting edge when a model’s coin replaces the baseline coin at odds fair for the baseline; positive values favor the model and negative values favor the baseline.The comparison requires predictions for the same outcomes in the same order.Source: §3.2 · p. 2
  • Within the exact coin domain, IMV orders predictions like mean log-likelihood and inversely to log loss, while its upper bound depends on baseline weight and outcome prevalence.Source: §3.2 · p. 2

Box 1: Vanilla IMV

  • Approximately 0.282 IMV is the example’s gain over a training-prevalence baseline, interpreted as expected profit of about $0.282 per $1 staked under baseline break-even odds.The example evaluates logistic-regression predictions on the held-out 30% of 1,000 simulated binary outcomes.Source: Box 1: Vanilla IMV · p. 2, §3.3 · p. 2

4 Methodology

The methodology applies IMV to feature attribution, multiclass comparisons, and model ablation, with each score interpreted relative to an explicit reference. SHAP-IMV uses refitted feature subsets, Multi-IMV contrasts classes, and Ablate-IMV compares full and ablated models.

4.2 SHAP-IMV for Feature Attribution

  • SHAP-IMV attributes feature gains using Shapley-weighted marginal changes in held-out IMV across feature subsets, compared with a constant-only reference.Subset models and the reference are refitted on shared cross-validation folds.Source: §4.2 · pp. 2–3, §4.1 · p. 2
  • SHAP-IMV satisfies Shapley efficiency, symmetry, dummy-feature, and additivity properties, but these do not guarantee unbiased estimation, causal identification, or social fairness.Exact enumeration requires 2^d fits per fold; the paper uses eight-predictor specifications to keep it tractable.Source: §4.2 · p. 3

4.2.1 Simulated SHAP-IMV example.

  • In the simulated example, SHAP-IMV assigns 0.209 to x1 and 0.056 to x2.The example evaluates two features using logistic regression and three stratified folds.Source: §4.2 · p. 3, §4.2.1 · p. 3

4.3.1 One-vs-All Comparisons.

  • One-vs-all Multi-IMV evaluates each class against the pooled remainder using the multiclass predictions’ original probability columns, without renormalization or separate binary classifiers.The simulated example reports mean held-out IMV over three stratified folds for 900 observations and three classes.Source: Box 2: SHAP-IMV · p. 3, §4.3.1 · p. 3, §4.3.4 · p. 4

4.3.2 The One-vs-One Multi-IMV Confusion Matrix.

  • Pairwise Multi-IMV retains instances from each class pair, renormalizes their probabilities, and reuses the fitted multiclass estimators rather than fitting a model per pair.The resulting matrix is symmetric, has a zero diagonal by convention, and contains baseline-relative probabilistic scores rather than confusion counts.Source: §4.3.2 · p. 4

4.3.3 Reading One-vs-All and Pairwise Scores.

  • One-vs-all scores measure class-specific gains against all other classes, whereas pairwise scores measure separation within a class pair; their populations and reference weights differ.Class balance bounds both scores, so the paper recommends comparing different classes or pairs relative to their bounds.Source: §4.3.3 · p. 4

4.4 Ablate-IMV for Model Component Comparisons

  • Ablate-IMV compares ablations with an uninformative 0.5-probability reference and also supports directional comparisons between model variants.Directional matrix entries use the column model’s reference weight as the denominator, so the matrix is generally asymmetric.Source: §4.4 · p. 4

4.4.1 Simulated Ablate-IMV example.

  • In the simulated ablation, the negative Full–Ablated entry indicates that the additional hidden layer does not improve held-out fit.The two neural networks were fitted from scratch on the same training observations and evaluated on the same test observations.Source: §4.4.1 · pp. 4–5

5 Experiments

The experiments cover feature attribution, multiclass prediction, and ablation across tabular, image, sensor-sequence, and graph tasks. They use multiple estimators and datasets, with cross-validation or held-out evaluations repeated across ten seeds.

5.1.1 Datasets.

  • The experiments span binary and multiclass tabular prediction, image parity, activity recognition, and graph classification, including six graph benchmarks.The SHAP-IMV specifications are restricted to eight predictors so exact enumeration remains tractable.Source: §5.1.1 · p. 5

5.1.2 Comparison Methods.

  • SHAP-IMV rankings are compared with SHAP, LIME, and Anchors, while Multi-IMV is reported alongside per-class Brier loss, accuracy, precision, and recall..Source: §5.1.2 · p. 5

5.1.3 Models and Training.

  • SHAP-IMV and Multi-IMV use five-fold stratified cross-validation across ten seeds; selected top-five subsets receive a separate ten-fold evaluation.The SHAP-IMV top-five feature selection precedes the refit evaluation folds.Source: §5.1.3 · p. 5
  • Ablation evaluations cover MNIST, UCI HAR, and MAGIC Gamma; the GNN case study uses a three-layer GCN trained for 200 epochs across ten seeds.GNN held-out metrics are reported at each epoch, including untrained epoch 0.Source: §5.1.3 · p. 5

5.2.1 Datasets for SHAP-IMV.

  • The SHAP-IMV datasets are Titanic, Wisconsin Diagnostic Breast Cancer, and Adult Income, each analyzed with eight specified predictors.Adult Income uses pooled original training and test files for cross-validation rather than the official test partition.Source: §5.2.1 · p. 5

5.2.2 Datasets for Multi-IMV.

  • Nursery, Car Evaluation, and Dry Bean supply the multiclass benchmarks, with their selected class frequencies retained for one-versus-rest and pairwise comparisons.Nursery and Car categorical inputs use integer codes, including ordinal codes for nominal predictors.Source: §5.2.2 · p. 6

5.2.3 Datasets for Ablate-IMV.

  • The ablation datasets pair MNIST odd/even classification and UCI HAR upstairs/downstairs activity with MAGIC Gamma gamma/hadron classification.MAGIC Gamma’s simulated class proportions do not represent gamma-event prevalence in observational telescope data.Source: §5.2.3 · p. 6

5.3 Comparison Methods and Metrics

The section compares feature-explanation methods and defines classification metrics used alongside probabilistic-fit comparisons. It also notes that the feature-selection comparison is descriptive, not a nested estimate of generalization.

5.3.1 Feature Explanation Methods.

  • SHAP-IMV refits feature subsets within cross-validation, while SHAP, LIME, and Anchors explain fixed fitted models using different scores and unequal explanation budgets.SHAP uses absolute contributions, LIME ranks surrogate-coefficient magnitudes, and Anchors scores rule inclusions adjusted for rule position.Source: §5.3.1 · pp. 6–7
  • Because explanation budgets differ and the comparisons are descriptive, they cannot establish uniform superiority; nested feature selection and fold-fitted preprocessing would be needed to assess generalization.Source: §5.3.1 · p. 7

5.3.2 Multiclass Evaluation Metrics.

  • Brier loss measures squared probability error; for binary outcomes, it uses the observed label and predicted positive-class probability.Source: §5.3.2 · p. 7
  • Accuracy, precision, and recall summarize classification decisions through correct classifications, correct positive predictions, and recovered actual positives, respectively.Source: §5.3.2 · p. 7

5.4 SHAP-IMV Results

SHAP-IMV-selected subsets had the highest mean accuracy for XGBoost and LightGBM, while LIME led for logistic regression. Method rankings largely agreed, but the Breast Cancer comparison was an exception.

5.4 SHAP-IMV Results

  • Across datasets, SHAP-IMV subsets had the highest mean accuracy for XGBoost (86.96%) and LightGBM (86.99%), while LIME led logistic regression (86.00%); Breast Cancer accuracy was slightly lower than SHAP or LIME for every estimator.Source: §5.4 · p. 7
  • The four methods largely agreed: SHAP-IMV shared at least four of five selected predictors with every alternative in all but one setting.It selected the same five predictors as at least one alternative in five of nine dataset-estimator settings, and its accuracy was never more than 0.5 percentage points below the best alternative.Source: §5.4 · p. 7
  • Feature rankings varied by estimator: sex led Titanic logistic regression, title led both boosting models, and radius and concave points ranked highest for Breast Cancer across estimators.Rankings describe predictive contributions within the specified models and feature sets, not causal effects.Source: §5.4 · p. 7

5.5 Multi-IMV Results

Multi-IMV distinguishes class-level probabilistic fit from conventional classification metrics, while pairwise comparisons reveal differences among class pairs. The results show how class frequency and evaluation scope shape these views.

5.5 Multi-IMV Results

  • For Car Evaluation’s rare Good class, logistic regression had 96.01% one-vs-rest accuracy but zero precision and recall, while mean Multi-IMV rounded to 0.00.XGBoost and LightGBM recalled Good at 89.13% and 95.22%; their Multi-IMV scores were 0.03 and 0.04, compared with 0.40 and 0.42 for Unacceptable.Source: §5.5 · p. 7

5.5.1 Nursery and Dry Bean Comparisons.

  • In Dry Bean, every estimator’s Horoz/Seker pairwise IMV exceeded Bombay/Dermason, largely reflecting their class-balance bounds; Dermason/Sira was furthest below its bound at 0.67 of 0.74.The scores compare probabilistic fit between pairs, not counts of classification errors.Source: §5.5 · p. 7
  • Figure 3 covers all four Car Evaluation classes, but the Nursery analysis retains only not_recom, priority, and spec_prior, excluding recommend and very_recom.Tables 4 and 5 provide additional per-class metrics for Nursery and Dry Bean.Source: §5.5.1 · p. 7

5.6 Ablate-IMV Results

Ablate-IMV compares architectural and feature-group variants with reference models on MNIST, UCI HAR, and MAGIC Gamma. Under the reported training budgets, removing a component need not reduce held-out fit.

5.6 Ablate-IMV Results

  • Figure 4 compares enhanced models in heatmap rows with reference models in columns, while bars compare each model against constant 0.5 predictions.MNIST and HAR ablate architecture, whereas MAGIC Gamma ablates feature groups; evaluations use the same test observations within each seed.Source: §5.6 · p. 8

5.6.1 MNIST: Architectural Ablation of a CNN.

  • Under one epoch, the MNIST linear model scores 0.751 IMV against the 0.5 baseline, versus 0.975 for the full CNN; removing dropout slightly raises the score to 0.980.The other ablations are closer to the full model, so component removal does not necessarily reduce held-out fit under this budget.Source: §5.6.1 · p. 10

5.6.2 UCI HAR: Recurrent and Temporal Components.

  • On UCI HAR, the mean-only MLP scores 0.411 against the 0.5 baseline, versus 0.919 for the full model; recurrent-state mean pooling scores 0.930.The full recurrent model has higher mean fit than the channel-mean MLP, but attention gives no gain over recurrent mean pooling under this budget.Source: §5.6.2 · p. 10

5.6.3 MAGIC Gamma: Feature-Group Ablation.

  • On MAGIC Gamma, feature-group variants remove geometry, size and concentration, moments, or orientation and distance while keeping the MLP architecture fixed.The variants use seed-specific stratified 75/25 splits and training-set standardization; the supplied passages do not report their comparative results.Source: §5.6.3 · p. 10

6 IMV and Model Training: A Case Study

The case study tracks IMV and threshold-based metrics on six graph-classification benchmarks over 200 training epochs. It evaluates fixed test graphs from an untrained model through the full training run, retaining across-seed ranges and negative IMV values.

6.1 Graph Datasets and Preprocessing

  • The six benchmarks are PROTEINS, NCI1, NCI109, Mutagenicity, AIDS, and DD, with dataset counts ranging from 1,113 to 4,337 graphs.The experiments use the TUDataset benchmarks; graph labels and task descriptions vary across protein and molecular datasets.Source: §6.1 · p. 11, Figure 5 · p. 12

6.2 Architecture and Training

  • Each seed uses a stratified 70/15/15 training, validation, and test split, and the three-layer GCN trains for 200 epochs across ten seeds.Validation metrics are recorded but not used for tuning or checkpoint selection; test curves follow the fixed training run.Source: §6.1 · p. 11, §6.2 · pp. 11–12, Figure 5 · p. 12

6.3 Epoch-Level Results

  • Figure 5 tracks test-set IMV, accuracy, precision, and recall from epoch 0 through 200, with seed means and epoch-wise minimum-to-maximum ranges.IMV uses the training-prevalence null; threshold-based metrics use 0.5 with class 1 positive. Negative IMV values and unsmoothed curves are retained, and the ranges are not confidence intervals.Source: Figure 5 · p. 12, §6.3 · p. 12

7 Computational Complexity Analysis

The complexity analysis measures the cost of computing vanilla_imv after predictions are available. It finds linear scaling with the number of test observations, with fixed overhead more noticeable at small sample sizes.

7 Computational Complexity Analysis

  • Computing two geometric-mean Bernoulli likelihoods takes O(n), followed by two scalar Brent searches; total time is O(n + k0 + k1).At fixed numerical precision, the searches do not operate on observation vectors, and the vectorized implementation uses O(n) auxiliary memory.Source: §7 · p. 13
  • At larger sample sizes, cost per observation varies little, consistent with linear scaling; fixed overhead raises per-observation cost at smaller sizes.The fixed overhead includes the two scalar root searches.Source: §7 · p. 13
  • Runtime measurements cover the complete vanilla_imv call on identical inputs, excluding model fitting and prediction; reported IQR describes timing variation, not IMV uncertainty.The timing procedure uses nine batches of repeated calls after a warm-up, with one numerical-library thread.Source: §7 · p. 13, Table 7 · p. 13

8 Conclusion

The conclusion presents SHAP-IMV, Multi-IMV, and Ablate-IMV as comparisons of probabilistic fit against a reference. The findings depend on baseline and evaluation choices, and exact attribution requires substantial refitting.

8 Conclusion

  • SHAP-IMV selected predictor subsets comparable to those chosen by SHAP, LIME, and Anchors, while its rankings changed with the estimator.The contributions are expressed as held-out gains in fit.Source: §8 · p. 13
  • Multi-IMV gains differed across classes, and Ablate-IMV showed that removing components need not reduce fit under the evaluated training budgets.The GNN case study also found that IMV and threshold-based metrics can change differently during training.Source: §8 · p. 13
  • The findings depend on the baseline, evaluation design, and coin-mapping domain; they do not guarantee better prediction or establish causal effects.Exact attribution requires substantial refitting.Source: §8 · p. 13
  • The imvpy package implements the three comparisons, and the IMV_ML repository provides notebooks and plotting code under GNU GPL v3.The notebooks retrieve cited datasets and generate local results.Source: §8 · p. 13

Related papers

Loading 2610.29960v1…