Why Does My Model Fail? Contrastive Local Explanations for Retail Forecasting

1 INTRODUCTION

• Regulatory developments and financial-industry interest provide additional motivation for explaining automated decisions.The passage notes that banks have been constrained to interpretable methods for credit assessment.Source: §1 · p. 1
• Prediction errors can reduce confidence and increase users’ preference for human forecasters, even after an algorithm outperforms a human.Source: §1 · p. 1
• At Ahold Delhaize, simple autoregressive models were in production while analysts showed interest in more complex forecasting techniques whose reduced interpretability was concerning when errors were large.Source: §1 · pp. 1–2
• Prior explanation methods can give similar explanations for large-error and reasonable predictions, motivating a method tailored to explaining large errors.The authors focus on errors in hindsight, rather than predicting them in advance or using explanations to improve the model.Source: §1 · p. 2
• MC-BRP presents bounds on important features that yield a reasonable prediction and the relationship between each feature and the target.Source: §1 · p. 2
• The study asks whether explanations support forward and counterfactual simulation, and whether they affect error understanding, deployment willingness, trust, and confidence in model performance.Source: §1 · p. 2
• The work evaluates MC-BRP with 75 participants and examines attitude differences between Practitioners and Researchers.The authors also analyze users’ understanding of the explanations and attitudes toward deploying the model.Source: §1 · p. 2

2 RELATED WORK

• Existing outcome-explanation work for tree ensembles addresses neither errors nor regression specifically, and the cited publications use binary classification tasks.Source: §2 · p. 2
• Unlike path-based counterfactual methods that select one nearest example, MC-BRP gives a range of feature values and does not assume full access to the original model.Source: §2 · pp. 2–3
• Influential-training-sample explanations can help identify harmful examples, but the cited approach is not contrastive and also aims to improve the model.Source: §2 · p. 3
• Shapley-based feature rankings are only a preliminary step in MC-BRP, which adds reasonable-prediction bounds and feature–target relationships.Source: §2 · p. 3
• LIME supplies local feature importances, but its regression approximation can add error and its selected features may be similar for large-error and reasonable predictions.The cited discussion notes that the target is typically of order 10^6.Source: §2 · p. 3
• The authors contrast their user study with prior explainability work that often neither draws on other disciplines’ explanation research nor evaluates with real users.They describe their own approach as rooted in philosophical literature and evaluated through a user study.Source: §2 · p. 3

3 METHOD

• MC-BRP treats large errors as potentially arising from unusual test-set feature values and defines reasonable and large-error cases by their prediction errors.The method relates the task to outlier detection and uses a threshold based on the first and third quartiles of the error set.Source: §3 · p. 3
• For each large-error instance, MC-BRP perturbs its important features within Tukey-fence bounds derived from reasonable cases and retains perturbations yielding reasonable predictions.It identifies the important features using LIME, though other feature-importance methods are also suitable.Source: §3 · pp. 3–4
• MC-BRP summarizes retained perturbations with feature means, standard deviations, and Pearson correlations to describe reasonable-value bounds and feature-target trends.The resulting bounds and trends are presented for each important feature; the example explanation shows input values outside their reasonable ranges.Source: §3 · p. 4

4 EXPERIMENTAL SETUP

• MC-BRP frames an explanation around a large prediction error, showing what contributed to it and what input changes could yield a reasonable prediction.The method is intended for users with varying ML expertise.Source: §4 · p. 4

4.1 Dataset and model

• The study predicts monthly sales for 563 stores using 45 features and 45,628 observations collected from 2010–2015.The data are split by year, with 2010–2013 for training and 2014–2015 for testing; the gradient boosting regressor achieves an R2 of 0.96.Source: §4.1 · pp. 4–5

4.2 Why existing solutions are insufficient

• LIME’s top-five features show little distinction between large-error and reasonable predictions; advertising costs ranks among them in 18.8% and 18.7% of cases, respectively.In preliminary testing, users could not make sense of error-focused objective questions with LIME explanations.Source: §4.2 · p. 5, Table 2 · p. 5

4.3 User study design

• The user study includes 75 participants, with 44 assigned to treatment and 31 to control; treatment participants see MC-BRP explanations while controls do not.Both groups answer subjective questions twice, while only treatment participants perform explanation-based forward and counterfactual simulations.Source: §4.3 · pp. 5–6

5.1 Objective questions

• 81.1% average accuracy shows users answered objective questions about MC-BRP explanations correctly; forward simulations reached 84.1% and counterfactual simulations 75.0%.The questions assessed whether users could simulate model outputs and identify input changes that would change the output.Source: Table 5 · p. 6, §5.1 · p. 6

5.2 Subjective questions

• Only understanding why the model makes large errors differed significantly between groups (χ2 = 18.2, α = 0.0001); treatment users agreed more than controls.The study found no statistically significant group differences in trust, confidence, or support for deployment.Source: Figure 2 · p. 7, §5.2 · p. 7
• Before explanations were shown, the groups had no statistically significant difference in initial attitudes toward the model.Participants had first seen a visual description comparing predicted and actual sales.Source: §5.2 · p. 7

6.3 User study limitations

• Treatment and control groups were slightly imbalanced by background, but the findings held after conditioning; future work proposes recruiting both groups concurrently.Table 6 reports the distribution of Practitioners and Researchers across the treatment and control groups.Source: §6.1 · p. 7, Table 6 · p. 7, §6.3 · p. 8

6.1 Comparing attitudes conditioned on background

• For both Researchers and Practitioners, treatment-group users agreed more with SQ1 than controls, indicating better understanding of large errors regardless of background.The difference was statistically significant for both groups, and stronger for Researchers.Source: §6.1 · p. 7, §6.2 · p. 8

6.2 Comparing attitudes in the treatment group

• Within the treatment group, Researchers were more favorable toward model use and trust, while Practitioners more often felt neutral or judged predictions unreasonable.The groups differed significantly on SQ2, SQ3, and SQ4; they did not differ significantly on SQ1.Source: §6.2 · p. 7, Figure 3 · p. 8

6.3 User study limitations

• The study lacked a baseline method because no existing method specifically explained regression prediction errors; MC-BRP can serve as a baseline for future work.The authors note that questions about errors would otherwise be unfair and confusing.Source: §6.3 · p. 8

• MC-BRP generates reasonable-prediction feature bounds and feature-target trends by perturbing instances and using Monte Carlo simulations and Pearson correlations.Source: §7 · p. 8
• 84.5% average accuracy showed users could interpret explanations through forward simulations, while counterfactual simulations showed 76.2% accuracy for actionability.Source: §7 · p. 8
• MC-BRP explanations improved understanding of large prediction errors, but did not significantly affect deployment support, trust, or perceived performance; these results held across user backgrounds.Source: §7 · p. 8
• Researchers and Practitioners differed significantly on deployment support, trust, and performance perceptions, but not on understanding large errors.Source: §7 · p. 8
• Future work will explore model abstention for unusual feature values and use a broader set of subjective questions to assess users’ impressions.Source: §7 · p. 8