Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction

1 Introduction

• The study addresses whether explanation quality affects human decision-making, an area less evaluated than explanations’ intrinsic relation to model properties.It tests the assumption that faithful explanations improve model use and explanations of irrelevant features reduce trust [35] [40] [29] [13].Source: §1 · p. 1
• Accurate and faulty saliency explanations alike did not significantly affect task accuracy or trust in model predictions.The study also found no significant effect on understanding; even obviously unreliable explanations did not significantly decrease trust.Source: §1 · pp. 1–2
• The randomized controlled trial compares faithful explanations of a high-quality model with spurious and input-agnostic random saliency maps.Strong explanations came from the strong model, while spurious explanations came from a model trained with spurious correlations.Source: §1 · p. 1, Figure 1 · p. 2

2.1 Task

• The age-prediction task uses APPA-REAL images and a strong model with 5.24 test-set mean absolute error (MAE).The study chose the task because both humans and models perform with some proficiency but are far from perfectly accurate, and because human and model guesses often disagree.Source: §2.1 · p. 2

2.2 Treatments

• Randomized treatment arms vary whether participants see a model prediction and which explanation or presentation accompanies it.The control provides no model assistance; the prediction baseline shows the model prediction without an explanation.Source: §2.2 · pp. 2–3, Figure 2 · p. 3
• Explanation treatments add integrated-gradients saliency maps to predictions from the same strong model, isolating explanation quality.The study compares strong maps, spurious maps that focus on face and background, and input-agnostic random maps; faulty explanations were motivated as a possible deterrent to trust.Source: §2.2 · pp. 3–4
• Design-based treatments test delayed predictions, personification, and a top-three age range as alternatives to explanation-based presentation.The delayed-prediction treatment records an initial guess before showing the model prediction; the range is described as wider when the model is more uncertain.Source: §2.2 · p. 4

2.3 Metrics

• The study measures absolute age-prediction error, agreement with the model as trust, decision time, and post-survey responses.Mixed-effects regressions estimate error and trust using treatment and image age as fixed effects and user- and image-specific random intercepts.Source: §2.3 · p. 4

2.4 Experiment Details

• Amazon Mechanical Turk data came from 1,058 participants, with 827.5 guesses per treatment and 93% power to detect a one-year difference at p < 0.05.The planned collection was 546 guesses per treatment to detect a one-year difference.Source: §2.4 · p. 4

3 Analysis and Results

• The analysis controlled for image- and user-specific effects and examined subsets defined by mean human and model errors of 8.65.The H+M- and H-M+ subsets contained 1,802 and 1,721 guesses, respectively.Source: §3 · pp. 4–5

3.1 How do model predictions and explanation quality affect model-in-the-loop accuracy?

• Predictions generally helped when humans were inaccurate but could hurt when humans were accurate and the model was inaccurate.Show Top-3 Range potentially improved accuracy by 4.55 in H-M+, but top-treatment pairwise differences were not statistically significant.Source: §3.1 · p. 5
• Adding explanations did not improve accuracy, and explanation-quality differences were small and statistically nonsignificant in every setting.Participants examined highlighted areas and features such as wrinkles but did not extract information that boosted performance.Source: §3.1 · pp. 5–6
• The authors note that results could change with extensive saliency-map training, whereas participants received only a short guide.They cite prior work in another task where training did not improve performance when predictions were also shown [26] [25].Source: §3.1 · p. 5
• Participants shown model predictions were more accurate by 2 years on average overall, while explanation-based treatments did not differ significantly from prediction alone.Pairwise differences between the top treatments were also not significant across the three settings shown.Source: §3.1 · p. 5, Figure 3 · p. 6

3.2 How does explanation quality affect human trust and understanding?

• No explanation treatment significantly changed trust, including when model predictions were inaccurate; even sizable estimated effects were nonsignificant.Strong explanations could increase trust by up to 1.08 years at the confidence-interval lower bound, while random explanations could decrease it by up to 1.48 years at the upper bound.Source: §3.2 · p. 6, Figure 4 · p. 7

3.3 How did humans incorporate model predictions and explanations into their decision making process?

• Participants rated explanation intelligibility similarly across quality levels, with means of 5.37/7 for strong, 5.05 for spurious, and 4.71 for random explanations.Many participants focused on prediction accuracy; only a few explicitly said explanations mediated their judgments.Source: §3.2 · p. 7, Figure 5 · p. 7, §3.3 · p. 7
• Among the top 75 guessers, 28.2% used the model prediction as a starting point and 21.8% adjusted their own initial guess based on it.The top 75 had a collective MAE of 5.19.Source: §3.3 · p. 7

Interpretable machine learning

• Interpretability methods include instance-level and global explanations, glass-box and black-box approaches, and feature attributions, natural language, and counterfactuals.The experiment uses input feature attributions [35] [40] [39] [32] [4].Source: §4 · p. 8
• Explanation evaluations have assessed correctness, user insights, and whether people can predict model behavior [40] [35] [11] [30].Some popular methods have also been found unreliable or unreasonably sensitive to minor input changes [21] [16].Source: §4 · p. 8

Model-in-the-Loop experiments

• Prior model-in-the-loop studies report mixed accuracy and trust effects, including cases where explanations helped, hurt, or failed to help users.Transparency also hindered error detection in an apartment-price task, echoing concerns about cognitive overload.Source: §4 · p. 8
• The authors identify prior work limited by simpler or non-neural models, small feature spaces, human-AI performance imbalance, and little investigation of certification.The cited studies include [34] [16] [7] [26] for model simplicity, [34] [49] for small feature spaces, and [7] [16] [25] [26] [34] [41] [49] for certification.Source: §4 · p. 8

Certification with explanations

• Certification examples include users detecting model limitations in ad recommendations and identifying the better of two models with 90% accuracy, but extensive human studies are lacking.The cited work also introduced model- and data-randomization tests for explanation specificity [3].Source: §4 · p. 8

Design of the human-AI system

• System-design factors such as stated accuracy, uncertainty displays, incentives, agency, and interactivity can matter, but are outside this study's scope.Prior work found stated accuracy can increase trust, while observed low accuracy can reduce it.Source: §4 · p. 8

• Adding explanations to model predictions did not significantly affect human accuracy, trust, or understanding, regardless of explanation quality.Even faulty explanations did not significantly decrease trust in the model.Source: §5 · p. 8
• Input feature attributions may be less useful for downstream decision-making than for researchers and engineers using interpretable ML methods to debug models.The paper notes that these methods have largely been used as debugging tools rather than as end-user communication.Source: §5 · p. 8
• Counterfactuals and natural-language explanations may be more effective, but should be evaluated by their downstream use when model-in-the-loop use is a goal.Source: §5 · p. 8

• The study's age-prediction task has high-stakes analogues, and its findings could also be used to improve ethically dubious demographic-prediction systems.The authors name facial-recognition surveillance and criminal identification as examples, and warn that complementary human and model biases may further harm marginalized communities.Source: Broader Impact · p. 9
• The authors hope the findings will help designers develop more useful explanations and human-AI systems, improving decision-system legibility for non-technical users and outcomes for affected people.Source: Broader Impact · p. 9

A.1 Datasets, Models, and Training Details

• The strong model was pretrained using IMDB-WIKI's 524,230 images and age labels, while the weak model was not pretrained on that dataset or trained to convergence on APPA-REAL.Strong and weak models used pretrained ImageNet weights; the spurious model was trained from scratch to tune it to spurious correlations.Source: §A.1 · p. 13
• Table S1 reports age-prediction model performance in mean absolute error; the Spurious model's slightly lower MAE than the Strong model is attributed to tuning for additional spurious correlations.Source: Table S1 · p. 13
• Training used Adam with a 0.001 learning rate, 224 × 224 images, and augmentation including noise, blur, rotations, scaling, translation, flips, and hue and saturation adjustments.Source: §A.1 · p. 13

A.2 Treatment Arms

• The Empathetic treatment used an avatar, and Table S2 lists the treatment arms; faulty-map No Acc treatments omitted the stated model performance.The No Acc treatments tested whether allowing users to judge model ability themselves would amplify the effect of faulty explanations.Source: §A.2 · p. 13, Table S2 · p. 14
• The Explain-strong, No Pred and Explain-weak, No Pred treatments showed saliency maps without the model prediction, and Explain-guide, No Pred presented a global guide derived from 200 maps.The guide identified regions around the eyes and facial lines as most important, followed by the mouth and nose, and was repeated for each image.Source: §A.2 · p. 13

• 40.3% of participants were female, and their mean age was 36.5 years (standard deviation 10.8).Source: §A.3 · p. 14

B Results

• 1.2 years increase in accuracy came from Explain-guide, No Pred, compared with a 1.8 years increase in the Prediction treatment.Explain-guide, No Pred was as helpful as the top-performing treatments when humans and models were inaccurate.Source: B Results · p. 14
• Instance-level explanation-only treatments had small, non-statistically significant effects on accuracy.The treatments were Explain-strong, No Pred and Explain-weak, No Pred.Source: B Results · p. 14
• Hiding model performance did not significantly increase or decrease the effect of faulty explanations, though it directionally appeared to increase trust when humans and models were inaccurate.The authors suggest that stating model performance may have emphasized the model's fallibility.Source: B Results · p. 14

C.1 The economic cost of Model-in-the-Loop predictions

• Under error × time, costs were Control 56.7, Empathetic 50.6, Explain-strong 47.2, Show Model Pred 43.5, and Show Top-3 Range 42.5.The authors present this as an illustrative example and note that time and cognitive load matter when designing human-AI systems.Source: §C.1 · p. 18

C.2 Combining human guesses and model predictions

• Hybrid models combining human guesses with model predictions were evaluated using cross-fold validation and hyperparameter search, with test MAE results reported in Table S6.The inputs were simply the model's prediction and the human's guess.Source: §C.2 · p. 18, Table S6 · p. 18
• A proposed follow-up is to derive coarse decision-tree strategies from hybrid models and test whether they could further improve accuracy.The paper gives a rule combining a person's estimate with a sufficiently greater model prediction as an example.Source: §C.2 · p. 18