PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Users Benefit From Interpretable Vision? A User Study, Baseline, And Dataset

Leon Sixt, Martin Schuessler, Oana-Iuliana Popescu, Philipp Weiß, Tim Landgraf

arXiv:2204.11642v1cs.LGcs.AIcs.CVcs.HC

TL;DR

Whether interpretability methods help users find model biases beyond inspecting inputs and predictions remains uncertain. The paper compares an output-based baseline with concept and counterfactual explanations in a controlled user study. The baseline was not outperformed overall, though counterfactuals helped with some attributes.

  • Problem

    Evidence is mixed on whether interpretability methods help users discover model biases better than inspecting model inputs and predictions, motivating direct user studies.

    Source: §1 · p. 1

  • Method

    The authors use the controllable TWO4TWO synthetic dataset and a bias-discovery task to compare an output-logit image-grid baseline with concept and INN counterfactual explanations.

    Source: §1 · p. 1, §5.1 · p. 6

  • Results

    Neither tested state-of-the-art method performed significantly better than the output-only baseline, although counterfactuals helped users identify the strongest model bias and some attributes more accurately.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results support evaluating explanation benefits through user studies of bias reasoning rather than relying only on technical evaluations or proxy tasks.

    Source: §1 · p. 1, §5.1 · p. 6, §6 · p. 9

    The study used a limited set of factors and only two bias configurations, so results may differ with other dataset configurations and visually salient biases.

    Source: §6 · p. 9

Abstract

from arXiv · show

A variety of methods exist to explain image classification models. However, whether they provide any benefit to users over simply comparing various inputs and the model's respective predictions remains unclear. We conducted a user study (N=240) to test how such a baseline explanation technique performs against concept-based and counterfactual explanations. To this end, we contribute a synthetic dataset generator capable of biasing individual attributes and quantifying their relevance to the model. In a study, we assess if participants can identify the relevant set of attributes compared to the ground-truth. Our results show that the baseline outperformed concept-based explanations. Counterfactual explanations from an invertible neural network performed similarly as the baseline. Still, they allowed users to identify some attributes more accurately. Our results highlight the importance of measuring how well users can reason about biases of a model, rather than solely relying on technical evaluations or proxy tasks. We open-source our study and dataset so it can serve as a blue-print for future studies. For code see, https://github.com/berleon/do_users_benefit_from_interpretable_vision

1 INTRODUCTION

The paper asks whether interpretability methods help users discover model biases more than inspecting inputs and predictions. It introduces a controlled synthetic dataset and tests explanation methods against an output-based baseline, finding no overall improvement over the baseline.

1 INTRODUCTION

  • The study asks whether modern interpretability methods help users discover model biases better than inspecting input/output pairs.Prior evidence on the usefulness of interpretability methods for bias discovery was mixed.Source: §1 · p. 1
  • TWO4TWO controls data-generating attributes and biases, while its output-logit-grouped image grid serves as a simple baseline explanation.The grid is designed to let users inspect attributes that may predict the target class.Source: §1 · p. 1
  • The main study (N=240) compared the baseline with automatically discovered concepts and INN-generated counterfactual interpolations.An initial study (N=50) found participants struggled to find both dataset biases using the baseline.Source: §1 · p. 1
  • None of the tested explanations outperformed the baseline, although counterfactuals helped users identify some features more accurately.Participants’ textual justifications also revealed usability issues across methods.Source: §1 · p. 2
  • The work contributes a controlled dataset generator, feature-importance measures, a bias-discovery study design, a baseline, and open-source study materials.The design is intended for lay users and includes measures for response quality.Source: §1 · p. 2

2 RELATED WORK

Prior work spans several vision-explanation approaches, but user studies have often evaluated accessibility or simpler tasks rather than bias discovery. The paper positions TWO4TWO as a dataset for human-subject evaluations of bias discovery.

Interpretable ML for Vision

  • Vision explanation approaches include saliency maps, example-based explanations, counterfactuals, activation concepts, and models with built-in interpretability.The paper focuses on counterfactual explanations and automatically discovered concepts.Source: §2 · p. 2

User Studies for Interpretability

  • Prior user studies tested concept accessibility or counterfactual forward prediction, while this work tests concept- and counterfactual-based methods on bias discovery with a challenging dataset.The authors describe their study as the first extensive test of these methods on that task.Source: §2 · p. 3

Synthetic Datasets for Interpretable Vision

  • TWO4TWO is presented as the first dataset designed explicitly for human-subject evaluations, supporting a unified bias-discovery evaluation.Earlier synthetic datasets could look artificial, make background cues conspicuous, or be difficult to extend to multiple biases.Source: §2 · p. 3

3 TWO4TWO: DATASETS WITH KNOWN FEATURE IMPORTANCE

TWO4TWO is a controllable synthetic dataset built to create multiple subtle biases and measure model reliance on known attributes. Its design combines ambiguous animal classes, selectively biased attributes, and model-based feature-importance measures.

Dataset Description

  • The dataset creates two abstract animal classes, Peeky and Stretchy, whose leg positions overlap from 0.48 to 0.52, making class identification ambiguous in that interval.Other continuously controllable attributes include posture, position, color, block shape, and background color.Source: §3 · p. 3
  • TWO4TWO aims to support quick participant training, multiple difficult-to-find biases, and challenges for interpretability methods.Its high number of controllable attributes and spatially overlapping or long-range features are central to that design.Source: §3 · pp. 3–4

Introducing Biases

  • Shape is biased outside the central leg-position range but uniform from 0.45 to 0.55; color adds class information in the overlap range and becomes uniform at more extreme positions.Background color and body posture are sampled independently of class.Source: §3 · p. 4

Measuring Ground-Truth Feature Importance

  • Feature importance is measured by changing one attribute at a time and comparing median absolute logit change and prediction flips.These measures reflect model influence, while an R2 fit distinguishes directed effects from flips with random direction.Source: §3 · p. 4
  • The trained model relied more on animal color than block shape, and color’s effect was more directed (R2=0.751) than shape’s (R2=0.307).The prediction-flip rate for unrelated background was only slightly lower than for block shape.Source: §3 · p. 4

4 MODEL AND EVALUATED METHODS

The study evaluates INN counterfactuals and automatically discovered concepts alongside a baseline. The INN uses an invertible feature mapping to generate counterfactual inputs, while concept explanations visualize selected correlated concepts.

INN Counterfactuals

  • The INN maps inputs to features for classification and uses its analytic inverse to reconstruct inputs; its test accuracy is 96.7%.The baseline and concept methods were also applied to this model.Source: §4 · p. 5
  • INN counterfactuals shift the feature vector along the classifier weight vector, then invert the shifted representation to visualize an output-changing input.The step size α controls the logit change, Δy = αw^T w, and each interpolation is shown as a grid row.Source: §4 · p. 5

Automatically-discovered Concepts

  • The concept method factorizes activations from layer 342 into 10 components and selects five most correlated with the logit score.That layer’s 8x8 feature maps balance spatial resolution and high-level information; selected correlations range from 0.21 to 0.34.Source: §4 · p. 5
  • Concepts are displayed as five example images per row with highlighted regions, and positive or negative contributions are separated into consecutive class rows.The authors avoided labeling concepts as class-related because their correlations with the model output were not strong and such labels could confuse participants.Source: §4 · p. 5

5 HUMAN SUBJECT STUDY

The human study tests whether explanations help users discover model-relevant features beyond a simple grid of inputs and predictions. The baseline outperformed concepts, while INN counterfactuals performed similarly overall and helped with some attributes.

5.1 DESIGN CONSIDERATIONS

  • The Two4Two study design aims to let lay participants learn the unfamiliar dataset quickly while avoiding prior domain knowledge as a confound.Its bias-discovery task is intended to be manageable without reducing evaluation to explanation accessibility alone.Source: §5.1 · p. 6
  • The baseline presents 50 images in five logit-sorted columns, offering users a simple comparison of model outputs across inputs.Participants received tutorials and comprehension checks, with exclusions for repeated failures or serious misunderstandings.Source: §5.1 · p. 6

5.2 EXPERIMENTAL DESIGN

  • The study task asked participants to rate five features as relevant or irrelevant, using accuracy as the primary measure and written justifications for qualitative analysis.The two studies used within-subject and between-subject designs, respectively; the second study compared three conditions and analyzed 192 valid responses.Source: §5.2 · pp. 6–7, §5.3 · p. 7

5.3 RESULTS

  • 84.5% for INN and 80.8% for BASE did not differ significantly, while CON scored 31.7%; Figure 4 summarizes correct-answer proportions by condition.Random answers would yield 50% accuracy.Source: Figure 4 · p. 7, Table 2 · p. 7, §5.3 · p. 8
  • INN users identified legs and color more reliably than BASE users, but did worse on background; shape and position did not differ significantly.The first study also found color less often than legs (49% vs. 86%), while shape accuracy was 74%.Source: §5.3 · pp. 7–8
  • Participants sometimes used probabilistic reasoning about color, recognizing that a feature could matter without appearing consistently in every example.This reasoning contrasted with deterministic expectations that all images should support a feature hypothesis.Source: §5.3 · p. 8
  • Concept users often expected complete, consistent highlighting, while overlapping features made it difficult to tell which attribute the highlighted regions represented.This confusion was especially apparent when animal features and background evidence appeared together.Source: §5.3 · p. 8
  • Some INN participants missed subtle background changes, while others noticed them but disagreed about whether they indicated relevance.An observer convnet measured background changes as 0.045, compared with 0.662 for legs’ position, 0.624 for shape, and 0.440 for color.Source: §5.3 · pp. 8–9, Figure 5 · p. 9

6 CONCLUSION

The two tested explanation methods did not significantly outperform the model-output baseline, although counterfactuals helped identify the strongest bias. The authors argue that user studies are needed to evaluate explanations, while noting limits in the study’s factors, task, and synthetic setting.

6 CONCLUSION

  • The study contributes a controllable-bias dataset, feature-importance verification methods, and a study design for evaluating interpretable vision with users.Its simple baseline relies only on model outputs.Source: §6 · p. 9
  • Budget constraints limited the experimental factors, and the effects of other dataset configurations and visual saliencies remain unclear.The binary-choice task and required textual justifications also limited understanding of participants’ issues.Source: §6 · p. 9
  • Neither tested method significantly outperformed the baseline, but counterfactuals helped users identify the model’s strongest bias.Some participants nevertheless rated the background’s relevance incorrectly when interpolation changes were subtle.Source: §6 · p. 9
  • The authors believe the findings may have implications for natural images, because TWO4TWO represents challenges such as spatially overlapping features in an abstract domain.They suggest poor performance on TWO4TWO would likely also indicate poor performance on natural data with spatially overlapping features.Source: §6 · p. 9
  • User studies can guide technical innovation by identifying where users struggle, though the authors describe them as laborious and expensive.They suggest more realistic controllable datasets, such as augmented reality, and invite work to beat the baseline.Source: §6 · p. 9

8 ETHICS STATEMENT

The study paid crowd workers above the UK minimum wage, obtained consent to use their answers, and collected no personal data. The authors state that their organization did not require online user studies to receive ethics review board approval.

8 ETHICS STATEMENT

  • Crowd workers were paid £11.45 per hour, above the stated UK minimum wage of £8.91.Source: §8 · p. 10
  • Participants consented to the use of their answers, and the study collected no personal data.Source: §8 · p. 10
  • The authors’ organization did not require ethics review board approval for online user studies.Source: §8 · p. 10

9 REPRODUCIBILITY STATEMENT

The authors describe reproducibility as challenging because study results depend on the dataset, model, and video tutorials. They report open-sourcing study materials and estimate the cost of reproducing the studies.

9 REPRODUCIBILITY STATEMENT

  • The authors identify dataset, model, and video-tutorial choices as factors on which human-study results depend.Source: §9 · p. 10
  • The authors report open-sourcing the dataset, model, videos, code, and study, with some materials to become public later.At the time described, links to the study and videos were in the Appendix; a machine-readable export, source code, and model were to be shared with reviewers and made public later.Source: §9 · p. 10
  • The authors estimate reproducing both studies would cost around 2500 GBP, excluding software licenses.Source: §9 · p. 10
  • A study comparing one condition against the baseline was estimated to cost around 1400 GBP.The authors report spending around 5000 GBP on the work, including prestudies, software licenses, and both studies.Source: §9 · p. 10

A.1 TWO4TWO DATASET DETAILS

TWO4TWO represents attribute distributions and class information, and its examples include rotation-biased sampling. The appendix also describes how classifier-space interpolations are visualized through predicted attribute values.

A.1 TWO4TWO DATASET DETAILS

  • Table 5 describes attribute distributions, whether attributes are biased across classes, and whether they add class information beyond leg position.Source: Table 5 · p. 14
  • Table 6 is identified as showing the effect of interpolating along the weight vector.Source: Table 6 · p. 15
  • Figure 6 plots observer-convnet attribute predictions for samples whose logits are linearly interpolated in classifier space.Each line represents one sample.Source: Figure 6 · p. 15
  • The rotation-biased sampler conditions rotation on object type; its listing caption says high positive rotations predict Stretchy and low negative rotations predict Peaky.Source: §A.1 · p. 16

A.2 ARCHITECTURE OF THE INVERTIBLE NEURAL NETWORK

The INN combines a supervised classifier representation with unsupervised flow training, while its final mapping is trained only on the unsupervised objective. A MobileNet-V2 predicts attribute values, with test performance reported using mean squared error.

A.2 ARCHITECTURE OF THE INVERTIBLE NEURAL NETWORK

  • The seven-block Glow-based INN uses 32 flow steps per block, and retains fade-out values for reconstructing counterfactuals through the inverse.Each flow step contains actnorm, an invertible 1 × 1 convolution, and an affine coupling layer; down-sampling preserves dimensions before fade-out maps channels to a standard normal for the unsupervised loss.Source: §A.2 · p. 16
  • The first 641 layers receive supervised and unsupervised training, while the remaining layers are optimized with the standard unsupervised flow objective.The model has 687 layers and 261 million parameters; its classifier uses the output of layer 641. The unsupervised objective matches a standard normal prior and uses the Jacobian log-determinant.Source: §A.2 · p. 16
  • Layer 342, used for concept explanations, is an affine coupling layer.Source: §A.2 · p. 16

A.3 SUPERVISED MOBILENET-V2

  • MobileNet-V2 predicts attribute values, and its test performance is measured by mean squared error in Table 7.Source: §A.3 · p. 17, Table 7 · p. 17

B LINKS TO MODEL, CODE, DATASET AND STUDY

The appendix provides exports of the model, code, datasets, and study materials. It also links the study conditions, tutorials, and preregistration records.

B LINKS TO MODEL, CODE, DATASET AND STUDY

C USER-STUDY LINKS AND VIDEOS

D USER-STUDY PREREGISTRATION AND HYPOTHESIS

  • The appendix links preregistrations for TWO4TWO validation and Study 2, and reports paying participants the Study 2 tariff, including 3.50GBP for the third comprehension task.Source: D USER-STUDY PREREGISTRATION AND… · p. 19

D.1 VALIDATION OF TWO4TWO

This section describes studies designed to test whether COLOR or SHAPE biases are harder to detect than LEGS, compare participants’ accuracy in identifying model-relevant attributes across three methods, and examine how participants used explanations when reasoning about attribute relevance.

D.1 VALIDATION OF TWO4TWO

  • The validation study tested whether participants identify COLOR or SHAPE as relevant less often than LEGS, using a grid of model predictions and relevant/irrelevant ratings.LEGS, COLOR, and SHAPE were ground-truth relevant; BACK and ROT were irrelevant.Source: §D.1 · p. 19

D.2 STUDY 2: CONCEPTS VS. BASELINE AND INN VS. BASELINE

  • Study 2 preregistered comparing baseline predictions (B), invertible-neural-network counterfactuals (CF), and prototypes (P), with hypotheses that P yields lower accuracy and CF higher accuracy than B.Participants were randomly assigned to one method; accuracy was the percentage of correctly classified relevant and irrelevant attributes.Source: §D.2 · p. 21
  • Study 2 displayed 50 images in a 10×5 grid: score-sorted predictions for B, prediction-changing counterfactual interpolations for CF, or correlated concepts with five examples per row for P.Displayed concepts had correlation of at least r=0.2 with model logits; the concepts were based on Zhang et al. (2020).Source: §D.2 · p. 21
  • For Study 2, the preregistered analysis used Kruskal-Wallis and focused Wilcoxon rank-sum tests if scores were non-normal, or ANOVA-based planned contrasts if assumptions were met.The plan assessed normality with Shapiro-Wilk and variance homogeneity with Levene’s test, using Welch’s ANOVA if the latter assumption failed.Source: §D.2 · p. 22
  • Study 2 planned 240 Prolific participants, randomly and equally assigned to conditions, and collected expertise ratings for descriptive accuracy comparisons across expertise levels.Eligibility included an academic degree and at least 90% Prolific approval; participants also completed comprehension checks, while low-effort or misunderstood responses could be excluded.Source: §D.2 · pp. 22–23
  • The authors planned open and axial coding of survey responses to explore how participants integrated explanations into reasoning about attribute relevance.Source: §D.2 · p. 23

Related papers

Loading 2204.11642v1…