PaperlayerView exact revision on arXiv

Source-linked AI summary

Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability

Vikram V. Ramaswamy, Sunnie S. Y. Kim, Ruth Fong, Olga Russakovsky

arXiv:2207.09615v2cs.CV

TL;DR

Concept-based interpretability has received less scrutiny of probe-dataset and concept choices than of new method development. This paper studies those factors and explanation complexity, finding probe-dependent explanations, hard-to-learn concepts, and a human preference for fewer than 32 concepts.

  • Problem

    Research on concept-based interpretability has emphasized new methods while giving less attention to probe datasets and the concepts that compose explanations.

    Source: §1 · p. 1

  • Method

    The paper analyzes overlooked factors in concept-based methods and uses a human study to examine how explanation complexity affects people’s reasoning.

    Source: §1 · p. 1, §5.1 · pp. 6–7

  • Results

    Different probe datasets profoundly change explanations; many concepts are harder to learn than their target classes, and people prefer explanations with fewer than 32 concepts.

    Source: §8 · p. 8

  • Takeaways & Limitations

    The authors recommend similar-distribution probe data, concepts more learnable than target classes, and explanations with fewer than 32 concepts.

    Source: §6 · p. 8

    Each method was tested in one setting, and all human-study participants were MTurk recruits with limited machine-learning backgrounds.

    Source: §7 · p. 8

Abstract

from arXiv · show

Concept-based interpretability methods aim to explain deep neural network model predictions using a predefined set of semantic concepts. These methods evaluate a trained model on a new, "probe" dataset and correlate model predictions with the visual concepts labeled in that dataset. Despite their popularity, they suffer from limitations that are not well-understood and articulated by the literature. In this work, we analyze three commonly overlooked factors in concept-based explanations. First, the choice of the probe dataset has a profound impact on the generated explanations. Our analysis reveals that different probe datasets may lead to very different explanations, and suggests that the explanations are not generalizable outside the probe dataset. Second, we find that concepts in the probe dataset are often less salient and harder to learn than the classes they claim to explain, calling into question the correctness of the explanations. We argue that only visually salient concepts should be used in concept-based explanations. Finally, while existing methods use hundreds or even thousands of concepts, our human studies reveal a much stricter upper bound of 32 concepts or less, beyond which the explanations are much less practically useful. We make suggestions for future development and analysis of concept-based interpretability methods. Code for our analysis and user interface can be found at \url{https://github.com/princetonvisualai/OverlookedFactors}

1. Introduction

Concept-based explanations offer people high-level views of model behavior, but their reliability and usability depend on overlooked choices. The study finds that probe datasets, concept learnability, and explanation size all matter.

1. Introduction

  • Concept-based methods use semantic concepts to provide global, high-level views of model behavior, but prior work has largely emphasized method development over factors affecting explanations.Interpretability matters for understanding deep networks, particularly in high-risk settings such as medical diagnosis.Source: §1 · p. 1
  • Different probe datasets can produce very different explanations for the same model and method, so the explanations are not determined by the model or method alone.The authors suggest choosing probe data with a distribution similar to the model’s training data.Source: §1 · p. 2
  • The study examines probe-dataset choice, concept learnability, and explanation complexity across four representative concept-based methods.The methods are NetDissect, TCAV, Concept Bottleneck, and IBD.Source: §1 · pp. 1–2
  • Some concepts used to explain a class are harder for models to learn than that class, and the authors argue learnability is necessary, though not sufficient, for correctness.They advocate using easily learnable concepts.Source: §1 · p. 2
  • Participants recognized fewer concepts as explanation size increased, and most preferred at most 32 concepts; concept explanations offered little to no accuracy advantage over examples.Recognition fell from 71.7%±27.7% with 8 concepts to 56.8%±24.9% with 32; prediction accuracy was 64.8% ± 23.9% with 8-concept explanations versus 60.0% ± 30.2% with example-based explanations.Source: §1 · p. 2

2. Related work

Related work spans local saliency explanations, concept and component labeling, and interpretable-by-design models. Prior evaluation work has focused more on saliency maps than on concept-based methods and their probe datasets.

2. Related work

  • Computer-vision interpretability includes saliency maps [15] [37] [45] [46], component labeling and concept-based methods [5] [16] [25] [39], and interpretable-by-design models [8] [10] [29] [34].The paper focuses on concept-based methods for image classification, leaving newer applications to image similarity and language models outside its scope.Source: §2 · pp. 2–3
  • The paper notes that future methods ideally would provide causal rather than purely correlation-based explanations.This is framed as an ideal, not as an established property of the methods studied.Source: §2 · p. 2
  • Prior evaluation protocols have studied sensitivity in post-hoc saliency maps [1] [2] [3] [23] [26] [41] [49], while the authors identify probe datasets and concepts as underexamined factors in concept-based explanations.They describe their work as the first to investigate those factors; prior work on explanation complexity was limited to decision sets [30].Source: §2 · p. 3
  • The authors recommend releasing analyses of concept-based methods alongside explainability fact sheets, building on calls for more release information about datasets and models [12] [33].They intend these materials to help researchers and practitioners understand, use, and build on the methods.Source: §2 · p. 3

3. Dataset choice: Probe dataset has a profound impact on the explanations

The authors compare Baseline, NetDissect, and TCAV explanations generated from ADE20k and Pascal probe datasets. They find that probe choice changes explanations and suggest these explanations are usable only for images from the probe dataset’s distribution.

3. Dataset choice: Probe dataset has a profound impact on the explanations

  • The study compares Baseline, NetDissect, and TCAV using ADE20k and Pascal subsets of Broden as probe datasets.The methods produce class-level concept combinations, neuron-level concept labels, and concept activation vectors, respectively.Source: §3 · p. 3
  • Across Baseline, NetDissect, and TCAV, ADE20k and Pascal produce very different explanations, limiting their use to images from the probe dataset’s distribution.For Baseline, the table compares the concepts with the largest coefficients across 10 scene classes.Source: §3 · p. 3, Table 1 · p. 4
  • 60 of 123 neurons receive very different NetDissect concept labels when labelled using ADE20k versus Pascal.The authors report examples of differences in neuron labels.Source: §3 · p. 3
  • 0.078 average cosine similarity separates TCAV vectors learned on ADE20k versus Pascal, despite mean ROC AUC above 85% for the selected concepts.The comparison covers 32 concepts with a base rate above 1% in both datasets combined.Source: Figure 2 · p. 4, §3 · p. 4

4. Concept learnability: Concepts used are less learnable than target classes

The section tests whether concepts used to explain classes are easier to learn than those classes. Across Broden/Places365 and CUB, concepts are harder to learn; IBD explanations also include concepts harder to learn than their scene classes.

4. Concept learnability: Concepts used are less learnable than target classes

  • Concept-based explanations implicitly assume that concepts are easier to learn than the classes they explain, an assumption the reported comparisons challenge.The section illustrates the assumption with a model learning bed before predicting bedroom.Source: §4 · pp. 4–5
  • 7.6% median normalized AP for Broden concepts vs. 37.5% for Places365 classes; 2.3% for CUB concepts vs. 65.9% for CUB classes.The authors evaluate learnability with normalized average precision, setting base rates to 1/365 and 1/200 for the respective comparisons.Source: Figure 3 · p. 5, §4 · p. 5
  • All 10 randomly chosen scene classes in the IBD analysis include at least one of their five highest-coefficient concepts that is harder to learn than the class.For some classes, including bathroom and kitchen, every concept in the explanation is harder to learn than the class.Source: §4 · p. 5
  • A significant fraction of concepts used by existing methods are harder to learn than target classes, and these concepts can also be hard for people to identify.Source: §4 · p. 5

5. Human capability: Human studies reveal an upper bound of 32 concepts

The human study tests how people handle concept-based explanations as their complexity increases, including their ability to recognize concepts, predict model outputs, and choose between simplicity and correctness. Participants preferred explanations with 8, 16, or 32 concepts, while recognition worsened and took longer as concept counts rose.

5. Human capability: Human studies reveal an upper bound of 32 concepts

  • The study tested whether participants could recognize concepts, predict model outputs, and trade off explanation simplicity against correctness.Participants chose among explanations with 4, 8, 16, 32, or 64 concepts for the preference task.Source: §5 · p. 6, §5.1 · p. 6

5.1. Human study design

  • For the recognition and prediction task, participants judged concepts for images and selected among four classes using explanations for only cases where model and explanation outputs matched.The study examined linear combinations of concepts and evaluated explanation form rather than a particular explanation method.Source: §5.1 · pp. 6–7

5.2. Key findings from the human studies

  • 71.7% ± 27.7% (8 concepts) to 56.8% ± 24.9% (32 concepts): mean recall fell as image time increased.Median time per image was 17.4 seconds for 8 concepts and 46.2 seconds for 32; prediction accuracy was 64.8% ± 23.9% (8 concepts), 63.2% ± 26.9% (16 concepts), and 63.6% ± 22.2% (32 concepts), barely above 60.0% ± 30.2% for example-based explanations.Source: §5.2 · p. 7
  • 82% of participants preferred explanations with 8, 16, or 32 concepts; only 6% preferred 64 concepts.The preference shares were 28%, 33%, and 21%, respectively; participants considered both simplicity and correctness.Source: §5.2 · p. 8

6. Discussion

The discussion recommends probe datasets similar to the model’s training distribution, concepts more learnable than target classes, and fewer than 32 concepts. It also notes that concept methods can highlight hard-to-learn concepts, suggesting they may capture correlations rather than causation.

6. Discussion

  • The authors recommend probe datasets similar to the model’s training distribution, concepts more learnable than target classes, and fewer than 32 concepts.Because large-scale probe datasets with concept labels are scarce, researchers are often left to use Broden or CUB; the authors call for diverse, high-quality probe datasets.Source: §6 · p. 8
  • Concept methods may highlight hard-to-learn concepts, suggesting they sometimes capture correlations rather than causations.Patch- and region-based alternatives are described as local explanations, not class-level global explanations.Source: §6 · p. 8

7. Limitations and future work

The authors qualify their findings because probe-dataset scarcity limited each method to one setting, and the human-study participants came from Amazon Mechanical Turk. They note that preferences in the second study part may differ for developers and lay end-users.

7. Limitations and future work

  • Probe-dataset scarcity limited each concept-based method to a single setting, so the authors plan to expand analyses as more datasets become available.NetDissect, TCAV, and IBD were tested on a Places365 scene classifier; Concept Bottleneck was tested on CUB.Source: §7 · p. 8
  • Because participants came from Amazon Mechanical Turk and had limited ML experience, Part 2 preferences may differ for developers and lay end-users.The authors suggest developers debugging models may be more willing to trade simplicity for correctness; they expect Part 1 concept-identification results to be less sensitive to participants’ ML expertise or role.Source: §7 · p. 8

8. Conclusion

The conclusion examines probe-dataset choice, concept learnability, and explanation complexity, finding that each raises limits on concept-based explanations. It also points readers to appendices with additional experimental and human-study material.

8. Conclusion

  • Probe-dataset choice profoundly influences explanations, which the authors say can only be used for images from the probe-dataset distribution.The conclusion cautions that the generated explanations may not generalize beyond that distribution.Source: §8 · p. 8
  • A significant fraction of concepts in explanations are harder for a model to learn than the target classes they aim to explain.Source: §8 · p. 8
  • People struggle to identify image concepts when given too many, and explanations with fewer than 32 concepts are preferred.Source: §8 · p. 8

Appendix

  • The appendices provide experimental setup details, additional probe- and concept-choice results, human-study information, and user-interface snapshots.Source: §8 · p. 8, Appendix · p. 10

A. Experimental details

The appendix documents the experimental protocols for TCAV, baseline explanations, and concept learning, along with computation times.

A. Experimental details

  • TCAV predicts ground-truth attributes from Places365-trained ResNet18 penultimate-layer features using logistic regression, with regularization selected by validation ROC AUC.The setup uses an l2 penalty and the liblinear solver.Source: A. Experimental details · p. 11
  • The baseline predicts the black-box model output from ground-truth image concepts using logistic regression with an l1 penalty to prioritize simpler explanations.For human studies, explanation complexities use 4, 8, 16, 32, or 64 concepts.Source: A. Experimental details · p. 11
  • Concept-learning models use ResNet18 features for ADE20k images and logistic regression for concepts with more than 10 positive samples.The regularization weight is selected by validation ROC AUC, and normalized AP is reported to compare concepts and classes with varying base rates.Source: A. Experimental details · p. 11
  • Each linear model took less than 2 minutes on a CPU, while feature extraction took less than 15 minutes on a GTX 2080 GPU.Source: A. Experimental details · p. 11

B. Probe dataset choice: more details

This section provides further detail on the claim that probe-dataset choice can significantly affect concept-based explanation outputs.

B. Probe dataset choice: more details

B.1. Varying the probe dataset

Using ADE20k and Pascal as probe datasets yields divergent concept-based results for both NetDissect and TCAV. The reported differences are not explained by widespread polysemantic neuron activation.

B.1. Varying the probe dataset

  • 69 of 123 strongly activated neurons receive very different concept labels when NetDissect uses ADE20k versus Pascal.Source: §B.1 · p. 11
  • Only 7 of the 69 differently labeled neurons are highly activated by both concepts, using an IOU threshold above 0.04.Source: §B.1 · p. 11
  • TCAV concept activation vectors learned from ADE20k and Pascal are not very similar overall, despite predicting the concepts well.The comparison covers 32 concepts with base rates of at least 1%.Source: §B.1 · p. 11

B.2. Difference in probe dataset distribution

This appendix examines how probe-dataset composition limits explanation coverage, then adds CUB concept-learning results and details the human-study design and interface.

B.2. Difference in probe dataset distribution

  • Concept base rates differ across ADE20k and Pascal, with wall, floor, dog, and pole highlighted for different classes in each dataset.For example, wall was highlighted for bow-window using ADE20k, while dog was highlighted for corn-field using Pascal.Source: §B.2 · p. 11, Figure 5 · p. 13
  • Images within the same scene labels also differ: Pascal bedrooms often show a person or animal on a bed, whereas ADE20k shows full bedrooms.Pascal tree-farm images also contain people, unlike ADE20k images.Source: §B.2 · p. 11, Figure 6 · p. 13
  • Among 123 penultimate-layer ResNet18 neurons marked highly activated by both datasets by NetDissect, 69 corresponded to radically different labels; the remainder corresponded to the same or very similar concepts.Source: Table 4 · p. 12
  • 63.7% accuracy on Pascal and 75.7% on ADE20k were upper-bound estimates for explaining a Places365-trained model using each probe dataset.The authors reset and retrained the final linear layer to emulate the original scene predictions; they interpret ADE20k as somewhat more similar to Places365 than Pascal.Source: §B.2 · pp. 11–13
  • 89.3% accuracy was achieved when the same upper-bound procedure used the CUB test dataset for a Concept Bottleneck model.The authors expected a reasonably high bound because the training and test distributions were expected to be similar.Source: §B.2 · p. 13

D. Human study details

  • The human study compared 8-, 16-, and 32-concept explanations with 10-image example explanations across matched inputs, using 125 participants in a between-group design.For the 32-concept condition, participants saw five images; in the other conditions, they saw ten.Source: D. Human study details · p. 14

E. User interface snapshots

  • The study interface asked participants to recognize concepts and guess model outputs, then choose a preferred simplicity-correctness trade-off and optionally give feedback.The example-based condition omitted the concept-method introduction and task preview.Source: E. User interface snapshots · pp. 14–15

C. Concepts used: more details

Loading 2207.09615v2…