Source-linked AI summary
Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, Rory Sayres
TL;DR
Deep-learning models are difficult to interpret because their internal states and low-level features do not readily express human concepts. The paper introduces CAVs and TCAV, which use example-defined concept directions and directional derivatives to quantify conceptual sensitivity. Experiments apply the approach to image classification and diabetic retinopathy, with evidence that CAVs capture intended concepts and provide useful model insights.
Problem
Deep-learning models have opaque internal states and often operate on low-level features rather than human-interpretable concepts, limiting concept-based understanding.
Method
TCAV learns Concept Activation Vectors from user-provided concept examples and uses directional derivatives to quantify conceptual sensitivity across classes without retraining the model.
Results
Experiments found that CAVs corresponded to intended concepts and supported insights across standard image classifiers and a diabetic-retinopathy application.
Takeaways & Limitations
TCAV provides a human-friendly, post hoc way to ask model-decision questions in terms of user-specified high-level concepts.
Takeaways & Limitations
Randomly chosen examples can produce meaningless CAVs, so TCAV requires significance testing to distinguish meaningful concepts.
Abstract
from arXiv · showhide
The interpretation of deep learning models is a challenge due to their size, complexity, and often opaque internal state. In addition, many systems, such as image classifiers, operate on low-level features rather than high-level concepts. To address these challenges, we introduce Concept Activation Vectors (CAVs), which provide an interpretation of a neural net's internal state in terms of human-friendly concepts. The key idea is to view the high-dimensional internal state of a neural net as an aid, not an obstacle. We show how to use CAVs as part of a technique, Testing with CAVs (TCAV), that uses directional derivatives to quantify the degree to which a user-defined concept is important to a classification result--for example, how sensitive a prediction of "zebra" is to the presence of stripes. Using the domain of image classification as a testing ground, we describe how CAVs may be used to explore hypotheses and generate insights for a standard image classification network as well as a medical application.
1. Introduction
The paper frames interpretability as a gap between human concepts and opaque, low-level model representations, then introduces TCAV to quantify concept sensitivity without retraining. The method uses user-defined examples and directional derivatives to produce global, human-friendly explanations.
- Interpretability matters for understanding, designing, debugging, and aligning machine-learning models with human values.
- Neural networks often operate on low-level features and internal activations that do not correspond to human-interpretable concepts.
- Users can define concepts with new example data, including concepts absent from the model’s training data.
- CAVs represent concepts as activation-space directions learned by a linear classifier separating concept examples from random counterexamples.
- TCAV applies directional derivatives to quantify a concept’s sensitivity for an entire class, such as the influence of stripes on zebra predictions.
- The approach is accessible, customizable, plug-in ready, and globally quantitative, and experiments examine classification models, dataset biases, human evaluations, and diabetic retinopathy.
2. Related work
Related interpretability methods commonly explain individual predictions through input features or local perturbations, while TCAV extends learned latent directions toward global, concept-based explanations for classes.
- Interpretability research includes general methods, neural-network-specific methods, and approaches leveraging neural networks’ local linearity.
- Postprocessing existing models can provide insights without restricting users to inherently interpretable models or retraining high-performance networks.
- Perturbation-based methods validate explanations by checking how model outputs change, but local explanations may differ across inputs within the same class.
- TCAV produces explanations for each class rather than only for individual data points.
- Saliency methods typically visualize pixel importance for one image, and studies report vulnerabilities involving randomized networks, preprocessing changes, and adversarial attacks.
- TCAV combines learned meaningful directions with directional derivatives to support on-the-fly hypothesis testing and global explanations.
3. Methods
The method represents user-defined human concepts as directions in a model’s activation space, then uses directional derivatives and aggregated scores to quantify conceptual sensitivity across inputs. Statistical testing filters unstable or meaningless concepts, while relative CAVs support comparisons between related concepts.
- TCAV computes a quantitative explanation of each concept’s relative importance to model prediction classes without retraining or modifying the model.The method uses directional derivatives to quantify prediction sensitivity and produces a final TCAVQ measure.
- Analysts define concepts with example inputs, including new user-provided data rather than only existing features, labels, or training data.This flexibility supports hypothesis exploration by non-expert and expert analysts.
- A CAV is the normal to a linear decision boundary separating activations from concept examples and random counterexamples.For a concept C, a binary linear classifier distinguishes layer activations from positive examples PC and negative examples N.
- Directional derivatives measure prediction sensitivity to movement toward a concept direction at an activation layer, producing a per-concept scalar rather than a per-feature metric.This differs from pixel-level saliency, which measures sensitivity to changes in individual input features.
- TCAVQ is the fraction of inputs in class k whose activation vector is positively influenced by concept C, with values from 0 to 1.The score depends on the sign of conceptual sensitivity and summarizes sensitivity globally for all inputs in a label.
- Repeated CAV training and two-sided testing are used to reject spurious concepts, while relative CAVs compare the importance of related concepts such as textures.The procedure typically uses 500 runs, tests whether scores differ from 0.5, and applies a Bonferroni correction with m = 2.
4. Results
Experiments show that CAVs align with intended concepts, expose model sensitivities and biases, and indicate where concepts emerge across network layers. TCAV also quantitatively confirms qualitative findings and filters spurious concept signals through statistical testing.
- 4.1.1. SORTING IMAGES WITH CAVS: CAV-based image sorting qualitatively confirms intended concepts and can reveal biases in the examples used to define them.For example, the highest-ranked necktie images under the ‘model women’ CAV all show women wearing neckties.
- 4.1.2. EMPIRICAL DEEP DREAM: Activation maximization produces patterns that reflect knitted texture, corgis, and Siberian husky concepts, enabling visualization of meaningful directions in a layer.The method optimizes patterns that maximally activate each CAV and compares them with semantic notions of the concept.
- 4.2.1. GAINING INSIGHTS USING TCAV: TCAV confirms expected concept sensitivities and reveals gender- and race-related signals in two widely used image-classification networks, including apron and ping-pong-ball associations.The tested concepts included color, texture, objects, gender, and race, none of which were network class labels.
- 4.2.1. GAINING INSIGHTS USING TCAV: Concept influence varies by depth: race signals strengthen near the final prediction layer, whereas texture influences zebra predictions earlier.Figure 4 uses proximity to the logit layer as an indication of more direct influence on predictions.
- 4.2.1. GAINING INSIGHTS USING TCAV: Statistical significance testing removes spurious CAVs, such as a dotted concept that otherwise produced high TCAVQ for zebra classes.After filtering, surviving CAVs consistently identified striped as the most important concept in that layer.
- 4.2.1. GAINING INSIGHTS USING TCAV: TCAV quantitatively confirms prior qualitative findings, identifying arms as more important for dumbbell predictions despite learning CAVs from only 30 images per concept.The result agrees with an earlier DeepDream observation of a dumbbell held by an arm.
- 4.2.2. TCAV FOR WHERE CONCEPTS ARE LEARNED: CAV classifier accuracy indicates where concepts are learned: abstract concepts become more accurate in higher layers, while simple concepts such as color remain accurate throughout.Accuracies are measured on a held-out test set one-third the size of the training set.
4.3. A controlled experiment with ground truth
The controlled experiment compares TCAV with saliency maps using networks trained on images with captions whose reliability varies. TCAV closely tracks the approximated concept ground truth, while saliency maps communicate concept importance unreliably to human evaluators.
- A controlled experiment with ground truth: The experiment trained four networks on zebra, cab, and cucumber datasets with caption noise parameter p ranging from 0 to 1.Caption noise controls the probability that a caption agrees with the image class, and caption-less-image performance approximates reliance on image concepts.
- Quantitative evaluation of TCAV: TCAVQ closely matches the approximated ground truth for whether networks rely more on image or caption concepts.For cab classification, image concepts dominate across models; for cucumber classification, reliance shifts between caption and image concepts as noise changes.
- Evaluation of saliency maps with human subjects: 52% of saliency-map judgments correctly identified which concept was more important, barely above the 50% random-chance baseline.In more than half of the conditions, viewers either found no significant difference or selected the wrong concept.
- Evaluation of saliency maps with human subjects: Very confident judgments were as common for incorrect as for correct saliency-map answers, making saliency-map interpretation potentially misleading.The two saliency-map methods also disagreed whenever one method correctly communicated the more important concept.
- TCAV for a medical application: The diabetic-retinopathy application found high TCAV scores for concepts relevant to level 4 and low scores for a non-diagnostic concept.For level 1, TCAV sometimes diverged from doctors’ heuristics and highlighted HMA alongside level-2-related concepts, consistent with frequent level-1-to-level-2 overprediction.
5. Conclusion and Future Work
TCAV provides a human-friendly linear interpretation of deep-learning internal states using natural concepts defined after training. Experiments suggest that CAVs correspond to intended concepts and provide insights across image-classification and medical applications.
- Conclusion: TCAV interprets a deep-learning model’s internal state in terms of natural high-level concepts.These concepts can be specified post hoc through user-provided examples rather than being known during training.
- Conclusion: Experiments provide evidence that CAVs correspond to their intended concepts and help explain predictions in standard and specialized classification models.The demonstrated applications include standard image classification networks and a diabetic-retinopathy model.
- Future Work: Future work includes applying TCAV to audio, video, and sequences, identifying adversarial examples, and automatically identifying concepts.The paper also proposes exploring networks that show super-human performance while remaining interpretable.
Appendix
The appendix contains additional experiments, results, and figures beyond the paper’s main sections.
- Appendix: The appendix presents other experiments and additional results and figures.It serves as a repository for supplementary analyses and visual material.
A. TCAV on adversarial examples
The adversarial-example experiment examines whether TCAV’s concept scores change for images that are deliberately altered to receive an incorrect class. Adversarial images classified as zebra show TCAVQ distributions different from normal zebra images.
- TCAV on adversarial examples: TCAVQ distributions for adversarial images classified as zebra differ from those of normal zebra images.The figure compares two types of adversarial images and their TCAVQ patterns.
- Adversarial examples: Adversarial examples are small, often visually imperceptible image changes that can alter a network’s predicted class.The experiment uses targeted single-step Fast Gradient Sign Method attacks to create two adversarial-image sets.
B. Additional Results: Insights and biases:
This section presents TCAV results for widely used image-classification networks and provides further results from Section 4.2.1.
- TCAV results are reported for widely used image-classification networks.
- Further results from Section 4.2.1 are provided.
D. Additional Results: Sorting Images with CAVs
The additional results include layer-wise TCAV analyses, layer-wise empirical DeepDream visualizations using CAVs, and image sorting with CAVs.
- Figure 12 reports TCAV results for each layer.
- Figures 13 and 14 show empirical DeepDream using CAVs for each layer in GoogLeNet.
- Figures 15–19 present additional results on sorting images with CAVs.