Source-linked AI summary
Towards Automatic Concept-based Explanations
Amirata Ghorbani, James Wexler, James Zou, Been Kim
TL;DR
Existing feature-based explanations make per-input importance judgments, while concept-query methods depend on human-selected labeled concepts and therefore have limited discovery. ACE automatically extracts higher-level visual concepts by segmenting class images, clustering segments in CNN activation space, and scoring concepts with TCAV; experiments report concepts that are meaningful, coherent, and important for prediction.
Problem
Feature-based explanations provide per-input feature importance, while existing concept methods require human-selected labeled examples, limiting automatic discovery of possible concepts.
Method
ACE automatically groups related image segments from a class into concepts and uses CNN activation-space similarity plus TCAV to assign concept importance.
Results
Human experiments and model analyses show that ACE discovers concepts that are meaningful, coherent, and important for neural-network predictions.
Takeaways & Limitations
The discovered concepts reveal potentially surprising correlations learned by the model and provide insights into its prediction behavior.
Takeaways & Limitations
ACE is evaluated on image data and assumes concepts appear as groups of pixels, leaving more abstract concepts and other data types for future work.
Abstract
from arXiv · showhide
Interpretability has become an important topic of research as more machine learning (ML) models are deployed and widely used to make important decisions. Most of the current explanation methods provide explanations through feature importance scores, which identify features that are important for each individual input. However, how to systematically summarize and interpret such per sample feature importance scores itself is challenging. In this work, we propose principles and desiderata for \emph{concept} based explanation, which goes beyond per-sample features to identify higher-level human-understandable concepts that apply across the entire dataset. We develop a new algorithm, ACE, to automatically extract visual concepts. Our systematic experiments demonstrate that \alg discovers concepts that are human-meaningful, coherent and important for the neural network's predictions.
1 Introduction
Feature-based explanations assign importance to individual input features, but their reliability and interpretability are limited. ACE addresses the open-ended concept-discovery problem by automatically identifying higher-level concepts across data and evaluating their importance.
- Feature-based methods estimate prediction importance by removing, perturbing, or altering individual input features, but studies identify reliability vulnerabilities.These methods can be sensitive to simple input shifts and adversarial perturbations.
- Concept-based explanations summarize model behavior with higher-level human-understandable units rather than individual pixels or features.Examples include concepts such as wheels or police logos for detecting police vans.
- Existing concept-query methods require humans to identify concepts and provide labeled examples, limiting discovery when the possible concept space is unlimited or unclear.Human choice can also introduce bias into the explanation process.
- ACE lays out principles for concept-based explanations and automatically aggregates related local image segments to identify meaningful, important concepts without requiring predefined concept labels.The method is evaluated with quantitative human experiments and model-focused analyses.
2 Concept-based Explanation Desiderata
The paper frames concepts as human-understandable units beyond individual features and proposes desiderata for evaluating their explanations. These desiderata emphasize meaningfulness and perceptual coherency, while remaining an incomplete starting point.
- Concepts are units intended to make model decisions more understandable than individual features, pixels, or characters.The paper avoids a precise definition and instead specifies desired properties for human-understandable explanations.
- Meaningfulness requires a concept to be semantically meaningful on its own and associated with similar meanings by different individuals.For images, a meaningful segment may contain a texture or object part rather than an isolated pixel.
- Coherency requires concept examples to be perceptually similar to one another while differing from examples assigned to other concepts.This criterion distinguishes internally consistent concept groups from mixed collections.
- The proposed properties are not claimed to be a complete desiderata set, but are presented as a useful starting point for concept-based explanations.
3 Methods
ACE is a global, unsupervised explanation method that extracts image concepts from class examples and assigns them importance with TCAV. It segments images, clusters similar segments in CNN activation space, removes outliers, and returns concept scores, with effectiveness limited by these processing stages and the CNN similarity assumption.
- ACE overview: ACE explains an entire classifier class without human supervision by extracting concepts from class images and returning each concept’s importance.It is a global explanation method rather than an explanation of only one individual input.
- Segmentation: ACE uses multi-resolution image segmentation to obtain segments spanning fine-grained textures and colors through coarser parts and objects.The paper states that three segmentation levels are sufficient in its experiments.
- Clustering: ACE maps resized segments into a CNN activation space, clusters nearby segments as concepts, and removes low-similarity outliers to improve coherency.The similarity measure works best with classifiers robust to scale and aspect ratio.
- Importance scoring: ACE uses TCAV to score a concept’s importance as the fraction of class images whose prediction score increases when representations are perturbed toward that concept.The final concept-importance stage could use another concept-importance score instead.
- Limitations: ACE’s ability to satisfy the desiderata depends on segmentation quality, clustering and outlier removal, and the reliability of CNN activation space as a similarity metric.
4 Experiments and Results
ACE extracts coherent, human-meaningful concepts across ImageNet classes, and experiments show these concepts are both interpretable and important to model predictions.
- ACE examples: ACE outputs the four most important concepts for each class from multiple complexity levels, including textures, object parts, wheels, windows, and spines.The examples were drawn from 100 randomly selected ImageNet classes, with three examples shown per concept in Figure 2.
- Human experiments: 56% of participants used the most frequent word or its synonyms to describe concepts, while 77% used one of the two most frequent words.Participants also selected same-concept segments over random segments 95.6% (14.3/15, ±1.0) of the time.
- Human experiments: 99% (14.9/15, ±0.3) of intruder-detection answers for discovered concepts were correct, compared with 97% (14.6/15, ±0.7) for hand-labeled concepts.The experiment used 30 participants and tested whether subjects could identify a conceptually different image.
- Model insights: ACE reveals intuitive, unintuitive, and part-specific correlations, such as police markings, basketball jerseys over balls, and carousel lights being more important than seats.The examples show that model-important concepts may reflect dataset regularities or distinguish separate parts of complex objects.
- Concept importance: Top-5 concepts retained within 80% of original accuracy, while removing them misclassified more than 80% of correctly classified samples.The evaluation used 1,000 ImageNet validation images and compared importance-ranked ordering with random and reverse orderings.
- Concept importance: For 20 classes, randomly stitched examples of four important concepts were classified correctly more than 80% of the time; over half exceeded 40%.The result was far above the stated random-chance rate of 0.001%, indicating that extracted concepts can suffice for classification without original image structure.
5 Related Work
Related work mainly explains individual predictions through feature importance, while concept-stitching experiments illustrate a distinct concept-level analysis of model behavior.
- Post-training explanations: Post-training explanation methods commonly estimate feature or training-sample importance for individual data-point predictions.These methods operate on inputs such as covariates, pixels, or other features.
- Concept-level analysis: Stitching important concepts can produce class predictions, with basketball jerseys, zebra skin, lionfish, and king snake patterns sometimes sufficient for Inception-V3 classification.This experiment tests whether important concepts can support prediction without preserving the original image structure.
- Limitations of feature-based explanations: Feature-based explanations have documented methodological and fundamental limitations, including similarity between explanations for trained and randomized models.Prior work also found that some methods reconstruct input images rather than estimate pixel importance.
6 Discussion
ACE automatically groups image features into meaningful, coherent, prediction-relevant concepts, but its scope is limited to image data and pixel-group concepts, with further tuning needed for class-specific granularity.
- 6 Discussion: The method has been evaluated only on image data and assumes concepts appear as groups of pixels.Applying concept-based explanations to text and extracting more complex or abstract concepts remain future directions.
- 6 Discussion: Class-specific tuning of ACE hyperparameters may better match the inherent granularity of concepts across different scene types.Nature scenes may contain fewer concepts than city scenes.
- 6 Discussion: ACE automatically groups input features into high-level concepts that are meaningful, coherent, and important for correct image predictions.Human experiments verified meaningfulness and coherency, while evaluation showed the concepts carry salient predictive signals.
A More Implementation Details
The implementation discovers concepts from sampled ImageNet images by multi-scale segmentation and clusters the resulting segments, with clustering choices evaluated empirically.
- A More Implementation Details: ACE uses 50 training images from each of 100 randomly selected ImageNet classes as concept-discovery data.Each image is segmented into 15, 50, and 80 SLIC super-pixel segments.
- A More Implementation Details: Each segment is resized to Inception-V3’s input size and mapped to its mixed_8-layer representation before concept clustering.Segments are padded with the network’s default zero value during resizing.
- A More Implementation Details: The implementation compares K-means, Affinity Propagation, and DBSCAN for grouping segments into coherent concepts while removing meaningless segments.Affinity Propagation often produced 30–70 clusters, followed by hierarchical simplification.
B ACE Considers Simple to Complex Concepts
ACE’s multi-resolution segmentation discovers concepts spanning simple visual properties and complex object or body parts, while grouping differently sized instances of the same concept.
- B ACE Considers Simple to Complex Concepts: Multi-resolution segmentation returns concepts ranging from colors and textures to body parts and objects.The method therefore captures concepts across multiple visual scales and abstraction levels.
- B ACE Considers Simple to Complex Concepts: ACE identifies differently sized instances as the same concept, such as two car windows where one is twice as large as the other.The supplementary examples include blue color, asphalt texture, car window, and human face concepts.
C Drawbacks of ACE
ACE can produce incoherent or duplicate concepts because of segmentation, clustering, or similarity errors, although reported incoherent concepts were not among the five most important concepts.
- C Drawbacks of ACE: ACE may return meaningless or non-coherent concepts because of segmentation, clustering, or similarity-metric errors.Such cases were described as rare, and incoherent concepts did not appear among the top five important concepts in the 100-class experiments.
- C Drawbacks of ACE: ACE may discover several human-perceived duplicates, such as separate wavy, calm, and shiny ocean-surface concepts with similarly high TCAV scores.It remains unresolved whether these distinctions reflect the network’s representations or should be merged into one ocean concept.
- C Drawbacks of ACE: Supplementary Figure 2 contrasts semantically inconsistent concepts with duplicated concepts discovered by ACE.The caption indicates that inconsistent concepts receive low or no TCAV scores, while duplicated concepts may still be discovered.
D Stitching Concepts
Supplementary examples show stitched images that the Inception-V3 network classified correctly.
- Correctly classified stitched images are shown as examples for the Inception-V3 network.
E More Examples of ACE
ACE examples are presented across 12 ImageNet classes, with four of the top-5 important concepts shown for each class.
- Examples cover 12 ImageNet classes, showing four of the top-5 important concepts for each class.
- Additional supplementary figures provide more examples of ACE across the displayed classes.
- The displayed class labels include Gorilla, Park Bench, Casette, Shopping Cart, Norfolk Terrier, and Jeep.