Towards Automatic Concept-based Explanations

Introduction

• As ML is used in fields such as medicine and commerce, understanding model predictions has become important and can be legally required [16].The passage also describes explainability as a required component of responsible ML use.Source: §1 · p. 1
• Feature-based explanations estimate prediction importance by removing or perturbing individual input features, but are unreliable and can fail to improve human understanding or trust.The passage also reports that identical explanations led human subjects to find evidence for contradictory conclusions.Source: §1 · p. 1
• Query-driven concept explanations depend on human-labeled examples, limiting discovery when possible concepts are unclear or unlimited and risking human bias in concept selection.These methods can test only concepts already identified and labeled by people.Source: §1 · p. 1

Our contribution

• ACE automatically identifies higher-level human-meaningful concepts important to the model by aggregating related local image segments across diverse data.The authors also lay out general principles for concept-based explanations and report quantitative human experiments and evaluations of ACE.Source: §1 · p. 2

2 Concept-based Explanation Desiderata

• Concept-based explanations should use units more understandable to humans than individual features, pixels, or characters.The authors use “concepts” for these units and note that defining a concept precisely is difficult.Source: §2 · p. 2
• The proposed properties are not claimed to be complete, but are offered as a starting point for concept-based explanations.Source: §2 · p. 2

• ACE explains an entire class without human supervision by extracting concepts from its images and returning each concept’s importance.Source: §3 · pp. 2–3

Automated concept-based explanations step-by-step

• ACE segments class images at multiple resolutions to capture concepts ranging from fine-grained textures and colors to coarser parts and objects.The experiments use three resolution levels to capture textures, object parts, and objects.Source: §3 · p. 3, Figure 1 · p. 3
• ACE maps resized segments into a CNN activation space, clusters them by Euclidean distance as concept examples, and removes cluster outliers to improve coherency.Prior work [44] motivates Euclidean distance in final-layer activation space as a perceptual similarity metric; segments are resized regardless of aspect ratio, as in Dabkowski & Gal [8]. The similarity measure works best with classifiers robust to scale and aspect ratio.Source: §3 · p. 3, Figure 1 · p. 3

How ACE is designed to achieve the three desiderata

• ACE uses super-pixel segmentation for low-cost, multiresolution automation, trading off against lower segmentation quality that can produce meaningless or overly broad segments.Semantic segmentation can produce meaningful classes but is computationally more costly and often cannot segment at different resolutions.Source: §3 · pp. 3–4

Automated concept-based explanations step-by-step

• ACE scores concepts with TCAV [20], the fraction of class images whose prediction score increases when activation representations are perturbed toward concept examples.The score approximates the average positive effect of a concept on class prediction using directional derivatives.Source: §3 · pp. 3–4

How ACE is designed to achieve the three desiderata

• ACE’s ability to satisfy the desiderata depends on segmentation, clustering and outlier removal, and especially the reliability of CNNs as a similarity metric.Source: §3 · p. 4

Examples of ACE algorithm

• ACE extracted concepts spanning different complexity levels, from lionfish spines and skin texture to car wheels and windows.The examples show the top four concepts for each of three ImageNet classes; the figure also illustrates a police van prediction using its tire and police logo.Source: §4 · p. 4, Figure 2 · p. 5

Human experiments

• Participants identified ACE concepts as coherent with 99% (14.9/15, ±0.3) accuracy, compared with 97% (14.6/15, ±0.7) for hand-labeled concepts.In the intruder-detection task, participants selected the conceptually different image from six choices.Source: §4 · pp. 4–5
• Participants preferred same-concept segments over random same-class segments 95.6% (14.3/15, ±1.0) of the time; descriptions converged on the top word and synonyms 56% of the time and the top two words 77%.These results indicate that participants found the concepts meaningful and used consistent verbal descriptions for them.Source: §4 · p. 5

Insights into the model through ACE

• ACE revealed intuitive prediction cues, dataset-linked cues, and separately important object parts.Police markings mattered for police-van predictions, while basketball jerseys outweighed the ball; jerseys appeared in 48 of 50 inspected training images, and carousel lights mattered more than seats.Source: §4 · p. 6, Figure 5 · p. 7

Examining the importance of important concepts

• Adding the top five concepts reached within 80% of original accuracy; removing them misclassified more than 80% of samples originally classified correctly.The evaluation used 1,000 randomly selected ImageNet validation images from the same 100 classes.Source: §4 · p. 6, Figure 4 · p. 7

Testing concept sufficiency

• Images stitched from the four top concepts were classified correctly more than 80% of the time for 20 classes, and more than 40% for over half of classes, versus 0.001% random chance.The result suggests the extracted concepts can be sufficient for the model’s predictions even without the objects’ original structure.Source: §4 · pp. 6–7

• Post-training methods explain already-trained models [41] [39], commonly by estimating feature or training-sample importance for individual predictions [33] [34] [43] [22].Source: §5 · p. 7
• The method explains each class using concepts present in its images without human-labeled examples, though rare returned concepts failed the desiderata.Source: §5 · p. 8
• Some feature explanations resemble those for randomized models, reconstruct inputs [38], invite confirmation bias [20], or scale poorly to high-dimensional data.Source: §5 · p. 8
• TCAV [20] estimates concept importance and IBD decomposes image predictions into conceptual components, but both require humans to provide concept examples.Source: §5 · p. 8

6 Discussion

• The experiments cover image data and concepts represented as pixel groups, leaving more abstract concepts difficult to extract automatically.The paper notes that concept-based explanations may also apply to text and other data types, an interesting direction for future work.Source: §6 · p. 8
• Future work includes tuning ACE’s hyperparameters separately for each class to better capture objects’ inherent granularity.For example, nature scenes may contain fewer concepts than city scenes.Source: §6 · p. 8
• ACE groups input features into high-level concepts that are coherent examples and important for correct image predictions, supported by human experiments and prediction validation.The discovered concepts also reveal potentially surprising correlations the model has learned, which may help promote safer use of machine learning.Source: §6 · p. 8

A More Implementation Details

• Using 100 random ImageNet classes and 50 training images per class, the method segmented images into 15, 50, or 80 super-pixels and extracted mixed_8 representations.Segments were resized to the Inception-V3 input size by padding with its default zero value, grayscale 117.5.Source: A More Implementation Details · p. 11
• The best clustering pipeline used K-means with K = 25, retaining each cluster’s 40 segments nearest its center; Affinity Propagation typically yielded 30–70 clusters, later simplified hierarchically.K-means, Affinity Propagation, and DBSCAN were tested.Source: A More Implementation Details · p. 11
• The method removed infrequent clusters and kept high-frequency, medium-frequency-and-popularity, or high-popularity clusters under discovery-image and cluster-size thresholds.High frequency required segments from more than half of discovery images; medium frequency required more than one-quarter and cluster size above the image count; high popularity required cluster size above twice the image count.Source: A More Implementation Details · p. 11

B ACE Considers Simple to Complex Concepts

• ACE identifies concepts ranging from simple color and texture to complex body parts and objects, with similar-abstraction concepts sharing similar semantic meanings in human evaluation.The paper reports that a human experiment verified the semantic similarity.Source: B ACE Considers Simple to Complex… · p. 11
• Multi-resolution segmentation detects concepts at varying sizes, including car windows of different sizes identified as the same concept.The examples also include blue color, asphalt texture, and human face.Source: Figure 1 · p. 11

C Drawbacks of ACE

• Segmentation, clustering, or similarity-metric errors can produce meaningless or incoherent concepts, but none ranked among the top five for any of the 100 experimental classes.The authors suggest incoherence may also reflect concepts that are similar to the network but not to humans.Source: C Drawbacks of ACE · p. 12
• ACE separately discovered three human-perceived duplicate ocean-surface concepts—wavy, calm, and shiny—with similarly high TCAV scores; whether the network distinguishes them or they should be merged remains open.The authors leave further combining them into one “ocean” concept for future work.Source: C Drawbacks of ACE · p. 12, Figure 2 · p. 12

D Stitching Concepts

• Supplementary Figures 3 and 4 show stitched images classified correctly by the Inception-V3 network.Source: Figure 3 · p. 13, Figure 4 · p. 14

E More Examples of ACE

• For 12 ImageNet classes, four of each class’s top-five important concepts are shown as ACE examples.Supplementary Figures 5–8 provide more examples.Source: E More Examples of ACE · p. 15, Figure 5 · p. 15, Figure 6 · p. 16
• Supplementary Figures 5 and 6 show more examples of ACE.Source: Figure 5 · p. 15, Figure 6 · p. 16
• Supplementary Figures 7 and 8 show more examples of ACE.Source: Figure 7 · p. 17, Figure 8 · p. 18