Source-linked AI summary
On Completeness-aware Concept-Based Explanations in Deep Neural Networks
Chih-Kuan Yeh, Been Kim, Sercan O. Arik, Chun-Liang Li, Tomas Pfister, Pradeep Ravikumar
TL;DR
DNN explanations often focus on local, low-level features, leaving unclear whether concepts sufficiently account for model behavior. The paper defines completeness, discovers concepts optimized for completeness and interpretability, and introduces ConceptSHAP for concept importance. Experiments report recovery of ground-truth concepts and interpretable insights across synthetic, image, and language data.
Problem
Existing DNN explanations can be local and unintuitive, while concept explanations lack a sufficient measure of how fully they account for model decisions.
Method
The paper defines completeness from prediction recovery using concept scores, discovers concepts with an interpretability regularizer, and quantifies concept attribution with ConceptSHAP.
Results
The method recovers ground-truth concepts correctly and outperforms compared methods on correct concepts and alignment score, while providing insights on image and language datasets.
Takeaways & Limitations
Completeness-aware discovery can yield concepts that are both sufficient for explaining DNN decisions and interpretable to humans.
Takeaways & Limitations
The approach assumes concept scores are sufficient statistics of model predictions and focuses on post-hoc explainability of pretrained DNNs.
Abstract
from arXiv · showhide
Human explanations of high-level decisions are often expressed in terms of key concepts the decisions are based on. In this paper, we study such concept-based explainability for Deep Neural Networks (DNNs). First, we define the notion of completeness, which quantifies how sufficient a particular set of concepts is in explaining a model's prediction behavior based on the assumption that complete concept scores are sufficient statistics of the model prediction. Next, we propose a concept discovery method that aims to infer a complete set of concepts that are additionally encouraged to be interpretable, which addresses the limitations of existing methods on concept explanations. To define an importance score for each discovered concept, we adapt game-theoretic notions to aggregate over sets and propose ConceptSHAP. Via proposed metrics and user studies, on a synthetic dataset with apriori-known concept explanations, as well as on real-world image and language datasets, we validate the effectiveness of our method in finding concepts that are both complete in explaining the decisions and interpretable. (The code is released at https://github.com/chihkuanyeh/concept_exp)
1 Introduction
The paper argues that feature-based explanations are often local and unintuitive, motivating concept-based explanations that capture global model reasoning. It introduces completeness-aware concept discovery and ConceptSHAP to find and assess interpretable concepts.
- Feature-importance explanations usually describe local predictions rather than globally explaining how a DNN reasons.
- Low-level input features, such as raw pixels, are not necessarily intuitive explanations for human understanding.
- The paper asks whether unsupervised concepts can be sufficiently predictive of DNN decisions and how that sufficiency can be measured.
- Completeness measures whether concept scores are sufficient statistics for recovering a model’s prediction behavior.
- The proposed discovery method jointly seeks complete concepts and semantic interpretability through an interpretability regularizer.
- ConceptSHAP quantifies contextualized concept importance, while experiments evaluate concept retrieval, coherency, and insights on synthetic, image, and language data.
2 Related Work
Related work organizes post-hoc explanations around features, samples, and counterfactuals, while latent-variable methods seek human-relatable dimensions under inductive biases.
- Post-hoc interpretability methods commonly attribute decisions to input features, observed samples, or input changes that alter outcomes.
- Explanation research also evaluates methods through human-centered and functional criteria.
- Latent-variable approaches use dimensionality reduction or related techniques to uncover higher-level, human-relatable concepts.
- Meaningful latent dimensions generally require inductive biases rather than arising from a completely unsupervised setting.
3 Defining Completeness of Concepts
The paper defines completeness by testing how accurately model predictions can be recovered from concept scores. Concepts are computed as thresholded similarities between activation parts and concept vectors.
- Problem setting: A pretrained DNN is decomposed into an input-to-intermediate representation Φ and an intermediate-to-output model h.
- Problem setting: For convolutional-style networks, intermediate activations can be represented as concatenated embeddings of potentially overlapping input parts.
- Choosing the representation: Higher layers encode more abstract concepts with larger receptive fields, whereas lower layers encode more specific concepts with smaller receptive fields.
- Concept scores: Each concept is a unit vector in activation space, and its score for an input part is a thresholded inner product measuring closeness to that concept.
- Concept scores: Concept scores are normalized and aggregated across input parts to form the representation used for completeness evaluation.
- Completeness score: Completeness is the best validation accuracy obtainable from concept scores, offset by random-prediction accuracy so its lower bound is zero.
- Completeness score: The score approximates the given model through its architecture and parameters, optimizing only a mapping from concept scores rather than retraining the backbone.
- Illustration: In the XOR example, omitting one required concept leaves prediction incomplete, whereas using all concepts yields completeness ηf(c1, c2, ..., cm) = 1.
4 Discovering Completeness-aware Interpretable Concepts
The method discovers concepts by optimizing prediction recovery while regularizing for coherent, distinct neighbors, then assigns global and class-specific importance with ConceptSHAP.
- Limitations of existing methods: Existing methods often rely on labeled concepts or automatically derived image segments and may use first-order sensitivity assumptions for concept importance.
- Our method: The method assumes spatial dependency in input data to help learn coherent concepts, particularly for images and language.
- Our method: The proposed method jointly optimizes concept vectors and a mapping function using a surrogate likelihood motivated by topic modeling.
- Our method: The interpretability regularizer encourages each concept’s top-K nearest patches to be coherent and different concepts to be dissimilar.
- Optimization: The optimization converges faster than training a model from scratch because it updates only concept vectors and the mapping function.
- ConceptSHAP: ConceptSHAP adapts Shapley values to quantify each concept’s contribution to the completeness score.
- ConceptSHAP: Shapley-based importance uniquely satisfies efficiency, symmetry, dummy, and additivity axioms.
- ConceptSHAP: Per-class ConceptSHAP evaluates concept contributions using completeness restricted to validation examples belonging to a specified class.
5 Experiments
Experiments evaluate whether the method discovers complete and interpretable concepts on synthetic, image, and language classification tasks. Across these settings, it recovers known concepts, achieves high completeness, and receives stronger human interpretability judgments than baselines.
- 5.1 Synthetic data with ground truth concepts: The method outperforms ACE, k-means, and PCA on correct concepts and automated alignment, while also obtaining the highest number of agreed concepts.Agreement measures whether users consistently retrieve the same concept from nearest neighbors.
- 5.1 Synthetic data with ground truth concepts: The method’s nearest neighbors consistently contain the corresponding ground-truth shape, such as shape 1 appearing in every nearest neighbor for concept 1.This provides a qualitative check that retrieved concepts align with the known synthetic factors.
- 5.2 Image classification: On AwA, the method discovers 53 concepts after removing near-duplicate vectors from an initial set of 70 and ranks class-relevant concepts with ConceptSHAP.ConceptSHAP and per-class saliency scores are computed for each concept and class.
- 5.2 Image classification: The method achieves the highest completeness on both the synthetic dataset and AwA, and its completeness objective substantially improves completeness over the ours-noc ablation.Figure 2 compares completeness as the number of discovered concepts m varies.
- 5.2 Image classification: In a 10-user study, average concept-identification accuracy is 0.733 for the method, compared with 0.267 for PCA and 0.6 for k-means.Users selected which of three candidate patches belonged to the same concept as four displayed top images.
- 5.3 Text classification: On IMDB, four discovered concepts have completeness 0.97, indicating that they are highly representative of the classification model.The text model is a four-layer CNN with 0.9 test accuracy, and each input part contains 10 consecutive words.
- 5.3 Text classification: The IMDB concepts separate negative sentiment patterns, with one criticizing movies directly and another using words such as “not,” “doesn’t,” and “even.”ConceptSHAP scores and frequent words are reported alongside nearest review neighbors.
6 Conclusions
The paper frames completeness as a measure of concept sufficiency and combines it with interpretability constraints to discover useful explanations. It also identifies joint learning of concepts and the model as a future direction.
- 6 Conclusions: Completeness quantifies how sufficient a concept set is for explaining model behavior, while interpretability constraints support discovering concepts that are both complete and interpretable.Experiments cover synthetic data with known concepts and real-world image and language data.
- 6 Conclusions: Jointly learning concepts with the model is proposed as a future direction for improving interpretability beyond post-hoc explanations.The current work focuses on post-hoc explanations of pre-trained DNNs.
7 Broader Impact
The paper positions concept-based explanations as a route toward more coherent and complete transparency for DNNs. It argues that broader, concept-level explanations can support understanding, model improvement, and deployment across data types.
- Concept-based explanations are presented as important for coherent and complete transparency into DNN decisions.
- Validating explanation sufficiency is described as a necessary but often overlooked check for model accountability.
- The method is data type agnostic and is demonstrated on both image and language data.
- The method explains models with a small number of concepts, using nearest training examples to help users interpret them.
- These broader explanations are positioned as useful for developers and data scientists seeking model insights and improvements.
8 Ackowlegement
The authors acknowledge support from DARPA.
- The work acknowledges support from DARPA via grant FA87501720152.
Appendix A Relation to PCA
The appendix relates completeness to PCA under highly restrictive assumptions. PCA can maximize a completeness surrogate in that setting, but its lack of semantic interpretability limits its role as a concept representation.
- Under strict conditions, the first m PCA vectors used as concept vectors maximize the L2 surrogate of the completeness score.
- The proposition assumes an isometric mapping, minimized loss, T = 1, a specified expectation relation, and a linear function l.
- The assumptions are extremely stringent and may not hold in general.
- When the assumptions fail, low intermediate-layer reconstruction error does not necessarily imply high prediction accuracy.
- PCA vectors minimize reconstruction error or maximize projected variance rather than being trained for semantic meaningfulness.
Proof of Proposition A.1
The proof establishes the PCA result by connecting PCA’s reconstruction-error minimization to the completeness surrogate under Proposition A.1’s assumptions. The conclusion is explicitly limited to that stringent setting.
- PCA’s basic properties provide the reconstruction-error minimization used in the proof.
- Under the proposition’s assumptions, the proof rewrites the reconstruction layer using the relation between concept projections and model representations.
- The resulting inequalities show that PCA vectors maximize the L2 surrogate of the completeness score.
- The proposition is intended to show equivalence between completeness and PCA’s information-preserving dimensionality reduction, not to establish practical generality.
Appendix B Shapley Axioms for ConceptSHAP
Appendix B states that ConceptSHAP is uniquely characterized by four axioms governing how concept importance relates to completeness.
- ConceptSHAP gives the unique importance assignment satisfying four axioms for a concept set and completeness score.The axioms are efficiency, symmetry, dummy, and additivity.
- Efficiency requires all concept importance values to sum to the total completeness score.
- Symmetry assigns equal importance to equivalent concepts whose completeness contributions match for every other-concept subset.
- The dummy axiom assigns zero importance to a concept that never changes completeness when added.
- Additivity requires importance under the sum of two completeness scores to equal the sum of their separate importance values.
Appendix C Additional Experiments Results and Settings
The appendix details alignment evaluation, hyperparameter sensitivity, nearest-neighbor studies, and qualitative results across synthetic, image, and language settings.
- Automated Alignment score on Synthetic Dataset: Alignment measures the best average matching accuracy between discovered concept vectors and the five ground-truth shapes.Each concept is tested in both directions, cj and −cj, and the direction with the higher alignment score is retained.
- Hyper-parameter Choice and Sensitivity: Both completeness and alignment exceed 0.93 when λ1 and λ2 range from 0.05 to 0.3 and β ranges from 0 to 0.3.The authors therefore report that the method is not sensitive to these hyperparameters over the tested ranges.
- User Study Setting and Discussion: The user study uses 20 users to evaluate whether nearest neighbors let people retrieve the correct ground-truth concepts.The appendix also describes averaging agreed answers across eight users for each method.
- User Study Setting and Discussion: Baseline methods fail through repeated concepts, incoherent concepts, or concepts unrelated to ground-truth concepts.The appendix links repeated or unrelated concepts to insufficient completeness and incoherence to insufficient coherency.
- Additional Nearest Neighbors for AwA: In AwA, many important concepts are shared across classes and are semantically meaningful, including grass, animal faces, ears, and background textures.The examples are selected among the three highest-ConceptSHAP concepts per class with R1(c) above 0.8.
- Additional Nearest Neighbors for NLP: In NLP, the nearest neighbors of concepts 1 and 2 are generally negative, whereas those of concepts 3 and 4 are generally positive.