Source-linked AI summary
Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectors
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A. Ehinger, Benjamin I. P. Rubinstein
TL;DR
CNNs are powerful but difficult to explain, especially in critical applications where understanding discovered features matters. The paper proposes an invertible concept-based explanation framework using matrix factorization, and finds that NMF-derived NCAVs provide strong interpretability and fidelity across computational and human-subject evaluations.
Problem
CNNs lack basic explainability, while existing concept-based methods such as ACE can produce inconsistent weights and make fidelity difficult to measure.
Method
The paper replaces ACE’s segment clustering with invertible matrix factorization, adds fidelity measurements, and evaluates interpretability through human-subject experiments.
Results
NCAVs from NMF are more interpretable than PCA CAVs and at least as interpretable as clustering CAVs, while PCA provides the best fidelity and NMF is close.
Takeaways & Limitations
The framework provides consistent concept-level explanations with measurable fidelity and supports both local and global explanations for pretrained CNNs.
Abstract
from arXiv · showhide
Convolutional neural network (CNN) models for computer vision are powerful but lack explainability in their most basic form. This deficiency remains a key challenge when applying CNNs in important domains. Recent work on explanations through feature importance of approximate linear models has moved from input-level features (pixels or segments) to features from mid-layer feature maps in the form of concept activation vectors (CAVs). CAVs contain concept-level information and could be learned via clustering. In this work, we rethink the ACE algorithm of Ghorbani et~al., proposing an alternative invertible concept-based explanation (ICE) framework to overcome its shortcomings. Based on the requirements of fidelity (approximate models to target models) and interpretability (being meaningful to people), we design measurements and evaluate a range of matrix factorization methods with our framework. We find that non-negative concept activation vectors (NCAVs) from non-negative matrix factorization provide superior performance in interpretability and fidelity based on computational and human subject experiments. Our framework provides both local and global concept-level explanations for pre-trained CNN models.
Introduction
CNN explanation methods have moved from input-level features toward concept-level representations, but ACE has consistency and fidelity limitations. ICE addresses these shortcomings with invertible matrix-factorization explanations and evaluates interpretability through human studies.
- Introduction: ACE can learn CAVs without labels, but discarded segments and distances to cluster centroids may lose information.
- Introduction: ICE replaces ACE’s clustering of segmented feature maps with matrix factorization to address inconsistent weights and difficult fidelity measurement.The inverse factorization enables measuring information loss and differences between explanation and target-model predictions.
- Introduction: The framework supports both local and global concept-level explanations, whereas ACE provides global class explanations only.
- Introduction: ICE provides consistent feature weights under stated limitations and introduces fidelity measurements that can guide explainer hyper-parameters.
- Introduction: The paper proposes measuring concept interpretability scientifically with human-subject experiments rather than intuition and selected examples.
A Framework for Concept-based Explanations
Concept-based explanations use feature maps from an internal CNN layer to form linear approximations, balancing interpretability against fidelity. The framework favors higher layers and reducer outputs because they better represent concepts and preserve graded information for inversion.
- A Framework for Concept-based Explanations: Concept-based explanations separate a CNN into a concept extractor and classifier, then approximate the classifier using feature maps from a selected layer.
- A Framework for Concept-based Explanations: Higher CNN layers are preferred because they encode higher-level concepts and generally yield more accurate estimated weights.The paper therefore assumes the last layer is the most suitable target layer.
- A Framework for Concept-based Explanations: Reducers output continuous concept scores, unlike ACE’s binary concept scores, enabling more accurate inversion and contribution analysis.
- A Framework for Concept-based Explanations: Interpretability requires human-meaningful features, while fidelity requires approximate predictions to match the explained CNN, creating a dual-objective trade-off.
Methodology
The framework factorizes CNN feature maps into non-negative concept directions and scores, then estimates concept importance to produce local and global explanations.
- Non-Negative Concept Activation Vectors: NMF represents flattened non-negative feature maps as concept scores, directions, and residual error, with factorization separating frequently appearing concept directions.The decomposition is V = SP + U, where P contains concept directions and U is the residual minimized by NMF.
- Non-Negative Concept Activation Vectors: The learned NCAV directions remain fixed after training, while new images are projected onto them to obtain concept feature scores.The scores measure how strongly each feature-map vector relates to the learned NCAV directions.
- Weight Estimation: NCAV feature importance is estimated from classifier derivatives, using feature maps related to the target concept to reduce estimation error.The estimated weight is an average derivative over the sampled feature-map area.
- Weight Estimation: Using the last layer before GAP and the dense layer makes NCAV weights input-independent because the classifier’s feature importance is PW.This choice is motivated by the consistent derivative of a linear classifier, and PW does not depend on the input feature map A.
- Visualization: For visualization, images with high concept scores become prototypes, while global explanations correlate image areas with concepts.Applying GAP to decomposed feature maps supplies concept scores used to select prototypes.
Evaluation
The evaluation measures both fidelity to the original CNN and human interpretability, comparing matrix-factorization reducers and clustering-based concepts across computational and human-subject experiments. NMF concepts are more interpretable than PCA concepts and at least as interpretable as clustering concepts, while PCA provides the strongest fidelity and clustering the weakest.
- Fidelity for Approximate Models: PCA provided the best fidelity for classification and regression, while NMF was close and clustering performed worst.Fidelity was evaluated across different numbers of concepts using classification and regression measurements.
- Fidelity for Approximate Models: As the number of concepts increases, approximate models provide more faithful predictions for both classification and regression.The evaluation varied the concept count from 5 to 50 in steps of 5 on ResNet50 models for ImageNet and CUB.
- Fidelity for Approximate Models: NMF and PCA approximate models incur a 2-5% accuracy drop with 30+ concepts when replacing the target model’s final layers.The approximate model achieves 100% fidelity against itself, but explanations can still be incorrect relative to the original model.
- Interpretability via Human Subject Experiments: Human participants evaluated concept interpretability through prediction and free-text description tasks rather than predefined labels.Task prediction measured correctly identified explanations, while description similarity used average pairwise cosine similarity of GloVe representations.
- Interpretability via Human Subject Experiments: NMF-derived NCAVs were more interpretable than PCA-derived CAVs and at least as interpretable as clustering-derived CAVs.The prediction-task difference between NMF and PCA was significant at p < 0.001; NMF and clustering were not reported as significantly different in the supplied results.
Related Work
Research on CNN explanations has progressed from input-level importance maps to internal feature-map concepts, while matrix factorization offers another route to interpretable representations.
- Input-level methods use linear approximations, image segments, pixels, gradients, or class-activation heatmaps to explain CNN predictions.
- Concept-level methods instead build linear classifiers on internal feature maps, using learned weights as concept activation vectors.
- ACE reduces reliance on labeled data through unsupervised k-means clustering, while visualization methods explore alternative feature-map representations and presentation choices.
- NMF is introduced as an interpretable matrix-factorization method for gathering information from CNN feature maps.
Conclusion
The proposed ICE framework uses feature-map concept vectors to produce invertible explanations for CNN predictions. NCAVs from NMF provide better overall explanations than clustering and PCA when fidelity and interpretability are considered together.
- Conclusion: NCAVs from NMF provide overall better explanations than clustering and PCA when fidelity and interpretability are both required.PCA provides better fidelity but lacks interpretability.
- Conclusion: ICE gathers interpretable concept vectors from CNN feature maps and inverts them back to predictions.
- Conclusion: The framework is based on the post-training explanation method ACE and is designed for concept-based explanations of CNN models.
Ethics Statement
Interpretability can support accountability and transparency by helping analysts understand CNN decision processes. However, approximate explanations may mislead users, and prototype images can create privacy risks without appropriate safeguards.
- Ethics Statement: Interpretability can support accountability and transparency by helping people understand decisions informed by machine learning.
- Ethics Statement: Approximate explanations can be incorrect and may mislead people about the reasons for decisions.
- Ethics Statement: Prototype images can cause unintended privacy disclosure unless public, consented sources or privacy protections are used.