Source-linked AI summary
CRAFT: Concept Recursive Activation FacTorization for Explainability
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, Thomas Serre
TL;DR
Attribution methods often reveal where a model looks without explaining what it sees, motivating more useful concept-based explanations. CRAFT automatically discovers and recursively refines concepts, estimates their importance with Sobol indices, and produces concept attribution maps through implicit differentiation. It improves human-centered utility in two of three test scenarios, while some technical limitations remain in prior factorization approaches.
Problem
Attribution methods have limited practical value because they identify influential image regions without explaining what the model sees there.
Method
CRAFT combines automated NMF concept discovery, recursive cross-layer decomposition, Sobol-based concept importance estimation, and implicit-differentiation concept attribution maps.
Results
CRAFT achieves higher human-centered utility scores than tested attribution methods and ACE in the first two scenarios, while no method exceeds the baseline in the third.
Takeaways & Limitations
CRAFT provides concept-based explanations that connect what a model uses with where those concepts appear, and its identified concepts were useful and meaningful to human experimenters.
Takeaways & Limitations
Attribution explanations can be incorrect, and prior factorization approaches face non-negativity and decomposition-nonuniqueness issues.
Abstract
from arXiv · showhide
Attribution methods, which employ heatmaps to identify the most influential regions of an image that impact model decisions, have gained widespread popularity as a type of explainability method. However, recent research has exposed the limited practical value of these methods, attributed in part to their narrow focus on the most prominent regions of an image -- revealing "where" the model looks, but failing to elucidate "what" the model sees in those areas. In this work, we try to fill in this gap with CRAFT -- a novel approach to identify both "what" and "where" by generating concept-based explanations. We introduce 3 new ingredients to the automatic concept extraction literature: (i) a recursive strategy to detect and decompose concepts across layers, (ii) a novel method for a more faithful estimation of concept importance using Sobol indices, and (iii) the use of implicit differentiation to unlock Concept Attribution Maps. We conduct both human and computer vision experiments to demonstrate the benefits of the proposed approach. We show that the proposed concept importance estimation technique is more faithful to the model than previous methods. When evaluating the usefulness of the method for human experimenters on a human-centered utility benchmark, we find that our approach significantly improves on two of the three test scenarios. Our code is freely available at github.com/deel-ai/Craft.
1. Introduction
CRAFT addresses limitations of attribution-based explanations by combining automated concept discovery with explanations of both what a model uses and where it appears. Its contributions include recursive concept decomposition, Sobol-based importance estimation, and concept attribution maps via implicit differentiation.
- Motivation: Attribution methods indicate where a model focuses but provide limited information about what triggered its decision.Recent human-centered benchmarks have found that such methods can fail or provide only marginal help.
- CRAFT: CRAFT uses NMF to automatically extract high-level concepts learned by deep neural networks and validates their practical utility with human psychophysics experiments.
- CRAFT: CRAFT recursively identifies concepts and sub-concepts across layers to select an appropriate level of granularity.The procedure starts near the top of the model and works upstream.
- CRAFT: Human experiments found that recursive decomposition produces more coherent sub-concepts and more refined, meaningful groups of points.
- CRAFT: CRAFT estimates individual concept importance with Sobol indices and generates concept attribution maps by backpropagating concept scores into pixel space with implicit differentiation.
2. Related Work
Prior work includes pixel attribution methods and concept-based approaches, but each has limitations involving interpretability, supervision, layer selection, or spatial representation. These limitations motivate CRAFT’s automated concept discovery and broader explanation pipeline.
- Attribution methods: Attribution methods generate importance maps for input variables, but they show where evidence lies without identifying what triggered the decision.
- Attribution methods: Attribution explanations can suffer from confirmation bias and sometimes appear useful despite being incorrect.Human-centered experiments have therefore been used to evaluate their usefulness.
- Concept-based methods: Concept-based explanations can be more interpretable, but early methods require manually curated concept images or labeled concept datasets.
- Concept-based methods: ACE discovers concepts without additional labeling or human supervision, yet requires heuristics, user-selected layers, single-cluster segment assignments, and outlier rejection.
- Concept-based methods: Kernel-level matrix factorization can treat the same concept in different image locations as distinct localized concepts.For example, grass at the bottom and top of an image may be separated.
3. Overview of the method
CRAFT discovers concepts from neural-network activations using NMF, recursively refining unclear concepts at earlier layers. It estimates concept importance with Sobol indices and differentiates through NMF to produce concept attribution maps.
- 3.1. Concept activation factorization.: CRAFT factorizes activations from image crops into a concept bank W and coefficients U using non-negative matrix factorization.The crops are selected from images matching the model’s predicted class, and the activations are computed at an intermediate layer.
- 3.2. Ingredient 1: A pinch of recursivity: Recursive decomposition refines difficult concepts into sub-concepts by selecting highly activating images and reapplying factorization at an earlier layer.The selected images are the 10% with the highest activation for the concept, based on the 90th percentile threshold.
- 3.2. Ingredient 1: A pinch of recursivity: CRAFT addresses layer-selection problems because concepts can amalgamate differently across network depths.The recursive procedure builds concept hierarchies by examining earlier layers when concepts remain difficult to understand.
- 3.3. Ingredient 2: A dash of sensitivity analysis: CRAFT uses Sobol indices to estimate concept importance from model-output variation under perturbations of concept coefficients.Perturbed activations are reconstructed by applying random masks to U before propagating them through the model.
- 3.4. Ingredient 3: A smidgen of implicit differentiation: The NMF-based approach has limitations because the factorization is generally non-unique and unconstrained approaches can prevent successful differentiation.The cited discussion contrasts NMF with unconstrained low-rank factorization and notes the resulting non-uniqueness of U and W.
- 3.4. Ingredient 3: A smidgen of implicit differentiation: Implicit differentiation through the NMF block enables concept scores to be backpropagated into pixel space as concept attribution maps.This supports both white-box attribution methods and black-box methods through a non-negative least-squares formulation.
4. Experimental evaluation
CRAFT was evaluated through human-centered utility experiments, psychophysics studies of recursivity, and fidelity analyses of concept-importance estimates. It outperformed tested attribution methods and ACE in two utility scenarios, while Sobol-based rankings produced more faithful estimates than TCAV.
- 4.1. Utility Evaluation: CRAFT achieved higher utility scores than all tested attribution methods and ACE in the first two of three real-world scenarios.No method exceeded the baseline in the third scenario, indicating that additional work is required there.
- 4.2. Validation of Recursivity: The psychophysics experiments evaluated whether recursive decomposition produced coherent concepts and improved understanding for different participant groups.Understanding improved significantly for non-experts but not experts.
- 4.1. Utility Evaluation: The utility benchmark measured how well explanations helped users infer classification rules that transferred to unseen images.Users predicted model decisions on novel images, with scores averaged over three sessions and normalized by an explanation-free baseline.
- 4.2. Validation of Recursivity: Participants chose recursive sub-concept clusters more than 74% of the time in both expert and non-expert groups.Both groups were significantly more likely to choose the sub-concept cluster, with p < 0.001.
- 4.3. Fidelity analysis: Sobol-based concept rankings produced steeper deletion and insertion curves than TCAV rankings.The fidelity evaluation reconstructed activations using important concepts and measured output changes after concept-space perturbations.
5. Conclusion
CRAFT automatically extracts human-interpretable concepts and explains model decisions by representing both what the model saw and where it saw it. Its recursive discovery, Sobol importance estimation, and implicit-differentiation maps were supported by human and fidelity evaluations.
- 5. Conclusion: CRAFT automatically extracts human-interpretable concepts from deep networks for per-class and per-image explanations.The method aims to highlight both what the model saw and where it saw it.
- 5. Conclusion: Recursive concept discovery identifies a granularity at which individual concepts are understandable, while Sobol indices estimate which concepts influence a class decision more accurately.These are two of CRAFT’s three proposed ingredients.
- 5. Conclusion: Implicit differentiation backpropagates through NMF blocks to generate concept-wise attribution maps independently of the attribution method used.Human psychophysics experiments supported the validity and usefulness of the concepts for experimenters.
A.1. Limitations
CRAFT’s limitations concern interpretability, measurement, factorization, and data requirements. The authors also identify confirmation and selection bias as risks and suggest avenues for improvement.
- Concept-based explanations lack a mathematical metric for direct comparison across techniques.The authors state that this forces reliance on time-consuming and expensive human-subject experiments.
- The authors suggest developing a metric as a priority for comparing concept-based explainability techniques.
- Concept clusters may be difficult to label clearly, potentially inducing confirmation and selection bias.Feature visualization may help, while image captioning is suggested as a possible improvement.
- NMF factorization is NP-hard to compute exactly, so CRAFT uses a tractable ADMM approximation that may yield non-unique solutions.The authors report low variance across runs but note the absence of a formal uniqueness guarantee.
- Relevant concept banks require enough samples from the class under study, which may affect explanation quality in data-frugal applications.
B. More results of CRAFT
Additional results compare CRAFT’s automatically discovered concepts with ACE and illustrate ranked concepts and feature visualizations across image classes. The examples show distinct, interpretable concept groupings and synthetic visualizations linked to target concepts.
- Qualitative comparison with ACE: CRAFT discovers distinct concept groupings for Church, Garbage truck, and English Springer images compared with ACE.Examples include vault, cross, and stained glass; dumpster, truck door, and two-wheeler; and eyes, nose, and fluffy ears.
- Qualitative comparison with ACE: CRAFT and ACE order concepts by importance, but CRAFT estimates importance with Sobol indices while ACE uses TCAV.The comparison also highlights segmentation-induced baseline bias as a weakness of ACE.
- Most important concepts: Figure S2 ranks the four most important CRAFT concepts for six classes: Chain saw, English springer, Gas pump, Golf ball, French horn, and Garbage truck.
- Feature visualization: Feature visualizations present synthetic images alongside image segments that maximize the target CRAFT concept.The visualizations are shown for chainsaw, English springer, and golf concepts.
- Feature visualization: CRAFT feature visualization optimizes an image to correspond to a concept from the concept bank using a dot-cossim loss and regularization.The reported settings use Fourier preconditioning with decay rate 0.8 and Adam with lr = 10^-1.
C.1. Alternating Direction Method of Multipliers (ADMM) for NMF
CRAFT computes NMF concepts with an ADMM-based alternating optimization scheme and uses implicit differentiation to obtain concept Jacobians. Sobol indices quantify concept influence through normalized output variation under stochastic perturbations.
- NMF and ADMM: NMF decomposes positive feature vectors A ∈ R^n×p into positive low-rank matrices U(A) and W(A), with r << min(n, p).
- NMF and ADMM: ADMM separates objective optimization from constraint satisfaction by introducing auxiliary variables and alternately minimizing over the U and W variable triplets.The equality constraints ˜U = U and ˜W = W become equal in the limit at convergence.
- NMF and ADMM: The ADMM procedure guarantees a monotonic decrease of the objective function ∥A − ˜U_t ˜W_t^T∥.
- Implicit differentiation: KKT conditions define an optimality function whose root gives the optimal NMF tuple, enabling implicit differentiation of U with respect to A.The method avoids backpropagating through every NMF-solver iteration and requires dual variables.
- Concept attribution maps: CRAFT computes concept attribution maps by backpropagating concept gradients from feature activations to input pixels.The two-stage process extracts features with A = h_l(X), differentiates the NMF stage, then computes ∇_XU_i and stacks the gradients.
- Sobol indices: Sobol indices estimate concept importance by perturbing concept coefficients with independent stochastic masks and a perturbation operator.The perturbation uses inpainting with baseline value zero, and total indices capture higher-order feature interactions.
- Sobol indices: Sobol indices quantify the importance of concept subsets as normalized contributions to model-output deviation and sum to one.
E. Human experiments
The human-experiment section introduces participant enrollment and the general experimental design used in the studies.
- Participants were enrolled before undertaking the general experimental design used across the studies.
E.1. Utility evaluation
The utility evaluation tested whether explanations help participants predict a model’s decisions on unseen images, using separate training and test phases across three scenarios.
- Utility experiment: The evaluation covered Husky versus Wolf, Leaves, and Kit Fox versus Red Fox scenarios with 84, 87, and 79 filtered participants, respectively.The scenario totals were reported after screening and filtering.
- Utility experiment: Each participant experienced only one condition to avoid experimental confounds.The design assigned participants to a single condition.
- Utility experiment: Participants trained on five explained samples, then predicted the classifier’s outputs for seven new samples without explanations.The main experiment comprised three training sessions, each followed by a brief test.
- Utility experiment: The study used a three-stage screening process covering practice performance, instruction comprehension, and attention checks.A catch trial was included in each testing phase to identify inattentive participants.
- Utility experiment: Participants included machine-learning experts and laypeople, with experts recruited from the authors’ laboratory or company and laypeople reporting no machine-learning expertise.The participant groups were defined by professional background and subject-matter expertise.
F. Fidelity experiments
The fidelity experiments compared concept-importance rankings across extraction and dimensionality-reduction methods using insertion and deletion metrics, and tested CRAFT on randomized weights.
- Fidelity experiments: Sobol outperformed TCAV for NMF in both deletion and insertion curves when estimating concept importance.Lower deletion values and higher insertion values indicate better estimation in the reported figure.
- Fidelity experiments: The experiments compared concept rankings from TCAV scores and Sobol indices using insertion and deletion metrics.The comparisons were performed on selected ILSVRC2012 classes with a pretrained ResNet50V2.
- Fidelity experiments: The study compared PCA, ICA, and RCA concept vectors alongside the NMF-based extraction setting.These dimensionality-reduction algorithms were evaluated through insertion and deletion measurements.
- Fidelity experiments: On a ResNet50V2 with randomized weights, CRAFT extracted concepts that differed drastically from those of trained models.The randomized-model sanity check is reported as evidence that CRAFT passes the check.
- Fidelity experiments: For the randomized Chain saw model, the extracted concepts were mainly based on color histograms.Figure S9 reports the three most relevant concepts for that class.