A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation

1 Introduction

• Attribution heatmaps locate important input features, while meaningful explanations also need to convey their semantic content.The cited user-study evidence is presented as showing limitations of attribution methods for meaningful explanations [16] [17] [18] [19] [21] [22] [23] [24] [25].Source: §1 · p. 1
• Concept-based methods seek high-level, human-interpretable concepts, but their early-stage development and informal concept definitions have hindered appropriate comparison metrics.Prior desiderata include meaningfulness, coherence, and importance.Source: §1 · p. 2
• The framework treats concept extraction as dictionary learning and concept importance scoring as attribution in concept space.It unifies the two steps shared by concept-based explainability methods.Source: §1 · p. 2
• The paper proposes evaluation metrics for extracted concepts, seven concept-importance methods with optimality guarantees, and adapted faithfulness benchmarks.It reports that Integrated Gradients, Gradient Input, RISE, and Occlusion achieve the highest theoretical scores for three faithfulness metrics with penultimate-layer decomposition.Source: §1 · p. 2
• Strategic cluster graphs represent the main visual strategies a model uses across a class, including espresso’s coffee-in-cup strategy as predominant.The figure also lists strategies for zucchini, including garden, flower, sliced, spotted-skin, and stacked examples.Source: Figure 1 · p. 2, §1 · p. 2

2 Related Work

• Kim et al. [26] define concepts as Concept Activation Vectors (CAVs) learned from concept examples and random counterexamples, then use TCAV directional derivatives to score category-level contributions.The approach requires substantial human effort to build a relevant concept-image database.Source: §2 · p. 3
• ACE [27] automates CAV extraction using centroids of activations from multi-scale image segments, but segments may bias explanations [34] [35].ACE uses TCAV to rank concepts by importance for an object category.Source: §2 · p. 3
• ICE [28] applies NMF to feature maps for localized concepts and uses TCAV scores, whereas CRAFT [29] applies NMF after global pooling and uses Sobol indices for global importance.SVD was also proposed for concept discovery [28], and explores entire subspaces rather than one-dimensional concepts.Source: §2 · p. 3

3 A Unifying perspective

• Concept-based methods extract concepts from class activations and assign global importance scores; the framework casts these steps as dictionary learning and attribution in concept space.The formulation is intended to clarify the methods and enable theoretical analysis and evaluation with metrics from other fields.Source: §3 · p. 3

3.1 Concept Extraction

• Dictionary learning represents activations as A ≈ UVT, with V as the concept basis and U as activation coordinates; methods impose different constraints on these matrices.ACE uses K-Means, PCA constrains V to be orthogonal, and NMF constrains U and V to be nonnegative.Source: §3.1 · p. 4
• PCA leads reconstruction, K-Means sparsity and OOD scores, while NMF balances tested metrics; K-Means and NMF are most stable, whereas PCA is highly unstable.K-Means’ centroid reconstructions stay near in-distribution instances but collapse the distribution; PCA can reconstruct negative, out-of-distribution activations.Source: §3.1 · pp. 5–6
• Across ResNet50 blocks B2–B5, concept extraction metrics favor the last layer.The figure compares PCA, NMF, and K-Means across layers; the authors report that the trend favors the last layer for decomposition.Source: §3.1 · p. 6, Figure 3 · p. 6

3.2 Concept importance

• Concept Attribution methods generalize attribution scores to concept space, assigning importance by how changes in concept coordinates affect the model output.The authors derive seven new CATs from existing attribution methods and adapt C-Deletion, C-Insertion, and C-µFidelity to assess their faithfulness.Source: §3.2 · p. 7
• In the penultimate layer, Gradient Input, Integrated Gradients, Occlusion, and RISE are optimal for C-Deletion, C-Insertion, and C-µFidelity.The authors’ guarantees rely on the penultimate-layer setting.Source: §3.2 · p. 8

3.1 Concept Extraction

• NMF, especially on the penultimate layer, offers the paper’s most appealing trade-off across tested extraction metrics compared with PCA and K-Means.The authors adopt NMF for subsequent concept extraction.Source: §3.1 · p. 6, §3.2 · p. 8

3.3 Unveiling main strategies

• The strategic cluster graph maps samples by concept-importance vectors and colors them by their top concept, revealing shared classification strategies and examples of misclassification.For lemons, peel texture is the most prevalent concept in 90% of samples; the “pile of round, yellow objects” concept has 40% mean accuracy, and shared peel texture or wedge shape explains cited errors.Source: §3.3 · p. 9

• The framework connects local concept importance to class-level strategies, which the strategic cluster graph visualizes across samples and uses to inspect failure cases.The paper describes the graph as revealing strategy prevalence and reliability.Source: §4 · pp. 9–10
• In the lemon example, peel texture is the predominant strategy among four concepts, and the graph links misclassified images to their corresponding strategies.The other concepts are lemon wedge shape, piles of round yellow objects, and green objects hanging on a tree.Source: Figure 6 · p. 10
• The authors argue that dictionary learning has potential benefits for automatic concept extraction and hope the work prompts further XAI advances.Source: §4 · p. 10

• Each attribution method maps concept coefficients u to per-concept importance scores φ(u), with higher scores marking more important concepts.The model signal g can be logits or cosine similarity with a sentence.Source: A Attribution methods for Concepts · p. 15
• Saliency [4] uses concept-space gradients, Gradient ⊙Input [6] multiplies gradients by the input, and Integrated Gradients [7] accumulates gradients from a zero baseline.Integrated Gradients uses 30 regularly spaced steps and a trapezoidal approximation.Source: A Attribution methods for Concepts · p. 15
• SmoothGrad [5] averages gradients at noisy concept inputs to reduce visual noise, whereas VarGrad aggregates the same noisy-input gradients using variance.Both use 30 samples with σ = 0.1 in the experiments.Source: A Attribution methods for Concepts · p. 15
• Occlusion estimates concept importance by omitting concepts one at a time and measuring the resulting prediction changes against a zero baseline.Source: A Attribution methods for Concepts · p. 15
• Sobol Attribution [9] scores a concept by the expected output variance remaining when other variables are fixed; experiments use 32 designs and the Jansen estimator.Source: A Attribution methods for Concepts · pp. 15–16
• HSIC [10] measures output dependence on concept presence across binary masks, while RISE [32] probes masked inputs to identify important features.For HSIC, the dependence calculation uses kernels on masks and outputs.Source: A Attribution methods for Concepts · p. 16

B Closed-form of Attributions for the last layer

• Saliency and Gradient-Input are derived first, and Integrated Gradients takes the Gradient-Input form while SmoothGrad takes the Saliency form.Source: B Closed-form of Attributions for the… · p. 16
• SmoothGrad computes the expectation of the gradient of g at u + δ, while VarGrad computes its variance, with δ sampled from N(0, Iσ) in both definitions.Source: B Closed-form of Attributions for the… · pp. 16–17
• In a last-layer linear system, VarGrad has zero attribution because the gradient is constant and therefore has zero variance.Source: B Closed-form of Attributions for the… · p. 16
• Occlusion and RISE use the Gradient-Input form, with RISE additionally involving multiplicative and additive constants.Source: B Closed-form of Attributions for the… · p. 17

C µ Fidelity optimality

• µFidelity evaluates the fidelity of concept importance, using Pearson correlation and comparisons that set selected coordinates of u to zero.Source: C µ Fidelity optimality · p. 17
• Gradient Input, Integrated Gradients, Occlusion, and RISE are optimal for µFidelity under a last-layer decomposition.Source: C µ Fidelity optimality · p. 17
• Any importance method of the stated form φ_i(u) = a(u_iV_i^TW) + b, with a ≥ 0, is also optimal and has correlation 1.Source: C µ Fidelity optimality · p. 17

D Optimality for C-Insertion and C-Deletion

• C-Insertion is a greedy selection process: at each step it adds the highest-scoring not-yet-selected concept under a cardinality constraint.Source: D Optimality for C-Insertion and… · p. 18
• The greedy algorithm sorts concepts by weight and returns a maximum-weight basis of the weighted matroid.Source: D Optimality for C-Insertion and… · p. 19
• The stated optimality result is restricted to concepts in the penultimate layer, where the decomposition is linear.Source: D Optimality for C-Insertion and… · p. 19
• Gradient Input, Integrated Gradients, Occlusion, and RISE are optimal for both C-Insertion and C-Deletion in the last layer.Source: D Optimality for C-Insertion and… · p. 19
• For C-Insertion, the weighting function is based on the model score, a linear combination of concepts; the proof relies on linearity.Source: D Optimality for C-Insertion and… · p. 19

E Sparse Autoencoder

• A neural network can produce U from activations A so that A is reconstructed linearly as UV^T, encompassing PCA and K-means.Source: E Sparse Autoencoder · p. 19
• NMF and K-means map A to U nonlinearly while reconstructing A linearly from U; the authors suggest this nonlinearity may aid effective concept extraction.Source: E Sparse Autoencoder · p. 20
• The proposed architecture uses dense layers, batch normalization, and ReLU to obtain positive scores and sparsity similar to NMF, without constraining V.Source: E Sparse Autoencoder · p. 20
• The dictionary V is initialized with truncated SVD and optimized with Adam at a 1e−3 learning rate; dictionary learning remains an area for exploration.Source: E Sparse Autoencoder · p. 20