Now You See Me (CME): Concept-based Model Extraction

1. Introduction

• DNN black-box behavior hinders adoption, especially in heavily regulated industries with high costs of error.Source: §1 · p. 1
• Feature-importance methods can be fragile to input or model-parameter perturbations and may not improve human understanding, trust, or error correction.They assign importance scores to features such as pixels, patches, or word vectors [4]; reported limitations include perturbation fragility and human-experiment findings [9].Source: §1 · p. 1
• CME uses concept-based extracted models to analyze DNNs, their learned concepts, and how those concepts inform predictions.The framework also aims to identify concept information that can improve predictive performance.Source: §1 · p. 1
• CME’s example separates pixel-to-concept processing from concept-to-class prediction, unlike a CNN’s direct pixel-to-class mapping.Source: Figure 1 · p. 2

2.1. Concept-based Explanations

• CME handles multi-valued concepts directly and combines latent information from multiple layers rather than requiring binarization or a single-layer choice.The paper notes that binarization can enlarge the concept space and fail to enforce mutually exclusive values; single-layer selection trades off low- and high-level concepts.Source: §2.1 · p. 2
• Existing concept explanations inspect learned concepts [13], provide class-specific accounts [14], or seek causal relations [15]; CME describes functional relations between concepts and outputs.Source: §2.1 · p. 2

2.2. Concept Bottleneck Models

• Unlike concept bottleneck models, CME analyzes existing DNNs, is described as computationally cheaper, can use partially labelled data, and can help learn which concepts a task requires.CBM approaches require concept annotations for all training data and concepts to be specified beforehand.Source: §2.2 · p. 2

2.3. Model Extraction

• Conventional extracted models use the original input representation, which can be difficult to interpret directly; CME represents decisions through human-understandable concepts.Extracted models can preserve statistical properties of the original model when their approximation fidelity is sufficiently high.Source: §2.3 · p. 3

3.1. Formulation

• The formulation assumes a concept representation and a mapping from each input to its ground-truth concept values.A concept representation defines distinct concepts and their possible values.Source: §3.1 · p. 3

3.2. CME

• CME represents an extracted model as a composition of an input-to-concept function and a concept-to-output function that approximates the original DNN.The input-to-concept function maps inputs to concept representations; the concept-to-output function maps those representations to output labels.Source: §3.2 · p. 3

3.3. Input-to-Concept (𝑝̂)

• For input-to-concept extraction, CME assumes access to DNN training data and labels plus concept labels for a small subset, then aggregates concept predictions across layers.The remaining training points are concept-unlabelled, forming a semi-supervised setting.Source: §3.3 · p. 3
• CME learns concept predictors from each layer’s hidden representations using semi-supervised multi-task learning, treating each concept as an independent task.For each concept, it selects the layer with the lowest predictive loss and uses that layer’s prediction for the concept representation.Source: §3.3 · pp. 3–4

3.4. Concept-to-Label (𝑞̂)

• The concept-to-label model is trained to predict the DNN’s output labels from concept labels predicted by the input-to-concept model.The extracted model can then be used to analyze how the DNN uses concept information when making predictions.Source: §3.4 · p. 4

4. Experimental Setup

• The study evaluates CME on dSprites and CUB, where only dSprites explicitly links the available concepts to the output task.Consequently, ground-truth concept information learned by the CUB model is unavailable.Source: §4 · p. 4, §4.2.3 · p. 5

4.1.1. Classification Tasks

• dSprites contains procedurally generated 64×64 black-and-white shapes spanning six independent concepts, and supports tasks predicting shape or shape-scale combinations.The two tasks test recognition of one concept and combinations of concepts, respectively.Source: §4.1 · p. 4, §4.1.1 · p. 4

4.1.2. Model

• 100.0 ± 0.0% accuracy was achieved by each dSprites CNN on its held-out task test set.Both models shared a three-convolutional-layer, two-dense-layer architecture with ReLUs, 50% dropout, and a softmax output.Source: §4.1.2 · p. 4

4.1.3. Ground-truth Concept Information

• For dSprites, the task definitions identify the ground-truth concepts as shape for Task 1 and shape and scale for Task 2.These are the concepts the models had to learn to achieve their task performances.Source: §4.1.3 · p. 4

4.2. Caltech-UCSD Birds (CUB)

• CUB has 11,788 images across 200 bird species; preprocessing refines its 312 binary annotations to 112 concepts, and the task predicts species.The concepts include attributes such as beak and wing color, shape, and pattern.Source: §4.2 · p. 4, §4.2.1 · p. 5

4.2.2. Model

• 82.7 ± 0.4% held-out test accuracy was achieved by the CUB model, an ImageNet-pretrained Inception-v3 fine-tuned end-to-end on CUB.The fully connected layers were excluded from pretraining.Source: §4.2.2 · p. 5

4.3.1. Net2Vec

• Net2Vec predicts binary concept presence from spatially averaged convolutional activations; its best-performing layers were selected for each task.For dSprites these were the third convolutional layer; for CUB, the final Inception block output.Source: §4.3.1 · p. 5

4.3.2. CBM

• The CUB concept bottleneck model benchmarks learnable concepts and task performance, but solves the classification task rather than approximating or analysing the CUB model.Its first submodel predicts concepts from images, its second predicts task labels from predicted concepts, and the two submodels and full model serve as p̂, q̂, and f̂ benchmarks.Source: §4.3.2 · p. 5, §4.3 · p. 5

5.1.1. dSprites

• On both dSprites tasks, CME predicted relevant concepts accurately and irrelevant concepts poorly, while Net2Vec performed much worse on relevant concepts.Figure 2 reports predictive accuracy for all concepts, averaged over five runs.Source: §5.1.1 · p. 6, Figure 2 · p. 6

5.1.2. CUB

• CUB concept F1 scores were 92 ± 0.5% for CBM, 86.3 ± 2.0% for CME, and 85.9 ± 2.3% for Net2Vec; MPO rated CME near Net2Vec and below CBM.All concepts were conservatively treated as relevant because their relevance to CUB labels was not specified.Source: §5.1.2 · pp. 6–7, Figure 3 · p. 7
• MPO measures the fraction of test samples with at least m relevant concepts mispredicted; CME’s MPO of 0.25 at m=4 means 25% had at least four errors.The figure compares MPO across m with an ideal-prediction curve; Net2Vec’s values were within 1% of CME’s for every m.Source: §5.1.2 · pp. 6–7, Figure 3 · p. 7

5.2. Task Performance

• For dSprites, CME extracted models achieved 99%+ fidelity and task performance; CUB models had lower scores, with CME and Net2Vec similar and CBM also low.The authors attribute CUB’s results to concept information learnable from the data being insufficient for high accuracy, so the original model relies on other information.Source: §5.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7

5.3. Intervening

• Using 32 important CUB concepts, intervention raised task performance by up to 14%, and the authors link undesired behavior more likely to data properties than model properties.The 32 concepts alone supported over 96% task accuracy with logistic regression; intervention set top-ranked concept predictions to ground truth values.Source: §5.3 · p. 8, Figure 4 · p. 8

5.4.1. Input-to-Concept (𝑝̂)

• Latent-space plots can reveal how concept values form manifolds, helping select suitable concept-prediction functions.Distinct clusters of concept values can indicate that a latent space is clustered with respect to those values.Source: §5.4.1 · pp. 8–9
• Concept representations vary across layers; CME can select different layers for different concepts as higher-layer manifolds become more unimodal.Figure 5 shows concept values across layers and marks CME’s selected layer for each concept.Source: Figure 5 · p. 8, §5.4.1 · p. 9

5.4.2. Concept-to-Output (𝑞̂)

• Inspecting 𝑞̂ functions exposes how concept information informs predictions, supporting checks against user expectations and model verification.The paper also identifies model debugging and knowledge extraction as uses of 𝑞̂ inspection.Source: §5.4.2 · p. 9

6. Conclusions

• CME uses concept-based extracted models to analyze learned concepts, how DNNs use concepts in prediction, and concepts that can improve predictive performance.The framework is described as model-agnostic and general-purpose.Source: §6 · p. 9
• CME assumes a fixed set of concept labels is available before model extraction begins.The authors propose exploring active learning to obtain informative concept labels interactively and reduce manual labeling effort.Source: §6 · p. 9
• The authors believe CME can provide granular concept-based analyses of DNN models.Source: §6 · p. 9

A. Concept Decomposition

• Concept decomposition explains model behavior by extracting concepts from high-dimensional inputs and then using them to predict labels.This bottom-up account uses human-understandable concepts and their interactions, paralleling how people may reason about tasks such as identifying bird species.Source: A. Concept Decomposition · p. 11
• Concept decomposition can in principle apply to any machine-learning model and task, although this work focuses on CNNs and tasks.Source: A. Concept Decomposition · pp. 11–12

A.1. CBMs

• CBMs are a special case that explicitly separates concept extraction from producing task labels.Non-CBM models can also show concept decomposition; the dSprites Task 2 model extracted relevant concepts in dense layers and used them for classification.Source: §A.1 · p. 12

A.2. CBMs & CME

• CBMs are more likely to rely on desired concepts during decisions and align with a user’s mental model of the task.Source: §A.2 · p. 12
• CME extracts concepts from pretrained DNNs using L*k concept predictors that can be simpler models trained on a fraction of the DNN training data.The authors state that training these predictors costs significantly less than training a bottleneck model on all training data, as in CBMs.Source: §A.2 · p. 12
• CME can use existing DNNs to discover relevant concepts and obtain concept annotations in a semi-supervised fashion, making both faster and cheaper.The passage contrasts this with the cost of producing CBM annotations and the incomplete availability of knowledge about relevant or sufficient concepts.Source: §A.2 · p. 12
• CME supports interaction with pretrained DNNs to refine concept-related knowledge when that information is expensive, difficult to obtain, or only partially known.Source: §A.2 · p. 12
• A CBM can be approximated with CME by using its concept-bottleneck output for p̂ and its task-label submodel for q̂.Source: §A.2 · p. 12

A.3. Further Discussion

• CME probes whether a DNN is concept-decomposable by approximating it with a model designed as concept extraction followed by task-label prediction.The premise is that concepts a DNN learns and uses for prediction will be present in its latent space.Source: §A.3 · p. 12
• Existing concept-decomposition approaches require concepts to be both sufficient for predicting task labels and learnable from the data.Source: §A.3 · p. 12
• The authors identify discovering sufficient concepts and checking their learnability as open directions for cases where those assumptions fail.They note that existing approaches do not discuss how to handle such cases, including the CUB task.Source: §A.3 · p. 12

B.1. Description

• Its 64×64 black-and-white images are generated from six independent ground-truth concepts: color, shape, scale, rotation, and x and y position.Source: §B.1 · pp. 12–13
• dSprites contains 737,280 images generated from all combinations of color, shape, scale, rotation, and x and y position.Source: §B.1 · pp. 12–13

B.2. Pre-processing

• Preprocessing retains every other x and y position value and every fifth rotation value, reducing the dataset to 36,864 samples while preserving concept value ranges and diversity.Source: §B.2 · p. 13

• The dSprites Task 1 visualization projects hidden spaces across layers and indicates which layers CME uses to predict the shape concept.Source: C. Input-to-Concept Functions · p. 13
• The CUB visualization covers six model layers and the five most important concepts identified in Section 5.3.Source: C. Input-to-Concept Functions · p. 13
• CUB concept values separate significantly better after Mixed_7c, but some remain mixed in later layers, indicating likely suboptimal concept extraction.Source: C. Input-to-Concept Functions · p. 13

• The extracted dSprites Task 2 model correctly differentiates output classes based on shape and scale concept values.Source: D. Concept-to-Output Functions · p. 13, Figure 9 · p. 15
• The Task 2 output comprises 18 classes from three shape and six scale concept values.Source: D. Concept-to-Output Functions · p. 13
• Figure 9 visualizes the decision tree extracted from the dSprites Task 2 model.Source: D. Concept-to-Output Functions · p. 13, Figure 9 · p. 15