Promises and Pitfalls of Black-Box Concept Learning Models

Motivation

• Concept Learning Models first map raw inputs to concepts and then map those concepts to predictions, aiming to retain interpretable decisions while matching black-box performance.The paper frames concepts such as increased tremors as human-interpretable intermediates for decisions such as Parkinson’s diagnosis.Source: §1 · p. 1

Information leakage

• Concept representations can encode information beyond their named concepts, making it difficult to isolate an individual concept’s influence on the downstream prediction.This problem has been demonstrated in Concept Bottleneck Models and can render their interpretations unreliable.Source: §1 · p. 1

Contribution

• Information leakage is more pervasive and consequential than prior literature indicated, including in CLMs trained with natural mitigation strategies for soft concept representations.Here, a soft concept is represented by a real-valued intermediate node.Source: §1 · p. 1
• The paper argues that existing approaches do not fully address leakage and suggests strategies to mitigate its effects in Concept Learning Models.The proposed recourse targets the effect of information leakage rather than treating concept representations as fully faithful by default.Source: §1 · p. 2

The Concept Bottleneck Model

• A Concept Bottleneck Model composes g: X → C, which maps inputs to concepts, with h: C → Y, which maps concepts to labels.The two neural-network functions can be trained independently, sequentially, or jointly.Source: §2 · p. 2
• Concept Bottleneck Models use k predefined concepts represented as vectors c_i ∈ R^k alongside inputs x_i ∈ R^d and downstream labels y_i ∈ R.The training data is given as {(x_i, y_i, c_i)}^n_i=1.Source: §2 · p. 2

Concept Whitening Model

• Concept Whitening divides a network f: X → Y into a feature extractor g: X → Z and a classifier h: Z → Y, with selected latent dimensions aligned to predefined concepts.The feature extractor is the Concept Learning Model component, even though the full model is trained for the downstream task.Source: §2 · p. 2

3. Extraneous Information Leakage in Soft Concept Representations

• Soft concept representations encode data-distribution information beyond predefined concepts, so downstream interpretations based on them can be flawed.Concept classification probabilities reflect distance from decision surfaces and thereby encode data distribution along perpendicular directions.Source: §3 · p. 2
• Sequentially trained CLMs still leak information: concept representations encode more than their concepts even when those concepts are irrelevant to the downstream task.In the parity experiment, the concepts “4” and “5” are absent from the data, yet the learned CLM outputs support substantially better-than-random prediction.Source: §3 · p. 2
• 86% accuracy is obtained from the top PCA component alone in the MNIST example, showing that leaked distributional information can support task prediction.The soft activations for concepts “4” and “5” preserve the first PCA dimension even though the concepts are task-independent.Source: §3 · p. 3
• The extent of leakage and the advantage of soft over true concept representations depends on hyperparameters, architecture, and training choices.This sensitivity means the severity of interpretability failure varies with modeling decisions.Source: §3 · p. 3
• AUC 0.999 is achieved by a jointly trained model with insufficient capacity for a relevant concept, versus 0.875 using ground-truth hard concept labels.The result indicates that soft representations of supervised concepts encode additional task-relevant information.Source: §3 · p. 4
• Adding latent dimensions does not prevent leakage: aligned bottleneck dimensions predict missing concepts, and learned feature importance can differ from the ground-truth concept-to-task model.For M2, dimensions aligned to x+ and y+ each predict z+ with AUC approximately 0.75; entanglement also makes x+ appear more important than it is.Source: §3 · pp. 3–4
• Concept whitening also fails to eliminate leakage: any latent dimension can predict any of three concepts, despite decorrelation and other purity criteria.Decorrelated representations can retain high mutual information, so downstream interpretations still fail to recover ground-truth feature importance.Source: §3 · p. 4

4. Avoiding the Pit-Falls of CLMs

• Hard one-hot concepts do not guarantee safety: random hard concepts can still capture data-distribution information on low-dimensional data manifolds.Thus leakage may remain an issue for black-box CLMs regardless of whether representations are soft or hard.Source: §4 · p. 5
• A larger soft-versus-hard prediction gap may indicate that the concept set is irrelevant to the task and should be replaced.The paper presents the absent-“4”/“5” parity setting as an example motivating new concept selection.Source: §4 · p. 5
• Concept whitening only enforces uncorrelated dimensions, so training should explicitly minimize mutual information between concept dimensions when independence is expected.The recommendation applies to both aligned and unaligned dimensions.Source: §4 · p. 5
• Even statistically independent latent concepts can be multiply encoded, motivating domain-expert supervision to improve transparency about input–concept relationships.Independence alone does not ensure that each learned dimension corresponds to only one concept.Source: §4 · p. 5

• Soft concept representations can encode undesirable additional information that negatively impacts the interpretability of downstream predictive models.Source: §5 · p. 5

• A Concept Bottleneck Model consists of g mapping inputs to concepts and h mapping concepts to labels.Source: A. Training of Concept Bottleneck Models · p. 6
• Independent training minimizes separate concept and label losses for g and h.Source: A. Training of Concept Bottleneck Models · p. 6
• Sequential training first minimizes g's concept loss, then minimizes h's label loss using g's outputs.Source: A. Training of Concept Bottleneck Models · p. 6
• Joint training minimizes label loss plus λ-weighted concept loss, with λ controlling the tradeoff between learning concepts and labels.Source: A. Training of Concept Bottleneck Models · p. 6

• The synthetic dataset contains two features, three mutually exclusive concepts, and a task boundary that cuts across concepts.Source: Figure 2 · p. 6, Representations Encode Data… · p. 6
• The experiment predicts task labels from predicted concepts using an MLP with three hidden layers of 32 ReLU units and a 0.5 classification threshold.Source: Representations Encode Data… · p. 6
• Raw concept labels predict task labels with 74.5% accuracy, while the feature-to-concept model reaches 87.0% concept-prediction accuracy.Source: Representations Encode Data… · pp. 6–7
• Because mutually exclusive concepts yield only two independent vectors, the three-dimensional activations can be represented in two PCA dimensions without information loss.Source: Figure 4 · p. 7, Representations Encode Data… · p. 7
• 99.3% accuracy is achieved by predicting task labels directly from features, compared with 95.9% from predicted concepts and 89.5% with sigmoid concept activations.Source: Figure 6 · p. 7, Figure 5 · p. 7, Figure 7 · p. 8
• The concept model largely preserves feature geometry and the task boundary, while sigmoid activations transform the boundary more severely and make optimization harder.Source: Representations Encode Data… · pp. 7–8

• The experiment sequentially trains a concept predictor and a downstream predictor, then evaluates their composition across increasing numbers of concepts over 10 runs.Hyperparameters are tuned to convergence for each concept count, with hard representations evaluated as a comparison.Source: C. Demonstration 2: Sufficiently Large · p. 8
• Random concepts are generated from hyperplanes defined by random coefficients and thresholds sampled from the observed dot-product range.Each hyperplane labels observations on opposite sides as true or false examples.Source: C. Demonstration 2: Sufficiently Large · p. 8
• 99% direct-pixel accuracy is approached as random concepts increase, showing that most feature information passes through the concept modeling process.The comparison uses a direct model of similar architecture on MNIST digit parity.Source: C. Demonstration 2: Sufficiently Large · p. 8
• Soft random concept representations outperform hard representations, and their downstream performance continues increasing without the same diminishing pattern.Hard representations also become increasingly predictive as more random concepts are added.Source: C. Demonstration 2: Sufficiently Large · p. 8

D. Demonstration 3: Representations Entangling Concepts

• The toy task defines concepts x+, y+, and z+ as coordinate signs, while the downstream label is positive when at least two concepts are positive.The dataset uses transformed coordinate features and separate training and test sets.Source: D. Demonstration 3: Representations… · pp. 8–9, Figure 9 · p. 9
• Soft random concepts become increasingly predictive of MNIST digit parity and approach the 99% accuracy of a direct pixel model as their number grows.Hard representations also gain predictiveness, but their performance begins to diminish as more concepts are added.Source: Figure 8 · p. 9
• Greater than 0.98 downstream AUC and greater than 90% accuracy are achieved with an incomplete two-concept bottleneck for all λ less than or equal to 0.1.A model using only the original two concepts obtains AUC 0.875 and accuracy 75%, while the missing concept limits its maximum expected accuracy to about 83%.Source: D. Demonstration 3: Representations… · p. 9
• A complete-concept CBM is evaluated alongside incomplete two- and three-node bottlenecks using AUC scores between bottleneck dimensions and the ground-truth concepts.The supplied tables describe alignments for the complete and incomplete concept sets at λ = 0.1.Source: D. Demonstration 3: Representations… · p. 10, Table 1 · p. 10, Table 2 · p. 10

E. Demonstration 4: Information Leakage in Concept Whitening Models

• The MNIST experiment classifies digits below 4 using concepts for digits 1, 2, and 3 on a balanced subset containing digits 1 through 6.The standard CNN uses convolution, batch normalization, pooling, global average pooling, and a linear prediction layer.Source: E. Demonstration 4: Information… · p. 10
• Correlation reduction occurs early in training, but high AUC scores for both aligned concepts are difficult to achieve consistently.A reported trial exceeded 0.9 AUC for both concepts, yet the authors argue that high AUC alone cannot establish concept purity.Source: E. Demonstration 4: Information… · pp. 10, 12, Figure 11 · p. 12
• The authors evaluate concept purity by comparing bottleneck outputs with ground-truth concepts and by inspecting correlations and cosine similarities in standard CNN and Concept Whitening representations.Figure 10 also varies λ to compare downstream-task and concept-prediction AUCs, with λ = 0.1 generally providing the best trade-off.Source: Figure 10 · p. 11, Figure 11 · p. 12, Figure 12 · p. 12
• AUC-based purity can appear strong because it uses one activation value, while the model can exploit the full activation map and retain information about other concepts.Thus, concept predictions based on the representation may contain leakage that the scalar AUC check does not reveal.Source: E. Demonstration 4: Information… · p. 12
• Purity-check networks predict all three concepts from any aligned or non-aligned Concept Whitening latent dimension with over 95% accuracy.This occurs despite high AUC scores for the aligned concepts and despite satisfying the reported purity checks from Chen et al. (2020).Source: E. Demonstration 4: Information… · pp. 10, 12

F. Demonstration 5: Concept Refinement

• Approximately 50% of grapefruits and apples sell, so the broad concepts “is grapefruit” and “is apple” are not predictive of sales on their own.The synthetic task models sales using fruit weight and acidity.Source: F. Demonstration 5: Concept Refinement · pp. 12–13, Figure 13 · p. 13
• Replacing them with “is acidic grapefruit” and “is small apple” makes the concept set perfectly predictive in the synthetic example.These sub-concepts reflect the task’s acidity and size decision boundaries.Source: F. Demonstration 5: Concept Refinement · p. 13
• Refining concepts can reduce weight and acidity to binary regions such as “is heavy” and “is acidic” while retaining predictive structure.The passage frames this as reducing the input space to a small number of predictive regions.Source: F. Demonstration 5: Concept Refinement · p. 13