Promises and Pitfalls of Black-Box Concept Learning Models

Anita Mahinpei; Justin Clark; Isaac Lage; Finale Doshi-Velez; Weiwei Pan

Source-linked AI summary

Promises and Pitfalls of Black-Box Concept Learning Models

Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, Weiwei Pan

arXiv:2106.13314v1Jun 24, 2021cs.LGcs.AI

TL;DR

Concept-learning models are intended to make predictions interpretable through predefined human concepts, but their representations may encode additional information. The paper analyzes how this leakage arises, shows that common mitigation strategies remain insufficient, and proposes ways to mitigate its interpretive effects.

Problem
Concept-learning models can encode information beyond predefined concepts, making downstream predictions misleadingly interpretable.
Method
The paper analyzes soft concept representations, tests sequential training and other leakage-mitigation strategies, and examines the mechanisms producing extra information.
Results
The paper finds that information leakage persists across soft concept representations and mitigation strategies, including settings without joint training; one model achieved AUC 0.999 versus 0.875 using ground-truth hard labels.
Takeaways & Limitations
Comparing soft and hard concept utilities can reveal that a concept set is irrelevant or needs modification, while domain-expert supervision can improve transparency.
Takeaways & Limitations
Information leakage may remain even with hard representations, because semantically meaningless random hard concepts can capture information about the data distribution.

Abstract

from arXiv · show

Machine learning models that incorporate concept learning as an intermediate step in their decision making process can match the performance of black-box predictive models while retaining the ability to explain outcomes in human understandable terms. However, we demonstrate that the concept representations learned by these models encode information beyond the pre-defined concepts, and that natural mitigation strategies do not fully work, rendering the interpretation of the downstream prediction misleading. We describe the mechanism underlying the information leakage and suggest recourse for mitigating its effects.

1. Introduction

Concept Learning Models map raw inputs to human-interpretable concepts before predicting outcomes, but their concept representations can encode additional information that makes downstream interpretations unreliable. This paper shows that information leakage is pervasive, persists under natural mitigation strategies for soft concepts, and motivates strategies to reduce its effects.

Motivation: Concept Learning Models first map raw inputs to concepts and then map those concepts to predictions, aiming to retain interpretable decisions while matching black-box performance.The paper frames concepts such as increased tremors as human-interpretable intermediates for decisions such as Parkinson’s diagnosis.
Information leakage: Concept representations can encode information beyond their named concepts, making it difficult to isolate an individual concept’s influence on the downstream prediction.This problem has been demonstrated in Concept Bottleneck Models and can render their interpretations unreliable.
Contribution: Information leakage is more pervasive and consequential than prior literature indicated, including in CLMs trained with natural mitigation strategies for soft concept representations.Here, a soft concept is represented by a real-valued intermediate node.
Contribution: The paper argues that existing approaches do not fully address leakage and suggests strategies to mitigate its effects in Concept Learning Models.The proposed recourse targets the effect of information leakage rather than treating concept representations as fully faithful by default.

2. Background

Concept Bottleneck Models compose an input-to-concept mapping with a concept-to-label mapping, while Concept Whitening aligns selected latent dimensions with predefined concepts inside a feature-extractor/classifier network.

The Concept Bottleneck Model: A Concept Bottleneck Model composes g: X → C, which maps inputs to concepts, with h: C → Y, which maps concepts to labels.The two neural-network functions can be trained independently, sequentially, or jointly.
The Concept Bottleneck Model: Concept Bottleneck Models use k predefined concepts represented as vectors c_i ∈ R^k alongside inputs x_i ∈ R^d and downstream labels y_i ∈ R.The training data is given as {(x_i, y_i, c_i)}^n_i=1.
Concept Whitening Model: Concept Whitening divides a network f: X → Y into a feature extractor g: X → Z and a classifier h: Z → Y, with selected latent dimensions aligned to predefined concepts.The feature extractor is the Concept Learning Model component, even though the full model is trained for the downstream task.

3. Extraneous Information Leakage in Soft Concept Representations

Soft concept representations encode data-distribution information beyond their named concepts, even under task-blind or mitigation-oriented training. This leakage improves downstream prediction but can make concept-based interpretations misleading.

3. Extraneous Information Leakage in Soft Concept Representations: Soft concept representations encode data-distribution information beyond predefined concepts, so downstream interpretations based on them can be flawed.Concept classification probabilities reflect distance from decision surfaces and thereby encode data distribution along perpendicular directions.
3. Extraneous Information Leakage in Soft Concept Representations: Sequentially trained CLMs still leak information: concept representations encode more than their concepts even when those concepts are irrelevant to the downstream task.In the parity experiment, the concepts “4” and “5” are absent from the data, yet the learned CLM outputs support substantially better-than-random prediction.
3. Extraneous Information Leakage in Soft Concept Representations: 86% accuracy is obtained from the top PCA component alone in the MNIST example, showing that leaked distributional information can support task prediction.The soft activations for concepts “4” and “5” preserve the first PCA dimension even though the concepts are task-independent.
3. Extraneous Information Leakage in Soft Concept Representations: The extent of leakage and the advantage of soft over true concept representations depends on hyperparameters, architecture, and training choices.This sensitivity means the severity of interpretability failure varies with modeling decisions.
3. Extraneous Information Leakage in Soft Concept Representations: AUC 0.999 is achieved by a jointly trained model with insufficient capacity for a relevant concept, versus 0.875 using ground-truth hard concept labels.The result indicates that soft representations of supervised concepts encode additional task-relevant information.
3. Extraneous Information Leakage in Soft Concept Representations: Adding latent dimensions does not prevent leakage: aligned bottleneck dimensions predict missing concepts, and learned feature importance can differ from the ground-truth concept-to-task model.For M2, dimensions aligned to x+ and y+ each predict z+ with AUC approximately 0.75; entanglement also makes x+ appear more important than it is.
3. Extraneous Information Leakage in Soft Concept Representations: Concept whitening also fails to eliminate leakage: any latent dimension can predict any of three concepts, despite decorrelation and other purity criteria.Decorrelated representations can retain high mutual information, so downstream interpretations still fail to recover ground-truth feature importance.

4. Avoiding the Pit-Falls of CLMs

The paper argues that common responses to leakage are insufficient and recommends using soft-versus-hard performance gaps diagnostically. It also advocates stronger disentanglement objectives and domain-expert supervision when concepts are believed to be independent.

4. Avoiding the Pit-Falls of CLMs: Hard one-hot concepts do not guarantee safety: random hard concepts can still capture data-distribution information on low-dimensional data manifolds.Thus leakage may remain an issue for black-box CLMs regardless of whether representations are soft or hard.
4. Avoiding the Pit-Falls of CLMs: A larger soft-versus-hard prediction gap may indicate that the concept set is irrelevant to the task and should be replaced.The paper presents the absent-“4”/“5” parity setting as an example motivating new concept selection.
4. Avoiding the Pit-Falls of CLMs: Concept whitening only enforces uncorrelated dimensions, so training should explicitly minimize mutual information between concept dimensions when independence is expected.The recommendation applies to both aligned and unaligned dimensions.
4. Avoiding the Pit-Falls of CLMs: Even statistically independent latent concepts can be multiply encoded, motivating domain-expert supervision to improve transparency about input–concept relationships.Independence alone does not ensure that each learned dimension corresponds to only one concept.

5. Conclusion

The paper analyzes how soft concept representations encode undesirable information beyond pre-defined concepts, undermining downstream interpretability. It also describes insights obtainable from understanding this leakage.

Soft concept representations can encode undesirable additional information that negatively impacts the interpretability of downstream predictive models.

A. Training of Concept Bottleneck Models

Concept Bottleneck Models use a concept-mapping function g and a concept-to-label function h, which can be trained independently, sequentially, or jointly. These strategies differ in how their loss functions are optimized.

A Concept Bottleneck Model consists of g mapping inputs to concepts and h mapping concepts to labels.
Independent training minimizes separate concept and label losses for g and h.
Sequential training first minimizes g's concept loss, then minimizes h's label loss using g's outputs.
Joint training minimizes label loss plus λ-weighted concept loss, with λ controlling the tradeoff between learning concepts and labels.

Representations Encode Data Distributions

In a synthetic task, predicted concept representations preserve substantial information about the original features and task boundary. Downstream performance therefore depends on representation and architectural choices, not solely on the concepts.

The synthetic dataset contains two features, three mutually exclusive concepts, and a task boundary that cuts across concepts.
The experiment predicts task labels from predicted concepts using an MLP with three hidden layers of 32 ReLU units and a 0.5 classification threshold.
Raw concept labels predict task labels with 74.5% accuracy, while the feature-to-concept model reaches 87.0% concept-prediction accuracy.
Because mutually exclusive concepts yield only two independent vectors, the three-dimensional activations can be represented in two PCA dimensions without information loss.
99.3% accuracy is achieved by predicting task labels directly from features, compared with 95.9% from predicted concepts and 89.5% with sigmoid concept activations.
The concept model largely preserves feature geometry and the task boundary, while sigmoid activations transform the boundary more severely and make optimization harder.

C. Demonstration 2: Sufficiently Large

Random concept representations can transmit information about the downstream task beyond the intended concept labels, with soft representations showing the strongest effect. As the number of random concepts increases, downstream performance approaches a direct pixel-based model.

The experiment sequentially trains a concept predictor and a downstream predictor, then evaluates their composition across increasing numbers of concepts over 10 runs.Hyperparameters are tuned to convergence for each concept count, with hard representations evaluated as a comparison.
Random concepts are generated from hyperplanes defined by random coefficients and thresholds sampled from the observed dot-product range.Each hyperplane labels observations on opposite sides as true or false examples.
99% direct-pixel accuracy is approached as random concepts increase, showing that most feature information passes through the concept modeling process.The comparison uses a direct model of similar architecture on MNIST digit parity.
Soft random concept representations outperform hard representations, and their downstream performance continues increasing without the same diminishing pattern.Hard representations also become increasingly predictive as more random concepts are added.

D. Demonstration 3: Representations Entangling Concepts

Toy and MNIST demonstrations show that concept bottlenecks can encode task-relevant information beyond their named concepts, especially when the concept set is incomplete or representations are soft. This leakage can preserve high downstream accuracy while undermining interpretability.

D. Demonstration 3: Representations Entangling Concepts: The toy task defines concepts x+, y+, and z+ as coordinate signs, while the downstream label is positive when at least two concepts are positive.The dataset uses transformed coordinate features and separate training and test sets.
D. Demonstration 3: Representations Entangling Concepts: Soft random concepts become increasingly predictive of MNIST digit parity and approach the 99% accuracy of a direct pixel model as their number grows.Hard representations also gain predictiveness, but their performance begins to diminish as more concepts are added.
D. Demonstration 3: Representations Entangling Concepts: Greater than 0.98 downstream AUC and greater than 90% accuracy are achieved with an incomplete two-concept bottleneck for all λ less than or equal to 0.1.A model using only the original two concepts obtains AUC 0.875 and accuracy 75%, while the missing concept limits its maximum expected accuracy to about 83%.
D. Demonstration 3: Representations Entangling Concepts: A complete-concept CBM is evaluated alongside incomplete two- and three-node bottlenecks using AUC scores between bottleneck dimensions and the ground-truth concepts.The supplied tables describe alignments for the complete and incomplete concept sets at λ = 0.1.

E. Demonstration 4: Information Leakage in Concept Whitening Models

Concept Whitening models can reduce correlation while still allowing latent dimensions to encode multiple concepts. Single-activation purity checks may therefore miss information leakage available through the full latent activation maps.

E. Demonstration 4: Information Leakage in Concept Whitening Models: The MNIST experiment classifies digits below 4 using concepts for digits 1, 2, and 3 on a balanced subset containing digits 1 through 6.The standard CNN uses convolution, batch normalization, pooling, global average pooling, and a linear prediction layer.
E. Demonstration 4: Information Leakage in Concept Whitening Models: Correlation reduction occurs early in training, but high AUC scores for both aligned concepts are difficult to achieve consistently.A reported trial exceeded 0.9 AUC for both concepts, yet the authors argue that high AUC alone cannot establish concept purity.
E. Demonstration 4: Information Leakage in Concept Whitening Models: The authors evaluate concept purity by comparing bottleneck outputs with ground-truth concepts and by inspecting correlations and cosine similarities in standard CNN and Concept Whitening representations.Figure 10 also varies λ to compare downstream-task and concept-prediction AUCs, with λ = 0.1 generally providing the best trade-off.
E. Demonstration 4: Information Leakage in Concept Whitening Models: AUC-based purity can appear strong because it uses one activation value, while the model can exploit the full activation map and retain information about other concepts.Thus, concept predictions based on the representation may contain leakage that the scalar AUC check does not reveal.
E. Demonstration 4: Information Leakage in Concept Whitening Models: Purity-check networks predict all three concepts from any aligned or non-aligned Concept Whitening latent dimension with over 95% accuracy.This occurs despite high AUC scores for the aligned concepts and despite satisfying the reported purity checks from Chen et al. (2020).

F. Demonstration 5: Concept Refinement

Concepts that are relevant to a prediction need not be predictive on their own when they omit important feature refinements. Splitting broad concepts into task-relevant sub-concepts can make the concept set predictive while reducing the input space to a small number of regions.

F. Demonstration 5: Concept Refinement: Approximately 50% of grapefruits and apples sell, so the broad concepts “is grapefruit” and “is apple” are not predictive of sales on their own.The synthetic task models sales using fruit weight and acidity.
F. Demonstration 5: Concept Refinement: Replacing them with “is acidic grapefruit” and “is small apple” makes the concept set perfectly predictive in the synthetic example.These sub-concepts reflect the task’s acidity and size decision boundaries.
F. Demonstration 5: Concept Refinement: Refining concepts can reduce weight and acidity to binary regions such as “is heavy” and “is acidic” while retaining predictive structure.The passage frames this as reducing the input space to a small number of predictive regions.