PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Concept Bottleneck Models Learn as Intended?

Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, Adrian Weller

arXiv:2105.04289v1cs.LGcs.AI

TL;DR

Concept bottleneck models are intended to make predictions through pre-specified concepts, but it is uncertain whether they satisfy interpretability, predictability, and intervenability. The authors assess CBM training variants and use post hoc interpretability methods; they find that learned concepts lack meaningful input-space representations and that joint CBMs may not meet the intended desiderata.

  • Problem

    CBMs aim to incorporate expert concepts into prediction, but it may be unrealistic to assume available concepts fully capture the relationship between inputs and targets.

    Source: §1 · p. 1

  • Method

    The authors investigate constrained-bottleneck CBM training variants and use post hoc interpretability methods to examine where concepts lie in input space.

    Source: §1 · pp. 1–2

  • Results

    The authors find that learned concepts do not appear to map to semantically meaningful input-space representations, and joint CBMs fall short on the three stated desiderata.

    Source: §1 · p. 2, §3 · p. 3

  • Takeaways & Limitations

    Under current methods, the independent CBM may be the only variant that achieves interpretability, predictability, and intervenability.

    Source: §5 · p. 4

    The intervenability result may depend on one-hot concept encoding, and the authors call for studying other concept representations.

    Source: §3 · p. 3

Abstract

from arXiv · show

Concept bottleneck models map from raw inputs to concepts, and then from concepts to targets. Such models aim to incorporate pre-specified, high-level concepts into the learning procedure, and have been motivated to meet three desiderata: interpretability, predictability, and intervenability. However, we find that concept bottleneck models struggle to meet these goals. Using post hoc interpretability methods, we demonstrate that concepts do not correspond to anything semantically meaningful in input space, thus calling into question the usefulness of concept bottleneck models in their current form.

1 INTRODUCTION

Concept bottleneck models route inputs through pre-specified concepts before predicting targets, aiming for interpretability, predictability, and intervenability. The authors question whether common training methods achieve these goals and report that learned concepts lack meaningful input-space representations.

1 INTRODUCTION

  • CBMs incorporate expert-defined concepts by mapping raw inputs to concepts and then concepts to targets, with interpretability, predictability, and intervenability as goals.Intervenability means replacing predicted concept values with ground-truth values to improve predictive performance.Source: §1 · p. 1
  • Neither joint nor sequential training with constrained bottlenecks satisfies the three desiderata, according to the authors.They suggest independent training may be the only way to achieve all three.Source: §1 · p. 1
  • Post hoc interpretability analyses find that pre-specified concepts do not appear to correspond to meaningful features in input space.The authors propose showing concepts to domain experts if they can be mapped to meaningful input-space features.Source: §1 · p. 2

2 PROBLEM SETUP

A concept bottleneck model composes an input-to-concept mapping with a concept-to-target mapping. The section distinguishes three training procedures and describes post hoc methods used to inspect their behavior.

2 PROBLEM SETUP

  • A CBM maps inputs x to concepts c with g, then maps c to targets y with f; training uses target and concept losses.The setup assumes inputs, targets, and pre-specified concepts are available in the training set.Source: §2 · p. 2
  • Independent training fits the input-to-concept mapping and separately fits the target predictor using ground-truth concepts.Sequential training instead fits the target predictor using concepts predicted by the learned input-to-concept mapping.Source: §2 · p. 2
  • Joint training learns g and f together by minimizing target loss plus a weighted concept loss.The parameter λ trades off the two losses.Source: §2 · p. 2
  • The authors use post hoc interpretability methods to identify input features relevant to concepts and concepts relevant to targets.Integrated Gradients is used for this analysis; Gradients and SmoothGrad are also used for saliency visualizations.Source: §2 · p. 2

3 CHECKING THE THREE DESIDERATA

The authors compare CBM variants with a concept-oracle reference and examine target–concept dependence and intervenability. Their results raise concerns about whether concepts are used as intended, while the intervenability account also calls for attention to concept representation.

3 CHECKING THE THREE DESIDERATA

  • With all concepts, the concept oracle achieves low error, though obtaining a full concept set that specifies the target well may be difficult.The authors note that joint CBM results are sensitive to λ and use the value from Koh et al. (2020).Source: §3 · p. 2
  • The joint CBM achieves much lower error than the concept oracle in the single-concept comparison, suggesting it uses target information beyond the concept bottleneck.The caption identifies Table 1 as OAI RMSE for each of ten single concepts, averaged over five initializations; standard deviations are below 0.01.Source: §3 · p. 2, Table 1 · p. 3
  • For CUB, saliency-map R^2 is 0.47 ± 0.04 for oracle–sequential, 0.24 ± 0.07 for oracle–joint, and 0.07 ± 0.04 for sequential–joint.The authors interpret the weak correspondence with the oracle as evidence that predicted concepts may serve as proxies for target information.Source: §3 · p. 3
  • The prose describes joint and sequential intervenability as inferior to independent, while Figure 1's caption says the reverse.The authors suggest the intervenability pattern may be an artifact of one-hot concept encoding and call for study of other representations.Source: §3 · p. 3, Figure 1 · p. 3
  • The authors conclude that joint CBMs fall short on interpretability and predictability, and may also fail intervenability because concepts are learned at the concept layer.They say the independent CBM may be the only variant meeting all three desiderata under current methods.Source: §3 · p. 3

4 CONCEPTS IN INPUT SPACE

The joint and independent CBMs’ concept saliency maps often miss the corresponding bird part, although the authors caution that current saliency methods may be ill-equipped to analyze concept bottlenecks.

4 CONCEPTS IN INPUT SPACE

  • The authors caution that existing saliency methods may be ill-equipped to study attribution for concept bottlenecks.They suggest developing methods better suited to this analysis.Source: §4 · p. 3
  • Both joint and independent CBMs’ wing-pattern maps attend to the entire bird rather than just the wing.Figure 2 shows this for the discrete wing-pattern values and their averaged concept map.Source: §4 · p. 3, Figure 2 · p. 4
  • The joint model’s leg-color concept does not attend to the bird’s leg across saliency methods.Figure 3 illustrates the joint model result.Source: §4 · p. 3, Figure 3 · p. 4
  • The independent model’s leg-color concept also does not attend to the bird’s leg across saliency methods.Figure 4 illustrates the independent model result.Source: §4 · p. 3, Figure 4 · p. 4

5 DISCUSSION

The discussion questions whether CBMs learn concepts as intended: under current methods, only the independent variant may meet all three desiderata, and the learned concepts lack semantically meaningful input-space representations.

5 DISCUSSION

  • Given current CBM methods, the independent variant may be the only one that achieves interpretability, predictability, and intervenability.The discussion notes that the joint CBM learns information about the target before the concept layer.Source: §5 · p. 4
  • Existing saliency methods find that none of the learned concepts map to semantically meaningful representations in input space.The authors frame this as a limitation of CBMs’ ability to meet the desiderata.Source: §5 · p. 4
  • If concepts mapped meaningfully into input space, domain experts could use such methods to discover target-relevant concepts that were not pre-specified.The authors present this as a possibility for future work.Source: §5 · p. 4

A.1 DATASETS AND HYPERPARAMETERS

The appendix describes an osteoarthritis grading task and a bird identification task, along with their concept representations and loss weights.

A.1 DATASETS AND HYPERPARAMETERS

  • The OAI task uses 36,369 x-rays to predict Kellgren-Lawrence grade from ten ordinal, clinically relevant concepts.The concepts are instance-specific, so examples with the same target may have different concept values.Source: §A.1 · p. 6
  • The CUB task uses 11,788 bird photographs to predict one of 200 species using 112 class-level, one-hot-encoded concepts.Training examples of the same class share the same concepts; categorical values such as wing color are represented by separate one-hot indicators.Source: §A.1 · p. 6
  • The concept-loss weight is λ = 0.01 for CUB and λ = 1 for OAI.The authors state that results depend strongly on λ; larger values enforce the expert-specified concepts more strongly.Source: §A.1 · p. 6

A.2 POST HOC INTERPRETABILITY

The post hoc analysis uses saliency maps to examine concept-to-input and concept-to-target relationships, with alternative ways to summarize categorical concepts.

A.2 POST HOC INTERPRETABILITY

  • Saliency methods applied to the target predictor identify which concepts matter to a target, while methods applied to the input-to-concept model identify where a concept lies in the input.The paper illustrates these analyses with bird wing and x-ray tibia concepts; Appendix B provides additional post hoc results.Source: §A.2 · p. 6
  • The analysis uses Integrated Gradients and also visualizes results with Gradients and SmoothGrad.The methods are applied to identify input features relevant to concepts and concepts relevant to targets.Source: §A.2 · p. 6
  • For categorical concepts, the authors summarize saliency maps either by averaging category maps or by weighting them with the softmax of the one-hot concept values.They use both alternatives for concepts such as beak color.Source: §A.2 · p. 6

B ADDITIONAL POST HOC RESULTS

Across three saliency methods, the models’ maps consistently focus outside the concept of interest, including examples for bill-shape and wing-pattern concepts.

B ADDITIONAL POST HOC RESULTS

  • Across Saliency (or Gradient), Integrated Gradients, and Smoothgrad, CBM maps consistently focus outside the concept of interest.The maps were computed on training images where both Joint and Independent models correctly predicted the illustrated concepts.Source: B ADDITIONAL POST HOC RESULTS · p. 7
  • The additional examples cover bill-shape concepts for Independent and Joint models and wing-pattern concepts for both models.Source: Figure 5 · p. 7, Figure 6 · p. 7, Figure 7 · p. 8

Related papers

Loading 2105.04289v1…