PaperlayerView exact revision on arXiv

Source-linked AI summary

Concept Bottleneck Models

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, Percy Liang

arXiv:2007.04612v3cs.LGstat.ML

TL;DR

End-to-end models generally do not support interaction through high-level concepts, motivating a model that predicts concepts before targets. The paper develops concept bottleneck models whose predicted concepts can be edited, and finds competitive accuracy across x-ray grading and bird identification while test-time concept correction improves accuracy.

  • Problem

    End-to-end models map raw inputs directly to targets and do not typically support querying or manipulating high-level concepts used by practitioners.

  • Method

    Concept bottleneck models predict human-specified concepts from inputs and then use those predicted concepts to make final predictions, allowing concept values to be edited at test time.

  • Results

    Concept bottleneck models achieve competitive accuracy with standard end-to-end models on x-ray grading and bird identification, while correcting concept mistakes at test time substantially improves accuracy.

  • Takeaways & Limitations

    The models support high-level concept interpretation, counterfactual explanations, and richer human-model collaboration through test-time intervention.

  • Takeaways & Limitations

    Concept bottleneck models require annotated concepts at training time, creating a trade-off between acquiring detailed annotations and acquiring additional training examples.

Abstract

from arXiv · show

We seek to learn models that we can interact with using high-level concepts: if the model did not think there was a bone spur in the x-ray, would it still predict severe arthritis? State-of-the-art models today do not typically support the manipulation of concepts like "the existence of bone spurs", as they are trained end-to-end to go directly from raw input (e.g., pixels) to output (e.g., arthritis severity). We revisit the classic idea of first predicting concepts that are provided at training time, and then using these concepts to predict the label. By construction, we can intervene on these concept bottleneck models by editing their predicted concept values and propagating these changes to the final prediction. On x-ray grading and bird identification, concept bottleneck models achieve competitive accuracy with standard end-to-end models, while enabling interpretation in terms of high-level clinical concepts ("bone spurs") or bird attributes ("wing color"). These models also allow for richer human-model interaction: accuracy improves significantly if we can correct model mistakes on concepts at test time.

1. Introduction

The paper introduces concept bottleneck models to make predictions interactable through human-specified concepts, addressing the limited concept-level interaction of end-to-end models. Across medical and vision tasks, these models retain competitive task performance while enabling interventions, counterfactual interpretation, and richer human-model collaboration.

  • End-to-end models map raw inputs directly to targets, making it difficult to query or manipulate concepts such as joint space narrowing and bone spurs.
  • Concept bottleneck models first predict human-specified concepts and then use the predicted concepts to produce the final target prediction.They are trained on inputs annotated with both concepts and targets, but concept annotations are not required at test time.
  • Editing predicted concepts and propagating those edits to the target enables test-time interventions and counterfactual explanations.For example, a radiologist can correct a predicted bone spur when it is actually an artifact.
  • On osteoarthritis grading and fine-grained bird identification, bottleneck models are comparable to standard end-to-end models while achieving high concept accuracies.The paper also reports improved robustness to covariate shifts for models guided to learn the right concepts.
  • Partially correcting concept mistakes at test time substantially improves model accuracy, while training methods produce different intervention trade-offs.The paper studies whether concept accuracy predicts intervention effectiveness and whether training choices affect intervention outcomes.

2. Related work

Related work situates concept bottleneck models among feature engineering, auxiliary concept objectives, causal modeling, and post-hoc concept interpretation. The paper distinguishes its models by learning input-to-concept mappings that form an intervention-capable prediction bottleneck.

  • Concept bottleneck models: Earlier concept bottleneck models were overtaken in predictive accuracy by end-to-end neural networks, but the approach has recently re-emerged for targeted tasks.
  • Feature engineering: Concept bottleneck models differ from feature engineering by learning mappings from raw inputs to high-level concepts rather than computing handwritten low-level features.
  • Concepts as auxiliary losses or features: Auxiliary concept losses and features in multitask or non-bottleneck models do not support intervention on individual concepts.
  • Causal models: The paper studies interventions on predicted concept values inside the model, not interventions on real-world concepts or causal effects on true labels.
  • Post-hoc concept analysis: Post-hoc concept analysis can indicate whether hidden representations encode concepts, but without intervention it cannot directly change what the model believes about a concept.

3. Setup

The setup represents prediction as a composition from inputs to a k-dimensional concept space and from concepts to the target, then compares independent, sequential, and joint training schemes with a standard model.

  • A concept bottleneck model predicts y through ˆc = g(x) and ˆy = f(ˆc), with g mapping inputs to k concepts and f mapping concepts to the final prediction.Task accuracy measures prediction of y, while concept accuracy measures prediction of c across concepts.
  • Independent bottleneck: The independent bottleneck learns the concept predictor and target predictor independently, with the target predictor trained using true concepts.At test time, the target predictor still receives predicted concepts.
  • Sequential bottleneck: The sequential bottleneck first learns the input-to-concept mapping, then trains the concept-to-target mapping using predicted concepts.
  • Joint bottleneck: The joint bottleneck minimizes a weighted combination of concept and task losses, with λ controlling their tradeoff.The standard model corresponds to λ → 0, while the sequential bottleneck can be viewed as λ → ∞.
  • The proposed construction turns an end-to-end network into a bottleneck model by resizing one layer to k concept units and selecting a training scheme.
  • Independent, sequential, and joint bottleneck models are comparable to standard end-to-end models on task error, while multitask learning further improves task error but does not permit interventions.

4. Benchmarking bottleneck model accuracy

Concept bottleneck models achieve competitive task performance while accurately predicting human-specified concepts across osteoarthritis grading and bird identification. They generally avoid a substantial task–concept accuracy trade-off and can be data-efficient, especially on OAI.

  • Applications: The evaluation covers radiologist-annotated knee x-rays for osteoarthritis grading and practitioner-annotated bird photographs for 200-way species classification.The OAI dataset contains 36,369 data points and the CUB dataset contains 11,788 photographs with 112 bird attributes.
  • Task and concept accuracies: Concept bottleneck models achieve comparable task accuracy to standard black-box models on both OAI and CUB despite the bottleneck constraint.On OAI, joint and sequential bottlenecks outperform the standard model in RMSE; on CUB, the joint model closes most of the gap.
  • Task and concept accuracies: Bottleneck models accurately predict individual concepts and achieve low average concept error across both tasks.Average Pearson correlations between predicted and true concepts are at least 0.87 for all bottleneck models.
  • Task and concept accuracies: The experiments do not observe a substantial trade-off between high task accuracy and high concept accuracy.This remains true when the bottleneck is trained jointly and its intermediate representation is pulled toward the provided concepts.
  • Data efficiency: On OAI, a sequential bottleneck model trained on approximately 25% of the full dataset performs similarly to the standard model.On CUB, joint bottleneck and standard models are more accurate throughout, with the joint model slightly better in lower-data regimes.

5. Benchmarking post-hoc concept analysis

The paper tests whether concepts can be recovered after training from standard and self-explaining models. Linear probes recover concepts less accurately than bottleneck models, and the authors emphasize that post-hoc concept prediction does not provide intervention capability.

  • Limits of post-hoc analysis: High concept accuracy is necessary but not sufficient for post-hoc interpretation in terms of concepts.Even accurate probes do not establish that changing a concept alone would change the model’s final prediction.
  • Post-hoc concept accuracy: Linear probes predict concepts less accurately from standard-model layers than bottleneck models read them from their bottleneck representations.On OAI, probe concept RMSE is 0.68 versus 0.53 for bottleneck models; on CUB, concept error is 0.09 versus 0.03.
  • Post-hoc concept accuracy: On OAI, linear probes on SENN also achieve 0.68 concept RMSE, lower than bottleneck-model concept accuracy.SENN learns representations encouraged to be interpretable without using the paper’s pre-specified concepts.
  • Implication: Concept bottleneck models directly train with practitioner-specified concepts rather than relying on their recovery from models trained without those concepts.The comparative post-hoc concept accuracy supports this approach when the relevant concepts are known in advance.

6. Test-time intervention

Test-time intervention lets users replace predicted concepts with corrected values and update the final prediction. Its effectiveness depends on concept alignment, model design, and the intervention setting.

  • Intervention procedure: Test-time intervention replaces selected predicted concepts with oracle-provided true values and recomputes the target prediction.Concepts are selected iteratively using a fixed validation-derived ordering.
  • Results: 2 concepts reduce task RMSE from >0.4 to ≈0.3 on OAI.Intervention substantially improves task accuracy in this setting.
  • Model trade-offs: The independent bottleneck performs better after replacing all k = 10 concepts, but slightly worse without intervention than sequential or joint bottlenecks.This reflects whether the c →y predictor is trained on true concepts or predicted concepts.
  • Model trade-offs: Intervention effectiveness is not determined by task and concept accuracies alone; inductive biases and c →y expressivity also matter.A joint model that over-prioritizes label accuracy can fail under intervention, while a linear c →y model is less effective than nonlinear alternatives.
  • CUB setting: CUB interventions require editing concept logits or grouped attributes, and their reported results are optimistic because oracle experts are assumed error-free.Only visible concepts are replaced with their true values.
  • Results: Intervention can improve task accuracy on CUB, but large gains require correcting several concept groups.The groups were queried in random order, so some queries were likely irrelevant to individual examples.

7. Robustness to background shifts

TravelingBirds tests whether bottleneck models withstand a shift that changes class-background associations between training and testing. Bottleneck models outperform standard models in this toy shift, but robustness depends on the concepts and shifts considered.

  • Dataset construction: TravelingBirds changes each bird class’s background association between training and testing.Birds are placed on Places backgrounds assigned by class during training, then the class-background mapping is shuffled at test time.
  • Results: Bottleneck models have substantially lower task error than standard models under the background shift.The benchmark reports both task and concept error.
  • Mechanism: Bottleneck models rely less on backgrounds because each concept appears across multiple background types and bird classes.Standard models instead exploit the training correlation between background and label, causing failures after the test shift.
  • Scope: The robustness advantage depends on the chosen concept set and the distribution shifts considered.The authors do not expect an arbitrary concept set to yield a more robust model.
  • Dataset construction: TravelingBirds is more adversarial than Waterbirds because it retains many classes and backgrounds while completely altering their test-time associations.Waterbirds collapses classes into waterbirds and landbirds and uses land or water backgrounds.

8. Discussion

Concept bottleneck models preserve competitive task accuracy while supporting concept-based reasoning, human intervention, and collaboration. The discussion highlights annotation costs, incomplete concepts, theoretical conditions, and open questions about intervention effectiveness.

  • Discussion: Concept bottleneck models can compete on task accuracy while enabling high-level reasoning and test-time human-model collaboration.The paper identifies medicine and other settings where experts can interact with models by correcting concepts.
  • Discussion: Concept annotations can trade off against additional training examples when detailed annotations are cheaper than acquiring new data.This tradeoff may be useful when collecting new medical examples requires invasive or expensive procedures.
  • Future work: Interactive human querying can help learn or refine concepts in tasks that are not normatively defined.Prior work used human-generated concepts for artist recognition, deceptive-review detection, and joke identification.
  • Future work: A direct x →y side channel can improve accuracy with incomplete concepts but prevents clean intervention on individual concepts.The connection may implicitly reason about concepts, motivating regularization or adversarial approaches.
  • Theoretical analysis: Theoretical analysis suggests independent bottlenecks have small asymptotic relative excess error when the number of concepts is much smaller than input dimensionality and concepts are less noisy than the target.The analysis compares two linear regressions for x →c and c →y with a single standard regression as training size grows.
  • Future work: Intervention effectiveness depends on training method and architecture, leaving open which factors drive successful interventions and how models should adapt from corrections.The paper proposes adaptive information-gain queries and learning from interventions as future directions.

Reproducibility

The paper provides code and executable experiment resources for reproducing the CUB experiments, while offering access procedures for the TravelingBirds and OAI data.

  • Reproducibility: Experiment-replication code is available on GitHub, with an executable version of the CUB experiments on CodaLab.The TravelingBirds dataset is also downloadable from the linked repository.
  • Reproducibility: The OAI dataset is not publicly released, but researchers can apply to access it through the NIH database.Access therefore requires an application rather than direct public download.

A. Datasets

The evaluation uses two datasets: longitudinal knee x-rays with radiologist-assessed osteoarthritis concepts and bird photographs with human-annotated attributes. Dataset construction includes reader adjudication, concept filtering, and handling of sparse or noisy annotations.

  • OAI: OAI inputs are knee x-rays paired with clinical osteoarthritis concepts and Kellgren-Lawrence grades assessed by radiologists.KLG is a five-level severity measure, while additional concepts describe osteoarthritis features across knee compartments.
  • OAI: The OAI dataset contains 4,172 patients and 36,369 knee observations divided into non-overlapping patient-level training, validation, and test sets.Each observation is one knee for one patient at one timepoint.
  • OAI: Sparse OAI concepts were removed because insufficient positive examples impaired concept prediction and lowered KLG accuracy.The filtering retained 10 clinical concepts and excluded 8 sparse concepts.
  • OAI: OAI reader disagreements were adjudicated under specified criteria, with KLG and joint-space narrowing receiving multi-reader review.Other clinical concepts were read by a single reader.
  • CUB: The CUB dataset contains 11,788 bird photographs from 200 species with 312 original binary attribute concepts and visibility annotations.Twenty percent of the official training data was randomly held out for validation.
  • CUB: CUB annotations were noisy and sometimes overlapping, so instance-level attributes were aggregated into class-level concepts by majority voting.The approximation can fail under visual occlusion and sexual or age dimorphism.
  • CUB: After filtering attributes present in at least 10 classes, the CUB concept set contained 112 binary concepts.This removed concepts that were too sparse after majority voting.

B.1. OAI model architecture and training

The OAI bottleneck models use a pretrained ResNet-18 to predict clinical concepts followed by an MLP for KLG prediction, while standard models omit concept-alignment loss. Training and intervention procedures are separately specified for OAI and CUB comparisons.

  • OAI architecture: OAI bottleneck models use an ImageNet-pretrained ResNet-18 with its last 12 convolutional layers fine-tuned on the dataset.The architecture follows the prior OAI setup except for separately tuned learning-rate choices.
  • OAI architecture: The bottleneck architecture regresses from ResNet features to concepts, then uses a three-layer MLP to predict KLG from those concepts.The standard model has the same general structure without loss encouraging the bottleneck layer to align with concepts.
  • OAI training: Training uses batch size 8, Adam optimization, grid-searched learning rates, decay every 10 epochs, and early stopping over 30 epochs.Model weights are selected using the validation epoch with lowest KLG RMSE.
  • CUB architecture: CUB uses an ImageNet-pretrained Inception V3 fine-tuned end-to-end with color jittering, horizontal flips, and random crops during training.Inference uses a center crop resized to 299 pixels.
  • Training: Hyperparameters are selected on validation data using task accuracy, or concept accuracy for the x →c component of sequential models.The search varies learning rates, schedules, and regularization strengths.
  • Training: Bottleneck concept losses weight concepts equally while correcting each binary concept loss for class imbalance.The average imbalance ratio is approximately 1:9.
  • Intervention: OAI intervention order is determined by validation-set improvement in KLG accuracy when each concept is corrected individually.The concepts are sorted by their separate accuracy gains.

C. Excess errors of independent vs. standard models

The analysis compares independent bottleneck and standard models in a well-specified linear regression setting through asymptotic excess error. It finds that independent bottlenecks can have lower excess error when concepts are low-dimensional and low-noise.

  • Independent bottleneck model: The independent bottleneck estimates X → C and C → Y with separate least-squares regressions, then composes them as ŶIB = X B̂ b̂.The first and second regressions use separate datasets with n1 and n2 training points.
  • Standard model: The standard model ignores concepts and directly estimates Y from X using one least-squares regression, producing ŶSM = Xv̂.Its regression expresses Y directly as a linear function of X with additive Gaussian noise.
  • Excess errors: Excess error is the difference between a model’s mean-squared error and the optimal estimator E[Y|X], compared as n1 = n2 = n tends to infinity.The analysis uses this quantity to compare the two models asymptotically.
  • Excess errors: The proposition compares the asymptotic excess errors of independent bottleneck and standard models under the specified linear-regression assumptions.The derivation proceeds by first obtaining each model’s expected error and then taking their ratio in the large-sample limit.
  • Excess errors: Independent bottlenecks have lower excess error when the concept dimension is low relative to input dimension and concept noise is low relative to output noise.The result is stated for the asymptotic relative excess error in the analytical setting.
Loading 2007.04612v3…