PaperlayerView exact revision on arXiv

Source-linked AI summary

A Closer Look at the Intervention Procedure of Concept Bottleneck Models

Sungbin Shin, Yohan Jo, Sungsoo Ahn, Namhoon Lee

arXiv:2302.14260v3cs.LG

TL;DR

The intervention procedure of concept bottleneck models remains comparatively unexplored despite its importance to practitioners. This paper develops and evaluates concept-selection strategies across different costs, intervention arrangements, implementations, and data characteristics. The analyses show that intervention effectiveness varies across these conditions, and the authors identify pitfalls in current practices.

  • Problem

    Many critical aspects of concept intervention in CBMs remain unexplored, despite the procedure's importance to practitioners.

    Source: §1 · p. 2, §8 · p. 9

  • Method

    The paper develops concept-selection criteria and evaluates intervention procedures across varied settings, including cost, intervention levels, implementation, and data characteristics.

    Source: §1 · p. 2, §3.2 · p. 2, §7 · p. 9

  • Results

    Intervention effectiveness varies with cost, data characteristics, and intervention granularity, and the paper identifies pitfalls in current practices.

    Source: §1 · p. 2, §7 · p. 9, §8 · p. 9

  • Takeaways & Limitations

    Choosing intervention criteria should reflect available budgets and test-time environments, and practitioners should account for dataset characteristics when deploying CBMs.

    Source: §7 · p. 9

    The study focuses on classification tasks; extending intervention strategies to regression with real-valued concepts or targets remains future work.

    Source: §7 · p. 9

Abstract

from arXiv · show

Concept bottleneck models (CBMs) are a class of interpretable neural network models that predict the target response of a given input based on its high-level concepts. Unlike the standard end-to-end models, CBMs enable domain experts to intervene on the predicted concepts and rectify any mistakes at test time, so that more accurate task predictions can be made at the end. While such intervenability provides a powerful avenue of control, many aspects of the intervention procedure remain rather unexplored. In this work, we develop various ways of selecting intervening concepts to improve the intervention effectiveness and conduct an array of in-depth analyses as to how they evolve under different circumstances. Specifically, we find that an informed intervention strategy can reduce the task error more than ten times compared to the current baseline under the same amount of intervention counts in realistic settings, and yet, this can vary quite significantly when taking into account different intervention granularity. We verify our findings through comprehensive evaluations, not only on the standard real datasets, but also on synthetic datasets that we generate based on a set of different causal graphs. We further discover some major pitfalls of the current practices which, without a proper addressing, raise concerns on reliability and fairness of the intervention procedure.

1. Introduction

Concept bottleneck models (CBMs) predict a target through human-interpretable concepts, which experts can correct at test time. Because even a small number of concept errors can substantially raise task error, the paper studies more effective intervention strategies and their conditions.

1. Introduction

  • On CUB, one incorrectly predicted concept is associated with a 25% increase in task error, while expert intervention can be costly.The figure describes average misclassification rate as the number of incorrectly predicted concepts increases.Source: Figure 1 · p. 1, §1 · p. 2
  • CBMs predict human-interpretable concepts before using them to make a target prediction, and experts can rectify predicted concepts at test time.The corrected concept values are used for the subsequent task prediction.Source: §1 · p. 1
  • Prior work largely focused on CBM variants and related issues, leaving critical aspects of the intervention procedure relatively unexplored.The authors develop concept-selection strategies and evaluate intervention across settings, including intervention levels, costs, and data characteristics.Source: §1 · p. 2

2. Related Work

Prior CBM work has pursued model variants for performance, interpretability, confounding, and information leakage, while some concurrent work proposed uncertainty-based interventions. This paper instead analyzes intervention procedures for standard CBMs in greater depth.

2. Related Work

  • Prior CBM variants address confounding, information leakage, disentangled or autoregressive representations, and the accuracy–interpretability trade-off.Examples include debiased CBMs, unsupervised concepts, and concept embedding models.Source: §2 · p. 2
  • Chauhan et al. (2022) and Sheth et al. (2022) present uncertainty-based methods for choosing concepts to intervene on.Source: §2 · p. 2
  • Whereas prior work mostly takes a model-centric perspective, this work provides in-depth analyses and comprehensive evaluations of intervention procedures in standard CBMs.Source: §2 · p. 2

3. Intervention Strategies

The paper formalizes CBM prediction and test-time concept correction, then compares criteria for choosing interventions. It also considers theoretical costs and intervention arrangements that vary by concept grouping and data access.

3.1. Preliminary

  • CBMs first predict concepts from input x, then predict target y from those concepts; intervention replaces selected predicted values with ground-truth values before the target prediction.Source: §3.1 · p. 2

3.2. Concept Selection Criteria

  • Concept-selection criteria assign scores and intervene in descending score order; RAND selects uniformly at random, while UCP prioritizes the highest-uncertainty predictions.LCP uses error against ground truth but is unavailable at test time; CCTP uses concept contribution to target outputs, and ECTP estimates expected change in the target predictive distribution.Source: §3.2 · p. 3

Expected uncertainty decrease in target prediction

  • EUDTP scores concepts by expected entropy decrease in the target predictive distribution and penalizes concepts with low expected entropy reduction after intervention.Source: Expected uncertainty decrease in… · p. 3

3.2.1. COST OF INTERVENTION

  • Theoretical intervention cost varies by selection criterion and includes inference costs for g and f plus expert time τ_i; CCTP is given as O(τ_g + 3τ_f + nτ_i).The cost expression assumes the backward pass on f costs the same as τ_f.Source: §3.2.1 · p. 3, Table 2 · p. 3

3.3. Levels of Intervention

  • Interventions can select individual concepts or related groups, and can allocate budgets per test case or share a budget across a batch.Group scores average the individual concept scores; batch allocation can focus effort on harder cases.Source: §3.3 · pp. 3–4

4. Evaluating Intervention Strategies

The section evaluates concept-selection strategies and shows that intervention effectiveness depends on cost assumptions, intervention granularity, training strategy, and concept representation. In most tested settings, informed selection outperforms random intervention, but important exceptions qualify that result.

4.1. Experiment Settings

  • The experiments use CUB, SkinCon, and synthetic datasets, following standard implementation protocols.The synthetic datasets support controlled experiments based on different causal graphs.Source: §4.1 · p. 4

Training

  • Training compares independent, sequential, and joint strategies, while conceptualization compares soft, hard, and sampled concept predictions.Hard predictions prevent information leakage in exchange for decreased prediction performance; sampled predictions ensemble five draws.Source: Training · p. 4

4.2. Evaluating Concept Selection Criteria

  • On CUB, correcting 20 concepts reduces task error less than 4% with RAND versus more than 16% with UCP; halving error takes 43 RAND corrections versus 12 with UCP.On SkinCon, ECTP leads among practical criteria, while LCP performs exceptionally well using ground-truth concept knowledge unavailable in practice.Source: §4.2 · p. 4, Figure 3 · p. 5

4.2.1. REFLECTING COST OF INTERVENTION

  • As α shrinks, ECTP loses effectiveness relative to RAND; at small β ECTP can also perform poorly, while realistic expert-intervention costs imply α is much greater than 1.ECTP recovers effectiveness as α grows; the small-α case is described as hypothetical, while changing β controls the relative costs of concept and target inference.Source: Figure 4 · p. 5, §4.2.1 · p. 5

4.3. Analyzing Intervention Levels

  • On CUB G+S, RAND needs 39.3% (11 out of 28) versus UCP’s 25.0% (7 out of 28) of groups to halve task error.Group-wise intervention is generally less effective than individual intervention at equal expense, though it may be cost-effective for mutually exclusive concepts; batch intervention is often more effective than single intervention, except for CCTP.Source: §4.3 · pp. 5–6, Figure 5 · p. 6

4.4. Considering Training and Conceptualization

  • Intervention is generally most effective with IND training, while EUDTP can underperform RAND under SEQ or JNT training.The authors suggest that training the target predictor without ground-truth concepts may explain why fixing predicted concepts works less well for those schemes.Source: §4.4 · p. 6
  • On SkinCon, intervention under SEQ, JNT, and JNT+P increases average task error regardless of the selection criterion.JNT already achieves low task error; the authors hypothesize that dataset characteristics and the limited bottleneck may contribute to the increase.Source: §4.4 · p. 6
  • HARD and SAMP may start with higher task error than SOFT, but UCP narrows the gap between conceptualization methods faster than RAND as interventions increase.This pattern is reported across training strategies and datasets.Source: §4.4 · p. 6

5. Analyzing Intervention with Synthetic Data

Synthetic causal-graph experiments show that intervention outcomes and selection-criterion rankings vary with data conditions. The analyses examine input noise, hidden concepts, concept diversity, and subgroup size.

5.1. Generating Synthetic Data

  • The synthetic-data framework uses three causal graphs to control input noise, hidden concepts, and probabilistically varying concept values.The three cases represent noisy inputs, concepts necessary for correct target prediction that are hidden, and within-class concept diversity.Source: §5 · p. 6, §5.1 · pp. 6–7

5. Analyzing Intervention with Synthetic Data

  • Real datasets differ in how much error intervention removes and in which selection criteria work best, motivating controlled synthetic experiments.The paper reports that intervening on all concepts brings CUB task error to 0%, while SkinCon error remains around 29%.Source: §5 · p. 6, §5.2 · p. 8

5.2. Results

  • At h = 50%, final task error remains around 13%, versus zero error at h = 0%, because hidden concepts leave target prediction without complete information.The authors note that hidden concepts are often present when constructing CBMs in realistic settings.Source: Figure 8 · p. 7, §5.2 · p. 7
  • With z = 2.0, reducing task error by half takes 17 intervention counts, versus 2 counts at z = 0.5; at z = 0.1, initial task error is near 0%.The initial task error increases with noise because concept prediction performs poorly.Source: Figure 8 · p. 7, §5.2 · p. 7
  • Subgroup size changes criterion rankings: EUDTP is less effective than RAND at γ = 1 but most effective at γ = 10, except for LCP.A subgroup is defined as classes with similar concept values; the authors suggest EUDTP works better when larger subgroups make target prediction uncertainty easier to reduce.Source: Figure 9 · p. 7, §5.2 · p. 7
  • More diverse concept values within a class increase initial and final task errors and make intervention less effective.The authors attribute this to the greater difficulty of learning discriminative representations; preprocessing concepts to have uniform values within a class may adversely affect fairness.Source: §5.2 · p. 7

6. Pitfalls of Intervention Practices

The section examines reliability and fairness pitfalls in intervention: nullifying uncertain concepts can raise task error, while majority voting can disadvantage minority samples.

6.1. Nullifying Void Concepts Increases Task Error

  • Nullifying void concepts by setting unsure values to zero can catastrophically fail, so the authors advise avoiding it for invalid cases.An invisible attribute does not necessarily mean the corresponding concept is false.Source: §6.1 · p. 8

6.2. Majority Voting Neglects Minorities

  • Majority voting can decrease task error on intervention while yielding biased predictions against minorities.Figure 11 contrasts results with and without majority voting.Source: Figure 11 · p. 8
  • For minority black terns with white underparts, majority voting can misguide predictions to black and favor correct targets only when their concept value changes to black.Intervention can aggravate the bias because the minority samples' task error decreases only when their predicted concept changes to the majority-voted value.Source: §6.2 · p. 8
  • The authors warn that majority voting on sensitive concepts such as race or gender can be problematic, and majority-vote values are unavailable for test-time intervention and are not considered fair.They defer addressing the performance–fairness trade-off to future work.Source: §6.2 · pp. 8–9

7. Discussion and future work

The discussion presents intervention analyses and an evaluation framework, while emphasizing that effectiveness depends on costs, data characteristics, and fairness considerations. It identifies classification as the scope of the work and points to other settings for future study.

In-depth analysis of intervention procedure

  • Intervention effectiveness depends not only on which concepts are selected, but also on what data is intervened on and under which environments.The authors propose further theoretical investigation of intervention strategies.Source: §7 · p. 9

Benchmark for evaluating concept selection methods

  • The evaluation protocol can assess new concept-selection methods, and the synthetic-data framework tests them under varied circumstances.The framework generates data based on different causal graphs.Source: §7 · p. 9

Analyzing the cost of intervention

  • A criterion's effectiveness can change with intervention cost, so selection should reflect available budgets and test-time environments, especially in extreme settings.The authors describe cases involving model size or labor-market conditions as hypothetical, while realistic settings have α much greater than 1.Source: §7 · p. 9

Identifying the effect of data on intervention

  • Intervention can be less effective on datasets with more hidden concepts or more diverse concept values within the same class.The authors advise practitioners to account for these dataset characteristics when developing and deploying CBMs.Source: §7 · p. 9

Reliability and fairness of intervention

  • The authors identify cases where intervention increases task error or biases predictions toward the majority, raising reliability and fairness concerns.They call for these problems to be addressed before adopting CBMs in practice.Source: §7 · p. 9

Extension of our work to other settings

  • The study focuses on classification; extending intervention strategies to regression with real-valued concepts or targets is proposed as future work.Other suggested directions include architectural variations such as hard autoregressive or concept-embedding models.Source: §7 · p. 9

8. Conclusion

The paper studies CBM intervention, developing concept-selection criteria and evaluating how their behavior varies with intervention levels, cost, training, conceptualization, and data characteristics. It also identifies pitfalls needing careful attention before use in realistic settings.

8. Conclusion

  • The intervention procedure has been understudied despite its practical importance, and selection criteria can behave differently across experimental factors.The authors call for further work on more effective and reliable intervention strategies.Source: §8 · p. 9

A.1. CUB

  • CUB experiments use binary bird-attribute concepts, majority-vote preprocessing, and retain 112 of 312 concepts after removing sparse ones.Majority voting reduces label noise but may harm fairness in some cases.Source: §A.1 · p. 11

A.2. SkinCon

  • SkinCon experiments use 3,230 skin-lesion images, 22 concepts, binary disease labels, and average per-class error for imbalanced classes.The dataset is split into 70% training, 15% validation, and 15% test data.Source: §A.2 · p. 11

A.3. Synthetic dataset

  • The synthetic data vary concept sparsity, similarity between class groups, input noise, hidden concepts, and concept-target variation.Classes within each group differ in two concept values; the default setup uses γ = 2 and k = 100.Source: §A.3 · pp. 11–12

B. Architectures and Training

  • CUB uses pretrained Inception-v3 for concept prediction and a one-layer MLP for task prediction, while SkinCon fine-tunes Deepderm and uses a one-layer MLP.Synthetic data use a three-layer MLP for concept prediction and a linear task predictor; each setup is trained over five random seeds.Source: B. Architectures and Training · p. 12

C. More on Reflecting Cost of Intervention

Intervention criteria can rank differently when intervention costs are included. In the tested realistic CUB setting, relative rankings remain the same, though the authors note that model and GPU differences could alter results.

C. More on Reflecting Cost of Intervention

  • As β decreases, ECTP becomes less effective because its cost includes O(τg + (2k + 2)τf + nτi), while RAND requires only one forward pass of f.The comparison fixes τi = 1, α = 1, and k = 100.Source: C. More on Reflecting Cost of… · p. 12, Figure 12 · p. 13
  • With CUB annotation and inference times included, RAND and ECTP retain their relative ranking, but model size or GPU environment may change the result.The setting uses τi ≈ 0.7 seconds, τg ≈ 18.7 × 10^-3 seconds, and τf ≈ 0.03 × 10^-3 seconds; ECTP first selects concepts with higher annotation costs when costs vary by concept.Source: C. More on Reflecting Cost of… · pp. 12–13, Figure 13 · p. 13

D. More Results on the Effect of Intervention Levels on Intervention

Intervention effectiveness varies with granularity and batching. Individual intervention is generally more effective than group-wise intervention, while batch intervention is at least competitive with single intervention across criteria.

D. More Results on the Effect of Intervention Levels on Intervention

E. More Results on the Effect of Training Strategies on Intervention

Training strategy affects which concept-selection criteria perform well, while effective criteria reduce differences between conceptualization strategies faster than RAND across the evaluated datasets.

E. More Results on the Effect of Training Strategies on Intervention

F. More Results on the Effect of Conceptualization Methods on Intervention

G. More Results on the Effect of Data on Intervention

Selection effectiveness varies with data conditions and intervention granularity; on CUB, omitting majority voting can make intervention increase task error.

G. More Results on the Effect of Data on Intervention

  • Under extremely high input noise or diversity, selection criteria work less effectively overall, and UCP is less effective than other criteria and farther from LCP (Figure 28).The authors assume uncertainty and prediction loss become less correlated when the concept predictor has very low accuracy.Source: G. More Results on the Effect of Data… · p. 16, Figure 28 · p. 17
  • 12 intervention counts were needed to halve task error at γ = 1, versus 5 corrected concepts at γ = 10 (Figure 29).With larger sub-group size, final task error increases, but intervention reduces it faster on mis-classified examples.Source: Figure 29 · p. 17, G. More Results on the Effect of Data… · p. 17
  • CCTP becomes less effective as concept sparsity approaches 50%, because concept contributions become less discriminative when nearly half the concepts determine prediction (Figure 29a).At higher sparsity, a few concepts determine the target prediction, letting CCTP prioritize the concept with the highest contribution.Source: G. More Results on the Effect of Data… · p. 16, Figure 29 · p. 17

H. More Results on Fairness of Majority Voting

Loading 2302.14260v3…