PaperlayerView exact revision on arXiv

Source-linked AI summary

Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off

Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, Mateja Jamnik

arXiv:2209.09056v2cs.LGcs.AI

TL;DR

Existing CBMs face trade-offs among task accuracy, interpretability, and effective interventions, especially when concept labels are incomplete. CEMs use supervised high-dimensional concept representations and paired embeddings. The experiments report competitive or better accuracy than standard DNNs, with interpretable concepts and effective, robust interventions.

  • Problem

    Existing CBMs can lose task accuracy when concepts omit information needed for prediction, and the introduction identifies trade-offs among accuracy, interpretability, and interventions, especially with incomplete concept annotations.

    Source: §1 · p. 2, §6 · p. 10

  • Method

    CEMs represent each concept with a supervised vector built from active and inactive embeddings, and introduce RandInt to expose models to interventions during training.

    Source: §3 · p. 3, §3.1 · p. 3, §5.4 · p. 9

  • Results

    Embedding-based CBMs achieve competitive or better downstream accuracy than DNNs without concept explanations; CEMs provide aligned concepts and effective, robust interventions.

    Source: §5.2 · p. 7, §5.4 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    The authors report that CEMs are more efficient in concept-incomplete settings, requiring fewer concept annotations and being more applicable to real-world tasks.

    Source: §6 · p. 10

    CBMs require carefully selected concept annotations during training, which can be as expensive as task labels to obtain.

    Source: §6 · p. 10

Abstract

from arXiv · show

Deploying AI-powered systems requires trustworthy models supporting effective human interactions, going beyond raw prediction accuracy. Concept bottleneck models promote trustworthiness by conditioning classification tasks on an intermediate level of human-like concepts. This enables human interventions which can correct mispredicted concepts to improve the model's performance. However, existing concept bottleneck models are unable to find optimal compromises between high task accuracy, robust concept-based explanations, and effective interventions on concepts -- particularly in real-world conditions where complete and accurate concept supervisions are scarce. To address this, we propose Concept Embedding Models, a novel family of concept bottleneck models which goes beyond the current accuracy-vs-interpretability trade-off by learning interpretable high-dimensional concept representations. Our experiments demonstrate that Concept Embedding Models (1) attain better or competitive task accuracy w.r.t. standard neural models without concepts, (2) provide concept representations capturing meaningful semantics including and beyond their ground truth labels, (3) support test-time concept interventions whose effect in test accuracy surpasses that in standard concept bottleneck models, and (4) scale to real-world conditions where complete concept supervisions are scarce.

1 Introduction

The introduction motivates concept-based models as a way to make accurate predictions more interpretable and open to human correction. It presents CEMs as an approach to the accuracy–interpretability trade-off, including when concept labels are incomplete.

1 Introduction

  • High-performing DNNs can fail to explain what they learn in human-understandable terms, motivating concern about their deployment.The introduction connects the accuracy-vs-interpretability trade-off to ethical and legal ramifications [7].Source: §1 · p. 2
  • CBMs predict high-level concepts before the downstream task, allowing predictions to be explained through concepts and experts to correct mispredicted concepts at test time.Concept labels may omit information needed for accurate downstream prediction, limiting task accuracy.Source: §1 · p. 2
  • Hybrid CBMs add unsupervised bottleneck units, but the introduction says this can hinder concept interventions and interpretability.The authors argue that this undermines user trust.Source: §1 · p. 2
  • CEMs are presented as overcoming the accuracy-vs-interpretability trade-off in concept-incomplete settings, with experiments showing competitive or better task accuracy than standard DNNs and effective interventions.The introduction also reports meaningful concept representations at least as well as vanilla CBMs.Source: §1 · p. 2

2 Background

CBMs map inputs to an intermediate concept vector and use that vector for downstream prediction. The section describes alternative training strategies, scalar and hybrid concept representations, and test-time interventions.

2 Background

  • A CBM maps input x to predicted concepts through an encoder and maps those concepts to a downstream label through a predictor.Training uses samples with input features, binary ground-truth concepts, and task labels.Source: §2 · pp. 2–3
  • CBMs can train the encoder and predictor sequentially, independently, or jointly with a weighted sum of cross-entropy losses.Source: §2 · p. 3
  • Boolean and Fuzzy CBMs represent each concept with a scalar aligned to one ground-truth concept; Hybrid CBMs append unsupervised dimensions to supervised concept dimensions.Boolean uses thresholding, while Fuzzy uses sigmoid activation.Source: §2 · p. 3
  • At test time, an expert can intervene by setting a predicted concept to its ground-truth value, potentially improving task performance.Such interventions can also construct concept-based counterfactuals.Source: §2 · p. 3

3 Concept Embedding Models

CEMs represent each concept with a supervised vector rather than a scalar, aiming to add learning capacity without making that capacity concept-agnostic. Their active and inactive embeddings support test-time interventions, while RandInt exposes the model to interventions during training.

3 Concept Embedding Models

  • CEMs use supervised high-dimensional vectors for concepts to add learning capacity when complete concept annotations are scarce.The authors contrast this with Hybrid CBMs, whose additional unsupervised bottleneck activations are concept-agnostic.Source: §3 · p. 3

3.1 Architecture

  • For each concept, CEM mixes positive and negative embeddings, giving evidence for and against concept activity and allowing intervention by switching embedding states.The architecture concatenates the mixed concept embeddings for the downstream label predictor.Source: §3.1 · pp. 3–4
  • A latent representation h feeds two concept-specific fully connected layers that learn embeddings in R^m for each concept.The two embeddings correspond to active and inactive concept states.Source: §3.1 · p. 3
  • A learnable scoring function estimates concept activity from the positive and negative embeddings.The score is a predicted probability for the concept.Source: §3.1 · p. 4
  • CEM trains jointly on task and concept prediction losses, with α controlling their relative importance.Source: §3.1 · p. 4

3.2 Intervening with Concept Embeddings

  • At test time, an expert corrects a concept prediction by replacing its mixed embedding with the embedding for the ground-truth state.For example, setting a predicted probability to 1 switches to the positive embedding.Source: §3.2 · p. 4
  • RandInt randomly applies independent concept interventions during training to improve the effectiveness of interventions at test time.The training strategy exposes the model to randomly corrected concept probabilities.Source: §3.2 · pp. 4–5
  • RandInt can be viewed as averaging over many CEMs, some receiving task and concept feedback for a representation and others receiving only concept-label feedback.Source: §3.2 · p. 5

4 Evaluating concept bottlenecks

This section introduces a metric for assessing concept-label alignment in multidimensional representations and proposes information-theoretic analysis of information flow through concept bottlenecks.

4 Evaluating concept bottlenecks

  • The section introduces a metric for concept quality in multidimensional representations and an information-theoretic analysis to help explain CBM underperformance.The analysis examines information flow at the concept level.Source: §4 · p. 5
  • CAS assesses whether learned concept representations faithfully reflect ground-truth labels by clustering each concept’s representations and measuring label homogeneity.It averages homogeneity across concepts and cluster counts to produce a normalized score in [0, 1].Source: §4 · p. 5
  • In practice, CAS varies the number of clusters, uses k-Medoids, and represents each Hybrid CBM concept with its individual activation and shared embedding.For Boolean and Fuzzy CBMs, CAS uses concept logits.Source: §4 · p. 5
  • Information Plane analysis tracks mutual information between inputs and concept representations, and between concept representations and task labels, across training epochs.The method uses these trajectories to study information flow at the concept level.Source: §4 · pp. 5–6
  • The authors conjecture that embedding-based CBMs preserve more input information in high-dimensional activations, while scalar representations compress input information and trade it off against task information.This proposed contrast is to be assessed through the mutual-information trajectories.Source: §4 · pp. 5–6

5 Experiments

The experiments assess task accuracy, concept interpretability, and interventions. Embedding-based CBMs achieve competitive or better task accuracy while providing aligned concept representations and effective interventions.

5 Experiments

  • The experiments ask whether concept embeddings affect task accuracy, align explanations with ground-truth concepts and add interpretability, and enable meaningful interventions.Source: §5 · p. 6

5.1 Setup

  • Evaluation covers XOR, Trigonometric, and Dot benchmarks, plus CUB with complete concept annotations and CelebA with six balanced, incomplete annotations.The study also compares against Boolean, Fuzzy, and Hybrid Joint-CBMs and capacity-matched DNNs without concept supervision.Source: §5.1 · p. 6

5.2 Task Accuracy

  • Embedding-based CBMs match or exceed non-concept DNN task accuracy and outperform Boolean and Fuzzy CBMs by up to +45% on Dot.The gains are emphasized for nonlinear or concept-incomplete tasks.Source: §5.2 · p. 7, Figure 3 · p. 7
  • Embedding-based CBMs show monotonically increasing I(X, Ĉ) during training, whereas Boolean, Fuzzy, and non-concept models tend to forget input information.In CEM, all input information flows through concepts; the authors connect this to higher alignment than Hybrid models.Source: §5.2 · p. 7, Figure 4 · p. 8

5.3 Interpretability

  • CEM concept-alignment scores are competitive with or better than Boolean and Fuzzy CBMs, while Hybrid alignment is up to 25% lower than CEM on Dot.The authors attribute Hybrid’s lower alignment to unsupervised activations that can contain information irrelevant to a given concept.Source: §5.3 · pp. 7–8
  • With only 25% of CUB concept annotations, CEM embeddings predict the remaining concepts at 94.33% ± 0.88% accuracy versus 91.83% ± 0.51% for Hybrid embeddings.The authors present this as evidence that CEM bottlenecks can serve as interpretable feature extractors.Source: §5.3 · p. 8
  • CEM’s “has white wings” embeddings separate active from inactive samples and cluster same-type birds together, unlike the less clearly separated Hybrid embeddings.The authors interpret the pattern as a hierarchy organized first by concept activation and then by task label.Source: §5.3 · pp. 8–9, Figure 5 · p. 8

5.4 Interventions

  • CEMs respond more effectively to interventions than scalar-based CBMs and tolerate a few incorrect settings, though performance deteriorates as errors accumulate.With RandInt, CEMs can outperform scalar-based CBMs when experts set large portions of the bottleneck.Source: §5.4 · p. 9, Figure 6 · p. 9

6 Discussion

The discussion presents CEM as a way to improve the accuracy–interpretability trade-off while retaining concept interventions. It reports gains across accuracy, concept representations, and interventions, alongside remaining limits and deployment risks.

6 Discussion

  • Benefits over vanilla CBMs and end-to-end black-box DNNs appear without drastically increasing CEM embedding size.The authors report this in an embedding-size ablation.Source: §6 · p. 10
  • The authors caution that stronger interpretable models may still pose risks of societal harm.Source: §6 · p. 10
  • CEMs offer state-of-the-art task accuracy, interpretable concepts aligned with human ground truths, effective interventions, and robustness to incorrect interventions.The authors describe these results as advancing the accuracy-vs-interpretability trade-off.Source: §6 · p. 10
  • CEMs require fewer concept annotations in concept-incomplete settings, which the authors say makes them more applicable to real-world tasks.CBM concept annotations can be as expensive to obtain as task labels.Source: §6 · p. 10
  • Concept alignment and task accuracy still have room to improve on challenging benchmarks such as CUB or CelebA, and inference and training resource use also need improvement.Source: §6 · p. 10

Checklist

The checklist records the paper’s stated scope, disclosures, and reproducibility materials. It also identifies synthetic data and notes that societal impacts are discussed.

Checklist

  • The authors say Sections 3 and 5 reflect the paper’s contributions and scope, while Section 6 discusses limitations and potential negative societal impacts.Source: Checklist · p. 14
  • Code and documentation were uploaded as a supplemental zip and made available in a public repository; training details are in Appendix A.6.Source: Checklist · p. 14
  • The checklist says error bars represent 95% confidence intervals of the mean unless otherwise specified, and computational resources are described in Appendix A.15.Source: Checklist · p. 14
  • The authors report that new assets are synthetic and mark consent, personally identifiable information, offensive content, and human-subject procedures as not applicable.Source: Checklist · p. 14

A.1 Concept Alignment Score Implementation Details

The Concept Alignment Score (CAS) generalizes concept predictive accuracy to embedding representations by measuring how coherently clusters correspond to known concept labels. Its implementation uses repeated homogeneity evaluations across cluster sizes.

A.1 Concept Alignment Score Implementation Details

  • CAS evaluates embedding concept representations by clustering samples and measuring whether each cluster contains samples with the same concept active or inactive.The score evaluates homogeneity for different cluster sizes using known concept labels.Source: §A.1 · p. 15
  • The homogeneity calculation estimates the conditional entropy of ground-truth concept labels given cluster labels using a contingency table of concept-class and cluster membership counts.Source: §A.1 · p. 15
  • The implementation uses δ = 50 to speed up CAS computation across datasets.Source: §A.1 · p. 15

A.2 Kernel Density Estimation of Mutual Information

The appendix estimates mutual information with kernel density estimation, treating noisy concept-encoding activations as a Gaussian mixture. It distinguishes input, task-label, and multi-label concept estimates and scales noise with representation dimension.

A.2 Kernel Density Estimation of Mutual Information

  • The KDE estimate assumes concept-encoding activity is a Gaussian mixture and adds Gaussian noise to the bottleneck activation vector to bound mutual information with the input.The approximation depends on evaluation samples being representative of the true input distribution.Source: §A.2 · p. 15
  • For input mutual information, the cited approach neglects conditional entropy because a neural-network layer is a deterministic function of its input; task-label mutual information retains a nonzero conditional-entropy term.Source: §A.2 · p. 15
  • Because concepts are generally multilabel, their mutual information estimate averages conditional entropies across concepts rather than using the mutually exclusive-label estimate used for task labels.The passage defines Pa,m as the number of samples with concept a taking value m and pa,m = Pa,m/n.Source: §A.2 · p. 16
  • The noise variance is set to σ^2 = ζ/100 to account for representation dimensionality, and estimates computed with natural logarithms are in nats.Here ζ is the dimension of the concept-encoding layer.Source: §A.2 · pp. 15–16

A.3.1 XOR problem

The XOR experiment draws two-dimensional inputs uniformly, thresholds each coordinate into a binary concept, and defines the task label as their XOR.

A.3.1 XOR problem

  • Uniform samples from the unit square yield two binary concepts by thresholding coordinates at 0.5, and the task label is their XOR.The concepts are c1 = 1[x1>0.5] and c2 = 1[x2>0.5]; y = c1 ⊕ c2.Source: §A.3.1 · p. 16

A.3.2 Trigonometric dataset

The appendix evaluates CEM on synthetic Trigonometric and Dot tasks and real-world CUB and CelebA data, examining performance, interventions, annotation scarcity, representations, and computational cost. It also assesses how embedding size affects performance and interventions and notes the use and public release of open-source software.

A.4 Effect of Concept Encoder Capacity

  • Using ResNet18 instead of ResNet34 reduces performance similarly across baselines, preserving their relative rankings on CUB and CelebA; CEM’s RandInt probability of pint = 0.25 provides good performance without interventions while enabling effective interventions.Intervention rankings with ResNet18 are reported to match those observed with ResNet34.Source: §A.4 · p. 18, Figure A.7 · p. 18, §A.5 · pp. 18–19

A.7 Task and Mean Concept Performance

  • Baselines achieve similar mean concept accuracy but distinct task accuracies; in CUB, No Concepts has higher mean CAS than Bool and Fuzzy CBMs, unlike CelebA, where black-box DNNs have lower CAS than CBMs.The authors attribute this CUB pattern to class-specific concept activations producing coherent representation clusters, a condition they say does not hold in CelebA.Source: §A.7 · p. 20, Figure A.10 · p. 20, Table 1 · p. 20

A.8 Concept Subsampling in CUB

  • With fewer CUB concept annotations, CEMs and Hybrid-CBMs’ task and concept accuracy are only mildly affected, unlike Bool and Fuzzy CBMs, although Hybrid-CBMs still do not respond effectively to human interventions.The authors describe this robustness as allowing a dramatic reduction in required concept annotations and their acquisition costs.Source: §A.8 · p. 21, Figure A.11 · p. 21

A.9 Bottleneck Representation Experiment Details

  • With 28 of 112 CUB concepts supervised, linear probes for the remaining 84 concepts achieve 91.83% ± 0.51% mean concept accuracy from Hybrid-CBM bottlenecks and 94.33% ± 0.88% from CEM bottlenecks.The trained models’ test task and mean concept accuracies were 77.15% ± 0.33% and 95.3% ± 0.31% for Hybrid-CBM, and 76.76% ± 0.27% and 95.47% ± 0.19% for CEM.Source: §A.9 · p. 21

A.10 More Qualitative Results

  • CEM concept embeddings appear less visually entangled than Hybrid-CBM embeddings, and adding concept probability to Hybrid-CBM embeddings still leaves them far from separable by ground-truth concept activation.The reported nearest-neighbor coherence also appears across different learned concept representations.Source: §A.10 · pp. 22–23, Figure A.12 · p. 22, Figure A.13 · p. 22

A.11 Computational Cost of CEM

  • CEM requires more FLOPs per training epoch than vanilla CBMs, but the reported increase is less than 10% on CUB and CelebA, and its training convergence time is not statistically significantly different from other baselines.RandInt adds little practical training time because its subroutine can be implemented as a multiplicative Bernoulli mask.Source: §A.11 · p. 23, Figure A.15 · p. 24

A.13 Intervention Experiment Details

  • In concept-incomplete CelebA, CEMs outperform Sequential and Independent CBMs by a large margin after correct interventions, while in complete CUB, most Fuzzy-CBM training regimes improve with correct interventions but can drop quickly when interventions are incorrect.The appendix also reports that CEM’s intervention advantage over Hybrid-CBMs persists when Hybrid-CBMs use RandInt; RandInt appears to hurt standard CBMs on incomplete tasks but slightly help on complete ones.Source: §A.12 · pp. 23–24, Figure A.16 · p. 24, §A.13 · p. 25

A.14 Embedding Size Ablation Study

  • With around 8–16 activations per embedding, CEMs perform better or competitively against equal-capacity black-box models and Hybrid-CBMs; except at m = 2 in CUB, they match or outperform black-box models, and except at very small sizes they outperform Hybrid-CBMs.On CUB, increasing RandInt probability slightly trades off validation task accuracy for improved intervention capability; the authors choose pint = 0.25 as a balance.Source: §A.5 · pp. 18–19, Figure A.9 · p. 19, §A.14 · p. 25
  • Unless embedding size is drastically constrained (e.g., m ≤4), CEM performance is stable, and concept interventions can let it surpass end-to-end black-box models.When end-to-end models outperform CEM, as for m = 2 on CUB, the task-accuracy difference is less than 1.5%; intervention effects are similar across moderately sized embeddings.Source: §A.14 · p. 26

A.15 Code, Licences, and Resources

  • Experiments used Python 3.7, PyTorch 1.11, Skelearn, and Matplotlib 3.5; code to recreate them was released in an MIT-licensed public repository.The libraries are identified as open source and BSD-licensed.Source: §A.15 · p. 26
  • Experiments ran on a private machine with 8 Intel Xeon Gold 5218 CPUs, 64GB RAM, and 2 Quadro RTX 8000 GPUs, using approximately 240 GPU hours.Source: §A.15 · p. 26
Loading 2209.09056v2…