Source-linked AI summary
When Are Concept Bottleneck Model Explanations Faithful and Compact?
Stefano Teso, Emanuele Marconato, Steve Azzolin, Antonio Vergari
TL;DR
CBM explanations are often heuristic, yet stakeholders need explanations that are both compact and faithful to the model’s decisions. The paper analyzes when such explanations are possible, proposes probabilistic concepts and group-lasso sparsity, and adapts exact explainers. It finds that many existing architectures require the full bottleneck for faithful explanations, while the proposed strategies can yield more compact ones.
Problem
Method
The paper analyzes how activation spaces and inference weights constrain faithful explanations, proposes probabilistic concepts and group-lasso sparsity, and adapts exact explainers for CBMs.
Source: §4 · p. 4, §3 · p. 4, §4.2 · p. 5Results
Many existing CBMs admit no faithful explanation smaller than the full bottleneck, while probabilistic modeling and structured sparsity enable substantially more compact weak AXps and EA avoids failures of heuristic strategies.
Source: §4 · p. 4, §5.3 · p. 8, §7 · p. 9Takeaways & Limitations
Large concept bottlenecks should not be equated with faithful interpretability; compact faithful explanations depend on model constraints and explanation methods.
Source: §4 · p. 4, §7 · p. 9The proposed strategies do not ensure small AXps for large bottlenecks, and excessive sparsification can lower downstream accuracy.
Source: §4.2 · p. 5
Abstract
from arXiv · showhide
Concept bottleneck models (CBMs) are neural classifiers that allow to explain their decisions via high-level concepts, potentially enabling understanding, steering and debugging. However, their explanations are often derived heuristically. Building on formal explainability, we argue they should also be faithful, i.e., not misreport which concepts actually matter. We show that, for widespread CBM architectures, including recent VLM-based variants, faithful explanations must include all concepts in the bottleneck, compromising interpretability when this is large. This result applies to both heuristic and faithful-by-construction formal explanations. To encourage the existence of compact faithful explanations, we suggest i) modeling concepts probabilistically as binary or categorical random variables (rather than logits), and ii) employing per-concept training-time sparsification via group lasso (rather than regular elastic net). We also extend algorithms from formal explainability to CBMs, and show they outperform natural heuristics in terms of guarantees and explanation size. Overall, our work warns against naive interpretability claims and provides formal conditions and practical strategies for ensuring CBMs are as interpretable as advertised.
1 Introduction
Concept bottleneck models (CBMs) aim to support understanding, steering, and debugging through high-level concepts, but their explanations need to be both compact and faithful. The paper examines when this is possible and introduces architectural and algorithmic strategies for compact faithful explanations.
1 Introduction
- Reliable CBM interaction requires explanations that are compact enough to understand and faithful enough not to mislead stakeholders.Bottlenecks may contain hundreds or thousands of activations, beyond what stakeholders can grasp.Source: §1 · p. 1
- Heuristic explanations can support different predictions with identical selected activations, so they may omit concepts the model uses.Figure 1 illustrates this with two bird-class predictions supported by the same selected activations.Source: §1 · pp. 1–2, Figure 1 · p. 2
- For many regular and VLM-based CBMs, faithful explanations cannot be compact unless the model is sufficiently constrained.The paper proposes probabilistic concept activations and group-lasso sparsification, and extends formal explainers to favor compactness while retaining faithfulness guarantees.Source: §1 · p. 2
2 Concept Bottleneck Models
CBMs map inputs to a bottleneck of concept activations, then map those activations to class scores. They can be trained with concept and label supervision, while VLM-aided variants provide an alternative when concept annotations are unavailable.
2 Concept Bottleneck Models
- A CBM uses a neural backbone to map an input to concept activations, then an inference layer to map them to per-class logits.The prediction is the class with the highest score; a common inference layer is linear.Source: §2 · p. 2
- CBM training uses losses for task labels and concepts, with independent, sequential, and joint training schemes differing in how modules are fit.Concept loss aligns learned concepts to human semantics, though the paper notes possible failure cases.Source: §2 · p. 3
- VLM-aided CBMs can obtain weak concept supervision by matching LLM-generated concept descriptions with images using a vision-language model.The resulting activations can train a backbone or be used directly for inference.Source: §2 · p. 3
3 Compactness and Faithfulness
CBM explanations must balance compactness with faithfulness: selecting a small subset by heuristic ranking does not guarantee that it justifies the prediction. The section adapts exact explainers to find subset-minimal faithful explanations.
3 Compactness and Faithfulness
- Common compactness heuristics select concepts by activation, class-weight magnitude, or their product, but provide no faithfulness guarantees for common k values.Taking the whole bottleneck as the explanation is often impractical because vocabularies can be large.Source: §3 · pp. 3–4
- Existing heuristic explanations may be unfaithful because the same selected activations can be compatible with different predicted classes.For example, the figure’s selected concepts support both Vesper Sparrow and Black-billed Cuckoo predictions.Source: §3 · p. 3
- A weak abductive explanation (AXp) is sufficient when matching its concepts rules out a counterexample with a different class; an AXp is also subset-minimal.Subset-minimality means no proper subset remains sufficient.Source: §3 · p. 4
- The exact algorithm EA checks whether candidate explanations admit a counterexample, using a linear program as its oracle for linear CBMs.EA+ grows a candidate using heuristic-informed groups, shrinks it to subset-minimality, and selects the smallest of three resulting AXps.Source: §3 · p. 4
4 The Trade-off Between Faithfulness and Compactness
Whether faithful explanations can be compact depends on the attainable activation space and inference-layer weights. The paper identifies common architectures that fail these constraints and proposes probabilistic concepts and group lasso to encourage smaller explanations.
4 The Trade-off Between Faithfulness and Compactness
- With unbounded activations and a nonzero weight difference outside an explanation, that explanation cannot be a weak AXp.For regular CBMs with raw-logit activations and dense linear layers, no faithful explanation smaller than the full bottleneck exists, regardless of algorithm.Source: §4 · p. 4
4.1 Existing CBMs vs. Proposition 1
- Many standard and VLM-based CBMs use unbounded activations, while elastic-net sparsity can still leave nonzero class-weight differences that prevent compact faithful explanations.LaBo’s bounded cosine-similarity activations alone do not guarantee compact explanations unless its inference weights are also constrained.Source: §4.1 · p. 5
4.2 Beyond Proposition 1 with Probabilistic Concepts and Group Lasso
- Probabilistic binary and categorical concepts bound activations, and categorical mutual exclusivity further limits counterexample activations.For a three-value color concept, if the red probability is 0.95, only the residual 0.05 probability mass can shift between green and blue when red is fixed.Source: §4.2 · p. 5
- Group lasso treats each activation’s weights as a group, encouraging concepts to contribute to no class and weight differences to become zero.This structured sparsity is intended to constrain the inference layer more directly than unstructured sparsity.Source: §4.2 · p. 5
- The proposed strategies shrink weak AXps, but excessive sparsification can lower downstream accuracy and neither strategy guarantees small AXps for large bottlenecks.Probabilistic modeling is reported not to incur an accuracy cost, while sparsification should be applied with care.Source: §4.2 · p. 5
5 Experiments
The experiments test whether probabilistic concepts and group-lasso sparsification enable compact faithful explanations, and compare EA+ with TopK heuristics. Probabilistic activations shrink explanations without a substantial F1 cost, while group lasso and EA+ show advantages in the reported comparisons.
5 Experiments
- Experiments span four datasets and compare regular CBMs with logit, ReLU, and probabilistic activations, plus VLM-CBMs, using a frozen OpenCLIP encoder and logistic inference layers.The inference layers are dense or sparsified with enet or glasso; the dataset set includes binary and categorical concepts.Source: §5 · p. 6
5.1 Q1: Probabilities Shrink Faithful Explanations At No Accuracy Cost
- ≤9 concepts on Derm7pt, ≤10 on WBCAtt, and ≤60 on CUB suffice for EA+ explanations with probabilistic activations, versus logit bottlenecks of 28, 31, and 312, with a worst-case F1 gap of 2 pp.For logits, no weak AXp smaller than the full bottleneck exists; probabilistic concepts shrink explanations without hindering accuracy.Source: §5.1 · pp. 6–7, Figure 2 · p. 7
5.2 Q2: Group Lasso Outperforms Elastic Net
- Elastic net sometimes increases explanation size, including by 58.4 pp for TopK-Wei and 68.5 pp for TopK-Prod on CUB; at α = 0.99, larger non-zero weights can let smaller activation changes flip predictions.For VLM-CBMs, enet often fails to shrink explanations, except for EA+ on LF-CBM (−16.6 pp), VLG-CBM (−30.7 pp), and LaBo (−52.7 pp).Source: §5.2 · p. 7
- Glasso consistently shrinks faithful explanations against the dense baseline and substantially outperforms enet on VLM-CBMs: EA+ sizes are 17.5% versus 82.9% for LF-CBM, 15.9% versus 68.7% for VLG-CBM, and 6.5% versus 46.9% for LaBo.Glasso’s EA+ reductions versus dense models are 22.3 pp on Derm7pt, 18.3 pp on WBCAtt, and 7.7 pp on CUB; it loses to enet by small margins in two reported cases.Source: §5.2 · pp. 7–8, Table 5 · p. 8
5.3 Q3: EA+ finds smaller faithful explanations
- EA+ finds smaller weak AXps at lower size budgets than TopK heuristics; for enet-sparsified VLM-CBMs, heuristic explanations average over 99% of the bottleneck, versus 82.9% for LF-CBM, 68.7% for VLG-CBM, and 46.9% for LaBo.On glasso-sparsified VLM-CBMs, EA+ sizes are 17.5%, 15.9%, and 6.5% for LF-CBM, VLG-CBM, and LaBo, respectively; TopK-Act and TopK-Prod benefit from categorical concepts’ mutual exclusivity, but that advantage disappears for binary concepts and VLM-CBMs.Source: §5.3 · p. 8
6 Related Work
This section distinguishes explanation faithfulness from concept alignment and positions the work among formal-XAI and CBM studies. It extends prior observations and identifies limits of existing algorithms and the scope of its own approach.
Related work
- Formal XAI seeks explanations with formal guarantees, but most methods focus on discrete features and therefore do not apply to CBMs with continuous concepts.The authors plan to investigate discrete-concept CBM variants in future work; algorithms for size-minimal AXps of linear classifiers are restricted to binary tasks.Source: §6 · p. 8
- Prior work on VLM-CBMs found that top-5 heuristic explanations can allow prediction flips; this work systematizes that observation and identifies architectural choices that support more compact weak AXps.Soria et al. [2026] introduced FXAI algorithms for prototype neural nets; their activation-ranking approach performs poorly for independent activations and VLM-CBMs.Source: §6 · p. 8
- Prior CBM transparency work studies whether learned concepts align with human concepts, whereas this work studies whether explanations faithfully identify the concepts determining predictions.The section distinguishes explanation faithfulness from concept alignment, which some works call “concept faithfulness.”Source: §6 · p. 9
7 Conclusion
The paper adapts formal explainability to CBMs and finds that many existing models do not admit compact faithful explanations. It describes algorithmic and modeling strategies for obtaining smaller faithful explanations.
7 Conclusion
- Many existing CBMs do not admit compact faithful explanations, while probabilistic modeling and structured sparsity enable substantially more compact weak AXps.Existing heuristics often fail to identify even weak AXps, whereas the exact algorithm does.Source: §7 · p. 9
A.1 Finding a Counterexample
- For each alternative class, the counterexample oracle solves a linear program and returns a valid counterexample only when its objective is positive.The candidate vector must preserve activations covered by the explanation and yield a different prediction.Source: §A.1 · p. 14
A.2 The Heuristic Algorithms
- TopK heuristics rank concepts by activation, absolute weight, or absolute score, but fixed-size selections are not guaranteed to be faithful.TopK* increases the prefix size until it finds a weak AXp, or returns the full bottleneck.Source: §A.2 · p. 14
A.3 The EA algorithm
- EA shrinks the full bottleneck into a subset-minimal weak AXp, making it an AXp, at the cost of exactly b oracle calls.It tests removing each activation in turn.Source: §A.3 · p. 15
A.4 The EA+ algorithm
- EA+ runs TopK* for three heuristics, shrinks each resulting weak AXp, and returns the smallest resulting AXp.Its binary-search variant reduces oracle calls for each heuristic from b to O(log2 b).Source: §A.4 · p. 15
B.1 The Independent Concept Activations Case
- For independent activations, an explanation smaller than the bottleneck is a weak AXp only under specific conditions on omitted-concept weights and the activation domain.The conditions differ across unbounded, lower-bounded, and bounded activation spaces.Source: §B.1 · p. 17
4. Bounded asymmetric: (S =×
- With unbounded activations, nonzero weight differences on omitted concepts can enable a label flip; bounded asymmetric domains instead test whether an alternative class can achieve a positive logit difference.A flip occurs exactly when the maximum alternative-class logit difference is positive.Source: §4 · pp. 17–18
B.2 The Probabilistic Concept Activations Case
- For categorical probabilistic activations, values of each concept share a simplex constraint, and a label flip is possible exactly when the maximum alternative-class gain is positive.This differs from the independent-activation case because alternative values of one categorical concept must sum to one.Source: §B.2 · p. 19
B.3 Discussion
The activation space and inference weights determine how many concepts a faithful explanation must cover. Bounding activations or reducing relevant weight differences can limit counterexamples, while probabilistic activations impose a stricter label-flipping condition.
B.3 Discussion
- With unbounded logits, a weak AXp must cover every activation outside the explanation having a nonzero weight difference to an alternative class; dense CBMs therefore produce large AXps.Group lasso can reduce nonzero weight differences and thereby shrink weak AXps.Source: §B.3 · p. 19
- For activations bounded below, AXps must cover concepts with positive weight differences, while negative differences may remain uncovered.Because CBMs do not control weight polarity during training, group lasso can reduce both positive and negative differences; the authors report improvements in Section D.5.Source: §B.3 · pp. 19–20
- In bounded symmetric settings, smaller activation ranges and weight differences leave less room for changing concepts to flip a label; the asymmetric case conveys essentially the same message.Cosine-similarity VLM activations lie in [−1, 1], while probabilistic activations lie in [0, 1]; elastic net and group lasso also shrink weight ranges.Source: §B.3 · p. 20
- Probabilistic activations impose a stricter label-flipping condition than purely bounded activations because their product-of-simplices structure restricts counterexamples.With the activation space fixed, reducing learned weight magnitudes is the available way to reduce counterexamples; the authors suggest future penalties targeting proposition quantities.Source: §B.3 · p. 20
C Experimental Details
The experimental setup compares explanation algorithms across probabilistic CBMs and cosine-similarity VLM-CBMs. Its counterexample analysis permits independently manipulated activations, including out-of-distribution cases, while retaining the categorical constraints of probabilistic concepts.
C.1 Concept Activation Spaces: In- vs. Out-of-distribution Counterexamples
- Each model is tested in its corresponding activation space, from unbounded logits and nonnegative ReLU activations to cosine similarities in [−1, 1] and probabilities within [0, 1].For probabilistic categorical concepts, activations within a concept sum to 1; other uncovered activations are manipulated independently.Source: §C.1 · p. 22
- The analysis treats explanations as needing robustness to both natural and stakeholder-constructed counterexamples, because restricting guarantees to natural cases requires a tractable approximation of the bottleneck's attainable outputs.Focusing only on natural counterexamples could in principle shrink AXps, but would weaken their guarantees.Source: §C.1 · p. 22
D.1 Extra: Algorithmic Comparison
- The comparison focuses on probability-based CBMs and cosine-similarity VLM-CBMs; CelebA is evaluated only with dense models because it has no categorical concepts for group lasso.The algorithmic comparison uses 100 fixed random test instances and considers explanation size, computational cost, and faithful subset-minimal outputs.Source: §D.1 · p. 23
D.1.1 EA+ Outputs Smaller AXps than EA
EA+ produces smaller AXps than EA and, despite sometimes making more oracle calls, remains computationally affordable in the reported experiments. Its subset-minimal AXp guarantee distinguishes it from heuristic alternatives.
D.1.1 EA+ Outputs Smaller AXps than EA
- EA+ always produces smaller AXps than EA; reductions are more noticeable for dense CBMs, with CUB dropping from nearly 170 to about 46 activations, while glasso-sparsified CUB drops from about 47 to 24.For VLM-CBMs, size differences are minimal for dense models, while sparse-model improvements are about 1.5 or 11.5 activations depending on vocabulary.Source: §D.1.1 · pp. 23–24
- EA+ can make up to three times as many oracle calls as EA on dense VLM-CBMs, though it makes fewer calls in some CUB settings.The dense VLM cases require individual tests during shrinking because their concept activations are independent.Source: §D.1.1 · p. 24
D.1.2 EA+ Is Computationally Affordable
- Unlike TopK heuristics, EA+ is guaranteed to return an AXp, and its outputs are also reported as smaller than those from the heuristics.The comparison uses TopK*bs, which automatically selects k to ensure a weak AXp and uses binary search to reduce oracle calls.Source: §D.1.2 · p. 24
- EA+ uses a logarithmic number of oracle calls to grow an explanation, then a linear number to ensure subset minimality; its reported worst case is 11 seconds on dense CUB.The additional calls arise from the checking phase, but the authors describe the runtime as affordable.Source: §D.1.2 · p. 24
D.1.3 EA+ Finds The Most Compact Explanations
Across regular and VLM-CBMs, EA+ yields the smallest average explanations among the compared algorithms. Sparsification sharply reduces explanation sizes for most algorithms, while dense cosine-similarity VLM-CBMs retain large explanations and small gaps between methods.
D.1.3 EA+ Finds The Most Compact Explanations
- EA+ achieves the smallest average explanation sizes across regular CBMs, with dense-model averages of 12.6 on CelebA, 8.3 on Derm7pt, 8.2 on WBCAtt, and 46.3 on CUB.On glasso models, its advantage over the runner-up is smaller on Derm7pt and WBCAtt, but about five activations on CUB.Source: §D.1.3 · p. 25, Table 10 · p. 25
- For VLM-CBMs, EA+ is smallest across the board on sparse models, by an average of 4 activations for LF-CBM and 5 for VLG-CBM.Dense VLM-CBMs produce large explanations with small differences between algorithms; the passage attributes this to cosine-similarity activations.Source: §D.1.3 · p. 25, Table 11 · p. 25
- EA+ offers a modest size advantage over competitors but is described as the most reliable alternative, with no obvious ranking among heuristics except for poor TopK-Act interactions with sparsification.The reported comparisons cover regular and glasso-sparsified CBMs as well as VLM-CBMs.Source: §D.1.3 · pp. 24–25
D.1.4 EA+ Always Finds AXps, Heuristics Very Often Do Not
Across these extended experiments, exact explanations outperform heuristics in subset-minimality, while probability modeling and group-lasso sparsification generally produce more compact explanations. Even after sparsification, faithful VLM-CBM explanations may remain too large for human interpretation.
D.1.4 EA+ Always Finds AXps, Heuristics Very Often Do Not
- EA+ finds subset-minimal AXps in every tested instance, whereas heuristics never reach 100%; their best dense-CBM rate is 37%, and the best glasso-CBM heuristic reaches only 1% on CUB.On Derm7pt, weight-based heuristics reach up to 85% with glasso, but performance remains dataset-dependent.Source: §D.1.4 · pp. 25–26, Table 12 · p. 25
D.2 Extended Q1: Logits vs. ReLUs vs. Probabilities
- Across datasets, probability activations enable more compact AXps than logits, with ReLU activations in between.CelebA’s binary concepts lack the simplex constraint present in categorical concepts, leaving the adversary less constrained and disadvantaging TopK-Act and TopK-Prod.Source: Figure 3 · p. 27, §D.2 · p. 27
D.3 Extended Q2: enet vs. glasso for Probabilities
- For probabilistic CBMs with categorical concepts, glasso improves compactness more consistently across algorithms than elastic net, while TopK-Act benefits less from either penalty.Elastic net can make TopK-Wei and TopK-Prod misbehave by learning larger non-zero weights.Source: §D.3 · p. 28, Figure 4 · p. 28
D.4 Extended Q2: Sparsity on VLM-CBMs
- Glasso shrinks explanations across VLM-CBM vocabularies for all algorithms except TopK-Act, while enet improves EA+ but not heuristic explanation sizes.For EA+ with enet, weak AXps are about 160 of 202 concepts for LF-CBM, 300 of 419 for VLG-CBM, and 5,000 of 10,000 for LaBo.Source: §D.4 · p. 29, Figure 5 · p. 29
- Despite glasso’s improvements, faithful VLM-CBM explanations of size 100 for VLG-CBM and up to 500 for LaBo are unlikely to be human-interpretable.The authors leave extensive evaluation of ways to address this limitation to future work.Source: §D.4 · p. 29
D.5 Extra: SelectiveCBM
Selective CBMs combine input-dependent gating with elastic-net sparsification, but their explanation sizes resemble ReLU CBMs and lag appropriately sparsified CBMs. From the AXp perspective, their gating remains equivalent to ReLU behavior.
D.5 Extra: SelectiveCBM
- Selective CBMs sparsify activations per input through gating and weights across inputs through an elastic-net penalty; the comparison uses CelebA and WBCAtt.The study compares them with dense ReLU CBMs and sparsified CBMs, using enet for CelebA and glasso for WBCAtt.Source: §D.5 · p. 30
- Selective CBMs improve AXp size only moderately over regular CBMs and underperform appropriately sparsified CBMs, with TopK-Act lagging the most.Their improvement chiefly benefits EA+, while the heuristics continue to perform poorly.Source: §D.5 · p. 30
- For AXps, SelectiveCBM’s learnable gate is equivalent to ReLU: neither prevents counterexamples when increasing a concept activation can flip the prediction.The counterexample threshold is zero for ReLU and a possibly non-zero positive learned threshold for Selective CBMs.Source: §D.5 · p. 30