Source-linked AI summary
Are Concept Bottleneck Models Effective as Decision-Support Systems?
Alessandro Bogani, Nicola Debole, Emanuele Marconato, Andrea Pugnana, Katya Tentori, Andrea Passerini
TL;DR
Evidence for CBMs as decision-support systems has been limited, especially regarding whether concept inspection and intervention improve human–AI collaboration. The paper addresses this gap with two large-scale user studies across contrasting binary classification tasks and finds benefits mainly when tasks are difficult, concepts are easy to identify, and users interact with the model.
Problem
Evidence remains limited on whether CBMs are effective decision-support systems and how inspecting or intervening on concepts affects reliance, confidence, and trust.
Method
Two user studies evaluate CBMs across familiar subjective email classification and unfamiliar objectively grounded bird classification, with progressively richer support conditions.
Results
CBMs, particularly their interactive component, can improve human–AI team performance beyond unaided humans and non-interpretable AI support under specific conditions.
Takeaways & Limitations
Benefits are more likely when users are uncertain, concepts are easily identifiable, and users actively engage with the interactive component.
Takeaways & Limitations
The studies examine only binary classification, leaving CBM effectiveness in multi-class settings unexplored.
Abstract
from arXiv · showhide
Concept Bottleneck Models (CBMs) are interpretable-by-design neural networks that detect human-understandable concepts from the input and use them to generate predictions. By allowing users to inspect the concepts underlying a prediction and explore how predictions change under alternative concept configurations, CBMs have emerged as one of the most prominent approaches to supporting human-AI collaboration. However, user studies investigating their actual effectiveness as decision-support systems remain limited. We present two large-scale user studies (N participants = 705, N observations = 6,959) evaluating how concept-based explanations and user interventions on the model's concepts affect the performance of the human-AI team in two distinct binary classification tasks. Our results show that CBMs, and particularly their interactive component, can improve human-AI team accuracy relative to both unaided human performance and performance with non-interpretable AI support. However, these benefits emerge only under certain conditions: classification tasks perceived as difficult, easily identifiable concepts, and active interaction with the model. We also discuss how inaccurate concept detection may undermine users' trust in the model. Overall, this work provides practical guidance for the deployment of CBMs as effective decision-support tools.
1 Introduction
CBMs address limited AI interpretability by exposing human-understandable concepts and enabling users to examine alternative concept configurations. This work studies when these features improve human–AI collaboration.
- CBMs base predictions on human-understandable concepts, addressing interpretability concerns in AI systems.
- Interactive concept inspection lets users assess how predictions change when concept configurations are modified.
- Empirical evidence remains limited on whether CBMs improve decision support and affect reliance, confidence, or trust.
- Two user studies compare CBM support across familiar, subjective email classification and unfamiliar, objectively grounded bird classification.
- CBMs, particularly their interactive component, can outperform unaided humans and non-interpretable AI support under specific task and interaction conditions.
- The work contributes an experimental paradigm for evaluating accuracy, decision confidence, and trust in human–AI collaboration.
2 Background
CBMs decompose prediction into concept detection followed by label prediction. Their modular structure also permits users to override detected concepts and obtain an updated prediction.
- A concept encoder g maps an input x to binary concept activations c, and a task predictor f maps those concepts to a label y.
- CBM components may be trained independently, sequentially, or jointly.
- At inference, users can inspect g(x), replace selected concepts with a modified vector ˜c, and update the prediction as ˜y = f(˜c).
3 Present Work
The paper evaluates CBMs in two binary classification studies using datasets with six human-interpretable concepts and experimental conditions ranging from no support to interactive concept support. Participants classify items while reporting confidence, and intervention and trust are measured where applicable.
- Two studies test CBM decision support for fraudulent-versus-legitimate emails and Le Conte’s-versus-Savannah sparrow images.
- Each dataset supplies labels and interpretable features, with six selected concepts used to reduce participants’ cognitive load.
- The email concepts include problem alert, time pressure, attachment interaction, update notification, operational tone, and reply request.
- The bird concepts include warm-colored eyebrow, warm-colored chest, plain sides, crested head, white throat, and striped chest.
- The implementation combines a deep concept encoder with binary SVM concept classifiers and a logistic-regression task predictor.
- The CBMs achieve 92.3% test accuracy on PhishFuzzer and 81.4% on CUB.
- Conditions range from no support and label-only support to non-interactive and interactive concepts, with users able to modify concept values only in IC.
- Participants classify 10 items, rate confidence, and receive no response-accuracy feedback; measured outcomes include accuracy, confidence, intervention count, and model trust.
4 Study 1 – Fuzzy concepts in a familiar task
In the familiar email-classification task, concept-based support improved accuracy over no support, but interaction did not add a significant accuracy benefit. Participants intervened more often when concepts were incorrect, while interactive support may have slightly reduced trust.
- Accuracy: When the CBM classified items incorrectly, accuracy was 68% with detected concepts versus 55% with label-only support.This interaction between support condition and CBM accuracy was significant (p = .002).
- Interaction: Participants intervened more often when detected concepts differed from ground truth (22%) than when they matched (9%), but interventions did not improve accuracy.Accuracy was 82% with at least one intervention and 83% without intervention (p = .248).
- Interaction: 36% of participants in the interactive condition never intervened, although excluding them did not alter classification-accuracy results.
- Confidence and trust: The support conditions did not significantly affect participants’ classification confidence or overall reported trust in the model.Confidence differences were nonsignificant (p = .588), and no AI-supported trust-item comparison reached significance before excluding non-interactive participants.
- Confidence and trust: Interactive concept support may have reduced trust in the model’s capabilities, with lower ratings than label-only support on one trust item.Median ratings were 3.5 in the interactive condition and 4.0 in the label-only condition (p = .007).
5 Study 2 – Well-defined concepts in an unfamiliar task
In the unfamiliar sparrow-classification task, concept-based support improved accuracy over no support, and active interaction produced higher accuracy than label-only support among participants who interacted. Interactive support also increased confidence, while inaccurate concept information remained a trust concern.
- Task characteristics: The study used a less familiar task with visual concepts intended to be more readily observable in the images.
- Accuracy: 82% accuracy with detected concepts and 83% with interactive concepts exceeded 73% with no support, whereas 79% with label-only support did not.The two concept-support conditions did not significantly differ from label-only support in the full sample.
- Task characteristics: CUB concept annotations were assigned at the species level rather than manually verified for each image, so selected images were restricted to clearly visible concepts and manually verified annotations.
- Interaction: Participants intervened more often when detected concepts differed from ground truth (33%) than when they matched (5%), and intervention trials reached 88% accuracy versus 79% without intervention.The intervention-frequency difference and accuracy difference were both significant (p < .001).
- Accuracy: After excluding non-interactive participants, interactive-support accuracy rose to 86% and exceeded label-only support, driven mainly by resistance to incorrect CBM labels.For incorrectly predicted items, accuracy was 65% versus 60%; for correctly predicted items, it was 91% versus 89%.
- Interaction: 35% of participants in the interactive condition never interacted with the concepts.
- Confidence and trust: Interactive support increased confidence relative to no support, with median ratings of 4 versus 3.Other pairwise confidence comparisons were not significant.
- Confidence and trust: Overall trust did not differ significantly across AI-supported conditions, but interactive participants gave lower ratings on one perceived-capability item than label-only participants.The medians were 4 and 5, respectively (p = .005), after excluding non-interactive participants.
6 Related Work
Prior work finds concept-based explanations interpretable and sometimes helpful, but its relevance to CBM decision support is constrained by task design, limited samples, and nonrepresentative applications. This paper addresses these gaps with larger studies in binary classification tasks.
- Concept Bottleneck Models: Recent CBM research has examined human-aligned concepts, sufficient task statistics, and interpretable task predictors.
- Concept-based XAI user studies: Prior user studies generally judged concept-based explanations adequate, informative, easy to interpret, and helpful for predicting or critically evaluating model outputs.
- Concept-based XAI user studies: Das et al. reported improved performance with concept-based explanations, but studied sequential game decisions rather than typical CBM classification problems.
- Concept-based XAI user studies: Furby et al. found some evidence that concept interaction can improve human-AI team performance, particularly when model accuracy is relatively low.Their study had 13 participants per condition, used a nonrepresentative Blackjack task, and required modification of inherent model features.
7 Conclusions
CBMs, particularly their interactive component, can improve human–AI team performance, but these benefits are most likely under specific task, concept, and engagement conditions.
- CBMs, especially their interactive component, can improve human–AI team performance beyond non-interpretable AI support and unaided human performance.
- Benefits are more likely when users are uncertain about the correct classification.Users confident in their own classifications have less reason to rely on CBM explanations.
- Benefits are more likely when concepts are objectively grounded and easy to understand.Fuzzy, subjective concepts may obscure the model’s rationale and reduce users’ trust.
- Approximately one third of participants in the IC condition never modified any concept values.Providing interaction opportunities alone was therefore insufficient for some users to use the feature.
- The work’s scope is limited to binary classification, leaving CBM effectiveness in multi-class settings unexplored.Multi-class tasks may reduce confidence but require more concepts and potentially increase cognitive load.
CBM specifics
The CBM uses separately trained concept extraction and task prediction, transforms concept scores into bounded activations, and permits users to toggle concepts during inference.
- The CBM independently trains its concept extractor and task predictor.
- The concept extractor maps inputs to concept activations using a frozen encoder followed by one binary SVM per concept.Each SVM outputs a decision score representing concept presence or absence.
- The image and text settings use different pretrained encoders, but the subsequent concept-extraction pipeline is identical.CUB uses CLIP ViT-L14, whereas PhishFuzzer uses a pretrained sentence transformer.
- At test time, concept logits pass through tanh before the task predictor outputs the positive-class probability.The positive class is predicted at probability 0.5 or above.
- Users can toggle any concept between active and inactive states to override the model’s predicted concept value.The intervention replaces the activation with the opposite polarity and clicking again restores the original value.
Datasets
The studies use separate email and bird-image training data, with concept predictors trained on broader data and task predictors trained on narrower binary-classification subsets.
- The PhishFuzzer concept predictor retains 1,064 cleaned emails, while its task predictor excludes 513 Spam emails.
- The CUB concept predictor uses 4,745 images outside the two target sparrow classes, with 28,470 annotations across six concepts per image.
- Table 1 summarizes statistics for the PhishFuzzer and CUB training datasets.
- The CUB task-prediction data consist of images from the Le Conte’s Sparrow and Savannah Sparrow classes.
A priori power analysis and data collection
The study used simulation-based power analysis and batch recruitment, but Study 2 had substantially more exclusions, especially in the no-support condition.
- The power analysis required 340 participants to detect a small three-way interaction with 82% statistical power.The target corresponded to 85 participants per AI-support condition.
- Study 2 excluded 209 of 551 participants (38%), compared with 38 of 401 (9%) in Study 1.Many exclusions followed failed attention checks, particularly among participants classifying unfamiliar bird species.
- After the first 400 participants, recruitment was adjusted because random assignment would have produced imbalanced conditions and less precise analyses.
Trust scale
The trust scale measured participants’ perceptions of system confidence, predictability, reliability, safety, efficiency, wariness, comparative ability, and decision-making preference using randomized 7-point items.
- Trust was assessed with eight questionnaire items covering confidence, predictability, reliability, safety, efficiency, wariness, comparative ability, and decision-making preference.Three items were reworded to invert their polarity and reduce acquiescence bias.
- The questionnaire included both positively and negatively worded statements about system performance and users’ willingness to rely on it.
- Each trust item used a 7-point Likert scale ranging from strong disagreement to strong agreement.
Previous experience with AI systems
The studies measured prior AI experience and examined confidence, trust, and behavioral responses to concept-detection errors across different CBM support conditions.
- Previous experience with AI systems: Participants reported AI experience ranging from little or no experience to developing AI systems professionally or personally.The four response options distinguished limited familiarity, occasional use, studied understanding, and system development.
- Confidence results: 25% of Study 1 responses received the highest confidence rating, with no significant frequency difference across conditions (p = .117).The lowest confidence rating accounted for 3% of responses and also did not differ across conditions.
- Confidence results: 12% of Study 2 responses received the highest confidence rating, occurring less often in NS than IC (p = .031).The overall condition effect was significant (p = .015), while other contrasts were not significant.
- Impact of inaccurate concept detection on trust: The error analysis was restricted to CUB because annotator agreement was 90% for CUB but 74% for PhishFuzzer.The exploratory effect was small and likely underpowered, with only 0–2 concept errors per trial.
- Impact of inaccurate concept detection on trust: As concept-detection errors increased, participants were less likely to follow correct CBM predictions in NIC (OR = 0.72), whereas errors had little impact in IC (OR = 1.12).The interaction between support condition and detection errors was significant, χ2(1) = 4.12, p = .042.