Human Uncertainty in Concept-Based AI Systems

1 INTRODUCTION

• Concept-based systems let people correct predictions through interpretable concepts, but typically assume interventions are always correct and confident; the paper studies uncertainty-aware interaction as a way to address human error and automation risks.The introduction frames uncertainty as a normal part of human reasoning and argues people should be able to express when they do not know.Source: §1 · p. 1
• The paper identifies failure modes in existing concept-based models under user uncertainty and empirically investigates training with uncertain concept labels as a mitigation for test-time uncertainty.It presents human uncertainty in interactive concept-based models as an understudied safety-critical problem.Source: §1 · pp. 1–2
• The authors develop UElic to collect rich human uncertainty over concepts and use it to create CUB-S, a relabeling of CUB intended as a challenge dataset for uncertain interventions.CUB-S is designed to address limitations of the existing dataset, while UElic is extensible for collecting real-world feedback.Source: §1 · p. 2

2 PRIMER ON CONCEPT-BASED SYSTEMS

• Concept-based models use human-interpretable concepts as intermediate representations for predicting task labels, aiming to improve interpretability and enable people to correct model mistakes.The supervised setting represents examples as input x, concepts c, and output y.Source: §2 · p. 2, §2.1 · p. 2

2.1 Notation

• Concepts are typically binary or categorical, but this work represents each concept as a value in [0, 1] to express its probability given the input.Categorical concepts can also be converted to binary concepts.Source: §2.1 · p. 2

2.2 Models

• The study focuses on joint-bottleneck CBMs and CEMs, which extend CBMs with supervised high-dimensional concept embeddings while still predicting concept values as intermediate steps.Joint-bottleneck CBMs learn input-to-concept and concept-to-output mappings together; CEMs can better use datasets with incomplete concept annotations.Source: §2.2 · p. 2

2.3 Interventions

• An intervention replaces a model’s predicted concept value with the human-provided value and recalculates the prediction; this work also permits uncertain, probabilistic edits.It considers Random and Skyline query policies, using Random unless otherwise noted; Skyline approximates an oracle upper bound on intervention efficacy.Source: §2.3 · p. 2

2.4 Critiques and Common Assumptions

• Although prior critiques identify several weaknesses of concept-based systems, this paper states that it is the first to directly study uncertainty in human users of these models.The cited weaknesses include information leakage, impure concept representations, spurious saliency maps, difficult concept definitions, and sensitivity to concept correlations.Source: §2.4 · p. 2

• How do existing concept-based systems handle human uncertainty introduced at test time?Source: §3 · p. 2
• How can systems be bolstered to better support human uncertainty at test time?Source: §3 · p. 2
• How does uncertainty’s level and form, from discrete annotations to continuous probabilities, affect performance?Source: §3 · p. 2
• The study spans simulated uncertainty in UMNIST and CheXpert [Irvin et al. 2019], coarse-grained scores in CUB, and richer human-elicited uncertainty in CUB-S.Source: §3 · p. 3
• Across datasets, the study compares models trained on binary, certain concepts but tested with uncertainty against the same models trained using uncertainty estimates in concept labels.Source: §3 · p. 3

4.1.1 Data.

• The experiments use CheXpert chest X-rays with 14 concepts, predicting “No Finding” from the other 13, and UMNIST mixtures of digits annotated with their values for a digit-sum task.CheXpert uncertain labels are set to 0.5 and unknown labels to 0; UMNIST’s δ parameter controls annotation uncertainty from fully certain at 0 to random at 1, and the same δ-smoothing is applied to CheXpert.Source: §4.1.1 · p. 3
• Evaluation measures task performance as the number of randomly selected concepts intervened on increases, using AUC for CheXpert and accuracy for UMNIST.Random intervention selection isolates the effect of uncertain interventions rather than evaluating intervention policies.Source: §4.1.2 · p. 3

4.2 Intervening with Uncertainty

• Even low simulated uncertainty causes significant intervention-performance drops in models not trained with uncertain concepts, with sharper declines as uncertainty increases.CEM performance drops particularly sharply at δ=0.6, showing that models trained on certain interventions may not generalize to uncertain users.Source: §4.2 · p. 4, Figure 2 · p. 4

4.3 Training with Uncertainty Can Improve Robustness

• Training with uncertainty can salvage intervention efficacy, particularly under distribution shift, even when the training and user uncertainty levels differ; the helpful level has a “sweet spot.”The results suggest uncertainty-aware training can better equip models for users’ uncertainty than training without uncertainty.Source: §4.3 · p. 4

4.4 Implications

• The results indicate that training with too much concept “softness” can harm performance, despite uncertainty training offering a reasonable way to capture value from uncertain interventions.The section emphasizes robustness under distribution shifts as a benefit of training with uncertainty.Source: §4.4 · p. 4

5.1 Taxonomy of Forms of Uncertainty

• Real human uncertainty may be epistemic or aleatoric and input-dependent or independent; this study distinguishes coarse confidence scales from direct fine-grained probability labels.The authors note that uncertainty expressed flexibly through language is outside the study’s scope.Source: §5.1 · p. 4

5.2.1 Experimental Set-Up.

• In CUB, annotators provide coarse, discrete confidence measures for annotations, so continuous interventions require a custom mapping whose choice affects downstream performance; a single measure cannot assign nuanced uncertainty to individual categorical concepts.The study filters 311 total binary concepts extractable from 28 categorical concepts down to 112 concepts. It evaluates task accuracy on bird-species classification.Source: §5.2.2 · p. 5, §5.2.1 · p. 5

5.2.2 How to Use Discrete Uncertainty Scores?

• Different imputations of “Probably” confidence and assumptions about broad versus narrow uncertainty can markedly change intervention quality.Broad uncertainty may substantially impair interventions by spreading probability over concepts where the annotator was actually certain; using population-level labels also yields markedly higher test-time performance than individual-level labels.Source: §5.2.2 · pp. 5–6, Figure 4 · p. 6, Figure 5 · p. 6

5.2.4 Training with Uncertainty.

• Training with aggregated uncertainty improves performance on softer, potentially noisier individual-level uncertainty, while broad-versus-narrow assumptions can also affect training-label efficacy.The reported improvements span gradations of uncertainty and are shown in Figure 6.Source: §5.2.4 · p. 6, Figure 6 · p. 7

5.3 Fine-Grained Uncertainty

• CUB-S provides nearly 5,000 richly uncertainty-labeled concept groups, addressing the scarcity of datasets with detailed labels over concept space.The authors present it as a formative resource for studying human uncertainty in concept-based models.Source: §5.3 · p. 7

5.3.1 Eliciting Human Uncertainty.

• UElic elicits probabilities over plausible concept attributes, and 89 participants used it to produce 4,984 soft categorical annotations across CUB’s 28 concepts.Participants selected plausible attributes and distributed their perceived probability across those options.Source: §5.3.1 · p. 7, §5.3.2 · p. 7

5.3.3 Richness in CUB-S.

• CUB-S annotations show that human uncertainty can depart sharply from uniform distributions, while annotators vary in calibration and exhibit genuine errors.Annotators tend to underestimate small probabilities and overestimate large ones; the estimated reference concepts are an approximation, not definitive ground truth.Source: §5.3.3 · p. 8
• CUB-S’s fine-grained uncertainty substantially challenges concept-based models; training on coarse-grained uncertainty mitigates some failures, but uncertainty can help or harm depending on the concept.These differences persist across training approaches, motivating adaptive, personalized query procedures and further multidisciplinary work.Source: §5.3.3 · p. 8, §5.3.5 · p. 8

6.1 Complementarity of Human and Machine Uncertainty

• Future systems should choose concepts and query times based on human–model uncertainty complementarity, and learn when interventions deserve trust because humans are not always consistent oracles.The section notes that cases difficult for a model may also be difficult for people, and even experts may lack certainty in situations such as occlusion.Source: §6.1 · pp. 8–9
• CEMs struggle with real human uncertainty; Skyline can use some signal, but some annotations hurt, while uncertainty-trained models may mitigate drops and uncertainty can help or harm interventions.Figure 12 shows both beneficial and harmful interventions; the illustrated model was trained on uncertain concepts with “Probably” probability = 0.7.Source: Figure 11 · p. 10, Figure 12 · p. 10

6.2 Treating Human (Mis)Calibration

• Users vary widely in calibration, which may guide trust and query choices; future work should study few-shot online post-hoc calibration and UElic teaching curricula without forcing certainty.The authors argue calibration should not deter uncertainty collection, especially where certainty is impossible, and identify calibration as a multidisciplinary opportunity spanning ML, cognitive science, UX design, and psychology.Source: §6.2 · p. 10

6.3 Scaling Uncertainty Elicitation

• Human uncertainty annotation is costly and time-consuming, requiring scalable elicitation and better simulators; more data is needed to characterize uncertainty forms and determine which to elicit for deployment.Model performance varies substantially with the form of uncertainty used.Source: §6.3 · p. 10

7 CONCLUSION

• Simulated and in-the-wild experiments demonstrate failure modes in popular concept-based systems handling coarse- and fine-grained uncertain feedback.The conclusion frames these failures as evidence for studying human uncertainty in concept interventions.Source: §7 · p. 10
• The authors offer UElic, a new interface, and CUB-S, a new challenge dataset, to support further study of human uncertainty in interventions.Source: §7 · p. 10
• Modeling human uncertainty at train and test time has the potential to improve the reliability and trustworthiness of concept-based models deployed in the wild.The conclusion notes that humans may be uncertain or make mistakes, even when their uncertainty is miscalibrated.Source: §7 · p. 10

Constructing UMNIST

• Each UMNIST sample contains p normalized 28×28 grayscale images of handwritten zeros or ones, with image values in [0,1].The input has shape x ∈ [0,1]^(28×28×p).Source: Constructing UMNIST · p. 12
• Each image has a binary concept annotation c_i indicating whether it is a one or zero, and the task label y is the number of ones, y = Σ_i c_i.The concept vector is c ∈ {0,1}^p and y ∈ {0,…,p}.Source: Constructing UMNIST · p. 12
• The construction introduces uncertainty by updating concept annotations, with δ ∈ [0,1] controlling the amount of dataset uncertainty.The supplied passage refers to an update rule but does not include its formula.Source: Constructing UMNIST · p. 12
• To reflect uncertain concepts in the inputs, each corresponding digit is mixed with a randomly selected MNIST training example of the opposite digit, using c_i as the mixing ratio.The approach is described as akin to Zhang et al. [2018].Source: Constructing UMNIST · p. 12
• For a zero input digit, the update combines it with a one sampled from the MNIST training distribution; for a one digit, it combines it with a sampled zero.The passage specifies the mixtures as (1 − c_i)x(:,:,i) + c_i z for zeros and c_i x(:,:,i) + (1 − c_i)z for ones, with z drawn from the corresponding empirical MNIST class distribution.Source: Constructing UMNIST · p. 12
• The paper focuses on p = 10 digits per sample, and Figure 13 presents dataset examples for varying δ.The MNIST class-conditional distributions are empirical training distributions.Source: Constructing UMNIST · p. 12

Training Details for UMNIST Experiments.

• For both CBMs and CEMs, the concept-to-label map is a ReLU two-layer MLP with layer sizes {20, p}, and joint training weights concept loss as much as task loss (α = 1) [2022].Each sample’s task loss is weighted according to its label’s empirical distribution.Source: Training Details for UMNIST Experiments · p. 12
• UMNIST models train on 20,000 samples with 20% for validation, test on 5,000 MNIST test samples, and use Adam at 10^-3, batch size 256, up to 50 epochs, and five seeds.Training stops early if validation loss does not improve for 15 epochs; no test digits are used to construct the training set.Source: Training Details for UMNIST Experiments · pp. 12–13

Training Details for CheXpert Experiments.

• CheXpert models train for 25 epochs on 25% of the training data, use an 80-10-10 train/validation/test split, and average results over five trials at learning rate 0.001 [Irvin et al. 2019].The split is drawn from the training partition because the test split lacks the uncertain concept label.Source: Training Details for CheXpert… · p. 13

Training Details for CUB-Based Experiments.

• CUB models follow the training settings of Espinosa Zarlenga et al. [2022], with one model run per seed because of computational complexity.Source: Training Details for CUB-Based… · p. 13

Details on Intervention Policies.

• Skyline assumes the true label and human interventions to test remaining single-concept edits for highest true-label probability, approximating an unrealistic best-possible oracle; Random selects an unqueried concept randomly.The policies consider querying or editing one concept at a time; Skyline follows the greedy oracle policy described by Chauhan et al.Source: Details on Intervention Policies · p. 13

Additional Results on Concept-Incomplete Variant of UMNIST

• The experiments used UMNIST with 50% of concept annotations available during training and set the concept weight loss α to 0.1 to improve CBM task accuracy.The CBM and CEM used the same configuration and architecture as the earlier UMNIST experiments.Source: Additional Results on… · p. 13
• In the concept-incomplete UMNIST experiments, both CBMs and CEMs lose performance as test-time uncertainty increases, while CBM interventions decrease test accuracy and uncertainty worsens that effect.The results suggest CEMs are relatively safer regardless of user uncertainty at intervention time.Source: Additional Results on… · p. 13

Additional CheXpert Investigations

• The investigations also include training with simulated uncertainty in Figure 15.Source: Additional CheXpert Investigations · p. 13
• The experiments vary the value assigned to CheXpert’s uncertain labels at both train and test time, alongside positive, negative, and unknown labels.The study explores the original uncertain annotations in CheXpert chest X-rays [Irvin et al. 2019].Source: Additional CheXpert Investigations · p. 13
• Figure 15 compares CEMs trained and tested with differing CheXpert uncertainty levels; test-time uncertainty improves intervention performance, while train-time uncertainty has minimal impact.Only 5% of annotations are uncertain, limiting the effect of train-time uncertainty; non-zero test-time values improve intervention accuracy because models distinguish uncertain from negative labels.Source: Additional CheXpert Investigations · pp. 13–14, Figure 15 · p. 14

Additional Details on CUB-S

• CUB-S uncertainty annotations cluster at 0 and 100, with a peak at the interface’s default 50; rounding and the default may explain these patterns.The authors suggest improving the interface to elicit a more accurate uncertainty distribution with a more nuanced balance of granularity.Source: Additional Details on CUB-S · p. 14, Figure 18 · p. 15
• Most annotators assign about 100 probability mass per concept, but some over-assign; concept-level distributions also vary, with eye color tightly centered near 100 and upperparts color or size more variable.Over-assignment is partly explained by concept groups with multiple relevant attributes, and may also reflect overestimated confidence; greater variation occurs for color concepts or ambiguous concepts such as size.Source: Additional Details on CUB-S · pp. 14–15, Figure 19 · p. 15, Figure 20 · p. 15
• Soft annotations reveal human interpretations and annotator-calibration differences absent from CUB’s hard labels, while filtered-out attributes retain substantial probability mass.The authors argue that filtered attributes may contain critical information about image content and encourage models and datasets that account for idiosyncrasies in uncertain annotations [Collins et al. 2022].Source: Additional Details on CUB-S · p. 15, Figure 21 · p. 16

Additional CUB Uncertainty Computational Experiments

• Uncertainty’s distribution across discrete confidence scores affects intervention efficacy under the Random policy.The paper reports that test-time uncertainty values strongly affect random-intervention performance: setting uncertainty to 0 prevents models from distinguishing negative from uncertain concepts, while nonzero values improve performance; test uncertainty matters more than training uncertainty.Source: Figure 16 · p. 14, Figure 17 · p. 14, Broad vs. Narrow Uncertainty · p. 15

Broad vs. Narrow Uncertainty.

• Training and intervening with population-level annotations yields the best reported performance, and moderate aggregate uncertainty improves robustness to test-time uncertainty.The authors note that whether interventions use individual- or population-level annotations matters, and connect the result to annotation design and how many annotators to elicit from.Source: Individual- vs Population-Level… · p. 15, Figure 27 · p. 18

CBMs and Simulated Uncertainty.

• CBMs struggle with test-time uncertainty on concept-incomplete and in-the-wild data, and perform comparatively worse than CEMs on CUB.These observations motivate the paper’s focus on CEMs.Source: CBMs and Simulated Uncertainty · p. 16, Figure 28 · p. 18
• Skyline selection can rapidly improve performance by choosing helpful uncertain annotations, but later selections can harm it, including for “upperparts” color.Which concepts Skyline prefers differs according to whether the CEM was trained without uncertainty or with uncertainty and different imputed “Probably” probabilities.Source: CBMs and Simulated Uncertainty · p. 16, Figure 29 · p. 19, Figure 30 · p. 19