PaperlayerView exact revision on arXiv

Source-linked AI summary

Concept-based Explainable Artificial Intelligence: A Survey

Eleonora Poeta, Gabriele Ciravegna, Eliana Pastor, Tania Cerquitelli, Elena Baralis

arXiv:2312.12936v1cs.AIcs.HC

TL;DR

Feature-level explanations may not capture what a network sees in terms that users can understand, while C-XAI lacks settled definitions and categorization. This paper reviews C-XAI, defines concepts and explanation types, and organizes approaches into nine categories; its taxonomy review aims to clarify their strengths and challenges.

  • Problem

    Deep neural networks’ complexity and opacity hinder understanding their decision-making process.

    Source: §1 · p. 1

  • Method

    The paper defines concepts and explanation types, then reviews and categorizes C-XAI approaches in a nine-category taxonomy.

    Source: §9 · p. 36

  • Results

    The review organizes C-XAI approaches into nine categories and offers insights into their strengths and challenges.

    Source: §9 · p. 36

  • Takeaways & Limitations

    The taxonomy provides a structured view of C-XAI approaches and their respective strengths and challenges.

    Source: §9 · p. 36

    Information leakage in concept representations can compromise interpretability.

    Source: §5.5 · p. 30

Abstract

from arXiv · show

The field of explainable artificial intelligence emerged in response to the growing need for more transparent and reliable models. However, using raw features to provide explanations has been disputed in several works lately, advocating for more user-understandable explanations. To address this issue, a wide range of papers proposing Concept-based eXplainable Artificial Intelligence (C-XAI) methods have arisen in recent years. Nevertheless, a unified categorization and precise field definition are still missing. This paper fills the gap by offering a thorough review of C-XAI approaches. We define and identify different concepts and explanation types. We provide a taxonomy identifying nine categories and propose guidelines for selecting a suitable category based on the development context. Additionally, we report common evaluation strategies including metrics, human evaluations and dataset employed, aiming to assist the development of future methods. We believe this survey will serve researchers, practitioners, and domain experts in comprehending and advancing this innovative field.

1 INTRODUCTION

Feature-level XAI can leave explanations difficult for non-experts to interpret, motivating explanations in terms of higher-level concepts. This paper reviews C-XAI methods and organizes their concepts, categories, selection guidance, and evaluation resources.

1 INTRODUCTION

  • Feature-level explanations may lack meaningful interpretation for non-experts, motivating explanations that describe what a network sees through higher-level attributes.The introduction argues that knowing where a network looks is not enough to explain what it sees.Source: §1 · pp. 1–2
  • The paper presents the first systematic review focused on C-XAI, addressing the absence of a survey distinctly centered on this domain.It reviews papers from 2017 to July 2023 in major conferences, selected journals, and related work.Source: §1 · p. 2
  • The review defines key terms, including concepts and concept-based explanations, and proposes a taxonomy of nine C-XAI categories.The categories are based on concept use during training, concept type, and the explanation required.Source: §1 · p. 2
  • It proposes guidelines for choosing a suitable category and analyzes methods across 13 dimensions, including their applicability and advantages.The review also compares methods within categories and discusses critics and comparisons.Source: §1 · p. 2
  • The paper reports metrics and datasets for assessing and developing C-XAI methods, and highlights early applications and future directions.Its planned coverage also includes human evaluations.Source: §1 · p. 3

2 DEFINITIONS

The section establishes shared definitions for C-XAI, classifying concepts and explanations while distinguishing post-hoc methods from explainable-by-design models.

2.1 What is a Concept?

  • The survey groups concepts into symbolic concepts, unsupervised concept bases, prototypes, and textual concepts.These vary in origin and representation: human-defined symbols, learned clusters, representative training examples, or textual descriptions.Source: §2.1 · pp. 3–4

2.2 What is a Concept-based Explanation?

  • Concept-based explanations fall into class-concept relations, node-concept associations, and concept visualizations.Class-concept relations express concept importance or logic rules linking multiple concepts to an output class.Source: §2.2 · pp. 4–5
  • Node-concept associations assign concepts to network units, either post-hoc through activation analysis or during training by requiring units to predict concepts.This identifies what internal units respond to in a sample.Source: §2.2 · p. 5
  • Concept visualizations highlight input features representing a concept; for non-symbolic concepts, they show which learned attributes or prototypes the network uses.They can complement other explanation types to connect concepts with a class or node.Source: §2.2 · p. 5

2.3 C-XAI Post-hoc Methods vs Explainable-by-design Models

  • Post-hoc methods explain existing models using concepts, whereas explainable-by-design models represent concepts explicitly in an intermediate layer that influences predictions.The explicit representation ties concepts to predictions and provides node-concept associations by design.Source: §2.3 · p. 6

3 C-XAI TAXONOMY

The taxonomy organizes C-XAI methods and models by their characteristics, applicability, and evaluation resources, then offers a question-based procedure for selecting an approach.

3.1 Dimensions of analysis

  • The survey characterizes C-XAI methods across 13 dimensions grouped by concept and explanation characteristics, applicability, and resources and evaluation.The groups cover how concepts are integrated and annotated, explanation type and scope, data and tasks, and evaluation and resource availability.Source: §3.1 · pp. 6–7
  • The dimensions distinguish post-hoc from explainable-by-design training, supervised, unsupervised, hybrid, or generated concept annotation, concept type, explanation type, and local or global scope.For by-design models, they also record concept employment and performance loss versus a black-box model.Source: §3.1 · pp. 6–7
  • Applicability dimensions record supported data, tasks, and network architectures, while evaluation dimensions track dataset releases, code or model availability, new metrics, and human evaluation.Supported data types include images, text, graphs, tabular data, videos, and time series.Source: §3.1 · p. 7

3.2 Guidelines for selecting a suitable method

  • Method selection begins by asking whether the model can be modified and whether task-related concept annotations are available.An unmodifiable existing model points to post-hoc methods; available annotations point to supervised methods, otherwise unsupervised ones are considered.Source: §3.2 · pp. 7–8
  • Further selection distinguishes hybrid or externally generated concepts and can branch on desired explanation type, dataset, or unsupervised concept representation.The procedure uses at most three questions to match a category to development conditions.Source: §3.2 · p. 8

4 POST-HOC CONCEPT-BASED EXPLANATION METHODS

This section surveys post-hoc concept-based methods for explaining trained models, covering supervised and unsupervised approaches that link concepts to classes, internal components, or input parts and use varied strategies to discover concepts and assess their importance or interpretability. It also notes their limits, including reliance on probe data, vulnerability to adversarial attacks, and the inability to guarantee that models truly use the concepts in their explanations.

4 POST-HOC CONCEPT-BASED EXPLANATION METHODS

  • These methods divide by concept supervision: annotated datasets support symbolic concepts, while unsupervised approaches extract concepts; supervised methods target class-concept relations or node-concept associations, and both can visualize concepts.Unsupervised methods generally focus on class-concept relations and concept visualization.Source: §4 · p. 8
  • Most reviewed methods provide global explanations, address classification, and use CNNs with image data; a few cover local explanations, text, graphs, videos, transformers, GNNs, or 3D-CNNs.Some methods provide both local and global explanations, and one provides only local explanations.Source: §4 · p. 9
  • Post-hoc methods suit pre-trained or predefined models and preserve predictive and generalization capabilities, but cannot guarantee that a model understands or uses its adopted concepts.They make model behavior more interpretable through terms more comprehensible to humans than raw input features.Source: §4 · p. 9

4.1 Supervised Concept-based Explainability Methods

  • Supervised post-hoc methods use samples annotated with symbolic concepts to assess which concepts a network learned, where, and how they influence its predictions.Class-concept methods examine relations between concepts and predictions, whereas node-concept methods associate individual hidden-node activations with concepts.Source: §4.1 · pp. 9–10

4.1.1 Class-Concept Relation methods.

  • Class-concept methods use external, class-related concept samples to study latent-space relations or the causal effect of inserting or removing concepts; T-CAV [56] uses a linear probe and requires linearly separable concepts.T-CAV measures conceptual sensitivity through class directional derivatives along concept activation vectors; its score is the fraction of class inputs with positive sensitivity.Source: §4.1.1 · p. 10
  • T-CAV enables humans to identify important concepts for networks trained on standard and corrupted features, while CAR relaxes T-CAV’s linear-separability assumption; in some settings, TCAR reports high class–concept association scores while TCAV scores are low.Source: §4.1.1 · pp. 10–11
  • IBD decomposes prediction evidence into semantically related concept components, and human evaluation found its explanations enhance CAM and Grad-CAM regarding model.Its optimization estimates concept importance for a class, with residuals capturing prediction variability not represented by learned concepts.Source: §4.1.1 · p. 11
  • Both CPM versions significantly outperform previous methods in assessing concept causality.CPM is initialized with black-box weights and trained to mimic its behavior; for counterfactual training, it uses factual inputs and information from counterfactual samples, which are unavailable at test time.Source: §4.1.1 · pp. 11–12

4.1.2 Node-Concept Association methods.

  • Results show that a hidden unit automatically learns to recognize a concept with precision higher than 70% on average across all hidden layers, while concepts shift from colours and textures in lower layers to objects and object parts in higher ones.Source: §4.1.2 · p. 12
  • Net2Vec [34] found that combining filters identifies concepts better than using single filters, while filters can encode multiple concepts and the method assumes linear concept-filter relationships.Net2Vec optimizes classification or segmentation from filter activations; its linear model may miss more complex, nonlinear alignments.Source: §4.1.2 · p. 13
  • GNN-CI extracts meaningful graph-classification concepts, including functional groups correlated with graph classes, and interpretable neuron activations positively correlate with correct predictions.It can describe neurons through concept compositions and explain the final class using a linear classifier trained on explained nodes.Source: §4.1.2 · p. 13

4.2 Unsupervised Concept-based Explainability methods

  • ACE [37] ranks image-segment clusters by T-CAV importance, and human evaluation found that samples grouped under the same concepts were coherent and generally received consistent labels.Segmentation, clustering, or similarity errors may still yield meaningless or incoherent concepts.Source: §4.2 · p. 14
  • On a synthetic dataset with complete concepts, Completeness-aware outperformed compared concept-discovery methods, including ACE, in discovering the complete concept set.It optimizes concept discovery for predictive sufficiency and promotes proximity between each concept and its closest neighboring patches.Source: §4.2 · pp. 14–15
  • ConceptSHAP modifies SHAP to measure concept importance for a class by replacing attribute importance with concept sufficiency.The survey describes it alongside Completeness-aware’s approach to discovering complete concept-based explanations.Source: §4.2 · p. 14
  • ICE uses NMF to extract concept vectors and a linear classifier’s weights for importance; its richer representation has lower approximation error and its concepts are more interpretable than PCA- or K-means-derived concepts.Source: §4.2 · p. 15
  • CRAFT [33] factorizes activations from random crops with NMF, recursively identifies concepts and sub-concepts, and uses Sobol scores for importance; human evaluation found higher utility than standard XAI and ACE in two of three scenarios.Source: §4.2 · p. 15
  • MCD discovers multidimensional concepts spanning hyperplanes across convolutional channels, and its SSC-based variant outperforms ICE and ACE on faithfulness and conciseness.Faithfulness measures prediction reduction after flipping the most important concepts; conciseness counts concepts needed to reach a given completeness score.Source: §4.2 · p. 15
  • DMA and IMA seek identifiable concepts: DMA guarantees recovery despite correlated components when their effects are disjoint, while IMA requires only orthogonality; neither provides concept explanations.Source: §4.2 · pp. 15–16
  • STCE [52] extends ACE to video by clustering 3D ConvNet features from class-specific supervoxels, assigns importance with T-CAV scores, and evaluates faithfulness by measuring accuracy drops after removing important concepts.Source: §4.2 · p. 16

4.3 Discussion

  • 91% vs 97% on the CUB dataset and 0.68 vs. 0.53 error on the OAI dataset show linear probes lag concept-trained models, so post-hoc explanations cannot guarantee symbolic concepts [61].Source: §4.3 · p. 16
  • Different probe datasets can produce different explanations for the same model and method, so the authors recommend datasets resembling the model’s training-data distribution [90].Source: §4.3 · pp. 16–17
  • Adversarial perturbations can change concept-based interpretations without changing classification, and attacks remain partly effective when transferred to a different black-box model.Source: §4.3 · p. 17

5 EXPLAINABLE-BY-DESIGN CONCEPT-BASED MODELS

This section surveys explainable-by-design concept-based models that build concepts into network architectures to support predictions and explanations. It covers concept-supervision approaches, unsupervised representations, and prototype-based, hybrid, and generative designs, discussing their reported results, tradeoffs, and limitations.

5 EXPLAINABLE-BY-DESIGN CONCEPT-BASED MODELS

  • Concept-based models use annotated, unannotated, partially annotated, or externally generated supervision; symbolic concepts may be trained jointly or instilled afterward, while unsupervised concepts may be clusters or prototypes.Hybrid approaches generally combine symbolic and unsupervised basis concepts; generated concept annotations are normally textual.Source: §5 · p. 17
  • Concept interventions let domain experts change predicted concept values and observe output changes, but these models require training from scratch and may have lower accuracy than standard black-box models.Intervention can support counterfactual explanations.Source: §5 · pp. 17–18

5.1 Supervised Concept-based Models

  • Supervised models use symbolic concept annotations to train an intermediate concept layer, either jointly from the same dataset or by instilling concepts learned from a separate dataset.Joint training is available when concept annotations are included in the main training data; instillation uses a separate dataset devoted to concept learning.Source: §5.1 · p. 18

5.1.1 Joint concept training.

  • In joint training, CBM inserts a concept bottleneck whose predictions feed the task function, and a double loss penalizes errors in both concepts and outputs.Concept interventions can alter predicted concepts and reveal counterfactual responses; paired class and concept annotations are required for this training strategy.Source: §5.1.1 · pp. 18–19
  • CBM’s generalization is lower than standard end-to-end models, especially with few available concepts, and class-concept relations are available only when its task function uses a single layer.The text attributes the generalization limitation mainly to representing each concept with a single neuron.Source: §5.1.1 · p. 19
  • LENs impose sparse task-function weights and provide first-order logic explanations linking task functions to influential concepts.The section also notes that LENs can work with tabular or textual inputs without a concept extractor and can explain clustering assignments.Source: §5.1.1 · p. 19
  • CEM matches black-box classification accuracy, remains accurate as the number of concepts decreases, and aligns with the true concept distribution nearly as well as CBM.CEM’s concept alignment is reported as much higher than the alignment provided by CEM’s comparison model, but the passage does not identify that model by name.Source: §5.1.1 · p. 20
  • ProbCBM outperforms CBM but not CEM in classification accuracy, while its concept prediction accuracy is on par with CBM and better than CEM.The authors report that integrating uncertainty through probabilistic embedding does not compromise concept prediction accuracy.Source: §5.1.1 · p. 20
  • CEM and Prob-CBM improve standard CBM performance but weaken interpretability because embedded concept dimensions lack direct symbolic meaning.DCR is introduced in response as an interpretable concept-embedding model.Source: §5.1.1 · p. 20
  • DCR [11] matches CEM task accuracy, exceeds CBM, recovers available ground-truth rules, and supports counterfactuals better than LIME and LEN with less perturbation-sensitive explanations.Its explanations are local, based on fuzzy rules applied to concept activations.Source: §5.1.1 · p. 20

5.1.2 Concept Instillation.

  • CW yields purer concepts than TCAV and IBD; CME beats Net2Vec in task accuracy but is comparable to CBM, while PCBM is slightly worse than its source model.CME introduces MisPrediction Overlap (MPO), which measures the portion of samples with at least m inaccurately predicted concepts; on MPO, CME is similar to Net2Vec but trails CBM. CW also finds more abstract and fundamental representations in lower layers.Source: §5.1.2 · pp. 21–22
  • ConceptTransformer matches end-to-end black-box classification and can boost performance when concepts and their relationships to classes are well-defined.The reported condition includes settings such as CUB-200-2011, where concepts and class relationships are known.Source: §5.1.2 · p. 22

5.2 Unsupervised Concept-based Models

  • Unsupervised concept models modify network representations to extract concepts without predefined symbols, using either clustered concept bases or prototypes in layer weights.The prototypes may be parts of frequently seen input samples.Source: §5.2 · p. 22

5.2.1 Unsupervised Concept Basis.

  • Interpretable CNNs use convolutional filters as unannotated object-part concepts, trained to maximize image–activation mutual information while encouraging low-entropy activations.The design aims for each filter to encode a distinct concept associated with one object category.Source: §5.2.1 · p. 23
  • Interpretable CNNs show superior interpretability and location stability to standard CNNs while matching simple CNNs in classification accuracy.Evaluation includes IoU against ground-truth object boxes and part-location stability across objects.Source: §5.2.1 · p. 23
  • SENN explanations are more explicit, faithful to ground-truth feature importance, and stable than LIME and SHAP[70], with accuracy slightly lower or on par with standard CNNs.SENN combines learned basis concepts with input-dependent class-concept relevance scores and encourages sparse, non-overlapping concepts.Source: §5.2.1 · pp. 23–24
  • BotCL achieves higher average task performance than SENN, ProtoPNet, and Completeness-aware, particularly with contrastive loss, and its concepts score well in purity, efficiency, and reconstruction error.Its consistency and mutual-distinctiveness losses encourage concepts to activate consistently across images and to capture different visual characteristics, respectively.Source: §5.2.1 · p. 24
  • SelfExplain [89] extracts text concepts from semantic-tree leaves and uses local and global interpretable layers to explain predictions through input concepts and concepts from training data.The two layers’ concept scores re-predict the class for explanations but are not used for the standard prediction.Source: §5.2.1 · p. 24
  • SelfExplain’s GIL and LIL layers improve classification across evaluated datasets, while human evaluators find its explanations more understandable, trustworthy, and adequate than standard XAI explanations.The authors attribute the classification gains partly to the multitask loss.Source: §5.2.1 · p. 24

5.2.2 Prototype-based Concepts.

  • Prototype-based models compare inputs with concepts representing training examples and use prototype similarities for prediction, with extraction designs ranging from autoencoders to shared or hierarchical prototypes.Prototype DL [66] uses an autoencoder and prototype classifier, and its interpretable architecture does not compromise predictive ability compared with traditional and non-interpretable models.Source: §5.2.2 · pp. 24–25
  • ProtoPNet [21] matches learned prototypical image parts to patches, uses weighted similarities for prediction, and provides class-concept relations and activation-map visualizations; its accuracy is up to 3.5% lower than non-interpretable baselines.Its prototype weights also indicate the importance of each prototype to a class.Source: §5.2.2 · p. 25
  • HPNet [45] organizes prototypes hierarchically to support classification and explanations at multiple granularities, and can classify previously unseen classes at an appropriate taxonomy level with a slight fine-grained accuracy drop versus a black-box model.It uses prototype layers for taxonomy nodes and heat maps to show localized image regions activating prototypes at different hierarchy levels.Source: §5.2.2 · pp. 25–26
  • ProtoPShare shares prototypes between classes through data-dependent merge-pruning, reducing prototype count and addressing instability when semantically similar class prototypes are far apart in latent space.Its two phases are initial training followed by pruning prototypes selected for merging based on similarity.Source: §5.2.2 · p. 26
  • ProtoPool shares prototypes across classes and replaces pruning with differentiable soft assignment, while focal similarity emphasizes salient features and projection maps prototypes to nearest training patches.Its soft assignments avoid ProtoPShare’s training-time-increasing merge-pruning step.Source: §5.2.2 · p. 26
  • ProtoPool achieved the highest accuracy with fewer prototypes on one dataset but comparable, lower accuracy on another; Deformable ProtoPNet achieved state-of-the-art accuracy, and users rated ProtoPool’s visualization more interpretable.ProtoPool outperformed ProtoPNet in accuracy when fewer prototypes were used.Source: §5.2.2 · p. 26
  • Deformable ProtoPNet addresses rigid prototypes by using adaptive parts that shift spatially with the input, allowing prototypes to account for object geometry and pose variation.Source: §5.2.2 · p. 26
  • ProtoPDebug uses expert feedback to flag confounding part-prototypes, then fine-tunes the model to penalize forbidden concepts and preserve valid ones, improving test classification over prior debuggers and ProtoPNet.Source: §5.2.2 · p. 27

5.3 Hybrid Concept-based Models

  • CBM-AUC combines supervised and unsupervised concepts through an M-SENN variant designed for scalability, outperforming SENN and CBM in task accuracy while producing semantically meaningful concept activations.M-SENN replaces SENN’s autoencoder decoder with a discriminator and shares intermediate network components with the encoder.Source: §5.3 · pp. 27–28
  • Learning Ante-hoc Explainable Models via Concepts adds a classifier over the concept encoder and a fidelity loss; it improves task accuracy over CBM and SENN on annotated and unannotated datasets.Source: §5.3 · p. 28
  • GlanceNets use a VAE to learn disentangled concepts that represent generative factors, improve concept alignment over CBM, and maintain task performance on par with CBM and SENN.When supervised concepts are available, they are mapped to learned concepts to preserve known semantics.Source: §5.3 · p. 28

5.4 Generative Concept-based Models

  • LaBO uses CLIP [88] for textual-concept and image embeddings; it retains accuracy with limited training data, while performance slightly decreases as data volume grows but remains competitive with black-box models.Source: §5.4 · p. 29
  • LF-CBM generates and filters concepts with GPT-3, uses CLIP-Dissect [81] for a concept matrix, and learns a sparse final predictor.The method learns concept weights to form a Concept Bottleneck Layer before training the final predictor.Source: §5.4 · p. 29
  • LF-CBM incurs only a small task-accuracy loss versus its neural-network backbones and outperforms interpretable PCBM.Source: §5.4 · p. 29

5.5 Discussion

  • Greater transparency in concept-based models can cost performance, although several approaches match complex black-box models while enabling concept-based decision explanations.Source: §5.5 · p. 30
  • Supervised concept representations can encode information beyond the concepts, including task-label representations when jointly trained with downstream tasks; this information leakage compromises interpretability and intervenability [74].Source: §5.5 · p. 30
  • Soft or fuzzy concept representations share leakage, and proposed mitigation strategies do not eliminate it; Mahinpei et al. [74] demonstrate leakage for CBM [61] and CW [22].The strategies include sequential training, adding unsupervised concept dimensions, and concept whitening.Source: §5.5 · p. 30
  • CBM interventions can increase task error when users set unsure concepts to zero, because a concept absent from an image may still be associated with its class [101].The discussion links this pitfall to soft concept representations and leakage.Source: §5.5 · p. 30
  • Majority voting to associate concept values with classes can bias outcomes toward the majority, and intervention may worsen minority treatment unless concepts are changed to majority values [32, 46, 61].Source: §5.5 · p. 30

6 EVALUATION

This section reviews how C-XAI methods are evaluated, including quantitative metrics, human assessments, datasets, concept effects and quality, and explanation usefulness. It also describes concept sourcing across dataset types and summarizes the availability of code and newly released datasets.

6.1 Quantitative evaluation

  • Quantitative metrics fall into two groups: measures of concepts’ contributions to predictions or task performance, and measures of concepts’ intrinsic quality.Concept quality metrics address inherent properties, relationships with internal network representations, and prediction errors.Source: §6.1 · p. 31
  • Prediction-focused metrics assess concepts’ effects on class decisions, such as T-CAV [37] [56] and CaCE [41], or their contribution to task performance, such as Completeness Score and Fidelity [33].The review also lists ConceptSHAP, Smallest Sufficient Concepts [37], and Concept Efficiency [32] among these metrics.Source: §6.1 · p. 31

6.2 Qualitative evaluation

  • Half of post-hoc approaches conduct human assessments, compared with only a quarter of explainable-by-design approaches.The review groups qualitative assessment around explanation understandability, concept coherence, and explanation utility.Source: §6.2 · pp. 31–32
  • Human evaluations assess whether explanations are understandable and plausible, including users’ judgments of justifiability, concept labels, trust, and communicated information.Some settings ask users to label concepts from a predefined vocabulary or provide free-text labels.Source: §6.2 · p. 32
  • Concept coherence can be assessed by intruder detection—identifying an image conceptually different from the rest [33] [37]—while utility can be measured by users’ accuracy predicting model decisions on novel images given explanations [33].Source: §6.2 · p. 32

6.3 Datasets

  • CUB is the predominant image dataset in C-XAI, BRODEN [13] was designed explicitly for concept-based XAI, and OAI is the only dataset identified as suitable for regression as well as classification.BRODEN contains over 63,000 images and 1,197 concepts across six categories.Source: §6.3 · p. 33
  • The reviewed text datasets mainly support binary or multiclass sentiment classification, and their concepts are often extracted from input text rather than supplied in advance.One cited approach represents concepts as non-terminal annotations [89].Source: §6.3 · pp. 33–34
  • BDD-OIA is the only video dataset with pre-established concepts; concepts are extracted through specific processes in the other two video cases, while tabular attributes serve as concepts and graph concepts are method-extracted.Source: §6.3 · p. 34

6.4 Resources

  • Nearly all reviewed methods except four made code publicly available on GitHub: the exceptions are post-hoc CaCE and Object Detection, and concept-based SENN and CBM-AUC.Source: §6.4 · p. 34
  • Four methods released novel datasets—ND, DMA, and IMA among post-hoc methods, and CME and Glance-Nets among concept-based models—while dataset creation can require C×N annotations.Class-level concept annotation can reduce the effort to N×Y annotations, where Y is the number of final classes; new dataset releases are infrequent because creating concept-annotated datasets is resource-intensive.Source: §6.4 · p. 34

7 APPLICATIONS

C-XAI applications span AI tasks and real-world domains, including out-of-distribution detection, medical diagnosis, education, and image generation. The examples show both domain-specific concept explanations and generation that can handle queries unseen in training.

Image generation

  • VAEL conditions image generation on a neuro-symbolic program over digit concepts, training concepts and a logic program to predict addition before decoding images that solve the query.The method also generalizes to unseen queries, including difference or factorization tasks.Source: §7 · p. 34

Out-of-distribution detection

  • Concepts extracted to explain a classifier were insufficient to explain an out-of-distribution detector’s prediction, motivating concepts that characterize the detector and separate in-distribution from out-of-distribution samples.One approach maps samples to human-defined concepts and analyzes rare features; another extracts unsupervised concepts that characterize detector decisions.Source: §7 · p. 34

Medical applications

  • In skin-cancer computer-aided diagnosis, TCAV found a strong correlation between DNN-learned concepts and concepts dermatologists routinely use.The authors suggest listing concepts that positively influenced the outcome may be sufficient for trusting CAD predictions.Source: §7 · p. 35

Education

  • RIPPLE combines a graph-based neural network for student-interaction time series with TCAV to interpret its internal state using six a-priori educational learning dimensions.The pipeline makes predictions about student success interpretable.Source: §7 · p. 35

8 PROMISING DIRECTIONS

The survey identifies methodological, evaluation, and application directions for C-XAI. These include extending concept representations and generative approaches, while addressing gaps in benchmarks, metrics, scalability, and integration into large-scale models.

Methodological directions

  • The authors propose exploring generative approaches for post-hoc methods, including using generative models to create concepts and LLMs to enhance supervised methods in assessing model causality.They also describe a concept generator producing representations, not necessarily textual, for use in an external, not necessarily visual, model.Source: §8 · p. 35
  • The review identifies a gap in node-concept association explanations for unsupervised methods, despite the potential difficulty of associating nodes with extracted concepts.The authors describe this as a valuable avenue for exploration.Source: §8 · p. 35
  • Extending concept representations to multimodal data such as text, images, and audio could enhance C-XAI versatility across domains.The survey also notes that the novel generative approach appears open to different types of fusion.Source: §8 · p. 35

Evaluation directions

  • The field lacks benchmark datasets and standardized metrics, hindering systematic comparisons; developing these resources could facilitate fair assessments of different methods..Source: §8 · p. 35

Application directions

  • The authors call for scalable concept-based methods and models that integrate with large-scale models and datasets, particularly for accountable decision-making in critical domains.Examples named include healthcare, finance, and autonomous systems.Source: §8 · p. 35

9 CONCLUSION

The conclusion presents the survey’s definitions and nine-category taxonomy as a framework for understanding C-XAI. Its review of the literature and highlighted datasets and metrics are intended to inform future research and comparisons.

Survey contribution

  • The survey defines types of concepts and concept-based explanations and introduces a systematic categorization framework for the C-XAI landscape.Source: §9 · p. 36

Taxonomy analysis

  • The authors reviewed and categorized C-XAI papers into nine taxonomy categories to provide insights into the strengths and challenges of different categories.Source: §9 · p. 36

Resources

  • The survey highlights datasets and metrics in C-XAI literature to catalyse future research and facilitate comparisons between methodologies.Source: §9 · p. 36

A GLOSSARY OF TERMS

This glossary aims to provide a shared reference for field terminology and consistent use of terms. It foregrounds concepts and concept-based explanations as ways to describe model behavior in human-understandable terms.

A GLOSSARY OF TERMS

  • Local explanations address an individual prediction, global explanations describe behavior across a dataset, and node/filter explanations detail individual network units.The glossary associates node/filter explanations with the features or relationships considered by targeted nodes.Source: A GLOSSARY OF TERMS · p. 40
  • Post-hoc explanations follow a prediction, model-agnostic techniques analyze inputs and outputs across architectures, and counterfactual explanations describe altered inputs that could change predictions.Counterfactual explanations use modified scenarios to show how input changes would yield different outcomes.Source: A GLOSSARY OF TERMS · pp. 40–41
  • The glossary also distinguishes saliency maps, which highlight important input regions, from concept scores and embeddings, which represent concept relevance or concepts numerically.It describes feature importance as the contribution of input features to model predictions.Source: A GLOSSARY OF TERMS · p. 41
  • A concept is a high-level, human-understandable abstraction of data features or patterns, while concept-based explanations express model predictions through such concepts rather than feature importance.The glossary presents concepts as a way to simplify information and communicate model behavior.Source: A GLOSSARY OF TERMS · p. 41
  • A concept-based model predicts using high-level concepts instead of raw data; a concept bottleneck forces predictions through an intermediate layer of predefined concepts.The glossary notes that this constraint may decrease classification performance.Source: A GLOSSARY OF TERMS · p. 41
  • Probing trains an auxiliary concept-specific model on latent representations, while concept intervention modifies predicted concepts to test hypothetical scenarios.The glossary says probing can uncover concepts learned by deep models and intervention can provide concept-based counterfactual explanations.Source: A GLOSSARY OF TERMS · p. 41

B FORMAL CONCEPT ANALYSIS

The section defines concepts using category theory and formal concept analysis. Its definition is broad enough to accommodate different concept typologies through the chosen attribute set.

B FORMAL CONCEPT ANALYSIS

  • Drawing on category theory, the paper defines a concept as a set of attributes agreeing on the intention of its extension.The extension is the objects belonging to the concept, and the intention is the attributes common to those objects.Source: B FORMAL CONCEPT ANALYSIS · p. 42
  • A formal context consists of objects, attributes, and an incidence relation recording which attributes apply to which objects.The paper denotes these as O, A, and I, respectively.Source: B FORMAL CONCEPT ANALYSIS · p. 42
  • A formal concept pairs object and attribute subsets whose derivations reproduce one another; choosing the attribute set allows the definition to cover previously identified concept types.The paper also characterizes this condition through double derivation.Source: B FORMAL CONCEPT ANALYSIS · p. 43

C METRICS DESCRIPTION

The section extends the metric analysis with short descriptions of metrics organized within categories.

C METRICS DESCRIPTION

C.1 Concept effect on class prediction and task performance

The section describes metrics for concept effects on class prediction and task performance. These metrics cover concept contribution, prediction sufficiency, and the effects of selecting or intervening on concepts.

C.1 Concept effect on class prediction and task performance

  • CaCE measures concepts’ causal effect on final class prediction, and ConceptSHAP measures a concept’s effect on completeness.Source: §C.1 · p. 43
  • Smallest Sufficient Concepts evaluates class accuracy using a subset of concepts and seeks the smallest such set.Source: §C.1 · p. 43
  • Smallest Destroying Concepts seeks the smallest concept set whose deletion most reduces prediction accuracy; STCE ranks concept importance for a class, and T-CAR measures class instances in a positive concept region.Source: §C.1 · p. 43
  • Sobol score measures a concept’s contribution and interactions to output variance, while a concept weight represents concept importance for a class in linear concept-to-task models.Source: §C.1 · p. 43
  • Concept Relevance denotes a concept’s predicted importance for a final class, while T-CAV counts class inputs with positive conceptual sensitivity.Conceptual sensitivity is how much a concept influences prediction for a given class.Source: §C.1 · p. 43
  • Completeness measures whether concept scores suffice for predicting model outcomes; fidelity counts agreement between original and approximate predictions, and faithfulness measures concepts’ predictive capacity.Completeness is based on the assumption that complete-concept scores are sufficient statistics of the model prediction.Source: §C.1 · p. 43
  • Concept efficiency concerns the number of concepts used for task prediction, while conciseness counts concepts needed to reach a target completeness and final-accuracy level in post-hoc methods.Richer concept representations mitigate the stated dependence of prediction capacity on concept count.Source: §C.1 · p. 44

C.2 Quality of Concepts

The section surveys measures of concept quality, covering alignment with ground truth, class coverage and purity, prediction errors, and relations between concepts and network representations.

C.2 Quality of Concepts

  • Completeness measures how well a concept covers a specific associated dataset class.Source: §C.2 · p. 44
  • Purity measures the ability to discover concepts that cover only a single class.Source: §C.2 · p. 44
  • Intersection over Union measures overlap between concept bounding boxes and a single node’s activation maps [13] [34]; calls it Part Intersection.Source: §C.2 · p. 44
  • Concept Error measures alignment with ground-truth concepts using the L2 distance between learned and ground-truth concepts [61]; calls it Explanation Error.The section groups it under concept prediction error.Source: §C.2 · p. 44
  • AUC score measures whether samples belonging to a concept rank higher than others, indicating concept purity [22].Source: §C.2 · p. 44
  • Misprediction-overlap metric (MPO) measures the test-set fraction with at least m relevant concepts predicted incorrectly [54].A larger MPO indicates a larger proportion of incorrectly predicted concepts.Source: §C.2 · p. 44

D METHODS RESOURCES

This section takes an orthogonal view of the methods, reporting the resources each uses according to the Section 3.1 categorization.

D METHODS RESOURCES

Loading 2312.12936v1…