Source-linked AI summary
Human-in-the-loop Extraction of Interpretable Concepts in Deep Learning Models
Zhenge Zhao, Panpan Xu, Carlos Scheidegger, Liu Ren
TL;DR
Automatically identifying human-understandable visual concepts that affect DNN decisions is challenging, limiting concept-based post-hoc interpretation. ConceptExtract combines user feedback, active learning, and visualization to extract user-defined concepts. Case studies and experiments report interpretable concepts and model-refinement strategies, while the authors note that full real-world evaluation remains needed.
Problem
Automatic approaches do not readily identify human-understandable visual concepts that affect model decisions, though such concepts are sought for interpreting DNN predictions.
Source: §1 · pp. 1–2Method
ConceptExtract uses visualization-assisted active learning to combine human feedback with concept-extractor training from image-patch representations.
Source: §1 · p. 2, §2.2 · p. 2, §4.2 · p. 4Results
Takeaways & Limitations
Concepts that reveal problematic cases, such as objects under shadows, can support clustering edge cases and developing mitigation strategies such as artificial-shadow augmentation.
Source: §1 · p. 2, §7.2 · p. 8A full evaluation in real-world application scenarios remains needed to complement the preliminary expert evaluation.
Source: §9 · p. 9
Abstract
from arXiv · showhide
The interpretation of deep neural networks (DNNs) has become a key topic as more and more people apply them to solve various problems and making critical decisions. Concept-based explanations have recently become a popular approach for post-hoc interpretation of DNNs. However, identifying human-understandable visual concepts that affect model decisions is a challenging task that is not easily addressed with automatic approaches. We present a novel human-in-the-loop approach to generate user-defined concepts for model interpretation and diagnostics. Central to our proposal is the use of active learning, where human knowledge and feedback are combined to train a concept extractor with very little human labeling effort. We integrate this process into an interactive system, ConceptExtract. Through two case studies, we show how our approach helps analyze model behavior and extract human-friendly concepts for different machine learning tasks and datasets and how to use these concepts to understand the predictions, compare model performance and make suggestions for model refinement. Quantitative experiments show that our active learning approach can accurately extract meaningful visual concepts. More importantly, by identifying visual concepts that negatively affect model performance, we develop the corresponding data augmentation strategy that consistently improves model performance.
1 INTRODUCTION
Model interpretation helps developers understand model weaknesses and identify possible roots of errors, supporting more responsible use and model refinement. ConceptExtract combines human-guided active learning with visual analytics to extract user-defined concepts for model diagnosis and comparison.
1 INTRODUCTION
- Understanding model weaknesses can help developers scrutinize decisions and address potential training-data errors or biases in real-world applications.The introduction connects interpretability with responsible scrutiny.Source: §1 · p. 1
- Interpretability is distinct from accuracy, and improved accuracy may not yield comparable gains in human insight into AI systems.The cited study by Bansal et al. makes this comparison.Source: §1 · p. 2
- For low-accuracy lost-cargo detection, potential error sources include road lighting and the cargo’s visual features, which can inform mitigation such as data augmentation.The authors describe root-cause identification as a role for model interpretation.Source: §1 · p. 2
- ConceptExtract combines a visual analytics interface with active learning to train concept extractors using human knowledge and limited labeling.The system supports patch exploration, interactive labeling, and model diagnostics and comparison.Source: §1 · p. 2
- Automatic concept clustering may group latent-space-near patches that are not semantically meaningful to people, motivating human input during concept extraction.The proposed framework applies to classification and segmentation examples and a range of architectures.Source: §1 · p. 2
2 BACKGROUND
The background introduces DNN architectures, latent representations for patch similarity, active learning for reducing labeling effort, and concept annotations as grounding for explanations.
2.1 Deep Neural Networks
- DNNs combine convolutional and nonlinear layers, often followed by a fully connected layer, and different layer configurations form distinct architectures.The paper describes these networks as taking pixel-valued inputs.Source: §2.1 · p. 2
2.2 Deep Embeddings
- Deep embeddings represent image patches in a selected low-dimensional hidden layer, where Euclidean distance is used as a perceptual similarity measure [54].The paper uses “deep embedding” interchangeably with “latent representation.”Source: §2.2 · p. 2
2.3 Active Learning
- Active learning iteratively queries users to label prioritized instances, retrains on those labels, and updates priorities to reduce the labeling burden.The paper reports that user selection from carefully laid-out images produces a more efficient training sequence than pure sequential active learning.Source: §2.3 · p. 2
2.4 Concept Annotations
- Concept annotations are potentially multiple object or pattern labels for an image, distinct from its classification label and unused in task-network training.They can provide grounding for model explanations.Source: §2.4 · p. 2
3 RELATED WORK
Related work spans post-hoc and inherently interpretable models, latent-space visualization, and prior quantitative and case-study evaluations of ConceptExtract.
Evaluation
- The paper reports two case studies and quantitative experiments across diverse machine-learning tasks and datasets.The introduction also states that experiments compare concept extraction speed with traditional active learning.Source: §3 · p. 2
Post-hoc explanations
Post-hoc and inherently interpretable models
- The paper focuses on post-hoc explanations for already-trained models, while noting that combining explanation objectives with popular architectures remains unresolved and can be time-consuming.It distinguishes this focus from inherently explainable models [8] [10] [17] [29] [52] [53].Source: §3 · pp. 2–3
Visualizing latent spaces
- ConceptExtract uses latent spaces both as inputs to concept-extraction networks and to arrange image patches for user exploration.Prior systems explored or compared latent spaces, including LSC and Spinner et al..Source: §3 · p. 3
4 TASKS AND WORKFLOW
ConceptExtract organizes model diagnosis around five developer tasks, then uses a visualization-assisted active-learning workflow to extract concepts and support interpretation and comparison.
4.1 Task Analysis
- Model developers use diagnosis to investigate whether errors are systematic, such as a segmentation model missing objects under tree shadows, and to guide refinement.The paper connects such analysis with understanding and anticipating model behavior.Source: §4.1 · p. 3
- The task analysis identifies five goals: summarize model behavior, explore patches, train and evaluate concept extractors, analyze concept effects, and compare models.The intended analyses include concept-specific prediction importance and performance differences across models.Source: §4.1 · p. 3
4.2 Workflow
- Preprocessing segments images into patches or super-pixels and extracts their representations from a selected target-model layer for concept learning.Individual concept-extractor networks share an architecture but have different weights for recognizing different concepts.Source: §4.2 · p. 3, Figure 2 · p. 4
- The patch view places visually similar samples near one another and lets users overlay accuracy, ground-truth, or predicted labels to focus exploration.Users can create concepts from the displayed patches and prioritize samples relevant to model performance.Source: §4.2 · p. 4, Figure 2 · p. 4
- Concept extractors score patches from 0 to 1; users filter uncertain or confidently predicted examples, with hard negatives highlighted as useful labels.Augmentation applies shape and color policies to labeled patches to reduce effort and support learning from few examples.Source: §4.2 · p. 4
- After concept learning, users analyze and compare models with TCAV scores and confusion matrices, including performance on images containing particular concepts.The workflow presents these as fine-grained model-analysis views.Source: §4.2 · p. 4
5 SYSTEM DESCRIPTION
ConceptExtract combines patch exploration, user-guided concept training, and concept-based diagnostics to support model interpretation and comparison. Its views connect visual patch selection to labeling, model analysis, and cross-filtering.
5.1 The Image Patch View
- The image patch view lays out latent-representation neighbors in a de-cluttered grid and overlays accuracy, predictions, or concept confidence to focus exploration.Users can filter to clusters or patches of interest, including those associated with wrong predictions.Source: §5.1 · p. 4
5.2 The Training View
- The training view lets users label positive and negative patches, crop concept examples, and save named concepts while training proceeds.Users can continue adding patches during training and save the extractor and retrieved images when a stage ends.Source: §5.2 · p. 5
5.3 The Model Analysis View Using Visual Concepts
- The model-analysis view uses TCAV bars and concept-specific confusion matrices to assess concept importance and model errors.For each concept, it compares predictions with ground truth on the top 50 patches by concept confidence; paired matrices also compare models.Source: §5.3 · p. 5
5.4 Other Views
- The cross-filter view charts patch-level features and links selections to the image patch view, supporting retrieval and comparison across concepts.Features include prediction accuracy and concept confidence scores, which can help identify confused patches and examine concept correlations.Source: §5.4 · p. 5
6 SYSTEM IMPLEMENTATION
The implementation separates visualization from storage and active learning, with a backend that prepares and stores patch-level model information. A stripe-pattern example illustrates the concept-retrieval workflow.
6 SYSTEM IMPLEMENTATION
- The backend implements target and comparison models and concept extractors in PyTorch, while Flask serves the application and React and D3 support the frontend.Super-pixels or patches are passed through the target model to obtain latent representations, with labels, predictions, and accuracy stored alongside them.Source: §6 · p. 5
- The system segments images into patches or super-pixels, scales them to the model input, and stores their latent representations and prediction metadata.The implementation uses scikit-image for super-pixel extraction and binary files for stored patch data.Source: §6 · p. 5
- For the stripe concept, active learning retrieves many super-pixels containing the pattern after the user identifies stripe examples associated with errors.The retrieved concept is then available for further model analysis.Source: Figure 4 · p. 6
7 EXAMPLE USAGE SCENARIOS
ConceptExtract is demonstrated on ImageNet classification and lost-cargo road segmentation, using human-defined concepts to inspect predictions, compare models, and investigate mitigation strategies. The examples include improved concept retrieval and model performance after shadow augmentation.
7.1 ResNet-101 for Image Classification
- The ImageNet study analyzes the ten worst-performing classes using 200 sampled images per class and Quickshift super-pixels.The concept extractors use the ResNet-101 final convolutional layer’s latent representations.Source: §7.1 · p. 6
- The ImageNet analysis finds that human-face features are relevant to tench predictions, while stripes inform feline predictions and accompany confusion among tiger cat, tiger, and tabby cat.The authors relate face reliance to training images showing people holding tench; stripe-containing images often yield mistakes among the three feline classes.Source: Figure 5 · p. 6, §7.1 · pp. 6–7
- 81.8% accuracy for the ensemble outperforms DenseNet (80.0%) and ResNet-101 (80.5%) on the ten selected classes.The ensemble averages the models’ class probabilities and corrects misclassifications made by both models.Source: Figure 6 · p. 7, §7.1 · p. 7
7.2 FCN and Semantic Segmentation
- For lost-cargo segmentation, the study uses 2,533 patches and finds that DenseNet-FCN performs worse on shadow-containing images, with shadow influencing predictions for all three labels.The study uses rectangle patches to preserve context and fine annotations as ground truth for concept evaluation.Source: §7.2 · p. 7
- Shadow augmentation improves the fine-tuned DenseNet-FCN and also improves IoU accuracy for DeepLabV3+.Artificial shadows are generated by reducing brightness on one side of a random boundary through a lost-cargo bounding box, with Gaussian blur around the boundary.Source: §7.2 · p. 7
- Dogs-concept precision improves significantly across active-learning stages, especially among the top 50 retrieved patches.Fine annotations serve as ground truth for evaluating the concept extractor.Source: §7.2 · p. 7, Figure 7 · p. 8
8 EXPERIMENTAL VALIDATION
Experiments show that human input improves concept-extractor quality, while the task model’s latent representations strongly affect which concepts can be extracted. Additional checks examine sampling, architecture, candidate prioritization, and image-labeling choices.
8.1 Concept Quality vs. Sampling Strategy
- Shadow augmentation makes the fine-tuned FCN more accurate, although the artificially generated shadows are not realistic.Table 1 reports IoU accuracy by semantic-segmentation class for baseline and shadow-augmented models.Source: Figure 8 · p. 8, Table 1 · p. 8
- All active-learning models outperform the baseline; standard objects reach 80% accuracy for the top 100 selections using about 3% of the data, while bobby cars approach upper-bound performance using 7% of positive data.Bobby cars remain harder to extract: even using all data, precision after the top 50 selections is below 0.5.Source: §8.1 · p. 8, Figure 9 · p. 9
8.2 Concept Quality vs. Latent Representation
- VGG-16 embeddings outperform FCN embeddings for all three concepts; for bobby cars and dogs, top-50 precision is roughly 0.7 and 0.9 versus about 0.4 and 0.5, respectively.The authors relate extraction limits to the quality of the task model’s latent representations.Source: §8.2 · p. 9, Figure 10 · p. 9
- The simple concept-extraction network can extract concepts, and the authors associate its limits with whether the task model’s representations distinguish those concepts.FCN embeddings do not separate cargo types such as bobby cars and dogs as readily as VGG-16 embeddings.Source: §8.2 · pp. 8–9
8.3 Other Considerations
- Extra hidden layers make convergence harder, removing layers lowers accuracy, confident-but-wrong base-model images are prioritized, and brushing performs similarly to lassoing.The authors choose brushing as the simpler labeling interface.Source: §8.3 · p. 9
9 LIMITATIONS, FUTURE WORK, AND CONCLUSION
The authors conclude that ConceptExtract’s visualization-assisted active-learning loop extracts customizable, human-understandable concepts. They identify unresolved questions about interface contributions, scale, and real-world evaluation.
9 LIMITATIONS, FUTURE WORK, AND CONCLUSION
- The prototype’s interface contributions remain unclear, including how clustering, module choices, and patch layout affect users’ selections.The authors identify these questions as future work for understanding the role of human expertise.Source: §9 · p. 9
- The current system supports limited training data and image patches, and its interface choices may not work effectively with more classes and images.The authors also say the impact of showing users more or fewer images needs further study.Source: §9 · p. 9
- A full evaluation in real-world applications is still needed to complement the preliminary expert evaluation.The authors also call for more studies of how machine-learning experts use concept-based explanations to understand and improve models.Source: §9 · p. 9
- The paper presents a visualization-assisted active-learning loop in ConceptExtract to extract customizable, human-understandable concepts for interpreting neural networks.The authors support this conclusion with case studies and experimental validations.Source: §9 · p. 9