Source-linked AI summary
Making deep neural networks right for the right scientific reasons by interacting with their explanations
Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, Kristian Kersting
TL;DR
Deep networks can achieve high accuracy by exploiting confounding factors, leaving their scientific decisions difficult to trust. The paper introduces XIL, in which scientists revise models through feedback on explanations, using counterexamples or RRR-based modification. Experiments show that XIL can avoid Clever Hans-like behavior and may improve predictive performance while influencing trust in model decisions.
Problem
Deep networks may achieve high accuracy by relying on confounding factors, while prior explanation methods identified such behavior without offering a correction solution.
Method
XIL adds scientists to the training loop, using feedback on explanations to augment training with counterexamples or modify models with the RRR loss.
Results
XIL can help avoid Clever Hans-like moments, revise models toward trustful decision strategies, and may improve predictive performance.
Takeaways & Limitations
Interactive explanation feedback can support models that are right for scientifically appropriate reasons and help users calibrate trust in their decisions.
Takeaways & Limitations
XIL assumes faithful explanations and faithful user feedback, and available data may not permit removing wrong reasons without reducing predictive performance.
Abstract
from arXiv · showhide
Deep neural networks have shown excellent performances in many real-world applications. Unfortunately, they may show "Clever Hans"-like behavior -- making use of confounding factors within datasets -- to achieve high performance. In this work, we introduce the novel learning setting of "explanatory interactive learning" (XIL) and illustrate its benefits on a plant phenotyping research task. XIL adds the scientist into the training loop such that she interactively revises the original model via providing feedback on its explanations. Our experimental results demonstrate that XIL can help avoiding Clever Hans moments in machine learning and encourages (or discourages, if appropriate) trust into the underlying model.
Explanatory Interactive Machine Learning (XIL)
Explanatory interactive learning (XIL) places domain experts in the training loop to correct models’ explanations, targeting predictions that rely on confounding factors. Across benchmark and plant-phenotyping tasks, XIL revised decision strategies toward biologically plausible regions while generally preserving or sometimes improving performance and supporting calibrated trust.
- XIL framework: XIL lets learners query users with predictions and explanations, then incorporates corrective feedback on labels or decision reasons.The framework covers right-right, wrong-wrong, and right-wrong cases, with the last being XIL’s main target.
- XIL framework: Counterexample-based XIL converts marked irrelevant components into examples that vary those components while retaining the model’s prediction.The strategy teaches the learner not to depend on components identified as irrelevant by the explanation.
- Benchmark results: 48% to 82% accuracy: a single counterexample sharply reduced confounder effects in the decoy fashion-MNIST experiment.With more counterexamples, counterexample training reached accuracy similar to the RRR method.
- Benchmark results: 78% default versus 73% XIL-trained test accuracy on the five-class PASCAL VOC subset, while feedback removed reliance on the source-tag confounder.Similar explanation-alignment benefits were observed on MSCOCO using hint-like extensions.
- Plant phenotyping: 99% HS and 88% RGB balanced accuracy exposed suspiciously strong plant-disease classification performance before explanation-based revision.The HS model used confounding dataset factors, and the authors caution that laboratory accuracy may not reflect performance outside the lab.
- Plant phenotyping: RRR revision redirected the HS model from confounding factors toward tissue and CLS regions, although performance slightly decreased and RGB background reliance persisted.The corrected HS-CNN still classified samples without visible symptoms.
- Trust study: Users trusted highly accurate machines less after witnessing wrong behavior, while correct explanations could increase trust at lower predictive performance.Users did not forgive incorrect explanations when predictions were correct, underscoring concern about being right for the wrong reasons.
- Overall findings: Across empirical results, XIL eliminated or reduced Clever Hans-like behavior and may improve predictive performance while shaping trust in the model.The paper presents this as an empirical conclusion rather than a guarantee for every dataset or setting.
Conclusion
The paper introduces explanatory interactive learning (XIL) to involve scientists in revising models through feedback on their explanations. It concludes that XIL can help avoid Clever Hans behavior, while noting interaction costs, performance trade-offs, and reliance on faithful explanations and feedback.
- Conclusion: XIL adds the scientist to the training loop, using feedback on model explanations to revise the model through counterexamples or the RRR loss.The approach is presented as a way to “un-Hans” machines by correcting their reasons for making predictions.
- Conclusion: The paper reports that users care strongly about Clever Hans-like behavior and that XIL can help avoid it.This conclusion is based on the paper’s experimental results and user study.
- Conclusion: XIL may require time-consuming explanation annotations and an uncertain number of user interactions to reach an acceptable state.The authors identify optimal query strategies as a future direction for reducing interaction effort.
- Conclusion: The data may prevent XIL from fully removing wrong reasons without reducing predictive performance.The authors therefore call for methods that keep any performance drop as small as possible.
- Conclusion: XIL assumes that faithful explanations can be computed and that user feedback is faithful.The paper notes that faithful explanations remain an active research problem, especially for deep learning.
Methods
The paper combines active learning, local explanations, explanation feedback, and model-revision strategies to correct misleading decision rationales. Methods include counterexamples, gradient-based regularization, and analysis of learned decision strategies.
- Active learning: Active learning selects unlabeled instances for expert labeling while updating the model with the acquired examples.The learner maintains labeled and unlabeled sets and aims to improve model quality while limiting query cost.
- Explanation methods: Local explainers such as LIME and grad-CAM explain individual predictions, enabling users to inspect model reasoning.LIME locally approximates decisions with interpretable components, while grad-CAM uses convolutional representations to produce spatial explanations.
- Counterexamples: Counterexamples randomize irrelevant features while preserving labels, encouraging classifiers to become orthogonal to correction directions.This strategy adds constraints that eliminate classifiers relying on the randomized, irrelevant feature.
- Datasets: Experiments use confounded fashion-MNIST and PASCAL VOC subsets, alongside RGB and hyperspectral plant images.The fashion-MNIST confounders are label-dependent in training but randomized in testing; the VOC subset contains watermark confounders for horse images.
- Strategy analysis: The study analyzes decision strategies by applying a discrete Fourier transform to grad-CAM heatmaps before spectral clustering.This pipeline is used to better differentiate strategies identified from individual samples.
Statement of ethical compliance
The study reports compliance with relevant ethical regulations and obtained informed consent from every user-study participant.
- The authors state that they complied with all relevant ethical regulations.
- The ethics statement identifies the TU Darmstadt Ethics Commission as the reviewing body.
- Each participant provided informed consent before the user study began.
Author information
The paper involves researchers from institutions in Germany and Italy, with contributions spanning study design, method development, biological interpretation, and dataset creation.
- The listed affiliations include the Technical University of Darmstadt and the University of Trento.
- Additional affiliations include the University of Bonn, the Institute of Sugar Beet Research, and LemnaTec GmbH.
- The authors designed the study, developed XIL extensions, interpreted data, drafted the manuscript, and created the phenotyping dataset.
- Patrick Schramowski and Wolfgang Stammer are listed as correspondence contacts.