Source-linked AI summary
Dependency Decomposition and a Reject Option for Explainable Models
Jan Kronenberger, Anselm Haselhoff
TL;DR
Deep neural networks can perform well but remain black boxes, and the dependencies between classification outputs and explanations are not established. The paper analyzes these dependencies and uses explaining attributes to verify predictions with a reject option. It reports that attribute-dependent models can match or improve on the reference, and that rejection increases accuracy across its models.
Problem
Deep networks are often treated as black boxes, and how classification outputs and explaining variables depend on one another remains an open question for explainable models.
Source: §1 · p. 1Method
The paper jointly learns class outputs and explaining variables, decomposes their dependencies, and uses attributes to verify predictions with an option to reject them.
Source: §1 · pp. 1–2Results
Models with attribute dependencies can have comparable or increased accuracy to the reference, and adding the reject option increases accuracy for all evaluated models.
Source: §5 · p. 6Takeaways & Limitations
The authors recommend models such as M-CDIA, where the classifier can benefit from an attribute dependency while retaining a low rejection rate.
Source: §5 · p. 6The supervised attributes may not capture the best representation from the network’s perspective, although they are guaranteed to be human-understandable.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Deploying machine learning models in safety-related do-mains (e.g. autonomous driving, medical diagnosis) demands for approaches that are explainable, robust against adversarial attacks and aware of the model uncertainty. Recent deep learning models perform extremely well in various inference tasks, but the black-box nature of these approaches leads to a weakness regarding the three requirements mentioned above. Recent advances offer methods to visualize features, describe attribution of the input (e.g.heatmaps), provide textual explanations or reduce dimensionality. However,are explanations for classification tasks dependent or are they independent of each other? For in-stance, is the shape of an object dependent on the color? What is the effect of using the predicted class for generating explanations and vice versa? In the context of explainable deep learning models, we present the first analysis of dependencies regarding the probability distribution over the desired image classification outputs and the explaining variables (e.g. attributes, texts, heatmaps). Therefore, we perform an Explanation Dependency Decomposition (EDD). We analyze the implications of the different dependencies and propose two ways of generating the explanation. Finally, we use the explanation to verify (accept or reject) the prediction
1. Introduction
Safety-critical use of deep neural networks calls for decisions users can understand, yet their predictions can be poorly calibrated and difficult to explain. This work jointly learns class predictions and explaining attributes, analyzes their dependencies, and uses attributes to verify predictions with a reject option.
1. Introduction
- Safety-critical applications need comprehensible predictions, while deep networks are often black boxes and their confidence estimates can be poorly calibrated [10].The introduction contrasts these models with classical methods whose decisions can be explained by introspection and whose temporal fusion methods can provide uncertainty measures.Source: §1 · p. 1
- Understanding internal representations may help make network decisions comprehensible and reduce the chance of malfunctions or successful attacks.The introduction also notes their potential use in developing decision systems that are not biased, unfair, or racist.Source: §1 · p. 1
- Attributes such as color, shape, and symbols can explain predictions as text or input regions and can be used with their joint probability to verify or reject a prediction.The approach adds an auxiliary explanatory model alongside the base network that performs the intrinsic task.Source: §1 · p. 1, Figure 1 · p. 2
- The paper jointly learns explaining attributes and classification outputs, then decomposes their dependencies to analyze relations between explanations and the intrinsic task.It also introduces appropriate network architectures and evaluates the approach on the German Traffic Sign Recognition Benchmark (GTSRB,).Source: §1 · pp. 1–2
2. Related Work
Prior explanation methods include feature visualization, attribution, and descriptive sentences, but can be difficult to understand or poorly tailored to a particular decision. Selecting useful attributes also requires prior knowledge, while unsupervised attributes may be hard to label or interpret.
2. Related Work
- Feature visualizations can be difficult for users to understand because they are high-dimensional, while image-caption systems may produce generic rather than decision-specific explanations [23].The paper addresses this problem using general as well as specific attributes.Source: §2 · p. 2
- Choosing prediction-explanation attributes is hard because it requires prior knowledge; handcrafted attributes can be simple [42], while network-selected unsupervised attributes may be difficult to label and falsely interpreted.Source: §2 · p. 2
- Input modifications and adversarial noise can confuse DNNs, and the method can detect unclear inputs when necessary attributes are missing.The passage distinguishes changes to real objects, such as contamination or graffiti, from adversarial attacks that add a small amount of noise [6] [9] [20].Source: §2 · p. 2
3. Methods
The method jointly models class outputs and explaining attributes, then decomposes their dependencies in alternative ways. It uses the resulting attributes to explain and verify predictions, with rejection when the verification condition is not met.
3. Methods
- The method jointly learns the output class and explaining attributes using a cross-entropy loss over their joint distribution.The parameters are divided into feature-extractor, classifier, and attribute components.Source: §3 · pp. 2–3, §3.1 · p. 3
3.1. Explanation Dependency Decomposition (EDD)
- Explanation Dependency Decomposition (EDD) compares alternative factorizations of the class–attribute distribution and their associated dependency assumptions.The reference model omits explaining variables; Table 1 summarizes dependencies and equations across the models.Source: §3.1 · p. 3, Table 1 · p. 3
3.1.1 Full Independence (M-FI)
- M-FI assumes independent classifiers, so attributes and class provide no additional information to one another given the input.Both branches may affect feature-extractor parameter updates, but their dedicated parameters are not updated by the other branch’s loss.Source: §3.1.1 · p. 3
3.1.2 Attributes with a Class-Dependency (M-IACD, M-DACD)
- M-IACD makes attributes class-dependent but mutually independent, while M-DACD also assumes attribute independence.M-IACD uses ground-truth labels during early training to address instability from noisy class outputs; its explanation branch can influence the classifier.Source: §3.1.2 · p. 3, Figure 2 · p. 4
3.1.3 Class with a Attribute-Dependency (M-CDIA, M-CDDA)
- M-CDIA makes the class depend on attributes while assuming the attributes are mutually independent; M-CDDA also preserves dependencies among attributes.This decomposition shifts focus toward using possible explanations to support classification performance.Source: §3.1.3 · p. 4
3.2. Explanation
- The model explains each image-specific decision through the presence or absence of attributes visible in that image.The predicted class and attributes form the most plausible class–explanation combination; the explanation may be affected by internal dependencies.Source: §3.2 · p. 4
3.3. Reject Option
- Attribute-based verification accepts a predicted class only when it is the sole class meeting the defined conditions; otherwise, it rejects.The conditions may be necessary or sufficient attributes, and the method uses Belnap’s True category for acceptance while the other categories trigger rejection.Source: §3.3 · pp. 4–5
4. Experiments
On GTSRB, the experiments compare dependency models using explanatory attributes and test attribute-based rejection. The results show that dependency choices affect classification and explanation performance, while rejection increases accuracy on accepted predictions.
4.1. Dataset
- Synthetic samples make up approximately 39% of GTSRB and vary attributes to reduce accidental learning of class-specific properties.For example, the samples include square signs with different main colors because the priority-road sign alone otherwise combines a square shape with yellow.Source: §4.1 · p. 5, Figure 3 · p. 5
4.2. Architecture
- The AlexNet-based 48×48 architecture predicts simple color, shape, and complex symbol or number attributes from progressively deeper feature layers.Colors are predicted after the second convolution layer, shapes after the third, and symbols and numbers after the fourth.Source: §4.2 · p. 5
4.3.1 Class predictions and explanations
- M-CDDA and M-CDIA improve class prediction over M-REF (92.20%), while M-DACD gives the best explanations and class-specific models predict classes less accurately.Full independence leaves classifier performance unchanged from M-REF but produces explanations of lower quality than the other models.Source: §4.3.1 · p. 5
4.3.2 Class predictions with reject option
- With the reject option, class-prediction accuracy increases by almost 6%; M-CDIA again achieves the best class-prediction accuracy with an 11.72% rejection rate, while M-DACD has the lowest rate, 10.15%.Source: §4.3.2 · p. 6
5. Conclusion
The conclusion presents attribute-based verification and rejection as ways to support, explain, and check network predictions. It recommends dependency models that benefit classification while keeping rejection rates low, and notes a boundary of supervised attributes.
5. Conclusion
- Additional attributes can support, verify, and explain network predictions; with suitable dependencies, they can also improve classification accuracy.The conclusion reports comparable or increased accuracy for models other than M-IACD, whose accuracy is slightly below M-REF.Source: §5 · p. 6
- The authors recommend models such as M-CDIA, where classification benefits from attribute dependencies and rejection rates remain low.They report that rejecting ambiguous inputs increases accuracy across their models and that verification gives a reason for rejection.Source: §5 · p. 6
- Supervised attributes are human-understandable but may not represent the network’s best possible internal representation.The dependency decomposition can also be used or combined with unsupervised explanation procedures.Source: §5 · p. 6