Source-linked AI summary
DeepView: Visualizing Classification Boundaries of Deep Neural Networks as Scatter Plots Using Discriminative Dimensionality Reduction
Alexander Schulz, Fabian Hinder, Barbara Hammer
TL;DR
Deep-network decisions can be difficult to understand, while many explainability approaches focus on individual inputs rather than decision boundaries and generalization alongside data. DeepView uses classifier-informed dimensionality reduction to visualize an approximate decision function, and the paper demonstrates it on CIFAR-10 and poisoned Fashion-MNIST, reporting strong projection accuracy with the Fisher metric.
Problem
Many explainability methods focus on single decisions, leaving comparatively little effort to visualize deep-network decision boundaries and generalization with data; deployed models also face vulnerability and fairness concerns.
Source: §1 · pp. 1–2Method
DeepView uses a Fisher-metric formulation that can handle non-differentiable classifiers and admits a faster approximation for complex models.
Source: §4.1 · p. 6Results
On CIFAR-10, Fisher-metric accuracy was 96.3% versus 18.3% for QkNN-E; Qdata was close to perfect, and Q¬data was 83.3%.
Source: §5.3 · p. 11Takeaways & Limitations
The authors suggest the visualizations may contribute to model improvement by revealing areas with insufficient data.
Source: §6 · p. 15The demonstrations use image data; extending the Fisher metric to temporal data remains an open question.
Source: §6 · p. 15
Abstract
from arXiv · showhide
Machine learning algorithms using deep architectures have been able to implement increasingly powerful and successful models. However, they also become increasingly more complex, more difficult to comprehend and easier to fool. So far, most methods in the literature investigate the decision of the model for a single given input datum. In this paper, we propose to visualize a part of the decision function of a deep neural network together with a part of the data set in two dimensions with discriminative dimensionality reduction. This enables us to inspect how different properties of the data are treated by the model, such as outliers, adversaries or poisoned data. Further, the presented approach is complementary to the mentioned interpretation methods from the literature and hence might be even more useful in combination with those. Code is available at https://github.com/LucaHermes/DeepView .
1 Introduction
Deep learning’s expanding use brings vulnerability, unfairness, and explainability concerns. The paper addresses the limited availability of visualizations that show deep-network decision boundaries and generalization alongside data.
1 Introduction
- Deep-learning systems can behave unpredictably under adversarial attacks or poisoning, and skewed training samples can produce biased or unfair models.These risks arise alongside deployment in areas including autonomous driving, medical diagnosis, and recommender systems.Source: §1 · p. 1
- Many explainable-AI methods examine individual decisions, while visualizing parts of a deep network’s decision boundary and generalization behavior with data remains comparatively underexplored.The paper identifies high-dimensional input spaces as a key obstacle to such scatter-plot visualizations.Source: §1 · p. 2
- DeepView combines classifier-informed discriminative dimensionality reduction with an approximate inverse mapping to visualize a trained network’s decision function alongside benchmark data.The authors also demonstrate the pipeline on popular deep-learning models and datasets.Source: §1 · p. 2
2 Related Work
The related work contrasts methods that visualize data groups with methods that depict classifier boundaries. It also situates DeepView within discriminative dimensionality reduction using the Fisher metric.
2 Related Work
- While many explainable-AI approaches explain single model decisions, only few provide a large-scale view of a trained model or visualize its decision boundaries; notable exceptions [15] project data interpreted through different task-solving strategies.Source: §2 · p. 2
- A separate approach depicts classifier boundaries using input-space density estimation, which the authors describe as infeasible for typical deep-network domains.Source: §2 · p. 3
- Prior work found Fisher-metric DiDi visualizations better or at least comparable to other formalizations, motivating the paper’s focus on that metric.Source: §2 · p. 3
3 DeepView: visualizing the decision function of a deep network
Directly plotting a classifier’s behavior is straightforward in two dimensions, but high-dimensional inputs make ordinary dimensionality reduction and exhaustive evaluation inadequate. DeepView approximates the decision function by combining classifier-informed projection with an inverse mapping and predictions on a projected grid.
3 DeepView: visualizing the decision function of a deep network
- In two dimensions, a classifier can be evaluated around observed data and its predicted labels and certainty encoded in the scatter-plot background.Source: §3 · p. 3
- For high-dimensional inputs, ordinary dimensionality reduction may preserve irrelevant structure, exhaustive classifier evaluation is infeasible, and projecting the full decision function risks information loss.For images, the paper gives brightness differences rather than object-relevant properties as an example of a structure ordinary reduction may preserve.Source: §3 · p. 3
3.1 DeepView: Proposed Visualization Scheme
- DeepView uses classifier-based DiDi to emphasize data properties relevant to the model, then approximates the decision function over a relevant subspace rather than the full input space.Its inverse dimensionality reduction is constructed to match the forward reduction.Source: §3.1 · pp. 3–4
- In the toy illustration, an orange island in the visualization indicates overfitting caused by a single orange training point.The authors say their novel implementations of projection and inverse projection enable the approach to visualize deep neural networks.Source: Figure 1 · p. 4, §3.1 · p. 4
- The pipeline projects data with Fisher UMAP, maps a regular two-dimensional grid back to input space, evaluates the network there, and displays predicted labels and predictive entropy.Source: §3.1 · p. 4
4 Dimensionality reduction
The section develops UMAP-based dimensionality reduction, makes it classifier-aware with a path-sensitive metric, and constructs a fitted inverse map.
4 Dimensionality reduction
- UMAP embeds data by minimizing the KL divergence between source-space and embedding edge probabilities, with source probabilities shaped by endpoint distance and local density.The embedding-space edge weights use a Student-t distribution.Source: §4 · p. 5
4.1 Discriminative dimensionality reduction
- The classifier-aware DiDi metric combines changes in class probabilities with source-space distance and measures shortest paths, enabling deep-network visualizations without gradients or density estimation.The paper presents this as a way to compute DiDi visualizations for high-dimensional data and deep networks.Source: §4.1 · pp. 6–7
4.2 Similarity coordinate embedding for inverse dimensionality reduction
- For inverse dimensionality reduction, the method fits π−1 by swapping UMAP’s source and embedding similarity roles; Euclidean training performed comparably to training with local Fisher-matrix approximations.The analysis considers Euclidean or Riemannian source metrics; under Gaussian input noise, the Euclidean cost approximation converges in mean exponentially fast as dimensionality increases.Source: §4.2 · pp. 8–9
5 Experiments
The experiments evaluate DeepView’s embedding and inverse mapping, then apply the visualization to CIFAR-10 and poisoned Fashion-MNIST data to inspect classifier behavior, adversarial examples, and backdoors.
5.1 Evaluating the Proposed Visualization
- The evaluation uses 5-nearest-neighbor leave-one-out accuracy to assess whether projected neighbors share the network’s classifications, and inverse-map label agreement on training and unseen positions.The latter measures are Qdata for positions used to train the inverse map and Q¬data for positions not used in its training.Source: §5.1 · pp. 9–10
5.2 Hyperparameter Selection
- The experiments set Euclidean regularization λ to 0.65 for both datasets; UMAP uses 30 neighbors and min_dist 1 for Fashion-MNIST.For π−1, a is set to the smallest value that does not cause a large drop in Qdata, and b is set to 1.Source: §5.2 · p. 10
5.3 Visualizing a ResNet-20 Network Trained on the CIFAR10 Dataset
- 96.3% with the Fisher metric versus 18.3% with Euclidean QkNN-E; Q¬data accuracy is 83.3%, while Qdata is described as close to perfect.Table 1 summarizes the evaluation criteria.Source: §5.3 · p. 11, Table 1 · p. 11
- On CIFAR-10, the orange class has complex-looking boundaries and the lowest true positive rate, 83.2%; an adversarial example is classified as orange inside a yellow-class area.The adversarial example was created using the sign of the model gradient and added to the 300 images.Source: §5.3 · p. 11
- Inverse-mapped CIFAR-10 positions expose a car-class pocket among ship-like images, while a bird-like marker is classified as ship.The authors suggest saliency maps for further investigation of a ship with an intense blue area that is misclassified.Source: §5.3 · p. 12
5.4 Fashion-MNIST with Backdoors
- Fashion-MNIST reveals a shoe grouping and T-shirt/Shirt overlap; inverse mapping identifies patterned bags classified as trousers, exposing the backdoor.The visualization includes 600 test points, including 20 backdoored samples.Source: §5.4 · pp. 12, 14
- Computing Fisher distances for the Fashion-MNIST visualization took around 90 seconds on a standard desktop with a consumer-level graphics card; direct computation scales quadratically with data size.The authors note that approximate schemes such as UMAP’s are applicable in principle.Source: §5.4 · p. 14
6 Conclusion
DeepView visualizes a smooth two-dimensional manifold of a deep network’s decision function, and the paper demonstrates it on two image-classification settings. The authors suggest these visualizations can inform model improvement, while noting that temporal data raises an open question for the Fisher metric.
6 Conclusion
- DeepView visualizes a smooth two-dimensional manifold of a deep neural network’s decision function on high-dimensional data such as natural images.The authors describe it as, to their knowledge, the first algorithm to do so.Source: §6 · p. 14
- The demonstrations describe layers trained on CIFAR-10 and a 4-layered ConvNet trained on poisoned Fashion-MNIST, illustrating how the method provides insight into the model and data.Source: §6 · p. 15
- Although demonstrated only on image data, the method is not restricted to images; extending its Fisher metric to temporal data remains an open question.Source: §6 · p. 15
- The authors suggest visualizations may help improve trained models by revealing areas with insufficient data.Source: §6 · p. 15
A Appendix: Proofs
The appendix establishes a relationship between the λ=0 DiDi metric and the classifier-induced Fisher metric, and analyzes an approximation to the inverse-projection cost. Its convergence guarantee assumes Gaussian noise in the input space.
A Appendix: Proofs
- For λ=0, the appendix relates the DiDi metric to the Fisher metric induced by the classifier.The proof invokes the corresponding curve-length relationship and their arc-length metric properties.Source: A Appendix: Proofs · p. 18
- The inverse-projection cost approximation f̂ is bounded by f and, under Gaussian input noise in mean, converges in mean exponentially fast as dimension increases.Source: A Appendix: Proofs · p. 18
- The appendix derives the approximation error as nonnegative, using that the relevant class probabilities lie between zero and one.Source: A Appendix: Proofs · p. 18