Source-linked AI summary
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Fred Hohman, Minsuk Kahng, Robert Pienta, Duen Horng Chau
TL;DR
Deep learning models are powerful but difficult to interpret, creating challenges for understanding their decisions, failures, and improvement needs. This paper surveys visual analytics research through a human-centered Five W’s and How framework, covering model understanding, visualization practices, and research directions. The survey synthesizes a rapidly growing field while identifying scalability and evaluation as important limitations and open challenges.
Problem
Deep neural networks’ complex, nonlinear structures make their learned representations and decision processes difficult to interpret, limiting understanding of model behavior.
Method
The paper conducts a comprehensive survey of deep learning visualization and visual analytics using a human-centered interrogative framework organized by Why, Who, What, How, When, and Where.
Results
The survey synthesizes visual analytics tools and techniques spanning model interpretation, training, user interaction, security, and related research directions.
Takeaways & Limitations
The survey helps visual analytics and deep learning researchers and practitioners quickly learn key aspects of this young, rapidly growing research area.
Takeaways & Limitations
The survey focuses on visual analytics and gives limited detailed treatment to algorithmic attribution and feature-visualization techniques because those works primarily contribute algorithms rather than interactive systems.
Abstract
from arXiv · showhide
Deep learning has recently seen rapid development and received significant attention due to its state-of-the-art performance on previously-thought hard problems. However, because of the internal complexity and nonlinear structure of deep neural networks, the underlying decision making processes for why these models are achieving such performance are challenging and sometimes mystifying to interpret. As deep learning spreads across domains, it is of paramount importance that we equip users of deep learning with tools for understanding when a model works correctly, when it fails, and ultimately how to improve its performance. Standardized toolkits for building neural networks have helped democratize deep learning; visual analytics systems have now been developed to support model explanation, interpretation, debugging, and improvement. We present a survey of the role of visual analytics in deep learning research, which highlights its short yet impactful history and thoroughly summarizes the state-of-the-art using a human-centered interrogative framework, focusing on the Five W's and How (Why, Who, What, How, When, and Where). We conclude by highlighting research directions and open research problems. This survey helps researchers and practitioners in both visual analytics and deep learning to quickly learn key aspects of this young and rapidly growing body of research, whose impact spans a diverse range of domains.
1 INTRODUCTION
Deep learning has achieved strong results and become easier to develop, but its complex decision processes create pressing needs for interpretation, debugging, and model understanding. Visual analytics organizes these needs through a human-centered Five W’s and How framework.
- Deep learning uses deep neural networks to learn structured data representations and now produces state-of-the-art results across diverse tasks.
- Open-source toolkits have lowered the barrier to building, training, and evaluating deep learning models, accelerating their adoption.
- Neural networks remain difficult to interpret because their learned representations and underlying decision processes lack transparency.
- Visual analytics supports insight into learned features and can help developers understand and debug deep learning models.
- The survey organizes deep learning visualization research around Why, Who, What, How, When, and Where to capture its needs, users, techniques, timing, and application contexts.
2 OUR CONTRIBUTIONS & METHOD OF SURVEY
The survey contributes a comprehensive review of deep learning visual analytics using a human-centered interrogative framework and a broad literature scope. It also distills research directions concerning interpretability, evaluation, human roles, and ethical AI use.
- The survey uses the Five W’s and How to position existing deep learning visualization works and describe their multifaceted contributions.
- Its human-centered interrogative process helps readers grasp key facets of a young research area and provides a basis for describing and developing new work.
- The literature review spans visualization, visual analytics, artificial intelligence, machine learning, deep learning, and computer vision venues, including relevant workshops.
- It identifies open problems in visual analytics capabilities, system evaluation, human participation, and the ethical use of AI applications.
- Survey Methodology & Summarization Process: The survey records metadata, approaches, design components, industry involvement, and open-source code before organizing works with the interrogative framework.
- The survey structure gives separate sections to the six interrogative questions and concludes with research directions and open problems.
3 COMMON TERMINOLOGY
The survey provides a terminology table to help readers follow technical terms and synonyms used throughout its discussion of deep learning.
- Table 3 tabulates relevant and common deep learning terminology as a quick reference for technical terms, meanings, and synonyms.
4 WHY VISUALIZE DEEP LEARNING
Deep learning visualization is motivated primarily by the need to understand model decisions and learned representations, while supporting interpretation, debugging, comparison, education, and applications across domains.
- Interpretability and Trust: Model understanding is sought to explain decisions and learned representations, helping people place trust in deep learning models.Neural networks’ large parameter counts and internal complexity make this especially challenging.
- Interpretability and Trust: Interpretability and explanation are not universally defined, making interpretations and explanations difficult to classify and qualify.An explanation may show predictions without revealing the mechanisms by which a model works.
- Interpretability and Trust: Human-centered explanation research argues that explanations should reflect how human observers understand and evaluate decisions, not only AI researchers’ intuitions.Miller’s work provides a social-science perspective on what constitutes a good explanation.
- Applied Contexts: Deep learning interpretation has been applied to domains including translation, social-problem analysis, autonomous driving, medical imaging, and urban imagery.The survey also catalogs techniques for embeddings, text, interpretability quantification, and image-based interpretation.
- Debugging and Improvement: Visualization supports iterative model development by helping developers tune architecture and hyperparameters, debug models, and improve performance.Systems address tasks such as identifying stable or unnecessary layers, degenerated filters, and training dynamics.
- Comparison and Selection: Visualization also supports model comparison and selection by exposing metrics and model behavior when choosing among successfully trained models.Relevant criteria include accuracy, loss, generalizability, and avoiding memorization or overfitting.
5 WHO USES DEEP LEARNING VISUALIZATION
Deep learning visualization serves three overlapping audiences: model developers, model users, and non-experts, with tools differing in technical depth and purpose.
- Audience Groups: The survey groups beneficiaries into model developers, model users, and non-experts according to deep learning knowledge, while noting that groups can overlap.A model developer may also use a non-expert educational tool to reinforce intuition.
- Model Developers: Model developers need technically focused tools for experimenting with, deploying, debugging, and controlling neural networks.Their tools may expose many hyperparameters and support large-scale model building.
- Model Developers: Developer-oriented systems include TensorBoard, DeepEyes, Blocks, ML-o-scope, and training-time visualization tools for model analysis and improvement.These systems address dataflow inspection, layer and filter diagnosis, convergence, overfitting, hyperparameter tuning, and classification results.
- Model Users: Model users apply known architectures, train smaller models, or use pretrained weights for domain-specific applications.ActiVis supports engineers and data scientists by unifying instance- and subset-level inspections of neuron activations.
- Non-experts: Non-experts generally use visualization for education or to understand AI-powered systems without prior deep learning knowledge.TensorFlow Playground and Teachable Machines use direct manipulation or simple interactive feedback to build intuition about neural networks.
6 WHAT TO VISUALIZE IN DEEP LEARNING
Visualizable deep learning components include architectures, parameters, activations, gradients, embeddings, instance groups, and model metrics, each exposing different aspects of model behavior.
- Architecture and Parameters: Model architecture and dataflow graphs show how operations are connected and how data moves through a network.The computational graph also defines training, testing, saving, and checkpointing behavior.
- Architecture and Parameters: Edges, weights, convolutional filters, and learned parameters reveal the signals and transformations adjusted during training.Weights connect neurons across layers, while convolutional filters generate smaller representations of input data.
- Activations and Embeddings: Neuron activations record how a trained network transforms inputs into intermediate signals and final outputs during inference.Activations can be studied for individual instances or collections of instances.
- Gradients: Backpropagation gradients can be visualized to inspect where output error is distributed through the network.Gradients flow opposite to the forward signal, from output toward input layers.
- Activations and Embeddings: High-dimensional activations can be treated as feature vectors, enabling visualization of embeddings and transformations across layers.Instance groups allow simultaneous activation computation and offer advantages beyond single-instance inspection.
- Model Metrics: Loss, accuracy, and related metrics summarize model state over training and provide indicators for progress, overfitting, and model comparison.These summaries are useful but abstract away subtle features of neural networks.
7 HOW TO VISUALIZE DEEP LEARNING
Existing systems use recurring visual encodings for neural network components, especially node-link diagrams for architectures and scatter plots for reduced embeddings, while addressing clutter and reduction quality.
- Node-link Diagrams: Network architectures and dataflow graphs are commonly represented as node-link diagrams, with nodes for operations or neurons and links for connections or weights.Large complex models can produce hairballs, motivating edge bundling, super-groups, and richer node summaries.
- Dimensionality Reduction and Scatter Plots: Activation embeddings are commonly reduced to two or three dimensions and plotted as scatter plots, with each point corresponding to a data instance.Original images may be placed at reduced coordinates to connect projected points with input examples.
- Node-link Diagrams: CNNVis uses bi-clustering-based edge bundling to reduce clutter caused by the many links in convolutional neural network visualizations.Node-link diagrams work better for smaller networks but face scalability problems in larger ones.
- Dimensionality Reduction and Scatter Plots: Dimensionality-reduction scatter plots depend strongly on the chosen algorithm and its hyperparameters, affecting the quality and interpretation of embeddings.The survey discusses differences between PCA and t-SNE, t-SNE sensitivity, and interaction techniques for finding nearby instances.
7.3 Line Charts for Temporal Metrics
Visual analytics uses temporal metrics and instance-level views to monitor training, diagnose errors, and inspect how individual or grouped data move through deep learning models.
- Temporal metrics: Line charts commonly plot loss, accuracy, and error measures over training epochs to diagnose long model-training processes.These metrics are recomputed after each epoch.
- Instance-level analysis: Instance-level observation scrutinizes a single image or text instance and its transformation through a model.Individual instances are easier to understand than entire datasets or high-dimensional embeddings.
- Instance-level analysis: Instances can serve as unit tests by revealing whether familiar examples are classified correctly and how specific failures occur.Satellite-imagery analysis uses translucent heatmaps to show areas with high population-prediction errors.
- Instance-level analysis: Text-instance analysis often colors characters according to activation magnitudes to identify examples requiring deeper inspection.Misclassified instances are among the cases that may warrant further scrutiny.
- Instance-level analysis: Blocks uses confusion matrices, matrix-level sorting, and a sample viewer to guide users toward important instances when predefined examples are unavailable.It reveals that class errors often occur in hierarchies.
- Group-level analysis: Group-level analysis sacrifices some detail while enabling evaluation of average and aggregate performance across groups.Systems including ActiVis analyze preset and user-defined groups, with many approaches focusing on text and LSTM models.
7.5 Interactive Experimentation
Interactive experimentation lets users manipulate inputs, architectures, and hyperparameters to observe model responses and build intuition about deep learning behavior.
- Interactive experimentation: Direct manipulation supports “what if?” experimentation by letting users change inputs and observe resulting model outputs.This approach is called explorable explanations and is used to make complex concepts and systems understandable.
- Models responding to input data: User-provided inputs can drive real-time visualizations of neural-network internals, including webcam data, hand-drawn MNIST digits, and selectable shapes.ShapeShop retrains models on selected shapes and visualizes learned features through class activation maximization.
- Models responding to input data: MNIST is a benchmark dataset of 28×28px handwritten digits used for image classification models.The dataset contains digits from 0 to 9.
- Models responding to input data: TensorFlow Playground explores simple neural networks through direct manipulation rather than programming to teach concepts and develop behavioral intuition.The tool focuses on simple, fully connected neural-network architectures.
- Models responding to input data: Other interactive systems let users write words while visualizing predicted next strokes and activations, or generate images from colored sketches.These examples extend interactive experimentation to textual prediction and generative image models.
- Interactive model design: Visual analytics systems expose hyperparameters for experimentation because their settings can strongly affect model performance and robustness.TensorFlow Playground lets users adjust the architecture of a simple fully connected neural network.
7.6 Algorithms for Attribution & Feature Visualization
Algorithmic visualization methods generate image-based explanations of neural-network behavior, while the survey distinguishes their impact from its primary focus on interactive visual analytics.
- Algorithmic techniques: Attribution methods highlight important image regions, whereas feature visualization generates new images intended to represent a learned class or feature.Both methods can generate images the same size as the input.
- Survey scope: These algorithmic techniques are usually non-interactive and are discussed only selectively because the survey focuses on visual analytics systems.The survey points readers to taxonomies and literature surveys for detailed treatment.
- Attribution and heatmaps: Heatmaps overlay images to show regions contributing to classification and their sensitivity.Visual backpropagation can provide these visualizations in real time for debugging self-driving-vehicle models.
- Attribution and heatmaps: Representation inversion reconstructs images from feature vectors to help inspect what convolutional neural networks have learned.Prediction difference analysis highlights evidence for or against a particular class.
- Feature visualization: Class activation maximization uses optimization, such as gradient ascent, to maximize a chosen neuron’s activation and generate synthetic images for a selected class.Subsequent work improved generated-image quality, including photorealistic synthesis.
- Feature visualization: Despite improved image quality, researchers remain skeptical about whether feature visualizations reliably capture consistent neuron meanings across inputs.The concern concerns the trustworthiness of the representation, not merely image appearance.
8 WHEN TO VISUALIZE IN THE DEEP LEARNING PROCESS
Deep learning visualization is useful both during training and after training. During training it supports monitoring and model improvement, while afterward it supports inspection of trained models and representations.
- When to visualize: Visualization can be used during training, after training, or at both stages of the deep learning process.The survey identifies these as distinct but non-mutually exclusive times for visualization.
- During training: During training, visualization helps users monitor how models learn higher-level features and track performance iteratively.The training process is described as foundational and iterative.
- During training: Browser-based systems query model status and dynamically update loss, accuracy, and training-time charts after each epoch.The visualizations run alongside training and expose the latest model state.
- During training: Visualized learning metrics can indicate whether training is succeeding, enabling early stopping that reduces development time and computational resources.Other systems support model steering and provide suggestions for improvement.
- During training: DeepEyes identifies stable and unstable layers and neurons for pruning, while Blocks exposes hierarchical class confusion and DGMTracker traces contributions to failing neurons.These systems target different forms of training diagnosis and model improvement.
- After training: After training, visualization systems inspect trained models, embeddings, activations, and recurrent-network comparisons.The Embedding Projector visualizes two- and three-dimensional embeddings, while ActiVis and RNNVis support post-training interpretation.
9 WHERE IS DEEP LEARNING VISUALIZATION
Deep learning visualization research spans application domains and a hybrid community connecting visualization, visual analytics, artificial intelligence, and deep learning. Its rapid growth has produced tools, techniques, workshops, open-source releases, and broader dissemination.
- Applications: Deep learning visualization has been applied across domains, including neural machine translation, reinforcement learning, social good, and autonomous systems.
- Models and data: Most surveyed work focuses on CNNs and image data, while newer studies visualize RNNs, LSTMs, and GANs.
- Research community: Bringing visualization and AI communities together has produced tools and techniques aimed at improving deep learning interpretability and democratization.
- Research community: The field is disseminated across visualization, visual analytics, artificial intelligence, and deep learning venues, including workshops because the area is relatively new.
- Open dissemination: Preprints, industry blogs, open-source code, and tools broaden access, although preprints may provide little quality control.
10 RESEARCH DIRECTIONS & OPEN PROBLEMS
The survey identifies open problems in interpretability, scalability, evaluation, human-AI collaboration, democratization, bias, and security. It calls for richer visual interfaces, scalable systems, stronger user studies, human participation, and responsible AI practices.
- 10.1 Furthering Interpretability: Future interpretability work should combine new visual representations, helpful interactions, and faster attribution and feature visualization techniques.The survey suggests that rich user interfaces combining these elements could produce major breakthroughs in understanding neural networks.
- 10.2 System & Visual Scalability: Deep learning visual analytics systems face visual and system scalability challenges with large datasets, millions of parameters, activations, embeddings, and complex models.Dimensionality reduction has limits on the number of points it can display, while deployed systems should support fast, real-time interactions.
- 10.3 Design Studies for Evaluation: Utility & Usability: Evaluation should assess visual utility, system usability, and interpretability, with design studies and comparisons involving AI experts and other users.The survey notes that many AI-community papers do not include user studies and that proposed methods benefit from showing superiority over tested alternatives.
- 10.4 Human-AI Pairing: Future tools could use human feedback and model steering to support human-AI complementarity and improve model development.Examples include what-if tools that expose experiment controls and systems that dynamically change or steer models during training.
- 10.4.2 Human-AI Pairing: The survey links AI democratization to educational tools, direct manipulation, clear explanations, robust programming libraries, and ethical use.It frames human participation and responsible deployment as continuing requirements for beneficial AI applications.
- 10.5 Detecting Bias: Visualization can help detect data, model, and human bias, including by inspecting datasets before training and examining discriminatory decisions.The survey cites Facets for dataset inspection and interactive work examining fair and unfair threshold classifiers.
- 10.6 Protecting Against Adversarial Attacks: Deep neural network classifiers can be fooled by subtle adversarial perturbations, motivating visualization for attack detection, understanding, and protection.Interactive tools can let users vary attack type and intensity while observing resulting misclassification.
11 CONCLUSION
The survey organizes deep-learning visualization research through a human-centered interrogative framework and synthesizes its broad, rapidly growing literature. It emphasizes visual analytics for understanding models and addressing bias and AI safety, while identifying research directions and open problems.
- The survey applies a human-centered interrogative framework to review visualization and visual analytics research in deep learning.
- Its scope extends beyond visualization venues to relevant work in artificial intelligence, machine learning, and computer vision.
- Visual analytics supports insight discovery and communication for discerning model bias, understanding models, and promoting AI safety.
- The survey highlights impactful research directions and open problems in this young, rapidly growing field.