Source-linked AI summary
ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models
Minsuk Kahng, Pierre Y. Andrews, Aditya Kalro, Duen Horng Chau
TL;DR
Deep neural networks are difficult to interpret, while existing visual tools inadequately address industry-scale model diversity and data volume. ActiVis combines participatory design with coordinated architecture, activation, instance, and subset views, and its scenario demonstrates discovery of class-specific misclassification patterns and confused question intents.
Problem
Existing visualization tools inadequately address the complexity, variety, and scale of industry-deployed deep-learning models and datasets.
Method
ActiVis is an interactive system that integrates a graph-based model overview with neuron-activation views supporting instance- and subset-level exploration.
Results
ActiVis helped identify distinct activation patterns for correctly versus misclassified instances and scattered patterns among “What is” questions, motivating models with different parameters.
Takeaways & Limitations
ActiVis supports model interpretation across multiple views and abstraction levels and has been deployed on Facebook’s machine-learning platform.
Abstract
from arXiv · showhide
While deep learning models have achieved state-of-the-art accuracies for many prediction tasks, understanding these models remains a challenge. Despite the recent interest in developing visual tools to help users interpret deep learning models, the complexity and wide variety of models deployed in industry, and the large-scale datasets that they used, pose unique design challenges that are inadequately addressed by existing work. Through participatory design sessions with over 15 researchers and engineers at Facebook, we have developed, deployed, and iteratively improved ActiVis, an interactive visualization system for interpreting large-scale deep learning models and results. By tightly integrating multiple coordinated views, such as a computation graph overview of the model architecture, and a neuron activation view for pattern discovery and comparison, users can explore complex deep neural network models at both the instance- and subset-level. ActiVis has been deployed on Facebook's machine learning platform. We present case studies with Facebook researchers and engineers, and usage scenarios of how ActiVis may work with different models.
1 INTRODUCTION
ActiVis addresses the difficulty of interpreting industry-scale deep neural networks by combining coordinated views for model architecture, neuron activations, and instance or subset exploration. Participatory design and an illustrative scenario show how these views help users identify and investigate classification issues.
- Motivation: Industry-scale deep neural networks remain difficult to interpret because they encode nonlinear hidden structures with huge numbers of parameters.This can leave users unable to understand or fix unsatisfactory model behavior.
- Motivation: Existing visualization work inadequately addresses the complexity, model diversity, and large datasets found in industry settings.The paper specifically notes that most existing visualization research targets image datasets, despite industrial tasks using text and numerical data.
- ActiVis: ActiVis unifies instance- and subset-level activation inspection with a graph-based model overview and localized layer inspection.Users can define subsets from raw attributes, transformed features, and output results, then drill down from the architecture graph to layer activations.
- Illustrative scenario: In the question-classification scenario, the neuron matrix showed distinctive class patterns while the projected activations showed incomplete class separation and possible misclassification.The matrix encodes neurons as columns and instances from different classes as rows; darker gray indicates stronger activation.
- Illustrative scenario: Correctly and incorrectly classified instances in the NUMber class had very different activation patterns, while “What is” instances showed scattered activations associated with confused intents.Susan used these findings to train additional models with parameters such as longer n-grams.
- Deployment and contributions: ActiVis integrates coordinated views to support flexible exploration of complex models at both instance and subset levels, and it was deployed on Facebook’s machine learning platform.A developer can enable visualization by adding only a few lines of code, while users access it through FBLearner Flow without additional code.
2 RELATED WORK
Prior work visualized individual instances, features, subsets, activations, model graphs, and dimensionality-reduced representations. The paper positions ActiVis as addressing gaps in jointly supporting these analysis strategies for scalable deep-learning interpretation.
- Instance-based exploration: Instance-based tools help users track how individual examples behave inside models and inspect prediction scores or parameter changes.Prior systems include model-agnostic tools and tools designed specifically for neural networks.
- Feature- and subset-based exploration: Feature- and subset-based exploration groups or slices instances to reveal relationships between data and model outputs.These approaches complement individual-instance inspection when features or groups provide more interpretable structure.
- Interactive visualization tools: Neural-network visualization tools commonly display filters or activations for individual layers to help users understand learned representations.This line of work includes interactive activation visualizations for deep networks.
- Scalable visualization systems: CNNVis improves scalability through neuron clustering, edge bundling, and class-level average activations, but does not support feeding instances for instance-based analysis.Its design summarizes convolutional-network structure rather than providing the individual-instance workflow described in the paper.
- Scalable visualization systems: Dimensionality-reduction systems project high-dimensional activations or filters into two- or three-dimensional spaces, sometimes adding similar-instance search or individual-neuron views.Examples include projected activation views, Embedding Projector, and ReVACNN.
3 ANALYTICS NEEDS FOR INDUSTRY-SCALE PROBLEMS
Participatory design with over 15 Facebook engineers, researchers, and data scientists examined how machine-learning users interpret results and shaped ActiVis’s design challenges. The sessions identified instance- and subset-based analysis as common and effective, while highlighting the need to support both.
- Participatory design: The ActiVis project used participatory design sessions with over 15 Facebook engineers, researchers, and data scientists and iteratively improved the system.The sessions focused on users’ visual analytics needs across multiple teams.
- Platform context: Facebook’s FBLearner Flow provides high-level result information such as precision and accuracy, while ActiVis extends interpretation toward deep-neural-network internals.FBLearner Flow supports model training and result analysis through a unified platform.
- Analysis strategies: The sessions found that instance- and subset-based strategies are both common and effective for interpreting machine-learning models.This finding echoed prior research and became a central design requirement.
- Instance-based analysis: Instance-based analysis lets users track familiar examples whose characteristics and ground-truth labels they know.It is especially useful when an individual instance is easy to interpret, such as text-only data.
- Subset-based analysis: Subset-based analysis can be more helpful when inspecting instances individually is tedious or numerical features are difficult to interpret.Slicing groups of instances supports analysis when individual examples do not readily reveal patterns.
- Design challenges: The resulting six design challenges were grouped into data, model, and analytics categories and were described as inadequately addressed by existing visualization tools.These challenges shaped ActiVis’s design goals.
C1. Diverse input sources and formats DATA
Industry deep-learning models may combine text, numerical features, images, and user information, while Facebook-scale training data can reach billions of rows and thousands of features. This diversity and volume constrain visualization design.
- Diverse input sources and formats: Deep-learning models can use text, numerical features, attached photos, and user information jointly in a single prediction task.Each data type may contribute many attributes to the model.
- High data volume: Facebook training data often reaches billions of rows and thousands of features, making many whole-dataset visualization tools unusable.The passage attributes this constraint to the sheer size of the data.
C3. Complex model architecture MODEL
Industry-scale deep learning models are complex and diverse, while visualization tools must accommodate varied data types and flexible subset definitions. Generalizable systems are needed to support practical model development and analysis.
- Industry models are often deep and wide, containing many layers, neurons, and operations that exceed simple linear-architecture assumptions.
- Visualization tools should work across different model types because small model changes can otherwise require substantial code changes or special-case handling.
- Flexible subset specification is desirable because users may need to define subsets from numerous input formats and features.
C6. Simultaneous need for performing instance- and subset-level
ACTIVIS combines instance- and subset-level exploration with coordinated activation, projection, and model views. These views support pattern discovery, comparison, and investigation across complex models and datasets.
- ACTIVIS unifies instance- and subset-level activation inspection to support comparison across multiple examples and groups.
- The system integrates a model-architecture overview with localized activation inspection so users can move between abstraction levels.
- ACTIVIS targets diverse industry-scale data and models through a flexible, scalable system design.
- Activation matrices place instances or subsets in rows and neurons in columns, with darker cells indicating stronger activation values.
- Users can sort neurons by subset-average activation to reveal patterns, such as instances correlated with the LOC class.
- The t-SNE projected view places instances with similar activation values closer together and links subset rows to their corresponding instances.
A: Overview of Model Architecture
ACTIVIS provides a computation-graph overview to help users understand complex model structure before inspecting selected components. The graph connects architectural exploration with corresponding activation views.
- The computation graph gives users an architectural overview before they move to model components of interest.
- Deep neural networks are represented as computation graphs, whose operators include building blocks such as convolution, matrix multiplication, and concatenation.
- ACTIVIS can represent operators and tensors as graph nodes, with data flowing from input on the left to output on the right.
- Users can zoom and pan the graph, inspect node names, and click nodes to display their corresponding activations.
B: Activation for Selected Node
Selecting graph nodes opens coordinated activation and instance views for examining model behavior. ACTIVIS supports comparisons across instances, subsets, layers, and large datasets through scalable interaction and computation strategies.
- Selecting a computation-graph node opens a neuron activation panel with node context, a matrix view, and a projected view.
- Users can compare multiple selected layers, whose projected activations become more discernible and clustered from input toward output.
- The instance selection panel summarizes predictions and helps users choose examples for activation comparison.
- Clicking instance boxes adds persistent activation rows, enabling comparisons among multiple instances and between instances and groups.
- ACTIVIS was deployed on Facebook’s machine learning platform, where developers can integrate it by adding a few lines of training-process code.
- User-defined functions allow subsets for varied data types, including image subsets defined from object-recognition outputs.
- ACTIVIS addresses scalability through subset analysis, matrix and projection views, selective precomputation, sampling, and linear-time activation-matrix computation.
- Earlier designs either restricted activation comparison to one instance or became overwhelming by allocating matrix blocks for many operators.
5 INFORMED DESIGN THROUGH ITERATIONS
ACTIVIS evolved through iterative design changes to support comparison, manage complex models, and reveal high-dimensional activation structure.
- Unifying instances and subsets: The initial design showed activations for all layers but allowed viewing only one instance at a time, preventing comparison across instances.Subset activations were represented using average values.
- Separating program and data: Showing multiple instances improved information access but became visually overwhelming for large, complex models and did not generalize well across models.Engineers also preferred focusing on only a few variable nodes.
- Separating program and data: ACTIVIS separated program and data so users could inspect selected variable nodes without displaying every activation in complex models.This design change followed concerns about scalability and model generality.
- Presenting 2-D projection of instances: A projected t-SNE view was added because neuron-level statistics and distributions could mislead by failing to capture high-dimensional activation patterns.The projection provided detail beyond average activations.
6 CASE STUDIES & USAGE SCENARIOS
Case studies with Facebook practitioners show ACTIVIS supporting test-case analysis, architecture exploration, activation-based debugging, and ranking-model investigation.
- Study setup: Three Facebook engineers and data scientists used ACTIVIS with their own text-classification datasets and models in 60-minute sessions involving workflows, hands-on exploration, and feedback.The participants represented software engineering and data-science roles with different analysis goals.
- Spot-checking models with user-defined instances and subsets: User-defined instances and subsets helped practitioners compare familiar test cases with positive and negative examples to assess model behavior.Bob and Dave valued specifying their own cases, supporting the need for subset-level visualization.
- Graph overview as a crucial entry point to model exploration: The computation graph served as an entry point for examining convolution operators, model parameters, and padding before inspecting activation patterns.This helped Bob understand internal model mechanisms in complex architectures.
- Visual exploration of activation patterns for evaluating model performances and for debugging hints: Activation visualizations exposed unexpectedly similar patterns, inactive neurons, and word-specific activation behavior that suggested possible debugging or training changes.Carol used these patterns to hypothesize parameter improvements, while Bob examined convolution outputs.
- Exploring ranking models: In a ranking-model scenario, similar class activations and many near-zero neurons prompted additional models with different parameter combinations, and some models improved.Subset activation patterns then linked important features to potentially predictive neurons.
7 DISCUSSION AND FUTURE WORK
The discussion identifies extensions for gradients, faster subset handling, automated subset discovery, input-dependent models, and longitudinal study of training impact.
- Visualizing gradients: ACTIVIS could be extended to visualize gradients, representing backward flow and helping developers locate neurons or datasets where models perform poorly.This would complement activation visualization of forward data flow.
- Real-time subset definition: Real-time subset definition is currently limited because new subsets require loading the dataset into RAM to test instance conditions.Dynamic filtering and searching are planned to speed subset definition and instance selection.
- Automatic discovery of interesting subsets: ACTIVIS does not automatically discover interesting subsets, motivating heuristics based on activation-pattern similarity or difference.For text data, candidate subsets could contain phrases related to an instance or class.
- Supporting input-dependent models: Supporting input-dependent models remains future work because their variable nodes change neuron counts according to each input.The extension would study relationships between neurons and subsets in such models.
- Understanding how ACTIVIS informs model training: The impact of ACTIVIS on model training workflows remains to be studied longitudinally, including whether sparse neuron matrices inform hyperparameter tuning.The paper presents this as planned future investigation rather than an established effect.
8 CONCLUSION
The paper concludes by presenting ACTIVIS as a deployed visual analytics system shaped by participatory design and evaluated through case studies and usage scenarios.
- Conclusion: ACTIVIS unifies instance- and subset-level exploration, integrates model architecture with localized activation inspection, and scales to industry-scale data and models.These three goals were distilled from participatory design sessions with over 15 Facebook researchers and engineers.
- Conclusion: ACTIVIS was deployed on Facebook’s machine learning platform and illustrated through case studies with engineers and data scientists plus usage scenarios across applications.The conclusion frames these materials as evidence of how the system may be used with different applications.