Source-linked AI summary
Towards Better Analysis of Machine Learning Models: A Visual Analytics Perspective
Shixia Liu, Xiting Wang, Mengchen Liu, Jun Zhu
TL;DR
Machine learning models are difficult to understand, while interactive model analysis lacks a comprehensive synthesis despite its growing range of tasks. The paper reviews the area, classifies work into understanding, diagnosis, and refinement, and discusses future research opportunities; it also notes distortion as a limitation of one diagnostic technique.
Problem
Machine learning models are often treated as black boxes because their functions and working mechanisms are incomprehensible or unclear.
Method
The paper provides an overview and classifies interactive model analysis techniques by target tasks, including understanding, diagnosis, and refinement.
Results
The review organizes interactive model analysis around understanding model behavior, diagnosing unsuccessful training, and guiding performance and robustness improvements.
Takeaways & Limitations
Interactive visualization is presented as a way to help experts understand, diagnose, and refine machine learning models.
Takeaways & Limitations
The confusion wheel can introduce distortion, although the technique provides guidance for performance improvement.
Abstract
from arXiv · showhide
Interactive model analysis, the process of understanding, diagnosing, and refining a machine learning model with the help of interactive visualization, is very important for users to efficiently solve real-world artificial intelligence and data mining problems. Dramatic advances in big data analytics has led to a wide variety of interactive model analysis tasks. In this paper, we present a comprehensive analysis and interpretation of this rapidly developing area. Specifically, we classify the relevant work into three categories: understanding, diagnosis, and refinement. Each category is exemplified by recent influential work. Possible future research opportunities are also explored and discussed.
1. Introduction
Interactive model analysis addresses the difficulty of understanding machine learning models whose functions and mechanisms are often unclear. The paper reviews this area, organizes techniques by target tasks, and discusses future research directions.
- Machine learning models are often treated as black boxes because their functions and working mechanisms are difficult to understand.
- Without clear understanding of how and why models work, building high-performance models typically relies on time-consuming trial and error.
- Interactive visualization has been used to support understanding and analysis of diverse machine learning models.
- The paper provides an overview of interactive model analysis and summarizes recent techniques according to their target tasks.
- It also discusses research opportunities and future directions for developing interactive model analysis techniques and systems.
2. Scope and Overview
Interactive model analysis integrates machine learning with interactive visualization to help experts understand, diagnose, and refine models. The paper frames these activities around the iterative machine learning pipeline and its goal of reducing human effort.
- Scope and Overview: The machine learning pipeline obtains data, extracts usable features, trains and tests a model, then refines it using evaluation results and expert experience.
- Scope and Overview: Building a reliable model through this iterative process is time-consuming and uncertain.
- Scope and Overview: These activities aim to reduce human effort when training a reliable and accurate model.
- Scope and Overview: Interactive model analysis combines machine learning models with interactive visualization techniques that translate models into understandable explanations for experts.
- Scope and Overview: Its scope covers three goals: understanding model behavior, diagnosing unsuccessful training, and guiding improvements in performance and robustness.
3. Discussion and Analysis of Existing Work
Existing interactive model-analysis work helps experts understand model components and learned representations, diagnose performance problems, and refine models through visual analytics. The approaches range from point-based and network-based views to tools for large CNNs, prediction errors, and training-sample selection.
- Recent efforts target understanding model operation, diagnosing failed training or inadequate performance, and refining models for better performance.
- 3.1. Understanding: Point-based visualizations represent learned representations as high-dimensional vectors and place similar components near one another using dimension reduction.Components can represent neuron outputs from a hidden layer; PCA and t-SNE are examples of dimension-reduction techniques.
- 3.1. Understanding: After training, test-sample representations show greater class separation, supporting the view that neural networks learn representations useful for class discrimination.
- 3.1. Understanding: Misclassified samples often appear as visual outliers with differently labeled neighbors, and many correspond to samples difficult for humans to classify.The example given is a digit 3 misclassified because it resembles some digit 5 images.
- 3.1. Understanding: Network-based techniques display neural networks as directed acyclic graphs, encoding information through node and edge size, color, and glyphs.They expose topology and help characterize neuron roles and interactions across layers, which point-based views cannot reveal.
- 3.1. Understanding: CNNVis scales CNN analysis by clustering layers and neurons, bundling edges, and visualizing learned features and activation patterns.It targets networks with thousands of neurons and millions of connections.
- 3.2. Diagnosis: Performance-diagnosis tools use prediction-score distributions and integrated views to identify misclassified samples and support feature-selection decisions.The confusion wheel uses histograms for class-specific prediction scores, while related tools combine multiple classifiers or views in one visualization.
- 3.2. Diagnosis: These visual tools can guide debugging and refinement by revealing changes between layers, suggesting ways to prevent training from getting stuck, and supporting informed training-sample selection.In one refinement example, selected samples produced a classifier with 97.43% accuracy.
4. Research Opportunities
The paper identifies future research opportunities spanning model understanding, online training analysis, uncertainty, explainability, and mixed-initiative guidance. These opportunities aim to better reveal model rationale, detect problems during training, quantify interacting uncertainties, and combine human and system guidance.
- Research Agenda: Existing methods are treated as an initial step, with further technical challenges and research opportunities left for exploration.The paper presents MutualRanker as enabling interactive refinement of uncertain results and positions the broader area as rapidly developing.
- Model Understanding: Model understanding needs better explanations of model structure, rationale, and decisions, especially where deep learning remains less explainable.The paper notes that model structure relates to functions and iteration-level performance, while existing approaches often fail to explain decisions.
- Online Training Process: Online training analysis should provide partial results for immediate exploration and support representative snapshot selection and comparison.The paper contrasts online analysis with examining only final results or a single snapshot, and identifies snapshot selection as a challenge.
- Online Training Process: Automatic anomaly detection during training remains challenging, particularly when supervising large deep networks can take more than one week.The paper proposes timely automatic notification while retaining interactive visualization to combine human anomaly detection with machine processing of large data.
- Mixed-Initiative Guidance: Mixed-initiative guidance requires unified mechanisms for system and user guidance, including provenance explanations, unified user/system data models, and conflict resolution.The paper highlights disagreements over training-sample labels as a concrete conflict requiring further exploration.
- Uncertainty: Future visual analytics should quantify uncertainty across data processing, model building, and visualization while modeling interactions between machine- and human-originated uncertainties.The paper describes uncertainty as arising at multiple stages and potentially increasing, decreasing, splitting, or merging throughout analysis.