Source-linked AI summary
Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models
Jiawei Zhang, Yang Wang, Piero Molino, Lezhi Li, David S. Ebert
TL;DR
Complex machine learning models are difficult to interpret and debug, while existing approaches often focus on particular model types or tasks. Manifold addresses this gap with a model-agnostic visual framework that compares model outputs, investigates symptom instances, and examines features through an iterative inspection–explanation–refinement workflow. Applications cover multiclass classification and regression, with expert feedback describing feature-level insights as useful for model iteration; the framework also has a learning curve for novice users.
Problem
Complex models are difficult to explain and debug, and existing approaches often do not extend beyond specific model types or tasks.
Method
Manifold uses only model inputs and outputs to support iterative inspection, explanation, and refinement through scatterplot and tabular visualizations.
Results
Manifold supports model comparison and diagnosis for multiclass classification and regression, including instance-level symptom analysis and feature-level explanations.
Takeaways & Limitations
Feature-level insights can guide feature engineering and model iteration while connecting model-level comparison with targeted diagnosis.
Takeaways & Limitations
The visual encodings may have a relatively long learning curve for novice users, requiring training and familiarization.
Abstract
from arXiv · showhide
Interpretation and diagnosis of machine learning models have gained renewed interest in recent years with breakthroughs in new approaches. We present Manifold, a framework that utilizes visual analysis techniques to support interpretation, debugging, and comparison of machine learning models in a more transparent and interactive manner. Conventional techniques usually focus on visualizing the internal logic of a specific model type (i.e., deep neural networks), lacking the ability to extend to a more complex scenario where different model types are integrated. To this end, Manifold is designed as a generic framework that does not rely on or access the internal logic of the model and solely observes the input (i.e., instances or features) and the output (i.e., the predicted result and probability distribution). We describe the workflow of Manifold as an iterative process consisting of three major phases that are commonly involved in the model development and diagnosis process: inspection (hypothesis), explanation (reasoning), and refinement (verification). The visual components supporting these tasks include a scatterplot-based visual summary that overviews the models' outcome and a customizable tabular view that reveals feature discrimination. We demonstrate current applications of the framework on the classification and regression tasks and discuss other potential machine learning use scenarios where Manifold can be applied.
1 INTRODUCTION
Manifold addresses the difficulty of interpreting and debugging increasingly complex machine learning models by providing a model-agnostic, interactive framework for comparing multiple models. Its workflow combines inspection, explanation, and refinement through visual summaries and feature-level analysis.
- Increasing model complexity makes it difficult for developers to explain model behavior and obtain evidence for development and debugging.
- Existing visual and interactive approaches commonly target a specific model type or task, limiting extension to complex industry-level scenarios.
- Manifold organizes model diagnosis and comparison into inspection, explanation, and refinement phases.
- A scatterplot-based summary compares model-pair diversity and complementarity while supporting inspection of symptom instances across classification and regression tasks.
- A tabular feature view supports feature discrimination and generates explanations that can inform subsequent model-development iterations.
- Manifold observes model inputs and outputs rather than internal logic, supporting diverse model types with consistent task-specific input and output formats.
2 RELATED WORK
Related work spans model interpretation, performance analysis, debugging, and human-in-the-loop refinement. Manifold differs by comparing models at the same instance level and supporting both classification and regression without requiring internal model access.
- Many deep-learning interpretation systems require access to internal calculations or structures, whereas Manifold targets broader model diagnosis without internal logic.
- Coarse performance statistics can bias cognition, while existing fine-grained approaches may present multiple models separately and omit instance-level behavior differences.
- Manifold drills down to symptom instances where models agree or disagree and applies its comparison technique to classification and regression.
- Prior diagnosis and debugging systems investigate issues through training instances, model-specific prediction behavior, or feature contribution analysis.
- Human-in-the-loop approaches incorporate user knowledge by interactively refining hyperparameters, features, training instances, or multiple models.
3 DOMAIN CHARACTERIZATION
Manifold characterizes model diagnosis as an iterative visual-analysis process covering inspection, explanation, and refinement. The framework focuses on finding suspicious instances, comparing features, and verifying changes through model iteration.
- The framework was designed collaboratively by machine-learning and visualization researchers through iterative conversations with domain experts.
- Model-development challenges include coding errors, understanding model strengths and weaknesses, comparing models for ensembling, and incorporating insights into iterations.
- Exploratory analysis helps users examine large-scale input data and uncover relationships between data and model output.
- The three analysis phases may overlap in practice but are separated for characterization and task-design purposes.
- Inspection (Hypothesis): Inspection compares coarse performance metrics and narrows analysis to an instance subset, including a symptom set of erroneous or suspicious results.
- Inspection (Hypothesis): Users can filter instances by shared features or metadata to obtain subsets with simpler, more interpretable feature distributions.
- Explanation (Reasoning): Explanation compares selected instances with relevant ground-truth groups to identify influential features, while acknowledging that explanations may not be causal.
- Refinement (Verification): Refinement verifies explanations by encoding extracted knowledge into the model and testing performance, using feature engineering or architectural adjustment depending on model type.
4 THE MANIFOLD FRAMEWORK
Manifold coordinates model-pair predictions in a small-multiple visual overview and links selected instances to feature-distribution comparisons. Its encodings support diagnosis across classification and regression without requiring model internals.
- Model Comparison Overview: Manifold combines a model comparison overview for finding symptom instances with a feature interpretation view for comparing distributions and generating explanations.The two dialogs are coordinated as an iterative diagnostic and comparative analysis workflow.
- Model Comparison Overview: The model comparison overview uses small multiples to organize pairwise model comparisons across task-specific dimensions.Rows encode model pairs, while columns encode an orthogonal dimension chosen for the use scenario or user preference.
- Multi-Class Classification: For classification, Cartesian coordinates encode each model’s prediction, correctness, agreement or disagreement, and prediction confidence.Points near the origin indicate lower prediction confidence; quadrants distinguish agreement from disagreement and TP, TN, FP, and FN cases.
- Multi-Class Classification: Quadrant and lasso selection support instance analysis across linked matrix cells, with selected-quadrant backgrounds indicating the red-blue composition.The available filters include ALL, UNION, and GT instance subsets.
- Multi-Class Classification: Contour visualization reduces overplotting and can reveal dense point clusters that are difficult to read in a scatterplot.A cluster of red points in Q1 is clearly revealed in the contour view but not the scatterplot view.
- Regression: For regression, residuals ε = ˆy - y are mapped to Cartesian coordinates, where sign indicates over- or under-prediction and distance from the origin indicates error magnitude.This differs from classification, where proximity to the origin represents lower prediction score rather than lower error.
- Feature Interpretation View: The feature interpretation table compares feature distributions across classes, while bar differences identify the most discriminative features between selected subsets.Column charts encode KL-divergence between distributions, and users can sort features or display the top K.
5 CASE STUDY
Two case studies illustrate Manifold in different usage scenarios using public datasets, supplemented by feedback from researchers and engineers who used the system in company-specific cases.
- Case Studies: The case studies use publicly available datasets instead of company-specific datasets because of data sensitivity.Researchers and engineers involved in the project were interviewed after using the system for company-specific use cases.
5.1 Multi-Class Classification
Manifold supports multi-class model comparison by exposing confidence, complementarity, erroneous instances, and feature-level explanations. The workflow links visual inspection to hypotheses about model combination and feature refinement.
- Experimental setup: The spooky author-identification case used 12 classification models to predict whether excerpts came from EAP, HPL, or MWS.GloVe was used with the input data.
- Model comparison: M3 and M10 were more confident on correctly predicted instances than M11, while M11 was more confident than M1 and M2.The comparison used density distributions of correctly predicted instances.
- Model comparison: M3 showed higher complementarity with M10 and M11, whereas it outperformed M0, M1, and M2 on most instances.The M3–M11 comparison suggested that combining the two models could improve overall confidence.
- Error diagnosis: Brushing and linking exposed false-positive and false-negative instances across classes while preserving fine-grained prediction-confidence information.Selected erroneous instances were linked across adjacent class cells for comparison.
- Feature explanation: Feature analysis found that old was frequent in C1 but potentially nondiscriminative across all classes, motivating removal or down-weighting during learning.The feature view compared distributions and term frequencies within the selected symptom set.
5.2 Regression
Manifold supports regression diagnosis by comparing residual patterns between candidate models and identifying features that distinguish systematic error subsets. The resulting feature encoders can feed a stacking model for iterative improvement.
- Experimental setup: The bike-sharing regression task used 11,000 training points and 6,500 test points, with RMSLE as the evaluation metric.The target was total bikes rented.
- Model comparison: M2 and M3 performed significantly better than the other three models and were selected for comparison.Residual error was encoded as ε = predict − actual.
- Residual analysis: M2 and M3 had narrower residual distributions near the origin, while the third quadrant contained more widely spread instances, indicating a tendency to under-predict.The quadrants represented combinations of over- and under-prediction by the two models.
- Feature analysis: Feature divergence showed that month, weather, and season were not major differentiators between jointly over-predicted and under-predicted subsets.Most categorical and binary features received low divergence scores.
- Model refinement: Feature encoders based on high-divergence features enabled a stacking model to learn to reduce residual errors.Gradient Boosting demonstrated this human-in-the-loop model-improvement workflow.
5.3 Domain Expert Feedback
Practitioners valued Manifold’s comparison-centric workflow for moving from model-level outcomes to suspicious instances and feature-level guidance. Feedback also identified an onboarding cost for its coordinate encoding.
- Participants: The assessment involved two AI researchers, three data scientists, and five machine-learning engineers who routinely iterated and debugged models.Participants represented text classification, regression, risk analysis, forecasting, and platform infrastructure.
- Observed workflow: Experts found multi-model debugging tedious because it required complex logical slicing and coarse summaries lacked fine-grained filtering guidance.They often analyzed models independently before combining results.
- Usability feedback: Two data scientists initially found the Cartesian coordinate encoding visually confusing without training.They specifically struggled to interpret probability scores in the negative half of the coordinate system.
- Limitations: The authors acknowledge that the new visual encodings may impose a long learning curve and require training for novice users.Experienced users later found the design effective for interactively selecting instances near the origin.
- User feedback: Users positively evaluated the workflow’s progression from model-level comparison through suspicious subsets to actionable feature-level information.They viewed feature-level insights as direct guidance for feature engineering and model iteration.
6 DISCUSSION
The discussion positions Manifold as a model-agnostic visual analytics framework, while identifying design choices, workflow observations, and limitations of more automatic alternatives.
- 6.1 Visual Analytics Rather Than Automatic Approaches: Manifold uses visual analytics rather than automatic clustering because clustering results can be difficult to interpret and act on during ensembling.The authors also report preliminary evidence that granular-cluster feature encoders may overfit training data.
- 6.2 Contributions to the State of the Art: Manifold treats models as black boxes, enabling comparative analysis without requiring knowledge of their internal mechanisms.Comparing correlated model results can also provide insights into a complex model through familiar reference models.
- 6.3 Design Iterations and Alternatives: User feedback motivated small multiples because holistic multi-model summaries risk severe information overload.The design process incorporated interviews with domain partners.
- 6.3 Design Iterations and Alternatives: Manifold focuses on pairwise model comparison because three-model encodings became confusing, complicated, and difficult to scale.The authors considered ternary, normalized-confidence, and three-dimensional alternatives before choosing model pairs.
- 6.4 Future Directions and Ongoing Efforts: The scatterplot encodes model complementarity by separating agreement from disagreement across quadrants, supporting ensemble analysis.A pixel-oriented extension can encode pairwise complementarity scores at larger scales.
- 6.4 Future Directions and Ongoing Efforts: In enterprise use, familiarized users sliced and compared subsets where models produced consistent or converse results.This observation comes from deployment within a core machine learning workflow and feedback from domain scientists.
7 CONCLUSION
The conclusion presents Manifold as a generic environment for comparing and debugging machine learning models. It emphasizes visual exploration as a way to help scientists handle complex models transparently and interpretably.
- 7 CONCLUSION: Manifold partitions instances by model correctness and confidence, identifies erroneous symptom instances, compares feature-level explanations, and supports iterative refinement.The framework is intended for a broad range of machine learning models.
- 7 CONCLUSION: The authors envision Manifold as a generic platform for transparent and interpretable manipulation of complex models by machine learning scientists.The system emerged from collaboration between visualization researchers and industry machine learning scientists.