Source-linked AI summary
StackGenVis: Alignment of Data, Algorithms, and Models for Stacking Ensemble Learning Using Performance Metrics
Angelos Chatzimparmpas, Rafael M. Martins, Kostiantyn Kucher, Andreas Kerren
TL;DR
Selecting a stacking configuration is difficult because many data, feature, algorithm, and model choices perform differently across metrics. StackGenVis combines a visual-analytics knowledge-generation model with tools for exploring those choices and building stacks. Two real-world use cases and interviews with three ML experts provide preliminary evidence of its applicability and effectiveness, while experts identify efficiency and scalability concerns.
Problem
Selecting a stacking configuration from many data, feature, algorithm, and model choices is a difficult trial-and-error process, and users also need to monitor training and choose suitable metrics.
Source: §1 · p. 2Method
StackGenVis adapts a visual-analytics knowledge-generation model to explore data, algorithms, and models, manage metrics and instances, and compare stack performance.
Source: §1 · p. 2, §4.3 · p. 6Results
Two real-world use cases address healthcare and sentiment/stance text, and interviews with three ML experts indicate encouraging results and applicability.
Source: §1 · p. 2Takeaways & Limitations
StackGenVis supports users in exploring and tracking stack development while combining performant and diverse models.
Source: §7 · p. 9Experts identified efficiency and scalability as major concerns; stacking's computational burden remains, and inserting visual analytics between levels adds to it.
Source: §6 · p. 9
Abstract
from arXiv · showhide
In machine learning (ML), ensemble methods such as bagging, boosting, and stacking are widely-established approaches that regularly achieve top-notch predictive performance. Stacking (also called "stacked generalization") is an ensemble method that combines heterogeneous base models, arranged in at least one layer, and then employs another metamodel to summarize the predictions of those models. Although it may be a highly-effective approach for increasing the predictive performance of ML, generating a stack of models from scratch can be a cumbersome trial-and-error process. This challenge stems from the enormous space of available solutions, with different sets of data instances and features that could be used for training, several algorithms to choose from, and instantiations of these algorithms using diverse parameters (i.e., models) that perform differently according to various metrics. In this work, we present a knowledge generation model, which supports ensemble learning with the use of visualization, and a visual analytics system for stacked generalization. Our system, StackGenVis, assists users in dynamically adapting performance metrics, managing data instances, selecting the most important features for a given data set, choosing a set of top-performant and diverse algorithms, and measuring the predictive performance. In consequence, our proposed tool helps users to decide between distinct models and to reduce the complexity of the resulting stack by removing overpromising and underperforming models. The applicability and effectiveness of StackGenVis are demonstrated with two use cases: a real-world healthcare data set and a collection of data related to sentiment/stance detection in texts. Finally, the tool has been evaluated through interviews with three ML experts.
1 INTRODUCTION
Stacking can combine heterogeneous models for predictive power, but selecting a configuration is difficult to understand and often involves trial and error. StackGenVis adapts a visual-analytics knowledge-generation model to support metric selection, data and model exploration, and stack comparison.
1 INTRODUCTION
- Stacking combines heterogeneous base models through a metamodel and can reduce bias and generalization error relative to single learning algorithms.It can combine bagging, boosting, and simpler algorithms in different layers.Source: §1 · pp. 1–2
- Choosing algorithms and models for a stack is difficult to explain and can require trial and error, motivating questions about efficient construction and process monitoring.The paper also asks how to choose and combine validation metrics for a given dataset.Source: §1 · p. 2
- StackGenVis supports exploratory construction of stacks by combining 11 algorithms and 3,106 models with 8 performance metrics, alongside metric selection, data wrangling, and model exploration.Its workflow also traces the resulting stack and compares its performance with the active metamodel.Source: §1 · p. 2
- The paper contributes a knowledge-generation model adapted for ensemble learning and StackGenVis, which visualizes models and predictions as high-dimensional vectors to explore performant, diverse models.The paper also describes its methodology and expert-interview outcomes.Source: §1 · p. 2
- Two real-world use cases are presented as confirming the effectiveness of multiple validation metrics and stacking-ensemble comparison, while expert interviews indicate encouraging results.The use cases cover healthcare and sentiment/stance detection in text.Source: §1 · p. 2
2 RELATED WORK
The related-work review identifies limited visual-analytics support for constructing and improving stacking ensembles. It contrasts StackGenVis with systems for bagging, boosting, regression, and model selection, highlighting its broader exploration of algorithms, metrics, instances, and models.
2 RELATED WORK
- The authors report finding no prior InfoVis or visual-analytics literature on constructing and improving stacking ensembles.The review compares prior tools for bagging, boosting, and buckets of models to establish the approach's novelty.Source: §2 · p. 2
2.1 Bagging and Boosting
- EnsembleLens targets anomaly detection, while BEAMES focuses on regression; StackGenVis instead explores multiple algorithms and metrics for broader ensemble construction.BEAMES offers four algorithms and three regression metrics, whereas StackGenVis adds instance manipulation and three feature-selection techniques.Source: §2.1 · p. 2
- iForest explains a specific algorithm and BoostVis diagnoses underperforming trees, while StackGenVis uses a model-agnostic approach and multiple metrics to select models.StackGenVis also ranks feature importance across many models.Source: §2.1 · pp. 2–3
- Compared with Schneider et al., StackGenVis aggregates alternative metrics over projections and supports user-defined metric weights, but does not directly manipulate ensembles.It focuses on exploring a large solution space before narrowing to well-performing and diverse models.Source: §2.1 · p. 3
2.2 Buckets of Models
- For model-bucket and related systems, StackGenVis aggregates overall and per-class performance across models rather than relying on pairwise comparison or regression prediction errors.The user can compare a set of models against the average of all models before choosing which to use.Source: §2.2 · p. 3
3 DESIGN GOALS AND ANALYTICAL TASKS
The design goals extend a visual-analytics knowledge-generation model to ensemble learning, linking computer-side data, visualizations, and models with human exploration, verification, and knowledge generation. Five analytical tasks translate these goals into support for stacking workflows.
3 DESIGN GOALS AND ANALYTICAL TASKS
- The adapted knowledge-generation model connects data, visualizations, and models with human loops of exploration, verification, and knowledge generation.The authors derive five design goals from prior work, experience, and literature reviews.Source: §3.1 · p. 3, §3 · p. 3
3.1 Design Goals: Visual Analytics to Support Ensemble Learning
- G1 supports human control of data, preprocessing, algorithms, and model predictions, enabling exploration and selection of effective, diverse models.The model and prediction space can be explored to improve stack performance; collaboration may also help experts share knowledge.Source: §3.1 · pp. 3–4
- G2–G5 call for exploration history and provenance, verification with annotations, coordinated views and guidance, and visual choices that address cognitive bias.The authors note visualization may introduce additional biases into an already complex setting involving different ML models.Source: §3.1 · p. 4
3.2 Analytical Tasks for Stacking
- Five tasks operationalize the goals: search algorithms, data, and models; preserve exploration history; manage metrics; compare stack stages; and offer alternative views.The tasks include metric selection suited to a dataset and feedback comparing the active stack with stored versions.Source: §3.2 · p. 4
4 STACKGENVIS: SYSTEM OVERVIEW AND APPLICATION
StackGenVis guides users through metric selection, algorithm and model exploration, data wrangling, and comparison of stored stacks. In the heart-disease example, it reached approximately 88% accuracy, exceeding the reported results of two prior ensembles.
4 STACKGENVIS: SYSTEM OVERVIEW AND APPLICATION
- StackGenVis organizes stack building into metric selection, algorithm exploration, data wrangling, model exploration, history tracking, and stack-performance comparison.The workflow uses selected metrics for validation throughout the building process.Source: §4 · p. 4
4.1 Data Sets and Performance Metrics
- The system offers eight metrics across threshold, ranking, and probability groups, allowing users to choose metrics and contribution weights for a data set.For the heart-disease example, the authors selected accuracy, precision, recall, f2-score, MCC, and log loss, with micro-averaging for applicable metrics.Source: §4.1 · pp. 5–6
4.2 Exploration of Algorithms
- Algorithm exploration uses aggregated performance and per-class views to compare algorithms, then lets users filter and tune model parameters before selecting algorithms for an initial stack.In the example, six KNN models selected through parameter filtering performed better than the average; six algorithms were selected for the initial stack.Source: §4.2 · p. 6
4.3 Data Wrangling
- Data wrangling supports instance manipulation and feature selection; removing a problematic diseased-class instance increased metamodel performance, while feature analysis led to disabling three less-important features.Permutation- and accuracy-based feature-importance searches are resource-intensive and can be disabled for larger data sets; some models performed slightly better with Chol, so it was retained.Source: §4.3 · p. 6
4.4 Exploration of Models
- Reducing model count too aggressively hurt performance when diversity was lacking, but selecting underperforming models for removal produced a stack with the same results as S3 and 30 fewer models.The final selection contained 174 models rather than 204; the prediction-space view also showed that all models performed worse on the selected difficult instances than on all instances.Source: §4.4 · pp. 6–7
4.5 Results of the Metamodel
- ≈88% accuracy with StackGenVis exceeded the prior majority-vote result of 85.48% and stacking result of ≈83% on the same data set.The authors present this as evidence that users can combine base models into a performant, diverse, and simpler stack.Source: §4.5 · p. 7
5 USE CASE
StackGenVis is applied to hypotheticality stance detection by iteratively selecting metrics and diverse models, then refining the stack against test data. The final stack outperforms the prior baseline on the reported test results.
5 USE CASE
- The workflow links algorithm exploration to metric selection and per-class comparison, then updates model and prediction spaces as users refine the stack.Source: Figure 7 · p. 8, Figure 8 · p. 8
- For the imbalanced 2,095-instance task, the authors emphasized g-mean over accuracy, ROC AUC over precision and recall, and macro-average for the smaller class.They found different algorithms performed better in different classes, so selecting diverse models improved per-class performance over choosing only top performers.Source: §5 · pp. 7–8
- The authors removed lower-performing, low-diversity models while preserving balance, reducing the stack from 200 to 170 and then 140 models.The final stack performed better than the earlier stacks and reached 91%–92% across metamodel validation metrics with 5-fold cross-validation.Source: §5 · p. 8
- Approximately 82% f1-score for prediction, compared with 54% reported for Skeppstedt et al.’s baseline approach.Source: §5 · p. 9
- 93.87% f1-score for the final stack’s weighted-average hypotheticality results, versus 66% for the baseline approach.Table 1 summarizes the stance-classification test results.Source: §5 · p. 8, Table 1 · p. 9
6 EVALUATION AND FUTURE WORK
Three ML experts were interviewed about StackGenVis after a walkthrough and hands-on interaction. They endorsed its iterative workflow and visualization, while identifying computational scalability and further evaluation as open concerns.
6 EVALUATION AND FUTURE WORK
- Semistructured interviews with three experts combined a system walkthrough, interaction with the iris data set, think-aloud feedback, and roughly one-hour sessions.Source: §6 · p. 9
- Experts viewed data wrangling and algorithm exploration as iterative: users may begin with either, and model accuracy can guide which instances to wrangle.Early metrics can also help benchmark strategies during collaboration between data scientists and domain experts.Source: §6 · p. 9
- Experts praised visualization for supporting direct interaction with algorithms and models, while noting that StackGenVis may feel overwhelming initially.They also connected visualization to comparing per-class performance; adapting metrics could extend the system to regression or ranking.Source: §6 · p. 9
- Computational burden remains a concern: stacking requires resources, and placing visual analytics between levels increases that burden.The authors identify progressive workflows for larger data sets, alternative metamodels, multi-layer stacks, and a quantitative user study as future work.Source: §6 · p. 9
7 CONCLUSION
StackGenVis aligns data, algorithms, and models through an interactive visual analytics system for constructing stacking ensembles. Real-world use cases and expert interviews provide a preliminary assessment of the system.
7 CONCLUSION
- StackGenVis aligns data, algorithms, and models in an interactive visual analytics system for building stacking ensembles from scratch.Source: §7 · p. 9
- Coordinated views, multiple perspectives, and training-history tracking help users develop complex stacks combining high-performing and diverse individual models.Source: §7 · p. 9
- Real-world data use cases demonstrated performance improvements and the process behind them, while expert interviews elicited workflow, visualization, and limitation feedback.Source: §7 · p. 9
- The interviews also identified system limitations for future development.Source: §7 · p. 9