Source-linked AI summary
Trust in AutoML: Exploring Information Needs for Establishing Trust in Automated Machine Learning Systems
Jaimie Drozdal, Justin Weisz, Dakuo Wang, Gaurav Dass, Bingsheng Yao, Changruo Zhao, Michael Muller, Lin Ju, Hui Su
TL;DR
AutoML automates data-science tasks, but trust remains a hurdle to wider adoption, and what information establishes data scientists’ trust is an open question. Across three studies, the paper finds that transparency features increase trust and understanding; participants also prioritize information differently, pointing to a need to accommodate preferences and domains.
Problem
As AutoML automates data-science tasks, trust remains a hurdle to wider adoption, and what information establishes data scientists’ trust is an open question.
Source: §1 · p. 1Method
Three studies—formative interviews, an experiment comparing transparency features, and a card-sorting task—examined AutoML trust and information needs.
Source: §1 · p. 2Results
Takeaways & Limitations
AutoML tools may need personalization to accommodate individual preferences or different domains of use.
Source: §6.1.2 · p. 9Participants were data-science students, so their trust-related information needs may not generalize to professional data scientists.
Source: §6.2 · p. 9
Abstract
from arXiv · showhide
We explore trust in a relatively new area of data science: Automated Machine Learning (AutoML). In AutoML, AI methods are used to generate and optimize machine learning models by automatically engineering features, selecting models, and optimizing hyperparameters. In this paper, we seek to understand what kinds of information influence data scientists' trust in the models produced by AutoML? We operationalize trust as a willingness to deploy a model produced using automated methods. We report results from three studies -- qualitative interviews, a controlled experiment, and a card-sorting task -- to understand the information needs of data scientists for establishing trust in AutoML systems. We find that including transparency features in an AutoML tool increased user trust and understandability in the tool; and out of all proposed features, model performance metrics and visualizations are the most important information to data scientists when establishing their trust with an AutoML tool.
1 INTRODUCTION
As data-science tasks become automated, this paper asks what information helps data scientists trust AutoML and tests transparency features and information priorities across three studies. It reports increased trust and understanding with transparency features, alongside a ranking of information needs.
1 INTRODUCTION
- AutoML automates tasks such as model selection, feature engineering, and hyperparameter tuning, amid a data-scientist shortage and a trust hurdle to broader adoption.Source: §1 · p. 1
- The authors define trust as confidence in and willingness to act on an AI decision aid, and examine how information about AutoML and its models relates to trust.They adopt an informational perspective in which having information ought to increase trust through greater transparency, while lacking it ought to decrease trust.Source: §1 · p. 2
- Three studies examined information needs: formative interviews, an experiment testing transparency features, and a card-sorting task ranking information.The interviews surveyed commercial AutoML products and identified information not commonly included in them.Source: §1 · p. 2
- The research asks whether transparency features affect trust and understanding, and which information is important or unimportant for trust in AutoML.Source: §1 · p. 2
- Transparency features increased users’ trust and understanding of AutoML, and the studies ranked information by its importance for establishing trust.The tested features included visualizations of input-data distributions and the feature-engineering process.Source: §1 · p. 2
2 RELATED WORK
The related work traces data science as a multistage, collaborative practice, the growth of AutoML, and efforts to make automated systems more transparent. It identifies a trust research gap and visualization challenges that inform this paper’s studies.
2.1 Human-in-the-loop Machine Learning
- Data science is described as multistage and collaborative, with craft-like work practices informing human-centered machine-learning tools and design guidelines.Prior work also studied notebook use and collaborative notebook features.Source: §2.1 · pp. 2–3
2.2 Automated Machine Learning (AutoML)
- AutoML automates model selection and optimization across data-science steps, while end-to-end tools and empirical user-experience studies have emerged.Prior work describes human–AI collaboration positively, while identifying trust and transparency as adoption concerns.Source: §2.2 · p. 3
2.3 Trust in Machine Learning and in AutoML
- Trust research specific to AutoML remains limited; earlier design suggestions call for showing both final model results and the pipeline steps and decisions.Source: §2.3 · p. 3
- Existing visualization systems do not provide an overview of how model pipelines were created, and combining pipeline overview with step-level detail is challenging.The challenge includes categorical search spaces, continuous hyperparameter dimensions, and nested steps.Source: §2.3 · pp. 3–4
- This paper uses low-fidelity prototypes and user evaluation to study transparency during AutoML preparation and modeling, leaving deployment-phase transparency for future work.Its three-study approach operationalizes transparency through data-oriented and process-oriented features, while prioritizing information needs to avoid cognitive overload.Source: §2.3 · p. 4
3 STUDY 1: THINK-ALOUD EVALUATION
A think-aloud pilot across four commercial AutoML tools found strong demand for transparency about data and models, alongside widespread difficulty understanding the tools’ processes.
3 STUDY 1: THINK-ALOUD EVALUATION
- Participants used four popular commercial AutoML products in random order and generated and evaluated models for the Titanic dataset.They had 30 minutes with each product and thought aloud while interacting with it.Source: §3 · p. 4
3.1 Results
- Transparency about data and models was highly desired across all four tools.The pilot findings informed a follow-up study of data-distribution and feature-engineering visualizations.Source: §3.1 · p. 4
- All four participants preferred a different tool, with some prioritizing data information and another emphasizing transparency about candidate models.Two participants noted limited support for examining raw or pre-processed data.Source: §3.1 · p. 4
- All participants expressed a lack of understanding about the processes used by the AutoML tools.Participants specifically questioned what had been done to the data and how feature engineering worked.Source: §3.1 · p. 4
4 STUDY 2: INCREASING TRUST VIA TRANSPARENCY FEATURES
Study 2 examined whether adding data-oriented and process-oriented transparency to an AutoML interface affected trust, building on feedback that these features were missing.
4 STUDY 2: INCREASING TRUST VIA TRANSPARENCY FEATURES
- The authors distinguish data-, process-, and model-oriented transparency, including data distributions, AutoML processes, and validation-set accuracy metrics.Source: §4 · p. 4
- The study compared a baseline AutoML interface with added insight into input-feature distributions and the feature-engineering process.These features reflected feedback from Study 1 participants that both were important but missing.Source: §4 · pp. 4–5
- The interface used screenshots from IBM Watson Studio AutoAI with company branding removed to allow the transparency features to be incorporated.Figure 2 shows examples of how the features were presented.Source: §4 · p. 5
4.1 Participants
The study’s final sample comprised 20 participants with prior machine-learning experience, most of whom were graduate students.
4.1 Participants
- The final sample was N = 20 after one of 21 recruited participants was excluded for insufficient machine-learning knowledge.Source: §4.1 · p. 5
- Fifteen participants were graduate students and five were undergraduates; 15 were male and five were female.Participants studied information technology, computer science, business analytics, mathematics, quantitative finance, and other disciplines.Source: §4.1 · p. 5
4.2 Procedure
Participants reviewed AutoAI-generated model information and completed a card-sorting task after receiving dataset and tool documentation.
4.2 Procedure
- The approximately one-hour study began with dataset and tool documentation, then proceeded to model review and card sorting.Participants received a $10 USD gift card for participation.Source: §4.2 · p. 5
4.2.1 Reviewing AutoML-produced models.
- Participants reviewed four AutoAI pipelines for the Census-Income dataset and judged whether they trusted any model enough for real deployment.The review included pipeline details such as performance metrics and confusion matrices.Source: §4.2.1 · p. 5
- Each participant saw the baseline V1 packet and one of two variants, V2A or V2B, which added input-data or feature-engineering visualizations.Packet order was randomized, with equal numbers seeing V1 or a V2 packet first.Source: §4.2.1 · p. 5, Table 1 · p. 5
4.3 Measures
The study measured trust and understandability with participant ratings and also asked whether participants would use an AutoAI pipeline in a real deployment.
4.3 Measures
- The study used “pipeline” and “model” interchangeably for AutoAI’s output, a binary classifier for loan approval decisions.Some, but not all, pipelines included feature engineering and hyperparameter optimization.Source: §4.3 · p. 5
- Trust was measured with four confidence and dependability items rated on 5-point Likert scales.The items asked whether participants considered the tool competent, trusted it, had confidence in its advice, and could depend on it.Source: §4.3 · p. 6
- A 9-item, 5-point Likert scale assessed understanding of the tool, its process, data, feature generation, models, metrics, and visualizations.The items included understanding data preprocessing and estimator selection.Source: §4.3 · p. 6
- Participants were asked whether they trusted any of the four pipelines enough to use one in a real deployment.The authors selected this measure because the task involved high-stakes loan approval.Source: §4.3 · p. 6
4.4 Results
Adding either transparency feature was associated with higher trust and understandability than the base interface, while the study did not establish which feature mattered more. Deployment choices also differed between the base and enhanced packets.
4.4.1 Reliablity.
- The trust scale had Cronbach’s α = 0.89, and the understandability scale had Cronbach’s α = 0.80.Both scales were treated as reliable, and averaged responses formed the trust and understandability outcome measures.Source: §4.4.1 · pp. 6–7
4.4.2 Effect of Transparency Features on Trust and Understandabil-
- In V1, trust ratings averaged 3.2 (.81) of 5, while understandability averaged 3.6 (.52) of 5.The authors characterized trust as neutral and noted room to improve understanding.Source: §4.4.2 · p. 7
- Trust in V2 packets (M (SD) = 4.1 (.62)) and understandability (M (SD) = 4.0 (.54)) were significantly higher than in V1.The differences were significant and large for trust, F[1, 19] = 19.5, p < .001, partial η2 = .36, and understandability, F[1, 19] = 21.9, p < .001, partial η2 = .49.Source: §4.4.2 · p. 7
- Trust ratings were V2A M (SD) = 3.9 (.65) and V2B M (SD) = 4.2 (.60), with no significant difference; understandability was 4.0 (.51) and 4.0 (.59), respectively.The study therefore did not resolve the relative importance of the two features.Source: §4.4.2 · p. 7
- In V1, 17 participants (85%) would not deploy any pipeline, whereas 16 participants (80%) would deploy one from V2.Deployment decisions correlated with trust (r = .67, p < .001) and understandability (r = .38, p = .02).Source: §4.4.3 · p. 7
4.4.3 Would Participants Deploy AutoML Models?
- Seven of 10 participants with V2A and 9 of 10 with V2B would deploy a pipeline; the difference was not significant.This result likewise did not distinguish the relative importance of the two transparency features.Source: §4.4.3 · p. 7
5 STUDY 3: ELICITING INFORMATION NEEDS
The card-sorting study asked data scientists to rank information needs for trust in AutoML. Performance information ranked highest, while participants also prioritized data-related information and process visibility.
5.2 Procedure
- The study used an open card sort: participants ranked 27 AutoML information cards from most to least important for trust and could add blank cards.Cards described information needs, such as viewing pre-processed data or knowing how features were engineered.Source: §5.2 · p. 7, §5 · p. 7
5.3 Results
- The researchers computed each card’s mean rank and categorized it by process, data, or model, and by whether it was visualized.The same participants had completed Study 2 earlier in the session; only four added information needs on blank cards.Source: §5.3 · p. 7, §5.1 · p. 7
5.3.1 Information Needs for Establishing Trust.
- Performance information for generated pipelines ranked most important for trust, both as a raw metric (#26) and in visual form (#27).Participants also rated pre-processing information (#10) and viewing how a pipeline was created (#23) as important.Source: §5.3.1 · pp. 7–8, Table 2 · p. 8
- Participants ranked input data distributions as more important than feature engineering information, despite no significant trust-rating difference between those features in Study 2.Source: §5.3.1 · p. 8
- Feature engineering ranked 19/27, while raw data and model evaluation metrics each ranked 11/27; the authors call for future research on this low interest.They note that raw data are transformed and engineered into features during data science work.Source: §5.3.1 · p. 8
6 DISCUSSION
The discussion reports that data-distribution visualizations and a feature-engineering graphic improved trust and understandability, while information preferences varied by user and intended use. It also identifies limits on how broadly the findings apply.
6 DISCUSSION
- Input-data distribution visualizations and a feature-engineering graphic significantly increased users’ trust and understandability of AutoML.The studies did not distinguish the two features statistically, although Study 3 participants ranked input-data distributions as more important.Source: §6 · p. 8
6.1 Implications for Design and Future Research — 6.1.1 Visualizations.
- Future design work should test visualizations for raw and processed data, including ways to inspect samples and compare data along a pipeline.The authors also suggest investigating annotation and communication needs around pipeline visualizations.Source: §6.1.1 · p. 8
6.1.2 Individual Differences and Personalization.
- Preferences varied even among 24 participants: six prioritized raw data, three preferred processed data, and two said raw data was least important.The authors therefore suggest allowing personalization to accommodate individual preferences or domains of use.Source: §6.1.2 · pp. 8–9
- Information needs depended on intended use: hyperparameter-optimization cards ranked lowest, but two participants said they mattered more for research than loan approval work.The authors caution against treating low rankings as evidence that this information is unimportant.Source: §6.1.3 · p. 9
6.1.3 Context of Use.
- Participants distinguished trust in an AutoML-produced model from trust in the AutoML tool itself; one retained tool trust despite rejecting a low-accuracy model.The authors identify this distinction as a topic for further research.Source: §6.1.3 · p. 9
6.2 Limitations
- Participants were students with prior data-science experience, which may not generalize to professional data scientists’ information needs.The authors call for validation with different user groups, particularly professionals given enterprise use of AutoML.Source: §6.2 · p. 9
- The study used a loan-application dataset, and future work could examine transparency features for presenting bias-detection and bias-mitigation information.The authors note the importance of loan decisions and the potential for AI algorithms to inherit discrimination against people of color and women.Source: §6.2 · p. 9
- The card-sorting task involved first-time AutoML users and may capture initial trust needs rather than information needs for maintaining trust.Participants also suggested rankings could change with continued tool use; the authors note that trust formation and retention may differ.Source: §6.2 · p. 9
7 CONCLUSION
The paper finds that adding information to AutoML tools significantly increased user trust and understandability. It offers a pool of transparency features for further study and recommends accommodating differences among users and domains.
7 CONCLUSION
- Increasing transparency by providing more information significantly increased user trust and understandability in the AutoML tool.The authors provide a pool of transparency features for further research and encourage exploration of ways to accommodate users’ domains, knowledge, and practices.Source: §7 · p. 9