PaperlayerView exact revision on arXiv

Source-linked AI summary

Whither AutoML? Understanding the Role of Automation in Machine Learning Workflows

Doris Xin, Eva Yiwei Wu, Doris Jung-Lin Lee, Niloufar Salehi, Aditya Parameswaran

arXiv:2101.04834v1cs.HCcs.LG

TL;DR

Auto-ML tools promise to make machine learning more accessible, but evidence about how people use them in practice is limited. The authors study experienced users’ workflows and find that human developers remain important alongside automation. They argue for a partnership between users and tools rather than complete automation.

  • Problem

    Auto-ML is growing as a way to make ML more accessible, but how its tools are used in practice and what users need remain insufficiently understood.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    The authors conducted a qualitative study of 16 people with professional experience applying Auto-ML to real-world use cases.

    Source: §1 · p. 1, §3.2 · p. 4

  • Results

    Human developers contribute domain knowledge and intuition that fills Auto-ML blind spots, and their involvement is important to effective, socially responsible ML use.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors argue that Auto-ML tools should support a productive partnership with users rather than aim for complete automation.

    Source: §5 · p. 13

    The participant population may not represent the overall Auto-ML user base.

    Source: §3.5 · p. 4

Abstract

from arXiv · show

Efforts to make machine learning more widely accessible have led to a rapid increase in Auto-ML tools that aim to automate the process of training and deploying machine learning. To understand how Auto-ML tools are used in practice today, we performed a qualitative study with participants ranging from novice hobbyists to industry researchers who use Auto-ML tools. We present insights into the benefits and deficiencies of existing tools, as well as the respective roles of the human and automation in ML workflows. Finally, we discuss design implications for the future of Auto-ML tool development. We argue that instead of full automation being the ultimate goal of Auto-ML, designers of these tools should focus on supporting a partnership between the user and the Auto-ML tool. This means that a range of Auto-ML tools will need to be developed to support varying user goals such as simplicity, reproducibility, and reliability.

1 INTRODUCTION

Auto-ML aims to make machine learning more accessible and reduce the manual effort of its iterative modeling process. A qualitative study of current users grounds recommendations in real practices and argues for preserving human agency and control rather than pursuing complete automation.

1 INTRODUCTION

  • Users must navigate iterative modeling choices across preprocessing, feature engineering, model types, hyperparameters, and empirical evaluation.The process is often perceived as a cumbersome trial-and-error “dark art.”Source: §1 · p. 1
  • The study interviewed 16 current Auto-ML users, ranging from hobbyists to industry researchers, about their use cases, work practices, needs, and perceptions.The interviews examined experience with and without Auto-ML and the roles of humans and automation in ML development.Source: §1 · p. 1
  • Human developers contribute domain knowledge, intuition that fills Auto-ML blind spots, and involvement important to effective, socially responsible ML use.The supplied passage continues this finding by arguing for a productive human–tool partnership rather than removing developers from the loop.Source: §1 · pp. 1–2
  • Recommendations draw on broad user perceptions and usage, aiming to preserve valuable tool benefits while emphasizing human agency and control.The authors argue complete automation is infeasible and describe the alternative as mixed-initiative ML.Source: §1 · p. 2

2 RELATED WORK

The related work situates Auto-ML among end-to-end tools, human-centered ML research, and human-in-the-loop systems. The paper builds on prior studies by examining experienced users’ real-world workflows and the roles humans continue to play.

2.1 Auto-ML Systems

  • Auto-ML offerings span open-source, cloud-hosted, and enterprise tools, with differing trade-offs in flexibility, transparency, compute support, and workflow coverage.Open-source tools offer flexibility but require users to provide compute; cloud offerings tend to be less configurable and transparent, while enterprise platforms emphasize feature completeness and operationalization.Source: §2.1 · pp. 2–3

2 RELATED WORK

  • The paper addresses limited evaluation of how the three types of Auto-ML tools are used in practice and seeks design considerations for human–AI collaboration.It focuses on adoption, usage, and bottlenecks users face.Source: §2.1 · p. 3

2.2 Human-Centered ML Work Practices

  • Prior human-centered Auto-ML work studies human guidance, tool perceptions, and trust, while this paper examines additional requirements and end-to-end workflows.The paper focuses on customizability, completeness, ease of use, efficiency, effectiveness, and generalizability.Source: §2.2 · p. 3
  • Earlier studies often used participants without Auto-ML experience and prototype interfaces focused on modeling, whereas this study examines experienced users in real-world applications.The authors report broader social context and hands-on human agency and control as findings beyond hypothetical-interface studies.Source: §2.2 · p. 3

2.3 Human-in-the-loop ML Tools

  • Human-in-the-loop research includes interactive feedback, model debugging, iterative development, and interpretation; this paper studies such capabilities in present-day Auto-ML tools.The paper reports that humans remain involved throughout the workflow and can iteratively influence ML outputs at multiple touch points.Source: §2.3 · p. 3

3 STUDY DESIGN

The study examines how people with real-world Auto-ML experience incorporate tools into their workflows. It uses interviews and collaborative coding, while acknowledging limits in sample representativeness and the depth of conceptual-perception findings.

3.1 Recruitment

  • Participants were recruited for prior experience applying Auto-ML to real-world use cases across varied domains.Recruitment used mailing lists and Slack, personal connections, and social media; eligibility was verified through experience with an Auto-ML tool.Source: §3.1 · pp. 3–4

3.2 Participants

  • Sixteen participants had professional Auto-ML experience and varied in organization, use case, job role, industry, and tools used.They averaged 10 years of programming experience and five years of ML experience, and came from three continents.Source: §3.2 · p. 4

3.3 Interview Procedure

  • Semi-structured interviews conducted from October 2019 to March 2020 lasted about an hour and covered roles, ML use cases, workflows, Auto-ML features, and tool perceptions.Fourteen interviews were remote and two were in person.Source: §3.3 · p. 4

3.4 Study Analysis

  • The researchers iteratively coded interview data, discussed themes, and established a shared codebook through consensus.They did not calculate inter-rater reliability, aiming to avoid potential marginalization of perspectives.Source: §3.4 · p. 4

3.5 Limitations

  • Interviews centered on one ML task, which provided concrete discussion but may have missed use-case diversity.Focusing on work practices supported deep understanding of workflow strategies but limited in-depth study of perceptions such as transparency and interpretability.Source: §3.5 · p. 4
  • The diverse participant sample may not represent the overall Auto-ML user base.The sample included 14 men and two women; the women’s share was slightly below the data-science profession’s gender ratio cited from the 2020 BCG report.Source: §3.5 · p. 4

4 RESULTS

The results section organizes interview findings around user and use-case categories, workflow tasks, tool benefits and deficiencies, and the roles of human developers and automation.

4 RESULTS

  • The section first groups users and use cases into high-level categories, then enumerates common reported ML workflow tasks.Source: §4 · p. 4
  • It presents perceived benefits of Auto-ML and deficiencies that may be addressed through system and UI improvements.The authors distinguish these deficiencies as having near-term solutions using existing techniques.Source: §4 · p. 4
  • The section identifies human developer roles that Auto-ML tools must respect and preserve, distinguishing them from deficiencies amenable to near-term fixes.The authors say automating these roles may require fundamental breakthroughs in knowledge representation and programming paradigms.Source: §4 · p. 4

4.1 User and Use Cases Segmentation

The study segments participants by ML expertise and use-case stakes to examine how context shapes Auto-ML use and perceptions. It characterizes tool categories across use cases, expertise, workflow-stage support, and participant ratings.

4.1 User and Use Cases Segmentation.

  • The segmentation provides context for interpreting participant behavior and sentiment about Auto-ML tools.The study focuses on how user expertise and use cases may shape tool use and perceptions.Source: §4.1 · p. 4

4.1.1 User Skill Levels.

  • Participants are grouped as ML Innovators with formal research experience, ML Engineers with applied ML training, or Novices without formal ML training.Innovators understand model mathematics; Engineers know popular models and tools; Novices are domain experts in non-ML fields.Source: §4.1.1 · p. 5

4.1.2 Use Case Categories.

  • Use cases span production applications with significant impact, lower-stakes industry prototypes, and academic research models.Production examples include financial decisions, recommendations, and fraud detection; prototype results do not directly affect business metrics or end-user experience.Source: §4.1.2 · p. 5
  • Figure 1 cross-tabulates tool categories with use cases and expertise, shows feature-based support for three workflow stages, and reports participant Likert scores.Proprietary-tool support ratings are omitted because the researchers lack full visibility into those tools; scores are treated as interval data based on their correspondence to quintiles.Source: §4.1.2 · p. 5, Figure 1 · p. 6

4.2 Data Pre-processing, Modeling, and Post Processing Tasks

Across the workflow, participants commonly combine automation with manual work: preprocessing is time-consuming and mostly manual, while modeling and post-processing involve automation alongside checking, adaptation, and deployment constraints.

4.2.1 Data Pre-processing.

  • Participants spend roughly 50% of their time on preprocessing, and perform roughly 80% of preprocessing tasks manually.They described preprocessing as crucial, while some wanted more automation and others considered its reliance on intuition and knowledge difficult to codify.Source: §4.2.1 · p. 6
  • Preprocessing includes data collection, labeling, wrangling, exploratory analysis, and feature engineering, with manual work persisting alongside available automation.Participants manually labeled data for intuition and turnaround; others crowdsourced labels. Most participants wrangled data, often using pandas, numpy, or SQL, while Auto-ML supported some users.Source: §4.2.1 · pp. 6–7
  • Missing-data imputation is common, but participants said context determines the appropriate technique; many also manually format data for Auto-ML ingestion.More than half of participants who performed feature engineering did so manually because solutions seemed incomplete or inefficient, or human intuition and domain knowledge seemed necessary.Source: §4.2.1 · pp. 6–7

4.2.2 Modeling.

  • During modeling, Auto-ML commonly handles hyperparameter tuning, model selection, and feature selection, with exceptions tied to scale and deployment constraints.P4 manually selected features before training because of petabyte-scale data, while P13 did so for models deployed to memory-limited edge devices.Source: §4.2.2 · p. 7
  • Participants also used Auto-ML to inform manual modeling, explore unfamiliar models, benchmark manual results, and check or correct automated outputs.P13 used it to check data quality and validate preprocessing; many participants modeled manually alongside Auto-ML to understand results and correct errors.Source: §4.2.2 · pp. 7–8

4.2.3 Post-processing.

  • Post-processing includes reporting, deployment, validation, interpretation, and less common tasks such as versioning and converting models into if-else rules.Tools often generated reports, but users modified them and prepared human-readable presentations; only hosted Auto-ML tools afforded automated deployment, while security review, unsupported features, complex logic, or supervised rollout led to manual deployment.Source: §4.2.3 · p. 8

4.3 Benefits of Auto-ML

Participants described Auto-ML as useful for speed, exploration, prototyping, and standardization, but its benefits varied with expertise, transparency, and confidence in performance. Their accounts support tools that aid human work rather than simply replacing it.

Enables and empowers novices.

  • Auto-ML enabled novices to use ML for business decisions, but curious users could face choice paralysis and longer development time.P9 reported a few percentage points of performance improvement alongside increased development time from exploring newly exposed model types.Source: §4.3 · p. 8

Standardizes the ML workflow for better reproducibility, code maintainability, knowledge sharing.

  • A predetermined search space and standardized workflow were associated with comparable models, reproducibility, maintainable code, and easier knowledge transfer.Hosted tools with extensive end-to-end reports could also serve as self-documenting knowledge-sharing platforms; lack of transparency, however, hindered one participant’s learning and contributed to another abandoning Auto-ML.Source: §4.3 · pp. 8–9

Prevents suboptimal results due to idiosyncrasies of ML Innovators.

  • Auto-ML helped some ML Innovators avoid suboptimal outcomes from their own idiosyncratic modeling choices.Participants described it as a “no assumptions approach” that could find models they would not have tried manually.Source: §4.3 · p. 9

Builds models more effectively and efficiently.

  • Participants rated improvement in efficiency at 4.1 and improvement in effectiveness at 3.88 on a five-point Likert scale.Some reported an order-of-magnitude reduction in development time, while effectiveness gains were often incremental and not compelling alone.Source: §4.3 · p. 9
  • Auto-ML let participants explore more models in the same time, with one participant comparing two or three manually reviewed models with 50 using Auto-ML.That participant reported gaining more insight into variable inputs and model behavior through the expanded experimentation.Source: §4.3 · p. 9

Enables rapid prototyping.

  • Auto-ML lowered the barrier to rapid prototyping by letting users gauge ML feasibility without setting up infrastructure, codebases, or full integration.For many industry users, however, its use remained prototyping rather than full development because they lacked confidence in its performance.Source: §4.3 · p. 9

4.4 Deficiencies of Auto-ML

Current Auto-ML tools leave substantial work to users, particularly around end-to-end workflow support and data wrangling. Participants also described limits in supported tasks and data, customization, and transparency, with trust and usability depending on human control.

End-to-end support

  • Cloud-hosted Auto-ML used in isolation leaves users to handle data ingress and egress, placing these tools below OSS in completeness despite built-in deployment and interpretability features.Participants also reported spending considerable time structuring data for cloud tools.Source: §4.4 · p. 10

Data preprocessing

  • Across tool categories, data preprocessing support often emphasizes feature engineering rather than the cleaning and wrangling tasks that consume users’ time.Participants sought automated data discovery, domain-specific cleaning and transformation, distributed processing, and dataset augmentation.Source: §4.4 · p. 10

Complex models, data types, and compute

  • Current tools mainly support classification and regression, focus on tabular data, and can fail under compute-intensive workloads or large datasets.Participants reported memory crashes, removing operators from search spaces, and limiting searches or returning to manual development to manage compute demands.Source: §4.4 · p. 10

Customizability

  • Customization presents a tension: users want more control over resources, cleaning, model search, and interpretability, but extensive options can overwhelm or prove unnecessary.Both too little and too much customizability contributed to low ratings.Source: §4.4 · pp. 10–11

Transparency and interpretability

  • Transparency is the lowest-rated quality for cloud solutions, yet visibility alone does not provide the understanding and agency participants need to trust models in high-stakes settings.Some participants switched to manual development so they could reason about and justify model design and selection decisions.Source: §4.4 · p. 11

4.5 Roles of the Human in Auto-ML

Participants described Auto-ML as a collaborative process in which humans guide searches, contribute contextual and domain knowledge, and supervise outputs. Human involvement helps address performance and efficiency limits while supporting safety, trust, and reliability.

4.5 Roles of the Human in Auto-ML

  • Humans remain indispensable because their contextual awareness, domain expertise, and ML experience support non-standard uses and stringent real-world requirements.The paper characterizes practitioners as contributors, mentors, and supervisors rather than users who can be removed from the workflow.Source: §4.5.1 · p. 11, §4.5 · p. 11

4.5.1 Humans boost Auto-ML’s performance and efficiency.

  • Humans use prior experience and heuristics to constrain Auto-ML’s search space, balancing model quality against compute and time limits.Participants limited model or hyperparameter choices; with fixed time, large untamed searches could skip search areas and weaken performance.Source: §4.5.1 · pp. 11–12
  • For complicated or non-standard tasks, participants used Auto-ML as a performance baseline or learning aid, then manually improved models or selected simpler pipelines.One participant reported that limiting a regression problem to elastic net worked better than the more complicated Auto-ML pipeline.Source: §4.5.1 · p. 12
  • Practitioners combine quick Auto-ML runs with manual feature work and repeated model iterations, using each to compensate for the other’s limits.One participant used an initial run to narrow features, then manually refined them and iterated before finalizing the training sample.Source: §4.5.1 · p. 12

4.5.2 Humans increase ML safety and prevent misuse of Auto-ML.

  • Practitioners supervise Auto-ML by validating its outputs and comparing them with manual models to check predictions, build confidence, and support trust with collaborators.Participants described using a standard approach model to show collaborators that an Auto-ML model performed better.Source: §4.5.2 · p. 12
  • Humans correct data and Auto-ML idiosyncrasies, including leakage and imbalance, while practitioners may return to manual development for mission-critical settings.Participants said they needed to reason about and justify model architecture, hyperparameters, and prediction decisions in high-stakes projects.Source: §4.5.2 · p. 13
  • Visibility into Auto-ML’s workings does not necessarily produce understanding or trust; participants cited unpredictability and lack of hands-on experience as reasons for manual involvement.One participant rated Auto-ML as extremely visible but still distrusted its model architecture and uncertainty.Source: §4.5.2 · p. 13

5 DISCUSSION

Auto-ML tools improve access to machine learning, but users do not want complete automation. The discussion argues for tools that preserve trust, understanding, agency, and productive human involvement while adapting to users and workflows.

Balance between Human Control and Automation

  • Complete automation is neither required nor desired; tools should support trust, understanding, and user agency instead of removing people from the process.The discussion also argues that full automation is infeasible and favors mixed-initiative ML solutions.Source: §5 · pp. 13–14

Lingua Franca for ML

  • Auto-ML tools should translate between business language and ML objectives because engineers interpret stakeholder problems and frame them for evaluation.The discussion notes that collaborators can have mismatched understandings of accuracy and its reporting.Source: §5 · pp. 13–14

Holistic platform

  • A holistic platform can make workflows easier, more efficient, and more interpretable through end-to-end support, reduced data transfer, and comprehensive lineage.The discussion proposes a common substrate for disparate Auto-ML tools and libraries rather than treating interoperability with external solutions as the default design.Source: §5 · p. 14

Adaptive UI

  • Interfaces should adapt to users’ skill and intent, revealing customization capabilities piecemeal rather than forcing users to choose among distinct modes.Blackbox interfaces can help novices and reduce maintenance overhead, while limited customizability and transparency can hinder trust and agency.Source: §5 · p. 14

Interactive Exploration

  • Interactive exploration should let users review prior search iterations, specify new subspaces, and work around the high latency of Auto-ML workloads.Current search-space customization is described as one-shot, leaving users to launch repeated rounds and track intermediate results manually.Source: §5 · p. 14

Balance between Human Control and Automation

  • Data preprocessing needs both near-term support for mechanical tasks and longer-term ways to express domain knowledge and integrate human intuition.Existing tools are described as lacking basic support for distributed processing and canonical transformations, while users find complex feature engineering out of reach.Source: §5 · p. 14
  • Human guidance can act as strong priors during hyperparameter search while leaving other subspaces open for machine exploration.An iterative human-machine dialog could help each side discover and fill the other’s blind spots, with the machine nudging users toward promising underexplored subspaces.Source: §5 · pp. 14–15

6 CONCLUSION

The conclusion reports a study of 16 practitioners and finds that their real-world Auto-ML workflows combine manual and automated work. It concludes that full automation is neither realistic nor desirable, and offers design guidance for effective human-AI partnership.

6 CONCLUSION

  • Sixteen practitioners’ accounts show that real-world Auto-ML workflows use hybrid manual-auto strategies, including hinting, safeguarding outputs, and preparing inputs.The study used semi-structured interviews with practitioners who had used Auto-ML in real-world applications.Source: §6 · p. 15
  • The participants’ work practices indicate that complete automation of machine learning is neither realistic nor desirable.The study describes forms of human-ML/AI partnership that vary with users’ motivations, needs, skill sets, and use cases.Source: §6 · p. 15
  • The study offers design guidelines for incorporating pragmatic guidance and empowering effective engagement with Auto-ML tools.The guidelines are aimed at next-generation Auto-ML tool developers.Source: §6 · p. 15
Loading 2101.04834v1…