Source-linked AI summary
Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Dakuo Wang, Justin D. Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, Alexander Gray
TL;DR
The paper asks how AutoAI may affect data scientists’ work and addresses limited attention to the broader practice-level implications of automating data science. Using semi-structured interviews with 20 IBM data scientists, it finds mixed perceptions alongside widespread expectations that AutoAI will shape the field, and envisions human-AI collaboration rather than simple replacement.
Problem
The paper examines how data scientists perceive AutoAI and how automation may affect their work practices, addressing limited attention to these broader implications.
Method
The authors conducted semi-structured interviews with 20 data scientists at IBM working across business contexts.
Results
Interviewees reported mixed perceptions: they valued faster pipeline construction but worried that AutoAI could reduce the technical skill needed for good data science.
Takeaways & Limitations
The paper argues that AutoAI should augment heterogeneous data science teams by teaching practices, recommending analyses, offering alternatives, and explaining its reasoning.
Takeaways & Limitations
The sample came from one large multinational technology company and may not represent the broader population of data scientists or data science workers.
Abstract
from arXiv · showhide
The rapid advancement of artificial intelligence (AI) is changing our lives in many ways. One application domain is data science. New techniques in automating the creation of AI, known as AutoAI or AutoML, aim to automate the work practices of data scientists. AutoAI systems are capable of autonomously ingesting and pre-processing data, engineering new features, and creating and scoring models based on a target objectives (e.g. accuracy or run-time efficiency). Though not yet widely adopted, we are interested in understanding how AutoAI will impact the practice of data science. We conducted interviews with 20 data scientists who work at a large, multinational technology company and practice data science in various business settings. Our goal is to understand their current work practices and how these practices might change with AutoAI. Reactions were mixed: while informants expressed concerns about the trend of automating their jobs, they also strongly felt it was inevitable. Despite these concerns, they remained optimistic about their future job security due to a view that the future of data science work will be a collaboration between humans and AI systems, in which both automation and human expertise are indispensable.
1 INTRODUCTION
AutoAI is presented as an emerging response to the growing demand for data science, but its effects on data scientists’ work remain contested. Interviews suggest that practitioners expect substantial automation while envisioning a future where human expertise and AI collaborate.
- Data science combines mathematics, statistics, machine learning, databases, cloud computing, and visualization to extract generalizable knowledge from data.
- AutoAI systems aim to automate tasks within the data science workflow, potentially addressing the gap between demand for and supply of data scientists.The passage also suggests these systems could help existing teams operate more productively.
- The study asks how data scientists perceive AutoAI and whether it will be incorporated into or displace aspects of their work.The researchers interviewed 20 self-described data scientists at IBM across varied business contexts.
- Interviewees expressed mixed reactions: some viewed extensive automation as harmful to their craft, while others welcomed AutoAI as a collaborator and teacher.Supporters associated AutoAI with reducing labor-intensive work and encouraging better practices.
- Nearly all interviewees considered AutoAI the future of data science and expected it to significantly affect the field.
- The paper identifies alternating scatter-and-gather collaboration, with AutoAI supporting individual productivity during artifact work and advising teams during idea-level collaboration.Its contributions also characterize AutoAI as a potential first-class collaborator across data science teams and stakeholders.
2 WHAT IS AUTOAI?
AutoAI refers to technologies that automate multiple stages of the iterative data science workflow, from preparing data and engineering features to selecting, optimizing, combining, and validating models. The authors illustrate this workflow through a demo that exposes both pipeline operations and model rankings.
- A typical data science workflow moves from data acquisition, cleaning, and labeling through feature engineering, model building, deployment, monitoring, and feedback.
- Model-building stages include selecting a model, optimizing hyperparameters, ensembling multiple models, and validating generalizability.
- Automation research targets each workflow stage, including data cleaning, feature generation, model selection, hyperparameter optimization, and ensembling.Examples cited include Deep Feature Synthesis, One Button Machine, AutoWEKA, Auto-sklearn, and TPOT.
- The demo interface shows a progress panel, a visualization of pipeline operations and produced models, and a leaderboard ranking models by a desired metric.The underlying AutoAI compute job may take from minutes to days to run.
3 RELATED WORK
Prior HCI and CSCW research examines human intervention, collaboration, tools, and automation in data science. This paper addresses a broader question: how introducing AutoAI may change data science practice and practitioners’ roles.
- Human intervention in data science: Related work studies how data science workers use rules, measurement plans, and human judgment when deciding what data are and how they are represented.
- Human intervention in data science: Researchers have described automation that preserves individual agency through shared representations editable by both people and algorithms.
- Human intervention in data science: Interactive machine learning complements AutoAI by improving human experiences for labeling data and soliciting richer information.
- Data science teams and disciplinary diversity: Prior work emphasizes domain knowledge and interdisciplinary teams because data science projects involve complex, situation-specific expertise.
- Data science tools: Notebooks and visual analytics support data exploration, coding, narrative, and model development but do not fully automate these activities.
- Research focus: This study focuses on active data science practitioners and examines whether AutoAI is merely another tool or a broader participant in data science practice.The authors note that data science includes many stakeholders beyond people who actively work with data or write code.
4 METHODOLOGY
The study used semi-structured interviews with 20 advanced IBM data scientists performing technical data-science work across varied business contexts. Interviews examined current practices, AutoAI familiarity and attitudes, and reactions to an AutoAI demonstration, while the authors note important sampling limits.
- Participants: The researchers interviewed 20 data scientists at IBM who actively performed data-science tasks as their primary job function.The sample included 11 women and 9 men, all advanced practitioners working on concrete business applications.
- Participants: Participants held “data scientist” roles and were actively working on data-science projects involving technical work.The recruitment focused on scientists and engineers who collect and clean data, train models, and deploy models, rather than broader leadership roles.
- Scope and limitations: The sample covered varied projects and business units, but excluded citizen data scientists, novices, executives, and clients who might hold different views.Findings were also shaped by settings involving clients, domain experts, regulated industries, and messy data.
- Sample description: The informant table presents project industries and seniority or management levels, with some participants represented across multiple industries.
- Procedure: The interview script covered current work practices, prior AutoAI experience and attitudes, and discussion prompted by an AutoAI system demonstration.Topics included job roles, projects, team size, collaboration, business objectives, perceived benefits, and future data-science work.
- Analysis: Interview data were analyzed through themes that were discussed and harmonized over several iterations, with examples selected from the source interviews.
5 RESULTS
The results move from informants’ individual and collaborative work practices to their mixed responses to AutoAI and their possible future relationships with it.
- Results overview: The results cover reported data-science work practices, responses to AutoAI, and possible positive and negative future relationships with AutoAI tools.
5.1 Work Practices of Data Scientists
Data scientists’ work combines technically intensive data preparation and modeling with extensive communication, domain expertise, education, and iterative collaboration. Their workflows alternate between individual analysis and group discussion, with work divided across models or components.
- Core work: Data scientists curate and clean data, write code, build and evaluate models, and communicate results with colleagues, clients, and stakeholders.Informants identified data preprocessing—including finding data, cleaning it, and engineering features—as their biggest challenge.
- Data science as a service: Consulting projects typically involve small squads working with clients and other stakeholders to produce six- or twelve-week proof-of-concept demonstrations.About a quarter of informants worked in this consulting setting.
- Data science as a service: All consulting data scientists reported that projects also educate client-organization members while developing data-science solutions.Clients also sought help establishing their own data-science practice.
- Constraints: Limited data and project time can push data scientists toward more interpretable models instead of more accurate ones.This trade-off arises when clients lack sophistication, data, or time for complex modeling.
- Domain expertise: Domain knowledge is important for cleaning specialized data, engineering features, and interpreting and validating model results.Healthcare examples required clinical knowledge to recognize equivalent drug references and quantities.
- Collaborative practices: Collaboration occurs mainly at the level of ideas, with clients and subject-matter experts contributing meaning while coding is generally performed individually.
- Collaborative practices: The “scatter-gather” workflow alternates initial and repeated group discussions with separate individual analyses of data.Subject-matter experts are consulted at gather points, while heads-down data work and coding occur in isolation.
- Division of labor: Teams divide work across alternative models or workflow components to avoid bottlenecks.Examples include separating feature engineering from model selection and assigning different models to different people.
5.2 Perceptions of AutoAI
Informants saw AutoAI as both an inevitable efficiency aid and a threat to technical depth, interpretability, and job security. They generally favored using it as a collaborative starting point that preserves human analysis, expertise, and feedback.
- General perceptions: 11 of 20 informants had heard of AutoAI, while 5 of 20 had tried it once or twice; reactions to its utility and impact were mixed.Informants viewed AutoAI as future-oriented and potentially time-saving, but also worried it could diminish technical depth.
- Perceived benefits: AutoAI’s most valuable potential is reducing the time from data to insights by providing an initial understanding of data and analytic alternatives.
- Perceived benefits: AutoAI can produce “good enough” models, provide a baseline for iteration, and demonstrate practices that help data scientists improve their own code.
- Perceived dangers: Informants often preferred hands-on work because data-science projects include teaching clients and exposing the reasoning behind models.
- Perceived dangers: Informants worried that widespread automation could weaken coding, mathematical understanding, data interpretation, and the technical depth of the profession.
- Mismatch with practice: AutoAI’s accuracy-focused optimization may not match goals of understanding feature relationships or reproducing models established in domain literature.
- Prior experience: One user reported that AutoAI improved the loss score over an original model and informed its recoding, while also fearing that continued use could threaten their job.
- Trust: Trust may fail when AutoAI gives unexplained errors, hides its reasoning, or prevents data scientists from analyzing and giving feedback on feature or model selection.
5.3 AutoAI Collaborate with Data Scientists
Informants envisioned AutoAI as a collaborator that automates tedious technical work while leaving human judgment, expertise, and interpretation essential. It could support individuals, teams, and non-data scientists as a baseline, advisor, and teaching tool, but raised concerns about flexibility, model quality, and skill loss.
- AutoAI as collaborator: AutoAI can automate high-effort portions of data science, including data cleaning, feature engineering, and model building, while humans continue higher-level work.Informants described dividing labor by workflow phase, such as AutoAI handling pre-processing while humans perform modeling.
- AutoAI as collaborator: A baseline from AutoAI can help data scientists compare candidate models, narrow their focus, and spend more time reporting results.Informants described AutoAI selecting promising models and settings or indicating which model types might perform better.
- AutoAI as collaborator: During team “gather” phases, AutoAI could recommend analyses, data labels, features, and models based on team needs, expertise, and unexplored gaps.It could monitor team efforts, steer work toward analyses others had not tried, and challenge humans to improve baseline models.
- AutoAI as collaborator: AutoAI could lower barriers for citizen data scientists and domain experts, but users might lose flexibility or build questionable models without sufficient knowledge.Informants linked these risks to limited understanding of data science, accuracy metrics, overfitting, and model interpretation.
- AutoAI as collaborator: Human subject-matter expertise remains necessary for data cleaning and feature engineering when understanding the data requires contextual knowledge and business logic.Informants emphasized that humans must remain involved and may add predictive features through intuition and domain-specific reasoning.
- AutoAI as collaborator: Explainability and generated code could make AutoAI a teacher by showing its reasoning, alternatives, and best practices to data scientists and other users.Such explanations could support trust, learning, and inspection of modeling choices.
6 DISCUSSION
The discussion contrasts AutoAI as an augmenting collaborator with AutoAI as a replacement for data scientists. The authors favor augmentation, while noting concerns about skill atrophy, limited sample representation, and the novelty of the technology.
- The Augmented vs. Automated Data Scientist: AutoAI may either augment data scientists’ productivity or act as a data scientist in its own right.The discussion uses these two roles to organize competing views of automation.
- The Augmented vs. Automated Data Scientist: Human input remains necessary because AutoAI cannot infer subject-matter expertise needed to understand the meaning of data.This supports shifting data scientists toward eliciting domain knowledge from subject-matter experts.
- The Augmented vs. Automated Data Scientist: Automating parts of the pipeline may cause data scientists to lose touch with their craft or miss valuable insights that active data exploration would reveal.The concern is that reduced hands-on use could contribute to skill atrophy and less exploration of the data.
- The Augmented vs. Automated Data Scientist: Explainability should be a core AutoAI design goal, with operations available for inspection, inquiry, and modification.AutoAI should explain model selection, alternatives considered, and reasons for excluding other models.
- The Augmented vs. Automated Data Scientist: Informants were skeptical that AutoAI could completely replicate data scientists, despite the appeal of automation for managers seeking cost savings.Their confidence in collaboration and continued human expertise supports developing AutoAI to augment rather than replace people.
- Limitations and Future Directions: The study’s informants came from one large multinational technology company, so their views may not represent practicing data scientists more broadly.The sample also omitted other data workers and stakeholders, whose perceptions remain open questions.
- Limitations and Future Directions: Because 75% of informants had never used AutoAI, many findings concern anticipated rather than experienced effects, requiring follow-up as adoption increases.The authors identify trust and individual- and group-level effects as areas for additional research.
7 CONCLUSION
Interviews with 20 data scientists revealed mixed perceptions of AutoAI: it could speed pipeline construction but might lower the technical-skill bar. The authors therefore advocate AutoAI that augments heterogeneous data-science teams through collaboration, recommendations, and explanations.
- 20 data scientists reported mixed perceptions of AutoAI, valuing faster pipeline construction while worrying it could lower the technical-skill bar.
- The authors argue that AutoAI should augment rather than automate the work practices of heterogeneous data-science teams.
- AutoAI could teach aspiring data scientists best practices and collaborate by recommending analyses, offering alternatives, and explaining its reasoning.