Source-linked AI summary
Who Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help?
Sasu Mäkinen, Henrik Skogström, Eero Laaksonen, Tommi Mikkonen
TL;DR
As ML moves toward frequent production delivery, this paper examines how much MLOps is needed in data scientists’ daily work. A survey of 331 professionals across 63 countries finds that most respondents still focus on data and models, while a smaller, more mature group is advancing toward infrastructure and deployment concerns.
Problem
The paper examines whether data scientists’ daily work still centers mainly on data and modeling or increasingly requires deployment and infrastructure practices for ML operations.
Method
The authors survey ML professionals about their backgrounds, activities, data, problems, and short-term goals, collecting 331 responses from 63 countries.
Results
Up to 40% of respondents work with both models and infrastructure, while most work involves relational and time series data and the leading short-term goals include production development and deployment.
Takeaways & Limitations
Most organizations remain focused on data and models, whereas more mature organizations with larger teams and stronger infrastructure orientations are advancing toward MLOps.
Abstract
from arXiv · showhide
Following continuous software engineering practices, there has been an increasing interest in rapid deployment of machine learning (ML) features, called MLOps. In this paper, we study the importance of MLOps in the context of data scientists' daily activities, based on a survey where we collected responses from 331 professionals from 63 different countries in ML domain, indicating on what they were working on in the last three months. Based on the results, up to 40% respondents say that they work with both models and infrastructure; the majority of the work revolves around relational and time series data; and the largest categories of problems to be solved are predictive analysis, time series data, and computer vision. The biggest perceived problems revolve around data, although there is some awareness of problems related to deploying models to production and related procedures. To hypothesise, we believe that organisations represented in the survey can be divided to three categories -- (i) figuring out how to best use data; (ii) focusing on building the first models and getting them to production; and (iii) managing several models, their versions and training datasets, as well as retraining and frequent deployment of retrained models. In the results, the majority of respondents are in category (i) or (ii), focusing on data and models; however the benefits of MLOps only emerge in category (iii) when there is a need for frequent retraining and redeployment. Hence, setting up an MLOps pipeline is a natural step to take, when an organization takes the step from ML as a proof-of-concept to ML as a part of nominal activities.
I. INTRODUCTION
MLOps extends continuous software engineering to machine-learning features embedded in larger software systems. The paper studies its importance through a survey of data scientists’ recent activities.
- ML work involves collaboration across organizational roles because deployed models interact with hosting software, access mechanisms, and monitoring.The paper situates MLOps within broader collaboration needs associated with continuous software engineering.
- MLOps applies continuous delivery practices to ML features, adding ML-specific actions to rapid software deployment.ML features commonly form only part of a larger system, making coordination with the surrounding software and context essential.
- The paper investigates MLOps’ relevance to data scientists’ daily work using responses from 331 ML professionals.Respondents reported what they had worked on during the previous three months.
II. BACKGROUND AND MOTIVATION
MLOps automates and monitors the ML lifecycle from data preparation through deployment, while ML operations must account for model-specific issues such as bias and drift. CD4ML formalizes this lifecycle through cross-functional, incremental delivery.
- MLOps advocates automation and monitoring across ML integration, testing, release, deployment, and infrastructure management.
- ML deployment requires monitoring for biases and drift, including changes that may emerge while the model is in use.
- ML development before model completion is described as relatively waterfall-like, while operationalizing the model follows conventional software practices.
- CD4ML organizes ML delivery into data preparation, model experimentation, and production deployment by data engineers, data scientists, and application developers.The approach uses code, data, and models in small, reproducible increments that can be reliably released.
III. SURVEY
The State of ML 2020 survey examined respondents’ backgrounds, ML problems, data types, short-term goals, and perceived obstacles. Its questions connect daily ML activities with the practical challenges MLOps may address.
- The survey asked about respondent demographics, ML problems, data types, and activities planned for the next three months.
- The survey was designed to understand difficulties data scientists encounter in their daily work.
- Respondents rated obstacles including data shortcomings, staffing, model development, cloud training, pipelines, deployment, experimentation, collaboration, version control, and organizational constraints.
A. Data Collection
The survey was conducted online in May 2020 and received 331 responses from 63 countries through community, newsletter, and LinkedIn recruitment channels.
- 331 responses were collected from professionals in 63 countries through an online survey conducted in May 2020.
- Participants were recruited through the Valohai community, Data Science Weekly newsletter, and LinkedIn advertisements and outreach.
- Eligible respondents held data science or machine-learning-related roles, including data scientist, machine learning engineer, data engineer, or head of data science.
B. Results
Survey respondents commonly worked across models and infrastructure, with relational and time-series data prominent and predictive analysis, time-series problems, and computer vision among the largest ML problem categories. Their near-term goals centered on production-oriented model development and deployment, while data issues were the leading perceived challenges.
- Up to 40% of respondents worked with both models and infrastructure, while relational and time-series data dominated their work.
- Predictive analysis, time-series data, and computer vision were the largest categories of problems companies solved with ML.
- Developing models for production, deploying them, optimizing them, and proving ML's potential were the four leading goals for the next three months.
- Messiness of data was the prime perceived challenge, followed by lack of data and data accessibility.
C. Analysis
The analysis groups ML organizations into data-centric, model-centric, and pipeline-centric stages. Most respondents were model-centric, whereas pipeline-centric organizations showed stronger infrastructure and retraining-related concerns, especially in larger firms.
- Maturity categories: Organizations were classified as data-centric, model-centric, or pipeline-centric according to their ML maturity.
- Maturity categories: Data-centric organizations focus on managing and utilizing data, while model-centric organizations build their first model and move it to production.
- Maturity categories: Pipeline-centric organizations have business-critical models in production and must scale continuous development while maintaining model quality.
- Maturity progression: The authors hypothesize that organizations generally progress from data-centric to model-centric and then pipeline-centric operations requiring automated pipelines.
- Survey analysis: Most respondents were model-centric, focusing on developing models and pushing them to production, while pipeline-centric considerations remained a minority.
- Pipeline-centric organizations: Among 22 organizations automating model retraining, data-centric tasks were no longer central concerns.
- Pipeline-centric organizations: Pipeline-centric organizations tended to have data science teams of 11–30 people, indicating strategic organizational commitment.
- Pipeline-centric organizations: Scale-ups and enterprises were the most pipeline-centric organizations, where MLOps was considered most valuable for reducing training-pipeline burdens.
IV. DISCUSSION
The survey indicates that ML is becoming more team-based and that data scientists’ roles are expanding toward infrastructure and deployment, although the full scope of ML expertise remains undefined.
- ML is moving from one-person proof-of-concepts toward more mature setups in which teams of developers collaborate on ML development.
- Data scientists’ skills are expanding beyond data science and models into ML infrastructure and deployment.
V. CONCLUSIONS
MLOps applies continuous delivery to ML, but ML-specific data, training, and model-management complexities require more than directly reusing DevOps tools. The survey finds that most respondents still focus on data and models, while more mature organizations are entering the MLOps arena.
- MLOps is becoming a requirement for companies that apply ML in production, but DevOps tools cannot be transferred to MLOps through a simple plug-and-play approach.
- As the number of models grows, maintaining coherence and quality becomes more complex, requiring systematic versioning of models and datasets.
- Most respondents still work with data and models, while pioneers entering the MLOps arena tend to have larger teams and a more infrastructure-oriented mindset.