Source-linked AI summary
Machine Learning Operations (MLOps): Overview, Definition, and Architecture
Dominik Kreuzberger, Niklas Kühl, Sebastian Hirschl
TL;DR
Industrial ML projects often struggle to automate operations and move from proof of concept to production, while MLOps remains conceptually ambiguous. The paper addresses this gap through mixed-method research combining literature, tooling, and expert evidence, producing a unified account of MLOps. It aggregates principles, components, roles, architectures, and workflows, defines MLOps, and identifies open challenges.
Problem
Many ML proofs of concept do not reach production, while research has emphasized model building over production-ready products and coordination of complex ML systems.
Method
The paper conducts mixed-method research combining a literature review, tool review, and expert interviews to conceptualize MLOps and its system elements.
Results
The study provides an aggregated overview of MLOps principles, components, roles, architectures, and workflows, and derives a definition of MLOps.
Takeaways & Limitations
The resulting guidelines and role responsibilities are intended to support researchers and professionals automating and operating ML products.
Takeaways & Limitations
MLOps requires many skilled roles, but the field faces a lack of highly skilled experts for these roles.
Abstract
from arXiv · showhide
The final goal of all industrial machine learning (ML) projects is to develop ML products and rapidly bring them into production. However, it is highly challenging to automate and operationalize ML products and thus many ML endeavors fail to deliver on their expectations. The paradigm of Machine Learning Operations (MLOps) addresses this issue. MLOps includes several aspects, such as best practices, sets of concepts, and development culture. However, MLOps is still a vague term and its consequences for researchers and professionals are ambiguous. To address this gap, we conduct mixed-method research, including a literature review, a tool review, and expert interviews. As a result of these investigations, we provide an aggregated overview of the necessary principles, components, and roles, as well as the associated architecture and workflows. Furthermore, we furnish a definition of MLOps and highlight open challenges in the field. Finally, this work provides guidance for ML researchers and practitioners who want to automate and operate their ML products with a designated set of technologies.
1 Introduction
The paper frames MLOps as a response to ML projects struggling to become production-ready because model-focused development has neglected operational coordination. It investigates MLOps holistically to clarify its meaning and support automation and operationalization.
- Many ML projects fail to progress from proof of concept to production because real-world ML systems and their infrastructure are difficult to coordinate.
- Existing ML research has focused on building models rather than production-ready products and the coordination required to operate complex systems.
- The paper examines how manual ML processes can be automated and operationalized through a holistic study of MLOps principles, components, roles, and architectures.
- The research asks what MLOps is and uses its findings to provide a definition intended to support common understanding of the term and related concepts.
- The authors aim to provide guidelines and precise responsibilities that may help more proofs of concept reach production with fewer design errors and more robust real-world predictions.
2 Foundations of DevOps
The paper situates MLOps within software engineering’s progression from process models toward DevOps, a paradigm addressing both social and technical issues in software development.
- Software engineering introduced process models such as waterfall and agile to deliver production-ready software products.
- DevOps emerged around 2008/2009 as a paradigm intended to reduce issues in software development while addressing organizational social and technical concerns.
3 Methodology
The study combines literature review, tooling review, and expert interviews to examine MLOps from academic and practical perspectives. It then uses these sources to conceptualize MLOps and its technical components.
- The methodology combines a structured literature review, a review of MLOps tools, and semi-structured interviews with experts from different domains.
- Literature Review: The literature review searched five scientific databases and selected 27 peer-reviewed articles after screening 1,864 retrieved articles and 194 papers in detail.
- Tool Review: The tooling review examined open-source tools, frameworks, and commercial cloud ML services to understand their technical components.
- Expert Interviews: Expert interviews used theoretical sampling to select experienced practitioners and obtain meaningful insights from a limited number of interviews.
- Expert Interviews: The interviews used semi-structured questions, soft laddering, recording, transcription, and open coding to analyze practitioner experiences.
4 Results
The results synthesize MLOps into principles, technical components, roles, and an architecture-oriented workflow. Together, these elements describe how ML systems can be designed, automated, deployed, and operated in production.
- The study structures MLOps insights into principles, components, roles, architecture, and workflow.The resulting conceptualization also provides a definition of MLOps.
- Principles: Nine principles guide MLOps realization and link to associated technical components.Figure 2 illustrates these principle-to-component associations.
- Principles: CI/CD automation builds, tests, delivers, and deploys ML artifacts while providing rapid feedback to developers.This feedback is described as increasing overall productivity.
- Principles: Workflow orchestration coordinates ML pipeline tasks through DAGs that encode execution order and dependencies.The workflow orchestration component represents execution order and artifact usage for individual workflow steps.
- Principles: Reproducibility, versioning, collaboration, metadata tracking, continuous training and evaluation, monitoring, and feedback loops support reliable ML operations.Versioning covers data, models, and code; monitoring and feedback loops support detecting changes and retraining.
- Roles: MLOps requires interdisciplinary roles spanning business, architecture, data science, data engineering, software engineering, DevOps, and ML/MLOps engineering.The ML/MLOps engineer combines cross-domain skills and operates infrastructure, workflows, deployment, and monitoring.
- Roles: The DevOps engineer and ML/MLOps engineer help automate workflows, deploy models, and monitor production systems.The ML/MLOps engineer additionally builds and operates ML infrastructure and manages automated workflow pipelines.
5 Architecture and Workflow
The paper presents a technology-agnostic, end-to-end MLOps architecture spanning project initiation, feature engineering, experimentation, automated workflow execution, and model serving. It assigns these activities to functional components and roles, with automation, tracking, deployment, monitoring, and feedback connecting the workflow.
- Architecture design: The architecture is technology-agnostic, allowing researchers and practitioners to select technologies and frameworks that fit their needs.The artifact is intended to provide general guidance rather than prescribe a specific technology stack.
- End-to-end architecture: The generalized architecture covers project initiation, feature engineering, experimentation, automated ML workflow execution, and model serving.These stages form the paper’s end-to-end process from identifying an ML opportunity through serving the model.
- Project initiation: During project initiation, business, architecture, ML, and data responsibilities are distributed across stakeholders, solution architects, data scientists, and data engineers.The data scientist derives the ML problem, while data and problem requirements are jointly clarified and raw data are assessed for distribution, quality, validation, and labeling.
- Feature engineering: The feature engineering pipeline transforms and cleans data, creates features, and ingests batch or streaming data into offline or online feature stores.Transformation and cleaning rules are defined and iteratively improved using feedback from experimentation or production monitoring.
- Experimentation: Experimentation uses feature-store data for preparation, validation, train/test splitting, algorithm and hyperparameter selection, and model training.Data scientists lead experimentation with software-engineering support, and required data changes are reported back to the data-engineering zone.
- Automated ML workflow: CI/CD automatically builds, tests, and delivers updated model or workflow code, while orchestration manages automated data extraction, preparation, validation, training, evaluation, and model registration.Training can repeat through hyperparameter adjustments, and the trained model is exported with configuration and environment files to the model registry.
- Tracking and reproducibility: Metadata tracking records training parameters, performance metrics, job details, artifact sources, and model lineage for registered models.Model lineage combines the lineage of data and code, including feature-data and model versions.
- Deployment and monitoring: After a model moves from staging to production, continuous deployment serves it, while monitoring can trigger retraining when drift, new data, or a schedule warrants it.Drift can be detected through distribution comparisons, and monitoring forwards the information to a scheduler that triggers the automated workflow.
6 Conceptualization
The paper conceptualizes MLOps as an engineering practice at the intersection of machine learning, software engineering, and data engineering. It defines MLOps as an end-to-end paradigm for productionizing ML systems through coordinated automation, reproducibility, monitoring, and feedback.
- Disciplinary intersection: MLOps is positioned at the intersection of machine learning, software engineering, DevOps, and data engineering.The paper presents this positioning as the conceptual synthesis of its literature and interview findings.
- Definition: MLOps is a paradigm encompassing best practices, concepts, and development culture for the end-to-end conceptualization, implementation, monitoring, deployment, and scalability of ML products.The definition treats MLOps as broader than a single tool or development technique.
- Operational principles: MLOps aims to productionize ML systems by bridging development and operations.Its stated principles include CI/CD automation, workflow orchestration, reproducibility, versioning, collaboration, continuous training and evaluation, metadata tracking, monitoring, and feedback loops.
7 Open Challenges
MLOps adoption faces organizational, ML system, and operational challenges spanning culture, expertise, scalability, automation, governance, and troubleshooting. These challenges constrain the development and operation of production ML products.
- Organizational challenges: MLOps requires a shift from model-driven work toward a product-oriented culture, but organizations often lack experts for key roles.The cited roles include architects, data engineers, ML engineers, and DevOps engineers.
- Organizational challenges: MLOps is a group process requiring multidisciplinary collaboration, yet silos, differing knowledge levels, and specialized terminology hinder communication.Decision-makers also need to recognize the business improvements associated with greater MLOps maturity and a product-focused mindset.
- ML system challenges: Fluctuating data volumes and demand make infrastructure requirements difficult to estimate and require flexible, scalable CPU, RAM, and GPU resources.The challenge is especially pronounced for ML training processes.
- Operational challenges: Manual operation is difficult across heterogeneous software and hardware stacks, while incoming data requires repeated retraining and robust automation.Operational work also produces artifacts requiring governance and versioning of data, models, and code.
- Operational challenges: Support is difficult because failures can involve many parties and components across ML infrastructure and software.Root-cause analysis therefore spans the interacting parts of the production system.
8 Conclusion
The paper addresses the limited progression of ML proofs of concept into production and the research focus on model building rather than operating complex ML systems. Using mixed methods, it identifies core MLOps aspects and derives a holistic definition intended to support ML projects.
- 8 Conclusion: Only a small number of ML proofs of concept progress into deployment and production, while real-world workflows remain largely manual.The paper also notes that academic work has focused more on model building and benchmarking than operating complex ML systems.
- 8 Conclusion: A mixed-method study of literature, tools, and eight experts identifies four main MLOps aspects: principles, components, roles, and architecture.These aspects form the basis for the paper’s holistic definition.
- 8 Conclusion: The resulting definition and associated concepts support a common understanding of MLOps for researchers and professionals.The paper presents this understanding as guidance for setting up ML projects.
Appendix
The appendix contains a list of evaluated technologies, a list of interview partners, and a figure showing the disciplines intersecting in the MLOps paradigm.
- Appendix: Table 1 lists the technologies evaluated in the study.
- Appendix: Table 2 lists the interview partners included in the study.
- Appendix: Figure 5 depicts the intersection of disciplines associated with the MLOps paradigm.