Source-linked AI summary
Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology
Stefan Studer, Thanh Binh Bui, Christian Drescher, Alexander Hanuschkin, Ludwig Winkler, Steven Peters, Klaus-Robert Mueller
TL;DR
CRISP-DM lacks machine-learning-specific tasks and quality-assurance guidance, while practitioners need lifecycle support for successful projects. The paper introduces CRISP-ML(Q), a six-phase process model with quality-assurance methods for managing ML development through monitoring and maintenance.
Problem
CRISP-DM lacks guidance on quality-assurance methodology for machine-learning development, while practitioners often lack state-of-the-art knowledge to mitigate risks and ensure project success.
Method
CRISP-ML(Q) guides practitioners through the entire iterative ML development lifecycle, including feasibility, data and feature decisions, maintenance, monitoring, and risk-oriented quality methods.
Results
The paper introduces CRISP-ML(Q) as a process model with quality-assurance methodology intended to increase efficiency and success in ML projects.
Takeaways & Limitations
The work supports establishing a standardized, industry- and application-neutral process model for ML development with quality-assurance methodology.
Takeaways & Limitations
The process and quality measures are not designed for safety-relevant systems, and legal issues are beyond the paper’s scope.
Abstract
from arXiv · showhide
Machine learning is an established and frequently used technique in industry and academia but a standard process model to improve success and efficiency of machine learning applications is still missing. Project organizations and machine learning practitioners have a need for guidance throughout the life cycle of a machine learning application to meet business expectations. We therefore propose a process model for the development of machine learning applications, that covers six phases from defining the scope to maintaining the deployed machine learning application. The first phase combines business and data understanding as data availability oftentimes affects the feasibility of the project. The sixth phase covers state-of-the-art approaches for monitoring and maintenance of a machine learning applications, as the risk of model degradation in a changing environment is eminent. With each task of the process, we propose quality assurance methodology that is suitable to adress challenges in machine learning development that we identify in form of risks. The methodology is drawn from practical experience and scientific literature and has proven to be general and stable. The process model expands on CRISP-DM, a data mining process model that enjoys strong industry support but lacks to address machine learning specific tasks. Our work proposes an industry and application neutral process model tailored for machine learning applications with focus on technical tasks for quality assurance.
1. Introduction
Machine learning projects often fail to meet sponsor expectations because existing process models provide insufficient guidance for ML-specific quality assurance and life-cycle management.
- 75 to 85 percent of practical ML projects do not match their sponsors’ expectations, according to surveys of leading technology companies.
- Industrial organizations need standards and process guidance specific to ML applications to support consistent product or service quality.
- CRISP-DM is widely used in industry but focuses on data mining rather than deployed ML models making long-term real-time decisions.
- Deployed ML models require monitoring and maintenance because performance can degrade as the operating environment changes.
- The proposed process model retains CRISP-DM’s industry and application neutrality while adding ML-specific quality assurance methodology.
2. Related Work
Related work provides process models and testing practices for ML and data mining, but none fully combines business understanding, complete life-cycle coverage, and quality assurance for practical ML applications.
- CRISP-DM organizes data mining activities into six phases with iterative backtracking, but it was designed for data mining projects.
- Existing ML process models address project challenges or production-readiness tests, yet omit either quality assurance methodology, business understanding, or complete project-phase coverage.
- Business understanding is presented as necessary for defining success criteria and feasibility before subsequent ML tasks.
- Prior quality-oriented data-mining work proposes traceability, test procedures, and test data during evaluation to assess product fitness for purpose.
- The paper addresses these gaps by devising a practical ML process model and providing curated references for detailed task analysis.
3. Quality Assurance in Machine Learning Projects
CRISP-ML(Q) structures ML application development around quality assurance, extending CRISP-DM with ML-specific life-cycle coverage and risk-mitigation practices.
- CRISP-ML(Q) targets applications where an ML model is deployed and maintained as part of a product or service.
- Business and data understanding are merged, while monitoring and maintenance become a separate phase compared with CRISP-DM.
- Quality assurance methodology is introduced for each process phase and task to mitigate risks affecting ML application success and efficiency.
- The quality-assurance flow instantiates requirements and constraints into tasks, identifies risks, and iteratively selects guidelines and checklists for mitigation.
- The processes and quality measures are not designed for safety-critical systems, which might require different or additional measures.
3.1. Business and Data Understanding
Business and Data Understanding jointly defines aligned objectives, verifies data quality, and assesses whether an ML project is feasible before development proceeds. The phase also documents requirements, data changes, risks, and review outputs for iteration or termination.
- The phase merges business and ML requirements because training-sample availability substantially influences application feasibility.
- Success criteria span business, ML, and economic levels and must be measurable and aligned with overall system requirements.
- Success Criteria: An ML success criterion translates a business objective into a minimum acceptable performance level, such as accuracy greater 97%.
- Success Criteria: Economic success can be represented by a KPI such as cost savings with automated quality check per part.
- Feasibility: Feasibility assessment examines training-data availability, size, and quality; insufficient sample size can lead to low out-of-sample performance and project termination or postponement.
- Data Quality: Data quality verification checks descriptions, requirements, formats, units, missing values, and production data against defined conditions.
3.2. Data Preparation
Data Preparation produces the dataset used for modeling while remaining iterative: later modeling or deployment findings can trigger backtracking. Quality assurance addresses representation, sampling, noise, missing values, feature construction, formats, and scaling.
- Data preparation produces the dataset for modeling and permits backtracking when later phases reveal erroneous data.
- Feature Selection: Feature selection should retain necessary features because underused or unstable features add error opportunities and increasing dimensionality requires exponentially more samples.
- Feature Selection: Feature selection methods comprise filter, wrapper, and embedded approaches, with selection bias as a documented risk.
- Feature Selection: Domain experts should inspect selected features because spurious correlations can produce impressive test performance without genuine generalization.
- Sampling: Sampling strategies can address class imbalance, but oversampling may overfit the minority class and undersampling must preserve majority-class characteristics.
- Noise filtering, imputation, feature engineering, augmentation, standardized formats, and normalization require evaluation because transformations can remove signal or introduce bias.
3.3. Modeling
The Modeling phase selects and trains models that satisfy project constraints, then evaluates them across performance and complementary quality properties. It also emphasizes baselines, reproducibility, documentation, and automated search or compact modeling where appropriate.
- Model selection uses business objectives, data, requirements, and constraints to identify a subset of appropriate models.
- Quality Measures: Models should be evaluated on performance, robustness, explainability, scalability, resource demand, and complexity, with weights adapted to the application.
- Model Selection: Lower-capacity models provide baselines before capacity is increased, since no model performs best across all problem classes.
- Model Training: Objectives, optimizers, regularization, and cross-validation jointly define the learning problem and support parameter adaptation, overfitting control, and generalization testing.
- Unlabeled data, transfer learning, compression, and ensembles offer alternatives for limited labels, compact models, or fault-tolerant decisions.
- Reproducibility: Reproducibility requires documenting algorithms, data, hyper-parameters, runtime environments, random seeds, experiments, and performance variability.
3.4. Evaluation
Evaluation verifies whether the model is trustworthy and meets predefined business and ML success criteria. It uses protected testing, robustness assessment, explainability checks, and expert judgment to support deployment, backtracking, or stopping.
- A disjoint blind test set reserved for final evaluation prevents test information from leaking into training or validation.
- Test data should represent the input distribution and invariances, while extensive testing addresses uncovered rare corner cases.
- Robustness: Robustness evaluation perturbs inputs or varies hyper-parameters to characterize sensitivity to noisy, wrong, or adversarial data.
- Explainability: Explainability can help identify errors and guide improvements such as enriching the dataset.
- Experts compare documented evaluation results with business and ML success criteria; unmet criteria can trigger backtracking or project termination.
3.5. Deployment
The deployment phase brings the model into practical use while addressing hardware constraints, production mismatch, usability, unforeseen errors, and rollout risk.
- Deployment applies the ML model in its designated field of application.
- Inference hardware is selected according to requirements or aligned with existing hardware, with cloud and embedded systems imposing different constraints.Optimization may target CPU/GPU availability, operating systems, or the ML workload.
- Production-condition evaluation incrementally tests whether production data, hardware, and environment match training assumptions.Each step calibrates the model to the deployed hardware and test environment to identify wrong assumptions and degradation causes.
- User acceptance and usability are assessed through prototypes and field tests that examine acceptance, usage rate, and user experience.User guides and disclaimers explain the system’s functionality and limitations.
- Fallback plans can address erroneous updates, bugs, outages, and system shutdowns by reverting to a previous model, baseline, or rule-based system.
- Incremental deployment strategies with model-and-data pipelines reduce the impact and repair cost of errors that escape prior evaluation.
3.6. Monitoring and Maintenance
Long-running ML applications require monitoring and maintenance because changing data distributions can degrade model performance and produce false predictions.
- Long-running ML models require lifecycle management because violations of training–inference distribution assumptions can degrade performance over time.
- Non-stationary feature or label distributions make training data stale and degrade model performance over time.The frequency of change depends on the domain, with stock-market data described as more volatile than elephant visual properties.
- The maintenance phase uses sequential monitor and update tasks to determine whether a model needs updating and whether the update succeeded.
- Monitoring compares incoming-data and predicted-label statistics with training-data statistics and can reject inputs that violate the defined schema.
- Updating collects newly labeled data, retrains or fine-tunes the model under changed distributions, and requires reevaluation before deployment.Fine-tuning the existing model may avoid training a completely new model from scratch, although labeling new data can be expensive.
- Deployment infrastructure tracks usage and performance, supports strategies such as A/B testing, and provides automatic or human-controlled fallback to previous models.
4. Discussion
The discussion presents CRISP-ML(Q) as a lifecycle process model that combines ML guidance with quality assurance methods intended to mitigate task-specific risks.
- CRISP-ML(Q) guides practitioners through the full ML development lifecycle, including iterative phases, tasks, maintenance, and monitoring.
- The model provides quality-oriented methods for mitigating risks identified for specific tasks.
- The proposed methods are characterized as best practices in ML projects across industry and academia.
- The authors argue that quality assurance is important because practitioners may focus solely on predictive performance while lacking state-of-the-art risk-mitigation knowledge.The discussion states that specialist literature is not covered in ML textbooks or academic curricula, leaving novices to industry practice without profound state-of-the-art knowledge.
5. Conclusions
The conclusion calls for standardizing and implementing CRISP-ML(Q), while limiting the current process and quality measures to non-safety-relevant systems.
- Standardizing a process model is identified as an important future step for enabling more successful ML projects and affecting the ML community.
- The process and quality measures are not designed for safety-relevant systems.
- The study of safety-relevant systems and discussion of legal constraints are left to future work.
- The authors encourage automotive and other industries to implement CRISP-ML(Q) and contribute knowledge toward a cross-industry standard process model.