The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practice

1. Introduction

• AI’s non-determinism creates uncertainty while production systems must remain safe, reliable, and accountable.Integrating models into complex production systems makes quality assurance a central concern.Source: §1 · p. 3
• End-to-end CI/CD pipelines can automate and monitor development, testing, deployment, and infrastructure management, but AI pipelines must also handle data and models.The paper notes that adapting these established software pipelines to AI requires further research because of added system-level complexity.Source: §1 · p. 3
• The paper seeks to synthesize current research on continuous AI development to establish evidence, identify knowledge gaps, and support further study.Source: §1 · p. 3

• A Multivocal Literature Review and follow-up interviews with academia and industry participants address three research questions about continuous AI development.Source: evant conceptual ideas, as well as… · p. 3
• The questions ask which pipeline terms are used and how they differ, which tasks and triggers pipelines involve, and what challenges arise in implementation, adaptation, and use.Source: evant conceptual ideas, as well as… · p. 4
• The paper proceeds from background and related work to methodology, findings, a TFX application, threats to validity, and conclusions with future work.Source: evant conceptual ideas, as well as… · p. 4

2.1. Continuous Software Engineering and DevOps

• CI integrates code changes while checking software quality; Continuous Delivery prepares tested features for a staging or test environment, while Continuous Deployment reaches production automatically.Source: §2.1 · p. 5
• DevOps for AI considers AI-development complexity such as data handling, while MLOps extends DevOps, lifecycle management spans data collection through deployment and monitoring, and CD4ML is its technical implementation.Source: §2.1 · p. 5

2.2. Related Work

• Prior reviews often drew on limited source sets, whereas this study analyzes over 150 papers and covers AI models across application contexts.Source: §2.2 · p. 8
• Related reviews did not comprehensively address this paper’s questions on terminology, pipeline tasks, triggers, and pipeline-related challenges.Source: §2.2 · pp. 8–10
• Earlier pipeline studies variously focus on specific tasks, maturity or automation levels, operationalization, or components, rather than continuous execution across the full pipeline.The review also notes work focused on lifecycle characteristics rather than continuous development tasks.Source: §2.2 · pp. 8–9
• This study extends DataOps work centered on data handling by examining tasks across the pipeline through deployment and monitoring.Source: §2.2 · p. 9
• Other related studies discuss general AI-development challenges, including data collection, quality, governance, drift, and non-technical concerns, rather than focusing on pipeline implementation.Source: §2.2 · p. 10

3. Methodology

• The study design combines an MLR to identify literature, taxonomy development to structure identified aspects, and interviews using deductive category definition.Source: §3 · p. 11, Figure 1 · p. 11

3.1. MLR

• The MLR includes grey literature because formal publications were not yet sufficient for this emerging software-engineering topic.Source: §3.1 · p. 11
• Formal sources began with 37 papers and were expanded through backward and forward snowballing to 79 papers; informal sources were selected separately.Source: §3.1 · p. 12
• The MLR used ten search terms, effort-bounded and theoretical-saturation stopping strategies, and identified 151 sources: 79 formal and 72 informal.Theoretical saturation was not reached in seven of twenty cases.Source: §3.1 · pp. 13–14

3.2. Taxonomy Creation

• A facet-based taxonomy groups extracted information into Data Handling, Model Learning, Software Development, and System Operations, allowing adaptation as the field evolves.Source: §3.1 · pp. 14–15, §3.2 · p. 15

3.3. Qualitative Analysis

• Nine participants were interviewed to assess the literature-derived information and taxonomy; semi-structured interviews covered definitions, tasks, taxonomy evaluation, and challenges.Responses were analyzed using summarizing qualitative content analysis and deductive categories based on the taxonomy.Source: §3.3 · pp. 15–16

4.1.1. DevOps for AI

• DevOps for AI is described as an agile practice for reducing the time and resources between deployment iterations [F59, F33, I8].Source: §4.1.1 · p. 18

4.1.2. CI/CD for AI

• CI/CD for AI extends validation and testing beyond code to data, schemas, and models, including semi-automatic iterative checks [I34, F33].Source: §4.1.2 · pp. 18–19

4.1.3. MLOps

• MLOps applies DevOps principles to ML systems and emphasizes cross-functional collaboration across teams [I34, F64, F54, F32, F46, I67, I59, I61, I27].Continuous Training automates retraining and serving using feedback and production data [I34, F33, I2, I36]; cited maturity models specify three or five levels [I34].Source: §4.1.3 · pp. 19–20

4.1.4. (End-to-End) Lifecycle Management

• End-to-end lifecycle management spans data collection through deployment and monitoring, iterating toward quality and stability while managing artifacts; unlike MLOps, it does not focus on team collaboration [F1, F16, F76, F19, F48, F33].The lifecycle also involves repeatedly executing pipeline tasks and managing, searching, sharing, and analyzing created artifacts [F70, F76, F7, F50, F40, F8].Source: §4.1.4 · pp. 20–21

4.1.5. CD4ML

• CD4ML is a technical implementation of MLOps that applies continuous-delivery principles across AI lifecycle management [F44, I18, I50, I49, I54, F44, I16, I65].Thoughtworks defines it around cross-functional teams producing ML applications in small, safe, reproducible increments [I50].Source: §4.1.5 · p. 21

4.2. Triggers

• Triggers are context-specific to the model, business requirements, and retraining strategy; teams may combine approaches while balancing update benefits against retraining effort [I35, F9, I14, F35, F65, I43, F59, F58].Participants described selecting thresholds through trial and error to minimize resource consumption.Source: §4.2 · p. 22

4.2.1. Feedback and Alert Systems

• Feedback, data changes, data drift, model-performance deterioration, and technical metrics such as latency can trigger pipeline runs [F22, F59, I63, I52, F15, F8, F45, F76, F30, F39, I45, I66, F75, I7].Periodic triggers or thresholds can prevent continuous retraining; participants differed on how much new data materially affects model outcomes [F2, F56, F15].Source: §4.2.1 · pp. 22–23

4.2.2. Orchestration Service and Schedule

• Orchestration services or schedules can trigger runs, trading responsiveness against efficient retraining frequency, resource allocation, and job ordering [F59, F16, I42, F8, F11, I34, F39, F60].One participant ran a pipeline weekly.Source: §4.2.2 · p. 23

4.2.3. Repository

• Repository commits or merges can trigger testing of data, model, or code changes; one participant recommended keeping data and code under the same source control [I5, I54, I10, I13, I38, I55].Source: §4.2.3 · pp. 23–24

4.2.4. Other Triggers

• Three of nine participants used human judgment to assess data sufficiency or retraining need, and infrastructure changes may also trigger updates to maintain model performance and functionality [F42, I42, I31, I5, F39, F29, I26, F72].Human assessment also considered whether improved quality still met users’ needs.Source: §4.2.4 · p. 24

4.3. Pipeline

• The pipeline combines four stages—Data Handling, Model Learning, Software Development, and System Operations—with feedback loops that use production information to improve models and data [F2].Task execution depends on organizational context and costs, and four of nine interviewed organizations did not fully automate their pipelines. Quality-test scope also varies across implementations.Source: §4.3 · pp. 24, 26

4.3.3. Software Development

• Software Development prepares the model for deployment through packaging, system-level quality assurance, and system versioning.Packaging produces deployable artifacts and may containerize the model and dependencies; system-level tests check the behavior of the broader software landscape before production deployment.Source: §4.3 · p. 24, §4.3.3 · pp. 36–37

4.3.1. Data Handling

• Data Handling curates data for training, where availability, quality, and preprocessing affect model quality [F2, F57, F6, I63].The pipeline should keep offline and production transformations and feature extraction consistent to prevent training-serving skew and leakage; it also validates data batches and tracks data, dependencies, and processing steps. In practice, full lineage may be infeasible, and participants report limited data documentation.Source: §4.3.1 · pp. 26–31

4.3.2. Model Learning

• Model Learning covers design, training, quality assurance, and improvement; Zhou et al. [F76] identify it as the pipeline’s most essential stage.Continuous testing can cause test-set overfitting, while model evaluation compares metrics with prior versions and may include fairness or explainability checks. Model-version storage may be constrained by resources or patient-data rules.Source: §4.3.2 · pp. 32–35

4.3.3. Software Development

• Software testing includes integration, API compatibility, stress, and robustness checks, but user acceptance testing requires human involvement; QFD is proposed to generate test cases from customer needs.A support vector machine generates test cases from a coverage matrix built using customer expectations and needs.Source: §4.3.3 · p. 38
• Software-level quality results and packaged models are stored, pipeline tasks are versioned and linked to model versions, and automated documentation handling remains a research gap.Examples include TFX storing a pipeline as an artifact or source code and ModelDB storing it as a sequence of actions.Source: §4.3.3 · pp. 38–39

4.3.4. System Operations

• Deployment follows successful pipeline tasks, selection of the best model—which may not be newest—and fulfillment of user-defined criteria; some pipelines also require human validation.A research gap is how to compare versions fairly when validation data or evaluation methods differ; an unseen dataset is one proposed approach.Source: §4.3.4 · p. 39
• Deployment environments range from cloud-managed platforms to disks or remote storage, while cross-platform operation requires abstraction of configuration management and low-level API calls.Cloud-managed platforms can scale for high-performance computing, low latency, and memory-intensive requirements; models should provide persistent output for a given input across environments.Source: §4.3.4 · pp. 39–41
• Continuous experimentation uses A/B tests, canary releases, or shadow deployments to gather runtime feedback; medical deployments should avoid affecting clinical performance or patient safety, and issues may require rollbacks.The approaches can also help identify potential network effects.Source: §4.3.4 · p. 40
• Production monitoring covers input/output data and drift, model performance, software KPIs, and business outcomes, with results mapped to model versions.Operational KPIs include response time, latency, throughput, resource use, and network statistics; business outcomes can include user engagement.Source: §4.3.4 · pp. 40–41

4.4. Challenges

• The 25 identified challenges are mapped to four stages or general requirements, with colors indicating how often each challenge appeared in the sources.Figure 5 uses yellow for five or fewer occurrences and dark blue for 26–30 occurrences.Source: §4.4 · p. 41, Figure 5 · p. 42, Table 7 · p. 43

4.4.1. Data Handling

• Data Handling challenges center on collection and integration, governance, local data restrictions, data quality, traceability, schema conversion, and machine-readable preparation steps.The most frequently mentioned data-stage issue was data collection and integration, with over 16 mentions.Source: §4.4.1 · p. 47

4.4.2. Model Learning

• Model Learning must balance update timing, testing and overfitting risks, latency, and versioning that tracks model evolution and its links to data and pipeline.Versioning should also help identify reasons for changes in performance and predictions.Source: §4.4.2 · p. 47

4.4.3. Software Development

• Software Development challenges include independent packaging, provenance for composite models and heterogeneous transformations, and backward and forward data-schema compatibility.The provenance challenge grows when combined models were trained on different datasets and transformations are highly heterogeneous.Source: §4.4.3 · p. 48

4.4.4. System Operations

• System Operations must address deployment compatibility, cross-model inference, monitoring and feedback loops, and differing resource and security needs across environments.The environment challenge appeared in 12 sources; edge-device hardware differences can require tailored containers.Source: §4.4.4 · p. 48

4.4.5. General Requirements for the Pipeline

• General requirements include flexible, customizable, reusable, fault-tolerant pipelines that can scale elastically and fit multi-tenant and security settings.Flexibility and customization reflect use-case-specific needs and preferred tools; scaling must account for infrastructure limits and heterogeneous device capabilities.Source: §4.4.5 · p. 49

5. Discussion

• TFX components are mapped to framework tasks, but practical pipelines and execution order vary with resource constraints, implementation difficulty, task costs, and organizational policies.TFX is described as a commonly mentioned pipeline, with an orchestrator such as Kubeflow managing and triggering components.Source: §5 · p. 50, Figure 6 · p. 51

5.1. Data Handling

• TFX data handling ingests and splits data, generates statistics and schemas, validates data quality and drift, then transforms features.ExampleGen handles multiple data sources; developers can modify generated schemas, and Transform performs feature engineering after validation.Source: §5.1 · pp. 52–53

5.2. Model Learning

• TFX model learning tunes hyperparameters before training and evaluates model behavior across data slices to compare candidates and select a suitable model.Trainer can warm-start from an existing model, while saved metadata supports comparison of multiple training runs.Source: §5.2 · p. 53

5.3. Software Development

• TFX’s primary pipeline does not explicitly handle software-development tasks such as packaging or system-level quality assurance; ML Metadata versions and documents software information.Source: §5.3 · p. 53

5.4. System Operations

• In system operations, InfraValidator checks model compatibility with a configured serving environment, Pusher deploys validated models, and ML Metadata records artifacts and provenance.The metadata store can identify features affecting evaluation metrics and enable rerunning only necessary components.Source: §5.4 · p. 54

6. Threats to validity

• Conclusion validity is restricted by the qualitative approach, while a tested interview plan and follow-up questions were used to improve interview reliability and reduce misunderstandings.Source: §6 · p. 55
• One researcher conducted the MLR; test-retest checks yielded 93,3% consistency for source selection decisions but a 23% average difference in information extraction.The authors note that the extraction difference can reflect categories with very few supporting sources.Source: §6 · p. 55
• Construct validity was addressed through literature-quality checklists, documented selection, contribution ratings, source consolidation, contrary findings, and broad search strings.Source: §6 · pp. 55–56
• The study covers AI, ML, and DL and several learning types, but does not consider special cases such as federated or transfer learning.Source: §6 · p. 56

• The review synthesizes 151 formal and informal sources with nine semi-structured interviews to compare five commonly used pipeline terms.Source: §7 · p. 57
• Potential pipeline triggers include feedback and alerts, scheduled orchestration, repository updates, and manual initiation.Source: §7 · p. 57
• The consolidated pipeline has four stages—Data Handling, Model Learning, Software Development, and System Operations—with tasks from data preparation through deployment and monitoring.Source: §7 · p. 57
• The paper maps 25 implementation, adaptation, and usage challenges across the stages, including data integration, reproducibility, schema compatibility, and multi-environment inference.Source: §7 · pp. 57–58
• Future work proposes platform demonstrators to compare pipeline tasks and track implementation, adaptation, and usage challenges.Source: §7 · p. 58

List of Formal References

• The cited references identify Ease.ML as a lifecycle management system and PADL as a modeling and deployment language for advanced analytical services [F1] [F3].Source: List of Formal References · pp. 66–67
• References address MLOps and DevOps practice, including hybrid-cloud diagnostics, AI-enabled application challenges, and data-science-to-delivery processes [F6] [F40] [F42] [F44].The listed titles also include work on what data scientists seek to accomplish with MLOps [F44].Source: List of Formal References · pp. 67, 72
• Production ML pipeline and training references include TFX, continuous training, data validation, continuous deployment, and CI services [F8] [F9] [F15] [F22] [F33].The titles cover production-scale platforms, validation, deployment, and continuous integration.Source: List of Formal References · pp. 67, 68, 69, 71
• References F45–F47 address enterprise AI architecture, trustworthy autonomous systems, and Earth-science model deployment.Source: List of Formal References · pp. 72–73
• References F48–F51 cover collaborative workflow, deep-learning lifecycle management, and end-to-end predictive-model development.Source: List of Formal References · p. 73
• References F52–F56 concern ML serving and data-management challenges in production machine learning.Source: List of Formal References · pp. 73–74
• References F57–F62 include work on data pipelines, edge analytics, and feasibility and generalization tools for ML development.Source: List of Formal References · pp. 74–75
• References F63–F69 cover applied ML deployment, IoT and edge systems, annotation, real-time applications, and MLOps.Source: List of Formal References · pp. 75–76
• References F70–F76 include model-management systems, intelligent sensors, MLflow, and an MLOps case study.Source: List of Formal References · pp. 76–77

List of Informal References

• The list includes practitioner materials on MLOps and applying CI/CD or DevOps practices to machine learning [I1], [I4], [I5], [I7], [I8], [I14].Titles include industrialized AI, Git-based CI/CD, and making machine learning operational.Source: List of Informal References · pp. 77–78
• The entries include an MLOps architecture using TFX, Kubeflow Pipelines, and Cloud Build [I2], work toward automating the AI Operations Lifecycle [I3], and titles on Uber ATG’s machine learning and Uber’s Michelangelo [I19], [I23].Source: List of Informal References · pp. 77, 79
• The list includes sources on continuous delivery and CI/CD for ML models, including Databricks/MLflow pipelines [I54], deployment pipelines [I55], and a CI/CD talk [I56].These entries span a platform summit, a company deployment pipeline, and a tech talk.Source: List of Informal References · p. 83
• Other entries cover deep-learning testing [I57], an MLOps introduction [I58], and adapting SOUP to ML models in certified medical systems [I59].The sources include both a research paper and instructional material.Source: List of Informal References · p. 84