PaperlayerView exact revision on arXiv

Source-linked AI summary

Challenges in Deploying Machine Learning: a Survey of Case Studies

Andrei Paleyes, Raoul-Gabriel Urma, Neil D. Lawrence

arXiv:2011.09926v3cs.LG

TL;DR

Deploying machine learning in production involves challenges that are not comprehensively covered in academic literature. This survey reviews reports and case studies across deployment stages, finding challenges at every step of the workflow and across broader concerns such as ethics, trust, law, and security.

  • Problem

    Academic literature does not widely cover challenges across the entire machine learning deployment pipeline, despite growing real-world deployment and reports of practical difficulties.

  • Method

    The survey reviews recent publications, reports, interviews, regulations, and case studies to map practical deployment challenges to stages of the machine learning workflow.

  • Results

    Practitioners face challenges at every step of the machine learning deployment workflow, including data management, model learning, verification, deployment, ethics, trust, law, and security.

  • Takeaways & Limitations

    The survey provides a cross-industry account of practical deployment challenges that can inform further work on approaches to address them.

  • Takeaways & Limitations

    The reviewed challenge set is far from complete, and further work could identify additional especially non-technical challenges through interviews with industry representatives.

Abstract

from arXiv · show

In recent years, machine learning has transitioned from a field of academic research interest to a field capable of solving real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. By mapping found challenges to the steps of the machine learning deployment workflow we show that practitioners face issues at each stage of the deployment process. The goal of this paper is to lay out a research agenda to explore approaches addressing these challenges.

1 INTRODUCTION

Machine learning’s growing adoption has created a need to understand the practical difficulties of deploying models in production. This survey reviews reports across industries, maps challenges to deployment stages, and aims to support research addressing them.

  • The authors review case studies to extract practitioners’ problems and concerns at each deployment stage, alongside cross-cutting issues affecting the whole workflow.These issues include ethics, law, end users’ trust, and security.
  • Professional surveys report model deployment times of 8–90 days for most companies, with 18% taking longer.An IDC survey of 2,473 organizations also identified lack of expertise, data bias, and high costs as primary reasons for failed AI deployments.
  • The academic literature covers the challenges of the entire machine learning deployment pipeline less broadly than business reports do.
  • The survey broadens prior general-purpose work by covering deployment case studies from a wide variety of industries.It also discusses commonly reported challenges in greater detail than the compared survey focused on information technology.
  • The survey considers case studies, field-specific reviews, and lessons-learned papers, focusing primarily on publications from the previous five years.The authors also refer to practical guidance reports, interview studies, and regulations, but conducted no interviews themselves.
  • The paper’s contribution is to show that practitioners face challenges at every stage of the machine learning deployment workflow.It seeks to raise awareness in the academic community and stimulate discussion about addressing these problems.

2 MACHINE LEARNING DEPLOYMENT WORKFLOW

The survey organizes industrial machine learning development into four workflow stages and examines issues at each stage. It also identifies requirements formulation as a cross-domain challenge outside the adopted workflow definition.

  • The adopted workflow contains four stages: data management, model learning, model verification, and model deployment.These cover data preparation; model selection and training; functional and performance checks; and infrastructure integration, maintenance, and updates.
  • Data management prepares the data needed to build a machine learning model.
  • Model learning covers model selection and training, while model verification checks functional and performance requirements.
  • Model deployment integrates the trained model into software infrastructure and includes maintenance and updates.
  • The survey discusses common issues at each workflow step and cross-cutting concerns affecting the full pipeline.Examples are drawn from different learning tasks where appropriate, and Table 1 summarizes the issues and concerns explored.
  • Requirements formulation is treated as a challenge across many domains but excluded from the adopted machine learning workflow.The workflow definition used by the survey does not consider it part of the machine learning workflow.

3 DATA MANAGEMENT

Data management is the first production-ML stage, but practitioners face substantial difficulty locating, integrating, labeling, validating, and profiling data. These challenges arise from distributed sources, missing or costly labels, limited coverage of real-world conditions, and undetected data-quality problems.

  • 3 DATA MANAGEMENT: Data preparation is the first production-ML stage and consumes more time and energy than practitioners often anticipate.Its effectiveness depends on training and test data as well as the algorithm.
  • 3.1 Data collection: Distributed services and heterogeneous sources make discovering, understanding, and joining relevant data into a usable dataset difficult.Firebird combined 12 datasets whose spatial identifiers required cleaning and correction.
  • 3.3 Data augmentation: Missing labels create a major augmentation challenge because annotation may require scarce experts, representative examples, or processing very large data volumes.In medical imaging, weak or noisy annotations can reduce model quality, while network traffic may reach 1.5 million packets per second per 1-GB/s interface.
  • 3.3 Data augmentation: Real-world systems may lack high-variance data, leaving models unprepared for unsafe situations and limiting transfer from laboratory or simulated settings.Autonomous-vehicle simulations can omit complex interactions such as friction, so small variations may prevent real-world transfer.
  • 3.3 Data augmentation: Well-designed annotation interfaces can improve the quality and overall success of applications collecting labels for medical images.The cited projects all benefited from a well-designed user interface.
  • 3.4 Data analysis: Data must be continuously validated and profiled because unnoticed errors, distribution shifts, and inconsistent types can propagate through the pipeline.Visualization is identified as particularly challenging for troubleshooting data quality and checking assumptions.

4 MODEL LEARNING

Model learning involves model selection, training, and hyper-parameter selection, where practical constraints often determine choices beyond predictive performance. The survey highlights trade-offs involving complexity, interpretability, hardware, computational cost, environmental impact, privacy, and feasible optimization bounds.

  • 4 MODEL LEARNING: Model learning covers model selection, training, and hyper-parameter selection, despite receiving substantial attention from academic research.The section identifies these as three practical steps within the deployment workflow.
  • 4.1 Model selection: Simpler models are often selected because they reduce deployment time, support feedback collection, require fewer resources, and can avoid overcomplicated designs.They are also favored in resource-constrained environments such as spacecraft and wireless cellular networks.
  • 4.1 Model selection: Interpretability can outweigh performance when models must express decisions in understandable business terms and support auditing.Decision trees provide transparent branching logic and are used for interpretable churn prediction in banking.
  • 4.1 Model selection: Deep learning is useful for offline analysis of large acquired datasets, but computational resource demands remain a blocker for onboard UAV deployment.This limits its use as an online processing instrument on UAVs.
  • 4.2 Training: $50k–$1.6m in cloud resources is the reported full-training cost for BERT, with training data size, parameter count, and operation count contributing to the total.The passage describes this cost as unaffordable for most research institutions and many companies.
  • 4.2 Training: Training also raises environmental and privacy concerns because growing computational demand increases emissions, while membership-inference attacks can identify training records.One neural-architecture-search training cycle was estimated to emit as much CO2 as four average cars over their lifetimes.
  • 4.3 Hyper-parameter selection: Hyper-parameter optimization is constrained by repeated training costs, incomplete knowledge for defining search spaces, and deployment hardware limits.Embedded and mobile models require joint optimization of accuracy with energy and memory constraints.

5 MODEL VERIFICATION

Model verification must assess functional requirements, robustness, generalization, and domain-relevant value rather than accuracy alone. The survey emphasizes that business metrics, regulation, data quality, and real-world testing create practical verification constraints.

  • 5 MODEL VERIFICATION: Verification should ensure generalization, edge-case handling, robustness, and compliance with functional requirements.The workflow includes requirement encoding, formal verification, and test-based verification.
  • 5.1 Requirement encoding: Higher model performance may not increase business value when proxy metrics such as clicks fail to reflect outcomes such as conversion.Booking.com therefore used domain metrics including conversion, customer-service tickets, and cancellations alongside accuracy.
  • 5.1 Requirement encoding: Verification metrics should reflect audience priorities, including bias and fairness or application-specific resource consumption.Examples include fairness validation and controlling spacecraft-resource consumption.
  • 5.2 Formal verification: Formal verification can use mathematical proofs or output-error bounds, but it rarely occurs in practice and is often replaced by regulatory quality standards.Regulatory frameworks specify what quality means and how models demonstrate compliance.
  • 5.3 Test-based Verification: Validation datasets derived from training splits may be insufficient for production deployment.Real-life testing can expose business-driven metrics but is difficult because of safety, security, and scale constraints.
  • 5.3 Test-based Verification: Simulation is cheaper, faster, and more flexible than real-world testing, but assumptions and small simulation-to-reality differences can cause drastic behavioral changes.For autonomous vehicles, validation of the model and simulation environment alone is not considered enough.
  • 5.3 Test-based Verification: Data quality must be monitored because errors introduced by decoupled data generation, code bugs, feedback loops, or changing dependencies can propagate downstream.Such issues are described as difficult to troubleshoot once they affect the pipeline.

6 MODEL DEPLOYMENT

Model deployment involves integration, monitoring, and updating, combining conventional software-maintenance concerns with challenges specific to data-driven systems.

  • Overview: Production ML systems require ongoing maintenance, with challenges shared with regular software services and others unique to ML.
  • Integration: Integration covers implementing models in a consumable, supportable form, while infrastructure construction largely remains a systems-engineering concern.
  • Integration: Code and model reuse can reduce duplicated engineering effort, infrastructure use, and deployment complexity while improving performance on individual tasks.
  • Monitoring: Monitoring remains an open problem because practitioners lack standard metrics and alarm triggers for evolving data, prediction bias, and model performance.
  • Monitoring: Production monitoring may require bespoke checks because model-specific needs make out-of-the-box end-to-end tooling inadequate.
  • Updating: Model updating must account for concept drift, delivery infrastructure, and behavioral changes that can affect users or downstream systems without software errors.

7 CROSS-CUTTING ASPECTS

Ethics, law, user trust, and security affect ML deployment across the workflow, with case studies showing risks from bias, weak governance, limited explainability, and attacks.

  • Ethics: Ethical responsibility should extend across the entire AI project workflow, informing both data collection and modeling activities.
  • Ethics: Hidden biases in training data can disadvantage people through proxy variables, gender defaults, social inequality, or imbalanced facial-analysis datasets.
  • Ethics: ML-generated creative content raises unresolved questions about authorship, originality, and harmful deepfake use.
  • Law: Data-protection regulation becomes especially demanding for sensitive information, while legislation may lag behind ML progress and become outdated.
  • Law: The Streams case shows that insufficiently specific agreements about patient data and involvement can trigger data-protection investigations.
  • End users’ trust: Trust can be fostered through early user involvement or explainability, which stakeholders and some regulations require for automated decisions.
  • Security: Feedback-loop exploitation can poison adaptive systems, while model-stealing attacks can extract equivalent models and threaten intellectual property.

8 DISCUSSION OF POTENTIAL SOLUTIONS

The survey proposes tools, services, standards, and broader process changes to address deployment challenges, while noting dependencies, adoption costs, and high-stakes limitations.

  • Discussion: The survey uses case studies across industries to identify recurring deployment pain points and frame research avenues for scalable ML adoption.
  • Tools and services: End-to-end platforms can support operational maintenance through data storage, retraining, deployment, and model hosting capabilities.
  • Tools and services: Testing tools target model robustness and quality assurance, while weak-supervision libraries address the difficulty of obtaining labels.
  • Tools and services: AutoML tools address model selection and hyperparameter tuning, but practical reports urge extreme caution in high-stakes decision-making.
  • Tools and services: Individual tools introduce dependencies that can accumulate into a maintenance burden and require practitioners to evaluate competing strengths and shortcomings.
  • Holistic approaches: ML projects require management and documentation approaches adapted to unique activities, datasets, and models that conventional software practices cannot directly handle.
  • Holistic approaches: Data-oriented architectures and MLTRL illustrate proposals for making data flows explicit and defining standards for robust ML systems.
  • Holistic approaches: Holistic approaches may ease ML deployment but demand substantial time investment, so risks and benefits should be assessed before adoption.

9 FURTHER WORK

The survey identifies several directions for extending its coverage and practical usefulness, including broader challenge discovery, industry comparison, and mapping reported problems to tools.

  • The reviewed challenges span every ML deployment stage but remain incomplete, especially regarding non-technical challenges.The authors propose interviews with industry representatives to identify additional deployment experiences.
  • Comparative analysis across industries could assess whether commonly reported challenges and solutions transfer between fields.The authors suggest combining quantitative and qualitative analyses of reported challenges with cross-industry comparison.
  • A comprehensive review of available tools and services could map them to reported challenges and help practitioners select appropriate solutions.The proposed resource would extend the paper’s brief discussion of existing tools.

10 CONCLUSION

The survey finds that practical deployment challenges affect every stage of the ML workflow and recur across industries. It calls for holistic, cross-disciplinary approaches while noting that deployment experience remains underreported in academic literature.

  • Practitioners face challenges throughout data management, model learning, verification, and deployment, alongside concerns spanning ethics, trust, law, and security.The survey illustrates these issues with case studies, experience reports, and academic literature from varied fields and industries.
  • The authors argue that ML research should address deployment problems holistically rather than leaving each applied field to develop separate approaches.They identify practical considerations as central to improving the deployment experience.
  • Cross-disciplinary dialogue with software engineering, human-computer interaction, systems, and policymaking could provide relevant solutions and inspiration.The authors connect ML’s deployment challenges with pain points previously experienced in these disciplines.
  • Deployment experience reports are relatively scarce in academic literature, leaving valuable industry knowledge unpublished.The authors encourage organizations to share such reports for wider learning, feedback, and self-reflection.
  • The survey draws on applied-data-science venues and journals spanning machine learning applications and artificial intelligence.The listed sources include ACM SIGKDD, IAAI, ECML PKDD, ICMLA, Applied Artificial Intelligence, and Neural Computing and Applications.
Loading 2011.09926v3…