Source-linked AI summary
Adoption and Effects of Software Engineering Best Practices in Machine Learning
Alex Serban, Koen van der Blom, Holger Hoos, Joost Visser
TL;DR
ML applications require stronger engineering guidance, but academic evidence on suitable practices is limited. The paper mines academic and grey literature to identify 29 practices and surveys 313 practitioners to measure adoption and perceived effects, finding that adoption patterns can support effect prediction and prioritised improvement. Survey-response bias and the gap between perceived and actual effects limit the conclusions.
Problem
ML applications require robust development, deployment, and maintenance, yet practitioner guidance is limited because much existing practice literature is grey literature.
Method
The study mined academic and grey literature to compile 29 ML software-engineering practices and surveyed practitioners about their adoption and perceived effects.
Results
Practice adoption was strongly dependent on intended effects, and combining adoption rates with model-derived practice contributions enabled prioritised improvements for effects such as traceability and software quality.
Takeaways & Limitations
The survey and analysis provide a quantitative basis for assessing and progressively improving practice adoption by ML teams.
Takeaways & Limitations
Survey answers may be affected by selection bias, and the study did not establish that perceived effects reflect actual effects.
Abstract
from arXiv · showhide
The increasing reliance on applications with machine learning (ML) components calls for mature engineering techniques that ensure these are built in a robust and future-proof manner. We aim to empirically determine the state of the art in how teams develop, deploy and maintain software with ML components. We mined both academic and grey literature and identified 29 engineering best practices for ML applications. We conducted a survey among 313 practitioners to determine the degree of adoption for these practices and to validate their perceived effects. Using the survey responses, we quantified practice adoption, differentiated along demographic characteristics, such as geography or team size. We also tested correlations and investigated linear and non-linear relationships between practices and their perceived effect using various statistical models. Our findings indicate, for example, that larger teams tend to adopt more practices, and that traditional software engineering practices tend to have lower adoption than ML specific practices. Also, the statistical models can accurately predict perceived effects such as agility, software quality and traceability, from the degree of adoption for specific sets of practices. Combining practice adoption rates with practice importance, as revealed by statistical models, we identify practices that are important but have low adoption, as well as practices that are widely adopted but are less important for the effects we studied. Overall, our survey and the analysis of responses received provide a quantitative basis for assessment and step-wise improvement of practice adoption by ML teams.
1 INTRODUCTION
The paper addresses limited practitioner guidance for engineering ML applications by cataloguing 29 practices and surveying their adoption and perceived effects. It finds that practice groups relate to intended effects and supports prioritising improvements using adoption and predictive-model evidence.
- 29 engineering best practices were distilled from academic and grey literature and classified as traditional, modified, or ML-specific.
- 313 practitioner responses measured adoption and the perceived effects of adopting specific practice sets across four effects.
- Practice groups showed strong dependency with intended effects, enabling predictive models to estimate outcomes such as traceability and software quality.
- Combining predicted practice contributions with adoption rates provides a method for prioritising practice improvements for targeted effects.
2 BACKGROUND AND RELATED WORK
ML engineering has distinctive challenges and limited academic guidance, while existing studies often focus on narrower practice areas or regionally biased evidence. This paper broadens the practice scope through a survey of practitioners and actionable, fine-grained recommendations.
- The literature on ML engineering practices is dominated by grey literature, while academic coverage remains relatively limited and its claims may lack strong empirical evidence.
- ML systems combine traditional maintenance problems with additional issues specific to ML components, including difficult testing, data management, and monitoring.
- Prior work includes technical-debt practices, testing and monitoring practices, developer checklists, architecture patterns, and organisational maturity models, but often addresses narrower or less actionable scopes.
- The study bridges this gap with a practitioner survey spanning varied backgrounds and a set of actionable, fine-grained best practices.
3 MINING PRACTICES FROM LITERATURE
The authors mined academic and grey literature using systematic searches and filtering, then organised extracted recommendations into a six-class taxonomy and a catalogue of 29 practices. They converted broad goals into actionable practices while retaining effects as higher-level outcomes.
- 64 query variants produced 21 relevant documents after quality screening, duplicate filtering, and snowballing from references.
- Extracted recommendations were organised into Data, Training, Deployment, Coding, Team, and Governance classes for the survey and analysis.
- The mining process abstracted actionable practices from recommendations and represented their higher-level goals as Effects, such as traceability.
- Six traditional software-engineering practices supplemented 23 initially selected practices, yielding 29 practices documented with references and adoption ranks.
4 STUDY DESIGN
The study used a cross-sectional survey of teams using ML components, refined through five pilot interviews, to measure practice adoption and perceived effects. Its questionnaire combined demographic filtering, practice-specific adoption questions, and four effect questions.
- The cross-sectional survey targeted teams using ML components and collected team, experience, organisation, and data-type information for grouping and filtering.
- 31 standardised questions mapped onto the practices, with multiple questions combined when they represented one practice such as continuous integration.
- Four final questions measured respondents’ perceived effects of adopting the practices, enabling tests of whether practice groups help achieve those effects.
- Five practitioners piloted the questionnaire, confirmed practice relevance, and suggested response refinements before broad distribution.
- The survey was distributed through a snowball strategy using professional contacts, publication authors, practitioner channels, and ML meetups.
5 FINDINGS ON PRACTICE ADOPTION
The survey examined practice adoption across participant demographics, practice types, and data types, finding higher adoption in larger and more experienced teams and among ML-specific practices. Adoption varied by organisation and data type, while common high-adoption practices focused on training objectives, metrics, versioning, monitoring, and reusable data-management scripts.
- Demographic differences: North American respondents reported significantly more adopted practices, while European adoption showed no striking differences from South America or Asia despite European over-representation.The authors therefore considered the regional imbalance unlikely to introduce substantial bias.
- Demographic differences: Tech companies showed higher complete adoption than other organisations, whereas research organisations had lower adoption overall but similar rates for non-deployment practices.The paper suggests research teams may develop prototypes where full adoption is unnecessary or partial adoption is sufficient.
- Demographic differences: Larger and more experienced teams generally adopted more practices, although teams with over five years of experience showed a slight increase in partial or non-adoption.The authors suggest this may reflect task distribution, diverse backgrounds, or limited awareness of newer practices among early ML practitioners.
- Practice types and rankings: Practices establishing and communicating training objectives and metrics ranked highest, followed by versioning, model monitoring during experimentation, and reusable data-management scripts.The five most-adopted practices were predominantly new or modified ML practices rather than traditional software-engineering practices.
- Practice types and rankings: Feature management, testing, automated hyper-parameter optimisation, and shadow deployment were among the least adopted practices, which generally required more effort or knowledge.Traditional code-quality practices such as regression testing and static analysis also had low adoption.
- Data types: Tabular data, text, images, and videos had similar overall adoption rates, while audio and time-series estimates were less reliable because they were less represented.Graph-data adoption rates were too unreliable to report because graphs were used by only 0.5% of respondents.
6 ANALYSIS OF PRACTICES AND EFFECTS
The analysis tests how practice groups relate to perceived effects using correlation, regression, and predictive models. Effects are accurately predictable from practice adoption, with non-linear models outperforming linear regression and practice importance informing improvement priorities.
- Practice correlations: Practice correlations reveal coordinated adoption patterns, including deployment practices clustering together and peer review correlating with collaborative team practices.Automated model deployment correlates with shadow deployment (ρ = 0.48) and automated rollbacks (ρ = 0.51).
- Regression analysis: All four regression models reject the null hypothesis that no practices determine the effect, with high R2 values across effects.Practice 25 has a traceability t-value of 6.29.
- Predictive models: RF models consistently outperform linear regression on test data, indicating that at least some practices have non-linear impacts on the studied effects.The evaluation used MSE, R2, and Spearman correlation on held-out test data.
- Predictive models: ρ ≥0.90 for predicted versus true outcomes across nearly all models, indicating that perceived effects can be accurately predicted from associated practice sets.Linear regression was the main exception.
- Practice importance: Shapley values from grid-search RF models quantify each practice’s contribution and are contrasted with adoption rankings to identify improvement priorities.Figure 3 compares practice importance with normalized adoption ranks; functional dependencies are not included.
7 DISCUSSION
The discussion compares adoption with previously identified engineering challenges and evaluates threats to validity. Testing remains a low-adoption challenge, while survey, literature, and perceived-effect measurements constrain interpretation.
- 7 DISCUSSION: Many previously identified challenges are supported by well-adopted practices, including artefact versioning, problem formulation, experiment monitoring, and outcome sharing.Artefact versioning ranks third in adoption.
- 7 DISCUSSION: Testing ML artefacts remains challenging because the corresponding testing practices have low adoption despite being identified as an engineering challenge.Practices 8 and 17 have low adoption in the study.
- 7 DISCUSSION: Feature management and automated hyper-parameter optimisation have low adoption despite literature recommendations, so their relevance requires further validation.The authors plan participant-validation interviews and additional data collection.
- Threats to validity: The catalogue may be biased because literature extraction and completeness of the identified practices remain subject to further participant validation.Researchers with different backgrounds and pilot interviews were used to limit this risk.
- Threats to validity: Survey findings may contain selection bias because Europe is over-represented and North American adoption patterns differ from other groups.The authors propose gathering more data to address this bias.
- Threats to validity: The effect analysis uses respondents’ perceived effects rather than measurements of actual effects, and the relationship between them remains unestablished.The authors identify this as an important topic for future research.
8 CONCLUSIONS AND FUTURE RESEARCH
The study compiles and evaluates 29 ML software-engineering practices through a survey of 313 respondents. It finds adoption differences across teams, strong predictive relationships with perceived effects, and a basis for prioritizing practice improvements, while calling for broader validation.
- 8 CONCLUSIONS AND FUTURE RESEARCH: The study combines literature mining with a survey of 313 respondents to measure adoption and perceived effects for 29 ML engineering practices.The practices are grouped into six categories.
- 8 CONCLUSIONS AND FUTURE RESEARCH: Larger teams adopt more practices, while traditional software-engineering practices have lower adoption than ML-specific practices.Technology companies also show higher adoption than non-technology organizations, government organizations, and research labs.
- 8 CONCLUSIONS AND FUTURE RESEARCH: Specific practice sets correlate positively with traceability, software quality, agility, and team effectiveness, and predictive models estimate these perceived effects with high accuracy.The conclusion summarizes the broader relationship analysis across the studied effects.
- 8 CONCLUSIONS AND FUTURE RESEARCH: Contrasting practice importance with adoption identifies which practices merit more or less attention, including logging production predictions with model versions and input data for traceability.The study presents this comparison as a basis for team or organizational improvement planning.
- 8 CONCLUSIONS AND FUTURE RESEARCH: Future work will expand respondents, improve measurement of AutoML-related effects, better cover traditional software-engineering practices, and conduct validation interviews.These plans address finer-grained analysis and catalogue validation.