PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices

Mojtaba Shahin, Muhammad Ali Babar, Liming Zhu

arXiv:1703.07019v1cs.SE

TL;DR

Research on continuous practices needed a synthesis of approaches, tools, challenges, and practices. This review analyzed 69 papers and identified 30 approaches and seven critical factors for adopting continuous practices.

  • Problem

    Research on continuous practices needed an evidential synthesis of its state of the art and potential future research areas.

    Source: §VII · p. 25

  • Method

    The authors systematically reviewed 69 papers published from 2004 to June 1, 2016, using predefined inclusion and exclusion criteria for analysis and synthesis.

    Source: §VII · p. 25

  • Results

    30 approaches across six CI/CD support functions and seven critical factors were identified, including testing effort and time, team awareness and transparency, and good design principles.

    Source: §VII · p. 26

  • Takeaways & Limitations

    Further research should investigate how continuous delivery or deployment adoption influences organizational architecting processes and outcomes.

    Source: D. Architecting for deployability · p. 24

    Study selection may be influenced by researchers’ subjective judgments about whether papers meet inclusion or exclusion criteria.

    Source: §VI · p. 25

Abstract

from arXiv · show

Context: Continuous practices, i.e., continuous integration, delivery, and deployment, are the software development industry practices that enable organizations to frequently and reliably release new features and products. With the increasing interest in and literature on continuous practices, it is important to systematically review and synthesize the approaches, tools, challenges, and practices reported for adopting and implementing continuous practices. Objective: This research aimed at systematically reviewing the state of the art of continuous practices to classify approaches and tools, identify challenges and practices in this regard, and identify the gaps for future research. Method: We used systematic literature review (SLR) method for reviewing the peer-reviewed papers on continuous practices published between 2004 and 1st June 2016. We applied thematic analysis method for analysing the data extracted from reviewing 69 papers selected using predefined criteria. Results: We have identified thirty approaches and associated tools, which facilitate the implementation of continuous practices in the following ways: (1) "reducing build and test time in continuous integration (CI)"; (2) "increasing visibility and awareness on build and test results in CI"; (3) "supporting (semi-) automated continuous testing"; (4) "detecting violations, flaws and faults in CI"; (5) "addressing security and scalability issues in deployment pipeline", and (6) "improving dependability and reliability of deployment process". We have also determined a list of critical factors such as "testing (effort and time)", "team awareness and transparency", "good design principles", "customer", "highly skilled and motivated team", "application domain", and "appropriate infrastructure" that should be carefully considered when introducing continuous practices in a given organization.

I. INTRODUCTION … B. Existing literature reviews

The paper reviews continuous integration, delivery, and deployment as related practices, aiming to synthesize how organizations adopt and implement them. It distinguishes this integrated scope from earlier reviews and outlines how its review addresses that gap.

I. INTRODUCTION

  • Continuous practices aim to accelerate software delivery without sacrificing quality, enabling frequent feedback and reliable releases, but organizational processes, practices, and tools may not be ready for adoption.CI integrates work multiple times per day; CDE and CD use automation to release value quickly and reliably to customers.Source: §I · p. 1
  • The SLR reviews 69 papers to classify approaches, tools, challenges, and practices, and to identify critical factors, context-sensitive guidance, and future research issues.Its contributions include an evidence-based guide for selecting approaches, tools, and practices suited to different contexts.Source: §I · pp. 1–2

A. Background

  • CI frequently integrates work and automates builds and tests, while CDE keeps applications production-ready after automated checks and delivers them to production-like environments.CDE requires CI and can reduce deployment risk, costs, and the time to obtain user feedback.Source: A. Background · p. 2
  • CD extends CDE by automatically deploying every change to production without a manual step; unlike CDE’s pull-based release, CD is push-based and may suit only some organizations or systems.CDE’s final deployment can be manual, whereas CD implies CDE but the converse is not true.Source: A. Background · p. 2

B. Existing literature reviews

  • Earlier CI reviews proposed a descriptive model of implementation variation and a framework of 14 organizational implications, while a case study examined five software companies.Stahl and Bosch’s review synthesized 46 primary studies into 22 clusters; Eck et al. studied organizational implications across 43 studies.Source: B. Existing literature reviews · p. 3
  • Other reviews covered rapid release, continuous deployment, and CDE, reporting scarce evidence for rapid-release advantages, organizational changes and customer or embedded-system challenges in CD, and seven categories of CDE problems and solutions.The CD mapping study identified 10 defining factors; the CDE review focused on production readiness rather than continuous automatic deployment.Source: B. Existing literature reviews · p. 3
  • This review used different selection criteria and both automated search and snowballing; among its 69 papers, only 2, 10, 7, and 12 overlapped with, respectively.It targeted empirical studies and excluded papers shorter than six pages, unlike prior reviews that included such papers.Source: B. Existing literature reviews · p. 3
  • The SLR integrates CI, CDE, and CD across approaches, tools, contextual factors, practices, and challenges, reporting a more comprehensive set of challenges and practices than prior reviews.It identified challenges not reported in, including lack of awareness and transparency, resistance to change, team dependencies, customer environments, and legacy-system dependencies; only five practices were shared with.Source: B. Existing literature reviews · pp. 3–4

C. Motivation for this SLR on continuous practices … D. Study selection

Because CI, delivery, and deployment are intertwined and often used interchangeably, this review examines them together to synthesize supporting approaches, tools, challenges, and practices. It applies a systematic review process to identify and assess relevant evidence.

C. Motivation for this SLR on continuous practices

  • The review examines integration, delivery, and deployment together because these intertwined practices are often difficult to distinguish and had not been systematically reviewed as a group.An integrated view was intended to clarify their challenges and support organizations in adopting the practices step by step.Source: C. Motivation for this SLR on… · p. 4

III. RESEARCH METHOD

  • The SLR followed three phases—protocol definition, review conduct, and reporting—with a protocol covering research questions, search, eligibility, selection, and synthesis.The authors describe SLR as a structured process for identifying, evaluating, and interpreting evidence.Source: §III · p. 4

A. Research questions

  • The research questions address approaches and tools, adoption challenges, and reported practices; the review defines approaches as technical methods and practices as shared social norms and activities.Approaches can support and improve practices.Source: A. Research questions · pp. 4–5

B. Search strategy

  • The search combined automatic retrieval across six digital libraries with reference-list snowballing, using pilot-tested terms for continuous practices and software-related concepts.The search terms were checked against known papers before being finalized; they targeted titles, keywords, and abstracts except in SpringerLink.Source: B. Search strategy · p. 5

3) Data sources

  • The six sources were IEEE Xplore, ACM Digital Library, SpringerLink, Wiley Online Library, ScienceDirect, and Scopus; SpringerLink searches covered full text, limited to its first 1,000 results.Its search engine did not support searching titles, abstracts, and keywords; Google Scholar was excluded for low precision and many irrelevant results.Source: 3) Data sources · p. 5

C. Inclusion and exclusion criteria

  • Eligibility was limited to peer-reviewed English papers of at least six pages reporting approaches, tools, or practices through empirical methods; duplicates and less mature versions were excluded.Editorials, position papers, reviews, tutorials, keynotes, and panel discussions were also excluded.Source: C. Inclusion and exclusion criteria · p. 5

D. Study selection

  • Selection narrowed 14,723 retrieved papers to 3,942 potential papers, 449 after title-and-keyword screening, and ultimately 69 included studies after snowballing, full-text review, and quality checks.Snowballing found 51 potentially relevant papers in references, of which 28 proceeded to the next phase; random selected papers were cross-checked by the second author.Source: D. Study selection · pp. 5–6

E. Data extraction and synthesis … A. Demographic attributes

The review combined descriptive statistics for demographic and contextual attributes with thematic analysis of approaches, challenges, practices, and critical factors. Its 69 included papers, published from 2004 to 1 June 2016, were concentrated in recent years and predominantly reported evaluation or validation research in industry settings.

E. Data extraction and synthesis — 1) Data extraction

  • Demographic and contextual items D1–D10 were analyzed descriptively, while D11–D14 were examined thematically; research types were classified into six categories.The six categories followed, and thematic analysis followed. Extracted information was stored in an Excel spreadsheet.Source: 1) Data extraction · p. 6
  • The thematic workflow developed initial codes, grouped them into themes, reviewed and refined themes, then defined concise names for challenges, practices, and critical factors.The researchers rechecked papers when needed and merged or dropped themes when evidence was insufficient.Source: 1) Data extraction · pp. 6–7

IV. RESULTS

  • Results synthesize information collected directly from reviewed papers with minimal interpretation; interpretation and reflection are reserved for the discussion.The results are presented as answers to the research questions.Source: §IV · p. 7

A. Demographic attributes

  • Of 69 papers published through 1 June 2016, 39 (56.5%) appeared in the last three years, and conferences accounted for 48 (69.5%), versus 14 journal papers (20.2%).The review found a steady upward trend over the last decade and few relevant papers before 2010.Source: A. Demographic attributes · p. 7, Figure 3 · p. 8
  • The papers appeared in 47 venues: IEEE Software published 7 (10.1%) and XP 6 (8.6%), while 40 venues each published papers totaling 40 of 69 (57.9%).The review notes that some venues were outside software engineering, indicating research across areas that require software development.Source: A. Demographic attributes · p. 7
  • Industry studies accounted for 64 of 69 papers (92.7%), and qualitative and quantitative analyses each appeared in 26 papers (37.6%).Six papers were conducted in academic settings, with [S40] counted in both settings; 14 papers (20.2%) used both analysis types, and three studies lacked specified analysis methods. Industry studies included real-world systems or industry settings.Source: A. Demographic attributes · pp. 7–8, Table 5 · p. 8
  • Evaluation and validation research comprised 49 of 69 papers (70.9%); 15 (21.7%) reported practitioner experience, and five proposed solutions.Evaluation and validation represented 25 (36.2%) and 24 (34.7%) papers, respectively; 22 of 25 validation papers used quantitative methods, while most evaluation papers used qualitative approaches. No papers were classified as philosophical or opinion papers because the review included empirical studies.Source: A. Demographic attributes · p. 7, Table 5 · p. 8

4) Application domains and project types

The reviewed studies covered 13 identified application domains, but domain and project-type reporting was often missing. Software/web development frameworks received the most attention, while greenfield and maintenance projects were reported at similar frequencies.

4) Application domains and project types

  • Of the studies reporting application domains, the review classified them into 13 domains, while 38 studies were categorized as unclear.A study could be assigned to multiple domains, and studies using multiple systems as case studies were counted once per system in Table 6.Source: 4) Application domains and project types · p. 8
  • Software/web development frameworks were the most common application domain (13 cases), followed by utility software and data management software.The prose identifies utility software as second-most common; data management software had 8 cases.Source: 4) Application domains and project types · p. 9
  • Greenfield projects were reported in 17 papers and maintenance projects in 16, while 36 papers did not report project type.Source: 4) Application domains and project types · p. 9

2) Increase visibility and awareness on build and test results in CI … C. RQ2. Which tools have been employed to design and implement deployment pipeline?

The review describes tools and approaches for making CI results easier to interpret, handling pipeline security, scalability and reliability, and assembling deployment toolchains. It reports that only 25 of 69 studies discussed tool integration, and found no standard pipeline or study implementing every identified stage.

2) Increase visibility and awareness on build and test results in CI

  • CI produces rapidly growing build and test information; SQA-Mashup visualizes it in dynamic and chronological views, while SQA-Profile tailors data and CIViT helps teams understand testing status and avoid duplicate effort.SQA-Mashup's views can take time to interpret and require professionals such as testers.Source: 2) Increase visibility and awareness… · p. 10
  • CIT frameworks partly automate test generation or support testing with unavailable SOA components, while NAFT automates repetitive testing; WECODE detects uncommitted conflicts and incremental integration reduces the impact of broken builds.Incremental integration reuses earlier component build results, allowing integration to continue despite build failures.Source: 2) Increase visibility and awareness… · p. 10

5) Address security and scalability issues in deployment pipeline

  • Security approaches isolate multitenant build jobs with virtualization and snapshots or integrate security patterns; Enterprise Continuous Integration splits large projects into modules with binary dependencies to address scalability.The security patterns use connect, disconnect, create, and delete primitives; each ECI module has its own CI while retaining single-project CI feedback.Source: 5) Address security and scalability… · p. 10

6) Improve dependability and reliability of deployment process

  • Cloud deployment reliability can be affected by infrastructure APIs and deployment tools; rolling-upgrade error handling and Process Oriented Dependability address reliability and sporadic-operation diagnosis, respectively.POD models sporadic operations as processes using collected metrics and logs.Source: 6) Improve dependability and… · pp. 10, 12
  • CI techniques select tests by code churn or prioritize suites to reduce testing costs or fault-detection delays; Morpheus filters feedback to changed code, while CiCUTS enables early integration testing and performance checks.CRTS targets pre-submit test costs, CTSP post-submit fault-detection delays, and CiCUTS uses emulation before components are complete.Source: 6) Improve dependability and… · p. 12
  • Rondo supports continuous deployment in dynamic environments by enabling deployment reproduction across sites, customization, and custom rollback.Source: 6) Improve dependability and… · p. 12

C. RQ2. Which tools have been employed to design and implement deployment pipeline?

  • Deployment pipelines use explicit stages and automation, though manual tasks may remain; no single standard exists, and only 25 of 69 studies (36.2%) discussed tool integration.The review found no primary study implementing a pipeline with all identified stages.Source: C. RQ2. Which tools have been employed… · p. 12
  • Subversion and Git/GitHub each appeared in six papers, Jenkins was the most reported CI server, and only four papers integrated testing tools; two integrated configuration-management tools.Reported pipeline tools also included SonarQube, JUnit, NUnit, Athena, TestLink, CUTS, HockeyApp, deoloyr, and Web Deploy.Source: C. RQ2. Which tools have been employed… · pp. 12–13

D. RQ3. What challenges have been reported for adopting continuous practices?

The review identified 20 challenges to adopting continuous integration, delivery, and deployment. They span team practices, investment and tooling, resistance to change, and organizational and technical constraints.

D. RQ3. What challenges have been reported for adopting continuous practices?

  • Insufficient team awareness and transparency, alongside weak coordination across developers, testers, and operations, can disrupt adoption and increase merge conflicts.The review emphasizes making project status, errors, feature quality, and completion timing visible to the team.Source: D. RQ3. What challenges have been… · pp. 13–14

b) Lack of Investment

  • Adoption is constrained by infrastructure and training costs, skill gaps, increased team workload, and tools that lack needed automation, reliability, security, or fit.Reported investments include resource upgrades, compute and bandwidth, deployment-pipeline maintenance, and team training; frequent releases can also increase pressure on operations, developers, and release teams.Source: b) Lack of Investment · p. 14

c) Change Resistance

  • Resistance to changed work habits and scepticism about continuous practices’ benefits can impede adoption, especially amid distrust, quality concerns, and added pressure.The review describes changing team mindsets as time-consuming and reports concerns about concurrent releases and confidence in application quality.Source: c) Change Resistance · p. 14
  • The challenge classification identifies CI testing gaps including incomplete automation, unstable tests, low coverage, poor test data, long runtimes, and test dependencies.The table also lists lack of test-driven development as a challenge.Source: c) Change Resistance · p. 15
  • For CI and CDE, the classification also reports component and architectural dependencies, including incompatible components, highly coupled designs, and frequent database-schema changes.It also identifies third-party components and difficulty finding autonomous requirements for frequent integrations.Source: c) Change Resistance · p. 15

d) Organizational Processes, Structure and Policies

  • Established organizational policies and cultures can obstruct continuous practices, while distributed teams may lack shared perceptions and visibility across development sites.The review notes that long-standing release cycles and practices can make organizational change difficult, and that separate integration environments may delay detection of integration issues.Source: d) Organizational Processes, Structure… · p. 16

2) Challenges for adopting CI practice

Testing difficulties and code-integration conflicts are major barriers to adopting CI. Incomplete test automation, poor test quality, and integration conflicts can slow feedback and impede reliable continuous deployment.

a) Testing

  • Incomplete test automation complicates CI because of inadequate infrastructure, laborious automation, hardware–software dependencies, and complex code–test dependencies.GUI testing tools can partly support rapid release, but reliability concerns still require QA members to manually check systems during automated testing [S36].Source: a) Testing · p. 16
  • Unreliable, numerous, low-coverage, or slow tests can impede deployment and erode confidence in continuous deployment [S4, S5, S6, S41, S45, S50, S62, S56].In large codebases, tests grow in number and run slowly, delaying developer feedback and undermining CI [S50]; user-interface tests can also be difficult to stabilize [S62].Source: a) Testing · p. 16

b) Merging Conflicts

  • Code-integration conflicts bottleneck CI, arising from third-party components, incompatible dependencies, limited knowledge of changed components, and highly coupled design [S4, S6, S21, S31, S41, S45].Teams may need extra effort to rewrite solutions when dependent components are incompatible or teams lack knowledge of changed components [S31].Source: b) Merging Conflicts · p. 16

3) Challenges for adopting CDE practice · 4) Challenges for adopting CD practice · 2) Practices for implementing CI

The review identifies architectural, team, customer, and domain constraints that hinder continuous delivery and deployment, alongside practices for improving CI and CDE through testing, planning, team structures, and shared release responsibility.

a) Lack of Suitable Architecture; b) Team Dependencies

  • Highly coupled architectures and cross-team dependencies hinder independent development and deployment in CDE and CD [S57, S60]; frequent database-schema changes challenge delivery, while one case linked unstreamlined schema changes to severe deployment-process bottlenecks [S6, S57, S58, S62].Source: a) Lack of Suitable Architecture · p. 17, b) Team Dependencies · p. 17

a) Customer Challenges

  • Customer environments, hardware and legacy-system dependencies, and customer preferences can prevent fully automated provisioning, testing, or continuous releases [S4, S6, S10, S29, S43, S56, S60, S62].Different configurations, limited access to production-like environments, compatibility issues, update concerns, and customer policies such as banking restrictions are cited barriers [S4, S10, S43, S56, S57].Source: a) Customer Challenges · p. 17

b) Domain Constrains

  • Domain constraints affect deployment frequency and method; safety-critical, embedded, and pervasive systems may not support the continuous deployment feasible for web applications [S4, S24, S56, S65].A case company successfully adopted CI but could not fully apply CD to safety-critical systems [S24].Source: b) Domain Constrains · p. 17

a) Improve Team Awareness and Communication; b) Investment

  • Reported enablers include team communication and knowledge sharing, planned and documented engineering processes, positive team mindset, and training or coaching [S6, S28, S31, S36, S37, S44, S47, S49].Examples include change logs, release-path rules, CI events, boot camps, and assigning new developers to release engineering teams [S5, S48, S57, S58].Source: a) Improve Team Awareness and… · p. 18, b) Investment · p. 18

c) Clarifying Work Structures

  • Clarifying work structures through defined roles, rotating responsibilities, dedicated or pilot teams, and explicit rules can support CI/CD adoption and release work [S1, S5, S26, S29, S30, S37, S39, S45, S46, S57, S58].Examples include build sheriffs, release managers, teams maintaining deployment pipelines, and rules requiring errors to be fixed or reverted promptly [S29, S39, S57, S58].Source: c) Clarifying Work Structures · pp. 18–19

2) Practices for implementing CI; b) Branching Strategies; c) Decompose Development into Smaller Units

a) Flexible and Modular Architecture

  • Architectures supporting CDE should let software features be developed and deployed independently; loosely coupled designs minimize change impact [S7, S12, S28, S30, S45, S51, S57], and component interfaces should be clearly defined [S5, S30].In three projects, architectural tactics included removing web services and collapsing the middle tier; one organization rearchitected its product to better adopt CI and CDE [S7, S45].Source: a) Flexible and Modular Architecture · p. 19

b) Engage all people in deployment process

  • Continuous delivery and deployment require developers and testers to take responsibility for software in production, including fixing post-release problems and participating in release periods [S6, S9, S43, S44, S48, S57, S58].Facebook required engineers who committed code to be on call during releases [S48].Source: b) Engage all people in deployment… · p. 21

4) Practices for implementing CD

The review identifies partial releases and customer involvement as practices for reducing release risk and enabling continuous deployment (CD). These practices include limiting or hiding features from users, adapting customer engagement and testing, and using product-owner-led processes.

a) Partial Release

  • Canary releases expose new versions incrementally to a small user subset, while dark deployments hide new functionality and rollback can restore a stable state.Facebook and OANDA released to small subsets of users; dark deployment aims to detect problems before end-users are affected. The review also identifies hiding or disabling problematic functionality as a risk-reduction practice.Source: a) Partial Release · p. 21
  • Small batches hid OnDemand dark-feature development, backward-compatible microservices addressed deployment delays, and an “empty release” introduced novice developers to continuous workflows.With small batches, the feature switch is turned on once the full feature is developed; an empty release initially runs a simple application such as “hello world.”Source: a) Partial Release · p. 21

b) Customer Involvement

  • Lead or pilot customers willing to receive updates, alongside renewed customer engagement models, were proposed to support transition to continuous deployment.Some studies defined lead customers as eager to explore continuous deployment; another recommended applying CDE or CD to companies willing to receive updates continuously.Source: a) Partial Release · p. 21
  • Continuous SCRUM used triage meetings and product-owner review to approve requirement changes, prioritize the backlog, and support sustainable weekly releases.The product owner ran the triage meeting and determined the triage committee.Source: a) Partial Release · p. 22
  • Customer testing was reported as helpful for resource-limited companies practicing CD and effective at finding lower-impact functional defects.The review identifies customer involvement in testing as a practice for adopting CDE and CD.Source: a) Partial Release · p. 22

V. DISCUSSION … D. Architecting for deployability

The discussion maps continuous-practice challenges to practices, identifies critical success factors and contextual gaps, and highlights architecting for deployability as an area needing further research.

A. Mapping of Challenges to Practices

  • The challenge-practice mapping shows that challenges and practices can support or exacerbate one another, with distributed teams complicating coordination and customer involvement supporting partial release.The mapping is intended to help readers identify which practices relate to particular challenges.Source: A. Mapping of Challenges to Practices · p. 22, Figure 5 · p. 23

B. Critical factors for continuous practices success

  • Testing (39.1%), team awareness and transparency (34.7%), good design principles (30.4%), and customer (24.6%) were the four most frequently identified critical success factors.Testing problems—including long-running or manual tests and frequent failures—undermined expected benefits, while only seven reviewed papers proposed test-automation approaches, tools, or practices.Source: B. Critical factors for continuous… · p. 22
  • Team awareness and transparency help teams detect conflicts before delivery and collaborate, while customer satisfaction—particularly for CD—and a highly skilled, motivated team also matter.Customer concerns include unavailable environments, extra computing resources, incompatibilities, and buggy releases; successful continuous automation also requires technical and soft skills, shared responsibilities, and ongoing learning.Source: B. Critical factors for continuous… · p. 22
  • Application domain (20.2%) and appropriate infrastructure (20.2%) are critical factors, with some domains limiting CD and missing or poor infrastructure contributing to implementation failure.Infrastructure includes tools, networks, technologies, and physical resources; CDE and CD require additional resources and automation support, including a deployment pipeline, with budget implications.Source: B. Critical factors for continuous… · p. 24

C. Contextual factor

  • Although 57.9% of reviewed papers came from industry settings, many omitted application and organizational context, limiting analysis and motivating more complete contextual reporting.The review included evaluation research (36.2%), validation research (34.7%), and practitioner experience reports (21.7%); contextual reporting can improve the credibility and interpretability of findings.Source: C. Contextual factor · p. 24

D. Architecting for deployability

  • Modular architecture, loosely coupled components, and clear interfaces support continuous practices, while heterogeneous environments heighten the challenge of independently developing, integrating, testing, and deploying applications.The paper identifies architecting as important and defines deployability as how reliably and easily an application, component, or service can be deployed to heterogeneous production environments.Source: D. Architecting for deployability · p. 24
  • Because only one reviewed paper explicitly examined deployability scenarios in upfront design, the authors call for more research on architecture decisions and deployability measurement, trade-offs, patterns, tactics, and styles.The paper notes that most deployment-related decisions in that study were architectural [S7].Source: D. Architecting for deployability · p. 24

E. Engineering deployment pipeline · VI. THREATS TO VALIDITY · VII. CONCLUSIONS AND IMPLICATIONS

The review synthesizes evidence from 69 papers on continuous practices, identifying implementation approaches, critical success factors, research gaps, and implications for researchers and practitioners. It finds that deployment-pipeline engineering and reporting remain underdeveloped despite substantial industrial evidence.

E. Engineering deployment pipeline

  • Only 36.2% of studies reported pipeline tool selection and integration, despite pipelines having no standard design and depending on organizational and project context.The review says enterprises need time and resources to select and integrate tools tailored to factors including team skills, organizational structure and budget, customer environments, and project domain.Source: E. Engineering deployment pipeline · pp. 24–25
  • Most reported tools target CI or stand alone, while increasing system size raises build and test demands and pipeline stages and integrations can be compromised by attackers.Of 30 approaches and tools, 24 target CI and 18 are stand-alone; the review calls for systematic pipeline architecture and tool selection.Source: E. Engineering deployment pipeline · p. 25

VI. THREATS TO VALIDITY

  • The review mitigated search and selection risks through six digital libraries, iterative and prior-review-informed search strings, snowballing, predefined criteria, author discussion, and cross-checking.Scopus was used to increase search coverage; inclusion disagreements were resolved through discussion, and other authors cross-checked a random selection of papers.Source: §VI · p. 25
  • Data extraction used a standardized form and author verification, but incomplete source information sometimes required subjective interpretation, potentially affecting results.Where possible, researchers consulted study, tool, and approach websites or training movies; other data items were interpreted only when explicitly reported.Source: §VI · p. 25

VII. CONCLUSIONS AND IMPLICATIONS

  • The 69-paper review found growing interest, with 39 papers (56.5%) published in the last three years; most were evaluation or validation studies conducted in industry.Evaluation papers numbered 25 (36.2%), validation papers 24 (34.7%), and 64 papers (92.7%) came from industrial settings.Source: §VII · pp. 25–26
  • The review identifies seven success factors, led by testing effort and time, and finds applications across domains and both greenfield and maintenance projects.Other factors, in importance order, are team awareness and transparency, good design principles, customer, skilled and motivated team, application domain, and appropriate infrastructure; software/web development frameworks and utility software received most attention.Source: §VII · p. 26
  • Thirty approaches support six continuous-practice needs, but only 12 were integrated and evaluated in deployment pipelines; two papers investigated pipeline security.The approaches address CI build and test time, result visibility, automated testing, fault detection, pipeline security and scalability, and deployment dependability.Source: §VII · p. 26
  • Researchers should report contextual details, study pipeline security, evaluate tools in real pipelines, and develop guidance for designing for deployability.The review found only two security studies and noted that just 12 approaches and tools had been integrated and evaluated in deployment pipelines.Source: §VII · p. 26

Appendix A. Selected studies

This appendix section lists selected studies, providing their titles, authors, and publication venues rather than reporting findings. It covers studies S53–S69 on continuous integration testing, DevOps and continuous deployment, infrastructure, security, and related practices.

Appendix A. Selected studies

  • Table 11 identifies the material as selected studies in the review.The entries include study titles, author names, and publication venues.Source: Table 11 · p. 27, Appendix A · p. 27
  • Other listed titles concern continuous deployment transitions, release engineering, and rapid releases, including technical and social challenges and a B2B case study.Examples include S6, S9, S29, and S43.Source: Appendix A · pp. 27–29
  • Listed CI-focused titles address automated integration flows, test automation frameworks, and continuous integration testing of service-oriented architecture solutions.Examples include S11, S38, and S40.Source: Appendix A · pp. 27–28
  • The bibliography lists work on robotic monitoring of agile software development teams, distributed continuous-integration scheduling, rule-based activity profiles for CI environments, and security of public continuous-integration services.Source: Appendix A · p. 28
  • The listed CI studies address service-oriented architecture simulation, combining execution-modeling tools with CI, and regression-test improvement.Their titles are “Surrogate: A simulation apparatus for continuous integration testing in service oriented architecture,” “CiCUTS,” and “Techniques for improving regression testing in continuous integration development environments.”Source: Appendix A · p. 29
  • The DevOps and deployment entries include embedded-systems adoption challenges, deployment at Facebook and OANDA, DevOps practices, and the Rondo tool suite.These are listed as studies S56–S59.Source: Appendix A · p. 29
  • Other entries cover DevOps adoption impediments, social testing for SME continuous delivery, automated pipeline testing, productized infrastructure, and code-churn-based test selection.These are listed as studies S60–S64.Source: Appendix A · pp. 29–30
  • The remaining entries address pervasive-system deployment requirements, secure-system patterns, fast automated-test feedback, cloud-operation error diagnosis, and feature toggles.These are listed as studies S65–S69.Source: Appendix A · p. 30

Appendix B. Data Extraction Form

Appendix B identifies the data extraction form used to relate items extracted from each study to the research questions.

Appendix B. Data Extraction Form

  • Table 12 presents the data items extracted from each study and their related research questions.Source: Table 12 · p. 30
Loading 1703.07019v1…