Source-linked AI summary
Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices
Mojtaba Shahin, Muhammad Ali Babar, Liming Zhu
TL;DR
Organizations increasingly need reliable, frequent software releases, but evidence on how to adopt and implement continuous integration, delivery, and deployment remained fragmented across approaches, tools, challenges, and practices. This study conducts an SLR with thematic analysis of 69 papers published from 2004 to 1 June 2016. It identifies 30 approaches and associated tools, plus seven critical factors for successful implementation, while acknowledging threats related to evidence retrieval and study selection.
Problem
The literature lacked an integrated, systematic synthesis of approaches, tools, challenges, and practices across continuous integration, delivery, and deployment.
Method
The study used a systematic literature review of 69 papers selected with predefined criteria and synthesized extracted data through thematic analysis.
Results
30 approaches and associated tools were identified, covering build and test efficiency, visibility, automated testing, fault detection, deployment-pipeline security and scalability, and deployment dependability and reliability.
Takeaways & Limitations
Seven critical factors should be considered when introducing continuous practices: testing effort and time, team awareness and transparency, design principles, customer, team skills and motivation, application domain, and infrastructure.
Takeaways & Limitations
The findings may be affected by threats including missing relevant papers and subjective judgment during study selection.
Abstract
from arXiv · showhide
Context: Continuous practices, i.e., continuous integration, delivery, and deployment, are the software development industry practices that enable organizations to frequently and reliably release new features and products. With the increasing interest in and literature on continuous practices, it is important to systematically review and synthesize the approaches, tools, challenges, and practices reported for adopting and implementing continuous practices. Objective: This research aimed at systematically reviewing the state of the art of continuous practices to classify approaches and tools, identify challenges and practices in this regard, and identify the gaps for future research. Method: We used systematic literature review (SLR) method for reviewing the peer-reviewed papers on continuous practices published between 2004 and 1st June 2016. We applied thematic analysis method for analysing the data extracted from reviewing 69 papers selected using predefined criteria. Results: We have identified thirty approaches and associated tools, which facilitate the implementation of continuous practices in the following ways: (1) "reducing build and test time in continuous integration (CI)"; (2) "increasing visibility and awareness on build and test results in CI"; (3) "supporting (semi-) automated continuous testing"; (4) "detecting violations, flaws and faults in CI"; (5) "addressing security and scalability issues in deployment pipeline", and (6) "improving dependability and reliability of deployment process". We have also determined a list of critical factors such as "testing (effort and time)", "team awareness and transparency", "good design principles", "customer", "highly skilled and motivated team", "application domain", and "appropriate infrastructure" that should be carefully considered when introducing continuous practices in a given organization.
I. INTRODUCTION
Continuous integration, delivery, and deployment are intertwined practices intended to accelerate software delivery while maintaining quality, yet prior literature had not been systematically synthesized across their approaches, tools, challenges, and practices. This SLR addresses that gap by integrating evidence from existing reviews and empirical studies.
- Background: Continuous practices aim to accelerate software development and delivery without compromising quality.CI integrates work frequently, while CDE and CD support quickly and reliably delivering software.
- Background: The practices are highly correlated and intertwined, making their meanings dependent on how organizations interpret and employ them.CI is described as a first step toward CDE, while CDE is necessary to support automatic and continuous deployment to production.
- Existing literature reviews: Prior secondary studies examined individual aspects of continuous software engineering, but no dedicated review had integrated approaches, tools, challenges, and practices across CI, CDE, and CD.Existing reviews covered CI, CDE, CD, or rapid release separately and used differing study sets and objectives.
- Research contribution: This SLR systematically reviewed 69 relevant papers to classify approaches, associated tools, challenges, and practices and identify critical factors for implementation.Its stated contributions include an accessible classification and guidance on factors to consider across software development and customer organizations.
- Background: Continuous delivery keeps applications production-ready after automated tests and quality checks, whereas continuous deployment automatically releases changes to production.CDE may retain a manual final deployment step; CD does not.
- Research contribution: The review treats CI, CDE, and CD together because their integrated analysis can clarify relationships among them and inform movement from one practice to another.The review emphasizes simultaneous coverage of technical, contextual, practice-related, and challenge dimensions.
C. Motivation for this SLR on continuous practices
The SLR addresses the need to synthesize intertwined continuous integration, delivery, and deployment practices through a systematic, evidence-based review. It uses predefined search and selection procedures to identify empirical evidence on approaches, tools, challenges, and practices.
- Motivation: CI, CDE, and CD are highly correlated and intertwined, making their meanings difficult to distinguish across organizational contexts.The review therefore investigates them together to understand their relationships and supporting approaches, tools, and practices.
- Review approach: The review uses an SLR process to identify, evaluate, and interpret evidence relevant to continuous practices.The protocol includes research questions, search strategy, and inclusion and exclusion criteria.
- Search strategy: Studies were retrieved through automatic searches across six digital libraries and complemented with snowballing of references.The search terms were developed through pilot searches, existing SLRs, and synonyms related to continuous software practices.
- Selection criteria: The review included peer-reviewed empirical studies and excluded non-English papers, short papers, duplicates, and non-empirical publication types.The criteria also excluded editorials, position papers, keynotes, reviews, tutorials, panel discussions, and papers lacking reported approaches, tools, or practices.
- Study selection: The selection process filtered thousands of retrieved papers through title and keyword screening, abstract and conclusion review, snowballing, and full-text quality assessment.The process began with 14,723 retrieved papers, was reduced to 3,942 potential papers after the SpringerLink limit, and selected 449 after initial screening.
E. Data extraction and synthesis
The review synthesized 69 studies, identifying publication and research-design patterns alongside approaches and tools supporting continuous practices. These approaches target faster CI feedback and testing, fault detection, deployment security, and process reliability.
- Publication and study characteristics: 69 primary papers published from 2004 to before 1 June 2016 were reviewed, with 11 of 14 journal papers appearing in 2015–2016.The studies were published across 47 venues; IEEE Software and XP were the leading venues, with 7 and 6 papers respectively.
- Publication and study characteristics: 49 of 69 papers (70.9%) reported evaluation or validation research, comprising 25 evaluation studies and 24 validation studies.Qualitative and quantitative approaches each appeared in 26 papers (37.6%), while 14 papers (20.2%) used both.
- Approaches and tools: CI acceleration approaches reduce build or test time through dependency isolation, test selection and prioritization, and hotspot identification.VMVM and VMVMVM isolate test dependencies; CRTS and CTSP use execution history for pre-submit testing; build-dependency and change-history analysis identifies C/C++ header files to optimize first.
- Approaches and tools: Other approaches increase CI feedback visibility, automate continuous testing, detect integration failures, secure deployment pipelines, and improve deployment reliability.Reported examples include deployment-pipeline test automation, WECODE, incremental integration with backtracking, virtualization-based build isolation, and analysis of cloud APIs and deployment tools.
D. RQ3. What challenges have been reported for adopting continuous practices?
The review identifies organizational, technical, coordination, customer, and infrastructure challenges that constrain adoption and implementation of continuous practices. Key difficulties include limited automation, dependencies, distributed-team coordination, cost, customer diversity, and resistance to frequent change.
- Organizational and coordination challenges: Insufficient awareness and transparency can disrupt transitions, while successful implementation requires stronger collaboration and coordination among team members.Distributed teams particularly face reduced visibility and inconsistent perceptions across development sites.
- Resource and infrastructure challenges: High costs arise from resource upgrades, computation, bandwidth, memory, automated acceptance testing, and infrastructure maintenance.These demands affect both software development and customer organizations.
- Technical challenges: Existing tools may provide inefficient code review, weak test automation, or vulnerable build and deployment capabilities.External resources and cloud-based deployment tools can increase errors and delays, while frequent tool changes create learning barriers.
- Testing challenges: Incomplete test automation remains difficult because of poor infrastructure, laborious manual-test automation, hardware–software dependencies, and limited test-driven development.Long-running, manual, or frequently failing tests can impede continuous practices.
- Dependency and deployment challenges: Dependencies among components, interfaces, teams, hardware, legacy applications, and customer environments can block independent delivery and deployment.Diverse customer sites and manual configuration make transferring software through continuous delivery difficult.
- Adoption and customer challenges: Customer preferences and policies may resist continuous releases because of frequent notifications, plug-in incompatibility, and increased bugs.Organizational culture, resistance to change, scepticism, and distrust can further impede adoption.
V. DISCUSSION
The paper positions continuous software engineering as an emerging research and practice area requiring systematic synthesis of reported approaches, tools, challenges, and practices. Its review aims to organize this body of knowledge for future methodological and practical use.
- Continuous practices are increasingly reported across software engineering, creating a need to systematically review and document their approaches, tools, challenges, and practices.
A. Mapping of Challenges to Practices
The review maps identified challenges onto reported practices to show which interventions address particular obstacles. The mapping also indicates that challenges and practices may depend on or exacerbate one another.
- Figure 5 maps challenges to practices, helping readers identify which reported practices relate to particular adoption and implementation obstacles.For example, flexible and modular architecture is expected to decrease dependencies in design and code.
B. Critical factors for continuous practices success
The review identifies 20 challenges, 13 practices, and 30 approaches with associated tools, then derives seven critical factors for successful continuous practices. Testing is the most frequently mentioned factor, while application domain and infrastructure define important boundaries for deployment.
- Review synthesis: 20 challenges, 13 practices, and 30 approaches with associated tools were identified across continuous integration, delivery, and deployment.The review found no one-to-one relationship between challenges and proposed practices, approaches, or tools.
- Critical-factor identification: Seven critical factors were derived by aggregating challenges, approaches, and practices associated with recurring adoption conditions.A factor was considered critical when cited in at least 20% of reviewed studies.
- Critical factors: Testing was the most frequently mentioned critical factor, cited in 27 papers (39.1%), followed by team awareness and transparency in 24 papers (34.7%).Good design principles were cited in 21 papers (30.4%), and customer-related factors in 17 papers (24.6%).
- Critical factors: Team awareness and transparency support visibility into project status, errors, build responsibility, and feature completion across the software development organization.
- Critical factors: Customer considerations are especially important for continuous deployment because customer organizations may not accept continuous releases or provide accessible environments.
- Scope and boundary conditions: Application domain and appropriate infrastructure constrain adoption, particularly when embedded-system characteristics make continuous deployment nearly impossible.Appropriate infrastructure includes tools, networks, technologies, build servers, test-automation servers, and physical resources.
C. Contextual factor
The reviewed papers often lacked organizational context, especially organization size and application domain, limiting analysis and interpretation. The authors recommend reporting such context to improve credibility and reader understanding.
- C. Contextual factor: Organizational size and application domain were generally underreported in the reviewed papers.The authors therefore dropped these attributes from data analysis and interpretation.
- C. Contextual factor: More complete contextual reporting could improve the quality and credibility of reported approaches, tools, and practices.It could also help readers understand the reported research.
D. Architecting for deployability
Sound architecture design is identified as an important factor in successfully adopting continuous integration, delivery, and deployment. However, deployability-oriented architecture decisions remain insufficiently studied.
- D. Architecting for deployability: Sound architecture design significantly influences the success of continuous integration, delivery, and deployment.Modular architecture, loosely coupled components, and clearly defined interfaces are reported as contributing factors, particularly for delivery and deployment.
- D. Architecting for deployability: Although practitioners discuss designing for testability and deployability, only one reviewed paper explicitly examined deployability scenarios during upfront architectural decisions.That study concluded that most deployment-related decisions were architectural.
- D. Architecting for deployability: Future research should investigate how continuous delivery and deployment adoption influences architectural processes and outcomes.Suggested questions concern early deployability measurement, quality-attribute trade-offs, and deployability-friendly architectural patterns, tactics, and styles.
E. Engineering deployment pipeline
Deployment pipelines enable continuous practices, but the literature provides limited detail on how enterprises design and implement them. Pipeline design must account for organizational, technical, customer, and domain-specific factors.
- E. Engineering deployment pipeline: Deployment pipelines are described as key enablers for enterprises adopting continuous practices.However, many industrially reported studies provide insufficient detail about enterprise pipeline design and implementation.
- E. Engineering deployment pipeline: 36.2% of included studies presented the tools used to implement deployment pipelines.The review notes that no standard or single pipeline exists.
- E. Engineering deployment pipeline: Pipeline design may be influenced by team skills, experience and structure, organizational structure and budget, customer environments, and project domain.Organizations consequently need to select and integrate varied open-source and commercial tools into tailored pipelines.
- E. Engineering deployment pipeline: High-performance infrastructure and appropriate tools are necessary but insufficient as software systems, builds, and test cases grow rapidly.The review calls for innovative approaches and tools to address this growth.
VI. THREATS TO VALIDITY
The review identifies threats involving literature search, study selection, data extraction, and data synthesis, while reporting broad findings about the evidence base and continuous-practice support.
- Search strategy: Researchers may miss relevant papers despite using six digital libraries, iterative search-string refinement, existing-review consultation, and snowballing.The search strategy was designed to mitigate, rather than eliminate, the risk of incomplete retrieval.
- Study selection: Study selection may be affected by subjective inclusion and exclusion judgments, although protocol adherence, author discussions, validation, and cross-checking were used to reduce bias.Disagreements were resolved through discussion, and additional authors cross-checked a random subset of selected papers.
- Data synthesis: Data synthesis required subjective interpretation when extracted papers lacked sufficient information, which might have affected extraction outcomes.The review combined quantitative and qualitative analysis and used measures to reduce interpretive bias.
- Evidence base: The review selected 69 papers using predefined criteria and found that most studies were industrial evaluation or validation research.Evaluation papers numbered 25 of 69 (36.2%), validation papers 24 of 69 (34.7%), and industrial studies 64 of 69 (92.7%).
- Findings: Thirty approaches and associated tools addressed six implementation goals, while seven critical factors were identified for successful continuous practices.The goals include reducing CI build and test time, improving visibility, automating testing, detecting faults, addressing deployment-pipeline security and scalability, and improving deployment reliability.
- Research gaps: The review reports gaps in contextual reporting, deployment-pipeline security research, real-pipeline evaluation, and guidance for architecting for deployability.Only two papers investigated deployment-pipeline security, and 12 of 30 approaches and tools were integrated and evaluated in deployment pipelines.
Appendix A. Selected studies
Appendix A lists the selected studies included in the review, covering continuous integration, delivery, deployment, testing, release engineering, architecture, and related practices.
- Selected studies: The appendix identifies the reviewed-study set as Table 11, followed by studies spanning continuous delivery, integration, and deployment.Listed examples include multi-customer project courses, SQA-Mashup, Vroom, and studies of continuous deployment.
- Delivery and deployment: Other selected studies examine continuous delivery and deployment through release practices, cloud environments, transition barriers, deployment architecture, and rapid releases.The list includes work on reliable high-frequency cloud releases, release engineering, transition toward continuous delivery, deployability design decisions, and rapid-release analytics.
- Continuous integration: Several selected studies address continuous-integration automation, build performance, monitoring, testing, scheduling, and activity profiling.Examples include automated integration flows, faster Java builds, BuildBot, distributed CI scheduling, SQA-Profiles, and test automation frameworks.
- Related concerns: The selected studies also cover security, quality assurance, architecture, dependencies, interaction testing, software measurement, and domain-specific applications.Examples include public CI security, architecture enforcement, technical dependencies, product-line interaction testing, software-product measurement, and computational chemistry.
Appendix B. Data Extraction Form
Appendix B defines the data items extracted from each selected study and maps them to demographic, research-question, and discussion categories.
- Appendix structure: Table 12 presents the complete set of extracted data items and their associated research questions.The appendix labels the table as data items extracted from each study and related research questions.
- Demographic data: The form records bibliographic and publication metadata, including authors, year, title, publication type, venue, and data-analysis type.These items support demographic characterization of the reviewed studies.
- Demographic data: Research type classifies studies as validation, evaluation, solution proposal, philosophical, opinion, or experience research.The classification is explicitly recorded as a demographic data item.
- Study context: The form captures study context, project type, and application domain to characterize where continuous-practice evidence was produced.Context distinguishes industry from non-industry cases, project type includes greenfield or maintenance, and domain covers applications used for reporting or validation.
- Review findings: It extracts techniques and tools, reported challenges, implementation practices, and critical factors linked to the review questions and discussion.Techniques and tools address RQ1 and RQ2; challenges address RQ3; practices address RQ4; critical factors support the discussion.