PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of DevOps Concepts and Challenges

Leonardo Leite, Carla Rocha, Fabio Kon, Dejan Milojicic, Paulo Meirelles

arXiv:1909.05409v4cs.SE

TL;DR

DevOps lacks a widely accepted definition, and this survey addresses that gap by synthesizing literature across engineers’, managers’, and researchers’ perspectives. It constructs conceptual maps, relates DevOps tools to concepts, and examines practical implications and challenges. The survey concludes that automation practices are relatively consolidated, whereas effective cross-department collaboration lacks consensus and dedicated tools.

  • Problem

    DevOps lacks a widely accepted definition, while prior surveys often address challenges from a single perspective and omit practical adoption implications.

  • Method

    The survey uses a protocol-based qualitative literature review and Grounded Theory coding to construct conceptual maps, classify tools, and organize implications and challenges.

  • Results

    Automation issues are relatively consolidated, but the literature lacks consensus on effectively empowering collaboration across departments and offers few tools for it.

  • Takeaways & Limitations

    The conceptual maps associate DevOps concepts with tools to support toolset selection, professional understanding, continuing education, improvement, and research-topic identification.

  • Takeaways & Limitations

    The survey could not exhaustively cover all DevOps sources and is affected by subjective and publication bias, including an absence of reported failed-adoption cases.

Abstract

from arXiv · show

DevOps is a collaborative and multidisciplinary organizational effort to automate continuous delivery of new software updates while guaranteeing their correctness and reliability. The present survey investigates and discusses DevOps challenges from the perspective of engineers, managers, and researchers. We review the literature and develop a DevOps conceptual map, correlating the DevOps automation tools with these concepts. We then discuss their practical implications for engineers, managers, and researchers. Finally, we critically explore some of the most relevant DevOps challenges reported by the literature.

1 INTRODUCTION

DevOps lacks a widely accepted definition, motivating this survey’s multidisciplinary conceptual framework for engineers, managers, and researchers. The survey reviews literature, relates concepts to tools, and examines practical implications and unresolved challenges.

  • DevOps lacks a widely accepted definition despite nearly a decade of discussion.
  • DevOps is defined as a collaborative, multidisciplinary organizational effort that automates continuous delivery while guaranteeing correctness and reliability.
  • Enterprises are widely adopting DevOps to improve their delivery processes, while the term also denotes an increasingly attractive professional role.
  • The survey addresses engineers’ needs to qualify for DevOps roles and re-architect systems, alongside managers’ needs to introduce and assess DevOps practices.
  • Its research problem is devising a conceptual framework covering DevOps tools, implications, and challenges from multiple perspectives, including practical adoption issues previous surveys did not cover.
  • The authors review academic and non-academic sources, construct conceptual maps, correlate tools with concepts, and discuss implications and literature-reported challenges.

2 DEVOPS

DevOps emerged in response to tensions between development and operations, extending agile practices toward collaboration, automated delivery, and runtime monitoring. Its practices address delays and reliability demands but also require architectural and process changes.

  • DevOps evolved from agile software development, whose early literature emphasized frequent releases but gave little attention to deployment-specific practices.
  • Separate development and operations silos create independent processes, tools, and knowledge bases, with ticket-based interfaces between departments.
  • Siloed responsibilities can produce long delays between code updates and deployment and ineffective problem-solving in which departments blame each other.
  • Conflicts, significant deployment times, and demands for frequent reliable releases led developers and operators to collaborate, giving rise to the term DevOps in 2008.
  • Continuous Delivery uses an automated pipeline in which committed versions pass compilation and automated tests before production release, while continuous deployment sends every passing version automatically.
  • DevOps initiatives also use automated runtime monitoring to improve performance, scalability, availability, and resilience, contributing to the emergence of Site Reliability Engineering.

3 STUDY DESIGN

The survey combines a protocol-based literature review with qualitative coding to construct a grounded conceptual framework of DevOps. The resulting maps organize concepts, tools, implications, and challenges for different professional perspectives.

  • 3 STUDY DESIGN: The study limits a rapidly growing literature through a search and selection protocol inspired by systematic reviews and analysis procedures based on Grounded Theory.
  • 3 STUDY DESIGN: The search covered major scientific repositories and Google Scholar using criteria concerning DevOps keywords, publication venues, language, length, and exclusion of literature reviews.
  • 3 STUDY DESIGN: The search found 198 papers; 45 irrelevant papers were discarded, leaving 153 relevant papers for analysis.
  • 3 STUDY DESIGN: At least two authors independently classified papers, resolving disagreements through individual examination and consensus, while additional guidelines determined core and relevant status.
  • 3 STUDY DESIGN: Snowballing added 14 core papers, and the study identified 50 core papers with reference codes used on the conceptual maps.
  • 3.2 Producing the conceptual framework: Grounded Theory open coding and constant comparison transformed excerpts into concepts and categories, producing maps whose nodes and links represent DevOps concepts and relationships.
  • 3.2 Producing the conceptual framework: The framework does not formulate a new theory; instead, it classifies tools, identifies practical implications by role, and discusses open challenges.
  • 3.2 Producing the conceptual framework: The maps are intended to support continuing education, reflection and continuous improvement, and identification of research topics and implications.

4 SOURCES OF KNOWLEDGE

DevOps knowledge is distributed across academic and practitioner sources, but the literature lacks dedicated venues and earlier reviews miss much of the field’s growth. This survey broadens and updates the evidence base by analyzing diverse sources and 50 core papers.

  • Sources of knowledge: The survey draws on peer-reviewed papers, books, talks, and practitioner-oriented sources because DevOps has few dedicated publication venues.Peer-reviewed papers are emphasized for academic quality, while books and talks provide accessible and up-to-date practitioner knowledge.
  • Peer-reviewed literature: No peer-reviewed journal, magazine, or event was found to be dedicated exclusively to DevOps.The International Workshop on Release Engineering and International Workshop on Quality-Aware DevOps were identified as potential publication venues.
  • Related surveys: Earlier DevOps SLRs, mostly conducted before 2015, miss much of the later publication growth and primarily examined pioneering studies.Their conclusions included low quality in initial studies, while this survey analyzed a broader and more recent body of work.
  • Related surveys: Previous reviews emphasized process and people, whereas this survey also covers delivery and runtime, including technical adoption issues such as automation, microservices, containerization, and toolset management.The survey also reports lessons concerning legacy systems, microservice adoption, delivery-pipeline automation, and autonomous toolset selection.
  • Other sources: The survey complements academic analysis with practitioner sources, including 238 English-language books identified through an Amazon.com search.Book publication roughly follows academic publication by about one year, and Continuous Delivery is identified as a seminal 2010 source for deployment-pipeline ideas.

5 FUNDAMENTAL CONCEPTS

The survey constructs a literature-grounded DevOps conceptual framework organized into process, people, delivery, and runtime categories. It connects these concepts to organizational perspectives, automation tools, and practical DevOps challenges.

  • Framework: The framework provides a structured set of DevOps concepts grounded in systematic literature analysis to support understanding of DevOps challenges.It consists of an overall conceptual map and four graph-based maps whose nodes represent concepts and arrows represent supported relationships.
  • Framework: The four categories are process, people, delivery, and runtime.Process covers business-related concepts, people covers collaboration and skills, delivery supports continuous delivery, and runtime addresses service stability and reliability.
  • Framework: Process and people align more closely with management, while delivery and runtime align more closely with engineering.Delivery is associated more with developers, whereas runtime is associated more with traditional operator responsibilities.
  • Process: Process concepts link frequent and reliable releases with business outcomes such as reduced risk and cost, regulatory compliance, product quality, and customer satisfaction.The survey attributes the connection between continuous delivery and quality or satisfaction to the short feedback cycle it provides.
  • People: People concepts center on collaboration between development and operations, breaking down silos and aligning organizational incentives.The paper identifies cultural change as raising questions about how organizations can shift away from siloed structures.
  • Delivery: Delivery concepts center on automating the deployment pipeline through versioning, automated testing, continuous integration, and configuration management.The delivery map also connects DevOps with microservices and related concerns such as backward compatibility and API versioning.
  • Delivery: Concrete tooling strategies remain debated, including Docker’s containerized approach and Chef or Puppet’s continuous configuration convergence.The survey considers delivery concepts more stable and accepted by the community than people concepts.
  • Runtime: Runtime concepts extend continuous delivery by targeting performance, availability, scalability, resilience, and reliability through infrastructure-as-code, virtualization, containers, cloud services, monitoring, and production experiments.The runtime perspective includes monitoring business or resource metrics and using failure injection as part of chaos engineering.

6 TOOLSET

The toolset discussion relates DevOps tools to collaboration, continuous delivery, reliability, organizational actors, and conceptual categories. It also highlights that tool selection and ownership remain difficult because tools are numerous, evolving, and embedded in unclear responsibilities.

  • DevOps tools are defined by goals of cross-department collaboration, continuous delivery, or software reliability.
  • The survey categorizes tools by examples, intended users, goals, and associated DevOps concepts to support organization-specific tool selection.
  • Tools for knowledge sharing: Collaboration tools such as GitLab and Rocket Chat support knowledge sharing, issue coordination, and conversation-driven automation between developers and operators.
  • Tools for knowledge sharing: The DevOps Periodic Table lists more than 100 tools but only eight for collaboration, suggesting that technical goals may eclipse people-focused concepts.
  • Tools for deployment automation: Deployment platforms such as PaaS and serverless services abstract infrastructure and suggest that specialized operations teams may not be necessary.
  • Tools for deployment automation: Containerization and configuration management differ because containers rebuild environments, while configuration management handles environment-specific settings that cannot be embedded in one image.
  • Actors: Tool ownership is unclear because cross-functional teams, shared deployment tools, and the wide variety of evolving tools blur developer and operator responsibilities.

7 IMPLICATIONS FOR ENGINEERS, MANAGERS, AND RESEARCHERS

The survey translates DevOps literature into practical implications for engineers, managers, and researchers, covering technical, organizational, cultural, and research concerns. It also reports delivery and trust outcomes while identifying adoption challenges and open research topics.

  • Implications for engineers: Engineers must adapt system architecture, collaboration, learning, incident handling, and implementation practices to support DevOps.The implications include learning evolving automation tools, coding for stability and security, and handling incidents, bugs, and failures.
  • Implications for engineers: Rollback capability is important after deployment, but complex integrations and database evolution can make rollback difficult; feature toggles and root-cause handling are alternatives.The literature identifies rollback as a recommended capability while also describing conditions that complicate it.
  • Implications for engineers: Automated testing remains difficult to implement fully, particularly for user interfaces, load testing, user experiments, and flaky-test control.Parallelizing test suites may be necessary to reduce testing time, but hardware availability and unreliable tests add complexity.
  • Implications for engineers: Deployment pipelines can require considerable infrastructure effort, ongoing artifact maintenance, and incremental implementation rather than a single-step rollout.Microservices may require multiple pipelines, while logs, configurations, build outputs, and temporary files must eventually be archived or removed.
  • Implications for managers: Managers must address adoption strategy, training, role definitions, assessment, and culture rather than treating DevOps as solely a tooling change.The survey highlights cross-functional skills, unclear DevOps-related job boundaries, cautious metric use, and management responsibility for continuous improvement.
  • Expected outcomes: Reported outcomes include increased delivery throughput, more predictable defect rates, sustained release frequency with fewer team members, and stronger contractor–client trust.These outcomes are reported in organizational experience studies involving continuous delivery and DevOps adoption.
  • Implications for researchers: Researchers are encouraged to study DevOps-related open topics, including strategies for transitioning software architecture and adopting associated practices.The survey positions these topics as opportunities for future academic investigation.
  • Open research topics: Distributed organizations continue to face interdepartmental communication challenges, while excessive communication and meetings can reduce productivity.The survey distinguishes effective communication from simply increasing communication volume.

8 UNRESOLVED CHALLENGES

The survey identifies unresolved challenges spanning architecture, organizational structure, measurement, tool selection, and continuous learning. The literature offers practical patterns, but lacks consensus on several organizational and technical decisions.

  • Architecture: Highly coupled monoliths obstruct continuous delivery, whereas independently deployable microservices support testability and deployability.Per-service test suites can reduce deployment blocking from long-running tests.
  • Architecture: Microservices introduce heterogeneity and require management patterns such as per-service pipelines, log aggregation, service registries, and correlation IDs.Additional patterns include segregating source code, configuration, and environment specifications, alongside versioned APIs and circuit breakers.
  • Architecture: Reusability can increase coupling and become a bottleneck for continuous delivery, so teams must balance reuse against component and team independence.The survey recommends discussing this trade-off rather than maximizing reusability by default.
  • Organizational structure: The literature does not establish whether organizations should preserve collaborating departments, create cross-functional teams, or form DevOps teams.DevOps teams may create another silo, although temporary teams focused on knowledge sharing are described as acceptable in some models.
  • Assessment: DevOps maturity models can identify investment areas, but industry adoption remains limited and organizations often lack reliable visibility into delivery practices.Metrics can become unreliable when used to punish teams, while inappropriate measurement may encourage behavior that diverts organizations from their goals.
  • Practitioner challenges: More than 100 evolving DevOps tools force professionals to prioritize tools according to personal and organizational demands, using conceptual maps and tool tables as guidance.A culture of autonomy and responsibility is also presented as a motivating factor for continuous learning, supported by social exchange around deployment results and trade-offs.

9 LIMITATIONS OF THIS STUDY

The survey’s literature review was broad but not exhaustive, and its paper-selection process remains vulnerable to subjective and publication bias. These constraints limit how completely the review can represent DevOps research and reported outcomes.

  • Scope: The review could not exhaustively cover all DevOps sources, excluding academic workshops and discussing only a few books.It focused on journal and conference papers and used snowballing to target primary literature.
  • Bias: Paper selection and classification may contain subjective bias despite review by at least two authors and oversight from senior researchers.The study also faces publication bias because positive results are more likely to be reported and accepted than negative results.

10 CONCLUSIONS

The survey organizes DevOps concepts, tools, and challenges across engineer, manager, and researcher perspectives. It finds stronger consolidation around delivery automation than around cross-department collaboration or organizational structure.

  • Conclusions: The survey links DevOps concepts with tools to support practitioners in choosing appropriate toolsets across researcher, manager, and engineer perspectives.The presentation also aims to clarify how DevOps affects daily activities for different professional profiles.
  • Conclusions: Technical delivery-automation issues are relatively consolidated, whereas effective cross-department collaboration lacks consensus and has few supporting tools.The survey identifies human collaboration and automation as DevOps’s two pillars.
  • Conclusions: The literature disagrees about microservice versioning, whether configuration management and containerization complement or compete, and whether maximizing reusability is preferable.Rollback practices also compete with the use of feature toggles in practitioner discussions.
  • Conclusions: No single organizational structure can be selected as best for every scenario among collaborating departments, cross-functional teams, and DevOps teams.The survey identifies real-organization studies of these structures and the DevOps role as promising future research.
Loading 1909.05409v4…