Source-linked AI summary
Where Are The Gaps? A Systematic Mapping Study of Infrastructure as Code Research
Akond Rahman, Rezvan Mahdavi-Hezaveh, Laurie Williams
TL;DR
The study addresses the need to identify research areas in the growing field of infrastructure as code. It conducts a systematic mapping study of IaC publications and identifies four studied topics, while highlighting further research on IaC defects and security flaws.
Problem
The study addresses the need to identify research areas and research problems in the growing field of infrastructure as code.
Method
The authors conduct a systematic mapping study using six scholar databases, filtering 33,887 publications to 31 IaC-related publications for qualitative topic analysis.
Results
The study identifies four IaC research topics: frameworks/tools, use of IaC, empirical studies, and testing.
Takeaways & Limitations
The findings identify research problems that could benefit from further study, including defects and security flaws in IaC.
Takeaways & Limitations
The authors acknowledge that generating topics can be subjective.
Abstract
from arXiv · showhide
Context:Infrastructure as code (IaC) is the practice to automatically configure system dependencies and to provision local and remote instances. Practitioners consider IaC as a fundamental pillar to implement DevOps practices, which helps them to rapidly deliver software and services to end-users. Information technology (IT) organizations, such as Github, Mozilla, Facebook, Google and Netflix have adopted IaC. A systematic mapping study on existing IaC research can help researchers to identify potential research areas related to IaC, for example, the areas of defects and security flaws that may occur in IaC scripts. Objective: The objective of this paper is to help researchers identify research areas related to infrastructure as code (IaC) by conducting a systematic mapping study of IaC-related research. Methodology: We conduct our research study by searching six scholar databases. We collect a set of 33,887 publications by using seven search strings. By systematically applying inclusion and exclusion criteria, we identify 31 publications related to IaC. We identify topics addressed in these publications by applying qualitative analysis. Results: We identify four topics studied in IaC-related publications: (i) framework/tool for infrastructure as code; (ii) use of infrastructure as code; (iii) empirical study related to infrastructure as code; and (iv) testing in infrastructure as code. According to our analysis, 52% of the studied 31 publications propose a framework or tool to implement the practice of IaC or extend the functionality of an existing IaC tool. Conclusion: As defects and security flaws can have serious consequences for the deployment and development environments in DevOps, along with other topics, we observe the need for research studies that will study defects and security flaws for IaC.
1. Introduction
IaC automates infrastructure configuration and provisioning, and its use and research interest have grown. This study maps the under-explored IaC research landscape to identify established and promising research areas.
- IaC context: IaC automatically configures system dependencies and provisions local and remote computing instances.
- IaC context: Organizations use IaC scripts and tools to provision cloud instances, manage databases, and manage user accounts.
- IaC context: Ambit Energy increased its deployment frequency by a factor of 1,200 using IaC scripts.
- Research gap: Interest in IaC has increased steadily after 2015, while systematic mapping evidence in the IaC domain remains limited.
- Study aim: The study conducts a systematic mapping study to identify addressed IaC research problems and problems needing further research.
- Study approach: Using six scholar databases and seven search strings, the researchers filtered 33,887 publications to 31 IaC-related publications and analyzed their topics and temporal trends.
- Contributions: The paper contributes a topic classification, temporal-trend evaluations, and an assessment of IaC-publication quality.
2. Background and Related Work
IaC applies software-engineering practices to infrastructure configuration, while systematic mapping studies classify and count research to reveal coverage. Prior IaC work spans tools, empirical studies, and testing, motivating a map of areas needing attention.
- Infrastructure as Code: IaC applies practices such as code review and version control to scripts that configure and provision infrastructure.
- Infrastructure as Code: IaC scripts use domain-specific languages, and organizations implementing DevOps widely use tools such as Puppet.
- Infrastructure as Code: A typical IaC workflow submits script changes to version control, triggers continuous integration, and runs static analysis and tests before integration.
- Systematic mapping studies: Systematic mapping studies classify papers and results into categories and count work in each category to show research coverage.
- Systematic mapping studies: Unlike systematic literature reviews, systematic mapping studies are exploratory rather than designed to synthesize answers to well-defined questions.
- Prior IaC research: Prior IaC research includes anti-patterns, convergence testing, co-evolution, test-suite reduction, automated correction, and automated testing frameworks.
- Prior IaC research: The mapping study is motivated by growing IaC research interest and aims to identify research areas needing attention.
3. Methodology
The study follows Petersen et al.’s systematic mapping guidelines through four methodological phases, including database searching, filtering, quality assessment, and topic analysis. It searches six scholar databases with seven strings, applies inclusion and exclusion criteria, evaluates publication quality, and uses qualitative analysis to identify topics.
- The systematic mapping study follows Petersen et al.’s guidelines and divides its methodology into four phases.
- Search: The search covers six scholar databases using seven search strings related to infrastructure as code, configuration as code, configuration scripts, DevOps, and IaC tools.The databases include IEEE Xplore, ACM Digital Library, IET Digital Library, Springer Link, ScienceDirect, and Wiley Online Library.
- Search validation: The search strings are validated with quasi-sensitivity against a quasi-gold set of potentially relevant peer-reviewed publications.The quasi-sensitivity metric is calculated as the number of search-string publications included in the quasi-gold set divided by the number of publications in that set.
- Quality and analysis: Two raters independently assess publication eligibility and record agreement using Cohen’s Kappa, resolving disagreements through discussion.The study also uses two raters to mitigate subjectivity in qualitative topic generation.
- Screening: The researchers filter search results using inclusion and exclusion criteria covering peer review, publication date, language, availability, and explicit IaC relevance.The study excludes non-peer-reviewed publications and work published before 2000, while requiring English text, download availability, and IaC relevance in the title, keywords, abstract, or introduction.
4. Results
The study narrows a large database search to 31 IaC-related publications, assesses their quality and threats to validity, and maps their topics, tools, and publication trends. The results show that IaC research is concentrated on frameworks and tools, while empirical studies and testing are less frequent and reporting of limitations is often incomplete.
- Search and selection: 33,887 publications were collected from scholar databases before duplicate removal and successive filtering produced the study set.The search process included six scholar databases and removed duplicates and other excluded publications before final screening.
- Search and selection: 31 publications were retained after title and abstract screening, author comparison, and resolution of disagreements.The screening process recorded Cohen’s Kappa of 0.81, described as ‘almost perfect’ agreement.
- Quality and validity: The quality assessment found scores above 3.0 for clearly stating aims, experimental design, findings, and actionable findings.These criteria correspond to quality checks Q1, Q3, Q8, and Q9.
- Quality and validity: Only 7 of 31 publications, or 22.5%, explicitly reported threats or limitations, indicating inadequate threat reporting.The study advocates better reporting of research threats in IaC publications.
5. Discussion
The mapping study finds that IaC research is relatively new and concentrated on implementing or extending IaC, while empirical studies, testing, defect analysis, and security remain limited.
- 31 IaC-related publications were identified, and the research area appears relatively new.
- Four topics emerged: Framework/Tool, Use of IaC, Empirical, and Testing.
- Framework/Tool was the most prevalent topic, followed by Use of IaC, indicating emphasis on implementing or extending IaC.
- Empirical studies and testing were infrequent compared with other software engineering research areas.
- No identified publication studied defects and security flaws in IaC.
- Future research avenues include anti-patterns, defect analysis, security, knowledge and training, and industry best practices.
6. Threat to Validity
The study’s validity is constrained by possible subjectivity in publication selection and topic generation, incomplete search coverage, and limited generalizability from the selected corpus.
- External Validity: Using six scholar databases excluded databases such as Scopus that may contain relevant IaC publications.
- External Validity: Seven search strings may have omitted IaC-related publications, although the quasi-sensitivity metric yielded 1.0.
- Conclusion Validity: Publication selection may be subjective and may miss IaC-related publications, although two raters independently assessed relevance.
- Conclusion Validity: Topic generation may be subjective, with two qualitative raters used to mitigate this threat.
- External Validity: The analysis depends on 31 publications collected on December 30, 2017, limiting generalizability.
7. Conclusion
The study maps IaC research across six scholar databases and identifies four principal topics, with research concentrated on tools and implementation. It concludes that defects, security flaws, and other underexplored areas warrant further study.
- IaC is described as a fundamental practice for implementing continuous deployment.
- Six scholar databases yielded 31 IaC-related publications filtered from 33,887 publications.
- The four topics are Framework/Tool, Use of IaC, Empirical, and Testing.
- Framework/Tool was the most prevalent topic, followed by Use of IaC.
- The authors advocate research on code quality issues such as defects and security flaws, alongside other IaC research avenues.