Where Are The Gaps? A Systematic Mapping Study of Infrastructure as Code Research

Background

• IaC automatically configures system dependencies and provisions local and remote instances, supporting automated deployment and cloud-based infrastructure management.Chef and Puppet provide utilities for configuring and provisioning software deployment infrastructure using cloud instances.Source: §1 · p. 2

Motivation

• IaC scripts helped Ambit Energy increase its deployment frequency by a factor of 1,200.Organizations such as Github, Mozilla, and Netflix use IaC utilities to provision cloud-based instances, manage databases, and manage user accounts.Source: §1 · pp. 2–3
• Because IaC research remains under-explored, a systematic mapping study can summarize the domain, provide an overview, identify potential research topics, reveal gaps, and group existing research.Source: §1 · p. 3

Contributions

• The paper contributes a list of IaC research topics, evaluations of publication and tool-use trends, and an assessment of IaC-related publication quality.Its research questions address studied topics, temporal publication trends, and temporal trends in the use of IaC-related tools.Source: §1 · pp. 4–5

Methodology

• The study searches six scholar databases with seven search strings, narrows 33,887 publications to 31 IaC-related publications, and applies quality assessment and qualitative analysis.It also evaluates overall and topic-wise publication trends and temporal trends in IaC-related tool use.Source: §1 · p. 5

2.1.1. Background on Infrastructure as Code (IaC)

• IaC applies software-engineering practices such as code review and version control to scripts that automatically provision infrastructure using domain-specific languages.Organizations commonly use tools such as Puppet to implement IaC.Source: §2.1.1 · p. 6

2.1.2. Background on Systematic Mapping Studies

• A systematic mapping study classifies papers into categories, counts work in each category, and maps research coverage, while remaining exploratory rather than synthesizing answers to predefined questions.This approach provides a basis for future research studies.Source: §2.1.2 · p. 7

2.2.1. Prior Research on IaC

• Prior IaC studies investigated anti-patterns, automated convergence testing, and co-evolution with other software artifacts; IaC scripts experienced frequent churn.Source: §2.2.1 · pp. 7–8
• The observed growing interest in IaC motivates this paper’s systematic mapping study of IaC research.The authors explicitly take motivation from this observation.Source: §2.2.1 · p. 8

2.2.2. Prior Research on Systematic Mapping Studies

• Systematic mapping studies of technical debt management (94 publications) and domain-specific languages (390 publications) found a need for dedicated management tools and limited emphasis on evaluating technique effectiveness.Source: §2.2.2 · p. 8
• Drawing on these precedents, the study maps IaC research to identify research areas that need attention in the field.The stated aim is to identify IaC research areas requiring further attention.Source: §2.2.2 · p. 9

3. Methodology

• The methodology follows Petersen et al. ’s systematic mapping guidelines and is organized into four phases.The supplied part covers the first three phases: search, inclusion and exclusion, and quality analysis.Source: §3 · p. 10

3.1. Phase One: Search

• The search phase queries six scholar databases with seven IaC-related search strings and validates their coverage using quasi-sensitivity against a quasi-gold publication set.The databases include IEEE Xplore, ACM Digital Library, IET Digital Library, Springer Link, ScienceDirect, and Wiley Online Library; the search strings cover IaC, configuration as code, configuration scripts, DevOps, Puppet, Ansible, Chef, and CFEngine.Source: §3.1 · pp. 10–12

3.2. Phase Two: Inclusion and Exclusion Criteria

• The inclusion and exclusion phase removes irrelevant, non-peer-reviewed, pre-2000, non-English, unavailable, and non-IaC publications before analysis.The retained publications must explicitly indicate IaC relevance in the title, keywords, abstract, or introduction, and are assessed for quality before answering the research questions.Source: §3.2 · pp. 13–14

3.3. Phase Three: Quality Analysis

• The quality-analysis phase applies Kitchenham et al. ’s criteria to assess whether publications clearly state their aims, units, and experimental designs.The criteria also address actionable findings, limitations, and presentation structure as indicators of publication quality.Source: §3.3 · p. 15
• Reported validity threats are classified using Wohlin et al. ’s four categories; conclusion validity concerns drawing conclusions without violating statistical assumptions and maintaining sufficient statistical power.Source: §3.3 · p. 16

3.4.1. Answer to RQ1: What topics have been studied in infrastructure as code (IaC)-related publications?

• RQ1 identifies IaC research topics by extracting raw text, generating verbatim initial codes, abstracting commonalities into topics, and resolving rater disagreements through discussion.Two raters independently generate topics, record Cohen’s Kappa [35], and allow each publication to relate to multiple identified topics.Source: §3.4.1 · p. 17

3.4.2. Answer to RQ2: What are the temporal publication trends for infrastructure as code (IaC)-related research topics?

• RQ2 measures temporal trends by counting IaC-related publications per year overall and separately for each identified topic.This produces two trend categories: an overall publication trend and topic-specific publication trends.Source: §3.4.2 · p. 18

3.4.3. Answer to RQ3: What are the temporal trends for the use of infrastructure as code (IaC)-related tools, as mentioned in IaC-related publications?

• RQ3 analyzes tool usage by counting a tool when it implements a framework, provisions a system, supports empirical research, or validates a framework or methodology.The analysis links reported tools to tasks and can reveal available scripts and repositories, such as Puppet scripts and GitHub sources, for future IaC empirical studies.Source: §3.4.3 · p. 19

4. Results

• The search yielded 33,887 publications; after removing duplicates, 14,015 remained, 10,567 were in English, 727 non-peer-reviewed publications were removed, and 31 studies were identified.Source: §4 · pp. 20–21
• Only 7 of 31 publications (22.5%) explicitly reported threats or limitations, indicating inadequate threat reporting in IaC research.The study recommends reporting threats according to Wohlin et al..Source: §4 · p. 24

4.1. Answer to RQ1: What topics have been studied in infrastructure as code

• Qualitative analysis identified four non-exclusive IaC research topics: framework/tool, use of IaC, empirical studies, and testing.A publication may belong to multiple topics, so the categories are not orthogonal.Source: §4.1 · p. 25
• Framework/tool studies were most common (16), followed by use of IaC (11), empirical studies (7), and testing (4).Empirical work included testing and non-testing studies, while testing research focused especially on idempotence and lacked empirical studies of coverage, practices, and techniques.Source: §4.1 · pp. 25, 27, 28, 29

4.2. Answer to RQ2: What are the temporal publication trends for infrastructure

• IaC research first appeared in 2012, reached a peak of 9 publications, and increased after 2014 for framework/tool, use, and empirical-study topics.Publication counts were insufficient to establish trends for the other topics.Source: §4.2 · pp. 29–30, Table 5 · p. 30, Table 6 · p. 30

4.3. Answer to RQ3: What are the temporal trends for the use of infrastructure as code

• RQ3 reports the IaC tools used in the 31 collected publications and begins by identifying their names.The supplied passage introduces the tool-use analysis but does not provide its findings.Source: §4.3 · p. 31

4.3. Answer to RQ3: What are the temporal trends for the use of infrastructure

• Twelve IaC tools appeared across 31 publications, with Chef used most often in 8 publications (25.8%).Source: §4.3 · p. 31
• Tool diversity was greatest for Framework/Tool studies, whereas Empirical Study and Testing publications mainly used Chef and Puppet.Source: §4.3 · p. 32
• From 2013 to 2017, Chef and Puppet were used more often than other IaC tools in the reviewed research.Source: §4.3 · pp. 32–33

5.1. Research in IaC: State of the Art

• The study identified 31 IaC-related publications from 9,840 search results, indicating that IaC research remains relatively new and is dominated by Framework/Tool studies.Empirical studies and testing are infrequent, and no publications address defects or security flaws.Source: §5.1 · pp. 34–35

5.2. Variety of Tools

• The 31 publications use 12 IaC tools, with Chef, Puppet, and Juju appearing most frequently and commercial tools providing promising research artifacts.Open-source repositories such as GitHub, PuppetForge, and Chef Cookbooks are suggested as useful sources for IaC research.Source: §5.2 · p. 35

5.3. Potential Research Avenues in IaC

• Future research should examine anti-patterns, defects, security flaws, knowledge and training, and industry best practices in IaC.Only one publication studied anti-patterns, no publications studied defects or security, and no study systematically characterized IaC best practices.Source: §5.3 · pp. 35–36
• Defect and security research should identify IaC characteristics and flaw types, assess consequences, and develop mitigation guidance.The paper notes that IaC defects can cause wide-scale outages and that security errors can compromise entire systems.Source: §5.3 · p. 36

5.4. Towards Better Reporting of Research Findings

• None of the publications achieved a perfect 4.0 quality score across all checks, although most offered actionable findings or suggestions.Future IaC research should follow reporting guidelines provided by experts.Source: §5.4 · p. 37

Internal Validity

• The search may omit relevant IaC publications because six databases and seven search strings were not comprehensive, although QSM yielded a score of 1.0.Other databases, including Scopus, were not considered.Source: §6 · pp. 37–38

Conclusion Validity

• Selecting IaC-related publications may be subjective and could miss relevant studies, so two raters independently determined which publications were related to IaC.Source: §6 · p. 38
• Generating topics through qualitative coding can be subjective, and the study mitigated this limitation by using two qualitative raters and established guidelines.Source: §6 · p. 38

External Validity

• Because the analysis covered 31 publications collected on December 30, 2017, and databases may omit relevant work, the findings have limited generalizability.The study mitigated this threat by using six scholar databases recommended by Kurhamm.Source: §6 · p. 38

• The study filtered 31 IaC publications from 33,887 records collected through six scholarly databases.Source: §7 · p. 39
• Qualitative analysis identified four topics: IaC frameworks/tools, IaC use, empirical IaC studies, and IaC testing.Source: §7 · p. 39
• Frameworks/tools were the most prevalent topic, followed by IaC use, indicating that research mainly implements or extends IaC practice.Source: §7 · p. 39
• The 31 publications used 12 tools, with Chef reported as the most frequently used tool.Source: §7 · p. 39
• Because IaC defects and security flaws can have serious consequences, the authors advocate research on code quality and other open research avenues.Source: §7 · p. 39