How are Software Repositories Mined? A Systematic Literature Review of Workflows, Methodologies, Reproducibility, and Tools

1 INTRODUCTION AND MOTIVATION

• Repository mining now draws on a larger, changing ecosystem of open-source projects, development tools, and data sources.Earlier work focused on CSV and SVN repositories; more recent work often uses Git repositories or sites such as Stack Overflow.Source: §1 · p. 1
• Changes in software development and stronger expectations for rigorous, reproducible research raise questions about whether workflows developed decades ago still fit current needs.The shift includes more attention to upstream and downstream dependencies and to tool-specific data pitfalls.Source: §1 · p. 1
• Data selection and sampling matter because selection can affect how generalizable repository-mining results are.The review therefore focuses especially on how researchers identify which software projects to analyze.Source: §1 · p. 1
• The review analyzes 286 relevant papers from three conferences, selected from 1,109 papers reviewed, to examine workflows, methodological choices, tools, and reproducibility.The authors manually analyzed papers after applying pre-selection criteria.Source: §1 · p. 2
• The paper examines tool use and dataset reuse, describes researchers’ sampling and retrieval workflows, and assesses reproducibility while proposing reporting recommendations.The analyzed papers and evaluation data are listed in an online appendix.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

• Repository-mining methodology progressed from early studies of version-control and issue-tracking data to specialized methods for Git repositories and broader data-analysis challenges.Prior literature also treated some methodological aspects as standalone contributions.Source: §2 · p. 2
• Earlier surveys examined data sources, research aims, methodology, and quality assessment, while later studies investigated data leakage and Git timestamp anomalies.The cited examples include Kagdi et al., data leakage, and timestamp anomalies [8].Source: §2 · p. 2
• Researchers can draw on many Git-hosting platforms, but the review describes GitHub as the dominant source, reflecting its greater data availability.The passage compares GitHub’s reported user and repository counts with GitLab’s user count.Source: §2 · p. 2
• GHTorrent, World of Code, and Boa provide precompiled project data, with World of Code monitoring multiple hosting sites and Boa limited to Java and Python projects.SOTorrent serves a similar precompilation purpose for public Stack Overflow dumps.Source: §2 · p. 2
• PyDriller and the GitHub REST API support data extraction, while GitHub Search (GHS) was designed to help researchers select and sample data.PyDriller extracts repository information such as commits, developers, and source code.Source: §2 · p. 2

3 METHODOLOGY

• Repository analysis can involve understanding development practices, retrieving and augmenting raw data, operationalizing measures, modeling them, and validating each step iteratively.The paper builds on prior workflow descriptions.Source: §3 · p. 3
• The review uses three conferences—MSR, ICSE, and ESEC/FSE—to include both a repository-mining venue and broader software-engineering research venues.The authors selected these conferences to support generalization across software-engineering research.Source: §3 · p. 3
• Papers from 2018–2020 were preselected using text searches and relevance criteria for GitHub, Stack Exchange, Stack Overflow, or specified mining tools, then manually screened for topical relevance.Papers mentioning the sources or tools only tangentially were removed; methodology and experimental-setup sections were analyzed.Source: §3 · p. 3
• The paper’s analysis tracked tools, sampling and data selection, dataset reuse and publication, replication packages, and the ordered workflow steps reported by each study.The review also examined whether researchers created custom tools.Source: §3 · p. 3
• The authors separately classified tools as resources enabling direct data access or retrieval and datasets as resources providing selected data without direct retrieval.They also assessed where researchers needed custom scripts across workflow steps and analyzed sampling practices.Source: §3 · p. 3
• The authors tracked replication-package claims and the availability of datasets, personal scripts, and other resources used to obtain results.The review treats replication packages as important for research involving complex data and extensive processing.Source: §3 · pp. 3–4
• The generalized workflow was developed inductively from methodology sections in about half the selected papers and checked against the remaining half; its stages are sampling, filtering, and retrieval.The authors note that earlier workflow descriptions did not capture contemporary studies selecting multiple projects.Source: §3 · pp. 3–4

4.1 RQ1 Results: Workflows

• Workflow paths did not always follow the listed order, and researchers sometimes combined approaches, such as supplementing a precompiled dataset with additional data they selected.The review also notes that the reported workflow statistics may be skewed because it did not deeply analyze external resources such as replication packages.Source: §4.1 · pp. 4–5
• Of 268 papers reporting data selection, 86.9% personally sampled at least some data; filtering occurred in 55.2% of papers reviewed, 63.6% detailed retrieval, and 79% of 182 retrieval-detailing papers partly relied on others’ work.Papers not stating whether they filtered were categorized as not filtering.Source: §4.1 · p. 5

4.2 RQ2 Results: Methodologies

• 20.7% of the 77 papers using datasets focused on bug-related datasets, while 19.5% focused on RepoReaper or F-Droid; 44.2% of datasets used had appeared in previous works.No single bug dataset was used in more than seven papers, and the authors say overall dataset reuse was limited.Source: §4.2 · p. 6
• Language was a sampling criterion in 64.2% of 265 papers and popularity in 45.3%; 29.7% sampled manually, while random sampling was the only traditional sampling form identified.The review found no two papers used random sampling in the same way; many first applied other restrictions, and some did not explain how they sampled randomly.Source: §4.2 · p. 6
• Random-sampling examples included selecting GitHub projects that met language and README criteria until 5,000 were collected, and stratifying Stack Overflow questions by language before sampling 210 pairs.The review says detailed, good examples were rare; random sampling often followed earlier sampling and was insufficiently described, making reproduction harder.Source: §4.2 · pp. 6–7

4.3 RQ3 Results: Tools

• 44.7% of 152 papers reporting tool use used the GitHub REST API, 21.7% used GHTorrent, 14.5% the Stack Overflow Data Dump, and 11.2% SOTorrent.No other listed tool appeared in more than 10% of papers; the authors characterize tool use as concentrated in a few tools and find little correlation between tools overall.Source: §4.3 · p. 7
• Researchers most often wrote scripts to filter or prune sampled data, were least likely to script sampling, and descriptions of script use were often incomplete.Among papers publishing replication kits, 145 included personal scripts, but no more than 78 reviewed papers specifically noted using scripts; the review cautions that it did not assess their intended use.Source: §4.3 · p. 8

4.4 RQ4 Results: Reproducibility

• 75.9% of the 286 papers met the review’s broad definition of publishing a replication package, which included making a dataset available even without other process information.The researchers checked stated package contents and resources at a glance but did not conduct an in-depth analysis of packages.Source: §4.4 · p. 8
• Among 205 working packages, 89.3% provided datasets, 70.7% included scripts, and 56.1% included additional resources.Twelve packages had non-working links and were excluded from contents statistics.Source: §4.4 · pp. 8–9
• Among 217 papers with packages, 86.6% claimed datasets, while 56.2% disclosed scripts and 39.6% disclosed results or other analysis resources.Dataset availability was the only type of package information shared consistently.Source: §4.4 · p. 8
• Package contents were not well defined, potentially making replication packages difficult or impossible to use.Source: §4.4 · p. 9

5.2 Reuse of Data and Tools

• Tool use clusters around a small subset, while datasets are often used to replicate prior work; large or labor-intensive curated datasets are notable exceptions.GHTorrent and SOTorrent are among the tools created for research; datasets with large samples or substantial curation are reused more often.Source: §5 · p. 9, §5.2 · p. 10

5.1 Standard for Reporting Methodologies

• Of 286 papers, 94 gave no insight into data retrieval and 175 lacked data-collection timestamp information; 217 provided replication packages, including 65 of the former and 122 of the latter.Source: §5.1 · p. 9

5.3 Implications for Tools

• Only 38 of 287 papers used any random sampling; World of Code offers a mechanism to make random sampling easier.Random sampling was difficult and often required substantial manual effort.Source: §5.3 · p. 10

5.4 Threats to Validity

• Validity risks include paper-selection bias, historical bias from the sampled years, and human error in manual categorization; the authors cite three conferences and a short period as mitigations.Source: §5.4 · p. 10

• The review analyzes 286 papers from recent conference proceedings to characterize workflows, methodologies, reproducibility, and tools.Source: §6 · p. 10
• Workflows are disjoint and often require multiple tools and custom scripts; 33% of papers omit data-retrieval information, and published datasets are rarely reused.Source: §6 · p. 10
• The authors recommend improving existing tools and reporting workflows through a methodology standard, addressing concerns about reproducibility and a replication crisis in software engineering research.Source: §6 · pp. 10–11