PaperlayerView exact revision on arXiv

Source-linked AI summary

Sampling Projects in GitHub for MSR Studies

Ozren Dabic, Emad Aghajani, Gabriele Bavota

arXiv:2103.04682v1cs.SE

TL;DR

MSR studies need repository samples that satisfy study-specific criteria, but GitHub APIs impose request limits and omit useful information. GHS addresses this gap with a continuously updated dataset and query tool containing 25 characteristics for 735,669 repositories across 10 languages. The dataset can be queried and exported through a web interface for project sampling.

  • Problem

    MSR researchers need to identify repositories meeting study-specific selection criteria, while GitHub APIs impose request limits and omit information such as commit counts.

  • Method

    GHS continuously mines 25 repository characteristics using the GitHub search API and a custom webpage crawler, with a query interface for selecting projects.

  • Results

    GHS provides information about 735,669 GitHub repositories written in 10 programming languages in a stable dataset, with continuously updated versions available online.

  • Takeaways & Limitations

    Researchers can query and export GHS to sample projects using combinations of repository selection criteria for MSR studies.

Abstract

from arXiv · show

Almost every Mining Software Repositories (MSR) study requires, as first step, the selection of the subject software repositories. These repositories are usually collected from hosting services like GitHub using specific selection criteria dictated by the study goal. For example, a study related to licensing might be interested in selecting projects explicitly declaring a license. Once the selection criteria have been defined, utilities such as the GitHub APIs can be used to "query" the hosting service. However, researchers have to deal with usage limitations imposed by these APIs and a lack of required information. For example, the GitHub search APIs allow 30 requests per minute and, when searching repositories, only provide limited information (e.g., the number of commits in a repository is not included). To support researchers in sampling projects from GitHub, we present GHS (GitHub Search), a dataset containing 25 characteristics (e.g., number of commits, license, etc.) of 735,669 repositories written in 10 programming languages. The set of characteristics has been derived by looking for frequently used project selection criteria in MSR studies and the dataset is continuously updated to (i) always provide fresh data about the existing projects, and (ii) increase the number of indexed projects. The GHS dataset can be queried through a web application we built that allows to set many combinations of selection criteria needed for a study and download the information of matching repositories: https://seart-ghs.si.usi.ch.

I. INTRODUCTION

MSR studies depend on selecting repositories that satisfy research-specific criteria, but GitHub’s APIs restrict querying and omit useful information. GHS addresses this by continuously indexing repository characteristics for scalable project sampling.

  • Motivation: Repository selection is crucial for obtaining generalizable findings and useful data points aligned with an MSR study’s research questions.Examples include requiring issue-tracker use or a minimum number of pull requests.
  • Motivation: GitHub’s search API allows 30 requests per minute, returns at most 100 results per request, and omits information such as repository commit counts.Searching basic information for public Java repositories would require approximately 160,000 requests, or about 88 hours.
  • Motivation: Existing archives such as GHTorrent and GHArchive do not provide a convenient way to sample projects using desired selection criteria.For example, computing repository-level counts in GHTorrent may require joins across multiple tables.
  • GHS: GHS is a dataset and tool that continuously mines 25 repository characteristics frequently used as MSR project-selection criteria.Examples include license, commits, contributors, issues, and pull requests.
  • GHS: GHS mines over 700,000 repositories across 10 programming languages and provides a query interface with export features.The stable dataset contains 735,669 repositories.

II. THE DATASET

GHS stores continuously updated repository information to support researchers in selecting projects for empirical studies. Its collection combines GitHub’s search API with a crawler that gathers additional information from repository webpages.

  • Dataset scope: GHS contains information about 735,669 public GitHub repositories for researchers selecting projects for empirical studies.The dataset mines, stores, and continuously updates 25 characteristics for each project.
  • Dataset scope: The dataset’s 25 characteristics are documented with descriptions, mining sources, and examples of their use in empirical studies.Table I lists these characteristics and their associated information sources.
  • Collection approach: GHS uses the GitHub search API and a custom crawler to collect information from repository homepages.The crawler supplements information unavailable through the search API.
  • Collection approach: Figure 1 presents the main steps of the GHS data-collection process.The paper subsequently details the process in its subsections.

A. Data Extraction

GHS combines API-based repository discovery with webpage crawling and periodic orchestration. Its extraction strategy uses language, time, and star filters, recursively partitions large result sets, and updates roughly 20,000 repositories daily.

  • Components: The data-collection process has three components: the GitHub API Invoker, GitHub Website Crawler, and Repository Miner.Together, they discover repositories, extract webpage information, and orchestrate recurring collection.
  • GitHub API Invoker: The GitHub API Invoker retrieves repositories by programming language and creation or update interval, while authenticated requests help manage API limits.It also monitors whether the mining token has exceeded its request limit.
  • Webpage extraction: The Website Crawler extracts webpage-sourced characteristics from repository landing, issues, and pull-request pages, while dynamic content can prevent complete capture.The crawler parses HTML with CSS selectors, primarily using jsoup.
  • Limitations: The crawler’s CSS-selector strategy may require updates if GitHub substantially changes its user interface.Generic selectors are used when possible to reduce this maintenance cost.
  • Continuous mining: The Repository Miner runs every six hours and either performs an initial historical collection or continues from the last mined date, excluding the newest two hours.The two-hour exclusion allows GitHub’s internal database time to synchronize newly created projects.
  • Filtering: GHS collects repositories written in selected languages, created or updated in selected intervals, and having at least 10 stars.The star threshold reduces the stored repository count and improves scalability, although stars are not a good proxy for quality or relevance.
  • Scalable retrieval: When an interval returns more than 1,000 repositories, the API Invoker recursively splits it in half and processes intervals through a priority queue.This addresses GitHub’s limit of returning only the first 1,000 results for a request.
  • Continuous mining: After extraction, GHS saves complete repository records to a database and can mine or update approximately 20,000 repositories per day.The algorithm iterates over repositories returned for intervals containing fewer than 1,000 results.

B. Data Storage

GHS stores repository data in MySQL and updates existing records rather than creating duplicates. A stable export contains 735,669 repositories across 10 languages, while the live dataset continues to update.

  • Storage: GHS stores collected repository characteristics in a MySQL database.When a previously mined repository changes, its corresponding rows are updated rather than duplicated.
  • Storage: Updating existing rows keeps repository data current but does not provide a historical evolution of the stored characteristics.The dataset therefore represents current information rather than a history of changes.
  • Stable dataset: The stable export dated January 28, 2021 contains 735,669 repositories written in 10 languages.The export is hosted on Zenodo.

C. Querying GHS

GHS combines configurable repository filters with a results interface that supports inspection and export. The filters cover repository identity, metadata, activity, popularity, and refinement criteria.

  • General filters: GHS lets users filter repositories by name strings, license, programming language, and issue labels.Issue-label collection was still under development and absent from the stable version.
  • Activity and popularity filters: Users can filter projects by history, activity periods, popularity indicators, and options such as excluding forks.Examples include numbers of commits and releases and activity within a specified time frame.
  • Results and export: The results page lists repositories satisfying the criteria and allows users to inspect and download them in different formats.The interface is illustrated with Apache Java repositories having at least 100 commits.
  • Interface overview: The interface presents the query GUI on the left and the results page with export options on the right.Together, the panels show the transition from specifying a repository sample to obtaining its results.

III. RELATED WORK

Prior tools provide GitHub data but are not principally designed for convenient repository sampling by precomputed selection criteria. GHS addresses this sampling focus relative to archival datasets and extraction tools.

  • Archival datasets: GHArchive records hourly GitHub events, but sampling repositories by creation year requires scanning roughly 8,000 event files.This makes it a valuable MSR data source but inconvenient for repository sampling.
  • Archival datasets: GHTorrent stores extensive GitHub data without query limits, but retrieving repository-level measures such as commit counts can require large multi-table queries.GHS instead focuses on basic repository information and precomputed selection criteria.
  • Archival datasets: Software Heritage preserves source code, including projects deleted from GitHub, but is not explicitly intended to simplify empirical project sampling.Its scope differs from GHS's sampling-oriented purpose.
  • Extraction tools: GitRepository extracts structured repository information through the GitHub API but does not provide a dataset, whereas GHS offers broader information through a GUI.The comparison concerns both data availability and sampling convenience.
  • Earlier projects: Public Git Archive contains approximately 180,000 repositories with at least 50 stars and, to the authors' knowledge, is not continuously updated.It was released in 2018.
  • Earlier projects: Orion combines projects from GitHub, Google Code, and Freecode but requires a custom DSL, and its project webpage is no longer accessible.These constraints distinguish it from GHS's web-based querying interface.

IV. FUTURE WORK

Future GHS work focuses on expanding language coverage and issue-label data, incorporating community-requested characteristics, and improving mining performance.

  • Future improvements: GHS plans to add more programming languages by changing its configuration file.Language expansion is presented as the first improvement direction.
  • Future improvements: The project plans to finalize collecting issue labels, which the GUI already supports but the crawler has not yet fully collected.Labels could support studies targeting repositories that explicitly use terms such as refactoring or documentation.
  • Future improvements: The open-source project plans to gather requests for additional repository characteristics through its issue tracker.The requested characteristics may be incorporated into GHS over time.
  • Future improvements: The authors plan to improve performance, especially the efficiency of data mining.This is the fourth stated improvement direction.

V. CONCLUSIONS

GHS is a dataset for simplifying MSR project sampling, with a stable release covering 735,669 repositories across 10 languages. Its dataset is continuously updated and available through an online querying interface.

  • Conclusions: 735,669 GitHub repositories written in 10 languages are included in the stable version of GHS.The dataset is available through Zenodo.
  • Conclusions: GHS is designed to simplify selecting projects for Mining Software Repositories studies.The online platform provides access to the latest continuously updated and expanded version.
  • Conclusions: The latest GHS version can be queried through an online interface at seart-ghs.si.usi.ch.The interface accompanies the continuously updated and expanded dataset.
Loading 2103.04682v1…