PaperlayerView exact revision on arXiv

Source-linked AI summary

Removing Noise or Introducing Bias? The Hidden Cost of MSR Filtering

Mohit Kaushik, Jyoti Bawa

arXiv:2609.18953v1cs.SE

TL;DR

Researchers use thresholds to remove experimental or toy repositories, but those filters can also alter sample demographics. The authors analyze a cleaned SEART dataset under commonly used thresholds and find maintenance, ecosystem, governance, and relational biases.

  • Problem

    Researchers filter experimental or toy repositories, but these thresholds can change sample demographics and statistical relationships, limiting generalizability.

    Source: §I · p. 1

  • Method

    The authors cleaned SEART data to a 1.57-million-repository baseline and examined repository composition and maintenance across popularity thresholds.

    Source: B. Data Collection, Preprocessing and… · p. 2

  • Results

    73.42% of baseline projects were inactive; filtering also shifted project composition and maintenance and inflated or degraded relationships between software metrics.

    Source: §IV · p. 3

  • Takeaways & Limitations

    The authors recommend stratified random sampling and refining noise criteria to avoid dropping genuine engineered projects.

    Source: §VI · p. 4

    The SEART baseline already applies a 10-star threshold, excluding GitHub’s absolute bottom tail; commits and stargazers are proxies affected by accumulation and bots.

    Source: §V · p. 3

Abstract

from arXiv · show

OSS platforms like GitHub serve as a primary data source for MSR research. As the platform is widely used by different users, spanning from student to developer, not all repositories are actual engineered projects. Therefore, to avoid such noise, researchers often apply several criteria, which may fundamentally change the sample demographics. To understand such biases, this study aims to uncover the hidden cost originating from these arbitrary thresholds or criteria. We analyzed 1.57 million repositories from the SEART platform and constructed several datasets from the thresholds often applied in MSR research. We identify the maintenance bias in these filtering processes, which masks the true abandonment (73.42%) realities of OSS projects. Also, these strategies favor some ecosystems and governance styles. Moreover, the sampling strategy also distorts the relationship between variables, suffering from relational biases. Therefore, to avoid such biases, the researchers should shift towards stratified sampling and refining the criteria for noise detection.

I. INTRODUCTION

GitHub is a major source for software engineering research, but its repositories include experimental projects that researchers often filter out. The study examines how such filters change sample demographics and introduce composition, maintenance, and relational biases.

I. INTRODUCTION

  • Researchers filter experimental or toy repositories using popularity or activity thresholds, including a 10-star minimum that still leaves substantial noise.Some studies apply a higher threshold, such as more than 500 stars, to further reduce noise.Source: §I · p. 1
  • Higher star thresholds can change sample demographics, favoring particular governance styles or ecosystems and altering statistical relationships that limit generalizability.Source: §I · p. 1
  • The study analyzes how arbitrary noise-reduction thresholds affect OSS composition, maintenance, and relational biases.Source: §I · p. 1

II. RELATED WORKS

Prior work documented the risks of treating GitHub repositories as uniform research objects and motivated filtering for noise. Related studies also show that star- and commit-based thresholds can lose genuine projects and distort samples and relationships.

II. RELATED WORKS

  • Kalliamvakou et al. identified 13 perils of mining GitHub, reporting that 37% of analyzed projects were noise or experimental and most were dead.Source: §II · p. 1
  • A review of 286 recent MSR studies found that 86.7% relied on non-probability sampling and 43.7% used popularity-based criteria to exclude repositories [12].Source: §II · p. 1
  • Star-based filters can discard genuine projects: a 10-star threshold had 97% precision but 32% recall, while 50- and 500-star thresholds had 14% and 0% recall.Munaiah et al. reported that the 10-star threshold lost 68% of genuine projects; Koch et al. found a weak correlation between stars and downloads.Source: §II · p. 2
  • High commit thresholds may select short-lived projects because younger projects can have higher monthly commit rates, and stricter thresholds can flip the activity-popularity relationship,.Raw commit volume can accumulate over time and be inflated by bots.Source: §II · p. 2

III. METHODOLOGY

The methodology section presents the study’s research questions, data source, and methodology.

III. METHODOLOGY

  • The section covers the research questions, data source, and methodology.Source: §III · p. 2

A. Research Questions

The study asks how popularity and activity thresholds affect repository representation and maintenance, and whether they alter relationships between developer activity and project popularity.

A. Research Questions

  • RQ1 asks how popularity- and activity-based thresholds alter programming-language, repository-age, and project-size representation.Source: A. Research Questions · p. 2
  • RQ2 asks how arbitrary commit and star thresholds change active-project age and project-abandonment rates.Source: A. Research Questions · p. 2
  • RQ3 asks how far noise filters change the statistical relationship between developer activity and project popularity.Source: A. Research Questions · p. 2

B. Data Collection, Preprocessing and Construction

The study starts from a cleaned baseline of 1.57 million repositories and constructs samples using popularity- and activity-based thresholds. It also compares filtered samples with a language-stratified random sample to examine correlations.

Data Collection, Preprocessing and Construction

  • The cleaned baseline contains 1.57 million repositories after removing null-attribute, forked, and empty projects from 1.9 million SEART repositories.The SEART dataset natively applied a 10-star threshold.Source: B. Data Collection, Preprocessing and… · p. 2
  • Star thresholds of 10, 20, 50, 100, 500, and 1,000 are used to assess project composition and maintenance bias.The measures include programming-language and license shares, median size, team size, and age.Source: B. Data Collection, Preprocessing and… · p. 2
  • For RQ3, the study varied activity-based commit thresholds from 10 to 1,000—specifically 10, 20, 50, 100, 500, and 1,000—and constructed three datasets, A, B, and C.Source: B. Data Collection, Preprocessing and… · p. 2
  • For RQ3, the study compared attribute correlations in datasets of 10,000 most active projects (top commits) and 10,000 randomly sampled repositories stratified by programming language, from a 1.57-million-repository baseline, to assess whether selection criteria inflate or degrade correlations.Source: B. Data Collection, Preprocessing and… · p. 3

IV. RESULTS

Popularity- and activity-based thresholds shift repository demographics and maintenance profiles, while filtering also changes measured relationships among repository attributes. The results report distinct patterns for star and commit thresholds.

Popularity- and activity-based thresholds

  • 7x: median project size rises as star thresholds increase, while the baseline contains 73.42% inactive projects whose share declines under stricter thresholds.Higher star thresholds also favor older, permissively licensed, and infrastructural projects; Python and JavaScript remain relatively steady while TypeScript and Go gain share and C++ drops out of the top five.Source: §IV · p. 3
  • ≈18x: median project size rises with commit thresholds, while inactive projects decline from 72.12% to 40.49%.Median contributors rise from 3 to 19 and age from 29.50 to 68.63 months; commit thresholds favor copyleft licenses and shift ecosystem shares.Source: §IV · p. 3

Relational bias

  • Filtering warps correlations: commits with size are inflated in popular and active samples, while commits–issues and issues–contributors correlations diverge by filtering strategy.For popularity-filtered versus activity-filtered samples, the latter correlations are 0.707 versus 0.331 and 0.692 versus 0.365, respectively; stargazer correlations degrade regardless of filtering. Spearman correlations were used because the dataset is described as nonlinear and Spearman as distribution-agnostic.Source: §IV · p. 3

V. DISCUSSION AND LIMITATIONS

The discussion frames filtering as a trade-off: criteria used to remove noise can introduce demographic and relational biases. The study’s baseline and proxy measures also constrain what its findings cover.

Discussion

  • Filtering can favor particular ecosystems and governance styles while changing statistical relationships, potentially leading models trained on top-N samples to learn sampling artifacts.The authors connect this concern to generalizability and prior work on GitHub project perils and variable relationships,, [12].Source: §V · p. 3

Limitations

  • The baseline excludes GitHub repositories below 10 stars, and unpopular repositories are not always noise.The study also uses commits and stargazers as activity and popularity proxies, which can be affected by accumulation bias and bots.Source: §V · p. 3

VI. CONCLUSION AND FUTURE WORK

The conclusion reports that filtering thresholds alter repository size, governance, and measured correlations. It recommends more representative sampling and more carefully refined noise criteria.

Conclusion

  • 7x: star-based thresholds inflate project size, while commit-based thresholds increase it by 18x; the approaches also favor different governance styles.Star-based thresholds favor permissive licenses and infrastructural projects, whereas commit-based thresholds favor different governance styles.Source: §VI · p. 3
  • Both threshold types can inflate or degrade correlations, introducing correlation bias.Table II is identified as reporting Spearman correlations across datasets, while Table I covers star- and commit-threshold progression.Source: §VI · p. 3, Table II · p. 4, Table I · p. 4

Future Work

  • The authors recommend avoiding unjustified top-N thresholds, using stratified random sampling, and refining noise criteria to avoid dropping real engineered projects.The stated aim is to preserve sample representativeness while removing noise.Source: §VI · p. 4
Loading 2609.18953v1…