PaperlayerView exact revision on arXiv

Source-linked AI summary

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

Joshua Owotogbe, Indika Kumara, Willem-Jan van den Heuvel, Damian Tamburri

arXiv:2608.28396v1cs.SE

TL;DR

Open-source ML robustness tools are important dependencies, but evidence about their maintenance, engagement, and long-term sustainability is limited. This study mines 28 GitHub repositories using repository-level software-engineering indicators and finds uneven engagement and maintenance, with activity concentrated in a small subset. The study also reports scope boundaries for interpreting repository activity and calls for treating robustness tools as evolving software systems.

  • Problem

    Evidence is limited about how open-source ML robustness tools are maintained, publicly engaged with, and sustained over time, despite their use in ML evaluation pipelines and AI assurance.

  • Method

    The study mines 28 GitHub repositories and analyzes functional scope, observable engagement, maintenance activity, and project longevity using established repository-mining methods and statistical analyses.

  • Results

    Five repositories were classified as active, 22 as inactive, and one as archived on January 21, 2026; six maintenance features remained significant after Holm correction.

  • Takeaways & Limitations

    Robustness tools should be treated as evolving software systems when considering their sustainability and continued use.

Abstract

from arXiv · show

Robustness evaluation is essential for deploying machine-learning (ML) systems in real-world settings, where models may face adversarial perturbations, distribution shifts, and other operational stressors. Many open-source tools, including Adversarial Robustness Toolbox, Foolbox, and Robustness Gym, support robustness testing and evaluation. However, little is known about how these tools are maintained, publicly engaged with, and sustained over time, even though practitioners may rely on them to select evaluation dependencies, reproduce robustness assessments, and provide evidence for AI assurance. We present an empirical study of the open-source robustness tooling ecosystem. Starting from a curated seed set derived from prior work, we systematically searched GitHub and identified 28 robustness-tool repositories. We analyzed repository artifacts to characterize observable community engagement, maintenance activity, and project longevity using established software-engineering metrics. Our results show that engagement and maintenance are unevenly distributed, with sustained activity concentrated in a small subset of repositories. At the data collection date of January 21, 2026, five repositories were classified as active, 22 as inactive, and one as archived. These findings highlight the need to treat robustness tools as evolving software systems.

1 Introduction

This study examines the sustainability of 28 open-source ML robustness-tool repositories, focusing on engagement, maintenance, and longevity. Engagement and recent development were concentrated in a small subset, with five repositories active, 22 inactive, and one archived on January 21, 2026.

  • 28 open-source ML robustness-tool repositories were mined to examine functional scope, community engagement, maintenance activity, and project longevity.
  • Engagement and recent development activity were concentrated in a small subset of repositories.
  • Five repositories were active, 22 inactive, and one archived under the 180-day commit-recency rule on January 21, 2026.
  • The estimated median time to 180-day inactivity was 40.63 months, approximately 3.39 years.
  • CleverHans had over 6,400 stars and 185 watchers but was classified as inactive because its default branch had no commits during the final 180 days.
  • The study adapts repository-mining methods from fairness-tool research, including repository discovery, 12 maintenance features, a 24-month window, and Mann-Whitney U comparison.

2 Background

The background defines ML robustness as reliable predictive performance under specified input changes and explains how robustness tools operationalize testing, evaluation, and mitigation. It frames sustainability as observable engagement and maintenance that must keep pace with evolving frameworks, dependencies, and adversarial practices.

  • 2.1 Machine Learning Robustness: Robustness is the capacity of a trained model to maintain reliable predictive performance under specified changes in input data.
  • 2.1 Machine Learning Robustness: Adversarial robustness concerns deliberate input modifications, whereas non-adversarial robustness concerns naturally occurring or synthetic deployment-time changes.
  • 2.2 Robustness Tools: Robustness tools support executable workflows for testing, evaluation, and mitigation, including attack generation, defense application, and benchmarking.
  • 2.2 Robustness Tools: Most tools target model training and evaluation, while some also support data preprocessing and depend on evolving ML-framework APIs and data abstractions.
  • 2.2 Robustness Tools: New attack methods and circumvented defenses create recurring maintenance demands alongside framework and dependency evolution.
  • 2.3 Open-Source Software Sustainability: Repository sustainability is assessed through community engagement and maintenance indicators such as stars, watchers, forks, commits, contributors, issue resolution, and pull-request activity.
  • 2.3 Open-Source Software Sustainability: This study classifies repositories using the GitHub archived flag and a reproducible 180-day commit-recency rule, while inactivity does not necessarily imply unusability or permanent abandonment.

3 Related Work

Prior work has proposed and evaluated ML robustness tools, but existing analyses largely treat them as static artifacts rather than evolving software projects. This study addresses the resulting gap by systematically examining maintenance, community engagement, and sustainability in open-source ML robustness tooling.

  • Repository mining: Repository-mining research uses commits, issues, pull requests, and related artifacts to characterize development dynamics, maintenance, project health, and sustainability.Prior studies also examine domain-specific practices across software ecosystems and programming environments.
  • Research gap: No prior study had systematically analyzed open-source ML robustness tools for maintenance activity, community engagement, and long-term sustainability within GitHub.The paper frames robustness tooling as a distinct ecosystem shaped by both adversarial threats and changing ML software environments.
  • Prior robustness tooling: Existing robustness-tool studies propose libraries, benchmarks, platforms, and analysis tools for adversarial evaluation and robustness assessment.Examples include adversarial attack and defense libraries, RobustBench, CANARY, and ROBY.
  • Prior robustness tooling: Surveys and comparative reviews cover robustness methods, benchmarks, open-source tools, and feature-oriented tool selection.These studies examine areas including adversarial attacks, defenses, evaluation metrics, privacy, community support, popularity, and versatility.
  • Research gap: Existing comparative analyses evaluate robustness tools at a point in time but do not examine continued maintenance, sustained contributor engagement, or ongoing compatibility with evolving frameworks and techniques.The gap concerns whether tools remain actively maintained, communities remain engaged, and tools continue functioning as ML frameworks and adversarial techniques change.
  • Study positioning: The study adapts established repository-mining procedures to investigate this previously unexamined open-source ML robustness ecosystem.Its practical motivation is to provide evidence about which tools remain actively maintained and which repository-level signals indicate sustainability risk.

4 Methodology

The study uses an observational repository-mining design combining systematic GitHub discovery, automated artifact extraction, and manual revision and classification. It analyzes open-source Python-based ML robustness repositories using data bounded by January 21, 2026, yielding a final dataset of 28 tools.

  • Study design: The methodology comprises repository discovery, repository mining, and manual revision and classification using publicly observable GitHub artifacts.All metadata, issue, pull-request, commit, and status analyses share a common collection date.
  • Study design: Annual trend analyses use complete calendar years 2019-2025 and exclude partial 2026 data.This restriction follows a repository-mining practice of analyzing complete temporal units.
  • Repository discovery: The search began with 11 established robustness tools and added a complementary GitHub search based on a 43-term vocabulary derived from tool and repository language.Search terms covered general robustness, adversarial ML, framework- or organization-specific expressions, and tool or package identifiers.
  • Repository discovery: The study limited discovery to Python-based repositories and used both repository-level and code-level GitHub REST API searches.Repository results were sorted by stars with up to 10 retained per term, while code searches retained up to 10 results under GitHub’s default relevance ranking.
  • Candidate screening: Repositories were retained only if they were publicly accessible, reusable software with a stated ML robustness-assessment purpose, sufficient documentation, and an OSI-approved license.Repositories failing any criterion were excluded before metric extraction.
  • Candidate screening: 334 unique candidate repositories remained after deduplication and exclusion of repositories already in the Seed Toolset.Candidates were combined by GitHub repository identifier before manual screening.
  • Repository mining: The researchers automatically collected repository metadata, ownership, project attributes, and activity metrics through the GitHub REST API.Activity measures included commits, contributors, stars, forks, watchers, issues, pull requests, and first- and last-commit timestamps.
  • Manual revision and classification: Manual inspection removed false positives and classified retained tools using repository artifacts, documentation, and linked publications.Exclusions included tutorials, non-reusable reproductions, isolated scripts, checkpoints or datasets without implementations, forks or duplicates, and unrelated software.

5 Findings

The study identified 28 robustness tools and found substantial variation in their scope, community engagement, and maintenance activity. Recent development was concentrated in a small subset, with five repositories active, 22 inactive, and one archived on January 21, 2026.

  • 5.1 ML Robustness Tool Dataset (RQ1): 20 of 28 tools (71.4%) had an associated research paper, while eight had none identified.
  • 5.1 ML Robustness Tool Dataset (RQ1): 28 robustness tools were identified, with most academic, Python-based, and focused on deep-learning models.Traditional machine learning, federated learning, graph learning, and multimodal vision–language models were less represented.
  • 5.2 Social Engagement: Sustained pull-request activity was concentrated in relatively few projects, while many repositories showed sporadic or short-lived contribution patterns.ART showed continuous activity from 2019 through 2024, whereas some tools had PR engagement confined to one or two years.
  • 5.2 Social Engagement: Community engagement was concentrated in a small subset of tools, especially ART, CleverHans, Foolbox, TextAttack, AdvBox, and PromptBench.Repository origin and age did not fully explain engagement variation.

6 Discussion

The findings show that robustness-tool sustainability depends on more than visibility or project age: maintenance must keep tools compatible with a changing ML ecosystem. The study therefore frames robustness tools as evolving software dependencies requiring ongoing assessment.

  • Visibility and Maintenance: High visibility does not guarantee continued maintenance: CleverHans had substantial historical engagement but was inactive under the 180-day commit-recency criterion.ART combined high stars, watchers, forks, and pull-request activity with sustained repository participation.
  • Compatibility and Upkeep: 54.41 days was the mean resolution time for API-related records, the longest duration in the keyword-based maintenance analysis.Backward-compatibility records also showed relatively long resolution times, whereas routine changelog and refactoring records were typically resolved more quickly.
  • Compatibility and Upkeep: Compatibility work is central to sustainability because tools must track changing ML framework APIs, dependencies, model interfaces, and evaluation workflows.Maintenance records frequently involved API changes, dependency updates, integration work, refactoring, deprecation handling, and backward compatibility.
  • Project Longevity: Project age alone is not a reliable indicator of sustainability because older tools can become inactive while newer tools can show recent maintenance.Longevity should be interpreted as one part of a repository’s trajectory, alongside maintenance attention, compatibility, and current evaluation needs.
  • Implications: Practitioners should evaluate maintenance state together with functional coverage and treat robustness tools as versioned, evolving dependencies in AI assurance documentation.Stars, forks, watchers, citations, and publication status indicate visibility or historical importance but do not guarantee current maintenance.

7 Threats to Validity

The study’s validity is constrained by its publicly observable GitHub sample, repository-level proxies, small unequal comparison groups, and imperfect status measures. These constraints limit generalization and require cautious interpretation of engagement, maintenance, and survival estimates.

  • Construct Validity: Cumulative engagement measures were not fully normalized by repository age, so comparisons between substantially different release years require caution.Older repositories have had more time to accumulate stars, watchers, forks, issues, and pull requests.
  • Construct Validity: Repositories differ substantially in functional scope, and small category sizes prevented reliable subgroup comparisons.Observed engagement and maintenance differences may therefore partly reflect tool scope rather than maintenance quality alone.
  • Internal Validity: The 180-day commit rule measures observable default-branch activity but cannot distinguish abandoned projects from stable tools maintained through other channels.The inactive label is not definitive evidence that software is unusable or abandoned.
  • External Validity: The study covers public GitHub projects and predominantly Python-based tools, so findings may not generalize to closed-source, non-GitHub, or non-Python tooling.
  • External Validity: The dataset is a systematically curated collection of publicly discoverable tools rather than an exhaustive census of the robustness-tool ecosystem.The search may have missed tools using different terminology or distribution channels.
  • Conclusion Validity: The Mann-Whitney comparisons involve only five active and 22 inactive repositories, limiting statistical power and stability of estimated group differences.The authors interpret significance as exploratory association rather than evidence of a large or causal effect, and call for replication on larger samples.

8 Conclusion

This repository-mining study examines 28 open-source ML robustness tools using observable engagement, maintenance, and longevity measures. Engagement and recent development were concentrated in a small subset, while activity differences and project survival varied across repositories.

  • Study scope and contributions: 28 open-source ML robustness tools were examined through repository mining, covering functional scope, observable GitHub engagement, maintenance activity, and project longevity.The study adapts established repository-mining methods and provides a curated dataset, statistical analyses, and a replication package.
  • Repository status: Engagement and recent development activity were concentrated in a small subset of repositories.The conclusion reports this concentration alongside uneven sustainability across the ecosystem.
  • Repository status: Five repositories were active, 22 inactive, and one archived on January 21, 2026, under the 180-day commit-recency rule.The inactive label indicates no recent default-branch commits and does not necessarily mean unusability or permanent abandonment.
  • Maintenance activity: Six of 12 maintenance features remained significantly different between active and inactive repositories after Holm correction.Unadjusted Mann-Whitney U tests identified differences in nine features; six remained significant after correction.
  • Maintenance activity: Maintenance-related records frequently concerned API changes, changelog updates, deprecation, refactoring, integration, and backward compatibility.API- and backward-compatibility-related records had relatively long observed resolution times, although categories overlapped and distributions were strongly skewed.
  • Project longevity: The estimated median time to 180-day inactivity was 40.63 months, while the estimated probability of remaining active fell from 92.44% after one year to 30.42% after five years.The Kaplan-Meier analysis covered 27 non-archived repositories, with 22 observed inactivity events and five right-censored active repositories.
  • Implications: Robustness tools should be managed as evolving dependencies rather than one-time evaluation artifacts.Teams should record the exact version or commit, preserve execution environments, monitor compatibility and maintenance signals, and prepare for upstream inactivity.
Loading 2608.28396v1…