Source-linked AI summary
A Framework for the Systematic Review of ML Assets in AI Registries
Alexandra González, Quim Motger, Xavier Franch, Silverio Martínez-Fernández
TL;DR
ML asset selection in AI registries lacks systematic retrieval methods, despite the assets’ widespread use and the registries’ heterogeneous, evolving structure. The paper adapts systematic-review phases to those registries and proposes a structured retrieval framework. It presents traceable, versioned retrieval as a way to make asset selection more transparent and reproducible.
Problem
ML asset selection lacks systematic retrieval practices, while AI registries are dynamic and heterogeneous and their asset relationships can be implicit.
Source: §1 · pp. 1–2Method
The paper adapts systematic-review principles to AI registries, treating ML assets as first-class analytical units and structuring retrieval around planning, conducting, and documenting.
Source: §1 · p. 2, §4 · p. 10Results
The paper proposes a structured framework for ML asset retrieval that supports more transparent selection than ad hoc practices.
Source: §5 · p. 10Takeaways & Limitations
Versioning retrieval makes asset identification, filtering, and justification a verifiable and repeatable component of empirical research.
Source: §4 · p. 10The framework is an empirically informed design, not yet a fully validated method.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Background: Modern software systems increasingly rely on Machine Learning (ML) assets (i.e., pre-trained models, datasets, benchmarks) for building, evaluating, and integrating ML-based systems. However, current exploration, selection and reuse practices of ML assets are not supported by systematic retrieval methodologies comparable to those used in traditional evidence synthesis. Consequently, in practice, ML asset selection is often presented as a settled design decision, supported by informal justification rather than a traceable, evidence-based, and updatable selection process. Aims: This paper explores how systematic review methods can support ML asset retrieval. In doing so, we aim to make their selection transparent and reproducible, grounded in explicit evidence, and ultimately better suited to its intended use. Method: We analyze established systematic review practices from scientific literature and adapt their phases (i.e., planning, conducting, and documenting) to Artificial Intelligence (AI) registries, treating ML assets as first-class units of analysis. The resulting framework integrates registry-aware search strategies, cross-registry schema alignment, and dependency-driven ML asset exploration. Results: We conceptualize ML asset retrieval as a systematic and reproducible process rather than an ad hoc activity, and propose a framework for structured ML asset discovery. \textbf{Conclusions:} This work illustrates how systematic review principles can be extended beyond scientific literature to support evidence synthesis over evolving AI registries.
1 Introduction
ML assets are widely used, but their selection is often ad hoc and poorly documented in AI registries that differ from the stable, standardized corpora assumed by traditional reviews. The paper proposes adapting systematic-review methods to these evolving, heterogeneous sources.
1 Introduction
- Software workflows use pre-trained models, datasets, and benchmarks to build, evaluate, and integrate ML-based systems.Researchers and practitioners find these assets in open registries such as Hugging Face and Kaggle.Source: §1 · p. 1
- Asset selection remains ad hoc, while documentation initiatives do not provide methodological guidance for retrieval across registries with heterogeneous metadata and traceability.Registry signals and links between models and datasets can be explicit or implicit, and the paper connects this network to production reliability and dataset bias.Source: §1 · pp. 1–2
- Traditional evidence-synthesis methods offer transparent, reproducible review procedures but do not directly fit dynamic AI registries with heterogeneous structures and evolving dependencies.The paper proposes adapting systematic-review principles to these sources.Source: §1 · p. 2
2 Related Work
Prior empirical studies select ML assets using practical, study-specific criteria, while ecosystem research analyzes artifact relationships rather than using them to guide systematic retrieval. The paper identifies a need for structured, transparent, and reproducible discovery.
2 Related Work
- Empirical studies often choose datasets, models, and benchmarks through ad hoc filters or pragmatic criteria rather than reproducible retrieval strategies grounded in registry structure.Examples include selecting models by size, type, release date, affiliation, training-data information, and popularity.Source: §2 · pp. 2–3
- AI ecosystem studies map artifact networks and metadata inconsistencies, but primarily analyze those structures rather than using them to support systematic retrieval.The cited work includes a cross-source knowledge graph, LLM ecosystem analysis, and cross-repository organization.Source: §2 · p. 3
- Together, these gaps motivate methodological support for structured, transparent, and reproducible ML asset discovery in evolving AI ecosystems.The paper frames this as a missing approach spanning ML asset use and AI ecosystem analysis.Source: §2 · p. 3
- Without explicit selection criteria, asset rationale is opaque, reproducibility and reuse are difficult, and repeating retrieval from evolving catalogs is effortful and error-prone.The paper also identifies the risk that ad hoc discovery may not yield the best assets for a task.Source: §2 · p. 3
3 The Review Process
The review process adapts systematic-review phases to ML assets by combining explicit planning, registry-aware retrieval and analysis, and traceable documentation. It also specifies periodic updates within a fixed review scope.
3.1 Plan the Review
- Plan the review by establishing its need, specifying asset-focused search questions, and defining and validating a protocol before execution.The protocol sets registries, access and search strategies, selection boundaries, quality assessment, extraction, and synthesis; pilot searches can inform refinement.Source: §3.1 · pp. 4–5
3.2 Conduct the Review
- Identify candidates with queries tailored to each registry’s search and metadata mechanisms, using APIs, snapshots, web interfaces, or hybrid retrieval as available.API omissions and rate limits can constrain retrieval, and documentation-based deduplication addresses mirrored or republished assets.Source: §3.2 · p. 6
- Select assets against protocol criteria by inspecting registry cards, since asset identifiers alone do not establish relevance and available signals vary across registries.Selection can consider task alignment, popularity signals, and maintenance activity; popularity requires caution because newer assets have had less time to accumulate engagement.Source: §3.2 · pp. 6–7
- Extend primary retrieval through backward and forward snowballing across asset provenance and reuse relationships, then apply the protocol’s selection criteria to newly found assets.Dependencies may appear in structured metadata or unstructured descriptions, so textual inspection may be needed.Source: §3.2 · pp. 7–8
- Assess evidence reliability and validity, then extract registry metadata and descriptive information into a unified schema while recording missing attributes.Quality assessment considers verifiability, methodological detail, and applicability; missing provenance or benchmarks may inform later interpretation.Source: §3.2 · pp. 8–9
- Synthesize extracted data by aggregating, normalizing, and categorizing it to answer the search questions; stratified sampling can represent an otherwise unmanageably large corpus.The sampling approach is intended to avoid bias toward dominant assets while retaining ecosystem variety.Source: §3.2 · p. 9
3.3 Document the Review
- Document the rationale, search strategy, selection criteria, deviations, and asset mappings, and preserve versioned snapshots of registry state for comparison over time.Because assets and their metadata change, reporting is framed as a sequence of reproducible snapshots rather than a single static extraction.Source: §3.3 · p. 9
3.4 Need of Iteration
- Periodically repeat conducting and documenting the review while keeping the validated plan fixed; adding registries or changing the protocol requires a new review instance.The loop refreshes retrieval and records snapshots to track changes within a consistent methodological scope.Source: §3.4 · p. 9
4 Discussion
The framework adapts evidence-synthesis principles to AI registries by treating ML assets, rather than documents or repositories, as interconnected units of analysis. It links traceable, versioned retrieval to reproducibility in empirical software engineering.
4 Discussion
- The framework shifts the unit of analysis from documents or repositories to ML assets and their dependencies in registries with sparse, inconsistent, heterogeneous metadata.The paper positions this as an adaptation of existing evidence-synthesis principles to challenges that traditional reviews and repository mining do not fully capture.Source: §4 · p. 10
- The framework extends multivocality beyond textual evidence to ML assets whose evaluation signals may not be peer-reviewed.The comparison is to Multivocal Literature Reviews, which incorporate grey literature alongside published literature.Source: §4 · p. 10
- Versioning asset retrieval makes identification, filtering, and justification part of a verifiable and repeatable empirical research process, extending transparency beyond code.The paper connects this to reproducibility in empirical software engineering.Source: §4 · p. 10
5 Conclusion & Future Work
The paper proposes a framework for systematic ML asset retrieval and presents it as an empirically informed design rather than a fully validated method. Future work will formalize and evaluate the approach and explore representing AI ecosystems as knowledge graphs.
5 Conclusion & Future Work
- The framework structures ML asset retrieval and supports more transparent selection than ad hoc practices.It accounts for the heterogeneous, dynamic, and weakly standardized nature of AI ecosystems.Source: §5 · p. 10
- Future work will formalize the methodology into a more operational framework by mapping its steps against existing literature.The mapping will identify established, partially addressed, and implicit practices.Source: §5 · p. 10
- The approach remains an empirically informed design rather than a fully validated method.Future work will investigate evidence of effectiveness, consistency across heterogeneous registries, and comparison with ad hoc selection practices.Source: §5 · pp. 10–11
- The authors envision a knowledge graph explicitly representing AI ecosystem models, datasets, and their relationships.They propose that this representation would enable ecosystem-level modeling and make retrieval strategies such as snowballing more systematic and straightforward.Source: §5 · p. 11
6 Data Availability
The article reports that it collected, generated, or analyzed no research artifacts.
6 Data Availability
- No datasets, code, or other research artifacts were collected, generated, or analyzed for this article.Source: §6 · p. 11