PaperlayerView exact revision on arXiv

Source-linked AI summary

Visible but Not Yet Curatable: Characterizing the Curatability of Compact and Derived Open LLM Artifacts

Yiyi Lu, Yilai Qian, Yucheng Jin

arXiv:2608.28819v1cs.HCcs.DL

TL;DR

Open LLM artifacts are distributed across scholarly and technical records, and prior work offers little collection-scale evidence about whether those records collectively support their curation. The paper defines record-level curatability and measures it across repositories and papers, finding that useful signals are common but coordinated evidence is not. It derives a minimal record design and complementary responsibilities for hubs, indexes, and libraries.

  • Problem

    Prior work often examines papers, repositories, model cards, datasets, or platform metadata in isolation, leaving little collection-scale evidence on whether they collectively support curation of open LLM artifacts.

    Source: §1 · pp. 1–2

  • Method

    The study defines curatability through four evidence dimensions and measures them by linking public repository metadata, paper-side links, provenance signals, release information, and cross-platform evidence.

    Source: §1 · p. 2

  • Results

    90.7% of paper records contain a useful curation signal, but only 18.1% combine usable provenance with executable release assets and 6.1% provide coordinated release-package evidence with direct paper–repository alignment.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study derives a minimal seven-field curatable record and complementary responsibilities for model hubs, scholarly indexes, and digital libraries.

    Source: §1 · p. 2

    The study measures public-record sufficiency, not model quality, adoption, complete genealogy, executable reproducibility, or preservation success.

    Source: §7 · p. 10

Abstract

from arXiv · show

Open Large Language Model (LLM) research increasingly produces compact and derived artifacts, such as adapters, quantized checkpoints, merged models, and distilled variants, that are distributed across papers, model hubs, model cards, code repositories, and release statements. Although these artifacts are publicly visible, digital libraries often lack sufficient evidence to identify, preserve, and cite them as coherent scholarly objects. We introduce a framework that conceptualizes curatability as a record-level property of distributed scholarly records and operationalizes it through four evidence dimensions: artifact identity, scholarly linkage, upstream evidence, and release assets. Guided by this framework, we conduct the first collection-scale characterization of open LLM curatability using a May 2026 snapshot of 191,375 public Hugging Face repositories and a core corpus of 2,214 scholarly papers. Our results reveal a pronounced visibility-to-curatability funnel. While 90.7% of paper records contain at least one useful curation signal, only 18.1% combine usable upstream evidence with concrete release evidence, and only 6.1% provide sufficiently coordinated evidence to support high-curatability records. Based on these findings, we derive a minimal seven-field curatable record and complementary responsibilities for model hubs, scholarly indexes, and digital libraries, providing practical guidance for improving the preservation and bibliographic control of open LLM artifacts.

1 Introduction

Open LLM artifacts are relational research objects whose identity and context are dispersed across public scholarly and technical records. This study frames their curation as a question of whether those records provide coordinated evidence, then measures that evidence at collection scale.

1 Introduction

  • Open LLM artifacts are relational objects whose functionality, provenance, and scientific meaning depend on explicit links to upstream models, resources, release assets, and publications.Their evidence is dispersed across papers, repositories, model cards, platform metadata, version histories, and release notes, complicating identification and citation.Source: §1 · p. 1
  • Prior work often examines papers, repositories, model cards, datasets, or platform metadata separately, leaving little collection-scale evidence on their combined support for artifact curation.Digital libraries need to preserve heterogeneous computational resources for transparency, reproducibility, and long-term accessibility.Source: §1 · pp. 1–2
  • The paper introduces curatability as a record-level measure based on artifact identity, scholarly linkage, upstream model evidence, and release assets.Its pipeline analyzes a May 2026 snapshot of 191,375 public Hugging Face repositories and a core set of 2,214 papers; the study also derives a minimal seven-field record and guidance for hubs, indexes, and libraries.Source: §1 · p. 2, Figure 1 · p. 2
  • 90.7% of paper records expose at least one useful metadata signal, but only 18.1% combine usable provenance with executable release assets and 6.1% provide coordinated release-package evidence and direct paper–repository alignment.The findings locate the main preservation challenge in fragmented context rather than absence of publicly released artifacts.Source: §1 · p. 2

2 Curatability Framework

The framework treats curatability as whether distributed public records collectively provide sufficient evidence for digital library curation. It evaluates that evidence through four dimensions while limiting assessment to public metadata interpreted according to a tiered hierarchy.

2 Curatability Framework

  • Curatability assesses whether distributed public records collectively provide sufficient evidence for digital library curation, rather than evaluating papers or repositories in isolation.It builds on principles including FAIR research objects, software citation, provenance models, and model documentation [17] [27].Source: §2 · p. 2
  • The four evidence dimensions are artifact identity, scholarly linkage, upstream provenance, and publicly available release assets.They cover identifying the target artifact, linking papers and repositories, tracing upstream relationships, and assessing assets such as weights, code, scripts, and evaluation resources.Source: §2 · p. 2
  • A record is curatable only when the dimensions jointly support identification, attribution, and preservation, using public metadata interpreted through a tiered evidence hierarchy.The hierarchy distinguishes repository metadata, paper–repository links, provenance statements, model cards, and release assets [13] [17].Source: §2 · p. 2

3 Related Work

Prior work supplies principles for documenting computational objects, describing releases, and representing provenance. This paper focuses instead on whether evidence across those records collectively supports curation of open LLM artifacts.

3.1 Computational Research Objects

  • Digital-library frameworks emphasize rich metadata for preserving computational research objects, but often assume the object to be curated is already identifiable.Open LLM contributions may comprise multiple derived artifacts distributed across papers, model hubs, repositories, and documentation.Source: §3.1 · p. 3

3.2 Documentation and Release of Open LLM Artifacts

  • Reproducibility and documentation work stresses coordinated releases and structured descriptions, while repository studies report variation in documentation practices.This analysis asks whether paper, repository, upstream-relation, and release-asset evidence co-occur as a curatable cross-system record.Source: §3.2 · p. 3

3.3 Provenance and Cross-platform Scholarly Records

  • Provenance models describe how objects are created and derived, but open LLM lineage evidence is often distributed across records or conveyed through contextual references.The paper extends typed-relation approaches by measuring whether scholarly linkage co-occurs with usable upstream and release evidence, using the linked cross-platform scholarly record as its unit of analysis.Source: §3.3 · pp. 3–4

4 Data and Measurement Design

The study assembles public repository, paper, and release traces, then converts them into distinct evidence fields and rule-based curatability thresholds. Its measurement design keeps record sources and units separate while using audits to calibrate judgment-sensitive coding boundaries.

4.1 Public Traces

  • The scoped May 2026 collection contains 191,375 public Hugging Face repositories and 2,214 eligible papers, alongside public paper, model-card, and repository code or release traces.The repository estimates describe a query-based snapshot, and eligible papers center on compact or derived models.Source: §4.1 · p. 4
  • Percentages use repository- or paper-level records; artifact-level claims are record-supported claims, not a census of unique artifacts.A paper may describe several artifacts, and a repository may contain several package forms.Source: §4.1 · p. 4

4.2 Evidence Pipeline

  • The pipeline preserves source context, extracts candidate evidence, normalizes and resolves records, then separates native, rule-derived, and audit-bound fields.LLM-extracted fields from unstructured paper text remain candidates until rule-based validation.Source: §4.2 · p. 4

4.3 Typed Evidence Fields

  • Artifact role identifies the object, while package form describes its distribution, distinguishing such roles as adapters or merges from adapter-only or multi-asset packages.The role categories also include base checkpoints, fine-tunes, and quantized or deployment packages.Source: §4.3 · pp. 4–5
  • L is a bibliographic relation gate: 0 means none, 1 one direct signal, and 2 corroboration; fuzzy family or size matches do not qualify.Source: §4.3 · p. 5
  • P=2 supports collection-level grouping and P=3 explicit lineage; R=2 means code or recipe, R=3 an asset, and R=4 a multi-asset package or full recipe.Release scores indicate visible evidence for inspection and preservation triage, not executable reproducibility or role-specific sufficiency.Source: §4.3 · p. 5

4.4 Calibration and Curatability Status

  • Audits reviewed 300 role, 200 upstream-evidence, and 50 release cases with single-expert decisions; they calibrate coding boundaries, not corpus prevalence.The study also tested query-bucket scope, linkage-signal composition, and alternative thresholds.Source: §4.4 · p. 5
  • Discovery-ready needs any nonzero signal; operational P+R needs P≥2 and R≥2; high-curatability adds L≥1 and R≥3, while preservation-review raises R≥4.Identity is assessed separately because role and package form are nominal; P and R are ordinal ladders, while L is a relation gate.Source: §4.4 · pp. 5–6

5 Results

The results assess artifact roles, scholarly linkage, upstream evidence, and release packages as distinct curation signals, then examine how often they co-occur. Across these dimensions, public records often expose useful traces, but coordinated evidence for curation is much less common.

5.1 RQ1: Can Repository Records Identify Artifact Roles?

  • The 300-repository audit finds 85.3% overall strict role support; fine-tune, adapter/LoRA, and quantized/deployment reach 100.0%, versus 71.7% for base and 55.0% for merge/mixed.For curation, records should retain both primary artifact role and package form rather than collapse distinct artifacts into a generic model label.Source: §5.1 · p. 6, Figure 3 · p. 7

5.2 RQ2: What Bibliographic and Linkage Evidence Is Visible?

  • Matched paper-to-Hugging Face evidence covers 5.6% of analysis-set records, despite 30.6% containing code-hosting links and 10.5% containing Hugging Face links.Visible links may point to code, dependencies, examples, or auxiliary resources rather than the model-hub record introduced by the paper.Source: §5.2 · p. 6
  • Repository fields support discovery, but curation requires typed scholarly relations distinguishing introduced artifacts, dependencies, baselines, background citations, and official releases.Task descriptors, family labels, and upstream fields are more visible than scholarly references or domain descriptors.Source: §5.2 · p. 6

5.3 RQ3: What Upstream or Family-Level Evidence Can Public Records Support?

  • Inventory-backed family signals cover 45.6% of records, while 28.0% of audited rows have strict evidence for one primary upstream family and 79.0% have usable family-level support.Family-level evidence can support collection grouping, but does not by itself support a provenance claim.Source: §5.3 · p. 7, Figure 4 · p. 7

5.5 RQ5: How Often Do the Required Evidence Fields Co-occur?

  • Only 136 of 2,214 records (6.1%) meet the high-curatability threshold, with the sharp drop concentrated at the asset-level release threshold.The operational provenance-plus-release region contains 401 records; the further decline reflects the requirement for more concrete release assets.Source: Figure 5 · p. 8, §5.5 · p. 8

5.4 RQ4: What Release Package Evidence Is Visible?

  • Release evidence ranges from statements to assets: 35.8% of records contain concrete code or asset evidence, while 4.5% provide multiple assets or a full recipe.The authors recommend recording release as a package, since a release statement or scripts may omit weights, data, or evaluation assets.Source: §5.4 · p. 8, Figure 5 · p. 8

5.5 RQ5: How Often Do the Required Evidence Fields Co-occur?

  • The funnel is an evidence-integration problem: typed record fields needed for citation, attribution, reuse assessment, and preservation triage remain uncoordinated.Table 5 presents the minimal curatable record as a coordination contract across existing systems, not a new centralized registry.Source: §5.5 · pp. 8–9, Table 5 · p. 9

6 Discussion

The discussion frames open LLM artifacts as visible but not yet well-formed curation records, then proposes a minimal record and distributed infrastructure responsibilities to coordinate their evidence.

6.1 From Visibility to a Minimal Curatable Record

  • The minimal record uses seven fields to resolve identity, scholarly-linkage, upstream-provenance, release-asset, and governance ambiguities identified in the funnel and audits.Artifact role and package form distinguish what an artifact is from how it is distributed; provenance tiers distinguish stronger derivation claims from family mentions.Source: §6.1 · p. 9
  • The record coordinates joinable public traces while retaining their sources and evidentiary strength, rather than creating a centralized registry or heavier publication burden.The intended uses include citation, attribution, reuse assessment, and preservation triage.Source: §6.1 · p. 9
  • Family names can indicate targets, parents, teachers, baselines, or background, while code links may expose scripts without weights, so records should retain relation types and evidence strength.A worked example applies the seven fields to MobileLLM-R1-360M using three evidence bundles for identity and paper relation, upstream evidence, and release assets.Source: §6.1 · p. 9, Table 6 · p. 10

6.2 Infrastructure Responsibilities and Design Implications

  • Model hubs should maintain artifact roles, package forms, release assets, and license/access fields; paper authors and scholarly indexes should type paper–repository relations.Upstream relations require both hub declarations and paper context, while libraries should preserve provenance tiers and curation-readiness status across sources.Source: §6.2 · p. 9
  • Digital libraries should expose structured release-package fields and preserve provenance evidence tiers rather than treating openness as a single access flag.The proposed interventions distinguish code, scripts, weights or checkpoints, adapters, datasets, evaluation assets, recipes, and environment details; FAIR packaging work [27] and the Model Openness Framework [32] make related points.Source: §6.2 · pp. 9–10

7 Scope and Limitations

The study characterizes public-record evidence sufficiency, with its repository analysis bounded to a May 2026 Hugging Face snapshot and its paper-level analysis to 2,214 records.

7 Scope and Limitations

  • The study measures public-record sufficiency, not model quality, adoption, complete genealogy, executable reproducibility, or preservation success.Paper-level results use the 2,214-record analysis set unless otherwise stated.Source: §7 · p. 10
  • The repository collection is a scoped May 2026 Hugging Face snapshot.The measured roles, package forms, provenance tiers, and release-package levels derive from visible records calibrated through audits.Source: §7 · p. 10
  • Private training logs, hidden checkpoints, unreported assets, and developer knowledge remain outside the study’s visible-record scope.Release tiers measure visible asset evidence rather than role-specific package sufficiency.Source: §7 · p. 10

8 Conclusion

The paper characterizes open LLM artifact curatability and finds a visibility-to-curatability funnel, motivating a minimal record that preserves evidence context and strength.

8 Conclusion

  • Only 6.1% of paper records provide sufficiently coordinated evidence to support provenance-aware curation, although 90.7% contain at least one useful curation signal.The analysis covers 191,375 Hugging Face repositories and 2,214 scholarly papers.Source: §8 · p. 10
  • The study introduces a record-level framework and collection-scale analysis of curatability for compact and derived open LLM artifacts.The proposed minimal record preserves artifact identity, scholarly linkage, provenance, release assets, evidentiary strength, and source context.Source: §8 · p. 10
  • The conclusion identifies evidence coordination across distributed scholarly records, rather than resource availability, as the primary preservation challenge for open LLM artifacts.It calls for digital libraries to coordinate evidence across the scholarly ecosystem rather than cataloging repositories or publications individually.Source: §8 · p. 10

9 Disclosure of AI Use

Generative AI tools assisted with manuscript language editing and code drafting, while the authors retain responsibility for the study’s scientific content.

9 Disclosure of AI Use

  • Generative AI tools assisted with language editing and improving manuscript clarity.The authors state that AI tools also supported code drafting and software development.Source: §9 · p. 10
  • The authors reviewed, tested, and validated all AI-generated code before using it in the study.Source: §9 · p. 10
  • The authors developed and verified the scientific content, analyses, interpretations, and conclusions and take full responsibility for them.Source: §9 · p. 10
Loading 2608.28819v1…