PaperlayerView exact revision on arXiv

Source-linked AI summary

A Private IPFS Data Sanctuary for Verifiable Digital Collection Objects

Joachim Snellings, Jasper Funk-Smit, Margot Belot, Eran Wolff, Sebastian Baldauf, Lisa Pfeiffer, Frederik Berger

arXiv:2610.07970v1cs.DCcs.NI

TL;DR

Scientific acquisition data need recovery that does not depend on one workstation or storage path. The paper evaluates a private seven-node IPFS sanctuary; its prototype recovered real raw folders by CID, ingested model weights, and recorded sustained successful local block access.

  • Problem

    During staging, acquisition data can depend on one workstation or storage path, creating a recoverability problem even while the physical specimen remains curated.

    Source: §1 · p. 2

  • Method

    The study evaluates a private, institution-controlled IPFS staging layer combining content-derived identity, replicated availability, and restricted peer participation.

    Source: §1 · p. 2

  • Results

    The seven-node prototype recovered two real raw-acquisition folders by CID, ingested a 156.1 GB model-weight folder, and logged successful local access in all 27,616 valid block-stat probes.

    Source: §3.2 · p. 3, §6 · p. 4

  • Takeaways & Limitations

    The collection object can retain its established UID while CIDs distinguish measurement states, with replication across failure domains and block addressing supporting distributed custody and a federated GLAM model.

    Source: §6 · p. 4

    The retrieval rates are single historical logical/materialization observations rather than physical throughput, and cross-institutional federation and full-zone restoration were not experimentally demonstrated.

    Source: §5 · p. 4

Abstract

from arXiv · show

Background: Galleries, libraries, archives and museums (GLAM) increasingly produce scientific collection data whose digital state must remain identifiable, recoverable, and verifiable after acquisition. A staging architecture should provide redundancy without requiring every measurement to depend on one workstation or storage path. Methods: We evaluated a private seven-node InterPlanetary File System (IPFS) deployment at the Museum f\"ur Naturkunde Berlin. Five Raspberry Pi 5 edge peers and two archive-class peers contributed NVMe-backed repositories across two wired zones linked by a shared wireless path. Two real raw-acquisition folders were recovered by content identifier (CID), a large machine-learning model-weight folder was ingested, and sustained telemetry from two edge and two archive nodes was compared. Results: Raw acquisition A (6.20 GB) materialized in 34.0 s (182 MB/s effective logical rate) and raw acquisition B (5.19 GB) in 14.092 s (368 MB/s). A 156.1 GB model-weight folder was recursively added in approximately 300 s (520 MB/s logical ingest). Four sustained telemetry series comprised 857.3 logged node-hours and 27,616 valid block-stat probes, all reporting successful local block access. Conclusion: The prototype demonstrates a practical institution-local distributed file system (DFS) for content-addressed acquisition staging. Museum-specific object identifiers can remain stable at the collection-object level while each imported measurement automatically receives a content-derived state identity. Replication intentionally distributes identical blocks across failure domains, while content addressing enables reuse of unchanged blocks within repositories. This combination supports verifiable digital collection objects and a federated storage model for collaborating GLAM institutions.

1 Introduction

Scientific acquisition data need a recoverable staging layer that does not depend on one workstation or storage path. This work evaluates a private IPFS sanctuary and asks whether content-derived measurement identities can coexist with museum object identifiers while supporting recovery and redundancy.

1 Introduction

  • During acquisition, quality control, reconstruction, and staging, dependence on one workstation or storage path creates a recoverability problem even when the specimen remains curated.The data include raw image sets, CT volumes, label images, meshes, and machine-learning artefacts.Source: §1 · p. 2
  • A private, institution-controlled IPFS staging layer combines content-derived identity, replicated availability, and restricted peer participation; its evaluation is specific to the museum topology, not a universal transfer-rate claim.IPFS addresses imported blocks through CIDs rather than only mutable server paths.Source: §1 · p. 2
  • The collection object keeps its museum UID while each imported measurement receives a root CID, distinguishing repeated digital states without another manually assigned measurement identifier.The research questions also ask whether this identity supports verifiable states and deliberate redundancy across storage zones.Source: §1 · p. 2

2 Methods

The methods describe a seven-node, NVMe-backed private IPFS cluster, its restricted two-zone network and telemetry, and three content operations used to evaluate recovery and ingestion. Reported logical rates are explicitly distinguished from physical network throughput.

2.1 Seven-node prototype

  • Seven physical nodes combined five Raspberry Pi 5 edge peers and two archive-class peers, with approximately 1.0–1.5 TB configured per peer and 7–10.5 TB nominal raw capacity before overhead and reserve.The edge peers each had 16 GB RAM and 2 TB NVMe; archive peers had at least eight CPU cores, 32 GB RAM, and 2 TB NVMe.Source: §2.1 · p. 2, Table 1 · p. 2

2.2 Topology, private swarm, and telemetry

  • Five edge peers shared one wired LAN and two archive peers another, joined by shared WLAN; a private swarm key and Cluster secret restricted participation, while read-only HTTP gateways exposed published content.The gateways sat behind an Nginx reverse proxy and did not grant IPFS or Cluster administrative access.Source: §2.2 · p. 2
  • Telemetry from two edge peers and both archive peers tracked node state and queried known local blocks, measuring local content access rather than peer-to-peer latency.Public bootstrap peers were removed; local mDNS discovery and DHT routing were used.Source: §2.2 · p. 2

2.3 Scientific workloads and evidence boundary

  • Three application-level operations used two real museum raw-acquisition folders and a larger machine-learning model-weight folder; logical size divided by materialization time yields an effective rate, not physical Ethernet or WLAN throughput.Target occupancy, provider count, and synchronized interface byte counters were not retained for historical retrievals.Source: §2.3 · p. 2

3 Results

Two real raw-acquisition folders were recovered by CID and a large model-weight folder was ingested; sustained telemetry recorded successful local block access. The logical rates do not establish physical network throughput, and the depicted federation is an architectural generalization.

3.1 Recovery and ingestion

  • Figure 1 depicts two-zone topology and conceptual federation across independent GLAM failure domains, not a WAN performance measurement; content identity remains independent of the serving zone.The federation layer is an architectural generalization.Source: Figure 1 · p. 3
  • 6.20 GB raw acquisition A materialized in 34.0 s at approximately 182 MB/s; 5.19 GB raw acquisition B in 14.092 s at approximately 368 MB/s; 156.1 GB model weights ingested in approximately 300 s at roughly 520 MB/s.These are single historical wall-clock observations and logical/materialization rates, not wire throughput.Source: §3.1 · pp. 2–3, Table 2 · p. 3

3.2 Sustained four-node telemetry

  • 857.3 combined logged node-hours and 27,616 valid block-stat probes were recorded; every valid probe reported successful local block access, while the asynchronous windows do not establish cluster uptime.Archive-node block-stat latencies were lower and tighter than the edge-node series; Edge-01 had a broader P90 tail despite a similar median to Edge-04.Source: §3.2 · p. 3, Table 3 · p. 3

4 Discussion

The discussion separates stable museum-object identity from content-derived identities for individual measurements, then considers how replication, block reuse, and federation affect storage and recovery. These are architectural claims with practical boundaries around persistence, networking, and unmeasured performance.

4.1 From museum object to verifiable digital measurement state

  • A museum UID remains attached to the physical object, while each imported measurement receives a distinct root CID that lets reconstruction reference the accepted acquisition instead of a mutable folder name.A failed acquisition can remain preserved without serving as the reconstruction source.Source: §4.1 · p. 3, Figure 2 · p. 3
  • For preservation, retain the folder root CID, relevant import parameters, and an independent fixity digest such as SHA-256.The UnixFS root can depend on directory representation and import or chunking parameters, rather than behaving as a conventional file hash.Source: §4.1 · p. 3

4.2 Redundancy and deduplication serve different purposes

  • Replication deliberately stores identical content across failure domains, while block-level reuse avoids storing another unique block when that block already exists in a repository.Reuse depends on workload and chunking; the experiment did not measure a deduplication ratio, and compressed or substantially rewritten files may share few blocks.Source: §4.2 · pp. 3–4

4.3 Federated GLAM storage

  • A shared CID can support agreed replicas across GLAM institutions without requiring participants to expose their complete repositories or use one central staging appliance.Institutions can retain local acquisition replicas and exchange agreed content, manifests, or derivatives, distributing storage responsibility incrementally; inter-institutional recovery was not benchmarked.Source: §4.3 · p. 4
  • A CID does not itself guarantee persistence: replica placement, pinning policy, credentials, and independent backup remain operational requirements.A slower WAN link affects recovery time, not the identity of recovered data; another zone can reseed peers after connectivity returns if it retains the required pinned blocks.Source: §4.3 · p. 4

4.4 Capacity and network scaling

  • The seven-node prototype configured approximately 7–10.5 TB of nominal raw capacity; aggregate LAN flows can scale until shared network or compute resources bottleneck, while WLAN peers share airtime.The historical experiment did not independently measure the WLAN ceiling.Source: §4.4 · p. 4

5 Limitations

The evaluation’s measurements and scope impose limits on how its performance and resilience results should be interpreted.

5 Limitations

  • The two raw-acquisition recovery values and model-weight ingest value are single historical observations, not repeated benchmarks.Target blockstore state, provider count, source/destination zone, and interface byte counters were not synchronized to those runs.Source: §5 · p. 4
  • The logical rates cannot be interpreted as physical network throughput, and the four sustained telemetry windows were asynchronous.Block-stat probes measure local blockstore access rather than peer-to-peer retrieval.Source: §5 · p. 4
  • The experiment measured no block-reuse ratio, and cross-institutional federation and full-zone restoration remain design extensions rather than demonstrated outcomes.Deduplication is discussed as an architectural property of content-addressed storage.Source: §5 · p. 4

6 Conclusion

The prototype combines content-derived measurement identity with distributed custody for scientific digitization in a GLAM institution. It demonstrates local acquisition recovery and model-weight ingestion, while positioning federation as a candidate for future evaluation.

6 Conclusion

  • Two real raw-acquisition folders were recovered by CID, a 156.1 GB model-weight folder was ingested, and telemetry documented prolonged operation of the local content-addressed layer.Telemetry came from two edge and two archive nodes.Source: §6 · p. 4
  • The museum object retains its established UID while each imported measurement receives a separate content-derived CID, keeping failed, repeated, accepted, and derived states distinguishable without overwriting the scientific record.Replication distributes selected states across failure domains, while block-level addressing enables reuse when identical blocks recur.Source: §6 · p. 4
  • Participating institutions could contribute storage capacity and recovery responsibility without changing the identity of shared digital collection objects.The authors position this as a candidate distributed and verifiable storage model; cost and inter-institutional recovery performance remain for future evaluation.Source: §6 · pp. 4–5

Abbreviations

The abbreviations cover the collection-data domain, storage system, and network terminology used in the paper.

Abbreviations

  • GLAM means Galleries, Libraries, Archives and Museums; NURI is the MfN URI used in its museum object-identification workflow; UID means unique identifier.These terms distinguish the institutional context and object-level identification terminology.Source: Abbreviations · p. 5
  • IPFS means InterPlanetary File System, DFS means Distributed File System, and CID means Content Identifier.Source: Abbreviations · p. 5
  • LAN and WAN mean Local Area Network and Wide Area Network; WLAN means Wireless Local Area Network, and NIC means Network Interface Controller.Source: Abbreviations · p. 5

Data, privacy, and reproducibility

The release package provides privacy-safe data and analysis sources for reproducibility while excluding operational identifiers and keeping raw logs under institutional control.

Data, privacy, and reproducibility

  • Privacy-safe content-operation and four-node telemetry tables are provided as data.Operational IP addresses, hostnames, raw PeerIDs, private swarm keys, Cluster secrets, credentials, and unreleased CIDs are excluded.Source: Data, privacy, and reproducibility · p. 5
  • Analysis and publication sources are included in the release package.The package identifies a public release repository.Source: Data, privacy, and reproducibility · p. 5
  • Reproducibility documentation specifies Kubo 0.36.0, IPFS Cluster 1.1.4, CRDT consensus, claim boundaries, and privacy controls.Institution-internal raw logs remain under local control.Source: Data, privacy, and reproducibility · p. 5
Loading 2610.07970v1…