PaperlayerView exact revision on arXiv

Source-linked AI summary

IPFS and Friends: A Qualitative Comparison of Next Generation Peer-to-Peer Data Networks

Erik Daniel, Florian Tschorsch

arXiv:2102.12737v3cs.NI

TL;DR

This survey examines emerging peer-to-peer data networks as alternatives to centralized storage and provides technical overviews of selected systems. It identifies shared building blocks, qualitatively compares designs, and derives challenges and research directions, finding varied approaches to file management, availability, and incentivization beyond broadly shared overlay structures.

  • Problem

    Decentralized peer-to-peer data networks have emerged as a response to data silos, but their technical concepts, building blocks, and differences require systematic examination.

  • Method

    The paper surveys selected networks, gives a deeper account of IPFS, reviews Swarm, Hypercore, SAFE, Storj, and Arweave, and develops a taxonomy from qualitative comparison.

  • Results

    The networks share overlay structures but use different approaches to file management, availability, and incentivization, with explicit incentives and replication or erasure coding commonly supporting availability.

  • Takeaways & Limitations

    The comparison identifies network architectures, file handling, information security, and incentivization as key building blocks for understanding and developing next-generation peer-to-peer data networks.

Abstract

from arXiv · show

Decentralized, distributed storage offers a way to reduce the impact of data silos as often fostered by centralized cloud storage. While the intentions of this trend are not new, the topic gained traction due to technological advancements, most notably blockchain networks. As a consequence, we observe that a new generation of peer-to-peer data networks emerges. In this survey paper, we therefore provide a technical overview of the next generation data networks. We use select data networks to introduce general concepts and to emphasize new developments. Specifically, we provide a deeper outline of the Interplanetary File System and a general overview of Swarm, the Hypercore Protocol, SAFE, Storj, and Arweave. We identify common building blocks and provide a qualitative comparison. From the overview, we derive future challenges and research goals concerning data networks.

I. INTRODUCTION

The paper surveys a next generation of peer-to-peer data networks that seeks to reduce centralized data silos through decentralized storage and sharing. It compares selected systems, extracts common building blocks, and identifies research challenges beyond blockchain-focused analyses.

  • Centralized cloud storage can create data silos affecting accessibility, availability, and confidentiality.
  • P2P data networks let peers directly share resources while jointly managing storage, replication, distribution, and exchange.
  • The paper studies a newer generation shaped by distributed ledgers, cryptocurrencies, Kademlia, and information-centric networking, with stronger attention to long-term availability.
  • The survey provides a comparative technical overview of selected systems and focuses on storage and content sharing independently of blockchain use.
  • It extracts differences in network organization, file lookup, decentralization, redundancy, privacy, and system purpose across the surveyed networks.

II. TECHNOLOGICAL PRECURSORS

The technological precursors establish the main mechanisms later used by next-generation P2P data networks: chunk-based exchange, structured overlays, distributed discovery, and explicit incentives.

  • The next generation builds on earlier P2P file sharing, Kademlia, and other precursor technologies while adding newer developments.
  • BitTorrent: BitTorrent distributes files through torrents that organize peers around a file and use trackers or DHTs for bootstrapping and peer discovery.
  • BitTorrent: BitTorrent splits files into chunks, and leechers become seeds after acquiring all chunks, allowing replication to expand distribution.
  • BitTorrent: Rarest-piece-first reduces chunk overlap, while tit-for-tat allocates upload bandwidth according to peer contributions.
  • Kademlia: Kademlia assigns peers identifiers, uses XOR distance to select neighbors, and organizes routing information into k-buckets arranged as a binary tree.

C. Information-Centric Networking

Information-centric networking shifts communication from locating hosts to requesting content, motivating content-based retrieval and caching while introducing new security and access-control challenges.

  • Information-centric networking: ICN changes networking from point-to-point location search toward direct content requests handled by interest and data packets.
  • Security challenges: Caching and content-focused communication require mechanisms for integrity, provenance, relevance, authentication, authorization, and privacy.
  • Self-certifying names: Self-certifying names use content hashes so clients can identify content and verify its integrity locally.
  • Self-certifying names: Self-certifying names alone do not establish an object’s provenance or origin, which may require cryptographic signatures and public-key infrastructure.
  • Access control: Once data is released, access control and revocation remain difficult; encryption can limit access but may require out-of-band key distribution.
  • Blockchains and DLT: Blockchain data structures link blocks with cryptographic hashes, while consensus secures consistency, but scalability and limited on-chain storage constrain purely DLT-based data networks.

A. General Functionality

IPFS combines content addressing, Merkle DAGs, peer discovery, provider lookup, and Bitswap for decentralized file exchange, while leaving persistence and redundancy to optional mechanisms.

  • Network functionality: IPFS uses libp2p, node identities, bootstrap nodes, Kademlia-based discovery, local discovery, encrypted connections, and swarm management.
  • File structure: IPFS splits objects into blocks identified by content identifiers and organizes them into Merkle DAGs whose root CID identifies the file.
  • File structure: Block deduplication lets file versions share unchanged blocks, so only differences need additional storage.
  • Provider lookup: The DHT locates providers for blocks, while providers periodically reannounce possession of stored content.
  • Bitswap: Bitswap exchanges blocks through want, have, and do-not-want lists and verifies received content by reconstructing its CID.
  • Availability: IPFS has no implicit repair or active replication: unpinned cached content may be garbage-collected, while Filecoin provides separate storage guarantees through markets and proofs.

B. Features and Extensions

IPFS extends content-addressed peer-to-peer storage with adaptable addressing, block identification, replication extensions, and multiple application uses. Its design combines several earlier P2P ideas while leaving privacy concerns unresolved.

  • Addressing and identification: IPFS uses multi-address and multi-hash structures to support multiple network transports and cryptographic hash functions.Multi-address encodes peer contact and transport information, while multi-hash supports different hash functions.
  • Addressing and identification: A CID identifies an IPFS block through its content hash plus metadata for hashing, encoding, and version information.
  • Extensions: IPFS Cluster adds a separate overlay that manages pinned data, configured replicas, repinning, and storage-aware node selection.IPFS itself does not ensure redundancy or availability.
  • Extensions: IPFS Cluster improves file availability without incentives, while Filecoin can incentivize others to store files.
  • Design and applications: IPFS supports blockchain-based applications, website archiving, content delivery, edge storage, and IoT storage, but can also be misused for ransomware and botnet coordination.
  • Design and applications: IPFS combines BitTorrent, Kademlia, Git, and ICN concepts, including content addressing and deduplication that may improve retrieval times and storage overhead.
  • Design and limitations: IPFS uses libp2p for peer discovery and connections and Bitswap for data exchange, while its broad protocol support complicates understanding and does not provide additional participant-privacy mechanisms.

IV. RELATED P2P DATA NETWORKS

Swarm is an Ethereum-dependent peer-to-peer platform that combines content-based addressing, structured chunk placement, and incentive mechanisms. Its design supports decentralized storage and streaming but retains storage, privacy, and ecosystem constraints.

  • Overview: The paper compares Swarm with other data networks and presents their main concepts in a general overview.
  • Architecture: Swarm uses content-based addressing, and unlike IPFS, the address also determines storage location through areas of responsibility.
  • Architecture: Swarm splits files into chunks arranged in a Merkle tree and routes requests through a Kademlia topology according to content addresses.
  • Incentives: Swarm balances data exchange with SWAP and uses postage stamps, SWEAR, and SWINDLE to support compliant storage and exchange.
  • Incentives: Postage-stamp lotteries reward storage nodes for proving chunk possession, while upload-linked stamps can limit pseudonymity.
  • Limitations: Swarm’s determined storage locations may create storage problems, and feeds may mitigate them through user-defined space, recovery, and pinning.
  • Limitations: Swarm benefits from Ethereum’s incentive structure and user base but requires dependence on Ethereum, while research on its mechanisms and use cases remains rare.

B. Hypercore Protocol/Dat

Hypercore supports mutable, incrementally versioned collections and selective access to large datasets, whereas SAFE emphasizes autonomous, encrypted, and privacy-oriented decentralized storage. Both retain important scope and security limitations.

  • Hypercore Protocol/Dat: Hypercore provides Git-like incremental versioning for mutable content and metadata, with separate networks for directory structures.
  • Hypercore Protocol/Dat: Hypercore was designed to share large mutable datasets, partly to prevent link rot and content drift in scientific literature.
  • Hypercore Protocol/Dat: Hypercore uses Hyperswarm, a Kademlia-based DHT, for peer discovery and shares data through a public key and derived discovery key.
  • Hypercore Protocol/Dat: Hypercore permits retrieval of specific versions and data regions, supports mutable collections of files, and provides confidentiality through encryption.
  • Hypercore Protocol/Dat: Hypercore lacks fine-grained authentication beyond the public key and still leaks metadata because the discovery key is only a pseudonym.
  • SAFE: SAFE is designed as an autonomous network with self-authentication, encrypted data types, dynamically organized Kademlia-based sections, and elder approval of events.
  • SAFE: SAFE stores and retrieves self-encrypted chunks through data maps, charges storage in Safecoin, and uses farming to reward nodes serving content.
  • SAFE: SAFE offers strong user control, privacy, and some anonymity, but remains in an alpha phase and its anonymity potential is still a development-stage conclusion.

D. Storj

Storj targets decentralized cloud storage with durability, low latency, security, and privacy through encrypted erasure-coded distribution. Its design improves retrieval resilience but introduces overhead, satellite centralization, and access-pattern concerns.

  • Design goals: Storj targets high durability, low latency, security, privacy, and low bandwidth consumption using end-to-end encryption and erasure codes.
  • Design goals: Storj provides Amazon S3 compatibility to support migration from centralized cloud storage and permits users to store, retrieve, delete, move, and copy data.
  • Architecture: Storj uses satellite, storage, and uplink nodes, with satellites administering storage and storage nodes holding encrypted data.
  • Architecture: Uploads are segmented, encrypted, erasure-encoded into stripes and pieces, then distributed in parallel across selected storage nodes.
  • Erasure coding: In a (k, n) erasure code, an object becomes n pieces and k pieces suffice for reconstruction; Storj additionally uses repair and churn buffers.
  • Incentives: Storj rewards storage nodes through audits and reputation rather than a blockchain-dependent mechanism to preserve low latency.
  • Erasure coding: Erasure coding adds storage overhead but allows retrieval from only the necessary pieces, while storage-node selection and reputation improve protection against breaches.
  • Limitations: Satellites partition availability and may promote centralization, while encrypted metadata can still leave access patterns exposed; research also reports satellite attack vectors.

E. Arweave

Arweave combines blockchain-like on-chain storage with cryptocurrency incentives, aiming for permanent, immutable, time-stamped data. Its replication incentives improve storage coverage, but opportunistic retrieval and sparse direct research remain limitations.

  • Arweave uses a blockchain-like blockweave for permanent on-chain storage and payment, with each block linking to its predecessor and a deterministically selected recall block.The weave is immutable and censorship-resistant, although nodes may refuse content.
  • Wildfire ranks peers by responsiveness and recent throughput, prioritizing high-performing connections and dropping low-ranked peers.
  • Arweave tokens reward miners and pay for transactions, supporting maintenance of the blockweave and its data.
  • Arweave provides time-stamping, transparency, incentives, and immutable storage through transactions with pseudonymous data authors.
  • Pseudo-random recall blocks incentivize nodes to store many blocks, increasing replication, while content filters allow nodes to choose which data they retain.
  • Content retrieval can become complicated because requests are sent opportunistically to nodes that may or may not store the requested content.
  • Direct research on Arweave is sparse, although blockchain research may be partly applicable to the protocol.

V. DISCUSSION OF BUILDING BLOCKS

The surveyed networks share core building blocks such as overlay communication, node identification, and content lookup, but implement them through substantially different architectures. Kademlia-based designs dominate, while Arweave instead uses gossip and peer ranking.

  • The survey identifies network architectures, file handling, information security, and incentives as building blocks and open-challenge areas for P2P data networks.
  • Every considered network builds an overlay network, and each uses a Kademlia-based DHT in some form, sometimes separating peer discovery from data exchange.
  • Nodes generally use self-determined key pairs or derived hashes for identification, while SAFE, BitTorrent, and Swarm use distinct identity schemes.
  • IPFS, Swarm, and SAFE organize networks differently despite using Kademlia: SAFE and Swarm are structured, whereas IPFS remains partly unstructured through encountered-peer connections.
  • Figure 11 provides an overview of the different network architectures.
  • Arweave does not use a DHT; it relies on Bitcoin-like gossip for neighbor discovery and Wildfire for tit-for-tat peer ranking and connection dropping.

B. File Handling and File Size

File handling varies across the surveyed networks, especially in chunking, storage location, lookup, and support for partial data. Most systems appear better suited to megabyte- or gigabyte-scale data than to single petabyte datasets.

  • File handling is more diverse than network organization and includes distinct storage and file-lookup mechanisms.
  • Chunking can increase metadata overhead while improving retrieval of large files; Arweave instead stores files as transactions within blockweave blocks.
  • Storage strategies differ: BitTorrent and Hypercore primarily exchange data, while IPFS, Swarm, SAFE, and Storj distribute or store pieces across nodes using different placement mechanisms.
  • IPFS and Swarm build Merkle trees or DAGs from file pieces, allowing retrieval and separate storage of individually addressed pieces.
  • Lookup is opportunistic in Arweave and IPFS, DHT-based in most other networks, and directed in BitTorrent and Hypercore because neighbors are expected to possess file data.
  • Data-size limits depend on whether data is initially stored at the source node or directly in the network, with Swarm using 4 kB chunks and Storj assuming objects of 4 MB or more.
  • Hypercore supports large datasets and partial sharing, while IPFS supports partial retrieval through its Merkle DAG; BitTorrent, SAFE, and Storj prevent partial data sharing.
  • The authors believe most networks suit megabyte- and gigabyte-scale data rather than single petabyte datasets, but require future performance measurements for confirmation.

2) Integrity:

The surveyed networks address integrity, availability, participation, and storage through hashes, replication, incentives, and compensation. Their designs differ in how they verify content, sustain redundancy, and motivate peers to store or exchange data.

  • Integrity: Hash functions verify distributed data by regenerating a hash and comparing it with a known value.IPFS, Swarm, and SAFE use hashes for content addressing, so retrieved content can be directly verified.
  • Availability: Node failures, maintenance, churn, and attacks can reduce availability, motivating replication, caching, active replication, or erasure coding.Erasure codes add per-chunk overhead but can reconstruct missing chunks without acquiring all chunks.
  • Availability: The surveyed networks generally prefer replication because it is simple, can require no coordination, and may reduce storage through block reuse.Erasure coding can provide similar availability with less overhead, but requires coordination and operates on file-level sets.
  • Incentivization: Open P2P networks use incentives to discourage free-riding and support participation, storage, and long-term availability.Hypercore is identified as an exception without an incentive mechanism; other systems use reputation, monetary incentives, or both.
  • Incentivization: Compensation mechanisms include cryptocurrencies, tokens, payment channels, storage rewards, audits, escrow, and penalties for misbehavior.Filecoin, Swarm, and Storj use payment-channel-like compensation for retrieval, while several systems reward nodes for storing data over time.

VI. RESEARCH AREAS AND OPEN CHALLENGES

The paper identifies performance, confidentiality and access control, security, and anonymity as open research areas, while noting that earlier P2P challenges remain relevant. Evidence and deployment are uneven across systems, with research especially concentrated on IPFS.

  • Cross-cutting challenges: Earlier P2P challenges—including deployment, naming, access control, DDoS protection, junk-data prevention, and churn protection—still apply to next-generation networks.The paper reports possible solutions but calls for more consideration of maturity, mechanism interaction, and adoption.
  • Performance: Performance research should examine read/write times, storage overhead, lookup, and churn resistance through simulations or tests.The paper notes that IPFS has attracted substantial analysis, whereas research on competitors is sparse.
  • Confidentiality and Access Control: Current systems provide some confidentiality and access control but are designed mainly for public rather than private data.Open questions include metadata exposure, access-control security, and deletion of personal, malicious, or illegal data.
  • Security: Security research must address known and emerging vulnerabilities, including an IPFS eclipse attack and a denial-of-service attack on Storj’s test network.The paper also discusses risks from large-scale storage control, unencrypted data, and arbitrary content.
  • Anonymity: Incentives and distributed-ledger transactions make strong anonymity difficult because they expose information about participants or requesters.The paper concludes that current-generation networks do not provide strong anonymity, despite advances in anonymous communication.

E. Naming

Naming systems must balance distribution, security, and human readability, but current data networks generally sacrifice at least one property. Self-certifying addresses improve integrity while making names less readable, so systems add separate naming services or accept trade-offs.

  • Naming: Zooko’s Triangle captures the difficulty of creating a namespace that is distributed, secure, and human-readable.The paper presents these three properties as the central naming challenge in distributed systems.
  • Naming: Existing systems generally sacrifice either distribution or human readability in their data addressing.Tracker-based BitTorrent and Storj lack distribution, while trackerless BitTorrent, Hypercore, IPFS, Swarm, and SAFE lack human-readable addresses.
  • Naming: Self-certifying data makes addresses depend on content, so changing the data changes its address and reduces human readability.Hypercore instead binds a data group to a public key and secures mutability through versioning.
  • Naming: Name resolution maps human-readable names or keys to self-certifying content and can support mutable or versioned data.IPFS provides IPNS for public-key-to-CID mappings and DNSLink for domain-name-to-IPFS-address mappings.
  • Naming: Blockchain-based naming systems claim to combine decentralization, human readability, and security, but forks and incomplete transaction finality can weaken security to an eventual property.The paper identifies ENS as an exception among the discussed systems that appears to provide a solution for Zooko’s Triangle.
  • Context: The next generation builds on earlier P2P experience while addressing long-term availability through newer design concepts.The paper frames this evolution as an improvement over the first generation’s known weakness in persistent availability.
Loading 2102.12737v3…