Source-linked AI summary
IPFS - Content Addressed, Versioned, P2P File System
Juan Benet
TL;DR
Existing distributed file systems and HTTP provide important deployment successes but leave peer-to-peer distribution techniques and mutable naming insufficiently integrated. IPFS introduces a peer-to-peer, version-controlled filesystem that combines these techniques around a content-addressed Merkle DAG, while S/Kademlia achieves a 0.85 lookup success rate with adversaries comprising half the nodes.
Problem
Existing file systems and HTTP have achieved broad use, but HTTP does not exploit many file-distribution techniques, and content-addressed objects still lack mutable naming.
Method
IPFS integrates distributed-file-system, peer-to-peer, and versioning techniques into a globally distributed system that models data as a content-addressed Merkle DAG.
Results
0.85 lookup success rate is achieved by S/Kademlia with an adversarial fraction as large as half of the nodes.
Takeaways & Limitations
IPFS provides a platform for distributing and versioning large data, with immutable objects, version histories, and globally distributed publishing through content-hash addressing.
Takeaways & Limitations
The exploration of alternative BitSwap strategies remains future work, although a sigmoid scaled by debt ratio works in practice.
Abstract
from arXiv · showhide
The InterPlanetary File System (IPFS) is a peer-to-peer distributed file system that seeks to connect all computing devices with the same system of files. In some ways, IPFS is similar to the Web, but IPFS could be seen as a single BitTorrent swarm, exchanging objects within one Git repository. In other words, IPFS provides a high throughput content-addressed block storage model, with content-addressed hyper links. This forms a generalized Merkle DAG, a data structure upon which one can build versioned file systems, blockchains, and even a Permanent Web. IPFS combines a distributed hashtable, an incentivized block exchange, and a self-certifying namespace. IPFS has no single point of failure, and nodes do not need to trust each other.
1. INTRODUCTION
Existing distributed file systems and the Web handle large-scale file distribution unevenly, while version-control systems offer collaboration workflows that file distribution systems lack. IPFS introduces a peer-to-peer version-controlled filesystem intended to reconcile these needs through a shared Merkle DAG.
- Prior systems: Peer-to-peer file-sharing systems achieved deployments exceeding 100 million simultaneous users, while academic distributed filesystems saw less comparable success.BitTorrent still supports tens of millions of nodes joining and leaving daily.
- Prior systems: HTTP dominates file transmission but does not incorporate many file-distribution techniques developed over the preceding fifteen years.Its backward-compatibility constraints make evolving Web infrastructure difficult.
- Motivation: New distribution challenges include petabyte datasets, cross-organization computation, high-volume media, massive-dataset versioning, and preventing accidental file disappearance.These challenges are summarized as making large amounts of data accessible everywhere.
- Motivation: Git provides distributed-friendly versioning and collaboration workflows that large-file distribution systems generally lack.These workflows model and implement operations on changing distributed data.
- IPFS contribution: IPFS introduces a peer-to-peer version-controlled filesystem that synthesizes lessons from prior successful systems and models all data within one Merkle DAG.The design aims to integrate these ideas into a cohesive system rather than treat them as isolated features.
2. BACKGROUND
IPFS builds on distributed hash tables, peer-to-peer exchange protocols, Merkle DAGs, version-control models, and self-certified namespaces. These components support lookup, cooperation among untrusted peers, immutable content addressing, versioning, and cryptographic naming.
- Distributed hash tables: Distributed hash tables coordinate peer-to-peer metadata and support efficient lookup with low coordination overhead and resistance to attacks.Kademlia queries average ⌈log2(n)⌉ nodes and is used in networks exceeding 20 million nodes.
- Distributed hash tables: Coral DSHTs store references to providers rather than forcing nearby nodes to store data, while regional clusters help find nearby data with lower lookup latency.Their relaxed API can avoid hotspots by returning a working peer rather than a complete value set.
- Secure routing: S/Kademlia derives node identities from PKI key pairs, signs messages, and uses disjoint lookup paths to improve security against malicious peers.It reports a success rate of 0.85 with an adversarial fraction as large as half the nodes.
- Block exchange: BitTorrent coordinates untrusted peers through contribution-based exchange and rare-piece prioritization, enabling non-seed peers to trade file pieces.Its standard tit-for-tat strategy remains vulnerable to exploitative bandwidth-sharing strategies.
- Version control: Git represents files, directories, and changes as immutable, content-addressed objects linked into a Merkle DAG.Version changes update references or add objects, and distribution transfers objects plus updated remote references.
- Self-certified filesystems: Self-Certified Filesystems use cryptographic naming so a filesystem name certifies its server and shared namespace allocation is not controlled by a centralized authority.The scheme addresses remote filesystems using a server network location.
3. IPFS DESIGN
IPFS is a peer-to-peer stack that integrates identity, networking, routing, exchange, content-addressed objects, versioned files, and mutable naming. Its self-describing cryptographic values support algorithm agility and compatibility as function choices evolve.
- Architecture: IPFS gives no node privileged status; nodes store objects locally, connect to peers, and transfer files and other data structures.The protocol is organized as sub-protocols for distinct functionality.
- Architecture: The stack includes identity, configurable networking, swappable routing, BitSwap exchange, Merkle DAG objects, Git-inspired versioned files, and self-certifying mutable names.Each layer addresses a separate function while the subsystems remain integrated.
- Exchange: BitSwap models block exchange as a market and weakly incentivizes data replication, with trade strategies designed to be swappable.A debt ratio makes exchange more lenient with previously successful relationships and stricter with unknown peers.
- Identities: Node identities derive from public keys and are checked by peers by comparing the hash of the public key with the claimed NodeId.The identity-generation procedure also applies a proof-of-work difficulty threshold.
- Cryptographic functions: Multihash stores the hash function identifier, digest length, and digest bytes so values can select suitable functions and remain compatible as choices evolve.The design separates cryptographic values from a fixed function set.
3.2 Network
IPFS uses a configurable network stack that can operate over different transports and provide reliability, connectivity, integrity, and authenticity. It represents network addresses with multiaddr strings that encode protocols and encapsulation.
- Network stack: IPFS supports multiple transport protocols, including WebRTC DataChannels for browser connectivity and uTP.The network layer is not tied to a single transport.
- Network stack: The network layer can add reliability with uTP or SCTP when underlying networks do not provide it.Reliability is treated as a configurable stack function.
- Connectivity: IPFS uses ICE techniques for NAT traversal and can operate over overlay networks without assuming access to IP.This broadens the underlying network environments in which the protocol can run.
- Security: Messages can optionally be checked for integrity with hash checksums and authenticity with HMAC using the sender’s public key.Both protections are optional network-stack features.
- Addressing: Multiaddr encodes network addresses and their protocols as byte strings, including encapsulated paths through multiple protocols.Examples include SCTP over IPv4 and SCTP over IPv4 proxied over TCP/IPv4.
3.3 Routing
IPFS uses a distributed shared hash table to locate peers and peers serving objects, while separating small metadata values from larger block references. Its routing layer is replaceable as long as it preserves the defined interface.
- Routing goals: IPFS routing finds peers’ network addresses and peers able to serve particular objects.The DSHT is based on S/Kademlia and Coral.
- Value storage: Small values up to 1KB are stored directly in the DHT, while larger values are represented by references to serving peers.The references are NodeIds of peers that can serve the block.
- Routing interface: The routing interface supports finding peers, storing and retrieving small metadata values, and locating peers that serve large values.The interface includes FindPeer, SetValue, GetValue, ProvideValue, and FindValuePeers.
- Routing interface: IPFS can swap its routing system for one suited to a given network, provided the replacement implements the specified interface.The paper contrasts DHTs in wide networks with static hash tables in local networks.
3.4 Block Exchange - BitSwap Protocol
BitSwap exchanges blocks through persistent want and have lists, using ledgers and probabilistic sending to discourage freeloading while preserving useful trading relationships. The protocol verifies blocks, updates ledgers, and manages peer connections through a defined lifecycle.
- Block exchange: BitSwap is a persistent marketplace where peers exchange wanted blocks for offered blocks across unrelated files.Unlike BitTorrent, it is not limited to blocks belonging to one torrent.
- Block exchange: When peers lack directly useful blocks, BitSwap gives lower priority to acquiring their own pieces while seeking pieces their peers want, encouraging rare-piece dissemination.This addresses cases where block distributions are not complementary.
- BitSwap strategy: Peers track byte balances, send to debtor peers with probability decreasing as debt increases, and temporarily ignore peers after declining to send.The default ignore_cooldown is 10 seconds.
- BitSwap strategy: The strategy aims to maximize exchange performance, prevent freeloading, tolerate unknown strategies, and remain lenient toward trusted peers.The paper presents a sigmoid scaled by a debt ratio as one practical choice, while leaving broader strategy exploration for future work.
- BitSwap strategy: The probability function drops quickly when a peer’s debt ratio exceeds twice its established credit.Figure 1 depicts probability of sending as r increases.
- BitSwap strategy: The debt ratio measures trust by permitting more debt for peers with successful exchange histories and restricting unknown or untrusted peers.The paper also states that this resists Sybil attacks, protects established relationships during temporary failures, and chokes deteriorating relationships.
- Ledgers: BitSwap ledgers record transfers between peers; mismatched ledgers are reinitialized, and nodes may retain or garbage-collect historical entries.Only current ledger entries are necessary for correct operation.
- Protocol lifecycle: Peer connections open by exchanging ledgers, exchange want lists and blocks while active, and close or temporarily ignore peers according to strategy decisions.Want lists are advertised on opening, periodically, after changes, and after receiving a new block.
3.5 Object Merkle DAG
IPFS builds a Merkle DAG of content-addressed objects on top of peer-to-peer block exchange. The object format supports traversal, caching, versioning, encryption, and arbitrary higher-level data structures.
- IPFS combines DHT routing and BitSwap block exchange to construct and distribute a Merkle DAG of objects.
- Content addressing identifies objects and links by multihash checksums, while checksum verification detects tampering or corruption.
- Deduplication stores identical objects only once, including shared portions of index objects and data.
- Objects contain opaque data plus named links with target hashes and sizes, allowing applications to use custom data formats.
- String paths resolve successive link names through the DAG, while hash-based paths remain accessible without a global root.
- A raw data field and common link structure support key-value stores, databases, linked documents, communications platforms, and blockchains.
- Retrieved objects can be cached locally, pinned for persistence, versioned through new hashes, and wrapped for encryption or signatures.
3.6 Files
IPFS defines file and version-control objects on the Merkle DAG, adapting Git’s model for distributed filesystem needs. It also offers multiple strategies for splitting and efficiently retrieving large, versioned files.
- IPFS models files with blocks, lists, trees, and commits, where a commit is a snapshot in a tree’s version history.
- The IPFS object model departs from Git to add fast size lookups, large-file deduplication, and commit embedding while retaining conversion compatibility.
- Blob objects store addressable file data without links, while commits capture snapshots whose referenced descendants preserve retrievable history.
- IPFS can integrate its object graph with Git tools or mount an IPFS tree as a Git repository through a translating filesystem.
- IPFS supports Rabin fingerprints, rsync rolling checksums, or user-specified functions for choosing file block boundaries and detecting changes.
- Tree caching and flattened trees mitigate the DHT, peer-connection, and block-retrieval overhead of long paths.
3.7 IPNS: Naming and Mutable State
IPNS adds mutable, self-certified names to IPFS’s immutable, content-addressed DAG. It publishes signed pointers through the routing system and offers aliases and DNS-based access for usability.
- IPNS addresses the missing mutable naming layer by retrieving changing state at a stable path while retaining immutable DAG objects.
- The immutable DAG provides hash retrieval, integrity checking, links, indefinite caching, bandwidth optimization, permanent links, and permanent backups.
- Self-Certified Names: A user receives a mutable namespace, publishes a signed object under it, and lets others verify the signature against the public key and NodeId.
- Self-Certified Names: IPNS publishes the object as an immutable IPFS object and stores its hash as mutable metadata in the routing system.
- Namespaces: Links in a published object act as sub-names, and commit objects may preserve access to older names when users choose that option.
- User-Friendly Names: Because long hash names are difficult to remember, IPNS supports symlink-like aliases, pronounceable Proquint identifiers, shortening services, and DNS TXT records.
3.8 Using IPFS
IPFS is designed as a general platform for distributed storage, synchronization, sharing, publishing, and computation. Its implementation targets libraries, command-line tools, and mounted filesystems.
- IPFS can serve as a mounted global filesystem under /ipfs and /ipns, or as a personal sync folder that versions, publishes, and backs up writes.
- Proposed uses include encrypted data sharing, versioned software package management, virtual-machine filesystems, databases, and linked communications.
- IPFS is also intended for integrity-checked, encrypted, and web content-delivery networks, as well as a Permanent Web.
- Implementations target an importable library, command-line object tools, and mounted filesystems using FUSE or kernel modules.
4. THE FUTURE
IPFS presents a decentralized infrastructure vision built by synthesizing successful distributed-systems ideas and coupling them into a broader platform. It could serve as a global versioned filesystem and support applications beyond file sharing.
- IPFS synthesizes successful distributed-systems ideas, with coupling and design synthesis as its main contribution beyond the novel BitSwap protocol.The paper characterizes this integration as the coupling of systems and synthesis of designs.
- IPFS is envisioned as decentralized Internet infrastructure on which many different kinds of applications can be built.
- At minimum, IPFS can function as a global, mounted, versioned filesystem and namespace or as a next-generation file-sharing system.
- The vision extends toward a web where publishing valuable information need not require the publisher to host it, and users can trust received content without trusting its publisher.