PaperlayerView exact revision on arXiv

Source-linked AI summary

The Lustre Storage Architecture

Peter Braam

arXiv:1903.01955v1cs.OS

TL;DR

Lustre addresses the challenge of designing a scalable cluster filesystem with global namespace support and modular storage capabilities. It outlines the architecture and related mechanisms, reporting robust error detection and recovery while noting that journal recovery remained difficult to stabilize.

  • Problem

    Existing filesystem designs faced difficulties providing a valid global namespace across multiple file servers while supporting scalable cluster storage.

    Source: Introduction · p. 18, §3.1 · p. 25

  • Method

    The paper outlines a modular Lustre architecture with layered networking, integrated metadata and object storage, and namespace bindings.

    Source: Introduction · p. 18, §2.6 · p. 23, §3.5 · p. 31

  • Results

    The qswnal is described as robust at detecting and recovering from errors, while clustered-directory access commonly requires a single RPC.

    Source: §4.4.2.5 · p. 46, §4.4.2.4 · p. 46, §6.7.3 · p. 84

  • Takeaways & Limitations

    The architecture supports scalable cluster filesystems and a global namespace that remains valid across workstation configurations.

    Source: Introduction · p. 18, §3.1 · p. 25

    Journal recovery remained a delicate project requiring substantial stabilization because of memory-pressure issues, particularly on low-memory machines.

    Source: §5.3 · p. 64

Abstract

from arXiv · show

This lengthy document often referred to as the "Lustre Book", contains a detailed outline of Lustre file system architecture, as it was created between 2001 and 2005, in accordance with the requirements from various users. Now, in 2019, most features have been implemented, but some only recently, and some along different lines of thought.

09/29/2005, HEAD · Glossary

The glossary defines Lustre’s architecture-specific acronyms, filesystem and clustering terminology, and related industry products and protocols. Together, these definitions describe metadata and object-storage services, caching, locking, networking, clustering, authorization, and interoperability concepts.

0.1. Lustre Specific Acronyms

  • The glossary defines metadata writeback caching as deferred flushing of application metadata updates and collaborative caching as shared read caching that enables client-to-client transfers and scalability.InterMezzo is identified as an early network filesystem with metadata writeback caching, while collaborative caching targets mostly read-only workloads.Source: §0.1 · p. 11
  • Lustre is the planned cluster filesystem with clustered metadata and full security, while Lustre Lite is an intermediate scalable filesystem milestone emphasizing high-performance I/O.The glossary also identifies load-balancing MDS systems and lock clients and servers as supporting distributed operation.Source: §0.1 · p. 11
  • Lustre-specific terms distinguish metadata services and clients from object-storage services and clients, including MDS/MDT/MDC and OST/OSC components.The MDS provides metadata services, while the OST provides object storage through an OBD server; MDC is the metadata client module and OSC communicates with an OST.Source: §0.1 · pp. 11–12

0.2. Filesystem, Clustering, and Networking Terminology

  • Networking terms distinguish request/response RPC, small-request and large-buffer NIO, and Portals-based networking abstractions, with UDP and VPN listed as protocol concepts.OS-bypass I/O can eliminate kernel-to-user buffer copying and avoid sleeping by polling for response arrival.Source: §0.1 · p. 12, §0.2 · pp. 12–13
  • Filesystem terminology covers administrative filesets, stronger transactional guarantees within fileset boundaries, and global namespaces that unify multiple exported filesystems.The glossary also defines barriers for synchronizing recovery phases, membership for tracking participating systems, and quorum for determining whether a group is authoritative.Source: §0.2 · pp. 12–13

0.3. Industry Products and Protocol Acronyms

  • The industry glossary situates Lustre among storage, filesystem, and networking technologies, including ADIO, DAFS, NFSv4, NASD, RDMA, iSCSI, and OSD standards work.ADIO is described as a low-level API for efficient MPI-I/O, while the OSD group works with ANSI T10 to define object-based storage-device standards.Source: §0.3 · pp. 13–14
  • Related systems and products provide comparison terminology for availability, caching, clustering, locking, and load balancing, including Coda, InterMezzo, Ensemble, IBM DLM, Kimberlite, Service Guard, and LVS.Coda and InterMezzo are associated with high availability and disconnected operation, while Ensemble provides group communication infrastructure and LVS manages service load balancing and redirection.Source: §0.3 · pp. 13–14
  • Additional industry entries define hashed or tree-based directories for fast name lookup and identify TUX as a scalable kernel-space web server whose optimizations might benefit Lustre.The glossary also lists security, directory, storage, and VFS-locking terms such as LDAP, extended attributes, SNIA, and lock pinning, releasing, acquiring, and revoking.Source: §0.3 · pp. 14–15

Architecture · Introduction

Lustre is a modular, scalable cluster filesystem architecture that separates clients, object storage, metadata services, and pluggable storage-management components. Its framework supports flexible deployment, failover, coherent access, SAN integration, and extensible metadata and storage services.

Introduction

  • Lustre provides a modular storage framework for scalable cluster filesystems, supporting networking, locking, storage targets, and deployments ranging from small environments to very large clusters.The system originated from Coda research and was advanced through industry and national-laboratory collaboration; Lustre Lite served as an independently usable intermediate target.Source: Introduction · p. 18

2.1. Lustre Components

  • Clients, Object Storage Targets, and Meta-Data Servers form Lustre’s three major system types, with shared framework modules and flexible separation or co-location.Clients perform filesystem operations, OSTs handle file-data I/O, and MDSs handle namespace operations; the subsystems can be deployed separately or symmetrically.Source: §2.1 · pp. 18–19

2.2. Object Storage Targets

  • OSTs store file-data objects, perform distributed block allocation, enforce client security, and expose interchangeable direct-storage drivers through a Portals-based network interface supporting RDMA.File metadata remains on MDSs while file-data objects reside on OSTs; direct drivers can use raw ext2 inodes or files in journal filesystems.Source: §2.2 · pp. 20–21

2.3. Meta-data Service

  • MDSs transactionally manage namespace metadata over networks, support failover and locking, and are designed to evolve toward load-balanced and stackable replicated or cached metadata services.Lustre Lite limits complexity to one MDS, while the full system plans load-balanced metadata processing and logical services that dispatch requests to replicated servers or local caches.Source: §2.3 · pp. 21–22

2.4. Client Filesystem

  • Client protocols provide authenticated, transaction-based metadata access with writebehind caching and coherent updates, while locking ranges from coarse file or subtree locks to fine extents and pathname locks.Lustre can provide UNIX update semantics, optionally relax data coherency when applications manage it, and transparently fail over client subsystems.Source: §2.4 · p. 22

2.5. Storage Management

  • Storage management uses stackable object modules to support migration, snapshots, enhanced security, active disk components, and dynamically changeable driver stacks.The architecture explicitly separates capabilities such as scalable allocation, security, and metadata control from a monolithic filesystem design.Source: §2.5 · pp. 22–23, §2.4 · p. 22

2.6. Lustre and SAN’s

  • SAN compatibility combines SANOST control and direct data transfer with OSTMDS unified metadata and object APIs, enabling clustered commodity Linux filesystems while sacrificing some Lustre scalability.OSTs retain security and allocation control while delegating data transfer to the SAN; OSTMDS can use an ordinary ext3 filesystem without reformatting.Source: §2.6 · p. 23

2.7. The Lustre Framework

  • The Lustre framework distinguishes the relatively static execution environment from modular, pluggable components, while the document records figure, text, and cross-reference revisions to its architecture description.This distinction follows from the flexibility provided by stacking and composing storage modules.Source: §2.7 · p. 24, §2.8 · p. 24

Global Namespaces for Filesystems - with Lee Ward

The section presents a self-describing global namespace that unifies filesets from multiple servers into one directory tree valid across clients, combining AFS-like traversal with automounter features for Linux filesystems.

Global Namespaces for Filesystems - with Lee Ward

  • The namespace uses filesystem-stored mount-point objects and addresses their traversal, persistent client caches, and interactions with server-local filesystems.InterMezzo and Lustre use a Linux namespace module to provide these features.Source: Global Namespaces for Filesystems -… · p. 25

3.1. Introduction

  • A global namespace unifies directory trees from multiple file servers while eliminating client configuration and remaining valid across workstations as configurations change.The namespace is intended to provide a single directory tree throughout an installation.Source: §3.1 · p. 25, Global Namespaces for Filesystems -… · p. 25
  • AFS traverses fileset mount-points within a mounted filesystem by obtaining location data from servers without client configuration; Coda, DCE/DFS, and Microsoft DFS use similar mechanisms.AFS namespaces can unify tens of thousands of filesets across administrative cells.Source: §3.1 · p. 25
  • Automounters detect mount-point traversal, consult mount maps, and execute Unix mounts, supporting heterogeneous combinations such as NFS and local filesystems.Mount maps may be stored in configuration files or NIS and LDAP directory services.Source: §3.1 · pp. 25–26
  • The design combines AFS advantages with newer automounter features to provide a more general global namespace across local filesystems, NFS, InterMezzo, and Lustre.Its implementation provides a roadmap for incorporating global namespaces into Linux environments.Source: §3.1 · p. 26

3.2. Fundamental Definitions and Requirements

This section defines requirements and design elements for a flexible global namespace, including filesystem and fileset mount-points, automounter support, namespace bindings, and location databases. It also describes InterMezzo namespace layouts, planned Lustre mount-object behavior, and emerging filtering namespace support.

3.2. Fundamental Definitions and Requirements

  • The global namespace should support Unix and fileset mount-points, multiple mount locations, local or networked configuration, cached activation and release, controlled retries, authorization, and timely updates.Mount-object handling should preserve Unix directory semantics, avoid lookup-triggered mounts, and allow standard filesystem APIs where possible.Source: §3.2 · p. 27

3.3. The Andrew FileSystem and Autofs

  • Recent autofs avoids mount storms by delaying mounts until entries are accessed or directories are opened, while adding normal indirect-map listings, mount-point trees, and parallel mounts.Autofs transparently mounts filesystems through mount-objects and can cache or release unused mounts, but still requires a separate filesystem type and external mount maps.Source: §3.3 · p. 28

3.4. Implementing and Handling Mount-Objects

  • The proposed system represents mount-objects as directories with setuid mode and mntinfo metadata, triggering user-level mounts on lookup or opendir while preserving link counts and supporting any underlying filesystem.Unlike AFS and autofs, mount-objects belong to the underlying filesystem, require no map files or external configuration, and can graft local and network filesystems together.Source: §3.4 · pp. 29–30

3.5. Namespace Bindings

  • Namespace bindings allow filesets and filesystems to be mounted at multiple locations by linking a mount-point dentry to the root inode of the bound namespace.The global-namespace mount option lets servers interpret mount-objects for clients while leaving them as ordinary directories when the option is absent.Source: §3.5 · pp. 31–32

3.6. The Fileset Location Database and the Automounter

  • The automounter obtains fileset location data from mntinfo or servers, applies local administrative restrictions and overrides, mounts with appropriate credentials, and performs expiry and invalidation through kernel ioctls.LDAP-based mount maps are expected as an implementation of server-side information lookup.Source: §3.6 · p. 32

3.7. The Global Namespace of the InterMezzo Filesystem

  • InterMezzo’s global namespace must combine filesets from multiple servers into one client directory tree by exploiting mount-points.InterMezzo uses disk filesystems as persistent server storage and client caches, including full or partial fileset replicas; this subsection also identifies combining server fileset namespaces as a required issue.Source: §3.7 · pp. 32–33
  • InterMezzo combines physical fileset namespaces from multiple caches into a global namespace under /izo and supports multiple namespaces.Clients can use partial caches for imported filesets and full replicas for locally stored filesets, while separate bindings expose logical and physical layouts.Source: §3.7 · p. 33
  • The namespace mechanisms support mixed filesystem types, including locally backed /tmp storage alongside globally visible temporary files.The described layout distinguishes local and global temporary directories within an InterMezzo-rooted installation.Source: §3.7 · p. 34

3.8. Mount-objects in Lustre

  • Lustre filesets are intended to use mount-objects whose traversal may query LDAP to locate a fileset’s metadata server.The passage presents this as a requirement or design description rather than a completed implementation.Source: §3.8 · p. 35

3.9. Implementation

  • A filtering global-namespace mechanism had begun implementation, while the changelog records figure and text updates and the original draft attribution.The implementation subsection remains a placeholder, and the changelog identifies P.D. Innes and L. Ward’s contributions.Source: §3.9 · p. 36, §3.10 · p. 36

Networking for Lustre

This section describes Lustre’s portable, scalable networking architecture, covering Portals-based messaging and bulk data movement, heterogeneous networks and RDMA, routing and gateway failover, request processing, connection management, and recovery.

4.2. Layering the Networking

  • The stack combines device drivers, Portals messaging, Niobuf data movement, and request processing to provide Lustre with a portable, scalable networking architecture.Its requirements include portability, clusters of 10,000’s of nodes, multiple network infrastructures, fast-network performance, OS bypass, recovery, and security.Source: §4.2 · pp. 37–38, §4.1 · p. 37

4.3.1. Portals Data Structures and Mechanisms.

  • Portals provides asynchronous put/get operations, packet matching, event queues, OS-bypass support, and network abstraction across kernel and user-level environments.Its portal tables and memory descriptors dispatch messages to buffers, while NALs connect the API to different network interfaces.Source: §4.2 · p. 39, §4.3.1 · pp. 39–40

4.2. Layering the Networking

  • The request-processing layer supports command or response packets with associated bulk buffers, single- or multi-host requests, callbacks, optional waits, timeouts, and error handling.This layer sits above data movement because Portals put/get primitives are too low-level for vectorized I/O-buffer transfers.Source: §4.2 · p. 39

4.3.2. Portals Internals.

  • Portals internals separate API and library domains through API and library NALs, enabling forwarding, packet dispatch, event generation, and protection-domain crossing.The four components are the Portals API library, Portals library, API NAL, and lib NAL.Source: §4.3.2 · pp. 41–42

4.3.3. RDMA Support in Portals NAL’s.

  • Portals supports RDMA through registered-memory handles exchanged during long sends or early binding, but proposed opaque-handle extensions are needed for memory descriptors and put/get operations.The proposed model matches common network-interface RDMA processing without changing the higher-level Portals API.Source: §4.3.3 · pp. 42–43

4.3.4. TCP - the Socknal.

  • Network-specific NALs adapt Portals to TCP, Quadrics, and VIA, using stream reassembly, zero-copy or DMA mechanisms, concurrent-request tuning, and connection registration.The Quadrics NAL was tuned for requests from up to 6,000 concurrent threads, while VIA’s scalability to 1000’s of nodes remained unclear.Source: §4.3.4 · p. 43, §4.3.5 · pp. 43–44, §4.3.6 · p. 44

4.4. Portals Routing

  • Portals routing lets nodes use multiple interfaces or gateway nodes to forward packets between networks, allowing Lustre filesystems to span heterogeneous clusters.Gateway nodes can shovel RPC messages between network-specific NALs such as QSWNAL and socknal.Source: §4.4 · p. 44, §4.4.2 · p. 44

4.4.2. Gateway Failover.

  • Redundant gateway support implements load balancing across equivalent routes and enables or disables gateways to maintain multi-cluster operation after gateway failures.Failure handling disables routes using the failed gateway until it is rebooted, within configurations where immediate gateway neighbors receive failure notifications.Source: §4.4.2 · p. 45

4.4.2.3. Impact on Lustre.

  • Gateway failover lets Lustre recover after dropped messages by disabling failed routes, using alternatives, and triggering normal recovery, although network healing may not precede Lustre timeouts.The design assumes at least two gateways per destination and requires Lustre to insulate abandoned RPCs and recover from failed recovery attempts.Source: §4.4.2.3 · pp. 45–46, §4.4.2.4 · p. 46

4.4.2.4. Summary of the scheme.

  • When a gateway fails, its peer disables it locally and issues an upcall so all nodes avoid it; after reboot and reconfiguration, administrators re-enable it network-wide.The router preserves topology while excluding the failed gateway, clears relevant NAL connections, and resumes routing when the gateway returns.Source: §4.4.2.4 · p. 46, §4.4.3 · pp. 47–48
  • qswnal robustly detects and recovers from errors, whereas socknal may miss gateway death because buffered TCP connections remain apparently healthy until later I/O or notification.Large zero-copy messages time out reliably, but small buffered messages and short TCP keepalives do not reliably expose failures.Source: §4.4.2.5 · pp. 46–47

4.4.2.5. Error handling.

  • The planned safeguard is for Lustre’s pinger to detect gateway failure, with NAL peer detection providing best-effort early avoidance and possible heartbeat or generalized notification improvements.Socknal heartbeats, socket-drain detection, and portals HELLO messages are considered, but dormant clients make heartbeat-based approaches problematic.Source: §4.4.2.5 · p. 47

4.4.3. Practical issues.

  • Portals routes messages across equivalent gateways, supports explicit gateway up/down control, preserves topology, load-balances routes, and leaves Lustre to retry after gateway-induced errors.Socknal and qswnal notify local routers and issue upcalls, while Lustre initially treats resulting failures as transient network errors.Source: §4.4.3 · pp. 47–48

4.5. Lustre Data Movement and Portals

  • Lustre uses Portals for client-server protocols carrying request, reply, and bulk packets, with statically assigned portals and increasing per-connection xids identifying requests and replies.Bulk transfers add page-specific xids so data identification remains independent of delivery order.Source: §4.5 · p. 49, §4.5.1 · p. 49, §4.5.2 · p. 49

4.5.3. Event Handling

  • Portals event queues dispatch incoming requests, replies, outgoing packets, and bulk completions through handlers that update state, invoke callbacks, wake threads, or free resources.Memory descriptors and user pointers connect events to services, waiting requests, and bulk descriptors; explicit thread yields could support interrupt-free processing.Source: §4.5.3 · p. 50, §4.5.4 · p. 50

4.5.5. Buffer Management.

  • Lustre uses receiver-managed, reusable buffers that append requests and replies in place, replace full buffers automatically, and avoid request-buffer copies, though connection-specific VI buffers threaten scalability.Buffers track active extents before returning to the available ring, while the proposed mitigation reassigns buffers when per-connection allocations are exhausted.Source: §4.5.5 · pp. 50–51

4.5.5. Buffer Management. / 4.5.6. Bulk Transport.

  • Bulk transport uses sink_niobuf, source_niobuf, and logical_niobuf descriptors to move object data symmetrically, support multiple extents, and enable optional remote DMA.Sources prepare buffers and receive completion notifications through Portals ACKs; remote DMA tokens allow direct memory access with lower overhead.Source: §4.5.6 · pp. 51–52

4.5.6. Bulk Transport.

  • A typical object write maps source pages, transfers remote extent descriptors, and uses acknowledgments and handlers to clean buffers, flush writes, and complete reads.For writes, sink events release pages to cache-flush daemons; for reads, received pages become up to date and unlocked after storage pages are prepared.Source: §4.5.6 · p. 52

4.6. Naming and structures used in Lustre Network / 4.6.1. Nodes and Addresses. / 4.6.2. Lustre Handles.

  • Lustre’s networking model names nodes, networks, interfaces, and peers with UUIDs and addresses, while opaque handles let authorized remote nodes locate and validate objects.Linux handles combine an object address with a random cookie, enabling fast validation and preventing forgery when object-lifetime races are avoided.Source: §4.6 · p. 53, §4.6.1 · p. 53, §4.6.2 · p. 53

4.6.3. Connections. / 4.6.4. Target. / 4.6.5. Export.

  • Connections track remote identity, peer addresses, message sequence numbers, connection handles, recovery generations, epochs, and boot counts, with reference-counted cleanup for unused connections.Targets expose device APIs through target-mode handlers, while exports maintain per-client state associated with target use and connections.Source: §4.6.3 · pp. 53–54, §4.6.4 · p. 54, §4.6.5 · p. 54

4.6.5. Export. / 4.6.6. Services. / 4.6.7. Client. / 4.6.8. Import.

  • Services dispatch incoming packets to target handlers, manage request threads, buffers, event queues, replies, and bulk portals, while clients and imports provide recovery-organized access to remote exports.Clients form a recovery tree, and imports instantiate local APIs whose requests cross to exports only at permitted levels.Source: §4.6.6 · p. 54, §4.6.7 · p. 54, §4.6.8 · p. 55

4.7. Request Processing / 4.7.4. Request Processing on the Client. / 4.7.5. The Connection Handshakes.

  • Request processing begins with authenticated connection handshakes that exchange secure connection pointers, generation and boot information, target mappings, connection identifiers, and per-client export state.Service connections map target UUIDs to export devices and return handles used on later messages.Source: §4.7 · p. 55, §4.7.5 · pp. 55–56

4.8. Error Handling / 4.8.1. NACK.

  • Portals error handling is intended to support recovery, including a proposed NACK when a receiver drops an ACK-requesting packet and events for transmission failure and packet reclamation.The transmission-failure event design remained under consideration.Source: §4.8 · p. 56, §4.8.1 · p. 56, §4.8.2 · p. 56

4.9. Changelog.

  • The documented evolution added concrete internal descriptions and high-level services, plus RDMA, security, scatter/gather vectors, policy handling, kernel-memory structures, and DLM and OST protocol updates.The changelog also records wire-level protocol documentation and clarification of coexisting user/kernel Portals instances.Source: §4.9 · pp. 56–57

Object Storage

This section presents the Object Storage architecture, including layered drivers, object namespaces and metadata, recovery mechanisms, snapshot volumes, and distributed read access through cache nodes. It also addresses implementation challenges involving coordination, memory pressure, aggregation, and distributed transactions.

5.1. Overview of the Architecture

  • Layered object drivers let storage applications aggregate devices, expose snapshots and versions, and reach clustered object storage through network clients and targets.Logical drivers can implement RAID-like aggregation, while client-target pairs package OBD commands across protocols and transport layers.Source: §5.1 · pp. 58–59

5.2. Fundamental Definitions

  • The architecture identifies devices and dependencies with UUIDs and represents OBD objects through self-describing structures containing device, driver, filesystem, and application metadata.OBDO structures require attribute and data operations, unlinking, device linkage, and driver-specific extension areas.Source: §5.2.2 · pp. 60–61, §5.2.4 · p. 63

5.2.2. Naming and Identification of Devices.

  • Object groups make namespaces non-flat, support snapshot grouping, and enable applications to request specific object identifiers when preserving references during filesystem migration.The proposal accepts greater implementation complexity and lets a copied object retain its original identifier, provided the destination does not already contain it.Source: §5.2.3 · pp. 61–62

5.2.4. Meta-data Handling.

  • Metadata handling is the architecture’s biggest challenge because ownership, selection, leasing, stale data, and performance trade-offs remain difficult across OBDs, logical drivers, and filesystems.The design distinguishes device, logical-driver, and application metadata, but explicitly leaves leasing schemes unresolved.Source: §5.2.4 · pp. 62–63

5.3. Journal Recovery of the Object-based Disk Filesystem

  • Journal recovery is possible and attractive, grouping related updates through transaction handles and writing transaction data to stable journal storage before asynchronous writes reach final locations.The proposed mechanism sacrifices durability and requires copy-on-write buffers, while recovery interacts subtly with aggregation.Source: §5.3 · p. 64, §5.3.1 · p. 64

5.3.2. Journal Modifications to the Ext2OBD Driver.

  • Journal support for Ext2OBD can group direct-driver updates, but page-cache versioning, memory pressure, logical-volume coordination, and distributed transactions require substantial unresolved design work.The page cache may need multiple copy-on-write page versions, and logical drivers may require optimistic concurrency methods for distributed transactions.Source: §5.3 · p. 64, §5.3.2 · p. 65

5.4. The Snapshot Logical Driver

  • The snapshot logical driver served as a case study that exposed interface requirements and led to their implementation, with separate snapshot volumes preserving a pure primary filesystem when needed.Snapshots stored within one volume require snapshot-aware access, whereas separate volumes keep the primary object volume consistent and filesystem-compatible.Source: §5.4 · pp. 65–66
  • Snapshot volumes represent filesystem state across ordered snapshot times using direct data objects and redirecting objects that select current or historical versions.Redirecting entries can point to the active object, snapshot objects, or unavailable future snapshots.Source: §5.4 · p. 66

5.4.1. The Behavior of the Snapshot Volume

  • Snapshot access is read/write for snapshot 0 but read-only for older snapshots, with reads selecting direct data or the direct object referenced by a redirector.Writes use object ctime to perform copy-on-write, freeze prior data, update snapshot pointers, and then write new data and metadata.Source: §5.4.1 · pp. 66–67

5.5. Caching OBD Driver

  • COBD distributes reads across client or dedicated cache nodes, reducing requests handled by target OSTs and improving read scalability.The driver uses existing OSC/OST infrastructure while caching nodes service client read requests without granting locks.Source: §5.5 · pp. 68–69, §5.5.1 · p. 69

5.5.1. Read Cache.

  • The read cache maintains mapped object pages under kernel memory pressure and uses read locks from the target OST to preserve consistency.The design may use swap-backed persistent storage for large caches, while filesystem-backed reuse after reboot was not planned immediately.Source: §5.5.1 · p. 69

5.5.2. Referral Module.

  • A referral module selects caching OSTs with compatible read locks and forwards requests by cache-node UUID, populating a cache when no copy is available.Selection may use a policy such as round-robin, and cache misses acquire a lock before standard OSC population.Source: §5.5.2 · pp. 69–70

5.6. Changelog

  • The changelog records figure and table updates, document revisions, a missing figure, the COBD section addition, and the original draft attribution.Entries credit P.D. Innes, Radhika Vullikanti, and P. Braam.Source: §5.6 · pp. 70–71

The Lustre Metadata Service

This section describes Lustre’s metadata service architecture for maintaining a coherent, globally consistent namespace across clustered clients and metadata servers. It covers APIs, caching and locking, distributed storage, transactional coordination, failover, recovery, and client-facing optimizations.

6.1. Introduction - Subsystems and Protocols

  • Lustre exposes a metadata API for creating, removing, renaming, and managing namespace elements, independently or alongside the object-storage API.The metadata API manages the filesystem namespace, while metadata and object services may run on the same or separate devices.Source: §6.1 · p. 72

6.2. The Meta Data Namespace

  • The metadata API provides every client with one consistent namespace view, representing entries as parent-fid, name, and inode-fid triples.Fids are never reused and uniquely identify inodes, simplifying namespace coherency; clients can also cache negative lookups using null fids.Source: §6.2.1 · pp. 72–73

6.2.2. API support for lookup.

  • Lustre supports element-level and subtree-level namespace synchronization, coupling locks to namespace structure and ordering lock acquisition before protected data use.Element-level lookup can combine data and locks in one mdc_enqueue RPC, whereas write-back synchronization requires revoking locks held within a protected subtree.Source: §6.2.2 · p. 74, §6.2.3 · p. 74

6.3.1. Attributes.

  • Metadata attributes include authorization, directory content, sizes, timestamps, data and directory locations, symbolic-link targets, and application-defined extended attributes.Extended attributes may be persistent or in-memory and can synchronize clustered CIFS and NFSv4 service metadata across nodes.Source: §6.3.1 · pp. 74–75

6.4. Updates to the namespace

  • Lustre’s namespace update paths separate client and MDS responsibilities: client-server mode discovers most changes through later lookups, while write-back mode records cached updates for reintegration.Reintegration covers create, unlink, rename, hard-link, and setattr operations; the MDS journals recovery information while locking affected parent directories.Source: §6.4.1 · p. 76, §6.4.2 · pp. 76–77

6.4.3. mds_open.

  • The mds_open intent path combines lookup, symbolic-link traversal, optional creation, opening, status reporting, and protected object creation for new files.Its response records executed decisions, errors, inode attributes, and a write lock when the client must create file objects.Source: §6.4.3 · p. 77

6.5. Write Back and Client Server Mode

  • Write-back locks reduce client and MDS resource usage but increase synchronization latency, whereas client-server locks support shared access among nodes.Lustre switches based on lock traffic and subtree activity, revoking conflicting locks and flushing cached updates when write-back ownership changes.Source: §6.5 · p. 77, §6.5.1 · p. 78, §6.5.2 · p. 78

6.5.2. Mechanisms.

  • Write-back reintegration batches update records without individual acknowledgments, but shared update sequences can block unrelated users when capabilities expire or are administratively revoked.The reintegration design resembles InterMezzo’s approach, while capability refusal is either an avoidable timeout problem or a potentially beneficial administrative control.Source: §6.5.3 · p. 78

6.6. MDS Recovery

  • Lustre’s metadata recovery preserves a consistent namespace across single MDS or temporary network failures by replaying completed-but-unpersisted requests in their original order.The design assumes simultaneous multi-node failures are rare, enabling performance gains, while requiring clients to distinguish and recover each request’s state.Source: §6.6 · p. 79

6.7. Clustering Metadata

  • Clustered metadata distributes inode groups across MDS servers through a logical metadata volume, unified metadata and recovery protocols, and directory objects that can span nodes.Directories store inode-group identifiers, and the client-side LMV selects the responsible MDC using request metadata while dispatching each command to one MDS.Source: §6.7 · p. 79, §6.7.1 · p. 80, §6.7.1.1 · p. 80

6.7.2. Resources

  • Lustre assigns moderately large object groups to nodes, journals each inode group for recovery, balances load through randomized distribution, and supports failover or resource adjustment.A load-balancing map identifies where each inode group is mounted, while growing directories fan out into data objects on multiple MDS servers.Source: §6.7.2 · p. 81, §6.7.3 · p. 83

6.7.1.2. Basics of the operations.

  • Clustered directories hash names into remote buckets; lookup commonly needs one RPC, while the worst case requires three, with disk reads equal to or one above large non-clustered directories.Directory splitting enables parallel operations but imposes penalties during splitting and subsequent renames or unlinks involving remote inodes.Source: §6.7.1.3 · p. 81, §6.7.3 · p. 84

6.7.5. Clustered MDS protocol

  • Clustered MDS operations extend creation, rename, link, and unlink across nodes using remote RPCs, resource locks, acknowledgments, and orphan management, while seeking shared recovery semantics.The design must avoid cyclic lock or acknowledgment dependencies, and remote inode creation requires commit-log recording or preallocated inodes for recovery.Source: §6.7.5 · p. 85

6.7.6.3. Failures of multiple MDS nodes.

  • Clustered recovery tracks execution, replies, and commits, using resending for unexecuted requests and replay for executed requests lost from persistent storage; multiple MDS failures require rollback to a consistent snapshot.Transaction logs record cross-MDS dependencies and support canceling committed records or undoing operations after simultaneous failures.Source: §6.7.6.1 · p. 86, §6.7.6.2 · p. 86, §6.7.6.3 · p. 86
  • MDS transactions coordinate locks, space and namespace checks across nodes, assign sequence numbers, journal participants for replay, and use leader-confirmed commits to manage undo records.A standby or ring-neighbor MDS can take over after failure, while periodic leader-driven snapshots converge on consistent committed transaction numbers and bound replay requests.Source: §6.7.6.3 · pp. 86–87

6.8. MDS Implementation

  • The MDS implementation requires post-recovery metadata consistency, pre-allocated file identifiers for write-back caching, and atomic execution of filesystem transactions with transaction-stream maintenance.Possible back ends are a metadata filesystem or clustered database, but client-visible APIs ultimately determine the design; filesystem storage enables reuse of recovery, locking, object, and security protocols.Source: §6.8 · pp. 87–88, §6.8.1 · p. 88

6.8.1. Meta-data File system.

  • Storing a file-set’s metadata on one OST simplifies implementation but can bottleneck heavy metadata traffic and prevents transactions such as links and renames from crossing file-set boundaries.A clustered database offers flexible metadata formats and high transaction throughput, but integrating it with the metadata server would require unusual glue and further exploration.Source: §6.8.1 · p. 88, §6.8.2 · p. 88

6.9. ODDS and ENDS

  • Lustre largely avoids frequent atime updates, while mtime changes remain POSIX-relevant and are coordinated between the MDS and OSTs for writes and truncates.An inode validity flag tells clients whether the MDS mtime is current; OST replies piggyback updated values, and close or setattr sends the final value back to the MDS.Source: §6.9.1 · p. 88, §6.9.1.1 · p. 89

6.9.1.1. The basic mtime access and update mechanism.

  • MDS mtime persistence uses asynchronous, batched commit callbacks to notify OSTs after disk commit, so a truncate may involve MDS initiation, OST execution, and client-to-MDS update calls.OST records persist the files and values under mtime management across crashes; MDS recovery queries all OSTs, clears pending records after processing, and may temporarily degrade performance without synchronous recovery.Source: §6.9.1.1 · p. 89, §6.9.1.2 · pp. 89–90

6.9.2. Server-based Lock Management.

  • Lustre’s server-based locking adds version-aware reacquisition, operation-based acquisition and revocation, and OS inode/dentry caching to optimize metadata concurrency.The protocol embeds modifications in lock requests and revocations, making it faster but more complex to recover from network or system failures.Source: §6.9.2 · p. 90

6.9.3. Fileid Caching.

  • File identifiers and names serve as separate primary-object families: clients translate names to file identifiers, while MDS updates pair parent identifiers with child names and lock revocations use identifiers.File-identifier-named objects and inode-resident attributes, directory pages, and I/O information support the corresponding caches.Source: §6.9.3 · p. 90

A Distributed Lock Manager for Lustre

This section describes Lustre’s distributed lock management and its integration with VFS and filesystem abstractions, covering lock processing, recovery, remastering, callbacks, transactional metadata, and related interfaces. It also addresses contextual and layered service structures, credentials, cancellation and version/value exchange, and areas requiring further refinement.

A Distributed Lock Manager for Lustre

  • Lustre extends a VAX Cluster-style distributed lock manager with policy-based extent locks and intent locks that can execute metadata operations instead of merely granting locks.Extent policies account for relationships among nested file extents, while intent processing can create a file and return a lock on the new object.Source: A Distributed Lock Manager for Lustre · p. 92, §7.5 · pp. 99–100

7.1. Resource Directories and Masters and Lock Owners

  • Resource trees assign each tree’s locks to one managing system, while hashing resource names into a weighted Lock Directory Weight Vector locates masters and permits new allocation when none exists.The directory is notified when a tree becomes free, and systems may appear multiple times in the vector to reflect differing capacity.Source: §7.1 · p. 93

7.2. Resource Directory API (Internal to the Lock Manager)

  • The resource-directory API returns an existing master or assigns the caller as master, releases resources after the final request, and maintains master-side queues and lock data.Non-master systems retain duplicate resource and lock state so locks can be remastered when a resource master leaves the cluster.Source: §7.2 · p. 94
  • Hierarchical locking requires ancestors to be locked before descendants and prevents parent dequeue while descendants remain, enabling coarse-grained parent locks and locally unique resource names.The model supports an inode-level master lock alongside differently modal extent locks.Source: §7.2 · p. 94

7.3. Foundations of the Locking Infrastructure

  • The locking infrastructure tracks granted, converting, and waiting requests, using compatibility rules to govern new acquisitions and conversions across six lock modes.A conversion can be granted only when compatible with all granted modes, and new requests also require empty ungranted conversion and new-request queues.Source: §7.3.1 · p. 95, §7.3.2 · p. 96

7.3.2. Lock Requests and Processing the Lock Queues

  • Lock processing prioritizes conversion requests over waiting requests, but can indefinitely postpone queued conversions or new requests; QUEUECONV, EXPEDITE, and NOQUEUE flags provide application-controlled alternatives.A second conversion request for one lock cannot coexist in the same conversion queues and may overwrite the first with a warning.Source: §7.3.2 · p. 97

7.4. The Completion and Blocking Callbacks

  • Completion callbacks report lock acquisition, conversion, cancellation, and blocking events, allowing cached-data subsystems to flush readers’ and writers’ data before releasing locks.Blocked requests receive a callback when the server receives the request and again when it grants the lock; blocking callbacks also deliver remote notifications.Source: §7.4.1 · p. 99, §7.4.2 · p. 99

7.6. Lock Services

  • Lustre lock services are intended to be available as a kernel library, a kernel daemon, or a new system call.This provides three service forms for exposing the lock manager.Source: §7.6 · p. 100
  • Lock services identify requests through status blocks and synchronously allocate lock IDs, enabling subsequent conversion or cancellation even before requests enter processing queues.Status blocks also carry application values and request conditions, while return values can be computed without contacting a remote resource manager.Source: §7.6 · pp. 100–101
  • The lock API supports acquisition, conversion, cancellation, parent-child relationships, completion routines, blocking triggers, and blocking or non-blocking calls.Cancellation can target a request, all sublocks of a parent, or all locks of a process; cleanup cancels queued requests and handles granted locks.Source: §7.6 · pp. 101–103

7.7. Lock Version/ Value Blocks

  • Lock version/value blocks let applications exchange resource versions: values may be written on release, returned on promotion, or left unchanged according to the operation.The lock value information is copied between request blocks and resource-held fields.Source: §7.7 · p. 104

7.8. Cluster Transitions and the DLM

  • Cluster transitions recompute resource directories, discard non-owned locks, re-register surviving masters, and reacquire non-mastered locks, while weighted remastering reduces unnecessary rebuild work.Leaving nodes should delete their locks, adjust directories, remaster affected trees, and grant locks previously blocked by those locks.Source: §7.8 · pp. 104–105

7.9. Dynamic Remastering

  • Dynamic remastering moves lock mastering according to system power, directory weight, and activity, using thresholds and benefit-versus-cost checks to control migrations.The manager evaluates resource trees every 8 seconds and can move up to five trees per run.Source: §7.9 · pp. 105–106

7.11. The Lustre VFS library

  • The Lustre VFS library provides a universal API for kernel drivers, user-level servers, other operating systems, and prospective SQL-based metadata servers.It is modeled on Linux VFS APIs and Lustre extensions.Source: §7.11 · p. 107

7.12. The structure of the library

  • The library executes operations within contexts specifying filesystem roots, address spaces, filesystem type, credentials, and context-provided translation operations, across low-, medium-, and high-level calls.Supported compilation targets include the Linux kernel and user-level filesystems, with conditional headers selected for each environment.Source: §7.12 · p. 107, §7.12.1 · p. 107

7.13.1. VFS structures.

  • The design also defines VFS and transactional structures, including dentries, inodes, mounts, handles, cookies, acknowledgment locks, transaction numbers, and object identifiers.The cited material identifies these structures but does not provide their full definitions.Source: §7.13.1 · p. 108, §7.13.2 · p. 108

7.13.2. Transactional information.

  • Transactional information tracks transaction numbers, object identifiers, acknowledgments, handles, and log cookies, but its independence from OBDS remains puzzling.The acknowledgment locks are used on the server side for tracking acknowledgments.Source: §7.13.2 · p. 108

7.13.3. Context and credentials.

  • Context and credential structures capture filesystem location, memory context, user and group credentials, capabilities, and a FID-to-dentry callback.The callback accepts inode, generation, group-related, and opaque data arguments.Source: §7.13.3 · pp. 108–109

7.14.1. Lustre operations.

  • Lustre operations provide context callbacks and locking functions for the MDS and obdfilter, but some remain unimported into LVFS or need more structure.The pdirops are intended for import into Lustre’s LVFS for MDS use, while key LVFS operations derive from Linux VFS or libc equivalents.Source: §7.14.1 · p. 109, §7.14.2 · p. 109

7.14.2. Linux VFS operations.

  • The Linux VFS integration reviews Lustre’s existing fsfilt API and proposes slight modifications to improve its integration with the fsfilt approach.The supplied passage identifies ll_lookup_one_len and frames the discussion as a brief review.Source: §7.14.2 · p. 110, §7.14.3 · p. 110

7.14.3. FsFilt api.

  • The FsFilt operations structure defines callbacks for transaction start, commit and waiting, attribute changes, ioctl, metadata access, page reads, callback handling, filesystem statistics, synchronization, and prepared writes.Source: §7.14.3 · p. 110
  • The FsFilt design supports filesystem-type operation discovery and registration, including demand loading when needed, but its abstraction boundaries and journaled read/write operations remain unresolved.The proposed revisions would pass fsfilt operations as arguments, rename functions with lvfs_ prefixes, and address missing journaled write operations.Source: §7.14.3 · pp. 110–111

Lustre Logging API

This section describes the Lustre logging API for coordinating persistent updates, replication, recovery, and configuration management through logs, catalogs, cookies, generations, and epochs. It covers interfaces and operations for creating, processing, traversing, versioning, and removing logs, as well as recovering authoritative MDS file sizes from OSTs.

8.1. Introduction

  • The API hides local versus remote use, supports stateless authorized access, permits log growth and removal, and prevents remote callers from assuming open logs persist.Logs are stored in log objects, currently implemented as files, and serve orphan recovery, RAID1 synchronization, and configuration updates.Source: §8.2 · p. 112, §8.1 · p. 112

8.3.1. Logs.

  • Logs may be identified by names or object identifiers, contain structured records with headers, opaque aligned bodies, tails, and allocation metadata, and scale through catalogs.Large logs can use hierarchical catalogs, while records are accessed by iteration or llog_cookie locations.Source: §8.3.1 · pp. 112–113

8.4. Llog connections and the Cancellation API

  • Transactional logs coordinate distributed updates: replicators complete related changes, then cancel originator records after their transactions commit.This mechanism is designed to preserve replicator completion despite originator crashes or replicator rollback, without requiring rollback invariance.Source: §8.4 · p. 114

8.4.1. Llog connections.

  • Connections use generations and RPCs to synchronize catalogs, while cancellation daemons distinguish old from new cookies and drop queued cookies after cancellation timeouts.Generation changes occur around llog_connect processing, allowing recovery and cancellation handling to recognize connection epochs.Source: §8.4.1 · p. 114, §8.4.2 · pp. 114–115

8.4.3. Normal operation.

  • During normal operation, originators log each replicator update, replicators return cancellation cookies after commit callbacks, and originators cancel records and clean empty logs.The protocol batches cookies into pages for cancellation RPCs, linking log creation, replicated transactions, commit callbacks, and cleanup.Source: §8.4.3 · p. 115

8.5. Llog recovery handling

  • Recovery processes only records from previous generations, replays unfinished operations, cancels records for completed operations, and requires atomic, idempotent log removal after failures.After processing completes, size recovery allows the MDS to resume caching correct file sizes; reconnecting replicators may reprocess logs that reappear.Source: §8.5 · p. 117, §8.5.1 · p. 117

8.7.11.1. Prototype.

  • The llog_next_block prototype supports locating the log block containing a requested record index.The function accepts a log handle, current and next indices, and an output pointer for the current position.Source: §8.7.11.1 · p. 120

8.7.11.3. Return Values.

  • llog_next_block returns the block containing next_idx while using curr_idx and its offset to optimize the search.Source: §8.7.11.3 · p. 121

8.8. Sample Method Table Descriptions

  • The obd_llog API provides setup, cleanup, add, and cancel operations through implementations that redirect via MDS paths or invoke relevant OSC devices.Striping information determines which OSCs generate replicated log records, while catalog methods initialize, clean, add, and queue cancellation records.Source: §8.8 · p. 121

8.8.2. obd_llog_cleanup(struct obd_device *).

  • Catalog setup activates log headers and handles from per-OSC log IDs, cleanup releases initialized handles, and origin_add or repl_cancel add records or queue cookie cancellation.origin_add uses LOV striping metadata to descend the LOV device and generate cookies; cancellation can be immediate with LLC_CANCEL_NOW.Source: §8.8 · p. 121, §8.8.2 · p. 121

8.9. Configuration Logs

  • Configuration logs begin at version 0, preserve one full current log plus change descriptors, and let clients mount from or incrementally refresh their configuration.Clients fetch the current log at startup; on refresh, they compare versions, fetch changes, and apply them with llog_process and a callback.Source: §8.9 · p. 122

8.10. Size Recovery

  • Size recovery uses MDS I/O epochs: clients flush data and submit size and OST cookies, after which the MDS updates cached size and ends the epoch.If the final closer lacks authoritative information, the MDS requires an additional setattr; the OST retains the inode and epoch state during recovery.Source: §8.10 · pp. 122–123

The Lustre Filesystem

The Lustre filesystem integrates client, metadata-server, and object-storage subsystems through dedicated protocols, locking, caching, and coordinated metadata and file I/O mechanisms. Its architecture defines Unix-compatible and parallel I/O, object-based storage and layouts, update and recovery processes, and adaptive strategies for consistency, performance, and flexibility.

9.1. Introduction - Subsystems and Protocols

  • Lustre’s core comprises Client FileSystem, Meta-Data Servers, and Object Storage Targets, which may run separately or together and communicate through pairwise protocols.CFS provides cluster-wide Unix file-sharing semantics; MDS handles metadata, OSTs handle file I/O, and an OST–MDS protocol supports pre-allocation and recovery.Source: §9.1.1 · p. 124

9.1.1. The Principal Subsystems.

  • MDS provides a load-balanced, scalable metadata service that converts client requests into journaled, batched persistent updates, including concurrent updates to one object.Batching can combine requests from one client’s writeback cache or from multiple clients operating concurrently.Source: §9.1.1 · p. 124

9.1.3. Locking.

  • A single lock-service module can support client and MDS protocols, OST file-I/O protocols, and coordination among MDS systems.OSTs serve file locks for Unix semantics, flock, and specialized collective-I/O locking, while MDSs locate resource masters and grant object locks.Source: §9.1.3 · pp. 124–125

9.2. Mount Interfaces

  • Lustre mounts through an artificial root directory containing mntinfo, allowing global namespace entry into the root fileset and permitting mounts without network connections.The default mntinfo value is fset::ROOT and can be overridden through mount arguments.Source: §9.2 · pp. 125–126

9.3. Reading Meta-data

  • Metadata reads separate inode, directory, and dentry data: inode attributes arrive during lookup, directory contents during readdir, and dentries are established for names.Lustre fetches attributes before iget, uses OBD-protocol attributes for compact platform-independent RPCs, and can defer striping metadata until file_open.Source: §9.3 · p. 126, §9.3.1 · pp. 126–127

9.3.1.1. Inode Numbers and File Identifiers.

  • Lustre must map large distributed file identifiers to stable 32-bit VFS inode numbers, but no injective mapping generally exists; separate fileset superblocks can ease the problem at the cost of cross-fileset hard-link recognition.The constraint primarily affects user programs that observe inode numbers through stat, while the kernel can retain larger internal identifiers.Source: §9.3.1.1 · p. 127

9.3.2. Directory Data

  • Clients cache directory data for lookups and notify MDS of persistent changes through update records, while MDS remains authoritative and supports multiple directory formats, including hashed layouts.Directory RPCs can fetch selected hash chains or pages, and readdir must restart from a valid point when directory versions change.Source: §9.3.2 · pp. 127–129

9.3.2.1. Page-based Directory Data

  • Page- and dentry-based directory caching relies on locks and revalidation: revoked directory locks may purge or repair dependent dentries, while negative and inode-less dentries require special handling.Lookup follows directory-data lookup, attribute retrieval, and dentry/inode instantiation; its lock requirements vary by intent, with operation-based locking trading server execution against client writeback caching.Source: §9.3.2.2 · p. 129, §9.3.2.3 · pp. 129–130, §9.3.3 · p. 130

9.3.3.1. Locking and Lookup.

  • Intent-aware locking treats lock/lookup requests as a protocol with callbacks and supports converting directory-data or attribute locks once lookup intent is known.The protocol uses descriptors containing operation intent.Source: §9.3.3.1 · p. 132, §9.3.3.2 · p. 132

9.4.1. Locking in Distributed and Cluster Filesystems.

  • Lustre offers Unix semantics across nodes, but Linux VFS deficiencies require intent-aware lookup, lock pinning, promotion, and release to preserve coordinated checks.The proposed VFS enhancement passes operation intent into lookup, lets filesystem methods acquire and pin locks, and provides a later release phase.Source: §9.4.1 · pp. 132, 134, 135

9.4.2. Update Records.

  • Update records use DAFS-style packets with l_fileid identifiers, remain in client memory, and group dependent updates so multiple streams can be generated in parallel.These records provide concise, replayable descriptions of filesystem updates and are especially important for writeback caching.Source: §9.4 · p. 132, §9.4.2 · p. 135

9.4.3. Adaptation to Concurrency: Writeback and Client-server Mode.

  • Lustre adapts metadata updates to concurrency: clients receive writeback privileges at low concurrency, while the MDS executes operations at high concurrency.The transition is roughly where Avg (remote execution) / [Avg (writeback processing) + Avg (lock establishment)] equals 1.Source: §9.4.3 · p. 136
  • In client-server processing, the MDS can execute an update during lock acquisition, while the client records the result and updates cached objects without further blocking.Intent-based processing can reduce lock acquisitions and bundle RPCs with lookup requests.Source: §9.4.3 · pp. 136–137

9.4.4. Writeback Caching and Lock Revocations.

  • Writeback updates are associated with affected objects and propagated when locks are revoked, caches shrink, or synchronous operations such as sync and fsync require flushing.Object-level locking allows updates to different objects to proceed without serialization, while transactions affecting one metadata object are generally serialized.Source: §9.4.4 · p. 137
  • Multiple users sharing one update sequence can block one another when capabilities expire or are administratively revoked, a concern especially relevant to busy shared-file servers.A wedged update may force a client reboot, motivating further discussion of updates originating from different users on one object.Source: §9.4.4 · p. 138

9.4.5. Meta-data Update Protocol Calls.; 9.5. MDS Handling of Client Meta-data Update Requests

  • Lustre serializes metadata updates into records sent to the MDS, with reintegration supporting both memory-cache completion and later stable-storage notification.The design supports client-server and writeback modes while distinguishing server acknowledgment from durable commitment.Source: §9.5 · p. 139, §9.4.5 · p. 139

9.5.1. Server-based Lock Management.

  • Operation-based locking embeds requested modifications in lock operations, enabling the MDS to revoke or grant locks while performing updates, but complicating failure recovery.Lock versioning can avoid unnecessary refreshes, while recovery remains more complex after network or system failures.Source: §9.5.1 · p. 139

9.5.2. Server Infrastructure.

  • The MDS infrastructure treats file identifiers and names as distinct primary-object families, translating client name operations into identifier-based requests and representing updates and revocations accordingly.Persistent metadata uses object-style inodes and directory pages, while file inodes reference storage objects rather than file data.Source: §9.5.2 · p. 140, §9.5.3 · p. 140

9.5.3. The MDS Backend.

  • MDS backends may use filtering filesystems, object filesystems, shared block filesystems, or clustered databases, but recovery must yield a consistent metadata filesystem with atomic transaction-stream execution.The clustered-object option reuses object, locking, and recovery protocols, while concentrating a fileset’s metadata on one OST can create heavy-traffic bottlenecks and prevents cross-fileset transactions.Source: §9.5.3 · pp. 140–141

9.6. The File Protocol; 9.6.2. File Inodes.; 9.6.2.1. File Inodes.

  • Lustre stores file data on object devices such as direct drives, OSC-accessed OSTs, or LOVs, with MDS-held opaque descriptors specifying objects and optional striping across hundreds or thousands of OSTs.The MDS supplies access-control capabilities, while LOV and device configurations are resolved through UUID-based resource descriptors.Source: §9.6 · p. 141, §9.6.2.1 · p. 142, §9.6.2.2 · p. 142

9.6.2.3. LOV: Logical Object Volumes.; 9.6.2.5. LOV Tables.

  • LOV metadata requires iterable object tables supporting migration, snapshot operations, failures, and rebuilds, with distributed consistency between MDS-held LOV objects and subobjects maintained asynchronously.The design supports marking affected logical objects bad after device failure and updating tables during iteration.Source: §9.6.2.3 · p. 142, §9.6.2.5 · p. 143, §9.6.2.6 · p. 143

9.6.2.7. LOV Descriptors and Storage Management.; 9.6.3. Opening and Creating Files.

  • Device descriptors and UUID references enable automatic reconfiguration for migration, replication, encryption, and replacement, provided descriptor references change atomically after validation.Opening a file obtains object, quota, storage-node, and striping metadata; parallel OST opens then establish handles for page-cache or OS-bypass I/O.Source: §9.6.2.7 · p. 143, §9.6.3 · p. 144

9.6.4. Reading and Writing Files: System Interface.

  • Lustre preserves Unix file-I/O semantics by revoking cached-page read locks before writes through a lock service on a subset of involved OST targets.Source: §9.6.4 · p. 145

9.6.5. Reading and Writing: OS Bypass.

  • Parallel I/O uses authenticated user-level handles cloned from kernel file descriptors, vectored interfaces mapped to object operations, synchronous OS-bypass writes, and no extent locking.Concurrent use of the parallel-I/O handle and system interfaces is undefined.Source: §9.6.5.1 · p. 145

9.6.5.1. Parallel I/O

  • MPI-IO support can efficiently implement file views and collective operations over object storage, while view-to-buffer and striping-to-buffer translations enable aggregation.Cluster file project data structures and algorithms are identified as suitable for these logical object modules.Source: §9.6.5.2 · p. 146

9.6.5.3. Collective Operations.

  • Collective I/O can track participating systems and completion while rejecting requests from nonparticipants, coordinating synchronized writes across the group.Source: §9.6.5.3 · p. 146

9.6.6.1. Adaptive Striping.

  • Adaptive striping could dynamically select balanced or less-busy storage resources from detected write patterns, extending beyond fixed descriptors and existing ROMIO/HDF5 support.Proposed controls include lock modes, O_DIRECT mounting, and fcntl functions for changing striping behavior.Source: §9.6.6.1 · p. 146, §9.6.6.2 · p. 146

9.6.6.2. Special API’s.

  • OST allocation policies should exploit LOV striping hints, while improved read-ahead is considered crucial because read performance is currently poor.Source: §9.6.6.3 · p. 147

9.6.6.3. Allocation Policies.

  • A collaborative cache can proxy file reads by publishing cache servers to clients and exposing cached file sections through lock data, using dedicated or client nodes.Source: §9.6.6.4 · p. 147

File striping configuration

This section describes Lustre’s configurable, layered I/O and striping mechanisms across OSTs, including RPC and page operations, caching, locking, and group I/O. It specifies the coordination, size-tracking, transactional, and recovery protocols used to maintain distributed file consistency and manage client-side caching.

10.2. Default LOV striping pattern under Lustre

  • The system-wide LOV default defines stripe size and stripe count at filesystem creation, with stripe count 0 spanning all available OSTs and 1 selecting a single OST.Default object placement is selected from the MDS-returned inode number to distribute files across OSTs, while 32-bit platforms prohibit stripe_size × stripe_count values reaching 4GB.Source: §10.2 · pp. 148–149

10.3. Striping configuration per file

  • Per-file striping can be set before allocation through ioctl(2) or lstripe, specifying stripe size, starting OST, and stripe count, but cannot be changed after objects are allocated.With O_LOV_DELAY_CREATE, applications must call LL_IOC_LOV_SETSTRIPE before another process opens the file; otherwise the default pattern is used, and without the ioctl the returned descriptor is unusable.Source: §10.3 · p. 149

10.1. Introduction

  • Striping across OSTs improves parallelism, but striping every file across every OST can reduce aggregate performance through contention, extra object creation, and additional RPCs.Single-OST striping suits many small files or many clients creating separate files, whereas large files written concurrently by multiple processes should span enough OSTs to keep them busy.Source: §10.4 · p. 150

10.5. Maximizing application read/write performance

  • O_DIRECT improves regular large-file I/O by batching up to 64kB in one RPC and bypassing the client page cache, while regular I/O sends separate 4096-byte chunks and can be 5-6x slower for a single thread.O_DIRECT requires reads and writes aligned to and in multiples of 4096 bytes; its benefit is smaller for smaller files.Source: §10.5 · p. 150

10.6.1. Page Cache.

  • Linux 2.4 writeback tracks dirty and locked state across superblocks, inodes, pages, and buffer_heads, moving data through dirty, locked, and clean lists during synchronization.The page-cache hooks prepare_write, commit_write, and writepage are the principal address-space operations involved in write caching.Source: §10.6.1.1 · p. 151

10.6.1.2. Lustre’s 2.4 Caching Attempt.

  • The Linux 2.4 client cache uses prepare_write and commit_write for partial-write preparation and dirty-page marking, while writepage batches dirty pages into Lustre I/O and throttles when memory is constrained.writepage must write dirty locked pages and unlock them, enabling progress for synchronization, write, and truncation paths.Source: §10.6.1.1 · p. 151, §10.6.1.2 · p. 152

10.6.1.4. Lustre Caching in Linux 2.6.

  • The Linux 2.6 caching and mmap design considers file-size checks, permission and DENYWRITE interactions, page-cache eviction callbacks, lock conversion on write faults, cancellation, invalidation, and dirty mmap writeback.The implementation discussion leaves synchronization between unmapping’s fput() and writeback of mmap-dirtied data unresolved, while LOV forwards pages to the appropriate OSC with a completion callback.Source: §10.6.1.4 · p. 152, §10.6.2.2 · p. 154

10.6.2.2. = implications for lustre =.

  • Lustre mmap support must coordinate page faults with DLM locks, but delayed page installation can race with lock cancellation and expose stale data.Because nopage does not identify read versus write faults, the initial design would acquire a PW lock, limiting shared read mmap behavior; a callback after mapping installation could help release lock references.Source: §10.6.2.2 · p. 154

10.6.4. OSC.

  • The OSC batches pages into RPCs by collecting enough pages and limiting simultaneous requests through max_pages_per_rpc and max_rpcs_inflight.Single-thread performance can reach 300MB/sec, but depends on tuning.Source: §10.6.4 · p. 155

10.6.5. OST I/O.

  • OST I/O uses synchronous direct I/O with limited threads, larger-than-64K transfers, and optional read-cache promotion, but Lustre’s discontiguous pages prevent direct API use unchanged.The implementation therefore requires an approach that accommodates protocol pages lacking physical contiguity.Source: §10.6.5 · p. 155

10.7. I/O Layering and API’s

  • The asynchronous OBD API binds I/O state to pages through cookies, enabling layered queueing, priority changes, callbacks, RPC batching, and concurrent I/O.Its goals are to shield layers, improve stacking and concurrency, and distinguish persistent page bindings from per-I/O arguments.Source: §10.7 · p. 155

10.7.1. OBD api layer.

  • The OBD API’s preparation, queueing, flag-setting, teardown, and synchronous operations manage page/object state, RPC formation, completion callbacks, and errors such as -EBUSY.Cookies identify pages, queueing may block for capacity, urgent and ready flags influence scheduling, and teardown fails while I/O remains in flight.Source: §10.7.1 · pp. 156–158

10.7.2. Example use of the obd I/O methods.

  • Typical page-cache writes register and queue dirty pages, while writepage marks ready and urgent pages; when caching cannot accept them, synchronous I/O builds and triggers RPCs.These paths respond to conditions including zero grants or max_dirty_mb set to zero.Source: §10.7.2 · p. 158

10.7.3. OSC state machine.

  • The OSC state machine inspects page lists after preparation, queueing, or flag updates and invokes callbacks to prepare pages, refresh counts, and fill per-RPC metadata.Callbacks can lock pages, determine whether I/O may complete immediately, and populate the obdo associated with each object list.Source: §10.7.3 · p. 159

10.8. Read-ahead

  • Read-ahead remains problematic because existing mechanisms provide poor batching, stop at in-flight pages, maintain only one window, and leave initiation and result storage unresolved.The proposed lower-change path tracks a read-ahead window in ll_inode_info and invokes it near ll_file_read().Source: §10.8 · pp. 159–160

10.9.1. Definitions.

  • KMS provides each client with a lower bound on distributed file size, reducing file-size lock contention while remaining insufficient as the exact size.For inactive files, size comes from server inode sizes; for active files, it combines server sizes with the maximum KMS_i across clients holding cached stripes.Source: §10.9.1 · p. 160, §10.9.2 · pp. 160–161

10.9.2.1. Size probes.

  • Size probes return the OST’s current KMS without relinquishing existing locks, allowing probes on multiple stripes to proceed in parallel without deadlock.The client requests a probe lock; the server uses a probe callback when other locks exist and returns the resulting size.Source: §10.9.2.1 · p. 161

10.9.4. Locks.

  • Lustre protects file-data extents, file size, and KMS with distributed locks distinguishing read protection from write/update protection.Read locks prevent conflicting writes, while write locks protect updates and permit size advancement beyond the locked extent.Source: §10.9.4.1 · p. 162

10.9.4.3. Stripes and locks.

  • For striped files, determining KMS requires scanning locks across all objects, and unallocated stripes may complicate finding the next lowest valid KMS.The design notes that storing a KMS with each object might improve this calculation.Source: §10.9.4.3 · p. 163
  • The ldlm_size value block stores the lock server’s last known file size, is returned on enqueue completion, and usually increases while remaining at least the protected object size.Cancellation and glimpse replies update it; truncation can move it backward.Source: §10.9.4.4 · p. 164

10.9.4.5. Operations on ldlm_sz.

  • Multiple cancellations or glimpse-callback replies produce the same final ldlm_sz regardless of reply order when their locks exclude the -1 extent.This makes the resulting lock-server size independent of callback or cancellation ordering.Source: §10.9.4.5 · p. 164

10.9.5.1. System Call Semantics.

  • The system-call protocol specifies EOF-aware reads, append and ordinary writes, size-changing truncation, and mmap behavior through corresponding size and extent locks.Mmap establishes file size at call time and does not support extending writes; later file-size changes need not notify the mapped area.Source: §10.9.5.1 · pp. 165–166

10.9.6.1. file read.

  • Read operations use a known size or lower-bound KMS; when matching locks do not establish the needed bound, Lustre performs a size probe before reading.For striped files, the KMS is calculated from covering read and write locks, but must still be validated when it lies below the requested interval.Source: §10.9.6.1 · pp. 166–167

10.9.6.4. prepare_write.

  • Prepare_write updates KMS or a local variable according to the range’s relation to KMS without updating i_size, while truncate executes through OST operations between MDS open and close.Truncation uses an MDS open by FID and moves network operations into ll_setattr_raw.Source: §10.9.6.4 · p. 167, §10.9.6.5 · p. 168

10.9.7.2. Pseudocode for update_kms.

  • The update_kms cancellation logic selects the highest remaining extent lock under the KMS semaphore, records its end, and safely unpins the lock before releasing synchronization.If no lock remains, the cached KMS is not replaced.Source: §10.9.7.2 · pp. 168–169

10.9.7.4. Pseudocode for ll_file_write.; 10.9.7.5. Pseudocode for ll_prepare_write.

  • The write path records whole-file lock knowledge as cached size information, updates inode size from the lock’s KMS, and zero-fills pages when the cached maximum size covers the write offset.Otherwise, ll_prepare_write reads the page from the OST before proceeding.Source: §10.9.7.4 · p. 169, §10.9.7.5 · pp. 169–170

10.9.8. MDS size management.

  • Lustre coordinates MDS and OST file-size authority by using MDS caching when files are inactive and consulting OST object sizes during write epochs.The MDS owns the authoritative size while files are not open for write; active I/O shifts size validity to OST-managed state.Source: §10.9.8 · p. 170, §10.9.8.1 · p. 170

10.9.8.1. Client protocol during normal operation.

  • On close, clients communicate valid sizes and recovery cookies to the MDS, which updates inode attributes when the last writer closes and retries through OSTs when necessary.The protocol avoids extra RPCs and prevents DONE_WRITING from racing ahead of CLOSE.Source: §10.9.8.1 · pp. 170, 172

10.9.8.2. OST Size Log records.

  • OST size-log records transactionally capture object size changes by I/O epoch, pin affected objects, and remain until the MDS commits the corresponding size update.Cookies identify records for cancellation, while newer epochs supersede older pinning state and prevent premature cleanup.Source: §10.9.8.2 · pp. 172, 174

10.9.8.3. MDS getattr handling.

  • The MDS serves cached inode sizes only after recovery has restored pending OST size updates and while no I/O epoch remains active for the inode.Otherwise, getattr processing must not mark the cached size as valid.Source: §10.9.8.3 · p. 175

10.9.8.4. Definition of epochs, epoch numbering and open recovery.

  • Lustre defines an I/O epoch so any client able to change OST object size through writes or truncates implies that the corresponding MDS inode is open for write.Epochs increase globally, begin on writable opens, and are continued during open replay when recovery supplies a prior epoch.Source: §10.9.8.4 · pp. 175–176

10.9.8.5. MDS I/O Epoch management.

  • An MDS epoch continues while at least one writer remains open and ends when the file is no longer open for write, balancing authoritative-size availability against OST logging frequency.Longer epochs delay restoration of MDS authority, whereas shorter epochs cause size-update records to be written more often.Source: §10.9.8.5 · p. 176
  • MDS epochs end only when a file has no openers or pending I/O, while close processing transfers size authority back to the MDS through a two-phase protocol.The MDS tracks open files, commits OST size changes through cancellation cookies, and uses epoch numbers that are nonzero while active and zero when inactive.Source: §10.9.8.5 · p. 177, §10.9.8.6 · p. 177

10.9.8.6. MDS commit, failures and log recovery.

  • MDS recovery preserves size consistency by requiring an OST size log record before each size-changing setattr and replaying logs before canceling records.Resent close requests remain safe after rollback because the OST record still exists; network or I/O failures can abort recovery or require manual intervention.Source: §10.9.8.6 · p. 178

10.9.8.7. Client Failure.

  • After client failure, the MDS can close the client’s files and obtain the final size, log cookies, and active epoch from OSTs before issuing the size update.This recovery path handles cases where the failed client could not provide the correct file size or the MDS lacked cancellation cookies.Source: §10.9.8.7 · p. 179

10.10. Group Locks

  • Group I/O locks let cooperating processes share a group identifier and concurrent write lock, while other readers wait, fail with -EWOULDBLOCK, or are bypassed by later locks from the same group.Processes explicitly flush and release group locks; locks are also dropped when files close or processes exit.Source: §10.10 · p. 180

10.11. Lock server location and recovery

  • Lustre places file locks on the OST holding each object, addressing cross-system failure handling through either I/O fencing or lease renewal.The design identifies lock revocation as the central issue when a single OST manages locks for all stripes.Source: §10.11 · p. 180, §10.11.1 · p. 180

10.11.1. Leases.

  • Lease safety uses brwtime = (OST request queue timeout) + (PtlGet timeout) + (brw_kiovec time) so clients stop writing before locks can be reassigned after failure.The DLM delays new locks until I/O and lock timeouts pass, while cache sizing and client count can adjust timeout estimates.Source: §10.11.1 · pp. 180–181

10.12. Grants; 10.12.1. Summary.

  • OST grants let clients use write-back caches only when OST space is available, with grant sizes adapting to writes, client counts, and free space.Clients begin with a minimum or zero grant, receive updates in replies, and fall back to synchronous I/O when grants are unavailable; large aggregate grants can still cause dirty-data buildup and lock timeouts.Source: §10.12.1 · pp. 181–182

10.12.2.1. OST side.; 10.12.2.2. OSC side.

  • The OST computes grants and rejects unaccounted writes with -ENOSPC when space is unavailable, while the OSC adjusts client cache limits and resubmits requests synchronously when caching is disabled.The OSC starts with a zero grant, so the first write is synchronous; after invalidation, grant support is not reenabled and the OST cannot directly determine whether the client still listens.Source: §10.12.2.1 · p. 182, §10.12.2.2 · p. 183

Recovery

This section describes Lustre’s recovery mechanisms for client, server, and transient network failures, with the goal of restoring consistent, performant operation without shutting down the entire filesystem. It covers failure detection, failover and fencing, request replay or reconstruction, client-state restoration, and coordinated recovery of persistent MDS/OST state.

Recovery

  • Lustre must recover from client, MDS, OST, and transient network failures while preserving filesystem consistency, performance, and availability.Client state is memory-based, whereas MDS and OST recovery must coordinate persistent state and monitor which peer systems remain operational.Source: Recovery · p. 184

11.2.2. Client failure.

  • Clients recovering from failures replay locks or requests, reconstruct lost replies, seek failover servers, and are evicted if they miss the recovery window.Client failure also requires revoking locks and resources; servers can use pings and last-heard state to detect unresponsive clients.Source: Recovery · p. 184, §11.2.2 · pp. 187–188

11.1. Recovery infrastructure

  • Timeout expiration on requests, locks, bulk I/O, or failed connections triggers recovery.A request timer starts when the request is sent; an expired timer indicates failure, while -ENOTCONN can trigger recovery after an initial connection failure.Source: §11.1.1 · p. 185

11.1.2. Basic support.

  • Recovery uses epoch, generation, incarnation, transaction, connection-level, and persistent last-received or last-committed state to fence servers and classify requests.Servers stop accepting new requests during recovery, and clients determine whether requests should wait, be replayed, resent, or have replies reconstructed.Source: §11.1.2 · p. 185, §11.1.3 · p. 185, §11.2 · p. 186

11.1.3. Client Request Management.

  • Client request management separates delayed, sending, and replay requests so recovery can prevent unsafe I/O and repeat operations lacking confirmation.Requests remain delayed until recovery completes, while unconfirmed operations may be replayed or resent according to server processing and commit state.Source: §11.1.3 · p. 185

11.2.1. Persistent State Recovery.

  • Journaling recovers server-local disk state, but Lustre must additionally reconcile MDS and OST metadata to eliminate orphaned or missing object references.Logical logs record object creation and deletion across MDS and OST operations, with cleanup after the corresponding commits complete.Source: §11.2.1 · p. 187

11.2.3. MDS/OST failure.

  • MDS recovery transparently handles short client or network failures, while long failures produce client errors and eviction even as other clients continue after a short interval.MDS failover activates a standby server through cluster high-availability software and can retransmit requests to re-establish lost replies.Source: §11.2.3 · p. 188

11.2.3.1. Design for failover OSTs.

  • OST failover aims to recover read/write calls transparently without interrupting the rest of the filesystem, using active standby processing, request replay, and client-side data retention until disk commit.If no failover server exists, the failed OST becomes inactive and clients stop creating objects there while the remainder stays accessible; failback returns a client group for load balancing.Source: §11.2.3.1 · pp. 190–191, §11.2.3 · p. 190
  • Timeouts activate recovery after reconnect attempts fail, while lost replies are reconstructed from persistent logs or memory to handle both MDS and network failures.Failover cannot initially distinguish server failure from transient network failure; reply reconstruction uses one mechanism for committed server-failure cases and executed network-failure cases.Source: §11.2.4 · pp. 191, 193, §11.3.7 · p. 198

11.3.1. Introduction.

  • MDS recovery replays client-visible transactions in their original order while preserving attributes and fids, compensating for journal rollback and replies lost before or after commit.Journaling may lose executed but uncommitted requests, so clients retain redundant request data; synchronous execution is available for multiple-failure scenarios.Source: §11.3.1 · pp. 193–194, §11.3.2 · p. 194
  • A single-threaded journaling MDS provides serial transaction histories, while DLM locks, acknowledgments, and update streams support atomicity, isolation, ordering, and concurrency.Lustre’s design aims to improve on strict and serializable execution by making transaction histories fully serial, although this carries performance implications for the MDS.Source: §11.3.1 · pp. 193–194

11.3.2. Request lifecycle.

  • Requests progress through executed, replied, and committed states, with replied and committed able to occur in either order, creating distinct recovery cases.A committed request no longer needs replay, whereas a request with only a reply may still require recovery handling.Source: §11.3.2 · p. 194

11.3.3. Tracking request execution and replies.

  • Update streams let the MDS track each client’s last executed request and reconstruct failures by replaying executed requests, handling lost replies, and retransmitting requests beyond the received boundary.At most one lost request or outstanding request/reply pair occurs per stream, and reply acknowledgment lets clients advance only after the preceding reply is known.Source: §11.3.3 · pp. 195–196

11.3.4. Tracking committed requests.

  • Transaction numbers and per-stream FIFO submission correlate client request order with backend commit order, allowing recovery to report committed boundaries and replay multiple streams in the original sequence.The MDS persists last committed and corresponding request numbers for each stream and reports them to clients during recovery and normal replies.Source: §11.3.4 · p. 196, §11.3.5 · p. 197

11.3.5. Replay of the transaction sequence.

  • During replay, Lustre sorts and executes requests from the last committed through the last replied transaction, while reply acknowledgments preserve server-created data and resolve cross-client ordering dependencies.A double client–MDS failure can create a replay gap; synchronous MDS execution is available when cascading recovery failures are unacceptable.Source: §11.3.5 · p. 197, §11.3.6 · p. 197

11.3.6. Reply acknowledgment .

  • Reply acknowledgments let the MDS identify whether a retransmitted request executed, reconstruct its reply, and enforce ordering across clients, including for failed or read-only requests.Acknowledgments protect transaction read/write sets with locks, and unacknowledged replies can be synchronized and flushed so their transactions become committed and non-replayable.Source: §11.3.6 · pp. 197–198, §11.3.7 · p. 198

11.3.7. Lost reply boundary.

  • When a reply boundary is lost, read-like calls can be re-executed, status-bearing metadata calls reconstruct replies from the last received log, and open-file state is recovered using transaction identifiers.Reconstructed open replies include the recorded disposition and status, with locks rebuilt from the list of open files.Source: §11.3.7 · pp. 198–199

11.4. Overview of recovery algorithm

  • Client failure is detected through timeouts or periodic pings; the server evicts unresponsive clients and cleans up their locks and file handles so cluster operation can resume.The pinger mechanism avoids waiting for every failed client to time out before restoring normal operation.Source: §11.4.1 · p. 199, §11.4.2 · p. 199

11.4.2. MDS or OST failures.

  • Recovery fences failed connections, reconnects clients to failover servers, and replays, resends, or cleans up requests according to server commit and receipt state.Clients use last_rcvd and last_committed values to classify requests, while servers expose a recovery window before normal processing resumes.Source: §11.4.2 · pp. 199–200
  • After server recovery, previously connected clients reopen files, replay known-lost transactions, and re-establish locks, while clients absent from the prior connection incarnation must discard cached state and return errors.The server cleans up state for clients that miss the recovery window or are evicted.Source: §11.4.2 · p. 200

11.5. Persistent state recovery

  • Lustre preserves MDS/OST persistent consistency with logical logs: OSTs record object creation, MDS records unlink operations, and recovery uses these records to detect and repair inconsistencies.The consistency problem arises because file creation and deletion update OST objects and MDS inode references in separate steps, allowing crashes to leave orphaned or missing objects.Source: §11.5 · p. 201

Lustre Security

This section presents Lustre’s security architecture and mechanisms for authentication, authorization, identity and credential management, encryption, auditing, and protection against security threats. It also specifies related configuration, interoperability, revocation, recovery, portability, and trust considerations.

12.1.1. Usability.

  • Lustre supports LDAP, Active Directory, NIS, specialized databases, or local /etc/passwd and /etc/group files through standard NSS interfaces, avoiding a mandated authorization engine.It also reuses existing ACL structures rather than introducing a new access-control scheme.Source: §12.1.1 · p. 202

12.1.2. Taxonomy.

  • Lustre addresses unauthorized information release, unauthorized modification, and denial of resource usage, but only partly addresses denial-of-service threats.Servers are largely trusted to maintain persistent data, while clients may send only encrypted file data to reduce server trust.Source: §12.1.2 · pp. 202–203, §12.1.3 · p. 203

12.1.3. Layering

  • Lustre’s layered security model separates trust interfaces, authentication, identity and group management, authorization, cross-realm usage, encryption, and auditing.GSS establishes security contexts; authorization checks identities and ACLs, while auditing can cover clients, metadata servers, object targets, and credential or encryption-key services.Source: §12.1.3 · p. 203

12.2. Lustre Networks and Security Policy

  • Lustre assigns each network a policy ranging from no security to GSS integrity, encrypted RPC data, or encrypted data, balancing performance with protection.Trusted cluster networks can avoid cryptographic work, whereas insecure networks use stronger policies; intermediate cases may rely on mechanisms such as VPNs outside Lustre.Source: §12.2 · p. 204

12.2.1. Binding GSS Security to Portals Network Interfaces.

  • Portals Network Interfaces bind Lustre network traffic to GSS policies, with connectionless interfaces configured at startup and connection-based acceptors selecting policies when connections arrive.The selected policies are invoked before sending traffic or after receiving Portals events, using network interfaces and event queues.Source: §12.2.1 · pp. 204–205

12.3. GSS Authentication & Trust Model

  • Lustre uses the GSS-API to authenticate users and protect network integrity, selecting efficient backends for trusted networks while supporting insecure networks.GSS provides limited-lifetime credentials, message integrity, and message privacy; the current backend is Kerberos 5.Source: §12.3 · p. 205, §12.3.1 · pp. 205–206

12.4. Process Identity and Authentication

  • Process Authentication Groups separate processes by authentication event rather than uid, mitigating root setuid attacks when hardened kernels prevent root from altering kernel memory.PAGs can also group processes with different uids or separate processes sharing one uid, and are inherited across fork with new groups created through login-related operations.Source: §12.4 · p. 207, §12.4.1 · p. 207, §12.4.2 · p. 208

12.4.3. Implementation.

  • Lustre’s PAG implementation can associate a shared security-context structure with processes, fetch credentials through GSS-daemon upcalls, and cache security contexts by PAG.Before authentication, processes receive only anonymous credentials; after authentication, client and server associate user identities and group memberships with the PAG.Source: §12.4.3 · pp. 208–209, §12.4.4.2 · p. 209

12.5. The user and Group Databases

  • Lustre authorizes users through security descriptors containing uid, group membership, client-provided identities, and server-resident attributes obtained during authentication and cached in kernel memory.Attributes can govern setuid/setgid/setgroups acceptance, identifier-only inode access, decryption-key access, and backup restoration.Source: §12.5 · p. 210, §12.5.1 · p. 210

12.5.2.2. Forwarding credentials

  • To support unmodified servers, Lustre associates a server-resident capability with a client, fileset, and principal, permitting forwarding of user IDs, group IDs, and supplementary groups.By default, no such triple has this capability; privileged GSS forwarding can instead refresh credentials associated with the user ID and PAG.Source: §12.5.2.1 · pp. 211–212, §12.5.2.2 · p. 212

12.6. Cross Realm Authentication and Authorization

  • Lustre resolves cross-realm UNIX identity conflicts by creating local accounts and supporting fileset policies for untranslated, lazy, or synchronous remote-ID translation.This addresses clients’ name lookup limitations and servers’ need to represent remote principals as inode owners or ACL entries.Source: §12.6 · p. 212, §12.6.1 · p. 213

12.6.2. Lustre handling of remote user id’s

  • For non-local clients, Lustre determines locality from network and fileset settings, maps authenticated principals to server IDs, and translates other IDs to a configured non-local user.Remote ACL manipulation uses textual names through lfs because normal ACL-changing system calls are unavailable for remote operations.Source: §12.6.2 · p. 213

12.7. MDS Authorization: Access Control Lists

  • Lustre uses Linux POSIX ACL mechanisms for MDS authorization, optionally applying a separate access-control list for encrypted files after POSIX ACLs grant inode access.Pathname traversal is authorized at parent directories, while fid reuse requires a short-lived MDS capability to limit fid-guessing attacks.Source: §12.7 · p. 214, §12.7.1 · p. 215

12.7.2. Implementation Details

  • Lustre leverages Ext3’s indirect ACL storage scheme on servers, while its encryption design stores the encrypted file key and signed sharing list with the file.The SFS control list associates groups with file keys encrypted under group-server public keys, although its described integrity handling remains incomplete.Source: §12.7.2 · p. 215, §12.9.2 · p. 216, §12.9.3 · p. 216

12.8. Auditing

  • Lustre audits clients, metadata servers, and object servers through the smfs filter, correlating OSS records with MDS inode FIDs and sending tunable logs to syslog.A separate tool combines server and client records to scan for anomalies, while parent-directory pointers support traceable pathnames.Source: §12.8 · p. 215

12.9. SFS Style Encryption of File Data

  • The proposed SFS-style encryption gives each file a permanent random key, encrypts data with countermode, and protects sharing through a group server and signed control list.The group server encrypts the file key, signs the sharing list, and returns the key after an application presents an authorized identity; the design notes remaining ACL-integrity concerns.Source: §12.9 · p. 216, §12.9.1 · p. 216, §12.9.2 · p. 216

12.9.3. The SFS Access Control List.

  • Lustre supports SFS-style group-key access control, including threshold escrow, recursive group-server decryption, audit trails, and extensible group definitions.Users present ACL identities and credentials to group servers, which verify membership and return decrypted file keys; group servers must therefore be trusted.Source: §12.9.3 · pp. 216–217, §12.9.3.1 · p. 217

12.9.3.2. Weaknesses Noted.

  • SFS encryption has weaknesses: counter-mode malleability requires mixing, while distributed keys can preserve user access permanently and therefore require ordinary ACLs.Lustre stores SFS ACLs in extended attributes and supports principal-managed group databases, but users retaining keys remain a theft risk.Source: §12.9.3.2 · p. 217, §12.9.4 · p. 217

12.9.4. Lustre SFS

  • Lustre’s simpler SFS implementation runs group-key services on MDS nodes, authorizes reads and writes with POSIX ACLs, and supports none, partial, or full file encryption.Encrypted directory names are proposed through client-side name encryption and MDS-held directory keys, with clients decrypting returned directory listings.Source: §12.9.5 · p. 218, §12.9.6 · p. 218

12.10. File I/O authorization

  • File I/O authorization uses MDS-issued capabilities signed for OSS verification, binding permitted operations to object identifiers, principals, versions, and validity periods.Capabilities normally require encryption in transit; GSS authentication, per-principal capabilities, and OSS blacklists strengthen authorization and revocation.Source: §12.10.1 · p. 218, §12.10.2 · p. 218

12.10.3. Multiple principals.

  • Capability revocation is immediate through OSS blacklists, but trusted clients must discard cached capabilities; untrusted clients may retain access while credentials remain valid.Including object versions can expire capabilities immediately, increasing security at the cost of slower setattr operations.Source: §12.10.4 · p. 219

12.10.5. Corner cases.

  • Cache flushes, open replay, and write-back caching create authorization corner cases because permission changes, replicated cookies, and recovery can cause errors or require client trust.Lustre rechecks permissions during replay, potentially evicting clients, while write-back authorization differs between master and cache OSS nodes.Source: §12.10.5.1 · p. 219, §12.10.5.2 · p. 219, §12.10.5.3 · p. 219

12.11.1. Recovery and the security server.

  • Lustre relies on external LDAP and Kerberos/GSS services whose redundancy and protocol recovery remain outside Lustre, while Lustre refreshes credentials for long-running jobs.Systems without IP services require policy-based security restrictions, but the design aims for one security infrastructure across less powerful operating systems.Source: §12.11.1 · pp. 219–220, §12.11.2 · p. 220, §12.11.3 · p. 220

12.12.3. MDS configuration options.

  • Security configuration supports network modes, principal and pool authorization, encryption and audit settings, remote Lustre identities, and OSS raw-object access permissions for specified principals, targets, and clients.Source: §12.12.2 · p. 221, §12.12.3 · p. 222, §12.12.4 · p. 222

Configuration Management

Lustre configuration management combines XML or LDAP descriptions with layered startup tools that translate system profiles into low-level device configuration. It also supports centralized administration, monitoring, failover recovery, and planned dynamic reconfiguration through Lustre control infrastructure.

13.1. Introduction; 13.2. Driver Organization; 13.2.1. Normal System Startup.

  • Configuration files describe Lustre systems, while lconf invokes lctl to load modules, start services, and configure devices across clients, OSTs, and MDSs.Startup proceeds through ordered levels from networking and locking to devices and mount points, while retryable network failures permit parallel cluster startup.Source: §13.1 · p. 223, §13.2.1 · p. 225

13.3. XML Configuration Files

  • XML configuration files, generated by lmc and interpreted by lconf, organize Lustre configuration by levels and can reduce misconfiguration through DTD syntax checks.The levels span networking, ldlm, OBDs, OSC/OST, MDC/MDS, LOV, and filesystem mount points; parsed entries generate equivalent lctl commands.Source: §13.3 · pp. 225–226

13.4. Lustre LDAP Schema; 13.4.1. Putting Together LDAP & lctl

  • LDAP-based configuration centralizes large-cluster data, assigns reusable profiles, and uses abstraction layers that can configure a 10,000 node cluster through translated low-level commands.Systems retrieve profiles matching their identities, while UUIDs associate related entries and help limit accidental misconfiguration.Source: §13.4 · pp. 227–228

13.5. Configuration Management Tools; 13.5.1. LMC.; 13.5.2. LConf.; 13.5.3. LCtl.; 13.5.4. LFS (Following the AFS/Coda fs/Cfs Tradition).

  • LMC creates or merges XML descriptors from node, network, and component information, while LConf selects node configuration, loads required modules, and invokes lctl for device setup.LCtl performs low-level configuration, filesystem mounting, troubleshooting, and debugging; LFS provides Lustre-specific filesystem operations.Source: §13.5 · p. 228, §13.5.1 · p. 229, §13.5.2 · p. 229

13.6. SNMP; 13.7. Dynamic Reconfiguration

  • SNMP can monitor device-specific /proc entries and compare lconf dumps with visible configuration, while planned dynamic reconfiguration includes kernel-triggered LDAP queries, device suspension, and client configuration refresh.The proposed suspend operation would let a device empty itself and queue requests during reconfiguration.Source: §13.6 · p. 229, §13.7 · p. 230

13.8. Configuration and High Availability; 13.9. Using Lustre for Monitoring and Configuration

  • High availability updates LDAP with a failed-over service’s new identity so clients can discover its address and recover pending transactions, while future designs expose configuration and monitoring through Lustre filesystems.The proposed filesystem interfaces include a default configuration mount, client monitoring exports, centralized namespace construction, and read-write tuning through /proc/sys.Source: §13.8 · p. 230, §13.9 · pp. 230–231

Configuration Management

This section describes Lustre configuration management across clients, MDSs, OSTs, and OSDs, covering layered tools, startup, migration, and component addition or removal. It also identifies scalability, recovery, synchronization, and deletion-management concerns.

14.2. Driver Organization

  • Lustre loads networking, class, lock-management, and role-specific drivers as kernel modules on clients, MDSs, and OSTs.OSTs layer ExtN, filtering, and object-target drivers; MDS metadata functionality is integrated in mds, while clients use filesystem, MDS/OST, and striping drivers.Source: §14.2 · pp. 232–233

14.3. Low Level Configuration Command API

  • Low-level configuration uses lctl to prepare formatted ext3 block devices, establish network connections, attach named UUID-bearing devices, and issue type-specific setup commands.Devices can use partitions, logical volumes, RAID devices, loop devices, or ramdisks; setup arguments identify components such as OSTs, MDSs, OSCs, MDCs, and LOVs.Source: §14.3.1 · p. 233, §14.3.2 · p. 234, §14.3.3 · pp. 234–235

14.4. XML Configuration

  • XML configuration addresses the scale and relational complexity of manual setup: lmc generates descriptors, and lconf applies them while loading modules and preparing devices.The XML DTD mirrors low-level configuration, while LMC commands create nodes, networks, MDSs, LOVs, OSTs, and mount points; LCONF also supports cleanup and dry-run debugging.Source: §14.4 · p. 235, §14.4.1 · p. 235, §14.4.2 · p. 237

14.4.2.2. Example, lov.sh.

  • A sample LMC script creates a one-node LOV configuration with an MDS, two OBDs, formatting, and a client mount point.The script writes lov.xml and uses LCONF-style configuration inputs for the local node, MDS, LOV, OST, and mount point.Source: §14.4.2.2 · p. 238

14.5. Normal System Startup

  • Startup activates subsystems in levels from network through devices, locking, targets, clients, LOVs, and mount points, while retryable failures permit parallel cluster startup.Clients contact MDS and OST systems, and MDSs contact OST systems during startup.Source: §14.5 · pp. 238–239

14.6. LDAP

  • LDAP stores configuration using a schema matching the XML descriptors, supports failover updates, and lets clients query the active MDS or OST after timeout-driven recovery.The access tools modify attributes and query identifiers including active and failover UUIDs, network addresses, and filesystem metadata.Source: §14.6 · p. 239, §14.6.1 · p. 240, §14.6.2 · p. 240

14.7.1. Introduction.

  • Configuration distinguishes an OST as a specific object-storage target from an OSD as its physical access path, and LOVTGTS stores OST UUIDs rather than OSD UUIDs.OSD UUIDs are used by configuration tools, whereas the actual OST UUID is used when setting up an OSD.Source: §14.7.1 · p. 242
  • OSTs can be added to existing OSDs or introduced with new OSDs, while one OST cannot currently be shared across multiple file systems.Multiple OSTs may share one OSD to provide different access controls, and migration moves all OSTs to the target OSD.Source: §14.7.1 · p. 242

14.7.2. Configuration Issues.

  • The configuration-management design was presented as a set of migrating-driver principles rather than a finalized solution.The text introduces the principles but does not provide their detailed implementation.Source: §14.7.3 · p. 242

14.7.3. Migration.

  • The proposed migrating driver would run on the target, create requested objects while migrating data, synchronize object numbering, iterate source objects, and pull data.Configuration management follows the DogFoodConfig approach, and inactive OST handling may reuse existing temporary-deletion code.Source: §14.7.3 · p. 242, §14.7.4 · p. 242

14.7.4. Addition and Removal.

  • Addition and removal must transparently coordinate OSTs, OSDs, in-flight requests, recovery, configuration changes, deleted identifiers, and scalable object cleanup without walking the MDS file system.A proposed alternative uses configuration-version matching with retries and client-driven deletion based on MDS fids embedded in OST objects.Source: §14.7.4 · p. 243

Redundant Object Storage Targets … Portals Diffs

This section specifies Lustre’s end-to-end architecture, interfaces, and operational mechanisms for distributed storage, metadata, locking, networking, caching, and I/O. It also details recovery, failover, configuration, monitoring, portability, and resource-management designs across clients, servers, and storage targets.

15.1. Introduction

  • Mirrored OSTs replicate objects across different targets, share read load, and use LOV-controlled writes with striping configuration stored on the MDS.The design mirrors every object on another OST and lets clients write through the LOV to both replicas.Source: §15.1 · p. 244, §15.2 · p. 244

15.2.2. Defining failures.

  • Writes succeed after reaching one OST, while replica failures are masked and repaired lazily through replication logs; recovery must resolve inconsistent or orphaned writes after power loss.The design identifies substantial RAID1 recovery issues and considers primary selection, per-object recovery, and logged synchronization.Source: §15.2.2 · pp. 244–245, §15.2.3 · p. 245

15.2.4. Sync on close

  • Disabling updates and new allocations on a partially failed mirrored pair reduces double-load risk, accelerates resynchronization, and prevents newly created objects from having only one replica.Recovery upcalls update allocation policy, while OST-resident replication records support resynchronization.Source: §15.2.5 · p. 246, §15.2.6 · p. 246

15.3. Other Mechanisms for use in an archive

  • Archive protections include pathname-aware load balancing that redirects reads to suitable replica servers, immutable inode attributes, synchronization on close, and scripts for rebuilding failed OST pairs.The listed implementation targets also include LOV mirrored writes, failure masking, recovery upcalls, and live-cluster OST support.Source: §15.3.1 · p. 246, §15.3.2 · p. 247, §15.4.1 · p. 247

16.1. Introduction

  • Collaborative caching addresses read scalability by allowing client or dedicated cache nodes to serve replicated pages, reducing requests directed to OST storage devices.The proposed COBD layer uses existing client and server infrastructure while adding caching nodes that service reads.Source: §16.1 · p. 248

16.3. New Architecture

  • The proposed architecture adds secondary caching OSTs backed by in-memory COBD pages, with target OSTs referring client reads to caches while retaining lock service at the target.Caching OSTs use the same code modules with different backends, and referred requests are restricted to read service.Source: §16.3 · p. 249, §16.3.1 · p. 250

16.3.1.1. Referral Policy and API for lock query

  • Referral policy queries lock holders and selects caching OSTs with compatible PR locks, optionally populating empty caches through standard OSC operations and distributing requests round-robin.The referral returns a caching OST hostname or UUID, with future optimization possible by returning referrals alongside read locks.Source: §16.3.1.1 · p. 251, §16.3.2 · p. 251

16.3.3. No Lock Service

  • COBD requires extent-specific non-optimistic locks, invalidation after writes, lock reacquisition for stale referrals, and integration with the client page cache to maintain consistency.The design also identifies cache-selection and extent-range policies, while persistent cache reuse after reboot is not immediately planned.Source: §16.3.1 · p. 250, §16.3.4 · p. 252, §16.3.5 · pp. 252–253

17.1.1. Layering.

  • The writeback cache adds a local MDS and WBC layer, allowing Lustre Lite to serve cached metadata locally while the WBC handles remote cache misses.The local MDS exposes the remote MDS API, while the WBC coordinates cache queries and acquisitions from the remote MDS.Source: §17.1.1 · p. 254, §17.1.3 · p. 255

17.1.4. Local updates.

  • Local updates create transactional or in-memory log records that a pressure-sensitive daemon propagates to the remote MDS and cancels after remote commit.Cached data is purged only after associated log entries are flushed, and cache fullness controls asynchronous or synchronous flushing.Source: §17.1.5 · p. 255, §17.1.6 · p. 255

17.2.1. Write back locks.

  • Subtree writeback locks validate cached namespaces through nanosecond-granularity ancestor change times, but conflicting remote operations revoke locks and may cancel writeback mode.Open objects subject to remote operations cannot use writeback mode unless operations commute with those remote operations.Source: §17.2.2 · p. 256, §17.2.3 · p. 256, §17.2.5 · p. 256

17.2.4. Lock revocation.

  • Lock revocation flushes writeback metadata logs and converts affected open MDS and OST objects to remotely opened locks; expiration likewise flushes remaining logs before cancellation.Subtree locks may be nested, and the first ancestor lock found is canceled after its associated update log is flushed.Source: §17.2.4 · p. 256, §17.2.6 · p. 257

Caching and XDSM

  • Lustre implements an XDSM API as part of its writeback cache, while the broader cache design addresses validation, reintegration, misses, and space management.The proposed cache wraps an underlying file system so cache modifications can proceed under distinct VFS inode locks.Source: Caching and XDSM · p. 258, §18.1 · p. 258

18.3.1. Mechanisms.

  • The cache uses persistent LRU-like records and extended inode attributes to evict eligible, nonbusy, nondirty, unpinned leaf objects while preserving transactional updates.A refined segment-level access scheme is considered more difficult, and files that cannot be cached entirely may be excluded.Source: §18.3 · p. 259, §18.3.1 · pp. 259–260

Lib Lustre - a Lustre Library Implementation

  • Liblustre provides portable Lustre access through a thin virtual file-system layer, sharing kernel implementation code and targeting POSIX systems from microkernels to Windows.The Cray Red- Storm implementation uses Sandia’s libsysio, while Windows support uses the Wine interface.Source: Lib Lustre - a Lustre Library… · p. 261, §19.1 · p. 262

19.1. Introduction

  • The portable library replaces Linux-specific build, configuration, kernel-API, and threading dependencies with cross-compilation, generic interfaces, wrapped C-library calls, and adaptable threading environments.GNU autoconf configures the source tree, with Linux presently identified as the supported build system.Source: §19.1 · pp. 262–263, §19.2 · p. 263

19.2. Build System

  • Lustre’s build system supports Linux, POSIX Lib Lustre, single-threaded Lib Lustre for microkernels, and Windows, with Portals compiled using the same cross-compilation flags.Linux is the default host target and may be omitted from configuration commands.Source: §19.2 · p. 263

19.3. Configuration Management

  • Configuration uses Linux ioctl calls to /dev/lustre, while user-space libraries use existing server configuration, lconf-generated dump files, and Unix-domain sockets for dynamic setup.Single-threaded clients require a startup function that performs all initialization, including filesystem registration and mounting.Source: §19.3.1 · pp. 263–264

19.4. PtlRpc module

  • The portable PtlRpc module preserves callback services, RPC waiting and timeout handling, and failed-RPC recovery across Linux kernel, multithreaded user-level, and single-threaded client environments.A potential limitation is that clients can wait on multiple event queues when multiple network interfaces are present.Source: §19.4 · p. 264

19.4.1. Modifications for liblustre.

  • Liblustre adapts event handling by reserving a non-overwriting serving queue, allowing callback-managed queues to wrap, and using PtlEQWait and PtlEQGet for conditions and urgent requests.The main thread can wait on only one event queue, while callbacks preserve events that must remain available.Source: §19.4.1 · p. 265

19.5. OSC & MDC

  • The object storage client and distributed lock service are considered readily portable, whereas the metadata client requires substantial interface rework; LibSysio provides asynchronous I/O completion through Portals events.The OSC design avoids non-Lustre types and structures, while metadata abstractions were recognized as important only late in the project.Source: §19.5 · p. 265, §19.6 · p. 265, §19.7 · p. 265

Quality Of Service - Implementation, APIs

20.2.1. QOS implementation.

  • The client normally allocates objects round robin but switches to QoS priority lists when thresholds are violated, prioritizing free blocks, object counts, and percentage of storage available.Priority lists are updated and rescanned at configurable intervals, with stale entries refreshed through statfs when necessary.Source: §20.2.1 · p. 268

20.2.2. Data structures and tunable parameters.

  • QoS state is carried in a 60-byte inline structure and maintained through sorted target lists, locks, timestamps, tunable intervals, and imbalance thresholds.The implementation limits statfs frequency because benchmarking showed that excessive statfs calls slow OSTs.Source: §20.2.2 · pp. 269–270

20.5.1. Read Ahead.

  • Read batching issues all pages for an ost_brw_read command before waiting, while OST read-ahead fills the page cache but adapts windows to traffic and QoS requirements.Client read-ahead should begin only after the OST starts the data stream, with tracing added for quality assurance.Source: §20.5.1 · p. 273

20.5.2.2. Known optimizations.

  • Range-based block allocation reduces repeated searches, journal activity, and fragmentation, with CPU studies expecting typical servers to exceed 100 MB/sec writes.The proposed bmap change allocates a range of blocks at once rather than one by one.Source: §20.5.2.2 · p. 274

20.5.3.1. File Size.

  • Storing final file sizes on the MDS after the last close avoids querying every OST, using replay logs and recovery procedures to preserve updates across failures.The same scheme can recover mtime and ctime values, though replay-log overhead may make the current algorithm preferable for small files.Source: §20.5.3.1 · pp. 274–275

20.5.4.2. Proposed changes to the read algorithm.

  • Sparse-file reads avoid transmitting zero-filled holes by using bmap and per-page status flags, preserving client transparency while reducing pages sent over the network.The OSC reconstructs zero-filled pages for unmapped regions after receiving status information, while Portals must handle partial bulk transfers and buffer mapping.Source: §20.5.4 · p. 275, §20.5.4.2 · p. 277, §20.5.4.3 · p. 277

20.5.5.2. Changes to replace PtlPut with PtlGet.

  • Replacing PtlPut with OST-issued PtlGet lets the OST pull write data after preparing sink pages, with completion signaled by PTL_EVENT_GET.The design also returns rapidly changing OST load information with each client request for application scheduling decisions.Source: §20.5.5.1 · p. 279, §20.5.6 · p. 279

20.5.6. OST Status return code.

  • OST replies can report operations per second, CPU utilization, free blocks, and network bandwidth by extending RPC reply state with an OST status structure.The OST handler tracks operations per second, and ptlrpc_reply attaches the status to every reply.Source: §20.5.6 · p. 280

20.6.1. Delayed object allocation.

  • LOV delayed allocation creates objects for larger files only when a stripe requires them, while retaining immediate allocation as an optional acceptance-test behavior.Collections of small files therefore use a single object.Source: §20.6.1 · p. 280

20.6.2. Object allocation and striping policies.

  • LOV striping policies expose per-directory or per-file controls over stripe size and the number of striped objects, enabling profiling of their performance effects.The QOS target-list logic orders targets using collected metrics while honoring requested stripe counts and preferred starting OSTs.Source: §20.3.5.4 · p. 272, §20.6.2 · p. 281
  • Object allocation and striping policies are expected to vary by workload, with profiling focused on stripe size and the number of clients writing stripes.CFS proposes simple mechanisms for changing file-system behavior, including settings on directories or open files and command-line configuration.Source: §20.6.2 · p. 281, §20.6.3 · p. 281

20.6.4. OST status information at MDS.

  • Lustre’s MDS can use OST space, load, and network-bandwidth status to distribute data and make QoS decisions, especially for multimedia applications.The MDS may periodically query OSTs, with frequency adapted to expected access patterns.Source: §20.6.4 · p. 281, §20.6.5 · p. 282

20.6.5. QOS from OST.

  • OST QoS can guarantee minimum or maximum bandwidth, temporarily exceed a minimum allocation without contention, and reduce service when competing guarantees become active.Clients receive QoS information on open and present it with subsequent requests so OSTs can prioritize service.Source: §20.6.5 · p. 282

20.7.1. Implementation mechanisms.

  • Lustre favors ioctls, mount options, and selected special APIs for file-system behavior, rejecting environment variables as inappropriate for kernel-facing configuration.Special APIs include cache-page invalidation, disabling lock fetches, and potentially reporting file sizes without file-size locks.Source: §20.7.1 · p. 283, §20.7.2 · p. 283

Introduction

  • Lustre’s implementation design uses POSIX and Linux standards, underlying journaling file systems, layered Portals networking, driver-specific APIs, distributed locking, and recovery protocols.Portals carries Lustre messages over NAL-supported devices, while handles, extent locks, and epoch, incarnation, and generation numbers support coordination and recovery.Source: Introduction · pp. 285–286

22.2. Multiple Portal Interfaces

  • Multiple Portals interfaces are initialized from a maintained interface list, with per-NAL event queues and peer structures selecting the correct interface for communication.Loaded NALs receive event-queue handles, while later-loaded NALs are ignored after ptlrpc initialization.Source: §22.2 · p. 287, §22.2.1 · p. 287, §22.2.2 · p. 288

22.2.2. Service Initialization.

  • Service initialization allocates per-NAL request queues and buffers, but uniform sizing can waste memory, while rotating queue checks prevent starvation across interfaces.Because incoming events are unsolicited, a full queue drops requests and can falsely trigger recovery by making senders suspect failure.Source: §22.2.2 · p. 288

22.3. Buffer Manipulation

  • Lustre messages use host-order transmission and receiver-side conversion, with early length validation, typed swabbing, and helper APIs for packing, sizing, locating, and safely unpacking buffers.Unpacking returns EINVAL when the supplied length cannot contain the message and its described buffers.Source: §22.3 · pp. 288–289, §22.3.1.4 · p. 289, §22.3.2.4 · p. 290

22.3.4.3–22.3.8.4. Message Swabbing

  • Lustre swabs incoming messages and request or reply buffers in place, skipping already-correct byte order and using structure-specific routines for fixed and variable-length data.Variable-length structures require separate swabbers for fixed and variable portions; arrays are processed element by element after their count is derived from buffer length.Source: §22.3.4.3 · p. 291, §22.3.6.4 · p. 291, §22.3.8.4 · pp. 292–293

22.3.5–22.3.7. Message Helpers

  • Message helpers allocate and pack counted buffers, validate bounded or NULL-terminated strings during unpacking, and return NULL or -ENOMEM on failure.String extraction accepts either an unrestricted NULL-terminated buffer check or a specified maximum length.Source: §22.3.5.4 · p. 291, §22.3.5.2 · p. 291, §22.3.5.3 · p. 291

22.4. Client Setup & Cleanup

  • Client setup resolves or allocates UUID-matched connections, increments connection references, initializes client control structures, and later decrements references while freeing client resources.A connection is returned with an incremented refcount, and releasing its final reference places it on the unused list.Source: §22.4 · p. 293, §22.4.1.4 · p. 293, §22.4.2.4 · p. 294

22.5.1. Server Request Processing

  • Server RPC processing uses a ring of preallocated request buffers, reference counts active requests, and recycles unlinked buffers when handlers finish.Reply buffers are allocated by request handlers and freed by a Portals event handler after replies leave the system; about 6,000 client threads use a few megabytes of preallocated memory.Source: §22.5.1.1 · p. 295

22.5.2. Service Initialization

  • Service initialization allocates request lists, buffers, portals, locks, event queues, and match entries, then wakes service threads to invoke request handlers.The request-buffer ring is statically set to RPC_RING_LENGTH value 10 and should exceed the number of request-handling threads.Source: §22.5.2.3 · pp. 295–296, §22.5.2.2 · p. 295

22.5.3–22.5.5. Service Lifecycle

  • Service lifecycle APIs attach worker threads, stop all service threads, and unregister services by freeing buffers and match entries, with cleanup errors possible for pending requests or failed unlinking.Thread startup may return -ENOMEM or kernel-thread errors, while unregistering can return PTL_NOINIT when PtlMEUnlink fails.Source: §22.5.3.4 · p. 296, §22.5.3.3 · p. 296, §22.5.4.4 · p. 296

22.6. Bulk Movement

  • Bulk transfers use registered page IOVs and Portals operations: PtlPut moves source data to remote sinks, while PtlGet retrieves remote data into local sinks.The backward-compatible PTL_MD_IOV option interprets the memory-descriptor start address as an IOV pointer, supporting transfers of up to 16 elements.Source: §22.6.2 · p. 297, §22.6.1 · p. 297

22.6.3–22.6.4. Bulk Operations

  • The ptlrpc bulk-put and bulk-get routines move all mapped pages as a single IOV, using PtlPut for reads and PtlGet for writes.Both operations require initialized page lists, page counts, connections, and portals, and can fail when memory or Portals components are uninitialized.Source: §22.6.3.4 · p. 298, §22.6.3.1 · p. 298, §22.6.4.2 · p. 299

22.7. Flowchart for Bulk Data Movement

  • Lustre bulk movement transfers data from a source to a sink after the client requests a server operation, using Portals registration and callbacks.Read and write operations reverse source and sink roles, while callback handlers clean up descriptors and wake processing that waits for completion.Source: §22.7 · p. 301, §22.7.1 · pp. 301–302, §22.7.2 · p. 303

22.7.1. Bulk data movement for reads

  • For reads, the client registers sink buffers, the server registers source buffers, and the server pushes data with PtlPut, followed by an acknowledgment.The read path uses ptlrpc_bulk_put to prepare a Portals memory descriptor with a bulk IOV; rendezvous handshakes are omitted from the figure.Source: §22.7.1 · p. 301, Figure 22.7.1 · p. 302

22.7.2. Bulk data movement for writes

  • For writes, the client registers source buffers, the server initiates transfer with PtlGet, and the client returns payload data to server sink buffers.The ptlrpc_bulk_get source waits for the sink to initiate and complete transfer, while the sink callback frees the descriptor and wakes waiting processing.Source: §22.7.2 · pp. 302–303, Figure 22.7.4 · p. 305
  • The earlier PtlPut-based write implementation had unsafe completion ordering, because acknowledgments could wake the OSC before the OST completed the write.The OSC could then perform dependent operations and discover later that the OST had not finished.Source: §22.7.2 · p. 303

22.8. RPC’s

  • Portals RPC functions send replies and errors, retransmit requests for protocol recovery, and transmit requests after preparing reply buffers, with documented allocation and initialization failures.ptlrpc_error uses ptlrpc_reply to send errors back to the client.Source: §22.8.1.4 · p. 304, §22.8.1.3 · p. 304, §22.8.2.4 · p. 305

22.9. Local RPC

  • Local RPC avoids network traversal by dispatching requests directly to local service handlers and copying bulk data between source and sink page lists.The client_obd local_rpc field is configured through probing, and local variants handle sending, replies, errors, and bulk movement.Source: §22.9 · p. 306

23.1. Introduction

  • Lustre messages encapsulate request and reply buffers inside Lustre message structures, which are themselves carried in Portals packet headers.Messages may contain multiple buffers or requests, including request headers, pathnames, and intent-locking requests.Source: §23.1 · p. 308

23.2. Portals message headers

  • Portals traffic uses PtlPut for Lustre requests, replies, and bulk reads, and PtlGet for bulk writes, with packet-specific headers carrying identifiers and match information.Each header has a fixed common portion and a type-dependent portion; xid values identify requests but may repeat during recovery retransmissions.Source: §23.2 · pp. 308–311, Table 2 · p. 310

23.3. Lustre Messages: RPC’s

  • Lustre packages requests and replies in Portals-backed messages, while bulk data uses raw Portals packets; messages carry buffers, require replies, and support recovery metadata.Messages and buffers are padded to 8-byte boundaries without counting padding in lengths; proposed fields include transaction, connection-generation, and protocol-version information.Source: §23.3 · pp. 311–312

23.4. OSC - OST Network Protocol

  • The OSC–OST protocol defines eleven operation codes with operation-specific buffers for connection management, object attributes, reads, writes, statistics, and object requests.Connections exchange target and client UUIDs plus an import handle; OBD objects describe object attributes, while read/write requests use per-extent I/O descriptors and remote buffers.Source: §23.4 · pp. 313–314
  • OSC–OST reads return per-page status through niobufs, whereas writes use client-prepared Portal match bits so the server retrieves bulk data without a Lustre RPC header.OST_STATFS replies report filesystem capacity through obd_statfs, including free and total file objects and blocks.Source: §23.4 · p. 314

23.5. Lustre DLM Network Protocol

  • The lock manager uses three regular calls and two callbacks, routing regular requests through affected-service portals and callbacks through the DLM RPC portal.Every lock request contains an ldlm_request or ldlm_reply, with lock descriptions specifying file extents and resource descriptors identifying the locked resource.Source: §23.5 · p. 315, §23.5.1 · pp. 315, 317

23.5.3. Message structures for the various locking operations.

  • Locking operations encode enqueue, cancellation, conversion, blocking callbacks, and grant callbacks through ldlm_request structures and operation-specific reply messages.Cancellation uses only lock_handle1; conversion supplies the requested post-conversion mode, while grant callbacks may change the requested resource name or lock mode.Source: §23.5.3.1 · p. 317, §23.5.3.2 · p. 317, §23.5.3.3 · p. 318

23.6. Client / Meta-data Server

  • The MDC–MDS protocol parallels OSC–OST messaging and assigns dedicated portals for requests, replies, and readdir bulk traffic alongside numbered metadata calls and update records.MDS messages commonly include an mds_body, while file identifiers and variable-length striping metadata support platform-independent metadata exchange.Source: §23.6 · pp. 318–319, §23.6.1 · p. 319

23.6.1. Meta-data Related Wire Structures.

  • Metadata update records begin with a 32-bit opcode and encode setattr, create, link, unlink, rename, and recovery operations using operation-specific records and supplementary name or striping buffers.Create messages can include file striping metadata or symbolic-link targets; rename carries source and destination names, while recreate is formatted like create with a different opcode.Source: §23.6.1 · p. 320, §23.6.2.1 · p. 320, §23.6.2.2 · p. 320

23.6.3. MDS Request record packet structure.

  • MDS request structures cover attribute lookup, open/close, page reads, and root discovery using file identifiers, handles, update records, and replies carrying metadata bodies.GETATTR behavior differs by caching mode: WB uses the file identifier, whereas CS uses the parent-directory identifier; open replies include a remote handle, security token, and inode attributes.Source: §23.6.3.1 · p. 322, §23.6.3.4 · p. 323, §23.6.3.5 · p. 323

23.7. Client - MDS/OST recovery protocol

  • Periodic OBD_PING messages let clients detect MDS/OST failures, join recovery or failover, and avoid losing cached updates when recovery windows expire.The opcode is understood by both OST and MDS nodes and has value 400 with empty request and reply bodies.Source: §23.7 · p. 324

NAL’s: Network Abstraction Layers for Portals

  • Portals uses a Network Abstraction Layer to support different networks, including RDMA, with asynchronous wire operations completed through lib_finalize notifications.NALs may rely on libraries, programmable NICs for protocol offload, or basic kernel device/IP-socket packet transmission.Source: NAL’s: Network Abstraction Layers for… · p. 326

24.2. KDAPL NAL

  • The KDAPL NAL initializes a provider-backed interface and event dispatcher, then uses queued events and service threads to handle connections, RDMA completions, and errors.Passive nodes accept connection requests, while active nodes connect to offered service points and process completion events.Source: §24.2 · p. 326, §24.2.1 · p. 327

24.2.3. Connection management. / 24.2.4. Receive handler.

  • KDAPL receive handling advances a connection state machine from headers to packet bodies, using Portals descriptors, CTS messages, and RDMA reads or writes for larger transfers.Events wake handler threads that analyze incoming data and advance the state machine until lib_finalize generates the completion event.Source: §24.2.4 · p. 329

24.2.5. Portals methods.

  • KDAPL Portals methods pin memory and may bind local and remote memory regions so cb_send can perform remote DMA, while RTS/CTS-style transfers are expected to scale better than repeated copying.The initial implementation may have scaling difficulties because small-packet sends can loop while copying data according to connection credits.Source: §24.2.5 · pp. 329–330

24.3. SockNAL / 24.3.1. Initialization. / 24.3.2. Interaction with the user space NAL library component. / 24.3.3. SockNAL Scheduler.

  • SockNAL initializes Portals and kernel data structures, creates CPU-scaled scheduler and reaper threads, and exchanges connection setup with a user-space socket component through ioctl.Schedulers move queued packets fairly across connections, while nonblocking transmission and reception use connection queues, socket-buffer callbacks, and a receive finite-state machine.Source: §24.3.1 · p. 331, §24.3.2 · p. 331, §24.3.3 · p. 331

25.1. Introduction

  • Stackable OBD drivers use a common stub API over storage devices, with driver-dependent semantics and integer status returns based on Linux i386 error numbers.OBD layers may include filter, snapshot, LOV, MDS, and caching drivers; heterogeneous systems must translate and encapsulate error values.Source: §25.1 · p. 334

25.2.1. iocontrol

  • The ioctl interface supports driver-specific administrative operations, including formatting, partitioning, snapshot and RAID manipulation, recovery, and LOV striping configuration.Outstanding questions concern user-space address parameters, which ioctls should be generic, and whether methods should use a generic upcall system.Source: §25.2.1.4 · p. 335, §25.2.1.5 · p. 335, §25.2.1.6 · p. 335

25.2.2. get_info / 25.2.3. set_info

  • The object API provides named device-parameter retrieval and updates, with get_info returning values such as block size and set_info rejecting unsupported parameters.The design calls for a simple database interface and agreement on mandatory information supplied by direct OBD drivers.Source: §25.2.2.4 · p. 336, §25.2.2.5 · p. 336, §25.2.2.6 · p. 336

25.2.4–25.2.7. Device lifecycle

  • OBD lifecycle methods attach, setup, cleanup, and detach configure and unconfigure drivers while enforcing ordering and preventing removal when devices or clients remain active.Attach associates device-specific methods, setup performs configuration, cleanup flushes or unmounts resources, and detach requires closed connections; the boundary between attach and setup remains unsettled.Source: §25.2.4.4 · p. 338, §25.2.5.4 · p. 338, §25.2.6.4 · p. 339

25.2.8–25.2.10. Connections and status

  • Connections establish per-client device state and are later closed by disconnect, while statfs reports device type, block size, capacity, and object counts.Logical drivers connect to child drivers during initialization, and snapshot replication can reduce reported free blocks and objects even without creating new objects.Source: §25.2.8.4 · p. 340, §25.2.8.5 · p. 340, §25.2.9.4 · p. 340

25.2.11.4. Description.

  • Preallocation reserves up to 32 objects for a connection to reduce round-trip latency and support writeback-cache space reservation, but it is implemented and not yet exploited.The number actually preallocated is returned and may be lower than requested.Source: §25.2.11.4 · pp. 341–342, §25.2.11.6 · p. 342

25.2.12.5. Examples.

  • The create operation allocates an object using OBDO hints for identifiers, mode, size, layout, encryption, and locality, although none of these creation hints are currently used.If no valid identifier is supplied, the drive selects an available one; preallocated identifiers can be requested explicitly.Source: §25.2.12.4 · p. 342, §25.2.12.5 · p. 343, §25.2.12.6 · p. 343

25.2.14.3. Return Values.

  • Destroy deletes specified objects, setattr writes only valid-flagged attributes, and getattr retrieves only requested metadata fields, while metadata sharing across drivers risks false sharing.Destroy flags could later support secure erasure or deletion-triggered backups, and sufficiently rich metadata remains an open issue.Source: §25.2.13.4 · p. 343, §25.2.13.5 · p. 343, §25.2.14.4 · p. 344

25.2.16.3. Return Values.

  • Open creates an object handle so the device can track usage and prevent destruction while the object is in use, whereas close validates the client connection and handle before closing it.The open request obtains object identifiers from the metadata service before contacting the target OST.Source: §25.2.16.4 · p. 345, §25.2.16.5 · p. 345, §25.2.17.4 · p. 346

25.2.18.3. Return Values.

  • Asynchronous brw performs page-aligned vector I/O with callbacks across driver start and finish phases, while OSC implementations support up to 16 I/O-vector entries.Reads currently transfer full pages, and writes were corrected to use the actual write length and avoid extending files with data beyond end-of-file.Source: §25.2.18.4 · p. 347, §25.2.18.5 · pp. 347–348

25.2.18.5. Examples.

  • The I/O path still has unresolved limitations: locked-page writes may require bounce-page handling, and sparse-file reads currently transmit zero-filled unmapped pages unnecessarily.The interface could be extended for raw, asynchronous, parallel, and collective I/O.Source: §25.2.18.6 · p. 348

25.2.19.3. Return Values.

  • The preprw/commitrw pair prepares page-aligned object I/O buffers and filesystem space, then writes or reads those buffers before commitrw persists writes and frees resources.Buffers are grouped by object, with preprw requiring page-aligned offsets and page-sized lengths; commitrw follows successful preprw.Source: §25.2.19.4 · p. 349, §25.2.19.6 · p. 350, §25.2.20.4 · p. 351

25.2.21.3. Return Values.

  • The enqueue and cancel APIs manage referenced locks by matching, converting, enqueuing, or releasing handles, although cancel only decreases reference counts rather than actually cancelling locks.enqueue may return any lock protecting the operation and may handle multiple objects; current issues include replacing extent parameters with a generic cookie and resolving callback header dependencies.Source: §25.2.21.4 · p. 352, §25.2.21.5 · p. 352, §25.2.22.2 · p. 353

25.2.23.3. Return Values.

  • The sync, punch, migrate, and copy methods provide device flushing, hole creation or truncation, object-to-object migration, and object copying, with page-alignment requirements for migration or copying.sync can flush a selected range or all outstanding object I/O, while current implementations migrate or copy entire objects despite requested partial-operation interfaces.Source: §25.2.23.4 · p. 353, §25.2.24.4 · p. 354, §25.2.24.5 · p. 354

25.2.23.6. Outstanding Issues.

  • Current limitations leave sync unused, truncate-through-punch incompletely specified, migration limited to whole-object data, and copy unable to honor offset and count for partial transfers.The snapshot driver uses migrate for copy-on-write movement, while the copy destination inherits size and block fields from the source.Source: §25.2.23.6 · p. 353, §25.2.24.6 · p. 354, §25.2.25.6 · p. 355

25.2.27.4. Description.

  • The iterate method invokes a repeat function for each existing object and supports resuming after interruption through startid, but it is non-atomic and requires the repeat function to execute on the device.The snapshot driver uses iteration to remove or restore snapshots by filtering objects through the repeat function.Source: §25.2.27.4 · p. 356, §25.2.27.6 · p. 356

25.3. Caching Nodes

  • Caching nodes improve read scalability through a COBD layered over OSC, honor only reads while target OSTs grant locks, and fall back to the target OST when data is uncached; OSC–OST writes currently use PtlPut.The caching-node read path is shown in Figure 25.3.1, where the client obtains a target-OST read lock and receives a cache-node referral.Source: §25.3 · p. 357, Figure 25.3.1 · p. 357, §25.4 · p. 357

25.4. A Diagram of Bulk Data Movement Between OSC and OST.

  • PtlGet at the sink is proposed as an alternative to PtlPut because it significantly simplifies bulk-transfer error handling and synchronous I/O.Source-side network failures still require an event so buffers can be released; the alternative is illustrated in Figure 25.4.2.Source: §25.4 · p. 357, Figure 25.4.2 · p. 358

25.5. Class Driver Interfaces

  • Class-driver interfaces expose ioctl-based access to the generic OBD interface, while selected devices may additionally use read/write commands for daemon upcalls.The class driver is a character device with open and close operations but no read/write methods by default, and it exports functions for other modules.Source: §25.5 · pp. 358–359
  • OBD class routines register and unregister device types, resolve devices and connections by name, index, UUID, or handle, and manage exports, imports, caches, and connection lifecycles.Registration can fail with EEXISTS or ENOMEM; unregistration can fail with EINVAL or EBUSY, while connection creation and lookup return NULL or error codes for invalid handles, devices, or memory exhaustion.Source: §25.5.2.3 · p. 359, §25.5.3.3 · p. 360, §25.5.4.3 · p. 360

25.5.9–25.5.16. Connection and Export Management

  • Connection routines allocate and initialize export state, validate handles through cached cookie information, disconnect individual or all device connections, and free associated buffers.class_connect returns EINVAL for NULL connection or device inputs and ENOMEM when export allocation fails; class_disconnect returns EINVAL when freeing a nonexistent client.Source: §25.5.9.4 · p. 361, §25.5.10.4 · p. 362, §25.5.11.4 · p. 362

25.6. Basic Support Functions

26.1. Introduction

  • The MDS stores Lustre inodes, attributes, and striping metadata, while MDC/MDT components handle client requests and persistent metadata access using locking APIs.While files are open, OSTs may have more accurate size and time attributes; the MDS becomes authoritative after the final close, requiring synchronization across MDS and OSTs under single failures.Source: §26.1 · p. 367
  • Lustre plans writeback metadata caching and clustered metadata servers to improve metadata performance, while evolving metadata APIs toward stackable first-class interfaces.The metadata APIs described are current and may evolve in future implementations.Source: §26.1 · p. 367

26.2. Attribute Management

  • File size is authoritative on the MDS for closed files, but open files, failed protocols, and recovery require aggregating object sizes across the file’s OST stripes.The design considers size locks, client callbacks, MDS size caching, and a single per-file lock manager to reduce the cost of distributed size computation.Source: §26.2.1 · p. 368

26.3.1. WB Cache organization.

  • Metadata writeback uses a local MDS-backed cache and a writeback client that serves cached operations locally, records updates, and later propagates them to the remote MDS.The cache may reside in memory or on persistent storage; cache misses use existing MDC APIs, while log records support recovery and are flushed under memory pressure or other requirements.Source: §26.3.1 · p. 370, §26.3.1.2 · p. 371, §26.3.1.3 · p. 371

26.3.3. Cache Purges

  • Directory caching is validated with locks, cached-range metadata, readdir cookies, and nanosecond-granularity subtree change times, while cache purges flush committed logs before removing data.Subtree locks follow a single-writer/multiple-reader model, can expire or nest, and are revoked when conflicting remote operations or noncommuting open-object operations occur.Source: §26.3.3 · p. 372, §26.3.4 · p. 372, §26.3.5.1 · p. 372

26.3.6. Preallocation

  • The metadata writeback design supports recovery by detecting failover, re-establishing writeback locks, reintegrating changes, and using preallocated objects whose identifiers can be transferred to clients.Lock revocation flushes writeback logs and converts locally open objects to remotely opened locks; preallocation can turn some MDS create operations into renames.Source: §26.3.5.4 · p. 373, §26.3.7 · p. 374

26.4. Clustered MDS implementation

  • Clustered metadata servers partition inode groups across nodes, use enhanced file identifiers and directory entries to route requests, and support directories split across multiple MDS nodes.A clustered metadata driver maintains MDS connections, while shared update and recovery protocols coordinate clients, MDS servers, and MDS-to-MDS operations.Source: §26.4 · p. 374, §26.4.1 · p. 375, §26.4.2 · p. 375

26.4.2. Data Structures.

  • Clustered MDS dispatches each request to one metadata server using inode groups, name hashes, directory offsets, extended attributes, and clustering descriptors.The LMV uses multiple MDC drivers, each managing traffic to one MDS; directory extended attributes describe how large directories are split across MDSs.Source: §26.4.3 · p. 375, §26.4.2 · p. 375

26.4.4. MDS implementation.

  • Directory creation and cross-directory updates may involve multiple MDSs, while directory splitting improves distribution but imposes penalties on splitting, renames, and later unlink operations.mkdir creates directories on another MDS; unlink, rmdir, and rename may span MDSs, whereas unaffected operations remain on one MDS when no split is needed.Source: §26.4.4 · pp. 376–377, §26.4.4.1 · p. 377

26.4.5. Recovery

  • Clustered-MDS recovery designates the node receiving a change request as coordinator, coordinates locks and commit checks across participants, and journals transaction sequence numbers for replay.The coordinator can split an oversized directory before committing; participating nodes commit in memory, journal replay records, and acknowledge before coordinator cancellation.Source: §26.4.5.1 · p. 377, §26.4.5.2 · p. 377

26.4.5.2. Mechanisms.

  • Transactions surviving on any participating node can be replayed on all nodes, while crash ordering uses acknowledgements as lock cancellation and clients replay lost transactions to their original MDS.Orphaned children are cleaned only after replay, and standby MDSs can take over through ring-based nearest-working-neighbor failover.Source: §26.4.5.2 · p. 377, §26.4.5.4 · p. 378

26.4.6. Locking

  • Locking orders multiple objects by ancestor, MDS number, and inode number, with global filesystem locks reserved for selected cross-directory renames and proofs applying to DLM protection locks.Cross-directory rename locking may use local and remote locks or an all-local alternative; shared ancestors can replace the global lock for some renames.Source: §26.4.6 · pp. 378–379

26.5. Metadata Client APIs

  • Metadata client operations use mdc_rpc_lock to prevent races across request and reply, while intent-based open performs server-side open work during lookup and avoids another RPC.The API includes intent enqueueing, root-status retrieval, attribute retrieval, filename-based attribute lookup, filesystem status, and attribute-setting requests.Source: §26.5 · p. 379, §26.5.1.3 · p. 381, §26.5.2.4 · p. 381

26.5.1.1. Prototype.

  • The metadata APIs define typed inputs for exports, identifiers, intents, locks, attributes, extended attributes, callbacks, and request outputs across their operation prototypes and parameter specifications.Lock callbacks can invalidate local caches, and attribute requests return selected fields plus LOV data or symbolic-link targets when applicable.Source: §26.5.1.1 · p. 380, §26.5.1.2 · p. 380, §26.5.3.2 · p. 382

26.5.1.3. Return Values.

  • API failures report conditions including invalid arguments or replies, insufficient memory, missing objects, I/O errors, interruptions, and timeouts, and failed lock handles cannot be trusted.Filesystem-status requests return status through osfs, while mdc_getstatus also returns the latest processed transaction number to support recovery.Source: §26.5.1.3 · p. 380, §26.5.2.4 · p. 381, §26.5.3.3 · p. 382

MDC return values

  • MDC operations report success or operation-specific failures including insufficient RPC-buffer memory, invalid arguments or packet types, missing objects, stale metadata, and storage exhaustion.Examples include ENOMEM, EINVAL, ENOENT, ESTALE, and ENOSPC across close, read, create, unlink, link, and rename.Source: §26.5.7.3 · p. 385, §26.5.8.3 · p. 385, §26.5.9.3 · p. 386

MDC metadata operations

  • MDC APIs create, close, read, unlink, link, and rename filesystem metadata through RPC requests to the MDS.Creation supports files, directories, symlinks, and special files; unlink may return object attributes for deleting corresponding OST objects.Source: §26.5.7.4 · p. 385, §26.5.8.4 · p. 385, §26.5.9.4 · p. 386

26.6. Logical Medata Volume API

  • The LMV driver provides metadata APIs that retrieve root and file attributes, dispatch creation and enqueue requests, and perform lookup, link, and intent-lock operations.Root-FID retrieval contacts the primary MDS, while attribute retrieval may split RPCs across MDSs holding objects associated with a FID.Source: §26.6 · p. 389, §26.6.1.4 · p. 389, §26.6.2.4 · p. 390

LMV return codes and locking

  • LMV metadata and locking calls return zero on success and otherwise pass nonzero errors to callers, while enqueue and intent-lock requests carry lock modes, intents, buffers, and callbacks.The callback-data release function always returns zero because its underlying lock-management callback reports no errors.Source: §26.6.1.3 · p. 389, §26.5.13.3 · p. 389, §26.6.2.3 · p. 390

LMV target routing

  • LMV routes operations to the appropriate underlying metadata driver using target selection based on FIDs, directory-name hashing, or the MDS holding a file handle.Name-based callback updates hash the name with the parent-directory object structure, while close locates the MDS owning the handle.Source: §26.6.3.2 · p. 391, §26.6.4.4 · p. 391

LMV creation and implementation scope

  • The LMV create operation dispatches creation to a selected metadata target, while one listed function is explicitly marked for removal rather than implementation.The lmv_create interface carries ownership, mode, device, symlink, security-context, FID, and extended-metadata information; another entry states, “Will not be implemented, to be removed.”Source: §26.6.5.1 · p. 392, §26.6.5.4 · p. 392, §26.6.6 · p. 392

27.1. Introduction

  • Lustre separates metadata locking at metadata servers from file-I/O locking at OSTs that store file objects.The lock manager implementation is organized around data structures, then APIs supporting these operations.Source: §27.1 · p. 400

27.2.1. Data Structures.

  • The DLM organizes client- and server-side namespaces as resource hash tables containing typed resources, lock queues, lock state, extent ranges, callbacks, remote handles, and wait queues.Namespaces distinguish client and server roles; resources track PLAIN, EXTENT, or MDSINTENT locks and maintain granted, converting, waiting, and temporary-RPC queues.Source: §27.2.1 · pp. 400–401

27.2.2. Data Structures: the Evaporation/Refcounting Problem.

  • Reference counting prevents locks and resources from being freed while referenced, with creation and destruction transitions coordinating access, callbacks, and list membership.Lock handles combine a structure address with a random cookie for server-side validation; resources are destroyed only when their reference counts reach zero.Source: §27.2.2 · p. 401, §27.2.3 · pp. 401–402, §27.2.4 · p. 403

27.2.5. Callbacks.

  • Callbacks use completion and blocking ASTs, server-to-client RPCs, and deferred work items to process lock-tree changes without holding the global namespace lock during lengthy callbacks.The lock manager must also tolerate heavily re-entrant calls and callback-related concurrency.Source: §27.2.5 · p. 403, §27.2.6 · p. 404

27.2.6. Blocking

  • Lock-manager updates cannot call blocking routines while structures are modified, and related replies, blocking ASTs, and completion ASTs may arrive in arbitrary orders.RPCs are placed on lr_tmp, while callback-based reply handling is favored over blocking; enqueue responses can indicate blocked converting, granted, or waiting locks.Source: §27.2.6 · p. 404, §27.2.7 · p. 404

27.3.1.1. Prototype.

  • The lock-enqueue APIs accept namespaces, resource identifiers, lock types, modes, extent cookies, callbacks, and output handles, returning errors or flags that describe blocking and lock changes.Intent locks may return a lock on a newly created child rather than the requested parent; invalid handles and allocation failures return EINVAL or ENOMEM.Source: §27.3.1.1 · p. 405, §27.3.1.3 · p. 405, §27.3.1.4 · p. 405

27.3.3.2. Parameters.

  • Lock matching searches a namespace’s resource queues for an existing lock, returns its handle, increments its reference count, and reports 1 on a match or 0 otherwise.Passing a null namespace supports duplicate-lock detection.Source: §27.3.3.4 · p. 408, §27.3.3.3 · p. 408

27.3.4.3. Return Values.

  • Lock lifecycle calls cancel locks, discard unused zero-reference locks, convert locks to new modes, and decrement references that can trigger pending callbacks.Cancellation can free the local lock copy and resource; unused-lock cancellation may be local-only, while conversion can block on granted or converting locks.Source: §27.3.4.4 · p. 408, §27.3.5.4 · p. 409, §27.3.6.4 · p. 409

27.3.10.3. Return Values.

  • Namespaces and resources are reference-managed: namespace creation enables client or server lock service, namespace freeing cleans resources and locks, and resource lookup hashes parent/name identities while reusing or creating entries.Resource release succeeds only when the reference count reaches zero and associated lock queues are empty.Source: §27.3.10.4 · p. 411, §27.3.11.4 · p. 411, §27.3.12.4 · p. 412

27.3.15.3. Return Values.

  • Lock and resource helpers maintain queue membership, references, descriptions, and handles: locks can be added or unlinked from granted/converting/blocked queues, converted to handles, and described by mode, extent, and version.Reference increments update local reader/writer counts according to locking mode, while lock release can destroy associated resources and connections.Source: §27.3.8.4 · p. 410, §27.3.15.4 · p. 413, §27.3.16.4 · p. 413

27.3.22.3. Return Values.

  • Changing a lock’s resource supports intent-based locking, allowing a metadata server to replace a parent-directory lock with a lock on a newly created directory.The operation returns 0 on success and ENOMEM when allocation of the new resource structure fails.Source: §27.3.22.4 · p. 415, §27.3.22.3 · p. 415

28.1. Introduction

  • The broader Lustre architecture comprises client, metadata-server, and object-storage-target subsystems, with metadata stored persistently through an underlying filesystem such as ext3 with extensions.The section also records documentation updates describing expanded explanations and pictures.Source: §27.4 · p. 416, §28.1 · p. 417
  • The implementation exposes a POSIX-compliant VFS layer over Linux caches and journaling filesystems, with metadata handled by the MDS and file I/O directed to OSTs.Lustre reuses Linux page-cache APIs and may write through a temporary page when the target page is locked.Source: §28.1 · p. 417, §28.2 · p. 417

28.3. File Locking Strategy

  • OSTs grant byte-range locks that permit concurrent access to different extents, while special [0,-1] and [offset,-1] locks protect file-size knowledge and size-changing operations.The optimistic lock manager grants the largest available nonconflicting extent and returns the file size with granted locks.Source: §28.3.1 · p. 419

28.3.3. Client Lock Namespace

  • LOV presents multiple OSTs as one device, stripes files across them, and decomposes file-extent lock requests into per-OSC requests for the corresponding stripes.A request for [0, -1] is expanded to every OSC in the striping pool.Source: §28.3.4 · p. 420

28.3.5. Lock refinements

  • Intent-based callbacks can return current file sizes from clients holding high-offset write locks without revoking those locks, reducing RPC traffic and unnecessary lock yanking.The optimization targets common size-query operations such as ls -l.Source: §28.3.5 · p. 422

28.4. Meta-data Locking

  • Metadata locking protects cached dentries, inode attributes, and directory data across clients, the MDS, and OSTs, using read, write, and exclusive locks according to each operation.Directory reads and lookups use read locks, updates use write locks, and rmdir requires an exclusive lock.Source: §28.4 · p. 422, §28.4.1 · p. 422

28.4.2. Writeback and Intent Based Updates

  • Intent updates keep directory write locks at the MDS, whereas writeback updates let clients modify cached directory data and batch transactions for later metadata flushes.Writeback locks can cover a directory subtree, requiring a lock tree that reflects directory hierarchy.Source: §28.4.2 · p. 423

28.5. Writeback Cache for File I/O

  • Writeback caching changes file writes from synchronous write-through to deferred, batchable flushes triggered periodically, by memory pressure, or when pages are needed.The implementation addresses Linux 2.5 and Linux 2.4 to support backporting.Source: §28.5 · p. 425, §28.5.1 · p. 425

28.5.2. Writeback Cache Design (Based on Linux 2.4).

  • Linux 2.4 writeback caching batches dirty pages for asynchronous OST updates, but requires space reservation to prevent later cache flushes from returning ENOSPC.Dirty pages are flushed through synchronization, periodic update, memory-pressure, or allocation-blocking paths; lock revocation also forces affected pages to disk.Source: §28.5.2 · p. 426

28.5.3. Writeback Cache Design (Based on Linux 2.5).

  • Linux 2.5 marks committed pages dirty in the writeback cache and later flushes them through pdflush or lock callbacks, with reservations preventing unsafe cache growth.A shared ll_writepages_internal helper writes either broad ranges for pdflush or exact file extents required during lock revocation.Source: §28.5.3 · pp. 426–427, §28.5.3.1 · p. 428

28.5.3.1. Periodic Flush of Dirty Data.

  • Periodic dirty-data flushing must pass the file extent being written back to ll_writepages, especially when a lock revocation targets only part of a file.The extent is carried alongside the permitted page count in write_back_control.Source: §28.5.3.1 · p. 428

28.6. System Call Implementation

  • Lustre system-call implementations trace client, MDS, LOV, and OST interactions in a cluster using logical-volume striping, covering mounting, metadata, data, and namespace operations.The described calls include sys_mount, sys_stat, sys_write, direct I/O, file creation, unlink, and mkdir.Source: §28.6 · p. 428, §28.6.1 · p. 429, §28.6.3 · p. 430

28.6.1. sys_mount.

  • During sys_mount, the client reconnects to cached or newly initialized MDS state, obtains LOV and root-file-set information, transfers root attributes, and establishes an analogous OSC connection.Per-client structures preserve mount counts, transaction and request results, open files, and granted locks for subsequent operations and recovery.Source: §28.6.1 · pp. 429–430

28.6.5. sys_open.

  • File creation acquires an MDS intent lock, creates striped objects across OSTs when needed, opens client-side and server-side handles, and stores striping metadata on the MDS.The open path retains per-client export data so requests can be replayed during filesystem recovery.Source: §28.6.5 · pp. 430–431, §28.6.6 · p. 431

29.1. Lustre Recovery Management

  • Lustre recovery uses connection levels, request lists, transaction numbers, and journaling to fence I/O, replay uncertain requests, and reconcile MDS–OST persistent state after failures.Requests move from sending_list to replay_list after timeout, while delayed requests remain blocked until recovery completes; journaling handles disk recovery, leaving orphan consistency to Lustre.Source: §29.1.1 · pp. 436–437, §29.1.2 · p. 438

29.2.1. Basic Recovery Scheme.

  • After MDS or OST failure, clients discover replacement servers through an upcall, suspend new I/O, reconnect, and replay retained requests according to transaction ordering.Transactions at or below last_committed can replay in any order, whereas later transactions are queued and processed strictly in order.Source: §29.2.1 · pp. 438–439
  • Lustre recovery separates replayable from resendable requests, restores open filehandles, and resumes normal processing after client recovery completes.A recovery barrier stops reply interpretation while pending requests are timed out; open and close replay preserves open-unlink and O_CREAT semantics.Source: §29.2.1 · p. 439, §29.2.2 · p. 439

29.2.2. Open request recovery handling.

  • Open and close requests use transaction numbers and retained references so failover preserves operation ordering and releases replay state only after close commits.Open requests remain ordered through committed transactions, while close callbacks transfer and eventually drop the file-handle reference.Source: §29.2.2.1 · pp. 439–440, §29.2.2.2 · p. 440

29.3. Server Recovery

  • Server failover uses shared storage with MDS active-standby takeover and OST active-active takeover, followed by recovery or load balancing when the original server returns.Servers initiate recovery after transfer or lock-related timeouts and when clients reconnect.Source: §29.3 · p. 441, §29.3.1 · p. 441

29.3.1. Basic Server Recovery Scheme.

  • During server recovery, reconnecting clients receive a recovery window, while late clients may be evicted and replay begins only after expected clients reconnect or signal LAST_REPLAY.The server tracks recovering clients, delays queued transaction processing, and prevents new non-recovery requests until replay completes.Source: §29.3.1 · p. 441, §29.3.2 · p. 442

29.3.2. Replay mode on the MDS server.

  • MDS replay processes requests by transaction number, recreates client-visible state and identical object identifiers, and aborts recovery on transaction gaps by evicting clients.The design trusts clients to replay state correctly and may wait for obd_timeout before treating a missing transaction as failure.Source: §29.3.2 · p. 442

29.4. OST/MDS Persistent State Recovery

  • OST/MDS persistent-state recovery addresses inconsistencies caused by failures during object creation, removal, or synchronization between OST objects and MDS inode metadata.The section proposes infrastructure and algorithms for repairing orphaned objects, missing objects, and inode/object mismatches.Source: §29.4.1 · pp. 442–443

29.4.2. Logging infrastructure.

  • Recovery logging uses sequential variable-length MDS and OST logs, 128-bit cookies, batched cancellation RPCs, and small log objects of up to 64kB.Cookies identify the log object, index, and offset; small objects reduce wasted space and recovery-time data traversal.Source: §29.4.2 · pp. 443–444

29.4.3. Create Logging.

  • Create logging records OST object identifiers against MDS file identifiers, cancels records after metadata commits, and repairs crash cases by replaying creates or scanning OST logs to destroy unreferenced objects.OST crashes can lazily recreate missing objects, with unsynced data potentially lost; MDS recovery synchronizes OSTs, scans uncommitted records, and removes orphan logs and objects.Source: §29.4.3 · p. 444, §29.4.3.1 · p. 444, §29.4.3.2 · p. 445

29.4.4. Unlink Logging.

  • Unlink logging records each stripe deletion per OST, returns cancellation cookies through the client, and destroys log objects after all entries commit.The MDS transactionally logs deletions, while OST log_cancel operations batch cookies and clear committed entries from MDS logs.Source: §29.4.4 · p. 447, §29.4.4.1 · p. 447

29.4.4.1. Normal Operation.

  • OST crashes can leave objects undestroyed or lose unlink cancellations, while MDS crashes can produce file holes or empty OST files.If the client remains available, replay resends destruction; otherwise the OST reads recovery logs and locally destroys committed objects.Source: §29.4.4.2 · p. 447, §29.4.4.3 · p. 447

29.4.4.4. Client Crash.

  • Client-crash recovery handles orphaned objects from interrupted create or unlink operations through normal MDS–OST coordination.Failures can occur before object creation is recorded on the MDS or before ost_destroy removes objects after inode deletion; the recovery protocol addresses both cases.Source: §29.4.5 · p. 448

29.5. Gateway Node Failover

  • Gateway failover removes failed gateways from routing decisions, broadcasts peer-death notifications, restores gateways after reboot, and retries requests after transient errors.Routing topology remains unchanged; only eligible routing-table entries are altered, and recovery still requires Lustre-side action after gateway failure.Source: §29.5 · pp. 448–449

29.6. Recovery APIs

  • Recovery APIs detect timed-out locks, trigger user-mode recovery, reconnect to failover targets, resend outstanding requests, and replay operations not durably committed.Recovery switches connections, performs required replay or resend processing, then returns the connection to LUSTRE_CONN_FULL for normal requests.Source: §29.6.1.4 · p. 450, §29.6.2.4 · p. 450, §29.6.3.4 · p. 451

29.6.8.3. Return Values.

  • Server connection handling reconnects existing client exports, starts recovery timing when appropriate, and rejects new clients until recovery completes.target_handle_connect either reconnects an existing export or creates a new one only after the device has completed recovery.Source: §29.6.8.4 · p. 453, §29.6.9.3 · p. 453

29.6.10.3. Return Values.

  • Lustre starts recovery for reconnecting clients, queues their requests, and evicts clients that fail to reconnect before the recovery timer expires.When recovery aborts, delayed replies are dropped and affected clients receive ENOTCONN; disconnect handling clears the client connection when needed.Source: §29.6.10.4 · p. 453, §29.6.9.4 · p. 453, §29.6.12.4 · p. 454

30.2. The basic offline lfsck implementation

  • Lustre’s offline lfsck uses MDS, directory-membership, and OST databases to identify orphaned objects and missing objects, then removes or reconstructs them through lost+found entries.The approach addresses damage beyond log recovery and can run as an occasional sanity check, while pre-created MDS objects require additional synchronization.Source: §30.2 · pp. 456–457, §30.2.1 · p. 457

30.2.1. Dealing with pre-created objects.

  • For pre-created objects, lfsck compares the last object IDs on MDS and OSTs, moves non-empty out-of-range objects to lost+found, removes empty ones, and deletes file-deletion llogs.If the MDS last ID exceeds the OST last ID, missing objects can be recreated at write time and reported.Source: §30.2.1 · p. 457, §30.2.2 · p. 457

30.2.4. The case of RAID1 OST’s.

  • RAID1 OST consistency checks can synchronize from the primary or from the object with the newer mtime, preferably offline because degraded-mode I/O continues using one OST.Group locks assist synchronization and ordinary I/O but do not fully solve the problem until the failed OST returns online.Source: §30.2.4 · pp. 457–458

30.3. Changes for an Online File System Checker

  • The online checker avoids LVM snapshots by scanning OST objects up to the last object number from the previous boot, thereby excluding race-prone newly created objects.This refinement avoids races and locks but depends on the current Linux OST implementation; future filesystem snapshots could provide an equivalent solution.Source: §30.3 · pp. 458–459

31.1. Architecture

  • Lustre configuration uses lmc to create DTD-conformant XML, lconf to translate it into lctl commands and load modules, and lctl to configure devices and mount the filesystem.The high-level design covers XML and DTD structure plus the main loops of lmc, lconf, and lctl.Source: §31.1 · p. 460, §31.2 · p. 461

31.2. High-level design

  • The high-level design aligns DTD entries, lmc-generated XML items, and lconf-generated commands, while DTD validation checks configuration syntax to reduce misconfiguration errors.The XML hierarchy permits repeated entities such as OSTs and OSDs and uses compound top-level structures.Source: §31.2 · pp. 462, 464

gen.XXX(...)

  • lmc is a Python utility that creates or merges XML descriptors through top-level add_XXX() functions and GenConfig methods for nodes, networks, storage devices, filesystems, and mount points.Its command flow dispatches through do_command(), add(), and element-specific generators; analogous elements follow the same flow.Source: §31.2 · p. 464, gen.XXX(...) · p. 465, §31.3 · p. 473

MDSDEV

  • lconf reads configuration from XML or LDAP, selects information for a specified or default node, invokes lctl for device setup, and automatically loads required Lustre modules.Its main loop obtains the database, sets the host, performs host-specific configuration, loads modules, creates system devices, writes configuration, and cleans up.Source: gen.XXX(...) · p. 466, MDSDEV · p. 468

31.3. low-level design

  • Lustre configuration uses a hierarchical XML structure validated by a DTD, with named UUID-bearing objects and UUID references connecting devices, networks, profiles, targets, and filesystems.The DTD defines compound main elements, basic content elements, and reference tags; XML is created by lmc and consumed by lconf.Source: §31.3 · pp. 469–471

3.3.1 GenConfig

  • GenConfig is lmc’s central class for adding XML configuration, providing methods for references, services, text, elements, networks, routing tables, and other Lustre components.The class is used by top-level functions that dispatch configuration-generation operations.Source: §3.3.1 · p. 473, §31.3 · p. 473

3.3.2 top-level functions

  • lmc’s top-level functions cover node, network, route, metadata, object storage, LOV, filesystem, and mount-point creation, supported by naming, UUID, lookup, reference, and filesystem utilities.The listed utilities include new_name(), new_uuid(), name2uuid(), findByName(), lookup(), lov_add_obd(), and ref_exists().Source: §3.3.2 · pp. 473–474, §3.3.3 · p. 474

3.4.1 the basic class module

  • lconf organizes device configuration through a parent module class and twelve child classes, supplemented by system-level, query, routing, and miscellaneous functions.The module class provides common initialization, information, cleanup, module-loading, and safety functions.Source: §3.4.1 · pp. 474–475, §3.3.3 · p. 474

3.4.3 system-level functions

  • The system-level support includes command execution and daemon helpers such as runcmd(), run(), and run_daemon().These functions are listed as part of lconf’s system-level functionality.Source: §3.4.3 · p. 475

31.4. Lustre configuration scenarios

  • Lustre’s lmc configuration assigns cluster_id, hostaddr, port, gateway_cluster_id, and target_cluster_id, generating XML network and routing entries for routed clusters.cluster_id identifies network membership; route tables select gateways and target clusters, while lo-hi specifies a node range.Source: §31.4 · pp. 477, 480, 481

32.1.1. NAME.

  • The lstripe utility creates a file with specified stripe size, starting OST, and stripe count; stripe count determines the number of OSTs on which the file is striped.The OST numbering begins at 0: 0 refers to OST1 and 1 to OST2. Examples create files with stripes only on OST1 or only on OST2.Source: §32.1.1 · p. 484, §32.1.3 · p. 484, §32.1.4 · p. 484

32.1.4. EXAMPLES.

  • For a 256k file12, striping across OST1 and OST2 yields two stripes with the first on OST1; file21 likewise has two stripes with the first on OST2.Source: §32.1.4 · p. 484

32.2.1. NAME.

  • llanalyze reads Lustre debug logs from standard input and can filter by process, locking, network, entry or exit markers, and RPC traces.Its output can be formatted with less -r, and an example extracts PID-specific trace markers into a separate log.Source: §32.2.1 · p. 485, §32.2.3 · p. 485, §32.2.4 · p. 485

Lustre Specific API calls

33.1.1. LL_IOC_GETFLAGS.

  • LL_IOC_GETFLAGS returns the flags set for a file, including LL_FILE_IGNORE_LOCK, which suppresses extent locking for POSIX semantics.The request uses a file descriptor and an integer pointer in the process address space.Source: §33.1.1.1 · p. 486, §33.1.1.2 · p. 486

33.1.2. LL_IOC_SETFLAGS.

  • LL_IOC_SETFLAGS updates file flags and supports LL_FILE_IGNORE_LOCK to disable extent locking for POSIX semantics.The interface is described as applying the supplied flag state to the file.Source: §33.1.2.1 · p. 486

33.1.2.2. Parameters.

  • Lustre exposes file flags for controlling behaviors such as whether file extents receive POSIX locks.The LL_FILE_IGNORE_LOCK flag disables extent locking for POSIX semantics.Source: §33.1.3.1 · p. 487

33.1.4. LL_IOC_GROUPLOCK

  • LL_IOC_GROUPLOCK obtains a group lock for cooperating processes, with one active group ID and additional group locks queued.Applications establish the group ID passed through the ioctl interface.Source: §33.1.4.1 · p. 487, §33.1.4.2 · p. 487

33.2. Striping Information

  • Lustre Lite stores striping metadata in the system.lustre_mds_objid extended attribute, enabling identification of the OSTs containing an inode’s stripes.Changing this attribute can make an object appear to lack a stripe.Source: §33.2 · p. 487

33.3.3. EXAMPLES.

  • The lustre-conf examples generate XML configurations for single systems, striped LOV deployments, multiple clients, routed MCR clusters, OST testing, and MDS failover.The examples configure nodes, networks, MDS and OST services, clients, routes, striping, and failover-related mounts.Source: §33.3.2 · p. 488, §33.3.3 · pp. 488, 489, 492

34.1. Introduction

  • The configuration-management framework represents Lustre systems in XML, can obtain configurations from files or LDAP, and supports SNMP management schemas.The chapter presents the DTD, LDAP schema, and SNMP schema as configuration-management references.Source: §34.1 · p. 497

34.2. Lustre DTD

  • The Lustre DTD defines top-level elements and references for nodes, networks, routes, profiles, devices, MDS and OST services, clients, LOV striping, and mountpoints.LOV device declarations require stripe size, stripe count, and pattern, with an optional stripe offset.Source: §34.2 · p. 497, <!-- Lustre Management DTD --> · pp. 497–498

34.3. LDAP Schema

  • LDAP stores Lustre configuration objects for nodes, profiles, networks, clients, MDSs, OSTs, devices, LOVs, mounts, MDCs, OSCs, and OBDs.MDS and OST failover systems query and update LDAP, including the active server field after service reactivation.Source: §34.3 · p. 500

34.3.1. LDAP Access API’s.

  • The client initiates recovery after timeout by querying LDAP for the currently active MDS or OST through lustre-query.lustre-modify updates LDAP attributes, while lustre-query supports searches and outputs identifiers, descriptions, addresses, active UUIDs, and failover UUIDs.Source: §34.3.1 · pp. 504–505

34.4. SNMP Schema

  • SNMP support combines an SNMP MIB describing Lustre components with an agent that retrieves and reformats matching data from /proc/lustre.The MIB is intended to provide implementation-independent, logical IP network-level information useful to administrators and end users.Source: §34.4 · p. 505

34.4.1. Lustre filesystem MIB.

  • The Lustre filesystem MIB defines tables for OSTs, MDSs, clients, and client-accessed OSTs, including identifiers, names, capacities, relationships, and row status.Client entries also describe LOV usage, stripe depth and factor, mount points, and connected MDSs.Source: §34.4.1 · pp. 505, 506, 507, 509, 510

35.1. Introduction

  • Lustre uses three broadly defined device classes—CLIENT, MDS, and OST—that can be co-located and assembled from multiple module combinations.Clients may use LOV for striping across multiple OSTs, while OSTs require a journalling backend such as extN or reiserFS.Source: §35.1 · p. 513
  • LprocFS exposes each Lustre mount point’s modular device stack and filesystem statistics through /proc/fs/lustre.It reports values including free and total disk space, free and total file counts, and basic block size.Source: §35.1 · p. 513

35.2. LprocFS

  • LprocFS materializes static, dynamic, module-level, OBD-instance, and other variables in a hierarchical /proc/fs/lustre interface.It requires compiled-in proc support, and initializing obdclass creates the hierarchy root before services can be used.Source: §35.2 · pp. 513–514

35.3. Data Structures

  • Each LProcFS variable records a name, read/write callbacks, and context data, with slash-separated names creating nested directories and current variables remaining read-only.Null-terminated variable lists populate newly registered hierarchies, while OBD entries are attached below module and device-specific roots.Source: §35.3 · pp. 515–517

35.2. LprocFS

  • LProcFS provides APIs to create, extend, and remove arbitrarily deep hierarchies under /proc/fs/lustre for modules, mount points, and OBD devices.Generic registration uses lprocfs_register, OBD devices use lprocfs_obd_attach, existing trees can receive dynamic variables through lprocfs_add_vars, and cleanup uses lprocfs_remove or lprocfs_obd_detach.Source: §35.3 · pp. 516–518

Lustre Kernel Modifications

  • Lustre introduces three significant Linux-kernel patches beyond trivial symbol exports: metadata locking, secure pointers, and socket data-sent callbacks.The secure-pointer mechanism validates slab allocation and matches a random number held by the owner before granting access; the socket callback reports when transmitted data can be freed.Source: Lustre Kernel Modifications · p. 519, §36.2 · p. 520, §36.3 · p. 520

36.1. Meta Data Locking

  • Lustre’s metadata-locking changes keep cached dentries valid and select appropriate locks by notifying the filesystem when dentries are accessed, released, and used in pathname resolution.The Linux 2.4 implementation minimized impact on other filesystems, while a simpler Linux 2.5 version embeds intents in nameidata.Source: §36.1 · p. 519

36.2. Secure Pointers

  • Secure pointers address validity and authenticity by validating slab-cache allocation and requiring a matching random number before remote access is granted.The design is not a kernel patch, and its eventual kernel integration was expected to involve possible modifications.Source: §36.2 · pp. 519–520

36.3. Socket Data Sent Callbacks

  • The socket callback patch notifies Lustre when TCP-sent data has left the system, but it was unlikely to enter Linux 2.5 unchanged because asynchronous socket I/O could supersede it.Without such notification, networking is forced to use synchronous writes to socket file descriptors.Source: §36.3 · p. 520

37.1. Evolution of Lustre modules

  • The module-evolution plan classifies anticipated changes as minor, serious additions, or substantial transformations that preserve existing features while displacing most of the implementation.The overview table presents the expected evolution of Lustre modules using these markers.Source: §37.1 · p. 521, Table 1 · p. 522

38.1. Design/performance uncertainties and alternatives

  • Open concerns include whether recovery scales to 1,000-node clusters without complex timeouts, metadata-lock deadlocks, Elan networking stability, Linux 2.5 performance, metadata-writeback flow control, and collaborative-cache failure recovery.A hierarchical tree-like recovery structure could distribute load but adds considerable complexity; metadata writeback needs back-pressure to limit interference with lock revocations.Source: §38.1 · pp. 523–524

Design Alternatives

  • The design chapter presents these decisions as selections among major architectural alternatives, APIs, and existing software, while management scalability to many 1000’s of nodes remained an open question.The alternatives include symmetric versus non-symmetric clustering and different message-passing APIs.Source: §38.1 · p. 524, Design Alternatives · p. 525

39.1. Architectural Choices

  • Lustre’s networking architecture was chosen over VIA, DAFS, Infiniband, and other alternatives because Portals provided multi-network support, remote DMA, request processing, and NAL and forwarding abstractions.VIA lacked sufficiently rich event delivery, DAFS lacked sufficient generality, and Infiniband was rejected as too complex and large.Source: §39.1 · p. 525
  • Lustre rejected fully symmetric clustering because membership, quorum, recovery, and trust models were complex or poorly scalable, while failover pairs and simpler clustering toolkits remained under consideration.The document also questions distributed metadata because large clusters complicate recovery and two-phase-commit updates can be slow or hang indefinitely.Source: §39.1 · p. 526
  • Lustre selected HTree-style hashed directories and an ext3-based layout because block-level subdivision better supports cluster load balancing and distributing block groups across metadata systems.The ext3 layout was preferred to XFS because its primitive structure enables subdivision, while ext3’s code size is 10% of ReiserFS and 3% of XFS with minimal performance differences reported.Source: §39.1 · p. 526, §39.2 · p. 527
  • Instead of adapting DAFS or NFSv4, Lustre implemented a simple new client-server protocol because those approaches handled asynchronous metadata recovery, storage networking, and object storage integration inadequately.The alternatives were considered cumbersome for Lustre’s requirements.Source: §39.1 · p. 527

Portals Diffs

  • Lustre maintained its own Portals version to support efficient kernel-service operation and guarantee that network operations complete in finite time, with documented API and NAL-interface differences.The documented branch was b_cray_portals_merge as of 19th Feb 2004.Source: Portals Diffs · p. 528

40.1. Differences at the API

  • Portals does not guarantee message ordering, but it guarantees a final completion event and finite-time completion; the current implementation supports 0.5MBytes per message across 128 fragments.Asynchronous unlinking can force an operation to complete and notify the application, while callbacks cannot invoke data-movement functions.Source: §40.1 · pp. 529–531

40.2. Differences at The Lib/NAL interface

  • At the lib/NAL interface, NAL send and receive routines report immediate startup failures or return PTL_OK and must finalize initiated operations with a ptl_err_t status in finite time.Kernel NALs also support non-contiguous page buffers through cb_send_pages() and cb_recv_pages().Source: §40.2 · pp. 532–533
  • Lustre Portals passes NALs the original scatter/gather descriptor plus an offset, avoiding dynamically allocated temporary descriptors for message subsections.NALs can request extracted descriptors when needed, while descriptors passed through the interface are read-only.Source: §40.2 · p. 533
  • RDMA-capable NALs can optimize PtlGet() by bypassing the normal REPLY message, directly transferring data into sink buffers for higher bandwidth and lower latency.The sending NAL simulates a reply to mark the sink buffer accessible, and completion is signaled after RDMA finishes or is abandoned safely.Source: §40.2 · pp. 533–534

40.3. Multiple Networks

  • The portals router connects clusters with different fabrics through multi-interface gateways, supporting redundant gateways for load sharing, failover, performance, and resilience.It maintains route, gateway, and NAL tables; gateway selection uses reachable target ranges and gateway state, while failures mark gateways down and trigger userland notification.Source: §40.3 · pp. 534, 536

40.4. Lustre Compatibility with Cray Portals

  • Lustre interoperates with Cray Portals by representing fragmented kernel buffers with physical-address iovecs, coalescing compute-node fragments when necessary, and emulating unsupported arrival-time fields.Cray’s completion-semantics option is used for Lustre-created MDs, while the event-queue callback drives message progress on compute nodes; the physical-address representation requires compatible architectures.Source: §40.4 · p. 537

Related papers

Loading 1903.01955v1…