The Lustre Storage Architecture

Peter Braam

Source-linked AI summary

The Lustre Storage Architecture

Peter Braam

arXiv:1903.01955v1Mar 5, 2019cs.OS

TL;DR

Lustre addresses the design of scalable cluster storage and global namespaces that avoid client configuration difficulties. It outlines a modular, layered architecture and reports robust message-error detection and recovery, while noting that journal recovery remains difficult to stabilize.

Problem
Existing global namespace systems have difficulties, motivating a directory tree valid across workstations without client configuration and after updates.
Method
The paper describes a modular storage framework, layered networking, integrated metadata/object storage, and a global namespace design combining AFS and automounter features.
Results
The qswnal is described as robust at detecting and recovering from errors, with lost messages detected through timeouts.
Takeaways & Limitations
The OSTMDS approach supports clustering commodity Linux boxes using ordinary ext3 filesystems without reformatting them.
Takeaways & Limitations
Journal recovery of OBDFS is delicate and may take a long time to fully stabilize, particularly on low-memory machines.

Abstract

from arXiv · show

This lengthy document often referred to as the "Lustre Book", contains a detailed outline of Lustre file system architecture, as it was created between 2001 and 2005, in accordance with the requirements from various users. Now, in 2019, most features have been implemented, but some only recently, and some along different lines of thought.

09/29/2005, HEAD · Glossary

The glossary defines Lustre-specific architecture, filesystem, clustering, networking, industry, and protocol terminology. It emphasizes metadata and collaborative caching, object-storage components, namespace and coordination concepts, and related systems and standards.

0.1. Lustre Specific Acronyms: Metadata writeback caching stores unflushed mkdir, create, setattr, and other metadata updates before they reach storage or servers.The glossary distinguishes metadata writeback from ordinary file-data caching.
0.1. Lustre Specific Acronyms: Collaborative caching shares read data among client subsets, enabling client-to-client transfers and scalability for mostly read-only workloads.COBD denotes Collaborative Cache.
0.1. Lustre Specific Acronyms: Lustre terminology separates metadata and object storage roles across the MDS, MDC, OST, and OSC components.MDS provides metadata services, while OST provides object storage through an OBD server instance.
0.2. Filesystem, Clustering, and Networking Terminology: Networking terminology includes Portals, Portal NALs, NIO, RDB, QSW, UDP, and VPN for message passing, request processing, and transport.OS-bypass I/O can avoid kernel-to-user buffer copies and process sleep while detecting responses.
0.2. Filesystem, Clustering, and Networking Terminology: Cluster coordination terms cover barriers, membership, quorum, lock revocation, and client-server request/response behavior.Quorum determines whether membership is large enough to represent an authoritative group, while lock revoke asks a client to release a held lock.
0.2. Filesystem, Clustering, and Networking Terminology: A fileset is an administratively exported file tree with stronger transactional guarantees inside its boundaries than across them.The global namespace unifies exported filesystems from multiple file servers into one directory tree.
0.3. Industry Products and Protocol Acronyms: The industry glossary situates Lustre among APIs, filesystems, storage protocols, clustering tools, authentication mechanisms, and standards.Examples include ADIO, Coda, DAFS, InterMezzo, NFSv4, NASD, RDMA, SNIA, and OSD-related work.
0.3. Industry Products and Protocol Acronyms: Related entries also document scalable directory designs, web-serving optimizations, Linux VFS locking, and high-availability failover systems.Hashed directories support fast name lookup, while TUX, Kimberlite, and Service Guard represent performance or availability references.

Architecture · Introduction

Lustre is a modular, scalable cluster filesystem whose clients, object storage targets, and metadata servers provide distinct but composable services. Its architecture supports flexible storage management, transactional metadata handling, security, failover, SAN integration, and deployment across clusters of different sizes.

Introduction: Lustre’s framework combines storage management, networking, locking, and mass-storage targets to support scalable cluster filesystems.The architecture is intended for small to very large clusters.
2.1. Lustre Components: The architecture comprises Clients, Object Storage Targets, and Meta-Data Server systems, each internally organized into modular components.Request processing and message-passing layers are shared across systems, while some modules are system-specific.
2.2. Object Storage Targets: Object Storage Targets provide file I/O, perform distributed block allocation, and enforce security for client access to data objects.This design distributes allocation metadata across OSTs and combines request processing with remote DMA.
2.2. Object Storage Targets: Lustre’s networking uses the Portals API and Network Abstraction Layers to support multiple transports, message events, and optional RDMA.The abstraction allows request processing to interoperate with different underlying network transports.
2.3. Meta-data Service: Metadata servers transactionally update backend metadata over a network interface, with failover services in Lustre Lite and planned load-balanced processing in full Lustre.The metadata design also envisions stackable services for replicated servers and local persistent caches.
2.4. Client Filesystem: Lustre exposes scalable allocation, security, and metadata control as independent abstractions rather than embedding them in one monolithic filesystem.The client filesystem also supports UNIX file-update semantics, multiple lock granularities, and transparent service failover.
2.5. Storage Management: Stacked object modules provide storage management functions including data migration, snapshots, enhanced security, and active disk components for data mining.A general framework manages and dynamically changes the driver stacks.
2.6. Lustre and SAN’s: SANOSC, SANOST, and OSTMDS modules extend Lustre to SANs and shared Linux filesystems, including commodity Linux clusters without filesystem reformatting.Combining SANOST and OSTMDS yields a commercial-cluster-oriented filesystem, though with less scalability than Lustre and Lustre Lite.

Global Namespaces for Filesystems - with Lee Ward

The paper presents a global namespace that unifies directory trees from multiple file servers into a self-describing tree valid across workstations and configuration updates. Its design combines AFS advantages with automounter features and provides an implementation roadmap for Linux environments.

Motivation and architecture: A global namespace unifies directory trees from multiple file servers while minimizing client configuration and remaining valid across installation-wide configuration updates.The namespace provides a single directory tree to all clients and is self-describing, requiring little or no client configuration.
Related systems: AFS demonstrated global namespaces by unifying tens of thousands of filesets distributed across multiple administrative cells.Users typically had personal home-directory filesets and project-associated filesets.
Related systems: Automounters create global namespaces by detecting mount-point traversal, consulting mount maps, and executing Unix mounts to graft heterogeneous filesystems into the directory tree.Mount maps may be stored in configuration files or directory services such as NIS or LDAP; automounters can combine NFS and local filesystem mounts.
Contribution and implementation: The proposed design combines AFS advantages with newer automounter features and outlines Linux integration using local filesystems, NFS, InterMezzo, and Lustre.The paper describes implementation details as somewhat involved and different from existing implementations.

3.2. Fundamental Definitions and Requirements

The architecture defines requirements for flexible global namespaces that integrate Unix filesystems and filesets while supporting caching, controlled activation, authorization, and timely updates. It uses directory-based mount-objects and namespace bindings to avoid lookup-triggered mount storms and support multiple mount locations.

Fundamental requirements: Global namespaces should include Unix filesystem and fileset mount-points, support multiple mount locations, and use both network-fetched and locally stored configuration data.Mount-points should be cached, releasable after idleness, invalidated by timeout or callback, and protected against retry storms.
Autofs mechanisms: Recent autofs versions list indirect-map entries normally and avoid mounting filesystems merely when mount-points are looked up, reducing mount storms.Earlier autofs versions triggered mounts during lookup, including during commands such as ls -l under an autofs-mounted directory.
Mount-object representation: The system represents mount-objects as directories rather than symbolic links, preserving directory link counts and avoiding object-type changes when they become mounted.This breaks with the AFS and Microsoft DFS tradition of representing mount-objects with symbolic links.
Filesystem integration: Mount-objects can reside in any filesystem, belong to the underlying filesystem rather than autofs, and require no map files to identify them.The architecture can graft a local /tmp filesystem into an NFS tree and mount a global /tmp/global directory underneath.
Multiple mount locations: Namespace bindings enable filesets and filesystems to be mounted at multiple points, while InterMezzo provides additional flexibility by exporting multiple namespaces.The text identifies bindings as the key concept for addressing file-server and persistent-cache issues.

Networking for Lustre

Lustre’s networking design presents a full stack that supports heterogeneous networks through a uniform request-processing API. It combines device drivers, Portals message passing, Niobuf data movement, and request processing with multi-interface connectivity and gateway failover mechanisms.

Design goals: The networking infrastructure targets different networks while providing a uniform high-level request-processing API to Lustre’s core filesystem modules.Its requirements include portability, scalability to clusters of 10,000’s of nodes, network portability, fast-network performance, OS bypass, recovery, and security.
Network stack: The full networking stack consists of device drivers, a Portals message-passing layer, a Niobuf data-movement layer, and a request-processing layer.The design places request-processing interfaces at the top and device drivers at the bottom, with the intermediate layers connecting them.
Portals message passing: Portals supplies asynchronous put and get operations, packet matching, event queues, OS-bypass mechanisms, and a network abstraction layer for multiple interfaces.Its mechanisms support memory management for 0-copy I/O, NIC-resident drivers, and crossing protection domains.
Network interoperability and routing: A node can use multiple interfaces and accept or initiate sessions over each, allowing systems attached to different network types to share Lustre file systems.The Portals router also implements load balancing over equivalent routes and enables or disables particular gateways.
Recovery: When messages are dropped, client or server timeouts trigger Lustre’s normal recovery actions, which can proceed immediately after a failed gateway is disabled if an alternative route exists.The networking behavior treats occasional in-transit message loss as recoverable through timeout detection and alternative routing.

Object Storage

Object storage is organized as layered software modules spanning storage applications, logical and direct object devices, and intermediary drivers. The architecture supports networked and clustered storage, richer object namespaces, snapshots, journal recovery, and caching.

Layered Object-Storage Stack: Storage Object Applications interface with logical or direct object devices, while intermediary drivers can aggregate multiple devices or export snapshots and object versions.Logical devices may expose the same interface as object-based storage devices, and RAID is an example of aggregation across direct drivers.
Network and Cluster Interfaces: Network OBD clients package commands for transport to targets, which translate received packets into methods on physical or logical object devices.Candidate protocols include IP, VIA, and SCSI, carried over transport layers such as Ethernet.
Network and Cluster Interfaces: Targets and network OBD drivers enable clustered object-based storage by implementing the OBD protocol across an interface.Clustered layouts use clients and targets to connect object storage devices.
Object Namespaces: The proposed namespace divides objects into groups, enabling applications such as filesystem snapshots to assign each snapshot its own group identifier.Group identifiers could be 32-bit or 64-bit integers, and the proposal considers explicit namespace retrieval and control mechanisms.
Object Namespaces: The architecture favors allowing applications to request given object identifiers, despite added implementation complexity from non-flat object-group namespaces.A drive may reject a requested identifier when that object already exists.
Advanced Object-Storage Services: The design includes journal recovery, snapshot logical drivers, and caching object drivers, while noting recovery’s stabilization challenges under memory pressure.Caching nodes can run on clients or dedicated servers and service reads, reducing requests handled by target OSTs.

The Lustre Metadata Service

Lustre’s metadata service manages the filesystem namespace while ensuring that all cluster clients see one coherent directory tree. It supports transparent recovery from single MDS failures and clustered metadata servers that distribute inode and directory metadata for scalability.

Metadata API: The metadata API adds, removes, renames, and accesses namespace elements, including files, directories, and symlinks.It can be exported by a Lustre device and run on the same or a separate device as the storage device.
Namespace Coherency: A single namespace view requires every client in the cluster to see exactly the same filesystem directory tree.Clients maintain coherent lookup results through caching combined with lock management.
Namespace Coherency: Lustre exposes never-reused file identifiers, ensuring that each inode is uniquely identified by its fid and simplifying namespace coherency.Namespace entries are represented by parent fid, name, and target inode fid triples.
Recovery and Availability: Lustre provides high availability by masking server and client failures, focusing transparent recovery on single-point MDS failures.The design assumes simultaneous failures of multiple nodes are rare, enabling performance increases.
Clustered Metadata Service: Clustered metadata servers manage inode groups, with directory formats and an LMV driver supporting metadata distribution across servers.Each inode group is a formatted filesystem with its own recovery journal and can fail over to another node; growing directories can be split across MDS servers.
Clustered Metadata Service: The worst case requires 3 RPC’s for directory and inode resolution, while the common case requires a single RPC because the directory inode is normally cached.The three-step path retrieves the directory attributes, locates the directory entry bucket, and finds inode attributes in the associated inode group.

A Distributed Lock Manager for Lustre

Lustre’s distributed lock manager adapts classical cluster locking with resource-tree organization, distributed resource mastering, and policy-based richer lock features. It provides mechanisms for locating masters, managing hierarchical lock state, remastering after cluster transitions, and exposing lock services through multiple forms and a universal API.

Resource and Lock Organization: A distributed resource directory hashes resource names to locate masters, assigns an unmastered resource to the caller, and releases it when its final lock is gone.The lock directory uses the Lock Directory Weight Vector, which can list stronger systems multiple times.
Distributed Mastering: Each resource master maintains granted, converting, and waiting request queues plus master lock data, while nonmasters retain duplicate state to support remastering after departures.The master handles requests from both itself and other systems in the same resource tree.
Resource and Lock Organization: Lustre organizes resources as trees, requiring locks on all ancestors before descendants and preventing ancestor dequeue while descendants remain.This supports hierarchical locking such as an inode master lock with child locks for file extents.
Extended Lock Features: Lustre extends standard lock services with policy-driven lock types and intent locks that can execute requested intents instead of granting locks.The architecture also treats extent locking relationships through lock types and policy functions.
Interfaces and Extensions: The planned architecture includes lock services in three forms, activity- and system-power-based master migration, and a universal LVFS API for access.These are described as supported forms or future improvements to the lock manager and access layer.

Lustre Logging API

The Lustre Logging API supports transactional logging for persistent updates spanning multiple systems, especially distributed commit, cancellation, and recovery. It uses remotely callable, stateless log operations with identifiable log objects, structured records, and generation- and cookie-based coordination between originators and replicators.

API requirements: Log access is designed to work through methods without exposing local versus remote execution, using stateless remote calls and authorization or authentication.Logs only grow, may be removed, and remote callers cannot assume an open log remains available.
Log and record model: Log objects are identified by names or llog identifiers, while records contain a 16-byte header, an opaque 32-bit-aligned body, and a 16-byte tail.A filesystem directory can translate names to identifiers, and records can be accessed by iteration or an llog_cookie containing the log identifier and record offset.
Log and record model: Large logs can use hierarchical catalogs, with catalog entries storing the identifiers of referenced log files.The catalog API builds on the lower-level Lustre log API.
Distributed commit: The API records an originator’s transaction in a log and cancels that record when the related replicator transaction commits.This ensures replicators complete their updates if the originator commits, even when systems crash or roll back.
Distributed commit: During normal operation, lop_add creates a log_cookie, the replicator commits its related transaction, and lop_cancel queues the cookie for cancellation at the originator.The replicator sends accumulated cancellation cookies when it has a full page.

The Lustre Filesystem

Lustre is a cluster filesystem built around client, metadata-server, and object-storage-target subsystems connected by dedicated protocols. Its architecture provides cluster-wide Unix semantics through distributed locking, adaptive metadata handling, and object-based I/O, while exposing bottlenecks for extremely heavy metadata traffic.

Architecture: Lustre’s core comprises the Client FileSystem, Meta-Data Servers, and Object Storage Targets, which may run separately or share systems.Each subsystem communicates through a corresponding protocol, with additional protocols for auxiliary subsystems.
Subsystem roles: The client filesystem provides cluster-wide Unix file sharing, while MDS handles inodes and directories and OST protocols handle I/O, striping, allocation, and security.MDS transforms requests into journaled, batched persistent metadata updates and scales across many client nodes.
Distributed locking: Lustre offers Unix semantics across multiple nodes through a shared lock service spanning client–MDS, client–OST, and inter-MDS protocols.OST systems serve file locks, including Unix, lock, flock, and specialized collective-I/O locks; writes revoke read locks on cached pages.
Metadata protocol: Lustre adapts metadata caching to concurrency: MDS grants client writeback privileges at low concurrency but performs operations directly when concurrency is high.Intent-aware processing can avoid multiple lookup locks, bundle RPCs with lookup processing, and preserve coordinated VFS semantic checks.
Limitations and parallel I/O: A single fileset’s metadata may bottleneck on one OST under extremely heavy metadata traffic, and filesystem transactions do not cross fileset boundaries.The architecture also supports efficient MPI-IO file views and collective operations through the object-storage framework.

File striping configuration

Lustre stripes files across multiple OSTs to improve parallel access, but effective performance depends on choosing an appropriate striping pattern. Striping may be configured system-wide or per file before objects are allocated, while striping every file across every OST can create contention and metadata overhead.

Striping rationale: Striping files across OSTs lets different clients access separate parts of one file concurrently, improving filesystem parallelism.The document emphasizes that a good striping pattern is needed to obtain maximum benefit.
Default configuration: The system-wide default defines stripe size and stripe count, with count 0 spanning all available OSTs and count 1 using a single OST.Stripe size is the number of bytes stored on each OST before I/O moves to the next OST.
Per-file configuration: Applications can set a distinct striping configuration at file creation through ioctl(2) or lstripe, including stripe size, starting OST, and stripe count.The lstripe utility selects subsequent OSTs consecutively, wrapping to OST #0 after the last OST.
Per-file configuration: A file’s striping pattern cannot be changed after objects are allocated, which normally occurs when the file is first opened.With O_LOV_DELAY_CREATE, the application must call LL_IOC_LOV_SETSTRIPE before using the returned file descriptor.
Performance trade-offs: Striping all files across all OSTs is counterproductive because it increases disk contention, object-creation work, and RPCs for file-size calculations and locking.The document instead stresses keeping OSTs occupied without indiscriminately spreading every file across them.

Recovery

Lustre recovery addresses client, MDS, OST, and transient network failures by restoring a consistent, performant file system. It uses timeout-triggered recovery, coordinated persistent-state management, request replay, and failover mechanisms.

Recovery scope: Lustre must recover from client, MDS, OST, and transient network failures while restoring a consistent and performant file system.Clients retain only memory-based state, whereas MDS and OST systems must coordinate recovery of persistent state.
Client recovery: During recovery, clients may re-establish locks, reopen files, and replay updates, but clients missing the recovery window are evicted.Afterward, the client recovery protocol classifies requests for delayed sending, completion confirmation, replay, or reply reconstruction.
Failure detection: Timeout expiration on requests, lock operations, bulk I/O, and failed reconnection replies triggers Lustre recovery.A request timer starts when the request is sent; recovery also follows an -ENOTCONN reply after an initial connection failure.
Recovery infrastructure: Epoch, generation, incarnation, and transaction identifiers determine recovery scope, fencing, request eligibility, and replay ordering.Servers stop accepting new requests during recovery, while communicated generation, incarnation, and epoch values fence hosts that cannot communicate.
Server consistency and failover: MDS/OST coherency is maintained through logical logs of selected operations communicated during recovery, ensuring object and inode references remain consistent.Failover OST systems use redundant shared storage to transparently recover application read/write calls after an OST failure.

Lustre Security

Lustre’s security architecture emphasizes integration with existing identity, group, and access-control mechanisms while supporting multiple authentication, authorization, encryption, and auditing layers. It adapts security enforcement to network trust and cross-realm use through GSS-API policies, local accounts, capabilities, and configurable identity translation.

Identity and administration: Lustre uses standard Unix user APIs and existing databases, supporting LDAP, Active Directory, NIS, and specialized sources through NSS switches.This reduces administrative burden and avoids mandating a particular authorization engine or user and group database.
Threats and trust: The security model addresses unauthorized information release, unauthorized modification, and denial of resource usage, although denial of service is only partly addressed.Servers are largely trusted, while clients may optionally send encrypted file data and remain responsible for protecting credentials.
Security layers: Lustre divides security responsibilities across network trust, authentication, group and user management, authorization, cross-realm usage, file encryption, and auditing.GSS-API interfaces include Kerberos 5, LIPKEY, and OPEN; auditing can cover clients, metadata servers, object storage targets, and credential or encryption-key services.
Network security: GSS-API authenticates users and protects network-message integrity, with security levels determined by backend and policy choices for trusted or insecure networks.Lustre seeks to avoid cryptographic work where network trust permits while supporting different performance-affecting security policies.
Cross-realm access: Cross-realm access uses local accounts and configurable fileset options that can avoid, defer, or synchronously perform remote user-ID translation.A server-resident capability associated with client, filesystem, and principal can forward user IDs, group IDs, and setgroup arrays.
Authorization and encryption: Lustre authorizes access through Linux kernel and filesystem POSIX ACL mechanisms, with a separate encryption-related access list optionally checked after inode authorization.This approach reuses existing ACL structures and preserves POSIX ACL semantics for users and locally represented cross-realm identities.

Configuration Management

Lustre configuration management separates system description from low-level device setup, using XML, lmc, and lconf to configure layered client, OST, and MDS components. LDAP-based management extends this model for large clusters by centralizing configuration, applying profiles, and reducing misconfiguration risk.

Configuration Management: Configuration proceeds in two stages: create a system description, then perform low-level device configuration across client, OST, and MDS layers.The configuration file is processed by lconf, which uses lctl to load modules, start services, and perform device-specific setup.
System Startup: Startup activates Lustre subsystems in ordered levels from network and devices to metadata, storage, striping, and mount points.Because connection and mount failures trigger retries without order sensitivity, cluster startup can proceed in parallel.
XML Configuration: XML configurations generated by lmc and implemented by lconf can reduce misconfiguration errors through syntax validation against Lustre’s DTD.Configuration is parsed and applied progressively by level, from networking and locking through storage targets, metadata, striping, and mount points.
LDAP Configuration: LDAP-based configuration centralizes data for large clusters, assigns default profiles, and helps avoid potentially disastrous storage-system misconfiguration.Small installations can use XML file-based management, while larger systems use LDAP repositories whose schemas match the XML descriptors.
LDAP Configuration: LDAP abstractions make configuring a 10,000 node cluster feasible and allow proxy or remote agents to translate high-level configuration into executable low-level commands.This supports systems without disks and can operate without XML or LDAP at execution time.

Configuration Management

Lustre configuration management layers loadable kernel drivers across clients, MDSs, and OSTs, then configures devices and connections through lctl. Because large clusters make low-level configuration cumbersome, XML descriptions generated by lmc and applied by lconf provide a higher-level alternative.

Driver layering: Loadable modules provide the common Lustre networking stack, class driver, and lock manager on client, MDS, and OST systems.Modules are loaded with Linux modprobe or insmod in a specified order.
Driver layering: OSTs layer ExtN, filter, and object-storage drivers, while MDS metadata functionality is integrated in the mds driver with fsfilt_extN supporting recovery.Clients use filesystem, MDS and OST client drivers, plus lov for striping over OSCs.
Storage targets: Each MDS and OST exports a target from a block device formatted as ext3, with IDE/SCSI partitions, logical volumes, RAID devices, loop devices, and ramdisks supported.An OST device requires no further configuration at this phase.
Network configuration: Network connections are established per remote system in an lctl session, while socknal-based OST and MDS systems require an acceptor program for TCP connections.The network commands support connecting to remote NIDs, UUID association, disconnection, and routing-table management.
Low-level configuration: Each Lustre device is configured in lctl by allocating a device number, attaching a named UUID-bearing driver device, and issuing a type-specific setup command.The attach command allows device names and UUIDs of up to 36 characters; setup arguments vary by device type.
XML configuration: XML configuration addresses the prohibitive scale of low-level setup; lmc generates configuration files and lconf implements them on systems.A simple DTD defines valid entries corresponding precisely to the low-level configuration.

Redundant Object Storage Targets … Portals Diffs

The document proposes a modular Lustre architecture spanning redundant storage, collaborative caching, metadata writeback, quality-of-service controls, recovery mechanisms, and extensible networking and APIs. It also describes implementation refinements for scalability, consistency, repair, configuration, observability, and portability across clustered systems.

Redundant Object Storage Targets: Mirrored OSTs replicate objects across two targets, mask single-target write failures, balance reads, and require logging and recovery to re-synchronize replicas.Writes may be declared successful after one OST succeeds, while failed pairs can be disabled to reduce double-load risk and accelerate re-synchronization.
Caching OBD; Lustre Metadata Write Back Caching: Collaborative caching distributes read requests across caching nodes using the COBD driver, while local metadata caching and writeback layers handle metadata misses and directory content management.Caching nodes may run on clients or dedicated servers, and the target OST can refer reads to suitable caching OSTs.
Caching and XDSM; Metadata API; File System Design: Lustre adds XDSM support, clustered metadata servers, enhanced filesystem metadata organization, and transaction replay mechanisms to improve metadata performance and consistency.Clustered metadata uses inode groups and enhanced fids to determine which metadata server services a request.
Quality Of Service - Implementation, APIs: Quality-of-service mechanisms aim to balance OST utilization, avoid filling targets, and provide application-specific guarantees, with status information returned to clients for scheduling decisions.The stated initial implementation focuses on uniform OST utilization and intelligent avoidance of targets that are filling up.
Further considerations; LOV Module; Lock Management API and Internals: Performance work reduces I/O overhead through batched reads, range block allocation, sparse-file optimizations, writeback caching, centralized file locking, and intent-based file-size updates.The design also proposes lazy allocation of objects for large files and configurable striping and allocation policies.
Networking API’s; Ptlrpc_bulk_put; Allocate iov; Ptlrpc_bulk_get; Fill MD with PtlGet; Portals Diffs: Portals and bulk-transfer changes add vectored I/O support, simplify sink-side error handling with PtlGet, guarantee completion events, and enable RDMA-specific optimizations.PTL_MD_IOV supports IOVs of up to 16 elements, while the router provides forwarding across fabrics through redundant gateways.
Recovery API’s; Lustre File System Repair; System Evolution: Recovery and repair infrastructure addresses persistent inconsistencies, orphaned objects, crash-induced holes, and transaction logging through replayable state and small demand-created log objects.The documented cases include MDS, OST, and client crashes during unlink and create operations.
Lustre Configuration; ECHO_CLIENT; MDSDEV; Manuals; Manpages; Lustre Specific API calls; Lustre procfs interface; Lustre Kernel Modifications; System Evolution; Design Alternatives: Lustre exposes modular stack composition and filesystem statistics through LprocFS, adds kernel and locking modifications, and organizes management, configuration, object-storage, metadata, and library interfaces.The architecture is described as highly configurable and modular, with protocol and gateway support spanning multiple networks and clusters.