Source-linked AI summary
Rucio - Scientific data management
Martin Barisits, Thomas Beermann, Frank Berghaus, Brian Bockelman, Joaquin Bogado, David Cameron, Dimitrios Christidis, Diego Ciangottini, Gancho Dimitrov, Markus Elsing, Vincent Garonne, Alessandro di Girolamo, Luc Goossens, Wen Guan, Jaroslav Guenther, Tomas Javurek, Dietmar Kuhn, Mario Lassnig, Fernando Lopez, Nicolo Magini, Angelos Molfetas, Armin Nairz, Farid Ould-Saada, Stefan Prenner, Cedric Serfon, Graeme Stewart, Eric Vaandering, Petya Vasileva, Ralph Vigne, Tobias Wegner
TL;DR
Scientific collaborations need to organize and access growing datasets distributed across heterogeneous, geographically separated infrastructure. Rucio addresses this through a policy-driven, modular data-management system whose concepts and architecture support ATLAS-scale operations. In production, it handled hundreds of petabytes, high transfer and interaction rates, and substantial database load, while volatile storage remains dependent on timely namespace updates.
Problem
Scientific data management must accommodate growing volumes, complex workflows, and heterogeneous storage and computing resources distributed across geographical and administrative boundaries.
Method
Rucio combines hierarchical data concepts, policy-driven management, modular architecture, and orchestration of storage transfers and workflow interactions.
Results
Rucio supported ATLAS production at nearly 450 Petabytes, with 25 million containers, 13 million datasets, and 960 million unique files by the end of 2018.
Takeaways & Limitations
Policy-driven management and a modular, horizontally scalable architecture enabled scientific collaborations to use distributed heterogeneous storage resources at scale.
Takeaways & Limitations
Volatile storage is not presumed to guarantee availability, so caching services must provide timely location updates or replicas may be flagged and removed.
Abstract
from arXiv · showhide
Rucio is an open-source software framework that provides scientific collaborations with the functionality to organize, manage, and access their data at scale. The data can be distributed across heterogeneous data centers at widely distributed locations. Rucio was originally developed to meet the requirements of the high-energy physics experiment ATLAS, and now is continuously extended to support the LHC experiments and other diverse scientific communities. In this article, we detail the fundamental concepts of Rucio, describe the architecture along with implementation details, and give operational experience from production usage.
1 Introduction
Rucio addresses the complexity of managing scientific data across heterogeneous, geographically distributed infrastructure by providing an integrated system for organization, management, access, and transfer. Developed for ATLAS, it coordinates storage, networks, workflows, and transfer services across the experiment’s distributed computing environment.
- 1 Introduction: Scientific data management must handle growing data volumes, complex workflows, heterogeneous resources, and geographically distributed sites spanning administrative domains.Data production, storage, analysis, and archiving may occur at different locations.
- 1 Introduction: Rucio provides a comprehensive solution that integrates data organization, management, access, and existing data-handling tools rather than focusing on a single task.Its scope includes activities such as transfers, synchronization, monitoring, and usage reporting.
- 1.2 ATLAS Distributed Computing: ATLAS uses Rucio as the principal distributed data-management system across more than 130 computing centers, integrated with PanDA and ProdSys.PanDA requests input availability, while Rucio registers newly created files and makes data available at destinations.
- 1.3 Storage, network, and transfer providers: Rucio operates over diverse storage systems and protocols connected by multiple research networks, with the WLCG providing 3 Tbps of total network capacity.The infrastructure includes systems such as EOS, dCache, XrootD, and S3-accessible storage, plus fallback routing over the commodity internet.
- 2.2 Namespace: Rucio’s namespace organizes files into datasets and containers, allowing collections to span multiple storage locations and files to belong to multiple datasets.Datasets group files for bulk operations, while containers group datasets into larger-scale organizational structures.
- 1.3 Storage, network, and transfer providers: Rucio uses historical throughput, packet-loss, and latency metrics to select transfer paths, while FTS performs third-party copies and Rucio groups, monitors, retries, and reports transfers.Multiple FTS servers can be orchestrated for improved parallelism and reliability.
2 Concepts
Rucio organizes distributed scientific data through hierarchical identifiers, account-based access, storage abstractions, and replication controls. Its namespace preserves identity while collections and replicas represent organization and physical availability.
- Namespace: DIDs comprise files, datasets, and containers, supporting overlapping hierarchies from individual files to large-scale collections.Datasets group files for bulk operations and may span multiple data centers; containers group datasets.
- Namespace: DID names combine a scope and name, with the pair globally unique and scopes partitioning the namespace.Scopes can separate instrumentation, simulation, and user-created data or support fine-grained permissions.
- Namespace: DIDs cannot be reused after use, requiring changed data to receive a new name to protect prior analyses from silent modification.Scientific data need not be immutable, but changed data must be renamed.
- Collections: Collections track open, monotonic, and complete states, governing whether content can change and whether all files have available replicas.Open collections accept additions, monotonic collections prohibit removal while open, and completeness is derived from replica availability.
- Accounts: Accounts authenticate users, groups, or activities and control access to scopes through multiple credential types in a many-to-many identity relationship.By default, accounts can read all scopes but write only to their own; privileged accounts can write across scopes.
- Storage: RSEs abstract globally addressable storage locations, while replication rules and volatile-storage updates determine whether replicas remain reliably represented.Volatile RSEs can change outside Rucio control, so stale location information may cause replicas to be flagged suspicious and removed.
3 Architecture
Rucio uses a distributed four-layer architecture in which clients interact through a server and core, while daemons perform asynchronous workflows over abstracted storage, transfer, and persistence systems.
- Architecture: Rucio is decomposed into clients, server, core, and daemons, with clients exposing interfaces and daemons handling continuous asynchronous workflows.The server provides authentication and APIs, while the core represents Rucio concepts and system state.
- Clients layer: The REST interface is Rucio’s main entry point, with Python clients providing authentication, token handling, wrappers, and higher-level helper functions.Command-line tools support both user operations and administrative commands.
- Server, core, and daemons: The server forwards requests to the core, while daemons asynchronously execute complex workloads and exclusively handle interactions with RSEs.This separation supports fast responses, lower server utilization, and background optimization of large workloads.
- Server, core, and daemons: Daemons use heartbeats for workload partitioning and automatic failover, redistributing work when daemons crash or additional instances start.Some daemons are optional, including consistency and external messaging daemons.
- Storage and transfer: RSEs abstract storage protocols, priorities, and attributes, while transfer tools provide generic submission, querying, and cancellation across transfer services.The storage abstraction hides distributed infrastructure complexity from Rucio components.
- Persistence layer: The persistence layer stores data and daemon application state in transactional databases, accessed through SQLAlchemy and supported across multiple RDBMS platforms.Rucio uses Alembic for schema upgrades and optimizes database interaction through indexing, reduced contention, and sharded daemon work.
4 Functionality details
Rucio combines configurable authentication and authorization with rule-driven replication, deletion, consistency checking, recovery, messaging, and monitoring workflows.
- Authentication and authorization: Rucio supports multiple authentication mechanisms and issues short-lived X-Rucio-Auth-Token tokens for subsequent REST operations.Expired tokens cause requests to be denied, while permissions can be customized per instance and operation.
- Replica management: Replica paths can be generated deterministically or non-deterministically using pluggable algorithms configured per RSE.The hash deterministic algorithm distributes files evenly across directories.
- Transfers: Replication rules create transfer requests that daemons submit, monitor, retry, and repair through an abstract transfer tool.Failed requests can mark rules STUCK, after which rule-repairer selects alternative destinations or retries the same destination.
- Data deletion: Expired or unreferenced replicas become eligible for deletion, and a deletion daemon removes them using greedy or non-greedy modes.Recent client downloads can keep expired replicas at an RSE during a configurable period.
- Consistency and recovery: Consistency tools identify lost files absent from storage and dark files present on storage but missing from the catalog.Removing dark files is important because accounting and quota depend on agreement between catalog and storage.
- Consistency and recovery: Rucio recovers bad replicas by injecting transfer requests from another copy after failures or checksum mismatches identify corruption or loss.Bad replicas may be marked by privileged accounts or automatically by Rucio.
- Messaging: Message queues carry typed events with JSON payloads to external services, supporting asynchronous workflow updates, monitoring, and analytics.Messages can describe operations such as transfers and deletions and be filtered by event type.
- Monitoring and analytics: Internal monitoring sends daemon metrics through statsd to Graphite and Grafana, while logs flow through Redis, Logstash, Elasticsearch, and HDFS.The monitoring architecture supports visualization, detailed searches, and long-term backup.
5 Operational experience
Rucio’s production deployment combines open-source development practices, horizontally scalable services, and monitoring across a large ATLAS data infrastructure. Operational measurements show sustained scale, responsive catalog access, and substantial redundancy for resilience.
- Development workflow: Rucio is distributed under Apache V2 and uses GitHub, Travis, time-based releases, automated testing, and human review.Pull requests are tested across Oracle, PostgreSQL, MySQL, and SQLite environments, with over 400 unit tests.
- Deployment and scalability: Multiple instances of services and daemons provide robustness and horizontal scalability until underlying database I/O becomes the limiting factor.The architecture’s scalability is governed by catalog entries and database throughput rather than the total volume of stored data.
- Deployment and scale: 450 Petabytes of ATLAS data, 25 million containers, 13 million datasets, 960 million files, 860 RSEs, and 1.2 billion replicas were managed by the end of 2018.The deployment includes CERN-hosted core infrastructure and additional FTS servers in the United States and United Kingdom.
- System performance: Rucio catalog operations show no discernible performance difference between files stored on disk and files stored on tape.The system also supports substantial database activity, including up to 3000 transactions per second and a 3.7 TB database volume.
- Data movement: At least 30 Petabytes were transferred monthly in 2018, peaking at 55 Petabytes in November, while failures were automatically recovered by Rucio.Monthly transfers averaged 50–70 million files, and deletion reached up to 100 million files or more than 30 Petabytes.
- System performance: 250 Hz average server interaction, spikes to 400–500 Hz, and response times below 50ms produced only 5–7% CPU utilization per REST API node.The 15-node service was almost one order of magnitude over-provisioned to preserve robustness during catastrophic data-center failures.
6 Advanced features
Rucio provides advanced workflows for dynamic data placement, automated rebalancing, and transfer-time analysis. These mechanisms adapt replicas and storage distributions to workload and infrastructure conditions, but their effectiveness depends on experiment-specific dataflows and monitoring.
- 6.1 Dynamic data placement: Dynamic data placement creates replicas using queued jobs, dataset popularity, existing replicas, configurable thresholds, and network metrics.The algorithm scans incoming jobs and records placement decisions for later analysis by operators and infrastructure providers.
- 6.1 Dynamic data placement: 60 percent of newly created replicas were reused within two weeks, and half of accessed datasets were accessed more than once over longer timescales.These measurements indicate that the placement algorithm created replicas that remained popular for several months.
- 6.2 Automated data rebalancing: Rucio automates data rebalancing through background rebalancing, RSE decommissioning, and operator-triggered manual rebalancing.The service addresses workflows that were historically carried out by human operators.
- 6.2 Automated data rebalancing: Background rebalancing equalizes primary-to-secondary replica ratios across RSEs to maintain comparable deletion capacity for new data.The algorithm repeatedly calculates average ratios and moves data from RSEs above the target distribution.
- 6.2 Automated data rebalancing: Decommissioning migrates all data from a selected RSE, while manual rebalancing moves an operator-specified volume during storage shortages or distribution changes.The decommissioning workflow is intended to make RSE removal faster and safer than labor-intensive manual migration.
- 6.3 Transfer-time analysis: Per-transfer trace records support statistical analysis of lifecycle stages, endpoint selection, scheduling, and network or storage optimization.The Transfer Time To Complete extension uses these data to estimate when newly created replication rules will finish.
7 Summary
Rucio addresses the difficulty of managing large data volumes and namespaces across distributed, heterogeneous storage through policy-driven data management and a modular, horizontally scalable architecture. Its operational experience supports current adoption across collaborations and continued development for HL-LHC and non-LHC communities.
- 7 Summary: Rucio provides usable, performant, scalable, and robust management for large data volumes and namespaces across distributed heterogeneous storage.Its design emerged from many years of operational experience addressing these challenges.
- 7 Summary: Policy-driven management lets users express requirements while enabling runtime optimization through self-instrumentation.Users need not determine how their requirements are achieved.
- 7 Summary: The modular, horizontally scalable architecture handles ATLAS load and permits selective component improvement without re-engineering the system core.
- 7 Summary: Rucio is in production with ASGC/AMS and Xenon1T, is being deployed by CMS and DUNE, and is evaluated by collaborations including LIGO and SKA.
- 7 Summary: Future development combines HL-LHC support with features for non-LHC communities, including neutrino and astronomy experiments.HL-LHC work targets smart content delivery, data staging, and caching, while non-LHC requests include mutable metadata, synchronization, WebDAV or HTTP transfer, workflow integration, embargoes, and publication-data links.