PaperlayerView exact revision on arXiv

Source-linked AI summary

Performance Evaluation of RED-ONION: A High-Speed Disk-to-Disk Transfer System

Keichi Takahashi, Hiroaki Kataoka, Takeo Hosomi, Ayahiro Takaki, Yasunori Kakizawa, Shuichi Ihara, Nobuaki Hashizume, Susumu Date

arXiv:2608.29053v1cs.DCcs.NI

TL;DR

Rapidly growing instrument data can be difficult to move from separate research facilities to computing centers using general-purpose interfaces. RED-ONION combines dedicated transfer infrastructure with end-to-end parallelization, and its prototype transferred a single 1 TB file at 90.08 Gbps over a transpacific path.

  • Problem

    Research data transfers have become a bottleneck because facilities and computing centers are separated while conventional interfaces do not match available network bandwidth.

  • Method

    RED-ONION combines DTNs, a dedicated high-bandwidth network, an all-flash parallel file system, and multi-threaded software with end-to-end storage and network parallelism.

  • Results

    90.08 Gbps was achieved for a single 1 TB disk-to-disk transfer over a transpacific path with a 150 ms round-trip time.

  • Takeaways & Limitations

    The system demonstrates wire-rate single-file transfer without relying on larger requests or aggregating throughput across many files.

Abstract

from arXiv · show

Modern experimental instruments produce data faster than general-purpose file transfer interfaces can move it, so delivery to the computing infrastructure has become a bottleneck in the research process. At many universities and research institutes, moreover, the instruments that generate research data and the high-performance computing systems that analyze it are separated both geographically and organizationally, because each demands its own expertise and installation environment. Connecting the two seamlessly is a pressing challenge for data-driven science. This article presents RED-ONION, a high-speed disk-to-disk transfer system that connects research facilities, on campus and beyond, to a computing center. The system combines data transfer nodes, a dedicated high-bandwidth network, an all-flash parallel file system, and multi-threaded transfer software that parallelizes network transmission and storage access. The design targets the wire rate both along the entire path, from the read on the sender storage to the write on the receiver storage, and for a single file between one pair of nodes rather than only in aggregate over many files or nodes. We describe the end-to-end optimizations across the transfer software, the operating system, and the storage that this requires. We evaluate a prototype deployed over a 100 Gbps transpacific path between Atlanta and Tokyo with a 150 ms round-trip time, on which a single 1 TB file transfer reached 90 Gbps, delivering a terabyte in approximately 95 s. Moving a dataset of this size therefore becomes a routine step, and the computing center serves an instrument as if the two were co-located.

I. INTRODUCTION

RED-ONION addresses the growing bottleneck of moving research data from geographically and organizationally separate experimental facilities to computing centers. It combines dedicated infrastructure and end-to-end optimization to target wire-rate disk-to-disk transfer for single large files and bulk small-file transfers.

  • Motivation: Terabytes-to-petabytes of daily data from modern instruments make rapid delivery to computing infrastructure increasingly important.Examples include DNA sequencers, radio telescopes, cryo-electron microscopes, and light sources.
  • Motivation: Separate expertise and infrastructure requirements keep experimental facilities and computing centers apart, while research data must still reach the computing center quickly.The data supports analysis, large-scale simulation, and machine learning.
  • Motivation: Conventional SCP/SFTP, object-storage APIs, and web interfaces do not match underlying network bandwidth, making transfer time a research-process bottleneck.The limitation becomes more consequential as data volumes grow.
  • Contribution: The paper optimizes the transfer path from sender storage to receiver storage for single large files and bulk small-file transfers at wire rate.The prototype evaluation uses a demanding transpacific path between the United States and Japan.
  • System: RED-ONION combines data transfer nodes, a dedicated high-bandwidth network, an all-flash parallel file system, and high-speed transfer software.The system is intended for campus facilities and can extend to off-campus research facilities over wide-area research networks.
  • Evaluation: 90 Gbps was achieved for a single 1 TB file over a transpacific prototype deployment between Atlanta and Tokyo.The result is reported for disk-to-disk transfer.

II. TECHNICAL CHALLENGES

RED-ONION treats disk-to-disk transfer as an end-to-end systems problem rather than a network-only problem. Its targets include wire-rate single-file performance, production availability, and scalability beyond aggregate multi-file or multi-node throughput.

  • End-to-end performance: Each stage from sender storage through the file system, page cache, network stack, interface, WAN, and receiver storage can independently throttle disk-to-disk throughput.Optimizing one stage in isolation can merely relocate the bottleneck.
  • End-to-end performance: Wire-rate performance must be sustained from reading sender-side storage through writing receiver-side storage, not only across the network.This defines the end-to-end optimization target.
  • Single-file transfer: RED-ONION targets wire-rate throughput for one file transferred between one pair of data transfer nodes.Prior work often achieved aggregate wire rate using many concurrent files or multiple nodes.
  • Production operation: The system must remain available during DTN or disk failures and scale to additional on-campus and off-campus sites.These requirements follow from its role as deployed research infrastructure rather than a research testbed.

III. HIGH-SPEED DISK-TO-DISK TRANSFER SYSTEM

RED-ONION is an architecture for disk-to-disk transfer built from DTNs, dedicated high-bandwidth networking, shared all-flash parallel storage, and multi-threaded transfer software. Its components jointly parallelize storage access and network transmission across the transfer path.

  • Architecture: A sender DTN reads and splits files into blocks, streams them over parallel TCP connections, and a receiver DTN reassembles and writes them to shared high-speed storage.The stored data can later move to archival storage or a supercomputer file system.
  • Architecture: DTNs provide the compute, network-interface, storage, and transfer-software endpoints for inter-site data movement.The architecture assigns these servers responsibility for exchanging data with peer sites.
  • Architecture: The dedicated network uses Ethernet at 100 Gbps or higher, with dedicated campus fiber or wide-area research-network paths.This separates the transfer path from general-purpose connectivity.
  • Architecture: Receiver-side high-speed storage is an all-flash parallel file system that stores received data and can move it to object or supercomputer storage.The DTNs mount this shared storage.
  • Architecture: Transfer software splits files into blocks, distributes them across connections, and performs storage reads and writes concurrently with network transmission.The design uses a high degree of parallelism across storage and networking operations.

B. DATA TRANSFER NODE AND HIGH-BANDWIDTH NETWORK

The DTN and storage design combines parallel hardware, resilient shared storage, and network tuning to sustain high-rate single-file transfers. Files are striped across storage targets so one file can use multiple servers and devices concurrently.

  • Data transfer nodes: A DTN dedicates one thread to each TCP stream, so CPU core count bounds the number of streams available to fill the link.Receiver DTNs use separate network interfaces for transfer traffic and storage traffic to avoid contention.
  • Data transfer nodes: Shared storage lets service fail over when a DTN fails, while adding DTNs increases site transfer capacity.Because one DTN is designed to fill the link, losing one primarily reduces concurrent service capacity rather than individual-transfer throughput.
  • High-bandwidth network: The network uses Ethernet at 100 Gbps or higher with end-to-end jumbo frames that reduce packet-processing overhead.Jumbo frames reduce the number of packets needed for a given throughput.
  • High-speed storage: EXAScaler spreads one file across multiple object storage targets so its I/O can be served by multiple object storage servers and devices in parallel.A parallel file system is needed because a single device or server usually cannot sustain the network wire rate.
  • High-speed storage: A file’s stripe count specifies the number of OSTs used, while its stripe size specifies consecutive bytes written to one OST before moving to the next.These parameters control the file layout across the parallel file system.
  • High-speed storage: RAID volumes and high-availability OSS pairs preserve storage service through individual-drive or OSS failures.Failed OSS storage targets can fail over to their partner OSS.

D. DATA TRANSFER SOFTWARE

RED-ONION uses Archaea as a multi-threaded transfer pipeline that overlaps storage access with network transmission. Its I/O modes and tuning choices target high-throughput disk-to-disk movement.

  • D. DATA TRANSFER SOFTWARE: Archaea uses TCP and a multi-threaded pipeline that decouples storage access from network transmission.One I/O thread queues content blocks while network threads transmit them over separate TCP streams.
  • D. DATA TRANSFER SOFTWARE: Blocks are handed from storage to network threads without additional copies because both paths use the same block size.
  • D. DATA TRANSFER SOFTWARE: Buffered I/O uses the page cache, direct I/O bypasses it, and asynchronous I/O combines direct I/O with Linux native asynchronous I/O.Asynchronous I/O keeps many operations in flight to hide storage latency.
  • D. DATA TRANSFER SOFTWARE: Reaching wire rate requires jointly tuning the transfer software, operating system, and storage system.
  • D. DATA TRANSFER SOFTWARE: The software configuration uses asynchronous I/O, disables encryption within the trusted network, enlarges transfer buffers, and increases content block size.The buffers absorb temporary rate differences between storage and network, while larger blocks amortize per-block overhead.

2) Operating System

The operating-system and storage configuration is tuned to keep a single file from being constrained by one storage target or by TCP’s bandwidth-delay limits.

  • 2) Operating System: TCP socket buffers must accommodate the bandwidth-delay product or the sender can stall while waiting for acknowledgments.
  • 2) Operating System: A single file is striped across all OSTs so its I/O is distributed across the storage system.
  • 2) Operating System: The stripe size is chosen so its product with the stripe count divides each I/O request size.This lets simultaneous RPCs reach every OST without leaving one idle.

A. EXPERIMENTAL SETUP

The prototype spans Atlanta and Tokyo with specialized transfer nodes, high-speed storage, and two evaluated network paths. The paths represent local-campus and overseas research-facility deployments.

  • A. EXPERIMENTAL SETUP: The prototype was deployed across two sites in the United States and Japan and evaluated during SC24.
  • A. EXPERIMENTAL SETUP: Atlanta DTNs use ten NVMe SSDs and dual-port 100 Gbps Ethernet, while the Tokyo DTN uses separate adapters for EXAScaler traffic and data transfer.
  • A. EXPERIMENTAL SETUP: The shared storage is an all-flash appliance with about 30 TB effective capacity, 35 GB/s aggregate read/write performance, four 100 Gbps links, four OSSs, and eight OSTs.
  • A. EXPERIMENTAL SETUP: The evaluation compares a loopback path for same-campus ingestion with a transpacific path for overseas-facility ingestion.
  • A. EXPERIMENTAL SETUP: The loopback path transfers between Atlanta DTNs over local NVMe, whereas the transpacific path transfers from Atlanta local NVMe to Tokyo shared EXAScaler over a 100 Gbps link.The transpacific path has an observed round-trip time of approximately 150 ms.

4) Monitoring

The evaluation measures end-to-end and instantaneous network throughput for 1 TB transfers under tuned conditions. Loopback results show that asynchronous and direct I/O approach the 100 Gbps wire rate, while buffered I/O is substantially slower.

  • 4) Monitoring: Prometheus samples at most once per minute, so vnStat records per-interface transfer volume every two seconds to resolve throughput over a transfer’s lifetime.
  • 4) Monitoring: The transpacific tuning uses a 220 GB total transfer buffer, 300 MB per-connection buffers, 64 MiB content blocks, and 1 GiB maximum TCP buffers.The TCP buffer setting is based on a roughly 1.75 GiB bandwidth-delay product at 100 Gbps and 150 ms round-trip time.
  • 4) Monitoring: Each path transfers a 1 TB file using buffered, direct, and asynchronous I/O while measuring effective end-to-end and instantaneous network throughput.
  • 4) Monitoring: Transpacific buffered transfers were aborted after throughput became steady, so their effective throughput uses only the completed portion of the file.

B. LOOPBACK PATH

The loopback and transpacific experiments show that asynchronous I/O best approaches wire-rate disk-to-disk transfer, while buffered and direct I/O are constrained by storage-side behavior.

  • Loopback path: 98.56 Gbps and 97.60 Gbps were achieved by asynchronous and direct I/O, respectively, on the 100 Gbps loopback path.Buffered I/O reached only 8.96 Gbps because page-cache write-back limited sustained throughput.
  • I/O mechanisms: Buffered I/O is limited by page-cache write-back, which makes sustained throughput depend on the kernel’s rate of flushing dirty pages to SSDs.This limitation appears on both the loopback and transpacific paths.
  • Transpacific path: 90.08 Gbps was achieved by asynchronous I/O for a single 1 TB transpacific transfer, compared with 12.08 Gbps for buffered I/O and 35.60 Gbps for direct I/O.The asynchronous transfer rose quickly to nearly 100 Gbps.
  • I/O mechanisms: Direct I/O keeps too few requests in flight to saturate the parallel file system’s object storage targets, whereas asynchronous I/O raises concurrency without enlarging the block size.Asynchronous I/O keeps many requests in flight, increasing storage-side concurrency.
  • Multi-file transfers: With concurrent files, buffered I/O flattened near 47 Gbps, direct I/O neared wire rate at four files, and asynchronous I/O reached 96.5 Gbps with one file.Asynchronous I/O sustained 88.9 Gbps in aggregate with two files.
  • Overall result: The 150 ms transpacific experiment was RED-ONION’s most demanding scenario, yet asynchronous I/O moved one file at 90 Gbps on the 100 Gbps link.The result indicates that the I/O bottleneck and single-file inefficiency were largely resolved within the tested scope.

V. RELATED WORK

Related transfer systems commonly optimize aggregate movement of multi-file datasets, while RED-ONION targets near-wire-rate transfer of a single large file between one pair of nodes.

  • Existing transfer tools: GridFTP, mdtmFTP, and XRootD primarily report aggregate throughput over concurrent transfers rather than the speed of an individual file.These systems use techniques including parallel TCP streams, multicore-aware scheduling, zero-copy, asynchronous I/O, and federated concurrent access.
  • Single-file transfers: 63% of Globus transfers involve one file, motivating RED-ONION’s focus on moving one terabyte-class file as fast as the link allows.Globus addressed this use case by adding chunking across several pairs of data transfer nodes.
  • Related performance: 77.25 Gbps was achieved by Zettar zx13 when moving 1 PB disk-to-disk on a 100 Gbps testbed with approximately 90 ms round-trip time.This represented 93.44% of the separately measured 82.67 Gbps storage throughput.
  • Comparison basis: Figure 8 compares instantaneous throughput for multi-file transpacific transfers across I/O modes and concurrency levels.The figure is framed around how throughput changes as the number of concurrent files increases.

B. HIGH-PERFORMANCE DATA TRANSFER PROTOCOLS

High-performance transfer approaches include UDP-based protocols, RDMA, and tuned TCP over dedicated networks. RED-ONION implements the latter approach with performance-tuned data transfer nodes and end-to-end optimization, while leaving production multi-tenant operation for future work.

  • UDP-based protocols: UDT, FASP, and HpFP replace TCP’s transport functions with UDP-based reliability, congestion control, pacing, or retransmission mechanisms.These designs target paths with high bandwidth-delay products, high packet loss, or both.
  • TCP assumption: On RED-ONION’s dedicated network, packet loss is essentially zero, and tuned TCP can fill available bandwidth when socket buffers and congestion control are configured appropriately.The Linux network stack is also described as continuously improving.
  • Science DMZ architecture: Science DMZ places performance-tuned data transfer nodes at a network edge without general-purpose security and policy bottlenecks, alongside continuous monitoring.RED-ONION is presented as an implementation of this architectural pattern.
  • RED-ONION result: 90.08 Gbps was achieved for a single 1 TB disk-to-disk transfer over a 100 Gbps Atlanta–Tokyo path with 150 ms round-trip time using asynchronous I/O.Buffered and direct I/O reached 12.08 Gbps and 35.60 Gbps, respectively.
  • Future work: Production deployment still requires request queuing, scheduling, and multi-tenant sharing across sites, data transfer nodes, networks, and storage.The article focuses on transfer performance between sites rather than these operational concerns.
Loading 2608.29053v1…