PaperlayerView exact revision on arXiv

Source-linked AI summary

Client-side transparent caching for remote ROOT data analysis

Dmytro Kovalskyi, Jan Eysermans, Mariarosaria D'Alfonso, Christoph Paus

arXiv:2609.00400v1cs.DChep-ex

TL;DR

Repeated remote-data analysis passes can be slowed by transferring and decompressing the same bytes while remote performance fluctuates. uCache provides a transparent local XRootD cache, and measured warm passes were substantially faster with minimal filling overhead.

  • Problem

    Repeated analysis passes spend time transferring and decompressing data already processed, while large datasets and variable remote-resource performance slow feedback.

  • Method

    uCache is a transparent XRootD client plugin that stores only requested data locally and can build recompressed, reordered replicas without server deployment or analysis-code changes.

  • Results

    1.6–8.8 times faster byte-cache serving and an additional 1.2–4.7 times improvement from replicas were measured, while cache filling added 0.1–1.2% waiting overhead in typical passes.

  • Takeaways & Limitations

    Warm-cache performance is more stable and less dependent on remote-source conditions, with benefits varying by workload, storage, network resources, format, and source distance.

  • Takeaways & Limitations

    Storage-path improvements do not reduce wall-clock time when warm passes are compute- or decompression-bound; on the LZMA dataset, a 5.5-fold reduction in byte-tier device operations changed time by less than 1%.

Abstract

from arXiv · show

High-energy physics analyses often process the same data as physicists refine algorithms and test new ideas. With data increasingly read from remote storage, each iteration is subject to network latency and depends on network bandwidth and shared-storage throughput, which can vary substantially under load. We present uCache (xrd-ucache), a transparent client-side cache implemented as an XRootD client plugin that requires neither server-side deployment nor changes to analysis code. It uses local storage on the analysis machine as a cache layer between the network and memory. The cache stores only the data actually read by an analysis. It can also rebuild cached data into a branch-aligned, recompressed form, eliminating most of the input/output and decompression costs of subsequent passes. We benchmark the cache using the Analysis Grand Challenge top quark pair analysis on public CMS Open Data compressed with zlib and LZMA. Filling the cache adds essentially no overhead compared with a direct read. Subsequent passes are 1.6-8.8 times faster from the byte cache and 2.1-15.7 times faster from the recompressed cache. For a typical analysis, a 1 TB cache suffices for datasets of 10-20 TB. The largest improvements occur when the remote data source is heavily loaded or geographically distant.

1 Introduction

HEP analyses repeatedly revisit large datasets, but remote access and shared-resource contention can slow feedback and restrict experimentation. uCache addresses this by caching only read data locally, without server changes or analysis-code modifications.

  • Iterative analysis cycles require repeated implementation, processing, inspection, and refinement of selections, observables, background models, biases, and unexpected features.
  • Faster feedback lets physicists test more alternatives, investigate unexpected observations, and identify weak approaches or possible biases earlier.
  • Tens to 100 TB-scale datasets, transient failures, and high resource demand can make complete results take considerable time.
  • Skimming and slimming reduce data volume, but a 3 to 4 orders-of-magnitude reduction can restrict experimentation and require time-consuming final flat ntuple production.
  • Remote XRootD reading exposes each pass to network latency, variable shared-origin throughput, and temporary server errors.
  • uCache uses local SSD or NVMe storage as a transparent XRootD client-side cache, storing only accessed data and optionally reorganizing it for repeated passes.

2 Design

uCache combines a sparse byte tier with an optional branch-major replica tier. The design fetches only requested data, supports integrity checks and fail-open behavior, and can locally transpose and recompress cached ROOT baskets for efficient warm reads.

  • Deployment: The plugin requires configuration only, with no analysis-code changes, server installation, administrator support, or elevated privileges.
  • Byte tier: The byte tier stores requested file ranges in 4 KiB pages indexed by URL, fetching only data actually requested and reusing overlapping cached ranges.
  • Byte tier: A complete pass fetches only 5.7–5.9% of the dataset on the reference setup, while warm passes are served locally without contacting the origin.
  • Reliability: CRC32C page checksums, atomic metadata updates, retries, and direct-origin fallback preserve access when cached data or the cache itself encounters errors.
  • Replica tier: The replica tier lays cached ROOT baskets out branch-major and can transcode LZMA-9 to ZSTD-1, reducing scattered reads and decompression work.
  • Replica tier: On the LZMA dataset, replica serving uses roughly 18 times fewer and 25 times larger reads, while replicas occupy 1.52 times the byte-only cache size.

3 Benchmarking method

The evaluation uses a realistic AGC CMS top-quark analysis across two compression regimes, multiple machines, storage devices, and remote sources. Measurements combine wall-clock results with uCache counters and protocol checks to distinguish cache, storage, network, and computation effects.

  • Workload: The benchmark runs the AGC CMS t¯t RDataFrame analysis with object selections, systematic variations, histogramming, and 15 NanoAOD branches across 9 concurrent graphs.
  • Datasets: Two datasets represent zlib-1 and LZMA-9 compression regimes, with the analysis reading about 6% of stored data: 101 GB and 143 GB per pass.
  • Test machines: Six machines span personal, workstation, facility, cloud, and Apple-silicon environments, with configurations and source round-trip conditions recorded during runs.
  • Measurements: Storage characterization measures sequential bandwidth, random 4 KiB reads, latency, and uCache fill and read paths at analysis concurrency.
  • Resource conditions: Shared-service sequential bandwidth varied from 153 and 158 MB/s in recent measurements to 315 MB/s in the first campaign.
  • Protocol: Protocol checks require low warm-pass origin share, zero CRC, fail-open, and validation failures, and report replica coverage for every warm-replica measurement.

4 Benchmarking results

Across suitable local-storage configurations, uCache adds little cost while filling and accelerates repeated analysis passes; the replica tier further reduces processing and decompression work. Gains depend on source conditions and cache-device capability, with slow or rate-limited storage potentially making caching slower than direct access.

  • First-pass overhead: 0.1–1.2% fill overhead is measured with suitable local cache storage, because cache writing overlaps waiting for remote data.The FNAL case reaches 3.1% when source delivery approaches the device’s sustained write rate.
  • Storage limitation: 6.9 times slower warm byte-tier and 3.0 times slower replica-tier access occurred on the rate-limited cloud volume.Its constrained I/O operations slowed caching below direct remote access; the fill cost reached 6.1 times the direct time.
  • Warm-pass performance: 1.6 times faster byte-cache access was achieved even on a workstation only 0.30 ms from CERN; LZMA-9 replicas added another 1.6 times speedup over bytes.Most of the replica gain came from reducing decompression work.
  • Warm-pass performance: Repeated analyses run faster on every tested dataset and machine configuration with sufficiently fast local storage, while absolute warm-cache times remain stable across remote-source conditions.Speedup ratios vary with network distance and remote-source load, so they are not fixed cache properties.
  • Remote-access overhead: 20 s of fixed setup time and about 3 ms per file are removed when cached files are served locally.The saving increases with source distance, reaching 27.8 s at 77.9 ms network latency versus 1.55 s at 0.30 ms.
  • Replica performance gains: 3.20 times more CPU instructions are required by the UL2016 byte cache than by its replica tier, primarily because of data decoding.For AGC 2015, the corresponding ratio is 1.31.
  • Storage trade-offs: 16% more disk space is required for zlib replicas and retained bytes, while the LZMA cache occupies 52% more than caching original bytes alone.The complete totals are 116.4 GB for zlib and 217.8 GB for LZMA.

4.4 Storage access pattern and the limiting resource

Cache performance depends on the resource limiting each run: request reduction helps IOPS-bound storage, while decompression, bandwidth, or source variability can dominate elsewhere. Replica construction is often inexpensive and rapidly repaid, but background building can compete with fully utilized analyses.

  • Storage access pattern and the limiting resource: 14–18 times fewer reads and 14–25 times larger reads characterize the replica layout, but local NVMe and SATA runs remain limited by decompression or storage bandwidth.Reducing operation count improves performance only when request rate limits the device.
  • Storage access pattern and the limiting resource: 14.4 times fewer operations of 14.7 times the size cut the constrained cloud-volume penalty by 2.3 times, yet the pass remained 3.0 times slower than direct reading.Across campaigns, delivered throughput varied, but fewer, larger requests were better on operation-rate-limited storage.
  • Remote data source variability: 2.05 times separated two remote fills 51 minutes apart, while warm-replica runs varied by only 0.96% across software versions and days.Remote throughput from CERN to one client ranged from 52 to 211 MB/s within an hour.
  • Remote data source variability: 6.9 ms of network distance did not ensure higher throughput: MIT sustained 29–53 MB/s, CERN about 110 MB/s, and FNAL 407 MB/s.The measurements attribute the larger effect to load on shared production storage rather than network distance alone.
  • Costs and break-even point: 0.1–1.2% fill overhead was measured on suitable storage, with 3.1% against the fastest source; a rate-limited cloud volume instead cost 6.1 times a direct read.The cache’s 10% first-pass overhead target was met on suitable configurations but not on the constrained cloud volume.

4.7 Comparison with XCache

The comparison evaluates uCache against XCache in both facility-style and local configurations, showing that uCache’s main advantages are sparse first-pass fetching, lower disk use, and its replica tier.

  • Experimental setup: A two-machine setup places XCache on the workstation and uCache with the analysis on a VM, testing local-network versus same-machine access.The 10 GbE link sustains 1.17 GB/s, and the dedicated XCache instance makes this comparison favorable to XCache.
  • First-pass behavior: Negligible first-pass overhead lets uCache reach warm serving without materially slowing the initial analysis pass.Filling uCache adds negligible overhead compared with direct reading.
  • Footprint and fill cost: 5.7% of the dataset is stored by uCache, whereas tuned XCache takes 22.6 times the direct-reading fill time and uses 3.1 times as much disk.Default XCache fetches about 88% of the full dataset despite the sparse access pattern.
  • XCache configuration: 13.2 times slower than direct reading is the warm-pass result for default XCache when its requested footprint exceeds available cache space.The default configuration repeatedly refetches complete files, transferring about 1.58 TB on every pass.
  • XCache configuration: Prefetch 0 improves XCache’s warm-pass time by 21.5 times relative to its default configuration, but makes the first pass 1.5 times slower.A 16-fold sweep of cache block size changes warm-pass time by only 1.8%.
  • Overall comparison: 14.2 times faster filling, 349 times fewer remote operations, and 37% of tuned XCache’s disk space characterize uCache in this setup.On repeat passes, uCache’s byte tier is 1.51 times faster than XCache and its replica tier is 2.44 times faster.
  • Performance mechanism: 395 MB/s for XCache and 595 MB/s for uCache remain below the 1172 MB/s network link, implicating cache-storage differences rather than network saturation.The VM’s local NVMe provides 2.8 times the 4 KiB operation rate and 3.6 times the bandwidth of the XCache storage path.
  • Conclusion: uCache’s advantages over XCache concentrate in first-pass cost, disk footprint, and replica-tier serving; same-storage warm byte serving is similar.The replica tier further reduces repeat-pass time after uCache reaches the cached state.

4.9 Sensitivity to the ROOT version

The study repeats the TTree measurement across four ROOT versions to separate cache behavior from analysis-software cost. ROOT version changes the analysis work substantially, while replica-tier timing varies little and relative cache gains are larger for slower ROOT versions.

  • Experimental design: Four ROOT versions are compared using the same 1453 files, origin, workstation, thread count, cache device, and evicted-pass procedure.The 6.40.02 measurement uses a development uCache build with preliminary XRootD 6 support.
  • Analysis cost: 530 to 723 s is the warm byte-tier range across identical data and hardware, a factor of 1.36.Instruction counts range from 55.9 to 75.6 trillion while IPC remains between 1.36 and 1.47.
  • Analysis cost: ROOT 6.36.02 is fastest; ROOT 6.34.04 executes 35% more instructions for the same physics, while ROOT 6.38.04 and 6.40.02 recover part of the improvement.The measured differences are attributed to work rather than execution efficiency.
  • Cache invariance: 6.56 million cache reads, the same vector-read chunks, 119.6 GB served, and two file opens per file occur across all four versions.These stable uCache counters support attributing runtime differences to ROOT rather than changing cache access.
  • Cache sensitivity: The warm replica-tier pass stays within a 7.8% band across versions, compared with 36% variation for the byte-tier pass.From ROOT 6.36 onward, replica timing varies by only 1.6%.
  • Cache sensitivity: 2.83 to 2.89 times faster replica serving occurs on slower ROOT versions, versus 2.56 to 2.60 times on ROOT 6.36.02.The reported relative gain is largest where ROOT leaves more removable work.

5 Known limitations

uCache’s benefits depend on workload, cache capacity, storage performance, input assumptions, and client configuration. The paper identifies graceful degradation when caches are undersized, but also boundaries where caching provides limited or no speedup.

  • Single-pass workloads: Single-pass jobs gain mainly from lower file-open latency and failed-read retries; larger speedups require repeated reads.Iterative analysis development is the primary target use case.
  • Compute-bound warm passes: Compute-bound warm passes do not become faster when computation or decompression, rather than I/O, limits wall-clock time.On LZMA data, a 5.5-fold reduction in byte-tier device operations changed wall-clock time by less than 1%.
  • Dataset dependence: Replica speedup varies from 1.2–2.4 times on zlib-1 data to 1.6–4.7 times on LZMA-9 data because removable decompression work depends on input compression.A single speedup is not representative without specifying the dataset and compression format.
  • Cache storage: A cache device limited to ∼1000 read operations per second made the byte tier 6.9 times slower than no cache and made filling take 6.1 times as long as direct reading.The replica tier reduces the penalty with fewer, larger reads, but its throughput was unstable across campaigns.
  • Cache capacity: When the working set exceeds capacity, the current policy declines new admissions instead of evicting the running job’s working set, producing stable partial caching.At 1.16× oversubscription, 87% of files were cached, with zero evictions and a 1.86-fold speedup across evicted warm passes.
  • Scope and assumptions: The benchmark assumes read-only inputs, one AGC top-quark analysis, and XRootD 5.6-or-newer clients targeting EL9-or-newer systems.Older clients fail open to uncached reads, while workflows modifying inputs inside the analysis loop are outside scope.

6 Related work

Related systems address different layers or deployment models. uCache provides end-user, client-side caching for sparse reads, complementing rather than replacing shared site caching and ROOT-level mechanisms.

  • Site-level caching: XCache/XrdPfc is shared site infrastructure, whereas uCache is intended for an individual physicist’s analysis machine.The paper directly compares default and tuned XCache configurations with uCache.
  • ROOT I/O caching: TTreeCache batches reads for one process in memory, while uCache persists data across processes and days below that layer.Both mechanisms remain enabled together in the measurements.
  • Whole-file caching: ROOT’s local file cache stores whole remote files, unlike uCache, which stores only accessed data; measured analyses read about 6% of opened-file bytes.The granularity difference is especially relevant for sparse access.
  • Format-specific caching: The DAOS RNTuple cache and uCache both retain accessed compressed data, but target different environments, activation models, and formats.The DAOS cache uses remote high-bandwidth cluster storage and explicit analysis enablement, while uCache is an XRootD plugin.
  • Facility and format approaches: XCache staging for CMS targets analysis-facility-scale delivery, contrasting with uCache’s end-user caching model.RNTuple separately improves ROOT’s physical data organization, allowing the paper to distinguish format and cache effects.
  • Benchmark context: CMS Open Data benchmarks and AGC provide precedent for the computing evaluation, while the subMIT facility paper supplies an independent apparatus comparison.These works situate the benchmark within established CMS analysis-performance studies.

7 Conclusions

uCache turns local storage into a durable, transparent cache for repeated remote analysis reads. The measurements show little fill overhead, faster and more stable warm passes, and benefits shaped by workload, storage, network, and data format.

  • Design: uCache stores only pages read by the analysis and can build branch-reordered, recompressed replicas without site infrastructure or analysis-code changes.It operates as a durable local layer between the page cache and remote infrastructure.
  • Fill overhead: 0.1–1.2% of the first pass was spent waiting on cache filling, rising to 3.1% against the fastest source and remaining within the 10% design target.The cache fills during the first useful analysis pass without requiring separate staging.
  • Warm-pass performance: 1.6 to 8.8 times faster byte serving and an additional 1.2 to 4.7 times replica improvement were measured on repeated passes.Warm-cache times reproduced to about a percent, unlike direct-reading times that fluctuated with remote conditions.
  • Storage and construction cost: 6 to 9% of the full dataset was stored, while background replica construction overlapped with filling and recovered its added work within the first repeated pass in measured cases.Construction competed directly with analysis only on a fully loaded host reading from a nearby fast source.
  • Scope of conclusions: The cache applies to both TTree and RNTuple, but gains vary with workload and available storage and network resources.RNTuple can reduce additional byte-cache benefit when nearby sources already keep up with the analysis.
  • Network locality: 245.5 s over a 10 GbE hop compared with 163.0 s for the local byte tier and 100.7 s for the local replica tier.The comparison demonstrates why local cache storage matters for this workload, while faster fabrics could change the arithmetic.
  • Comparison with shared caching: uCache uses about one third as much disk as tuned XCache in the measured workload while providing similar warm byte-serving performance on the same machine and device.Shared XCache offers pooled capacity and cross-user sharing, whereas uCache serves individual users from local storage.

8 Software and data availability

The paper reports software and measurement provenance across multiple uCache releases, with comparisons constructed within the same release and session.

  • Software availability: uCache is open source under the MIT license, and the measurements span releases 0.18.1 through 0.21.0.No reported ratio crosses releases; each comparison divides quantities measured in the same session with the same release.

Declaration on the use of AI-assisted technologies

Claude Code was used in developing uCache and drafting the manuscript, including its tables and figures, under author direction and validation.

  • Claude Code assisted both uCache development and manuscript drafting, including tables and figures.The authors specified requirements, directed changes, and validated implementation behavior.
  • The authors validated the software using automated unit, differential, crash-recovery, and fuzz tests under memory and race detectors.

A.1 UL2016: official CMS Open Data UL2016 NanoAODv9

The UL2016 benchmark dataset uses published CMS Open Data NanoAODv9 records mapped to AGC sample roles, with documented physics caveats and a partial record availability boundary.

  • The UL2016 dataset maps AGC sample roles onto published UL2016 CMS NanoAODv9 records selected for generator/tune fidelity.
  • The mapping is an I/O replacement rather than a physics reproduction of the AGC result.
  • 30 of 117 advertised files are available for WJetsToLNu_1J record 69717 on the portal.The totals used are 1456 files, 2412 GB, and 1.30 × 10^9 events across 26 records.
  • Four tape-resident FNAL files were excluded, representing 0.27% of files and 0.22% of bytes.The remaining records were replicated to T1_US_FNAL_Disk and verified by per-file size.
  • Table 18 lists the 26 CMS Open Data UL2016 NanoAODv9 records used, with record 69717 counted by its 30 published files.

A.2 AGC 2015: AGC 2015 NanoAOD

The AGC 2015 dataset is a fixed 787-file NanoAOD input list derived from 13 published CMS 2015 Open Data MINIAODSIM records.

  • 787 NanoAOD files totaling 1777 GB and 0.94 × 10^9 events comprise the AGC 2015 fixed input list.
  • The AGC team derived these files from 13 published CMS 2015 Open Data MINIAODSIM records.
  • The files are served from the CERN Open Data EOS instance and fixed by the AGC repository’s nanoaod_inputs.json.

B Software versions and reproduction

The paper records software versions, analysis configuration, XCache comparison settings, and cache-validation procedures to support reproduction across machines and campaigns.

  • Software versions: uCache results use releases 0.19.1, 0.20.0, 0.21.0, or 0.18.1 depending on the campaign and comparison.
  • Software versions: ROOT 6.36.02 and XRootD 5.8.3 are used on most Linux systems, while the Mac mini uses ROOT 6.38.04 and XRootD 5.9.1.
  • Analysis configuration: The benchmark uses the AGC cms-open-data-ttbar RDataFrame implementation with input lists specified in Appendix A.
  • XCache comparison: The XCache comparison varies default and tuned prefetch, RAM, read-ahead, and queue settings on a two-machine setup.
  • Reproduction protocol: Warm-pass timing uses verified page-cache eviction and, for repeated campaigns, reports the median of three evicted passes.Run validity checks include origin share, CRC, fail-open, validation, and coverage.
  • Dataset provenance: Table 19 documents the 13 parent CMS 2015 MINIAODSIM records from which the AGC 2015 NanoAOD input list was derived.
Loading 2609.00400v1…