Source-linked AI summary
Operational Experience and R&D results using the Google Cloud for High Energy Physics in the ATLAS experiment
Fernando Barreiro Megino, Kaushik De, Johannes Elmsheuser, Alexei Klimentov, Mario Lassnig, Miles Euell, Nikolai Hartmann, Tadashi Maeno, Verena Martinez Outschoorn, Jay Ajitbhai Sandesara, Dustin Sell
TL;DR
ATLAS faces projected order-of-magnitude growth in LHC computing demand, motivating work to close the gap between future needs and expected resources. The paper evaluates Google Cloud as an ATLAS site and uses elastic resources for R&D; it reports successful integration with ATLAS workflow and data management systems and demonstrated scaling beyond 100k concurrently running vCPUs in PanDA.
Problem
ATLAS faces projected order-of-magnitude growth in LHC computing demand, creating a gap between future needs and expected resources.
Source: §1 · p. 2Method
The study evaluates Google Cloud as a complete ATLAS computing site and uses elastic cloud resources for R&D across GPUs, ARM CPUs, and columnar analysis.
Source: §1 · p. 3Results
Google Cloud services were successfully integrated with PanDA and Rucio, and scaling beyond 100k concurrently running vCPUs in PanDA was demonstrated.
Source: §5 · p. 19Takeaways & Limitations
The cloud-site solutions are not ATLAS-specific and may be considered by other communities for HL-LHC analysis facilities.
Source: §5 · p. 19Google Cloud Storage’s HTTP interface supports only one range per request, requiring concurrent single-range requests for columnar data reads.
Source: §4.3 · p. 18
Abstract
from arXiv · showhide
The ATLAS experiment at CERN relies on a worldwide distributed computing Grid infrastructure to support its physics program at the Large Hadron Collider. ATLAS has integrated cloud computing resources to complement its Grid infrastructure and conducted an R&D program on Google Cloud Platform. These initiatives leverage key features of commercial cloud providers: lightweight configuration and operation, elasticity and availability of diverse infrastructure. This paper examines the seamless integration of cloud computing services as a conventional Grid site within the ATLAS workflow management and data management systems, while also offering new setups for interactive, parallel analysis. It underscores pivotal results that enhance the on-site computing model and outlines several R&D projects that have benefited from large-scale, elastic resource provisioning models. Furthermore, this study discusses the impact of cloud-enabled R\&D projects in three domains: accelerators and AI/ML, ARM CPUs and columnar data analysis techniques.
1. Introduction
ATLAS relies on a global Grid to process and analyze LHC data, while projected HL-LHC demands motivate R&D on complementary computing resources. This paper describes a Google Cloud project evaluating a complete ATLAS site and cloud-enabled R&D, while leaving its TCO results for separate discussion.
1. Introduction
- Distributed computing underpins HEP computing, supporting results such as the Higgs boson discovery.The passage cites the discovery as an example [1,2].Source: §1 · p. 2
- HL-LHC computing demands are expected to increase by an order of magnitude, prompting R&D to close the gap between future needs and expected resources.The passage attributes the increase to a 5–7x luminosity increase and a 4–5x increase in event size.Source: §1 · p. 2
- US ATLAS began a Google collaboration in 2017, launched a supported next phase in 2020, and started the ATLAS-Google project studied here in May 2022.The 15-month project used a subscription model with a fixed monthly cost and resource targets averaged over the subscription period.Source: §1 · p. 3
- The project evaluated a complete Google Cloud ATLAS site, ARM and GPU architectures, and the facility’s total cost of ownership.This paper focuses on technical integration and R&D results; the TCO study is discussed elsewhere [10].Source: §1 · p. 3
- The R&D projects used GPUs and large-memory resources for neural simulation-based inference, ARM CPUs for benchmarking and validation, and clusters for columnar analysis.The columnar-analysis work compared file formats.Source: §1 · p. 3
2. ATLAS Distributed Computing overview
The Worldwide LHC Computing Grid (WLCG) provides the distributed computing and storage infrastructure for LHC experiments, including ATLAS. Rucio and PanDA manage data and workloads across this infrastructure, which includes Grid, HPC, and cloud resources.
2. ATLAS Distributed Computing overview
- The WLCG is a global collaboration of more than 170 computing centers, organized in tiers, that provides the resources needed to handle LHC experiments’ exabyte of data.The passage identifies CERN’s Tier-0 as the primary hub for storage and initial processing.Source: §2 · p. 3
- A typical Grid site combines a Computing Element for workload execution, a Storage Element, and internal and external network bandwidth.The components rely on in-house technologies and middleware developed by several Grid initiatives.Source: §2 · p. 4
- Rucio and PanDA provide ATLAS users a central entry point that hides the complexity of distributed resources with different Grid, HPC, and cloud configurations.The overview diagram depicts this high-level arrangement.Source: Figure 1 · p. 4
- Rucio manages ATLAS data and its locations, while PanDA schedules and manages job execution across distributed resources.PanDA interacts with Rucio for data locations, replicas, and registration of job outputs; together they manage Grid, HPC, and cloud resources.Source: §2 · pp. 4–5
3. Re-imagining and operating an ATLAS Grid site with cloud technologies
The section describes adopting cloud-native services to operate an ATLAS cloud site integrated with Grid storage, workload, and analysis systems. Elastic provisioning supported simulation and reprocessing workloads, while networking work addressed transfer costs and explored Full Chain and peering.
3.1. Considerations when adopting cloud technologies
- Cloud choices span lift-and-shift and managed services; elasticity, Spot pricing, and diverse processors support variable workloads and quicker access to specialized hardware.Source: §3.1 · pp. 5–6
3.2. Technology Choices
- The site chose cloud-native services and standard protocols over installing and managing Grid Storage and Compute Elements, aiming to use cloud capabilities while reducing operational effort and cost.Source: §3.2 · p. 6
3.2.1. Storage integration
- Object Stores integrate with Rucio and FTS through signed URLs, enabling time-limited access to specific resources over standard HTTP without provider-side storage maintenance.Source: §3.2.1 · pp. 6–7
3.2.2. Compute integration
- Managed Kubernetes runs batch and interactive pods, autoscaling queued workloads; PanDA submits via Harvester, while JupyterHub users can launch Dask clusters for parallel analysis.CVMFS is mounted across pods; queues include x86, GPU, and ARM work on Spot instances with transparent PanDA retries.Source: Figure 2 · p. 7, §3.2.2 · pp. 7–8
3.3. Operational experience
- At 100,000 vCPUs, two 50-million-event campaigns each took around a day versus about a week on the full Grid; 15% of 2022 reprocessing ran on 10,000 vCPUs within a week.The site’s 5% wall-clock failure rate, including preemption and infrastructure errors, aligned closely with pledged ATLAS resources; R&D queues also used large-memory nodes and GPUs.Source: §3.3 · pp. 8–9, Figure 3 · p. 9
3.4. Networking considerations
- Egress and inter-region transfers incur costs; Full Chain keeps processing stages colocated at some cost to flexibility, while cloud–LHCONE/LHCOPN peering remains complex and needs further option evaluation.Source: §3.4 · p. 10
4. R&D Projects
Cloud elasticity supported ATLAS R&D across AI/ML, ARM CPUs, and columnar analysis, provisioning resources for workloads with distinct compute and memory needs.
4. R&D Projects
- R&D used elastic resources through PanDA or interactive Jupyter and Dask, developing an ML-based analysis technique, accelerating ARM software migration, and investigating compact columnar formats.Source: §4 · p. 11
4.1.1. Application to neural simulation-based inference analyses in the ATLAS experiment
- An NVIDIA T4 GPU trained one NN almost 5× faster than a CPU using the same O(1M) simulated events, a difference relevant to ensembles of thousands of NNs.The paper notes that only a small percentage of WLCG sites offer GPU resources and their configurations are not homogeneous.Source: §4.1 · p. 11, Figure 4 · p. 12
- Cloud resources made an NSBI analysis with over a billion NN parameters tractable, using GPUs and nodes with over 1 TB of memory beyond ATLAS Grid capacity.NSBI approximates otherwise intractable probability densities and density ratios; the Hessian computation used row-by-row parallelization with auto-differentiation and just-in-time compilation.Source: §4.1.1 · pp. 11–12, §4.1.2 · p. 13
4.2. Usage of ARM CPUs
- ARM CPUs benchmarked competitively against tested Intel and AMD processors, and ATLAS software passed physics validation on ARM.The validation covered reconstruction and full Geant4 detector simulation workflows; ATLAS sites began considering ARM capacity.Source: §4.2 · pp. 14–15
4.3. Columnar data analysis
- For remote column reads, Google Cloud Storage’s single-range HTTP interface required concurrent requests; a custom asynchronous uproot source scaled better than the default thread pool.With 10 threads, scattered reads of 10% of a file took ten times longer than downloading the whole file; 100 concurrent aiohttp connections performed best in these tests.Source: §4.3 · p. 18
- Parquet files processed twice as fast as ROOT TTree files for simple filtering and object selection, with both tasks mainly limited by data reading.The reported advantages include metadata reading, larger compression units, and column layout; the tests used Parquet because ROOT TTree settings could not address all these factors.Source: §4.3 · p. 18
- Scaling was almost linear through 300 cores, with no further speedup beyond 700; longer object-selection tasks finished mostly within 10 minutes on 4,000 cores.The scaling test used 10% of the files; its 2–5-second tasks required a 32 GB scheduler node to extend to several thousand cores.Source: §4.3 · pp. 18–19, Figure 8 · p. 19
- Cloud elasticity enabled rapid iteration for interactive columnar analysis, with results for each iteration obtained within minutes for debugging and optimization.Jupyter and Dask setups were not yet widely implemented across Grid sites.Source: §4.3 · p. 19
5. Summary and Conclusions
Google Cloud services were integrated into ATLAS workflow and data management systems and operated at medium-sized Grid-site scale, with cloud R&D continuing into future analysis and data-processing work.
5. Summary and Conclusions
- Google Cloud services integrated with ATLAS workflow and data management systems PanDA and Rucio, and operation demonstrated more than 100k concurrently running vCPUs in PanDA.The services operated at the scale of a current medium-sized Grid site.Source: §5 · p. 19
- The described solutions are not ATLAS-specific and can be adapted by other HEP or non-HEP communities, including for possible HL-LHC analysis facilities.Source: §5 · p. 19
- Future work plans to continue using cloud resources to study novel analysis and data-processing techniques and optimize data exports from cloud data centers to WLCG sites with research network providers.Source: §5 · p. 20