PaperlayerView exact revision on arXiv

Source-linked AI summary

A Roadmap for HEP Software and Computing R&D for the 2020s

Johannes Albrecht, Antonio Augusto Alves, Guilherme Amadio, Giuseppe Andronico, Nguyen Anh-Ky, Laurent Aphecetche, John Apostolakis, Makoto Asai, Luca Atzori, Marian Babik, Giuseppe Bagliesi, Marilena Bandieramonte, Sunanda Banerjee, Martin Barisits, Lothar A. T. Bauerdick, Stefano Belforte, Douglas Benjamin, Catrin Bernius, Wahid Bhimji, Riccardo Maria Bianchi, Ian Bird, Catherine Biscarat, Jakob Blomer, Kenneth Bloom, Tommaso Boccali, Brian Bockelman, Tomasz Bold, Daniele Bonacorsi, Antonio Boveia, Concezio Bozzi, Marko Bracko, David Britton, Andy Buckley, Predrag Buncic, Paolo Calafiura, Simone Campana, Philippe Canal, Luca Canali, Gianpaolo Carlino, Nuno Castro, Marco Cattaneo, Gianluca Cerminara, Javier Cervantes Villanueva, Philip Chang, John Chapman, Gang Chen, Taylor Childers, Peter Clarke, Marco Clemencic, Eric Cogneras, Jeremy Coles, Ian Collier, David Colling, Gloria Corti, Gabriele Cosmo, Davide Costanzo, Ben Couturier, Kyle Cranmer, Jack Cranshaw, Leonardo Cristella, David Crooks, Sabine Crépé-Renaudin, Robert Currie, Sünje Dallmeier-Tiessen, Kaushik De, Michel De Cian, Albert De Roeck, Antonio Delgado Peris, Frédéric Derue, Alessandro Di Girolamo, Salvatore Di Guida, Gancho Dimitrov, Caterina Doglioni, Andrea Dotti, Dirk Duellmann, Laurent Duflot, Dave Dykstra, Katarzyna Dziedziniewicz-Wojcik, Agnieszka Dziurda, Ulrik Egede, Peter Elmer, Johannes Elmsheuser, V. Daniel Elvira, Giulio Eulisse, Steven Farrell, Torben Ferber, Andrej Filipcic, Ian Fisk, Conor Fitzpatrick, José Flix, Andrea Formica, Alessandra Forti, Giovanni Franzoni, James Frost, Stu Fuess, Frank Gaede, Gerardo Ganis, Robert Gardner, Vincent Garonne, Andreas Gellrich, Krzysztof Genser, Simon George, Frank Geurts, Andrei Gheata, Mihaela Gheata, Francesco Giacomini, Stefano Giagu, Manuel Giffels, Douglas Gingrich, Maria Girone, Vladimir V. Gligorov, Ivan Glushkov, Wesley Gohn, Jose Benito Gonzalez Lopez, Isidro González Caballero, Juan R. González Fernández, Giacomo Govi, Claudio Grandi, Hadrien Grasland, Heather Gray, Lucia Grillo, Wen Guan, Oliver Gutsche, Vardan Gyurjyan, Andrew Hanushevsky, Farah Hariri, Thomas Hartmann, John Harvey, Thomas Hauth, Benedikt Hegner, Beate Heinemann, Lukas Heinrich, Andreas Heiss, José M. Hernández, Michael Hildreth, Mark Hodgkinson, Stefan Hoeche, Burt Holzman, Peter Hristov, Xingtao Huang, Vladimir N. Ivanchenko, Todor Ivanov, Jan Iven, Brij Jashal, Bodhitha Jayatilaka, Roger Jones, Michel Jouvin, Soon Yung Jun, Michael Kagan, Charles William Kalderon, Meghan Kane, Edward Karavakis, Daniel S. Katz, Dorian Kcira, Oliver Keeble, Borut Paul Kersevan, Michael Kirby, Alexei Klimentov, Markus Klute, Ilya Komarov, Dmitri Konstantinov, Patrick Koppenburg, Jim Kowalkowski, Luke Kreczko, Thomas Kuhr, Robert Kutschke, Valentin Kuznetsov, Walter Lampl, Eric Lancon, David Lange, Mario Lassnig, Paul Laycock, Charles Leggett, James Letts, Birgit Lewendel, Teng Li, Guilherme Lima, Jacob Linacre, Tomas Linden, Miron Livny, Giuseppe Lo Presti, Sebastian Lopienski, Peter Love, Adam Lyon, Nicolò Magini, Zachary L. Marshall, Edoardo Martelli, Stewart Martin-Haugh, Pere Mato, Kajari Mazumdar, Thomas McCauley, Josh McFayden, Shawn McKee, Andrew McNab, Rashid Mehdiyev, Helge Meinhard, Dario Menasce, Patricia Mendez Lorenzo, Alaettin Serhan Mete, Michele Michelotto, Jovan Mitrevski, Lorenzo Moneta, Ben Morgan, Richard Mount, Edward Moyse, Sean Murray, Armin Nairz, Mark S. Neubauer, Andrew Norman, Sérgio Novaes, Mihaly Novak, Arantza Oyanguren, Nurcan Ozturk, Andres Pacheco Pages, Michela Paganini, Jerome Pansanel, Vincent R. Pascuzzi, Glenn Patrick, Alex Pearce, Ben Pearson, Kevin Pedro, Gabriel Perdue, Antonio Perez-Calero Yzquierdo, Luca Perrozzi, Troels Petersen, Marko Petric, Andreas Petzold, Jónatan Piedra, Leo Piilonen, Danilo Piparo, Jim Pivarski, Witold Pokorski, Francesco Polci, Karolos Potamianos, Fernanda Psihas, Albert Puig Navarro, Günter Quast, Gerhard Raven, Jürgen Reuter, Alberto Ribon, Lorenzo Rinaldi, Martin Ritter, James Robinson, Eduardo Rodrigues, Stefan Roiser, David Rousseau, Gareth Roy, Grigori Rybkine, Andre Sailer, Tai Sakuma, Renato Santana, Andrea Sartirana, Heidi Schellman, Jaroslava Schovancová, Steven Schramm, Markus Schulz, Andrea Sciabà, Sally Seidel, Sezen Sekmen, Cedric Serfon, Horst Severini, Elizabeth Sexton-Kennedy, Michael Seymour, Davide Sgalaberna, Illya Shapoval, Jamie Shiers, Jing-Ge Shiu, Hannah Short, Gian Piero Siroli, Sam Skipsey, Tim Smith, Scott Snyder, Michael D. Sokoloff, Panagiotis Spentzouris, Hartmut Stadie, Giordon Stark, Gordon Stewart, Graeme A. Stewart, Arturo Sánchez, Alberto Sánchez-Hernández, Anyes Taffard, Umberto Tamponi, Jeff Templon, Giacomo Tenaglia, Vakhtang Tsulaia, Christopher Tunnell, Eric Vaandering, Andrea Valassi, Sofia Vallecorsa, Liviu Valsan, Peter Van Gemmeren, Renaud Vernet, Brett Viren, Jean-Roch Vlimant, Christian Voss, Margaret Votava, Carl Vuosalo, Carlos Vázquez Sierra, Romain Wartel, Gordon T. Watts, Torre Wenaus, Sandro Wenzel, Mike Williams, Frank Winklmeier, Christoph Wissing, Frank Wuerthwein, Benjamin Wynne, Zhang Xiaomei, Wei Yang, Efe Yazgan

arXiv:1712.06982v5physics.comp-phhep-ex

TL;DR

HEP’s ambitious experimental programme requires major software and computing R&D to acquire, manage, process, and analyse large data volumes. The paper sets out a coordinated programme for improving software efficiency, scalability, and performance while addressing community, training, career, and sustainability challenges.

  • Problem

    Future HEP experiments face large data-processing demands, simulation challenges, heterogeneous communities, and differing production and analysis requirements.

  • Method

    The paper proposes a coordinated software and computing R&D programme spanning fast simulation, vectorised particle transport, community practices, training, and career support.

  • Results

    The roadmap identifies improved software efficiency, scalability, and performance, including vectorised transport and fast simulation, as priorities for coping with future challenges.

  • Takeaways & Limitations

    Sustainable progress requires coherent community solutions, shared libraries and projects, investment in expertise and training, and recognition of software careers.

  • Takeaways & Limitations

    Training must balance transferable industry-standard skills with the experiment-specific training needed to accomplish scientific goals.

Abstract

from arXiv · show

Particle physics has an ambitious and broad experimental programme for the coming decades. This programme requires large investments in detector hardware, either to build new facilities and experiments, or to upgrade existing ones. Similarly, it requires commensurate investment in the R&D of software to acquire, manage, process, and analyse the shear amounts of data to be recorded. In planning for the HL-LHC in particular, it is critical that all of the collaborating stakeholders agree on the software goals and priorities, and that the efforts complement each other. In this spirit, this white paper describes the R&D activities required to prepare for this software upgrade.

1 Introduction

Particle physics’s broad experimental programme requires major investments in detector hardware and software R&D to acquire, manage, process, and analyse growing datasets. For HL-LHC, the community is coordinating software priorities and R&D to improve performance, extend physics reach, and sustain software over the programme’s lifetime.

  • The coming experimental programme spans Higgs and BSM physics, heavy flavour, dark matter, neutrino physics, quark-gluon plasma, and unexplored phenomena.
  • The programme requires commensurate software R&D for acquiring, managing, processing, and analysing data alongside investments in new or upgraded detector hardware.
  • HL-LHC is scheduled to begin in 2026 and may collect about 30 times more data than the LHC has produced so far.The existing LHC dataset is already close to an exabyte, making simple scaling of current solutions inadequate.
  • The HEP Software Foundation organised a community white-paper process to coordinate software goals and priorities for HL-LHC and complement stakeholder efforts.The process involved LHC experiments and wider HEP software and computing communities, with an estimated 250 participants and topical working groups.
  • The roadmap targets improved software efficiency, scalability, and performance; new computing approaches that extend detector physics reach; and long-term software sustainability.It also addresses data and knowledge preservation and attracting expertise through career recognition and training.
  • Investing in the roadmap may benefit other technically similar projects, particularly astrophysics experiments such as SKA, CTA, and LSST.

2 Software and Computing Challenges

HL-LHC data volumes and evolving computing hardware create software and resource challenges that cannot be addressed by simply scaling current solutions. The roadmap therefore emphasizes efficient resource use, adaptable architectures, coordinated software development, and sustained community investment.

  • Software sustainability and people: Existing HEP software is largely written for x86 64 and serial processing, creating sustainability concerns as original authors leave the field and code becomes difficult to maintain.The experiments have produced tens of millions of lines of code through contributions from thousands of physicists and computing professionals.
  • Resource and technology pressures: The amount of data experiments can collect and process will be limited by affordable software and computing, making resource efficiency central to future physics reach.The text specifically links future physics reach to how efficiently computing resources are used.
  • Resource and technology pressures: Power, storage, disk I/O, network use, and hierarchical data access require software that can exploit changing architectures and distributed resources.Candidate approaches include vectorisation, thread-based programming, GPUs, machine learning, and flexible use of different architectures.
  • Software sustainability and people: Human challenges require training, documentation, support, community coordination, common libraries, and recognition for software-specialist careers.The paper presents coherent community action as critical while acknowledging that some developments will remain experiment-specific.

3 Programme of Work

The programme of work proposes coordinated R&D across event generation, simulation, and shared software to meet rising precision and computing demands. It combines algorithmic improvements, efficient filtering and reweighting, concurrency, and sustained support for common tools and expertise.

  • Event generators: Future event generators must deliver higher theoretical precision while addressing the computational demands this precision creates.The roadmap identifies this as the central challenge for the next decade.
  • Event generators: Negative event weights and computational complexity make NLO samples require many more generated, simulated, and reconstructed events than LO samples.This is inconsistent with experiment budgets that allocate similar numbers of Monte Carlo events as real-data events.
  • Community support: Long-term generator development and career opportunities are considered critical because future experiments operate on multi-decade timescales.The paper states that increased effort alone is not guaranteed to produce the desired result.
  • Event generators: Reweighting event samples could reduce BSM sample-generation time, but requires the updated model’s phase space to be a subset of the original and faces additional NLO issues.Negative event weights remain a technical issue for reweighting.
  • Event generators: Generator efficiency can improve through integrated filtering that produces several orthogonal output streams from one inclusive event generation.The proposal targets CPU waste caused by repeatedly generating large inclusive samples for separate filters.
  • Detector simulation: Simulation R&D must support 150 times more data than Run 1 while serving diverse experiments and increasingly complex detectors within available computing budgets.The paper identifies geometry, physics coverage, accuracy, and performance as simultaneous demands.
  • Detector simulation: Speeding up critical Geant4 elements and revisiting digitisation can benefit multiple applications while creating opportunities for code sharing among experiments.The paper presents these gains as worth shared investment.

Current Practices

The simulation programme combines Geant4 maintenance with R&D on performance, modularity, fast simulation, digitisation, transport, and reproducible parallel computation. These efforts target the computational demands of HL-LHC and intensity-frontier experiments while preserving validated physics capabilities.

  • Current Practices: Intensity-frontier experiments require large simulations of beamlines, neutrino fluxes, low-energy neutrons, and rare backgrounds across broad experimental phase space.Accurate detector simulation is needed for neutrino-energy reconstruction, while Muon g-2 and Mu2e face very low signal-to-background ratios.
  • Current Practices: Geant4 remains the primary simulation toolkit, requiring continuous maintenance, physics-model improvement, and performance optimisation for current and future experiments.The toolkit supports nearly every HEP experiment, while alternative engines must first become available, integrated, and validated.
  • Research and Development Programme: GeantV pursues multithreaded, vectorised particle transport by grouping similar tracks across events and combining SIMD coding with improved data locality.Vectorised geometry, navigation, and physics libraries are developed independently so they can also support Geant4.
  • Research and Development Programme: Fast Simulation aims to combine full and fast simulation for different particles in one event while reducing computing time without unacceptable physics-accuracy loss.The framework is intended to remain flexible across simulation approaches.
  • Research and Development Programme: The roadmap targets optimised digitisation for HL-LHC and DUNE, alongside modular Geant4 components, refined physics models, and reproducible pseudorandom generation.A stated milestone is fully tested and validated optimised digitisation code by 2022; modularisation supports integrating only used components.

3.3 Software Trigger and Event Reconstruction

Future trigger and reconstruction systems must process more complex events at much higher rates while maintaining efficient, high-resolution reconstruction. The roadmap therefore emphasises heterogeneous computing, scalable software practices, analysis facilities, and sustainable support for the full event-processing ecosystem.

  • 3.3 Software Trigger and Event Reconstruction: ATLAS and CMS target hardware-trigger output up to 1 MHz, while LHCb and ALICE will stream typical proton-proton collision rates of 30–40 MHz to software triggers.Higher event complexity also increases the signals that software trigger algorithms must handle.
  • 3.3 Software Trigger and Event Reconstruction: Open-source tools and continuous integration already enable automated code-quality and performance checks, but scaling these checks to high statistics remains challenging.The analysis ecosystem also faces sustainability risks because maintenance is currently supplied by relatively few institutions.
  • 3.3 Software Trigger and Event Reconstruction: Tracking dominates reconstruction CPU needs at high pileup, while calorimetric reconstruction, particle flow, and particle identification also contribute significantly in some experiments.Reconstruction output is stored for analysis, with disk usage typically reaching tens to hundreds of petabytes per experiment.
  • 3.3 Software Trigger and Event Reconstruction: The identified R&D areas include vectorisation, architecture-specific algorithms, multithreading, and software capable of exploiting increasingly parallel and heterogeneous computing platforms.The programme links algorithm design and programming techniques to the hardware architectures on which they will run.
  • 3.3 Software Trigger and Event Reconstruction: The baseline analysis model reduces data successively to compact datasets, while new tools such as Jupyter and scikit-learn are being integrated into the established ecosystem.The intended outcome is flexible analysis with reduced time to insight and broader use of open-source components.

Current Practices

HEP analysis must reduce centrally managed datasets that are too large for local delivery while shortening the time to insight. The roadmap combines interoperable open-source tools, Python support, remote and query-based analysis facilities, declarative programming, preservation, and machine learning.

  • Current Practices: Run 2 analysis data can reach hundreds of terabytes, motivating successive reduction toward datasets small enough for low-latency access, potentially on a laptop.Retaining intermediate datasets is part of the proposed response to repeated analysis needs.
  • Current Practices: Python is prioritised for analysis because its simpler interfaces and extensive open-source ecosystem can reduce analysis-code complexity and time to insight.The roadmap seeks long-term maintenance and ROOT bindings with the ease of native Python modules.
  • Current Practices: Declarative analysis lets scientists specify intended data transformations while underlying services manage execution, including parallel processing.This model removes the need for analysts to define the event loop directly.
  • Research and Development Programme: Dedicated analysis facilities are intended to integrate relevant technologies and provide latency-optimised, stable environments for experimenters.Prototypes include fast caching, additional storage layers, cluster-management tools, and cloud or Big Data orchestration systems.
  • Research and Development Programme: The roadmap calls for dynamically pluggable open-source tools, data-exchange mechanisms, stronger Python support, functional or declarative analysis prototypes, and an Interpretation Gateway.The programme also includes analysis preservation, reinterpretation tools, and a blueprint for facility design and support.
  • 3.5 Machine Learning: Machine-learning R&D targets reconstruction, real-time classification, analysis-stage modelling, and significant data compression with negligible loss of physics utility.The programme also evaluates which ML tools and trade-offs are appropriate relative to traditional software.

Research and Development Roadmap and Goals

The roadmap identifies machine learning, data management, and new analysis paradigms as complementary R&D priorities for coping with HL-LHC data and computing demands. It targets improved efficiency and scalability across heterogeneous resources while preserving affordable scientific output.

  • Machine Learning: Orders-of-magnitude speedups in generative fast simulation are promising, but current approaches have not yet reached the required accuracy.GANs and VAEs are being explored as alternatives that learn high-dimensional feature distributions from existing samples.
  • Machine Learning: Machine learning R&D spans tracking, particle identification, simulation, anomaly detection, triggering, and resource optimisation.The programme also includes ML middleware, data formats, and ML-as-a-Service for HEP workflows.
  • Analysis and Computing: New analysis paradigms are needed because scientific reach depends on how efficiently data can be accessed and processed under changing technology and budget constraints.The roadmap frames these paradigms as having potential to enable new analysis methods and increase scientific output.
  • Data Management: HL-LHC data volumes and storage requirements are increasing faster than projected technology gains, making storage and analysis costs potential constraints on physics reach.Annual storage requirements are expected to increase by a factor close to 10, with storage remaining a major computing cost driver.
  • Data Management: The roadmap combines data organisation, management, and access to improve efficiency and scalability in the many-exabyte era.It considers heterogeneous storage, data placement, WAN access, and data-lake approaches alongside evolving analysis models.

Research and Development Programme

The programme proposes coordinated demonstrations and prototypes for affordable data storage, access, scheduling, and processing across heterogeneous HEP resources. It combines data-layout, caching, network, storage, and workload-management studies with common infrastructure planning.

  • Data Organisation and Access: The programme aims to demonstrate that HL-LHC data can be stored, accessed, and analysed affordably despite increased volume and complexity.It calls for proof-of-concept work followed by full implementations where successful.
  • Data Organisation and Access: Data-organisation studies will compare column-wise and row-wise layouts, Spark-like access, just-in-time decompression, and mappings across storage and memory hierarchies.These studies are intended to assess performance for different access patterns and hardware architectures.
  • Data Placement and Granularity: Data-placement R&D will quantify caching and placement benefits for reconstruction, analysis, and machine-learning workloads.The plan includes evaluating resource utilisation, catalogues, distribution, event granularity, and recoverability on volatile resources.
  • Global Optimisation: The roadmap examines tactical and archival storage, content delivery methods, data lakes, and opportunistic compute to reduce infrastructure costs and optimise access latency.Proofs of concept are planned for 2020, with broader implementations contingent on successful results.
  • Resources and Infrastructure: Scheduling and infrastructure must accommodate dynamic HPC, cloud, volunteer, GPU, and FPGA resources while accounting for regional funding differences.The roadmap also proposes quantitative system cost-performance modelling and software-defined-network integration.
  • Common Services: The roadmap proposes common data-management functionality, scalable heterogeneous workload scheduling, and a quasi-interactive analysis facility integrated with experiment systems.These actions target consolidation, finer-grained processing, and new physics-analysis models.

3.8 Data-Flow Processing Framework

Future HEP processing frameworks must support parallel execution across heterogeneous processors and integrate the needs of physicists, production managers, and facility operators. The programme moves from shared architectural concepts to common production-quality libraries.

  • Data-Flow Processing Framework: Future frameworks must parallelise reconstruction and simulation across multi-core, many-core, GPU, TPU, and tiered-memory systems.These resources must be integrated with storage and high-speed networks.
  • Data-Flow Processing Framework: Legacy implementations and differing stakeholder needs complicate common framework development across experiments.Existing frameworks share concepts but use historically different implementations and processing models.
  • Review and Architecture: By 2018, the roadmap targeted agreement on architectural concepts after reviewing concurrency libraries, functional programming, domain-specific languages, and experiment use cases.The review was intended to establish building blocks for next-generation frameworks.
  • Prototypes: By 2020, prototypes should demonstrate shared components, heterogeneous-resource scheduling, GPU use, data-model changes, scalable I/O, and workload-management integration.The prototypes were intended to inform HL-LHC Computing Technical Design Reports.
  • Production Libraries: By 2022, production-quality framework libraries usable by several experiment frameworks were expected, with at least one major paradigm shift anticipated over the five-year period.Cross-experiment workshops were planned to discuss the impact of evolving paradigms.
  • Conditions Data: Conditions data is non-event information required to simulate, digitise, or reconstruct detector events, and its modest volume still demands caching because many distributed jobs access it.Its volume is expected at terabyte scale with update rates typically O(1)Hz.
  • Conditions Data: Long-term maintainability remains a challenge, motivating consensus requirements and a common next-generation conditions database among experiments.Earlier Run 1 and Run 2 shortcomings contributed to complex implementations.

Current Practices

Current practice separates conditions-data payloads from metadata and increasingly tests online reconstruction, while visualisation remains fragmented across experiment-specific tools. The roadmap therefore favours maintainable common formats, client-server architectures, and community-driven visualisation packages.

  • Conditions Data: An Interval Of Validity specifies when a conditions-data payload applies, using a run number, luminosity section, or universal timestamp.A qualified conditions set combines payloads with their associated validity intervals for the workload’s required time span.
  • Conditions Data: Separating payloads from metadata permits independent caching, alternative backend technologies, and simpler preservation of small metadata exports.Payloads are accessed through unique hash identifiers, while tags and intervals select them.
  • Conditions Data: Online reconstruction is being tested by ALICE and LHCb for Run 3, potentially separating conditions-data distribution for reconstruction and analysis jobs.Running reconstruction in the HLT also increases pressure on the conditions-data infrastructure.
  • Conditions Data: Conditions-database R&D was planned for completion by 2020 to inform HL-LHC TDRs and enable Run 3 deployment addressing current limitations.Planned work included CVMFS filesystem views, industry technology evaluation, and REST-based implementations.
  • Visualisation: Visualisation faces duplication and maintenance problems because experiments use multiple specialised tools and formats.Common data formats and client-server architectures could broaden device support and reduce dependence on experiment frameworks.
  • Visualisation: Event displays combine event-based access, detector-geometry visualisation, and user interactivity.Applications may be fully integrated with experiment frameworks or lightweight cross-platform tools using simplified data.
  • Visualisation: The roadmap proposes exporters, interface packages, common formats, and community-driven tools for production visualisation beyond 2020.A client-server architecture would ease use of WebGL technologies and game engines.

3.11 Software Development, Deployment, Validation and Verification

The roadmap promotes coordinated, sustainable software development using common tools, robust validation, training, and technology choices that remain adaptable as platforms evolve. It proposes community mechanisms to track technologies, share practices, and improve portability and performance.

  • Software development and management: HEP software development must support geographically distributed contributors while maintaining code quality, reproducibility, and maintainability.The roadmap covers code organisation, documentation, issue tracking, artefact building, deployment, licensing, and validation.
  • People and practices: Training and documentation are essential because HEP software is produced by large, heterogeneous communities ranging from specialist developers to physicist programmers with limited formal training.The roadmap calls for accessible, experiment-neutral training and documentation that supports both newcomers and review by contributors.
  • Software development and management: Standard, non-HEP-specific tools and continuous integration can streamline contributions and automatically test code, quality, policy, and platform compatibility.The guidance favours widely documented open-source tools, git-based workflows, and testing before merge requests are accepted.
  • Technology strategy: Technology choices should remain generic and agile because the platforms offering the best value over the coming decade are uncertain.The roadmap recommends tracking external developments, adopting industry standards where possible, and using demonstrators with explicit success metrics.
  • Community coordination: A development forum and stronger links with external software communities would coordinate technology discussion, demonstrators, training, and knowledge exchange.The proposal highlights the Concurrency Forum model and engagement with communities such as SciPy, SCxx, RSE, and WSSSPE.
  • Technology strategy: Portable solutions for heterogeneous vector hardware and shared software metrics are proposed to improve performance, portability, and cross-community learning.The programme includes C++ refactoring tools, portable vectorisation prototypes, and common infrastructure for profiling and code metrics.

3.12 Data and Software Preservation

Data and software preservation aims to retain the data, knowledge, software, and workflows needed to reproduce and reuse HEP results. The roadmap proposes integrated preservation tools and prototypes, while noting that experiment-wide policies for the CERN Analysis Preservation Portal remain unsettled.

  • Motivation and scope: Preserving data and the knowledge behind scientific results is necessary to prevent the large investment in particle physics experiments from being lost.Knowledge preservation includes the analysis context and workflows needed to understand and reproduce results.
  • Motivation and scope: Preserved software and workflows can support collaboration, transfer knowledge to new analysts, automate submissions, and enable reuse beyond the originating experiment.External reuse includes outreach, education, and reinterpretation of published results in new contexts.
  • Motivation and scope: Preservation systems must accommodate different reuse levels and needs across central experiment workflows and individual analysts.The paper states that preserving analysis knowledge is difficult because analysts and experiments manage workflows in highly varied ways.
  • Current practices: LHC experiments have not yet established a formal policy determining whether use of CAP is required or merely encouraged.All experiments support some internal mechanisms for preserving knowledge surrounding physics publications.
  • Current practices: Existing preservation activity includes studies of common services for bit preservation, executable environments, and structured analysis metadata, consistent with FAIR data principles.LHC experiments also provide selected public datasets, with CMS releasing substantial AOD-format datasets for new analyses.
  • R&D programme: The roadmap calls for an analysis ecosystem that captures preservation metadata and supports reuse, alongside preservation and validation of large-scale production workflows.Proposed activities include a CAP demonstration, container-based workflow execution, use-case collection, sandbox evaluation, and executable preservation prototypes.

Current Activities

Current security activities address trust and policy, operational security, authentication and authorisation, and technology evolution through collaboration across HEP and the wider research-and-education community. The planned work updates infrastructure, strengthens incident response, and supports broader identity and interoperability needs.

  • Security priorities: HEP security efforts are organised around trust and policies, operational security, authentication and authorisation, and technology evolution.These areas are being pursued by groups that extend beyond the HEP community.
  • Trust and policies: The roadmap proposes policies aligned with EU data protection and frameworks for trustworthy interoperability as infrastructure incorporates diverse providers and cloud gateways.The existing X.509-focused trust model must adapt to modern data-exchange designs, including commercial providers and hybrid clouds.
  • Operational security: Community-driven incident response policies, security operations guidance, and shared threat intelligence are proposed to address increasingly sophisticated attacks.The paper calls for a global Research and Education forum and reference guidance for Security Operation Centre deployment.
  • Operational security: Security coordination should expand participation and streamline communication through maintained contacts, secure channels, and incident-response mechanisms.The proposed capabilities include communication workflows within and outside HEP.
  • Operational security: HEP security teams should integrate more closely with home-organisation and site-security teams.The roadmap identifies this integration as necessary for adequate coordination between HEP and institutional security capabilities.
  • Authentication and authorisation: Authentication and authorisation infrastructure should be overhauled to support token translation, identity-provider proxies, membership tools, and a wider range of user identities.The roadmap also supports work through FIM4R and related research authentication and authorisation initiatives.

4 Training and Careers

Training, career recognition, and community coordination are essential to building the software expertise HEP needs. The roadmap proposes broad training, modern collaborative methods, and career support for software specialists.

  • Career support: The HSF treats developer skills, successful careers, and community-wide software expertise as essential goals.
  • Training: The roadmap proposes permanent training infrastructure, software schools, and new active and peer-training approaches.
  • Career support: Career support includes job opportunities, advancement paths, and recognition of software development and training as scientific research.
  • Training: Training should serve everyone from novice programmers to advanced developers across the full data-processing chain.
  • Training: Training should become an integral part of major R&D projects, supported by dedicated materials and collaborative platforms.
  • Career support: Recognition remains difficult because institutional promotion practices often undervalue computing expertise and training contributions.

5 Conclusions

The roadmap argues that HL-LHC software and computing require a step change rather than incremental scaling. It prioritises coordinated, sustainable R&D spanning processing, data, analysis, development practices, and people.

  • Conclusions: HL-LHC computing needs exceed incremental code changes and facility scaling because data volumes and hardware diversity are increasing under constrained budgets.
  • Conclusions: The community calls for greater investment in skilled people, recognised career paths, and training across HEP software and computing.
  • Conclusions: Sustainable software choices should be evaluated for deployment consequences, cost effectiveness, and coordination with distributed computing partners.
  • Conclusions: Highest-priority work includes simulation, reconstruction, supporting frameworks, data formats, event content, and distributed data and workload management.
  • Conclusions: Machine learning, parallelised data processing, modern development techniques, common tools, training, and data preservation are identified as important future directions.
  • Conclusions: R&D proposals should be evaluated against the roadmap’s charges while retaining agility for disruptive changes that cannot be planned.
  • Conclusions: The HSF enabled the Community White Paper by helping overcome institutional, regional, and experimental subdivisions.

A List of Workshops

The Community White Paper process used a sequence of topical and general workshops across Europe and the United States. These meetings developed working-group proposals and community consensus for the roadmap.

  • Workshops: The process began with a January 2017 HSF workshop at SDSC/UCSD covering general and topical working-group sessions.
  • Topical workshops: Working-group meetings addressed software triggers, event reconstruction, machine learning, intensity-frontier computing, and visualisation.
  • Topical workshops: Further meetings covered machine learning, analysis tools and ROOT, event-processing frameworks, and the final general CWP workshop.

B Glossary

The glossary defines the paper’s principal facilities, software projects, computing concepts, organisations, networks, and data-analysis terms. Together, these entries establish the terminology used across the roadmap.

  • Data and analysis: AOD is a summary of a reconstructed event containing sufficient information for common physics analyses.
  • Organisations and programmes: CWP is the HSF-organised community strategy and roadmap for HEP software and computing R&D in the 2020s.
  • Software and simulation: Geant4 simulates particle passage through matter, while GeantV is an R&D project exploiting new CPU parallelism for detector simulation.
  • Facilities: HL-LHC is a proposed 2026 LHC upgrade targeting tenfold higher luminosity and improved sensitivity to rare or statistically marginal measurements.
  • Organisations and programmes: The HSF facilitates international coordination and common efforts in HEP software and computing.
  • Analysis tools: Jupyter notebooks combine executable code with human-readable analysis descriptions, equations, figures, and results.
  • Computing concepts: Machine learning enables computer-based prediction without explicit programming and includes methods such as boosted decision trees and neural networks.

The HEP Software Foundation

The supplied section materials identify a large collaboration involving numerous authors and institutions across high-energy physics laboratories, universities, and computing centers. The affiliations span Europe, North America, South America, Asia, Africa, and Australia.

  • The HEP Software Foundation: Participating institutions include major laboratories and universities across Europe and North America.Examples include CERN, Brookhaven National Laboratory, Fermilab, MIT, and multiple European universities and research institutes.
  • The HEP Software Foundation: The affiliations also include institutions in South America, Asia, Africa, and Australia.Examples include the Centro Brasileiro de Pesquisas Físicas, Chinese Academy of Sciences, National Taiwan University, University of Cape Town, and the University of Melbourne-affiliated listing is not present in the supplied passages.
  • The HEP Software Foundation: The collaboration includes dedicated computing organizations alongside physics departments and laboratories.Named examples include CNAF, the Center for High Performance Computing, the Port d’Informació Científica, and the Center for High Throughput Computing.
Loading 1712.06982v5…