A Roadmap for HEP Software and Computing R&D for the 2020s

• The programme targets Higgs precision and BSM searches, heavy-flavour physics and matter-antimatter differences, dark matter, neutrinos, and quark-gluon plasma.HL-LHC experiments ATLAS and CMS pursue Higgs-sector measurements and BSM searches, while LHCb and Belle II study heavy-flavour physics; ALICE, CBM and PANDA probe quark-gluon plasma.Source: §1 · p. 3
• The experimental programme requires commensurate investment in software R&D to acquire, manage, process and analyse data, alongside investments in detector hardware.Source: §1 · p. 5
• The HL-LHC’s sharply increasing data volume and complexity require a software upgrade alongside hardware upgrades, beyond simply scaling current solutions.ATLAS and CMS are expected to collect about 30 times more data than the LHC has produced to date, against a backdrop of evolving computing hardware and more sophisticated analyses.Source: §1 · p. 5
• The CWP roadmap prioritizes software efficiency, scalability and performance; new approaches that extend physics reach; sustainability and preservation; and expertise, recognition and training.The HSF planning exercise was intended to coordinate stakeholder goals and priorities for the HL-LHC and other HEP experiments on a similar timescale.Source: §1 · pp. 5–6
• The CWP process involved an estimated 250 participants and engaged LHC experiments, other HEP experiments, national computing organisations and the wider software and computing community.It ran from a January 2017 kick-off workshop to a final workshop in June 2017, with intermediate topical workshops; working groups covered careers, data, computing, software and physics topics.Source: §1 · p. 6
• The roadmap may benefit technically similar projects, particularly astrophysics surveys and observatories including SKA, CTA and LSST.Source: §1 · p. 6

2 Software and Computing Challenges

• Much experiment-specific software is poorly maintained, ill-documented, and untested, despite supporting data flow from acquisition and triggering through reconstruction and analysis.The codebase spans tens of millions of lines, was largely built for x86 64 and serial processing, and has lost many original authors.Source: §2 · p. 7
• Run 3 and HL-LHC will sharply increase processing demands, with LHCb handling over 40 times today’s collisions, ALICE reading Pb-Pb at 50 kHz, and pileup reaching about 200.HL-LHC is anticipated to deliver about 300 fb-1 annually, while ATLAS and CMS plan to record 5–10 times as many events as today.Source: §2 · p. 8
• Future data collection and processing—and HL-LHC physics reach—will be limited by affordable computing and how efficiently resources are used.The resource increase is estimated by extrapolating the Run 2 computing model.Source: §2 · pp. 8–9
• By 2027, under a flat funding profile, projected ATLAS needs exceed available resources by a factor 4 in CPU and a factor 7 in disk.The estimate accounts for technology improvements following current trends.Source: Figure 4 · p. 9, §2 · pp. 9–10
• Stalled single-core progress, power constraints, and alternative processors make concurrent programming and support for low-power and heterogeneous architectures increasingly important.Potential architectures include many-core CPUs, GPUs, AArch64 devices, and possibly FPGAs; GPU programming brings generation-specific programming and memory-handling challenges.Source: §2 · p. 10
• HEP must modernize serial code with concurrent frameworks, parallel algorithms, and machine learning, while flexibly using HPC/cloud resources and hierarchical distributed data.Efficient network use requires closer application integration and software for distributed data and workload management, software distribution, and data access.Source: §2 · pp. 10–11
• Sustainable progress requires investment, training, documentation, software leadership, and shared community solutions beyond experiment-specific development.The paper also points to potential links with astronomy, genomics, computing science, and Big Data; the HSF promotes common projects.Source: §2 · p. 11

3.1 Physics Generators

• Higher-precision predictions make computationally demanding higher-order event generators increasingly necessary, with integration challenges spanning 10–15 dimensions, negative weights, cancellations, memory use, and initialization time.NNLO generation is expected to be more widely required by the HL-LHC, but computational cost has limited its use.Source: Scope and Challenges · pp. 12–13

Current Practice

• NLO generation can sharply increase CPU needs, while negative weights require many more events than LO samples and strain budgets that allow similar simulation volumes to real data.Current NLO use is limited until schemes reduce or avoid negative weights.Source: Current Practice · p. 13

Research and Development Programme

• Sustained generator and shared-library improvements require technical support, experienced optimisation specialists, stronger theory–experiment collaboration, and recognition of external software contributions.The passages call for medium- to long-term placements with generator projects and shared investment in tools such as LHAPDF, HepMC, and Rivet.Source: Research and Development Programme · p. 14, Current Practice · p. 14
• Generator R&D prioritizes improved theoretical algorithms and reweighting, which can reduce BSM sample generation time but requires phase-space coverage and solutions to NLO and negative-weight issues.Long-term support and career opportunities are considered critical; reweighting requires the updated model’s phase space to be a subset of the original and can require extra event-file information at NLO.Source: Research and Development Programme · p. 15
• Concurrency and HPC work should address integrators, phase-space factorization, thread-hostile code, scaling, initialization overhead, and memory use for high-multiplicity tree-level generation.Parallelism added after the fact can scale poorly on HPC machines; factorizing multi-particle phase-space integration into lower-dimensional integrals is proposed as a way to parallelize.Source: Research and Development Programme · pp. 15–16
• Generator-side filtering could reduce wasted CPU by directing one inclusive event generation into several orthogonal output streams rather than regenerating samples for each stream.The proposal targets workflows that filter large inclusive samples before detector simulation and reconstruction.Source: Research and Development Programme · p. 16

3.2 Detector Simulation

• Simulation must meet demand for much larger samples, diverse physics programmes, and complex future detectors while respecting computing budgets; Fast Simulation may improve runtime if physics accuracy remains acceptable.HL-LHC experiments expect to collect 150 times more data than in Run 1, and sustained specialist expertise is needed to develop and maintain simulation codes.Source: Scope and Challenges · pp. 16–17
• Detector-simulation R&D spans physics-model review, vector and GPU optimization, Fast Simulation, shared geometry tools, background modeling, and digitization improvements and code sharing.The model review may improve precision and runtime; architecture work includes SIMD vectorisation and HPC applications, while Fast Simulation work includes common tuning and validation frameworks.Source: Scope and Challenges · p. 17

Current Practices

• Geant4 remains essential for detector simulation, with continuing maintenance and physics improvements; electromagnetic work includes model review and a Goudsmit-Saunderson model reported to outperform current models in accuracy and speed.The roadmap also prioritizes reviewing electromagnetic models, including hadron and ion models, for HL-LHC and FCC studies, while Geant4 is maintained until alternative engines are integrated and validated.Source: Current Practices · pp. 17–20
• FTFP BERT and QGSP BERT provide alternative hadronic model combinations for systematic studies, while preliminary results indicate Geant4 shower lateral profiles are too narrow.CALICE calorimeter test beams and LHC data inform validation; intermediate- and low-energy models and further QGS improvements are being pursued.Source: Current Practices · p. 18
• Intensity Frontier experiments need accurate neutrino energy-deposition and rare-background simulation at scale, while detector-specific digitisation offers parallelism and opportunities for shared code.Geant4 is the primary simulation toolkit for these experiments; shared efforts include LArTPC digitisation and silicon radiation-damage modeling.Source: Current Practices · p. 19

Research and Development Programme

• GeantV groups similar tracks across events for SIMD transport, with vectorised geometry, navigation, and physics; experiment applications were slated to integrate and validate these capabilities toward HL-LHC production.VecGeom was integrated into Geant4, and a proof-of-concept transport engine was planned for 2018; experiment milestones include multithreaded running and beta-engine integration in 2020, with a possible production engine by 2026.Source: Research and Development Programme · pp. 20–22
• Geant4 modularisation aims to simplify framework integration and component reuse, alongside physics-model extensions that improve coverage, accuracy, and CPU and memory efficiency.Planned model work includes improved hadronic cascades and electromagnetic interactions, optical-photon transport, and an EPOS string model for multi-GeV to multi-TeV interactions.Source: Research and Development Programme · pp. 21–22
• Pileup R&D addresses storage and I/O burdens by studying real zero-bias overlays and improving pre-mixing, which can combine minimum-bias events at digitisation level.Zero-bias overlay must contend with data availability, nonlinear signal effects, and alignment consistency; pre-mixing reduces disk and network use per event.Source: Research and Development Programme · pp. 22–23
• Fast Simulation aims to combine full and fast simulation within events while saving computing time without unacceptable physics loss; GANs offer a route to correlated, non-Gaussian shower fluctuations.The programme includes assessing machine-learning approaches, developing ML-based simulation for some observables, and demonstrating common infrastructure across detector configurations.Source: Research and Development Programme · p. 23
• The programme seeks flexible, high-performance shared digitisation examples and validated optimised code, while developing sequential and vectorised PRNG implementations to replace existing ROOT and CLHEP libraries.The PRNG effort responds to reproducibility challenges on highly parallel infrastructures and targets transition to a common library in ROOT and Geant4.Source: Research and Development Programme · pp. 23–24

3.3 Software Trigger and Event Reconstruction — Scope and Challenges

• Without major changes to reconstruction approaches or algorithms, HL-LHC resource needs would exceed expected budgets as event complexity and rates rise.The challenge includes maintaining reconstruction efficiency and resolution at high pileup, particularly for low-pT objects.Source: Scope and Challenges · p. 24
• ATLAS and CMS target hardware-trigger outputs up to 1 MHz, while LHCb and ALICE plan to stream 30–40 MHz to software triggers.Real-time analysis is driven by the need to examine more signal than can be written for traditional processing, without reducing sensitivity or introducing bias.Source: Scope and Challenges · pp. 24–25

3.3 Software Trigger and Event Reconstruction — Current Practices

• Experiments use substantial online and offline computing facilities, with tracking often dominating reconstruction CPU at high pileup and tiered data formats supporting event skimming and analysis.CPU use is typically tens or hundreds of thousands of x86 64 cores, while disk storage is tens to hundreds of PB per experiment; automated calibration and alignment systems support data taking and reconstruction.Source: Current Practices · pp. 25–26

3.3 Software Trigger and Event Reconstruction — Research and Development Programme

• R&D targets SIMD vectorisation, multithreaded throughput, and heterogeneous architectures, with benchmarks to compare physics-per-dollar-perwatt and computer-science collaboration to guide algorithm choices.GPU and FPGA deployments also require production demonstrations and assessment of the costs of maintaining multiple implementations.Source: Scope and Challenges · p. 25, Research and Development Programme · pp. 26–27
• Reconstruction R&D must scale algorithms to high event complexity, develop fast techniques for granular and timed detectors, and improve throughput for targets such as charged-particle tracking.High pileup at HL-LHC or FCC-hh may require new algorithms, including advanced machine learning, to maintain or improve physics measurements and object identification.Source: Scope and Challenges · p. 24, Research and Development Programme · pp. 26, 28
• R&D must extend automated statistical, technical, and physics-performance QA/QC, while real-time analysis saves signal-associated event content and enables offline analysis chains to run in real time.Research includes compression and custom data formats, calibration and validation tools, and frameworks for analysts to deploy such analyses without compromising data-taking quality.Source: Research and Development Programme · p. 27

3.4 Data Analysis and Interpretation — Scope and Challenges

• Analysis workflows must retain rapid, flexible iteration on compact datasets while adapting to larger samples, and enable new open-source tools to interoperate with the ROOT-based ecosystem.R&D also covers file formats, compression, data access, and workflows suited to future computing infrastructures.Source: Scope and Challenges · pp. 28–29
• Analysis systems must support reproducibility, provenance tracking, data and software preservation, and reinterpretation across theoretical models, while sustaining training and a maintainable tool ecosystem.The paper notes that ecosystem maintenance is concentrated in a few institutions and calls for policies to retire obsolete components and attract contributors to new tools.Source: Scope and Challenges · pp. 29–30

Current Practices

• LHC analyses typically reduce centrally managed event data into flat ntuples and histograms, while laptop-sized datasets, nearby intermediates, optimized formats, and analysis trains address access and resource constraints.The current workflow has produced more than 2,000 publications during Runs 1 and 2; analysis trains reduce repeated reads by running analyses together in shared batch jobs.Source: Current Practices · p. 30
• The R&D programme calls for Python to become a sustainably supported first-class language and for open-source tools to interoperate with the existing analysis ecosystem.It includes evolving ROOT bindings toward native-Python ease of use, dynamic component exchange, and an Interpretation Gateway for repositories such as HEPData [85] and preservation and reinterpretation tools.Source: Current Practices · pp. 31–32
• Apache Spark exemplifies cluster-based analysis that combines sequential data reduction with interactivity and batch processing, while remote querying can produce analysis outputs without a reduction campaign.These approaches may simplify data access and parallelism or provide checkpoints through transient data; their role and resource optimization remain to be evaluated.Source: Current Practices · p. 31
• Functional or declarative analysis lets scientists state desired data transformations while underlying systems manage event iteration and optimize diverse resources.The paper calls for continued R&D, including ROOT’s TDataFrame, and a prototype programming paradigm.Source: Current Practices · pp. 31–32
• HL-LHC analysis facilities are envisioned as stable, latency-optimized environments integrating HEP technologies, with prototypes and evaluations planned for fast caching, storage layers, and cloud or Big Data orchestration.The programme also calls for Run 3 scale testing and a blueprint for remaining facility development, system design, and support.Source: Current Practices · p. 32

3.5 Machine Learning

• HEP ML R&D aims to adapt external toolkits to domain-specific applications while identifying problems that require new formulations and sharing expertise, lessons, and training across teams.Teams are expected to help others develop and deploy experiment-specific algorithms and use established ML tools.Source: §3.5 · p. 33, Scope and Challenges · p. 33

Scope and Challenges

• ML applications could speed reconstruction, extend real-time analysis and physics reach, and compress data, with GPUs and dedicated processors enabling new approaches.The proposed scope includes replacing computationally expensive pattern-recognition and parameter-extraction steps, including charged-track and vertex reconstruction, and expanding trigger applications as HL-LHC collision rates rise.Source: Scope and Challenges · p. 34

Current Practices

• HEP analysis commonly uses TMVA for signal/background classification, while analysts are beginning to adopt scikit-learn [74] and Keras; collaboration-specific ML work has also prompted cross-experiment coordination.Examples include ML in reconstruction and identification, and the Inter-experiment Machine Learning forum has fostered closer collaboration between LHC and non-LHC groups.Source: Current Practices · pp. 34–35

Research and Development Roadmap and Goals

• The roadmap targets image-based particle identification and particle-property extraction, alongside track-recognition approaches that could scale linearly with LHC intensity.For LArTPCs, it calls for a proof of concept and comparison of deep-learning architectures by 2020; it also identifies jet-flavour tagging and many-core track-recognition efforts.Source: Research and Development Roadmap and… · pp. 35, 37
• The roadmap calls for evaluating ML-compatible data formats and bridging strategies, network-aware ML optimisation, and tailored hardware and instances to support ML-as-a-Service adoption in HEP.Desired format attributes include fast reads and writes, sparse readability, compressibility, and broad ML-community adoption; network uses include congestion prediction and WAN path optimisation.Source: Research and Development Roadmap and… · p. 36
• The roadmap also targets operational and analysis bottlenecks through anomaly detection, generative fast simulation, real-time ML, and neural-network integration of the Matrix Element Method.Unsupervised anomaly detection could alert on deviations across many variables to support pre-emptive maintenance; an initial generative-simulation attempt was orders of magnitude faster but had not reached required accuracy. Trigger work must balance complexity with inference-time constraints, while DNNs are candidates for computationally intensive MEM integration.Source: Research and Development Roadmap and… · pp. 36–37

3.6 Data Organisation, Management and Access

• Data-intensive science depends on how quickly data can be accessed and processed, motivating budget-compatible computing models that integrate emerging analysis paradigms as data volumes and computing technologies change.The paper says these changes could enable new analysis methods and increase scientific output.Source: §3.6 · p. 37

Scope and Challenges

• HL-LHC annual storage requirements are expected to rise by close to a factor of 10, outpacing projected technology gains, while storage costs rival compute costs and may constrain physics reach.The LHC experiments already provision about an exabyte of storage, approximately half archival and half disk.Source: Scope and Challenges · pp. 37–38
• High-speed WANs, streaming, and dynamic cloud or HPC resources motivate data models for federated access and opportunistic computing, while ML and query workloads alter where data must be accessed and produced.Storage and data management must also match growing event complexity and reconstruction needs so they do not impede improvements in computing work.Source: Scope and Challenges · pp. 38–39
• Storage R&D must accommodate heterogeneous and tactical resources, assess lower-latency archive alternatives and reliability, and pursue shared solutions without sacrificing data access or fit to varied use cases.The paper also calls for collaboration beyond HEP, including with SKA and CTA, during their formative design phases.Source: Scope and Challenges · p. 39
• DOMA treats data organisation, management and access together rather than independently, creating opportunities for global optimisation, efficiency and scalability in the many-exabyte era.The paper notes that mature LHC computing has increasingly pursued optimisations spanning previously separate data areas.Source: Scope and Challenges · pp. 40–41

Research and Development Programme

• Event-level sub-file granularity should be tested against file-based access for performance, storage and network use, and recoverability under preemption on cloud spot or dynamic HPC resources.The planned access patterns include simulation, reconstruction, and analysis.Source: Research and Development Programme · p. 41
• A proof of concept should evaluate column- versus row-wise data organisation, MapReduce- or Spark-like access, and just-in-time decompression mapped to hardware data flows.The programme targets completion by 2020 to enable full implementations in subsequent years.Source: Research and Development Programme · p. 41
• Placement optimisation, including caching, should be assessed for reconstruction and machine-learning workloads, alongside varied-quality storage, archival solutions, and CDN or NDN delivery.Proofs of concept for varied storage quality are targeted by 2020, while CDN and NDN approaches are proposed for longer-term study.Source: Research and Development Programme · pp. 41–42

Scope and Challenges

• Naive HL-LHC scaling would require almost an order of magnitude more resources than technology evolution alone can provide, amid competing science needs and regional funding differences.The challenge is to find a feasible configuration of facilities and computing sites.Source: Scope and Challenges · pp. 42–43
• Dynamic HPC, volunteer, and cloud resources and diverse CPUs, GPUs, and FPGAs must be integrated with reliable storage and networks as computing models become more network-centric.Research network capacity trends show growth of two orders of magnitude by HL-LHC times, compared with expected annual cost-adjusted increases of 15% for CPU and 25% for disk.Source: Scope and Challenges · p. 43
• An evolutionary migration model should assess proposed changes’ hardware and human costs and software, workload, and physics impacts, with potential value for other data-intensive sciences.The model is needed because experiments will continue using the current system during the transition to HL-LHC.Source: Scope and Challenges · pp. 43–44

Current Practices

• Current-practice work will quantify WLCG performance and costs, define a federated data-centre or data-lake concept, and establish how SDNs can manage data transfers and access.The data-lake study targets lower storage operating costs and network management while maintaining CPU efficiency.Source: Current Practices · p. 45
• Current-practice efforts will consolidate common data-management functionality, develop scalable scheduling for heterogeneous resources and finer-grained processing, and prototype quasi-interactive analysis integrated with experiment systems.The data-management work recognises the connection between storage-system elements and metadata.Source: Current Practices · pp. 45–46

3.8 Data-Flow Processing Framework — Scope and Challenges

• Frameworks must support parallel processing across heterogeneous processors and tiered memory, integrate with workflow and data management, and accommodate software with its own parallelization model.Common development is desirable but constrained by legacy work, stakeholder needs, and security requirements.Source: Scope and Challenges · p. 46
• HEP experiments use distinct frameworks, including Gaudi, CMSSW, art, basf2, Marlin, FairROOT, AliROOT, and the developing O2; most support task-based parallelization [115].Processing models use threads, tasks, processes, or inter-process communication, trading off programming constraints, execution efficiency, resource adaptation, throughput, and latency.Source: Scope and Challenges · p. 47
• Frameworks have largely run as independent single-core batch jobs, while CMS uses threading and ATLAS uses multi-process fork-and-copy-on-write; future scaling may span nodes, an approach O2 is pioneering with microservices or function-as-a-service.The CMS and ATLAS changes address rising memory needs per process; changing hardware makes the interaction between workflow and workload management and framework applications important to examine.Source: Scope and Challenges · pp. 47–48

3.8 Data-Flow Processing Framework — Research and Development programme

• By 2018, review concurrency libraries, integration of generator and simulation models, functional or domain-specific descriptions, and differences among frameworks and experiment use cases; by 2020, prototype shared architectures for HL-LHC TDRs.The prototypes should demonstrate heterogeneous scheduling and co-processor use, data-model changes for vectorization and concurrent I/O, and workload-management integration with storage, network, and HPC resources; migration paths toward shared implementations are also part of the work.Source: Research and Development programme · p. 48
• By 2022, deliver production-quality framework libraries usable by several experiments, covering demonstrated areas and continuing cross-experiment discussion of an anticipated major paradigm shift.The roadmap expects at least one major paradigm shift within the five-year period.Source: Research and Development programme · p. 49

3.9 Conditions Data — Scope and Challenges

• Conditions data is terabyte-scale with typically O(1)Hz updates, yet distributed jobs can generate O(10) kHz reads, requiring caching and efficient mapping of data and metadata to files.CVMFS offers embedded caching, snapshotting, and branching; inefficient caching can affect reconstruction, especially when Run 4 conditions data is linked to luminosity at minute-level granularity.Source: Scope and Challenges · p. 49
• A more maintainable conditions-data design uses a content-agnostic server with client-side intelligence, REST interfaces, and multiple proxies or servers for resilient access.ATLAS and CMS are working on a common next-generation database, while Belle II has developed a solution based on the same concepts and architecture.Source: Scope and Challenges · p. 50
• Event-level work allocation can reduce opportunities to optimize conditions-data access at job level and increase infrastructure pressure, but its impact remains uncertain.The event service is being used by ATLAS to use HPC resources for simulation or processing; further experience is needed to assess its conditions-data effects.Source: Scope and Challenges · p. 50

Current Practices

• Conditions data use payloads indexed by validity intervals and tags, with global tags resolving each subsystem and interval to one payload through a unique hash key.A fully qualified set covers the workload’s required time span; the interval may be a run number, luminosity-section ID, or universal timestamp.Source: Current Practices · p. 50

Research and Development Programme

• The conditions R&D programme targets completion by 2020, covering CVMFS-based file-system views, evaluation of industry technologies, and REST-based approaches for ATLAS and ALICE.ATLAS plans to migrate from COOL and study client-side complexity; ALICE plans to explore REST as an alternative or complement to CVMFS.Source: Research and Development Programme · p. 51

Scope and Challenges

• Event displays trade detailed framework integration against lightweight portability: integrated applications access experiment data but are complex, while generic-format applications can lose detail and require extra processing.The choice of graphical backend also affects which devices can be supported.Source: Scope and Challenges · p. 52
• Visualization challenges include sustaining experiment-specific tools, defining common data formats, broadening device support through client-server architecture, and choosing 3D-rendering libraries suited to required output and interaction.For non-event displays, open-source and industry-standard tools can provide generic solutions.Source: Scope and Challenges · p. 52

Current Practices

• HEP event displays typically provide event-based data access, detector-geometry visualization, and user interaction for selecting data and setting object-property cuts.Geometry may come from experiment frameworks or use a simplified description for speed or portability.Source: Current Practices · pp. 52–53

Research and Development Roadmap

• Visualization R&D will bridge experiment frameworks to community-standard packages and formats, making applications and event displays less dependent on experiment-specific software.Exporters and interface packages are intended to retain access to detailed experiment data while connecting it to common tools.Source: Research and Development Roadmap · p. 54
• The visualization roadmap includes demonstrators and a client-server interface for geometry and event data, potentially using an API or service to stream events.The proposed interface is being developed with the Data Access and Management Working Group.Source: Research and Development Roadmap · p. 54
• After the visualization design work is completed by 2020, the focus will shift to production-ready community tools, supported by annual workshops for knowledge exchange and collaboration.The tools may take advantage of new data-access services.Source: Research and Development Roadmap · p. 54

Scope and Challenges

• For large, distributed collaborations, streamlined development processes should enable contribution and collaboration while maintaining code quality, reproducibility, and maintainability with minimal effort.Software development and project management are continuous efforts that share practices with non-HEP software industries.Source: Scope and Challenges · pp. 54–55
• The development scope spans distributed workflows, software quality and sustainability, deployment with operations, small- and large-scale validation, and licensing that supports interoperability.Source: Scope and Challenges · p. 55
• HEP’s diverse experiments, funding, expertise, and resources complicate shared software practices, while centrally managed production and fast-moving user analysis have markedly different lifecycles.Solutions may not adapt well to both use cases.Source: Scope and Challenges · p. 55
• Practical development should use supported platforms, established tools such as Git and CMake, continuous integration, automated validation and deployment, and repository-based documentation.Git and social coding sites support code sharing and review; continuous integration can run tests and quality or policy checks at low human cost.Source: Scope and Challenges · p. 56

Research and Development Programme

• The R&D vision is to keep HEP current with wider open-source technologies through a technology forum, accessible developer training, and demonstrators evaluated against clear success metrics.The roadmap also calls for generic technology choices and ongoing review so the community can adapt to uncertain future platforms.Source: Research and Development Programme · p. 57
• Short-term projects include a common software forum, continued packaging prototypes, and practical advice and templates for setting up packages.The forum is modeled on the Concurrency Forum, which helped establish demonstrators and prototypes for parallel data processing frameworks.Source: Research and Development Programme · p. 57
• The agenda also calls for experiment-neutral core-skills training, stronger links beyond HEP, case studies of software successes and failures, and stronger package publication and citation records,.Source: Research and Development Programme · p. 58
• Longer-term projects target C++ refactoring, portable vector hardware, industry standards, performance instrumentation, shared software-data infrastructure, and statistical tools for regression testing.Performance work should build on existing tools such as igprof where possible.Source: Research and Development Programme · p. 58

3.12 Data and Software Preservation — Scope and Challenges

• Preserving data and analysis knowledge can enable new analyses, revisit published results, support internal collaboration, and provide reuse for education and external research.Reanalysis can help resolve conflicts between results or evaluate new theoretical models; accessible data and software can also support outreach, phenomenology, and machine-learning engagement.Source: Scope and Challenges · p. 59

3.12 Data and Software Preservation — Current Practices

• LHC experiments offer differing data releases, from CMS AOD and ALICE data to LHCb reconstructed or ntuple data and ATLAS publication and reinterpretation material, while formal CAP requirements remain unsettled.All experiments support some public access and internal preservation mechanisms for knowledge surrounding physics publications.Source: Current Practices · p. 60
• Preservation R&D calls for prototype ecosystems with embedded capture tools and a CAP interface, guided by reuse use cases and evaluation of sandbox and freezing technologies.The roadmap also calls for coordination with other working groups and tracking analysis evolution toward Big Data environments.Source: Current Practices · p. 61
• The roadmap also targets multi-container workflow execution, preservation and validation of production executables, integration into new tools, and standardized reusable records through HEPData, Rivet, or other tools.The proposed scheme could preserve sufficiently detailed records of many LHC Run 2 research outputs.Source: Current Practices · p. 61
• The 2020–2022 preservation goal is an analysis ecosystem enabling reuse and systems that preserve and validate large-scale production workflows.The proposed deployed solutions support any analysis conductible in the ecosystem.Source: Current Practices · p. 61

3.13 Security — Scope and Challenges

• HEP must adapt its X.509-dependent trust architecture for diverse cloud and HPC partners, protect personal-data exchange, and support federated authentication while retaining community control of authorisation.The proposed direction includes reducing personal-data processing, using standards to ease integration, and mapping federated identities and grid attributes to external services.Source: Scope and Challenges · pp. 62–64
• Because sophisticated attacks are costly to counter alone, HEP should share threat intelligence and incident response with external organisations and build a global R&E security forum.The paper says HEP is well positioned to contribute because of its security operations and global network, while closer ties between HEP and campus security are needed.Source: Scope and Challenges · pp. 62–63
• Security planning must address emerging attack vectors such as IPv6 and IoT, and clarify trust, vulnerability-management, and incident-response responsibilities across public, private, and hybrid clouds.The paper notes possible incompatibilities between community incident response and commercial providers’ “pay-for-what-you-break” model.Source: Scope and Challenges · p. 63

Current Activities

• Current work spans shared trust policies, operational security coordination, and WLCG evaluation of authentication and authorisation.WISE offers a forum for sharing security practices; attribute flows remain problematic, while REFEDS and AARC address identity assurance and federation security capability through Sirtfi.Source: Current Activities · pp. 64–65

Research and Development Programme

• Groups working across trust, operational security, authentication and authorisation, and technology evolution are driving security R&D for the coming decade.These groups generally extend beyond the HEP community.Source: Research and Development Programme · p. 65

Trust and Policies

• Define policies aligned with new EU data-protection requirements, develop trustworthy interoperability frameworks, and promote community-driven incident-response policies.Source: Trust and Policies · p. 65

Operational Security and threat intelligence

• Provide SOC deployment guidance for HEP sites and help establish a global Research and Education incident-response forum.The programme also calls for taking action based on shared threat intelligence; it describes global-community response as the only effective solution to global security threats.Source: Operational Security and threat… · pp. 65–66
• Expand participating-organisation capacity and communication workflows, integrate HEP and home-site security teams, and prepare defendable services against sophisticated criminal and state threats.Source: Operational Security and threat… · p. 66

Authentication and Authorisation

• Raise trust in non-X509 federated identities to X.509-equivalent confidence and help set future identity-federation directions through FIM4R.At that confidence level, non-X509 identities could be alternatives to grid certificates and CERN accounts.Source: Authentication and Authorisation · p. 66
• Overhaul authentication and authorisation infrastructure with Token Translation, Community IdP-SP Proxies, and membership-management tools to support more user identities in WLCG services.Source: Authentication and Authorisation · p. 66

4 Training and Careers

• The HSF’s mission calls for developer training, career paths with advancement opportunities, and recognition of software development as scientific research on equal footing with other activities.It includes support for schools and permanent training infrastructure, job-opportunity listings, and advocacy for researchers who specialize in software.Source: §4 · p. 67

4.1 Training Challenges

• New technologies require HEP specialists and broad community adoption, while the close user–developer relationship and reciprocal understanding of physics and software remain important.The community must implement field-specific solutions because some advanced requirements are unique to HEP.Source: §4.1 · p. 67
• Training must span novice to advanced audiences and data-processing tasks, while balancing transferable skills with experiment-specific needs and recognizing the effort of trainers.Training can improve career prospects and make HEP more attractive, but experiments also need to focus on skills required for their own goals.Source: §4.1 · pp. 67–68

4.2 Possible Directions for Training

• Because conventional schools require substantial time and travel and are hard to scale, HEP should expand accessible lab-based core training and make training integral to major R&D projects.Examples include the CERN School of Computing, Bertinoro School of Computing, GridKa School of Computing, LHCb StarterKit, and ALICE Juniors; such events also foster networking.Source: §4.2 · p. 68
• Active and peer training, collaborative notebooks, and selected online courses can broaden practical learning, though developing a supported HEP-relevant course repository requires dedicated effort.The HSF experimented with WikiToLearn, and ROOT has begun providing notebook-based training material.Source: §4.2 · pp. 68–69
• Cross-experiment Q&A could broaden expertise beyond experiment-run systems, but a viable approach has not emerged because of technical and support-effort challenges.Stack Overflow is given as an example of a valuable Q&A system.Source: §4.2 · p. 69

4.3 Career Support and Recognition

• Weak institutional recognition of software and training work harms computing specialists’ career prospects and reduces the community’s ability to engage them in training.Organizations need to adapt promotion and hiring expectations; well-engineered software can reduce resource consumption and increase physics reach, while recognition and incentives depend on local conditions.Source: §4.3 · pp. 69–70

5 Conclusions

• HEP software and computing require a step change in effort, including investment in skilled people, recognized career paths, and training for expert and non-expert contributors.The paper says the bulk of non-expert physicists’ contributions are vital to success.Source: §5 · p. 70
• The community should replace duplicated development with shared, open-source solutions, supported across HEP and strengthened by links to academia, industry, and computer science.Technology choices should account for deployment consequences with distributed-computing partners; the HSF can facilitate common work.Source: §5 · pp. 70–71

Improvements in software efficiency, scalability and performance

• Critical efficiency priorities include simulation, reconstruction and their frameworks, suitable data formats and event content, and optimized distributed computing.The paper identifies simulation and reconstruction as consuming the bulk of experiments’ CPU cycles, and reducing storage requirements as vital.Source: Improvements in software efficiency,… · p. 71

Enable new approaches that can radically extend physics reach

• New simulation and reconstruction techniques, machine learning, and parallelized Big Data analysis could open fruitful approaches for extending physics reach.The paper calls for sharing machine-learning knowledge widely across subdomains.Source: Enable new approaches that can… · p. 71

Ensure the long-term sustainability of the software

• Modern development practices, common tools and training, data preservation, and shared support can improve software quality and sustain analyses for decades.Project proposals should be evaluated against the roadmap’s charges, while the community remains agile enough to adapt to disruptive changes.Source: Ensure the long-term sustainability of… · pp. 71–72

HEP Software Foundation Workshop

• The first HSF workshop supported the CWP process; the final general workshop brought working groups together to present status and plans, build consensus, and prepare for writing.The first workshop included plenary and parallel topical sessions; the final workshop was held in Annecy in June 2017.Source: HEP Software Foundation Workshop · pp. 73–74

Software Triggers and Event Reconstruction WG meeting

• The Software Triggers and Event Reconstruction working group met alongside Connecting the Dots, focused on forward-looking pattern recognition and machine-learning algorithms for HEP.Source: Software Triggers and Event… · p. 73

IML Topical Machine Learning Workshop

• The Machine Learning CWP working group met alongside the Inter-experimental Machine Learning workshop, which facilitated communication about machine-learning R&D in LHC experiments.Source: IML Topical Machine Learning Workshop · p. 73

Community White Paper Follow-up at FNAL

• The FNAL follow-up engaged Intensity Frontier computing and software researchers, discussed CWP plans and HL-LHC commonalities, and included a visualisation workshop with HEP and industry participation.The visualisation workshop explored current HEP tools and possible future directions.Source: Community White Paper Follow-up at FNAL · pp. 73–74

DS@HEP 2017 (Data Science in High Energy Physics)

• The Machine Learning CWP working group also met at DS@HEP, a workshop series facilitating communication about machine-learning R&D in HEP.Source: DS@HEP 2017 (Data Science in High… · p. 74

HEP Analysis Ecosystem Retreat

• The HSF analysis ecosystem retreat addressed HEP analysis tools and ROOT, considering both the current landscape and the CWP’s 5–10-year horizon.Source: HEP Analysis Ecosystem Retreat · p. 74

CWP Event Processing Frameworks Workshop

• The Event Processing Frameworks CWP workshop focused on drafting an initial framework white paper, with representatives of most current-practice frameworks participating.Source: CWP Event Processing Frameworks Workshop · p. 74

• AOD is a reconstructed-event summary containing sufficient information for common physics analyses.Source: B Glossary · p. 75
• The CWP is an HSF-organized community effort describing a strategy and roadmap for HEP software and computing R&D in the 2020s.Source: B Glossary · p. 75
• Geant4 simulates particle passage through matter, while GeantV is an R&D project to exploit newer CPUs’ parallelism for detector simulation.Source: B Glossary · p. 76
• HEPData is an open-access repository for scattering data from experimental particle physics.Source: B Glossary · p. 77
• The proposed HL-LHC upgrade for 2026 aims to increase LHC luminosity tenfold, up to 10^35 cm^-2 s^-1.Source: B Glossary · p. 77
• Machine learning enables computers to learn without explicit programming and includes algorithm classes such as boosted decision trees and neural networks.Source: B Glossary · p. 78
• A PRNG is an algorithm that generates number sequences whose properties approximate those of random-number sequences.Source: B Glossary · p. 79
• SDN encompasses network technologies intended to make networks as agile and flexible as virtualized server and storage infrastructure.Source: B Glossary · p. 79

B Glossary

• The glossary defines Energy reactions as collisions among lepton, photon, and hadron combinations, and the Standard Model as a theory of fundamental particles and their interactions.The Standard Model is described as the currently dominant theory explaining elementary particles and their dynamics.Source: B Glossary · p. 79
• SIMD performs the same operation on multiple data points simultaneously, while Intel TBB supports task-parallel C++ programs that use multicore performance.SIMD describes computers with multiple processing elements; TBB is a widely used C++ template library for task parallelism.Source: B Glossary · pp. 79–80
• SWAN provides interactive data mining in the CERN cloud through Jupyter notebooks, and TMVA supplies ROOT-integrated machine-learning tools for multivariate classification.TMVA is described as a standalone project for processing and parallel evaluation of sophisticated classification techniques.Source: B Glossary · p. 80
• VecGeom is a vectorised geometry library for particle-detector simulation, WebGL renders interactive browser graphics, and XRootD is a fast, low-latency software framework.The supplied XRootD definition describes it as a generic software suite for fast, low-latency use.Source: B Glossary · p. 80
• VOs organize centrally managed user groups for WLCG authorizations, while WLCG links computing centres to store, distribute, and analyse LHC data using X.509 authentication.WLCG comprises more than 170 computing centres in 42 countries; X.509 uses electronic certificates based on private and public keys.Source: B Glossary · p. 80
• x86 denotes the 64-bit version of the x86 instruction set.Source: B Glossary · p. 80