A Comparison of CPU and GPU implementations for the LHCb Experiment Run 3 Trigger

1 Introduction

• The upgraded system processes the full 30 MHz detector output in software, enabling tracker-based HLT1 selections and improving hadronic efficiency relative to the previous calorimeter-based trigger.Run 3 operates at approximately 2 · 10^33 cm−2s−1 and µ around 6.Source: §1 · p. 5

1.1 DAQ and HLT architectures

• The CPU-only and hybrid options both use asynchronous event building, HLT processing, disk buffering, and later HLT2 reconstruction.The HLT2 software and processing architecture are identical in both cases.Source: §1.1 · p. 6, §1.2 · p. 9

1.1.1 CPU-only architecture

• The CPU-only architecture sends the full 32 Tb/s detector data stream from EB servers to a dedicated HLT1 CPU farm before buffering selected output.The HLT1 process reduces the event rate by a factor of 30-60, while the disk buffer retains output during alignment and calibration.Source: §1.1.1 · p. 7
• Advances in server technology reduce the EB from 500 TDR servers to around 173, allowing three TELL40 boards per server and simplifying future upgrades.The EB servers assemble detector fragments into multi-event packets for downstream processing.Source: §1.1.1 · p. 7

1.1.2 Hybrid architecture

• The hybrid architecture replaces the HLT1 CPU farm with GPUs in EB servers, reducing output traffic by a factor of 30-60 and allowing cheaper on-board network interfaces.HLT2 continues to run on the CPU-based EFF after selected events are written to the disk buffer.Source: §1.1.2 · pp. 7–8, Figure 2 · p. 9

1.2 Real Time Analysis

• Run 3 real-time analysis uses HLT1-selected events and a substantial disk buffer so HLT2 can apply offline-quality reconstruction after alignment and calibration.The buffer must also cover recovery from unforeseen operational issues without data loss.Source: §1.2 · pp. 8–9

2.1 Use of storage and computing resources during and outside datataking

• Resource-use assumptions allocate CPU resources to HLT processing during data-taking and simulation outside it, while GPUs are restricted mainly to in-fill HLT1 processing.EB nodes are reserved for event building during data-taking because their memory bandwidth is needed for that task.Source: §2.1 · p. 10

2.2 Existing and pledged HLT2 computing resources

• Run 3 HLT2 resources are quantified using a 380-HEPSPEC reference server, with roughly 1450 equivalent servers available and 1200 more pledged.These older servers are intended for HLT2 because equipping them for HLT1 would not be cost-effective.Source: §2.2 · p. 10, §2.3 · p. 10

2.3 Event building and data-flow

• The CPU-only event-building path must distribute the full 32 Tb/s stream to HLT1 nodes and connect them optically to EB nodes and at least 30 storage servers.A cost-effective CPU node uses two sockets with 64 physical cores and two 100 Gb/s network interfaces.Source: §2.3 · pp. 10–11
• The hybrid path executes HLT1 on GPUs in EB nodes, reducing the EB-to-disk-buffer rate to 0.5-1.0 Tb/s and substantially decreasing optical network requirements.The EB servers can use their on-board 10 Gigabit interfaces for the reduced traffic.Source: §2.3 · p. 11

2.4 Disk buffer boundary conditions

• The disk buffer must sustain 1 MHz initially, with a possible 2 MHz upgrade, corresponding to 120 GB/s for 120 kB events and an effective disk rate of 35-40 MB/s.The lower effective rate accounts for reduced hard-disk performance as the buffer fills.Source: §2.4 · p. 11
• At least 2880 physical disks are required with minimal redundancy, or 3120 with adequate redundancy, because usable capacity is only about 80% of nominal disk size.The passage states that adding disks or using larger disks is not an attractive budget option.Source: §2.4 · p. 11

3 HLT1 sequence

• Both implementations include Velo decoding and reconstruction, primary-vertex finding, muon identification, and a simplified Velo-only Kalman filter using the forward momentum estimate.These components use conceptually similar or identical algorithms in both architectures, so only minor physics differences are expected for several stages.Source: §3 · pp. 11–12
• CPU and GPU implementations use different data layouts and event models for UT and SciFi decoding, and different algorithms for Velo-UT segment reconstruction.These differences lead to slight differences in physics performance.Source: §3 · p. 12
• Both tracking implementations parameterize particle trajectories in the magnetic field and use the initial Velo-UT momentum estimate to accelerate long-track reconstruction.The CPU matches Velo-UT segments to SciFi doublets, whereas the GPU matches them to triplets.Source: §3 · p. 12
• The CPU tracking algorithm applies a 490 MeV transverse-momentum threshold when defining search windows, contributing to differences between CPU and GPU performance.The passage links this choice, together with the Velo-UT algorithm differences, to somewhat different performance.Source: §3 · p. 12
• The sequence covers most bottom- and charm-physics use cases, but several specialized algorithms were preliminary and not fully optimized for this comparison.Their inclusion produced similar relative slowdowns for CPU and GPU implementations, so they were not expected to change the conclusions.Source: §3 · p. 12
• Both implementations include representative HLT selections such as displaced-vertex finding, monitoring, and decision-report writing, whose throughput differences are comparatively small.The specific components are not exactly identical between the architectures.Source: §3 · p. 12
• A global event cut removes 7% of the busiest minimum-bias events before reconstruction to limit occupancy-sensitive tracking effects.The criterion uses the total UT and SciFi hit count, although another subdetector could be chosen in Run 3.Source: §3 · p. 13

• Throughput counts events processed per second in steady state, including HLT1 input, reconstruction, selection, and final output handling.The measurements use the same minimum-bias samples, while CPU results cover both HLT1 AMD nodes and HLT2 reference nodes.Source: §4 · p. 13
• 171 kHz is the measured CPU HLT1 throughput, compared with 92 kHz for GPU HLT1 under the stated measurement procedure.HLT1 throughput includes reconstruction, selection, and saving of trigger candidates.Source: §4 · p. 13
• 471 Hz is the measured CPU HLT2 throughput on an AMD EPYC node, versus 134 Hz on a Quanta node.The HLT2 measurement includes reconstruction but not selection or saving of trigger candidates.Source: §4 · p. 13
• O(170) CPU servers or O(330) GPU cards can implement the HLT1 architectures, making both options viable at system scale.These estimates follow from the measured throughput values used throughout the comparison.Source: §4 · p. 13

5 Physics performance

• The comparison uses identical algorithms, common output formats, and shared physics-performance definitions compiled for both CPU and GPU execution.The CPU version of the GPU code agrees with the GPU version to within 10−4−10−3.Source: §5 · pp. 13–14

5.1 Tracking efficiencies and ghost rates

• Tracking studies use B and D signal samples and restrict analysis to reconstructible long tracks with 2 < η < 5, pT ≥0.5 GeV/c, and p ≥3 GeV/c.Electrons are excluded because their performance was not optimized to the same extent, although checks did not change the conclusions.Source: §5.1 · p. 14, §5 · p. 14, Table 1 · p. 14
• The GPU and CPU implementations show very similar tracking performance, with integrated single-track efficiency agreeing to better than 1%.CPU efficiency is higher at low momenta, GPU efficiency at high momenta; the difference is attributed to algorithm tuning rather than technology.Source: §5.1 · pp. 14–15
• Both implementations have similar Velo efficiency trends versus distance of closest approach and primary-vertex z position.Efficiency losses at large distance of closest approach arise from beamline-favoring search windows and can be recovered at moderate throughput cost.Source: §5.1 · p. 15
• Ghost-track fractions are very similar between the CPU and GPU implementations in the studied long-track samples.Ghost tracks are defined as fake signal combinations that cannot be assigned to a true particle.Source: §5.1 · p. 15

5.2 Track parameter resolution

• The GPU momentum resolution is worse by 0.15–0.2% at low momenta, due to suboptimal momentum-parameterization tuning that can be recovered without throughput loss.Impact-parameter resolution remains very similar between technologies.Source: §5.2 · p. 15

5.3 Primary vertex reconstruction efficiency and resolution

• Primary-vertex efficiency and resolution are close to identical, with average minimum-bias resolutions of σx = σy ∼14 µm and σz ∼87 µm.Vertices are counted as reconstructed when found within 2 mm of their true position.Source: §5.3 · p. 15

5.2 Track parameter resolution

• Track reconstruction efficiencies are very similar, with only a minor momentum dependence in long-track efficiency and limited impact on overall trigger efficiency.The trigger is robust because most HLT1 selections use only a subset of tracks from a signal decay.Source: Figure 3 · p. 16

5.4 Muon ID efficiency

• The GPU muon-identification efficiency improves by up to 10% absolute at small pseudorapidities, while its pion misidentification rate improves by up to 5% there.CPU misidentification performance is a couple of percent better at higher momenta; similar results are expected after throughput-neutral tuning.Source: §5.4 · pp. 16–17, Figure 4 · p. 17, Figure 5 · p. 17

5.5 HLT1 Efficiency

• TrackMVA performance is expected to be similar because CPU and GPU implementations use the same minimum-transverse-momentum and impact-parameter-significance requirements.This line selects tracks meeting those common requirements.Source: §5.5 · pp. 18–19
• TwoTrackMVA can differ at the percent level because the CPU uses an MVA classifier while the GPU implementation does not yet include one.The expected difference is stated for a given background retention.Source: §5.5 · p. 19
• DiMuonLowMass and SingleHighPtMuon are implemented identically across architectures, supporting comparable performance for these selections.DiMuonLowMass targets very low-momentum displaced-track pairs, while SingleHighPtMuon selects identified high-pT muons.Source: §5.5 · p. 20
• CPU- and GPU-based HLT1 trigger efficiencies are comparable versus parent transverse momentum and decay time, and when integrated across the samples.The comparison covers TrackMVA, TwoTrackMVA, DiMuonLowMass, and SingleHighPtMuon selections.Source: §5.5 · pp. 18, 20, Table 2 · p. 20

5.6 HLT1 rates

• TrackMVA, DiMuonLowMass, and SingleHighPtMuon have comparable trigger rates between CPU and GPU implementations.Rates are defined from positive decisions over sampled minimum-bias events, scaled by 30 MHz.Source: §5.6 · pp. 20–21
• 912 ± 52 kHz for GPU HLT1 versus 798 ± 48 kHz for CPU HLT1, with both within the required 1–2 MHz output range.The inclusive difference is largely due to the distinct TwoTrackMVA implementations.Source: §5.6 · pp. 21–22
• The TwoTrackMVA rate discrepancy is explained by the different implementation of that trigger line across the CPU and GPU projects.The figure reports rates for all four selections.Source: Figure 11 · p. 21, §5.6 · p. 22

5.7 Global Event Cut efficiency

• Identical global event cuts produce identical CPU and GPU GEC efficiencies across all samples.Both implementations require fewer than 9750 total SciFi and UT clusters.Source: §5.7 · p. 22
• The integrated GEC efficiency is 0.75 ± 0.01 for Z →µ+µ−, about 85% for the studied B and D samples, and 0.931 ± 0.003 for minimum-bias events.The B and D values have statistical uncertainties of approximately 1%.Source: §5.7 · p. 22

5.8 Summary of the physics performance

• The GPU- and CPU-based implementations have close to identical performance overall, with observed differences attributed to efficiency, fake-rate, or misidentification-rate tuning.These tuning differences are expected to be neutral in throughput, leaving economic and developer-time costs for the architecture decision.Source: §5.8 · pp. 22–23, Figure 13 · p. 23

6.1 Nominal cost-benefit

• The analysis estimates costs for processing 30 MHz at nominal luminosity using relative Quanta-node units and includes HLT1 compute, network, HLT2, and disk-buffer considerations.The nominal analysis assumes full luminosity from the start of data taking and accounts for out-of-fill HLT2 use of CPU nodes.Source: §6 · p. 23, §6.1 · p. 23, Table 4 · p. 24

6.2 Disk buffer simulation

• The disk-buffer study samples correlated fill and gap patterns, generates 1000 toys per datapoint, and selects budget-valid configurations that maximize HLT1 output rate.The buffer is sized to cover 95% of the toys using a 100 kB event size and scanned HLT1 and HLT2 rates.Source: §6.2 · p. 24

6.2.1 Quantification of HLT1 and HLT2 througputs

• The throughput inputs include 175 AMD 7502 nodes for 30 MHz HLT1 and 2050 Quanta-equivalent nodes providing 275 kHz of additional HLT2 throughput.The AMD nodes also provide 82.5 kHz of out-of-fill HLT2 throughput, while the Quanta-equivalent nodes are used only for HLT2.Source: Table 5 · p. 25, §6.2.1 · p. 25, §6.2.2 · p. 25

6.2.3 Results

• 660 kHz and 790 kHz are the maximum affordable HLT1 throughputs for the CPU-only and hybrid scenarios, respectively.These maxima occur within the regions of affordability when the disk buffer is fully used and remaining resources are spent on HLT2.Source: §6.2.3 · pp. 25–26, Figure 14 · p. 26, Table 7 · p. 26

6.2.4 Results assuming a factor two performance increase in HLT2

• Doubling HLT2 throughput raises the maximum affordable HLT1 throughput to 1185 kHz for CPU-only and 1425 kHz for hybrid scenarios.The study assumes this improvement from preliminary reconstruction optimizations pending validation.Source: §6.2.4 · p. 26, Table 8 · p. 26, Figure 15 · p. 27

6.3 Deferred purchasing

• Deferred purchasing supports buying half the nominal HLT1 processing capacity for 2022 because early Run 3 operation is expected at lower luminosity.The hybrid option can initially use one GPU card per event-builder node, preserving flexibility for newer GPU generations and limited event-builder slots.Source: §6.3 · p. 27

7 Conclusion

• CPU-only and GPU-based HLT1 proposals are both viable, with similar raw throughput and physics performance.The choice is not obvious from performance alone.Source: §7 · pp. 27–28
• LHCb selected a hybrid HLT with a GPU-based first stage after weighing immediate cost savings, future performance opportunities, flexibility, and technology-maintenance overhead.The paper identifies GPU progress and heterogeneous-computing expertise as longer-term benefits considered in the decision.Source: §7 · p. 28

• The authors declare no financial or personal conflicts of interest involving third parties affected by the article.Source: Conflict of interest statement · p. 28