Operation of the ATLAS trigger system in Run 2

• The TDAQ system decides in real time whether collision data are recorded, directly affecting the dataset available for physics analyses.Source: §1 · p. 3
• Run 2 brought higher luminosities, more interactions per bunch crossing, and 13 TeV collisions, increasing rates from interesting physics processes.Run 2 also included proton–proton and several heavy-ion collision datasets.Source: §1 · p. 3
• The paper covers trigger operation, selection evolution, special runs, real-time configuration, software validation, monitoring, and reconstructed-data quality.Source: §1 · p. 3

• ATLAS combines an inner tracker, superconducting solenoid, electromagnetic and hadronic calorimeters, and a muon spectrometer with toroidal magnets.Source: §2 · p. 3
• The inner detector provides charged-particle tracking for |η| < 2.5 using pixel, silicon-strip, and transition-radiation tracker technologies.The TRT extends track reconstruction to |η| = 2.0 and supplies electron-identification information from transition-radiation hits.Source: §2 · pp. 3–4
• The calorimeters cover |η| < 4.9 and provide electromagnetic and hadronic measurements through barrel, endcap, and forward modules.Source: §2 · p. 4
• The muon spectrometer measures muon deflection with precision tracking and dedicated trigger chambers covering |η| < 2.4 for triggering.Source: §2 · p. 4

• The two-stage trigger reduces data from the collision stream through Level-1 hardware selection and High-Level Trigger software processing.Accepted events are buffered, processed, and sent to permanent storage for offline reconstruction.Source: §3 · pp. 4–6
• Level-1 combines calorimeter, muon, topological, and detector-subsystem inputs to identify candidate physics objects and form the trigger decision.Its calorimeter firmware includes pedestal corrections to mitigate out-of-time pile-up, especially for missing transverse momentum.Source: §3 · pp. 4–5
• The Central Trigger Processor applies simple and complex dead-time algorithms to keep Level-1 accepts within detector readout and buffer constraints.At the end of Run 2, the total peak inefficiency was about 1% at a Level-1 rate of 90 kHz.Source: §3 · p. 5
• Level-1 accepts up to 100 kHz within 2.5 μs and identifies detector regions of interest for the High-Level Trigger.Source: §3 · p. 5
• The High-Level Trigger uses fast rejection followed by more precise, CPU-intensive reconstruction on approximately 40 000 processing units.Processing units typically make decisions within a few hundred milliseconds, with full-detector data requested for some objects such as missing transverse momentum.Source: §3 · p. 5
• The Fast TracKer was commissioned during Run 2 but was not used by the High-Level Trigger for trigger decisions.Source: §3 · p. 6

4.2 LHC fill patterns in Run 2

• Changing filling schemes altered luminosity, pile-up, and trigger rates, requiring trigger adjustments and luminosity levelling to preserve efficiency under resource limits.The 2017 8b4e schemes reduced colliding bunches by 30%, while increased bunch intensity raised pile-up to 80 interactions compared with up to 60 previously.Source: §4.2 · pp. 7–9

4.1 The LHC fill cycle

• The LHC nominal cycle comprises injection, ramp, squeeze and adjust, stable beams, dump, and ramp-down phases; collisions occur during stable beams, which occupied approximately 50% of Run 2.Source: §4.1 · p. 7, Figure 2 · p. 8

4.3 Bunch groups

• Bunch groups classify bunch crossings by beam configuration and support physics, background, cosmic-ray, noise, calibration, and technical trigger conditions.ATLAS monitors filling schemes with beam pick-ups and generates bunch-group sets for each scheme before checking them against measured beam positions.Source: §4.3 · p. 9

4.4 The ATLAS run structure

• Run startup uses finite-state-machine transitions to boot, configure, connect, and apply run-dependent settings before releasing the trigger inhibit.The start transition assigns the run number and completes final configuration before events flow through the system.Source: Figure 4 · p. 10, §4.4 · p. 10
• An ATLAS run is a stable data-acquisition period divided into luminosity blocks, the smallest units that can be declared good or bad for physics analysis.Each luminosity block represents approximately constant luminosity and detector conditions.Source: §4.4 · p. 10

• 24/7 control-room operation is supported by remote on-call experts responsible for efficient collection of high-quality data.Source: §5 · p. 11
• Operation coordinators organize weekly on-call experts and daily meetings to oversee smooth and efficient trigger data-taking.Source: §5 · p. 11
• The trigger desk, online, and trigger-menu roles provide configuration, operational support, and preparation of active triggers with their prescale factors.Source: §5 · p. 11
• Online-release and reprocessing teams validate and deploy software through nightly tests and large-sample simulations of the L1 and HLT systems.Source: §5 · p. 11
• Data-quality, debug-stream, and signature-specific experts assess recorded data, recover problematic events, and monitor trigger performance for physics signatures.Source: §5 · p. 11
• L1 subsystem experts monitor operation and data quality for L1Calo, barrel and endcap muons, and the central trigger processor, while the group also supports technical and commissioning runs.Source: §5 · p. 11

6 The Run-2 trigger menu and streaming model

• Trigger chains combine an L1 item with HLT reconstruction and kinematic selections, while prescales control each chain’s accepted-event probability within the trigger menu.Source: §6 · p. 12
• The complete selection set had to fit detector readout, HLT processing, and permanent-storage limits while reflecting physics priorities and available resources.Source: §6 · p. 12
• Unprescaled single-electron and single-muon thresholds were targeted near 25 GeV, alongside coverage of signatures supporting the ATLAS physics programme and B-physics.Source: §6 · p. 12

6.1 The trigger menu evolution in Run 2

• The 2018 menu added to and improved the 2017 menu after luminosity reached 2.0 × 10^34 cm^-2 s^-1, because similar pile-up and resource estimates allowed continued operation.Source: §6.1 · p. 13

6.2 Cost monitoring framework

• Cost monitoring sampled 10% of L1-accepted events to measure CPU and data-flow usage, predict trigger rates, and estimate HLT processing needs for menu planning.Source: §6.2 · p. 13

6.3 Run-2 streaming model

• The inclusive streaming model allowed events to enter multiple streams, including express, debug, calibration, TLA, and monitoring streams, with PEB reducing content for selected detector regions.Source: §6.3 · pp. 13–14, Table 1 · p. 15
• Physics streams retained full detector information, while TLA streams stored reduced event content and HLT objects, enabling higher accept rates without comparable output-bandwidth growth.Source: §6.3 · pp. 13–15
• In 2018, physics data were recorded at an average rate of approximately 1.2 kHz, comprising about 1 kHz of main physics data and 200 Hz of BLS data.Source: §6.3 · p. 15

7 Special data-taking configurations

• Special configurations required dedicated trigger menus adapted to bunch conditions, detector settings, calibration goals, and tightened rate, bandwidth, and CPU limits.Source: §7 · pp. 15–16

7.1 Runs with few bunches

• Low-bunch runs were constrained by IBL wire-bond resonances, so menus simulated the fixed-frequency veto limit and reduced rates when nominal selections exceeded detector protection limits.Source: §7.1 · p. 16, Figure 8 · p. 18
• During the lower-luminosity phase, supporting trigger chains were prescaled to reduce the L1 rate without affecting primary physics triggers.Source: §7.1 · p. 18

7.2 Luminosity calibration runs

• Luminosity calibration used a minimum-bias trigger and partial event building to read out inner-detector data for five selected bunches at approximately 5 kHz.Source: §7.2 · pp. 18–19

7.3 Enhanced bias runs

• Enhanced-bias data used representative L1 items plus a random trigger, with invertible prescale weights restoring an effective zero-bias spectrum without an HLT selection.Source: §7.3 · p. 19
• The enhanced-bias menu added 300 Hz for about one hour and recorded around one million events to estimate rates of primary, supporting, and backup trigger chains.Source: §7.3 · p. 19

8 Condition updates in the HLT

• HLT algorithms depend on calibration, alignment, and other conditions stored in COOL with validity intervals that can change during data-taking.Source: §8 · p. 19

8.1.1 Online beam spot

• Beam-spot parameters were continuously estimated from HLT jet-triggered primary-vertex histograms and updated when statistically significant position or width changes occurred.Source: §8.1.1 · pp. 19–20

8.1.2 Online luminosity

• LUCID-driven luminosity updates were initiated when average luminosity deviated by more than 5%, supporting bunch-dependent calorimeter and particle reconstruction corrections.Source: §8.1.2 · p. 20

8.1.3 Updates of trigger prescales

• Prescale changes required synchronized HLT-farm updates and a luminosity-block-to-prescale-key history so delayed processing could apply the correct configuration.Source: §8.1.3 · p. 20

8.2 Conditions update procedure

• New conditions became valid from the next luminosity block, with markers carried through event data prompting HLT processes to retrieve the corresponding COOL objects.Source: §8.2 · p. 21

9 Trigger configuration

• The TriggerDB records hardware and software settings needed to interpret recorded triggers offline and reproduce their behaviour.It stores L1 and HLT menus, algorithm parameters, software release information, prescales, and bunch group sets.Source: §9 · pp. 21–22

9.2 The TriggerTool

• TriggerTool allows experts to browse, edit, upload, and reproduce configurations, while prescale changes create new PSKs associated with an existing SMK.Algorithm-parameter changes require stopping the current run and starting a new one; prescale or bunch-group changes can be deployed during a run.Source: §9 · p. 21, §9.2 · p. 23

9.1 The trigger database design

• The SMK, BGK, and L1/HLT PSKs uniquely define the trigger configuration used for each run and are saved per luminosity block for offline analysis.Any configuration change produces a new SMK while shared primitives remain linked rather than duplicated.Source: §9.1 · p. 22

9.3 The TriggerPanel

• TriggerPanel manages the active SMK and PSKs, supporting standby, emittance-scan, and physics scenarios with luminosity-based alias suggestions confirmed by the shifter.Selected keys are sent to the CTP and then forwarded to the HLT.Source: §9.3 · p. 24, Figure 12 · p. 25, Figure 13 · p. 26

9.4 Automatic prescaling of L1 trigger items

• The Auto-Prescaler automatically creates and applies L1 prescale keys when detector noise threatens trigger-rate limits, then reverts them after conditions improve.Its rule checker evaluates monitoring data and its orchestrator generates and applies the updated prescale set; the system also supports non-physics beam modes.Source: §9.4 · p. 25

10 Online release validation

• The validation cycle combines nightly bug tests, weekly reprocessing of about 1 million enhanced-bias events, and expert assessment of efficiencies, CPU use, and predicted menu rates.Releases are also validated for changed running conditions and signed off using online and offline monitoring before deployment.Source: §10 · pp. 27–28
• For 957 HLT chains, predicted-to-recorded rate differences are mostly Gaussian within ±3 significance, while small negative tails arise from seed and luminosity-scaling biases.Mean fractional statistical uncertainties are 10% for predicted chain rates, 2% for recorded rates, and 1% for the total HLT-rate prediction.Source: §10 · p. 27, Figure 14 · p. 28
• Enhanced-bias reprocessing can rerun HLT algorithms and L1 simulation on raw data, producing trigger decisions and reconstructed outputs for release evaluation.This supports validation of newly introduced L1 items as well as HLT software changes.Source: §10 · pp. 27–28
• Reprocessing about 1 million enhanced-bias events takes around 24 hours and roughly 7000 CPU hours, using about 4 GB of virtual memory per job and 0.2% of ATLAS grid resources.Source: §10 · p. 28

11 Debug stream processing

• Debug streams classify HLT timeouts, processing-unit timeouts, errors, crashes, missing data, duplicates, truncated results, force accepts, and late events for offline investigation.Different failure types have different recovery prospects, with missing-data events generally not recoverable unless corrective fixes remove the cause.Source: §11 · pp. 28–29
• Debug-stream fractions decreased from 2.4 × 10^-4 in 2015 to 2 × 10^-7 in 2018, making them negligible relative to the physics-stream event sample.The annual fractions were 3.5 × 10^-5 in 2016 and 3.2 × 10^-6 in 2017.Source: §11 · p. 29, Table 2 · p. 30
• Offline recovery reruns the trigger with the online configuration while removing the constraint that caused debug-stream recording, and software patches can fix algorithmic failures before rerunning.Recovery rates were 97.8%, 99.5%, 98.9%, and 96.5% in 2015–2018, respectively.Source: §11 · p. 30
• Automatic recovery is not attempted when a run contains over 100k debug-stream events, because this usually indicates a persistent software issue requiring correction first.Unrecoverable events associated with substantial detector problems can exclude affected luminosity blocks from physics analyses.Source: §11 · p. 30

12.1 Rate monitoring

• Trigger rates provide an early warning because they respond sensitively to detector malfunctions, beam issues, and instantaneous-luminosity changes.The Trigger Rate Presenter monitors and archives individual L1 and HLT rates before and after prescales, together with stream rates.Source: §12.1 · p. 31, Figure 15 · p. 32
• Xmon overlays luminosity-scaled rate predictions with live rates, allowing shifters to identify trigger or detector problems quickly during data-taking.Predictions use fitted luminosity and pile-up dependences from previous runs and current online conditions.Source: §12.1 · p. 31, Figure 16 · p. 33

12.2 Online data quality monitoring

• DQMD compares live trigger histograms with references using automatic tests such as the Kolmogorov–Smirnov test and flags agreement as green, yellow, or red.Reference histograms come from good runs with similar conditions and are updated as conditions change.Source: §12.2 · pp. 31–32, Figure 17 · p. 33

12.3 P-BEAST and shifter assistant

• P-BEAST presents rates, bandwidths, prescale keys, HLT capacity, memory use, luminosity, and dead time over time, giving shifters a rapid view of system status and history.It was heavily used during 2017 luminosity levelling with the 8b4e filling scheme to check trigger-resource limits.Source: §12.3 · p. 34, Figure 18 · p. 34

12.4 Data-taking anomalies

• High rates can create backpressure when HLT processing or debug-stream output exhausts available resources, so affected chains are typically prescaled or disabled until the cause is addressed.Run-2 data losses were typically caused by excessive rates from changing LHC conditions or sub-detector problems.Source: §12.4 · p. 35

Offline assessment

• Express-stream histograms reconstructed within a day provide the main offline DQ check, comparing trigger distributions, efficiencies, resolutions, and trigger objects with offline references.When express-stream statistics are insufficient, the final assessment uses equivalent histograms from the full physics stream, typically available a few days later.Source: §13 · p. 36

Defect handling

• Data-quality defects record luminosity blocks with non-optimal acquisition, distinguishing tolerable abnormalities from intolerable problems that make data unsuitable for physics.Examples of intolerable defects include large detector-coverage losses and incorrect trigger configurations.Source: §13 · p. 36
• The Good Runs List is built from all detector and trigger defects, generally vetoing any luminosity block marked by an intolerable defect.This converts DQ assessments into the run and luminosity-block selection used by physics analyses.Source: §13 · p. 36

Data-quality performance

• 99.6% trigger data-quality efficiency was achieved over the full Run 2 13 TeV proton–proton dataset, with 139 fb−1 deemed good for physics analysis.The efficiency is luminosity-weighted over stable-beam periods intended for physics use.Source: §13 · p. 36

Offline assessment

• Reference-based web displays flag histogram agreement as green, yellow, or red, while periodic reference updates account for changing trigger menus and detector conditions.The example shows hadronically decaying τ-lepton candidates reconstructed in the HLT.Source: §13 · p. 36, Figure 19 · p. 37

Data-quality performance

• Trigger data-quality losses were mainly associated with L1 hardware or configuration problems, while TDAQ data-taking losses declined from 3.16% in 2015 to 0.671% in 2018.L1 issues included detector-coverage problems and a CTP-cable problem.Source: §13 · pp. 36–37, Table 3 · p. 37

14 Conclusion

• The trigger system successfully supported proton–proton, heavy-ion, and special-run data-taking with minimal losses from operational problems.Validation, monitoring, and data-quality procedures operated in both online and offline environments.Source: §14 · p. 38
• 139 fb−1 of high-quality 13 TeV proton–proton collision data were collected across peak luminosities of 0.5 × 10^34 to 2.1 × 10^34 cm−2 s−1 and 28–60 pile-up collisions.The conclusion attributes this coverage to the trigger’s performance and flexibility.Source: §14 · p. 38
• Trigger operation maximized the physics use of delivered collisions while respecting maximum readout rates, available HLT computing resources, and sustainable storage bandwidth.These constraints shaped how the system exploited Run 2 collision data for analyses.Source: §14 · p. 38