CheckMATE 2: From the model to the limit

• CheckMATE automatically calculates limits for new-physics models and integrates Monte Carlo event generation, unlike the original version, which required users to generate events themselves.The program combines event generation, detector simulation, and analysis within its workflow.Source: PROGRAM SUMMARY · p. 2
• The simplest output states whether a model is ruled out at 95% CLs, while more complex metrics can combine multiple signal regions.Source: PROGRAM SUMMARY · p. 2
• CheckMATE implements only a subset of available experimental results, and processing 10000 events requires about 245 s on a single core.Running time scales approximately linearly with the number of input events.Source: PROGRAM SUMMARY · p. 2

• Users must cite Delphes 3 [1], FastJet [2], the anti-kt jet algorithm [4], and the CLS prescription when using CheckMATE.Source: Important Note · p. 2
• Users must cite all experimental analyses used to set limits, including relevant implementation references for analyses developed by non-CheckMATE authors.Source: Important Note · p. 2
• When CheckMATE generates events internally, users must cite MadGraph5 aMC@NLO [6] for hard matrix elements and Pythia 8.2 [7] for showering or matching.Source: Important Note · p. 2
• Externally generated .hepmc or .lhe files require citation of the Monte Carlo event generator that produced them.Source: Important Note · p. 2
• Analyses using mT2, MCT-family variables, topness, or Super-Razor require citations to the corresponding libraries and methods [8] [10] [12] [13] [14] [15] [16].These include mt2 bisect, MctLib, and the topness library; Super-Razor is separately identified as a cited analysis variable.Source: Important Note · pp. 2–3

• Recast analyses provide general model coverage by simulating events, detector responses, experimental cuts, and predefined signal regions, unlike faster simplified-model tools limited by tested topologies.Simplified-model limits can be severely weakened for longer or asymmetric decay chains and may be inaccurate for different angular correlations or intermediate particles.Source: §1 · p. 5
• CheckMATE 2 integrates MadGraph5 aMC@NLO and Pythia 8, so users provide an SLHA or UFO file while event generation runs internally.The integration supports supersymmetric models through SLHA files and other models through UFO files produced by tools including FeynRules, SARAH, or LanHEP.Source: §1 · p. 5
• Hadronised event samples require substantial storage, with 10,000 events occupying ∼1–∼10 GB and large-scale disk I/O limiting concurrent processing.Previously, recast tools required users to supply externally generated, showered, hadronised events with underlying-event modelling.Source: §1 · p. 5
• Internal event generation makes single-process execution up to ∼40% faster and enables simultaneous cluster jobs from one hard disk without performance loss.Avoiding stored event files removes a major I/O bottleneck in multi-process processing.Source: §1 · pp. 5–6
• CheckMATE 2 provides over 60 LHC analyses spanning the 7, 8, and 13 TeV runs, plus 14 TeV high-luminosity analyses for studying ultimate LHC reach.The available analyses let users move beyond current-data limits toward projected reach for a model.Source: §1 · p. 6
• The updated AnalysisManager adds more LHC kinematic observables and tools that simplify statistical analysis and investigation of new search ideas.These improvements accompany the expanded analysis collection.Source: §1 · p. 6

2. General Program Flow

• The Python script configures event generation, showering, detector simulation, and recasting, then reports signal-to-limit ratios and whether each model point is allowed.The computation-heavy tasks are delegated to C++ modules and external programs, while Python processes the final results against published 95% CL limits.Source: §2.1 · p. 6, §2 · p. 6

2.2. FRITZ

• FRITZ connects MadGraph5 aMC@NLO, Pythia 8, Delphes, and analysis modules while passing intermediate events on-the-fly instead of storing them on disk.Users can still explicitly write Pythia or Delphes events when required.Source: §2.2 · pp. 6, 8

2.3. Event Use or Generation:

• CheckMATE 2 supports externally generated events, Pythia-only generation, MadGraph5 aMC@NLO generation followed by Pythia showering, and externally supplied LHE files showered by Pythia 8.The integrated event-generation functionality can yield a factor-of-3 speed gain over CheckMATE 1 depending on the benchmark and parallelization.Source: §2.3 · pp. 8–9

2.4. Delphes

• Delphes simulates calorimetry, tracking, and jet reconstruction, while CheckMATE internally performs final-state identification efficiencies and extends event objects for generator-level tagging.Detector-level ROOT objects are normally passed directly to the analysis framework rather than saved.Source: §2.4 · p. 9

2.5. AnalysisHandler

• AnalysisHandler applies analysis-dependent isolation, identification, and tagging requirements, using separate ATLAS and CMS handlers tuned for 7, 8, 13, and 14 TeV studies.It constructs the required identification working-point lists and can apply simplified b- and τ-tagging algorithms.Source: §2.5 · pp. 9–10

2.6. Analyses

• The analysis framework processes events through overlap removal and signal-region cuts, storing efficiencies and signal yields computed from luminosity, cross section, and acceptance.Weighted events are incorporated when calculating efficiencies.Source: §2.6 · p. 10

2.7. Evaluation and Output

• CheckMATE compares predicted signal yields with experimental limits using either model-independent S95 limits or calculated CLs values, and optionally computes likelihoods for parameter fits.With multiple signal regions, exclusion is determined by the region with the best expected sensitivity under the background-only hypothesis.Source: §2.7 · p. 10

Parameter specification

• CheckMATE parameters can be supplied in an input file or as command-line -parameter value pairs, but command-line setup supports only one process.Multiple processes require an input file or the add feature.Source: §3 · p. 11

Analysis selection

• Analysis selection supports individual analyses, collision energies, experiments, and comma-separated combinations sharing the same center-of-mass energy.Available energy selectors include 7TeV, 8TeV, 13TeV, and 14TeV; incompatible energies abort the run.Source: §3 · p. 11

Invisible particles and Pythia8Process

• Additional parameters declare non-MSSM invisible particles and generate common SUSY production processes through Pythia8Process, while other combinations require an explicit Pythia 8 input file.Supported process groups include gluino, squark, stop, sbottom, electroweakino, coloured-SUSY, and all-SUSY production; the default PDF is used and the underlying event is switched off.Source: §3 · pp. 11, 14

Result-file columns

• Result-file parameters expose event totals, signal-region yields, weighted-event uncertainty inputs, efficiencies, exclusion metrics, CLs, likelihoods, and related uncertainties.The total- and signal- prefixes distinguish full samples from events passing signal-region cuts, while sumofweights2 supports statistical uncertainties for weighted events.Source: §3 · pp. 11, 14

add mode

• CheckMATE supports adding new processed events or processes to an existing run by reusing its name and output directory, with settings restored and results accumulated.Add mode can be repeated and is controlled by OutputExists or -oe add.Source: §3 · p. 14

Likelihood

• Likelihood mode computes signal-region likelihood ratios using nuisance parameters for ∆B and ∆S, while summed likelihoods require independent regions because overlaps are not checked.The likelihood outputs remain −1 unless the corresponding calculation is enabled; likelihood scans can identify best-fit points and confidence intervals. Ref. [59] provides an example, and Ref. discusses parameter analyses.Source: §3 · p. 14

4.1. Benchmark Model

• The example benchmark contains a 1.5 TeV gluino, eight 1.5 TeV first- and second-generation squarks, and a stable 100 GeV bino-like neutralino LSP.Gluinos decay democratically to same-flavour light quark pairs and the neutralino, while squarks decay to associated quarks and the neutralino; equal gluino branching ratios are phenomenologically justified.Source: §4.1 · p. 15
• The run illustrates four event-generation inputs: MadGraph5 aMC@NLO plus Pythia 8, Pythia 8 alone, pre-generated .lhe files, and multiple showered .hepmc files.These correspond respectively to squark-antisquark, squark-pair, gluino-pair, and associated gluino-squark production, with multiple files averaged for the latter process.Source: §4.1 · p. 16

4.2. Initialising and Starting CheckMATE

• The parameter card combines common run settings with one process block per production mode, using an SLHA spectrum for internally simulated events and process-specific event or cross-section inputs.A fixed random seed makes results deterministic; KFactor rescales the MadGraph5 aMC@NLO leading-order cross section, while .hepmc inputs require an explicit cross section.Source: §4.2 · pp. 17–18
• The benchmark point is excluded with r = 2.24273341815 in signal region SR02_3j of analysis atlas_1405_7875 after 40 analyses.The predicted signal exceeds the 95% upper limit, agreeing with the experimental collaboration’s result in Figure 9 of Ref. [61].Source: §4.2 · p. 20
• Changing m̃q and m̃g in the SLHA file allows users to map allowed and excluded regions of parameter space after testing individual points.For users focused only on exclusion status, the final result is generally sufficient.Source: §4.2 · p. 20

4.3. Structure of the Results Folder

• The results folder stores screen-equivalent output in result.txt alongside module-specific directories for internal settings, event steering, generation, detector simulation, and analyses.The internal files support reproducing the Python instance and add-mode runs, while the folder is created only after successful completion.Source: §4.3 · p. 20, §4.3.1 · p. 20

4.3.2. Folder

• FRITZ logs runtime output, errors, module initialization, event-loop progress, cross sections, and successful finalization for each tested event file.The example produces five log files because four processes include one process represented by two separate .hepmc files.Source: §4.3.2 · p. 21
• The example links one PythiaHandler to separate ATLAS and CMS DelphesHandlers, whose detector-level outputs feed analysis handlers applying tagging, isolation, and signal-region categorisation.Generator cross sections normalize the analyses; generator statistical uncertainty is handled internally, while additional systematic uncertainty is supplied through XSectErr.Source: §4.3.2 · pp. 22, 24

4.3.3. Folder

• MadGraph5 aMC@NLO stores process-specific working files, while CheckMATE supplies the proc, parameter, and standardized run-card inputs needed for event generation.The run card incorporates user settings such as requested event count and centre-of-mass energy, while the SLHA parameter card can already exist as an input file.Source: §4.3.3 · p. 25

4.3.4. Folder

• Pythia 8 cards configure showering or hadronisation from SLHA parameters or MadGraph events, and its cross-section output matches the values used for analysis normalisation.The run also preserves random-number-generator states for reproducing or independently restarting event sequences.Source: §4.3.4 · pp. 26–27

4.3.5. Folder

• Delphes stores detector-simulation logs and normally uses standardized detector cards, with modified cards generated here only to impose the fixed random seed 10.The modified cards differ from standard cards by the added set RandomSeed 10 line.Source: §4.3.5 · p. 27

4.3.6. Folder

• Analysis outputs include cutflow files for stepwise selection efficiencies and signal files for final signal-region yields, with weighted-event sums supporting statistical-error evaluation.Signal files feed the subsequent evaluation, while cutflows are primarily useful for validation; the example’s cross-section error is set to 20% of the total cross section.Source: §4.3.6 · pp. 28–29

4.3.7. Folder

• CheckMATE combines process results, propagates statistical and systematic errors, compares signal regions with experimental limits using the conservative r-limit, and records the most sensitive regions.The evaluation stores observed and expected backgrounds, predicted signal events, limits, and r values in total_results.txt, then identifies the largest expected sensitivity per analysis.Source: §4.3.7 · pp. 30–31
• CheckMATE identifies three analyses with sizeable r-values targeting large jet multiplicity and missing transverse energy: ATLAS [61], ATLAS [62], and CMS [63].The CMS analysis uses the αT variable to identify BSM events with large hadronic activity.Source: §4.3.7 · p. 32
• r_obs^cons ≈ 2.2 in SR02 3j of ATLAS 1405 7875 excludes the tested model because the value exceeds 1.CheckMATE selects the most sensitive signal region and uses the observed result for the final exclusion decision.Source: §4.3.7 · p. 32
• The example illustrates how users can test various BSM models through multiple input methods and inspect the resulting output files.The described workflow is intended to let standard users test models without much effort.Source: §4.3.7 · p. 32

Signal regions and correlations

• The 190 ATLAS 8 TeV signal regions include searches such as the 27-region one-lepton stop search [71], but correlations between regions can invalidate naive statistical combinations.Correlations are especially strong for signal regions with similar final states and kinematic cuts and should be included in any combination.Source: §5 · p. 32

Validation and coverage

• The available analyses include 8 and 13 TeV ATLAS and CMS searches, with ATLAS outnumbering CMS because particle-efficiency validation was optimized for ATLAS.The listed analyses are validated against published cut-flows, distributions, and/or exclusion-limit plots, and the tables report signal-region counts and integrated luminosities.Source: Table 3 · p. 33, Table 4 · p. 34, Table 5 · p. 34

Implemented analyses

• About 60 analyses span inclusive SUSY, third-generation, electroweak, monojet/monophoton, vector-like top, vector-boson-fusion, and SM cross-section searches.Most searches target missing transverse momentum, while the monojet and monophoton analyses also probe compressed spectra, simplified dark matter models, large extra dimensions, and Higgs portals.Source: §5 · pp. 32, 36

Validation limitations

• Some analyses are only partially validated, so users should exercise caution when relying on signal regions that have not been completely tested.Outstanding validation issues may reflect insufficient information from the experimental collaborations.Source: §5 · p. 36

High-luminosity studies

• Official ATLAS 14 TeV high-luminosity studies for 300 and 3000 fb−1 cover squark, gluino, stop, sbottom, chargino, and neutralino pair production.These studies were included because current searches already push gluino and first-generation squark exclusion limits well beyond the TeV scale and high-luminosity limits are expected to improve.Source: §5 · pp. 35–36, Table 7 · p. 35

Single-instance performance

• 245 seconds versus 340 seconds reduced processing time for 10,000 gluino pairs, demonstrating CheckMATE 2’s improved performance over CheckMATE 1.The benchmark used identical Pythia settings and sparticle spectra at 8 TeV, with events processed through a single inclusive supersymmetry search [61].Source: §6 · p. 36

Performance mechanism

• FRITZ improves performance by bypassing HepMC or STDHEP generation and detector-level ROOT storage, directly interfacing Pythia, Delphes, and AnalysisHandler without disk I/O.This design avoids the intermediate file-writing and reading steps used in the CheckMATE 1 setup.Source: §6 · pp. 36–37

Concurrent performance

• 544 seconds versus 1,802 seconds reduced average processing time for ten simultaneous instances, showing CheckMATE 2’s advantage when concurrent ROOT-file operations bottleneck CheckMATE 1.CheckMATE 1 required detector-level ROOT files to be written before analysis, whereas CheckMATE 2 immediately processed reconstructed objects.Source: §6 · pp. 36–37

7.1. Prototyping New Analyses

• Analysis prototyping now allows analyses to be added without signal-region data, enabling Standard Model background simulation before defining expected backgrounds and observations.After backgrounds are calculated, users enter total Standard Model backgrounds for each signal region, and AnalysisManager internally computes S95 limits for rapid new-physics tests.Source: §7.1 · p. 37

7.2. Kinematical Variables

• CheckMATE provides common LHC kinematical variables through Delphes and ROOT, including transverse momentum, energy, pseudorapidity, boosts, and TLorentzVector functionality.It also implements mass-reconstruction variables such as MT 2 and derivatives [10] [11] and MctLib [12] [13], with the full list in Table 8 and a review in Ref. [103].Source: §7.2 · p. 37

• CheckMATE 2 integrates Monte Carlo event generation, enabling direct testing from UFO models to LHC results while significantly reducing CPU load.This release improves both the ease and speed of model investigation compared with its predecessor.Source: §8 · p. 37
• Over 50 additional analyses cover most LHC searches involving missing energy, while 14 TeV high-luminosity studies expose the ultimate LHC reach for models.Source: §8 · p. 37
• The AnalysisManager now internally calculates S95 values, producing only small differences from experimental values and ensuring statistical consistency across analyses.Source: §8 · p. 37
• CheckMATE 2 facilitates user-developed analyses through easier background inclusion and a substantially expanded library of kinematical variables.Source: §8 · p. 38
• The program remains a continuously evolving snapshot, with planned fast parameter scans, jet-multiplicity matrix-element merging, and systematic correlations for combining signal regions.Source: §8 · p. 38

A. Installation Instructions

• CheckMATE uses external programs and libraries divided into always-required and optional categories, with an interactive online tutorial for selectively skipping or updating installation steps.The online tutorial is especially useful when required programs are already installed or optional functionality is unnecessary.Source: A. Installation Instructions · p. 39

A.1. Required Packages

• CheckMATE requires Python 2.7.X with X>3, ROOT v5.34.36 or later, and Delphes v3.3.3 or later [1].Python 3 is not supported, and already installed packages allow the corresponding tutorial sections to be skipped.Source: §A.1 · p. 39
• The installation tutorial is tested on Ubuntu 16.04, while other operating systems may require changed flags or additional system libraries.The source files remain usable on other systems, with documentation recommended for troubleshooting.Source: §A.1 · p. 39

A.2. Optional Packages

• Optional event-generation support uses Pythia 8 [7] and/or MadGraph5 aMC@NLO [6], with HepMC required only for saving generated events in HepMC format.MadGraph5_aMC@NLO is installed by downloading and unzipping it, whereas HepMC and Pythia 8 are compiled when needed.Source: §A.2 · pp. 40–41

A.3. Installing CheckMATE

• CheckMATE is configured with ROOT and Delphes plus any installed optional packages, while gzip include and library paths are needed only for gzip-enabled Pythia 8.Delphes and ROOT are mandatory; optional --with commands can be omitted when the corresponding programs are absent.Source: §A.3 · p. 41
• The supplied CMSSM test spectrum has tan β = 10, m0 = 100 GeV, m1/2 = 250 GeV, A0 = −100 GeV, positive µ, and SUSY masses of 100–600 GeV.This example is used for the final installation test run.Source: §A.3 · p. 41
• r = 1.15744708205 excludes the CMSSM test scenario at 95% confidence in atlas_1405_7875 SR03_a.4jl-.The command uses only 100 Monte Carlo events and reports the result within seconds.Source: §A.3 · p. 41

B. Statistical Analysis in CheckMATE

• CheckMATE addresses the p-value for observing N events when the Standard Model predicts B ± ∆B and new physics adds S ± ∆S, using CLs with a likelihood-ratio discriminator.The procedure is described in detail in Ref. [117].Source: B. Statistical Analysis in CheckMATE · p. 41

B.1. 1-bin Likelihood and Test Statistics

• With perfectly known S and B, the event count likelihood is given by a Poisson distribution.Source: §B.1 · p. 41
• CheckMATE models uncertain background values with a probability density P(B_unc.|B, ∆B), combining independent background errors into a single ∆B.The framework can be extended to multiple independent uncertainty sources.Source: §B.1 · p. 42
• Background uncertainty is parameterized as B_unc.(θ), where the nuisance parameter θ follows P(θ|θ̃) and θ̃ initially satisfies B_unc.(θ̃) = B.The value of θ̃ changes during the CLs calculation.Source: §B.1 · p. 42
• CheckMATE assumes B_unc.(θ) = B exp(θ∆B/B) with Gaussian θ, yielding the intended mean and standard deviation for small ∆B/B while preventing negative backgrounds for large uncertainties.Source: §B.1 · p. 42
• Independent signal and background uncertainties define an extended likelihood with signal-strength modifier µ, which distinguishes signal and background hypotheses as S_unc(θS) varies.Source: §B.1 · p. 42
• CheckMATE uses a Profile Likelihood Ratio, with globally maximizing parameters ˆµ, ˆθS, and ˆθB and fixed-µ nuisance maximizers defining qµ.Larger qµ indicates lower compatibility between the observation and null hypothesis; for large event rates, Wilks’ theorem [118] gives an asymptotic χ2 distribution.Source: §B.1 · p. 42
• Because the likelihood-ratio numerator cannot be evaluated analytically, CheckMATE computes the test statistics by finding derivative roots with scipy.optimize.root.Source: §B.1 · p. 42

B.2. Confidence Levels and p-values

• CLs yields conservative signal limits: under-fluctuations reduce 1−CLB, increase CLs, and weaken exclusions, preventing false exclusion of signals below background uncertainties.CheckMATE excludes models with too-small CLs and evaluates the required Poisson-Gaussian integrals numerically.Source: §B.2 · p. 43

B.3. Model Independent Limits S95

• CheckMATE derives observed S95 by solving CLs = 0.05 with Pegasus regula-falsi, while expected limits use background expectations from the observation.The model-independent approximation removes signal-uncertainty terms and uses λ(µ, θB) ≡ µS + B exp(∆BθB).Source: §B.3 · pp. 43–44

B.4. Likelihood

• Combined likelihoods sum the Eq. (5) test statistic over bins without the one-sided µ restriction, but require orthogonal signal regions.For non-orthogonal analyses, users must manually select relevant orthogonal regions before summing likelihoods.Source: §B.4 · p. 44

C. Tuning

• CheckMATE’s tuning updates lepton efficiencies and b-tagging, incorporates recent ATLAS parametrisations, and implements new lepton energy resolutions in 13 and 14 TeV Delphes cards.Efficiencies are evaluated internally by the corresponding AnalysisHandlers.Source: C. Tuning · p. 44

C.1. Electrons

• Electron identification implements loose, medium, and tight ATLAS operating points with transverse-energy-dependent efficiencies, while energy and momentum smearing follow ATLAS parametrisations.The implementation currently models the main ET dependence and stores the functions in the ATLAS 13 and 14 TeV Delphes cards.Source: §C.1 · p. 44, Figure 2 · p. 45

C.2. Muons

• Muon efficiencies are approximately 0.99 for loose and 0.97 for tight candidates, except at |η| < 0.1 where efficiency is 0.6, with negligible pT dependence.Muon resolution coefficients are provided for 13 and 14 TeV setups, with separate rapidity regions for the HL option.Source: §C.2 · p. 45

C.3. B-Tagger

• B-tagging balances signal efficiency against background rejection, distinguishing charm from light-jet rejection because charm is harder to separate from b-jets.ROC curves describe this trade-off, whose preferred working point depends on each analysis’s required signal quantity and purity.Source: §C.3 · pp. 45–46, Figure 3 · p. 46

• The model uses separate ROC curves for light-jet and c-jet rejection, parameterised by signal efficiency and the corresponding background rejections.Here, ¯ϵS denotes signal efficiency, while ¯rlight/c denotes light- and c-jet background rejection.Source: §C.3.1 · p. 46
• Tagging probabilities depend on jet transverse momentum for signal, light-quark, and D∗-meson jets at each chosen ROC working point.The corresponding measurements are shown in Figure 4.Source: §C.3.1 · p. 46, Figure 4 · p. 47
• Signal efficiencies are reduced by 15% overall, suppressed further at large energies, and light-quark tagging rates by 20% to match experimental cutflows.The signal parametrisation also uses separate data sets for low and high energies and adapts to two η regions for light-quark rejection.Source: §C.3.1 · p. 47
• The pT-dependent distributions, defined at ¯ϵS = 0.7, are linearly rescaled to the selected signal or background efficiency.Light- and charm-jet efficiencies are parameterised as functions of η through coefficients listed in Table 10.Source: §C.3.1 · pp. 47–48, Table 10 · p. 48

13 TeV ATLAS b-tagger

• For Run–2 ATLAS analyses, CheckMATE uses efficiencies fitted to the MV2c20 algorithm [128], retaining the Run–1 b-jet-efficiency pT dependence but changing the mistag rates.This defines the Run–2 implementation relative to Run–1.Source: §C.3.2 · p. 48

Efficiency parameterization

• Light- and charm-jet efficiencies are parameterized as functions of η and pT, with coefficients tabulated separately for each jet type.The η dependence uses polynomial functional forms, while the pT dependence is piecewise and depends on b-efficiency; coefficients are provided in Tables 10–12.Source: §C.3.2 · p. 48
• The combined pT- and η-dependent efficiencies are normalized using 12 million t¯t events and include b-efficiency-dependent scale functions.Outside b-efficiency range [0.6, 0.85], parameters are frozen to 60% or 85%, with scaling determined exclusively by the scale functions.Source: §C.3.2 · pp. 48–49

Validation

• Validation against 13 TeV analyses using b-jets found good agreement with the published results.The validation used the b-tagger model described for these analyses.Source: §C.3.2 · p. 49