PaperlayerView exact revision on arXiv

Source-linked AI summary

CheckMATE 2: From the model to the limit

Daniel Dercks, Nishita Desai, Jong Soo Kim, Krzysztof Rolbiecki, Jamie Tattersall, Torsten Weber

arXiv:1611.09856v2hep-ph

TL;DR

Testing many new-physics models against LHC data is difficult because recasting requires externally generated events and substantial storage and I/O. CheckMATE 2 integrates event generation into the model-testing loop, expands analysis coverage, and improves performance and analysis-development tools. The release supports direct testing from SLHA or UFO inputs, up to approximately 40% faster single-process running, and more than 60 LHC analyses.

  • Problem

    Recasting LHC searches requires externally generated events, imposing substantial storage and I/O demands when testing models.

  • Method

    CheckMATE 2 integrates MadGraph5 aMC@NLO and Pythia 8 into a complete model-testing loop accepting SLHA or UFO inputs.

  • Results

    Up to approximately 40% faster single-process running is achieved, alongside coverage of over 60 LHC analyses and 14 TeV high-luminosity studies.

  • Takeaways & Limitations

    Users can test models against current LHC data, investigate ultimate LHC reach, and more easily develop new searches.

  • Takeaways & Limitations

    The release is a snapshot of an evolving program, and implemented signal regions may not be statistically independent across analyses.

Abstract

from arXiv · show

We present the latest developments to the CheckMATE program that allows models of new physics to be easily tested against the recent LHC data. To achieve this goal, the core of CheckMATE now contains over 60 LHC analyses of which 12 are from the 13 TeV run. The main new feature is that CheckMATE 2 now integrates the Monte Carlo event generation via Madgraph and Pythia 8. This allows users to go directly from a SLHA file or UFO model to the result of whether a model is allowed or not. In addition, the integration of the event generation leads to a significant increase in the speed of the program. Many other improvements have also been made, including the possibility to now combine signal regions to give a total likelihood for a model.

PROGRAM SUMMARY

CheckMATE automatically calculates limits for new-physics models, with integrated event generation and statistical outputs including combined signal regions.

  • CheckMATE automatically calculates limits for new-physics models against LHC searches.Its simplest output states whether a model is ruled out at 95% CLs.
  • The new version integrates Monte Carlo event generation, replacing the requirement that users generate events externally.The program now incorporates this step internally.
  • CheckMATE can combine multiple signal regions and provide more complicated statistical metrics for model testing.
  • Only a subset of available experimental results has been implemented.

Important Note

CheckMATE relies on several external physics, detector-simulation, and statistical tools, with required citations for incorporated analyses and libraries.

  • Users are instructed to cite the experimental analyses and relevant implementation references used to set limits.
  • The program incorporates Pythia 8, MadGraph5 aMC@NLO, FastJet, and the anti-kt jet algorithm in relevant workflows.
  • Specialized analyses use libraries for mT2, MCT-family discriminants, topness, and Super-Razor variables.

1. Introduction

CheckMATE 2 addresses the storage and performance costs of externally generated events by integrating event generation and broadening the available analysis coverage.

  • 1. Introduction: Recasting simulates model-specific events, passes them through detector simulation, applies experimental cuts, and tests predefined signal regions.This approach is slower than simplified models but supports a broad range of theories.
  • 1. Introduction: CheckMATE 2 integrates MadGraph5 aMC@NLO and Pythia 8 so users can provide an SLHA or UFO model and generate events internally.
  • 1. Introduction: 10,000 hadronised events require approximately 1–10 GB of storage, creating substantial disk and I/O demands.On shared cluster storage, reading and writing events limits performance for 2–3 simultaneous processes.
  • 1. Introduction: Up to approximately 40% faster single-process running results from avoiding intermediate event-file storage.The design also permits many simultaneous jobs from one hard disk without reduced performance.
  • 1. Introduction: Over 60 experimental analyses cover the 7, 8, and 13 TeV LHC runs, with additional 14 TeV high-luminosity studies.These studies support investigation of the ultimate LHC reach for a model.
  • 1. Introduction: CheckMATE 2 also improves tools for new-search development, including background inclusion, additional observables, and easier statistical analysis.

2. General Program Flow

CheckMATE 2 processes models through event generation, showering, detector simulation, and analysis, with data passed between modules largely on the fly. It supports internally generated or externally supplied events and can optionally retain intermediate outputs.

  • Program flow: CheckMATE’s Python scaffolding configures event generation, showering, detector simulation, and recasting, while computation-heavy modules run in C++.The scaffolding reads user input and prepares configuration files for FRITZ and AnalysisHandler.
  • FRITZ: FRITZ connects MadGraph5 aMC@NLO, Pythia 8, Delphes, AnalysisHandler, and requested analyses according to the supplied data and settings.It is the core C++ program coordinating the processing chain.
  • Event use or generation: CheckMATE 2 automatically links MadGraph5 aMC@NLO and Pythia 8 for event generation, while also supporting externally generated event files.Supported inputs include externally produced .hepmc or .hep files and externally generated .lhe files.
  • Event use or generation: Pythia 8 can generate BSM events entirely within CheckMATE, using either a full .in file or a simplified SUSY process syntax.The full-input mode exposes Pythia 8 parameters, while the SUSY shortcut uses default showering and hadronisation settings.
  • Detector simulation and analysis: Generated or supplied events proceed directly to Delphes and analysis routines, and intermediate events are not stored unless explicitly requested.Delphes creates detector-level objects, which are then processed by the AnalysisHandler; optional output switches can write event files to disk.
  • Performance: A factor-of-3 speed gain over CheckMATE 1 is reported depending on the benchmark model and number of parallel runs.The improvement is associated with passing intermediate data on the fly instead of storing and rereading it from disk.

3. Full List of CheckMATE Parameters

CheckMATE exposes parameters for selecting analyses, configuring event generation and detector processing, controlling result files, and enabling likelihood calculations. The parameter system supports command-line or input-file configuration, with documented constraints and caveats for selected workflows.

  • Parameter input: Parameters can be supplied through the input file or command line, although command-line setup supports only one process.The limitation arises because only one -p command can be provided.
  • Analysis selection: Analysis selectors can target individual analyses, collision energies, or experiment-specific groups, and combined selectors must use the same centre-of-mass energy.Examples include 7, 8, 13, and 14 TeV selections and ATLAS- or CMS-specific groups.
  • Detector simulation: Non-MSSM invisible particles must be explicitly declared so Delphes does not treat them as exotic hadrons depositing energy in the hadronic calorimeter.This is relevant for BSM models such as a Higgs portal model with a stable scalar.
  • Result files: Result-file parameters control which analysis, event, process, and total-result columns CheckMATE records for later investigation.Available columns include event counts, weights, efficiencies, backgrounds, exclusion ratios, confidence limits, and likelihood quantities.
  • Result files: Weighted-event uncertainties require sumofweights2, which stores the sum of squared weights, while total- and signal-prefixed quantities refer to different event subsets.The total prefix refers to the full input sample, whereas signal quantities refer to events passing the relevant signal-region cuts.
  • Likelihood: Likelihood and CLS outputs remain −1 unless their corresponding calculations are enabled, and likelihood ratios can be summed across signal regions.CheckMATE supports likelihood-based model fits in addition to exclusion tests.
  • Run management and caveats: Adding new results requires reusing the original run’s name and output directory, while pre-clustered or parton-level LHE inputs are not recommended.The documented warning is that such inputs may produce substantial efficiency and acceptance errors.
  • Event generation: Pythia8Process provides shortcuts for common SUSY production modes, including coloured, electroweak, or all listed SUSY processes.Other processes or combinations require an explicit Pythia 8 .in file; default parton distributions are used and the underlying event is switched off.

4. Example: Running CheckMATE and Understanding the Results

The example demonstrates how CheckMATE combines multiple event-generation inputs, detector simulation, and experimental analyses to determine whether a supersymmetric model is excluded. For the tested point, the most sensitive signal region yields an exclusion consistent with the experimental result.

  • Model and inputs: The example tests a simplified supersymmetric model with 1.5 TeV gluino and first- and second-generation squarks, plus a 100 GeV stable bino-like neutralino LSP.The gluino decays to first- and second-generation quark–antiquark pairs and the neutralino, while squarks decay to quarks and the neutralino.
  • Event generation: CheckMATE supports several production-mode workflows, using MadGraph5_aMC@NLO and Pythia 8 internally or accepting pre-generated LHE and HepMC files.Squark–antisquark events use MadGraph5_aMC@NLO plus Pythia 8; squark-pair events use Pythia 8 alone; gluino-pair and associated production use externally generated inputs.
  • Run configuration: The run requires a shared SLHA spectrum file and separate process blocks specifying the event source and production-process settings.The parameter block defines the run name, SLHA file, analyses, and random seed, while process blocks describe the four production modes.
  • Practical qualification: Combining different input modes in one run is illustrative; in practice, users would normally use the same tool setup for the different hadronic SUSY final states.This qualification limits how the example’s mixed event-generation configuration should be interpreted.
  • Results: After detector simulation and 40 analyses, CheckMATE reports the model excluded in signal region SR02_3j of analysis atlas_1405_7875.The reported result evaluates r as signal divided by the 95% CL limit on signal.
  • Results: The conservative observed ratio is about 2.2, so CheckMATE selects SR02_3j because robscons exceeds the exclusion threshold of 1.The example result agrees with the experimental collaboration, and changing the gluino and squark masses allows users to map allowed and excluded parameter regions.

5. Available Analyses

CheckMATE provides a broad set of validated LHC analyses spanning multiple energies, final states, and BSM signatures, with additional high-luminosity projections. Users should account for incomplete validation and correlations when interpreting or combining signal regions.

  • Coverage: The search library spans inclusive SUSY, third-generation, electroweak, and monojet or monophoton final states.These categories target gluino and squark production, stops and sbottoms, electroweakinos and sleptons, and compressed or dark-matter-related scenarios.
  • Coverage: CheckMATE also includes vector-like top, vector-boson-fusion rapidity-gap, and selected Standard Model cross-section analyses.Long-lived-particle and heavy-Higgs searches are not included in the current version.
  • Signal regions: 190 signal regions are employed for the ATLAS 8 TeV searches alone, including 27 in one stop search.Signal regions can cover varied mass hierarchies, but regions across analyses are often statistically correlated.
  • Validation: Most listed analyses are fully validated against published cut-flows, distributions, or exclusion-limit plots, while some remain partially validated.Partially validated analyses require caution, especially when the relevant signal regions have not been completely tested.
  • Coverage: CheckMATE covers ATLAS and CMS searches at 8 and 13 TeV, with about 60 analyses available overall.The implemented analyses are grouped by experiment and collision energy.
  • High-luminosity projections: The library adds official ATLAS 14 TeV high-luminosity studies for 300 and 3000 fb−1 covering squark, gluino, stop, sbottom, chargino, and neutralino production.These studies extend the program toward projected high-luminosity LHC reach.

6. Performance Studies

CheckMATE 2 improves performance by processing detector-level objects directly in memory rather than repeatedly writing and reading event files. Concurrent runs particularly expose the bottleneck caused by ROOT file operations in CheckMATE 1.

  • Implementation: CheckMATE 2 bypasses HepMC or STDHEP files and avoids storing detector-level objects in ROOT files before analysis.FRITZ interfaces Pythia, Delphes, and the AnalysisHandler without intermediate hard-disk operations.
  • Performance comparison: Concurrent simulations show a substantial speed advantage for CheckMATE 2 over CheckMATE 1.The comparison used ten simultaneous Pythia instances and 20,000 generated events with identical settings.
  • Performance comparison: CheckMATE 2 processes reconstructed detector-level objects immediately as ROOT objects, avoiding the simultaneous ROOT-file write bottleneck.This in-memory processing is identified as the source of the demonstrated performance gain.

7. Analysis Manager

The AnalysisManager gains tools for prototyping new LHC analyses, calculating backgrounds and S95 limits, and accessing common kinematic and mass-reconstruction variables. These additions support rapid development and testing of analyses within CheckMATE.

  • Scope of updates: The updates are described as additions to the existing AnalysisManager, with the original manual retained for broader details.The paper focuses on listing the changes rather than repeating the full program description.
  • Analysis prototyping: AnalysisManager now permits analysis prototyping without initially supplying signal-region data.Users can first simulate Standard Model backgrounds and add background contributions for each defined signal region.
  • Limit calculation: After backgrounds are calculated, users can enter total Standard Model backgrounds and have AnalysisManager internally calculate S95 limits.The resulting limits allow new-physics models to be tested quickly.
  • Kinematic tools: CheckMATE provides common kinematic variables through Delphes and ROOT interfaces, including transverse momentum, energy, pseudorapidity, and boosts.These variables are directly available from Delphes objects and their methods.
  • Kinematic tools: The program also includes mass-reconstruction variables such as MT2 and its derivatives, alongside MctLib.A full list is documented in the program’s kinematic-variable table and documentation.

8. Summary

CheckMATE 2 expands and streamlines model testing against LHC data through integrated event generation, broader analysis coverage, statistical consistency, and improved user-analysis support.

  • Integrated Monte Carlo event generation improves CheckMATE 2’s ease and speed for testing models against recent LHC data.The integration also significantly reduces the CPU load required to investigate models.
  • Over 50 additional analyses cover the vast majority of LHC searches involving missing energy.The release also includes high-luminosity 14 TeV studies for investigating the ultimate LHC reach.
  • Internally calculated S95 values keep results statistically consistent across all included analyses.The internally calculated values differ only slightly from the experimental values.
  • Table 8 lists the kinematical observables available in the public alpha version of CheckMATE.
  • The AnalysisManager makes it easier to include backgrounds and develop new LHC searches.Its library of kinematical variables has also grown significantly.
  • CheckMATE 2 remains a continuously evolving program, with further developments planned for parameter scans, jet merging, and systematic correlations.The planned correlation treatment is intended to support proper combinations of multiple analyses.

A. Installation Instructions

The installation requires the core analysis and detector-simulation packages, while event-generation and event-storage components are optional; a supplied test run demonstrates successful use.

  • A.1. Required Packages: CheckMATE requires Python 2.7.X with X>3, ROOT v5.34.36 or later, and Delphes v3.3.3 or later.Python 3 is not supported at the stated time.
  • A.1. Required Packages: ROOT is compiled and installed first, followed by Delphes, which must load the compiled ROOT libraries.Parallel compilation uses four processes in the tutorial and can be adjusted to available cores.
  • A. Installation Instructions: The installation tutorial has been tested on Linux under Ubuntu 16.04, though other operating systems may require changed flags or additional libraries.
  • A.1. Required Packages: A successful Delphes installation is indicated by the presence of libDelphes.so in its directory.
  • A.2. Optional Packages: Pythia 8 and MadGraph5 aMC@NLO support event generation, while HepMC is needed only to store generated events in HepMC format.HepMC installation and its connection to Pythia 8 are optional.
  • A.2. Optional Packages: Pythia 8 is compiled with optional HepMC support, and its build directory should contain the necessary libraries after installation.
  • A.3. Installing CheckMATE: CheckMATE is then configured with ROOT and Delphes, while optional paths for HepMC, Pythia 8, MadGraph5 aMC@NLO, and gzip may be omitted when unused.Delphes and ROOT remain required.

B. Statistical Analysis in CheckMATE

CheckMATE evaluates cut-based collider hypotheses with a likelihood framework that incorporates signal and background uncertainties through nuisance parameters and a profile likelihood ratio.

  • The statistical task is to evaluate the p-value for observed events given Standard Model background and new-physics signal expectations.CheckMATE uses the CLs prescription with a likelihood-ratio discriminator.
  • 1-bin Likelihood and Test Statistics: With precisely known signal and background, the event count likelihood is modeled by a Poisson distribution with mean S+B.
  • 1-bin Likelihood and Test Statistics: Background uncertainty is represented by a nuisance parameter whose distribution is based on the combined background error ΔB.The algorithm can in principle be extended to multiple independent error sources.
  • 1-bin Likelihood and Test Statistics: CheckMATE models uncertain backgrounds as B exp(θΔB/B) with Gaussian θ, preventing unphysical negative backgrounds for large uncertainties.For small relative uncertainties, this lognormal model approximates a Gaussian background with mean B and standard deviation ΔB.
  • 1-bin Likelihood and Test Statistics: Independent signal and background uncertainties are incorporated into an extended likelihood with a signal-strength modifier μ.The modifier helps distinguish signal and background hypotheses as the uncertain signal changes.
  • 1-bin Likelihood and Test Statistics: The profile likelihood ratio maximizes the likelihood globally over μ and nuisance parameters, then compares it with the likelihood at fixed μ.Larger qμ indicates lower compatibility between the observation and the null hypothesis.
  • 1-bin Likelihood and Test Statistics: CheckMATE numerically finds the roots of the test-statistic derivatives because the likelihood-ratio numerator cannot be evaluated analytically.

B.2. Confidence Levels and p-values

CheckMATE derives signal and background p-values by generating hypothetical experiments under fitted nuisance parameters, then uses CLs to set conservative exclusions and supports combined likelihoods across orthogonal signal regions.

  • Confidence levels and p-values: CheckMATE generates hypothetical event counts with Poisson fluctuations and nuisance parameters with Gaussian fluctuations to evaluate signal-hypothesis p-values.The resulting fraction of experiments with a test statistic at least as extreme as observed defines CL_S+B.
  • Confidence levels and p-values: The CL_S+B construction can falsely exclude signals much smaller than the background systematic uncertainty when the observed count fluctuates unusually low.This motivates using a conservative prescription based on the model-independent upper limit S95 in ordinary CheckMATE runs.
  • Confidence levels and p-values: A background-only p-value is evaluated with µ = 0 while retaining the signal-hypothesis test statistic q_µ=1, preventing a false use of q_µ=0.This separates the hypothesis used for pseudo-experiment expectations from the signal hypothesis being tested.
  • Confidence levels and p-values: CLs excludes models with small CLs and weakens limits when data are in tension with the background-only hypothesis.Under perfect Standard Model agreement, 1 − CLB equals 0.5 and CLs is close to CL_S+B.
  • Likelihood combination: CheckMATE can sum likelihood test statistics over signal regions to return a combined likelihood, but only for orthogonal regions.For non-orthogonal analyses, users must manually select the relevant orthogonal regions before summing.
  • Detector modeling: The detector-modeling updates include ATLAS lepton efficiencies, energy resolutions, and electron identification across loose, medium, and tight working points.Electron efficiency is modeled primarily as a function of transverse energy, with weaker rapidity dependence.

C.3.1. 8 TeV

The 8 TeV b-tagging model parameterizes signal and background efficiencies using ROC working points and transverse-momentum-dependent tagging probabilities, with additional tuning against analysis cutflows.

  • 8 TeV b-tagging: A chosen ROC working point specifies signal efficiency and corresponding light- and charm-jet background rejections.The model uses these ROC relations to obtain tagging probabilities for the selected working point.
  • 8 TeV b-tagging: CheckMATE tunes the b-tagging parameterization against cutflows from analyses requiring b-tagging.The D* meson jet tagging probability is tuned to 0.4 because it is roughly twice that of other charm mesons.
  • 8 TeV b-tagging: Signal, light-jet, and c-jet tagging efficiencies depend on the jet candidate’s transverse momentum.The dependence is shown separately for signal and light-jet/c-jet backgrounds.
  • 8 TeV b-tagging: The b-tagging model includes a 15% reduction in overall signal-efficiency normalization and an added high-energy efficiency decrease.These adjustments were introduced to improve agreement with experimental results.

C.3.2. 13 TeV

For 13 TeV analyses, CheckMATE models b-tagging efficiencies with MV2c20-based fits and η- and pT-dependent functions, validating the resulting predictions against published analyses.

  • 13 TeV b-tagging: CheckMATE uses efficiencies fitted to the ATLAS MV2c20 algorithm for Run–2 analyses.The b-jet efficiency retains the Run–1 pT dependence, while mistag rates differ from Run–1 values.
  • 13 TeV b-tagging: Light- and charm-jet efficiencies depend on both pseudorapidity and transverse momentum.Separate functional forms and coefficient tables are provided for light and charm jets.
  • 13 TeV b-tagging: The η-dependent efficiency uses polynomial functions whose coefficients are modulated by the selected signal efficiency and ROC curves.The index i denotes either charm or light jets.
  • 13 TeV b-tagging: Outside the signal-efficiency range 0.6–0.85, parameters are frozen at 60% or 85% and scaling is supplied only by the ROC functions.The pT- and η-dependent efficiencies are then combined and normalized using 12 million t̄t events.
  • 13 TeV validation: The 13 TeV b-tagger was validated on implemented analyses using b-jets and found to agree well with published results.This validation covers the subset of CheckMATE 2’s 13 TeV analyses that use b-jets.
Loading 1611.09856v2…