PaperlayerView exact revision on arXiv

Source-linked AI summary

MadMiner: Machine learning-based inference for particle physics

Johann Brehmer, Felix Kling, Irina Espejo, Kyle Cranmer

arXiv:1907.10621v2hep-phhep-exphysics.data-anstat.ML

TL;DR

High-dimensional LHC measurements face an intractable likelihood, while established approaches either reduce observables or simplify detector response. MadMiner automates matrix-element-informed machine-learning inference, and its example analysis reports stronger expected operator limits than total-rate and standard-histogram analyses.

  • Problem

    Realistic simulations make the likelihood intractable because calculating it requires integrating over enormous latent parton, shower, and detector histories.

    Source: A. LHC measurements as a… · p. 4

  • Method

    MadMiner implements an analysis chain that uses Monte Carlo event information, including matrix-element-related quantities, to train neural likelihood or likelihood-ratio estimators.

    Source: §1 · p. 19, §V · p. 28

  • Results

    The example analysis found stronger expected limits on effective operators with machine-learning techniques than with total-rate and standard-histogram analyses.

    Source: §V · p. 28

  • Takeaways & Limitations

    The methods retain high-dimensional event data and realistic shower and detector simulation, with fast event evaluation after training.

    Source: §V · p. 28

    When simulations are mismodeled, scale factors and nuisance parameters may be needed, and problematic phase-space regions may require selection cuts.

    Source: Appendix A · p. 30

Abstract

from arXiv · show

Precision measurements at the LHC often require analyzing high-dimensional event data for subtle kinematic signatures, which is challenging for established analysis methods. Recently, a powerful family of multivariate inference techniques that leverage both matrix element information and machine learning has been developed. This approach neither requires the reduction of high-dimensional data to summary statistics nor any simplifications to the underlying physics or detector response. In this paper we introduce MadMiner, a Python module that streamlines the steps involved in this procedure. Wrapping around MadGraph5_aMC and Pythia 8, it supports almost any physics process and model. To aid phenomenological studies, the tool also wraps around Delphes 3, though it is extendable to a full Geant4-based detector simulation. We demonstrate the use of MadMiner in an example analysis of dimension-six operators in ttH production, finding that the new techniques substantially increase the sensitivity to new physics.

I. INTRODUCTION

LHC likelihoods depend on high-dimensional event data but are generally intractable because realistic simulations integrate over enormous latent histories. MadMiner automates matrix-element-informed machine-learning inference while retaining high-dimensional observables and realistic detector simulation.

I. INTRODUCTION

  • Restricting analyses to a few hand-picked summary statistics discards information in other directions of phase space.The passage notes that even well-motivated variables may fall short of fully differential analyses.Source: §I · p. 2
  • Matrix Element Method and Optimal Observable approaches retain high-dimensional observables but simplify shower and detector response, while facing radiation challenges and costly event integrals.These approximations make the likelihood tractable but can produce suboptimal results.Source: §I · p. 2
  • Classifier-based analyses can suppress backgrounds yet fail to maximize parameter precision when kinematic distributions vary across the parameter space.The described strategy typically follows event classification with a histogram-based measurement.Source: §I · p. 3
  • Likelihood-free machine-learning methods use simulated samples to handle high-dimensional observables without summary statistics or simplifications to the physics or detector response.Good performance can require many simulations; matrix-element-informed approaches additionally support fast likelihood evaluation after training.Source: §I · p. 3
  • MadMiner automates the analysis workflow, including event simulation, detector response, backgrounds, and systematic uncertainties.It wraps MadGraph5_aMC, Pythia 8, and Delphes 3 for phenomenological studies and can extend to Geant4-based detector simulation.Source: §I · p. 3

A. LHC measurements as a likelihood-free inference problem

  • Realistic simulation chains make the likelihood intractable because it requires integrating over enormous latent parton, shower, and detector histories.The resulting challenge arises when inferring model parameters from observed events.Source: §I · p. 2, A. LHC measurements as a… · p. 4

B. Learning the likelihood function

MadMiner supports neural estimators of likelihoods and likelihood ratios, including methods that use additional information from Monte Carlo events. The methods differ in their estimator targets and training losses, and the resulting estimates feed established statistical procedures for parameter fits and limits.

B. Learning the likelihood function

  • The supported estimator targets include likelihoods, likelihood ratios, and ratios parameterized by both numerator and denominator model points.The denominator for a likelihood-ratio estimator can use a reference-parameter distribution.Source: B. Learning the likelihood function · p. 5
  • One network parameterized by theory parameters can interpolate across the parameter space and borrow statistical power from nearby points.The passage contrasts this with training separate networks at each grid point.Source: B. Learning the likelihood function · p. 5
  • MadMiner implements neural density estimation, including masked autoregressive flows, and Carl, which estimates the likelihood and likelihood ratio, respectively.These approaches train from samples generated at different parameter points.Source: B. Learning the likelihood function · p. 5
  • Joint likelihood ratios and scores from simulated events define losses that train estimators of likelihoods or likelihood ratios with less training data.The additional simulator information is used to improve estimator precision.Source: B. Learning the likelihood function · pp. 5–7
  • Scandal, Cascal, Rolr, Alice, Rascal, and Alices differ in loss functions and whether they use the joint score, joint likelihood ratio, or both.Rascal and Alices use both forms of simulator information.Source: B. Learning the likelihood function · p. 7
  • After training, likelihood estimates support best-fit and exclusion calculations; toy experiments guarantee statistically correct limits, whereas asymptotic limits rely on accurate network training.Toy experiments can be computationally expensive, while the asymptotic strategy is efficient.Source: B. Learning the likelihood function · p. 7

C. Learning locally optimal observables

MadMiner’s Sally and Sallino methods estimate locally optimal observables—the score—instead of reconstructing the full likelihood. The score is sufficient near a reference parameter point, but that optimality is local and depends on training conditions.

C. Learning locally optimal observables

  • Near the reference point, score components are sufficient statistics, so they retain the full likelihood’s power for measuring parameters in that neighborhood.This local result motivates using the score as a compact set of observables.Source: C. Learning locally optimal observables · pp. 7–8
  • Unlike the Matrix Element Method, the machine-learning approach can accommodate realistic shower and detector simulations and be evaluated efficiently.The comparison is made for estimating the likelihood ratio using approximate parton-level likelihoods.Source: C. Learning locally optimal observables · p. 7
  • A neural network estimates the score from event data, and the score is a vector of observables evaluated at a fixed reference parameter point.Its components are event-level values, unlike the likelihood and likelihood ratio, which remain functions of the parameters.Source: C. Learning locally optimal observables · pp. 7–8
  • Sally and Sallino train score estimators with a loss involving the joint score, which converges to the true score when minimized.After training, Sally supports multivariate score histograms, while Sallino uses one-dimensional histograms of score projections.Source: C. Learning locally optimal observables · p. 8
  • With efficient training and enough data, Sally and Sallino yield statistically optimal limits near the reference point, but may lose power farther away.The size of the region where score components remain sufficient depends on higher derivatives of the log likelihood and is not known in advance.Source: C. Learning locally optimal observables · p. 8

D. The Fisher information

The Fisher information matrix offers a compact measure of measurement sensitivity and is closely connected to the score. It supports local likelihood comparisons and interpretable sensitivity summaries, including treatment of nuisance parameters.

D. The Fisher information

  • The Fisher information summarizes analysis sensitivity in a compact, interpretable form and is useful for quickly comparing channels or optimizing an analysis.It is an alternative to best-fit points and exclusion limits when assessing sensitivity.Source: D. The Fisher information · p. 8
  • For small parameter displacements, the local Fisher distance approximates the log-likelihood ratio, and the inverse Fisher information is the minimum covariance of an estimator under the Cramér-Rao bound.Greater Fisher information corresponds to more precise parameter measurement.Source: D. The Fisher information · p. 9
  • The Fisher information is invariant to reparameterizations of observables, transforms covariantly with parameters, and adds across phase-space regions.This allows the differential information distribution to show where analysis power arises; nuisance parameters can be profiled with a matrix operation.Source: D. The Fisher information · p. 9
  • For particle-physics processes, the Fisher information separates information in the overall rate from power in kinematic distributions.The rate term depends on luminosity, cross section, and parameter derivatives; the kinematic term uses the score components.Source: D. The Fisher information · p. 9

E. Practical analysis aspects

MadMiner supports broad process and model coverage while providing practical tools for event simulation, inference setup, and neural-network training. Its morphing functionality can accelerate computations for suitable models, but some features and modeling choices have specific scope limits.

E. Practical analysis aspects

  • MadMiner supports almost all perturbative high-energy processes that can run in MadGraph5_aMC, including UFO models, provided the model has finitely many parameters and reweightable parton-level event weights.The approach is not fundamentally limited to leading order, though negative weights in some subtraction schemes can cause numerical instabilities.Source: E. Practical analysis aspects · p. 9
  • For factorized matrix elements, morphing uses evaluations at benchmark points to interpolate event weights exactly elsewhere in parameter space.It is often applicable to effective field theories; MadMiner automates the linear algebra after users specify maximal parameter powers.Source: E. Practical analysis aspects · p. 10
  • MadMiner can optimize morphing bases for numerical stability, but some features are available only with morphing, and that setup can substantially reduce computational cost.The basis optimization samples configurations over a user-specified region and selects one minimizing the sum of squared morphing weights.Source: E. Practical analysis aspects · p. 10
  • Pythia 8 supports showering and hadronization, while Delphes 3 and smearing functions provide detector simulation, with modular extension to Geant4-based tools.Users can define fixed-length event observables from Delphes objects or smeared parton-level momenta, and can add external observable-calculation code.Source: E. Practical analysis aspects · pp. 10–11
  • Analyses can combine signal and background processes with unrestricted parton-level final states, including backgrounds that depend on model parameters.Fully data-driven backgrounds are unsupported, although MC samples can have data-driven normalization.Source: E. Practical analysis aspects · p. 11
  • Nuisance parameters model simulation uncertainties and enter inference like physics parameters; MadMiner currently supports scale and PDF uncertainties, but not new-physics effects on shower-splitting probabilities.Final inference profiles or marginalizes nuisance parameters using established statistical procedures.Source: E. Practical analysis aspects · p. 11
  • MadMiner trains configurable feed-forward networks in PyTorch using method-specific losses and several gradient-based optimizers.Network inputs and outputs depend on the method, and parameter rescaling can be useful when parameters span very different magnitudes.Source: E. Practical analysis aspects · pp. 9, 12
  • Ensemble variance can diagnose optimization and finite-training-sample uncertainty, but it is not guaranteed to cover all sources of bias and variance.Bootstrap resampling can make the ensemble variance reflect uncertainty from finite training samples.Source: E. Practical analysis aspects · p. 12

F. Recommendations for getting started

Choosing a starting method depends on whether the analysis needs a full likelihood or locally optimal observables. The paper recommends Sally for dominant linear effects and Alices when nonlinear effects dominate, with a trade-off between power, complexity, and pipeline changes.

F. Recommendations for getting started

  • Full likelihood or likelihood-ratio reconstruction can yield the best limits with sufficient data, expressive networks, and successful global optimization, but demands more training and pipeline change.The approach may require more training data and hyperparameter experiments than estimating observables.Source: F. Recommendations for getting started · p. 12
  • Locally optimal observables are simpler, scale better to high-dimensional parameter spaces, need fewer training samples, and require minimal changes to existing pipelines.Their limitation is that they are optimal only when dominant signatures enter linearly in model parameters; otherwise they may lose power and yield worse limits.Source: F. Recommendations for getting started · pp. 12–13
  • Sally is suggested as a starting point when dominant new-physics effects are linear in parameters, whereas Alices is suggested when nonlinear contributions dominate.For Alices, the paper recommends initially balancing the two loss terms and scanning α over several orders of magnitude.Source: F. Recommendations for getting started · p. 13

III. USING MADMINER

MadMiner provides open-source infrastructure and tutorials for analyses using the described inference techniques. Its workflow specifies the physics setup, benchmarks, and systematics, then generates weighted events and showered, hadronized outputs.

III. USING MADMINER

  • MadMiner is open source and offers interactive tutorials, API documentation, a Docker environment, and reusable Reana workflows.The package can be installed with pip; shower and detector studies additionally require Pythia 8 and Delphes 3.Source: §III · p. 13

A. Analysis specification and event generation

  • Users define the process and parameter space, choose benchmarks (optionally using morphing), specify PDF and scale systematics, and save settings in HDF5.Morphing can automatically select additional benchmarks to minimize the expected size of morphing weights.Source: A. Analysis specification and event… · p. 13
  • MadMiner generates events with MadGraph reweighting at all benchmarks, stores weights and parton information, then uses Pythia 8 to shower and hadronize events into HepMC files.The workflow can launch one generation run or multiple runs with different run cards or benchmark samples.Source: A. Analysis specification and event… · pp. 13, 15

B. Detector effects and observables

MadMiner moves from detector-level event processing through training-data preparation and machine-learning estimation to inference and Fisher-information analysis. Its asymptotic-limit implementation does not currently support systematic uncertainties.

B. Detector effects and observables

  • DelphesReader supports detector-level analysis with Pythia 8 and Delphes 3, while LHEReader supports parton-level analysis using transfer functions for shower and detector effects.Users can define observables and cuts; MadMiner parses them for simulated events and extracts the relevant event weights.Source: B. Detector effects and observables · p. 15

C. Sample unweighting and augmentation

  • SampleAugmenter reweights and unweights events, calculates joint likelihood ratios and scores, and produces algorithm-specific training arrays.Multiple samples can be combined and shuffled before augmentation.Source: C. Sample unweighting and augmentation · p. 15

D. Machine learning

  • Configurable PyTorch neural estimators train independently of simulations and MadMiner files, expose likelihood, ratio, or score estimates, and can be ensembled for checks and uncertainty estimation.Training data can be moved to an external system with GPU support.Source: D. Machine learning · p. 16

E. Inference

  • AsymptoticLimits calculates expected p-values using an Asimov dataset and observed p-values from event lists, with likelihoods estimated from histograms or trained estimators and tests based on Wilks’ theorem.The class currently does not support systematic uncertainties.Source: E. Inference · p. 16

F. Fisher information

  • FisherInformation calculates information from a ScoreEstimator, histograms, or truth-level particles, and provides phase-space distributions plus projected or profiled nuisance-parameter treatments.The histogram options cover one or two kinematic variables.Source: F. Fisher information · pp. 16–17

IV. PHYSICS EXAMPLE

The physics example studies dimension-six operators in tth production, using a high-dimensional final state with missing energy despite its lower expected sensitivity than other channels. It moves from a one-dimensional illustration to validation and a realistic analysis.

IV. PHYSICS EXAMPLE

  • The chosen fully leptonic top and diphoton Higgs signature provides high-dimensional event data with non-trivial missing energy, though it is not expected to be the most sensitive channel.The paper contrasts it with semileptonic tth production and Higgs production in gluon fusion.Source: §IV · p. 17
  • The example comprises a one-parameter, one-observable illustration, a parton-level toy validation against a calculable likelihood, and a realistic analysis with shower, detector, and high-dimensional data.The realistic analysis uses a three-dimensional parameter space.Source: §IV · p. 17
  • The analyses use MadMiner v0.4 with leading-order event generation at 14 TeV, NLO-normalized rates, and the SILH-basis dimension-six operators implemented in the HEL FeynRules model.The three scenarios use different simulation settings, summarized in Table II.Source: §IV · p. 17, Table II · p. 18

A. Illustration of analysis techniques

  • The simplified illustration studies the dimension-six operator cG through its effect on the effective Higgs-gluon coupling and uses θ = 100 cG, with θ = 0 representing the Standard Model.Three benchmarks support morphing of event weights and cross sections across parameter space; the analysis keeps only diphoton transverse momentum observable.Source: A. Illustration of analysis techniques · pp. 17–18
  • The illustration simulates events with MadGraph5_aMC@NLO, showers and hadronizes them with Pythia 8, and models detector response with Delphes 3.The detector simulation uses the HL-LHC card suggested by the HL/HE-LHC working group.Source: A. Illustration of analysis techniques · p. 18

1. Signal only

In the one-dimensional examples, MadMiner estimates likelihood ratios and scores from simulated events, integrating over latent variables in the observable-level fit. The examples show agreement with histogram estimates while illustrating the effects of backgrounds and binning.

1. Signal only

  • For signal events, the joint likelihood ratio is higher for the BSM sample, which has more high-pT,γγ events; latent variables produce substantial scatter at fixed observable values.The Alices estimate fits the observable-level ratio by integrating out latent variables, while joint-score gradient information can help convergence with less data.Source: §1 · pp. 18–19

2. Adding backgrounds

  • Neural-network likelihood-ratio estimates generally agree excellently with histogram estimates, while avoiding the few-bin bias and many-bin fluctuations associated with limited Monte Carlo statistics.The Alices model learns the continuum limit corresponding to infinitely many histogram bins.Source: Figure 3 · p. 20, §2 · pp. 20–21
  • The Sally score estimate agrees excellently with histogram results, and background events form a horizontal score line in the one-dimensional comparison.The score is evaluated at the Standard Model reference point.Source: Figure 3 · p. 20, §2 · p. 21
  • Adding ttγγ backgrounds increases variance in the joint likelihood-ratio data, so more training events are needed for the neural network to converge.The inference technique remains fundamentally valid in the presence of backgrounds.Source: §2 · p. 20

B. Validation at parton level

The parton-level study tests whether MadMiner’s score and likelihood-ratio estimators recover known truth in a setting where all parton momenta and flavours are observable. Both estimators closely match their ground-truth values in this validation.

B. Validation at parton level

  • Making all latent variables observable equates the likelihood ratio and score with their joint versions, enabling direct comparison to ground truth.The validation omits parton shower and detector response; these approximations permit ground-truth calculation but are not required by MadMiner’s inference techniques.Source: B. Validation at parton level · p. 21
  • The validation setup makes the parton momenta and flavours of initial- and final-state particles observable.The study does not simulate parton showering or detector response, which would prevent access to the ground truth for realistic neutrinos or detector modelling.Source: B. Validation at parton level · p. 21, Figure 4 · p. 22
  • The study considers three dimension-six operators affecting top and Higgs couplings in ttH production.The parameter vector is θ = (cu, 100 cG, 100 cuG); two coefficients are rescaled, and event weights and cross sections are morphed from nine benchmark points.Source: B. Validation at parton level · pp. 21–22
  • The Sally score and Alices likelihood-ratio predictions closely match their ground truths in the parton-level validation.The comparison uses 1.25 million events; the score comparison focuses on t2(x), corresponding to θ2 = 100 cG.Source: B. Validation at parton level · p. 22, Figure 4 · p. 22

C. Realistic physics analysis

The realistic analysis applies score-based Fisher information to high-dimensional ttH event data, then compares sensitivity across parameter directions and analysis inputs. Kinematic information breaks a rate-only degeneracy, while systematic uncertainties mainly reduce sensitivity in the rate-sensitive direction.

C. Realistic physics analysis

  • The realistic study includes ttH signal and continuum ttγγ background, with the background sensitive to cuG but independent of cu and cG.It neglects subleading backgrounds, including fake photons or leptons, and includes Pythia 8 showering, Delphes 3 detector response, and PDF and scale uncertainties.Source: C. Realistic physics analysis · p. 23
  • The selected event data comprise 48 observables, and the analysis compares score or likelihood-ratio estimates with a pT,γγ histogram and total-rate baseline.For 3 ab−1, the stated selection yields 24.5 expected SM ttH signal events and 33.6 ttγγ background events.Source: C. Realistic physics analysis · p. 23

1. Fisher information

  • Near a reference point, squared Fisher distance approximates −2 times the expected log likelihood ratio and can be translated into an expected exclusion p-value.The approximation applies when the theory parameters are sufficiently close to the reference point.Source: §1 · p. 23
  • Kinematic information breaks the rate-only degeneracy, constraining the rate-sensitive direction almost as well as the rate measurement.A rate measurement alone constrains only one direction in theory space and is blind in the other.Source: §1 · p. 24, Figure 5 · p. 24
  • Systematic uncertainties, dominated by scale uncertainty, mainly reduce sensitivity in the rate-sensitive direction; the orthogonal direction is affected only slightly.The reduction is more pronounced for total-rate and pT,γγ-histogram information than for the full multivariate information.Source: §1 · pp. 24–25, Figure 5 · p. 24
  • The narrow 2σ bands across ten independently trained score estimators provide a sanity check on training stability.The networks use resampled training data and independent random seeds; wider bands would indicate numerical issues or insufficient training data.Source: Figure 5 · p. 24, §1 · p. 25
  • Fisher information on Ou peaks near pT,γγ = 100 GeV, while information on OuG and especially OG shifts toward the high-energy tail.The Ou peak marks a compromise between signal-to-background ratio and rate; the other operators’ kinematic effects are larger at high energy.Source: §1 · p. 25, Figure 6 · p. 25
  • The cG score is largest around the Higgs peak at pT,γγ ≳ 100 GeV, and background-dominated regions approach t2(x) = −0.58.The learned score is a non-trivial function of the high-dimensional observable space, not simply of mγγ and pT,γγ.Source: §1 · pp. 25–26, Figure 6 · p. 25

2. Exclusion limits

The exclusion limits show that kinematic information and the machine-learning methods can strengthen constraints beyond rate-only or single-histogram analyses. MadMiner packages the inference workflow and, in the ttH example, yields stronger expected limits than total-rate and standard-histogram analyses.

2. Exclusion limits

  • The full-likelihood and Fisher-information limits differ, indicating relevant dimension-six squared terms that contribute 1%–10% to the total rate in the 68% CL region and more in high-energy phase space.The full-likelihood analysis includes nonlinear effects and squared dimension-six amplitudes, unlike the local Fisher-information approximation.Source: §2 · pp. 26–27
  • For the pT,γγ analysis, shape information complements rate information by removing rate-only blind directions, and the three methods’ results agree closely.This agreement provides a cross-check among the histogram, Alices, and Sally approaches.Source: §2 · p. 27
  • Kinematic information breaks the rate-only degeneracy between disjoint best-fit regions, including the cu = 0 and cu = −4/3 solutions, and even a single-variable pT,γγ histogram improves sensitivity.The cu = −4/3 solution corresponds to a sign-flipped top Yukawa coupling with the same total cross section.Source: §2 · pp. 27–28
  • Alices gives the strongest expected Wilson-coefficient limits, while Sally’s score components outperform pT,γγ; Fig. 8 compares both with rate-only and histogram limits.The Alices advantage away from the Standard Model reference point is conditional on successful network training.Source: Figure 8 · p. 27, §2 · p. 28

V. CONCLUSIONS

  • MadMiner implements all steps in the analysis chain and supports high-dimensional event data without low-dimensional summary statistics, modeling realistic shower and detector simulation without approximations to underlying physics.In a high-luminosity LHC example analysis of three effective operators in tth production, its machine-learning techniques yielded stronger expected limits than total-rate and standard-histogram analyses.Source: §V · p. 28

Appendix A: Frequently asked questions

The FAQ explains how simulated event histories carry likelihood information and how the methods are used and checked. It also states limits tied to simulation quality and the construction of exclusion limits.

Appendix A: Frequently asked questions

  • An event history spans a simulated collision through showering and detector interactions; the joint likelihood ratio and score quantify how its likelihood changes as theory parameters vary.A given history can be compatible with different parameter values while being more or less likely under them.Source: Appendix A · p. 29
  • Networks use full-simulation observables as inputs and parton-level joint likelihood-ratio or score labels, which are unbiased but noisy estimators of their true counterparts including shower and detector effects.The paper states that a network trained appropriately can therefore learn shower and detector effects.Source: Appendix A · p. 29
  • For signal-background classification, parameter-dependent hypotheses can make score-based methods useful; when signal and background share phase space, joint-ratio or score methods need less training data, but otherwise offer no advantage over binary classifiers.For hypotheses without additional parameters, Carl, Rolr, or Alice techniques can learn event-level signal or background probabilities.Source: Appendix A · pp. 29–30
  • Mismodeling can be handled with quantified nuisance uncertainties or suitable cuts in problematic regions, but wholly untrustworthy simulations with unquantifiable uncertainty do not support accurate results.A data-driven normalization is also suggested when kinematics are trusted but the overall normalization is not.Source: Appendix A · p. 30
  • Although a neural network’s internal structure is not directly interpretable, its learning target is connected to the matrix element, and the paper recommends multiple checks of successful training.Listed checks include validation loss, independent network seeds, and comparisons of estimated score or likelihood-ratio expectations with known values.Source: Appendix A · p. 30
  • Neyman limits built with toy experiments have a coverage guarantee even if the estimated likelihood, ratio, or score is wrong; the stated risk is lost statistical power, not excessive optimism.The passage contrasts this guarantee with limits based on asymptotic likelihood-ratio properties.Source: Appendix A · p. 30

Related papers

Loading 1907.10621v2…