The CMS statistical analysis and combination tool: COMBINE

CMS Collaboration

Source-linked AI summary

The CMS statistical analysis and combination tool: COMBINE

CMS Collaboration

arXiv:2404.06614v2Apr 9, 2024physics.data-anhep-ex

TL;DR

Statistical analyses require a flexible and reusable framework for modeling data, uncertainties, and diverse physics questions. This paper presents COMBINE, a configurable tool built around datacards and physics models, and documents its statistical procedures and applications. COMBINE supports likelihood-based analyses across CMS use cases, while its coverage is bounded by assumptions such as approximately correct interval coverage absent large non-Gaussian uncertainties and special treatment near physical boundaries.

Problem
Statistical analyses require a general tool that can support diverse searches and measurements while representing statistical models, parameters of interest, and systematic uncertainties.
Method
COMBINE constructs likelihood-based statistical models from user-provided datacards and configurable physics models, then provides command-line workflows for multiple analysis procedures.
Results
COMBINE has become the main CMS statistical-analysis tool and supports confidence or credible intervals, profile likelihoods, goodness-of-fit tests, and combined analyses.
Takeaways & Limitations
Different physics models can be applied to the same datacard, facilitating reinterpretation in terms of different parameters of interest.
Takeaways & Limitations
Confidence intervals have approximately correct coverage without large non-Gaussian uncertainties, while physical-boundary best fits require Feldman–Cousins intervals for improved coverage properties.

Abstract

from arXiv · show

This paper describes the COMBINE software package used for statistical analyses by the CMS Collaboration. The package, originally designed to perform searches for a Higgs boson and the combined analysis of those searches, has evolved to become the statistical analysis tool presently used in the majority of measurements and searches performed by the CMS Collaboration. It is not specific to the CMS experiment, and this paper is intended to serve as a reference for users outside of the CMS Collaboration, providing an outline of the most salient features and capabilities. Readers are provided with the possibility to run COMBINE and reproduce examples provided in this paper using a publicly available container image. Since the package is constantly evolving to meet the demands of ever-increasing data sets and analysis sophistication, this paper cannot cover all details of COMBINE. However, the online documentation referenced within this paper provides an up-to-date and complete user guide.

1 Introduction

COMBINE provides command-line workflows and human-readable datacards for constructing statistical models in high-energy physics. Originally developed for Higgs searches, it has become a generic tool supporting diverse analyses and combinations when data sets are statistically independent.

COMBINE provides command-line interfaces for common high-energy-physics statistical workflows and represents statistical models in human-readable datacards.These features support methodological consistency and efficient investigation of potential issues without limiting analysis complexity.
COMBINE can combine likelihoods to increase sensitivity in searches or measurements when the data sets are statistically independent.
Originally developed for Higgs-boson searches in proton-proton collisions, COMBINE now supports broader statistical analyses beyond Higgs measurements.Its later applications include supersymmetry searches, Higgs-property measurements, and Standard Model parameter measurements.
The paper summarizes COMBINE's statistical methods and capabilities, while directing readers to online documentation for complete and up-to-date guidance.
The paper covers installation, statistical models, analysis types, physics models, execution, and procedures such as likelihood estimation, interval calculation, and goodness-of-fit testing.

2 Installation

COMBINE can be built within CMSSW or as a standalone package, with dependencies including ROOT, ROOFIT, and several numerical libraries. A precompiled Docker image provides an alternative installation route and includes reproducible examples.

The package relies on ROOT, ROOFIT, and additional libraries including VDT, GSL, EIGEN, BOOST, and GZIP.
COMBINE may be compiled within a CMSSW environment or as a standalone package.CMSSW provides a versioned set of dependencies.
A precompiled COMBINE version is available as a Docker container image.
At writing, COMBINE v9.2.0 and the example datacards and inputs were available in the container's data/tutorials/CAT23001 directory.Results involving random sampling should be consistent across runs but not identical.

3 The statistical model

COMBINE constructs a probability model for observed data using primary observables, parameters of interest, nuisance parameters, and auxiliary measurements. Its likelihood assumes statistical independence and supports both Bayesian and frequentist calculations.

COMBINE's primary task is to construct a statistical model p(data;⃗Φ) encoding the probability density of observed data through model parameters.
The parameter space is divided into parameters of interest and nuisance parameters that represent theoretical and experimental uncertainties.Examples include uncertainties in process cross sections and luminosity calibration.
The resulting likelihood is used in both Bayesian and frequentist calculations, including early upper-limit determinations for the Higgs-boson cross section.
Auxiliary observables provide probability distributions for nuisance parameters and can be reinterpreted as posterior distributions used when generating pseudo-data.COMBINE assumes uniform priors for nuisance parameters.
Primary observables are organized into statistically independent channels, which may be event counts in final states or continuous observables such as invariant masses.

4 Supported analysis types

COMBINE uses human-readable datacards to specify statistical models for counting and shape analyses, including observables, probability densities, processes, channels, and uncertainties. These inputs can be converted, combined, and extended with nuisance parameters that propagate systematic effects into the model.

Datacards: Datacards define observables, their probability densities, observed data, and whether the analysis uses counting, binned, or unbinned distributions.They are COMBINE’s primary plain-text input and can describe pdfs with histograms or parametric functions.
Datacards: COMBINE converts datacards into binary ROOT RooWorkspace representations, either through text2workspace.py or automatically when a datacard is supplied.The output uses the input datacard name with a .root extension unless -o specifies another name.
Datacards: Multiple datacards can be combined into one model while preserving the readability of their individual components.Same-named nuisance parameters are treated as one parameter when their auxiliary-observable pdfs match.
Counting analyses: Counting analyses model one channel’s total event count with a Poisson distribution whose expected yield can depend on signal and background parameters.A counting datacard declares channels, processes, rates, observations, and systematic uncertainties affecting those rates.
Counting analyses: Counting-analysis uncertainties are encoded through nuisance parameters and auxiliary probability terms, with log-normal and gamma forms used for common rate and simulation-statistics uncertainties.Datacard entries specify the uncertainty name, effect, and associated p(y; ν); examples include luminosity, signal cross section, and limited simulated-event uncertainties.

5 Physics models

COMBINE separates the statistical model encoded in a datacard from the physics model that defines parameters of interest and their effects on signal processes. This supports alternative parameterizations, including generic and Higgs-specific models, while allowing additional constraints on model parameters.

Physics-model construction: Physics models define parameters of interest and how they affect signal processes declared in a datacard.Different physics models can be used with the same datacard, enabling reinterpretation in terms of different parameters of interest.
Default model: The default physics model uses one parameter of interest, r, to multiply the rate of every signal process.This model is selected when the -P option is not specified.
Model construction: The statistical model is fully determined after COMBINE combines the datacard with the selected physics model.A multi-signal datacard can represent multiple signal processes, channels, and systematic uncertainties within the resulting model.
Generic models: Generic models are implemented in Python classes that define parameters of interest and the dependence of signal and background yields on them.The package includes example models, including floatingXSHiggs for several dominant Higgs production modes.
Higgs models: The floatingXSHiggs model can define separate parameters for production modes such as ggH, qqH, VH, WH, ZH, and ttH.The parameter ranges are specified in the same Python class, and each parameter is associated with a production process through a model function.
Parameter constraints: Datacards can include generic constraints that multiply the statistical probability by additional constraint terms.These constraints can impose restrictions from external theoretical or experimental information and have been used for regularization in unfolded measurements.

6 How to run COMBINE

The combine executable performs statistical routines using a model supplied through a datacard or ROOT file. Users select the routine with the -M option and can inspect available methods through --help.

Running COMBINE: The combine executable performs statistical routines using the statistical model constructed by COMBINE.It accepts either a text datacard or ROOT input file.
Command syntax: The command combine <datacard.[txt|root]> -M <Method> selects the input model and statistical calculation.The Method argument specifies which calculation to perform.
Command syntax: The --help option displays the available method and executable options.This provides the command-line list of supported choices before running a calculation.

6.1 Generic minimizer options

Many COMBINE methods optimize a profile negative-log-likelihood by profiling nuisance parameters at fixed parameter-of-interest values. CascadeMinimizer supports sequential algorithms and discrete nuisance-parameter minimization.

Likelihood optimization: Many COMBINE methods use numerical optimization of the likelihood function.These methods typically operate on the profile negative-log-likelihood.
Likelihood optimization: The profile negative-log-likelihood profiles nuisance parameters by maximizing the likelihood at fixed parameter-of-interest values.The profiled nuisance parameters are the values that maximize the likelihood for each fixed parameter set.
Minimization control: CascadeMinimizer performs sequential minimization using different algorithms and can minimize over discrete nuisance parameters.This supports methods such as discrete profiling.
Minimization control: Table 4 lists the available combinations of minimizer and algorithms in COMBINE.The table serves as a reference for the optimization configurations supported by the package.

6.2 Output from COMBINE

COMBINE reports most statistical results both in the terminal and in a ROOT file containing a TTree named limit. The output filename records method and user-selected metadata, while standard branches store the main result and its uncertainty.

Output files: Most COMBINE methods print computation results to the terminal and save them in a ROOT file containing a TTree named limit.This provides both immediate command-line output and a persistent machine-readable result.
Output files: Output files follow the pattern higgsCombine$NAME.MethodName.mH$MASS.[$WORD$VALUE].root.NAME, MASS, and optional keyword-value pairs are populated from command-line and datacard settings.
Output contents: The limit and limitErr branches always represent the main result of a statistical routine and its estimated uncertainty.These branch names retain that meaning even when the routine does not calculate a frequentist limit.
Output contents: Table 5 documents the TTree branches contained in COMBINE output ROOT files.The branch structure provides the schema for reading saved results.

6.3 Pseudo-data generation

COMBINE can use observed data by default or generate pseudo-data sets for statistical distributions and optimization studies. It supports ordinary random pseudo-data, Asimov data sets for expected frequentist outcomes, and configurable nuisance-parameter sampling.

COMBINE normally uses observed data but can substitute pseudo-data sets to study statistical quantities or perform analyses blind to observations.
The --toys <N> option generates N pseudo-data sets from the statistical model and runs the selected routine once for each set.Each pseudo-data set is constructed by generating random values of the observables.
The --toys -1 option produces an Asimov data set whose parameter estimates equal the values used to generate it.Asimov data sets estimate expected frequentist limits and confidence intervals more efficiently when valid.
Asimov data sets use expectation values for binned counting or shape analyses, or large samples of weighted events for unbinned analyses.Expected bin counts may be non-integer because factorial terms are omitted when constructing the likelihood.
By default, nuisance parameters are sampled from p(νk|yk), while --toysFrequentist switches to a parametric bootstrap for N > 0.
Pseudo-data generation can be separated from statistical analysis by saving data sets to a ROOT file and supplying them to COMBINE later.The paper presents these procedures as only part of COMBINE’s available functionality.

6.4 Frequentist limits and confidence intervals

COMBINE provides frequentist upper limits and confidence intervals through likelihood-based test statistics, pseudo-data distributions, and CLs-based searches. HybridNew supports exact pseudo-data calculations, while AsymptoticLimits uses asymptotic approximations.

HybridNew calculates upper limits and Feldman–Cousins intervals using test statistics based on the likelihood function.
COMBINE supports LEP-, TEV-, and LHC-style test-statistic prescriptions with different likelihood ratios and nuisance-parameter treatments.
The LHC-style statistic uses a profile-likelihood ratio and is set to zero when the maximum-likelihood estimate satisfies ˆµ > µ.
For each tested µ, COMBINE evaluates pµ and pb from pseudo-data distributions, computes CLs = pµ/(1 −pb), and finds the upper limit where CLs = α.A bisection algorithm searches for the value corresponding to the 100(1 −α)% confidence level.
The example HybridNew calculation gives Limit: r < 0.346362 +/- 0.0134581 @ 95% CL.
Figure 3 compares eqLHC(µ = 0.4) distributions for µ = 0 and µ = 0.4, while Figure 4 displays CLs interpolation and its uncertainty for the 95% CL limit.
COMBINE improves upper-limit accuracy by interpolating an exponential function across nearby CLs results, while AsymptoticLimits uses asymptotic approximations for eqLHC(µ).

6.5 Significance calculation

COMBINE estimates signal significance by generating pseudo-data under the background-only hypothesis, evaluating the observed test statistic, and converting p0 to a standard-normal significance. An asymptotic method avoids pseudo-data generation when applicable.

HybridNew estimates signal significance by generating pseudo-data under the background-only hypothesis µ = 0.The null hypothesis represents absence of the signal contribution.
The observed q0 is compared with the q0 distribution from pseudo-data under µ = 0 to determine the null p-value.
The example HybridNew calculation reports Significance: 2.54397 -0.0146063/+0.0151701 and Null p-value: 0.00548 +/- 0.000233452.
The Significance method converts p0 to a standard-normal significance using an asymptotic approximation for f(q0|µ = 0).This can avoid pseudo-data generation when the number of events is large.
The example asymptotic calculation reports Significance: 2.56729.
The output ROOT file stores significance and its estimated uncertainty in limit and limitErr, while --pval stores the p-value instead.

6.6 Bayesian upper limits and credible regions

COMBINE calculates Bayesian upper limits and credible regions from posterior probabilities using numerical or Markov-chain integration. Users can choose priors and average independent chains to estimate limits and their uncertainties.

Bayesian calculations in COMBINE are based on the posterior probability p(⃗µ), with the prior for parameters of interest specified by the user.
BayesianSimple uses numerical integration, whereas MarkovChainMC uses Markov-chain integration and marginalizes nuisance parameters for more complex models.
Available priors include the default flat prior, the inverse-square-root prior 1/sqrt(r), ROOT formula expressions, and RooFit probability-density objects.
Both Bayesian methods compute the 100(1 −α)% credible upper limit µup, with α controlled by the --cl=1 −α option.
The Datacard 1 example reports Limit: r < 2.21031 +/- 0.0133576 @ 95% credibility using 100 Markov chains.
Saving Markov chains with --saveChain enables estimation of posterior distributions and derivation of credible intervals or regions.

6.7 Maximum likelihood estimates and scans

COMBINE’s MultiDimFit method estimates parameters and confidence intervals through several scan algorithms based on the likelihood and profile-likelihood ratio. Examples with Datacard 5 illustrate one- and two-parameter fits, correlations, and algorithm-specific outputs.

Method: MultiDimFit evaluates the negative-log-likelihood function to obtain maximum likelihood estimates and confidence-interval estimates for parameters of interest.The method supports several approaches and has been used for CMS measurements.
Algorithms: The supported algorithms include none, singles, contour2d, random, fixed, and grid, each producing different parameter estimates, intervals, or scan outputs.The grid algorithm evaluates q(⃗µ) over a specified set of parameter points, while contour2d constructs two-dimensional regions.
Coverage: These interval algorithms have approximately correct coverage when large non-Gaussian uncertainties are absent; physical-boundary best fits call for Feldman–Cousins intervals.The boundary case is identified as requiring improved coverage properties.
Example results: For Datacard 5, the best-fit values are r_ggH = +0.882 -0.749/+0.795 and r_qqH = +4.683 -2.746/+3.464 at 68%.These values are reported from the singles algorithm using the floatingXSHiggs physics model.
Intervals: The reported measurement format is ˆµ −∆−µ/+∆+µ, with ∆±µ determined from the 68% confidence intervals.The nuisance-parameter fits distinguish default pre-fit values from maximum-likelihood post-fit values.
Example results: The r_qqH estimate is more precise than r_ggH because the dijet channel has a larger signal-to-background ratio, while contributions across channels correlate the parameters.The correlation is visualized through q(rggH, rqqH) and the corresponding two-dimensional confidence region.

6.8 Goodness of fit tests and diagnostics

COMBINE provides goodness-of-fit tests and diagnostics based on the statistical model, pseudo-data, likelihood fits, and nuisance-parameter impacts. The examples show how the tool evaluates compatibility, uncertainty constraints, and parameter sensitivity.

Goodness of fit: GoodnessOfFit tests compatibility between observed data and the statistical model, using the model as the null distribution without a well-defined alternative hypothesis.The observed statistic and its null distribution are obtained through separate commands, with pseudo-data used for the latter.
Goodness of fit: COMBINE supports saturated, Kolmogorov–Smirnov, and Anderson–Darling test statistics, all based on binned data.For parametric shape analyses, automatic binning uses 100 uniform bins per observable unless another binning is specified.
Goodness of fit: For Datacard 2, the saturated-test example gives a p-value of 0.73 from pseudo-data generated with the template analysis.The test-statistic distribution peaks near the number of observable bins, as expected for this statistic.
Diagnostics: ChannelCompatibilityCheck compares a nominal signal-rate fit with alternate fits that assign separate rates to the model’s channels.The output includes nominal and alternate estimates and a chi-squared-like compatibility variable.
Diagnostics: FitDiagnostics performs fits with the signal-strength parameter free and fixed to zero, returning its estimator, uncertainty, and additional fit information.Its outputs support comparisons between pre-fit and post-fit nuisance parameters and uncertainty constraints.
Diagnostics: Nuisance-parameter impacts quantify how shifting a nuisance parameter by its uncertainty changes a parameter-of-interest estimate, with sign indicating correlation direction.Large impacts typically identify nuisance parameters contributing substantially to the total uncertainty.

7 Summary

COMBINE is CMS’s main statistical-analysis tool, built around configurable statistical models and command-line workflows. It supports a broad range of inference and goodness-of-fit procedures, with comprehensive online documentation.

Summary: After a decade of development, COMBINE became the main tool used for statistical analysis of CMS data.The package is based on ROOT, ROOFIT, and ROOSTATS.
Summary: Users construct models from a text file and a configurable physics model containing parameters of interest and nuisance parameters for systematic uncertainties.The resulting model supports analyses of statistical data through a common configuration framework.
Summary: COMBINE performs confidence or credible interval calculations, profile-likelihood evaluations, and goodness-of-fit tests.The online documentation provides comprehensive capability and installation instructions.

Institut f¨ur Hochenergiephysik, Vienna, Austria

This section lists authors and institutional affiliations contributing to the paper, including universities, laboratories, and research centers across multiple countries.

Affiliations: The author list includes contributors affiliated with universities and research institutes in Belgium, Brazil, Bulgaria, Chile, China, Colombia, Croatia, Cyprus, the Czech Republic, and Egypt.The listed affiliations span Antwerp, São Paulo, Beijing, Hefei, Split, Prague, Cairo, and other locations.
Affiliations: Additional affiliations include institutions in Estonia, Finland, France, Georgia, Germany, and Greece.The entries identify universities, national laboratories, and specialized research centers.