Source-linked AI summary
HistFitter software framework for statistical data analysis
M. Baak, G. J. Besjes, D. Cote, A. Koutsman, J. Lorenz, D. Short
TL;DR
HistFitter addresses the need for flexible, scalable statistical analysis of large particle-physics datasets and implements the control-, validation-, and signal-region strategy in a programmable framework. It supports complex probability-density models, multi-model bookkeeping, statistical interpretation, and publication-quality presentation, while rigorous validation and shared uncertainties connect background fits to signal-region predictions. Its scope includes dependence of expected limits on observed data and best-effort support.
Problem
Large particle-physics analyses require flexible methods to manage complex models, many signal hypotheses, and large collections of histograms and auxiliary measurements.
Method
HistFitter uses programmable Python/C++ configurations and modular probability-density-function components to build, fit, validate, interpret, and present multiple data models.
Results
HistFitter provides integrated control, validation, and signal-region treatment, organized multi-model testing, statistical interpretation, and publication-ready tables and plots.
Takeaways & Limitations
The framework supports complete particle-physics analyses and large collections of signal-hypothesis tests through a unified, statistically rigorous workflow.
Takeaways & Limitations
Expected upper limits depend indirectly on observed data because observed-data background-level estimates are used for both observed and expected determinations.
Abstract
from arXiv · showhide
We present a software framework for statistical data analysis, called HistFitter, that has been used extensively by the ATLAS Collaboration to analyze big datasets originating from proton-proton collisions at the Large Hadron Collider at CERN. Since 2012 HistFitter has been the standard statistical tool in searches for supersymmetric particles performed by ATLAS. HistFitter is a programmable and flexible framework to build, book-keep, fit, interpret and present results of data models of nearly arbitrary complexity. Starting from an object-oriented configuration, defined by users, the framework builds probability density functions that are automatically fitted to data and interpreted with statistical tests. A key innovation of HistFitter is its design, which is rooted in core analysis strategies of particle physics. The concepts of control, signal and validation regions are woven into its very fabric. These are progressively treated with statistically rigorous built-in methods. Being capable of working with multiple data models at once, HistFitter introduces an additional level of abstraction that allows for easy bookkeeping, manipulation and testing of large collections of signal hypotheses. Finally, HistFitter provides a collection of tools to present results with publication-quality style through a simple command-line interface.
1 Introduction
HistFitter is a programmable framework for constructing, fitting, interpreting, and presenting complex statistical data models. It integrates particle-physics analysis regions, multi-model bookkeeping, and publication-quality output.
- HistFitter performs complete statistical analyses from a single user-defined configuration file by coherently combining tools from several sources.
- Its built-in control, signal, and validation regions constrain, extrapolate, and validate data-model predictions with statistically rigorous methods.
- HistFitter tracks numerous data models and their input histograms before and after adjustment, enabling organized tests and parameter scans across signal hypotheses.
- It provides methods for evaluating signal significance, assessing likelihood-fit quality, and producing publication-quality tables and plots.
- The paper details HistFitter’s extensions, including programmable model construction, multiple probability-density-function instances, statistical fits, and fit-result presentation.
2 Data analysis strategy
The analysis strategy uses control, validation, and signal regions to normalize backgrounds, test extrapolations, and estimate signal-region yields. HistFitter implements this strategy through shared model parameters, transfer factors, and statistically rigorous validation.
- Particle-physics analyses use simulated and data-driven background and signal predictions to interpret observations and search for unobserved processes.
- Signal regions are selected for potential excesses, while control regions constrain dominant backgrounds and are designed for high purity with minimal signal contamination.
- Validation regions test background predictions between control and signal regions, balancing statistical significance, signal contamination, and extrapolation assumptions.
- HistFitter shares probability-density-function parameters across control, signal, and validation regions, allowing simultaneous modeling and fitting of these regions.
- After control-region normalization, HistFitter propagates model modifications to validation and signal regions, while validation regions remain excluded from the fit constraint.
- Signal-region comparisons are conventionally made only after satisfactory validation-region agreement, a procedure used to validate extrapolations and avoid premature prediction-driven bias.
- Transfer factors convert observed control-region counts into signal-region background estimates using ratios of simulated signal-region and control-region contributions.
- Systematic uncertainties can partially cancel in transfer-factor extrapolations, leaving control-region statistical and residual extrapolation uncertainties in signal-region predictions.
3 HistFitter software framework
HistFitter is a configurable framework that turns input data and user-defined models into fitted, tested, and presented statistical analyses. Its bookkeeping and model-management design supports complex analyses and collections of signal hypotheses.
- Processing sequence: HistFitter processes user-defined configurations and input data through histogram generation, PDF construction, fitting, statistical testing, and result presentation.Histograms are prepared with ROOT, PDFs are constructed by HistFactory and stored with datasets and configurations, then RooFit and RooStats perform fits and tests.
- Processing sequence: A single configuration file can run processing steps individually or consecutively, supporting transitions from cut selection to statistical significance determination.This flexibility allows analyses to regenerate only the histograms needed during early development or proceed from data description to higher-level statistical results later.
- Configuration and bookkeeping: HistFitter’s configuration and bookkeeping machinery tracks multiple fit configurations containing PDFs, region data, and metadata for building, fitting, visualizing, and interpreting models.The configuration manager can hold any number of fitConfig objects, each representing one complete data-model workflow.
- Configuration and bookkeeping: By generating lists of data models, HistFitter enables organized hypothesis tests over grids of signal models through an additional abstraction layer.The configManager acts as a factory of factories, producing fitConfig objects that construct PDF objects.
- Configuration and bookkeeping: HistFitter reduces memory use by identifying and reusing histograms shared across models, while external storage avoids regenerating them during reruns.Background samples are often shared across models, whereas signal samples tend to be unique; cached histograms also support workload sharing.
- Statistical modeling: HistFactory models nominal predictions and systematic variations for multiple signal and background processes across multiple regions, with nuisance parameters interpolating systematic templates.The likelihood combines Poisson event-count terms with distributions constraining systematic uncertainties, while signal and background predictions are kept positive.
4 Programming of Probability Density Functions
HistFitter uses programmable, object-oriented components to construct probability density functions of nearly arbitrary complexity. Its channels, samples, and systematic uncertainties can be assembled flexibly across analyses and input formats.
- Fit configuration: HistFitter’s fitConfig class constructs highly complex PDFs from channels, samples, and systematic uncertainties with intuitive configuration code.A fitConfig combines these objects with input histograms and steers HistFactory’s creation of a RooSimultaneous PDF.
- Object assembly: The trickle-down mechanism adds samples and systematics to multiple channels or samples, simplifying complex PDF configurations.Users can add objects directly to fitConfig, while identical names declare correlations among samples or systematic uncertainties.
- Channels: Channels represent CRs, SRs, or VRs and may contain event counts or binned distributions defined by selections and observable ranges.HistFitter simultaneously fits CR and SR channels while excluding VRs from the fit constraint.
- Samples: Samples link PDF components to raw data and support TTree, Float, and Histogram inputs, which are converted into histograms before PDF construction.TTree inputs provide flexibility but require more processing and disk I/O; Float inputs support quick tests, while Histograms support external frameworks and rebuilding PDFs.
- Systematic uncertainties: HistFitter provides multiple systematic methods, including extensions derived from basic HistFactory methods for ATLAS Run-1 use cases.Systematic objects can be defined for individual samples or propagated across multiple samples and channels.
5 Performing fits
HistFitter supports background-only, model-dependent signal, and model-independent signal fits using combinations of control, signal, and validation regions. Shared parameters and expanded fit results propagate background estimates and uncertainties across regions for rigorous validation and testing.
- Common fit strategies: HistFitter supports three common strategies: background-only, model-dependent signal, and model-independent signal fits.These strategies support validation and hypothesis testing with different assumptions about signal models and region usage.
- Background-only fit: The background-only fit estimates backgrounds in SRs and VRs from CR data without assuming a signal model or using SR and VR observations.Its predictions therefore allow unbiased comparisons between predicted and observed event counts in those regions.
- Model-dependent signal fit: The model-dependent signal fit simultaneously uses CRs and SRs to study a specific signal model, setting exclusion limits without excesses or measuring signal properties when excesses occur.Multiple statistically independent, non-overlapping SRs can be combined, generally improving or preserving exclusion sensitivity.
- Model-independent signal fit: The model-independent signal fit uses CRs and SRs with a dummy signal and a single-bin SR to set upper limits without assumptions about signal distribution across bins.The same strategy supports background-only hypothesis tests that quantify excess significance independently of a particular signal model.
- Extrapolation and validation: RooExpandedFitResult enables shared and unshared nuisance parameters, correlations, and uncertainties to be extrapolated across regions for rigorous VR cross-checks.Changes or uncertainty reductions from the CR fit are automatically reflected in validation-region predictions.
6 Presentation of results
HistFitter provides generalized visualization, plotting, and table-generation tools for inspecting fits, uncertainties, validation regions, and background predictions. These outputs support both analysis workflows and publication-quality presentation.
- Presentation tools: HistFitter provides generalized scripts and plotting functions for every built model, callable through single-line commands.The tools require no further coding for each model.
- Visualization of fit results: Fit-result visualizations can show multi-bin control regions before and after fitting, with adjusted background normalizations, shapes, and uncertainties.The post-fit distributions are adjusted to best describe observed data across all bins.
- Visualization of fit results: Validation-region pull distributions compare observed and predicted event counts using systematic and statistical uncertainties combined in quadrature.Average negative or positive pulls indicate overestimated or underestimated background predictions, respectively.
- Event yield and uncertainty tables: HistFitter produces publication-ready tables containing observed events, fitted background components, and systematic-uncertainty breakdowns.Example tables report predictions across signal regions and distinguish total from component uncertainties.
- Event yield and uncertainty tables: The framework includes methods for propagating or refitting nuisance parameters to determine systematic uncertainties on background predictions.The two methods use covariance propagation or quadratic differences between nominal and nuisance-fixed fits.
7 Interpretation of results
HistFitter supports several frequentist hypothesis tests for signal models, model-independent limits, and potential excesses. Its tools scan signal strengths, save grid results, and visualize observed and expected exclusions.
- Statistical tests: HistFitter provides four statistical tests using fit setups that include both control and signal regions.The tests are implemented through RooStats classes and their results can be interpreted in plots and tables.
- Signal model hypothesis test: Signal-model tests evaluate a specified new-physics model against the background-only assumption using a signal-strength parameter.A signal strength of zero represents background only, while one represents background plus signal.
- Signal model hypothesis test: HistFitter collects observed and expected CLs values and p-values across multiple signal scenarios, retaining results only after a successful initial free fit.The scenarios can form a model grid by varying parameters of a supersymmetry model.
- Signal model hypothesis test: 95% confidence-level exclusion limits can be displayed over two-dimensional SUSY model grids using the CLs prescription.The axes represent gluino and neutralino masses, with expected and observed contours shown separately.
- Signal strength upper limit: Signal-strength upper limits are found by scanning CLs values, refining the interval, and converting the resulting limit into an excluded cross-section limit.A 95% CL upper limit usually corresponds to a CLs threshold of 5%.
- Model-independent upper limit: Model-independent fits constrain the number of beyond-the-Standard-Model events in each signal region and can yield visible-cross-section limits.Visible cross section is defined as acceptance multiplied by reconstruction efficiency and production cross section.
8 Public release
HistFitter is publicly available with source-code documentation, setup guidance, and working examples.
- Public release: The HistFitter software package is publicly available through its website and includes code documentation, a tutorial, and working examples.The package requires ROOT release v5.34.20 or greater.
9 Conclusion
HistFitter is a programmable framework for constructing, fitting, testing, and presenting complex data models. Its particle-physics region strategy, multi-model support, and command-line presentation tools were developed for ATLAS analyses.
- Conclusion: HistFitter builds and tests data models of nearly arbitrary complexity from user-defined input configurations.It automatically constructs PDFs, fits them to data, and interprets them with statistical tests.
- Conclusion: The framework integrates control, signal, and validation regions with statistically rigorous treatment tailored to complete particle-physics analyses.These concepts are embedded in the framework's design rather than handled as separate add-ons.
- Conclusion: HistFitter supports simultaneous work with multiple data models, aiding searches for new phenomena in large experimental datasets.This capability introduces an additional level of abstraction for such analyses.
- Conclusion: Command-line tools produce publication-style presentations of analysis results based on experience with ATLAS Run-1 data.The framework includes tools and options for presenting end results.
A Example configuration
The example configuration builds a single-bin SR counting experiment with background, data, systematic uncertainties, and separate discovery and exclusion fit configurations. It also specifies input files, event selections, weights, statistical-test settings, and signal models.
- The example defines a single-bin counting experiment with one channel called SR.
- Two background samples, A and B, and a data sample, Data, are extracted from fileA.root and fileB.root.
- The configuration applies treeSys and weightSys systematic uncertainties, with weightSys applied only to sample A, plus a luminosity uncertainty applied to both background samples.
- Separate Discovery and Exclusion fit configurations are created, using a dummy signal for discovery and a specific Signal sample for exclusion.
- The model specifies input luminosity scaling, event-selection cuts, nominal event weights, sample colours, and asymptotic frequentist upper-limit settings with 20 scan points.
- The Exclusion configuration adds a theory-normalized Signal sample from signal.root with norm factor mu_SIG constrained between 0 and 5.