Source-linked AI summary
Approximating Likelihood Ratios with Calibrated Discriminative Classifiers
Kyle Cranmer, Juan Pavez, Gilles Louppe
TL;DR
Simulators can link theoretical parameters to complex observations while leaving likelihood evaluation unavailable. The paper uses calibrated, parameterized discriminative classifiers to approximate likelihood ratios, with conclusions tied to classifier effectiveness and calibration.
Problem
Simulators describe complex processes linking parameters θ to observations, while likelihood evaluation is unavailable.
Method
The paper reformulates generalized likelihood-ratio tests using a parameterized family of discriminative classifiers trained and calibrated with a simulator.
Results
The calibrated classifier family can approximate the likelihood ratio when the likelihood is unavailable.
Takeaways & Limitations
The approach provides a classifier-based reformulation of generalized likelihood-ratio testing.
Takeaways & Limitations
Approximation quality depends on either the calibrated ratio approximation being good or the discriminator being effective.
Abstract
from arXiv · showhide
In many fields of science, generalized likelihood ratio tests are established tools for statistical inference. At the same time, it has become increasingly common that a simulator (or generative model) is used to describe complex processes that tie parameters $θ$ of an underlying theory and measurement apparatus to high-dimensional observations $\mathbf{x}\in \mathbb{R}^p$. However, simulator often do not provide a way to evaluate the likelihood function for a given observation $\mathbf{x}$, which motivates a new class of likelihood-free inference algorithms. In this paper, we show that likelihood ratios are invariant under a specific class of dimensionality reduction maps $\mathbb{R}^p \mapsto \mathbb{R}$. As a direct consequence, we show that discriminative classifiers can be used to approximate the generalized likelihood ratio statistic when only a generative model for the data is available. This leads to a new machine learning-based approach to likelihood-free inference that is complementary to Approximate Bayesian Computation, and which does not require a prior on the model parameters. Experimental results on artificial problems with known exact likelihoods illustrate the potential of the proposed method.
1 Introduction
Simulators can connect theory parameters and measurement processes to high-dimensional observations, but their likelihoods are often impossible or impractical to evaluate directly. The paper shows that likelihood-ratio inference can be extended to this setting using calibrated discriminative classifiers.
- Simulators describe complex processes linking theory parameters and measurement apparatus to high-dimensional observations, while direct likelihood evaluation is often impossible or computationally impractical.
- The paper shows that likelihood ratios are invariant under dimensionality reductions to one dimension when the transformation is monotonic with the likelihood ratio.
- The method provides a machine-learning approach to likelihood-free inference that can operate in a frequentist setting without a prior over model parameters.
- Calibrated discriminative classifiers can construct equivalent generalized likelihood-ratio test statistics when training data come from a generative model.
- In the LHC context, simulator-generated data are used to approximate densities at parameter points, with interpolation used for parameterized likelihood-based tests.
- The approach extends classifier use from fixed binary hypotheses to composite hypotheses parameterized by θ within established likelihood-based inference techniques.
2 Likelihood ratio tests
Likelihood-ratio testing requires evaluating parameter-specific densities, but simulators may generate observations without exposing those densities. The paper shows that a one-dimensional classifier-based reduction can preserve likelihood-ratio information and support calibrated likelihood-free inference.
- Motivation: Likelihood-ratio tests compare parameterized data densities, but simulator-based models may provide samples without directly evaluable likelihoods.The motivating setting includes complex simulations and forward-only particle-detector simulation.
- Likelihood-ratio invariance: A reduction s: R^p → R preserves likelihood-ratio information when s is monotonic with the density ratio.Under this condition, the ratio can be recovered through the induced one-dimensional density.
- Classifier construction: The likelihood-ratio problem can therefore be recast as supervised classification between samples generated under θ0 and θ1.A classifier supplies the reduction map, and consistent probabilistic classification algorithms can learn it.
- Calibration: Calibration can recover the density ratio even when the learned classifier is imperfect up to a monotonic transformation.This motivates calibration of the classifier-score density rather than relying directly on (1 − s(x))/s(x).
- Limitations and model choice: If the learned score is not monotonic with the density ratio, the induced density may fail to factorize and produce artifacts in the ratio approximation.With enough training data, tree-based models can nevertheless achieve accurate results through their universal-approximator capacity.
- Calibration: The approach factorizes learning into estimating a classifier score and its one-dimensional induced density, which is simpler than learning p(x|θ).The reduction retains the informative content of the ratio while projecting observations into one dimension.
3 Generalized likelihood ratio tests
The paper extends likelihood-ratio estimation to composite hypotheses and mixture models using classifier-based, one-dimensional density-ratio approximations. It also describes parameterized classification, calibration, and diagnostics for likelihood-free inference.
- Likelihood-ratio reduction: High-dimensional likelihood ratios can be reproduced from univariate classifier-score densities when the reduction is monotonic with the likelihood ratio.The reduction uses s(x; θ0, θ1) and compares p(s(x)|θ0) with p(s(x)|θ1).
- Composite hypotheses: Composite hypothesis testing requires maximizing likelihoods over parameter spaces for both null and alternative hypotheses.The generalized likelihood ratio supports hypothesis tests and confidence intervals, including settings with nuisance parameters.
- Composite hypotheses: The maximum-likelihood estimate is preserved by maximizing a likelihood ratio against a fixed denominator whose support covers the model support.The parameterized transformation must remain monotonic with the corresponding likelihood ratio.
- Parameterized classification and calibration: Parameterized classifiers take observations and hypothesis parameters as inputs, enabling a smooth classification score across parameter values.A single learning stage is trained with inputs (x, θ0, θ1) and targets y, after which the generative model supplies calibration data.
- Mixture models: Mixture-model likelihood ratios can be decomposed into pairwise classification problems between component distributions.The decomposition relates the high-dimensional ratio to a calibrated ratio based on the univariate density of the corresponding classifier.
- Diagnostics: Approximation quality requires diagnostics because the true likelihood ratio is unavailable in the likelihood-free setting.Checking whether fitted quantities are independent of the denominator parameter θ1 can indirectly probe approximation quality, but indistinguishability can also reflect an ineffective discriminator.
4 Examples and applications
The paper evaluates calibrated classifier-based likelihood-ratio approximations on one-dimensional mixtures and multidimensional parameter-inference problems. Across these examples, calibration and mixture decomposition improve approximation quality, while approximate likelihoods support inference consistent with exact results.
- The examples use artificial problems with known exact likelihoods before considering likelihood-free inference with multidimensional data.
- 4.1 Likelihood ratios of mixtures of normals: Mixture decomposition lets classifiers focus on easier pairwise component comparisons, resulting in more accurate approximations.
- 4.1 Likelihood ratios of mixtures of normals: Calibrating the classifier score through univariate density estimation is key to obtaining accurate likelihood-ratio approximations.
- 4.1 Likelihood ratios of mixtures of normals: The decomposed approximation reproduces the overall distribution of the exact log-likelihood ratio, although low-probability artifacts and an underestimated maximum remain.
- Once trained and calibrated, the approximation can be applied to any dataset D, making ensemble tests computationally efficient.
- 4.1 Likelihood ratios of mixtures of normals: The approximate likelihood yields an unbiased maximum-likelihood estimator with essentially the same variance as the exact MLE in the mixture example.
- 4.2 Parameterized inference from multidimensional data: For five-dimensional data with two parameters, the method estimates (α̂ = 1.008, β̂ = −1.004) after 50 likelihood evaluations, near the true values α = 1 and β = −1.
- 4.2 Parameterized inference from multidimensional data: In the multidimensional example, well-trained and calibrated likelihood curves agree with the exact likelihood within the Gaussian-process uncertainty band, and the ROC curve is almost identical.
5 Related work
Prior work addresses likelihood approximation from generative samples, machine-learning hypothesis testing, and density-ratio estimation. The proposed approach instead uses a parameterized classifier while separating dimensionality reduction from calibration.
- Neal (2007): Neal’s approach estimates θ per observation through regression before composing the classifier, which can lose information because one observation may weakly identify θ.The paper contrasts this with using the full dataset and postponing classifier evaluation until θ is tested.
- Parameterized classifiers: The proposed method embeds θ directly in the classifier and evaluates it when the likelihood is evaluated, rather than eliminating θ-dependence.This avoids the regression-based composition discussed for Neal’s method.
- Theoretical distinction: The paper states that its result is exact for an ideal classifier and that classifier quality is factorized from density calibration.Inference can remain valid with a non-ideal classifier, although power may be lost.
- Machine-learning hypothesis testing: The work extends machine-learning likelihood-ratio estimation from the Neyman–Pearson setting to generalized likelihood-ratio tests.Related studies use i.i.d. samples from each hypothesis or scalar maps of high-dimensional features for other statistical goals.
- Density-ratio estimation: Unlike methods that model high-dimensional numerator and denominator densities, the proposed method estimates a one-dimensional projected density, which is described as easier.The projection preserves the informative content of the likelihood ratio under the paper’s construction.
- Density-ratio estimation: The method separates dimensionality reduction from calibration, whereas related approaches estimate the likelihood ratio directly from x in one step.The paper notes convergence results under some assumptions for some direct-estimation approaches.
6 Conclusions
The paper reformulates generalized likelihood ratio tests using parameterized discriminative classifiers calibrated with a simulator, enabling likelihood-ratio approximation without direct likelihood evaluation. The approach supports likelihood-free parameter inference in both Bayesian and frequentist settings, while calibration remains challenging when performed continuously over parameters.
- Parameterized discriminative classifiers trained and calibrated with a simulator can approximate likelihood ratios when p(x|θ) cannot be directly evaluated.
- The method provides an alternative to Approximate Bayesian Computation for likelihood-free parameter inference and does not require specifying a prior.
- Unlike approaches that learn the posterior conditional on D, the trained approach can be applied to any observed data D.
- The approach separates approximation of the target likelihood from calibration, allowing the former to leverage advances in supervised classification.
- Calibration is difficult to perform continuously in θ, with strategies depending on parameter dimensionality, likelihood complexity, and simulator-running issues.
A Probabilistic classification for building s
This appendix characterizes probabilistic classification through its optimal regression function and connects that function monotonically to density ratios. The result supports the dimensionality-reduction framework, while discrete zero-one-loss classifiers do not satisfy the required theorem conditions.
- For squared error loss, the best probabilistic classifier is the conditional probability P(Y = 1|X = x), expressed using class priors and class-conditional densities.
- With equal class priors, the optimal regression function simplifies to a ratio of the two class-conditional densities.
- The resulting regression function satisfies the theorem's conditions because it is a monotonic transformation of the likelihood ratio.
- Exponential, binomial log-likelihood, cross-entropy, and squared hinge losses similarly produce classifiers monotonic with the density ratio.
- Discrete-output classifiers minimizing zero-one loss do not satisfy the theorem's conditions.