PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Inference

Larry Wasserman, Aaditya Ramdas, Sivaraman Balakrishnan

arXiv:1912.11436v4math.STstat.MEstat.ML

TL;DR

The paper addresses the lack of generally valid inference for irregular models, where standard likelihood-ratio methods can have intractable limiting distributions. It introduces split likelihood-ratio procedures that use data splitting to construct tests and confidence sets with finite-sample guarantees. The framework applies to arbitrary parametric models and some nonparametric settings, with extensions to sequential inference and nuisance-parameter problems.

  • Problem

    Irregular models such as mixture models are difficult for standard inference because likelihood-ratio statistics can have intractable limiting distributions and guaranteed-coverage confidence sets may be unavailable.

  • Method

    The split LRT splits data, estimates from one part, and evaluates likelihood on the other to construct universal tests and confidence sets without regularity conditions.

  • Results

    The procedures provide finite-sample type-I-error and coverage guarantees, including for mixture models, while extending to some nonparametric and sequential settings.

  • Takeaways & Limitations

    Universal inference is most useful when standard asymptotic methods are difficult or impossible to apply, including complex composite-null tests and nonparametric confidence sets.

  • Takeaways & Limitations

    When the true density is outside the model collection, the universal confidence set cannot generally contain it; in regular models, the method may also be non-optimal.

Abstract

from arXiv · show

We propose a general method for constructing hypothesis tests and confidence sets that have finite sample guarantees without regularity conditions. We refer to such procedures as "universal." The method is very simple and is based on a modified version of the usual likelihood ratio statistic, that we call "the split likelihood ratio test" (split LRT). The method is especially appealing for irregular statistical models. Canonical examples include mixture models and models that arise in shape-constrained inference. Constructing tests and confidence sets for such models is notoriously difficult. Typical inference methods, like the likelihood ratio test, are not useful in these cases because they have intractable limiting distributions. In contrast, the method we suggest works for any parametric model and also for some nonparametric models. The split LRT can also be used with profile likelihoods to deal with nuisance parameters, and it can also be run sequentially to yield anytime-valid $p$-values and confidence sequences.

1 Introduction

Standard likelihood-ratio and asymptotic-Gaussian inference relies on large samples and regularity conditions, leaving irregular models without a general method. Mixture-model inference illustrates the difficulty: likelihood-ratio statistics can have intractable limiting distributions, while the paper proposes finite-sample-valid procedures.

  • Likelihood-ratio tests and asymptotically Gaussian confidence intervals are standard tools whose validity often depends on large-sample theory and regularity conditions.
  • When regularity conditions fail, statistical inference is typically handled ad hoc because no general method is available.
  • Mixture models are irregular settings where constructing tests and confidence intervals is particularly difficult.
  • For testing mixture orders, the likelihood-ratio statistic has an intractable limiting distribution, making provable type-I-error control elusive.
  • The paper introduces a universal method intended to provide tests and confidence sets for arbitrary parametric models and some nonparametric models.

2 Universal Inference

Universal inference splits data so one part constructs an estimator and the other evaluates likelihood, yielding confidence sets and tests with finite-sample guarantees. The split LRT extends across parametric and some nonparametric settings, while crossfitting, profile likelihoods, and sequential use broaden the framework.

  • The method splits data into D0 and D1, using an estimator from D1 and likelihood calculations on D0 to construct universal confidence sets.The initial estimator may be an MLE, Bayes estimator, robust estimator, or another estimator.
  • Theorem 1 guarantees finite-sample coverage of at least 1 − α for the universal confidence set.
  • The split LRT compares an estimator from D1 with the null MLE from D0, while crossfitting swaps the data roles and combines the resulting procedures.
  • Theorem 3 states that split and crossfit LRTs control type-I error at level α uniformly over the null.
  • The framework extends to nonparametric null and alternative classes without requiring parametric assumptions or a relationship between the classes.
  • The procedures are universal because their guarantees hold in finite samples without regularity conditions.
  • Their proof uses Markov’s inequality on the exponential of the statistic, yielding an exponential-tail argument rather than an asymptotic likelihood-ratio distribution.
  • Compared with classical likelihood-ratio sets, split procedures evaluate an out-of-sample likelihood, so threshold comparisons alone do not determine relative behavior.

3 Sanity Check: Regular Models

In regular models, the universal confidence set retains the optimal 1/√n diameter rate when the initial estimator is √n-consistent, although its high-dimensional radius can be larger than the classical set. Nonparametric diameter guarantees additionally require entropy and estimator-quality conditions.

  • Under regularity assumptions including identifiability, smoothness, compactness, and nondegenerate Fisher information, optimal confidence-set diameter is O(1/√n).
  • The universal confidence set achieves the same O(1/√n) rate when the initial estimator is √n-consistent.Under the theorem’s conditions, the MLE satisfies this consistency requirement, though the result also permits more general estimators.
  • Nonparametric Hellinger-diameter control requires the model’s bracketing entropy to be sufficiently small and the initial estimator to be sufficiently close to the truth.
  • Under the stated entropy and estimator conditions, Theorem 6 upper-bounds the split LRT confidence-set diameter by a rate involving η_n and ϵ_n.
  • In the multivariate Normal example, both classical and universal confidence sets have squared radii with the same scaling in dimension d and sample size n.
  • For large d and constant α, the universal set’s squared radius is about 4 times larger than the classical set’s.

4 Examples

The examples apply universal testing to mixture, dependence, shape-constrained, and sieve-based inference, while also exposing when data splitting can reduce usefulness.

  • Mixture models: Mixture-model testing remains difficult because the likelihood-ratio statistic has an intractable limiting distribution and finite-sample type-I-error control has been elusive.
  • Model selection using sieves: Sieve-based sequential testing supports model selection while controlling type-I error despite testing multiple hypotheses and stopping at a random index.The procedure can select mixture-component counts and other nested models.
  • Nonparametric example: Shape constrained inference: For log-concavity testing, the universal method provides, to the authors’ knowledge, the first finite-sample-guaranteed test for the problem.The null MLE is computable in polynomial time, while membership in the resulting confidence set can be checked efficiently even if the set is hard to represent explicitly.
  • Positive dependence (MTP2): The universal test applies to Gaussian MTP2 and Ising MTP2 testing because the required null MLEs can be computed efficiently.The method also extends to testing MTP2 log-concave densities against log-concave alternatives.
  • Crossfitting can beat splitting: uniform distribution: In the uniform model, the split confidence set can be [0, ∞) with probability 1/2, whereas crossfitting dominates splitting in this example.The split and crossfit intervals are nevertheless rate-optimal up to a constant factor, with expected length O(θ log(1/α)/n).

5 De-randomization

The method’s random data split can make inferences unstable, so the paper develops multi-split variants that preserve validity while reducing split-induced variability. These variants trade computation and, potentially, power or flexibility against stability.

  • Split variability: The universal method’s random split can make final inferences unstable because different splits may produce different results.This instability is described as a “p-value lottery.”
  • Validity-preserving aggregation: Multiple split statistics can be combined while preserving the key expectation bound Eθ∗[Tn] ≤1, regardless of dependence between them.This permits inference with a combined statistic without requiring independent splits.
  • Variants: K-fold inference averages likelihood-ratio statistics after estimating on complementary folds, with alternative choices for which fold evaluates the ratio.The paper presents this as a natural variant of the universal method.
  • Variants: Subsampling repeats random K-fold procedures to reduce algorithmic variance, whereas all-splits removes split randomness but is computationally infeasible.The paper presents all-splits as statistically promising despite its computational cost.
  • Practical trade-offs: The flexibility to average variants creates a risk that analysts cherry-pick the variant producing the highest likelihood ratio after inspecting results.The passage identifies this as a broader issue associated with choosing among averaging procedures.
  • Practical trade-offs: Cross-fitting can substantially outperform splitting in the uniform example, although the authors do not recommend universal tests there because standard intervals are well studied.The example is intended to contrast split and crossfit approaches rather than motivate universal inference in that model.

6 Extensions

The universal framework extends beyond the basic split LRT to profile likelihoods, computational relaxations, robustified and smoothed likelihoods, and conditional likelihoods for non-i.i.d. data. These extensions preserve the central inference strategy while broadening the models and parameters it can address.

  • Profile likelihood and nuisance parameters: Profile likelihoods let the split-likelihood approach construct confidence sets for a target component while treating other parameter components as nuisance parameters.For θ=(θu,θn), setting g(θ)=θu yields inference for θu alone.
  • Computational relaxations: Convex relaxations can make otherwise nonconvex likelihood optimization more tractable while yielding a valid split LRT through an upper-bounded test statistic.The cited construction uses a relaxation F0 of the null likelihood L0.
  • Computational relaxations: Upper-bounding the null maximum likelihood is sufficient for performing universal inference, so exact computation of the null MLE is not always necessary.The paper connects this idea to relaxations of difficult optimization problems.
  • Robustness: Powered likelihoods Lη with 0<η<1 can replace the ordinary likelihood, allowing the universal methods to be used with a robustified likelihood.The extension relies on the nonnegativity of the η-Renyi divergence.
  • Smoothed likelihoods: Smoothed likelihoods provide an alternative when the MLE is inconsistent or does not exist because the likelihood is unbounded.The smoothed MLE is defined as a KL projection and supports an analogous smoothed split LRT.
  • Non-i.i.d. data: Conditional likelihoods extend the construction to non-i.i.d. settings whenever the likelihood on one data block can be calculated conditional on the other.Sampling without replacement from a large urn is given as an example.

7 Misspecification, and convex model classes

For convex model classes, the universal method provides results under misspecification and supports practical testing when maximum likelihood can be computed or bounded. Under misspecification, coverage centers on a KL projection only when that projection lies in the model.

  • Convex model classes: Convex model classes include mixtures, moment-constrained, monotonic, unimodal, and symmetric density families where maximum likelihood may be computable by convex optimization.The paper lists several examples of convex density classes and notes that convex optimization can make universal procedures applicable.
  • Misspecification: Under misspecification, the universal confidence set is analyzed relative to the reversed information projection of the true density onto the convex model.The reversed information projection minimizes KL divergence from the true density over the model.
  • Misspecification: The universal set cannot generally contain the reversed information projection because that projection need not belong to the model, but it covers it when it does.This is a scope condition on the misspecification guarantee.
  • Misspecification: If the data arise from q outside the model and a minimizing density p∗ lies inside the convex model, then Pq(p∗∈Cn) ≥1−α.This is the stated coverage guarantee under the specified convexity and projection conditions.
  • RIPR split LRT: The RIPR split LRT controls type I error for convex density sets, with supθ0∈Θ0 Pθ0(Rn>1/α)≤α.The result applies when the convex-set test is defined and the relevant likelihood construction is available.
  • RIPR split LRT: When implementable and when the RIPR lies in the null set, the RIPR split LRT can be more powerful than the MLE split LRT.The MLE split LRT remains more broadly applicable because it requires fewer convexity and computational conditions.

8 Anytime p-values and confidence sequences

Sequential versions of the split LRT use likelihood-ratio processes and martingale arguments to support inference over time. They yield anytime-valid p-values and confidence sequences that remain valid under stopping, peeking, and post-hoc time selection, although computation can grow with time.

  • Sequential extension: The split LRT has a sequential extension that produces anytime-valid p-values and confidence sequences.The construction is based on repeatedly updating likelihood-ratio statistics as data arrive.
  • Sequential testing: The running MLE LRT has type I error at most α, with supθ∗∈Θ0 Pθ∗(τθ∗<∞)≤α.Equivalently, with probability at least 1−α, the test never stops under the null.
  • Martingale validity: Under the null, the running likelihood-ratio statistic is bounded by a nonnegative martingale, which supplies the time-uniform validity argument.The null MLE maximizes the denominator among null parameters, yielding Mt≤Lt.
  • Martingale validity: Ville’s inequality converts the martingale bound into a guarantee against ever crossing the rejection threshold over time.The paper identifies this as a time-uniform analogue of Markov’s inequality for nonnegative supermartingales.
  • Anytime-valid p-values: Both pt=1/Mt and p̄t=min_s≤t 1/Ms can serve as p-values that are uniformly valid over time.The running-minimum version supports the event-based formulation of anytime validity.
  • Anytime-valid p-values: Anytime-valid p-values remain valid at data-dependent stopping times and arbitrary post-hoc random times, allowing peeking and optional continuation without repeated-testing correction.This validity follows from the sequential martingale construction.
  • Confidence sequences: The confidence sequences Ct and C̄t satisfy simultaneous coverage, including coverage at stopping times and arbitrary random times.The stated guarantee is Pθ∗(∃t:θ∗∉C̄t)≤α.
  • Further extensions: Sequential extensions also support nuisance parameters through profile likelihoods, non-i.i.d. data through conditional likelihoods, and robustness through powered likelihoods.The paper gives sequential t-tests with unknown Gaussian variance as one example.

9 Conclusion

The paper concludes that split likelihood ratio methods provide simple tests and confidence sets with finite-sample guarantees, especially when standard asymptotic methods are difficult to apply. It identifies open directions involving power, optimality, and semiparametric extensions.

  • Split likelihood ratio methods yield simple tests and confidence sets with finite-sample guarantees.
  • These methods are most useful for complex composite null testing and nonparametric confidence sets, where standard asymptotic methods are difficult or impossible to apply.
  • The authors plan simulations to study test power and confidence-set size across models.
  • They do not expect the test to be rate optimal in all cases, but suggest it may have properties analogous to the generalized likelihood ratio test.
  • Future work includes extending profile-likelihood variants to semiparametric problems with finite-dimensional parameters and infinite-dimensional nuisance parameters.

A.1 Proof of Claim (20)

The proof establishes local equivalence between Hellinger distance and parameter-space ℓ2 distance, then uses this relation and entropy bounds to control the split-LRT set.

  • The proof relates Hellinger distance between distributions to ℓ2 distance between parameters.
  • Hellinger distance is upper bounded by parameter ℓ2 distance throughout the parameter space.
  • Within a sufficiently small neighborhood of θ∗, Hellinger distance is also lower bounded by parameter ℓ2 distance.
  • The local equivalence lets the proof translate distributional neighborhoods into parameter neighborhoods near θ∗.
  • A local Hellinger bracketing-entropy bound, combined with Wong and Shen’s likelihood-ratio result, controls the split-LRT set’s diameter.

A.1.2 Proof of Lemma 14

The proof bounds local metric entropy in two regimes and uses likelihood expansions around the MLE to establish the needed split-LRT diameter control.

  • For small Hellinger radii, Lemma 13 converts Hellinger proximity into a parameter-space bound.
  • The small-radius entropy calculation is sufficient when the radius is at least C/√n.
  • For larger radii, compactness supplies a crude global metric-entropy bound without requiring a Hellinger-to-parameter lower bound.
  • Conditioning on an MLE error of order 1/√n, the proof expands the likelihood around θ∗ using the empirical-process term Gn and h = √n(bθ − θ∗).
  • The quadratic and empirical-process terms are both Op(1), completing the required likelihood bound.

B Proof of Theorem 6

The proof combines likelihood-ratio bounds with a high-probability event for the split estimator to exclude distributions sufficiently far from the truth from the confidence set.

  • The argument controls the Hellinger diameter dn of the split-LRT set.
  • It conditions on the event χ2(p∗, bp1) ≤ C1η2n, which holds with probability at least 1 − κ/2 for sufficiently large C1.
  • Wong and Shen’s theorem provides a uniform likelihood-ratio bound over distributions at least ϵn away from p∗.
  • With probability at least 1 − κ/2, any p satisfying H(p,p∗) ≥ M(ϵn + ηn + log(1/α)/n) is excluded from the confidence set.
Loading 1912.11436v4…