PaperlayerView exact revision on arXiv

Source-linked AI summary

Two Modeling Strategies for Empirical Bayes Estimation

Bradley Efron

arXiv:1409.2677v1stat.ME

TL;DR

Empirical Bayes seeks to estimate conditional parameter distributions by learning from parallel observations, but its theoretical basis is uncertain. This paper compares modeling on the parameter and observation scales and develops formulas to assess their frequentist accuracy. Its examples show that neither strategy dominates: f-modeling can be ineffective for discontinuous parameter-scale questions, while g-modeling can be useful for more general inference.

  • Problem

    Empirical Bayes must learn a prior from parallel data, but its theoretical basis is uncertain and its accuracy needs assessment.

    Source: §1 · pp. 1–2

  • Method

    The paper compares f-modeling and g-modeling and develops computational formulas for assessing the frequentist accuracy of estimates from both strategies.

    Source: §1 · p. 2

  • Results

    Across examples, the comparison is not one-sided: g-modeling is useful for more general inference, while f-modeling is ineffective for discontinuous θ-scale questions such as the prostate-data probability example.

    Source: §1 · p. 2, §7 · p. 16

  • Takeaways & Limitations

    Parallel experiments carry prior information in their other cases, and extracting it efficiently remains an ongoing empirical Bayes problem for which both modeling strategies are useful.

    Source: §1 · p. 2, §7 · p. 17

    For the Figure 4 setting, f-modeling is expected to be difficult, and the Section 5 approach ameliorates but does not cure those difficulties.

    Source: §4 · p. 8

Abstract

from arXiv · show

Empirical Bayes methods use the data from parallel experiments, for instance, observations $X_k\sim\mathcal{N}(\Theta_k,1)$ for $k=1,2,\ldots,N$, to estimate the conditional distributions $\Theta_k|X_k$. There are two main estimation strategies: modeling on the $\theta$ space, called "$g$-modeling" here, and modeling on the $x$ space, called "$f$-modeling." The two approaches are described and compared. A series of computational formulas are developed to assess their frequentist accuracy. Several examples, both contrived and genuine, show the strengths and limitations of the two strategies.

1. INTRODUCTION

Empirical Bayes must learn an appropriate prior from parallel data rather than assume it, and this paper compares modeling that learning on the parameter scale and the observation scale. It develops computational accuracy formulas and uses numerical examples to show that neither strategy dominates across applications.

1. INTRODUCTION

  • The central empirical Bayes task is learning an appropriate prior from ongoing statistical experience, rather than knowing it by assumption.The paper frames this learning as requiring both Bayesian and frequentist modeling strategies.Source: §1 · p. 1
  • G-modeling works on the θ scale, while f-modeling works on the x scale where empirical Bayes data arrive.The paper notes that theoretical work has more often used g-modeling, while applications have more often used f-modeling.Source: §1 · p. 1
  • Across its examples, the comparison is not one-sided: g-modeling becomes useful beyond classic applications, while inverting x-space observations to recover g has an ill-posed computational character.The paper concludes that both strategies are useful because empirical Bayes calculations are inherently difficult.Source: §1 · p. 2
  • The paper uses discrete models to avoid the rates-of-convergence difficulties associated with estimating g from observed X values in the deconvolution problem.It distinguishes empirical Bayes estimation of nonlinear functionals of g from deconvolution work centered on linear functionals.Source: §1 · p. 2
  • The paper develops Theorems 1–4 as computational formulas for assessing the frequentist accuracy of both f-model and g-model estimates.The authors note that accuracy can be poor and argue that assessing it is an important part of empirical Bayes analysis.Source: §1 · p. 2

2. A DISCRETE MODEL OF BAYESIAN INFERENCE

The discrete model places prior mass on a finite set of parameter values and connects that prior to a finite set of observations through sampling probabilities. This formulation makes Bayes calculations amenable to matrix operations and sets up the paper’s f-modeling development.

2. A DISCRETE MODEL OF BAYESIAN INFERENCE

  • The model restricts both parameter values and observations to finite discrete sets to simplify f-modeling computations.The authors say the discrete framework allows familiar matrix calculations in place of functional equations.Source: §2 · pp. 2–3
  • The prior probabilities g and sampling probabilities P induce the observation probabilities f through f = Pg.Here P contains the sampling probabilities linking each parameter value to each observation value.Source: §2 · p. 2
  • In the Figure 1 example, a discretized mixture prior under normal translation sampling produces the triangular-shaped marginal density f(x).The figure displays the prior g(θ) above and its corresponding f(x) below, on different scales.Source: §2 · p. 2, Figure 1 · p. 3
  • The discrete model expresses posterior probabilities of θ given an observation using Bayes’ rule and supports posterior expectations of a parameter of interest.The authors present this finite formulation as a simplifying framework, including for continuous-data examples such as the prostate data.Source: §2 · pp. 2–3

3. BAYES RULE IN TERMS OF f

This section rewrites posterior expectations in terms of the observation marginal f, which can be estimated from the observed sample, and develops frequentist accuracy calculations for that approach. Its examples show that f-based estimation can be noisy, especially for discontinuous targets, and that direct parameter-scale sampling can be far more efficient when available.

3. BAYES RULE IN TERMS OF f

  • Under the full-rank assumption on P, the section inverts f = Pg to express Bayes’ rule in terms of f; other definitions of A are possible.The discussion notes that the case rank(P) < m is addressed in Section 4.Source: §3 · p. 3
  • The observed category counts provide a multinomial sufficient statistic, and substituting the empirical estimate of f yields a nonparametric estimate of the posterior expectation.The section uses a delta-method calculation to approximate the estimate’s frequentist standard error.Source: §3 · p. 4
  • Direct sampling on the θ scale would usually make the estimate less variable than the f-based estimate and automatically enforce constraints such as monotonicity.The paper emphasizes that empirical Bayes data, however, arrive on the x scale rather than the θ scale.Source: §3 · p. 4
  • For c0 = 0.1, the three example parameters require N = 1936, 8281 and 187 million, respectively, for the f-based estimate; the discontinuous third target destabilizes the calculation.The abrupt discontinuity at θ = 0 leads to violent cycling in U; the section points to regularization as a way to improve accuracy.Source: §3 · p. 5
  • 25,600 Θk observations suffice for coefficient of variation 0.1 when directly estimating Pr{θ ≤0|x = 2.5}; direct sampling is almost always more efficient than f sampling.The efficiency difference is a factor of 86 for parameter (3), but less than a factor of 3 for parameter (1), t(θ) = θ.Source: §3 · p. 5
  • The accuracy results rely crucially on independent sampling; nonindependence effectively reduces the effective sample size N.The paper notes that independence assumptions are often tacitly made in the empirical Bayes literature.Source: §3 · p. 5

4. REGULARIZED f-MODELING

Regularized f-modeling uses smooth parametric estimates of the marginal distribution to reduce the noise of fully nonparametric empirical Bayes estimates. Truncating the inversion can reduce variability, but the choice of truncation level trades bias against variability and may perform poorly for difficult priors.

4. REGULARIZED f-MODELING

  • Parametric f-modeling reduced coefficients of variation by factors of 3, 5, and 100 for the three parameters, compared with nonparametric estimates.The example used a natural-spline structure matrix with five degrees of freedom.Source: §4 · p. 7
  • Increasing r decreases bias but increases variability; the choice is crucial for parameter (3) but unimportant for parameter (1).Choosing r too small can badly bias the estimate; the supplied Table 2 caption describes this bias–variability tradeoff for parameter (3).Source: Table 2 · p. 7, §4 · pp. 7–8
  • For a smooth true prior, g_r approaches g rapidly as r increases, and bias is a problem only for quite small r.Figure 3 depicts approximations for r = 6, 9, and 12 alongside the true g.Source: §4 · p. 8, Figure 3 · p. 8
  • For a prior with a 90% atom at zero and 10% uniform mass, g-error exceeds 1.75 even at r = 21, with too little weight at zero and erratic, sometimes negative, estimates elsewhere.The authors expect empirical Bayes estimation to be usually difficult in this setting.Source: §4 · p. 8
  • This difficult prior resembles one used in familiar false discovery rate analyses, and the alternative regularization in Section 5 ameliorates—but does not cure—the estimation difficulty.Source: §4 · p. 8

5. MODELING THE PRIOR DISTRIBUTION g

The section develops parametric g-modeling of the prior and formulas for assessing its frequentist accuracy. Its performance depends on the inference target: it can outperform f-modeling for discontinuous parameters, yet accurate estimation for a spike-at-zero prior may require very large samples.

5. MODELING THE PRIOR DISTRIBUTION g

  • Parametric g-modeling represents the prior through an exponential-family model, estimates its parameters by maximum likelihood, and assesses posterior-estimate uncertainty with approximate standard deviations.The multinomial observations are drawn from f=Pg, where g is determined by the model parameters.Source: §5 · pp. 8–10
  • The multinomial g-model likelihood is not an exponential family in the observed counts, a theoretical disadvantage relative to the Poisson formulation used for f-modeling.The section nevertheless derives approximate covariance and standard-deviation formulas for g-model estimates.Source: §5 · pp. 9–10
  • For t(θ)=θ, f-modeling has somewhat smaller standard deviations, but for Pr{θ≤0|x}, g-modeling is far superior.Both comparisons use six-degree-of-freedom spline models; the section notes that g-modeling often has lower standard deviations for discontinuous parameters, with important exceptions.Source: §5 · p. 11
  • The approach enforces positive prior probabilities, can automatically enforce constraints on f, and avoids choosing the f-modeling truncation parameter r.It also avoids the need for Bayes rule in terms of f.Source: §5 · p. 11
  • For large-scale testing with a big prior atom at θ=0, g-scale modeling accommodates it by adding e0 to Q; the example uses true g=0.9e0+0.1·uniform and Q=ns(θ,5) augmented by e0.Here e0 is a column with 1 at θ=0 and zeros elsewhere, and Q is a 31 × 6 matrix.Source: §5 · p. 11
  • For estimating Pr{θ=0|x} under that mixture, reasonably accurate g-model estimates may require sample sizes on the order of 10,000.With N=5000, the estimated mass at zero was 0.920 versus the true value 0.903, but nonnull peaks at θ=±2 were artifacts.Source: §5 · pp. 11–12

6. CLASSIC EMPIRICAL BAYES APPLICATIONS

Classic empirical Bayes applications often estimate posterior quantities directly from the marginal distribution of observations, without estimating the prior. The examples show that f-modeling suits some such targets, while g-modeling and f-modeling can differ substantially in uncertainty for other targets.

6. CLASSIC EMPIRICAL BAYES APPLICATIONS

  • Some classic problems, including false discovery rates and the James–Stein estimator, permit Bayesian inferences to be computed directly from the marginal observation distribution, making g-modeling unnecessary for those calculations.The section also discusses Robbins’ Poisson estimation and Tweedie’s extension to general exponential families.Source: §6 · pp. 13, 15
  • Tweedie’s formula estimates E{θ|x} from the marginal density f(x), without explicit reference to the prior g(θ).For the prostate data, the estimate is nearly zero between −2 and 2; for the largest observed z-value, 5.29, the estimate is 4.09.Source: §6 · pp. 13–14
  • In the prostate-data estimate of E{θ|x}, f-modeling and g-modeling closely approximate Tweedie’s estimate, while f-modeling has noticeably lower standard deviations for large |x|.Bootstrap standard deviations for Tweedie estimates closely follow the f-modeling standard deviations and support the delta-method calculations.Source: §6 · p. 14
  • For prostate-data local false discovery rates, g-modeling has considerable uncertainty, with coefficients of variation around 25% even at N=6033, while f-modeling estimates fail for every tested r.For the upper false discovery rate, f-modeling substantially reduces standard deviations across the x-scale.Source: §6 · p. 15
  • Setting π0=1 gives an upper false discovery rate that exceeds 1 near x=0, the stated penalty for that choice.The passage describes dividing its values by their maximum as a simple correction.Source: §6 · p. 15

7. DISCUSSION

The discussion contrasts f-modeling’s straightforward fitting on observed data with g-modeling’s more difficult fitting but simpler posterior computations. It concludes that f-modeling suits a limited class of inference targets, while more general targets can remain difficult to estimate accurately.

7. DISCUSSION

  • f-modeling fits directly to observed X values using familiar methods, whereas g-modeling requires difficult nonconvex fitting but makes subsequent Bayesian computations straightforward.In the nonparametric case, f-modeling uses the vector of observed proportions, while g-modeling requires difficult nonparametric maximum-likelihood calculations.Source: §7 · p. 16
  • f-modeling dominates practice within the limited set of cases where posterior inferences are simple functions of the marginal distribution f(x).Within this special class, the Bayesian computational difficulties of f-modeling disappear.Source: §7 · p. 16
  • Improved f-modeling strategies may be feasible, including alternatives to the pseudo-inverse used in the paper.The discussion notes that pseudo-inverses of P other than A are available.Source: §7 · pp. 16–17
  • For Pr{|θ| ≥1.5|x} in the prostate data, g-modeling provides moderately accurate estimates, while f-modeling is ineffective for this discontinuous θ-scale target.Figure 8 reports g-model estimates and their standard deviations; the discussion describes the estimates as yielding some useful information.Source: Figure 8 · p. 16, §7 · p. 16
  • Outside traditional applications, empirical Bayes estimation can be ill-posed: even g-modeling may need more than N = 20,000 observations for CV 0.1 when x exceeds 2.Reducing the degrees of freedom for Q can improve accuracy, but risks increased bias.Source: §7 · p. 17
  • Higher-dimensional modeling is more difficult, and even a study as large as the prostate investigation may not provide enough information for accurate empirical Bayes estimation.Source: §7 · p. 17
  • Parallel experiments provide their own prior information: the other N −1 cases furnish information for analyzing each (Θk,Xk) pair.The discussion frames efficient extraction of this information as an ongoing area of study.Source: §7 · p. 17
Loading 1409.2677v1…