Bayes Estimators with Performance Comparable to Empirical Bayes Estimators and Improved Local Robustness

I. INTRODUCTION

• The study focuses on the EB estimator using marginal maximum-likelihood hyperparameter estimation, a method valued for balancing estimation accuracy and computational cost.For Gaussian weighting families with parameterized covariance matrices, this estimator reduces to kernel-based regularized estimation used in system identification.Source: §I · p. 1
• The paper constructs a generalized Bayes estimator whose XMSE matches the corresponding empirical Bayes estimator, while improving local robustness to hyperparameter perturbations.The construction characterizes a weighting function through pointwise optimization; the paper also analyzes sensitivity and computational complexity.Source: §I · p. 2

II. PRELIMINARIES AND PROBLEM STATEMENT

• The linear-regression setup estimates an unknown parameter vector from a known regression matrix and measurements, with performance assessed by mean squared error.The stated assumptions include a fixed model order at least as large as the true order, a known full-rank deterministic regression matrix, and Gaussian noise with known variance.Source: §II · p. 2

A. Bayes and empirical Bayes estimators

• A generalized Bayes estimator uses a weighting function under squared-error loss, while an empirical Bayes estimator substitutes a data-estimated hyperparameter into that estimator.The paper uses marginal-likelihood maximization to estimate the hyperparameter and refers to the resulting estimator as the EB estimator.Source: A. Bayes and empirical Bayes estimators · pp. 2–3
• For Gaussian weighting families, the EB estimator and its hyperparameter cost function have closed-form expressions, and the kernel matrix defines a quadratic-penalty regularized least-squares estimator.The Gaussian likelihood and weighting function are conjugate, yielding the stated closed-form expressions.Source: A. Bayes and empirical Bayes estimators · p. 3

B. Motivation and problem statement

• Small shape-hyperparameter perturbations are associated with increased sample MSE and decreased average FIT, illustrating the local sensitivity motivating the paper.In the simulation example, the unperturbed EB estimator outperforms maximum likelihood on the reported sample means of sample MSE and average FIT.Source: B. Motivation and problem statement · pp. 3–4, Figure 1 · p. 4
• The proposed alternative is a Bayes estimator designed to perform comparably to empirical Bayes without direct plug-in dependence on estimated hyperparameters.The paper notes that hyperparameter estimation errors, including those from numerical optimization, may cause finite-sample performance degradation and numerical fragility.Source: B. Motivation and problem statement · p. 4
• XMSE provides an analytical large-sample design criterion because an analytical finite-sample MSE expression for the empirical Bayes estimator is generally beyond reach.XMSE is a second-order measure of the difference between an estimator's MSE and the maximum-likelihood estimator's MSE.Source: B. Motivation and problem statement · p. 4
• The design question is whether a nonnegative weighting function can make a Bayes estimator's XMSE equal that of the corresponding EB estimator for general weighting families and a general Σ.Prior results addressed only special structures for Σ or the EB weighting family; the general case is identified as the problem addressed here.Source: B. Motivation and problem statement · p. 4

III. DESIGN OF BAYES ESTIMATORS WITH THE SAME XMSE AS EB ESTIMATORS

• Under regularity conditions, the generalized Bayes estimator's XMSE separates into squared excess bias and excess variance, whose sign depends on the curvature of log weighting.The paper imposes positivity, smoothness, and tail-growth conditions on the weighting function to support the analysis.Source: §III · p. 4

A. XMSE of an EB estimator ˆθEB(ˆηEB)

• Theorem 1 gives an explicit XMSE expression for the EB estimator, including an added excess-variance term induced by hyperparameter estimation.Under Assumptions 1–2 and 4, that term makes EB hyperparameter estimation a performance penalty.Source: A. XMSE of an EB estimator ˆθEB(ˆηEB) · p. 5

B. Design of weighting function

• For any EB weighting family, profiling over its hyperparameter yields a Bayes counterpart with the same XMSE and no direct dependence on the EB hyperparameter estimate.This is the characterization established by Theorem 2.Source: B. Design of weighting function · p. 5

C. Case study: Gaussian EB weighting families

• For Gaussian EB families, the complete solution to the matching PDE is a pointwise upper envelope and is generally non-Gaussian because its profiled hyperparameter varies with θ.The complete family of solutions is given by (21).Source: C. Case study: Gaussian EB weighting… · p. 5

1) Scale-shape parametrization:

• In the scale-shape parametrization, P(η)=cK(α), where c controls overall scale and α controls covariance shape.The scale can be profiled out in closed form, leaving an (nη − 1)-dimensional optimization for η⋆(θ), versus an nη-dimensional EB hyper-parameter problem.Source: 1) Scale-shape parametrization · pp. 5–6
• The scale-shape result specializes to weighting functions previously obtained for fixed K and, when Σ=K=I_nθ, to the result in [15, Theorem 1].The fixed-K case matches [12, Corollary 10.1].Source: 1) Scale-shape parametrization · p. 6
• Positive rescaling of K(α) inversely rescales c⋆(α; θ), leaving the resulting weighting function unchanged.The scale-shape decomposition is therefore non-unique without changing π⋆(θ).Source: 1) Scale-shape parametrization · p. 6

2) Diagonal parametrization:

• The diagonal parametrization imposes a positive lower bound η, avoiding the normal-Jeffreys prior's singularity at zero and yielding a well-defined Bayes estimator.Without the bound, the normal-Jeffreys prior is locally nonintegrable near zero and the Bayes-estimator integrals diverge.Source: 2) Diagonal parametrization · p. 6
• For sufficiently small ε, the designed Bayes estimator's XMSE is expected to be close to that of the EB estimator.This comparison is given for the lower-bounded diagonal parametrization.Source: 2) Diagonal parametrization · p. 6

IV. LOCAL SENSITIVITY OF EB AND BAYES ESTIMATES TO HYPER-PARAMETER PERTURBATIONS

• The Bayes estimate is at most second-order sensitive to hyper-parameter perturbations, while the EB estimate is at most first-order sensitive.The analysis holds the observed data fixed; the reduced sensitivity suggests reduced performance degradation under small perturbations.Source: §IV · pp. 6–7

A. General computational structures

• The Bayes implementation computes η⋆(θ) for each integration point, either in closed form or by numerically solving the design problem.The Bayes estimate generally requires numerical approximation; the Gaussian-family implementation uses importance sampling.Source: A. General computational structures · p. 7
• Neither estimator is uniformly computationally preferable; relative cost depends strongly on conjugacy and the EB weighting-family parametrization.The paper analyzes general computational structures and then specializes to Gaussian and Student-t weighting families.Source: A. General computational structures · p. 7, §V · p. 7

B. Gaussian EB weighting family with (12)

• For the Gaussian weighting family, conjugacy with the Gaussian likelihood gives closed-form EB hyper-parameter cost and estimate expressions.This enables efficient computation of the EB estimate.Source: B. Gaussian EB weighting family with… · p. 7
• When the proposal qG(θ) approximates the target distribution well, it can improve numerical accuracy and stability.The proposal exploits the Gaussian weighting family's structure at η=η⋆(θ̂ML).Source: B. Gaussian EB weighting family with… · pp. 7–8

O(M B

• For the Gaussian weighting family with parametrization (12), the Bayes estimate can be computationally comparable to or more efficient than EB for fixed N and nθ.The comparison depends on the number of importance-sampling samples and EB cost evaluations.Source: O(M B · p. 8

C. Student-t EB weighting family

• The Student-t weighting family is heavy-tailed and imposes less shrinkage on large model parameters than a Gaussian weighting family.Its degrees of freedom ν is assumed known.Source: C. Student-t EB weighting family · p. 8
• Because the Student-t weighting family is non-conjugate to the Gaussian likelihood, the EB estimate and its hyper-parameter cost are computed using importance sampling.The proposal is a Gaussian Laplace approximation to the posterior proportional to p(Y|θ)π(θ|η).Source: C. Student-t EB weighting family · p. 8

10 Reuse the IS samples and IS weights to compute

• The Bayes estimate constructs the proposal once, while EB constructs it MEB times, making the Bayes computational advantage more evident for larger MEB.For the remaining computations, the Bayes estimate is described as computationally comparable or more efficient under the stated comparison.Source: §10 · p. 9
• For other non-conjugate weighting-likelihood pairs that require repeated η-dependent EB proposals, similar Bayes computational advantages can be expected.The paper attributes the comparison to the repeated proposal construction required by EB in these settings.Source: §10 · p. 9

1) Test data and simulation setup:

• The Gaussian simulations use the Example 1 systems and data with nθ = 20 and N = 80, and perturb both scale and shape hyper-parameters.The scale perturbation is applied on a logarithmic scale to keep the perturbed scale positive.Source: A. Gaussian EB weighting family · p. 9, 1) Test data and simulation setup · p. 9

A. Gaussian EB weighting family

• For the Gaussian family, EB and Bayes have quite close estimation accuracies, both outperform ML, and Bayes has smaller degradation under scale and shape perturbations.Bayes computation time is comparable to EB's in this implementation.Source: 2) Simulation results · p. 9, Table I · p. 9, Figure 2 · p. 10

1) Test data and simulation setup:

• The Student-t simulations use 100 generated parameter realizations with nθ = 50 and N = 200, and perturb the positive hyper-parameter on a logarithmic scale.Each parameter and design collection is evaluated with 100 Monte Carlo simulations.Source: 1) Test data and simulation setup · p. 10

2) Simulation results:

• For the Student-t family, Bayes and EB have comparable estimation accuracy, both outperform ML, and Bayes is locally more robust to hyper-parameter perturbations.Bayes computation time is about 12 times lower than EB's in this implementation.Source: 2) Simulation results · p. 10, Figure 4 · p. 10, Table II · p. 10

VII. CONCLUSION

• The generalized Bayes estimator matches EB's XMSE and is at most second-order sensitive to hyper-parameter perturbations, versus at most first-order sensitivity for EB.The paper analyzes both estimators' computational complexities and reports that Bayes can be comparable or more efficient under certain conditions.Source: §VII · p. 11
• Constructing generalized Bayes estimators that outperform EB remains a potential extension; under general settings, XMSE improvement leads to a nonlinear PDE that generally requires numerical methods.The paper suggests using its constructed estimator as a baseline and adding shrinkage through its weighting function.Source: §VII · p. 11

B. Proof of Theorem 1

• The proof expresses the EB estimate solely as a function of the ML estimate, then derives the limiting hyper-parameter and its derivative needed for the XMSE calculation.The appendix substitutes these limits into the cited XMSE result to obtain the expression for the EB estimator's XMSE.Source: A. Proof of Lemma 1 · p. 11, B. Proof of Theorem 1 · pp. 11–12
• Uniform convergence of the objective over the feasible hyper-parameter set, together with compactness and uniqueness, yields consistency of the minimizing hyper-parameter.The proof first establishes convergence of the objective using local and tail bounds.Source: B. Proof of Theorem 1 · pp. 11–12
• For sufficiently large samples, the estimated hyper-parameter is an interior minimizer and satisfies the first-order optimality condition used to derive its limit derivative.The argument uses consistency and differentiates the optimality condition with respect to the parameter.Source: B. Proof of Theorem 1 · p. 12

C. Proof of Theorem 2

• The implicit function theorem establishes that the uniquely selected hyper-parameter ηb⋆(θ) is twice continuously differentiable near θ0.The proof uses an interior minimizer and a nonsingular second derivative with respect to η.Source: C. Proof of Theorem 2 · p. 12
• The constructed weighting function π⋆(θ) is shown to satisfy the regularity conditions required for its XMSE to be well-defined.The proof verifies positivity and differentiability, then checks the additional stated condition.Source: C. Proof of Theorem 2 · pp. 12–13
• Matching the excess-bias and excess-variance components establishes that the constructed Bayes estimator and EB estimator share the same XMSE.The proof matches the bias component through the optimality condition and derives the variance expression using the chain rule.Source: C. Proof of Theorem 2 · p. 13

D. Proof of Lemma 3

• Regularity of P(η) supplies positivity, differentiability, and uniform derivative bounds for the Gaussian weighting families required by the assumptions.The proof uses compactness of Dη and regularity of P(η) to bound P(η) and its derivatives.Source: D. Proof of Lemma 3 · p. 13

E. Proof of Corollary 2.1

• The constructed weighting function π⋆(θ) gives the complete family of solutions to the XMSE-matching PDE, including its Gaussian specialization.Substitution reduces the Gaussian case to the PDE in (19).Source: E. Proof of Corollary 2.1 · p. 13, D. Proof of Lemma 3 · p. 13

F. Proof of Corollary 2.2

• For fixed α, minimizing the cost over c yields its first-order optimality condition and the resulting reduced cost function.The derivation substitutes the optimizing c into Wb(c, α; θ).Source: F. Proof of Corollary 2.2 · p. 13

G. Proof of Corollary 2.3

• With noncompact Dη = [η, ∞)nη, coercivity ensures minimizers exist, and the unique selected coordinates satisfy [ηb⋆(θ)]k = max{[θ]k^2, η}.The proof establishes coercivity for both the EB cost and Wb, then substitutes the coordinate-wise minimizer.Source: G. Proof of Corollary 2.3 · pp. 13–14

• Assumption-based domination and Gaussian-type tail decay are used to establish finiteness of the integrals needed in the proof.The proof notes that the tail can dominate polynomial growth.Source: H. Proof of Theorem 3 · p. 14
• The EB estimator is locally bounded in its response to hyperparameter perturbations because its gradient is continuous near the marginal maximum likelihood estimate.Differentiation under the integral sign and continuity of the resulting derivatives support this argument.Source: H. Proof of Theorem 3 · p. 14
• A Taylor remainder bounds the change in the Bayes weighting function by a Hessian bound times the squared hyperparameter perturbation.The proof then uses denominator control to bound the difference between the two Bayes estimates.Source: H. Proof of Theorem 3 · p. 14
• For sufficiently small perturbations, the Bayes-estimate denominators remain positive, so the perturbed and unperturbed ratio estimators are well-defined.The proof uses this control when decomposing their difference.Source: H. Proof of Theorem 3 · pp. 14–15

I. Complexity analysis of ˆθBayes(π⋆) in Section V-B

• In Section V-B, computing the maximum-likelihood estimate uses a Cholesky factorization costing O(n_θ^3) and two triangular solves costing O(n_θ^2).The Bayes proposal calculation also includes O(M_b n_θ) evaluations and a factorization with stated cost O(n_θ^3 + M_b n_θ).Source: §I · p. 15
• In Section V-B, each importance sample requires O(n_θ^2) generation and O(M_b n_θ) evaluation of f^W_b to obtain its hyperparameter value.Computing the associated importance weight is stated to cost O(n_θ) per sample.Source: §I · p. 15

J. Complexity analysis of ˆθEB(ˆηEB) in Section V-C

• In Section V-C, constructing the EB proposal costs O(M_L n_θ^2), followed by importance sampling repeated M_EB times; reusing samples and weights leaves a weighted-average estimator computation.The proposal construction forms a Hessian and computes its inverse at cost O(n_θ^3).Source: J. Complexity analysis of ˆθEB(ˆηEB)… · p. 15

K. Complexity analysis of ˆθBayes(π⋆) in Section V-C

• In Section V-C, the Bayes proposal-construction step costs O(M_b n_θ + [N + M_L]n_θ^2).The remaining importance-sampling and estimator-approximation complexity is described as analogous to the Section V-B analysis.Source: K. Complexity analysis of ˆθBayes(π⋆)… · p. 15