PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimation in Gaussian Noise: Properties of the Minimum Mean-Square Error

Dongning Guo, Yihong Wu, Shlomo Shamai, Sergio Verdu

arXiv:1004.3332v1cs.IT

TL;DR

The paper asks how MMSE behaves as a function of SNR and input distribution for arbitrary variables observed through Gaussian noise. It develops posterior-tail, incremental-channel, and analytic tools to characterize this behavior, then derives distributional comparisons and channel-information consequences. The results include positive-SNR smoothness, conditional-moment derivative formulas, concavity in the input distribution, and a single-crossing property between Gaussian and non-Gaussian MMSE curves.

  • Problem

    The paper studies the mathematical properties of MMSE for arbitrary input distributions observed through additive Gaussian noise, including its dependence on SNR and the input distribution.

  • Method

    The paper analyzes Gaussian-channel posteriors and an incremental-SNR representation that converts ordinary MMSE into conditional MMSE and enables regularity and derivative analyses.

  • Results

    MMSE is infinitely differentiable at every positive SNR for every input distribution, is real analytic under certain conditions, is concave in the input distribution, and has a Gaussian/non-Gaussian single-crossing property.

  • Takeaways & Limitations

    These MMSE properties provide simple proofs of Gaussian-input optimality for scalar Gaussian wiretap and broadcast channels and of the entropy power inequality when one variable is Gaussian.

  • Takeaways & Limitations

    The paper notes that real analyticity at zero SNR for all inputs cannot coexist with MMSE being injective over all random variables modulo shifts and reflections.

Abstract

from arXiv · show

Consider the minimum mean-square error (MMSE) of estimating an arbitrary random variable from its observation contaminated by Gaussian noise. The MMSE can be regarded as a function of the signal-to-noise ratio (SNR) as well as a functional of the input distribution (of the random variable to be estimated). It is shown that the MMSE is concave in the input distribution at any given SNR. For a given input distribution, the MMSE is found to be infinitely differentiable at all positive SNR, and in fact a real analytic function in SNR under mild conditions. The key to these regularity results is that the posterior distribution conditioned on the observation through Gaussian channels always decays at least as quickly as some Gaussian density. Furthermore, simple expressions for the first three derivatives of the MMSE with respect to the SNR are obtained. It is also shown that, as functions of the SNR, the curves for the MMSE of a Gaussian input and that of a non-Gaussian input cross at most once over all SNRs. These properties lead to simple proofs of the facts that Gaussian inputs achieve both the secrecy capacity of scalar Gaussian wiretap channels and the capacity of scalar Gaussian broadcast channels, as well as a simple proof of the entropy power inequality in the special case where one of the variables is Gaussian.

I. INTRODUCTION

The paper develops a detailed study of MMSE for arbitrary inputs observed through additive Gaussian noise, treating it both as an SNR function and an input-distribution functional. It establishes regularity, derivative, distributional, and information-theoretic properties with applications to Gaussian channels and the entropy power inequality.

  • Motivation: The paper studies the MMSE of estimating an arbitrary random variable from an observation contaminated by additive Gaussian noise.MMSE is considered as a function of SNR for fixed input distribution and as a functional of the input distribution for fixed SNR.
  • Properties: The posterior input distribution conditioned on a Gaussian-channel observation decays at least as quickly as some Gaussian density.This posterior tail behavior supports bounds on conditional and unconditional moments of the estimation error.
  • Regularity: For every input distribution, MMSE is infinitely differentiable at positive SNR; under certain conditions, it is real analytic and approximable by its Taylor series.Regularity at zero SNR requires additional conditions.
  • Derivatives: The first three SNR derivatives of MMSE are expressed through average central moments of the input conditioned on the output.The result is also extended to conditional MMSE.
  • Distributional properties: MMSE is concave in the input distribution at fixed SNR, and Gaussian and non-Gaussian MMSE curves cross at most once over positive SNR.The single-crossing statement holds regardless of the variances of the compared inputs.
  • Applications: MMSE properties yield simple proofs of the Gaussian-input optimality of scalar Gaussian wiretap and broadcast channels and of the Gaussian-variable case of Shannon’s entropy power inequality.The paper also connects MMSE to information measures and applications including filtering, detection, and power allocation.

B. The Conditional MMSE and SNR Increment

This section introduces conditional MMSE and shows that Gaussian observations can be combined through an incremental channel. The resulting SNR-addition identity translates ordinary MMSE into conditional MMSE and supports later bounds and regularity results.

  • Conditional MMSE: Conditional MMSE is the mean-square estimation error when side information U is available to the estimator.It can be represented as an average of ordinary MMSE values under the conditional input distributions indexed by U.
  • SNR increment: Two independent Gaussian observations of X combine into an equivalent observation whose SNR is the sum of the component SNRs.The same SNR-addition property applies when one observation is treated as side information.
  • Consequences: The incremental-channel identity is used to translate MMSE at any SNR into conditional MMSE at a smaller SNR.The paper identifies this result as key to subsequent MMSE regularity and derivative results.
  • SNR increment: Proposition 3 states that mmse(X, snr + γ) equals the conditional MMSE at increment γ given an observation at SNR snr.The proof uses a cascade of Gaussian channels and the fact that the lower-SNR observation is a physical degradation of the higher-SNR observation.
  • Posterior bounds: At nonzero SNR, the input can always be estimated with finite mean-square error, regardless of whether the input distribution has finite moments.The linear estimate Y/√snr has mean-square error 1/snr.
  • Posterior bounds: For any input distribution, the posterior distribution conditioned on a nonzero-SNR Gaussian observation is sub-Gaussian, with bounded posterior moments.This provides the moment control used in the section’s estimation-error bounds.

III. SMOOTHNESS AND ANALYTICITY

The MMSE is smooth as a function of positive SNR for every fixed input distribution, including distributions without finite moments. Under additional conditions it is real analytic, while zero-SNR regularity needs stronger assumptions.

  • Smoothness: The function mmse(X, snr) is smooth on (0, ∞) for arbitrary fixed input distribution.This smoothness enables the derivative calculations developed in the following section.
  • Analyticity: Under certain technical conditions, MMSE is real analytic in SNR at positive values and can be arbitrarily well approximated by its Taylor expansion.Regularity at zero SNR requires additional conditions.
  • Zero SNR: At zero SNR, the n-th derivative depends on the first n + 1 moments of the input, so infinite right differentiability follows when all moments are finite.The zero-SNR expansion therefore has stronger moment requirements than positive-SNR smoothness.
  • Smoothness: MMSE is infinitely differentiable at every positive SNR for every input distribution, even when the input mean and variance are infinite.The proof removes moment assumptions using the incremental-SNR result and posterior moment bounds.
  • Proof strategy: When all input moments are finite, the smoothness proof differentiates Gaussian-weighted integral expressions and exchanges derivatives with expectations.The Gaussian density ensures products with polynomials remain bounded, supporting this interchange.

B. Real Analyticity

The paper characterizes when the MMSE is real analytic by extending it through power series and imposing conditions that control complex-plane behavior. These conditions explain why analyticity holds at positive SNR for broad input classes but may fail at zero SNR.

  • Definition: Real analyticity means representability by a convergent power series in a neighborhood of the expansion point.Equivalently, the function must extend to an open complex disk through that power series.
  • Transfer to SNR: Analyticity of mmse(X, a^2) at a implies analyticity of mmse(X, snr) at snr = a^2.At zero, the Taylor series is even; at positive SNR, the result follows by composing with the real-analytic map snr ↦ √snr.
  • Interpretation: The analyticity conditions control the effect of the imaginary part of a on the magnitude of the observation density h0(y; a).For real a, h0 remains positive and decays no faster than a Gaussian, but complex values may introduce zeros that obstruct extension.
  • Examples and boundary: For finite-alphabet, exponential, or Gaussian inputs, the stated conditions hold for every a ≠ 0, yielding real analyticity at all positive SNR.The binary example shows that analyticity at zero need not follow: the MMSE cannot be extended to points on the imaginary axis.

IV. DERIVATIVES

The paper derives MMSE derivatives with respect to SNR from conditional moments, extending the information–estimation relationship to higher derivatives. These formulas remain valid for every input at positive SNR because Gaussian-channel posterior moments are finite.

  • MMSE expansion: The MMSE Taylor expansion around snr = 0+ is developed through third order, with coefficients determined by input moments.For the first three derivatives at zero SNR, finite second, third, and fourth moments are required, respectively.
  • MMSE expansion: For snr > 0, finiteness of input moments is unnecessary because conditional moments are finite.The expansion can therefore be lifted from zero SNR to arbitrary positive SNR using the SNR-incremental result.
  • Derivative formulas: The derivatives of the MMSE are expectations of polynomials in conditional moments Mi.These moments are well-defined for positive SNR, and symmetry of the input induces symmetry for odd-indexed conditional moments.
  • Gaussian example: For a standard Gaussian input, M2 = (1 + snr)^−1, M3 = 0, and M4 = 3(1 + snr)^−2, making the derivative formulas immediate.The conditional input distribution remains Gaussian in this case.
  • Mutual information: The same derivative framework extends the information–estimation relationship to derivatives of mutual information.The resulting formulas hold when the corresponding expectations exist, and analyticity follows under the conditions of Proposition 8.

C. Derivatives of the Conditional MMSE

The paper develops conditional-MMSE extensions, establishes concavity in the input distribution, and shows that normalized sums become progressively easier to estimate toward the Gaussian limit.

  • Conditional MMSE: The first three MMSE derivatives extend to conditional MMSE, with the result formulated for jointly distributed (X, U).The extension is stated as a direct generalization of the derivative result.
  • Concavity in Input Distribution: For fixed SNR, MMSE is concave as a functional of the input distribution.The proof uses a Bernoulli mixture and observes that revealing the mixture component can improve the estimator.
  • Conditioning Reduces MMSE: Additional side information cannot increase MMSE, because an informed estimator can discard that information.For fixed positive SNR, equality holds if and only if X and U are independent.
  • Normalized Sums: The MMSE of normalized sums of i.i.d. finite-variance variables is monotone with the number of summands and converges to the Gaussian-input MMSE.The convergence follows from the central limit theorem.
  • Gaussian Inputs Are Hardest to Estimate: At every SNR, a non-Gaussian input has strictly smaller MMSE than a Gaussian input with the same variance, with equality only for Gaussian inputs.The result applies when the input variance is no greater than the Gaussian comparison variance.

E. The Single-Crossing Property

The single-crossing property constrains how MMSE curves compare with Gaussian benchmarks: their difference can cross zero at most once, and the result extends to conditional MMSE.

  • Single-Crossing Property: For any input, its MMSE curve crosses the standard Gaussian MMSE curve (1 + γ)^−1 at most once over positive SNR.The difference is defined relative to the standard Gaussian benchmark.
  • Single-Crossing Property: If f(γ) is negative, it is strictly increasing; after a zero at snr0, it remains nonnegative for larger γ, while tending to zero as γ grows.The same three properties hold against the Gaussian MMSE σ2/(1 + σ2γ) for any σ.
  • Proof Mechanism: The proof uses positivity of the derivative while the MMSE difference is negative, preventing a second zero crossing.The argument applies the derivative formula and Jensen’s inequality.
  • Conditional Extension: The single-crossing property extends to conditional MMSE.The paper notes that this extension also covers the conditional formulation.
  • High-SNR Behavior: High-SNR MMSE can decay faster than exponentially for sufficiently skewed binary inputs, but non-Gaussian inputs need not decay faster than Gaussian inputs.The asymptotic behavior can therefore differ substantially across non-Gaussian distributions.

VI. APPLICATIONS TO CHANNEL CAPACITY

The paper applies MMSE ordering and single-crossing to Gaussian channel-capacity problems, obtaining simple proofs for Gaussian wiretap secrecy capacity and Gaussian broadcast capacity.

  • Secrecy Capacity of the Gaussian Wiretap Channel: Gaussian inputs achieve the secrecy capacity of the scalar Gaussian wiretap channel under the power constraint.The proof uses that standard Gaussian input maximizes MMSE at every SNR, then substitutes the Gaussian MMSE into the secrecy-rate expression.
  • Secrecy Capacity of the Gaussian Wiretap Channel: For the Gaussian wiretap channel, the secrecy capacity is represented using the standard Gaussian input and the channel SNRs.The supplied expression includes the stated logarithmic capacity formula.
  • The Gaussian Broadcast Channel: The Gaussian broadcast-channel proof uses MMSE single-crossing to order mutual informations associated with Gaussian and binary inputs.The paper presents this as an alternative to conventional EPI-based proofs.
  • The Gaussian Broadcast Channel: Gaussian inputs achieve the capacity region of scalar degraded Gaussian broadcast channels.The construction uses an auxiliary variable with a Markov-chain relationship and Gaussian marginals.
  • The Gaussian Broadcast Channel: The broadcast-channel rate bound for the weaker receiver is established through the power-dependent parameter α and the Gaussian benchmark expression.The supplied rate expression is 1/2 log(1 + snr2) − 1/2 log(1 + α snr2).

C. Proof of a Special Case of EPI

The paper uses the single-crossing property of MMSE curves to prove a special case of the entropy power inequality and connect the technique to conditional EPI and broadcast-channel capacity. It also identifies unresolved questions about whether the MMSE transform uniquely determines an input distribution.

  • Proof of a Special Case of EPI: The single-crossing property implies the entropy power inequality when one summand is Gaussian.The proof analyzes an integral whose integrand changes sign at most once, making the integral positive under the stated conditions.
  • Proof of a Special Case of EPI: The same proof technique applies to conditional EPI, which can establish the capacity region of the scalar broadcast channel.
  • Open Question: The MMSE transform maps an input distribution to its MMSE function over all SNR values, but its injectivity remains an open question.
  • Open Question: Conjecture 1 states that identical MMSE curves characterize a zero-mean distribution up to reflection.
  • Open Question: Real analyticity at zero SNR and injectivity cannot both hold for all random variables because distinct distributions can share all moments.For sub-Gaussian variables, Carleman’s condition makes the moments determine the distribution uniquely.

APPENDIX A PROOF OF PROPOSITION 5

The appendix proves Proposition 5 by bounding conditional moments and establishing integrability for the functions used in the MMSE analysis. The argument relies on Gaussian-channel conditioning, moment bounds, and standard inequalities.

  • Moment Bounds: The proof begins with a Gaussian-channel representation and uses Jensen’s inequality to bound conditional quantities.
  • Moment Bounds: A moment-generating-function characterization, Chernoff’s bound, and the union bound control the tails of the relevant random variables.
  • Conditional Bounds: Conditional moments are bounded by combining the conditional expectation and conditional absolute moments.
  • Structural Lemma: The functions g_i are finite weighted sums of specified elementary forms, proved by induction on i.
  • Integrability: Absolute integrability follows by applying generalized Hölder and Jensen inequalities together with independence of X and the Gaussian noise.

APPENDIX D PROOF OF PROPOSITION 8 ON THE ANALYTICITY

The analyticity proof first treats sub-Gaussian inputs using Gaussian derivative bounds and uniform integrability, then extends the argument to arbitrary inputs at positive SNR through an incremental-SNR representation.

  • Sub-Gaussian Inputs: For sub-Gaussian X, Gaussian derivative bounds give real analyticity with infinite radius of convergence for the Gaussian density factor.
  • Sub-Gaussian Inputs: Moment and Hermite-polynomial bounds control the analytic expansions and justify applying Fubini’s theorem.
  • Analytic Extension: The functions h0 and h1 are analytic in the SNR parameter, and a nonvanishing denominator makes their ratio analytic on a complex disk.
  • Analytic Extension: Uniform integrability is obtained from L2 boundedness, allowing the analytic approximants to converge uniformly to the MMSE.
  • Conclusion: The MMSE is real analytic in SNR for sub-Gaussian inputs and, at every positive SNR, the proof extends to inputs without sub-Gaussianity.The positive-SNR extension conditions the input on a Gaussian-channel observation, yielding sub-Gaussian residuals whose moment growth depends on the fixed increment.
  • Conclusion: The remaining estimates use Jensen’s and Fubini’s theorems together with bounds on products of the auxiliary variables M_i.

APPENDIX E PROOF OF PROPOSITION 9 ON THE DERIVATIVES

The appendix derives MMSE derivatives at arbitrary SNR by rewriting the quantity through posterior distributions and using known zero-SNR derivatives. It obtains expressions through the third derivative.

  • Derivative Strategy: The incremental-channel technique used for mutual information derivatives also supports derivatives of estimation-theoretic quantities.
  • Zero-SNR Expansion: For zero-mean, unit-variance inputs with finite higher-order moments, MMSE admits a Taylor expansion near zero SNR.
  • Zero-SNR Expansion: An arbitrary finite-moment input is centered and normalized using its mean and variance before applying the expansion.
  • Arbitrary-SNR Derivatives: Posterior MMSE representations transfer the known zero-SNR derivatives to arbitrary SNR by averaging over the observation.
  • Arbitrary-SNR Derivatives: The same posterior-averaging argument yields the second and third MMSE derivatives.
Loading 1004.3332v1…