PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Correntropy for Robust Adaptive Filtering

Badong Chen, Lei Xing, Haiquan Zhao, Nanning Zheng, José C. Príncipe

arXiv:1504.02931v1stat.MLcs.IT

TL;DR

The paper addresses the limitation of using only a Gaussian kernel in correntropy. It generalizes correntropy with a generalized Gaussian density, proposes GMCC and a GMCC adaptive-filtering algorithm, and reports zero probability of divergence with theoretically supported and desirable simulated performance.

  • Problem

    Correntropy’s conventional Gaussian kernel is not always the best choice for robust estimation and adaptive filtering.

  • Method

    The paper uses a generalized Gaussian density as the correntropy kernel, proposes GMCC, and derives a stochastic-gradient adaptive-filtering algorithm with convergence analysis.

  • Results

    The GMCC algorithm can achieve zero probability of divergence, has a derived steady-state EMSE, and shows desirable performance in simulations.

  • Takeaways & Limitations

    Generalized correntropy provides a flexible framework that includes Gaussian correntropy as a special case and supports robust adaptive filtering.

Abstract

from arXiv · show

As a robust nonlinear similarity measure in kernel space, correntropy has received increasing attention in domains of machine learning and signal processing. In particular, the maximum correntropy criterion (MCC) has recently been successfully applied in robust regression and filtering. The default kernel function in correntropy is the Gaussian kernel, which is, of course, not always the best choice. In this work, we propose a generalized correntropy that adopts the generalized Gaussian density (GGD) function as the kernel (not necessarily a Mercer kernel), and present some important properties. We further propose the generalized maximum correntropy criterion (GMCC), and apply it to adaptive filtering. An adaptive algorithm, called the GMCC algorithm, is derived, and the mean square convergence performance is studied. We show that the proposed algorithm is very stable and can achieve zero probability of divergence (POD). Simulation results confirm the theoretical expectations and demonstrate the desirable performance of the new algorithm.

I. INTRODUCTION

The paper motivates replacing the Gaussian correntropy kernel with a generalized Gaussian density and develops GMCC for robust adaptive filtering. It establishes generalized correntropy properties, derives the GMCC algorithm, analyzes convergence, and reports zero POD with desirable simulated performance.

  • Motivation: Non-Gaussian noise motivates non-quadratic error costs because MSE can degrade considerably outside Gaussian settings.Lower-order measures are more robust to heavy-tailed impulsive noise, while higher-order measures can be desirable for light-tailed non-Gaussian noise.
  • Related adaptive costs: LMF can converge faster and achieve lower steady-state MSD than LMS in light-tailed noise, but its stability depends on signal, noise, and initialization conditions.The LMF stability guarantee is not established generally.
  • Related adaptive costs: The sign algorithm is robust to large impulsive noises, but its convergence speed and steady-state performance are generally inferior.This illustrates the trade-off between robustness and adaptive-filtering performance.
  • Correntropy: Correntropy is a nonlinear local similarity measure that is insensitive to outliers, making it a robust adaptation cost for heavy-tailed impulsive noise.With a Gaussian kernel, its induced metric behaves like L2, L1, and L0 norms as data move from relatively small to far from the origin.
  • Generalized correntropy: The Gaussian kernel is not always the best choice, so the paper uses the generalized Gaussian density as a correntropy kernel, not necessarily satisfying Mercer’s condition.The generalized family includes Gaussian and Laplace distributions as special cases and approaches a uniform density as α→∞.
  • Generalized correntropy: Generalized correntropy yields a generalized correntropy-induced metric whose order-α loss behaves like different norms, from Lα to L0, across data ranges.GMCC-based estimation is described as smoothed MAP estimation, with MAP and LMP estimation as extreme cases.
  • GMCC adaptive filtering: The paper applies GMCC to adaptive filtering, derives a stochastic-gradient GMCC algorithm, analyzes mean-square convergence, and derives a theoretical steady-state EMSE.The optimal GMCC filtering solution resembles the Wiener solution, but error nonlinearity weights the autocorrelation matrix and cross-correlation vector.
  • Results: The GMCC algorithm can have zero probability of divergence, and simulations confirm theoretical expectations and desirable performance.The paper presents stability, steady-state performance, and Monte Carlo simulation results.

B. Properties

The generalized correntropy has symmetry, positivity, boundedness, metric-related structure, and limiting behaviors governed by its order and kernel parameters.

  • Generalized correntropy is symmetric, positive, and bounded above by GGGD(0).
  • For sufficiently small λ, generalized correntropy is approximately affine in the α-order absolute error moment.
  • When 0 < α ≤ 2, generalized correntropy is a second-order statistic of a nonlinear feature-space mapping.
  • The generalized correntropy induced metric defines a metric on the N-dimensional sample-vector space when 0 < α ≤ 2.
  • The GCIM behaves like different L_α-to-L_0 norms across regions of the data surface.
  • As λ approaches infinity, minimizing the generalized loss approximately becomes minimizing the L_0-norm; as λ approaches zero, it approaches an α-power loss.
  • The generalized loss is concave for 0 < α ≤ 1 and convex for α > 1 at nonzero error coordinates.

III. GENERALIZED MAXIMUM CORRENTROPY CRITERION

The generalized maximum correntropy criterion extends correntropy-based estimation by maximizing similarity under a generalized Gaussian kernel, linking estimation to smoothed density modes and power-loss limits.

  • GMCC estimates a function of observations by maximizing generalized correntropy between the target and its estimate.
  • GMCC therefore yields a smoothed maximum a posteriori estimate.
  • The GMCC estimator is the maximizer of a conditional density smoothed by convolution with the generalized Gaussian kernel.
  • As β approaches zero, GMCC estimation becomes MAP estimation.
  • The GMCC estimator can also be characterized by maximizing the smoothed error PDF at zero.
  • As β approaches infinity, GMCC estimation becomes least mean p-power estimation with p = α.

IV. ADAPTIVE FILTERING UNDER GMCC CRITERION

Under the GMCC criterion, linear adaptive filtering represents outputs through a weight vector and input vector, then maximizes a generalized correntropy cost of the resulting error.

  • The optimal weight vector is obtained by maximizing the GMCC cost over the filter weights.
  • The GMCC adaptive-filter cost is the expected generalized Gaussian function of the instantaneous error.
  • The linear filter output is y(i) = W^T X(i), and the error is e(i) = d(i) − W^T X(i).
  • The input vector comprises current and delayed samples, X(i) = [x(i), x(i−1), …, x(i−m+1)]^T.

B. Optimal solution

The optimal GMCC filter weights are characterized through error-dependent weighting, while Gaussian inputs and limiting parameter regimes recover classical least-squares solutions.

  • The optimal weight vector maximizing GMCC is characterized by an error-dependent weighting function applied to input correlations.
  • For α = 2 and λ approaching zero, the GMCC solution converges to the Wiener solution.
  • For zero-mean Gaussian input and desired processes, the GMCC-optimal solution equals the Wiener solution.
  • Under Gaussian errors, maximizing generalized correntropy is equivalent to minimizing error variance, yielding the Wiener solution.

C. Adaptive algorithm

The GMCC algorithm is derived by stochastic-gradient optimization of the generalized maximum correntropy criterion, with an error-dependent update that attenuates large errors. It includes MCC and LMP/LMS-related algorithms as special or limiting cases and is designed to improve robustness to outliers.

  • Algorithm derivation: The GMCC algorithm is obtained from the cost function using stochastic-gradient adaptation.Its update uses the prediction error, its sign, the input vector, and an error-dependent exponential factor.
  • Special cases: When α=2, GMCC becomes the original MCC algorithm.When λ approaches zero, GMCC reduces to the LMP algorithm with p=α; α=2 further yields LMS.
  • Interpretation: GMCC can be viewed as an LMP algorithm with p=α and a variable step-size.The variable step-size depends on the current error and decreases as the error grows.
  • Robustness: As the error magnitude tends to infinity, the GMCC variable step-size tends to zero, so large errors have little influence on filter weights.This property is stated to make GMCC robust to large outliers or impulsive noise.
  • Implementation: The GMCC algorithm has nearly the same computational complexity as LMP, requiring only the additional calculation of an exponential term.Variable-step-size and normalized GMCC variants are also identified.
  • Stability analysis: Under the stated signal model and assumptions, the mean-square analysis establishes decreasing and converging weight-error power under suitable step-size conditions.The paper also gives a scalar zero-noise example in which GMCC never diverges, while the general vector noisy-case POD analysis remains open.

B. Steady-State Mean Square Performance

The steady-state mean-square analysis derives an approximate EMSE expression for GMCC using independence assumptions and a Taylor expansion around the disturbance noise. The approximation is reliable when the steady-state a priori error is small.

  • Steady state: The resulting steady-state relation expresses EMSE through expectations of the nonlinear error function and the input covariance matrix.The steady-state form removes the time index because the error distributions are assumed stationary.
  • EMSE analysis: The steady-state EMSE is defined as the limiting mean-square a priori error.The analysis derives an approximate analytical expression for this quantity.
  • Assumptions: The derivation assumes zero-mean independent noise and a zero-mean a priori error independent of the noise.These are assumptions A1 and A2 used in the Taylor-based analysis.
  • Approximation: A Taylor expansion of the nonlinear error function around the noise yields an approximation involving first- and second-order derivatives.Third- and higher-order terms are treated as negligible when the a priori error is sufficiently small.
  • Validity boundary: A theoretical steady-state EMSE can be evaluated for a given noise distribution, but its accuracy decreases when step-size or noise power makes the a priori error large.Under those conditions, the neglected higher-order terms are no longer negligible.

VI. SIMULATION RESULTS

Simulations evaluate GMCC stability, steady-state EMSE, and convergence under several noise distributions and impulsive outlier levels. The results support the predicted stability, EMSE behavior, and robustness advantages over LMP-family methods.

  • Stability: GMCC did not diverge in the POD simulations, unlike the comparison LMF behavior, matching the theoretical expectation.POD evaluation used 1000 independent Monte Carlo simulations with 1000 iterations each.
  • Steady-state EMSE: Steady-state EMSE increased with both step-size and noise variance.Theoretical and simulated EMSEs were compared across these two conditions.
  • Steady-state EMSE: For small step-size and noise variance, simulated EMSE closely matched the theoretical value; discrepancies grew at larger settings.This behavior agrees with the analytical approximation's stated validity boundary.
  • Experimental setup: The impulsive-noise model combines a low-variance process with a substantially higher-variance outlier process selected by a binary occurrence process.The simulations varied A(i) across Gaussian, Binary, Laplace, and Uniform distributions.
  • Distributional comparisons: GMCC-family algorithms were more stable than LMP-family algorithms, with LMP failing to converge in the example when p>4.GMCC with α≠2 could significantly outperform original MCC, and α=6 performed best for Binary or Uniform A(i).
  • Outlier robustness: When outlier-noise variance increased from 15 to 100, most LMP-family algorithms diverged while GMCC-family algorithms continued to work well.The larger-outlier experiment used Uniform A(i) for the displayed convergence results.

VII. CONCLUSION

The paper generalizes correntropy by replacing the Gaussian kernel with a generalized Gaussian density and develops GMCC for adaptive filtering. The resulting algorithm has analyzed mean-square behavior, strong stability indications, and favorable simulation performance, while the generalized framework includes original correntropy as a special case.

  • Contribution: Generalized correntropy uses the generalized Gaussian density as a flexible kernel that is not necessarily a Mercer kernel.The Gaussian-kernel correntropy is recovered as a special case.
  • Contribution: GMCC is proposed as an optimality criterion and applied to derive an adaptive filtering algorithm.The paper studies the algorithm's mean-square convergence behavior.
  • Results: Theoretical analysis derives a steady-state EMSE and presents a simple example with zero probability of divergence.Monte Carlo simulations confirmed the theoretical results and reported favorable GMCC performance.

APPENDIX A

Appendix A establishes metric and limiting properties of the generalized correntropy-induced measure and analyzes curvature of the associated generalized correntropy loss.

  • Metric properties: The GCIM function defines a Euclidean distance in the Hilbert space.
  • Metric properties: GCIM is a metric in the sample vector space, satisfying nonnegativity and the triangle inequality.The appendix also identifies GCIM with a Euclidean distance in Hilbert space.
  • Limiting behavior: As λ becomes large, the relevant right-hand side approaches zero when x_i ≠ 0, supporting the stated limiting result.The bound uses ε as a small positive number arbitrarily close to zero.
  • Loss curvature: For 0 < α ≤ 1, the generalized correntropy loss has a nonpositive Hessian-related bound, while for α > 1 the corresponding bound is nonnegative.The appendix states these opposite inequalities for the loss with respect to e.

APPENDIX E

Appendix E analyzes invexity of the generalized correntropy loss and connects generalized correntropy maximization with conditional-density maximization.

  • Invexity: Invexity is characterized through an inequality involving function values, gradients, and a vector-valued function q(x_1, x_2).The appendix gives the defining inequality for all x_1 and x_2 in the domain.
  • Loss analysis: For α > 1, the generalized correntropy loss is differentiable with respect to e.
  • Loss analysis: The generalized correntropy loss has a stationary point at e = 0 under the stated bounded-error condition.The derivation uses the sign function and the condition |e_i| ≤ M.
  • Density interpretation: The generalized correntropy maximizer is also an argmax of the conditional density under the stated symmetry argument.The appendix derives this correspondence through integrals involving the generalized Gaussian function and conditional density.
Loading 1504.02931v1…