Source-linked AI summary
Kernel Risk-Sensitive Loss: Definition, Properties and Application to Robust Adaptive Filtering
Badong Chen, Lei Xing, Bin Xu, Haiquan Zhao, Nanning Zheng, Jose C. Principe
TL;DR
The paper addresses limitations of existing similarity measures for robust learning with non-Gaussian data. It proposes KRSL in kernel space, applies it to adaptive filtering through MKRSL, and reports faster convergence, higher accuracy, and maintained outlier robustness, with theory supported by simulations.
Problem
Second-order measures are limited for non-Gaussian data, while correntropy's non-convex loss surface can cause slow convergence and poor accuracy.
Method
The paper defines KRSL in RKHS, applies it to adaptive filtering, develops MKRSL, and analyzes its mean square convergence.
Results
KRSL offers faster convergence and higher accuracy than correntropy-based methods while maintaining robustness to outliers; simulations confirm the theoretical results and better convergence performance.
Takeaways & Limitations
KRSL provides a robust kernel-space criterion whose performance surface supports improved adaptive-filtering convergence compared with existing algorithms.
Abstract
from arXiv · showhide
Nonlinear similarity measures defined in kernel space, such as correntropy, can extract higher-order statistics of data and offer potentially significant performance improvement over their linear counterparts especially in non-Gaussian signal processing and machine learning. In this work, we propose a new similarity measure in kernel space, called the kernel risk-sensitive loss (KRSL), and provide some important properties. We apply the KRSL to adaptive filtering and investigate the robustness, and then develop the MKRSL algorithm and analyze the mean square convergence performance. Compared with correntropy, the KRSL can offer a more efficient performance surface, thereby enabling a gradient based method to achieve faster convergence speed and higher accuracy while still maintaining the robustness to outliers. Theoretical analysis results and superior performance of the new algorithm are confirmed by simulation.
I. INTRODUCTION
The paper motivates kernel-space similarity measures for non-Gaussian data and introduces KRSL to improve the performance-surface trade-off between convergence and robustness. It applies KRSL to adaptive filtering through MKRSL and analyzes its convergence, with simulations supporting the theoretical results.
- Motivation: Second-order measures such as correlation and MSE are tractable under Gaussian and linear assumptions but cannot fully quantify similarity for non-Gaussian variables.
- Motivation: Kernel-space information-theoretic costs can capture higher-order statistics and improve solutions over MSE in non-Gaussian and nonlinear signal processing.
- Motivation: Correntropy reduces the influence of outliers, but its loss surface can be highly non-convex, causing slow convergence and poor accuracy.
- Contribution: KRSL is a kernel-space risk-sensitive similarity measure with a bounded, potentially more convex surface controlled by kernel bandwidth and a risk-sensitive parameter.
- Contribution: The paper develops MKRSL for robust adaptive filtering, analyzes mean square convergence, and uses Monte Carlo simulations to confirm theoretical results and demonstrate performance.
B. Kernel Risk-Sensitive Loss
KRSL is designed to retain robustness while improving the optimization surface of correntropy-based losses. Its properties characterize boundedness, limiting relationships to C-Loss, MSE, and l0 behavior, and convexity controlled by λ and σ.
- Motivation: C-Loss is robust to large outliers but can be highly non-convex, creating a trade-off between convergence behavior and robustness when adjusting σ.
- Definition: KRSL preserves the traditional risk-sensitive-loss form while defining the measure in kernel space; empirical KRSL replaces the expectation with a finite-sample average.
- Properties: KRSL is symmetric, positive, bounded, and reaches its minimum if and only if X = Y.
- Limiting cases: For small λ, KRSL is approximately equivalent to C-Loss, while for large σ it is approximately equivalent to MSE.
- Convexity: The empirical KRSL is convex whenever ∥e∥∞ ≤ σ, and larger λ generally expands the convex range beyond that region.
- Limiting cases: As σ becomes very large, empirical KRSL behaves like a squared L2 norm, whereas as σ becomes very small it behaves approximately like an L0 norm.
A. Performance Surface
The MKRSL criterion defines an adaptive-filtering performance surface with an optimal-solution condition, while its surface can be more favorable for gradient search than C-Loss.
- Setup: The adaptive-filtering setup estimates an unknown FIR weight vector from observed responses, inputs, and additive noise under the KRSL cost.The error is e(i) = d(i) − W^T X(i), and the optimum minimizes J_KRSL(W).
- Surface comparison: Figure 1 compares contours and gradients of the C-Loss and KRSL performance surfaces.The figure presents C-Loss in panel (a) and KRSL in panel (b).
- Optimal solution: The MKRSL optimal solution satisfies the theorem’s stationary-condition equation under the stated invertibility requirement.The relevant weighted input-correlation matrix must be invertible.
- Optimal solution: As σ → ∞, the MKRSL solution becomes the well-known Wiener solution, although the finite-bandwidth equation is not closed form.The right-hand side depends on W through e(i).
- Surface comparison: The C-Loss surface is flat far from the optimum and sharp near it, which may slow gradient-based convergence from distant initial estimates.The comparison uses m = 2, σ = 2.0, λ = 10, and N = 10000 under white Gaussian inputs and noise.
B. Robustness Analysis
The MKRSL criterion is analyzed for robustness to impulsive noise, with explicit scalar-case bounds showing accurate estimation when most samples have small noise despite arbitrarily large outliers.
- Setting: The robustness analysis considers scalar FIR identification, where both the weight and input are scalars.The authors state that analogous vector-case norm bounds are difficult to derive.
- Assumptions: The assumptions require more than half the samples to have noise bounded by ε_v, while remaining samples may contain arbitrarily large outliers.A nonzero input-amplitude lower bound is also assumed for the bounded-noise samples.
- Bounds: Theorem 2 bounds the scalar estimation error as |W_MKRSL − W_0| ≤ ξ under the stated assumptions.The expression for ξ is given in the paper’s theorem.
- Bounds: When σ exceeds the specified threshold, Corollary 1 gives |W_MKRSL − W_0| ≤ ρε_v.The bound can remain small when ε_v is small, even with N − M arbitrarily large outliers, provided M > N/2 samples have small noise.
C. Stochastic Gradient Adaptive Algorithm
The MKRSL adaptive algorithm is a stochastic-gradient LMS variant with an error-dependent variable step size. Its step size can favor convergence for positive risk sensitivity while vanishing for large errors, preserving robustness.
- Algorithm: The paper derives a stochastic-gradient adaptive-filtering algorithm from the instantaneous MKRSL cost.W(i) denotes the estimated weight vector and η = μ/σ^2 is the step-size parameter.
- Algorithm: The resulting MKRSL algorithm is an LMS method with variable step size η(i) = η exp(λ(1 − κ_σ(e(i))))κ_σ(e(i)).The variable step size depends on the instantaneous error through the kernel function.
- Step-size behavior: For λ > 0, η(i) may peak away from zero error, potentially yielding faster convergence and better accuracy.When λ → 0+, the algorithm approaches the MCC algorithm.
- Step-size behavior: For every λ, η(i) approaches zero as |e(i)| → ∞, making the MKRSL algorithm insensitive to large errors.This error-dependent attenuation is the stated robustness mechanism for impulsive errors.
- Complexity: The MKRSL algorithm has almost the same computational complexity as MCC, with only the extra calculation of exp(λ(1 − κ_σ(e(i)))).The added term is the principal extra computational demand identified by the authors.
- Step-size behavior: Figure 2 plots η(i) against e(i) for different λ values with σ = η = 2.0.The key comparison is how λ changes the location and shape of the step-size response.
D. Mean Square Convergence Performance
The paper derives transient and steady-state mean-square convergence results for MKRSL under explicit independence, Gaussianity, and small-error assumptions, using numerical integration or Taylor expansion when closed forms are unavailable.
- Transient Behavior: Theoretical convergence analysis characterizes MKRSL transient behavior through an energy-conservation relation and a dynamic recursion for weight-error power.
- Transient Behavior: Theorem 3 derives the transient relation assuming independent noise and input, Gaussian a priori error, and uncorrelated input-energy and nonlinear-error terms.
- Steady-State Behavior: Because hG(.) and hU(.) generally lack closed forms, their values can be obtained by numerical integration, while Taylor expansion provides an approximate EMSE solution.
- Steady-State Behavior: The steady-state EMSE is characterized as a positive solution of an equation involving the functions hG(.) and hU(.).
- Steady-State Behavior: Theorem 4 further assumes zero-mean independent noise, independence of a priori error and noise, and negligible third- and higher-order steady-state error terms.
IV. SIMULATION RESULTS
The simulation section tests the theoretical analysis and evaluates the performance of the proposed MKRSL algorithm.
- Simulations are presented to confirm the theoretical analysis and demonstrate the performance of the proposed MKRSL algorithm.
A. Verification of Theoretical Results
Simulations verify that MKRSL theory closely predicts convergence and steady-state EMSE, while comparative experiments show MKRSL outperforming the tested alternatives.
- Verification of Theoretical Results: Theoretical weight-error-power convergence curves match simulations averaged over 1000 Monte Carlo runs across different parameter settings.
- Verification of Theoretical Results: Theoretical steady-state EMSE predictions agree closely with simulated convergence curves across parameter settings, step-sizes, noise variances, and noise distributions.
- Performance Comparison with Other Algorithms: The comparison evaluates MKRSL against LMS, SA, LMMN, LMM, and GMCC under Gaussian, binary, uniform, and sine-wave noise conditions.
- Performance Comparison with Other Algorithms: MKRSL significantly outperforms the other tested algorithms after matching their initial convergence speeds and selecting parameters for desirable performance.
C. Effects of the Parameters λ and σ
MKRSL performance depends strongly on the risk-sensitive parameter λ and kernel bandwidth σ, while simulations report robustness to outlier amplitudes and frequencies and better convergence than existing algorithms.
- Parameter effects: λ and σ both significantly influence MKRSL convergence, and desirable performance requires appropriate parameter settings.The paper leaves optimal parameter selection open and suggests manual tuning or trial-and-error methods.
- Experimental settings: Convergence curves are evaluated under Gaussian, binary, uniform, and sine-wave distributions of A(i).Figure 6 averages weight-error-power convergence curves over 100 independent Monte Carlo runs.
- Outlier effects: MKRSL remains robust to outlier amplitudes, with performance potentially improving as outlier variance increases.This behavior is reported from simulations using different outlier variances and occurrence frequencies.
- Outlier effects: The steady-state weight error power increases only slightly from 0.0096 to 0.013 as outlier frequency rises from 0% to 30%.The simulations characterize MKRSL as robust to outlier occurrence frequencies.
- Algorithm performance: Simulation results confirm theoretical predictions and show better convergence performance than some existing algorithms.The reported comparisons concern convergence performance in the adaptive-filtering experiments.
APPENDIX A PROOF OF THEOREM 2
The appendix proves the theorem by showing that the KRSL objective at the MKRSL solution is lower than at the reference solution and excludes sufficiently distant parameter values.
- Proof strategy: The proof begins from the objective inequality J_KRSL(W_MKRSL) ≤ J_KRSL(W_0).This establishes the comparison between the MKRSL estimate and the reference parameter W_0.
- Proof strategy: It suffices to show J_KRSL(W) > J_KRSL(W_0) whenever |W − W_0| > ξ.The argument reduces the desired parameter-error bound to an objective-separation condition.
- Proof conditions: The proof invokes Assumption 2 and the bound |v(i)| ≤ ε_v for indices in I(ε_v).These conditions support the intermediate inequality used to establish objective separation.
- Proof completion: The final inequality uses ε_v ≥ |v(i)| and exp(−exp(λ)) < 0, completing the proof.The appendix identifies these bounds as the basis for the concluding comparison.