PaperlayerView exact revision on arXiv

Source-linked AI summary

A Novel Family of Adaptive Filtering Algorithms Based on The Logarithmic Cost

Muhammed O. Sayin, N. Denizcan Vanli, Suleyman S. Kozat

arXiv:1311.6809v1cs.LG

TL;DR

Adaptive filters face a trade-off between convergence, stability, and robustness to impulsive noise. The paper introduces a relative logarithmic cost framework and derives LMLS and LLAD algorithms that continuously combine error measures. LMLS matches LMF convergence with a larger stability bound, while LLAD matches LMS in impulse-free noise and remains robust under impulsive interference.

  • Problem

    Existing adaptive filters trade convergence and steady-state behavior against stability or robustness: LMF has stability issues, while least-squares methods degrade under impulsive interference and SA can converge slowly.

  • Method

    The paper introduces a relative logarithmic error cost that continuously combines lower- and higher-order error measures and derives the LMLS and LLAD algorithms.

  • Results

    LMLS achieves comparable convergence performance with LMF and a larger step-size stability bound; LLAD matches LMS in impulse-free noise and outperforms SA under impulsive interference.

  • Takeaways & Limitations

    The logarithmic framework provides adaptive algorithms that combine convergence benefits with stability or impulsive-noise robustness across the evaluated scenarios.

Abstract

from arXiv · show

We introduce a novel family of adaptive filtering algorithms based on a relative logarithmic cost. The new family intrinsically combines the higher and lower order measures of the error into a single continuous update based on the error amount. We introduce important members of this family of algorithms such as the least mean logarithmic square (LMLS) and least logarithmic absolute difference (LLAD) algorithms that improve the convergence performance of the conventional algorithms. However, our approach and analysis are generic such that they cover other well-known cost functions as described in the paper. The LMLS algorithm achieves comparable convergence performance with the least mean fourth (LMF) algorithm and extends the stability bound on the step size. The LLAD and least mean square (LMS) algorithms demonstrate similar convergence performance in impulse-free noise environments while the LLAD algorithm is robust against impulsive interferences and outperforms the sign algorithm (SA). We analyze the transient, steady state and tracking performance of the introduced algorithms and demonstrate the match of the theoretical analyzes and simulation results. We show the extended stability bound of the LMLS algorithm and analyze the robustness of the LLAD algorithm against impulsive interferences. Finally, we demonstrate the performance of our algorithms in different scenarios through numerical examples.

EDICS Category: MLR-LEAR, ASP-ANAL, MLR-APPL

The paper develops a logarithmic error-cost framework that continuously combines lower- and higher-order error measures, targeting the convergence, stability, and robustness trade-offs of adaptive filters.

  • Motivation: Least-squares methods can be unstable or vulnerable to impulsive noise, while robust lower-order methods often converge more slowly.LMF stability depends on input and noise power and initial weights; SA is robust but can converge slowly.
  • Analysis scope: The framework covers system identification and supports analysis of transient, steady-state, and tracking performance.The paper considers an unknown system vector, input and desired signals, and a logarithmically normalized error cost.
  • Logarithmic cost framework: The proposed logarithmic cost continuously combines different error orders without the breaking point of switched-norm methods.Its logarithmic term is less steep for small perturbations, while the cost resembles the conventional function for large errors.
  • Analysis scope: The logarithmic cost has the same optimal solution as the underlying conventional cost when the latter is non-negative and convex.The paper attributes this to the positive semidefiniteness of the logarithmic-cost Hessian.
  • Logarithmic cost framework: For small errors, the logarithmic cost uses higher-order error statistics; for large errors, it approaches the conventional cost function.This design aims to retain convergence benefits for small perturbations while preserving conventional behavior for larger errors.

III. NOVEL ALGORITHMS

The general logarithmic update applies a differentiable normalization to conventional error costs, with a design parameter controlling the transition toward the underlying cost.

  • General update: The general steepest-descent update is obtained from the gradient of the logarithmic cost, using step size µ and design parameter α.Both parameters are positive, and α controls the asymptotic cut-off behavior.
  • Design parameter: Larger α reduces the logarithmic-term weight, making the resulting algorithm behave more like one minimizing the conventional cost.The parameter is introduced to determine the asymptotic cut-off value.
  • General update: The stochastic version replaces the expected conventional cost with a sample-wise function to produce adaptive updates.The framework covers LMS, sign, and normalized updates through different choices of the conventional cost.

A. The Least Mean Logarithmic Square (LMLS) Algorithm

The LMLS update behaves like LMF for small errors and LMS for large perturbations, while the logarithmic LLAD construction combines LMS and SA for robust adaptation.

  • LMLS algorithm: For small errors, LMLS resembles an LMF update; for large errors, it behaves like LMS.This combines lower steady-state error from fourth-order statistics with the stability associated with least-squares updates.
  • LMLS algorithm: The LMLS design combines LMS and LMF behavior continuously according to the error amount rather than through an artificial mixing parameter.The paper presents this as an alternative to mixed LMF + LMS algorithms.
  • LLAD algorithm: The sign algorithm is robust to impulsive interference but has slower convergence because it uses the lowest convex error power.The logarithmic framework modifies the absolute-error cost to improve this convergence behavior.
  • LLAD algorithm: LLAD combines LMS and SA into one robust algorithm with improved convergence performance.The paper states that α can be optimized to improve performance over SA in impulsive-noise environments.

C. Normalized Updates

The paper introduces normalized updates to reduce dependence on input correlation statistics and derives normalized LMLS and NLLAD algorithms within the logarithmic-cost framework.

  • Normalized updates are introduced to provide independence from input data correlation statistics under certain settings.
  • The normalized least mean logarithmic square algorithm is derived from the new objective and update framework.
  • The normalized least logarithmic absolute difference algorithm is obtained analogously for the absolute-difference cost.
  • The weighted-energy recursion is formulated with a symmetric positive definite weighting matrix Σ and design parameter α.
  • Transient analysis uses Gaussian, independence, and decorrelation assumptions for the regressor, noise, estimation error, and weighted error.

A. Transient Analysis

The transient analysis develops learning-curve recursions for MSD and EMSE using Gaussian assumptions, weighted-energy identities, and state-space representations.

  • The analysis evaluates the weighted-error recursion term by term and derives a general form using a Gaussian-variable lemma.
  • For white regressor data, the MSD recursion is closed because hU(et) and hG(et) are tabulated and the right-hand side depends only on MSD.
  • The state-space recursion represents transient behavior for correlated regression data through a coefficient matrix built from the characteristic polynomial of R.
  • The state vector’s top and second entries yield the time evolution of MSD and the learning curves for EMSE, respectively.
  • The transient analysis provides the basis for subsequent steady-state EMSE and MSD analysis of LMLS and LLAD.

B. Steady State Analysis

The steady-state analysis derives EMSE and MSD expressions for LMLS and LLAD, then compares theoretical predictions with simulations across step sizes and Gaussian settings.

  • The steady-state analysis derives EMSE and MSD expressions for LMLS and LLAD from the transient recursions under stated asymptotic assumptions.
  • For α = 1, the LMLS EMSE expression matches the EMSE of the LMF algorithm, with smaller roots matching simulations.
  • Different α values can yield similar steady-state convergence by changing the step size, while smaller α produces a slower convergence rate.
  • For sufficiently small α, LLAD achieves similar steady-state convergence performance to LMS under the zero-mean Gaussian error-signal assumption.
  • Theoretical steady-state MSD matches simulation results for LMLS and LLAD across plotted step sizes and Gaussian system-identification settings.

C. Tracking Performance

The tracking analysis models time-varying filter weights as a random walk and derives tracking EMSE expressions for LMLS and LLAD.

  • The non-stationary analysis assumes a random-walk model for the time-varying parameter vector, driven by a zero-mean process with covariance Q.
  • The random-walk model preserves the definition of the a priori error used in the tracking analysis.
  • The LMLS tracking EMSE is the same as the LMF tracking EMSE under the stated assumption.
  • A corresponding tracking EMSE expression is derived for LLAD through its stated assumption.
  • The paper frames the subsequent comparison around stability bounds and robustness against impulsive noise.

A. Stability Bound for the LMLS Algorithm

The analysis establishes that the logarithmic-cost framework provides at least the conventional step-size stability bound, while LMLS extends LMS stability and remains comparable to LMF. For LLAD, the analysis models impulsive noise and shows that optimizing α improves convergence and agrees with simulations.

  • Stability analysis: The introduced algorithms have a step-size bound at least as large as their corresponding conventional algorithms.The effective step size satisfies µ′ ≤ µ irrespective of α.
  • Stability analysis: LMLS extends the LMS stability bound while achieving performance comparable to LMF, which has stability issues.The comparison concerns the same LMS bound with β = 1.
  • LLAD robustness analysis: The impulsive-noise analysis specifically evaluates LLAD because its cost uses f(e_t) = |e_t| and targets improved steady-state performance over SA.Gaussianity and Price’s Theorem are not applicable in the impulsive-noise setting.
  • LLAD robustness analysis: Increasing the impulse frequency ν_i causes larger steady-state EMSE, while optimizing α can minimize that EMSE.The paper derives an approximate optimum design parameter for impulsive noise.

VI. NUMERICAL EXAMPLES

Numerical experiments compare LMLS, LLAD, LMS, LMF, and SA under impulse-free and impulsive-noise conditions using matched steady-state MSD settings. LMLS matches LMF while extending stability, and LLAD matches LMS without impulses but remains robust when impulses prevent LMS convergence.

  • Scenario 1 (impulse-free environment): LMLS and LMF achieve comparable convergence, while LMLS outperforms LMS for relatively small step sizes.The impulse-free comparison uses σ_n^2 = 0.01 and α = 1.
  • Scenario 1 (impulse-free environment): LMLS remains comparable to LMS at larger step sizes, whereas LMF is unstable in that setting.The comparison uses µ_LMLS = 0.1 and µ_LMS = 0.0047.
  • Scenario 1 (impulse-free environment): LLAD has convergence performance comparable to LMS and improves the convergence performance of SA in impulse-free noise.The comparison is made using MSD trajectories for LLAD, SA, and LMS.
  • Scenario 2 (impulsive noise environment): In impulsive-noise environments, LMS does not converge, while LLAD performs better than SA.The experiments use 1%, 2%, and 5% impulse frequencies with σ_ni^2 = 10^4.
  • Concluding remarks: The experiments support the paper’s conclusion that LMLS extends stability and LLAD is robust against impulsive interferences while improving over SA.The authors report that the theoretical analyses match the simulation results across the evaluated scenarios.

APPENDIX A EVALUATION OF hG(et)

Appendix A evaluates the auxiliary hG(e_t) terms for the LMLS and LLAD updates. The derivations use the definitions of g(e_t), Gaussian-error assumptions, and special-function identities.

  • The LMLS Algorithm: The LMLS derivation obtains hG(e_t) by substituting the evaluated expectation terms into the update expression.The calculation uses σ_e^2 and the complementary error function erfc(·).
  • The LLAD Algorithm: The LLAD derivation obtains hG(e_t) through the corresponding expectation calculation for its logarithmic absolute-difference update.The derivation uses the imaginary error function erfi(z) and exponential integral Ei(x).

APPENDIX B EVALUATION OF hU(et)

Appendix B evaluates hU(e_t) for the LMLS and LLAD algorithms. Both derivations rely on interchanging integration and differentiation, with the LLAD result also using the earlier LLAD evaluation.

  • The LMLS Algorithm: For LMLS, hU(e_t) is derived by applying the interchange of integration and differentiation property.The derivation explicitly invokes this property for θ(e_t, α).
  • The LLAD Algorithm: For LLAD, hU(e_t) follows the same derivation pattern and uses the previously obtained LLAD expression.The final step applies the result identified as equation (31).
Loading 1311.6809v1…