PaperlayerView exact revision on arXiv

Source-linked AI summary

Combinations of Adaptive Filters

Jerónimo Arenas-García, Luis A. Azpicueta-Ruiz, Magno T. M. Silva, Vitor H. Nascimento, Ali H. Sayed

arXiv:2112.12245v1eess.SYcs.LGcs.SY

TL;DR

Running multiple adaptive filters can raise computational-cost concerns; this overview reviews combination schemes and practical mixing-layer estimation. Across its studied scenarios, the schemes offer competitive performance against application-specific state-of-the-art methods.

  • Problem

    Running two or more filters in parallel can be a concern where computational cost is at a premium.

    Source: §VII · p. 19

  • Method

    The paper reviews adaptive-filter combination schemes and practical methods for estimating the time-varying combination layer.

    Source: §XII · p. 28

  • Results

    Across all studied scenarios, combination schemes offer competitive performance against state-of-the-art methods for each application.

    Source: §XII · p. 28

  • Takeaways & Limitations

    Their simplicity and competitive performance make combination structures attractive for demanding applications requiring enhanced performance.

    Source: §XII · p. 28

    The theoretical performance results concern optimally adjusted combinations; practical mixing-parameter estimation adds noise, which can justify convexity constraints.

    Source: §III · p. 6

Abstract

from arXiv · show

Adaptive filters are at the core of many signal processing applications, ranging from acoustic noise supression to echo cancelation, array beamforming, channel equalization, to more recent sensor network applications in surveillance, target localization, and tracking. A trending approach in this direction is to recur to in-network distributed processing in which individual nodes implement adaptation rules and diffuse their estimation to the network. When the a priori knowledge about the filtering scenario is limited or imprecise, selecting the most adequate filter structure and adjusting its parameters becomes a challenging task, and erroneous choices can lead to inadequate performance. To address this difficulty, one useful approach is to rely on combinations of adaptive structures. The combination of adaptive filters exploits to some extent the same divide and conquer principle that has also been successfully exploited by the machine-learning community (e.g., in bagging or boosting). In particular, the problem of combining the outputs of several learning algorithms (mixture of experts) has been studied in the computational learning field under a different perspective: rather than studying the expected performance of the mixture, deterministic bounds are derived that apply to individual sequences and, therefore, reflect worst-case scenarios. These bounds require assumptions different from the ones typically used in adaptive filtering, which is the emphasis of this overview article. We review the key ideas and principles behind these combination schemes, with emphasis on design rules. We also illustrate their performance with a variety of examples.

I. PROBLEM FORMULATION AND NOTATION

The section introduces adaptive-filter combinations as a way to combine component estimates while adapting their relative weights. It outlines their modeling assumptions, design flexibility, potential robustness, and computational tradeoff.

I. PROBLEM FORMULATION AND NOTATION

  • Under a linear regression model with i.i.d. noise independent of the regressors, steady-state MSE, EMSE, and MSD are defined as limiting values as n increases.The model allows the optimal solution to vary over time.Source: §I · p. 2
  • Different adaptive schemes are distinguished by their update functions and any additional state information required for filter updates.Source: §I · p. 2

II. A BASIC COMBINATION OF TWO ADAPTIVE FILTERS

  • Two component filters process the same input and reference signals, while a combination layer adapts their weights to optimize overall performance.The combined output uses the component outputs and a mixing parameter.Source: §II · pp. 2–3
  • Convex combinations constrain the mixing parameter to [0, 1], whereas affine combinations allow any real value; adapting it helps track model drift.Source: §II · p. 3
  • Most algorithms remain usable with unequal component lengths by zero-padding the shorter filter; explicitly constructing the extended vector is unnecessary.Source: §II · p. 3
  • Combinations can increase robustness when filtering conditions are unknown, imprecise, or changing over time.Source: §II · p. 4
  • Combining diverse filters can enhance performance beyond an individual filter, while same-family alternatives can let the mixture favor better parameter values over time.The parameters may include step size or forgetting factor.Source: §II · p. 4
  • The main cost is adapting both component filters, although joint-adaptation strategies can require only a slight increment over one filter.Most mixing-parameter adaptation schemes add little computational cost.Source: §II · p. 4

III. OPTIMUM MIXING PARAMETER AND COMBINATION PERFORMANCE

The section derives optimal affine and convex mixing parameters by minimizing the combination’s MSE, then characterizes performance across four operating cases. The theoretical results concern optimally adjusted combinations; practical mixing-parameter adaptation adds noise.

III. OPTIMUM MIXING PARAMETER AND COMBINATION PERFORMANCE

  • The cross-EMSE cannot be larger in absolute value than both individual EMSEs simultaneously.Source: §III · p. 5
  • When convexity constraints are active, the convex combination selects the better component, while the affine combination can outperform both components.This describes cases 1 and 2 in the four-regime analysis.Source: §III · pp. 5–6
  • In case 3, affine and convex combinations share an optimum mixing parameter and outperform both components when cross-EMSE is small.The passage attributes the gain to weak correlation between component a priori errors and reduced estimation-error variance.Source: §III · p. 6
  • Case 4 yields the same EMSE as either component regardless of the mixing parameter, but requires exact equality of the two EMSEs and their cross-EMSE.The passage says this condition is rarely encountered in practice.Source: §III · p. 6
  • These performance results assume optimally adjusted affine and convex combinations; practical mixing-parameter estimation adds noise that can justify convexity constraints.Source: §III · p. 6

A. Cross-EMSE of LMS and RLS filters

This section reviews steady-state EMSE and cross-EMSE results for LMS and RLS filters under a random-walk model of the optimum solution. The model exposes a tradeoff between filter performance and tracking ability, while also implying divergence of the optimum solution's expected squared norm.

A. Cross-EMSE of LMS and RLS filters

  • Under a random-walk optimum with zero-mean changes of covariance Q, steady-state EMSE reflects a tradeoff between filter performance and tracking ability.The model assumes q(n) has covariance Q = E{q(n)q⊤(n)}.Source: A. Cross-EMSE of LMS and RLS filters · p. 6
  • Energy-conservation approximations give LMS and RLS steady-state EMSE, their optimal adaptation parameters, and each family’s minimum achievable EMSE in Table II.The table covers LMS step size µ, RLS forgetting factor 1 − β, their optimal values, and ζo(∞).Source: A. Cross-EMSE of LMS and RLS filters · p. 6, Table II · p. 7
  • The random-walk model implies that the optimum solution’s expected squared norm diverges as time increases, although it is used for simplicity and yields similar results to more realistic tracking models.Source: A. Cross-EMSE of LMS and RLS filters · p. 6
  • Table III summarizes steady-state cross-EMSE for the LMS and RLS adaptive-filter families.Source: Table III · p. 7

B. Combination of two LMS filters with different step sizes

Combining LMS filters with different step sizes can improve tracking across changing solution speeds. The examples show broader low-NSD ranges than individual filters, while the combination cannot beat the optimum single filter when the change rate is known.

B. Combination of two LMS filters with different step sizes

C. Improving the tracking performance of LMS and RLS filters

The section examines tracking when the change covariance Q shifts between conditions favoring LMS and RLS. It shows that combining the filters can achieve EMSE values unavailable to either filter alone.

C. Improving the tracking performance of LMS and RLS filters

IV. ESTIMATING THE COMBINATION PARAMETER

The section reviews gradient-based ways to learn the mixing parameter, including unconstrained affine updates and convex updates that keep the mixture weight in range. It discusses normalization, stability, and the trade-offs that guide rule selection.

IV. ESTIMATING THE COMBINATION PARAMETER

  • The ideal linear mixing parameter minimizes the combined MSE, but is unrealizable, motivating practical gradient-based updates for affine and convex combinations.The mixing layer can be viewed as a second adaptive filter, although its component-output difference has strongly correlated, time-varying power.Source: §IV · p. 10
  • In aff-LMS, a large step size can cause instability during initial convergence, while time-varying difference-signal power makes step-size selection difficult and can leave EMSE away from optimum.Constraining λ(n) to be at most one can address the initial instability, but does not necessarily attain optimal EMSE.Source: §IV · p. 10
  • Power-normalized aff-LMS uses a low-pass estimate of the component-output difference power, is less sensitive to the filtering scenario, and converges in mean square for 0 < µλ < 2.The unnormalized rule is obtained by setting p(n) = 1.Source: §IV · p. 10, Table IV · p. 11
  • Convex rules adapt an auxiliary parameter through an activation function so λ remains in (0, 1); cvx-LMS and cvx-PN-LMS update that parameter by gradient descent.The power-normalized version is reported as more robust to SNR changes and as simplifying selection of µa relative to cvx-LMS.Source: §IV · pp. 10–11
  • The factor λ(n)[1 − λ(n)] reduces gradient noise near the convex-weight boundaries, while saturation of a(n) ensures a minimum adaptation level.The saturation bounds a(n) and therefore constrains λ(n) to a corresponding interval.Source: §IV · pp. 11–12
  • Affine combinations can achieve smaller EMSE than convex ones in some cases, but convex combinations may be preferred unless theory predicts significant affine gains.Because convex weights are constrained, cvx-LMS and cvx-PN-LMS do not diverge, permitting large µa values; gradient noise still calls for appropriate step-size selection.Source: §IV · pp. 11–12

A. Convergence properties of combination filters

The simulations compare affine and convex power-normalized combination rules for two NLMS filters, assessing convergence against the optimal mixing parameter and steady-state excess error. The convex power-normalized rule approaches optimal mixing across the tested step sizes without a larger residual error trade-off.

A. Convergence properties of combination filters

  • The experiment combines NLMS filters with µ1 = 0.5 and µ2 = 0.01, and uses µλ = µa/800 for aff-PN-LMS to obtain comparable steady-state error.The optimum solution is a stationary length-7 vector, with input covariance R = 1/7 I and observation-noise SNR of 20 dB.Source: A. Convergence properties of… · p. 12
  • cvx-PN-LMS nearly matches optimal mixing for all three tested µa values and reaches the slow filter's roughly −50 dB final EMSE after first reaching the fast filter's −30 dB level.The combination results are averaged over 1000 experiments.Source: A. Convergence properties of… · p. 12
  • The faster convergence of cvx-PN-LMS does not come at the cost of larger steady-state residual error than aff-PN-LMS.The steady-state excess-error comparison is relative to the slow filter component.Source: A. Convergence properties of… · p. 12, Figure 4 · p. 13
  • The effective cvx-PN-LMS step size, µaλ(n)[1 − λ(n)], grows during component switching and shrinks at steady state, supporting fast adaptation and low residual error.The plots include the evolution of λ(n)[1 − λ(n)].Source: A. Convergence properties of… · pp. 12–13, Figure 4 · p. 13

B. Benefits of power normalized updating rules

The section compares power-normalized combinations, combinations against variable-step-size filters, and schemes that let component filters communicate. Across these examples, combinations can remain competitive under changing conditions, while coefficient transfer or feedback offers ways to reduce lag between components.

B. Benefits of power normalized updating rules

  • Across the tested SNRs and solution-change speeds, cvx-PN-LMS performs as well as the best component filter, whereas cvx-LMS degrades at fast changes and lower SNR.The comparison uses NLMS components with µ1 = 0.5 and µ2 = 0.01; a similar behavior is reported for aff-LMS versus aff-PN-LMS.Source: B. Benefits of power normalized… · p. 14, Figure 5 · p. 14

C. Combination schemes versus VSS LMS

  • The VSS comparison uses two LMS filters with µ1 = 0.01 and µ2 = 0.001, and evaluates a 10-tap filter across five optimum step sizes.The optimum solution-change covariance varies across the evaluated conditions.Source: C. Combination schemes versus VSS LMS · p. 15, Figure 6 · p. 15
  • The convex combination delivers the best possible performance across five optimum single-LMS step sizes, including values below µ2 and above µ1, and exceeds standard LMS and the compared VSS schemes overall.The VSS parameters were optimized over the interval [µ2, µ1].Source: C. Combination schemes versus VSS LMS · p. 15, Figure 6 · p. 15

V. COMMUNICATION BETWEEN COMPONENT FILTERS

  • Weight transfer and weight feedback are two communication schemes for equal-length component filters; feedback copies the combined weight vector to both components periodically.Weight transfer communicates in one direction, while feedback returns the combined weights to both filters.Source: §V · pp. 16–17, Figure 7 · p. 16
  • A slow filter can lag during initial convergence or after an abrupt optimum-weight change; leaking or copying coefficients from the fast filter can avoid waiting for it to catch up.Figure 8 illustrates a simple-copy example with a change at n = 5 × 10^4.Source: §V · pp. 16–17, Figure 8 · p. 17
  • Coefficient transfer can be triggered when λ(n) ≥ λ0; periodic copying uses intervals N0 ≥ 2 to avoid leaving slow-filter adaptation irrelevant.Copying has lower computational cost than gradual leakage, while feedback requires computing the combined weights every N0 samples.Source: §V · pp. 16–17, Table V · p. 16

VI. COMBINATION OF SEVERAL ADAPTIVE FILTERS

Combining more than two adaptive filters can make parameter selection less critical and alleviate multiple design compromises at once. The paper reviews hierarchical and one-layer topologies, which differ in how they combine filter outputs and adapt their mixing parameters.

VI. COMBINATION OF SEVERAL ADAPTIVE FILTERS

  • Combining K filters can simplify parameter selection and address several compromises simultaneously, including step size, filter length, and nonlinear-model choice.For example, a third filter with step size one can improve robustness when Tr{Q} > 10^-3.Source: §VI · p. 17

The hierarchical scheme

  • A hierarchical scheme combines pairs of filters in layers, adapting each mixing parameter to minimize its local combined-error power.For four filters, the outputs of two lower-layer combinations are combined in an upper layer.Source: The hierarchical scheme · p. 18, Figure 9 · p. 18

The one-layer scheme

  • A one-layer scheme combines K filter outputs directly; affine forms adapt K −1 parameters, while convex forms constrain K weights to be positive and sum to one.The convex form uses a softmax mapping of auxiliary parameters to enforce those constraints,,.Source: The one-layer scheme · pp. 18–19, Figure 9 · p. 18
  • Hierarchical mixing parameters can reveal which of two models differing in one setting is favored, providing information about the filtering scenario.The setting may be step size or filter length.Source: The one-layer scheme · p. 19

VII. REDUCED COST COMBINATIONS

Combination schemes can add computational cost, but companion filters and shared computations can reduce that burden. The section also describes a reduced-wordlength difference-filter implementation and illustrates how a mixing parameter can characterize signal modality.

Use a low-cost filter as companion to a high-cost one

  • A combination of RLS and LMS improves tracking over a single RLS filter with only about a 10% cost increase over lattice RLS.The estimate compares lattice RLS at about 16M multiplications, 8M additions, and 8M divisions with LMS at about 2M additions and 2M multiplications.Source: Use a low-cost filter as companion to… · p. 19

VII. REDUCED COST COMBINATIONS

Take advantage of redundancies in the component filters

  • Difference-filter implementations exploit similar component weights to use about half the first filter’s wordlength for the difference filter, with little to no performance degradation.The reduced wordlength applies to costly second-filter operations; the table is identified as a reduced-cost combination using two LMS filters.Source: Take advantage of redundancies in the… · p. 20, Table VI · p. 20

VIII. SIGNAL MODALITY CHARACTERIZATION

  • In a convex NLMS–NNGD prediction combination, λ(n) near one indicates a linear input process, while a value near zero suggests a nonlinear one.The mixing parameter is adapted to minimize prediction square error; its evolution tracks changes among linear autoregressive and benchmark nonlinear signals.Source: §VIII · p. 21, Figure 10 · p. 21

IX. ADAPTIVE BLIND EQUALIZATION

The examples use combinations to adaptively select between blind and decision-directed equalization, and to manage uncertainty about system sparsity. They report automatic equalizer switching and complementary performance-cost properties for two sparse-identification schemes.

IX. ADAPTIVE BLIND EQUALIZATION

  • The blind/DD combination automatically switches to decision-directed mode when equalization is sufficient, avoiding a preset transition-MSE threshold.The blind algorithm addresses initial convergence and abrupt channel changes; DD mode is used for lower error after steady-state MSE becomes sufficiently low.Source: §IX · pp. 21–22
  • Across 1000 runs of 256-QAM equalization at 40 dB SNR, the combination better detects when to switch to DD mode and shows lower switching-time variance than other methods.The channel includes an abrupt change; switching is considered complete when a run reaches −16 dB MSE.Source: §IX · p. 22, Figure 11 · p. 22
  • The blind/DD example extends combinations beyond MSE-based adaptation by using blind criteria based on high-order moments, including at the combination layer.The paper notes that similar use may apply in settings such as blind source separation.Source: §IX · p. 23

X. SPARSE SYSTEM IDENTIFICATION

  • The sparse-identification schemes address uncertainty in sparsity: ZA-LMS can outperform LMS for highly sparse systems, whereas LMS performs better for dispersive systems.The sparsity degree may be unknown beforehand or vary over time.Source: §X · p. 23
  • Scheme A combines ZA-NLMS and standard NLMS; Scheme B weights nonoverlapping coefficient-block outputs with shrinkage factors that adapt to minimize error power.In Scheme B, factors tending toward zero shrink blocks whose mean-square deviation is high relative to the unknown-system energy, and act as support estimators.Source: §X · pp. 23–24, Figure 12 · p. 24
  • At Q = 256, Scheme B costs less than Scheme A and achieves lower MSD for highly and medium sparse systems.Scheme A remains robust across sparsity levels, performing as well as its best component filter; Scheme B improves as block length decreases.Source: §X · p. 24

XI. ACOUSTIC ECHO CANCELLATION

Volterra filters can model nonlinearities with memory, but their kernel count and settings introduce tradeoffs, especially when echo-path distortion is uncertain or changes. The described combinations select linear and nonlinear components and report strong performance across changing distortion conditions.

XI. ACOUSTIC ECHO CANCELLATION

  • Volterra filters model a large class of nonlinearities with memory, but their performance depends on kernel count, step size, and kernel memory.When only linear distortion is present, adapting nonlinear kernels can make a Volterra filter perform worse than a simple linear adaptive filter.Source: §XI · pp. 25–26
  • Combination of Volterra filters mixes outputs from different filters, while combination of kernels replaces each kernel with a combination of kernels using complementary settings.Both approaches address Volterra-filter compromises; the kernel-combination approach has similar performance but lower computational cost.Source: §XI · p. 26
  • The echo canceller combines two linear kernels with different step sizes for fast reconvergence and low steady-state error, and a quadratic kernel with a zero-output virtual kernel to handle nonlinear distortion.The mixing parameters weight the linear-kernel estimates and the nonlinear echo estimate.Source: §XI · p. 26, Figure 15 · p. 27
  • The proposed nonlinear echo canceller achieves the best reported performance across the tested linear-to-nonlinear conditions and reconverges after an abrupt room-response change.It favors linear filters for t ≤ 30 s, incorporates the quadratic kernel for t > 90 s, and slightly outperforms linear and nonlinear filters during 30 s < t ≤ 90 s.Source: Figure 16 · p. 27, §XI · p. 28

XII. CONCLUSIONS AND OPEN PROBLEMS

The review finds that combinations perform competitively across the studied application scenarios, with power-normalized algorithms simplifying parameter selection and improving robustness when scenario statistics are partly unknown. It identifies open problems in component-filter design, computational cost, and extension to other domains.

XII. CONCLUSIONS AND OPEN PROBLEMS

  • Practical combination algorithms need methods to adjust the combination layer for potentially time-varying filtering conditions.The review covers popular schemes, their theoretical properties, and their limits.Source: §XII · p. 28
  • Across studied scenarios, combinations offer competitive performance against each application's state-of-the-art methods; power-normalized algorithms simplify parameter selection and improve robustness when scenario statistics are partly unknown.The article illustrates the approach with examples across a variety of applications.Source: §XII · p. 28
  • An open problem is selecting and designing component filters with reduced cross-EMSE to minimize overall EMSE.Source: §XII · p. 28
  • Another open problem is reducing combined-scheme complexity toward that of an individual filter.Source: §XII · p. 28
  • The review also calls for extending combinations to application domains with different compromises and performance tradeoffs.Source: §XII · p. 28
Loading 2112.12245v1…