Source-linked AI summary
Low-rank Matrix Factorization under General Mixture Noise Distributions
Xiangyong Cao, Qian Zhao, Deyu Meng, Yang Chen, Zongben Xu
TL;DR
LRMF commonly uses L1- and L2-based losses for Laplace- and Gaussian-like noise, but real data may contain more complex noise. This paper proposes penalized mixtures of Exponential Power distributions, with a Markov random field extension, and reports stronger performance across varied complex-noise tasks than competing methods.
Problem
L1- and L2-based LRMF losses address Laplace- and Gaussian-like noise, respectively, but complex noise in applications may not fit either assumption well.
Source: §I · p. 1Method
The paper proposes PMoEP, which models LRMF noise with a penalized mixture of Exponential Power distributions, and PMoEP-MRF, which embeds a Markov random field to facilitate local continuity.
Source: §I · p. 2, A. PMoEP-MRF Model · p. 6Results
Across synthetic and real complex-noise scenarios, including face modeling, hyperspectral restoration, and background subtraction, PMoEP performs better than current LRMF methods.
Source: §VI · p. 12Takeaways & Limitations
The proposed methods can automatically learn the number of mixture components from data and are evaluated on synthetic data and three real-data applications.
Source: §I · p. 2, §VI · p. 12
Abstract
from arXiv · showhide
Many computer vision problems can be posed as learning a low-dimensional subspace from high dimensional data. The low rank matrix factorization (LRMF) represents a commonly utilized subspace learning strategy. Most of the current LRMF techniques are constructed on the optimization problems using L1-norm and L2-norm losses, which mainly deal with Laplacian and Gaussian noises, respectively. To make LRMF capable of adapting more complex noise, this paper proposes a new LRMF model by assuming noise as Mixture of Exponential Power (MoEP) distributions and proposes a penalized MoEP (PMoEP) model by combining the penalized likelihood method with MoEP distributions. Such setting facilitates the learned LRMF model capable of automatically fitting the real noise through MoEP distributions. Each component in this mixture is adapted from a series of preliminary super- or sub-Gaussian candidates. Moreover, by facilitating the local continuity of noise components, we embed Markov random field into the PMoEP model and further propose the advanced PMoEP-MRF model. An Expectation Maximization (EM) algorithm and a variational EM (VEM) algorithm are also designed to infer the parameters involved in the proposed PMoEP and the PMoEP-MRF model, respectively. The superseniority of our methods is demonstrated by extensive experiments on synthetic data, face modeling, hyperspectral image restoration and background subtraction.
I. INTRODUCTION
LRMF commonly uses Gaussian- or Laplace-oriented losses, but real data can contain more complex noise that neither fits well. The paper introduces penalized mixtures of Exponential Power distributions for LRMF, with automatic component selection and a Markov random field extension.
I. INTRODUCTION
- Gaussian- and Laplace-oriented LRMF losses are limited for complex noise, including the sparse and Gaussian-like noise described in hyperspectral images.The paper notes that L2-based methods are sensitive to outliers and non-Gaussian noise, while L1 is optimal only for Laplace-like noise.Source: §I · pp. 1–2
A. PMoEP model
- PMoEP models LRMF noise as a mixture of Exponential Power distributions and uses penalized likelihood to select mixture components.The model uses a series of sub- and super-Gaussian candidates; each EP component can represent different distribution shapes, with Laplace and Gaussian as special cases.Source: §I · p. 2, A. PMoEP model · p. 3
I. INTRODUCTION
- The PMoEP-MRF extension embeds a Markov random field to facilitate local continuity of noise components.The paper describes the continuity as applying along space and time in video.Source: §I · p. 2
III. LRMF WITH MOEP NOISE
- The paper presents EM for PMoEP and variational EM for PMoEP-MRF, and reports proving their convergence.It states that experiments evaluate the proposed models against previous methods.Source: §I · p. 2
II. RELATED WORK
- Earlier LRMF approaches based on L2, L1, or MoG noise have limits: they are optimal for particular distributions or remain limited on complex real noise.Finite MoG mixtures also face limitations in approximating distributions such as Laplace and in selecting the number of components.Source: §II · p. 2, §I · p. 2
B. EM algorithm for PMoEP model
The PMoEP EM procedure alternates between estimating mixture assignments and updating model parameters. Its factor updates use an augmented Lagrange multipliers procedure, and components with zero mixing weight are removed.
B. EM algorithm for PMoEP model
- The E-step computes posterior component assignments, and the M-step updates mixture proportions, precision parameters, and low-rank factors.The algorithm summary also removes components whose mixing proportion reaches zero.Source: B. EM algorithm for PMoEP model · pp. 4–5
- The factor-update subproblem is solved using an augmented Lagrange multipliers method with alternating updates.The procedure updates U and V via SVD and updates the auxiliary variable L through scalar subproblems.Source: B. EM algorithm for PMoEP model · p. 4
- The paper summarizes PMoEP estimation as an iterative EM algorithm that returns model parameters, the final component count, and posterior probabilities.Its initialization includes candidate shape parameters and an initial component count.Source: B. EM algorithm for PMoEP model · p. 5
C. Convergence Analysis of EM algorithm
The paper analyzes convergence for the PMoEP EM algorithm using a penalized objective. It states that the proof is provided in Appendix A.
C. Convergence Analysis of EM algorithm
- The section presents a convergence analysis for the proposed EM algorithm for the PMoEP model.The passage states the analysis's scope without specifying its result.Source: C. Convergence Analysis of EM algorithm · p. 5
- The convergence analysis examines the sequence generated by Algorithm 2 using a penalized objective formed from the likelihood and a penalty on mixture proportions.The passage introduces the objective as P(Θ) = l(Θ) − P(π; λ).Source: C. Convergence Analysis of EM algorithm · p. 5
- The paper states that the proof of the convergence result is in Appendix A.The supplied passage identifies the proof location but does not give the theorem's conclusion.Source: C. Convergence Analysis of EM algorithm · p. 5
D. Implementation Issues
PMoEP uses preset mixture settings and selects λ from candidate values with a modified BIC criterion. Experiments use 4–10 initial components and shape parameters spanning 0.1 to 2.
D. Implementation Issues
- Experiments set Kstart to 4–10 based on a coarse empirical estimate of noise complexity and choose shape parameters from 0.1–2.The vector p has length Kstart.Source: D. Implementation Issues · p. 5
- The modified BIC criterion selects λ from a series of candidate values.The passage notes that this criterion has been proven consistent for estimating component number in finite Gaussian mixtures.Source: D. Implementation Issues · p. 5
- The modified BIC criterion uses the number of non-missing elements and estimates of component count, mixture weights, and precision parameters.The component, weight, and precision estimates maximize the PMoEP objective for a given λ.Source: D. Implementation Issues · p. 5
IV. PMOEP WITH MARKOV RANDOM FIELD
The paper introduces PMoEP-MRF as an advanced model, then presents a variational EM algorithm and analyzes its convergence.
IV. PMOEP WITH MARKOV RANDOM FIELD
- The PMoEP-MRF section proposes an advanced model and a corresponding variational EM algorithm.It also provides convergence analysis for the algorithm.Source: §IV · p. 5
- The section's algorithmic treatment includes convergence analysis of the proposed variational EM algorithm.Source: §IV · p. 5
A. PMoEP-MRF Model
PMoEP-MRF incorporates a local smoothness prior over noise-component labels, then uses variational EM because the conditional distribution required by EM is intractable. Its variational updates optimize a lower bound and reuse the PMoEP M-step for model parameters.
A. PMoEP-MRF Model
- The MRF prior encodes spatial and temporal smoothness by favoring similar noise-component labels at neighboring pixel entries.The model specifies a neighborhood N(i,j) and uses τ=10 in experiments; ψ is higher when paired labels match at 0 or 1.Source: A. PMoEP-MRF Model · pp. 5–6
B. Variational EM algorithm for PMoEP-MRF model
- Variational EM is used because the required conditional distribution P(Z|E) is not tractable.The method seeks a variational distribution close to P(Z|E) in KL divergence; its lower bound equals log L(E) when the distributions match.Source: B. Variational EM algorithm for… · p. 6
- The variational optimization alternates between updating the variational distribution and optimizing U, V, π, and η.The variational parameters γ_ij are normalized across mixture components, and their E-step update satisfies a fixed-point relation.Source: B. Variational EM algorithm for… · p. 6
- The VEM M-step updates U, V, π, and η in the same way as the PMoEP EM algorithm.The algorithm also updates γ, removes components according to π, and iterates until convergence.Source: B. Variational EM algorithm for… · p. 6
C. Convergence Analysis of Variational EM algorithm
The convergence analysis guarantees convergence of Algorithm 3 because its iterates are upper bounded and its variational objective increases monotonically.
C. Convergence Analysis of Variational EM algorithm
- Algorithm 3's sequence of variational parameters and model parameters is upper bounded, guaranteeing convergence.Source: C. Convergence Analysis of Variational… · p. 7
- The objective J(RE) increases monotonically at each iteration because the two update propositions each guarantee an increase.Source: C. Convergence Analysis of Variational… · p. 7
V. EXPERIMENTAL RESULTS
Synthetic experiments compare PMoEP and related LRMF methods across several noise settings. PMoEP performs especially well under complex noise, while estimated noise distributions match the ground truth in the illustrated cases.
A. Synthetic simulations
- The experiments average six measures over 30 randomly generated rank-4 matrices with 20% missing entries and several noise settings.The settings include Gaussian, exponential-power, Laplace, sparse, and two mixture-noise cases.Source: A. Synthetic simulations · pp. 7–8
- PMoEP performs best under complex noise, leads under Laplace and exponential-power noise, and shares top performance with other methods under Gaussian or sparse noise.The authors attribute its complex-noise performance to flexibility in modeling unknown noise.Source: Table II · p. 7, A. Synthetic simulations · p. 8
- In the illustrated synthetic cases, PMoEP-estimated noise probability densities match the ground-truth densities.The figure includes zoomed views of indicated portions.Source: A. Synthetic simulations · p. 8, Figure 3 · p. 8
B. Face modeling
In face modeling, PMoEP and PMoG outperform competing methods on faces with large dark regions. In hyperspectral restoration, PMoEP removes complex noise, including stripes and severe corruption, across the tested images.
B. Face modeling
- PMoEP and PMoG outperform other methods on faces containing large dark regions with significant shadow, saturation, and amplified camera noise.All compared methods can remove cast shadows and saturation in the face images.Source: B. Face modeling · p. 8
C. Hyperspectral Image Restoration
- The hyperspectral tests use the Urban and Terrain datasets, reshaping and stacking image bands into data matrices for evaluation.The reported matrices are 94249 × 210 for Urban and 153500 × 210 for Terrain.Source: C. Hyperspectral Image Restoration · pp. 8–9
- PMoEP produces clean, smooth reconstructions for some striped or deadline-affected bands and recovers severely polluted bands where competing methods fail.The comparisons cover bands from both the Urban and Terrain datasets.Source: C. Hyperspectral Image Restoration · p. 9, Figure 8 · p. 9
- PMoEP yields smoother horizontal mean profiles, indicating more effective horizontal-stripe removal than the other methods.The profiles compare band 207 in Urban and band 152 in Terrain.Source: C. Hyperspectral Image Restoration · p. 10, Figure 12 · p. 10, Figure 13 · p. 10
- PMoEP more properly extracts sparse strips, sparse deadlines, and dense Gaussian noise from corrupted hyperspectral images than the competing methods.The figure displays original bands and extracted noise maps, with positive and negative values shown in yellow and blue.Source: C. Hyperspectral Image Restoration · p. 10, Figure 14 · p. 10
D. Background Subtraction
On video background subtraction, PMoEP and PMoEP-MRF outperform other methods on complex-background sequences, while performances are nearly alike on simple backgrounds. The conclusion places these results within a broader claim of improved performance under complex noise.
D. Background Subtraction
- Evaluation measures foreground-support recovery with precision and recall, combines them using F-measure, and thresholds the recovered foreground at the value maximizing F-measure.Source: D. Background Subtraction · p. 11
- PMoEP and PMoEP-MRF outperform other methods on Hall, Curtain, Fountain, WaterSurface, and Escalator, while all methods perform almost the same on four simple-background sequences.PMoEP is second best on average; PMoEP-MRF slightly improves average performance over PMoEP.Source: D. Background Subtraction · p. 11, Table III · p. 11
- PMoEP-MRF's slight average improvement over PMoEP is attributed to modeling spatial and temporal smoothness under foreground with a Markov random field.Source: D. Background Subtraction · p. 11
VI. CONCLUSIONS
- The conclusion reports better PMoEP performance across varied synthetic and real complex-noise scenarios, including face modeling, hyperspectral restoration, and background subtraction.It also states that the methods can automatically learn the number of mixture components from data.Source: §VI · p. 12
APPENDIX A PROOF OF THEOREM 1
The appendix proof states that an iteration sequence is nondecreasing and bounded above, while the accompanying distribution subsection notes equivalent forms of the exponential power distribution.
APPENDIX A PROOF OF THEOREM 1
- The proof's displayed steps invoke an expectation inequality and refer to the E and M steps of Algorithm 1.Source: Appendix A · p. 12
- The proof states that the sequence of likelihood values is nondecreasing and bounded above, and therefore converges to a constant.Source: Appendix A · p. 12
A. Three different forms of Exponential Power Distribution
- The exponential power distribution is presented in three equivalent forms.Source: A. Three different forms of… · p. 12
B. Draw Samples from Exponential Power Distribution
Sampling from the exponential power distribution is divided by whether its shape parameter p is at least one or below one, with different procedures described for the two cases.
B. Draw Samples from Exponential Power Distribution
- Sampling from the exponential power distribution is divided into cases p ≥ 1 and 0 < p < 1.Source: B. Draw Samples from Exponential Power… · p. 12
1) case 1:
- For p ≥ 1, the paper adopts a sampling method from,,.Source: 1) case 1 · p. 12
2) case 2:
- For 0 < p < 1, the paper uses the method in and applies slice sampling from because one of the two sampling steps is difficult to sample directly.Source: 2) case 2 · p. 12