PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian Robust Tensor Factorization for Incomplete Multiway Data

Qibin Zhao, Guoxu Zhou, Liqing Zhang, Andrzej Cichocki, Shun-ichi Amari

arXiv:1410.2386v2cs.CVcs.LG

TL;DR

Incomplete tensor data with outliers requires separating global low-rank structure from sparse local corruption without relying on difficult rank or tuning-parameter choices. The paper develops a fully Bayesian robust CP factorization with hierarchical priors and variational inference, reporting stronger predictive performance and robustness than competing methods across synthetic and real-world data.

  • Problem

    Existing tensor methods struggle with missing data, outliers, and unknown CP rank, while robust completion approaches are sensitive to tuning parameters.

  • Method

    A fully Bayesian generative model separates a low-CP-rank tensor from an elementwise sparse outlier tensor and learns posteriors using efficient variational inference.

  • Results

    The method outperforms state-of-the-art methods in predictive performance and robustness to outliers on synthetic and real-world datasets, even when competitors use groundtruth-tuned parameters.

  • Takeaways & Limitations

    Automatic rank determination, sparsity adaptation, and evidence-based low-rank–sparse balancing provide tuning-parameter-free model selection.

Abstract

from arXiv · show

We propose a generative model for robust tensor factorization in the presence of both missing data and outliers. The objective is to explicitly infer the underlying low-CP-rank tensor capturing the global information and a sparse tensor capturing the local information (also considered as outliers), thus providing the robust predictive distribution over missing entries. The low-CP-rank tensor is modeled by multilinear interactions between multiple latent factors on which the column sparsity is enforced by a hierarchical prior, while the sparse tensor is modeled by a hierarchical view of Student-$t$ distribution that associates an individual hyperparameter with each element independently. For model learning, we develop an efficient closed-form variational inference under a fully Bayesian treatment, which can effectively prevent the overfitting problem and scales linearly with data size. In contrast to existing related works, our method can perform model selection automatically and implicitly without need of tuning parameters. More specifically, it can discover the groundtruth of CP rank and automatically adapt the sparsity inducing priors to various types of outliers. In addition, the tradeoff between the low-rank approximation and the sparse representation can be optimized in the sense of maximum model evidence. The extensive experiments and comparisons with many state-of-the-art algorithms on both synthetic and real-world datasets demonstrate the superiorities of our method from several perspectives.

I. INTRODUCTION

Tensor factorization represents multidimensional data through latent interactions, but incomplete observations, unknown CP rank, and outliers make robust modeling and parameter selection difficult. The paper proposes a fully Bayesian framework that jointly recovers low-rank structure and sparse outliers while automatically learning rank and sparsity.

  • Tensor factorization captures interactions among latent factors in multidimensional data and supports applications including face recognition, social network analysis, completion, and brain signal processing.
  • Existing incomplete-tensor methods often require manually specified rank, which can underfit or overfit observations and degrade predictive performance.CP-rank determination is difficult and has been shown to be NP-hard.
  • Outliers and non-Gaussian noise occur frequently in image and video data, while robust tensor methods remain sensitive to tuning-parameter selection.
  • The proposed probabilistic model separates partially observed data into a low-CP-rank tensor for global information and a sparse tensor for outliers.It applies shared hierarchical sparsity priors to latent factors and elementwise hierarchical Student-t modeling to the sparse term.

IV. BAYESIAN ROBUST CP FACTORIZATION

The model treats incomplete tensor observations as noisy sums of a low-CP-rank component and a sparse outlier component. Hierarchical priors induce shared column sparsity in latent factors and elementwise sparsity in outliers, while variational Bayesian inference estimates posteriors and missing-entry predictions.

  • Model specification: The observation model represents Y as a low-CP-rank tensor X plus sparse outliers S and isotropic Gaussian noise, with only observed entries available for inference.
  • Model specification: CP factorization expresses the latent tensor as a sum of R rank-one tensors formed from factor matrices across the N modes.
  • Model specification: A shared precision matrix across factor matrices uses hierarchical priors to induce column-wise sparsity and infer the effective CP rank from an initially large latent dimension.
  • Model specification: Each sparse-tensor entry receives an independent precision hyperparameter, and increasing that precision drives the corresponding outlier element to zero.
  • Model specification: The Gaussian–Gamma hierarchy provides a sparsity-inducing marginal model related to a Student-t distribution while retaining conjugate inference structure.
  • Inference: Fully Bayesian posterior inference over model parameters yields a predictive distribution for unobserved tensor entries rather than only point estimates.

B. Model Learning via Bayesian Inference

The model uses variational Bayesian inference because exact inference is analytically intractable, obtaining efficient closed-form posterior approximations through mean-field factorization and conjugacy.

  • Inference framework: Variational Bayesian inference approximates the intractable exact posterior with efficient closed-form posterior distributions.The approximation minimizes KL divergence between q(Θ) and p(Θ|YΩ).
  • Inference framework: Mean-field approximation factorizes the posterior into separate distributions over model variables.The individual factors have explicitly derivable functional forms under the model’s conjugate exponential-family structure.
  • Inference framework: Conjugate exponential-family structure enables explicit derivation of the individual variational factors under the mean-field assumption.

1) Posterior distribution of factor matrices:

The factor-matrix posteriors are updated from observed-data likelihood messages and hierarchical-prior messages, with covariance terms computed from the other mode factors. Model fitness controls the balance between prior information and current-data information.

  • Posterior factor updates: Each mode-n factor posterior combines messages from the observed-data likelihood with messages from its hierarchical prior.The resulting row-wise posterior distributions are Gaussian and factorized across rows.
  • Posterior factor updates: The covariance update uses Khatri-Rao products of all mode factors except mode n, summed over observed-entry indices.This structure ensures that only factors interacting with the updated factor contribute.
  • Posterior factor updates: Factor covariance balances the expected prior precision with covariance information from the other factor matrices.The balance is controlled by expected noise precision, which is related to model fitness.
  • Posterior factor updates: Observed values serve as combination coefficients when updating each factor mean from the other mode factors.Larger observations imply greater similarity among the corresponding latent factors in this interpretation.
  • Hyperparameter update: The variational posterior of each rank hyperparameter λr is an independent Gamma distribution.

2) Posterior distribution of hyperparameters

The hyperparameter updates adaptively prune unnecessary factor components and independently regulate sparse residual entries, while the sparse tensor captures information not explained by the low-rank approximation.

  • Factor hyperparameters: Small factor components induce large posterior precision and are progressively shrunk to zero, while larger components are enhanced.This mechanism effectively prunes components and supports automatic CP-rank determination.
  • Sparse-tensor hyperparameters: The sparse-tensor posterior is derived from Gaussian entry priors with individual precision hyperparameters and updated variational parameters.
  • Sparse-tensor hyperparameters: The sparse tensor captures residual information not explained by the low-rank CP approximation, including individual non-Gaussian noise patterns.Its magnitude depends on the sparse prior hyperparameter and Gaussian-noise precision.
  • Sparse-tensor hyperparameters: Smaller expected sparse-entry magnitudes produce larger γ values, forcing those entries toward zero more strongly.Larger-magnitude entries receive the opposite treatment and are further enhanced.
  • Noise hyperparameter: The variational posterior of the noise precision τ is a Gamma distribution updated from observed-data messages and its hyperprior.

5) Posterior distribution of hyperparameter

The noise-precision posterior is updated using the expected squared Frobenius norm of model residuals, with explicit evaluation enabled by an independence lemma for factor rows.

  • Residual expectation: Exact evaluation of the residual expectation requires an auxiliary independence result for rows of independently distributed factor matrices.
  • Residual expectation: The auxiliary lemma makes the posterior expectation of model residuals explicitly computable within the variational updates.
  • Noise-precision update: The expected noise precision τ is updated from residual information measured by the squared Frobenius norm.

6) Lower bound of model evidence:

The variational lower bound provides an explicit model-evidence objective and convergence monitor. Its hyperparameters can be adapted to outlier sparsity by maximizing this bound.

  • Lower bound formulation: The variational lower bound equals the posterior expectation of the joint log density plus the entropy of q.With the derived parametric q distributions, the bound has an explicit evaluable form.
  • Convergence monitoring: The lower bound should not decrease during iterative re-estimation and can therefore be monitored for convergence.The bound is also interpreted as a lower bound on the log marginal likelihood.
  • Prior settings: Small fixed top-level hyperparameters produce noninformative or Jeffreys priors.The listed hyperparameters are usually set to very small values or zero.
  • Hyperparameter adaptation: The sparsity-related hyperparameters aγ0 and bγ0 are automatically adapted to different outlier types by maximizing the lower bound.This optimization is performed with respect to the sparsity hyperparameters.

7) Initialization of model inference:

Inference is initialized with hyperparameters, factor-related quantities, sparse-tensor variables, and an initial rank. The algorithm then updates posteriors, evaluates the lower bound, prunes components, and iterates to convergence.

  • Initialization rationale: Variational Bayesian inference is guaranteed only to converge to a local minimum, making initialization important for avoiding poor local solutions.The paper specifically emphasizes choosing an initialization point for the top-level hyperparameters.
  • Hyperparameter initialization: The top-level hyperparameters aτ0, bτ0, aγ0, and bγ0 are set to 10^-6, yielding a noninformative prior.The resulting initial expectations are E[Λ] = I, E[τ] = 1, and E[γ_i1...iN] = 1.
  • Factor initialization: Factor matrices can be initialized either by random draws from N(0, I) or using SVD-based quantities.The SVD scheme uses left singular vectors and diagonal singular values from mode-n matricizations.
  • Rank initialization: The rank R is usually initialized at its maximum possible value, although it can also be set manually.The maximum is bounded by R ≤ min_n P_n, with P_n defined from the other mode dimensions.
  • Inference procedure: Algorithm 1 initializes model variables, updates posterior factors and hyperparameters, evaluates the lower bound, reduces the model, and repeats until convergence.Posterior updates proceed from bottom to top, indicating message passing begins with observed data.

C. Predictive Distribution

The method approximates analytically intractable predictions for missing entries with a Student-t distribution and provides predictive uncertainty. Its inference cost is linear in observations, with additional rank-dependent terms and complete-data simplifications.

  • Predictive distribution: Missing-entry predictions are approximated from variational posteriors over all parameters, yielding a Student-t predictive distribution.The predictive distribution is conditioned on observed entries.
  • Predictive uncertainty: Predictive uncertainty is available analytically through the variance of the predictive distribution.The variance is expressed using the Student-t degrees of freedom and scale-related quantity Ψ.
  • Computational complexity: The overall incomplete-data algorithm has complexity O((R^2M + R^3) Σ_n I_n), scaling linearly with data size and polynomially with tensor rank.Here M is the number of observations, R is model complexity, and R ≪ M generally.
  • Large-scale extension: For extremely sparse or large-scale data, the paper indicates that approximate inference can instead be updated per observed entry.This is presented as an alternative strategy to the main inference procedure.

V. ADVANTAGES

The fully Bayesian framework learns model parameters and the low-rank–sparse tradeoff automatically, while providing posterior uncertainty and an efficient deterministic inference algorithm.

  • Automatic learning: The method is tuning-parameter free, learning all model parameters automatically from observed data.This contrasts with methods requiring predefined rank or penalty parameters and nuclear-norm regularization tuning.
  • Model selection: Automatic rank determination can discover the ground-truth CP rank, while automatic sparsity modeling adapts to various outliers or non-Gaussian noises.The low-rank–sparse tradeoff is learned by maximizing model evidence.
  • Uncertainty quantification: Posterior uncertainty over model parameters and missing-entry predictions can help prevent overfitting and provide confidence information.The method uses full posteriors rather than point estimates of factor matrices and predictions.
  • Inference efficiency: The deterministic Bayesian inference algorithm empirically converges quickly and has computational complexity that scales linearly with data size.The efficiency claim concerns the algorithm's dependence on data size.

VI. EXPERIMENTAL RESULTS

Synthetic and real-world experiments evaluate BRTF for recovery, completion, rank estimation, runtime, and video background modeling. Across these settings, BRTF is reported to be robust to outliers and missing data while automatically estimating rank and maintaining high efficiency.

  • Synthetic validation: BRTF significantly outperforms competing methods under all tested outlier conditions for recovering the low-rank tensor and latent factors.The comparison includes outliers both much larger than, and within the range of, the true signals.
  • Synthetic validation: BRTF performance is unaffected by outlier percentage and magnitude, confirming adaptation to varied outlier types.The experiment considers varying outlier percentages and two outlier magnitudes.
  • Synthetic validation: BRTF correctly estimates CP rank when the true rank exceeds data dimensions, whereas CP-ARD overestimates rank under outlier corruption.With a 30 × 30 × 30 tensor and true rank R = 50, BRTF also significantly outperforms alternatives in RRSE and FME while showing high efficiency.
  • Synthetic validation: BRTF achieves the best completion performance among competing methods and remains stable across varying missing ratios.The results also report accurate ground-truth tensor-rank estimation in all tested missing-data cases.

B. Video Background Modeling

BRTF separates video backgrounds and foregrounds while handling severe missingness and outliers. It remains effective without tuning parameters and also performs robustly on non-Gaussian facial-image noise.

  • Background and foreground separation: BRTF successfully separates background and foreground across the evaluated surveillance sequences.It completely separates a temporarily stationary person in one Hall frame, while other methods capture the person in both components.
  • Robust completion: With 90% of pixels missing, BRTF recovers backgrounds more effectively than RegL1ALM, PRMF, and HORPCA.Other methods exhibit more severe ghost effects, whereas BRTF is reported as unaffected by the missing pixels and outliers.
  • Automatic inference: BRTF works fully automatically, whereas competing methods require time-consuming tuning-parameter selection.This comparison concerns the simultaneous completion and anomaly-detection experiments.
  • Facial-image denoising: BRTF and HORPCA produce satisfactory reconstructions under non-Gaussian facial-image noise, outperforming CP-ALS, HOSVD, and Tucker-ARD.BRTF still outperforms HORPCA in recovery performance even though HORPCA parameters were tuned using groundtruth.
  • Facial-image denoising: BRTF is computationally more efficient than HORPCA in facial-image denoising because HORPCA requires multiple tuning runs.Tucker-ARD provides automatic model selection but is reported to perform poorly on non-Gaussian noise and to have low computational efficiency.
Loading 1410.2386v2…