PaperlayerView exact revision on arXiv

Source-linked AI summary

Singularities of Non-negative Matrix Factorization and their application to Bayesian inference

Naoki Hayashi, Yota Maeda, Yasushi Esaki

arXiv:2609.34043v1stat.MLcs.LGmath.STstat.TH

TL;DR

NMF is a singular statistical model, and its RLCT governs leading terms in Bayesian generalization error and free energy. The paper analyzes local factorization geometry to derive an RLCT upper bound that is tighter than prior bounds, with an exact value in a stated special case.

  • Problem

    The RLCT governs leading terms in Bayesian generalization error and free energy, while previous NMF bounds leave room for tighter evaluation.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    The paper analyzes local RLCTs using analytic coordinates around zeros of the NMF loss and establishes a bound under smooth, positive, bounded priors.

    Source: §3 · p. 4, §4 · p. 12

  • Results

    The paper establishes a tighter RLCT upper bound than previous work and obtains an exact RLCT value when the model and true distribution are equal.

    Source: §3 · p. 4, §6 · p. 21

  • Takeaways & Limitations

    The bound can improve RLCT-based model-selection and posterior-computation methods and sharpen asymptotic evaluations of Bayesian generalization error and free energy.

    Source: §1 · p. 2, §3 · p. 4

    The theorem requires a smooth, positive, bounded prior on the zero set; commonly used Poisson–gamma priors may have zeros and fail this assumption.

    Source: §3 · pp. 4–5

Abstract

from arXiv · show

Non-negative matrix factorization (NMF) is a singular statistical model whose Bayesian asymptotics are governed by the real log canonical threshold (RLCT). We study the local geometry of the factorization map and derive an upper bound for the RLCT of NMF. Let $H$ be the model inner dimension and $H_0$ the non-negative rank of the true $M\times N$ matrix. Assuming that the true matrix admits a strictly positive factorization of inner dimension $H_0$ in the interior of the parameter domain, we prove, for smooth positive priors, that $\lambda\leq \{(H-H_0)\min(M,N)+H_0(M+N-H_0)\}/2$. This bound strictly improves the previous bound when $H_0\geq3$. The proof uses a local analytic normal form that separates independent linear coordinates from a residual matrix product. When $H=H_0$ also equals the ordinary rank of the true matrix, we obtain the exact value $\lambda=H_0(M+N-H_0)/2$. Under the standard assumptions of singular learning theory, these results bound the leading coefficients of the expected Bayesian generalization error and the Bayesian free energy.

1 Introduction

NMF represents non-negative data matrices, but its non-unique factorizations make optimization sensitive to initialization and prone to local solutions. The paper studies the RLCT to tighten existing bounds relevant to Bayesian NMF.

1 Introduction

  • NMF factors a non-negative matrix into two non-negative matrices, supporting applications across several data-analysis fields.The cited examples include image recognition, text mining, signal processing, bioinformatics, consumer analysis, and recommender systems.Source: §1 · p. 1
  • Because infinitely many factor pairs can represent the same matrix, objective-function optimization depends strongly on initialization and often reaches a local solution.Bayesian NMF uses Gibbs sampling to compute parameter posteriors and can assess fluctuations in estimates; mean-field variational inference can also be formulated.Source: §1 · p. 1
  • NMF is a singular model, whose likelihoods and posteriors cannot be approximated by normal distributions; Bayesian inference can yield smaller generalization loss than maximum-likelihood or maximum-a-posteriori estimation.The passage notes that most machine-learning statistical models are singular.Source: §1 · p. 2
  • The paper derives a tighter RLCT upper bound for NMF than previous bounds, addressing a conjectural evaluation involving the non-negative rank.The authors say the bound gives a more accurate asymptotic characterization of Bayesian generalization error and improves RLCT-based model-selection and posterior-computation methods.Source: §1 · p. 2

2 Bayesian inference framework

The framework evaluates Bayesian inference through free energy and generalization error, whose asymptotic behavior is governed by the RLCT.

2 Bayesian inference framework

  • Bayesian inference uses a dataset of independent variables drawn from the true distribution and a model with a prior over its parameters.The framework denotes the marginal likelihood and posterior by Z_n and φ*(w|D_n), respectively.Source: §2 · p. 2
  • The posterior expectation of the statistical model is called the Bayesian predictive distribution.It is denoted p*(x|D_n).Source: §2 · p. 2
  • Free energy evaluates inference accuracy through the marginal likelihood, while Bayesian generalization error evaluates prediction accuracy on unseen data through the Bayesian predictive distribution.The free energy is defined using the negative log marginal likelihood.Source: §2 · p. 3
  • The RLCT λ determines the leading asymptotic terms of free energy and generalization error, linking it to hyperparameter tuning and sufficient-sample-size estimates.The framework gives F_n = nS_n + λ log n + O_p(log log n).Source: §2 · p. 3

3 Main Theorem

The theorem bounds NMF’s RLCT under stated prior and model assumptions, then uses that bound to constrain Bayesian generalization error and free energy when KL divergence is equivalent to the factorization loss.

Assumptions

  • The setup requires a compact semianalytic parameter set with nonempty interior and a prior that is locally integrable and simultaneously resolvable with KL.The true factors have positive entries, and their product has non-negative rank H0.Source: §3 · pp. 3–4

RLCT definition

  • The RLCT is defined as λ when the largest pole of the specified meromorphic continuation is −λ.The continuation has negative rational poles under the stated assumptions.Source: §3 · p. 4

Main Theorem

  • For smooth, positive priors bounded on the zero-loss set, Theorem 3.1 bounds λ by ((H−H0)min(M,N)+H0(M+N−H0))/2.The setup assumes H≥H0 and a nonempty zero-loss set.Source: §3 · p. 4

Corollary 3.1

  • When KL(q | p) is equivalent to Φ(U,V), the RLCT result yields inequalities for Bayesian generalization error and free energy.The paper relates these quantities to prediction on unseen data and inference accuracy through the marginal likelihood.Source: §3 · pp. 4–5

Corollary 3.2

  • For gamma priors, Corollary 3.2 gives an RLCT upper bound with min(MϕU,NϕV) in its dimension term, although such priors may have zeros.Zeros may violate Theorem 3.1’s prior condition.Source: §3 · p. 5

Relation to prior work

  • The bound improves previous NMF upper bounds, partially addresses an exact-RLCT conjecture, and supports a tighter lower bound for variational approximation error.The latter follows by applying Corollary 3.2 with Kohjima’s result to a previous proof.Source: §3 · p. 6

4 Proof of the Main Theorem

The proof bounds the NMF RLCT by analyzing the true factorization and extra components, using a local analytic normal form in the H = H0 case. When H = H0 also equals the true matrix rank, it derives an exact RLCT.

Proof setup

  • The proof chooses a zero attaining the RLCT minimum for H = H0 and defines a = rank(U0), b = rank(V0), r = rank(W0), and c = H0 - a - b + r.These ranks and c parameterize the local normal form.Source: §4 · p. 7

Lemma 4.2

  • The derivative of the factorization map has rank MN - (M - a)(N - b), since its image comprises matrices with a zero lower-right block.This identifies the independent linear directions used in the local-coordinate construction.Source: §4 · pp. 7, 9

Lemma 4.3

  • Lemma 4.3 gives analytic local coordinates in which the factorization error is generated by independent linear coordinates and a residual product XY.When M = a, N = b, or c = 0, the residual product is zero.Source: §4 · pp. 7, 12

Lemma 4.4

  • For H = H0, Lemma 4.4 bounds the RLCT obtained by restriction to a sufficiently small neighborhood of (U0, V0) by H0(M + N − H0)/2, assuming the prior is positive and bounded there.Source: §4 · p. 7

4 Proof of the Main Theorem

  • λ = H0(M + N − H0)/2 when H = H0 equals the ordinary rank of the true matrix.In this case, the proof sets a = b = r = H0 and c = 0, then applies the minimality of the RLCT at (U0, V0).Source: §4 · p. 15
  • The upper-bound strategy follows earlier studies, using the triangle inequality and RLCT properties to bound the original loss.The proof notes that multiplying a function by a positive constant leaves its RLCT unchanged, and that an upper bound on a function gives an upper bound on its RLCT.Source: §4 · p. 15
  • λ ≤ ((H − H0)min(M,N) + H0(M + N − H0))/2 for the NMF model.The proof bounds the original loss using a sum of the true-factorization and extra-component losses, then evaluates each contribution using preceding lemmas.Source: §4 · pp. 15–16

5 Discussion

The discussion compares the derived RLCT bound with prior bounds, considers how knowledge of the true factor ranks can tighten it, and relates the results to an earlier conjecture.

5.1 Tightness of the derived bound

  • The derived RLCT upper bound is tighter than the previous study’s bound across the stated odd- and even-H0 cases.For even H0, the passage states the comparison for H0 ≥ 2 or H0 = 0.Source: §5.1 · pp. 16–17

5.2 Rank assumptions on the true parameter matrices

  • Knowing rank(U0), rank(V0), and rank(U0V0) yields an even tighter bound, with equality when H = H0.The discussion notes that model selection often uses H as a control variable, making H = H0 a natural setting for the main theorem.Source: §5.2 · p. 17

5.3 Relation to the Conjecture on the RLCT of NMF in Previous Work

  • The discussion partially resolves the earlier RLCT conjecture [9]: equality with reduced-rank regression holds when H = H0 = r and can also hold for some H = H0 > r.For H = H0 > r, the stated equality conditions include b = r when M − N > H − r, and a = r when N − M > H − r.Source: §5.3 · p. 18

6 Conclusion

The paper establishes a tighter upper bound on the RLCT of NMF and obtains its exact value over a broader range than previously known when the model and true distribution are equal.

6 Conclusion

  • The study establishes a tighter upper bound on the RLCT of NMF than previous work.This is the paper’s stated principal contribution.Source: §6 · p. 21
  • When the model and true distribution are equal, the study obtains the exact RLCT over a broader range than previously known.Source: §6 · p. 21
  • Future work includes determining exact RLCT values when the model is redundant and conducting large-scale numerical experiments.Source: §6 · p. 21
Loading 2609.34043v1…