PaperlayerView exact revision on arXiv

Source-linked AI summary

Dimensionality Reduction for Binary Data through the Projection of Natural Parameters

Andrew J. Landgraf, Yoonkyung Lee

arXiv:1510.06112v1stat.MLstat.ME

TL;DR

Squared-error PCA is conceptually ill-suited to binary responses, while prior logistic PCA factorization complicates scoring new cases. This paper projects saturated-model natural parameters instead; simulations show LPCA can match or outperform LSVD in estimating true probabilities when its settings are chosen properly.

  • Problem

    Squared-error PCA is conceptually ill-suited to binary responses, and factorization-based logistic PCA requires refitting to score new cases.

    Source: §1 · p. 2

  • Method

    The method projects saturated-model natural parameters to form logistic PCA loadings, without factorizing the natural-parameter matrix.

    Source: §7 · p. 28

  • Results

    For every simulated dataset, some m and estimated rank combination gave LPCA mean squared error as small as or smaller than any LSVD estimate.

    Source: §6.1.3 · p. 23

  • Takeaways & Limitations

    The resulting principal component scores are linear combinations of the data and can be quickly calculated on new observations.

    Source: §7 · p. 28

    The MM algorithm can reach a local minimum but does not generally guarantee a global minimum.

    Source: §5.1 · p. 15

Abstract

from arXiv · show

Principal component analysis (PCA) for binary data, known as logistic PCA, has become a popular alternative to dimensionality reduction of binary data. It is motivated as an extension of ordinary PCA by means of a matrix factorization, akin to the singular value decomposition, that maximizes the Bernoulli log-likelihood. We propose a new formulation of logistic PCA which extends Pearson's formulation of a low dimensional data representation with minimum error to binary data. Our formulation does not require a matrix factorization, as previous methods do, but instead looks for projections of the natural parameters from the saturated model. Due to this difference, the number of parameters does not grow with the number of observations and the principal component scores on new data can be computed with simple matrix multiplication. We derive explicit solutions for data matrices of special structure and provide computationally efficient algorithms for solving for the principal component loadings. Through simulation experiments and an analysis of medical diagnoses data, we compare our formulation of logistic PCA to the previous formulation as well as ordinary PCA to demonstrate its benefits.

1 Introduction

The section motivates logistic PCA as a likelihood-based alternative to applying squared-error PCA to binary data, then introduces a formulation that projects saturated-model natural parameters. Unlike the previous factorization-based approach, it avoids case-specific parameters and makes new-data scores simple to compute.

1 Introduction

  • Ordinary PCA’s squared-error reconstruction is conceptually ill-suited to binary responses, motivating likelihood-based dimensionality reduction.Generalized PCA uses exponential-family likelihoods, and logistic PCA applies the Bernoulli likelihood to binary data.Source: §1 · pp. 1–2
  • The earlier logistic PCA factorizes natural parameters and assigns each case a latent factor, so its parameter count grows with the number of cases.Scoring new data requires another factorization, which is prone to overfitting.Source: §1 · p. 2
  • The proposed formulation projects saturated-model natural parameters, extending Pearson’s low-dimensional, minimum-error view of PCA to binary data.It retains scores that are linear functions of the data and reduces to standard PCA for Gaussian data.Source: §1 · p. 2
  • The new formulation keeps parameter count independent of the number of observations and computes new-data scores by matrix multiplication.It also gives explicit solutions for some specially structured data matrices.Source: §1 · p. 2
  • The paper derives two practical algorithms for the non-convex optimization: iterative spectral decompositions using MM and a convex relaxation.The methods are applied to simulated and real datasets to examine advantages and trade-offs.Source: §1 · p. 2

2 Background

The background presents PCA as both a minimum-reconstruction-error representation and a Gaussian factor-model solution. It then describes generalized PCA and earlier binary-data extensions based on factorization of natural parameters.

2 Background

  • Pearson’s PCA represents data in a lower-dimensional space with minimum mean squared error; its loadings are the leading covariance eigenvectors.Under a Gaussian model, the corresponding low-rank factorization is given by truncated SVD.Source: §2 · p. 3
  • Generalized PCA extends this likelihood-based subspace approach to exponential-family data by factorizing the natural-parameter matrix as Θ = ABT.Its objective is to minimize negative log-likelihood subject to a rank constraint.Source: §2 · pp. 3–4
  • Earlier binary-data methods include logistic PCA, a binary factor model, and sparse logistic PCA; estimating scores for new cases requires additional computation in the described methods.The section also notes an application of logistic PCA to recommendation systems.Source: §2 · p. 4

3 New Formulation of Generalized PCA

The section reformulates generalized PCA as projection of saturated-model natural parameters, specializing the approach to Bernoulli data and extending it to exponential-family settings. It contrasts the resulting logistic PCA with factorization-based methods and describes dimension selection and projection geometry.

3 New Formulation of Generalized PCA

  • The formulation views standard PCA as projecting saturated-model natural parameters, which equal the data under a Gaussian likelihood.For Gaussian data, minimizing squared error is equivalent to minimizing deviance.Source: §3 · pp. 4–5

3.1 Alternative formulation to logistic PCA

  • For binary data, logistic PCA approximates saturated-model Bernoulli natural parameters by mQ and projects them to minimize Bernoulli deviance.Here qij = 2xij − 1 and the projection uses orthonormal loadings.Source: §3.1 · p. 5

3.2 Generalized PCA formulation

  • The same projection framework extends to one-parameter exponential-family distributions using the canonical link and can accommodate columns from different families.Examples include Bernoulli, Poisson, and Gaussian data.Source: §3.2 · pp. 5–6

3.3 Comparison to previous techniques

  • Unlike logistic SVD’s low-rank factorization, the proposed method solves for loadings alone, preserving distinct roles for cases and variables.The formulation’s bounded scores can act as an implicit regularizer, whereas logistic SVD’s case-specific factors can overfit.Source: §3.3 · pp. 6–7
  • For new observations, logistic PCA computes scores with matrix multiplication, whereas logistic SVD requires fitting a logistic regression for each case.Fast scoring can matter for repeated cross-validation, large batches, or real-time prediction.Source: §3.3 · pp. 6–7
  • The tuning parameter m controls the extremity of fitted probabilities: larger values move them toward 0 or 1, while smaller values move them toward 0.5.Cross-validation is reported as an effective way to choose m, and simulations show LPCA can match or outperform LSVD in estimating true probabilities when m and k are chosen properly.Source: §3.3 · pp. 7–8

3.4 Number of Principal Components

  • Logistic PCA adapts variance-explained and scree-plot approaches to deviance, but its loading matrices need not be nested across ranks.For large d, non-sequential selection procedures may take a long time, so the authors recommend stopping once cumulative deviance explained is high.Source: §3.4 · p. 8

3.5 Geometry of the Projection

  • In the two-variable illustration, the linear projection occurs in natural-parameter space, then inverse-logit transformation yields a nonlinear probability-space curve.The fitted probabilities appear closer to the original values than standard PCA except for (0, 0), and standard PCA projections can fall outside [0, 1].Source: §3.5 · pp. 9–10, Figure 1 · p. 9

4 Logistic PCA for Patterned Data

The section derives first-order conditions for logistic PCA and uses them to characterize solutions for independence and compound-symmetric data structures. These cases provide analogues to familiar algebraic results for standard PCA under stated conditions.

4 Logistic PCA for Patterned Data

  • Because logistic PCA solutions are less algebraically understood than standard PCA solutions, the section derives necessary conditions and identifies solutions under assumptions on data structure.These results aim to clarify logistic PCA's behavior on patterned data.Source: §4 · p. 10

4.1 First-Order Optimality Conditions

  • The logistic PCA first-order conditions are nonlinear in U, and their U-dependent matrix prevents a closed-form eigen-decomposition solution.The conditions nevertheless allow explicit results for special cases.Source: §4.1 · p. 11

4.2 Independence

  • For a column with mean 1/2, the lth standard basis vector el satisfies logistic PCA's first-order optimality conditions regardless of m.When the column mean differs from 1/2, el can nearly satisfy the optimality conditions for sufficiently large m.Source: §4.2 · p. 11
  • Among standard-basis loadings, the column with mean closest to 1/2 yields the greatest deviance decrease for k = 1.For larger k, choosing the k columns closest to 1/2 gives the greatest decrease among loading matrices made from standard basis vectors.Source: §4.2 · p. 12

4.3 Compound symmetry

  • When QTQ is compound symmetric and condition (4.4) holds, the equal-weight loading vector satisfies logistic PCA's first-order conditions.Condition (4.4) requires fitted probabilities to be representable as an affine function of the q_ijs; it is more likely to hold with fewer columns, and is satisfied when d ≤ 4 under compound symmetry.Source: §4.3 · pp. 12–13

5 Computation

The section presents two algorithms for the non-convex loading problem: majorization-minimization (MM) and a convex relaxation over the Fantope. Their guarantees and the Fantope’s fit-versus-interpretability trade-offs shape when each approach is useful.

5 Computation

  • The two algorithms guarantee, respectively, a local solution for the original problem and a global solution for a relaxed version.The original loading objective is non-convex and has an orthonormality constraint.Source: §5 · p. 13

5.1 Majorization-minimization (MM) algorithm

  • MM majorizes the deviance with a quadratic surrogate and updates loadings through eigenvectors, decreasing deviance to a local minimum.The quadratic bound uses the Bernoulli variance upper bound of 1/4.Source: §5.1 · pp. 14–15

5.2 Convex relaxation

  • The alternative optimizes over the convex hull of rank-k projection matrices, the Fantope, using accelerated gradient descent with O(1/t^2) convergence.The Fantope is the set of matrices H satisfying 0 ⪯ H ⪯ I_d and tr(H) = k.Source: §5.2 · pp. 15–16

5.2.1 Discussion of the Convex Formulation

  • Fantope solutions can be full rank even at k = 1, yet are useful for probability estimation; increasing m can approach saturated fit and fit new cases as well as training data.The authors suggest selecting m by masking entries and assessing matrix-completion accuracy as future work.Source: §5.2.1 · pp. 18–19

6 Numerical Examples

The numerical examples compare LPCA with LSVD and PCA in simulations and patient-diagnosis data, examining probability estimation, fit, prediction, algorithm behavior, and loadings.

6.1.2 Fantope versus MM

  • Fantope converged quickly to the same minimum across initializations, but MM beat its projected solution for every initialization and was more sensitive to initialization.The range of MM deviance values was fairly narrow in this example.Source: §6.1.2 · p. 20, Figure 2 · p. 21

6.1.3 Estimate of true probabilities

  • Probability-estimation MSE depended on concentration: low concentration favored higher m and correct-or-higher rank, high concentration favored lower m and rank, while φ = 1 required tuning both rank and m.In a sparse simulation with mean probability 0.1, lower-than-true ranks performed better overall, possibly because predicting zeros was reasonably accurate.Source: §6.1.3 · pp. 21–23, Figure 3 · p. 22
  • With five-fold cross-validation to choose m, LPCA was more accurate at high and middle concentration, while LSVD was more accurate at low concentration unless the estimated rank was at least the true rank.The authors note that choosing the number of components remains necessary for both methods.Source: §6.1.3 · p. 23, Figure 4 · p. 24

6.2.1 Selecting number of principal components

  • LSVD’s cumulative deviance explained reached 90% at 24 components, while marginal contributions for LPCA and LSVD leveled off after two; the authors chose two for manageable analysis.The figure also includes standard PCA.Source: §6.2.1 · pp. 24–25, Figure 5 · p. 25

6.2.3 Evidence of LSVD overfitting through principal component regression

  • On new patients, LPCA loadings reconstructed data much better than PCA loadings; in controlled regression simulations, LSVD performed worse than PCA and LPCA on training and test data.The LSVD–LPCA MSE gap was larger on test data, but the authors caution that LPCA is not guaranteed to generalize better in every setting.Source: §6.2.2 · p. 25, §6.2.3 · pp. 26–27, Figure 7 · p. 27

6.2.2 Quality of fit

  • For patient-diagnosis fit, LPCA outperformed PCA at equal parameters per component; LSVD explained more deviance after a few components, but LPCA looked favorable when compared by free-parameter count.The authors attribute the LSVD comparison partly to its extra parameters in A.Source: §6.2.2 · p. 25, Figure 6 · p. 26

6.2.4 Interpretation of loadings

  • Two LPCA components showed high loadings for serious co-occurring ICU conditions, and subject-matter experts found meaning in the components.The first emphasized acute kidney and respiratory failure; the second included several circulatory diseases.Source: §6.2.4 · p. 28

7 Discussion

The discussion presents logistic PCA as a projection of saturated-model natural parameters, retaining linear scores and simple scoring on new data. It also notes extensions to other exponential-family data and an open question in sparse-loadings optimization.

7 Discussion

  • The formulation projects natural parameters from the saturated model, yielding principal components that are linear combinations of the data and can be computed on new data.The paper contrasts this with earlier logistic PCA formulations that extend SVD.Source: §7 · p. 28
  • Two algorithms minimize deviance, and their performance is demonstrated on simulated and real data.The figure caption identifies two logistic-PCA components for the patient-diagnosis data.Source: §7 · p. 29, Figure 9 · p. 29
  • The formulation can extend to other exponential-family distributions using their deviance and saturated-model natural parameters.Source: §7 · p. 29
  • Sparse loadings remain an open optimization question, although adding a sparsity constraint to the Fantope solution is described as straightforward.The paper says further research is needed to find the best way to solve for the loadings.Source: §7 · p. 29

A.1 Calculation of gradient for logistic PCA

This appendix derives the deviance gradients with respect to the loading matrix and the main-effect vector, using standard matrix and vector differentiation.

A.1 Calculation of gradient for logistic PCA

  • The appendix derives the deviance gradient with respect to U using standard matrix derivative rules.Source: §A.1 · pp. 30–31
  • For the U derivative, the fitted natural parameter is written as μ_j + [UUᵀ(θ̃_i − μ)]_j.Source: §A.1 · p. 30
  • The appendix also derives the deviance gradient with respect to μ using standard vector differentiation.It defines e_j as the length-d standard basis vector with 1 in position j.Source: §A.1 · p. 31

A.2.1 Theorem 1

The theorem’s special case takes the component direction to be a standard basis vector and gives an analytic choice of main effects; the derivation also identifies an uncorrelated-columns condition and a residual that shrinks as m increases.

A.2.1 Theorem 1

  • For u = e_l, the main effects have an analytic solution, with μ̂_j = logit X̄_j for j ≠ l and μ̂_l arbitrary because it does not affect deviance.Source: §A.2.1 · p. 31
  • The derivation uses the assumption that X_j and X_l are uncorrelated; in the noted case, the squared norm of the residual can be made arbitrarily small by increasing m.Source: §A.2.1 · p. 32
  • With μ = μ̂ and u = e_l, the first-order optimality conditions are satisfied exactly under the stated construction.Source: §A.2.1 · pp. 31–32

A.2.2 Theorem 2

For a basis-vector component, the theorem characterizes the deviance across columns and identifies the column-mean choice that yields the lowest deviance.

A.2.2 Theorem 2

  • With u = e_l and μ̂_j = logit X̄_j for j ≠ l, the lth column’s deviance does not depend on its column mean, and total deviance depends on the other d − 1 columns.Source: §A.2.2 · pp. 32–33
  • The deviance of column j decreases as X̄_j moves farther from 1/2.Source: §A.2.2 · p. 33
  • Choosing l whose column mean is closest to 1/2 yields a fit with the lowest deviance.Source: §A.2.2 · p. 33

A.2.3 Theorem 3

The appendix establishes optimality and gradient properties for structured logistic PCA solutions, then compares convergence and probability estimates across simulated scenarios. LSVD generally has lower or equivalent deviance, while LPCA typically converges in fewer iterations.

A.2.3 Theorem 3

  • For k = 1 without main effects, compound symmetry of the relevant matrix yields an optimal loading vector proportional to 1_d and equal fitted probabilities across columns.The proof establishes compound symmetry and shows the resulting loading vector satisfies the first-order optimality conditions.Source: §A.2.3 · pp. 33–34

A.2.4 Theorem 4

  • The symmetrized gradient is Lipschitz continuous under the bound established in Theorem 4.The proof obtains the bound using inequalities including Cauchy-Schwarz and a preceding lemma.Source: §A.2.4 · p. 35

A.3.1 Number of iterations to convergence

  • LPCA generally requires fewer convergence iterations than LSVD, which often reaches the maximum; LSVD iterations rise with component count for φ ∈ {1, 3}.For LPCA, iterations generally increase as m increases.Source: Figure 10 · p. 36, §A.3.1 · p. 36

A.3.2 Simulation results for more sparse situation

  • With ¯p = 0.1, results are mostly like the ¯p = 0.5 setup, but estimating below the true rank degrades performance less severely.The sparse-situation estimates included main effects.Source: Figure 11 · p. 37, §A.3.2 · p. 37

A.3.3 Deviance comparison of LPCA and LSVD

  • LSVD has lower or equivalent deviance across scenarios, with its advantage smallest at φ = 0.01 and greater for concentrated probabilities and higher estimated ranks.The deviance comparison uses the ¯p = 0.5 setup.Source: §A.3.3 · pp. 37–38, Figure 12 · p. 38
Loading 1510.06112v1…