PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagonal Attenuation: A Finite-Sample Correction for PCA

Qiang Sun

arXiv:2609.05796v1stat.MLcs.LGmath.STstat.MEstat.TH

TL;DR

Finite-sample covariance fluctuations can rotate PCA away from its population target, motivating a correction that preserves sample cross-covariances while reducing sample variances. The paper derives diagonal attenuation from random-mask reconstruction and extends it beyond the mask-derived range. Both attenuation paths improve on PCA across the reported data–budget cells, with one achieving the largest mean gain in every cell.

  • Problem

    With limited data, sample fluctuations can rotate the fitted PCA subspace away from its population target, and coordinatewise sample-variance errors contribute to that rotation.

    Source: §1 · p. 2

  • Method

    The paper derives diagonal attenuation by averaging full-output reconstruction loss over random input masks, then studies a directly tunable path that preserves sample cross-covariances while reducing sample variances.

    Source: §1 · p. 2

  • Results

    Both attenuation paths improve on PCA in all six data–budget cells, and one has the largest mean gain in every cell.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Attenuation can reduce sampling leakage; balanced marginals yield an explicit risk theorem, while general covariances require accounting for changing eigengaps and population cost when the target moves.

    Source: §3.2 · p. 9, §5 · p. 14

    The main risk theory is fixed-dimensional and Gaussian, while the real-data studies cover local image, frequency, and temporal coordinates.

    Source: §5 · p. 14

Abstract

from arXiv · show

Principal component analysis (PCA) can rotate away from its population target when a covariance matrix is estimated from limited data. We introduce diagonal attenuation, which preserves sample cross-covariances while reducing coordinatewise sample variances. The method is revealed exactly by averaging a linear full-output reconstruction loss over random input masks; studying the correction directly extends it beyond the range attainable by masking. We isolate the part of the random coupling between retained and omitted population directions that is contributed by sample-variance errors, and show how attenuation can reduce the resulting rotation. Under balanced marginal variances, we derive an explicit expected-risk theorem, uniform over the attenuation path for all sufficiently large finite samples, and obtain the asymptotically risk-minimizing strength. For general covariances, we characterize when attenuation leaves the population PCA subspace unchanged and give a risk theorem that also accounts for changing eigengaps and the population cost when the target moves. Simulations track this tradeoff from exact preservation back to PCA. Across local image patches, speech spectra, and smartphone acceleration, both mask-derived and direct attenuation improve PCA under two fitting-sample budgets, and one of them has the largest mean gain among seven methods in every data--budget cell. The full path selects strengths beyond the mask-derived boundary on $63\%$--$95\%$ of the subsamples.

1 Introduction

The paper targets finite-sample PCA rotation caused in part by coordinatewise sample-variance errors, and introduces diagonal attenuation to reduce those errors while retaining sample cross-covariances. It extends masking-derived attenuation into a directly tunable path and compares its clean reconstruction risk with PCA.

1 Introduction

  • Coordinatewise sample-variance errors can couple retained and omitted population directions, while diagonal attenuation reduces sample variances without changing sample cross-covariances.Masking derives the correction, and direct attenuation extends it beyond the mask-derived range; the theory also accounts for changes to the population target and eigengaps.Source: §1 · p. 2, Figure 1 · p. 3
  • Both attenuation paths improve on PCA in all six data–budget cells, and one has the largest mean gain in every cell.The comparisons cover local image patches, speech spectra, and smartphone acceleration.Source: §1 · p. 3

1.1 Related work

  • Unlike diagonal-target shrinkage, diagonal attenuation retains observed cross-covariances and weakens only the sample diagonal; this paper studies complete covariance draws and clean PCA risk.The related-work discussion distinguishes this setting from noisy or incomplete low-rank matrix recovery, masked-target objectives, and broader linear reconstruction maps.Source: §1.1 · pp. 3–4

1.2 Notation

  • The notation defines diag(A) as retaining the diagonal, off(A) as A−diag(A), P_A as a leading rank-d projector, and α as attenuation strength.The mask rate is m.Source: §1.2 · p. 4

2 Diagonal attenuation for PCA

This section defines the clean PCA target and risk, derives diagonal attenuation by averaging over random input masks, and extends the resulting correction to a directly tunable estimator path. It then characterizes when attenuation preserves or moves the population PCA target.

2.1 PCA target and clean risk

  • The clean reconstruction risk is evaluated on a fresh, unmasked observation, with the population covariance defining the PCA target.The section introduces the target and risk for rank-d orthogonal projectors.Source: §2.1 · p. 4

2.2 From input masking to diagonal attenuation

  • Averaging the full-output reconstruction loss over independent Bernoulli masks makes its optimizers leading rank-d projectors of the attenuated sample covariance.The correction retains sample cross-covariances and multiplies sample variances by 1−α.Source: §2.2 · p. 5
  • Direct attenuation extends the mask-derived range 0≤α<1/2 to 0≤α≤1, with ordinary PCA at α=0 and diagonal-deletion PCA at α=1.The full interval can be searched by held-out reconstruction with PCA included as a candidate.Source: §2.2 · p. 5

2.3 Population target under attenuation

  • The population cost B_pop(α) is the reconstruction cost remaining without sampling error, and it is zero exactly when the attenuated population projector remains optimal for PCA risk.The cross-block vanishes exactly when the target subspace is invariant under the attenuated population covariance.Source: §2.3 · p. 5
  • For α>0, the target remains the unique leading rank-d population projector exactly when attenuation neither mixes target and discarded subspaces nor reverses their spectral ordering.The analysis assumes Gaussian sampling and fixed dimension with a positive target eigengap; finite-moment extensions are given in the supplement.Source: §2.3 · p. 6

3 Why attenuation can improve finite-sample PCA

Attenuation changes sampling-induced coupling between retained and omitted directions, reducing variance-related rotation under balanced marginals. For general covariances, whether it helps also depends on eigengaps and any population cost from moving the target.

3.1 Balanced marginals: reducing sampling-induced rotation

  • Balanced marginals keep the population subspace and eigengaps fixed, isolating sample-fluctuation couplings across the rank cutoff as the source of estimated-subspace change.The retained–omitted coupling is a_jk−αb_jk, where b_jk is the sample-diagonal contribution.Source: §3.1 · p. 7
  • The residual coupling’s first-order rotation is divided by its population eigengap, and its squared value contributes to clean reconstruction risk.A larger eigengap weakens the effect of the same residual coupling.Source: §3.1 · p. 7
  • The sample-diagonal component’s nonnegative overlap with the unattenuated coupling explains why small attenuation reduces residual-coupling variance, while excessive attenuation can increase it.The latter effect comes from the quadratic term in the variance expression.Source: §3.1 · p. 8
  • For n≥N, expected risk is lower than PCA when 2αCΣ−α²VΣ > 2C0/√n; the risk-minimizing strength can exceed the mask-derived range.Every fixed 0 < α < min{2CΣ/VΣ, 1} improves on PCA for all sufficiently large finite samples.Source: §3.1 · p. 8

3.2 General covariance: changing eigengaps and population leakage

  • For general covariances, target preservation allows unequal marginal variances when population leakage vanishes and the block gap remains positive across the attenuation interval.Attenuation may still change eigengaps even when the population PCA subspace is unchanged.Source: §3.2 · p. 9
  • The general risk analysis collects sampling leakage, converts it to rotation using attenuated eigengaps, then weights that rotation by the original PCA eigengaps.The attenuated gap operator maps retained–omitted entries by their attenuated eigengaps.Source: §3.2 · p. 9
  • Under target preservation, expected risk is lower than PCA for n≥N when K(0)−K(α) > 2C/√n, accounting for residual-coupling variation and attenuated eigengaps.When D=τI_p, this risk curve reduces to the balanced-case pairwise formula.Source: §3.2 · p. 10
  • When attenuation moves the target, it helps only when the finite-sample gain pays for the population reconstruction cost; increasing leakage shifts the preferred strength toward PCA.The signed finite-sample correction around the moved subspace can reflect sampling rotation toward or away from the original target.Source: §3.2 · p. 10

4 Numerical studies

The numerical studies test the predicted risk tradeoff as attenuation preserves or moves the population target, then compare both attenuation paths with five alternatives under shared fitting-sample budgets. Both paths improve PCA across all six real-data cells, with one path achieving the largest mean gain in each.

4 Numerical studies

  • The controlled Gaussian experiments compare balanced, unequal target-preserving, and continuously moving-target covariances, with 3,000 samples per setting.The real-data studies use N=64 and 128, five-fold reconstruction validation, and official test sets across image patches, speech spectra, and acceleration signals.Source: §4 · p. 10, §4.1 · pp. 10–11

4.1 Finite-sample risk across preserved and moving targets

  • The theoretical risk curve tracks Monte Carlo risk in both target-preserving designs; best simulated attenuation lowers scaled PCA risk by 16.3% and by 63.4% at α=0.90 in the unequal design.Across 13 movement levels, the theoretical and Monte Carlo minimizing strengths differ by at most 0.025; preferred attenuation falls to α=0 as target movement grows.Source: Figure 2 · p. 11, §4.1 · pp. 11–12

4.2 Fixed-budget studies on image patches and ordered signals

  • The real-data comparison uses shared fitting observations and validation folds to assess attenuation against PCA, correlation PCA, HeteroPCA, and coordinate- and block-masked-target estimators.Direct attenuation selects α>1/2 on 94%–95% of image-patch fits and 63%–80% of ordered-signal fits.Source: §4.2 · pp. 12–13
  • Both attenuation paths have positive mean gains with intervals above zero in all six data–budget cells, and one leads in every cell.Direct attenuation leads in five cells; mask-derived attenuation leads for Phoneme at N=128.Source: Figure 3 · p. 13, §4.2 · p. 13, Table 1 · p. 13

5 Discussion

Diagonal attenuation reduces sample-variance-driven couplings that can rotate the fitted subspace, but its risk benefit must be weighed against changes to eigengaps and population targets. The main risk theory is fixed-dimensional and Gaussian, while the real-data studies cover three modalities.

5 Discussion

  • Attenuation can weaken random retained–omitted couplings arising from coordinatewise sample-variance errors before they rotate the fitted subspace.Its tradeoff includes altered eigengaps and, when the target moves, a population reconstruction cost.Source: §5 · p. 14
  • The explicit balanced-marginal risk theorem applies when attenuation changes sampling leakage without moving the target or its eigengaps.For general covariances, the risk comparison must account for changing eigengaps and any population cost of target movement.Source: §5 · p. 14
  • The main risk theory is fixed-dimensional and Gaussian; the three real-data studies cover local image, frequency, and temporal coordinates.The supplement extends the balanced formula under finite-moment conditions; structured attenuation beyond a common diagonal strength is left for future work.Source: §5 · p. 14

AI use statement

The author reports that generative-AI tools assisted with literature discovery and synthesis, proof cross-checking, code review, and language editing.

AI use statement

  • The author reports AI assistance with literature discovery and synthesis, proof cross-checking, code review, and language editing, while taking responsibility for the manuscript.The author states that cited primary sources, derivations, and reported computations were verified.Source: AI use statement · p. 14

Supplementary Material

The supplement supplies proofs and extensions for the main results, then compares reconstruction objectives and reports empirical details. Its proof guide maps results to arguments involving mask marginalization, population geometry, and risk expansions.

Supplementary Material

  • The supplement proves exact mask marginalization, develops population geometry, and establishes target-preserving, target-moving, and balanced-marginal risk results.It also extends the balanced theorem beyond Gaussian observations and establishes a validation fallback guarantee.Source: Supplementary Material · p. 15
  • The supplement compares the full-output projector with masked-target and unconstrained linear reconstruction objectives, and reports a non-Gaussian stress test and implementation details.It also provides full real-data tables.Source: Supplementary Material · p. 15
  • The proof guide maps the estimator derivation to leading-eigenspace optimization and the risk results to projector rotation and clean reconstruction risk.Gaussian pairwise moments yield the quadratic risk curve for retained-to-omitted residual couplings.Source: Supplementary Material · p. 16

S.1 Mask marginalization and estimator construction

Averaging the masked-input loss yields a PCA estimator based on a diagonally attenuated sample covariance. Its attenuation strength is determined by the mask rate.

S.1 Mask marginalization and estimator construction

  • The proof obtains the estimator by taking expectations of the masked reconstruction loss over independent Bernoulli-q input-mask coordinates.The derivation uses equality of the two scalar cross terms before taking the expectation.Source: §S.1 · pp. 16–17
  • Minimizing the averaged loss over rank-d projectors is equivalent to maximizing the trace against the attenuated sample covariance.Ky Fan’s variational principle identifies its leading rank-d projector as the optimizer.Source: §S.1 · p. 17
  • The attenuation parameter maps to the Bernoulli mask rate as α = (1 − q)/(2 − q) = m/(1 + m).The positive prefactor q(2 − q) does not affect the optimizer for q > 0.Source: §S.1 · p. 17

S.2 Population geometry under attenuation

Attenuation can either preserve the population PCA target or move it along the path T_α = Σ − α·diag(Σ), with population cost quantified when preservation is approximate.

S.2 Population geometry under attenuation

  • The population target is evaluated along T_α = Σ − α·diag(Σ), with exact preservation and approximate preservation treated separately.The section characterizes exact preservation and quantifies the population cost when preservation is only approximate.Source: §S.2 · p. 17

S.2.1 Exact preservation with unequal marginals

Balanced marginal variances are sufficient but not necessary for attenuation to preserve the population PCA subspace. An unequal-marginal example retains the target across the full path.

S.2.1 Exact preservation with unequal marginals

  • Balanced marginal variances suffice for exact preservation, but they are not required.The unequal-marginal design provides an example of preservation without balance.Source: §S.2.1 · p. 17
  • The unequal-marginal example retains the PCA target throughout the attenuation path, with infimum cutoff gap 7/2 − 3/2 > 1.37.Its diagonal is D = diag(4, 4, 2.5, 2.5), and the leading subspace is span{u1, u2}.Source: §S.2.1 · p. 17
  • Exact preservation holds when the target subspace and its complement are invariant under D and the cutoff gap remains positive.Under these conditions, T_α is block diagonal in that decomposition and its leading rank-d projector is uniquely the population target.Source: §S.2.1 · p. 17

S.2.2 Population bias along the attenuation path

The population cost of attenuation is controlled along a path with a positive cutoff eigengap. Movement remains small when diagonal cross-subspace coupling is limited and the gap stays separated.

S.2.2 Population bias along the attenuation path

  • Population movement remains small when diagonal cross-subspace coupling is limited and the cutoff eigengap stays separated along the path.The analysis follows T_t = Σ − tD from the original population projector and requires a positive cutoff gap.Source: §S.2.2 · p. 17
  • Larger eigengaps reduce rotation from residual directions, so distant residual directions require no separate tail penalty.The local population-bias expression makes this gap dependence explicit.Source: §S.2.2 · p. 18
  • When attenuation moves the target, the population term remains in the expected-risk decomposition alongside the sampling contribution.Exact preservation removes the population term; the target-moving result retains it and quantifies sampling around the attenuated target.Source: §S.2.2 · p. 18

S.3 Proofs of the finite-sample risk results

The supplement uses a common projector perturbation argument for both target-preserving and target-moving risk theorems, then specializes the target-preserving coefficient to balanced marginal variances.

S.3 Proofs of the finite-sample risk results

  • A common projector perturbation argument supports both target-preserving and target-moving risk theorems, with the balanced-marginal case making the sampling-leakage calculation explicit.Source: §S.3 · p. 18

S.3.1 A common clean-risk expansion

The section develops a common clean-risk framework for PCA targets and diagonal attenuation, covering balanced and general settings, validation selection, and distinctions among reconstruction objectives. It also reports robustness and fixed-budget real-data studies.

S.3.1 A common clean-risk expansion

  • A two-matrix projector expansion provides a common clean-risk approximation whether or not the population target is preserved, with uniform remainder control over compact families having bounded norms and gaps.The lemma is based on expressing the perturbed leading subspace as a graph over the original one.Source: §S.3.1 · pp. 18–19

S.3.2 Target-preserving expected risk

  • When attenuation preserves the population PCA target, the common expansion gives a uniform expected-risk result across the attenuation path under a positive gap.The expected cubic remainder is O(n^-3/2), uniformly in α, and the stated bound covers all sample sizes after enlarging its constant for finitely many initial sizes.Source: §S.3.2 · pp. 19–20

S.3.3 Total risk around a moving population target

  • The moving-target theorem separates population displacement from sampling-induced rotation, whose second-order risk includes both variance and average displacement.The displacement contribution can be signed: attenuation improves on PCA to first order when its sampling-risk reduction exceeds its population movement cost; if that cost has a unique zero at α = 0, oracle risk minimizers converge to zero as sample size grows.Source: §S.3.3 · pp. 20–22

S.3.4 Balanced marginals and the explicit risk curve

  • Under balanced marginal variances, the target-preserving risk coefficient becomes an explicit pairwise quadratic, yielding an optimizable attenuation curve and eventual finite-sample improvement when that curve beats PCA.The quadratic is strictly convex; in the flat-spike benchmark, risk is nonincreasing throughout 0 ≤ α ≤ 1.Source: §S.3.4.2 · pp. 24–25

S.4.1 Risk expansion beyond Gaussian sampling

  • With balanced marginals and finite eighth moments, the risk expansion extends beyond Gaussian sampling using the actual covariance between rotation fluctuations and their diagonal components.The direct leading-risk oracle is clipped to [0, 1], while the mask-path oracle is clipped to [0, 1/2]; a fixed attenuation level improves leading risk when 0 < α < 2Cν/Vν, subject to α < 1/2 on the mask path.Source: §S.4.1 · pp. 25–26

S.4.2 Validation selection with a PCA fallback

  • Independent validation selects among fitted attenuation candidates with a finite-sample guarantee, and the candidate set’s inclusion of PCA provides an exact fallback as validation size grows.The guarantee conditions on training-fitted rank-d projectors and requires no post-selection refitting.Source: §S.4.2 · p. 26

S.5 Alternative masking and reconstruction objectives

  • The section delineates the full-output orthogonal-projector objective from masked-target reconstruction and unconstrained linear autoencoders, then introduces the general block masked-target projector loss.Source: §S.5 · p. 27, §S.5.1 · p. 27

S.5.1 The general block masked-target objective

  • Group masking attenuates within-group covariance blocks, and its masked-target objective is nonlinear in P and generally not a spectral estimator.The objective can be compared locally with diagonal attenuation using the same retained–omitted direction pairs.Source: §S.5.1 · pp. 27–28

S.5.2 Masked-target random slopes at the PCA target

  • Under balanced marginal variances, diagonal attenuation has zero population leakage and retains PCA curvature, whereas masked-target PCA requires stationarity and positive curvature.In the two-dimensional example, PCA is the masked-target solution when λ1/λ2 ≥ 3 and is not even a local minimum when 1 < λ1/λ2 < 3.Source: §S.5.2 · pp. 29–30

S.5.3 Linear reconstruction beyond the projector constraint

  • Beyond projectors, the section characterizes optimal asymmetric rank-at-most-d maps and positive-semidefinite tied maps; the asymmetric optimum is a best rank-d approximation to C_n,q.Under population balance, the asymmetric map’s range is the PCA subspace, but its raw reconstruction map is generally not the PCA projector.Source: §S.5.3 · pp. 30–31

S.5.4 Additional related context

S.6.1 Robustness beyond Gaussian observations

  • The moment-risk prediction’s sign and curvature persist for Gaussian, standardized Student t10, and standardized centered-exponential observations in the controlled design.At n = 1024, the largest absolute difference between simulated scaled risk and the moment prediction was 1.02.Source: §S.6.1 · p. 32, Figure S.1 · p. 33

S.6.2 Fixed-budget real-data comparisons

  • Masked-target fits use two-start optimization rather than a certified global optimum; 86 of 1,163 nonfallback coordinate-masked fits reached the iteration cap.None of the 175 nonfallback block-masked fits reached the cap.Source: §S.6.2 · p. 33
  • Both diagonal-attenuation paths have positive mean gains with intervals above zero in all six real-data cells, while the compared alternatives do not reproduce this uniform improvement.The comparison covers local image patches, speech spectra, and smartphone acceleration.Source: §S.6.2 · p. 33, Table S.1 · p. 34, Table S.2 · p. 34
  • The direct path selects strengths beyond 1/2 on 63%–95% of fits, while every search retains PCA as a candidate.Mean selected strengths are 0.351–0.479 on the mask-derived path and 0.615–0.931 on the full direct path.Source: §S.6.2 · p. 33, Table S.3 · p. 34
Loading 2609.05796v1…