PaperlayerView exact revision on arXiv

Source-linked AI summary

Search Dimension in Unlabeled Projection Pursuit: A Scaling Law for Subspace Restriction

Rares Dimitrie Grozavescu, Mark Girolami

arXiv:2609.37917v1cs.LGmath.STstat.MLstat.TH

TL;DR

Unrestricted projection pursuit can find an uninformative direction in a large Gaussian complement, even when the latent regime remains recoverable. The paper compares exact-subspace search with data-driven subspace estimation and measures their threshold scaling. At fixed search dimension, the measured exponent is close to 1/8, but the observed search-dimension dependence differs and is not identified by the tested range.

  • Problem

    In high dimensions, projection pursuit searches many directions, most of which may carry no information about the latent structure, creating a risk of a spurious empirical optimum.

    Source: §1 · p. 1

  • Method

    The paper characterizes unrestricted-search failure and compares exact-subspace search with data-driven subspace estimation in a controlled model.

    Source: §1 · p. 2, §5 · p. 4, §5.2 · p. 7

  • Results

    At fixed search dimension, the measured threshold-ratio exponent is 0.156 (95% CI [0.118, 0.174]), close to 1/8, while varying search dimension gives 0.325 and an unresolved functional form.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Restricting an unlabeled search to the known signal subspace can remove a high-dimensional failure mode without changing the population target on the negative-kurtosis branch.

    Source: §8 · p. 8

    The ς^-8 dependence comes from a sufficient-bound analysis of the fourth-moment criterion, not an information-theoretic lower bound for the full inference problem.

    Source: §8 · p. 8

Abstract

from arXiv · show

Projection pursuit searches for a direction along which the data look least Gaussian. When the observation space contains a large Gaussian complement, the empirical objective can be minimized by a direction that carries no signal, with empirical kurtosis as low as at the truth. Sample splitting exposes rather than repairs this failure. Appending coordinates independent of the latent regime degrades the search while leaving Bayes recoverability unchanged. Restricting the search to the column space of a known forward operator removes the failure exactly on the negative-kurtosis branch. Estimating a principal subspace from the data is the alternative. In a controlled two-component model, the leading sufficient scalings differ in the gain with which the operator transmits the discriminant: $\varsigma^{-4}$ for covariance-spike estimation and $\varsigma^{-8}$ for fourth-moment search. At fixed search dimension, the measured threshold ratio collapses onto $n/p^2$ with exponent $0.156$, close to the predicted $1/8$. This is an empirically supported scaling motivated by sufficient bounds, not a proved asymptotically tight law. When the search dimension is varied, the measured exponent is $0.325$, substantially larger than $1/8$, and the tested range does not identify its functional form. The crossing location also depends on calibration and model configuration. Under a downstream excess-error criterion, the scaling largely disappears.

1 Introduction

Unrestricted minimum-kurtosis projection pursuit can select an uninformative direction from a large Gaussian complement, despite unchanged information about the latent regime. The paper analyzes this failure, compares exact-subspace search with data-driven subspace estimation, and tests how their thresholds scale.

1 Introduction

  • With Gaussian-complement dimension at least n −1 for even n, empirical excess kurtosis reaches −2 on an uninformative direction, although its population kurtosis is zero.Before this exact degeneracy, spurious minima already deepen with searched dimension.Source: §1 · p. 1
  • At fixed search dimension, the measured threshold-ratio exponent is 0.156 (95% CI [0.118, 0.174]), close to 1/8; varying search dimension yields 0.325.The tested range does not distinguish power-law, logarithmic, and linear forms; crossing also changes with calibration and model configuration.Source: §1 · p. 2
  • Known-subspace restriction removes the population failure on the negative-kurtosis branch, while a finite-sample direction-recovery bound motivates comparison with data-driven subspace estimation.The paper distinguishes exact-subspace search from the separate cost of estimating a useful search subspace.Source: §1 · p. 2
  • Sample splitting cannot repair a search-limited procedure, and under downstream excess-error thresholds the direction-recovery scaling largely disappears.The latter criterion is less sensitive to weak-signal direction differences.Source: §1 · p. 2

2 Setup and exact reference

The paper studies minimum-kurtosis projection pursuit in a linear–Gaussian inverse model with a finite Gaussian-mixture prior. It also specifies a Bayes classifier reference and a two-component experimental setting.

2 Setup and exact reference

  • The setup uses a linear–Gaussian inverse model with a finite Gaussian-mixture prior for the latent variable.The whitened observation is written as e y = S−1/2y = eGz + ˜ε, with ˜ε ∼ N(0, I_p).Source: §2 · p. 2
  • The signal subspace is R, with dimension r, and the Bayes classifier for the Gaussian-mixture model serves as the exact reference.Downstream procedures are evaluated on held-out data without held-out regime labels during fitting.Source: §2 · p. 2
  • Main experiments use two equally weighted Gaussian components with isotropic within-component covariance and mean separation along one latent coordinate.For two components, the population excess kurtosis is parameterized by weights, projected separation, and projected variances.Source: §2 · p. 2

3 The operator determines the population search space

The forward operator identifies the signal subspace, while its orthogonal complement contains only independent standard Gaussian noise. Mixing in that complement dilutes population excess kurtosis, making restriction to the known subspace lossless for population minimization on the negative-kurtosis branch.

3 The operator determines the population search space

  • Directions in R⊥ contain no latent information, and the orthogonal complement is independent of the latent variable and projected signal.This identifies the geometric source of the unrestricted-search failure.Source: §3 · p. 3
  • On the negative-kurtosis branch, adding an orthogonal Gaussian component attenuates excess-kurtosis magnitude, so restricting search to R preserves the population minimum.The dilution relation is given as κ(u) = κa λ(θ)^2.Source: §3 · p. 3
  • If the relevant in-subspace kurtosis is positive, the minimum instead lies in the Gaussian complement; the direction-recovery experiments use the negative branch.The dilution mechanism also extends beyond equal mixture weights, with amplitude vanishing at specified weight and covariance-separation boundaries.Source: §3 · p. 3

4 Finite-sample search and spurious minima

Uniform finite-sample control depends on the dimension searched, so exact restriction to the signal subspace replaces ambient dimension p with subspace dimension r. An unrestricted search can nevertheless attain the empirical kurtosis floor on a direction containing no signal, with spurious minima appearing even before that floor is attainable.

4 Finite-sample search and spurious minima

  • Uniform concentration is controlled by dsearch: unrestricted search uses p, whereas exact restriction to R uses r.The sub-Gaussian scale bound does not depend on ambient dimension p.Source: §4 · p. 3
  • For V ≥2, the local coefficient 2|κ⋆|/V makes the direction-recovery bound sharp to first order.The coefficient is the local coefficient of sin2 d at u⋆.Source: §4 · p. 4
  • The leading term gives a sufficient sample-size scaling up to logarithmic and problem-dependent factors; the untruncated bound also has a second-order branch with stronger dsearch dependence.The measured thresholds lie in the regime where the leading branch is relevant.Source: §4 · p. 4

4.1 Two failure modes of the unrestricted search

  • For even n and Gaussian-complement dimension at least n −1, an uninformative direction attains empirical excess kurtosis −2 while its population kurtosis is zero.This is the exact degeneracy of the unrestricted empirical search.Source: §4.1 · p. 4
  • Before the exact floor is attainable, the Gaussian complement can already contain spurious minima, and their depth grows with the searched dimension.The empirical objective increasingly favors directions without signal as irrelevant dimensions are added.Source: §4.1 · p. 4
  • Fourth-moment pursuit remains useful when separation is expressed through covariance, a regime outside the main scaling experiment.The second-moment route is cheaper in the mean-separated, well-conditioned setting used for that comparison.Source: §4.1 · p. 4

5 Comparing exact-subspace search with subspace estimation

The section compares an oracle search in the true signal subspace with PCA-based subspace estimation, measuring how their direction-recovery thresholds scale and how calibration affects their crossing. The measured rank dependence differs from the leading prediction, and its functional form remains unresolved.

5 Comparing exact-subspace search with subspace estimation

  • The oracle exact-subspace search isolates the cost of finding a direction in a known search space, whereas PCA plus search also pays to estimate that space.The comparison concerns direction recovery; downstream excess-error thresholds largely lose the scaling because prediction error is less sensitive to weak-signal direction differences.Source: §5 · pp. 4, 6
  • The comparison assumes two components, isotropic latent covariance, and a discriminant that is the covariance spike determining the PCA threshold.A stronger nuisance spike can let PCA recover another direction first; covariance-only separation does not supply the competing second-moment signal.Source: §5 · p. 5
  • At fixed search dimension, the threshold ratio follows a power law in n/p^2, with a confidence interval containing the predicted 1/8 exponent.The collapse fit covers 11 configurations; the rank comparison covers 28.Source: §5 · p. 5, Table 1 · p. 5
  • At matched p and n, the KPP threshold has rank exponent 0.325 versus the predicted 1/8, while the PCA threshold shows no corresponding rank dependence.The tested range establishes rank dependence but does not distinguish power-law, logarithmic, and linear forms.Source: §5 · pp. 5–6

5.1 The threshold crossing depends on calibration

  • The threshold-ratio crossing is not universal: changing the angular criterion shifts it by a factor of four, and changing the strongest covariance spike shifts it by two orders of magnitude.Anisotropic latent covariance changes the prefactor non-constantly across ranks, so no single crossing applies.Source: §5.1 · p. 6, Figure 1 · p. 6, Table 3 · p. 6

5.2 Calibrating the angular criterion

  • Referencing the angular criterion to a fixed fraction of the null median removes the observed anomaly and leaves the measured exponent stable across tested fractions.False-positive-rate calibration alone does not fully remove the effect because the null distribution changes in scale as well as tail mass.Source: §5.2 · p. 6

6 Sample splitting does not repair the search

Sample splitting can validate only directions the fitting-stage search proposes, so it does not repair unrestricted-search failure. Adding latent-independent coordinates preserves Bayes decisions while degrading that search.

6 Sample splitting does not repair the search

  • Appending coordinates independent of the latent regime enlarges the unrestricted search space without changing exact Bayes decisions.The added coordinates degrade search performance despite unchanged Bayes recoverability.Source: §6 · p. 7
  • Doubling the points in the split arms leaves the unrestricted direction largely unchanged while improving the restricted arm.In this regime the unrestricted procedure is search-limited and the restricted procedure is sample-limited.Source: §6 · p. 7
  • The operator-based subspace succeeds in almost every configuration, but exact-rank restriction can perform worse when the discriminant is not the recovery-determining spike.In the single failing cell, Baik–Ben Arous–Péché rank selection recovers the discriminant, while supplying the exact rank spends extra dimensions on noise.Source: §6 · p. 7
  • Out-of-sample kurtosis for the unrestricted procedure remains near zero while the restricted procedure approaches κ⋆; unrestricted directions also become nearly orthogonal to u⋆ as ambient dimension increases.Figure 2a shows the split search following the unrestricted full-sample arm, while the restricted arm approaches κ⋆.Source: Figure 2 · p. 7, §6 · p. 7
  • A larger validation sample can rank u⋆ above the selected candidate, but cannot recover it if the fitting-stage search never proposed it.With nB = 800 validation noise can favor a spurious candidate; with nB = 12800 validation ranks u⋆ correctly.Source: Figure 2 · p. 7, §6 · p. 7

7 Related work

Prior work studies projection-pursuit contrasts, high-dimensional projection behavior, and subspace restriction in several settings. This paper instead focuses on finite-sample effects of the searched observation space in unlabeled projection pursuit.

7 Related work

  • Projection pursuit traces to Friedman and Tukey (1974) and Huber (1985), while minimum-kurtosis mixture separation was studied by Peña and Prieto (2001).Kurtosis is also a standard fourth-moment contrast in independent component analysis (Hyvärinen and Oja, 2000).Source: §7 · p. 7
  • Bickel et al. (2018) and Montanari and Zhou (2025) study high-dimensional Gaussian projections, while Mukherjee et al. (2023) prove recovery for Wasserstein projection pursuit under linear p/n scaling.The latter uses a different index, but shares the high-dimensional search over many directions.Source: §7 · pp. 7–8
  • The paper asks whether a known Gaussian complement can be removed from an unlabeled projection-pursuit search without changing its population target.It contrasts this observation-space restriction with prior subspace-based approaches in other settings.Source: §7 · pp. 7–8
  • Radojicic et al. (2021) analyze asymptotic properties of blind kurtosis- and skewness-based discriminant estimators, whereas this paper studies finite-sample effects of the searched observation space.Likelihood-informed subspaces reduce parameter-space dimension in specified inverse problems; this paper restricts observation-space directions in an unlabeled search.Source: §7 · p. 8

8 Discussion and limitations

The discussion separates information in the observations from the costs of subspace estimation and fourth-moment direction search. It qualifies the scaling comparison as sufficient-bound motivated and limited in scope, with key lower-bound and functional-form questions unresolved.

8 Discussion and limitations

  • Intrinsic information, useful-subspace estimation, and fourth-moment direction search are distinct quantities with different statistical costs.Known-subspace restriction can remove a high-dimensional failure without changing the population target on the negative-kurtosis branch.Source: §8 · p. 8
  • The ς^-8 dependence comes from a sufficient-bound analysis, not an information-theoretic lower bound for the full inference problem.A two-point argument yields ς^-4, and the analysis does not establish a matching ς^-8 lower bound.Source: §8 · p. 8
  • A lower bound specific to empirical fourth-moment minimizers remains open and would need to control contrast fluctuations over the searched set.Search-dimension dependence enters through that supremum.Source: §8 · p. 8
  • The theory and scaling comparison cover two-component mixtures with the discriminant on a single left singular direction, not M > 2.Unequal weights are covered by the population calculation; extending to multiple components requires a different estimand and subspace-recovery analysis.Source: §8 · p. 8
  • The functional form of search-dimension dependence and the exponent for spurious-minimum depth remain unresolved; theory concerns exact global minimizers while experiments use multi-start gradient descent.Restart experiments support that reported thresholds are not optimizer artifacts but do not establish global optimization guarantees.Source: §8 · p. 8
  • The comparison concerns direction recovery, but the scaling largely disappears when thresholds use downstream excess error.Prediction error is less sensitive to weak-signal direction differences.Source: §8 · p. 8

AI Use Statement

The authors report using generative AI tools during manuscript preparation and state that they reviewed and verified the assisted material.

AI Use Statement

  • Generative AI supported conceptual discussion, proof checking, literature search, drafting, editing, and organization during manuscript preparation.The authors reviewed and verified all AI-assisted material used in the manuscript.Source: AI Use Statement · p. 8
  • Generative AI was not used to generate experimental data or determine the reported experimental results.Source: AI Use Statement · p. 8
  • The authors take responsibility for the final content, including mathematical statements, experimental results, citations, and interpretations.The statement says generative AI systems are not authors of the work.Source: AI Use Statement · p. 8

Subspace Restriction in Unlabeled Projection Pursuit: A Scaling Law and Where It Fails — Supplementary Material

The supplementary material develops concentration and geometry results for searching within a subspace, explaining why the Gaussian complement can distort unrestricted kurtosis search and when restriction preserves the population optimum.

B.4 Theorem 1 (uniform concentration)

  • Uniform concentration is formulated over a searched subspace, with its dimension entering the bound through L_n = d_search log(3n) + log(18/δ).The truncated-criterion argument requires n ≥ C′σ^8(log n)^3L; the stated powers and logarithmic factors are not optimized.Source: A Statements moved from the main text · p. 10, §B.4 · pp. 12–13

B.1 Lemma 1 (noise-only complement)

  • The signal and orthogonal-complement projections of the Gaussian noise are independent, leaving the complement as noise-only Gaussian variation.The argument uses the signal lying in R and zero cross-covariance between orthogonal projections of Gaussian noise.Source: §B.1 · p. 11

B.2 Lemma 2 (exact dilution)

  • When the population kurtosis is negative, adding an independent Gaussian complement reduces its magnitude, and every population minimizer in the plane remains in the signal subspace.For positive kurtosis, the minimum instead occurs in the complement, where the projection contains no regime information.Source: §B.2 · p. 11

B.3 Lemma 3 (variance floor and uniform sub-Gaussianity)

  • The moment bounds depend on operator and latent-distribution quantities, not ambient dimension; in the laboratory setting, the bound is invariant across the p and r sweep.The variance floor supports uniform control over directions.Source: §B.3 · p. 12

B.5 Lemma 5 (uniform non-clipping)

  • The untruncated empirical kurtosis equals the truncated criterion throughout a subspace when every projected observation stays within the clipping threshold.The condition is expressed using the norm of the projected, sample-centered observations.Source: §B.5 · p. 13

B.6 Theorem 2 (the untruncated criterion)

This section extends uniform concentration to the untruncated criterion, derives angular guarantees under global quadratic growth, and analyzes exact and approximate spurious minima in the Gaussian complement.

B.6 Theorem 2 (the untruncated criterion)

  • For the untruncated criterion, the sufficient sample size for an angular guarantee becomes quadratic in d_search, and removing truncation costs a factor d_search.The second-order term L^2/n cannot generally be replaced by one linear in L; a single observation can drive large uniform deviation.Source: §B.6 · p. 14

B.7 Lemma 4 (global quadratic growth)

  • Under the stated two-component, equal-spherical-covariance model, the population kurtosis has a unique minimizer and global quadratic growth around it.The minimizing direction is ±e_Ge_1; the result is specialized to the balanced two-component setting.Source: §B.7 · p. 15

B.8 Corollary 1

  • A uniform two-sided deviation bound converts objective error into an angular guarantee for the empirical minimizer.The corollary requires the deviation η to be at most |κ⋆| sin^2 θ_0/V when V ≥ 2.Source: §B.8 · p. 15

B.9 Proposition 1 (exact degeneracy)

  • With even n and p − r ≥ n − 1, an unrestricted direction in the Gaussian complement can attain empirical kurtosis −2, below the population minimizer almost surely.The corresponding population complement kurtosis is zero; for r ≤ n − 2, no direction in R attains that empirical floor.Source: §B.9 · p. 16

B.11 Remark 1 (necessity of truncation)

  • The untruncated criterion can also be dominated by a direction aligned with one large observation, producing uniform deviation at least c m^2/n − 3 with high probability for m ≤ n.The remark identifies this as a single-observation failure as p grows.Source: §B.11 · p. 17

B.12 Scope and limitations of the theoretical results

  • The main failure result concerns minimum-kurtosis search over the full sphere; global-growth angular guarantees require balanced weights and equal spherical component covariances.Several concentration and degeneracy results hold for general mixtures, but for M > 2 the minimum-kurtosis direction need not be unique, and all reported measurements use M = 2.Source: §B.12 · p. 17
  • The theory gives sufficiency for n ≳ d_search polylog and establishes failure for even n ≤ p − r + 1; beyond n = p, the depth law is measured.For the truncated criterion, the stated spurious depth is ≍ log(m ∧ n^a)/n under m ≤ n^2.Source: §B.12 · p. 17

C Numerical checks

The numerical checks test the concentration bounds, lower-bound constructions, and sensitivity of the measured search thresholds. They support dimension-dependent differences between unrestricted and restricted searches while showing that some scaling details remain unresolved.

C Numerical checks

  • The global-growth inequality holds across all 54 tested configurations, with its smallest slack −3.3e−16 at d = 0, where equality holds.Source: C Numerical checks · p. 17
  • The fitted decay in n is −0.71 [−0.78, −0.63] over the full sphere and −0.50 [−0.52, −0.48] over the signal subspace; at fixed r, p-dependence is +1.03 [+0.85, +1.19] versus −0.02 [−0.14, +0.10].The restricted-set decay matches the −1/2 leading-term exponent, and its measured deviation does not depend on p at fixed r.Source: C Numerical checks · p. 17, Figure 3 · p. 18
  • On the restricted search grid, the untruncated deviation decays with n at −0.50 [−0.52, −0.48] and has p-dependence −0.02 [−0.14, +0.10].The no-clipping condition held throughout the grid, so truncated and untruncated objectives were pointwise identical there.Source: C Numerical checks · p. 17

D Numerical checks for the lower bounds

  • Across the saturation-free depth measurements, the fitted m-dependence was not identified: replacing log m with log log m gave R2 = 0.9569 versus 0.9589.The floor-aware fit used D = 6.36 m/n, with C ∈ [6.23, 6.49].Source: D Numerical checks for the lower bounds · p. 18
  • Below the lower-boundary estimate, certificates occurred in 100% of 13 cells; within a factor of two above it, they occurred in 81%, and beyond that in 0%.The constructed complement witness attained empirical kurtosis −2 to 4.4e−16 across 16 configurations in the stated even-n regime.Source: D Numerical checks for the lower bounds · p. 18

E Experimental details; F Information used by each method

  • The laboratory configurations preserve population Bayes error and |κ⋆| across p and r; the operator-restricted comparison uses the known forward-model subspace, while true generative parameters are reserved for the Bayes oracle.Methods are evaluated on held-out data, and test labels are withheld during fitting and matching.Source: E Experimental details · p. 19, F Information used by each method · p. 19

G Optimization budget; I Algorithm

  • Increasing the restart budget 16-fold produced identical discovery thresholds and no systematic angular-error change; the reported kurtosis search uses five random starts and 250 iterations.The algorithm compares unrestricted, operator-restricted, and empirical-covariance PCA-restricted search spaces.Source: G Optimization budget · p. 20, I Algorithm · p. 20

L Extended sweeps

  • Threshold exponents remain similar across angular criteria, but crossings shift; anisotropy and calibration prevent a transportable crossing constant, and the tested range does not identify the search-dimension dependence.The fixed-angle thresholds have fitted exponents 0.142, 0.150, and 0.145, while crossings move from 1.20 to 0.60 to 0.30; calibrated criteria differ fivefold between two configurations.Source: L Extended sweeps · p. 21
Loading 2609.37917v1…