PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Select Features with Manifold Deep Unfolding PCA

Jingjing Liu, Pei Tai, Xianchao Xiu, Wanquan Liu

arXiv:2610.19436v1cs.CVcs.LG

TL;DR

Unsupervised feature selection must identify informative features without labels, but existing approaches face manual parameter tuning or high training-data demands, while deep unfolding and manifold preservation remain underexplored in UFS. The paper combines sparse PCA models, ADMM-based deep unfolding, and manifold optimization; its methods generally outperform competitors in clustering.

  • Problem

    UFS seeks informative features without labels, while combining sparsity, adaptive parameter learning, and manifold preservation in deep unfolding remains challenging.

    Source: §I · pp. 1–2

  • Method

    MDUFS-Net unfolds ADMM for an ℓ2,0-regularized PCA model into a trainable network with manifold optimization; MDUFS-Net+ adds ℓ0 regularization.

    Source: §I · p. 2, §V · p. 11

  • Results

    Across real-world clustering, both methods generally outperform competitors; MDUFS-Net+ raises average ACC by 1.91% over MDUFS-Net, and their average NMI gains over the best comparison method are 3.64% and 5.19%, respectively.

    Source: C. Real-world Results · pp. 8–9

  • Takeaways & Limitations

    The methods preserve advantages of sparse modeling and deep learning while achieving effective feature selection measured by ACC and NMI.

    Source: §V · p. 11

Abstract

from arXiv · show

Unsupervised feature selection (UFS) is of great significance in image processing and pattern recognition. Nevertheless, existing model-based methods often rely on manually tuned hyperparameters, while deep learning-based methods generally require massive training data. To address these issues, this paper proposes an efficient manifold deep unfolding feature selection network (MDUFS-Net). Technically, we first incorporate the $\ell_{2,0}$-norm into the classic principal component analysis (PCA) framework, then unfold the alternating direction method of multipliers (ADMM)-based iterative algorithm into an end-to-end trainable network, thereby enabling automatic parameter learning. Moreover, a manifold optimization strategy is developed to improve the numerical robustness. On this basis, we further extend MDUFS-Net into an enhanced variant called MDUFS-Net+ by introducing the $\ell_{0}$-norm for more discriminative feature representations. Extensive experiments validate that our proposed MDUFS-Net and MDUFS-Net+ outperform state-of-the-art methods in the UFS task, with promising performance in clustering. Our code is available at https://github.com/xianchaoxiu/MDUFS.

I. INTRODUCTION

Unsupervised feature selection seeks representative features without labels, but sparse PCA methods still rely on manually chosen parameters, while deep UFS networks can lack an explicit connection to mathematical models. This paper combines sparse PCA with ADMM deep unfolding and manifold optimization, then adds element-wise sparsity in MDUFS-Net+.

I. INTRODUCTION

  • High-dimensional data can contain redundant features that raise storage and computation costs or degrade downstream learning, motivating label-free selection of informative subsets.Source: §I · p. 1
  • Sparse PCA methods impose sparsity to select features, but their regularization or penalty parameters are typically set manually, limiting generalization capability.Prior approaches include ℓ2,p- and ℓ2,0-based models.Source: §I · p. 1
  • Deep unfolding maps iterative optimization into stage-wise trainable networks, preserving model interpretability while enabling adaptive parameter learning.The introduction identifies this as a way to bridge the gap between deep networks and mathematical models.Source: §I · p. 1
  • MDUFS-Net unfolds ADMM for an ℓ2,0-regularized PCA model, learns parameters end to end, and uses manifold optimization to maintain projection-matrix orthogonality and enhance stability.Source: §I · p. 2
  • MDUFS-Net+ adds element-wise sparsity to row-wise sparsity, with the stated aim of improving feature-selection performance in real-world scenarios.Source: §I · p. 2

A. Sparse PCA

  • Sparse PCA uses a projection matrix whose row sparsity selects features, while its reconstruction term preserves data information.The cited SPCAFS formulation also discusses ℓ2,p regularization and the need to tune its regularization and optimization parameters.Source: A. Sparse PCA · p. 2

B. Deep Unfolding

Deep unfolding maps iterative optimization into stage-wise trainable networks, retaining model interpretability while enabling adaptive parameter learning. This section applies that framework to PCA-based unsupervised feature selection through MDUFS-Net.

B. Deep Unfolding

  • Deep unfolding had not been investigated for UFS, particularly PCA models with non-convex sparsity regularization.The paper positions its approach as bringing this model-driven framework to that setting.Source: B. Deep Unfolding · p. 2

III. THE PROPOSED METHODS

  • The proposed-method section presents mathematical models and deep unfolding networks for MDUFS-Net and MDUFS-Net+.The network frameworks are illustrated in Fig. 2.Source: §III · p. 3, Figure 2 · p. 3

A. MDUFS-Net

  • MDUFS-Net unfolds ADMM updates into network stages with learnable λk and αk, replacing parameters typically predefined in iterative optimization.The method addresses the sensitivity to parameter settings and lack of stability described for the iterative model.Source: A. MDUFS-Net · p. 3

1) Update Xk+1

Each unfolding stage updates the model variables while preserving the orthogonality constraint on X. The method uses a manifold-based update for X and a row-wise thresholding operation for Y.

1) Update Xk+1

  • The X update uses a manifold projection-based strategy because the constraint X⊤X = I prevents direct Euclidean updates from preserving feasibility.QR factorization is used to obtain an orthogonal factor when the updated estimate does not exactly satisfy the constraint.Source: 1) Update Xk+1 · pp. 3–4
  • Algorithm 1 iterates through the unfolding stages and outputs the final X and Y variables.The algorithm initializes the data, stage-wise parameters, and starting variables before its repeated updates.Source: 1) Update Xk+1 · p. 4
  • A manifold gradient update embeds orthogonality preservation into each unfolding stage, using a constraint-compatible search direction and a learnable step size.The Euclidean gradient is projected to obtain the search direction, followed by a stage-specific step size and QR-based retraction.Source: 1) Update Xk+1 · pp. 3–4
  • After X is updated, Y is obtained through a closed-form row-wise hard-thresholding operation.HardNet implements the row-wise operation at each unfolding stage, with 2λk/αk controlling the sparsity threshold.Source: 1) Update Xk+1 · p. 4

3) Update Λk+1:

The MDUFS-Net stage updates the Lagrange multiplier after calculating X and Y. A linear combination module implements this dual-variable update.

3) Update Λk+1:

  • After Xk+1 and Yk+1 are calculated, the stage updates the Lagrange multiplier Λ.This is the final variable update described in the MDUFS-Net unfolding procedure.Source: 3) Update Λk+1 · p. 4
  • The multiplier update is implemented by a linear combination module using the stage’s inputs.Linear(·) represents the linear combination module for updating the dual variable at the k-th stage.Source: 3) Update Λk+1 · p. 4
  • Algorithm 1 summarizes the deep unfolding framework for MDUFS-Net.Its procedure includes the multiplier update after the X and Y updates.Source: 3) Update Λk+1 · p. 4

B. MDUFS-Net+

MDUFS-Net+ extends MDUFS-Net with an element-wise ℓ0 regularizer to address redundant elements in the model solution. It retains the manifold gradient update and adds a variable update based on element-wise hard thresholding.

B. MDUFS-Net+

  • MDUFS-Net+ adds an ℓ0-norm regularizer because the MDUFS-Net solution may contain redundant elements.The ℓ0-norm counts the number of nonzero elements in a matrix.Source: B. MDUFS-Net+ · p. 4
  • The Y update follows the same procedure as in MDUFS-Net, while the Λ update also remains unchanged.The MDUFS-Net+ unfolding procedure additionally updates Π.Source: B. MDUFS-Net+ · p. 4
  • The extended X-subproblem adds a quadratic penalty associated with the new variable Z, modifying the Euclidean gradient and projected search direction.The orthogonality-preserving manifold gradient update and QR-based retraction otherwise follow the MDUFS-Net procedure.Source: B. MDUFS-Net+ · pp. 4–5
  • The Z update uses element-wise hard thresholding to solve its subproblem.The thresholding operation is formulated within the deep unfolding scheme.Source: B. MDUFS-Net+ · p. 5

C. Loss Function

The loss uses differentiable surrogate functions because the original ℓ2,0- and ℓ0-norm terms cannot be optimized directly by gradient-based backpropagation. A smoothing parameter for the ℓ2,0 surrogate is adjusted during training.

C. Loss Function

  • The original ℓ2,0- and ℓ0-norm terms cannot be optimized directly by gradient-based backpropagation, so differentiable surrogate functions are used.Source: C. Loss Function · p. 5
  • For the ℓ2,0-norm surrogate, σ controls its approximation to the original norm and is gradually adjusted during training.Source: C. Loss Function · p. 5
  • MDUFS-Net+ also uses a differentiable ℓ0-norm surrogate and has its own loss function.Source: C. Loss Function · p. 5

IV. NUMERICAL STUDIES

The numerical studies compare MDUFS-Net and MDUFS-Net+ with representative unsupervised feature-selection methods. They include synthetic and real-world results, ablations, and further discussion.

IV. NUMERICAL STUDIES

  • The evaluation compares both proposed networks with LapScore, SOGFS, RNE, UDFS, SPCAFS, FSPCA, FEN-PCAFS, and BSUFS.Implementations are obtained from the AutoUFSTool toolbox or the respective authors.Source: §IV · p. 5
  • The synthetic comparisons include visual feature-selection results for Dartboard1 and Diamond9.Source: Figure 3 · p. 6, Figure 4 · p. 6
  • The study covers dataset descriptions and parameter settings, synthetic and real-world results, ablations, and further discussion.Source: §IV · p. 6

1) Dataset Descriptions:

The experiments use synthetic and real-world datasets spanning several application domains. Methods are evaluated with ACC and NMI across feature counts, with repeated k-means runs used to report mean and standard deviation.

1) Dataset Descriptions:

  • Two synthetic datasets and eight real-world datasets are used, with real-world data spanning speech, deep learning, biological information, image processing, and time-series analysis.Source: 1) Dataset Descriptions · p. 6

2) Parameter Settings:

  • The proposed networks use the same architecture and training strategy across datasets, with 15 unfolding stages and 100 Adam epochs at an initial learning rate of 1 × 10^-4.Learnable parameters are optimized by gradient-based training, and sparse-regularization lower bounds depend on dataset feature dimensionality.Source: 2) Parameter Settings · pp. 6–7

3) Evaluation Metrics:

  • ACC and NMI are evaluated with 10–100 selected features in steps of 10; k-means is repeated 50 times, and mean and standard deviation are reported.Source: 3) Evaluation Metrics · p. 7

B. Synthetic Results

On real-world datasets, the proposed methods generally outperform competitors, while synthetic results and ablations examine feature selection and the roles of sparsity and manifold components.

B. Synthetic Results

  • On Diamond9, only MDUFS-Net and MDUFS-Net+ identify the two most discriminative features; on Dartboard1, five methods identify appropriate features.The authors report reliable feature-selection results for both proposed methods on the synthetic datasets.Source: B. Synthetic Results · pp. 7–8

C. Real-world Results

  • Compared with MDUFS-Net, MDUFS-Net+ raises average ACC by 1.91% and lowers average standard deviation from 1.56% to 1.11%.The methods generally maintain higher ACC curves than competing methods; MDUFS-Net+ has the highest average ACC in most cases, followed by MDUFS-Net.Source: C. Real-world Results · pp. 8–9
  • Against the best-performing comparison method, MDUFS-Net and MDUFS-Net+ improve average NMI by 3.64% and 5.19%, respectively.MDUFS-Net has relatively lower NMI on MNIST and HAR; the authors attribute this to strong feature correlations and overlapping class distributions.Source: C. Real-world Results · p. 9

1) Effect of Sparsity Regularization Terms:

  • Using both ℓ2,0- and ℓ0-norm regularization achieves the best ACC across datasets; COIL20 improves from 44.42% to 65.23%, and ISOLET from 40.20% to 64.16%.The authors describe row sparsity and element-wise sparsity as complementary.Source: 1) Effect of Sparsity Regularization… · p. 9

2) Effect of Manifold Optimization:

3) Effect of Deep Unfolding:

The paper evaluates the effect of deep unfolding and stage-wise parameter learning, then discusses stability, unfolding depth, and runtime. Its conclusion summarizes MDUFS-Net and MDUFS-Net+ and identifies future directions.

3) Effect of Deep Unfolding:

  • Deep unfolding improves both ACC and NMI across all five evaluated datasets; on COIL20, ACC rises from 60.81% to 65.23%.The comparison is conventional ADMM with fixed parameters versus MDUFS-Net+.Source: 3) Effect of Deep Unfolding · pp. 9–10

4) Effect of Parameter Learning:

  • Stage-wise independent parameter learning consistently outperforms parameter sharing, supporting flexible adaptation across unfolding stages.The comparison is reported in Table VII.Source: 4) Effect of Parameter Learning · p. 10

1) Model Stability:

  • Across 50 clustering runs, both proposed methods achieve better ACC with more compact distributions, and MDUFS-Net+ is especially stable on ISOLET and PIE.The paper attributes MDUFS-Net+’s stability improvement to ℓ0-norm-driven sparse feature selection reducing redundant-feature influence.Source: 1) Model Stability · p. 11

2) Unfolding Sensitivity:

  • Performance generally improves and then stabilizes as unfolding stages increase; 15 stages offer the best or comparable results on most datasets and are the default.Some datasets remain competitive with fewer stages, balancing performance and computational cost.Source: 2) Unfolding Sensitivity · p. 11

3) Runtime:

  • Runtime varies substantially by dataset, but MDUFS-Net and MDUFS-Net+ maintain acceptable computational costs across the tested datasets.The compared methods use different implementations and hardware: MATLAB on CPU versus Python with GPU acceleration.Source: 3) Runtime · p. 11

V. CONCLUSION

  • MDUFS-Net and MDUFS-Net+ combine sparse PCA, ADMM-based deep unfolding, and manifold optimization, achieving effective feature selection in ACC and NMI.The ℓ2,0-norm enables feature selection, while the ℓ0-norm suppresses redundant feature elements.Source: §V · p. 11

V. CONCLUSION:

  • Future work proposes extending the methods to spiking neural networks to improve computational efficiency and exploring large language models with UFS for complex datasets.The paper presents the latter direction as having potential to enhance feature-selection capabilities.Source: §V · p. 11

Related papers

Loading 2610.19436v1…