PaperlayerView exact revision on arXiv

Source-linked AI summary

When Features Become Instances: Inverted Contrastive Learning for Unsupervised Feature Selection

Utsab Ghosh, Roshni Chakraborty

arXiv:2609.00782v1cs.AIcs.CL

TL;DR

UFS must identify informative features without labels, but existing methods rely on indirect structural or reconstruction-based proxies. ICLFS instead learns feature-wise contrastive representations, uses projector-space norms for saliency, and refines rankings with redundancy-aware correction. Across 12 benchmarks, it achieves the best clustering accuracy on 10 datasets and remains competitive on the other two.

  • Problem

    Unsupervised feature selection lacks class labels, making feature utility difficult to define and requiring indirect structural, reconstruction, or surrogate objectives.

  • Method

    ICLFS inverts the data matrix, contrasts multiple masked positive views with a shuffled negative view, ranks features by projector-space embedding norms, and applies Laplacian-Gated Ranking Correction.

  • Results

    10 of 12 benchmark datasets achieve the best clustering accuracy for ICLFS against classical and neural baselines under the standard clustering-based UFS evaluation protocol.

  • Takeaways & Limitations

    Feature-wise contrastive representation consistency provides a strong and broadly effective alternative to neighborhood-, cluster-, and reconstruction-based UFS formulations.

  • Takeaways & Limitations

    The current framework is ranking-based; future work will examine alternative subset criteria, task-adaptive strategies, and hybrid selection mechanisms.

Abstract

from arXiv · show

Unsupervised feature selection seeks a compact subset of informative features without access to class labels, making feature utility difficult to define. Existing UFS methods therefore rely on indirect structural criteria, such as similarity preservation, locality, sparsity, cluster geometry, or reconstruction quality. In this paper, we instead study UFS through representation consistency and propose Inverted Contrastive Learning for Unsupervised Feature Selection (ICLFS), a feature-wise contrastive framework that reformulates UFS as a representation learning problem over features rather than samples. ICLFS first inverts the data matrix so that each feature is represented by its sample-profile vector, then constructs multiple masked positive views together with a shuffled negative view, and learns projector-space representations that remain consistent across these structured perturbations under an InfoNCE-based objective. Motivated by recent findings that cosine-based and InfoNCE-based training affect embedding norms, we use projector-space embedding magnitude as the saliency signal for ranking features. The resulting norm-based ranking is subsequently refined through Laplacian-Gated Ranking Correction, which suppresses locally redundant candidates while preserving salient ones. Extensive experiments on 12 benchmark datasets show that ICLFS achieves the best clustering accuracy on 10 datasets against both classical and neural baselines under the standard clustering-based UFS evaluation protocol, while remaining competitive on the other two. These results show that feature-wise contrastive representation consistency provides a strong and effective alternative to neighborhood, cluster, and reconstruction-based UFS formulations.

1 Introduction

Unsupervised feature selection must identify informative features without labels, so existing methods infer utility from indirect structural criteria. ICLFS instead learns feature-wise representation consistency and ranks features by embedding magnitude, achieving the best clustering accuracy on 10 of 12 benchmark datasets.

  • High-dimensional unlabeled data contain noisy, redundant, and weakly informative features that can impair clustering and complicate analysis.
  • Existing UFS methods infer feature usefulness through similarity preservation, locality, sparsity, cluster geometry, or other intrinsic structural criteria.
  • ICLFS reformulates UFS as feature-wise contrastive representation learning by inverting the data matrix and treating features rather than samples as learning instances.
  • Projector-space embedding norms provide the saliency signal, while Laplacian-Gated Ranking Correction suppresses locally redundant candidates and preserves salient ones.
  • 10 of 12 benchmark datasets achieve the best clustering accuracy for ICLFS against classical and neural baselines under the standard clustering-based UFS protocol.On the remaining datasets, RELATHE remains competitive and LS-CAE is stronger on ALLAML.

2 Related Work

Related UFS methods commonly rely on graph, spectral, discriminative, reconstruction, or compressive objectives, while contrastive learning has largely remained sample-centered. ICLFS addresses this gap by modeling feature-level relational consistency and combining learned representations with explicit redundancy-aware refinement.

  • Classical UFS methods use graph and spectral structure, including local-neighborhood preservation and global similarity-graph consistency.
  • Other classical approaches jointly learn feature importance with latent discriminative or clustering structure, often using sparsity or spectral regularization.
  • Deep UFS methods frequently optimize reconstruction, surrogate prediction, or differentiable selection objectives rather than feature-level relational consistency.
  • Most contrastive methods treat samples as instances and do not directly address feature-centric saliency in unsupervised feature selection.
  • ICLFS combines feature-wise contrastive learning, structured multi-view perturbations, inter-feature interaction modeling, and post hoc redundancy-aware filtering.

3 Proposed Approach

The proposed approach formalizes UFS on an unlabeled sample-by-feature matrix as selecting a fixed-size subset that preserves informative data structure. Because labels are unavailable, selection relies on structure inferred from dependencies, similarities, neighborhoods, or learned representation consistency.

  • UFS starts from an unlabeled matrix whose rows represent samples and columns represent measured features.
  • The objective is to select exactly k feature indices and form a reduced representation from the corresponding columns.
  • The reduced matrix should retain the most informative structure of the original data according to an unsupervised selection criterion.
  • Without class labels, relevant structure may be inferred from feature dependencies, sample similarity, local neighborhoods, or learned representation consistency.

3.2 Overview

ICLFS is organized as a four-component pipeline that converts the data into feature-wise views, learns contrastive embeddings, and refines norm-based feature rankings. The stages progress from inversion and view construction through neural optimization to final redundancy-aware selection.

  • Component I: Preprocessing and Feature-wise Inversion: Component I normalizes the input and inverts the data matrix so each feature is represented across the full sample set.
  • Component II: Contrastive Task: Component II creates multiple masked positive feature maps and a shuffled negative map to define the unsupervised contrastive task.
  • Component III: Neural Model: Component III processes the anchor and generated maps with self-attention, encoding, projection, and optimization to learn feature-wise embeddings.
  • Component IV: Selector: Component IV ranks features by projector-space norm and applies Laplacian-Gated Ranking Correction to obtain the final selected subset.

3.3 Component I: Preprocessing and Feature-wise Inversion

Component I standardizes the sample-wise matrix and transposes it so each feature becomes a learning instance represented by its profile across all samples.

  • Preprocessing: ICLFS standardizes X across samples to give each feature zero mean and unit variance before representation learning.This reduces scale imbalance and prevents larger raw magnitudes from dominating subsequent learning.
  • Feature-wise inversion: The method transposes the standardized matrix into X⊤, whose rows correspond to features rather than samples.The inverted map has shape X⊤∈R^d×n.
  • Feature-wise inversion: Each feature is represented by an n-dimensional sample-profile vector that serves as one feature instance.This representation allows the model to operate on feature profiles rather than sample instances.
  • Feature-wise inversion: The inverted feature map lets all features be processed jointly so subsequent interactions can capture dependencies among them.This representation supports learning a feature-scoring and ranking mechanism.

3.4 Component II: Contrastive Task

Component II creates multiple views for each feature: masked positives preserve feature identity under partial observation, while a shuffled negative disrupts sample-aligned structure for contrastive training.

  • Contrastive objective: The view set is designed to identify features whose representations remain stable under structured perturbations without labels or predefined targets.This stability motivates using projector-space embedding norms for later saliency ranking.
  • View construction: Four masked feature maps provide positive views of each feature, and one shuffled feature map provides an explicit negative view.These views are generated from the inverted feature map and stacked for training.
  • Contrastive design: The shuffled negative is more comparable to the anchor than an unrelated feature because it changes structure without changing the underlying feature.The contrast therefore separates structure-preserving from structure-disrupting transformations of the same feature.
  • View construction: The positive views preserve each underlying feature’s identity under partial observation, whereas shuffling preserves marginal values but disrupts sample alignment.The exact masking ratios and overlap settings are specified separately in Section 4.3.
  • Contrastive design: For each feature, the anchor is paired with four positive views and one negative view, while other feature instances act as implicit negatives.The corresponding row-wise views are collected in the generated feature-map tensor M.

3.5 Component III: Neural Model

Component III uses self-attention and row-wise nonlinear encoding to produce feature embeddings, then applies a residual projector whose normalized outputs support InfoNCE while unnormalized magnitudes support ranking.

  • Encoder: The encoder first applies self-attention across feature rows so each feature can attend to other feature instances.Each row contains the feature’s sample-profile vector.
  • Encoder: A two-stage row-wise multilayer perceptron transforms the attention-mixed map into latent embeddings for all features.The first stage uses batch normalization, LeakyReLU, and dropout before the final encoder embedding matrix.
  • Projector: The projector’s normalized outputs define the InfoNCE space, while its unnormalized embedding magnitudes provide the feature-ranking signal.This separates similarity optimization from magnitude-based saliency assessment.
  • Projector: The projector maps encoder embeddings into a contrastive space through residual multilayer perceptron blocks followed by a final linear projection.The projector output is Z(V)∈R^d×d_z, with one embedding row per feature.
  • Model scaling: The architecture is scaled according to dataset sample size because sample profiles determine effective input dimensionality and available training signal.The paper uses two fixed sample-aware decorrelation regimes based on observed differences between smaller- and larger-sample datasets.
  • Optimization: InfoNCE aligns views of the same feature and separates them from mismatched views, while contrastive training affects the raw embedding magnitudes used for ranking.The framework therefore uses norm-sensitive contrastive behavior as part of feature saliency estimation.

3.6 Component IV: Selector

The selector ranks features by the magnitude of raw anchor-side projector embeddings, then applies a cardinality-specific Laplacian correction to reduce locally poor candidates without retraining.

  • Feature Selection Criterion: The anchor projector output is used because its feature rows correspond to the original transposed feature representation.
  • Feature Selection Criterion: Projector-space embedding norms provide the initial global saliency ranking for all features.The ranking is computed from the raw anchor-side projector output and sorted in descending norm order.
  • Laplacian-Gated Ranking Correction: LGRC corrects the norm ranking because individual embedding magnitudes do not explicitly measure local sample-neighborhood preservation.The correction is applied post hoc to top-ranked candidates without retraining the neural model.
  • Laplacian-Gated Ranking Correction: For each target cardinality, LGRC forms a candidate pool from the highest-ranked features and draws replacements from the remaining reserve list.The pool size is controlled by a fixed multiplier α ≥ 1.
  • Laplacian-Gated Ranking Correction: Features with Laplacian scores above the fixed percentile threshold are iteratively replaced by reserve features until the pool satisfies the gate or the reserve is exhausted.Lower Laplacian scores indicate stronger preservation of local sample-neighborhood structure.
  • Laplacian-Gated Ranking Correction: After correction, the retained pool is restored to descending projector-norm order, and its top k features form the final subset.

4 Experiments

The experiments evaluate ICLFS and baselines on a diverse 12-dataset benchmark using downstream clustering accuracy. The protocol combines repeated k-means runs with dataset-level selection of the best evaluated subset size.

  • Evaluation Protocol: Feature-selection quality is assessed through downstream clustering performance using k-means on the selected features.Clustering accuracy is computed after optimally matching cluster assignments to ground-truth labels with the Hungarian algorithm.
  • Evaluation Protocol: For each feature cardinality, clustering accuracy is averaged across 20 k-means runs, and each dataset reports the highest mean among six evaluated subset sizes.
  • Datasets: The benchmark contains 12 datasets spanning image, biomedical, text, and mass-spectrometry domains, with 60–7000 samples and 1024–10,000 features.It includes binary-class and multi-class settings across heterogeneous structural regimes.
  • Implementation Details: Training uses PyTorch with Adam for 100 epochs, learning rate 10^-3, weight decay 10^-4, and temperature τ = 0.05.Masked views use keep ratios of 0.90, 0.60, and 0.50, with 0.10 overlap for the complementary pair.
  • Implementation Details: ICLFS uses two sample-size-aware regimes that vary encoder, projector, and decorrelation settings across lower- and higher-sample datasets.The lower-sample regime covers seven datasets, while the higher-sample regime covers five.
  • Implementation Details: The post hoc correction uses α = 1.5, q = 0.75, and a 3-nearest-neighbor cosine-distance affinity graph with heat-kernel weighting.The full inverted data matrix is processed jointly, with view terms accumulated before each optimizer update.
  • Baselines: The evaluation compares ICLFS with LS, MCFS, NDFS, SPEC, CAE, and LS-CAE under the same feature-selection protocol.

5 Results and Analysis

ICLFS is evaluated against UFS baselines, through component ablations, and under alternative feature-scoring criteria. It performs best on 10 of 12 datasets, while core representation-learning components contribute more than the final ranking refinement and projector-norm ranking is strongest overall.

  • Comparison with Existing Research Works: ICLFS achieves the highest clustering accuracy on 10 of 12 benchmark datasets, tied for second on RELATHE and lower on ALLAML.On RELATHE, ICLFS reaches 55.46%, matching LS-CAE; on ALLAML, it reaches 66.60% versus 71.11% for LS-CAE.
  • Comparison with Existing Research Works: ICLFS reaches 83.97% on PROSTATE, 52.33% on NCI9, and 67.66% on LUNG, exceeding the strongest competing baselines on each dataset.The corresponding baselines are CAE at 60.78%, LS at 41.08%, and MCFS at 59.26%.
  • Ablation Study: Removing decorrelation, attention, or structured masking produces substantially larger losses than removing LGRC alone, with the largest reported drop reaching 26.91% on PROSTATE.The ablations support structured multi-view contrastive learning, attention-based feature interaction, and projector-space decorrelation as important components.
  • Alternate Selection Criterion: Projector-norm scoring is strongest overall, winning on ARCENE, PCMAC, WARPPIE10P, and COIL20, while Weighted Sum is best on PROSTATE at 84.31% versus 83.97% for z-norm.On WARPPIE10P, z-norm reaches 44.14% versus 43.36% for Weighted Sum and 41.38% for Harmonic-hz; on COIL20, it reaches 68.68% versus 66.91% and 66.77%.

6 Conclusion

ICLFS reformulates unsupervised feature selection as feature-wise representation learning, using projector-space embedding magnitudes for ranking and Laplacian-Gated Ranking Correction to reduce redundancy. Across 12 benchmark datasets, it achieves the best clustering accuracy on 10 and remains competitive on the other two, while broader selection mechanisms remain future work.

  • 6 Conclusion: ICLFS treats features rather than samples as learning instances in a contrastive representation-learning framework.It inverts the data matrix, uses masked positive views and a shuffled negative view, and optimizes feature-wise representations with InfoNCE.
  • 6 Conclusion: Projector-space embedding magnitudes provide the saliency signal for feature ranking, followed by decorrelation regularization and Laplacian-Gated Ranking Correction.The correction stage is intended to improve feature separation and reduce redundancy.
  • 6 Conclusion: On 12 benchmark datasets, ICLFS achieves the best clustering accuracy on 10 datasets and remains competitive on the remaining two.The comparison includes both classical and neural baselines under the standard clustering-based UFS evaluation protocol.
  • 6 Conclusion: Future work will extend the framework beyond ranking-based selection toward alternative subset criteria, task-adaptive strategies, and hybrid mechanisms.These extensions would build directly on learned feature representations.

Declaration of Competing Interest

The authors report no relevant financial or non-financial competing interests.

  • Declaration of Competing Interest: The authors disclose no relevant financial or non-financial interests.
Loading 2609.00782v1…