Source-linked AI summary
Infinite Latent Feature Selection: A Probabilistic Latent Graph-Based Ranking Approach
Giorgio Roffo, Simone Melzi, Umberto Castellani, Alessandro Vinciarelli
TL;DR
Feature selection must remain effective across heterogeneous datasets because learning performance depends heavily on feature choice. The paper introduces a probabilistic latent graph method that models relevancy and ranks features across graph paths representing possible subsets. Across diverse benchmarks, it reports strong robustness and the highest performance levels in challenging scenarios compared with the evaluated methods.
Problem
Feature selection methods need robustness across heterogeneous datasets because machine-learning performance depends heavily on the selected features.
Method
The method models feature relevancy as a latent variable in a PLSA-inspired process and ranks features by evaluating graph paths corresponding to feature subsets.
Results
The approach attained the highest performance levels across challenging scenarios and was reported as more robust than the comparison methods.
Takeaways & Limitations
The study supports probabilistic latent graph-based ranking as a robust feature-selection approach across diverse benchmark settings.
Takeaways & Limitations
The latent model assumes each feature contains two topics, Relevancy and Irrelevancy, and initializes their priors accordingly.
Abstract
from arXiv · showhide
Feature selection is playing an increasingly significant role with respect to many computer vision applications spanning from object recognition to visual object tracking. However, most of the recent solutions in feature selection are not robust across different and heterogeneous set of data. In this paper, we address this issue proposing a robust probabilistic latent graph-based feature selection algorithm that performs the ranking step while considering all the possible subsets of features, as paths on a graph, bypassing the combinatorial problem analytically. An appealing characteristic of the approach is that it aims to discover an abstraction behind low-level sensory data, that is, relevancy. Relevancy is modelled as a latent variable in a PLSA-inspired generative process that allows the investigation of the importance of a feature when injected into an arbitrary set of cues. The proposed method has been tested on ten diverse benchmarks, and compared against eleven state of the art feature selection methods. Results show that the proposed approach attains the highest performance levels across many different scenarios and difficulties, thereby confirming its strong robustness while setting a new state of the art in feature selection domain.
1. Introduction
Feature selection addresses the dependence of learning performance on feature choice and is important across diverse computer vision and other data settings. The paper proposes a probabilistic latent graph-based method that ranks features over paths representing possible subsets, then evaluates it broadly against established methods.
- Feature selection can improve prediction performance, reduce model cost, and expose regularities in data by selecting informative features.
- The proposed algorithm models feature relevancy as a latent variable and represents features as nodes in an affinity graph.
- The method ranks features by considering possible feature subsets as graph paths and using matrix power-series convergence to bypass combinatorial enumeration.
- Its pipeline quantizes raw feature distributions, learns graph weights with a PLSA variation and EM, and then applies infinite feature selection for ranking.
- The evaluation compares the method with 11 state-of-the-art selectors across 10 publicly available benchmarks spanning biological, handwritten-character, text, sentiment, and object-recognition data.
2. Related Work
Feature-selection methods are commonly divided into filters, wrappers, and embedded methods, with the proposed approach belonging to the filter category. Related filter strategies differ in how they estimate feature quality, including neighborhood discrimination, class separation, and mutual information.
- Feature-selection research addresses increasingly high-dimensional domains containing hundreds to tens of thousands of variables and relatively few training examples.
- Filters score intrinsic data properties independently of the classifier, whereas wrappers evaluate subsets with classifiers and embedded methods integrate selection into model training.
- ReliefF is an iterative randomized supervised filter that estimates feature quality from how nearby samples are differentiated.
- Fisher scores features independently using the ratio of inter-class separation to intra-class variance.
- Mutual Information selects features using the information shared between feature-value distributions and class membership.
3. Our Approach
The approach learns feature relationships through a PLSA-inspired latent model, then ranks features by aggregating evidence across paths in a weighted graph. Discriminative quantization converts feature values into tokens before graph weighting and infinite-path analysis.
- 3.2. From co-occurrences to graph weighting: The method builds an undirected feature graph whose edge weights are learned potential values representing pairwise feature relevance.Each adjacency-matrix element is formed from the estimated relevancy probabilities of the two connected features.
- 3.1. Discriminative Quantization process: Feature distributions are quantized into a smaller vocabulary of tokens using discriminative scores based on class means and standard deviations.The process uses T = 6 intervals, from poorly represented to well-represented samples.
- 3.2. From co-occurrences to graph weighting: A PLSA-inspired model represents each feature as a mixture of relevant and irrelevant latent topics over token co-occurrences.The model estimates P(t|z) and P(z|f) using maximum likelihood and the EM algorithm.
- 3.3. Probabilistic Infinite Feature Selection: Feature subsets are modeled as paths through graph nodes, including walks that may revisit nodes and edges.The cost of a path is defined by the joint probability of its participating nodes.
- 3.3. Probabilistic Infinite Feature Selection: Considering paths of every length is equivalent to considering feature subsets of any cardinality, and matrix power series aggregate them analytically.Because the infinite sum can diverge, a regularization factor weights paths by length and ensures convergence for appropriate choices.
4. Experiments and Results
Experiments evaluate ILFS across diverse datasets and feature-selection settings, comparing it with 11 state-of-the-art methods. The results emphasize strong ranking performance and robustness across heterogeneous scenarios, with statistical tests supporting the reported comparisons.
- Experimental design: The evaluation spans datasets with few training samples, many features, sparse or dense data, unbalanced classes, overlapping classes, and noisy samples.
- Experimental design: The experiments compare 11 state-of-the-art feature-selection methods using image-recognition and miscellaneous benchmark settings.
- Deep representation with pretraining: ILFS achieved the best mAP on VOC-2007 and remained among the top three methods on VOC-2012 when selecting 50% of the features.
- Miscellaneous datasets: ILFS outperformed all competitors at every tested feature cardinality in the miscellaneous-dataset experiment, while other methods varied substantially across datasets.
- Reliability and validity: Each Table 4 accuracy aggregates SVM results from 20 data splits and five feature-subset sizes, totaling 100 tests per method.
- Reliability and validity: Across ten experiments, ILFS achieved seven successes, with the probability of outperforming competitors more than seven times by chance reported as 4.82 · 10^-3.
5. Conclusion
The conclusion presents ILFS as a probabilistic feature-selection method that ranks features across all possible subsets while modeling relevancy through a PLSA-inspired process. It reports robust, statistically significant performance across challenging scenarios and identifies automatic subset selection and multi-interval validation as future directions.
- ILFS ranks features by considering all possible feature subsets, thereby bypassing the combinatorial problem.
- The method models feature relevancy with a PLSA-inspired process whose mixing weights P(z|f) weight a graph of features.
- The weighted graph ranks each feature according to its importance and the importance of its neighbors.
- ILFS reportedly outperforms the comparison methods in robustness and ranking quality to a statistically significant extent across challenging scenarios.
- Future work includes automatic subset selection through absorbing Markov chains and validation over multiple T intervals.