Source-linked AI summary
Semi-supervised Feature Analysis by Mining Correlations among Multiple Tasks
Xiaojun Chang, Yi Yang
TL;DR
High-dimensional feature selection is challenged by noisy or correlated features, scarce labels, and independent treatment of related tasks. The paper proposes SFMC, which combines semi-supervised manifold learning with multi-task and joint feature modeling. Across video classification, image annotation, human motion recognition, and 3D motion analysis, SFMC outperforms compared algorithms.
Problem
Feature selection must handle noisy or correlated high-dimensional features while existing methods rely on labeled data and often treat related tasks independently.
Method
SFMC combines semi-supervised feature selection, manifold learning, joint feature evaluation, and trace-norm sharing across multiple related tasks.
Results
SFMC outperforms the compared feature-selection algorithms across video classification, image annotation, human motion recognition, and 3D motion data analysis.
Takeaways & Limitations
The framework supports representative feature selection when each task has insufficient labeled training data.
Abstract
from arXiv · showhide
In this paper, we propose a novel semi-supervised feature selection framework by mining correlations among multiple tasks and apply it to different multimedia applications. Instead of independently computing the importance of features for each task, our algorithm leverages shared knowledge from multiple related tasks, thus, improving the performance of feature selection. Note that we build our algorithm on assumption that different tasks share common structures. The proposed algorithm selects features in a batch mode, by which the correlations between different features are taken into consideration. Besides, considering the fact that labeling a large amount of training data in real world is both time-consuming and tedious, we adopt manifold learning which exploits both labeled and unlabeled training data for feature space analysis. Since the objective function is non-smooth and difficult to solve, we propose an iterative algorithm with fast convergence. Extensive experiments on different applications demonstrate that our algorithm outperforms other state-of-the-art feature selection algorithms.
1 INTRODUCTION
The paper addresses limitations of high-dimensional feature selection by combining semi-supervised learning, multi-task knowledge, and joint feature-correlation modeling. SFMC uses labeled and unlabeled data, manifold learning, and a fast iterative solver for multimedia applications.
- Motivation: High-dimensional representations often contain noisy or correlated features that can degrade subsequent classification and clustering.Feature selection reduces noisy and redundant information while lowering representation dimensionality.
- Motivation: Manual labeling of large training sets is unrealistic, motivating feature selection that exploits both labeled and unlabeled data.Semi-supervised learning addresses the limited availability of labeled training samples.
- Motivation: Independent task-wise feature selection fails to mine correlations among multiple related tasks.The paper therefore incorporates multi-task learning into feature selection.
- Proposed approach: SFMC combines semi-supervised feature selection and multi-task learning while considering correlations between different features.The framework is intended to select representative features with insufficient labeled data per task.
- Application process: The approach represents videos as feature vectors, learns sparse coefficients using feature relationships and related-task knowledge, then classifies testing videos with selected features.The process is illustrated for video classification in Figure 1.
- Optimization: A fast iterative algorithm is proposed because the objective function is non-smooth and difficult to solve.Convergence experiments report that the algorithm converges within very few iterations.
2 RELATED WORK
Related work identifies three recurring limitations of classical feature selection: reliance on labeled data, one-by-one feature selection, and independent treatment of tasks. Semi-supervised, manifold, and multi-task methods address parts of this landscape but leave room for their integration.
- Feature selection: Classical feature selection evaluates and ranks features individually, using labeled data and treating each task independently.These choices limit use of unlabeled data, feature correlations, and shared information across related tasks.
- Feature selection: l2,1-norm regularization is widely used to select features jointly through joint sparsity.Prior methods apply it to feature-selection regularization or loss functions.
- Semi-supervised learning: Semi-supervised learning exploits unlabeled data to learn data structure and can reduce the human labor required for labeling.Manifold-learning approaches use graph Laplacian structure for semi-supervised analysis.
- Multi-task learning: Multi-task learning learns related tasks with a shared representation and has been applied to multimedia analysis.Prior multi-task feature selection leverages shared information, but related prior algorithms were implemented only with labeled data.
3 METHODOLOGY
The methodology formulates feature selection across multiple tasks with task-specific losses, sparsity, shared-task regularization, and manifold structure. It predicts labels for all training data and optimizes the non-smooth objective through iterative updates.
- Problem formulation: The framework models t tasks, each with n_l training samples and m_l labeled samples, using a task-specific feature-selection matrix W_l.The concatenated matrix W supports regularization of common components across tasks.
- Problem formulation: The objective combines task loss, l2,1-norm sparsity, manifold regularization, and trace-norm transfer of information across tasks.Least-square loss is adopted, while trace-norm minimization leverages shared knowledge among related tasks.
- Manifold regularization: Manifold regularization uses a Laplacian built from local neighborhoods, manifold structure, and local discriminant information.The construction uses k-nearest-neighbor cliques and centers data with a centering matrix.
- Label propagation: Predicted label matrices F_l assign label vectors to all training data, while U_l gives labeled samples high weight and unlabeled samples unit weight.The experiments approximate ∞ with 10^6 for labeled samples.
- Regularization: The l2,1-norm jointly evaluates feature informativeness within each task, while the trace norm transfers information among task-specific selection functions.This produces batch-style feature evaluation rather than independent one-feature selection.
- Optimization: Because the l2,1-norm and trace norm prevent a closed-form solution, SFMC solves the objective through iterative optimization steps.The optimization algorithm computes Laplacian and selection matrices during its updates.
13 repeat
The paper optimizes its non-smooth objective with an iterative algorithm and proves convergence through monotonic decrease of the objective value. The algorithm is reported to converge within very few iterations.
- Convergence analysis: Theorem 1 states that the objective function value monotonically decreases in each iteration until convergence.The convergence proof derives an inequality for successive iterations and establishes the decrease property.
- Convergence analysis: The convergence argument concludes that Algorithm 1 converges because its objective function value decreases after each iteration.Equation (22) is used to establish the decrease after each iteration.
4 EXPERIMENTS
Experiments evaluate the proposed feature-selection algorithm across four multimedia tasks using multiple datasets, labeled-data settings, and comparisons with baseline methods. Across these evaluations, it generally performs best, especially when labeled data are scarce.
- Experimental scope: Experiments cover video classification, image annotation, human motion recognition, and 3D motion data analysis across four datasets.The datasets are CCV, NUS-WIDE, HMDB, and HumanEva.
- Baselines: The comparisons include All Features, Fisher Score, FSNM, SPEC, FSSI, LSDF, and SFSS.These baselines span unsupervised, supervised, multi-task, and semi-supervised feature-selection approaches.
- Evaluation protocol: Training sets use varying labeled-data percentages, and results are averaged over five independent repetitions.For each dataset, n samples are selected for training, m% are labeled, and the remaining data are used for testing.
- Video classification: The proposed algorithm consistently achieves the best video-classification performance, with a 6.6% advantage over the second-best method for subject 2 at 5% labeled data.Its advantage becomes smaller as more training data are labeled.
- Image annotation: The proposed method outperforms the compared algorithms for image annotation and is more competitive with less labeled training data.Image annotation results are reported for 1%, 5%, and 10% labeled training data.
5 CONCLUSION
The paper presents a semisupervised feature analysis method that mines feature correlations and shared information across related tasks, with an iterative solver for its difficult objective. Across multiple applications, experiments report that it outperforms the compared algorithms.
- The method mines correlations between different features and leverages shared information across multiple related tasks.
- An iterative and effective algorithm is proposed because the objective function is non-smooth and difficult to solve.
- The evaluation covers video classification, image annotation, human motion recognition, and 3D motion data analysis.
- The proposed method outperforms the other compared algorithms across these applications.