Source-linked AI summary
A deep matrix factorization method for learning attribute representations
George Trigeorgis, Konstantinos Bousmalis, Stefanos Zafeiriou, Bjoern W. Schuller
TL;DR
The paper addresses whether complex mappings in Semi-NMF contain hierarchical information about hidden attributes that one-level clustering cannot interpret. It proposes Deep Semi-NMF and partially supervised Deep WSF to learn attribute-oriented representations, and reports improved clustering and classification performance over Semi-NMF and other compared NMF variants.
Problem
Single-level clustering may not interpret complex hierarchical structure and implicit lower-level attributes in the mapping between a dataset and its Semi-NMF representation.
Method
Deep Semi-NMF factorizes the data mapping into multiple factors to learn hidden representations, while Deep WSF incorporates partial labels for known attributes through a hierarchy and graph regularization.
Results
Deep Semi-NMF outperforms the compared NMF-based techniques for face-image clustering, while Deep WSF learns better attribute features and improves classification with partial attribute information.
Takeaways & Limitations
The models support clustering and classification according to multiple known or unknown dataset attributes, including settings with partially annotated or mixed-source data.
Takeaways & Limitations
The paper identifies other applications, including multi-source speech recognition and multilinear extensions, as future work.
Abstract
from arXiv · showhide
Semi-Non-negative Matrix Factorization is a technique that learns a low-dimensional representation of a dataset that lends itself to a clustering interpretation. It is possible that the mapping between this new representation and our original data matrix contains rather complex hierarchical information with implicit lower-level hidden attributes, that classical one level clustering methodologies can not interpret. In this work we propose a novel model, Deep Semi-NMF, that is able to learn such hidden representations that allow themselves to an interpretation of clustering according to different, unknown attributes of a given dataset. We also present a semi-supervised version of the algorithm, named Deep WSF, that allows the use of (partial) prior information for each of the known attributes of a dataset, that allows the model to be used on datasets with mixed attribute knowledge. Finally, we show that our models are able to learn low-dimensional representations that are better suited for clustering, but also classification, outperforming Semi-Non-negative Matrix Factorization, but also other state-of-the-art methodologies variants.
1 INTRODUCTION
The paper extends Semi-NMF from a single clustering-oriented representation to deep hierarchies that model hidden attributes and complex data structure. It also introduces Deep WSF for partially labeled attributes and reports improved clustering and classification representations.
- Semi-NMF motivation: Semi-NMF constrains only H to be nonnegative, enabling lower-dimensional features with a soft clustering interpretation.Z represents cluster centroids, while H represents soft membership indicators.
- Deep Semi-NMF: Deep Semi-NMF factorizes the mapping between X and H into multiple factors to learn hidden representations and latent attributes.The motivation is that complex multimodal data may contain hierarchical attributes such as identity, pose, and expression.
- Deep WSF: Deep WSF incorporates partial attribute labels through graph regularization, with each hierarchy level corresponding to a known attribute.It is designed for datasets with mixed attribute knowledge and can use partially annotated data sources.
- Relation to prior work: The paper positions its contribution against hierarchical clustering and multi-label learning by targeting partially annotated datasets with known and unknown attributes.It states that this is the first work to automatically discover representations for different attributes in a multimodal face-clustering application.
2 BACKGROUND
The background presents matrix factorization as constrained decomposition, then explains Semi-NMF's clustering interpretation and its relation to NMF-based feature-learning methods. It also distinguishes the paper's Deep WSF setting from prior semi-supervised NMF approaches.
- Matrix factorization: Matrix factorization decomposes X into basis or loading matrix Z and feature or component matrix H under method-specific constraints.SVD imposes no sign restrictions, whereas NMF requires all involved matrices to be nonnegative.
- Semi-NMF: Semi-NMF allows mixed signs in X and Z while restricting H to nonnegative components.This relaxes NMF's non-negativity constraints while preserving a factorization suitable for clustering.
- Semi-NMF: Viewing Z as centroids and H as datapoint indicators gives Semi-NMF a clustering interpretation.With orthogonal H, the method becomes equivalent to k-means; without orthogonality, it is a soft clustering method.
- Semi-NMF optimization: The Semi-NMF objective is optimized by alternating updates of Z and H while imposing non-negativity only on H.The update for Z uses the Moore–Penrose pseudo-inverse of H, and numerical stabilization uses a small epsilon.
- Related NMF variants: Multi-layer NMF learns feature hierarchies but lacks the clustering interpretation attributed to Deep Semi-NMF.The paper reports that Deep Semi-NMF outperforms Multi-layer NMF and all compared models for face-image clustering feature learning.
- Semi-supervised NMF: Deep WSF is compared with CNMF and DNMF, which incorporate label information respectively as hard feature constraints or Fisher-criterion discriminant information.These methods represent prior semi-supervised non-negative matrix factorization approaches.
3 DEEP SEMI-NMF
Deep Semi-NMF factorizes data into multiple non-negative latent representations, allowing different layers to support clustering according to distinct attributes. Greedy layerwise pretraining followed by fine-tuning, with optional nonlinearities, is used to learn this hierarchy.
- Model formulation: Deep Semi-NMF factorizes X into m+1 factors, producing a hierarchy of implicit representations.Each implicit representation is constrained to be non-negative and retain a clustering interpretation.
- Hierarchical interpretation: A three-layer face model can represent identity, expression, and pose through progressively composed mappings.The factors Z3, Z2Z3, and Z1Z2Z3 correspond respectively to mappings involving expressions, poses, and face images.
- Hierarchical interpretation: The model aims to learn the latent attribute hierarchy and representations suited to clustering at each layer.The authors also hypothesize that the final layer can improve clustering for the attribute with the lowest variability.
- Optimization: Layerwise pretraining decomposes X and successive feature matrices before alternating-minimization fine-tuning reduces reconstruction error.The authors state that pretraining greatly improves training time.
- Nonlinear extension: Nonlinear functions between layers increase expressibility and can represent latent attributes that are non-linearly separable in the input space.The nonlinear formulation is intended to better approximate the manifolds on which the data lie.
4 WEAKLY-SUPERVISED ATTRIBUTE LEARN-
Deep WSF incorporates partial labels for known attributes through graph-based regularization, while WSF-MA combines multiple attribute constraints in one representation. These methods support mixed attribute knowledge and can improve supervised and clustering outcomes.
- Deep WSF: Deep WSF learns a hierarchy whose levels correspond to known attributes with only partial labels available.It incorporates attribute information through graph regularization.
- Reported outcomes: The proposed Deep WSF initialization improves Deep Semi-NMF identity clustering, while Deep WSF improves face classification for pose, expression, and identity.These outcomes are reported for the illustrated face-learning setting.
- Graph regularization: Graph regularization encourages final representations of samples known to share a label to remain close.When no expert information exists for an attribute, its regularization term has no influence on optimization.
- Optimization: Deep WSF training initializes factors greedily with WSF or Semi-NMF and fine-tunes them until convergence.The linear version can use multiplicative updates, while nonlinear variants use gradient-based optimization.
- Multiple-attribute extension: WSF-MA learns one representation from multiple attributes by constructing separate Laplacian graphs with different regularization factors.This extends WSF to datasets with multiple attribute constraints.
5 OUT-OF-SAMPLE PROJECTION
The learned Deep Semi-NMF or Deep WSF model can project an unseen sample into a lower-dimensional embedding while keeping learned weight matrices fixed. Two projection strategies are presented, differing in how they treat non-negativity.
- Projection setup: An out-of-sample point x* is mapped to a lower-dimensional embedding using learned weights and features from the trained model.The projection is applied after training either the unsupervised or weakly supervised model.
- Projection methods: Basis matrix reconstruction projects x* into the linear space defined by Z but does not guarantee a non-negative embedding.This method is described as having precedent in NMF-based work.
- Projection methods: Non-negativity update rules learn the new features while keeping all learned weight matrices fixed.The resulting feature layers are constrained to be non-negative.
6 EXPERIMENTS
Experiments evaluate Deep Semi-NMF and Deep WSF on synthetic and face datasets, testing hierarchical representations, clustering, reconstruction, initialization, and classification. Deep Semi-NMF generally improves clustering over competing methods, while reconstruction error remains comparable to Semi-NMF.
- Datasets and evaluation: Deep Semi-NMF is evaluated on CMU PIE and XM2VTS face datasets, with experiments using pixel intensities, image gradient orientations, clustering, and classification.CMU PIE contains 2,856 images of 68 subjects, while XM2VTS contains 2,360 images of 295 subjects.
- Synthetic multimodal data: A second deep-factorization layer finds a projection that makes the synthetic dataset’s classes linearly separable, unlike simple Semi-NMF.The synthetic example uses pose and identity labels for the first and second layers, respectively.
- Model configurations: Two-layer models use 625 features in the first hidden representation and 20–70 features in the second, balancing speed and accuracy.Additional experiments vary the number of layers while fixing the final layer at 40 components.
- Reconstruction error: Semi-NMF consistently achieves lower reconstruction error than the other methods, while Deep Semi-NMF does not significantly reduce error relative to equivalent Semi-NMF models.The authors associate Semi-NMF’s reconstruction advantage with allowing mixed signs in Z.
- Clustering results: Deep Semi-NMF significantly outperforms every compared method in pixel-intensity clustering accuracy across the evaluated datasets.The comparison uses k-means on the learned feature representations.
- Clustering results: With image gradient orientations, Deep Semi-NMF outperforms single-layer Semi-NMF, and all comparisons except CMU PIE pose are statistically significant at p ≪ 0.01.The gradient-orientation features produce a mixed-sign input matrix.
7 CONCLUSION
The paper introduces Deep Semi-NMF to learn hierarchical attribute representations for clustering and reports improved final-layer clustering on face-image datasets. Deep WSF incorporates known attribute information and supports auxiliary-dataset pretraining, while future work considers broader applications and multilinear extensions.
- Deep Semi-NMF automatically learns a hierarchy of attributes and representations suited for clustering according to those attributes.
- The method learns high-level final-layer representations for clustering the least-variable attribute, outperforming the considered NMF-based techniques on two face-image datasets.
- Deep WSF incorporates known attributes, including partially annotated or mixed-source information, to learn better features for multiple attributes simultaneously.
- Deep WSF also supports auxiliary-dataset pretraining to speed learning and uncover better representations for the attribute of interest.
- Future work will examine applications such as multi-source speech recognition and multilinear extensions of the framework.