Source-linked AI summary
Deep Subspace Clustering Networks
Pan Ji, Tong Zhang, Hongdong Li, Mathieu Salzmann, Ian Reid
TL;DR
Subspace clustering methods often assume linear subspaces, while practical data may occupy nonlinear structures and predefined kernels may be poorly suited. DSC-Nets learn an explicit nonlinear mapping and self-expressive affinities jointly, and experiments report significant improvements over state-of-the-art methods.
Problem
Existing subspace clustering methods primarily target linear subspaces, while practical data can form nonlinear subspaces and predefined kernels may not provide suitable feature spaces.
Method
DSC-Nets combine deep auto-encoders with a differentiable self-expressive layer between encoder and decoder to jointly learn nonlinear representations and data-point affinities.
Results
DSC-Nets significantly outperform state-of-the-art methods across face and object clustering benchmarks; DSC-Net-L2 achieves 2.67% error for 38 Extended Yale B subjects and 5.34% error on COIL20.
Takeaways & Limitations
The framework directly learns affinities within one neural network while handling challenging nonlinear clustering settings and small-size datasets.
Takeaways & Limitations
The self-expressive layer has N^2 parameters and can dominate network size, making direct training from scratch difficult for networks with millions of parameters.
Abstract
from arXiv · showhide
We present a novel deep neural network architecture for unsupervised subspace clustering. This architecture is built upon deep auto-encoders, which non-linearly map the input data into a latent space. Our key idea is to introduce a novel self-expressive layer between the encoder and the decoder to mimic the "self-expressiveness" property that has proven effective in traditional subspace clustering. Being differentiable, our new self-expressive layer provides a simple but effective way to learn pairwise affinities between all data points through a standard back-propagation procedure. Being nonlinear, our neural-network based method is able to cluster data points having complex (often nonlinear) structures. We further propose pre-training and fine-tuning strategies that let us effectively learn the parameters of our subspace clustering networks. Our experiments show that the proposed method significantly outperforms the state-of-the-art unsupervised subspace clustering methods.
1 Introduction
The paper addresses subspace clustering when data lie on nonlinear subspaces, introducing DSC-Nets to learn an explicit mapping and affinities jointly. Experiments on face and object datasets report significant improvements over state-of-the-art methods.
- Motivation: Subspace clustering aims to cluster data drawn from a union of low-dimensional subspaces without supervision.The problem has applications including image segmentation, motion segmentation, and image clustering.
- Limitations of Existing Methods: Existing methods mainly target linear subspaces, although practical face images can form nonlinear subspaces because of non-Lambertian reflectance and varying pose.Kernel methods address nonlinear cases, but kernel selection is largely empirical and may yield unsuitable feature spaces.
- Proposed Approach: DSC-Nets learn an unsupervised explicit nonlinear mapping adapted to subspace clustering using deep auto-encoders.The encoder maps data points into a latent space through multiple layers.
- Proposed Approach: A self-expressive layer between encoder and decoder represents each sample as a linear combination of other samples and directly learns affinities.The layer is a fully connected, bias-free layer without nonlinear activations.
- Training: The method uses pre-training and fine-tuning strategies to learn DSC-Net parameters unsupervised with limited data.The strategies initialize and optimize the network after auto-encoder pre-training.
- Evaluation: DSC-Nets significantly outperform state-of-the-art subspace clustering methods on face clustering with Extended Yale B and ORL and object clustering with COIL20 and COIL100.These experiments cover both face and general object clustering.
2 Related Work
Related work spans traditional subspace clustering, auto-encoders, and an early combination of deep learning with subspace-clustering priors. The paper positions DSC-Nets as a more direct integration of learned representations and affinities.
- Subspace Clustering: Traditional subspace clustering estimates pairwise affinities, forms an affinity matrix, and then applies normalized cuts or spectral clustering.Methods are broadly categorized as factorization, higher-order model, and self-expressiveness approaches.
- Auto-Encoders: Auto-encoders nonlinearly transform data into latent spaces and typically optimize reconstruction through encoder and decoder components.Deep auto-encoders have been used for dimensionality reduction, image denoising, and initializing deep embedding networks.
- Deep Learning and Subspace Clustering: Prior work combined deep auto-encoder features with sparse subspace clustering or k-means, applying clustering after feature extraction rather than directly learning affinities within one network.That approach used SIFT or HOG features and a fully connected deep auto-encoder.
3 Deep Subspace Clustering Networks (DSC-Nets)
DSC-Nets combine stacked encoders, a self-expressive layer, and stacked decoders to learn nonlinear representations and affinities jointly. The network is trained through auto-encoder pre-training and full-network fine-tuning, then uses the learned layer for spectral clustering.
- 3.1 Self-Expressiveness: Self-expressiveness represents each point in a linear subspace as a linear combination of other points from that subspace.For a data matrix X, coefficients in C encode these combinations, optionally excluding each point itself.
- 3.1 Self-Expressiveness: Self-expressiveness applies only to linear subspaces, motivating an explicitly learned mapping that makes nonlinear subspaces more separable.The paper uses deep auto-encoders to learn this mapping rather than relying on predefined kernels.
- 3.2 Self-Expressive Layer: The network inserts a self-expressive layer at the junction between encoder and decoder to encode self-expressiveness in the latent representation.The encoder output ZΘe is transformed before entering the decoder.
- 3.2 Self-Expressive Layer: The self-expressive layer is a fully connected linear layer whose weights correspond to C and are used to construct affinities between all data points.The layer contains linear neurons without nonlinear activations, and its weights can be optimized jointly with encoder and decoder parameters through backpropagation.
- 3.2 Self-Expressive Layer: DSC-Net-L1 and DSC-Net-L2 apply ℓ1 and ℓ2 regularization, respectively, to the self-expressive-layer parameters.These parameters are the elements of C.
- 3.3 Network Architecture: The architecture uses stacked encoders, a self-expressive layer, and stacked decoders, with convolutional auto-encoders advocated for image clustering.Fully connected auto-encoders are also compatible with the self-expressive layer.
- 3.3 Network Architecture: For 1,000 input images, the self-expressive layer has 10^6 parameters versus 14,671 encoder-decoder parameters in the stated example, so it typically dominates network size.This parameter scaling reflects the N^2 size of the self-expressive layer.
- 3.3 Network Architecture: Because direct training from scratch is difficult for networks with millions of parameters, the method pre-trains the auto-encoder and then fine-tunes the full network on all data.Fine-tuning uses gradient descent with Adam and a learning rate of 1.0 × 10^-3 in the reported experiments.
4 Experiments
The experiments evaluate the method on face and object image clustering datasets against several subspace-clustering baselines. DSC-Net methods achieve lower clustering errors, remain more stable as cluster counts increase, and outperform competing methods across the reported settings.
- Experimental setup: The evaluation compares DSC-Net-L1 and DSC-Net-L2 with LRR, LRSC, SSC, KSSC, SSC-OMP, EDSC, AE+SSC, and AE+EDSC on four standard datasets.The datasets are Extended Yale B and ORL for faces, and COIL20 and COIL100 for objects.
- Extended Yale B Dataset: Extended Yale B contains 38 subjects with 64 images each, and experiments test robustness across increasing numbers of clusters.The study evaluates K ∈ {10, 15, 20, 25, 30, 35, 38} subjects using consecutive subject groups.
- Extended Yale B Dataset: 2.67% clustering error on Extended Yale B for 38 subjects is around 1/5 of the best-performing baseline EDSC.DSC-Net-L2 remains relatively stable as the number of subjects increases, while most baselines degrade dramatically.
- ORL Dataset: ORL is more challenging because facial expressions and details introduce greater non-linearity, and DSC-Net errors are roughly half those of EDSC and SSC with pretrained features.The competing methods’ error rates remain around twice as high as those of the proposed methods.
- COIL20 and COIL100 Datasets: 5.34% clustering error on COIL20 is roughly 1/3 of the AE+EDSC error, while DSC-Net-L2 achieves the lowest error on both COIL20 and COIL100.COIL20 and COIL100 contain diverse object images captured across changing viewing angles and evaluated over 20 and 100 classes, respectively.
5 Conclusion
The paper combines a deep auto-encoder with a self-expressive layer for subspace clustering and proposes training strategies for the resulting network. Experiments show improved clustering accuracy over state-of-the-art methods, including on small datasets.
- The self-expressive layer harnesses the self-expressiveness property of a union of subspaces within a deep auto-encoder framework.
- Pre-training and fine-tuning strategies train the network in challenging small-data settings, including the ORL dataset.
- The proposed methods significantly improve clustering accuracy over state-of-the-art subspace clustering solutions on several standard datasets.