Source-linked AI summary
Local Deep-Feature Alignment for Unsupervised Dimension Reduction
Jian Zhang, Jun Yu, Dacheng Tao
TL;DR
Dimension-reduction methods need to preserve locality while learning deep representations, but existing approaches do not fully combine these capabilities. LDFA learns deep features with local SCAEs, aligns neighbourhood-specific features globally, and provides explicit embedding for new samples; experiments report competitive performance across visualization, clustering, and classification.
Problem
Existing auto-encoders capture global rather than local characteristics, while smoothness regularization models locality only indirectly and manifold methods generally provide shallow features.
Method
LDFA constructs a neighbourhood for each sample, learns local deep features with SCAEs, and aligns them with global features through local affine transformations.
Results
Experiments in image visualization, clustering, and classification show that LDFA is competitive with several well-known dimension-reduction techniques.
Takeaways & Limitations
The experiments support using SCAEs to capture local data characteristics in unsupervised deep learning, including when neighbourhoods contain few samples.
Abstract
from arXiv · showhide
This paper presents an unsupervised deep-learning framework named Local Deep-Feature Alignment (LDFA) for dimension reduction. We construct neighbourhood for each data sample and learn a local Stacked Contractive Auto-encoder (SCAE) from the neighbourhood to extract the local deep features. Next, we exploit an affine transformation to align the local deep features of each neighbourhood with the global features. Moreover, we derive an approach from LDFA to map explicitly a new data sample into the learned low-dimensional subspace. The advantage of the LDFA method is that it learns both local and global characteristics of the data sample set: the local SCAEs capture local characteristics contained in the data set, while the global alignment procedures encode the interdependencies between neighbourhoods into the final low-dimensional feature representations. Experimental results on data visualization, clustering and classification show that the LDFA method is competitive with several well-known dimension reduction techniques, and exploiting locality in deep learning is a research topic worth further exploring.
I. INTRODUCTION
The paper motivates LDFA by combining locality preservation with unsupervised deep learning, addressing the limitations of methods that learn only global structure or only shallow local features.
- Proposed framework: The paper proposes LDFA, which aligns local features learned from neighbourhood subsets to obtain low-dimensional representations.The framework constructs a neighbourhood for each sample, learns local features, and aligns them across local coordinate systems.
- Advantages: LDFA uses stacked contractive auto-encoders to preserve locality, regularize learning from small neighbourhoods, and reduce the difficulty of robust feature learning.The authors argue that local similarity limits the number of variations that each embedding function must model.
- Motivation: Traditional AEs learn one transformation for all samples, capturing global characteristics while overlooking local characteristics that are useful in pattern recognition.The paper notes that real-world data often exhibit multiple Gaussian distributions, whose components reflect local characteristics.
B. Locality-Preserving Learning
Locality-preserving methods represent neighbourhood relationships during dimension reduction, but the paper identifies a gap in combining this capability with deep feature learning.
- Neighbourhood-based learning: Manifold-learning methods preserve local characteristics through neighbourhood graphs connecting each sample to nearby samples.These graphs are processed by local estimators that transfer locality into low-dimensional space.
- Representative methods: LLE preserves reconstruction weights, while LE and LPP preserve proximity between nearby samples in the reduced space.These methods use different linear or nonlinear mechanisms to encode neighbourhood relationships.
- Discriminative locality: Some manifold methods incorporate discriminative information by defining neighbourhoods using data similarity, identity, or pose.These constructions support discriminative feature learning or person-independent pose estimation.
- Limitation: Manifold algorithms can be reformulated as single-layer nonlinear networks, which limits their ability to discover deep-level features.Their objectives are deliberately designed for one-layer learning, making direct stacking unsuitable for deep feature learning.
- Deep-locality gap: Existing smoothness regularization preserves locality only indirectly because it was originally designed mainly to improve robustness.The paper therefore motivates explicitly combining locality preservation with deep learning.
A. Contractive Auto-Encoder (CAE)
Contractive auto-encoders learn reconstructive feature representations while regularizing sensitivity to data variation; stacking them produces a deeper feature extractor for local neighbourhoods.
- Auto-encoder: An auto-encoder maps input data to a feature representation and decodes that representation to reconstruct the input.The encoder uses parameters W and b, while the decoder uses transposed weights and bias c.
- Stacked architecture: A two-layer SCAE applies successive encoding layers to transform x1 into x3 before reconstructing x1.The superscript l identifies the network layer, and adjacent layer outputs serve as subsequent inputs.
- Contractive regularization: The CAE adds smoothness regularization to make feature learning insensitive to data variations while retaining reconstruction ability.This regularization is intended to produce robust low-dimensional features.
- Role in LDFA: The proposed framework uses local SCAEs to extract deep-level features separately from each neighbourhood.These local features are later aligned into global representations.
B. Objective Function of Local Deep-Feature Alignment
LDFA jointly learns deep features within sample neighbourhoods and aligns those local representations into a shared global coordinate system.
- Local feature learning: Each neighbourhood contains a sample and several of its closest neighbours, and its local SCAE extracts deep-level features.The objective includes a local deep-feature learning term for all neighbourhoods.
- Global alignment: The top-layer features learned by each local SCAE are neighbour-wise, so they must be converted into global deep features.The method assumes an affine transformation relates local features to their global counterparts.
- Global alignment: A local affine transformation matrix aligns each neighbourhood’s local coordinate system with the global coordinate system.The alignment seeks global features that preserve locality characteristics contained in the local features.
- Optimization: The overall LDFA objective combines local SCAE feature-learning error with the alignment error across neighbourhoods.The combined formulation jointly represents local deep learning and global alignment.
C. Optimization
LDFA optimization proceeds in two stages: local SCAEs learn neighbourhood features, then local features are aligned into global representations through an eigenvalue-based formulation.
- Stage 1: Local feature learning: Stage 1 learns local deep features by training a series of SCAEs for the constructed neighbourhoods.Each CAE layer is pre-trained, the SCAE is initialized from those parameters, and the deep network is optimized with gradient descent and back-propagation.
- Stage 2: Global alignment: Stage 2 aligns the local features to form global feature representations.
- Stage 2: Global alignment: The optimal local alignment matrix is computed as Ai = HiTi(HL_i)†, using the Moore-Penrose generalized inverse of the local features.
- Stage 2: Global alignment: The global representations H are selected so that each local representation Hi matches its corresponding global subset HSi while minimizing overall alignment error.The selection matrices Si identify the data contained in each neighbourhood, and the alignment objective is rewritten as trace(HΦHT).
- Optimization outcome: The final low-dimensional representations H are obtained by solving the resulting eigenvalue problem.Algorithm 1 computes neighbourhoods, trains local SCAEs, derives local features and alignment matrices, constructs Φ, and solves for H.
D. Embedding a New Data Sample
LDFA supports explicit out-of-sample embedding by attaching a fully connected alignment network to each local SCAE and selecting the network associated with the nearest training sample.
- Explicit embedding: LDFA constructs an explicit embedding function for each local neighbourhood to map new samples into the learned low-dimensional subspace.
- Uniform deep network: A one-layer fully connected feed-forward network maps each top-layer local feature representation to its global counterpart.The network uses sigmoid activation and replaces the original local affine-transformation matrix.
- Uniform deep network: Each local SCAE and its fully connected network form a uniform deep neural network for explicit globally aligned embedding.The SCAE parameters initialize the first L layers, while the fully connected network initializes the L + 1th layer before fine-tuning.
- Out-of-sample procedure: For a new sample, LDFA finds its nearest training sample and applies that sample’s corresponding embedding function.The resulting output is the new sample’s low-dimensional feature representation.
IV. EXPERIMENTS
The experiments evaluate LDFA for visualization, clustering, and classification, using local SCAE training, global alignment, and explicit out-of-sample embedding procedures.
- Experimental evaluation: LDFA is evaluated as a dimension-reduction preprocessing method for data visualization, clustering, and classification.The study reports both qualitative and quantitative experimental results.
- Training procedure: Training computes each neighbourhood, learns a local deep SCAE, extracts top-layer local features, and derives alignment matrices.
- Training procedure: The method constructs the global alignment matrix Φ, solves the eigenvalue problem for H, and builds fully connected networks between local and global features.
- Out-of-sample procedure: Out-of-sample embedding locates the new sample’s nearest neighbour, obtains its neighbourhood, and applies the corresponding trained network.
- Evaluation metric: Clustering accuracy is measured by purity, defined as the ratio of correctly clustered samples to the total number of samples.The purity formulation compares clustered data with the original class assignments.
A. The Data Sets
The experiments use seven benchmark image datasets spanning digits, faces, birds, dogs, and general objects, with repeated random sampling for statistical evaluation.
- Benchmark datasets: The benchmark suite contains MNIST Digits, USPS Digits, Olivetti Faces, UMist Faces, NABirds, Stanford Dogs, and Caltech256.
- Dataset configuration: Dataset descriptions report class counts, total sample counts, data dimensions, and the images selected for training and testing.
- Parameter analysis: Clustering-accuracy analysis varies the number of local SCAE layers and output feature dimensions while fixing neighbourhood size to 10.
- Dataset configuration: Only 100 of NABirds’ 400 species are included in the experiments.
- Evaluation protocol: All experiments are repeated 10 times with randomly selected images, and the statistical results are shown using box plots.
B. Data Visualization and Clustering
LDFA is evaluated through visualization and clustering on small and larger image datasets. It is close to t-SNE in visualization, often outperforms competing methods in clustering, and is sensitive to neighbourhood size.
- Method implications: LDFA can work well with limited data because local SCAEs learn from small, similar neighbourhoods and use regularization to estimate parameters.The authors also state that locality reduces the number of local embedding-function variations and the need for complex networks.
- Clustering: LDFA outperforms PCA, LTSA, LLC, SAE, and SCAE on MNIST and USPS clustering, but remains inferior to t-SNE.On Olivetti and UMist Faces, LDFA achieves the best clustering performance.
- Clustering: On Olivetti and UMist Faces, LDFA achieves the best clustering performance among the compared methods.SCAE also outperforms SAE on these face datasets, which the authors associate with their similar structure and texture differences.
- Clustering: LDFA outperforms the other five algorithms on clustering accuracies for the NABirds, Stanford Dogs, and Caltech-256 datasets.These larger datasets contain varied object poses and sizes, often with cluttered natural-scene backgrounds.
C. Data Classification
The paper evaluates whether LDFA produces discriminative low-dimensional representations for classification. Across multiple datasets and classifiers, LDFA generally achieves the strongest results, with several stated exceptions.
- Evaluation design: The classification experiments compare feature representations rather than maximizing accuracy, so some state-of-the-art classifiers are deliberately omitted.The classifiers are tuned to generate their best results within this comparison design.
- MNIST: LDFA outperforms SAE with the same network structure on MNIST classification across the tested four-layer structures.The best SAE structure is 784-512-256-128-64, and classification is repeated 10 times.
- Common datasets: LDFA achieves the highest classification accuracy in most cases on MNIST Digits, USPS Digits, Olivetti Faces, and UMist Faces.Exceptions occur for naive Bayes classification on USPS Digits and UMist Faces.
- LBP features: On NABirds, Stanford Dogs, and Caltech-256 using LBP descriptors, LDFA produces the highest and best mean classification accuracy for each classifier.The only exception is random-forest classification on Stanford Dogs.
- HOG features: Using HOG descriptors, LDFA produces the best results on NABirds, Stanford Dogs, and Caltech-256 except for AdaBoost and LDA on Stanford Dogs.The HOG descriptors are first reduced to 500 dimensions through PCA before comparison.
V. CONCLUSION
LDFA combines local SCAE-based feature learning with global affine alignment for unsupervised dimension reduction, and supports explicit embedding of new samples. Experiments applied it to visualization, clustering, and classification, with results indicating that local deep features improve unsupervised deep-learning performance.
- LDFA defines neighbourhoods, learns local deep features with SCAEs, and aligns them with global features through local affine transformations.
- The method provides an explicit approach for mapping new data into the learned low-dimensional subspace.
- LDFA was used as a preprocessing step for image visualization, image clustering, and image classification.
- SCAE extracted discriminative local deep features robustly from small neighbourhoods with few network layers.
- The experiments support using SCAE to capture local data characteristics to improve unsupervised deep-learning performance.