Source-linked AI summary
Deep Learning of Part-based Representation of Data Using Sparse Autoencoders with Nonnegativity Constraints
Ehsan Hosseini-Asl, Jacek M. Zurada, Olfa Nasraoui
TL;DR
The paper addresses limited part-based feature learning in autoencoders by imposing nonnegativity constraints on autoencoder and deep-network weights. It proposes NCAE with sparse representations, evaluates it across image and text datasets, and reports more interpretable part-based features, improved reconstruction, and better prediction performance than related approaches.
Problem
The paper seeks an autoencoder representation that disentangles hidden structure through sparse, part-based features while supporting effective deep-network training.
Source: §I · pp. 1–2Method
NCAE trains autoencoders with nonnegative weights, stacks pretrained layers, and fine-tunes a softmax classifier with constrained negative weights.
Source: §I · p. 2, §II · p. 2, B. Deep Learning using the Nonnegative… · p. 4Results
Across image and text evaluations, NCAE learns sparse, interpretable part-based features, improves reconstruction quality, and improves deep-network prediction relative to the reported comparison methods.
Source: 1) Learning Part-based Representation… · pp. 6, 8, B. Supervised Learning · p. 12, §IV · p. 12Takeaways & Limitations
Nonnegativity provides an autoencoder representation that decomposes inputs into parts and is associated with better discrimination in subsequent network layers.
Source: §I · p. 2, B. Supervised Learning · p. 12
Abstract
from arXiv · showhide
We demonstrate a new deep learning autoencoder network, trained by a nonnegativity constraint algorithm (NCAE), that learns features which show part-based representation of data. The learning algorithm is based on constraining negative weights. The performance of the algorithm is assessed based on decomposing data into parts and its prediction performance is tested on three standard image data sets and one text dataset. The results indicate that the nonnegativity constraint forces the autoencoder to learn features that amount to a part-based representation of data, while improving sparsity and reconstruction quality in comparison with the traditional sparse autoencoder and Nonnegative Matrix Factorization. It is also shown that this newly acquired representation improves the prediction performance of a deep neural network.
I. INTRODUCTION
The paper develops a nonnegativity-constrained autoencoder to learn sparse, part-based representations and improve reconstruction and deep-network prediction. It motivates this approach through autoencoder pretraining, additive representations, and comparisons with related methods.
Motivation
- Deep networks learn complex mappings but are difficult to train because random initialization can become trapped in poor solutions.Greedy layer-wise pretraining was proposed to improve this bottleneck and guide supervised fine-tuning.Source: §I · p. 1
Autoencoder learning
- Autoencoders reconstruct inputs from encoded representations, and stacked pretrained layers provide initialization for deep networks.Lower reconstruction error at each layer is described as producing a more accurate representation and better initialization.Source: §I · p. 1
Part-based representation
- Part-based representation decomposes data into additive components, potentially disentangling hidden structure in high-dimensional data.The motivation draws on sparse coding, NMF, and evidence that visual data is represented hierarchically and by parts.Source: §I · pp. 1–2
Contribution
- The proposed method imposes nonnegativity during autoencoder learning to obtain sparse, part-based features and better reconstruction.The approach is extended to stacked autoencoders and a softmax classifier with nonnegative weight constraints.Source: §I · p. 2
Evaluation
- The method is evaluated against SAE, NNSAE, NMF, DAE, and DpAE on image and text datasets, including representation and classification performance.The reported comparisons cover latent features, reconstruction error, sparsity, and deep-network prediction.Source: §I · p. 2
II. METHODS
The methods section describes an autoencoder that maps inputs to hidden representations and reconstructs them, optimizing reconstruction while controlling representation size and sparsity. The paper also uses layer-wise pretraining to construct deep networks.
Autoencoder architecture
- An autoencoder maps an input vector x to a hidden representation and then maps that representation to a reconstructed vector.The hidden mapping uses parameters θ1 = {W1, b1}, while the decoder uses θ2 = {W2, b2}.Source: §II · p. 2
- The hidden and output layers use logistic sigmoid mappings, with W1 and W2 defining the corresponding weight dimensions.The hidden representation h lies in [0,1]^n′ and the reconstruction x̂ lies in [0,1]^n.Source: §II · p. 2
Objective
- Average reconstruction error is used as the cost function for optimizing the autoencoder parameters.The training set size is represented by m.Source: §II · p. 2
Representation
- Limiting hidden dimension n′ produces a compressed representation intended to help reveal latent structure in high-dimensional data.The paper treats parameter constraints as a way to control the learned representation.Source: §II · p. 2
Sparsity
- Sparsity is encouraged by constraining the average activation of each hidden unit to a small positive parameter p using KL divergence.This limits hidden-unit activity and supports sparse representations.Source: §II · pp. 2–3
A. Part-based Representation Using a Nonnegativity Constrained Autoencoder (NCAE)
NCAE adds a nonnegativity penalty to autoencoder training so data can be represented through sparse parts combined additively. Its objective combines reconstruction, sparsity, and penalties on negative weights.
Part-based representation
- Part-based representation combines decomposed parts additively, and constraining W to be nonnegative supports this form of representation.The paper frames nonnegative weights as a mechanism for decomposing data into parts.Source: A. Part-based Representation Using a… · p. 3
NCAE objective
- NCAE replaces the standard weight-decay term with a quadratic penalty that encourages nonnegative weights.The resulting objective is designed to penalize negative weights during training.Source: A. Part-based Representation Using a… · p. 3
- Minimizing the NCAE objective reduces reconstruction error, increases hidden-layer sparsity, and reduces nonnegative-weight violations.These effects are stated as consequences of minimizing the combined cost function.Source: A. Part-based Representation Using a… · p. 3
Optimization
- Weight and bias updates use gradients of the NCAE objective within backpropagation.The derivative with respect to weights contains three terms, with the learning rate η controlling updates.Source: A. Part-based Representation Using a… · p. 3
Evaluation setup
- On MNIST, NCAE receptive fields are compared with SAE, NNSAE, and NMF, while decoding filters are compared with SAE.The visualizations include weight histograms for the learned filters.Source: Figure 2 · p. 4, Figure 3 · p. 4
B. Deep Learning using the Nonnegative Constrained Autoencoder (NCAE)
The deep NCAE network is built by stacking pretrained autoencoder layers with a softmax classifier, followed by supervised fine-tuning under nonnegativity constraints.
- Each autoencoder layer is pretrained separately, using the previous layer’s hidden activities as input to the next.Source: B. Deep Learning using the Nonnegative… · p. 4
- The final autoencoder’s hidden activities serve as input to a softmax regression classifier trained in supervised mode.Source: B. Deep Learning using the Nonnegative… · p. 4
- The softmax cost function constrains negative input weights using a penalty defined for the classifier’s weight columns.Source: B. Deep Learning using the Nonnegative… · p. 4
- Supervised fine-tuning stacks the pretrained NCAE and softmax layers to optimize classification accuracy, while constraining only softmax-layer negative weights.Source: B. Deep Learning using the Nonnegative… · p. 4
- Batch gradient descent with L-BFGS minimizes the autoencoder, softmax, and deep-network objective functions.Source: B. Deep Learning using the Nonnegative… · p. 5
III. EXPERIMENTAL RESULTS
The experiments evaluate unsupervised feature learning and classification using three benchmark image datasets and one text corpus.
- The evaluation uses MNIST handwritten digits, ORL face images, small NORB object recognition, and the Reuters 21578 document corpus.Source: §III · p. 5
- The experimental section reports both feature-learning tests and classification performance for a deep network built from NCAE components.Source: §III · p. 5
- The parameter-settings table records the algorithm configurations used in the experiments.Source: Table I · p. 5
A. Unsupervised Feature Learning
Unsupervised feature learning compares NCAE with sparse autoencoder, nonnegative sparse autoencoder, and NMF representations using image receptive fields and text features.
- Three-layer NCAE features are compared with three-layer SAE, NNSAE, and NMF basis images for image data.Source: A. Unsupervised Feature Learning · p. 5
- For text data, the first-layer weights represent groups of words used to evaluate whether the learned features connect to topics.Source: A. Unsupervised Feature Learning · p. 5
- The algorithms are tuned over ranges of regularization parameters to minimize their respective objective functions.Source: A. Unsupervised Feature Learning · p. 5
- The reported sparseness histograms measure the criterion across 196 receptive fields and 196 decoding filters.Source: Figure 4 · p. 5, Figure 5 · p. 5
1) Learning Part-based Representation of Images:
Across MNIST, ORL Faces, and small NORB images, NCAE learns sparse, localized, part-based features and achieves strong reconstruction relative to SAE, NNSAE, and NMF.
MNIST
- NCAE receptive fields on MNIST are more sparse and localized than those learned by SAE, NNSAE, and NMF.The features capture basic digit structures such as strokes and dots.Source: 1) Learning Part-based Representation… · pp. 5–6
- 1.8799 reconstruction error: NCAE outperforms SAE and NNSAE and is similar to NMF on MNIST test digits.NCAE reconstructions are less blurred than NMF reconstructions.Source: Figure 6 · p. 6, 1) Learning Part-based Representation… · p. 6
- NCAE hidden activations are sparser than SAE activations for different numbers of MNIST hidden neurons.Sparsity is measured using KL divergence with p=0.05.Source: 1) Learning Part-based Representation… · p. 6, Figure 7 · p. 6
ORL Faces
- NCAE learns sparse face features containing recognizable parts such as eyes, noses, and mouths, whereas SAE, NNSAE, and NMF mainly learn holistic features.Constraining negative weights to zero produces fewer darker pixels in the receptive fields.Source: Figure 9 · p. 7, 1) Learning Part-based Representation… · p. 8
- Increasing the nonnegativity penalty coefficient α makes ORL face features sparser and decomposes faces into smaller parts.The tested α values range logarithmically from 0.003 to 0.3.Source: 1) Learning Part-based Representation… · p. 8, Figure 11 · p. 8
Small NORB
- NCAE learns more localized edge-like features on small NORB than the more holistic basis images learned by NMF.The comparison uses receptive fields from SAE, NNSAE, NCAE*, and NMF.Source: 1) Learning Part-based Representation… · p. 9, Figure 12 · p. 9
2) Semantic feature discovery from text data:
NCAE extracts semantic features from Reuters documents by grouping informative words around latent topics. Its representations organize related documents and place meaningfully related topics closer in semantic space.
Feature preparation
- NCAE represents Reuters documents in a TF-IDF vector space after removing very rare and very frequent words and selecting informative features.The corpus contains 11,413 documents and 12,317 word dimensions before dimensionality reduction.Source: 2) Semantic feature discovery from… · p. 9
Semantic feature discovery
- With 200 input neurons and 15 hidden neurons, NCAE groups semantically related words around latent topic features.The examined words are ranked by their connecting weights to hidden neurons.Source: 2) Semantic feature discovery from… · pp. 9–10
- NCAE identifies seven distinguishable Reuters topics, including ship, crude, earn, acq, money-fx, grain, and trade.Each topic is associated with its top weighted words.Source: Figure 13 · p. 10, 2) Semantic feature discovery from… · p. 10
Semantic organization
- NCAE groups related Reuters documents together, with meaningfully related topics closer in the semantic space.The document visualization uses a 15-dimensional representation projected with t-SNE.Source: 2) Semantic feature discovery from… · p. 10, Figure 14 · p. 10
B. Supervised Learning
The authors pretrain deep networks by stacking NCAE autoencoders and then fine-tune them with a softmax classifier. NCAE improves classification before fine-tuning across all three datasets and after fine-tuning on two of them, while part-based first-layer features support class discrimination.
Training procedure
- Deep networks are pretrained by stacking NCAE autoencoders, followed by softmax training and supervised fine-tuning.Classification is evaluated on MNIST, NORB, and Reuters-21578, with results averaged over 10 experiments.Source: B. Supervised Learning · pp. 10–11
Classification results
- NCAE pretraining yields significantly better accuracy than other networks on all three datasets before fine-tuning.The comparisons include SAE, NNSAE, DAE, and DpAE-based networks.Source: B. Supervised Learning · p. 11
- After fine-tuning, NCAE remains superior on two of the three datasets; on NORB, its accuracy is not significantly different from DAE and DpAE.The NORB result limits the post-fine-tuning advantage to two datasets.Source: B. Supervised Learning · pp. 11–12
Optimization
- NCAE converges faster than the other networks without using more pre-fine-tuning iterations.All networks were trained for 400 iterations before fine-tuning.Source: B. Supervised Learning · pp. 11–12
Representation and discrimination
- NCAE first-layer features decompose data into distinct parts, whereas other networks learn more holistic features that provide less class separation.The paper links this part decomposition to better discrimination between classes in later layers.Source: B. Supervised Learning · p. 12
IV. CONCLUSION
The paper proposes NCAE, a deep autoencoder trained with nonnegative weight constraints in both unsupervised learning and supervised fine-tuning. Across four datasets, the method is evaluated against autoencoder and matrix-factorization baselines for representation quality and prediction.
- NCAE applies nonnegative weight constraints during unsupervised autoencoder training and supervised fine-tuning.Nonnegativity is motivated by its promotion of additive features and part-based data representation.Source: §IV · p. 12
- The evaluation covers MNIST, ORL faces, small NORB objects, and the Reuters-21578 text corpus.The study examines reconstruction error, part-based feature representation, hidden-encoding sparsity, and prediction performance.Source: §IV · p. 12
- NCAE features capture part-based representations while achieving lower reconstruction error and better hidden-encoding sparsity than SAE and NMF.The reported comparison concerns the unsupervised learning stage.Source: §IV · p. 12
- Deep-network prediction performance is compared with SAE, NNSAE, DAE, and DpAE.Source: §IV · p. 12