PaperlayerView exact revision on arXiv

Source-linked AI summary

Relational Autoencoder for Feature Extraction

Qinxue Meng, Daniel Catchpoole, David Skillicorn, Paul J. Kennedy

arXiv:1802.03145v1cs.LGstat.ML

TL;DR

High-dimensional feature extraction needs representations that preserve useful information, but traditional autoencoders do not explicitly model relationships among samples. The paper proposes relational autoencoders that combine reconstruction with relationship modeling and extends them to major autoencoder variants. On MNIST and CIFAR-10, the experiments show lower reconstruction loss, more robust features, and improved classification results.

  • Problem

    Traditional autoencoders reconstruct data but do not explicitly model relationships among samples, which may affect results using original and new features.

  • Method

    The paper proposes RAE, extends relationship modeling to SAE, DAE, and VAE, and trains the model by iteratively adding layers and updating parameters with SGD.

  • Results

    3.8% classification error: RAE achieved the lowest error rate, followed by GAE at 5.7% and BAE at 8.9%.

  • Takeaways & Limitations

    Considering data relationships can decrease reconstruction loss, generate more robust features, and improve classification results on the evaluated datasets.

Abstract

from arXiv · show

Feature extraction becomes increasingly important as data grows high dimensional. Autoencoder as a neural network based feature extraction method achieves great success in generating abstract features of high dimensional data. However, it fails to consider the relationships of data samples which may affect experimental results of using original and new features. In this paper, we propose a Relation Autoencoder model considering both data features and their relationships. We also extend it to work with other major autoencoder models including Sparse Autoencoder, Denoising Autoencoder and Variational Autoencoder. The proposed relational autoencoder models are evaluated on a set of benchmark datasets and the experimental results show that considering data relationships can generate more robust features which achieve lower construction loss and then lower error rate in further classification compared to the other variants of autoencoders.

I. INTRODUCTION

High-dimensional data makes dimensionality reduction important for mitigating over-fitting and computational complexity while exposing intrinsic structure. The paper proposes RAE to combine feature reconstruction with relationships among data samples and extends it across autoencoder variants.

  • High-dimensional datasets can cause over-fitting, high computational complexity, overly complicated models, and the curse of dimensionality.
  • Feature extraction creates new features by combining original features, whereas feature selection retains only a subset and may lose information.
  • Projection-based methods reduce dimensionality but can change relationships among data samples, motivating relationship-aware dimensionality reduction.
  • Traditional autoencoders minimize reconstruction loss but do not explicitly model relationships among samples.
  • RAE minimizes losses for both data features and their relationships, filters weak relationships with activation functions, and extends the approach to SAE, DAE, and VAE.
  • The paper evaluates RAE against GAE and compares original autoencoders with their relationship-aware extensions on benchmark datasets.

II. LITERATURE REVIEW

Autoencoders learn low-dimensional representations by reconstructing inputs and can be stacked or made nonlinear. Their development includes variants addressing training, robustness, and representation-distribution concerns, while relationship modeling remains a distinct focus.

  • Autoencoders map original data into a low-dimensional representation and reconstruct the input by minimizing reconstruction loss.
  • Nonlinear activation functions allow autoencoders to learn more useful features than linear feature extraction methods, although training can take longer.
  • Improved deep-architecture training renewed interest in autoencoders, while stacked layers can introduce overfitting through increasingly influential deeper-layer weights.
  • Traditional autoencoders generate new features by minimizing reconstruction loss between X and X′.
  • Denoising autoencoders learn from corrupted inputs, whereas variational autoencoders target desired distributions for hidden-layer representations.
  • Contractive autoencoders increase sensitivity to slight variations, and maintaining relationships among samples is described as another way to achieve that sensitivity.

III. PRELIMINARIES

Feature extraction maps high-dimensional data to fewer features, while a basic autoencoder uses an encoder-decoder pair to learn and reconstruct a reduced representation. The relational formulation adds relationship reconstruction to the objective.

  • A. Feature Extraction: Feature extraction transforms data from a high-dimensional space into a relatively low-dimensional space, using linear or nonlinear mappings.
  • A. Feature Extraction: A relational autoencoder minimizes reconstruction loss while maintaining relationships among the original and reconstructed samples.
  • B. Basic Autoencoder: A traditional autoencoder generates reduced features Y by reconstructing the original dataset X as X′ and minimizing their difference.
  • B. Basic Autoencoder: The encoder maps input X to hidden representation Y, while the decoder maps Y back to reconstruction X′.
  • B. Basic Autoencoder: Nonlinear activation functions make the encoder or decoder nonlinear; identity activation yields linear projection or reconstruction.
  • B. Basic Autoencoder: Training optimizes parameters θ to minimize reconstruction loss, using squared error for linear reconstruction and cross-entropy for nonlinear reconstruction.

C. Stacked Autoencoder

Stacked autoencoders add hidden layers to generate different levels of features. Their encoding and decoding processes compose layer-specific functions and can be trained greedily layer by layer.

  • C. Stacked Autoencoder: Stacked autoencoders add hidden layers to generate different levels of new features representing the original data.
  • C. Stacked Autoencoder: The stacked encoder composes the encoding function f_i across layers, while the decoder composes corresponding decoding functions g_i.
  • C. Stacked Autoencoder: A stacked autoencoder can be trained using a greedy layerwise feed-forward approach.

IV. RELATIONAL AUTOENCODER (RAE)

RAE extends autoencoder feature extraction by reconstructing both data and relationships among samples. It uses similarity-based relationships, thresholds weak similarities, and trains the resulting model iteratively.

  • RAE minimizes reconstruction loss for both the input data and relationships among data samples.The relationship terms use R(X) and R(X′) to represent sample relationships before and after encoding and decoding.
  • Data relationships are represented by similarities computed as X multiplied by X^T.
  • A threshold t filters out weak and trivial relationships to improve computational efficiency.
  • RAE uses squared error as its loss function.
  • The training procedure initializes a neural network, adds configured layers, optimizes the RAE loss with SGD, and stops when loss improvement is at most ε.

A. Extension to Sparse Autoencoder (SAE)

The relational extension of Sparse Autoencoder incorporates data relationships while retaining weight-decay regularization to keep network weights small.

  • Sparse Autoencoder uses weight-decay regularization to limit increasing hidden-neuron weights in deeper layers.
  • Relational Sparse Autoencoder extends SAE by considering data relationships in its objective function.SAE’s weight-decay term constrains the weight matrix to have small elements.

B. Extension to Denoising Autoencoder (DAE)

Relational Denoising Autoencoder extends denoising training by reconstructing both corrupted data and relationships derived from the data. The corruption considered is additive isotropic Gaussian noise.

  • Denoising Autoencoder corrupts input features, reconstructs the destroyed input, and minimizes reconstruction loss against the original data.
  • Relational Denoising Autoencoder adds relationship reconstruction between XX^T and the corresponding corrupted relationship ˜X˜X^T.
  • The considered corruption is additive isotropic Gaussian noise, with ˜X = X + ∆.

C. Extension to Variational Autoencoder (VAE)

Variational Autoencoder differs from other autoencoders by assuming a distribution for latent neurons and minimizing divergence between posterior and latent distributions. Its encoding and decoding processes define the probabilistic reconstruction pathway.

  • VAE assumes a distribution for latent neurons and minimizes the difference between posterior and latent distributions using Kullback-Leibler divergence.
  • The relational VAE extension is defined after specifying the latent-distribution assumption and encoding-decoding processes.
  • qφ(Y|X) encodes input X into the probability of latent representation Y, while pθ(X|Y) decodes Y to reconstruct X.

V. DATASETS

The evaluation uses two image benchmarks: MNIST handwritten digits and CIFAR-10 labeled images. The experiments compare autoencoder models using these datasets.

  • MNIST contains 60,000 training examples and 10,000 test samples of handwritten digits.
  • CIFAR-10 contains 60,000 images across 10 classes, with 50,000 training and 10,000 test samples.
  • The first experiment compares RAE with BAE and GAE using reconstruction loss and classification accuracy.
  • Extended relational variants are compared with their corresponding original autoencoders to estimate the effects of modeling relationships.

A. Experiment Setting

The experiments use standardized autoencoder configurations and evaluate reconstruction and classification across MNIST and CIFAR-10. RAE performance is examined as the relationship-weighting parameter α varies.

  • Experiment Setting: All autoencoders use tied weights, sigmoid activations, and stochastic gradient descent for 400 epochs.The tied-weight configuration is W′ = W^T.
  • Experiment Setting: As network depth increases, layer widths decrease toward a predefined threshold l_t, with Xavier initialization used to improve training efficiency.The neuron-count rule is described using n_i and n_i+1 for adjacent layers.
  • Experiment Setting: Reconstruction loss is measured by Mean Squared Error, while classification uses softmax regression on extracted features with 10-fold cross validation.
  • Experiment Setting: α ranges from 0 to 1 in steps of 0.02 and controls the relative weights of data and relationship reconstruction.BAE and GAE lack α, so their reconstruction loss remains unchanged as α varies.
  • Experiment Setting: As α increases, RAE shifts from data reconstruction toward relationship reconstruction, with information loss eventually increasing when relationships are over-emphasized.
  • Experiment Setting: RAE achieves the lowest classification error rate at 3.8%, compared with 5.7% for GAE and 8.9% for BAE.

C. Comparing extend autoencoder variations to original ones

The study extends relational modeling to multiple autoencoder variants and compares each extension with its original counterpart. The reported results link relationship modeling to lower reconstruction loss and more robust features.

  • C. Comparing extend autoencoder variations to original ones: Considering data relationships decreases reconstruction loss relative to the corresponding original autoencoder variants.
  • C. Comparing extend autoencoder variations to original ones: The authors report that lower reconstruction loss produces more robust and meaningful features that contribute to improved classification results.
  • C. Comparing extend autoencoder variations to original ones: The extended variants are Relational Sparse Autoencoder, Relational Denoising Autoencoder, and Relational Variational Autoencoder.
Loading 1802.03145v1…