PaperlayerView exact revision on arXiv

Source-linked AI summary

A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines

David Charte, Francisco Charte, Salvador García, María J. del Jesus, Francisco Herrera

arXiv:1801.01586v1cs.LGcs.NE

TL;DR

Increasing input dimensionality can degrade machine-learning performance, creating interest in compact, higher-level feature fusion and in choosing among many autoencoder variants. This paper reviews ANN-based autoencoders for nonlinear feature fusion, proposes a taxonomy, compares them with classical techniques, and supplies guidelines, software discussion, and case studies. Its practical scope is specifically ANN-based autoencoders, especially those intended for nonlinear feature fusion, with MNIST used to compare configurations.

  • Problem

    Increasing input dimensionality can degrade machine-learning performance, while numerous autoencoder variants support different applications and feature-fusion roles.

  • Method

    The paper reviews ANN-based autoencoders, proposes a taxonomy, compares them with classical feature-fusion techniques, and provides design guidelines, software discussion, and case studies.

  • Results

    The paper provides empirical MNIST comparisons across autoencoder configurations, including differing activation functions, optimizers, and models.

  • Takeaways & Limitations

    The review offers guidance for selecting an appropriate autoencoder model based on criteria including units, loss functions, and activation functions.

  • Takeaways & Limitations

    The review focuses on autoencoders whose basic model is an ANN, especially those aimed at nonlinear feature fusion, and assumes lossy compression is tolerable in compression scenarios.

Abstract

from arXiv · show

Many of the existing machine learning algorithms, both supervised and unsupervised, depend on the quality of the input characteristics to generate a good model. The amount of these variables is also important, since performance tends to decline as the input dimensionality increases, hence the interest in using feature fusion techniques, able to produce feature sets that are more compact and higher level. A plethora of procedures to fuse original variables for producing new ones has been developed in the past decades. The most basic ones use linear combinations of the original variables, such as PCA (Principal Component Analysis) and LDA (Linear Discriminant Analysis), while others find manifold embeddings of lower dimensionality based on non-linear combinations, such as Isomap or LLE (Linear Locally Embedding) techniques. More recently, autoencoders (AEs) have emerged as an alternative to manifold learning for conducting nonlinear feature fusion. Dozens of AE models have been proposed lately, each with its own specific traits. Although many of them can be used to generate reduced feature sets through the fusion of the original ones, there also AEs designed with other applications in mind. The goal of this paper is to provide the reader with a broad view of what an AE is, how they are used for feature fusion, a taxonomy gathering a broad range of models, and how they relate to other classical techniques. In addition, a set of didactic guidelines on how to choose the proper AE for a given task is supplied, together with a discussion of the software tools available. Finally, two case studies illustrate the usage of AEs with datasets of handwritten digits and breast cancer.

1. Introduction

The paper introduces autoencoders as nonlinear feature-fusion models within the broader landscape of dimensionality reduction and representation learning. It reviews their models, relationships to classical techniques, design guidelines, software, and illustrative experiments.

  • Autoencoders: Autoencoders encode input data into codifications and train decoders to reconstruct the inputs as closely as possible.Their encoder-decoder structure supports both coding and reconstruction within the same model.
  • Motivation: As input dimensionality grows, machine-learning behavior often degrades, motivating feature engineering and dimensionality-reduction methods.The paper attributes this degradation mainly to the curse of dimensionality.
  • Feature engineering: Feature construction and extraction complement feature selection because variables that are individually uninformative may become useful in combination.Feature selection is also described as an NP-hard combinatorial problem and may evaluate variables independently.
  • Feature engineering: Feature engineering includes selecting existing variables or automatically constructing new representations through transformations.The paper distinguishes automated feature learning from broader feature engineering and notes that expert choices may still be needed.
  • Feature fusion: Feature fusion combines variables to remove redundant and irrelevant information, including through manifold-learning and ANN-based methods.The paper presents autoencoders as models that can learn new representations for this purpose.
  • Paper scope: The paper contributes a global AE taxonomy, mathematical model descriptions, comparisons with classical feature-fusion methods, application reviews, design guidelines, software coverage, and case studies.The MNIST case study examines how different AE architectures and parameters affect results, with reproducible scripts provided.

2. Autoencoder essentials

Autoencoders are typically symmetric encoder–decoder networks trained to reconstruct inputs while learning constrained encodings. Their structure, activations, and restrictions determine whether they support nonlinear feature fusion and compact representations.

  • General structure: Autoencoders are trained to reconstruct their inputs while imposing restrictions that prevent simple copying through the network.They are commonly symmetric neural networks with the middle layer representing the input encoding.
  • General structure: An autoencoder maps input x to encoding y through an encoder and then to reconstruction r through a decoder.The reconstruction has the same dimension as x, while y may be higher- or lower-dimensional.
  • Activation functions: Activation functions shape nonlinear representations: linear undercomplete autoencoders with squared-error loss recover the principal components obtained by PCA.Sigmoid and hyperbolic-tangent functions are common, while ReLU can degrade reconstruction and SELU is presented as an alternative.
  • Network structure: An undercomplete autoencoder uses a lower-dimensional encoding to force compact feature fusion, while an overcomplete model requires additional restrictions to avoid learning the identity function.Properly restricted overcomplete autoencoders can also produce compact encodings.
  • Network structure: A shallow autoencoder has one hidden encoding layer, whereas a deep autoencoder has multiple hidden layers and may be trained layer by layer or as a deep network.The encoder and decoder generally contain matching layer counts, with units arranged in reverse order.
  • Taxonomy and scope: The paper’s taxonomy classifies autoencoders by the properties induced in their encodings and reviews approaches including regularization and noise tolerance for feature fusion.Its scope emphasizes ANN-based autoencoders whose objective is nonlinear characteristic fusion.

3. Autoencoders for feature fusion

Autoencoders learn nonlinear feature combinations by encoding inputs into latent representations and decoding them for reconstruction. Their training minimizes reconstruction loss, with regularization, optimization algorithms, and layer-wise initialization supporting useful models, especially deep ones.

  • Foundations: AEs learn useful representations by nonlinear feature fusion, usually mapping data into a lower-dimensional latent space.The encoded representation can be used to map new instances into latent feature space.
  • Foundations: A basic AE is a feed-forward neural network with encoder and decoder layers, commonly arranged symmetrically around hidden representations.Activation functions and weight matrices need not be identical between the two halves.
  • Objective function: The AE objective optimizes weights and biases to minimize a per-instance reconstruction loss over the training samples.Mean squared error is typical, while cross-entropy is preferred when inputs are binary or modeled as bits.
  • Training: SGD and variants such as AdaGrad, RMSProp, and Adam optimize AE parameters, while backpropagation computes the required gradients.L-BFGS and conjugate gradient are alternative optimizers not based on SGD.
  • Training: Weight decay limits parameter growth to improve generalization, and tied weights reduce parameters while preserving the desired architecture.Tied matrices can make training faster.
  • Training: Deep AEs can be initialized by greedily training successive shallow AEs, unrolling the encoder symmetrically, and fine-tuning the complete network.Each successive layer uses representations computed by forward passes through previously trained layers.

3.2. Regularization

Regularized autoencoders impose desired properties on learned representations, including sparsity and local stability. Sparse autoencoders penalize deviations from target activation rates, while contractive autoencoders penalize encoder sensitivity to input perturbations.

  • Overview: Basic AE encodings do not generally have special properties, so regularization adds penalties for undesired behaviors.The penalty is incorporated into the objective function.
  • Sparse autoencoder: Sparse representations keep most encoding values at or near zero, motivated partly by efficient coding behavior observed in simple visual-cortex cells.Sparse codes can remain meaningful even when overcomplete, unlike basic overcomplete AEs that may copy inputs.
  • Sparse autoencoder: Sparse AEs penalize the divergence between each unit’s desired average activation ρ and its observed mean activation ˆρi.The average activation ˆρi depends on encoder parameters and the training set.
  • Sparse autoencoder: With ρ = 0.2, the KL penalty is lowest near the target activation and grows rapidly as activation moves away, tending to infinity at 0 and 1.The figure characterizes the penalty as a function of a hidden unit’s average activation.
  • Contractive autoencoder: Contractive AEs penalize the encoder Jacobian’s Frobenius norm to make encodings less sensitive to small input changes.This encourages local invariance and can facilitate discovery of lower-dimensional manifold structures.
  • Contractive autoencoder: For a linear encoder, L2 weight decay induces contraction by maintaining small weights; nonlinear encoders can also contract through saturated hidden units.The contractive AE can be sampled by perturbing points along encoder-defined tangent directions.

3.3. Noise tolerance

Noise-tolerant autoencoders address the instability and reconstruction limitations of standard AEs on corrupted inputs. Denoising AEs corrupt inputs during training but reconstruct the originals, whereas robust AEs alter the loss to reduce sensitivity to non-Gaussian noise.

  • Motivation: Standard AEs do not guarantee stable latent features or noise removal when reconstructing new samples.Denoising and robust variants target these limitations.
  • Denoising autoencoder: Denoising AEs learn robust features by reconstructing partially destroyed inputs.Their latent feature spaces are intended to resist corrupted inputs beyond simple denoising applications.
  • Denoising autoencoder: During denoising training, selected input features are set to 0, but reconstruction error is computed against the original uncorrupted input.The network is therefore trained to infer missing values.
  • Denoising autoencoder: Denoising AEs can be overcomplete and require no additional restrictions to learn meaningful codes; after training, inputs are encoded without corruption.Layer-wise training is possible for denoising AEs with multiple hidden layers.
  • Denoising autoencoder: Alternative corruption processes include additive Gaussian noise and salt-and-pepper noise, which respectively offset components or replace them with minimum or maximum values.The corruption process is not limited to setting features to zero.
  • Robust autoencoder: Robust stacked AEs use a correntropy-based loss and are less affected by non-Gaussian noise than standard AEs.Correntropy is less affected by outliers than MSE, supporting greater resilience to non-Gaussian noise.

3.4. Domain specific autoencoders

Autoencoder variants adapt architecture, objectives, or latent distributions to images, sequences, generative modeling, relational structure, class separation, and hierarchical inputs. These models extend the basic encoder-decoder beyond generic feature fusion.

  • Data-specific models: Convolutional AEs replace fully connected layers with convolutions to account explicitly for the two-dimensional structure of image data.The convolution operation uses shared weights for forward processing and reconstruction.
  • Data-specific models: LSTM AEs encode sequences into fixed-size representations and decode them to reconstruct the original sequence, supporting large sequential data such as video.The decoder attempts to extract the sequence in inverse order.
  • Generative models: Variational AEs replace deterministic encoder and decoder functions with stochastic mappings to approximate latent-variable distributions.Because they support sampling from the learned distribution, applications commonly involve generating new instances.
  • Generative models: Adversarial AEs impose a prior distribution on encodings by jointly training an AE and a discriminator that distinguishes codes from prior samples.The encoder is trained to fool the discriminator, encouraging encodings to follow the imposed distribution.
  • Other architectures: Relational AEs reconstruct both samples and relationships, using a weighted sum of sample-reconstruction and relation-reconstruction errors.This differs from merely adding a penalty term for relationships.
  • Other architectures: Discriminative AEs use class information to gather positive samples and push negative samples away, with object detection as their main objective.They optimize a hinge loss and reconstruct positive instances better than negative ones.
  • Other architectures: Autoencoder trees, dual AEs, cross-correlation AEs, and recursive AEs modify the encoder-decoder structure for soft routing, dual representations, co-occurrences, or sentence sentiment.These architectures use decision trees, linked latent spaces, multiplicative combinations, or tree-like recursive encodings.

4. Comparison to other feature fusion techniques

AEs are positioned among linear, nonlinear, supervised, unsupervised, convex, and nonconvex feature-fusion methods. Their distinguishing advantages include learning nonlinear combinations, incorporating selected structural objectives, and mapping new instances after training.

  • AEs generally perform unsupervised, nonconvex feature fusion, whereas some alternatives are supervised or optimize convex objectives.
  • Linear approaches: Linear AEs minimizing mean quadratic error learn PCA components, while general AEs can learn nonlinear combinations and overcomplete representations.
  • Linear approaches: LDA seeks class-separating linear combinations under distributional assumptions, whereas AEs may encode broader meaningful information without those assumptions.
  • Nonlinear approaches: AEs usually optimize a global reconstruction measure rather than pairwise distances, although their objectives can be combined with MDS.
  • Nonlinear approaches: Isomap preserves neighborhood and geodesic distances, while LLE preserves local structure by reconstructing points from neighbors.
  • Nonlinear approaches: Contractive and denoising AEs can learn manifold-related representations, and unlike Isomap and LLE, AEs map new instances into latent space after training.

5. Applications in feature learning and beyond

AEs support feature learning across predictive, compression, anomaly-detection, hashing, and visualization tasks. Their applications span images, biomedical signals, language, time series, documents, and multimodal data, with reconstruction quality sometimes imposing a lossy-compression requirement.

  • AE feature fusion can support classification, regression, clustering, compression, anomaly detection, hashing, and visualization.
  • Classification: Stacked denoising AEs produce compact, higher-level features for classifying cancer nuclei in large, variable tissue images.
  • Classification: AE variants have been applied to multimodal video classification, multilabel classification, and semi-supervised text classification.
  • Data compression: AE compression is approximate, so it is useful when lossy compression is permissible; applications include images, biometric signals, language, and high-resolution time series.
  • Abnormal-pattern detection: Basic and denoising AEs have been compared with PCA and Kernel PCA for spacecraft anomaly detection, while other AEs target outliers and fake accounts.
  • Hashing: Semantic hashing uses binary encoding units as document addresses, and related binary AEs support fast similar-image search.

6. Guidelines, software and examples on autoencoder design

The paper offers design guidelines, software guidance, and MNIST examples for selecting AE architectures, objectives, regularizations, and training choices. The examples show that encoding size, optimizer, activation, and model type materially affect reconstruction and latent representations.

  • Guidelines: AE design choices include architecture, encoding length, activations, loss function, regularization, and combinations of model modifications.
  • Architecture: Very small encodings may prevent successful shallow-AE training, whereas larger encodings improve reconstruction accuracy; convolutional layers suit images and LSTM components suit sequential data.
  • Activations and loss function: Loss and activation should be matched: linear or ReLU outputs suit mean squared error, while logistic outputs suit cross-entropy with normalized data.
  • Regularizations: Weight decay can reduce overfitting, sparse coding can increase structural flexibility, and contraction regularization can help when data lie on a lower-dimensional manifold.
  • Software: The paper surveys open-source frameworks including TensorFlow, Caffe, Torch, MXNet, and Keras for constructing and training AEs.
  • Comparison of optimizers: Adam achieved the best objective-function results among the compared optimizers, closely followed by AdaGrad, with slightly faster convergence also observed for Adam.
  • Comparison of optimizers: SGD failed to recover digits, AdaGrad produced relatively poor reconstructions, and Adam and RMSProp performed better with little difference between them.
  • Comparison of activation functions: Sigmoid reconstructions were poor, tanh was closer to the inputs, and ReLU and SELU were slightly sharper in this normalized image setting.

7. Conclusions

The paper reviews AEs as nonlinear feature-fusion tools, proposes a taxonomy, compares them with classical methods, and surveys broader applications. It concludes with selection guidelines, software resources, and empirical comparisons across MNIST configurations, illustrated alongside case studies.

  • The paper frames nonlinear automated feature fusion as a way to combine features into representations used by machine learning models.
  • The paper introduces AE foundations, proposes a taxonomy based on inferred-model properties, and details feature-fusion models in comparison with classical techniques.
  • It also lists AE variants and applications beyond feature fusion, reflecting the growing number of proposals based on fundamental AE models.
  • Guidelines cover units, loss and activation functions, model selection, software, and empirical MNIST comparisons across configurations, optimizers, and models.

Appendix A.1. Breast Cancer Diagnosis (Wisconsin)

The Wisconsin breast cancer dataset provides a two-dimensional visualization example, but its class imbalance and limited sample size create considerations for autoencoder training and generalization.

  • The dataset is used briefly to illustrate a two-dimensional visualization generated by a basic autoencoder.
  • WDBC contains 569 patient instances with 30 numeric input features and two tumor classes, benign and malignant.The malignant class represents 37.3% of instances and the benign class 62.7%.
  • The 30 attributes summarize ten cell-nucleus traits using their mean, standard error, and largest value.
  • Class imbalance may bias the autoencoder toward reconstructing benign instances more accurately, while the small dataset may limit generalization for some deep learning techniques.Benign instances contribute more to the loss function because they are more numerous.

Appendix A.2. MNIST

MNIST is a standard deep-learning benchmark consisting of normalized grayscale handwritten-digit images, with substantial training data that supports higher-level feature extraction.

  • MNIST is widely used to benchmark deep-learning techniques and neural architectures because higher-level features are reasonably easy to extract from its simple images.
  • MNIST contains 60,000 handwritten-digit images, split into 50,000 training instances and 10,000 test instances.
  • Each image is a 28x28 grayscale input with 784 variables, whose pixel values are rescaled to the [0, 1] interval.
  • State-of-the-art work on MNIST achieves an error rate of around 0.2%.
Loading 1801.01586v1…