PaperlayerView exact revision on arXiv

Source-linked AI summary

Concrete Autoencoders for Differentiable Feature Selection and Reconstruction

Abubakar Abid, Muhammad Fatih Balin, James Zou

arXiv:1901.09346v2cs.LGstat.ML

TL;DR

Feature selection must identify original measurements that can replace expensive or redundant full-feature collection. The concrete autoencoder makes subset selection differentiable while learning a decoder for reconstruction, and it generally outperforms prior methods across datasets, including a large gene-expression setting.

  • Problem

    Feature selection seeks informative original features because measuring all features can be expensive or difficult, whereas dimensionality reduction does not directly eliminate redundant measurements.

  • Method

    The concrete autoencoder uses a Concrete selector layer with annealed temperature as an encoder and a neural-network decoder to select discrete features and reconstruct inputs.

  • Results

    Concrete autoencoders generally outperform state-of-the-art methods for feature selection and data reconstruction across varied datasets.

  • Takeaways & Limitations

    On a gene-expression dataset with more than 100,000 samples and 10,000 features, selected features outperformed the state-of-the-art gene subset.

  • Takeaways & Limitations

    Selected features do not provide p-values or statistical significance quantification and should be validated through hypothesis testing or domain knowledge.

Abstract

from arXiv · show

We introduce the concrete autoencoder, an end-to-end differentiable method for global feature selection, which efficiently identifies a subset of the most informative features and simultaneously learns a neural network to reconstruct the input data from the selected features. Our method is unsupervised, and is based on using a concrete selector layer as the encoder and using a standard neural network as the decoder. During the training phase, the temperature of the concrete selector layer is gradually decreased, which encourages a user-specified number of discrete features to be learned. During test time, the selected features can be used with the decoder network to reconstruct the remaining input features. We evaluate concrete autoencoders on a variety of datasets, where they significantly outperform state-of-the-art methods for feature selection and data reconstruction. In particular, on a large-scale gene expression dataset, the concrete autoencoder selects a small subset of genes whose expression levels can be use to impute the expression levels of the remaining genes. In doing so, it improves on the current widely-used expert-curated L1000 landmark genes, potentially reducing measurement costs by 20%. The concrete autoencoder can be implemented by adding just a few lines of code to a standard autoencoder.

1. Introduction

Feature selection reduces data collection by identifying informative original features, but standard dimensionality reduction does not directly remove redundant measurements. The concrete autoencoder addresses this gap with differentiable subset selection and reconstruction, outperforming prior methods across datasets.

  • Feature selection can eliminate irrelevant or redundant measurements when collecting the full feature set is expensive or difficult.
  • Standard dimensionality-reduction methods preserve information in fewer dimensions but do not select original features for directly reducing experimental costs.
  • The concrete autoencoder performs end-to-end feature subset selection and imputation using deep autoencoders and a Concrete-distribution relaxation.
  • 20 of 784 MNIST pixels were selected unsupervised, and the decoder generally identified digits while capturing stylistic details such as the swirl in “2”.
  • The method generally outperformed state-of-the-art methods for feature selection and data reconstruction across varied datasets.
  • Related Works: Embedded methods incorporate feature selection into another algorithm’s learning phase, whereas regularization-based unsupervised methods include UDFS, MCFS, and AEFS.

2. Problem Formulation

The problem is to choose a fixed-size subset of features and learn a reconstruction function that minimizes reconstruction loss. The discrete subset search is combinatorial and can be NP-hard, while decoder complexity also affects both error and feature choice.

  • Global feature selection chooses k features from d-dimensional data and learns a function mapping the selected features to a reconstruction of all d features.
  • With n samples, the objective becomes selecting k columns of X whose submatrix can reconstruct the original data matrix.
  • The discrete optimization over feature subsets grows exponentially with d and is NP-hard even for linear-regression reconstruction functions.
  • More expressive nonlinear reconstruction functions can lower reconstruction error but may make optimization more difficult.
  • Mean-squared reconstruction error is used because it is smooth and differentiable and serves as a proxy for clustering and classification analyses.

3. Proposed Method

The concrete autoencoder replaces a standard encoder with k Concrete-based selector nodes and combines them with a neural-network decoder to learn discrete features for reconstruction. Temperature annealing enables exploration of feature combinations early in training and convergence toward informative individual features later.

  • The concrete autoencoder adapts a standard autoencoder by replacing its encoder with a k-node Concrete selector layer for discrete feature selection.The decoder remains a user-configurable neural network.
  • As temperature approaches zero, each selector node converges from a weighted feature combination to one individual input feature.At test time, the selected feature is the input dimension with the largest corresponding α value.
  • Each selector node samples a continuous relaxation of a one-hot vector, enabling differentiation through the feature-selection parameters.The relaxation is controlled by temperature T, and reparametrization permits gradients through the Concrete variables.
  • During training, the decoder reconstructs the input from selector outputs using the same neural-network architecture used at test time.The encoder has one neuron for each feature to be selected, while the decoder computes x̂ = fθ(u).
  • A high temperature supports stochastic exploration, whereas a low fixed temperature can cause poor local minima; annealing balances both regimes.The schedule exponentially decays temperature from T0 to TB across training epochs, supporting exploration early and convergence later.

4. Experiments

Experiments compare concrete autoencoders with unsupervised feature-selection baselines across public datasets, evaluating reconstruction and classification on hold-out data. The method generally achieves the lowest reconstruction error, while its selector parameters also expose related feature groups; in gene-expression inference, it improves imputation and can match landmark-gene error with fewer genes.

  • Annealing schedules: The annealing schedule strongly affected MNIST feature selection: exponential decay reached a suitable minimum, whereas high, low, or abrupt schedules produced larger or suboptimal errors.The experiment used k = 20 selected features and tracked temperature, selector convergence, and validation reconstruction error.
  • Experimental setup: Experiments compare concrete autoencoders with UDFS, MCFS, AEFS, PFA, and Laplacian score on multiple public datasets.The evaluation uses train, validation, and test splits, with test-set reconstruction error and classification accuracy as metrics.
  • Nonlinear decoders: The concrete autoencoder consistently achieved the lowest reconstruction error and high classification accuracy with nonlinear decoders on ISOLET.Reconstruction networks were trained for 200 epochs, with hidden-layer size selected using validation accuracy.
  • Linear decoders: With linear regression reconstruction, the concrete autoencoder had the lowest reconstruction error on almost all datasets and generally high classification accuracy.The comparison used selected features and a standard unregularized linear regressor evaluated on a hold-out test set.
  • Interpreting related features: Concrete selector parameters can identify related features, as shown by spatially clustered top-3 pixel groups for each of 20 MNIST selector nodes.The authors suggest applying the same analysis to correlated genes in gene-expression datasets.
  • L1000 gene-expression case study: About 3% lower gene-expression imputation error was obtained than with 943 L1000 landmark genes across reconstruction architectures.Using linear regression, approximately 750 CAE-selected genes achieved landmark-gene-level MSE, reducing measured genes by roughly 20%.

5. Discussion

The discussion presents concrete autoencoders as scalable, general differentiable feature selectors that outperform competing methods while remaining easy to implement. It also identifies the lack of statistical significance measures as a limitation requiring validation through additional analysis.

  • Method: Concrete autoencoders use Concrete random variables and reparametrization to backpropagate through stochastic discrete feature selection.The stochastic selector explores subsets of a specified size while minimizing the chosen loss.
  • Novelty: They differ from competing methods that use regularization to encourage sparse feature selection.
  • Empirical results: Across six public datasets, concrete autoencoders outperformed many complex feature-selection methods for reconstruction and classification.This pattern persisted with a single linear decoder, indicating usefulness beyond nonlinear reconstruction networks.
  • Extensions and scalability: Concrete autoencoders scale to datasets with many samples or high dimensionality and can incorporate supervised losses or feature-specific costs.A gene-expression dataset contained more than 100,000 samples and 10,000 features; cost terms can trade accuracy against measurement expense.
  • Usability: Implementing the architecture requires modifying only a few lines of standard autoencoder code, with runtime similar to a standard autoencoder.
  • Limitation: Concrete autoencoders do not provide p-values or statistical significance quantification, so selected features should undergo hypothesis testing or domain-specific validation.

A. Selected Features for Single Classes in MNIST

The experiments apply concrete autoencoders to single-digit MNIST subsets, selecting ten informative pixels for each subset. Figure 7 specifically concerns images of the digit 0.

  • Experiment setup: The single-class MNIST experiments select k = 10 features for each digit subset.
  • Digit 0: For digit 0 images, Figure 7 visualizes the ten most informative pixels selected by a concrete autoencoder.
  • Reference: The figure description directs readers to the main paper’s Figure 1 for additional information.

B. Pseudocode for a Concrete Autoencoder

The pseudocode trains a concrete autoencoder by selecting k features, reconstructing the full input with a decoder, and updating selector and decoder parameters through backpropagation and reparametrization.

  • Training inputs: Training takes a dataset X, feature count k, decoder fθ, epoch count B, learning rate λ, and initial and final temperatures.
  • Initialization: The algorithm initializes selector parameters α and decoder parameters θ before iterating over training epochs.
  • Feature selection: It forms XS by horizontally concatenating the k selected feature columns from X.
  • Optimization: The loss compares fθ(XS) with X, and gradients update θ by backpropagation and α through the reparametrization trick.
  • Outputs and use: Training returns the learned reconstruction function and trained Concrete parameters for later feature selection and imputation.

C. Classification Accuracies for Feature Selection Methods with Linear Reconstruction

The experiments compare concrete autoencoders with linear decoders against other feature-selection methods using linear reconstruction, selected features, and test-set classification accuracy.

  • Experimental protocol: Each method selects k = 50 features, then uses a standard unregularized linear regressor to impute the original features.
  • Results: Concrete autoencoders generally achieve the lowest reconstruction error and high, though not always highest, classification accuracy.
  • Classification evaluation: Table 2 reports test-set classification accuracies across six public datasets, using a one-hidden-layer neural-network reconstruction setup.For the mice protein dataset, the selection size is k = 10 rather than k = 50.

D. Examples of Feature Groups in MNIST Digits

Concrete autoencoders select informative pixel groups in MNIST, with single-digit patterns revealing correlated pixels that can indicate stroke direction.

  • Single-class digit selections form correlated pixel groups that are more likely to belong to the same stroke.These patterns are especially striking when the autoencoder is trained on individual digit classes.
  • k = 10 informative pixel groups were selected for images of digit 0 in MNIST.The figure compares these results with Fig. 5 in the main paper and includes panels for several digits.

E. Architecture for GEO Dataset Experiments

The GEO experiments use large reconstruction networks and train the concrete autoencoder with a linear decoder while annealing the selector temperature.

  • GEO reconstruction networks used 9000 hidden-layer neurons, dropout rate 0.1, 100 epochs, batch size 256, and a learning rate decayed by 0.95 each epoch.The initial learning rate was 0.001.
  • The concrete autoencoder used a linear decoder and was trained for 5000 epochs with temperature decreasing from 10 to 0.01.

F. Supervised Concrete Autoencoders

Supervised concrete autoencoders replace reconstruction with classification: selected features feed a classifier trained by cross-entropy, but MNIST imputation is generally worse than direct reconstruction training.

  • The supervised variant replaces the decoder’s reconstruction network with a neural network classifier.Its training pseudocode remains similar to that of the standard concrete autoencoder.
  • The classifier receives k selected features formed by concatenating the corresponding columns of the training feature matrix.
  • Training minimizes cross-entropy between true labels and classifier logits, updating classifier and Concrete parameters by backpropagation and reparametrization gradients.The procedure returns the trained classifier and Concrete parameters.
  • Supervised MNIST training produced representative imputations, but imputation quality was generally worse than when optimizing reconstruction error directly.
  • k = 20 pixels were selected from the 784 MNIST pixels for supervised experiments.The figure shows the selected features and input images with only those features retained.
Loading 1901.09346v2…