PaperlayerView exact revision on arXiv

Source-linked AI summary

A Deep Learning Approach to Structured Signal Recovery

Ali Mousavi, Ankit B. Patel, Richard G. Baraniuk

arXiv:1508.04065v1cs.LGstat.ML

TL;DR

The paper addresses structured signal recovery from under-sampled measurements, where conventional CS relies on handcrafted representations and computationally complex recovery algorithms. It introduces SDA-based deep learning with linear or mildly nonlinear measurements and learned representations, reporting improved recovery performance in certain cases and faster reconstruction than competing methods. The approach remains limited by conditional-independence assumptions in modeling natural images.

  • Problem

    Structured signal recovery must reconstruct signals from under-sampled measurements, while CS uses handcrafted representations and computationally complex convex or greedy recovery methods.

    Source: A. Motivation · p. 2

  • Method

    The paper uses stacked denoising autoencoders to learn structured representations from training data and recover signals from linear or mildly nonlinear measurements.

    Source: §II · p. 2, B. SDA + Nonlinear Measurement Paradigm · p. 3, §V · p. 7

  • Results

    The SDA methods improve recovery performance in certain cases and beat competing methods in reconstruction time without solving an optimization problem.

    Source: B. Comparison with Other Methods · p. 6, §V · p. 7

  • Takeaways & Limitations

    Deep architectures can learn representations that outperform prespecified transformations and can provide efficient structured-signal recovery.

    Source: A. Motivation · p. 2, B. Comparison with Other Methods · p. 6

    GRBMs and denoising autoencoders treat input-image components as conditionally independent given the hidden layer, limiting natural-image modeling.

    Source: §V · p. 7

Abstract

from arXiv · show

In this paper, we develop a new framework for sensing and recovering structured signals. In contrast to compressive sensing (CS) systems that employ linear measurements, sparse representations, and computationally complex convex/greedy algorithms, we introduce a deep learning framework that supports both linear and mildly nonlinear measurements, that learns a structured representation from training data, and that efficiently computes a signal estimate. In particular, we apply a stacked denoising autoencoder (SDA), as an unsupervised feature learner. SDA enables us to capture statistical dependencies between the different elements of certain signals and improve signal recovery performance as compared to the CS approach.

A. Motivation

The paper frames structured signal recovery as an ill-posed inverse problem and proposes deep learning to learn representations and recover signals beyond conventional CS approaches.

Problem

  • Structured signal recovery seeks x from under-sampled measurements y = Γ(x), with M ≪ N and sufficient signal structure.The operator may be linear or nonlinear; sparse signals with linear measurements are a central CS case.Source: A. Motivation · p. 1
  • The paper organizes the challenge around recovering the signal, designing the measurement operator, and finding a representation that exposes structure.These questions are contrasted with CS solutions based on optimization or greedy algorithms, random matrices, and prespecified transformations.Source: A. Motivation · p. 1

Motivation

  • The study evaluates deep neural networks for recovering structured signals, particularly images, from under-sampled random linear measurements.Its motivation is the success of deep architectures in image representation, and it compares the approach with state-of-the-art CS algorithms.Source: A. Motivation · p. 1
  • CS recovery replaces an NP-hard ℓ0 optimization with convex or greedy methods, often requiring hundreds of iterations.The paper proposes replacing these iterative procedures with a feed-forward deep neural network for faster recovery.Source: A. Motivation · p. 2
  • Prespecified bases such as wavelets and DCT are handcrafted and restrictive in capturing complex dependencies within signals.The paper therefore uses deep learning to automate representation learning from training examples and reports better dimensionality reduction representations than prespecified transformations.Source: A. Motivation · p. 2

Novelty

  • The paper presents itself as an early study of structured signal recovery from under-sampled measurements using deep learning.Earlier deep-learning inverse-problem studies cited here focused on denoising, noisy-pattern removal, and super-resolution.Source: A. Motivation · p. 2

II. STACKED DENOISING AUTOENCODERS FOR STRUCTURED SIGNAL RECOVERY

The proposed recovery framework has two modes: fixed linear measurements following traditional CS and nonlinear adaptive measurements inspired by neural-network architectures.

Evaluation

  • The proposed method is evaluated against state-of-the-art approaches from the CS framework.The comparison is conducted later using simulation results.Source: §II · p. 2

Measurement paradigms

  • The method considers fixed linear measurements as the traditional CS measurement paradigm.This scenario uses the conventional measurement setting before introducing the nonlinear alternative.Source: §II · p. 2
  • It also introduces nonlinear adaptive compressive measurements inspired by neural-network architecture and capability.The paper states that incorporating measurement nonlinearity enhances overall recovery performance in its simulations.Source: §II · p. 2

A. SDA + Linear Measurement Paradigm

For linear measurements, the paper trains a three-layer SDA to map measurement vectors to original signals, using nonlinear feed-forward transformations and supervised paired data.

Training setup

  • The linear paradigm represents measurements as y = Φx and learns a nonlinear mapping from measurement vectors to original signals.Training uses paired measurements and signals in Dtrain, with performance tested on paired examples in Dtest.Source: A. SDA + Linear Measurement Paradigm · pp. 2–3

Architecture

  • A three-layer SDA applies a nonlinearity to the affine transformation at each layer.The first hidden layer receives the measurement vector as input, and the paper uses the sigmoid nonlinearity.Source: A. SDA + Linear Measurement Paradigm · p. 3
  • The first layer computes its hidden representation with weight matrix W1 and bias vector b1, with computational cost O(MN).The paper notes that this matches the cost of one iteration of an iterative CS recovery algorithm.Source: A. SDA + Linear Measurement Paradigm · p. 3
  • The second hidden layer and output layer apply further nonlinear affine transformations to produce the estimate x̂.The output and network parameters define a nonlinear mapping used with mean squared error training.Source: A. SDA + Linear Measurement Paradigm · p. 3

Optimization

  • Backpropagation minimizes the training loss, and Figure 1 depicts the SDA structure fed by linear measurements.The architecture is designed as a feed-forward recovery network rather than an iterative optimization procedure.Source: A. SDA + Linear Measurement Paradigm · p. 3

B. SDA + Nonlinear Measurement Paradigm

The nonlinear-measurement SDA adds the measurement mapping as an adaptive network layer, allowing the learned measurement structure to fit the training signal class.

III. PROBABILISTIC RELATION BETWEEN SDA AND COMPRESSIVE SENSING

The paper interprets SDA-based recovery probabilistically: layerwise denoising learns representations of the signal distribution, then feedforward reconstruction replaces explicit CS optimization.

  • The network receives compressed measurements or original images at its first layer, then propagates latent representations through subsequent layers.Source: §III · p. 3
  • Layerwise pre-training trains each denoising autoencoder to reconstruct its input from a noisy version before training the next layer.Source: §III · p. 4
  • The SDA’s probabilistic interpretation connects autoencoder energy with the free energy of a corresponding Gaussian-visible, binary-hidden RBM.Source: §III · p. 4
  • Training adjusts weights and biases so reconstruction error is small for signals drawn from the underlying distribution, not only the training set.Source: §III · p. 4
  • Unlike CS formulation (1), a trained autoencoder reconstructs compressed data through feedforward computation without solving an optimization problem.Source: §III · pp. 4–5
  • After layerwise pre-training, supervised fine-tuning minimizes image reconstruction MSE from compressed measurements.Source: §III · p. 5

IV. SIMULATION RESULTS

The experiments compare structured-signal recovery methods using reconstruction quality and recovery speed, with PSNR as the quality measure.

  • The simulation study compares the proposed framework with state-of-the-art CS recovery algorithms.Source: §IV · p. 5
  • The comparison evaluates both reconstruction quality through PSNR and recovery speed.Source: §IV · p. 5
  • The supplied passage describes the evaluation criteria but reports no numerical simulation outcome.Source: §IV · p. 5

A. Implementation

The implementation trains networks on small image patches to control autoencoder complexity, then reconstructs large images from overlapping patches.

  • Growing image size requires larger fully connected autoencoders, increasing backpropagation complexity and overfitting risk.Source: A. Implementation · p. 5
  • The method therefore trains networks for small sub-images while reconstructing large images patch by patch.Source: A. Implementation · p. 5
  • Testing uses overlapping sub-images and averaging to reconstruct larger images.Source: A. Implementation · p. 5
  • Training uses 32 × 32 grayscale patches from ImageNet images, without overlap during training.Source: A. Implementation · p. 5
  • Figures 3 and 4 depict overlapping sub-images and the test images used for Table I.Source: Figure 3 · p. 5, Figure 4 · p. 5
  • Each layer is corrupted with zero-mean Gaussian noise of standard deviation 0.2 during denoising pre-training, followed by backpropagation fine-tuning.Source: A. Implementation · p. 5

B. Comparison with Other Methods

The comparison finds no universal reconstruction-quality winner: deep-learning methods perform especially well on irregular images, while D-AMP benefits from smooth images with similar patches. L-SDA and NL-SDA recover images far faster because they use feed-forward inference instead of optimization.

V. CONCLUSION AND FUTURE WORK

The paper concludes that stacked denoising autoencoders provide a learned framework for structured-signal recovery with linear or mildly nonlinear measurements. It identifies conditional independence in GRBMs as a limitation and calls for models that capture pixel relationships and scale to large images.

  • The framework learns structured representations from training data, supports linear and mildly nonlinear measurements, and efficiently computes signal estimates.Source: §V · p. 7
  • SDA captures statistical dependencies among signal elements and improves recovery performance compared with the CS approach.Source: §V · p. 7
  • GRBMs conditionally treat input-image components as independent given the hidden layer, limiting their modeling of natural images.Source: §V · p. 7
  • Future work should develop models that scale to large images and capture relationships between pixel intensities rather than assuming conditional independence.Source: §V · p. 7

Related papers

Loading 1508.04065v1…