Source-linked AI summary
A Deep Learning Approach to Structured Signal Recovery
Ali Mousavi, Ankit B. Patel, Richard G. Baraniuk
TL;DR
The paper addresses structured signal recovery from under-sampled measurements, where conventional CS relies on handcrafted representations and computationally complex recovery algorithms. It introduces SDA-based deep learning with linear or mildly nonlinear measurements and learned representations, reporting improved recovery performance in certain cases and faster reconstruction than competing methods. The approach remains limited by conditional-independence assumptions in modeling natural images.
Problem
Structured signal recovery must reconstruct signals from under-sampled measurements, while CS uses handcrafted representations and computationally complex convex or greedy recovery methods.
Method
The paper uses stacked denoising autoencoders to learn structured representations from training data and recover signals from linear or mildly nonlinear measurements.
Results
The SDA methods improve recovery performance in certain cases and beat competing methods in reconstruction time without solving an optimization problem.
Takeaways & Limitations
Deep architectures can learn representations that outperform prespecified transformations and can provide efficient structured-signal recovery.
Takeaways & Limitations
GRBMs and denoising autoencoders treat input-image components as conditionally independent given the hidden layer, limiting natural-image modeling.
Abstract
from arXiv · showhide
In this paper, we develop a new framework for sensing and recovering structured signals. In contrast to compressive sensing (CS) systems that employ linear measurements, sparse representations, and computationally complex convex/greedy algorithms, we introduce a deep learning framework that supports both linear and mildly nonlinear measurements, that learns a structured representation from training data, and that efficiently computes a signal estimate. In particular, we apply a stacked denoising autoencoder (SDA), as an unsupervised feature learner. SDA enables us to capture statistical dependencies between the different elements of certain signals and improve signal recovery performance as compared to the CS approach.
A. Motivation
The paper frames structured signal recovery as an ill-posed inverse problem and proposes deep learning to learn representations and recover signals beyond conventional CS approaches.
- Problem: Structured signal recovery seeks x from under-sampled measurements y = Γ(x), with M ≪ N and sufficient signal structure.The operator may be linear or nonlinear; sparse signals with linear measurements are a central CS case.
- Problem: The paper organizes the challenge around recovering the signal, designing the measurement operator, and finding a representation that exposes structure.These questions are contrasted with CS solutions based on optimization or greedy algorithms, random matrices, and prespecified transformations.
- Motivation: The study evaluates deep neural networks for recovering structured signals, particularly images, from under-sampled random linear measurements.Its motivation is the success of deep architectures in image representation, and it compares the approach with state-of-the-art CS algorithms.
- Motivation: CS recovery replaces an NP-hard ℓ0 optimization with convex or greedy methods, often requiring hundreds of iterations.The paper proposes replacing these iterative procedures with a feed-forward deep neural network for faster recovery.
- Motivation: Prespecified bases such as wavelets and DCT are handcrafted and restrictive in capturing complex dependencies within signals.The paper therefore uses deep learning to automate representation learning from training examples and reports better dimensionality reduction representations than prespecified transformations.
- Novelty: The paper presents itself as an early study of structured signal recovery from under-sampled measurements using deep learning.Earlier deep-learning inverse-problem studies cited here focused on denoising, noisy-pattern removal, and super-resolution.
II. STACKED DENOISING AUTOENCODERS FOR STRUCTURED SIGNAL RECOVERY
The proposed recovery framework has two modes: fixed linear measurements following traditional CS and nonlinear adaptive measurements inspired by neural-network architectures.
- Evaluation: The proposed method is evaluated against state-of-the-art approaches from the CS framework.The comparison is conducted later using simulation results.
- Measurement paradigms: The method considers fixed linear measurements as the traditional CS measurement paradigm.This scenario uses the conventional measurement setting before introducing the nonlinear alternative.
- Measurement paradigms: It also introduces nonlinear adaptive compressive measurements inspired by neural-network architecture and capability.The paper states that incorporating measurement nonlinearity enhances overall recovery performance in its simulations.
A. SDA + Linear Measurement Paradigm
For linear measurements, the paper trains a three-layer SDA to map measurement vectors to original signals, using nonlinear feed-forward transformations and supervised paired data.
- Training setup: The linear paradigm represents measurements as y = Φx and learns a nonlinear mapping from measurement vectors to original signals.Training uses paired measurements and signals in Dtrain, with performance tested on paired examples in Dtest.
- Architecture: A three-layer SDA applies a nonlinearity to the affine transformation at each layer.The first hidden layer receives the measurement vector as input, and the paper uses the sigmoid nonlinearity.
- Architecture: The first layer computes its hidden representation with weight matrix W1 and bias vector b1, with computational cost O(MN).The paper notes that this matches the cost of one iteration of an iterative CS recovery algorithm.
- Architecture: The second hidden layer and output layer apply further nonlinear affine transformations to produce the estimate x̂.The output and network parameters define a nonlinear mapping used with mean squared error training.
- Optimization: Backpropagation minimizes the training loss, and Figure 1 depicts the SDA structure fed by linear measurements.The architecture is designed as a feed-forward recovery network rather than an iterative optimization procedure.
B. SDA + Nonlinear Measurement Paradigm
The nonlinear-measurement SDA adds the measurement mapping as an adaptive network layer, allowing the learned measurement structure to fit the training signal class.
- The nonlinear SDA treats mapping the original signal to measurements as an additional first layer.This extra layer allows the measurement structure to adapt to the training set.
- With sufficient natural-image data, the measurement matrix may adapt to the class of signals being acquired.
- The measurement nonlinearity can be sigmoid, another nonlinearity, or identity, supporting nonlinear and adapted linear measurements.The nonlinearity used for measurements may differ from those in other network layers.
- The nonlinear SDA uses four parameterized layers and outputs a reconstructed signal through MNL(x, ΩNL).
- Figure 2 presents the SDA architecture for the nonlinear-measurement setting.
III. PROBABILISTIC RELATION BETWEEN SDA AND COMPRESSIVE SENSING
The paper interprets SDA-based recovery probabilistically: layerwise denoising learns representations of the signal distribution, then feedforward reconstruction replaces explicit CS optimization.
- The network receives compressed measurements or original images at its first layer, then propagates latent representations through subsequent layers.
- Layerwise pre-training trains each denoising autoencoder to reconstruct its input from a noisy version before training the next layer.
- The SDA’s probabilistic interpretation connects autoencoder energy with the free energy of a corresponding Gaussian-visible, binary-hidden RBM.
- Training adjusts weights and biases so reconstruction error is small for signals drawn from the underlying distribution, not only the training set.
- Unlike CS formulation (1), a trained autoencoder reconstructs compressed data through feedforward computation without solving an optimization problem.
- After layerwise pre-training, supervised fine-tuning minimizes image reconstruction MSE from compressed measurements.
IV. SIMULATION RESULTS
The experiments compare structured-signal recovery methods using reconstruction quality and recovery speed, with PSNR as the quality measure.
- The simulation study compares the proposed framework with state-of-the-art CS recovery algorithms.
- The comparison evaluates both reconstruction quality through PSNR and recovery speed.
- The supplied passage describes the evaluation criteria but reports no numerical simulation outcome.
A. Implementation
The implementation trains networks on small image patches to control autoencoder complexity, then reconstructs large images from overlapping patches.
- Growing image size requires larger fully connected autoencoders, increasing backpropagation complexity and overfitting risk.
- The method therefore trains networks for small sub-images while reconstructing large images patch by patch.
- Testing uses overlapping sub-images and averaging to reconstruct larger images.
- Training uses 32 × 32 grayscale patches from ImageNet images, without overlap during training.
- Figures 3 and 4 depict overlapping sub-images and the test images used for Table I.
- Each layer is corrupted with zero-mean Gaussian noise of standard deviation 0.2 during denoising pre-training, followed by backpropagation fine-tuning.
B. Comparison with Other Methods
The comparison finds no universal reconstruction-quality winner: deep-learning methods perform especially well on irregular images, while D-AMP benefits from smooth images with similar patches. L-SDA and NL-SDA recover images far faster because they use feed-forward inference instead of optimization.
- Table I reports no obvious winner across 10 images, although L-SDA, NL-SDA, and O-NL-SDA outperform D-AMP and min-TV in some cases.
- Deep-learning methods outperform D-AMP on irregular images with few similar patches, whereas D-AMP performs better on smooth images with many similar patches.
- NL-SDA and O-NL-SDA outperform Tiled D-AMP in all but one case in Table I.
- At M/N = 0.4, reconstructed examples report PSNR values of 29.96 dB for SDA+Linear Measurements, 31.15 dB for SDA+Nonlinear Measurements, and 38.48 dB for D-AMP.
- L-SDA and NL-SDA are nearly 1,000,000 times faster than D-AMP at an under-sampling ratio of 0.4.
- For under-sampling ratios below 0.06, SDA-based methods outperform D-AMP in successful-recovery probability; at larger ratios, D-AMP performs better.
- At an under-sampling ratio of 0.06, NL-SDA begins outperforming D-AMP after 3.5 × 10^4 backpropagation iterations.
V. CONCLUSION AND FUTURE WORK
The paper concludes that stacked denoising autoencoders provide a learned framework for structured-signal recovery with linear or mildly nonlinear measurements. It identifies conditional independence in GRBMs as a limitation and calls for models that capture pixel relationships and scale to large images.
- The framework learns structured representations from training data, supports linear and mildly nonlinear measurements, and efficiently computes signal estimates.
- SDA captures statistical dependencies among signal elements and improves recovery performance compared with the CS approach.
- GRBMs conditionally treat input-image components as independent given the hidden layer, limiting their modeling of natural images.
- Future work should develop models that scale to large images and capture relationships between pixel intensities rather than assuming conditional independence.