PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Compressed Sensing

Yan Wu, Mihaela Rosca, Timothy Lillicrap

arXiv:1905.06723v2cs.LGeess.SPstat.ML

TL;DR

Compressed sensing is constrained by sparse-signal assumptions and slow reconstruction, even though generative models relax sparsity while retaining costly optimization. The paper introduces Deep Compressed Sensing, jointly training measurement and generation functions with reconstruction optimization via meta-learning, and derives GAN and semi-supervised GAN variants. It reports improved reconstruction performance and speed, plus GAN improvements and semantically meaningful latent spaces.

  • Problem

    Compressed sensing is limited by its sparsity assumption and costly reconstruction, while generative compressed sensing removes sparsity but remains slow.

  • Method

    Deep Compressed Sensing jointly trains measurement and generation functions and the latent reconstruction optimization process via meta-learning, extending measurement objectives to GAN variants.

  • Results

    The framework improves reconstruction performance and speed, includes a GAN model with improved performance, and yields semantically meaningful latent spaces in semi-supervised GANs.

  • Takeaways & Limitations

    GANs can arise as a special case of the measurement-error framework, enabling discriminator-guided latent optimization and extensions to semi-supervised GANs.

  • Takeaways & Limitations

    Generative compressed sensing typically requires hundreds or thousands of gradient-descent steps and several restarts, and random measurement matrices can be sub-optimal for highly structured signals.

Abstract

from arXiv · show

Compressed sensing (CS) provides an elegant framework for recovering sparse signals from compressed measurements. For example, CS can exploit the structure of natural images and recover an image from only a few random measurements. CS is flexible and data efficient, but its application has been restricted by the strong assumption of sparsity and costly reconstruction process. A recent approach that combines CS with neural network generators has removed the constraint of sparsity, but reconstruction remains slow. Here we propose a novel framework that significantly improves both the performance and speed of signal recovery by jointly training a generator and the optimisation process for reconstruction via meta-learning. We explore training the measurements with different objectives, and derive a family of models based on minimising measurement errors. We show that Generative Adversarial Nets (GANs) can be viewed as a special case in this family of models. Borrowing insights from the CS perspective, we develop a novel way of improving GANs using gradient information from the discriminator.

1. Introduction

Compressed sensing is flexible and sample efficient but is limited by sparse-signal assumptions and slow reconstruction. Deep Compressed Sensing jointly trains measurement, generation, and reconstruction processes, while also yielding GAN variants and faster, more accurate recovery.

  • Motivation: Compressed sensing separates encoding and decoding into measurement and reconstruction, enabling flexible, sample-efficient recovery from low-dimensional measurements.It has been applied to noisy or expensive measurements such as MRI and single-pixel imaging.
  • Motivation: CS application is hindered by the sparsity assumption and slow reconstruction, despite generative models relaxing the sparsity requirement.The generative-model approach improved performance over Lasso but still required costly optimization.
  • Contributions: Deep Compressed Sensing trains deep neural networks within the compressed-sensing framework for both measurement and reconstruction.The framework jointly trains a generator and the optimization process used for online reconstruction.
  • Contributions: Meta-learned reconstruction is more accurate and orders of magnitudes faster than previous models.The approach targets both reconstruction quality and the cost of iterative optimization.
  • Contributions: The framework derives GAN training from measurement-error minimization and uses discriminator-guided latent optimization to improve GAN performance.The non-saturated generator loss −ln (D(G(z))) emerges as a measurement error, and the framework extends to semi-supervised GANs with semantically meaningful latent spaces.

2. Background

Compressed sensing reconstructs signals from low-dimensional measurements, but classical CS requires sparsity and computationally intensive optimisation. Generative models relax sparsity through latent-space reconstruction, yet reconstruction remains slow and typically relies on random measurements.

  • 2.1. Compressed Sensing: Classical CS recovers sparse signals from under-determined measurements using random matrices whose projections preserve distances with high probability.The Restricted Isometry Property supports accurate reconstruction by minimising measurement error under a sparsity constraint.
  • 2.2. Compressed Sensing using Generative Models: Classical reconstruction is computationally intensive, while generative-model CS reconstructs by optimising a latent representation z and setting the signal to Gθ(z).The latent-space formulation replaces direct optimisation of the signal with optimisation over the generator’s representation.
  • 2.1. Compressed Sensing: Sparsity assumptions restrict CS to signals or representations that are sparse in known bases, which cannot adapt to arbitrary data distributions.Fourier and wavelet bases only partially relax the restriction because their domains must already be known to be sparse.
  • 2.2. Compressed Sensing using Generative Models: Generative-model CS replaces sparsity with a pretrained generator whose architecture and data-adapted weights constrain outputs to a low-dimensional manifold.This constraint supports low reconstruction error with high probability under a generalised Set-Restricted Eigenvalue Condition.
  • 2.2. Compressed Sensing using Generative Models: Latent reconstruction remains slow because the non-convex optimisation commonly requires hundreds or thousands of gradient steps and several random restarts.This runtime burden motivates training the optimisation procedure itself with meta-learning.
  • 2.2. Compressed Sensing using Generative Models: CSGM also relies on random measurement matrices, which can be sub-optimal for highly structured signals such as natural images.The background notes that learned measurements can perform significantly better.

3. Deep Compressed Sensing

Deep Compressed Sensing trains neural measurement and reconstruction functions together, using meta-learning to make latent reconstruction faster and more accurate. The framework also derives GAN variants by defining measurement objectives beyond the RIP.

  • Meta-learning: Meta-learning trains the latent optimisation procedure by back-propagating through gradient descent steps.The online updates operate over latent variables rather than model parameters, which are usually fewer and therefore quicker to update.
  • Meta-learning: Only 3-5 gradient descent steps were sufficient to improve upon baseline methods.The efficiency gain comes from implicitly performing second-order optimisation through the latent optimisation steps.
  • Measurement learning: Training the generator with only measurement error can produce a null-space solution, so DCS enforces the RIP through a measurement loss.The training samples both real data and generated samples so the learned distance-preserving property applies to both.
  • Framework: DCS learns both measurement and reconstruction functions as deep neural networks, rather than relying on a fixed random projection.The measurement function is parameterized as Fφ, while the generator Gθ reconstructs signals from measurements.
  • GAN extensions: DCS extends measurement objectives beyond the RIP, including an adversarial objective that yields a GAN formulation.When latent optimisation is disabled, the resulting model is identical to a vanilla GAN; additional latent optimisation significantly improved GAN training in experiments.
  • GAN extensions: The semi-supervised extension replaces the binary discriminator with a K + 1-class classifier, reserving one class for generated data.When T > 0, latent optimisation moves representations toward regions associated with particular classes, providing a way to train conditional GANs.

4. Experiments

Experiments show that jointly learned measurement functions and meta-learned reconstruction improve recovery efficiency, while latent optimisation strengthens GAN performance across evaluations.

  • 4.1. Deep Compressed Sensing for Reconstruction: DCS significantly outperformed the Bora et al. baseline while using only 3 gradient-descent steps without restarting.The baseline used hundreds or thousands of steps with several restarts.
  • 4.1. Deep Compressed Sensing for Reconstruction: Optimised neural-network measurement functions achieved the best reconstruction performance across all evaluated scenarios.Learned linear projections improved over random projections, and learned neural-network projections improved further.
  • 4.1. Deep Compressed Sensing for Reconstruction: DCS reconstruction errors were 5.3 and 3.4 on MNIST with 10 and 25 measurements, versus 3.8 and 2.5 for Uncertainty Autoencoders.On CelebA, DCS errors were 23.4 and 18.5 with 20 and 50 measurements, versus 27 and 22.
  • 4.2. CS-GANs: CS-GANs achieved better Inception Score and FID, with less variance, than vanilla GANs across 144 hyper-parameter settings.The compared models differed only in the number of latent optimisation steps: 0 versus 3.
  • 4.2. CS-GANs: CS-GANs also improved upon the competitive Spectral-Normalised GAN baseline, with deeper, Batch-Normalised, and Spectral-Normalised architectures improving results further.The comparison used reproduced baseline results alongside the authors’ replication.
  • 4.2. CS-GANs: Semi-supervised CS-GAN experiments on MNIST used 11 measurements and produced samples with a semantically meaningful latent space.Ten measurements represented the MNIST classes and one represented generated samples.

5. Discussion

The discussion presents DCS as a framework that jointly learns measurements, generation, and reconstruction, extending compressed sensing toward flexible neural objectives and generative modeling.

  • 5. Discussion: DCS jointly trains the measurement function, generator, and latent optimisation procedure through meta-learning.The framework combines compressed sensing with deep neural networks while retaining online minimisation of measurement errors.
  • 5. Discussion: The framework derives a family of models, including a GAN model, by varying the properties imposed on measurements.The GAN variant uses discriminator-guided latent optimisation.
  • 5. Discussion: Unlike autoencoder-based combinations, DCS preserves online measurement-error minimisation in generic neural networks.This preserves flexible optimisation of measurements and latent representations for different objectives.
  • 5. Discussion: Learned measurement functions can represent properties difficult or impossible to obtain from random or hand-crafted projections.The authors connect this capability to broader problem classes involving online error minimisation.
  • 5. Discussion: The paper illustrates the framework on relatively simple tasks and anticipates applications including style transfer, MRI, and unsupervised anomaly detection.These applications are presented as potential future beneficiaries rather than evaluated results.

A. Experiments Detail

Experiments use Adam optimization, fixed training settings, and three latent-optimization steps as the default reconstruction configuration.

  • A. Experiments Detail: Models were trained for 4 × 10^5 steps with batch size 64 and 100-dimensional generator latent representations.Latent optimisation used three gradient-descent steps with initial step size 0.01.

A.1. Reconstruction Experiments

Reconstruction experiments use established generators and compare random linear projections with neural-network measurement functions across MNIST and CelebA.

  • A.1. Reconstruction Experiments: MNIST used a two-layer 500-unit-per-layer MLP generator, while CelebA used a DCGAN generator.Both generators followed the setup of Bora et al. for comparison.
  • A.1. Reconstruction Experiments: Measurement functions included random linear projections, MNIST MLPs, and the CelebA DCGAN discriminator.The neural-network measurement functions used two-layer MLPs for MNIST and a DCGAN discriminator for CelebA.

A.2. GAN experiments

The GAN experiments reused established architectures for MNIST, semi-supervised GANs, CIFAR, and Spectral Normalised GANs, while evaluating multiple optimisation settings and repeated runs. Inception Score and Fréchet Inception Distance were averaged across repeated evaluations using 5,000 random samples.

  • MNIST and semi-supervised GAN experiments used the same MLP generator and discriminator architecture.
  • CIFAR experiments used DCGAN with Adam parameters β1 = 0.5 and β2 = 0.9.
  • CIFAR hyper-parameters varied generator and discriminator learning rates, latent variable sizes, and mini-batch sizes, with two replicas per combination.The tested learning rates were {1 × 10−4, 2 × 10−4, 3 × 10−4}, latent sizes were {100, 200}, and mini-batch sizes were {32, 64}.
  • Spectral Normalised GANs used the deeper discriminator from Miyato et al. and selected learning rate 1 × 10−4 with Adam β2 = 0.999 through grid search.The reported settings were those that most stably achieved the best results.
  • Inception Scores and Fréchet Inception Distances averaged 10 evaluations based on 5,000 random samples each.
Loading 1905.06723v2…